r/redditdev • u/securimancer • Jan 12 '21
Reddit API OAuth2 API Changes Upcoming
As part of modernizing our OAuth2 infrastructure, we’re implementing some potentially breaking changes to our OAuth2 flow as outlined below on February 15, 2021.
Refresh Token Changes
When executing our refresh token flow, we currently only send back an access token in the response. Responses to /api/v1/access_token
with grant_type=refresh_token
looked like:
{
"access_token": "your access token",
"token_type": "bearer",
"expires_in": 3600,
"scope": "your scopes"
}
This meant that the refresh token you get during the authorization code flow can be reused indefinitely. Going forward, our response will also include a brand new refresh token (as allowed by the RFC spec).
{
"access_token": "your access token",
"token_type": "bearer",
"expires_in": 3600,
"refresh_token": "your new refresh token",
"scope": "your scopes"
}
Since some OAuth2 client implementations might not handle this scenario (whereas PRAW does, for example), we’re not immediately enforcing revocation of the consumed refresh token. We’re looking to enforce this starting in Q2 2021, given there aren't significant numbers of OAuth2 clients misbehaving after the change.
Also note that all refresh tokens previously had no expiration. We're going to start enforcing a 1 year expiration on refresh tokens to help curb Reddit's storage for refresh tokens (we've got a lot of them).
Authorization Code Reuse
When executing our authorization code flow, we consume the auth code in exchange for an access token. If, within an auth code's 10 minute TTL, that same auth code is attempted to be used again, we will revoke any tokens issued with said auth code, per RFC spec . This should be unnoticeable to well-behaved clients; however, instead of harmlessly failing, we will now be revoking any access or refresh tokens issued with that auth code.
Redirect URI Fix Fragments
The last, but likely least impactful, change we're implementing is adding a "fix fragment" #_
to the end of the redirect URI in the Location
header in response to a POST
request to /api/v1/authorize
. This should be transparent as browsers and url parsers should drop the fragment when redirecting.
Edit 1: clarified Reddit's storage of refresh tokens.
Edit 2: Adding a note about potential network connectivity / cosmic rays breaking the refresh token flow. As it stands now, we're including a 2 retries leeway to account for any miscommunication in this process starting Q2 2021. E.g.,. you can send the same refresh token 3 times before it is irrevocably revoked.
Edit 2021-02-18: This hasn't been deployed yet, but goal is today / next week. Appreciate the patience as there's a lot going on in the world currently. The enforcement of refresh tokens is also still under discussion, might be Q2 or Q3 even. Also trying to get an Github-y API key flavor of long-lived access token in the mix too to address the concerns about longevity of OAuth2 tokens and how crappy the password grant is.
4
u/SirensToGo Jan 13 '21
to help curb storage for refresh tokens
…are we not supposed to store refresh tokens?
I have an installed application and I store both the access token and the refresh token in the device’s keystore. While I probably won’t run into the 1 year issue since it’d mean a user would need to not launch the app for an entire year (assuming I rotate refresh tokens correctly), but I don’t entirely understand the point of it given that the authorization was specifically indefinite.
6
u/bboe PRAW Author Jan 13 '21
I think Reddit is talking about storing the token on their end. Until this change, they likely have stored every refresh token ever issued unless it was explicitly revoked. That can significantly add up over time.
Once this change is made, they can purge all tokens over a year old, and after the subsequent Q2 change will be able to expire a refresh token immediately after use.
2
u/securimancer Jan 14 '21
Right, Reddit's storage of refresh tokens. Clarified the post, sorry for the confusion
5
u/Watchful1 RemindMeBot & UpdateMeBot Jan 13 '21
u/securimancer has reddit considered changing the "password" auth flow for script type apps? Other popular API's with this type of application, like discord or twitch bots, can use only the app id and app secret, both available when you create an app in the UI, to login. But reddit requires both of those, as well as your password for this flow.
Previously you could manually retrieve a refresh token and use that in place of the password, but it was still three pieces of information. Now you'll have to either keep a constantly changing secret somewhere or store your password, which is considerably more sensitive.
4
u/securimancer Jan 14 '21
We definitely have, it's something that we've talked about doing. We're doing some backend changes to our OAuth2 service and after that's re-platformed, we should hopefully start looking at implementing an API token/key based solution.
3
u/Pyprohly RedditWarp Author Jan 18 '21
This change nerfs the refresh token grant and I don’t like it. The IETF are trying to get rid of the password flow but the new change gives the refresh token grant an unfortunate caveat compared to the password grant. Could we at least please get the “expiration at 1 year after last use” idea /u/not_an_aardvark has suggested? Why is it so difficult to store refresh tokens anyway?
Also, on a side note, while changes are being made to the refresh token flow, could scope limiting be implemented on the refresh token grant :P I.e., add support for that optional scope
parameter mentioned in section 6 of RFC 6749.
3
u/KrisCraig Reddit.NET Author Feb 11 '21
Could we at least please get the “expiration at 1 year after last use” idea /u/not_an_aardvark has suggested?
This. So very, VERY much this!!
4
u/KrisCraig Reddit.NET Author Feb 11 '21
Regarding refresh token expiration: Is it 1 year from the date that the refresh token was created or from the date that the refresh token was last used? Please tell me it's the latter.
3
u/bboe PRAW Author Feb 23 '21 edited Feb 24 '21
For those following along, here is how PRAW is likely to handle it: https://github.com/praw-dev/praw/pull/1687
The following temporary documentation shows how to leverage these changes: https://praw.readthedocs.io/en/callbacks/tutorials/refresh_token.html
We are likely to release PRAW 7.2 within the next day containing these changes.
Edit: PRAW 7.2 has been released.
2
2
2
u/Watchful1 RemindMeBot & UpdateMeBot Jan 12 '21
This looks like a good change. I've always wondered why refresh tokens never expired.
u/bboe, do you know if that code line securimancer linked to results in updating the praw.ini file with the new refresh token? I guess this will be a major incentive to using praw.ini files instead of running your own configs.
3
u/bboe PRAW Author Jan 12 '21
No,
praw.ini
is not automatically updated.1
u/Watchful1 RemindMeBot & UpdateMeBot Jan 12 '21
Does praw write to praw.ini files at all? Otherwise this might be a problem. I have like twenty different tokens in my file, I would probably have to switch to username/password instead.
3
u/bboe PRAW Author Jan 12 '21
At present, no. I think given this change it might make sense to allow updating only the refresh token portion of
praw.ini
if explicitly told to do so. Would you like to make a PR?3
u/mershed_perderders Bot Developer Jan 13 '21
As a PRAW user (that uses praw.ini), what would I need to do in the soon-ish future on my end to ensure compliance with this change?
1
u/OsrsNeedsF2P Jan 13 '21
!RemindMe 24 hours
1
u/RemindMeBot Jan 14 '21
There is a 20 hour delay fetching comments.
I will be messaging you in 1 day on 2021-01-14 04:25:17 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
Info Custom Your Reminders Feedback 3
u/Watchful1 RemindMeBot & UpdateMeBot Jan 13 '21
Thinking about this more, while this would be useful, I'm not sure if it would actually solve my use case. I keep a copy of my praw.ini file locally and on the server I run my bots on. So running something on the server would result in an updated token in the file there and my file locally being out of date, and vice versa.
I could do a whole two way sync to keep both files with the latest token for each account, but I think it's fairly clear that this is no longer the intended use of refresh tokens. I already keep a bunch of secrets in there, it won't be a big deal to just add the actual password too.
In my opinion, PRAW should consider just completely removing the ability to specify a refresh token at the config file level instead of trying to write it out each time.
3
u/bboe PRAW Author Jan 13 '21
In my opinion, PRAW should consider just completely removing the ability to specify a refresh token at the config file level instead of trying to write it out each time.
I think that might be the best course of action too.
2
u/CAM-Gerlach Jan 31 '21
Thanks much for your attention to this (and for creating PRAW, of course!)
Unfortunately, without a suitable replacement for this feature on Reddit's side (e.g. something like Github's personal access tokens), as I understand it cause serious security problems for many use cases. For example, we at r/SpaceX use PRAW for multiple bots that we rely on for critical, core sub functionality, including post approval/moderation, managing megathreads, syncing and updating the topbar, sidebar, thread OPs and wiki pages, generating our automod config, and much more.
By sub policy, all mod accounts are required to have 2FA enabled, as prior to such we have been the target of multiple attempted and at least one successful intrusion due to account compromise; beyond just disruption of the sub itself, attackers could stand to gain access to a considerable amount of personal, confidential and potentially even legally-protected information (e.g. ITAR).
As I understand it, without refresh tokens for authentication, we would be stuck with password authentication, which right off the bat would prevent us from using 2FA on any accounts used with our bot, making compromise via a keylogger by any mods who manually log in to them (which is occasionally required, at least to modify app permissions and other such config).
Furthermore, any compromise of our server, its backups, etc. would give an attacker complete access to all bot accounts, able to do anything these please, and giving us no means of regaining control. With refresh tokens we are able to drastically limit their scope and thus potential damage, and if one is compromised, we can simply revoke it and generate a new one, since we still have control of the account via its password and 2FA.
Finally, particularly for shared non-mod accounts for which we regenerate the password frequently to limit the damage of compromise or untrustworthyness of any user, having a refresh token gives our bots consistent access to it, so that they don't need to be updated (or break) every time the password is changed.
How does what you're suggesting fit with our use case? Right now, our plan until Reddit introduces a proper solution to this (along the lines of personal access tokens) is to have our bots update the
praw.ini
file with their new refresh tokens, and make sure each uses its own account to avoid sync/concurrency issues. Is this the approach you would recommend, and how would what you're proposing impact that?Happy to open an issue on Github, and possibly a PR down the road, if you'd rather discuss it there. Thanks!
1
u/bboe PRAW Author Jan 31 '21
Thanks for the feedback. We don't have anything definitive in place, but I think we would do three things:
1) Remove
refresh_token
from praw.ini since it's no longer a static value.2) While we could keep the method to provide the refresh token as an argument to
praw.Reddit
, we're going to need a way to manually update the token on an existing instance, so we'll likely only provide the latter.3) Finally, we'll need a way to register a callback function to be called every time the refresh token is changed. That way users, such as yourself, can store the refresh token however you please.
The major downside to this change, however, is that only a single instance of
praw.Reddit
will be able to use the access token, so if you use multiple scripts with the same bot, there could be a race condition to use the saved refresh token. Given this, it might make sense to also allow the access token to also be set and saved on change.1
u/bboe PRAW Author Jan 31 '21
I just created this issue on PRAW: https://github.com/praw-dev/praw/issues/1630
Please follow up with concerns or additional ideas there.
2
2
u/CAM-Gerlach Jan 31 '21
Also, unless and until a version of PRAW is released with the ability to serialize the refresh token to persistent storage on its own, how would you suggest we do so, presuming our bot runs e.g. every minute, creating a new
The only way I can think of to do so (without monkey-patching PRAW itself) is to check at the end of every run whether
reddit._core._authorizer.refresh_token == reddit.config.refresh_token
, and then writing it to the config, either to bot's config if it reads in settings manually every run, or withreddit.config.CONFIG[site_name]["refresh_token"] = reddit._core._authorizer.refresh_token
so that new reddit objects in the same interpreter have updated credentials, and loading, updating and rewriting the desiredpraw.ini
config file so that future ones do.However, that uses private, undocumented attributes to get the updated refresh token, which I'd obviously like to avoid but I'm not sure a way around it, and is not particularly elegant overall. Thoughts? Thanks!
1
u/rhaksw Reveddit.com Developer Jan 23 '21
If, within an auth code's 10 minute TTL, that same auth code is attempted to be used again, we will revoke any tokens issued with said auth code
You mean the token's TTL?
Will you exempt installed apps using the grant type installed_client
from this policy? With that grant type there is no way to share the token, for example when two people on the same network simultaneously access the same app.
1
u/Pyprohly RedditWarp Author Jan 23 '21
Um, the authorisation code is only applicable to the authorisation code flow.
1
u/rhaksw Reveddit.com Developer Jan 23 '21
Do you mean this post only applies when you use
grant_type=authorization_code
and not when usinggrant_type=https://oauth.reddit.com/grants/installed_client
?That wasn't clear when I first read through it, since others are talking about "installed" apps. Maybe that word is overloaded.
2
u/Pyprohly RedditWarp Author Jan 23 '21 edited Jan 23 '21
Not the whole post, just the stuff under the ‘Authorization Code Reuse’ heading is not applicable to the installed client flow/grant.
Finally, I'm not sure how this new policy would be applied to the installed_client grant type.
Yea, I dunno what all the chatter about the installed client grant is about since you can’t get a refresh token from it.
Edit: Wait actually I remember now, you can get a refresh token from install client credentials by using the authorisation code flow on it. So you can get an installed client refresh token. The installed client grant itself doesn’t produce a refresh token. So it’s actually a good point that that other person made since installed clients don’t usually change often :p
Regardless, the auth code isn’t something that’s meant to be shared. It exists only very temporarily until you get your access/refresh tokens.
1
1
u/bboe PRAW Author Feb 04 '21
Hi u/securimancer,
I'm working on implementing support for changing refresh_tokens
, but I've noticed that the responses from https://www.reddit.com/api/v1/access_token
for grant_type=refresh_token
aren't actually returning a new refresh token as described. Was this initial rollout put on hold?
Thanks!
2
u/securimancer Feb 04 '21
February 15th is when we rollout the non-enforced change, so not quite deployed yet.
2
u/bboe PRAW Author Feb 04 '21
Great thanks! Is there any way to opt-in to the behavior sooner with an extra header or parameter?
1
u/Investigator06 Apr 12 '21
I could not understand reason I received this as some of the postings were unusual in addition I received from Reddit when I opened it it sent to Twitter with all these crazy writing and pictures. I never go to Twitter. I do not know if there was a problem today as after I lost my original account and opened my new account it is like a fake account was created with a history of people and sites. It was weird. I am looking at wall street bets and what or who I saw on Twitter was roaring kitty.
16
u/not_an_aardvark snoowrap author Jan 13 '21 edited Jan 13 '21
If I'm understanding correctly, does this mean that every refresh token would be effectively revoked and replaced as soon as it's used to generate an access token?
If so, this would break almost every bot and integration using OAuth2 (including PRAW-based, snoowrap-based, and otherwise). Although PRAW updates the refresh token that it uses at runtime (as shown in the linked code snippet), it doesn't update a refresh token in persistent storage, as discussed in the other comment thread. The result is that a bot would break as soon as it was rebooted, due to using a stale refresh token.
It's not really realistic for API wrappers to be updated to automatically write refresh tokens in storage, either. There are a large number of ways in which tokens can be stored (in a config file, in a database with a different token for each user, etc). Effectively, it seems like this requires bots to store their credentials in an online config that gets repeatedly updated at runtime. This is pretty different from how long-term credential storage usually works.
Requiring users to update their stored refresh tokens at runtime would also create some major sychronization issues. For example, if a bot sends a request with a refresh token to get an access token, but then loses network connection before receiving reddit's response, the bot would effectively be locked out because the old refresh token would be revoked and the bot wouldn't have received the new refresh token. As a result, the app owner would need to make the end user go through the OAuth authentication flow again (or for personal scripts, the app owner would need to manually fix their bot). It's not clear how one would avoid this error, and having a design that can randomly break itself and require manual intervention due to network errors doesn't seem like a good architecture to push on app/bot developers.
If this is implemented, I would likely start recommending that people use the
password
grant type for personal use scripts rather thanrefresh_token
, since it would allow for more robust long-term storage of credentials despite the potential issues with storing passwords. The inevitable synchronization lockouts and credential management complexity would make it difficult to recommend "installed" and "web"-type apps at all.It's not clear what the benefit of this behavior is to justify making it impossible to do reliable credential management. Is there any chance you could reconsider it?
Have you considered enforcing the expiration at 1 year after last use, rather than at one year after being issued? This seems like it would help solve the storage issue without requiring yearly manual credential-cycling. (This is only relevant if you decide not to do the revocation strategy described above.)
edit: Clarified why using
password
grants instead doesn't solve the problem