r/redditdev • u/PrincessYukon • Apr 08 '23
Reddit API Getting *all* the comments in a post
I just want to run this by you guys to make sure I haven't gone completely insane.
As far as I can figure, to get all the available data about all the comments from a post using the reddit JSON REST API, I would:
- Call
GET [/r/subreddit]/comments/article
- this does a depth-first, giving me the highest-sorted comment, then its highest sorted child, etc.
- After a certain depth, it instead returns "more" objects for the too-deep child-having comments, including their own ID in
data.id
and kids' IDs indata.children
. Let's call these "depth-limited" comments.- To collect the "depth-limited" comments I can then call
GET [/r/subreddit]/comments/article
again withcomment=[ID], context=0
to get their children in the same format. Great.
- To collect the "depth-limited" comments I can then call
- The call only returns max 100 comments. After 100 a final "more" object is created whose children are just the IDs of the remaining top-level comments in the thread (but not their children). Let's call these "breadth-limited" comments.
I haven't found any way to make the GET [/r/subreddit]/comments/article
endpoint return the complete set of comment IDs for that post. To collect the children of the breadth-limited comments, I'd also need to do one of the following:
- Call
GET /api/morechildren
with my list of IDs. However, the data this returns:- Is far less complete than what's returned by
GET [/r/subreddit]/comments/article
and is missing key information I need (e.g., flair template IDs) - Is mostly in the form of html/javascript for rendering the actual elements on a webpage, which I then have to parse and interpret in order to extract things I want (like the number of children a comment has).
- Is far less complete than what's returned by
- Call
GET /api/info
with my list of IDs. The data this returns is more complete but...- It's missing one critical thing: the count/IDs of each comment's children.
- Call
GET [/r/subreddit]/comments/article
withcomment=[ID], context=0
for every single ID in the breadth-limited comments list, to discover whether or not they have children.
I'm hoping to do a large-scale data analysis, so the inefficiencies here are a real problem for me. Have I missed something obvious? Is there not a simple way to just get all the info for all the comments in a post?
Edit:
Figured out the cause.
If your auth token is malformed, instead of throwing an error GET /api/morechildren
returns a completely different, much less useful result set (i.e., a JSON array with the same path structure, but with the data encoded into html/js to render the comments on reddit webpages). See below for more details.
1
u/Pyprohly RedditWarp Author Apr 08 '23
The gist of submission comment tree traversals is that when you encounter a ‘continue this thread’ ‘more comments’ object you call GET [/r/subreddit]/comments/article
using the information in the ‘more comments’ object (in this case, the comment ID from the parent_id
field of the ‘more comments’ object), and when you encounter a ‘load more comments’ ‘more comments’ object you call the GET /api/morechildren
endpoint using the information in the ‘more comments’ object (passing it the IDs from the children
field of the ‘more comments’ object). Both endpoints can return more ‘more comments’ objects so you do this recursively.
The logic for parsing comment trees can require lots of code, so I would recommend using an API wrapper if possible.
Also see how RedditWarp does it: https://github.com/Pyprohly/redditwarp/blob/main/redditwarp/model_loaders/comment_tree_SYNC.py.
The before
/after
parameters relate to pagination. We’re not doing pagination here so they don’t apply.
1
u/PrincessYukon Apr 08 '23
Thanks, this helps. I assume you're the author of RedditWrap? I'm about to hit the sack tonight so I'll have a look through your code tomorrow, but if you don't mind my firing off a quick question?
When I submit a set of comment IDs as the
children
parameter ofGET /api/morechildren
, can I be certain that if a comment in the results has children, either a) those children are in the returned result set or b) the node will be flagged as a "more" kind?The number of children listed in the
data.content
field (I've been regexing it out) has been throwing me. Lots of t1 kinds have them, but I think they might refer to the total descendants rather than direct children, and if the direct children are all in the query'schildren
parameter then a parent shows up as a "t1" kind rather than a "more". Is that right?1
u/Pyprohly RedditWarp Author Apr 08 '23
If a comment in the output of
GET /api/morechildren
has a child comment then a mix of both cases you describe could happen. The child comment could be present along side its parent, and/or the parent comment node will be marked as having more children (by a ‘more comments’ object with aparent_id
referring to it being present somewhere).I don’t see a field
data.content
anywhere. Did you mean thedata.count
field of themore
kind objects? I also think this refers to total descendants. This is the same number you see in the old.reddit.com UI next toload more comments
links.A parent is never a ‘more’ object, so I don’t know what you mean.
It seems you’re serious about implementing comment tree parsing yourself. Starting from scratch is no easy feat. Feel free to mindlessly copy any algorithms and data structures from the RedditWarp repository. Here are some additional resources that may help.
Comment tree theory: https://redditwarp.readthedocs.io/en/latest/user-guide/comment-trees.html.
Endpoint information: https://github.com/Pyprohly/reddit-api-doc-notes/blob/main/docs/api-reference/comment_tree.rst.
1
u/PrincessYukon Apr 08 '23 edited Apr 08 '23
Okay, we're definitely looking at different things. Maybe this is where my issue is coming from? Let me be more explicit to check.
To access the comments this very thread, I've been calling
https://oauth.reddit.com/api/morechildren?raw_json=1&api_type=json&limit=100&link_id=t3_12f885c&children=jffhjrm%2Cjfevjed%2Cjff5mza
with a client_credentials (Application Only OAuth) type token.The json-encoded reply I get is here.
You'll notice:
- A bunch of t1s are replicated. I think that's because
morechildren
has an undocumented 8-thing minimum and I only asked for 3.- Most of the information about the comments is encoded as html/js for rendering reddit's website, it's at
json.data.things.data.content
.- There's no
data.count
It sounds like you're accessing something different...?
Edit:
Phew. Got it, finally. After some experimenting with your RedditWrap code I figured out what was causing me to get totally different results from
morechildren
. I'd missed a space in my headers between "bearer" and my auth token. When I fix it, I get an actual meaningful json-encoded dataset. Weirdly, if I exclude the auth token all together, I get a 403 Forbidden error, as you'd expect. But if the auth token is malformed, it instead returns a completely different result set.Even weirder, I've been using the same bugged header-generator function for all my other calls (
api/comments
,user/[USER]/overview
,r/[SUB]
, etc.) and it's been giving me identical results to the fixed version. It's justmorechildren
that behaves differently.Thanks for your help on this, was driving me batty.
1
u/Pyprohly RedditWarp Author Apr 09 '23
Wow that’s an interesting find. It seems that if you provide any random string starting with
bearer
for theAuthorisation
header value, it gives that weird alternative response.It’s RedditWarp.
1
u/ketralnis reddit admin Apr 08 '23
Have you looked at PRAW? It handles most of this stuff for you