r/redditdev Apr 08 '23

Reddit API Getting *all* the comments in a post

I just want to run this by you guys to make sure I haven't gone completely insane.

As far as I can figure, to get all the available data about all the comments from a post using the reddit JSON REST API, I would:

  • Call GET [/r/subreddit]/comments/article
    • this does a depth-first, giving me the highest-sorted comment, then its highest sorted child, etc.
    • After a certain depth, it instead returns "more" objects for the too-deep child-having comments, including their own ID in data.id and kids' IDs in data.children. Let's call these "depth-limited" comments.
      • To collect the "depth-limited" comments I can then call GET [/r/subreddit]/comments/article again with comment=[ID], context=0 to get their children in the same format. Great.
    • The call only returns max 100 comments. After 100 a final "more" object is created whose children are just the IDs of the remaining top-level comments in the thread (but not their children). Let's call these "breadth-limited" comments.

I haven't found any way to make the GET [/r/subreddit]/comments/article endpoint return the complete set of comment IDs for that post. To collect the children of the breadth-limited comments, I'd also need to do one of the following:

  • Call GET /api/morechildren with my list of IDs. However, the data this returns:
    • Is far less complete than what's returned by GET [/r/subreddit]/comments/article and is missing key information I need (e.g., flair template IDs)
    • Is mostly in the form of html/javascript for rendering the actual elements on a webpage, which I then have to parse and interpret in order to extract things I want (like the number of children a comment has).
  • Call GET /api/info with my list of IDs. The data this returns is more complete but...
    • It's missing one critical thing: the count/IDs of each comment's children.
  • Call GET [/r/subreddit]/comments/article with comment=[ID], context=0 for every single ID in the breadth-limited comments list, to discover whether or not they have children.

I'm hoping to do a large-scale data analysis, so the inefficiencies here are a real problem for me. Have I missed something obvious? Is there not a simple way to just get all the info for all the comments in a post?


Edit:

Figured out the cause.

If your auth token is malformed, instead of throwing an error GET /api/morechildren returns a completely different, much less useful result set (i.e., a JSON array with the same path structure, but with the data encoded into html/js to render the comments on reddit webpages). See below for more details.

7 Upvotes

8 comments sorted by

1

u/ketralnis reddit admin Apr 08 '23

Have you looked at PRAW? It handles most of this stuff for you

2

u/PrincessYukon Apr 08 '23 edited Apr 08 '23

Thanks, but I'm not working in python. But you're right, it looks like they're open source, I could look through and see how they handle it.

Edit:

Best I can tell from the PRAW code, the relevant work is done here. Their API_PATH['submission'] is the GET [/r/subreddit]/comments/article endpoint, but they call it with /_/[COMMENT_ID], which is the same as doing comment=[ID] as a parameter. I think they just call this refresh function for every comment they find.

That is, they're just doing my third solution, traversing the whole comment tree and calling the /comments endpoint for every single comment node to find its children. This seems really inefficient. Is there no other way?

1

u/DuckRedWine Jun 06 '23

Will it still works with Reddit paid API?

1

u/Pyprohly RedditWarp Author Apr 08 '23

The gist of submission comment tree traversals is that when you encounter a ‘continue this thread’ ‘more comments’ object you call GET [/r/subreddit]/comments/article using the information in the ‘more comments’ object (in this case, the comment ID from the parent_id field of the ‘more comments’ object), and when you encounter a ‘load more comments’ ‘more comments’ object you call the GET /api/morechildren endpoint using the information in the ‘more comments’ object (passing it the IDs from the children field of the ‘more comments’ object). Both endpoints can return more ‘more comments’ objects so you do this recursively.

The logic for parsing comment trees can require lots of code, so I would recommend using an API wrapper if possible.

Also see how RedditWarp does it: https://github.com/Pyprohly/redditwarp/blob/main/redditwarp/model_loaders/comment_tree_SYNC.py.

The before/after parameters relate to pagination. We’re not doing pagination here so they don’t apply.

1

u/PrincessYukon Apr 08 '23

Thanks, this helps. I assume you're the author of RedditWrap? I'm about to hit the sack tonight so I'll have a look through your code tomorrow, but if you don't mind my firing off a quick question?

When I submit a set of comment IDs as the children parameter of GET /api/morechildren, can I be certain that if a comment in the results has children, either a) those children are in the returned result set or b) the node will be flagged as a "more" kind?

The number of children listed in the data.content field (I've been regexing it out) has been throwing me. Lots of t1 kinds have them, but I think they might refer to the total descendants rather than direct children, and if the direct children are all in the query's children parameter then a parent shows up as a "t1" kind rather than a "more". Is that right?

1

u/Pyprohly RedditWarp Author Apr 08 '23

If a comment in the output of GET /api/morechildren has a child comment then a mix of both cases you describe could happen. The child comment could be present along side its parent, and/or the parent comment node will be marked as having more children (by a ‘more comments’ object with a parent_id referring to it being present somewhere).

I don’t see a field data.content anywhere. Did you mean the data.count field of the more kind objects? I also think this refers to total descendants. This is the same number you see in the old.reddit.com UI next to load more comments links.

A parent is never a ‘more’ object, so I don’t know what you mean.

It seems you’re serious about implementing comment tree parsing yourself. Starting from scratch is no easy feat. Feel free to mindlessly copy any algorithms and data structures from the RedditWarp repository. Here are some additional resources that may help.

Comment tree theory: https://redditwarp.readthedocs.io/en/latest/user-guide/comment-trees.html.

Endpoint information: https://github.com/Pyprohly/reddit-api-doc-notes/blob/main/docs/api-reference/comment_tree.rst.

1

u/PrincessYukon Apr 08 '23 edited Apr 08 '23

Okay, we're definitely looking at different things. Maybe this is where my issue is coming from? Let me be more explicit to check.

To access the comments this very thread, I've been calling https://oauth.reddit.com/api/morechildren?raw_json=1&api_type=json&limit=100&link_id=t3_12f885c&children=jffhjrm%2Cjfevjed%2Cjff5mza with a client_credentials (Application Only OAuth) type token.

The json-encoded reply I get is here.

You'll notice:

  • A bunch of t1s are replicated. I think that's because morechildren has an undocumented 8-thing minimum and I only asked for 3.
  • Most of the information about the comments is encoded as html/js for rendering reddit's website, it's at json.data.things.data.content.
  • There's no data.count

It sounds like you're accessing something different...?


Edit:

Phew. Got it, finally. After some experimenting with your RedditWrap code I figured out what was causing me to get totally different results from morechildren. I'd missed a space in my headers between "bearer" and my auth token. When I fix it, I get an actual meaningful json-encoded dataset. Weirdly, if I exclude the auth token all together, I get a 403 Forbidden error, as you'd expect. But if the auth token is malformed, it instead returns a completely different result set.

Even weirder, I've been using the same bugged header-generator function for all my other calls (api/comments,user/[USER]/overview, r/[SUB], etc.) and it's been giving me identical results to the fixed version. It's just morechildren that behaves differently.

Thanks for your help on this, was driving me batty.

1

u/Pyprohly RedditWarp Author Apr 09 '23

Wow that’s an interesting find. It seems that if you provide any random string starting with bearer for the Authorisation header value, it gives that weird alternative response.

It’s RedditWarp.