file_content_explanations.md

All data is retrieved through the official reddit API. Content from private or quarantined subreddits is not included. One JSON object represents one post or comment. The objects are modified slightly, as described below. Some modifications are only introduced in later dumps. This does not apply to the previous pushshift dumps (2023-03 and earlier).

Auto generated JSON schemas are available here.

2023-04+

• All objects are sorted by created_utc, id (secondary) in ascending order
• Text is encoded as UTF-8
• JSON keys are sorted
• The body_html field is removed
• retrieved_on is added, to indicate when the object was retrieved (UTC timestamp in seconds)

2023-11+

All content has been retrieved a second time, with a 36 hour delay. The original and the new content are merged, in the following way:

• The following fields are updated to the new value:
  archived, controversiality, downs, hide_score, locked,
  num_comments, num_crossposts, over_18, score, score_hidden,
  spoiler, stickied, ups, upvote_ratio
• Any new meta data is stored in the optional _meta field
• The timestamp of the 2nd retrieval is stored in _meta.retrieved_2nd_on
• If a thing has been deleted in the updated version, _meta.was_deleted_later is set to true
• If a thing was initially deleted, but is now available, _meta.was_initially_deleted is set to true and the body or selftext field is restored
• In both of the above cases, _meta.removal_type contains more information about the removal reason
• If the text content of a thing has changed, _meta.is_edited is set to true
• If a thing was initially unavailable, but now is, _meta.note is set to "initially_unavailable"

2024-04+

Archives will now only be released as .zst files, now that I'm changing by database and API architecture, and am no longer using .zst_blocks files.