Skip to content

Conversation

@terencedignon
Copy link

This adds community endpoints (community_info, community_members, community_moderators, community_tweets + raw variants) and CLI flags.

I had to tweak the parser so community member responses with missing fields still parse cleanly.

To test, run examples/community_scraping.py


Example trimmed response:

{
    "id": 1501272736215322629,
    "id_str": "1501272736215322629",
    "name": "New York Mets",
    "description": "Your baseball source for all Mets news, updates, in-game highlights and content. For Mets fans by Mets fans #LGM",
    "memberCount": 27105,
    "moderatorCount": 0,
    "rules": [
      {"id_str": "1501272736383090692", "name": "Keep Tweets on topic.", "description": ""},
      {"id_str": "1521944352750280704", "name": "No Trolls", "description": ""},
      {"id_str": "1877106707501035784", "name": "No Politics", "description": ""},
      {"id_str": "1939637317636858332", "name": "No Hate Speech/Slurs", "description": "Ridiculous that this needed to be added. You will be banned"}
    ],
    "topicId": null,
    "topicName": "Baseball",
    "isNsfw": false
  }

And a typical member item from community_members:

  {
    "community_role": "Member",
    "rest_id": "1118575978169737217",
    "core": {"screen_name": "GottaBelievePod", "name": "We Gotta Believe"},
    "legacy": {...minimal legacy fields...}
  }

Closes #273.

@terencedignon
Copy link
Author

gentle ping on this. Again, no worries if you think this is out of scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: adding support for scraping communities

1 participant