Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add edge-case handling when retriving prolific user history #145

Open
3 tasks
FoxxMD opened this issue May 18, 2023 · 0 comments
Open
3 tasks

Add edge-case handling when retriving prolific user history #145

FoxxMD opened this issue May 18, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@FoxxMD
Copy link
Owner

FoxxMD commented May 18, 2023

Due to limits in reddit's infra basically every search vector is limited to returning 1000 items.

The consequence of this is that when CM retrieves a user's profile history only the last 1000 submissions/comments/overview items can possibly be retrieved.

In the event the user is prolific and the fetch window is time-based it is possible we run out of items before the time window is reached. Additionally, reddit's listing response doesn't differentiate between max exhaustion and actual "no more items".

Document limitation

  • This should be added to the docs so operators/moderators are aware of the current limitation

Detect and handle max exhaustion cases

  • We should detect that 1000 is not inclusive when the window is time-based
  • Add config option to window to fail if exhausted or make do with given data?

Handle single subreddit source use-case with different search

If the window is filtering to a specific subreddit we can use a different search vector to (probably) get more results. This is because user profile search requires us to get all activities and then filter to subreddit -- as opposed to searching a subreddit only returns activities from that sub to begin with...

If max exhaustion occurs in this scenario we could fallback to getting a listing with this query:

https://www.reddit.com/r/SUBREDDIT/search/?q=author%3AUSERNAME&restrict_sr=1&sr_nsfw=&type=comment
@FoxxMD FoxxMD added the bug Something isn't working label May 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant