Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initial link suggestion work #251

Closed
wants to merge 1 commit into from
Closed

initial link suggestion work #251

wants to merge 1 commit into from

Conversation

jacobryas4
Copy link
Contributor

@jacobryas4 jacobryas4 commented Aug 31, 2023

here is the initial work for the link suggestion feature. a high level overview of whats included:

  • dapp_scraper.py: this file scrapes for IPFS links, then fetches the data from each link via a public IPFS gateway. it then saves the data as JSON in a JSON file.
  • dapps.py: here we create the schema for the weaviate DB
  • config.py: added the new dapps_index here
  • index_widget.py: modified this file to use the new dapps_index - QUESTION: do we still use the scraped_sites_index? if so, i will need to add it back in here in some way

This is initial work but it does seem to be working. some outlying issues to be addressed:

  • dapps doesn't seem to have that much data. Right now, it pulls data for about a dozen dapps. This is enough to prove the concept but probably need more robust data moving forward
  • the frontend only intermittently recognizes/displays the URL as an actual link and when it does, the link opens in the same tab instead of a new one.
  • i need to do some more testing to figure out how good it is. I did some specific testing ("what is {x} protocol?") and then moved to more vague questions ("what is a service i can use to do {pretty specific items in the description of the data}?"). I'm sure there is room for some fine tuning

@jacobryas4 jacobryas4 closed this Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant