Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade the TagComponent to the KnowledgeComponent #7070

Merged
merged 45 commits into from
Oct 19, 2022
Merged

Upgrade the TagComponent to the KnowledgeComponent #7070

merged 45 commits into from
Oct 19, 2022

Conversation

drew2a
Copy link
Contributor

@drew2a drew2a commented Sep 27, 2022 •
edited
Loading

This is a draft PR that upgrades the TagComponent to the KnowledgeGraphComponent with adding a Linux snippet.
Related to #6214

TODO:

  • Apply a more generic DB structure for the KG
  • Update the networking part of TagCommunity
  • Write a migration
  • Fix tags case sensitivity
  • Rename TagCommunity to KnowledgeCommunity
  • Add a type field for Resource
  • Created a basic validation experiment to check correctness of the knowledge gossip logic
  • Update the REST endpoints to process statements instead of tags

Migration:

  1. Remove all received entities
  2. Regenerate locally created entities

Questions:

  1. How to store the tags database? If we store the Tags DB locally, we will have quick responses for local and remote search queries. The drawback is that this DB will use a lot of disk space (more than 5Gb with infinitive growth). If we store the Tags DB distributively then the usage of disk space will be minimal, but metadata retrieval will become a more complex and time-consuming task.
  2. Which mechanism will we use for knowledge retrieval? Just random gossip (like popularity community)?
  3. How quickly knowledge spreads across the network?

Ubuntu (1 snippet)
image

Ubuntu (3 snippets)
image

Debian and Linux Mint (10 snippets)

image image

Ref:

@drew2a drew2a changed the title WIP: DB scheme for the 'relation' field (Tags) WIP: Add DB scheme for the relation field (Tags) Sep 27, 2022
@drew2a drew2a force-pushed the feature/add_content_items branch 2 times, most recently from c8ac9f4 to 3aeb3c1 Compare September 27, 2022 12:41
@drew2a drew2a force-pushed the feature/add_content_items branch 3 times, most recently from a67b90f to c5d73fa Compare October 3, 2022 11:48
@drew2a drew2a changed the title WIP: Add DB scheme for the relation field (Tags) WIP: Add snippet for Ubuntu Oct 3, 2022
@drew2a drew2a added this to the 7.13.0 milestone Oct 17, 2022
@drew2a drew2a marked this pull request as ready for review October 17, 2022 15:44
@drew2a drew2a requested a review from a team October 17, 2022 15:44
@drew2a drew2a changed the title WIP: Upgrade the TagComponent to the KnowledgeComponent Upgrade the TagComponent to the KnowledgeComponent Oct 17, 2022
@devos50
Copy link
Contributor

devos50 commented Oct 18, 2022

It seems that the Windows tests are failing on tests related to tags: https://github.com/Tribler/tribler/actions/runs/3266579335/jobs/5370515178. Is this expected?

@drew2a drew2a marked this pull request as draft October 18, 2022 08:56
@drew2a
Copy link
Contributor Author

drew2a commented Oct 18, 2022

It seems that the Windows tests are failing on tests related to tags: https://github.com/Tribler/tribler/actions/runs/3266579335/jobs/5370515178. Is this expected?

@devos50 it looks like a bug

@drew2a drew2a marked this pull request as ready for review October 19, 2022 12:52
@drew2a drew2a requested a review from devos50 October 19, 2022 13:10
@drew2a
Copy link
Contributor Author

drew2a commented Oct 19, 2022 •
edited
Loading

@devos50 the Windows tests are fixed!

@drew2a drew2a merged commit 499d559 into Tribler:main Oct 19, 2022
kozlovsky
kozlovsky reviewed Oct 19, 2022
View reviewed changes
Copy link
Contributor

@kozlovsky kozlovsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me the PR looks good

@synctext synctext mentioned this pull request Nov 9, 2022
Closed
@synctext
Copy link
Member

synctext commented Nov 9, 2022

With this pull request we put Tribler on the path of P2P-style "semantic search".

@drew2a drew2a mentioned this pull request Apr 20, 2023
Merged
@drew2a drew2a mentioned this pull request May 1, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kozlovsky kozlovsky kozlovsky left review comments

@devos50 devos50 devos50 approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
7.13.0
Development

Successfully merging this pull request may close these issues.
4 participants
@drew2a @devos50 @synctext @kozlovsky