Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Add canonical portal for duplicate ontologies #770

Merged
Merged
Show file tree
Hide file tree
Changes from 70 commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
6e87b96
simplify the header component to use content instead of a new section
syphax-bouazzouni May 8, 2024
f3870cd
update dropdown component to a custom title section instead of text
syphax-bouazzouni May 8, 2024
b21d74b
update browse page to use Dropdown component not bootsrap one
syphax-bouazzouni May 8, 2024
a4759c8
add color option to square badge component
syphax-bouazzouni May 8, 2024
2733312
add text and bg colors options to ontology browse card component
syphax-bouazzouni May 9, 2024
cc6c288
add federation helper code and config sample
syphax-bouazzouni May 9, 2024
8a8bb8e
add portals filters in the browse page
syphax-bouazzouni May 9, 2024
382582b
use federation helpers to get federation ontologies information
syphax-bouazzouni May 9, 2024
6ab2e86
remove the filter using index code from submission filter as no used
syphax-bouazzouni May 9, 2024
5f8de42
refactor the filter_using_data function to be faster by using an hash
syphax-bouazzouni May 9, 2024
4dd2740
update the browse analytics cache to change depending on portals
syphax-bouazzouni May 9, 2024
692c36e
add categories and groups ids on hover to know its origin when federated
syphax-bouazzouni May 9, 2024
2910b13
use the last part of ids for browse counts independently of its origin
syphax-bouazzouni May 9, 2024
cb26cb2
remove binding.pry from final federation code
syphax-bouazzouni May 10, 2024
7879468
add an error message if one of the external portal is down
syphax-bouazzouni May 10, 2024
8d8b948
Merge branch 'development' into feature/federate-search-page
Bilelkihal Aug 29, 2024
f9f4810
display federated results in search page
Bilelkihal Aug 29, 2024
e403973
display federated search results in different colors
Bilelkihal Aug 30, 2024
0ad89fa
add portals param in federated search
Bilelkihal Sep 6, 2024
5db6602
add federation checks in search page
Bilelkihal Sep 6, 2024
e68d651
merge federated search results
Bilelkihal Sep 9, 2024
065570b
show portals names in federated search result chips in the form 'Agro…
Bilelkihal Sep 9, 2024
2a33a0d
sort federated search results by string similarity
Bilelkihal Sep 9, 2024
102b0a2
Merge branch 'development' into feature/federate-search-page
syphax-bouazzouni Sep 11, 2024
62e1cc0
fix merge development to federated search issues
Bilelkihal Sep 12, 2024
07f0e44
clean search result component
Bilelkihal Sep 12, 2024
2e5dd7b
internationalize: results from other portals in search and browse page
Bilelkihal Sep 12, 2024
d1b228a
fix performance issue in federated search
Bilelkihal Sep 12, 2024
bb96959
merge results using class id and ontology acronym
Bilelkihal Sep 13, 2024
edb6937
extract federation enabled into a helper in federated search
Bilelkihal Sep 13, 2024
d986607
refactor search result elem function
Bilelkihal Sep 13, 2024
d285722
add realtime benchmark for search federated search
Bilelkihal Sep 13, 2024
4d0f827
add portal names and colors in top of federated search results
Bilelkihal Sep 13, 2024
6f71685
make sort by string similarity not case sensitive
Bilelkihal Sep 13, 2024
37d16a8
Merge branch 'development' into feature/federate-search-page
syphax-bouazzouni Sep 19, 2024
fc8ec7e
remove duplicated federation configuration
Bilelkihal Sep 20, 2024
f7a4e47
extract portal button in home page to federation helper, fix the styl…
Bilelkihal Sep 20, 2024
f8742b6
fix icons colors in federation
Bilelkihal Sep 23, 2024
52e9b27
display a message in the federated search results when a portal is no…
Bilelkihal Sep 27, 2024
bf5fe76
disable input chips in federated search page for the portals that are…
Bilelkihal Sep 27, 2024
3366fca
show tooltip for disabled input chips in the federated search
Bilelkihal Sep 27, 2024
d2177f9
clean search controller federation code
Bilelkihal Oct 1, 2024
5c7d434
update ontologies api ruby client to the latest version of federated …
Bilelkihal Oct 1, 2024
cb8276d
fix federated browse icons colors
Bilelkihal Oct 2, 2024
d320de2
add tooltip for portal button in federated search
Bilelkihal Oct 2, 2024
02ce7cb
open external ontologies links in new tab in federated browse
Bilelkihal Oct 2, 2024
cd83029
use portal button helper in federated browse
Bilelkihal Oct 2, 2024
aacfd56
cache federation status call
Bilelkihal Oct 3, 2024
d17c31b
update chip helpers to support disabled state
Bilelkihal Oct 3, 2024
7ca6359
use turbo frame for federation input chips, and use federation portal…
Bilelkihal Oct 3, 2024
aa7913c
add skelton loading animation for federation input chips
Bilelkihal Oct 3, 2024
bf8224f
cache federation input chips separately
Bilelkihal Oct 4, 2024
d15ae62
use cached federation input chips in browse page
Bilelkihal Oct 4, 2024
6f989f6
fix browse page federation inputs section title style
Bilelkihal Oct 4, 2024
3d2a63c
initialize federation portals input chips in the home page (asynch in…
Bilelkihal Oct 4, 2024
22eeca9
Merge branch 'development' into feature/federate-search-page
syphax-bouazzouni Oct 4, 2024
701879a
clean federated search aggregator code
Bilelkihal Oct 7, 2024
965cab5
move chip skelton to components helper
Bilelkihal Oct 7, 2024
f3cb39b
internationalize portal is not responding message
Bilelkihal Oct 7, 2024
fba0daf
clean federation portal status cache method
Bilelkihal Oct 7, 2024
b4f7816
clean federation stimulus controller code
Bilelkihal Oct 8, 2024
2a21b8c
display federated browse errors as warning instead of danger (orange …
Bilelkihal Oct 8, 2024
b037ab3
remove related code to fedrated portals status to put it in another PR
Bilelkihal Oct 8, 2024
69fe323
use the key "collection" instead of "results" in federated search res…
Bilelkihal Oct 8, 2024
71f92bf
method to get the canonical portal for an ontology using pullLocation
Bilelkihal Oct 9, 2024
108ed98
display federated ontologies in browse page based on the canonical po…
Bilelkihal Oct 9, 2024
8f00782
Merge branch 'development' into feature/federation-ontology-canonical…
syphax-bouazzouni Oct 16, 2024
2f4c8c4
clean canonical portal function in federation helper
Bilelkihal Oct 16, 2024
61d8cfd
extract canonical portal ontology choice in a function for clarity
Bilelkihal Oct 16, 2024
4dbba55
display search page results based on the canonical portal of the onto…
Bilelkihal Oct 17, 2024
1d748c8
move federation canonical to federation helper
syphax-bouazzouni Oct 22, 2024
6a2aba4
clean swap portal attributes function in federation helper
Bilelkihal Oct 23, 2024
a5c855f
fix and clean apply canincal portal function
Bilelkihal Oct 23, 2024
8b2182f
extract count portals into a separate function from external_canonica…
Bilelkihal Oct 23, 2024
70dfcd8
Merge branch 'development' into feature/federation-ontology-canonical…
syphax-bouazzouni Oct 23, 2024
55d0b71
put again search canonical logic the search_aggregator file and refac…
syphax-bouazzouni Oct 23, 2024
a86ba77
do the search canonical logic only if enabled
syphax-bouazzouni Oct 23, 2024
5a533f1
fix rest_hostname raising an exception that REST_URI does not exist
syphax-bouazzouni Oct 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,8 @@ gem "flag-icons-rails", "~> 3.4"

# Custom API client
gem 'ontologies_api_client', git: 'https://github.com/ontoportal-lirmm/ontologies_api_ruby_client.git', branch: 'development'

# Ruby 2.7.8 pinned gems (to remove when migrating to Ruby >= 3.0)

gem 'ffi', '~> 1.16.3'
gem 'net-ftp', '~> 0.2.0', require: false
gem 'net-http', '~> 0.3.2'
Expand Down
5 changes: 4 additions & 1 deletion app/controllers/concerns/search_aggregator.rb
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,16 @@ def aggregate_results(query, results)

all_ontologies = LinkedData::Client::Models::Ontology.all(include: 'acronym,name', include_views: true, display_links: false, display_context: false)

all_submissions = LinkedData::Client::Models::OntologySubmission.all(include: 'pullLocation', include_views: true, display_links: false, display_context: false)

search_results = grouped_results.map do |group|
format_search_result(group, all_ontologies)
end

search_results = merge_sort_federated_results(query, search_results) if federation_enabled?

search_results = apply_canonical_portal(search_results, all_submissions)
syphax-bouazzouni marked this conversation as resolved.
Show resolved Hide resolved

search_results
end

Expand Down Expand Up @@ -91,7 +95,6 @@ def search_result_elem(class_object, ontology_acronym, title)
result
end


def ontology_name_acronym(ontologies, selected_acronym)
ontology = ontologies.select { |x| x.acronym.eql?(selected_acronym.split('/').last) }.first
"#{ontology.name} (#{ontology.acronym})" if ontology
Expand Down
6 changes: 5 additions & 1 deletion app/controllers/concerns/submission_filter.rb
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ def merge_by_acronym(submissions)
if ontologies.size.eql?(1)
ontology = ontologies.first
else
ontology = ontologies.select { |x| helpers.internal_ontology?(x[:id]) }.first || ontologies.first
ontology = canonical_portal_ontology(ontologies)
end

ontology[:sources] = ontologies.map { |x| x[:id] }
Expand All @@ -80,6 +80,10 @@ def merge_by_acronym(submissions)
merged_submissions
end

def canonical_portal_ontology(ontologies)
syphax-bouazzouni marked this conversation as resolved.
Show resolved Hide resolved
ontologies.select { |x| helpers.internal_ontology?(x[:id]) }.first || ontology_from_portal(ontologies, canonical_portal(ontologies))
end

def filter_submissions(ontologies, query:, status:, show_views:, private_only:, languages:, page_size:, formality_level:, is_of_type:, groups:, categories:, formats:)
submissions = LinkedData::Client::Models::OntologySubmission.all(include: BROWSE_ATTRIBUTES.join(','), also_include_views: true, display_links: false, display_context: false)

Expand Down
51 changes: 51 additions & 0 deletions app/helpers/federation_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -124,4 +124,55 @@ def federation_external_class?(class_object)
end


def canonical_portal(ontologies)
portal_counts = Hash.new(0)
# Count occurrences of each portal in the pull_location URL
ontologies.each do |ontology|
federated_portals.keys.each do |portal|
if ontology[:pullLocation]&.include?(portal.to_s)
portal_counts[portal] += 1
end
end
end
# Determine the portal with the most occurrences
canonical_portal = portal_counts.max_by { |_, count| count }&.first

canonical_portal
end

def apply_canonical_portal(search_results, all_submissions)
search_results.each do |result|
next if result[:root][:portal_name].nil? || result[:root][:other_portals].blank?

root_link = result[:root][:link].split('?').first
candidates = [result[:root][:link]] + result[:root][:other_portals].map { |p| p[:link].split('?').first }

portal_counts = Hash.new(0)
Bilelkihal marked this conversation as resolved.
Show resolved Hide resolved
candidates.each do |candidate|
submission = all_submissions.find { |s| s.id.split('/')[0...-2].join('/') == candidate }

federated_portals.keys.each do |portal|
portal_counts[portal] += 1 if submission&.include?(portal.to_s)
Bilelkihal marked this conversation as resolved.
Show resolved Hide resolved
end
end

canonical_portal = portal_counts.max_by { |_, count| count }&.first
next if canonical_portal.nil? || result[:root][:portal_name].eql?(canonical_portal.to_s)

canonical_portal_result = result[:root][:other_portals].find { |r| r[:name] == canonical_portal.to_s }
swap_portal_attributes(result[:root], canonical_portal_result) if canonical_portal_result
end
end

def swap_portal_attributes(root_portal, new_portal)
Bilelkihal marked this conversation as resolved.
Show resolved Hide resolved
root_portal[:link], new_portal[:link] = new_portal[:link], root_portal[:link]
root_portal[:portal_name], new_portal[:portal_name] = new_portal[:portal_name], root_portal[:portal_name]
root_portal[:portal_color], new_portal[:portal_color] = new_portal[:portal_color], root_portal[:portal_color]
root_portal[:portal_light_color], new_portal[:portal_light_color] = new_portal[:portal_light_color], root_portal[:portal_light_color]
end

def ontology_from_portal(ontologies, portal)
ontologies.select{|o| o[:id].include?(portal.to_s)}.first
end

end
Loading