Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor: Centralize mapping creation #157

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
0cb252f
replace the readers by attr_read
syphax-bouazzouni Jul 27, 2022
5e01d03
remove useless returns
syphax-bouazzouni Jul 27, 2022
782355a
auto lint changes
syphax-bouazzouni Jul 27, 2022
fb2fb07
add the subject_source_id and object_source_id attributes
syphax-bouazzouni Jul 27, 2022
67b90b2
update owlapi-wrapper
syphax-bouazzouni Aug 3, 2022
7eb3424
add concerns glob loader
syphax-bouazzouni Aug 3, 2022
32e6639
add MetadataExtractor concern with extract_version and ontology_iri
syphax-bouazzouni Aug 3, 2022
4996384
use the new extract_metadata and remove the old extract_version
syphax-bouazzouni Aug 3, 2022
c1e485b
update owlapi-wrapper to 1.4.1 upstream
syphax-bouazzouni Aug 3, 2022
6eddae6
Merge branch 'pr/feature/add-extract-ontology-uri-to-submissions' int…
syphax-bouazzouni Aug 5, 2022
764f45e
add the concerns requires
syphax-bouazzouni Aug 8, 2022
0078bfa
add the mapping bulk load tests
syphax-bouazzouni Aug 8, 2022
e99a5cc
change the mapping model to enforce :list for relation
syphax-bouazzouni Aug 8, 2022
763679c
auto-lint of the mappings.rb file
syphax-bouazzouni Aug 8, 2022
c8b68d5
add the Mappings::BulkLoad concern
syphax-bouazzouni Aug 8, 2022
73b15a8
extend the Mappings module with the Mappings::BulkLoad concern
syphax-bouazzouni Aug 8, 2022
d479dc2
revert enforce relation to be :list
syphax-bouazzouni Aug 8, 2022
5b2a233
add check_mapping_exist before the creation
syphax-bouazzouni Aug 19, 2022
089f75b
make the bulk_loader return errors for each unsaved mapping
syphax-bouazzouni Aug 19, 2022
737add9
fix unspecified mapping defaults
syphax-bouazzouni Aug 19, 2022
c7114cc
remove 'process' from the given mappings
syphax-bouazzouni Aug 19, 2022
a40e5e1
rescue mapping process creation errors
syphax-bouazzouni Aug 19, 2022
b3a4d92
fix mappings tests by deleting them after test
syphax-bouazzouni Aug 19, 2022
03fc368
rescue the loading errors in the tests
syphax-bouazzouni Aug 19, 2022
0d524c8
remove 'process' from the given mapping to load in the tests
syphax-bouazzouni Aug 19, 2022
b47236b
save backup_rest_mapping id when creating
syphax-bouazzouni Sep 2, 2022
d0b58ce
fix test_mapping_ontologies_not_found
syphax-bouazzouni Sep 14, 2022
a80ae8c
centralize the mapping creation in the Mappings::Creator module
syphax-bouazzouni Sep 14, 2022
fcaf6d4
extend Mappings with the Creator module
syphax-bouazzouni Sep 14, 2022
bb1a8de
auto lint mappings module
syphax-bouazzouni Sep 14, 2022
5014a11
auto lint mappings models and add validate_size enforce to class_urns
syphax-bouazzouni Sep 14, 2022
2b656eb
re-use the create_mapping method in the BulkLoader
syphax-bouazzouni Sep 14, 2022
7b67f9d
bring ontology acronym if class found
syphax-bouazzouni Sep 14, 2022
7868dd6
Merge branch 'master' into pr/refactor/centralize-mapping-creation
syphax-bouazzouni Sep 14, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Merge branch 'master' into pr/refactor/centralize-mapping-creation
syphax-bouazzouni authored Sep 14, 2022

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 7868dd6bed68797086f5a9ac5e0faceb403c98a6
26 changes: 12 additions & 14 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
GIT
remote: https://github.com/ncbo/goo.git
revision: fd7d45cb862c5c2c1833b64a5c8c14154384edc2
revision: 562826ba21f7da641159071531375776a1414207
branch: master
specs:
goo (0.0.2)
@@ -32,8 +32,8 @@ GEM
multi_json (~> 1.3)
thread_safe (~> 0.1)
tzinfo (~> 0.3.37)
addressable (2.8.0)
public_suffix (>= 2.0.2, < 5.0)
addressable (2.8.1)
public_suffix (>= 2.0.2, < 6.0)
ansi (1.5.0)
ast (2.4.2)
bcrypt (3.1.18)
@@ -50,7 +50,7 @@ GEM
launchy (~> 2.1)
mail (~> 2.7)
eventmachine (1.2.7)
faraday (1.10.0)
faraday (1.10.1)
faraday-em_http (~> 1.0)
faraday-em_synchrony (~> 1.0)
faraday-excon (~> 1.1)
@@ -110,7 +110,7 @@ GEM
omni_logger (0.1.4)
logger
parallel (1.22.1)
parser (3.1.2.0)
parser (3.1.2.1)
ast (~> 2.4.1)
pony (1.13.1)
mail (>= 2.0)
@@ -119,7 +119,7 @@ GEM
pry (0.14.1)
coderay (~> 1.1)
method_source (~> 1.0)
public_suffix (4.0.7)
public_suffix (5.0.0)
rack (1.6.13)
rack-test (0.8.3)
rack (>= 1.0, < 3)
@@ -137,17 +137,17 @@ GEM
rexml (3.2.5)
rsolr (1.1.2)
builder (>= 2.1.2)
rubocop (1.31.2)
rubocop (1.35.0)
json (~> 2.3)
parallel (~> 1.10)
parser (>= 3.1.0.0)
parser (>= 3.1.2.1)
rainbow (>= 2.2.2, < 4.0)
regexp_parser (>= 1.8, < 3.0)
rexml (>= 3.2.5, < 4.0)
rubocop-ast (>= 1.18.0, < 2.0)
rubocop-ast (>= 1.20.1, < 2.0)
ruby-progressbar (~> 1.7)
unicode-display_width (>= 1.4.0, < 3.0)
rubocop-ast (1.19.1)
rubocop-ast (1.21.0)
parser (>= 3.1.1.0)
ruby-progressbar (1.11.0)
ruby2_keywords (0.0.5)
@@ -169,7 +169,7 @@ GEM
eventmachine (~> 1.0, >= 1.0.4)
rack (>= 1, < 3)
thread_safe (0.3.6)
tzinfo (0.3.60)
tzinfo (0.3.61)
unf (0.1.4)
unf_ext
unf_ext (0.0.8.2)
@@ -178,8 +178,6 @@ GEM
macaddr (~> 1.0)

PLATFORMS
x86_64-darwin-18
x86_64-darwin-21
x86_64-linux

DEPENDENCIES
@@ -213,4 +211,4 @@ DEPENDENCIES
thin

BUNDLED WITH
2.3.14
2.3.15
Binary file added bin/owlapi-wrapper-1.4.0.jar
Binary file not shown.
64 changes: 41 additions & 23 deletions lib/ontologies_linked_data/metrics/metrics.rb
Original file line number Diff line number Diff line change
@@ -33,10 +33,12 @@ def self.metrics_for_submission(submission, logger)
metrics.properties = prop_count
logger.info("properties finished")
logger.flush

# re-generate metrics file
submission.generate_metrics_file(cls_metrics[:classes], indiv_count, prop_count)
submission.generate_metrics_file2(cls_metrics[:classes], indiv_count, prop_count, cls_metrics[:maxDepth])
logger.info("generation of metrics file finished")
logger.flush

rescue Exception => e
logger.error(e.message)
logger.error(e)
@@ -46,35 +48,51 @@ def self.metrics_for_submission(submission, logger)
metrics
end

def self.max_depth_fn(submission, logger, is_flat, rdfsSC)
max_depth = 0
mx_from_file = submission.metrics_from_file(logger)
if (mx_from_file && mx_from_file.length == 2 && mx_from_file[0].length >= 4)
then
max_depth = mx_from_file[1][3].to_i
else
logger.info("Unable to find metrics providing max_depth in file for submission #{submission.id.to_s}. Using ruby calculation of max_depth.")
roots = submission.roots

unless is_flat
depths = []
roots.each do |root|
ok = true
n=1
while ok
ok = hierarchy_depth?(submission.id.to_s,root.id.to_s,n,rdfsSC)
if ok
n += 1
end
if n > 40
#safe guard
ok = false
end
end
n -= 1
depths << n
end
max_depth = depths.max
end
end
max_depth
end

def self.class_metrics(submission, logger)
t00 = Time.now
submission.ontology.bring(:flat) if submission.ontology.bring?(:flat)

is_flat = submission.ontology.flat
roots = submission.roots

max_depth = 0
rdfsSC = nil
unless is_flat
depths = []
rdfsSC = Goo.namespaces[:rdfs][:subClassOf]
roots.each do |root|
ok = true
n=1
while ok
ok = hierarchy_depth?(submission.id.to_s,root.id.to_s,n,rdfsSC)
if ok
n += 1
end
if n > 40
#safe guard
ok = false
end
end
n -= 1
depths << n
end
max_depth = depths.max
rdfsSC = Goo.namespaces[:rdfs][:subClassOf]
end
max_depth = max_depth_fn(submission, logger, is_flat, rdfsSC)

cls_metrics = {}
cls_metrics[:classes] = 0
cls_metrics[:averageChildCount] = 0
7 changes: 7 additions & 0 deletions lib/ontologies_linked_data/models/ontology_submission.rb
Original file line number Diff line number Diff line change
@@ -416,6 +416,13 @@ def generate_metrics_file(class_count, indiv_count, prop_count)
end
end

def generate_metrics_file2(class_count, indiv_count, prop_count, max_depth)
CSV.open(self.metrics_path, "wb") do |csv|
csv << ["Class Count", "Individual Count", "Property Count", "Max Depth"]
csv << [class_count, indiv_count, prop_count, max_depth]
end
end

def generate_umls_metrics_file(tr_file_path=nil)
tr_file_path ||= self.triples_file_path
class_count = 0
5 changes: 3 additions & 2 deletions test/models/test_ontology_submission.rb
Original file line number Diff line number Diff line change
@@ -956,7 +956,7 @@ def test_submission_metrics
assert_equal 0, metrics.classesWithMoreThan25Children
assert_equal 18, metrics.maxChildCount
assert_equal 3, metrics.averageChildCount
assert_equal 3, metrics.maxDepth
assert_equal 4, metrics.maxDepth

submission_parse("BROTEST-METRICS", "BRO testing metrics",
"./test/data/ontology_files/BRO_v3.2.owl", 33,
@@ -1006,12 +1006,13 @@ def test_submission_metrics
assert_equal 63, metrics.properties
assert_equal 124, metrics.individuals
assert_equal 0, metrics.classesWithOneChild
assert_equal 7, metrics.maxDepth
#cause it has not the subproperty added
assert_equal 474, metrics.classesWithNoDefinition
assert_equal 0, metrics.classesWithMoreThan25Children
assert_equal 0, metrics.maxChildCount
assert_equal 0, metrics.averageChildCount
assert_equal 0, metrics.maxDepth


#test UMLS metrics
acronym = 'UMLS-TST'
You are viewing a condensed version of this merge commit. You can view the full changes here.