Merge pull request #22 from vivo-community/develop

Develop
vivo-community · Jun 22, 2021 · 4d7c020 · 4d7c020
2 parents 18df6eb + 234cbbf
commit 4d7c020
Show file tree

Hide file tree

Showing 88 changed files with 1,576 additions and 974 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,8 +2,18 @@
 All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
+
+
+## [1.1.0] - 2021-06-22
 ### Added
-- Query ORCID and filter for current employees.
+- query ORCID for person and current employees at ROR organization
+- refactor all queries to follow three steps: source, extraction, mapping to vivo-rdf
+- check GraphQL queries for response code 200 and additionally if response body contains error messages (no data)
+- make all controller use HTTP-Get requests
+- centralize input validation
+- export to VIVO in chunks
+- simplify error handling
+- document methods in swagger-UI and remove response section
 
 ## [Renamed Project] - 2021-05-25
 As new datsources were integrated and the name datacitecommons2vivo was not reflecting

diff --git a/DockerfileBuild b/DockerfileBuild
@@ -3,7 +3,7 @@ FROM maven:3.6.3-jdk-11-slim AS build-env
 WORKDIR /app
 COPY ./pom.xml ./pom.xml
 COPY ./src ./src
-RUN mvn clean install -Dmaven.test.skip=true && cp target/*.jar app.jar
+RUN mvn clean install -Dmaven.test.skip=true && cp target/generate2vivo-*.jar app.jar
 
 #-------- NEXT STAGE -----------
 

diff --git a/README.md b/README.md
@@ -1,17 +1,43 @@
 [![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
 
 ## generate2vivo
-generate2vivo is a Data Ingest Tool for the Open-Source-Software VIVO. 
-It queries metadata from the Datacite Commons PID-Graph and the ROR API, 
-maps it to the VIVO ontology using [sparql-generate](https://ci.mines-stetienne.fr/sparql-generate/index.html) and optionally imports it into a VIVO instance.
+generate2vivo is an extensible Data Ingest Tool for the Open-Source-Software VIVO. 
+It currently queries metadata from Datacite Commons, ROR and ORCID
+and maps them to the VIVO ontology using [sparql-generate](https://ci.mines-stetienne.fr/sparql-generate/index.html).
+The resulting RDF data can be exported to a VIVO instance directly or returned in a HTTP response.
 
+- [Available queries](#available-queries)
+  + [Datacite Commons](#datacite-commons)
+  + [ROR](#ror)
+  + [ORCID](#orcid)
 - [Installation](#installation)
-- [Usage](#usage)
-    + [Datacite Commons](#datacite-commons)
-    + [ROR](#ror)
 - [Run in Command Line](#run-in-command-line)
 - [Extensible](#extensible)
 
+### Available queries
+The datasources and queries that are currently available are listed below.
+
+##### Datacite Commons
+For Datacite Commons the following queries are available:
+* `organization` : This method gets data about an organization by passing a ROR id.
+* `organizationPlusPeople`: This method gets data about an organization and its affiliated people by passing a ROR id.
+* `organizationPlusPeoplePlusPublications`:This method gets data about an organization and its affiliated people and their respective publications by passing a ROR id.
+* `person`: This method gets data about a person by passing an ORCID id.
+* `personPlusPublications`: This method gets data about a person and their publications by passing an ORCID id.
+* `work`: This method gets data about a work by passing an DOI.
+
+##### ROR
+For ROR there are 2 queries available:
+* `organization`: This method gets data about an organization by passing a ROR id.
+* `organizationPlusChildren`: This method gets data about an organization and all their sub-organizations by passing a ROR id.
+
+##### ORCID
+For ORCID the following queries are available:
+* `personPlusWorks`: This method gets data about a person and their works by passing an ORCID id.
+* `currentEmployeesPlusWorks`: This method gets data about an organization's current employees and their works by passing a ROR id.
+
+
+
 ### Installation
 1. Clone the repository to a local folder using `git clone https://github.com/vivo-community/generate2vivo.git`
 2. Change into the folder where the repository has been cloned. 
@@ -33,32 +59,16 @@ maps it to the VIVO ontology using [sparql-generate](https://ci.mines-stetienne.
 
 5. A minimal swagger-ui will be available at `http://localhost:9000/swagger-ui/`.
 
-### Usage
-Go to `http://localhost:9000/swagger-ui/` in your browser and choose Datacite Commons or ROR as a data source. 
-
-##### Datacite Commons
-For Datacite Commons there are 2 queries available:
-* `getOrganizationPlusPersons`: Queries Datacite Commons for the organization and its affiliated people.
-* `getPersonPlusPublications`: Queries Datacite Commons for the person and its affiliated publications. 
-
-##### ROR
-For ROR there are 2 queries available:
-* `getOrganization`: Queries ROR for the organization.
-* `getOrganizationPlusChildren`: Queries ROR for the organization and all of its sub-organizations recursively.
-
-The program will return a 200 Status, if the data was imported to VIVO or if you chose not to provide your VIVO details,
-it will return the RDF-data as a result in format JSON-LD.
-
 ### Run in Command Line
 Alternatively you can run the queries from the command line using the sparql-generate executable JAR-file.
 All queries are placed in folder `src/main/resources/sparqlg` and come with a `sparql-generate-conf.json`. 
 Its structure and use are explained in detail on the [sparql-generate website](https://ci.mines-stetienne.fr/sparql-generate/language-cli.html).
 
 ### Extensible
-The software is easily extensible, meaning you can add and remove datasources without touching the code.
+The software is easily extensible, meaning you can add and remove datasources.
 
 For example, if you are not interested in using Datacite Commons, just remove the folder from `src/main/resources/sparqlg`
-and the respective controller in the package `eu.tib.controller` and it's gone.
+and the respective controller in the package `eu.tib.controller`.
 
 On the other hand, if you would like to add a datasource:
 * add a folder with your queries under `src/main/resources/sparqlg` and include a `sparql-generate-conf.json` 

diff --git a/pom.xml b/pom.xml
@@ -13,8 +13,8 @@
 
     <groupId>eu.tib</groupId>
     <artifactId>generate2vivo</artifactId>
-    <version>1.0.0</version>
-    <description>Data Ingest from different datasources like Datacite Commons or ROR to VIVO</description>
+    <version>1.1.0</version>
+    <description>Extensible Data Ingest Tool for VIVO. Contains data sources like Datacite Commons, ORCID and ROR.</description>
 
     <properties>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

diff --git a/src/main/java/eu/tib/controller/DataciteCommonsController.java b/src/main/java/eu/tib/controller/DataciteCommonsController.java
@@ -1,17 +1,16 @@
 package eu.tib.controller;
 
+import eu.tib.controller.validation.InputValidator;
 import eu.tib.service.ResponseService;
 import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
 import io.swagger.annotations.ApiParam;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.http.ResponseEntity;
 import org.springframework.util.StopWatch;
 import org.springframework.validation.annotation.Validated;
-import org.springframework.web.bind.annotation.PostMapping;
-import org.springframework.web.bind.annotation.RequestMapping;
-import org.springframework.web.bind.annotation.RequestParam;
-import org.springframework.web.bind.annotation.RestController;
+import org.springframework.web.bind.annotation.*;
 
 import javax.validation.Valid;
 import javax.validation.constraints.Pattern;
@@ -27,9 +26,29 @@ public class DataciteCommonsController {
     @Autowired
     private ResponseService responseService;
 
-    @PostMapping(value = "/getOrganizationPlusPersons", produces = "application/json")
-    public ResponseEntity<String> getOrganizationPlusPersons(
-            @Valid @Pattern(regexp = "^https://ror.org/\\d{2}[a-z0-9]{5}\\d{2}")
+    @ApiOperation(value = "Retrieve organization data from Datacite Commons", notes = "This method gets data about an organization from Datacite Commons by passing a ROR id.")
+    @GetMapping(value = "/organization", produces = "application/json")
+    public ResponseEntity<String> getOrganization(
+            @Valid @Pattern(regexp = InputValidator.ror)
+            @ApiParam("Complete ROR URL consisting of https://ror.org/ plus id")
+            @RequestParam String ror) {
+
+        final String id = "sparqlg/datacitecommons/organization";
+        log.info("Incoming Request for " + id + " with ror: " + ror);
+        StopWatch stopWatch = new StopWatch(id);
+        stopWatch.start(id);
+
+        ResponseEntity result = responseService.buildResponse(id, Collections.singletonMap("ror", ror));
+
+        stopWatch.stop();
+        log.info(id + " took " + stopWatch.getTotalTimeSeconds() + "s");
+        return result;
+    }
+
+    @ApiOperation(value = "Retrieve data about an organization and its affiliated people from Datacite Commons", notes = "This method gets data about an organization and its affiliated people from Datacite Commons by passing a ROR id.")
+    @GetMapping(value = "/organizationPlusPeople", produces = "application/json")
+    public ResponseEntity<String> getOrganizationPlusPeople(
+            @Valid @Pattern(regexp = InputValidator.ror)
             @ApiParam("Complete ROR URL consisting of https://ror.org/ plus id")
             @RequestParam String ror) {
 
@@ -44,10 +63,48 @@ public ResponseEntity<String> getOrganizationPlusPersons(
         log.info(id + " took " + stopWatch.getTotalTimeSeconds() + "s");
         return result;
     }
+    @ApiOperation(value = "Retrieve data about an organization and its affiliated people and their respective publications from Datacite Commons", notes = "This method gets data about an organization and its affiliated people and their respective publications from Datacite Commons by passing a ROR id.")
+    @GetMapping(value = "/organizationPlusPeoplePlusPublications", produces = "application/json")
+    public ResponseEntity<String> getOrganizationPlusPeoplePlusPublications(
+            @Valid @Pattern(regexp = InputValidator.ror)
+            @ApiParam("Complete ROR URL consisting of https://ror.org/ plus id")
+            @RequestParam String ror) {
+
+        final String id = "sparqlg/datacitecommons/orga2person2publication";
+        log.info("Incoming Request for " + id + " with ror: " + ror);
+        StopWatch stopWatch = new StopWatch(id);
+        stopWatch.start(id);
 
-    @PostMapping(value = "/getPersonPlusPublications", produces = "application/json")
+        ResponseEntity result = responseService.buildResponse(id, Collections.singletonMap("ror", ror));
+
+        stopWatch.stop();
+        log.info(id + " took " + stopWatch.getTotalTimeSeconds() + "s");
+        return result;
+    }
+
+    @ApiOperation(value = "Retrieve data about a person from Datacite Commons", notes = "This method gets data about a person from Datacite Commons by passing an ORCID id.")
+    @GetMapping(value = "/person", produces = "application/json")
+    public ResponseEntity<String> getPerson(
+            @Valid @Pattern(regexp = InputValidator.orcid)
+            @ApiParam("Complete Orcid URL consisting of https://orcid.org/ plus id")
+            @RequestParam String orcid) {
+
+        final String id = "sparqlg/datacitecommons/person";
+        log.info("Incoming Request for " + id + " with orcid: " + orcid);
+        StopWatch stopWatch = new StopWatch(id);
+        stopWatch.start(id);
+
+        ResponseEntity result = responseService.buildResponse(id, Collections.singletonMap("orcid", orcid));
+
+        stopWatch.stop();
+        log.info(id + " took " + stopWatch.getTotalTimeSeconds() + "s");
+        return result;
+    }
+
+    @ApiOperation(value = "Retrieve data about a person and their publications from Datacite Commons", notes = "This method gets data about a person and their publications from Datacite Commons by passing an ORCID id.")
+    @GetMapping(value = "/personPlusPublications", produces = "application/json")
     public ResponseEntity<String> getPersonPlusPublications(
-            @Valid @Pattern(regexp = "^https://orcid.org/\\d{4}-\\d{4}-\\d{4}-\\d{4}")
+            @Valid @Pattern(regexp = InputValidator.orcid)
             @ApiParam("Complete Orcid URL consisting of https://orcid.org/ plus id")
             @RequestParam String orcid) {
 
@@ -62,4 +119,23 @@ public ResponseEntity<String> getPersonPlusPublications(
         log.info(id + " took " + stopWatch.getTotalTimeSeconds() + "s");
         return result;
     }
+
+    @ApiOperation(value = "Retrieve data about a work from Datacite Commons", notes = "This method gets data about a work from Datacite Commons by passing an DOI.")
+    @GetMapping(value = "/work", produces = "application/json")
+    public ResponseEntity<String> getWork(
+            @Valid @Pattern(regexp = InputValidator.doi)
+            @ApiParam("DOI of the publication")
+            @RequestParam String doi) {
+
+        final String id = "sparqlg/datacitecommons/work";
+        log.info("Incoming Request for " + id + " with doi: " + doi);
+        StopWatch stopWatch = new StopWatch(id);
+        stopWatch.start(id);
+
+        ResponseEntity result = responseService.buildResponse(id, Collections.singletonMap("doi", doi));
+
+        stopWatch.stop();
+        log.info(id + " took " + stopWatch.getTotalTimeSeconds() + "s");
+        return result;
+    }
 }
diff --git a/src/main/java/eu/tib/controller/OrcidController.java b/src/main/java/eu/tib/controller/OrcidController.java
@@ -1,17 +1,16 @@
 package eu.tib.controller;
 
+import eu.tib.controller.validation.InputValidator;
 import eu.tib.service.ResponseService;
 import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
 import io.swagger.annotations.ApiParam;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.http.ResponseEntity;
 import org.springframework.util.StopWatch;
 import org.springframework.validation.annotation.Validated;
-import org.springframework.web.bind.annotation.PostMapping;
-import org.springframework.web.bind.annotation.RequestMapping;
-import org.springframework.web.bind.annotation.RequestParam;
-import org.springframework.web.bind.annotation.RestController;
+import org.springframework.web.bind.annotation.*;
 
 import javax.validation.Valid;
 import javax.validation.constraints.Pattern;
@@ -27,18 +26,41 @@ public class OrcidController {
     @Autowired
     private ResponseService responseService;
 
-    @PostMapping(value = "/getCurrentEmployees", produces = "application/json")
-    public ResponseEntity<String> getCurrentEmployees(
-            @Valid @Pattern(regexp = "^Q[1-9]\\d*$")
-            @ApiParam("Wikidata id for a research organization starting with Q")
-            @RequestParam String wikidata) {
+    @ApiOperation(value = "Retrieve data about a person and their works from ORCID", notes = "This method gets data about a person and their works from ORCID by passing an ORCID id.")
+    @GetMapping(value = "/personPlusWorks", produces = "application/json")
+    public ResponseEntity<String> getPersonPlusWorks(
+            @Valid @Pattern(regexp = InputValidator.orcid)
+            @ApiParam("Complete Orcid URL consisting of https://orcid.org/ plus id")
+            @RequestParam String orcid) {
+
+        final String id = "sparqlg/orcid/person";
+        log.info("Incoming Request for " + id + " with orcid: " + orcid);
+        StopWatch stopWatch = new StopWatch(id);
+        stopWatch.start(id);
+
+        String orcid_id = orcid.replaceFirst("https://orcid.org/","");
+        ResponseEntity result = responseService.buildResponse(id,
+                Collections.singletonMap("orcid", orcid_id));
+
+        stopWatch.stop();
+        log.info(id + " took " + stopWatch.getTotalTimeSeconds() + "s");
+        return result;
+    }
+
+
+    @ApiOperation(value = "Retrieve data about an organization's current employees and their works from ORCID", notes = "This method gets data about an organization's current employees and their works from ORCID by passing a ROR id.")
+    @GetMapping(value = "/currentEmployeesPlusWorks", produces = "application/json")
+    public ResponseEntity<String> getCurrentEmployeesPlusWorks(
+            @Valid @Pattern(regexp = InputValidator.ror)
+            @ApiParam("Complete ROR URL consisting of https://ror.org/ plus id")
+            @RequestParam String ror) {
 
         final String id = "sparqlg/orcid/employees";
-        log.info("Incoming Request for " + id + " with wikidata: " + wikidata);
+        log.info("Incoming Request for " + id + " with ror: " + ror);
         StopWatch stopWatch = new StopWatch(id);
         stopWatch.start(id);
 
-        ResponseEntity result = responseService.buildResponse(id, Collections.singletonMap("wikidata", wikidata));
+        ResponseEntity result = responseService.buildResponse(id, Collections.singletonMap("ror", ror));
 
         stopWatch.stop();
         log.info(id + " took " + stopWatch.getTotalTimeSeconds() + "s");

diff --git a/src/main/java/eu/tib/controller/RORController.java b/src/main/java/eu/tib/controller/RORController.java
@@ -1,17 +1,16 @@
 package eu.tib.controller;
 
+import eu.tib.controller.validation.InputValidator;
 import eu.tib.service.ResponseService;
 import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
 import io.swagger.annotations.ApiParam;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.http.ResponseEntity;
 import org.springframework.util.StopWatch;
 import org.springframework.validation.annotation.Validated;
-import org.springframework.web.bind.annotation.PostMapping;
-import org.springframework.web.bind.annotation.RequestMapping;
-import org.springframework.web.bind.annotation.RequestParam;
-import org.springframework.web.bind.annotation.RestController;
+import org.springframework.web.bind.annotation.*;
 
 import javax.validation.Valid;
 import javax.validation.constraints.Pattern;
@@ -27,13 +26,14 @@ public class RORController {
     @Autowired
     private ResponseService responseService;
 
-    @PostMapping(value = "/getOrganizationPlusChildren", produces = "application/json")
-    public ResponseEntity<String> getOrganizationPlusChildren(
-            @Valid @Pattern(regexp = "^https://ror.org/\\d{2}[a-z0-9]{5}\\d{2}")
+    @ApiOperation(value = "Retrieve data about an organization from ROR", notes = "This method gets data about an organization from ROR by passing a ROR id.")
+    @GetMapping(value = "/organization", produces = "application/json")
+    public ResponseEntity<String> getOrganization(
+            @Valid @Pattern(regexp = InputValidator.ror)
             @ApiParam("Complete ROR URL consisting of https://ror.org/ plus id")
             @RequestParam String ror) {
 
-        final String id = "sparqlg/ror/orga2children";
+        final String id = "sparqlg/ror/organization";
         log.info("Incoming Request for " + id + " with ror: " + ror);
         StopWatch stopWatch = new StopWatch(id);
         stopWatch.start(id);
@@ -45,13 +45,14 @@ public ResponseEntity<String> getOrganizationPlusChildren(
         return result;
     }
 
-    @PostMapping(value = "/getOrganization", produces = "application/json")
-    public ResponseEntity<String> getOrganization(
-            @Valid @Pattern(regexp = "^https://ror.org/\\d{2}[a-z0-9]{5}\\d{2}")
+    @ApiOperation(value = "Retrieve data about an organization and all their sub-organizations from ROR", notes = "This method gets data about an organization and all their sub-organizations from ROR by passing a ROR id.")
+    @GetMapping(value = "/organizationPlusChildren", produces = "application/json")
+    public ResponseEntity<String> getOrganizationPlusChildren(
+            @Valid @Pattern(regexp = InputValidator.ror)
             @ApiParam("Complete ROR URL consisting of https://ror.org/ plus id")
             @RequestParam String ror) {
 
-        final String id = "sparqlg/ror/organization";
+        final String id = "sparqlg/ror/orga2children";
         log.info("Incoming Request for " + id + " with ror: " + ror);
         StopWatch stopWatch = new StopWatch(id);
         stopWatch.start(id);