Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faster combined query for retrieving datasets via API #9684

Merged
merged 16 commits into from
May 24, 2024
Merged

Faster combined query for retrieving datasets via API #9684

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
2968ba2
combined query for retrieving datasets with API
ErykKul Jun 28, 2023
c8a1352
Merge branch 'IQSS:develop' into 9683_get_dataset_api_in_single_query
ErykKul Jun 28, 2023
cdc3e3d
Merge branch 'IQSS:develop' into 9683_get_dataset_api_in_single_query
ErykKul Jun 29, 2023
6083ead
better error handling - should fix failed integration test
ErykKul Jun 29, 2023
8e75241
find deep in file listing
ErykKul Jul 4, 2023
ca5b28c
Merge branch 'develop' into 9683_get_dataset_api_in_single_query
ErykKul Sep 1, 2023
2f98f23
Merge branch 'IQSS:develop' into 9683_get_dataset_api_in_single_query
ErykKul Sep 6, 2023
53d0d85
Merge branch 'IQSS:develop' into 9683_get_dataset_api_in_single_query
ErykKul Sep 19, 2023
9599f44
merged develop branch
ErykKul Apr 17, 2024
773e2e2
fixed error type: badRequest -> notFound
ErykKul Apr 17, 2024
6d0a1ba
if global Id not found, try alt global Id
ErykKul Apr 19, 2024
f558e33
Merge branch 'IQSS:develop' into 9683_get_dataset_api_in_single_query
ErykKul Apr 22, 2024
60c53af
find deep disabled for the files API
ErykKul Apr 22, 2024
1eea73c
Merge branch 'IQSS:develop' into 9683_get_dataset_api_in_single_query
ErykKul May 2, 2024
3b97add
removed newline
ErykKul May 6, 2024
fb4eb8b
Merge branch 'IQSS:develop' into 9683_get_dataset_api_in_single_query
ErykKul May 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,8 @@ public Dataset findDeep(Object pk) {
.setHint("eclipselink.left-join-fetch", "o.files.dataFileTags")
.setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas")
.setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas.fileCategories")
//.setHint("eclipselink.left-join-fetch", "o.files.guestbookResponses")
.setHint("eclipselink.left-join-fetch", "o.files.fileMetadatas.varGroups")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this part of the optimization here? I would guess this isn't often accessed, so I'm not sure how much of a performance help it would be.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had seen in in the query log even when it was not used. For large datasets it does make a difference (one query less for each file).

//.setHint("eclipselink.left-join-fetch", "o.files.guestbookResponses
.setHint("eclipselink.left-join-fetch", "o.files.embargo")
.setHint("eclipselink.left-join-fetch", "o.files.retention")
.setHint("eclipselink.left-join-fetch", "o.files.fileAccessRequests")
Expand Down
40 changes: 30 additions & 10 deletions src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import edu.harvard.iq.dataverse.engine.command.exception.RateLimitCommandException;
import edu.harvard.iq.dataverse.externaltools.ExternalToolServiceBean;
import edu.harvard.iq.dataverse.license.LicenseServiceBean;
import edu.harvard.iq.dataverse.pidproviders.PidUtil;
import edu.harvard.iq.dataverse.locality.StorageSiteServiceBean;
import edu.harvard.iq.dataverse.metrics.MetricsServiceBean;
import edu.harvard.iq.dataverse.search.savedsearch.SavedSearchServiceBean;
Expand Down Expand Up @@ -371,31 +372,50 @@ protected DataverseLinkingDataverse findDataverseLinkingDataverseOrDie(String da
}

protected Dataset findDatasetOrDie(String id) throws WrappedResponse {
return findDatasetOrDie(id, false);
}

protected Dataset findDatasetOrDie(String id, boolean deep) throws WrappedResponse {
Long datasetId;
Dataset dataset;
if (id.equals(PERSISTENT_ID_KEY)) {
String persistentId = getRequestParameter(PERSISTENT_ID_KEY.substring(1));
if (persistentId == null) {
throw new WrappedResponse(
badRequest(BundleUtil.getStringFromBundle("find.dataset.error.dataset_id_is_null", Collections.singletonList(PERSISTENT_ID_KEY.substring(1)))));
}
dataset = datasetSvc.findByGlobalId(persistentId);
if (dataset == null) {
throw new WrappedResponse(notFound(BundleUtil.getStringFromBundle("find.dataset.error.dataset.not.found.persistentId", Collections.singletonList(persistentId))));
GlobalId globalId;
try {
globalId = PidUtil.parseAsGlobalID(persistentId);
} catch (IllegalArgumentException e) {
throw new WrappedResponse(
badRequest(BundleUtil.getStringFromBundle("find.dataset.error.dataset.not.found.bad.id", Collections.singletonList(persistentId))));
}
datasetId = dvObjSvc.findIdByGlobalId(globalId, DvObject.DType.Dataset);
ErykKul marked this conversation as resolved.
Show resolved Hide resolved
if (datasetId == null) {
datasetId = dvObjSvc.findIdByAltGlobalId(globalId, DvObject.DType.Dataset);
}
if (datasetId == null) {
throw new WrappedResponse(
notFound(BundleUtil.getStringFromBundle("find.dataset.error.dataset_id_is_null", Collections.singletonList(PERSISTENT_ID_KEY.substring(1)))));
}
return dataset;

} else {
try {
dataset = datasetSvc.find(Long.parseLong(id));
if (dataset == null) {
throw new WrappedResponse(notFound(BundleUtil.getStringFromBundle("find.dataset.error.dataset.not.found.id", Collections.singletonList(id))));
}
return dataset;
datasetId = Long.parseLong(id);
} catch (NumberFormatException nfe) {
throw new WrappedResponse(
badRequest(BundleUtil.getStringFromBundle("find.dataset.error.dataset.not.found.bad.id", Collections.singletonList(id))));
}
}
if (deep) {
dataset = datasetSvc.findDeep(datasetId);
} else {
dataset = datasetSvc.find(datasetId);
}
if (dataset == null) {
throw new WrappedResponse(notFound(BundleUtil.getStringFromBundle("find.dataset.error.dataset.not.found.id", Collections.singletonList(id))));
}
return dataset;
}

protected DatasetVersion findDatasetVersionOrDie(final DataverseRequest req, String versionNumber, final Dataset ds, boolean includeDeaccessioned, boolean checkPermsWhenDeaccessioned) throws WrappedResponse {
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/edu/harvard/iq/dataverse/api/Datasets.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ public interface DsVersionHandler<T> {
@Path("{id}")
public Response getDataset(@Context ContainerRequestContext crc, @PathParam("id") String id, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response, @QueryParam("returnOwners") boolean returnOwners) {
return response( req -> {
final Dataset retrieved = execCommand(new GetDatasetCommand(req, findDatasetOrDie(id)));
final Dataset retrieved = execCommand(new GetDatasetCommand(req, findDatasetOrDie(id, true)));
final DatasetVersion latest = execCommand(new GetLatestAccessibleDatasetVersionCommand(req, retrieved));
final JsonObjectBuilder jsonbuilder = json(retrieved, returnOwners);
//Report MDC if this is a released version (could be draft if user has access, or user may not have access at all and is not getting metadata beyond the minimum)
Expand Down Expand Up @@ -480,7 +480,7 @@ public Response getVersionFiles(@Context ContainerRequestContext crc,
@Context UriInfo uriInfo,
@Context HttpHeaders headers) {
return response(req -> {
DatasetVersion datasetVersion = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers, includeDeaccessioned);
DatasetVersion datasetVersion = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId, false), uriInfo, headers, includeDeaccessioned);
DatasetVersionFilesServiceBean.FileOrderCriteria fileOrderCriteria;
try {
fileOrderCriteria = orderCriteria != null ? DatasetVersionFilesServiceBean.FileOrderCriteria.valueOf(orderCriteria) : DatasetVersionFilesServiceBean.FileOrderCriteria.NameAZ;
Expand Down
Loading