Generate derivatives for FileSets when Valkyrie mode is enabled

[tpendragon]

Descriptive summary

The pub/sub and transaction architecture doesn't currently generate derivatives for Valkyrie resources, resulting in no thumbnails or manifests.

Rationale

Valkyrie resources should be able to generate derivatives.

Expected behavior

Derivatives generate.

Actual behavior

Derivatives don't generate.

Steps to reproduce the behavior

Prerequisites

Configure Hyrax:

1. Edit /config/initializers/hyrax.rb and set:

  config.collection_model = "Hyrax::PcdmCollection"
  config.query_index_from_valkyrie = true
  config.index_adapter = :solr_index

2. restart rails app

To reproduce

1. Navigate to: Dashboard -> Works
2. Click button: Add new work
3. Select type: Monograph
4. File in required metadata and other required steps
5. Add an image file
6. Click button: Save

This forwards to the new work's show page. Look at the list of items at the bottom. There is no thumbnail.

Related work

Found as part of #5480

[tpendragon]

Notes:

Most derivatives store locally - I think we can just use the logic in CreateDerivativesJob for all of those. The test app doesn't have ffmpeg, but has enough to run PDF and (maybe image) derivatives - so we should be able to run that and make sure the files exist on disk. Unfortunately that's not the case for ExtractedText - extracted text persists to a File node in Fedora. See:

hyrax/app/models/concerns/hyrax/file_set/derivatives.rb

Lines 45 to 47 in f940a32

	Hydra::Derivatives.source_file_service = Hyrax::LocalFileService
	Hydra::Derivatives.output_file_service = Hyrax::PersistDerivatives
	Hydra::Derivatives::FullTextExtract.output_file_service = Hyrax::PersistDirectlyContainedOutputFileService

The PersistDirectlyContainedOutputFileService creates a node in Fedora via ActiveFedora. If we want to maintain that we'll have to find a nice way to replace that logic. I think we can use the Wings storage adapter? I don't know if that'll work right with nurax-pg and a normal storage adapter.

I think my strategy is going to be to get everything but extracted text to work.

[jlhardes]

#4788 - Hyrax::Works do not show thumbnail in Dashboard -> Works might provide helpful info for this issue, even though it seems to be focused on the Monograph work type on nurax-dev that is Wings.

[gamontoya]

@rjkati I believe this is closeable. This is what I'm seeing on Koppie after editing my local hyrax.rb.

[rjkati]

In the Valkyrie update meeting, we determined that thumbnails and derivatives should be displaying. Currently, thumbnails and derivatives are not displaying on koppie or nurax-pg. Softserv will spend time today digging through the nurax-pg code to troubleshoot #5594 and will look at this ticket during that investigation.

[gamontoya]

@rjkati For my purposes, are we expecting to see image thumbnails (not generic place holders) like I see on dev-nurax?

[rjkati]

@gamontoya, yes that's correct. We should see image thumbnails on pg-nurax as well.

[gamontoya]

@rjkati Still not seeing thumbnails on pg-nurax. Can you confirm or should I only test on Koppie after updating hryax.rb?

[tpendragon]

Our understanding is that this is only a problem on nurax-pg, not locally. Whoever grabs this will probably have to connect with @dlpierce

[dunn]

this is working for me on nurax-pg:

[dlpierce]

Thumbnails should be working in koppie/nurax-pg, except that the thumbnails in the catalog view do not show unless the work is edited after ingest. I almost have a PR ready to trigger a work reindex to fix this.

[rjkati]

I'm now seeing thumbnails for images, but not PDFs

Example: https://pg.nurax.samvera.org/concern/generic_works/8765b6ee-db0b-47b9-a5c7-e8efbc3dd4b9?locale=en

[dunn]

@rjkati is that only happening on nurax-pg?

[rjkati]

@dunn yes it's only happening on nurax-pg. I tested this morning locally using koppie and got thumbnails to display for PDFs.

[dlpierce]

Nurax uses the official ruby base image that is based on Debian. Debian still only provides ImageMagick 6 which has restrictions on PDF operations. Koppie uses Alpine which has a more recent ImageMagick.

[rjkati]

@dlpierce for testing purposes, should we expect that there will be no thumbnails for PDFs on pg.nurax (maybe all the nuraxes?) until further notice? If so, I can make a note in the testing spreadsheet to verify thumbnails with images.

[dlpierce]

I changed the PDF policy on Nurax to make this work. There will also be a PR to address reindexing the work after its thumbnail has derivatives created.