Object store cache file not updated after job completes with outputs_to_working_directory

[mvdbeek]

from @selten on element:

I recently upgraded to Galaxy 23.0. Additionally I added an extra object store to Galaxy (Amazon S3). I have outputs_to_working_directory set to true. When a file is uploaded, Galaxy will immediately create the file on S3 and in the object store cache, this is set to 0 bytes. Once the upload tool finishes, Galaxy will upload the file to S3 but it seemingly doesn't update the cached file to the new version. Has anyone seen this type of behaviour? Is there a known misconfiguration on my end? The object_store_cache isn't writable directly from the job, could that cause this issue?

I think we may simply not be doing that right. Definitely needs at least a test case.

[selten]

The extended metadata strategy seems to play a role here. Now that I switched to directory, it all seems to work better again.

[mvdbeek]

awesome though that you were using extended, I'll prioritize a fix

[mvdbeek]

The object_store_cache isn't writable directly from the job, could that cause this issue?

that could be an issue yes. The metadata script attempts to update the cache location, but it does not fail when the file couldn't be updated (

galaxy/lib/galaxy/objectstore/s3.py

Line 680 in cbb0119

log.exception("Trouble copying source file '%s' to cache '%s'", source_file, cache_file)

).

I've written a simple test that checks whether the cache gets updated, and it does, but that does indeed require write access to the cache location.

I think the way this works currently is convenient because for a subsequent job the file simply exists in the cache so it won't be pulled in again.
There are 2 things we can do:

• We should check if pre-creating the output in the object store is still required. If it is not (or if this can be set up to not be required) then we can just keep the logic as it is: update if possible from the worker (which might be closer to the data than the job handler, so in principle that might be preferrable). If the worker fails to push the file at least it won't exist in the cache and will be pulled in ... when submitting the next job ?
• We could check that the object in the object store matches the cached file ... this is probably only feasible in the job finish code