[STABLE] Fix job file stageout interruptibility

[natefoo]

The job files api is used by Pulsar when Pulsar is configured to initiate staging operations on the remote end. Without the upload module, restarting processes while a stageout is occurring can result in job failure (or possibly retrying the stageout from the beginning, a costly operation when the file is large).

Additionally, the staged out files are moved from their temporary received paths to their final destinations. This prevents the temporary space from growing excessively large and also causes the move to be instantaneous source and destination are on the same filesystem.

Here is the corresponding nginx config:

location /_job_files {
    # Pass non-stageouts directly to the API method
    if ($request_method != POST) {
        rewrite "" /api/jobs/$arg_job_id/files last;
    }
    upload_store /galaxy/upload_job_files;
    upload_store_access user:rw; # group:rw all:rw;
    upload_pass_form_field "";
    upload_set_form_field "__${upload_field_name}_path" "$upload_tmp_path";
    upload_pass_args on;
    upload_pass /_upload_job_files_done;
}

location /_upload_job_files_done {
    internal;
    rewrite "" /api/jobs/$arg_job_id/files;
}

The nginx config should work if you have set the following in galaxy.ini:

nginx_upload_job_files_store = /galaxy/upload_job_files
nginx_upload_job_files_path = /_job_files

Despite the warnings on IfIsEvil, this use of if in the nginx config appears to be safe.

Related trello card: https://trello.com/c/usD7jzcT

[blankenberg]

👍

[jmchilton]

Fix - huh? I think you may be pulling a Blankenberg :).

Nonetheless 👍

[natefoo]

Fix - huh? I think you may be pulling a Blankenberg :).

Shhhhhh.

[blankenberg]

Hey, wait, I uh...ok..

[jmchilton]

We broke non-nginx uploads here. The files got lost in the endpoint description.