Use public inputs in example files

[dshiga]

Previously, the example inputs files were pointing to a private bucket in our dev project. This PR changes them to point to inputs in a public bucket instead.

See:
https://elastc.com/c/QN-0A7Mr/539-remove-environment-specific-params-from-example-inputs

Related PR: HumanCellAtlas/pipeline-tools#18

[samanehsan]

Looks good to me!

[samanehsan]

The inputs for optimus have changed so there are now three different arrays for fastq files instead of an array of arrays: https://github.com/HumanCellAtlas/skylab/blob/master/pipelines/optimus/Optimus.wdl#L16

I just noticed this after looking at Marcus's PR that changes the test inputs file: https://github.com/HumanCellAtlas/skylab/pull/86/files#diff-c53d14ec9a922a0786f2647a9480b7b3R6

[ambrosejcarr]

Confirmed, could you please fix my screw-up and update this to be:

  "Optimus.r1": [
      "gs://hca-dcp-mint-test-data/10x/demo/fastqs/pbmc8k_S1_L007_R1_001.fastq.gz",
      "gs://hca-dcp-mint-test-data/10x/demo/fastqs/pbmc8k_S1_L007_R1_001.fastq.gz"
  ],
  "Optimus.r2": [
      "gs://hca-dcp-mint-test-data/10x/demo/fastqs/pbmc8k_S1_L007_R2_001.fastq.gz",
      "gs://hca-dcp-mint-test-data/10x/demo/fastqs/pbmc8k_S1_L007_R2_001.fastq.gz"
  ],
  "Optimus.i1": [
      "gs://hca-dcp-mint-test-data/10x/demo/fastqs/pbmc8k_S1_L007_I1_001.fastq.gz",
      "gs://hca-dcp-mint-test-data/10x/demo/fastqs/pbmc8k_S1_L007_I1_001.fastq.gz"
  ]

[samanehsan]

@ambrosejcarr should this be data from the 10x or Optimus folder here?

[ambrosejcarr]

10x is the data type, so is appropriate here.

[dshiga]

Okay, I fixed that, could use another look @ambrosejcarr.

[ambrosejcarr]

I guess this is fine to do find/replace, but in the example that was given, they point to real files that a mint dev could inspect, whereas the changed files don't exist.

Other opinions on this @samanehsan? I guess for public users the existence of these files is a bit pointless since they can't access them anyway (unless we moved them the hca-dcp-mint-test-data bucket

[dshiga]

I actually copied over those output files to the public bucket. Are you sure they're not there? I may have missed some or given the wrong paths here.

[ambrosejcarr]

Oops, my fault! Sorry. You're right.

[ambrosejcarr]

This looks good to me. One comment that does not directly relate to this, is that I would prefer we didn't propagate the disorganization from the other bucket.
@jishuxu can you confirm that the statements I make below are accurate?

These are all human, and should have been generated from the GRCh38_Gencode files:

gs://hca-dcp-mint-test-data/reference/rsem_ref/
gs://hca-dcp-mint-test-data/reference/GRCh38_gencode.v27.refFlat.txt
gs://hca-dcp-mint-test-data/reference/gencode.v27.rRNA.interval_list

If accurate, lets move them into gs://hca-dcp-mint-test-data/reference/GRCh38_gencode

Second, we should eventually remove the below index and replace it with GRCh38, since we're going to use GENCODE.

gs://hca-dcp-mint-test-data/reference/Hg19_kco/

[jishuxu]

Yes, we can move these annotation files into one folder. As for the kco reference bundle, we probably won't use them any more. it is okay to remove from our system.

[dshiga]

jenkins retest