feat: Add code for validating fragement file

[Bento007]

Reason for Change

• prototype cellxgene schema CLI changes needed to validate fragments single-cell#721
• All validataion step related to fragments have been implemented

Changes

• add cli command

$ cellxgene-schema process-fragment -h                                                                                                    
Usage: cellxgene-schema process-fragment [OPTIONS] H5AD_FILE FRAGMENT_FILE

  Check that an ATAC SEQ fragment follows the cellxgene data integration schema. If
  validation fails this command will return an exit status of 1 otherwise 0. When
  the '--generate-index' tag is present, the command will generate a tabix
  compatible version of the fragment and tabix index. The generated fragment will
  have the file suffix .bgz and the index will have the file suffix .bgz.tbi.

Options:
  -i, --generate-index  Generate index for fragment
  -h, --help            Show this message and exit.

• add requirements to run dask in local distributed mode
• add validation steps for ATAC-SEQ fragment file
• add tabix index generation steps.

Testing

• added unit test for all validation steps.

pip install git+https://github.com/chanzuckerberg/single-cell-curation/@tsmith/10x-ATAC#subdirectory=cellxgene_schema_cli

Remaining Work

chanzuckerberg/single-cell#724

[github-actions]

This PR has not seen any activity in the past 2 weeks; if no one comments or reviews it in the next 3 days, this PR will be closed.

[github-actions]

This PR was closed because it has been inactive for 17 days, 3 days since being marked as stale. Please re-open if you still need this to be addressed.

[github-actions]

This PR has not seen any activity in the past 2 weeks; if no one comments or reviews it in the next 3 days, this PR will be closed.

[github-actions]

This PR has not seen any activity in the past 2 weeks; if no one comments or reviews it in the next 3 days, this PR will be closed.

[codecov]

Codecov Report

Attention: Patch coverage is 89.14286% with 19 lines in your changes missing coverage. Please review.

Project coverage is 89.78%. Comparing base (b7e96bf) to head (873a155).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1095      +/-   ##
==========================================
- Coverage   89.88%   89.78%   -0.10%     
==========================================
  Files          19       20       +1     
  Lines        2194     2369     +175     
==========================================
+ Hits         1972     2127     +155     
- Misses        222      242      +20     

Components	Coverage Δ
cellxgene_schema_cli	90.60% <89.14%> (-0.23%)	⬇️
migration_assistant	91.26% <ø> (ø)
schema_bump_dry_run_genes	79.74% <ø> (ø)
schema_bump_dry_run_ontologies	99.53% <ø> (ø)

[ivirshup]

Suggested change

	obs = ad.read_h5ad(anndata_file, backed="r").obs
	with h5py.File(anndata_file) as f:
	obs = ad.io.read_elem(f["obs"])

^ This will read less, and could be significantly faster depending on how much other data is in the file.

If you are on anndata < 0.11 read_elem is under anndata.experimental

Similar below

[ivirshup]

I think it looks good! I can help with the dask TODOs if you'd like.

I would like to see at least one test that reads and validates the output bgz/ tbi files. I would like this test to be parameterized over both index implementations (notably, write_bgzip_cli is not covered by tests). I would probably make the indexing method explicitly selectable to make this easier to test.

Broadly, I think more tests checking the processed outputs would be good (I think there's just test_process_fragment right now?).

Some ideas:

• Check that the output file is actually sorted as we expect
• Access the file via the index and make sure the results are as expected

[ivirshup]

The docs for this say:

The contents of filename have to be sorted by contig and position - the method does not check if the file is sorted.

The input you are providing here should be contiguous by contig, but not necessarily sorted since we don't know the order in which the tasks will complete. Right?

I assume this is fine, but maybe a comment noting the behavior is different from the docs?

[Bento007]

added a comment in prepare_fragment

[nayib-jose-gloria]

how was this number determined? are we planning on running this many cores for the atac-seq validation container?

[Bento007]

This is number of jobs we send to the dask scheduler at a time. The dask configuration will determine the compute used. The value of 4 was determined by running the sample fragment a few times with a decreasing number of jobs until it didn't freeze.

[ivirshup]

Suggested change

	temp_data = Path(write_path) / f"temp_{chromosome}.tsv"
	temp_data = Path(write_path) / f"temp_{chromosome}.tsv.gz"

I think this file should be compressed to keep storage requirements during processing down.

Ideally, I would use a better compressor than GZIP (like zstd) but I'm not sure what you'll have installed here.