Skip to content

release 53.0

Compare
Choose a tag to compare
@dozy dozy released this 21 Jan 16:23
· 1256 commits to master since this release
  • a wrapper object npg_pipeline::product to represent a product
  • use products attribute to drive p4_stage1, seq_alignment and autoqc
  • create composition.json files to guide archiving
  • p4 params files for seq_alignment moved from no_cal/laneN to no_cal
    (changes run folder structure when merging lanes)
  • cluster_count and seqchksum_comparator checks now done at run level instead
    of lane level
  • upfront definition of all products
  • generic runfolder scaffolding for any products
  • since the top-level qc directory is no longer required, the tileviz
    directory is moved to the analysis directory
  • reshuffle of roles in npg_pipeline::roles:
    npg_pipeline::roles::business::base merged into npg_pipeline::base;
    npg_pipeline::roles::business::flag_options moved to
    npg_pipeline::base::options, a number of pipeline options from other
    modules moved to this role;
    npg_pipeline::roles::accessors moved to npg_pipeline::base::config;
    helper functions moved to a new role - npg_pipeline::function::util
  • ref_adapter_pre_exec_string method renamed to repos_pre_exec_string
  • metadata_cache_dir method, formerly in npg_pipeline::roles::business::base,
    removed; npg_pipeline::function::p4_stage1_analysis module, the only user
    of this function, switched to use the relevant accessor from the
    npg_pipeline::runfolder_scaffold role
  • minor changes for bcfstats qc check
  • executor type (lsf or wr) can be specified in the configuration file
  • wr executor:
    set per-job priority;
    increase priority for p4 stage 1 job and its predecessors;
    set priority of status and start-stop jobs to zero so that
    they are executed immediately, but still within dependencies
    and memory constraints;
    map queues to arbitrary wr options, in particular, a special queue
    for p4_stage1 maps to a specific cloud host flavour
  • correction of build method for rpt_list attribute in product
  • make bam_cluster_count_check pipeline job dependent on
    qc_spatial_filter (in function_list_central.json)
  • archival daemon - limit number of simultaneously archived NovaSeq runs
  • wr executor - explicitly propagate pipeline's environment to jobs
  • illumina archiver job:
    exclude discontinued verbose attribute and paths that are not needed
    for the minimal work this loader is doing now;
    remove LSF preexec requesting that the job is a unique runner since
    db queries are much simpler now
  • change signature of the autoqc archival job in line with extended
    functionality of the autoqc db loader (ability to find JSON files
    in the run folder)
  • change components_as_products method of npg_pipeline::product to
    return a list with one item when there is only one component in
    the composition (instead of an empty list)
  • tileviz index file with links to lane-level tileviz reports is created
  • seq_alignment supports HISAT2 aligner for RNA libraries
  • explicit iRODS destination collection is set for iRODS loaders,
    /seq/illumina/runs/RUN_ID for NovaSeq runs and /seq/RUN_ID
    for the rest
  • explicitly use iRODS loader from an 'old' dated directory for
    old style runfolders
  • a new function, archive_run_data_to_irods, to publish run-level non-product data to iRODS
  • modify run_data_to_irods_archiver module to ensure the interop files go to a dedicated directory
  • additional tags for NovaSeq in dbic_fixtures