Entropy panel mk2

[jameshadfield]

For the full story please read through the commit messages. I'll try to highlight the main UI changes here, but perhaps it's better to just jump in and use the review app with fresh eyes.

Main UI changes

Overlapping CDSs are better shown

Click for details

By using opacity and jittering them based on the frame the CDS is in. The aim is not to have a perfect genome viewer (e.g. see how much space a "proper" genome browser needs!) but to expose what's there and allow you to hover over them / click them (clicking a CDS selects it, see next section).

nCoV, current Auspice on top, this PR on bottom; much better communication of the CDSs present (they were always there in the JSON, just visually obscured).

Hovering over the CDSs is now possible too:

Selecting a CDS focuses the entropy panel on the translated CDS

Click for details

When viewing a CDS we now have the main axis represent this and only this CDS, always shown going L→R, with tick labels representing amino acids. This better represents biology as we are able to ignore any artifacts upstream of a translation. The UI to view a CDS is to either click on a CDS or select a genotype color-by for that CDS.

This screenshot (top, bottom) shows the pre-capsid CDS for HepB. In the top view other CDSs are overlapping so most of the bars you see are from a different CDS (pre-capsid CDS is the orange coloured CDS, and then the green CDS is overlapping), and it was not possible to see the actual diversity of pre-capsid¹.

¹ I guess in this example the CDSs are in frame, so the diversity is identical, but barcharts from different CDSs obscuring each other has always been a problem in auspice

Segmented CDSs (nCoV example)

Click for details

Top view is our current genome annotation for nCoV, bottom is an example of what it could be (incomplete, as I've only added in 2 of the 16 proteins cleaved from the polyproteins)

Specifically, we can now view ORF1ab (PP1ab) in its entirety:

(The main axis now has two segments, which is what we have been referring to as ORF1a and ORF1b, separated by a -1 ribosomal frameshift.)

And since overlapping CDSs are better shown we can (finally!) show all the proteins which are produced, e.g. RdRp (a complex one, since it spans the -1 frameshift). The entropy panel conveys the expected lack of diversity here well I think.

CDSs which wrap around the origin (HepB example)

Click for details

Most of HepB's CDSs wrap the origin! Displaying the translation (i.e. the top half of the panel) is no different from any segmented gene, it's the zooming that's tricky here. My solution was to ~invert the brush; perhaps easier to explore than to explain!

Segmented CDSs may have codons which span multiple segments (referred to as the phase of the segment), and we handle this well now:

Entropy panel state is decoupled from the color-by

Click for details

Previously the entropy panel was tightly coupled to the color-by. It is still linked to it, but it can also operate independently if (and only if) you start interacting with it. Any change to the (genotype) color-by will bring it back in sync. For instance, I can now select Spike 452 color-by (which will always jump to that position in the entropy panel + select Spike), and then use the UI to select another CDS in the entropy panel without changing the colorBy. If (via the sidebar) I change to a different Spike position then the entropy panel will jump back to show Spike.

Proposed JSON schema

The current proposal is mostly backwards compatible, except segmented CDSs are ignored by previous Auspice versions. For instance here's how ORF1ab looks:

  "ORF1ab": {
    "gene": "Multiple CDSs with the same gene will be grouped together",
    "strand": "+",
    "segments":[
      {"start": 266, "end": 13468},
      {"start": 13468, "end": 21555}
    ],
    "display_name": "Text here is shown when hovering on the CDSs"
  },

And you can specify a CDS which wraps around the origin by using an end value beyond the end value of the genome (or create the segments yourself, either way works):

"nuc": {
  "start": 1, "end": 3182, "strand": "+"
},
"envL": {
  "start": 2850, "end": 4019, "strand": "+", "gene": "S"
},

Test datasets

Available via the review app:

• Moneypox (no changes from staging site version) - useful for a genome with lots of CDSs
• Zika (no changes from staging site version) - a very simple genome
• HepB - a short genome but complex: overlapping CDSs, many CDSs which wrap the origin
• ncov - A modified nCoV dataset which removes our ORF1a / 1b artifact and shows the ORF1ab polyprotein (2 segments, one with a -1 frameshift), the PP1a polyprotein (aka ORF1a) and 2 of the 16 proteins cleaved out of those polyproteins. Created using a small script to modify a nextclade nCoV tree so it shouldn't be taken as 100% perfect!

Things still to do (either here or in subsequent work):

styling

• The y-jitter (currently using the CDS's frame) still results in hard-to-click / hard-to-see CDSs. I'm not sure of the ideal solution here, but we could probably allow a little more height in overall entropy panel (perhaps calculated at load time depending on how complicated the genome map is). Please come up with good suggestions!!!
• I must better communicate which CDS is selected - currently I change the text/stroke colour, but it's not clear enough.
• CDS text overlapping
• "Simple" genomes (e.g. zika) have too much whitespace because all CDSs are in a single frame
• The zoom elements are a little bit too small - originally the grey brush element extended above the annotations (but behind them), but this obscured the fact that you could pan the zoom by dragging the grey part. Arguably it's so small now it's still obscured.

Negative strand CDSs

• All that's required is writing the code to handle the projection from genome space to protein space. And a whole lot of testing.

misc

• I've done a lot here to remove duplicate / conflicting redux state but there is one which still exists: state.metadata.genomeAnnotations (used by the tree info boxes to order the genes).
• Sometimes the entropy zoom URL params aren't quite right (e.g. missing when they should be there or vice versa)
• Button to expand zoom (where possible)
• Remove the wireframes / console.log statements
• Help popup (similar to the ones we have in the sidebar)
• Failing tests
• Changelog
• Add documentation / tutorial? (I won't let this hold up the PR, but it is important)

future work?

• For nucleotide positions we used to check if it fell inside a gene, which allowed on-hover info boxes to show information such as "Nucleotide x (Codon y in protein z)". This never considered the case where a nucleotide fell in multiple genes (CDSs), so it was somewhat misleading. It was a generally useful idea, so we should bring it back in an improved form. It's plausible that we could find the AA positions for all CDSs which overlapped that nucleotide.
• Main axis uses ticks which go above and below the axis, with an additional label to communicate the corresponding nt position in the genome (information which exists in the on-hover box)
• Mousewheel zooming has been disabled. I should bring it back, but it needs a proper debounce.

Issues closed

Closes #1596
Closes #1651
Closes #1145
Closes #299
Closes #1516

[rneher]

this is very cool! Looking forward to finally having this in auspice!

A few points, some of which you touch upon above.

• When using the MPXV build, the zoom elements for the CDS get very small. We might need some way to zoom into the genome first. Or maybe put the zoom elements for the CDS on the CDS and keep the genome zoom below?
• as you say, negative strand CDS are not properly accounted for. The majority of MPXV genes in the first third of the genome are on the negative strand. Auspice used to know about that.
• we could add localAnnotation to the CDS along side the ranges to specify things that should only be shown in the CDS view, like RBD.

[jameshadfield]

Thanks @rneher

-ve strand CDSs are now working, except that zooming within a CDS is backwards (will fix tomorrow). I've abandoned the use of frame for the y-offsets and instead just stack them as efficiently as possible to avoid any CDSs overlapping (in the bottom view, anyways, the top view used too much space that way so they overlap a little bit).

we could add localAnnotation to the CDS along side the ranges to specify things that should only be shown in the CDS view, like RBD.

Yeah! I played around with this yesterday, and I think it'll be very helpful. Here's how it looked for Spike (clicking on the local annotation toggled the highlighting). I'm going to leave it out of this PR, or at least get this PR finished before adding it in. I'd have confidence that we should add it to the JSON spec now tho.

[jameshadfield]

Thanks for all the reviews. I think this PR is at a good point to wrap up and release which I plan to do ~early next week (and we can keep improving this after merge if we notice things).

Would it be possible to add an AA or NT label for the plotted units?

Good idea! I've added a AA pos label when a CDS is selected, but having "NT" felt a bit too cramped. I've also made the panel title indicate whether we're viewing the genome or a CDS, and the little help icon also explains the coordinates.

Could it be a unidirectional mouse cursor when at the end of the rope? ... Giving handle triangles a white center and gray outline could help to indicate that they are draggable.

Yup! Both done.

The two labels for ORF1ab ... are a bit confusing to me. I think these should just be "ORF1a" and "ORF1b"

(Discussed elsewhere) the best way forward is to continue to annotate these as separate CDSs, rather than a single segmented CDS which I demoed here. This allows people to continue to label mutations in an ORF1b coordinate system which has become quite common.

How difficult would it be to draw faint lines from the gray bounding box on the bottom track to the left and right edges of the top track?

Is there a reason for why CDSs are strictly rectangles in the lower track, while they have the helpful arrows on the top track?

Both possible -- I tried the arrows approach on the lower track and it looked quite messy (lots of aliasing), but I'm sure if we played around with it enough it'd work. I haven't tried the faint lines but think it's worth pursuing. The zooming approach will need further work to support large (>200kb) genomes well.

I'd think replacing "↵ nuc" with "RESET ZOOM" (or "RESET LAYOUT") would be more appropriate.

Done. It's a bridge too far to make this greyed out at the present moment, but we can tackle that aspect later on.