lib-classifier: refactor single image viewer with VisXZoom #6390
Conversation
… to SingleImageViewerContainer
mcbouslog
changed the title
|
|
||
| import SVGImage from '../SVGComponents/SVGImage' | ||
|
|
||
| function calculateAdjustedViewBox({ naturalWidth, naturalHeight, transformMatrix }) { |
There was a problem hiding this comment.
I think this calculation needs additional work, specifically to handle zoomed out panning. When zooming out, then panning, the image jumps around erratically.
| <VisXZoom | ||
| height={naturalHeight} | ||
| panning={panning} | ||
| setOnPan={setOnPan} | ||
| setOnZoom={setOnZoom} | ||
| width={naturalWidth} | ||
| zoomConfiguration={DEFAULT_ZOOM_CONFIG} | ||
| zoomingComponent={SingleImageCanvas} | ||
| zooming={zooming} | ||
| {...singleImageCanvasProps} | ||
| /> |
There was a problem hiding this comment.
See https://github.com/zooniverse/front-end-monorepo/blob/master/packages/lib-classifier/src/components/Classifier/components/SubjectViewer/components/ScatterPlotViewer/components/ZoomingScatterPlot/ZoomingScatterPlot.js#L206-L224 for existing implementation of VisXZoom. constrain could potentially be added to the SingleImageViewer.
|
|
||
| - [`VisXZoom`](components/SVGComponents/VisXZoom) - An extension of the VisX library's pan and zoom functional component. Uses a transformation matrix, boolean control for zoom and pan individual to turn the functions on and off as needed, has configurability for zoom direction, minimum zoom, maximum zoom, zoom in value, and zoom out value, and supports zoom to point. Currently implemented with the `ScatterPlotViewer`, the scatter plots used by the `VariableStarViewer`, and the scatter plot used by `DataImageViewer`. It uses the [`ZoomEventLayer`](components/SVGComponents/ZoomEventLayer) which is a transparent SVG rectangle that has event listeners for double click, on wheel, on key down, on mouse down, on mouse up, on mouse enter, and on mouse leave. Subject viewers using this should set as props the width, height, left and top points of the SVG area to render, as well as, the child subject viewer component to render, a zoom configuration object, and optionally a custom [constrain](https://airbnb.io/visx/docs/zoom#Zoom_constrain) function enabling the ability to constrain pan and zoom directionality and/or pan dimensions. An example of this implementation is used with the [`ZoomingScatterPlot`](components/ScatterPlotViewer/ZoomingScatterPlot). The zoom configuration can be set in the workflow configuration [`subject_viewer_config`](https://github.com/zooniverse/front-end-monorepo/blob/master/docs/arch/adr-27.md) object or in the JSON of a JSON subject. (NOTE: TODO is to finish building out support for configuration being set on the workflow configuration object). | ||
| - [`SVGPanZoom`](components/SVGComponents/SVGPanZoom) - A port of the PFE pan and zoom functionality being used with the `SingleImageViewer` and subsequently the `MultiFrameViewer` and anywhere else the `SingleImageViewer` may be used. It does scale transformations on the SVG view box. | ||
| - [`VisXZoom`](components/SVGComponents/VisXZoom) - An extension of the VisX library's pan and zoom functional component. Uses a transformation matrix, boolean control for zoom and pan individual to turn the functions on and off as needed, has configurability for zoom direction, minimum zoom, maximum zoom, zoom in value, and zoom out value, and supports zoom to point. Currently implemented with the `SingleImageViewer`, the single image viewer is used by `DataImageViewer`, `FlipbookViewer`, `ImageAndTextViewer`, `MultiFrameViewer`, and `SeparateFramesViewer`, as well as the `ScatterPlotViewer`, the scatter plots used by the `VariableStarViewer`, and the scatter plot used by `DataImageViewer`. It uses the [`ZoomEventLayer`](components/SVGComponents/ZoomEventLayer) which is a transparent SVG rectangle that has event listeners for double click, on wheel, on key down, on mouse down, on mouse up, on mouse enter, and on mouse leave. Subject viewers using this should set as props the width, height, left and top points of the SVG area to render, as well as, the child subject viewer component to render, a zoom configuration object, and optionally a custom [constrain](https://airbnb.io/visx/docs/zoom#Zoom_constrain) function enabling the ability to constrain pan and zoom directionality and/or pan dimensions. An example of this implementation is used with the [`ZoomingScatterPlot`](components/ScatterPlotViewer/ZoomingScatterPlot). The zoom configuration can be set in the workflow configuration [`subject_viewer_config`](https://github.com/zooniverse/front-end-monorepo/blob/master/docs/arch/adr-27.md) object or in the JSON of a JSON subject. (NOTE: TODO is to finish building out support for configuration being set on the workflow configuration object). |
There was a problem hiding this comment.
Just out of curiosity is the 'todo' at the end in a Github Issue or referenced somewhere else in the code?
There was a problem hiding this comment.
I think #2110 is the related issue. I'm not sure the NOTE belongs in the README though. I could remove the NOTE or update it to include the specific issue?
| const imageLocationUrl = subject?.locations[currentFrame]?.url | ||
|
|
||
| const { img, error, loading, subjectImage } = useSubjectImage({ | ||
| src: imageLocationUrl, |
There was a problem hiding this comment.
Changing this from defaultFrameLocation to imageLocationUrl means useSubjectImage() hook is called on every frame because imageLocationUrl is determined by currentFrame rather than defaultFrame. This is undesired behavior because it directly calls onSubjectReady() every time a volunteer flips to a new frame. onSubjectReady() resets the rotation between frames and pushes new metadata to the classification.
Regarding rotation, we want volunteers to be able to rotate the subject and 'play' the flipbook with a subject rotated. rotation should not be reset in between every frame.
I noticed the metadata bug because earlier this year it was introduced by #5754 and fixed by #5952. On this branch, when I used my flipbook test project (listed in the FEM active projects for reference), this is what a submitted classification looks like when I flipped through 12 frames. Huge metadata causes classification exports to hang.
Questions - Is the change from defaultFrameLocation to imageLocationUrl intentional and required for the new SingleImageViewer with VisXZoom? How can each viewer ensure onSubjectReady() is called only once per subject, and not per frame? I didn't look into ImageAndTextViewer, MultiFrameViewer, and SeparateFramesViewer in detail yet, but I want to include those in this discussion too.
There was a problem hiding this comment.
Questions - Is the change from defaultFrameLocation to imageLocationUrl intentional and required for the new SingleImageViewer with VisXZoom?
I don't think it's required. I'll try refactoring so onSubjectReady (onReady passed from SubjectViewer) is only called once per subject and report back.
|
When Sarah converted the single image viewer to To see this, try drawing while zoomed in on this branch, particularly with any tool that uses dragging motions/gestures. Here's a screen recording with the rectangle tool, dragging the resize handles while zoomed in. You can also reproduce this with the cat face ellipse tool in I Fancy Cats. Screen.Recording.2024-11-22.at.14.33.49.movI think one big barrier to a major refactor was a lack of tests, in turn due to lack of SVG support in |
|
There's an undocumented feature, used by Seabird Watch and Penguin Watch, which you should add to your manual drawing tool tests. When you're making a large number of marks, you don't want to be constantly switching between zoom mode and drawing mode. For this reason, you can control pan-and-zoom with the keyboard, from a focused drawing mark. To see this in action, try placing marks, while zooming and panning with the keyboard, at either of the following links:
You should be able to mark a penguin, then zoom and pan from the keyboard while continuing to mark penguins on the zoomed picture (on crowded pictures, you really need to zoom in, so this feature isn't optional.) Screen.Recording.2024-11-22.at.10.15.02.movThis PR breaks that keyboard accessibility for drawn marks. |
|
One last comment: on this branch, drawn marks scale up with the image when you zoom in. Drawn marks should ignore the scale transformation and stay the same size at all zoom levels (see #6066 for one attempt to fix this for stroke widths. You can also style SVG shapes so that they are independent of scale.) |
|
Thank you @eatyourgreens , the comments and videos are very helpful! To summarize I think you've identified 3 issues:
I'll report back on progress. |
You're welcome. For that first one, I can also reproduce it with the point tool, without drag handles. Drop a point on the image somewhere, then drag and move it while zoomed in. The pointer movement, within the subject viewer, isn't properly mapped to the image. It looks like the pointer coordinates are still scaled to the subject viewer dimensions, not the new, larger image dimensions. Dragging only works when the subject viewer and image are the same size. Could be a missing transformation somewhere on the |
|
I can recreate the noted 2 and 3 issues, but I'm having trouble recreating 1 (marks with drag and move handles aren't scaled and working as expected)? I'm using Chrome v131, but I don't think it'd be browser specific. ZoomInAndDrawRectangle.mp4 |
|
Sorry, I forgot to say that my video was Firefox 132. SVG implementations do vary between browsers, unfortunately. EDIT: I think the drawing tools are only using bits of the SVG 2 spec that are well-supported across the major browser engines. |
|
I just tried a few different browsers. Scaling and dragging seems to be ok in Chrome or Safari, but broken in Firefox. |
|
This might be relevant. Dragging interactions get the current transformation matrix from a Creating a mark here gets the CTM from the These refs don’t point to the same element, so that could account for differences in behaviour between creating a mark and dragging a mark. They probably should be using the same element and transformation matrix. I don't think I've noticed this before. At least, I can't find an issue for it, and I'm sure I would have opened an issue for something like this. |
|
Update: dragging a zoomed-in mark is broken in Firefox on the production release. I'll open an issue. It's not a bug that's been introduced on this branch. The reason is that the Update to the update:
|
| transform={rotationTransform} | ||
| > | ||
| <svg | ||
| ref={canvasLayer} |
There was a problem hiding this comment.
This ref is causing the broken dragging behaviour in Firefox. It causes dragging to get the screen CTM from this svg element, not the rect that's actually used for drawing. I've removed it in #6491, in favour of setting svgContext.canvas inside the InteractionLayer component.
| maxWidth={maxWidth} | ||
| tabIndex={0} | ||
| viewBox={viewBox} | ||
| xmlns='http://www.w3.org/2000/svg' |
There was a problem hiding this comment.
xmlns is ignored by HTML parsers, so you can remove it. See the note at the top of the MDN SVG element page.
Package
Linked Issue and/or Talk Post
Describe your changes
review mobile/pinch handlingnew functionality will be investigated and potentially a separate PRNotes
How to Review
Helpful explanations that will make your reviewer happy:
+to zoom in) doesn't work, but it does in the linked test project, not sure whyChecklist
PR Creator - Please cater the checklist to fit the review needed for your code changes.
PR Reviewer - Use the checklist during your review. Each point should be checkmarked or discussed before PR approval.
General
yarn panic && yarn bootstrapordocker-compose up --buildand FEM works as expectedGeneral UX
Example Staging Project: i-fancy-cats
New Feature
Refactoring