Colorscheme1.1 Heatmap

[lanzhenw]

As a user, I can configure the heatmap field and float field colorscale through either providing a named colorscale or manually defining a value color range list. The field will first look for field specific colorscale defined, and then check defaultColorscale of the session.

Set colorscale for a field on UI:

global-field.mov

Set colorscale with manual defined list:

manual.list.mov

Set colorscale for a float type field to render the embeddings:

float_embeddings.mov

Example schema:

        import fiftyone as fo
        import fiftyone.zoo as foz

        dataset=foz.load_zoo_dataset("quickstart")

        # Store a custom color scheme for a dataset
        dataset.app_config.color_scheme = fo.ColorScheme(
            color_by="field",
            color_pool=["#ff0000", "#00ff00", "#0000ff"],
            fields=[
                {
                    "path": "ground_truth",
                    "fieldColor": "#ff00ff",
                    "colorByAttribute": "label",
                    "valueColors": [{"value": "dog", "color": "yellow"}],
                    "maskTargetsColors": [{"intTarget": 2, "color": "#ff0000"}, {"intTarget": 12, "color": "#99ff00"}],
                }
            ],
            label_tags={
                "fieldColor": "#00ffff",
                "valueColors": [
                    {"value": "correct", "color": "#ff00ff"},
                    {"value": "mistake", "color": "#00ff00"},
                ]
            },
            colorscales = [

                {
                    "path": "float_field",
                    "list": [{"value": 0, "color": 'rgb(0, 0, 255)'}, {"value": 1, "color": 'rgb(0, 255, 255)'}],
                },
                {
                    "path": "heatmap",
                    "name": "hsv",
                },
            ],
            multicolor_keypoints=False,
            opacity=0.5,
            show_skeletons=True,
            default_mask_targets_colors=[{"intTarget": 1, "color": "#FEC0AA" }, {"intTarget": 2, "color": "#EC4E20"}],
            default_colorscale={"name": "sunset", "list": None}
        )

        session = fo.launch_app(dataset)

What changes are proposed in this pull request?

This PR also includes changes in the backend session workflow of how colorscheme is handled.
We added id field to the object (needs migration)

How is this patch tested? If it is not, please explain why.

• I can define a default colorscale for the dataset. If a field does not have specific setting, default colorscale should be used.
• I can define field level colorscale for float field and heatmap field.
• Json viewer should capture instant changes. I can modify the schema in json viewer directly.
• I can save a schema to the dataset. When loading a new session, it will use the dataset default setting.

How to create a dataset with heatmap?

import os
import numpy as np
import fiftyone as fo
import fiftyone.zoo as foz

def random_kernel(metadata):
    h = metadata.height // 2
    w = metadata.width // 2
    sign = np.sign(np.random.randn())
    x, y = np.meshgrid(np.linspace(-1, 1, w), np.linspace(-1, 1, h))
    x0, y0 = np.random.random(2) - 0.5
    kernel = sign * np.exp(-np.sqrt((x - x0) ** 2 + (y - y0) ** 2))
    return fo.Heatmap(map=kernel, range=[-1, 1])

dataset = foz.load_zoo_dataset("quickstart").select_fields().clone()
dataset.compute_metadata()

for sample in dataset:
    heatmap = random_kernel(sample.metadata)

    # Convert to on-disk
    map_path = os.path.join("/tmp/heatmaps", os.path.basename(sample.filepath))
    heatmap.export_map(map_path, update=True)

    sample["heatmap"] = heatmap
    sample.save()

session = fo.launch_app(dataset)

bugs:

• Regression: session.color_scheme = fo.Colorscheme(color_by="field",
  color_pool=["#ff0000", "#00ff00", "#0000ff"]) is not working.

• Initial load: rgb is not loaded until we open the color panel and change certain settings

• datasetColorscheme fragment does not reliably update when save as default

• Release Notes

Is this a user-facing change that should be mentioned in the release notes?

• No. You can skip the rest of this section.
• Yes. Give a description of this change to be included in the release
  notes for FiftyOne users.

(Details in 1-2 sentences. You can just refer to another PR with a description
if this PR is part of a larger change.)

What areas of FiftyOne does this PR affect?

• App: FiftyOne application changes
• Build: Build and test infrastructure changes
• Core: Core fiftyone Python library changes
• Documentation: FiftyOne documentation changes
• Other

[benjaminpkane]

I defer to @manivoxel51 on approval. Code wise, this looks pretty complete! Any fixes could happen in separate PRs from my perspective

[codecov]

Codecov Report

Attention: 6823 lines in your changes are missing coverage. Please review.

Comparison is base (beaf840) 15.63% compared to head (00088fa) 15.66%.
Report is 6 commits behind head on develop.

Files	Patch %	Lines
app/packages/looker/src/elements/imavid/index.ts	6.04%	435 Missing ⚠️
.../components/ColorModal/colorPalette/Colorscale.tsx	0.00%	263 Missing ⚠️
...nents/ColorModal/controls/ManualColorScaleList.tsx	0.00%	262 Missing ⚠️
.../components/ColorModal/controls/ColorscaleList.tsx	0.00%	253 Missing ⚠️
...p/packages/looker/src/lookers/imavid/controller.ts	4.89%	233 Missing ⚠️
...src/components/Filters/StringFilter/Checkboxes.tsx	0.00%	208 Missing ⚠️
...p/packages/core/src/components/ColorModal/utils.ts	0.00%	187 Missing ⚠️
...nents/ColorModal/colorPalette/GlobalColorscale.tsx	0.00%	176 Missing ⚠️
app/packages/looker/src/lookers/imavid/index.ts	11.35%	164 Missing ⚠️
app/packages/state/src/hooks/useCreateLooker.ts	9.52%	152 Missing ⚠️
... and 148 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3804      +/-   ##
===========================================
+ Coverage    15.63%   15.66%   +0.03%     
===========================================
  Files          672      726      +54     
  Lines        77832    80365    +2533     
  Branches      1041     1069      +28     
===========================================
+ Hits         12168    12593     +425     
- Misses       65664    67772    +2108     

Flag	Coverage Δ
app	15.66% <17.09%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[manivoxel51]

Nice work!💥

Tested the app with heatmaps and didn't notice any blocking issues. I agree that we can crush any known bugs as a followup PR.

[manivoxel51]

@lanzhenw, @benjaminpkane e2e is failing - can we get those fixed before merging?

[manivoxel51]

Two more things before merging

• Please remove all console.logs

• nit: there are 00 in the list value input.

e use single 0

[lanzhenw]

@manivoxel51 I fixed the e2e tests and the typing issue.

typing 000 => 0
typing 02 => 2
typing 0.00 => 0.00
typing 0 => 0

Screen.Recording.2023-11-20.at.8.22.06.AM.mov

[manivoxel51]

we'll investigate the failing test - locally passing - CI failing