Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dvc exp show reads and combines wrong metrics, params #9956

Open
shcheklein opened this issue Sep 17, 2023 · 2 comments
Open

dvc exp show reads and combines wrong metrics, params #9956

shcheklein opened this issue Sep 17, 2023 · 2 comments
Labels
A: experiments Related to dvc exp bug Did we break something?

Comments

@shcheklein
Copy link
Member

shcheklein commented Sep 17, 2023

Probably the same applies to plots- need to check.

In a monorepo scenario.

We moved everything from the example-get-started-experiments repo into a subdir, and created a new commit in a new branch:

commit 3ebcab8550e5ba1d582f2799979cea6ac2bb0731 (HEAD -> subdir, origin/subdir)
Author: Ivan Shcheklein <[email protected]>
Date:   Sun Sep 17 12:36:31 2023 -0700

    move project to subdir

commit aca214d8511539e22550e2d8e02925d1a3c1bfad (origin/main, origin/HEAD, main)
Author: Alex Kim <[email protected]>
Date:   Mon Sep 11 07:34:17 2023 +0000

    Run dvc.yaml pipeline
...

Here is the new layout:

``` . ├── README.md └── subdir ├── data │   ├── pool_data │   │   ├── images │   │   │   ├── REGION_1-24_0_1024_0_1024.jpg │   │   │   ├── REGION_1-24_0_1024_1024_2048.jpg │   │   │   ├── REGION_1-24_0_1024_2048_3072.jpg │   │   │   ├── REGION_1-24_0_1024_2816_3840.jpg │   │   │   ├── REGION_1-24_1024_2048_0_1024.jpg │   │   │   ├── REGION_1-24_1024_2048_1024_2048.jpg │   │   │   ├── REGION_1-24_1024_2048_2048_3072.jpg │   │   │   ├── REGION_1-24_1024_2048_2816_3840.jpg │   │   │   ├── REGION_1-24_1136_2160_0_1024.jpg │   │   │   ├── REGION_1-24_1136_2160_1024_2048.jpg │   │   │   ├── REGION_1-24_1136_2160_2048_3072.jpg │   │   │   ├── REGION_1-24_1136_2160_2816_3840.jpg │   │   │   ├── REGION_2-19_0_1024_0_1024.jpg │   │   │   ├── REGION_2-19_1024_2048_0_1024.jpg │   │   │   ├── REGION_2-19_1136_2160_0_1024.jpg │   │   │   ├── REGION_2-4_0_1024_2816_3840.jpg │   │   │   ├── REGION_2-4_1024_2048_1024_2048.jpg │   │   │   ├── REGION_2-4_1024_2048_2048_3072.jpg │   │   │   ├── REGION_2-4_1024_2048_2816_3840.jpg │   │   │   ├── REGION_2-4_1136_2160_1024_2048.jpg │   │   │   ├── REGION_2-4_1136_2160_2048_3072.jpg │   │   │   ├── REGION_2-4_1136_2160_2816_3840.jpg │   │   │   ├── REGION_3-25_0_1024_0_1024.jpg │   │   │   ├── REGION_3-25_0_1024_1024_2048.jpg │   │   │   ├── REGION_3-25_0_1024_2048_3072.jpg │   │   │   ├── REGION_3-25_0_1024_2816_3840.jpg │   │   │   ├── REGION_3-25_1024_2048_0_1024.jpg │   │   │   ├── REGION_3-25_1024_2048_1024_2048.jpg │   │   │   ├── REGION_3-25_1136_2160_0_1024.jpg │   │   │   ├── REGION_3-25_1136_2160_1024_2048.jpg │   │   │   ├── REGION_3-8_0_1024_2048_3072.jpg │   │   │   ├── REGION_3-8_0_1024_2816_3840.jpg │   │   │   ├── REGION_3-8_1024_2048_0_1024.jpg │   │   │   ├── REGION_3-8_1024_2048_1024_2048.jpg │   │   │   ├── REGION_3-8_1024_2048_2048_3072.jpg │   │   │   ├── REGION_3-8_1024_2048_2816_3840.jpg │   │   │   ├── REGION_3-8_1136_2160_0_1024.jpg │   │   │   ├── REGION_3-8_1136_2160_1024_2048.jpg │   │   │   ├── REGION_3-8_1136_2160_2048_3072.jpg │   │   │   ├── REGION_3-8_1136_2160_2816_3840.jpg │   │   │   ├── REGION_4-12_0_1024_1024_2048.jpg │   │   │   ├── REGION_4-12_0_1024_2048_3072.jpg │   │   │   ├── REGION_4-12_0_1024_2816_3840.jpg │   │   │   ├── REGION_4-12_1024_2048_0_1024.jpg │   │   │   ├── REGION_4-12_1024_2048_1024_2048.jpg │   │   │   ├── REGION_4-12_1024_2048_2048_3072.jpg │   │   │   ├── REGION_4-12_1024_2048_2816_3840.jpg │   │   │   ├── REGION_4-12_1136_2160_0_1024.jpg │   │   │   ├── REGION_4-12_1136_2160_1024_2048.jpg │   │   │   ├── REGION_4-12_1136_2160_2048_3072.jpg │   │   │   ├── REGION_4-12_1136_2160_2816_3840.jpg │   │   │   ├── REGION_4-22_0_1024_0_1024.jpg │   │   │   ├── REGION_4-22_0_1024_1024_2048.jpg │   │   │   ├── REGION_4-22_1024_2048_0_1024.jpg │   │   │   ├── REGION_4-22_1024_2048_1024_2048.jpg │   │   │   ├── REGION_4-22_1024_2048_2048_3072.jpg │   │   │   ├── REGION_4-22_1024_2048_2816_3840.jpg │   │   │   ├── REGION_4-22_1136_2160_1024_2048.jpg │   │   │   ├── REGION_4-22_1136_2160_2048_3072.jpg │   │   │   ├── REGION_4-22_1136_2160_2816_3840.jpg │   │   │   ├── REGION_5-15_0_1024_1024_2048.jpg │   │   │   ├── REGION_5-15_1024_2048_0_1024.jpg │   │   │   ├── REGION_5-15_1136_2160_0_1024.jpg │   │   │   ├── REGION_5-21_0_1024_2048_3072.jpg │   │   │   ├── REGION_5-21_1136_2160_0_1024.jpg │   │   │   ├── REGION_5-7_1024_2048_2048_3072.jpg │   │   │   ├── REGION_5-7_1136_2160_1024_2048.jpg │   │   │   ├── REGION_5-7_1136_2160_2048_3072.jpg │   │   │   ├── REGION_5-7_1136_2160_2816_3840.jpg │   │   │   ├── REGION_6-13_0_1024_1024_2048.jpg │   │   │   ├── REGION_6-13_0_1024_2048_3072.jpg │   │   │   ├── REGION_6-13_1024_2048_2048_3072.jpg │   │   │   ├── REGION_6-13_1136_2160_2048_3072.jpg │   │   │   ├── REGION_6-18_0_1024_0_1024.jpg │   │   │   ├── REGION_6-18_0_1024_1024_2048.jpg │   │   │   ├── REGION_6-20_0_1024_2048_3072.jpg │   │   │   ├── REGION_6-20_1024_2048_1024_2048.jpg │   │   │   ├── REGION_6-4_0_1024_1024_2048.jpg │   │   │   ├── REGION_7-3_0_1024_2048_3072.jpg │   │   │   ├── REGION_7-3_0_1024_2816_3840.jpg │   │   │   ├── REGION_7-3_1024_2048_0_1024.jpg │   │   │   ├── REGION_7-3_1024_2048_1024_2048.jpg │   │   │   ├── REGION_7-3_1024_2048_2048_3072.jpg │   │   │   ├── REGION_7-3_1136_2160_1024_2048.jpg │   │   │   ├── REGION_7-3_1136_2160_2048_3072.jpg │   │   │   ├── REGION_8-24_0_1024_1024_2048.jpg │   │   │   ├── REGION_8-24_0_1024_2816_3840.jpg │   │   │   ├── REGION_8-24_1024_2048_1024_2048.jpg │   │   │   ├── REGION_8-24_1024_2048_2048_3072.jpg │   │   │   ├── REGION_8-24_1136_2160_1024_2048.jpg │   │   │   └── REGION_8-24_1136_2160_2048_3072.jpg │   │   └── masks │   │   ├── REGION_1-24_0_1024_0_1024.png │   │   ├── REGION_1-24_0_1024_1024_2048.png │   │   ├── REGION_1-24_0_1024_2048_3072.png │   │   ├── REGION_1-24_0_1024_2816_3840.png │   │   ├── REGION_1-24_1024_2048_0_1024.png │   │   ├── REGION_1-24_1024_2048_1024_2048.png │   │   ├── REGION_1-24_1024_2048_2048_3072.png │   │   ├── REGION_1-24_1024_2048_2816_3840.png │   │   ├── REGION_1-24_1136_2160_0_1024.png │   │   ├── REGION_1-24_1136_2160_1024_2048.png │   │   ├── REGION_1-24_1136_2160_2048_3072.png │   │   ├── REGION_1-24_1136_2160_2816_3840.png │   │   ├── REGION_2-19_0_1024_0_1024.png │   │   ├── REGION_2-19_1024_2048_0_1024.png │   │   ├── REGION_2-19_1136_2160_0_1024.png │   │   ├── REGION_2-4_0_1024_2816_3840.png │   │   ├── REGION_2-4_1024_2048_1024_2048.png │   │   ├── REGION_2-4_1024_2048_2048_3072.png │   │   ├── REGION_2-4_1024_2048_2816_3840.png │   │   ├── REGION_2-4_1136_2160_1024_2048.png │   │   ├── REGION_2-4_1136_2160_2048_3072.png │   │   ├── REGION_2-4_1136_2160_2816_3840.png │   │   ├── REGION_3-25_0_1024_0_1024.png │   │   ├── REGION_3-25_0_1024_1024_2048.png │   │   ├── REGION_3-25_0_1024_2048_3072.png │   │   ├── REGION_3-25_0_1024_2816_3840.png │   │   ├── REGION_3-25_1024_2048_0_1024.png │   │   ├── REGION_3-25_1024_2048_1024_2048.png │   │   ├── REGION_3-25_1136_2160_0_1024.png │   │   ├── REGION_3-25_1136_2160_1024_2048.png │   │   ├── REGION_3-8_0_1024_2048_3072.png │   │   ├── REGION_3-8_0_1024_2816_3840.png │   │   ├── REGION_3-8_1024_2048_0_1024.png │   │   ├── REGION_3-8_1024_2048_1024_2048.png │   │   ├── REGION_3-8_1024_2048_2048_3072.png │   │   ├── REGION_3-8_1024_2048_2816_3840.png │   │   ├── REGION_3-8_1136_2160_0_1024.png │   │   ├── REGION_3-8_1136_2160_1024_2048.png │   │   ├── REGION_3-8_1136_2160_2048_3072.png │   │   ├── REGION_3-8_1136_2160_2816_3840.png │   │   ├── REGION_4-12_0_1024_1024_2048.png │   │   ├── REGION_4-12_0_1024_2048_3072.png │   │   ├── REGION_4-12_0_1024_2816_3840.png │   │   ├── REGION_4-12_1024_2048_0_1024.png │   │   ├── REGION_4-12_1024_2048_1024_2048.png │   │   ├── REGION_4-12_1024_2048_2048_3072.png │   │   ├── REGION_4-12_1024_2048_2816_3840.png │   │   ├── REGION_4-12_1136_2160_0_1024.png │   │   ├── REGION_4-12_1136_2160_1024_2048.png │   │   ├── REGION_4-12_1136_2160_2048_3072.png │   │   ├── REGION_4-12_1136_2160_2816_3840.png │   │   ├── REGION_4-22_0_1024_0_1024.png │   │   ├── REGION_4-22_0_1024_1024_2048.png │   │   ├── REGION_4-22_1024_2048_0_1024.png │   │   ├── REGION_4-22_1024_2048_1024_2048.png │   │   ├── REGION_4-22_1024_2048_2048_3072.png │   │   ├── REGION_4-22_1024_2048_2816_3840.png │   │   ├── REGION_4-22_1136_2160_1024_2048.png │   │   ├── REGION_4-22_1136_2160_2048_3072.png │   │   ├── REGION_4-22_1136_2160_2816_3840.png │   │   ├── REGION_5-15_0_1024_1024_2048.png │   │   ├── REGION_5-15_1024_2048_0_1024.png │   │   ├── REGION_5-15_1136_2160_0_1024.png │   │   ├── REGION_5-21_0_1024_2048_3072.png │   │   ├── REGION_5-21_1136_2160_0_1024.png │   │   ├── REGION_5-7_1024_2048_2048_3072.png │   │   ├── REGION_5-7_1136_2160_1024_2048.png │   │   ├── REGION_5-7_1136_2160_2048_3072.png │   │   ├── REGION_5-7_1136_2160_2816_3840.png │   │   ├── REGION_6-13_0_1024_1024_2048.png │   │   ├── REGION_6-13_0_1024_2048_3072.png │   │   ├── REGION_6-13_1024_2048_2048_3072.png │   │   ├── REGION_6-13_1136_2160_2048_3072.png │   │   ├── REGION_6-18_0_1024_0_1024.png │   │   ├── REGION_6-18_0_1024_1024_2048.png │   │   ├── REGION_6-20_0_1024_2048_3072.png │   │   ├── REGION_6-20_1024_2048_1024_2048.png │   │   ├── REGION_6-4_0_1024_1024_2048.png │   │   ├── REGION_7-3_0_1024_2048_3072.png │   │   ├── REGION_7-3_0_1024_2816_3840.png │   │   ├── REGION_7-3_1024_2048_0_1024.png │   │   ├── REGION_7-3_1024_2048_1024_2048.png │   │   ├── REGION_7-3_1024_2048_2048_3072.png │   │   ├── REGION_7-3_1136_2160_1024_2048.png │   │   ├── REGION_7-3_1136_2160_2048_3072.png │   │   ├── REGION_8-24_0_1024_1024_2048.png │   │   ├── REGION_8-24_0_1024_2816_3840.png │   │   ├── REGION_8-24_1024_2048_1024_2048.png │   │   ├── REGION_8-24_1024_2048_2048_3072.png │   │   ├── REGION_8-24_1136_2160_1024_2048.png │   │   └── REGION_8-24_1136_2160_2048_3072.png │   ├── pool_data.dvc │   ├── test_data │   │   ├── REGION_1-24_0_1024_0_1024.jpg │   │   ├── REGION_1-24_0_1024_0_1024.png │   │   ├── REGION_1-24_0_1024_1024_2048.jpg │   │   ├── REGION_1-24_0_1024_1024_2048.png │   │   ├── REGION_1-24_0_1024_2048_3072.jpg │   │   ├── REGION_1-24_0_1024_2048_3072.png │   │   ├── REGION_1-24_0_1024_2816_3840.jpg │   │   ├── REGION_1-24_0_1024_2816_3840.png │   │   ├── REGION_1-24_1024_2048_0_1024.jpg │   │   ├── REGION_1-24_1024_2048_0_1024.png │   │   ├── REGION_1-24_1024_2048_1024_2048.jpg │   │   ├── REGION_1-24_1024_2048_1024_2048.png │   │   ├── REGION_1-24_1024_2048_2048_3072.jpg │   │   ├── REGION_1-24_1024_2048_2048_3072.png │   │   ├── REGION_1-24_1024_2048_2816_3840.jpg │   │   ├── REGION_1-24_1024_2048_2816_3840.png │   │   ├── REGION_1-24_1136_2160_0_1024.jpg │   │   ├── REGION_1-24_1136_2160_0_1024.png │   │   ├── REGION_1-24_1136_2160_1024_2048.jpg │   │   ├── REGION_1-24_1136_2160_1024_2048.png │   │   ├── REGION_1-24_1136_2160_2048_3072.jpg │   │   ├── REGION_1-24_1136_2160_2048_3072.png │   │   ├── REGION_1-24_1136_2160_2816_3840.jpg │   │   └── REGION_1-24_1136_2160_2816_3840.png │   └── train_data │   ├── REGION_2-19_0_1024_0_1024.jpg │   ├── REGION_2-19_0_1024_0_1024.png │   ├── REGION_2-19_1024_2048_0_1024.jpg │   ├── REGION_2-19_1024_2048_0_1024.png │   ├── REGION_2-19_1136_2160_0_1024.jpg │   ├── REGION_2-19_1136_2160_0_1024.png │   ├── REGION_2-4_0_1024_2816_3840.jpg │   ├── REGION_2-4_0_1024_2816_3840.png │   ├── REGION_2-4_1024_2048_1024_2048.jpg │   ├── REGION_2-4_1024_2048_1024_2048.png │   ├── REGION_2-4_1024_2048_2048_3072.jpg │   ├── REGION_2-4_1024_2048_2048_3072.png │   ├── REGION_2-4_1024_2048_2816_3840.jpg │   ├── REGION_2-4_1024_2048_2816_3840.png │   ├── REGION_2-4_1136_2160_1024_2048.jpg │   ├── REGION_2-4_1136_2160_1024_2048.png │   ├── REGION_2-4_1136_2160_2048_3072.jpg │   ├── REGION_2-4_1136_2160_2048_3072.png │   ├── REGION_2-4_1136_2160_2816_3840.jpg │   ├── REGION_2-4_1136_2160_2816_3840.png │   ├── REGION_3-25_0_1024_0_1024.jpg │   ├── REGION_3-25_0_1024_0_1024.png │   ├── REGION_3-25_0_1024_1024_2048.jpg │   ├── REGION_3-25_0_1024_1024_2048.png │   ├── REGION_3-25_0_1024_2048_3072.jpg │   ├── REGION_3-25_0_1024_2048_3072.png │   ├── REGION_3-25_0_1024_2816_3840.jpg │   ├── REGION_3-25_0_1024_2816_3840.png │   ├── REGION_3-25_1024_2048_0_1024.jpg │   ├── REGION_3-25_1024_2048_0_1024.png │   ├── REGION_3-25_1024_2048_1024_2048.jpg │   ├── REGION_3-25_1024_2048_1024_2048.png │   ├── REGION_3-25_1136_2160_0_1024.jpg │   ├── REGION_3-25_1136_2160_0_1024.png │   ├── REGION_3-25_1136_2160_1024_2048.jpg │   ├── REGION_3-25_1136_2160_1024_2048.png │   ├── REGION_3-8_0_1024_2048_3072.jpg │   ├── REGION_3-8_0_1024_2048_3072.png │   ├── REGION_3-8_0_1024_2816_3840.jpg │   ├── REGION_3-8_0_1024_2816_3840.png │   ├── REGION_3-8_1024_2048_0_1024.jpg │   ├── REGION_3-8_1024_2048_0_1024.png │   ├── REGION_3-8_1024_2048_1024_2048.jpg │   ├── REGION_3-8_1024_2048_1024_2048.png │   ├── REGION_3-8_1024_2048_2048_3072.jpg │   ├── REGION_3-8_1024_2048_2048_3072.png │   ├── REGION_3-8_1024_2048_2816_3840.jpg │   ├── REGION_3-8_1024_2048_2816_3840.png │   ├── REGION_3-8_1136_2160_0_1024.jpg │   ├── REGION_3-8_1136_2160_0_1024.png │   ├── REGION_3-8_1136_2160_1024_2048.jpg │   ├── REGION_3-8_1136_2160_1024_2048.png │   ├── REGION_3-8_1136_2160_2048_3072.jpg │   ├── REGION_3-8_1136_2160_2048_3072.png │   ├── REGION_3-8_1136_2160_2816_3840.jpg │   ├── REGION_3-8_1136_2160_2816_3840.png │   ├── REGION_4-12_0_1024_1024_2048.jpg │   ├── REGION_4-12_0_1024_1024_2048.png │   ├── REGION_4-12_0_1024_2048_3072.jpg │   ├── REGION_4-12_0_1024_2048_3072.png │   ├── REGION_4-12_0_1024_2816_3840.jpg │   ├── REGION_4-12_0_1024_2816_3840.png │   ├── REGION_4-12_1024_2048_0_1024.jpg │   ├── REGION_4-12_1024_2048_0_1024.png │   ├── REGION_4-12_1024_2048_1024_2048.jpg │   ├── REGION_4-12_1024_2048_1024_2048.png │   ├── REGION_4-12_1024_2048_2048_3072.jpg │   ├── REGION_4-12_1024_2048_2048_3072.png │   ├── REGION_4-12_1024_2048_2816_3840.jpg │   ├── REGION_4-12_1024_2048_2816_3840.png │   ├── REGION_4-12_1136_2160_0_1024.jpg │   ├── REGION_4-12_1136_2160_0_1024.png │   ├── REGION_4-12_1136_2160_1024_2048.jpg │   ├── REGION_4-12_1136_2160_1024_2048.png │   ├── REGION_4-12_1136_2160_2048_3072.jpg │   ├── REGION_4-12_1136_2160_2048_3072.png │   ├── REGION_4-12_1136_2160_2816_3840.jpg │   ├── REGION_4-12_1136_2160_2816_3840.png │   ├── REGION_4-22_0_1024_0_1024.jpg │   ├── REGION_4-22_0_1024_0_1024.png │   ├── REGION_4-22_0_1024_1024_2048.jpg │   ├── REGION_4-22_0_1024_1024_2048.png │   ├── REGION_4-22_1024_2048_0_1024.jpg │   ├── REGION_4-22_1024_2048_0_1024.png │   ├── REGION_4-22_1024_2048_1024_2048.jpg │   ├── REGION_4-22_1024_2048_1024_2048.png │   ├── REGION_4-22_1024_2048_2048_3072.jpg │   ├── REGION_4-22_1024_2048_2048_3072.png │   ├── REGION_4-22_1024_2048_2816_3840.jpg │   ├── REGION_4-22_1024_2048_2816_3840.png │   ├── REGION_4-22_1136_2160_1024_2048.jpg │   ├── REGION_4-22_1136_2160_1024_2048.png │   ├── REGION_4-22_1136_2160_2048_3072.jpg │   ├── REGION_4-22_1136_2160_2048_3072.png │   ├── REGION_4-22_1136_2160_2816_3840.jpg │   ├── REGION_4-22_1136_2160_2816_3840.png │   ├── REGION_5-15_0_1024_1024_2048.jpg │   ├── REGION_5-15_0_1024_1024_2048.png │   ├── REGION_5-15_1024_2048_0_1024.jpg │   ├── REGION_5-15_1024_2048_0_1024.png │   ├── REGION_5-15_1136_2160_0_1024.jpg │   ├── REGION_5-15_1136_2160_0_1024.png │   ├── REGION_5-21_0_1024_2048_3072.jpg │   ├── REGION_5-21_0_1024_2048_3072.png │   ├── REGION_5-21_1136_2160_0_1024.jpg │   ├── REGION_5-21_1136_2160_0_1024.png │   ├── REGION_5-7_1024_2048_2048_3072.jpg │   ├── REGION_5-7_1024_2048_2048_3072.png │   ├── REGION_5-7_1136_2160_1024_2048.jpg │   ├── REGION_5-7_1136_2160_1024_2048.png │   ├── REGION_5-7_1136_2160_2048_3072.jpg │   ├── REGION_5-7_1136_2160_2048_3072.png │   ├── REGION_5-7_1136_2160_2816_3840.jpg │   ├── REGION_5-7_1136_2160_2816_3840.png │   ├── REGION_6-13_0_1024_1024_2048.jpg │   ├── REGION_6-13_0_1024_1024_2048.png │   ├── REGION_6-13_0_1024_2048_3072.jpg │   ├── REGION_6-13_0_1024_2048_3072.png │   ├── REGION_6-13_1024_2048_2048_3072.jpg │   ├── REGION_6-13_1024_2048_2048_3072.png │   ├── REGION_6-13_1136_2160_2048_3072.jpg │   ├── REGION_6-13_1136_2160_2048_3072.png │   ├── REGION_6-18_0_1024_0_1024.jpg │   ├── REGION_6-18_0_1024_0_1024.png │   ├── REGION_6-18_0_1024_1024_2048.jpg │   ├── REGION_6-18_0_1024_1024_2048.png │   ├── REGION_6-20_0_1024_2048_3072.jpg │   ├── REGION_6-20_0_1024_2048_3072.png │   ├── REGION_6-20_1024_2048_1024_2048.jpg │   ├── REGION_6-20_1024_2048_1024_2048.png │   ├── REGION_6-4_0_1024_1024_2048.jpg │   ├── REGION_6-4_0_1024_1024_2048.png │   ├── REGION_7-3_0_1024_2048_3072.jpg │   ├── REGION_7-3_0_1024_2048_3072.png │   ├── REGION_7-3_0_1024_2816_3840.jpg │   ├── REGION_7-3_0_1024_2816_3840.png │   ├── REGION_7-3_1024_2048_0_1024.jpg │   ├── REGION_7-3_1024_2048_0_1024.png │   ├── REGION_7-3_1024_2048_1024_2048.jpg │   ├── REGION_7-3_1024_2048_1024_2048.png │   ├── REGION_7-3_1024_2048_2048_3072.jpg │   ├── REGION_7-3_1024_2048_2048_3072.png │   ├── REGION_7-3_1136_2160_1024_2048.jpg │   ├── REGION_7-3_1136_2160_1024_2048.png │   ├── REGION_7-3_1136_2160_2048_3072.jpg │   ├── REGION_7-3_1136_2160_2048_3072.png │   ├── REGION_8-24_0_1024_1024_2048.jpg │   ├── REGION_8-24_0_1024_1024_2048.png │   ├── REGION_8-24_0_1024_2816_3840.jpg │   ├── REGION_8-24_0_1024_2816_3840.png │   ├── REGION_8-24_1024_2048_1024_2048.jpg │   ├── REGION_8-24_1024_2048_1024_2048.png │   ├── REGION_8-24_1024_2048_2048_3072.jpg │   ├── REGION_8-24_1024_2048_2048_3072.png │   ├── REGION_8-24_1136_2160_1024_2048.jpg │   ├── REGION_8-24_1136_2160_1024_2048.png │   ├── REGION_8-24_1136_2160_2048_3072.jpg │   └── REGION_8-24_1136_2160_2048_3072.png ├── dvc.lock ├── dvc.yaml ├── model.tar.gz ├── models │   ├── model.pkl │   └── model.pth ├── notebooks │   └── TrainSegModel.ipynb ├── params.yaml ├── requirements.txt ├── results │   ├── evaluate │   │   ├── dvc.yaml │   │   ├── metrics.json │   │   └── plots │   │   ├── images │   │   │   ├── REGION_1-24_0_1024_0_1024.png │   │   │   ├── REGION_1-24_0_1024_1024_2048.png │   │   │   ├── REGION_1-24_0_1024_2048_3072.png │   │   │   ├── REGION_1-24_0_1024_2816_3840.png │   │   │   ├── REGION_1-24_1024_2048_0_1024.png │   │   │   ├── REGION_1-24_1024_2048_1024_2048.png │   │   │   ├── REGION_1-24_1024_2048_2048_3072.png │   │   │   ├── REGION_1-24_1024_2048_2816_3840.png │   │   │   ├── REGION_1-24_1136_2160_0_1024.png │   │   │   └── REGION_1-24_1136_2160_1024_2048.png │   │   └── images.dvc │   └── train │   ├── dvc.yaml │   ├── metrics.json │   ├── params.yaml │   └── plots │   └── metrics │   ├── dice_multi.tsv │   ├── eval │   │   └── loss.tsv │   └── train │   └── loss.tsv ├── sagemaker │   ├── code │   │   ├── inference.py │   │   └── requirements.txt │   └── deploy_model.py └── src ├── data_split.py ├── endpoint_prediction.py ├── evaluate.py └── train.py ```

If we now run dvc exp show -A (inside subdir), it outputs:

 Experiment                 Created        results/evaluate/metrics.json:dice_multi   train.loss   eval.loss   results/train/metrics.json:dice_multi   step   evaluate/dice_multi   base.random_seed   data_split.te>
 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────>
  workspace                  -                                               0.88572     0.021562    0.023693                                  0.9027     10                     -   42                 ['REGION_1'] >
  subdir                     12:36 PM                                        0.88572     0.021562    0.023693                                  0.9027     10                     -   42                 ['REGION_1'] >
  main                       Sep 11, 2023                                    0.87753     0.027895    0.019998                                 0.87784      8                     -   42                 ['REGION_1'] >
  ├── 56b79f2 [plush-kine]   11:53 AM                                        0.88572     0.021562    0.023693                                  0.9027     10                     -   42                 ['REGION_1'] >
  └── b9c0670 [welsh-gust]   11:48 AM                                        0.88161     0.023979    0.025817                                 0.89612     10                     -   42                 ['REGION_1'] >
  b471fc8                    Sep 09, 2023                                          -     0.025721     0.02894                                 0.87858      8               0.86488   42                 ['REGION_1'] >
  be23273                    Sep 08, 2023                                          -     0.025721     0.02894                                 0.87858      8               0.86488   -                  -            >
  da0c571                    -                                                     -            -           -                                       -      -                     -   -                  -            >
  2138a12                    -                                                     -            -           -                                       -      -                     -   -                  -            >
  746c173                    -                                                     -            -           -                                       -      -                     -   -                  -            >
  4f3c0f7                    -                                                     -            -           -                                       -      -                     -   -                  -            >
 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

It's merging metrics (e.g. train.loss across commits from the new monorepo layout, and the old one). Relative paths for those metrics are the same indeed, but they belong to different projects.

Not sure how critical it is. E.g. if it's possible to create a situation when it clearly reads and merges file from to co-evolving projects.

DVC Version

DVC version: 3.21.1 (pip)
-------------------------
Platform: Python 3.11.4 on macOS-13.3.1-arm64-arm-64bit
Subprojects:
	dvc_data = 2.16.1
	dvc_objects = 1.0.1
	dvc_render = 0.6.0
	dvc_task = 0.3.0
	scmrepo = 1.3.1
Supports:
	http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.9.1, boto3 = 1.28.17)
Config:
	Global: /Users/ivan/Library/Application Support/dvc
	System: /Library/Application Support/dvc
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc (subdir), git
Repo.site_cache_dir: /Library/Caches/dvc/repo/d6ca19490601310289a3f89f16c515ac
@shcheklein shcheklein added bug Did we break something? A: experiments Related to dvc exp labels Sep 17, 2023
@dberenbaum
Copy link
Collaborator

In 3ebca, the metrics are only found in the subdirectory, and not in the root results dir, correct?

If that's the case, I think it's expected, and it should be the same in plots after #9898. If each commit has only one metric/plot with that name, they will be merged.

@shcheklein
Copy link
Member Author

It makes sense in some cases, in some cases it might be creating some weird behavior. I don't know what the right solution is. It was a bit unexpected to see, but I also see that it's a non-trivial task to come with one-fits-all semantics here.

In case of monorepo and subdir, but introducing a subdir we are effectively changing names of the metrics, plots, etc, w/o even moving them. E.g. let's say subdir existed before, but was part of the main repo, and now we want to isolate it - we don't move any files at all, just do extra dvc init --subdir; git commit. So if before metrics were being merged in one way, now because of their name change they will start being merged with some metrics in root directory.

A more practical example. Let's say we have an ensemble of models and all of them have acc + the final acc at the root. If we introduce subdir then acc in it will start getting merged with acc in the root, at least in the previous commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: experiments Related to dvc exp bug Did we break something?
Projects
None yet
Development

No branches or pull requests

2 participants