Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some metric tables are empty #716

Open
Tracked by #164
eu9ene opened this issue Jul 1, 2024 · 6 comments
Open
Tracked by #164

Some metric tables are empty #716

eu9ene opened this issue Jul 1, 2024 · 6 comments
Labels
bug Something is broken or not correct weights and biases Intergration with Weights and Biases

Comments

@eu9ene
Copy link
Collaborator

eu9ene commented Jul 1, 2024

For example for https://wandb.ai/moz-translations/da-en

Screenshot 2024-06-24 at 9 39 47 AM
@eu9ene eu9ene added bug Something is broken or not correct weights and biases Intergration with Weights and Biases labels Jul 1, 2024
@eu9ene
Copy link
Collaborator Author

eu9ene commented Jul 1, 2024

Also see #688 (comment)

@vrigal
Copy link
Contributor

vrigal commented Jul 8, 2024

This definitely looks like an issue with Weight & Biases, as I am able to list data from the API for those tables.
It is certainly related to missing bar charts by the way. I noticed two types of empty tables (rows are blue, chrf and comet scores):

  • No row:
    • aV89TBTaTtiG0M39A5i04Q / teacher 1 / flores_aug-mix_devtest_table (W&B run 9nmh995a)
  • Rows with empty values:
    • N1O85rIASLmCwKfUpCvTlw / teacher-ensemble / flores_devtest_table (W&B run 0mnjufnk)

Here is the script I used to retrieve tables via the API:

>>> import json
>>> import wandb
>>> run = api.run(f"moz-translations/da-en/9nmh995a")
>>> artifact = next(
>>>     (artifact for artifact in list(run.files())[::-1] if artifact.name.startswith("media/table/flores_aug-mix_devtest_table")
>>> )
>>> json.load(artifact.download())
{'columns': ['Metric', 'Value'],
 'data': [['bleu_detok', 49.15], ['chrf', 72.12], ['comet', 0.9006]]}
>>> run = api.run(f"moz-translations/da-en/0mnjufnk")
>>> artifact = next(
>>>     (artifact for artifact in list(run.files())[::-1] if artifact.name.startswith("media/table/flores_devtest_table")
>>> )
>>> json.load(artifact.download())
{'columns': ['Metric', 'Value'],
 'data': [['bleu_detok', 48.93], ['chrf', 72.16], ['comet', 0.9013]]}

@eu9ene
Copy link
Collaborator Author

eu9ene commented Aug 8, 2024

Ok, so to summarize we can see the values in the the artifact https://wandb.ai/moz-translations/da-en/artifacts/run_table/run-9nmh995a-flores_augmix_devtest_table/v0/files/flores_aug-mix_devtest_table.table.json but the table is empty in UI:

Screenshot 2024-08-08 at 4 30 40 PM

@eu9ene
Copy link
Collaborator Author

eu9ene commented Aug 9, 2024

After a conversation with W&B support, they suggested a workaround that works for me: resetting the personal workspace.

Screenshot 2024-08-09 at 2 39 51 PM

@vrigal could you try this too for your workspace? It appears to also fix #688

@vrigal
Copy link
Contributor

vrigal commented Aug 12, 2024

@eu9ene unfortunately I still see some tables with "No rows to display" in your workspace (but I see it has been saved on 2024-08-09T21:32:05.000Z). Even with resetting cache or changing the web browser.
I cannot change the workspace on this dashboard, but it is surely related.

@eu9ene
Copy link
Collaborator Author

eu9ene commented Aug 19, 2024

I reported this and W&B team is working on the fix but there's no ETA yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is broken or not correct weights and biases Intergration with Weights and Biases
Projects
None yet
Development

No branches or pull requests

2 participants