Group logs online evals #708
Conversation
| group_logs_client.open(resume=True) | ||
|
|
||
| # Restore existing metrics data | ||
| data = list_existing_group_logs_metrics(group_logs_client.wandb) |
There was a problem hiding this comment.
We run many evals tasks at the same time. So there's a chance of race condition here, right? I'm thinking maybe we can pursue an approach with building an automatic report based on the metrics we already write to the runs.
There was a problem hiding this comment.
This is indeed possible, and would mean that the second eval metric overrides the first one. However, the chances of this happening are small: listing takes ~1.5s and publishing takes ~2s (which is small compared to the runtime of eval tasks). Relying on runs would probably be a better approach here.
There was a problem hiding this comment.
I agree chances are low but still, as we discussed, let's explore the approach with reporting not to republish the evals again.
vrigal
force-pushed
the
group-logs-online-evals
branch
from
82e7c53 to
e0cee4c
Compare
Based on #660
Closes #697
This was more complex than I thought, as it is not possible to increment an existing table from the Python client.
I ran the
eval.pyscript for every evaluation task of groupN1O85rIASLmCwKfUpCvTlw. Results are displayed on https://wandb.ai/teklia/da-en/runs/5ljd4a4n