Use active_run_lock(run) to protect updates of aggregated data.

[vdbergh]

We changed the active_run_locks to be reentrant.

[ppigazzini]

Looks good on DEV

[vdbergh]

@ppigazzini Can you check in the server log if there are messages like this

fishtest/server/fishtest/rundb.py

Lines 1202 to 1209 in a51582a

	if run["finished"]:
	info = "Update_task: task {}/{} belongs to a finished run".format(
	run_id, task_id
	)
	print(info, flush=True)
	self.set_inactive_task(task_id, run)
	self.buffer(run, True)
	return {"task_alive": False, "info": info}

I am trying to debug this https://tests.stockfishchess.org/actions?action=log_message&user=&text=cores

[vondele]

Very rare, and not very recent:

$ sudo journalctl -u fishtest@6543 --since "30 days ago" | grep "belongs to a finished run"
May 05 07:23:32 tests.stockfishchess.org pserve[28672]: Update_task: task 663723214b68b70d8580c189/890 belongs to a finished run
May 18 09:32:22 tests.stockfishchess.org pserve[22090]: Update_task: task 66458ee993ce6da3e93b5b90/1079 belongs to a finished run

But matching the two events you highlight

[vdbergh]

Thanks. I will fix this.

[vondele]

This could possibly happen if the worker sends an update_task, the rundb is updated, but the full transaction fails (e.g. nginx timeout), in which the worker sends the update_task again?

[ppigazzini]

extended to 180 days ago

$ sudo journalctl -u fishtest@6543 --since "180 days ago" | grep "is not equal to 0"
May 05 07:19:11 tests.stockfishchess.org pserve[28672]: The run object 663723214b68b70d8580c189 does not validate: run['cores'] (value:7) is not equal to 0
May 18 09:28:04 tests.stockfishchess.org pserve[22090]: The run object 66458ee993ce6da3e93b5b90 does not validate: run['cores'] (value:7) is not equal to 0

[vdbergh]

This could possibly happen if the worker sends an update_task, the rundb is updated, but the full transaction fails (e.g. nginx timeout), in which the worker sends the update_task again?

No this happens because for performance reasons update_task() and request_task() are not synchronized. But with PR #2020 they are at least partially synchronized so that we can now do something in request_task(). See #2021 .