The execution builder is offline

[come-maiz]

Description

I'm testing teku + nethermind + mev-boost in kiln following this guide:
https://github.com/flashbots/mev-boost/wiki/Testing

Every now and then, Teku logs this error:

2022-07-02 18:17:50.099 WARN  - The execution builder is offline: java.util.concurrent.TimeoutException. Block production will fallback to the execution engine.
2022-07-02 18:17:50.211 INFO  - The execution builder is back online. It will be used for block production.

I don't know if this is a problem on mev-boost or teku. mev-boost seems to be running without issues constantly logging a 200 on the status endpoint. I'm starting my investigation on this here, can you please guide me?

Steps to Reproduce (Bug)

Follow the steps in:
https://github.com/flashbots/mev-boost/wiki/Testing

Expected behavior: Teku starts producing blocks built by the mev relay.

Actual behavior: This error message is logged:

2022-07-02 18:17:50.099 WARN  - The execution builder is offline: java.util.concurrent.TimeoutException. Block production will fallback to the execution engine.
2022-07-02 18:17:50.211 INFO  - The execution builder is back online. It will be used for block production.

Frequency: Like 3 times in 5 minutes.

Versions (Add all that apply)

• Software version:

~/workspace/kiln/teku$ ./build/install/teku/bin/teku --version
teku/v22.6.1+21-ga046c52/linux-x86_64/-privatebuild-openjdk64bitservervm-java-11

$ java --version
openjdk 11.0.15 2022-04-19
OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1)
OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1, mixed mode, sharing)

• OS Name & Version:

$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"
PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• Cloud VM, type, size: digital ocean

[tbenr]

I think that this is due to the fact that mev-boost in latest version is actually calling the status endpoint against relays to check their availability. Our timeout setting for status call could be now too short (if I remember correctly it is 1 second) for the full roundtrip so we may timeout before you provide a 200 to us.

Btw the timestamps you're reporting are a bit strange: we call the status every slot (12s) so I don't quite understand how we log the timeout and a success so close. We will look at it.

[tbenr]

This has been mitigated with flashbots/mev-boost#185
What we can do on our side is to tune our timeout if needed. I'll keep it opened while we double check if a different value is needed

[ajsutton]

Checking in on this one - it doesn't seem like we're having ongoing issues and can close this?

[ajsutton]

Going to close this one. It looks like the meg-boost mitigation was sufficient.