-cache.triesinmemory flag ignored or doesn't work

[adamf-sendblocks]

This is precisely the same as #676 - we have two bor nodes (v1.5.2) running with identical configurations in identical containers. The failure to support getting traces from blocks older than latest - 127 is inconsistent; our production node has been supporting older blocks so we haven't touched it since syncing the two, but our dev node isn't.

We've played a lot with the configurations, and from our experiments it looks like sometimes it takes, and sometimes it doesn't.

May be relevant: we have a mix between a config.toml file and command line arguments, and the only item in the toml file is the jsonrpc timeout (because there's no server command flag for it). I initially thought that the mix might be why it doesn't work, but it appears to be irrelevant as we've gotten it to work with the toml file, and it's currently not working with the toml file.

System information

Bor client version: v1.5.2

Heimdall client version: v1.0.4

OS & Version: Linux Ubuntu 22.04.2 LTS

Environment: Polygon Mainnet

Type of node: Sentry

Additional Information: some mods were made for tracing, but nothing functional was changed

Overview of the problem

See #676, it's identical.

The following are our command line arguments (managed by docker yaml, hence the formatting):

entrypoint:
      - docker-entrypoint.sh
      - /usr/bin/bor
      - server
      - -config
      - /home/config/bor.config.toml
      - -identity=sb-bor-mainnet-node1
      - -datadir=/bor-home
      - -bootnodes=${BOR_BOOT_NODES}
      - -bor.heimdall=http://heimdallrest:${HEIMDALL_REST_PORT}
      - -chain=mainnet
      - -discovery.dns=${BOR_DISCOVERY_DNS}
      - -port=${EL_P2P_PORT}
      - -http
      - -http.addr=0.0.0.0
      - -http.vhosts=*
      - -http.corsdomain=*
      - -http.port=${EL_RPC_PORT}
      - -http.api=eth,net,web3,admin,debug,bor
      - -ws
      - -ws.addr=0.0.0.0
      - -ws.port=${EL_WS_PORT}
      - -ws.api=eth,net,web3,admin,debug,bor
      - -ws.origins=*
      - -cache=${BOR_BLOCK_CACHE_SIZE}
      - -cache.triesinmemory=${BOR_BLOCK_LOOKBACK}
      - -metrics
      - -pprof
      - -pprof.addr
      - 0.0.0.0
      - -metrics.prometheus-addr
      - 0.0.0.0:7071

BOR_BLOCK_CACHE_SIZE has been either 12288 or 20480, we've tried values of BOR_BLOCK_LOOKBACK from 256 through to 10000 with no (consistent) success.

This is the contents of the toml file:

[jsonrpc]
    [jsonrpc.timeouts]
        read = "2m0s"
        write = "2m0s"
        idle = "2m0s"

Logs / Traces / Output / Error Messages

We're not seeing any warnings or errors, the logs are the same between the two nodes we're running.