Add error handling for every external call

[trizin]

Handle unexpected errors such as:
web3.exceptions.BlockNotFound

So agents do not exit on random errors and keep running.

DoD:

• for every external call, there's an error-handling mechanism in place

[trentmc]

It will be useful to enumerate the specific behaviors to catch. So that this issue has a clearer definition of done (DoD). cc @trizin

[trizin]

Essentially, we need to integrate an error-handling mechanism wherever we make external calls. Depending on the nature of the error encountered:

• If the subgraph is down: Quit the app.
• For temporary query errors: Retry query up to x times.
• etc.

[trentmc]

OK! This is better. And I guess the DoD is:

• this is complete when: for every external call, there's an error-handling mechanism in place

[trizin]

Another common error:

  File "/app/pdr_backend/dfbuyer/main.py", line 133, in <module>

    main()

  File "/app/pdr_backend/dfbuyer/main.py", line 127, in main

    log_loop(block)

  File "/app/pdr_backend/dfbuyer/main.py", line 114, in log_loop

    block = web3_config.w3.eth.get_block(blockno, full_transactions=False)

  File "/usr/local/lib/python3.8/site-packages/web3/eth/eth.py", line 390, in get_block

    return self._get_block(block_identifier, full_transactions)

  File "/usr/local/lib/python3.8/site-packages/web3/module.py", line 73, in caller

    result = w3.manager.request_blocking(

  File "/usr/local/lib/python3.8/site-packages/web3/manager.py", line 250, in request_blocking

    return self.formatted_response(

  File "/usr/local/lib/python3.8/site-packages/web3/manager.py", line 223, in formatted_response

    apply_null_result_formatters(null_result_formatters, response, params)

  File "/usr/local/lib/python3.8/site-packages/web3/manager.py", line 97, in apply_null_result_formatters

    formatted_resp = pipe(params, null_result_formatters)

  File "cytoolz/functoolz.pyx", line 680, in cytoolz.functoolz.pipe

  File "cytoolz/functoolz.pyx", line 655, in cytoolz.functoolz.c_pipe

  File "/usr/local/lib/python3.8/site-packages/web3/_utils/method_formatters.py", line 794, in raise_block_not_found

    raise BlockNotFound(message)

web3.exceptions.BlockNotFound: Block with id: '0x257b39' not found.