archive node retries

[deepthiskumar]

Added retry logic for database writes to support concurrent database transactions (described here). This also requires setting transaction isolation level to 'serializable' and can be done in two ways:

1. set it as the default isolation level in postgresql.conf:
   default_transaction_isolation = 'serializable'
2. set it for a specific database using the following query
   ALTER DATABASE <DATABASE NAME> SET DEFAULT_TRANSACTION_ISOLATION TO SERIALIZABLE ;

They need to be done as a separate step before archive node starts writing (and before bringing up the database instance in the case of option 1). I think setting it at the db level makes it explicit and therefore is a better option, but i'm not quite sure where in the codebase this configuration needs to live

[psteckler]

Can you confirm that make archive_blocks still builds (there are additional arguments to the make_block_*_aux functions)?

[psteckler]

Should there be logs for the commit and rollback?

[deepthiskumar]

yup, added it

[deepthiskumar]

Can you confirm that make archive_blocks still builds (there are additional arguments to the make_block_*_aux functions)?

Can you confirm that make archive_blocks still builds (there are additional arguments to the make_block_*_aux functions)?

There was a type error, fixed it

[psteckler]

maybe accumulate errors?

[deepthiskumar]

In addition to logging it as warnings (the else case)? The error in this line is logged at the call site

[psteckler]

probably good enough

[bkase]

do you want to immediately retry without any backoff timeout? Maybe it's worth waiting a few milliseconds? Though I'm not sure what else that will affect. Up to you...

[deepthiskumar]

Don't see a problem adding a backoff timeout of a few milliseconds. It should help if there are multiple commit attempts

[O1ahmad]

2. set it for a specific database using the following query
   `ALTER DATABASE <DATABASE NAME> SET DEFAULT_TRANSACTION_ISOLATION TO SERIALIZABLE ;`

... I think setting it at the db level makes it explicit and therefore is a better option, but i'm not quite sure where in the codebase this configuration needs to live

Agreed and something like below added to archive node's db-bootstrap jobs should work for this:

      - name: update-transaction-isolation-level
        image: bitnami/postgresql
        command: ["bash", "-c"]
        args: ["PGPASSWORD={{ .Values.postgresql.postgresqlPassword }} psql --username {{ .Values.postgresql.postgresqlUsername }} --host {{ tpl .Values.archive.postgresHost . }} --port {{ .Values.archive.postgresPort }} --dbname {{ .Values.archive.postgresDB}} -c ALTER DATABASE {{ .Values.archive.postgresDB }} SET DEFAULT_TRANSACTION_ISOLATION TO SERIALIZABLE ;"]

[O1ahmad]

1. set it as the default isolation level in postgresql.conf:
   `default_transaction_isolation = 'serializable'

Alternatively, we can update the postgres settings set in terraform with extended configs:

postgresqlExtendedConf = {
      default_transaction_isolation ="serializable"
}

[aneesharaines]

!approved-for-mainnet