sandbox cannot bring up indexer

[algochoi]

Subject of the issue

It looks like sandbox cannot bring up indexer on the develop branch using the latest commit 6fb9512:

Starting sandbox for: harness
see sandbox.log for detailed progress, or use -v.
* docker containers started!
* waiting for services to initialize.
the following did not start:
* indexer node
One or more services failed to start.

Docker logs:

Starting indexer against algod.
{"error":"connecting to postgres: failed to connect to `host=indexer-db user=algorand database=indexer_db`: dial error (dial tcp 172.21.0.2:5432: connect: connection refused)","level":"error","msg":"could not init db, connecting to postgres: failed to connect to `host=indexer-db user=algorand database=indexer_db`: dial error (dial tcp 172.21.0.2:5432: connect: connection refused)","time":"2022-12-13T18:18:19Z"}
Starting indexer against algod.
Algod not responding... waiting.
Connecting to algod:4001 (172.21.0.3:4001)
saving to 'genesis'
genesis              100% |********************************|  2134  0:00:00 ETA
'genesis' saved
Connecting to algod:4001 (172.21.0.3:4001)
wget: can't open 'genesis': File exists
Connecting to algod:4001 (172.21.0.3:4001)
wget: can't open 'genesis': File exists
Algod not responding... waiting.
Connecting to algod:4001 (172.21.0.3:4001)
wget: can't open 'genesis': File exists
Algod not responding... waiting.
Connecting to algod:4001 (172.21.0.3:4001)
wget: can't open 'genesis': File exists
Algod not responding... waiting.
Connecting to algod:4001 (172.21.0.3:4001)
wget: can't open 'genesis': File exists
Algod not responding... waiting.
{"level":"info","msg":"Found Importer: algod","time":"2022-12-13T18:19:34Z"}
{"level":"info","msg":"Found Processor: block_evaluator","time":"2022-12-13T18:19:34Z"}
{"level":"info","msg":"Found Exporter: postgresql","time":"2022-12-13T18:19:34Z"}
{"level":"info","msg":"Starting Pipeline Initialization","time":"2022-12-13T18:19:34Z"}
{"__type":"importer","_name":"algod","level":"info","msg":"Algod Importer added http prefix to NetAddr: http://algod:4001","time":"2022-12-13T18:19:34Z"}
{"level":"info","msg":"Initialized Importer: algod","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":142,"msg":"trackerDBInitialize upgrading database schema from version 0 to version 9","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":146,"msg":"trackerDBInitialize performing upgrade from version 0","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.(*trackerDBSchemaInitializer).upgradeDatabaseSchema0","level":"info","line":248,"msg":"upgradeDatabaseSchema0 initializing schema","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":146,"msg":"trackerDBInitialize performing upgrade from version 1","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":146,"msg":"trackerDBInitialize performing upgrade from version 2","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":146,"msg":"trackerDBInitialize performing upgrade from version 3","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":146,"msg":"trackerDBInitialize performing upgrade from version 4","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.(*trackerDBSchemaInitializer).upgradeDatabaseSchema4","level":"info","line":396,"msg":"upgradeDatabaseSchema4: deleted 0 rows","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":146,"msg":"trackerDBInitialize performing upgrade from version 5","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.(*trackerDBSchemaInitializer).upgradeDatabaseSchema5.func1","level":"info","line":421,"msg":"upgradeDatabaseSchema5 upgraded 1 out of 5 accounts [ 20.0% ]","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":146,"msg":"trackerDBInitialize performing upgrade from version 6","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.(*trackerDBSchemaInitializer).upgradeDatabaseSchema6.func1","level":"info","line":487,"msg":"upgradeDatabaseSchema6 upgraded 1 out of 5 accounts [ 20.0% ]","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":146,"msg":"trackerDBInitialize performing upgrade from version 7","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":146,"msg":"trackerDBInitialize performing upgrade from version 8","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"trackerdb.go","function":"github.com/algorand/go-algorand/ledger.runMigrations","level":"info","line":207,"msg":"trackerDBInitialize database schema upgrade complete","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"catchpointtracker.go","function":"github.com/algorand/go-algorand/ledger.(*catchpointTracker).initializeHashes","level":"info","line":1560,"msg":"initializeHashes rebuilding merkle trie for round 0","time":"2022-12-13T18:19:34Z"}
{"__type":"processors","_name":"block_evaluator","file":"catchpointtracker.go","function":"github.com/algorand/go-algorand/ledger.(*catchpointTracker).initializeHashes","level":"info","line":1679,"msg":"initializeHashes rebuilt the merkle trie with 5 entries in 1.804715ms","time":"2022-12-13T18:19:34Z"}
{"level":"info","msg":"Initialized Processor: block_evaluator","time":"2022-12-13T18:19:34Z"}
{"__type":"exporter","_name":"postgresql","level":"info","msg":"Running 0 migrations.","time":"2022-12-13T18:19:34Z"}
{"__type":"exporter","_name":"postgresql","level":"info","msg":"Setting status: Migrations Complete","time":"2022-12-13T18:19:34Z"}
{"__type":"exporter","_name":"postgresql","level":"info","msg":"Migration finished successfully.","time":"2022-12-13T18:19:34Z"}
{"level":"info","msg":"Initialized Exporter: postgresql","time":"2022-12-13T18:19:34Z"}
serving on :8980
{"level":"info","msg":"serving on :8980","time":"2022-12-13T18:19:34Z"}
{"level":"info","msg":"Enable all parameters flag is set to: false","time":"2022-12-13T18:19:34Z"}
{"level":"info","msg":"Pipeline round: 0","time":"2022-12-13T18:19:34Z"}
{"__type":"exporter","_name":"postgresql","level":"info","msg":"adding block 0","time":"2022-12-13T18:19:34Z"}
{"level":"info","msg":"Pipeline round: 1","time":"2022-12-13T18:19:34Z"}
⇨ http server started on [::]:8980
{"level":"error","msg":"HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2022-12-13T18:20:34Z"}
{"level":"info","msg":"Pipeline round: 1","time":"2022-12-13T18:20:35Z"}
{"level":"error","msg":"HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2022-12-13T18:21:35Z"}
{"level":"info","msg":"Pipeline round: 1","time":"2022-12-13T18:21:36Z"}
{"level":"error","msg":"HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2022-12-13T18:22:36Z"}
{"level":"info","msg":"Pipeline round: 1","time":"2022-12-13T18:22:37Z"}

Using the previous commit works properly. This is using config.dev, and it looks like the CI also fails for this build: https://app.circleci.com/pipelines/github/algorand/sandbox/122/workflows/47ca40c9-7032-436a-ae85-74a25c39208f/jobs/1826

Reverting the sandbox file gives the same docker output and initial error message, but indexer seems to work properly when transactions are sent in dev mode and correctly sync to the right round. e.g.

indexer - health
{
  "data": {
    "migration-required": false
  },
  "db-available": true,
  "is-migrating": false,
  "message": "2",
  "round": 2,
  "version": "(unknown version)"
}

[algochoi]

Closed by #162