Sentinel-2 pair picking can exceed the maximum lambda runtime of 900 s (15 min)

[jhkennedy]

In its-live-monitoring-test, processing the sentinel2-valid-2,json integration test message (shown below) pulls so many potential secondary pairs that the lambda is timing out even when setting the maximum timeout of 15 minutes.

{
  "name": "S2B_MSIL1C_20240430T142739_N0510_R139_T24VUR_20240430T162937",
  "id": "bb62af19-0e23-4f90-87be-a1e936a117db",
  "path": "products/2024/4/30/S2B_MSIL1C_20240430T142739_N0510_R139_T24VUR_20240430T162937",
  "timestamp": "2024-04-30T14:27:39.024Z",
  "datatakeIdentifier": "GS2B_20240430T142739_037345_N05.10",
  "sciHubIngestion": "2024-04-30T17:19:38.004Z",
  "s3Ingestion": "2024-04-30T18:14:57.807Z",
  "tiles": [
    {
      "path": "tiles/24/V/UR/2024/4/30/0",
      "timestamp": "2024-04-30T14:34:42.769Z",
      "utmZone": 24,
      "latitudeBand": "V",
      "gridSquare": "UR",
      "datastrip": {
        "id": "S2B_OPER_MSI_L1C_DS_2BPS_20240430T162937_S20240430T143400_N05.10",
        "path": "products/2024/4/30/S2B_MSIL1C_20240430T142739_N0510_R139_T24VUR_20240430T162937/datastrip/0"
      }
    }
  ],
  "datastrips": [
    {
      "id": "S2B_OPER_MSI_L1C_DS_2BPS_20240430T162937_S20240430T143400_N05.10",
      "path": "products/2024/4/30/S2B_MSIL1C_20240430T142739_N0510_R139_T24VUR_20240430T162937/datastrip/0"
    }
  ]
}

The test stack is much more verbose, setting the logging level to DEBUG -- it's possible that setting logging to INFO would keep us under the timeout...

But I think that this is currently a blocking issue.

[jhkennedy]

Switching from DEBUG to LOG logging level in the test environment brought this down to just over 5 minutes, and sentinel2-valid-1.json ran in 1.5 minutes.

I will leave this open to monitor over time as we'll probably want to address it, but this is not a blocking issue anymore. We can monitor and determine how often this is a real problem.

Another strategy would be to reduce the search bounding box so that it doesn't overlap with neighbor tiles, significantly reducing the number of results to check.