Data too large error from very large data products

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

When I did a harvest of a data set with some very large data products, I get a data too large error and the data is not loaded into the Registry.

🕵️ Expected behavior

I expected the loaded would be nominally loaded into the Registry.

📜 To Reproduce

1. Download TBD data product
2. Attempt to harvest the product
3. Note the error

[ERROR] LIDVID = urn:esa:psa:em16_tgo_acs:data_raw:acs_raw_hk_nir_20170907t000000-20170907t055959::3.0, 
Message = [parent] Data too large, data for [indices:data/write/bulk[s]] would be [16591820628/15.4gb], 
which exceeds the limit of [16287753830/15.1gb]. Current usage: [16591415264/15.4gb], new bytes reserved: [405364/395.8kb], 
usages [request=0/0b, fielddata=0/0b, in_flight_requests=405364/395.8kb, accounting=613644/599.2kb]

🖥 Environment Info

Linux

📚 Version of Software Used

3.7.6

🩺 Test Data / Additional context

TBD

🦄 Related requirements

No response

⚙️ Engineering Details

No response

[alexdunnjpl]

@jordanpadams what's the best way to get a copy of the label for this product?

[jordanpadams]

@alexdunnjpl a ping is out to the user.

[alexdunnjpl]

@jordanpadams looking deeper into this error, it appears to be due to imminent exhaustion of the JVM heap on OpenSearch, rather than any one request/product being too large. (Presumably RAM allocation is currently 16GB on that node)

The fix here is to bump up the instance size to cope with peak throughput, and/or incorporate pause/retry behaviour in harvest.

Closing as a duplicate of #125 on that basis, since the fix for that is a fix for this.

[jordanpadams]

@alexdunnjpl nice sleuthing. 🎉

[alexdunnjpl]

@sjoshi-jpl I see that psa is currently r5.4xlarge.search (128GB RAM) - did this get bumped up from r5.xlarge.search (16GB RAM) at some point recently?

[sjoshi-jpl]

@alexdunnjpl yes this was recently bumped up based on our last conversation with @jordanpadams and @tloubrieu-jpl as we discussed how PSA could be as large / resource intensive as GEO.