ADIOS 2.8 Breaks Old BP4 Files (Compression)

[ax3l]

The ADIOS 2.8 release series breaks reading of 2.7 (and ealier) BP4 files.

We use c-blosc compression with ADIOS2 in openPMD (WarpX, PIConGPU, etc al.). The ADIOS 2.8 series cannot read files written with earlier ADIOS2 releases anymore if we used compression.

Please see openPMD/openPMD-api#1293 for details.

@pnorbert et al., can you please add a backwards compatible read mode so we can continue to read our existing BP4 files?

[ax3l]

@chuckatkins this will also affect our production usage of ParaView. if possible, please hold back on upgrading to ADIOS2 v2.8 in ParaView until this bug is fixed in ADIOS2.

[eisenhauer]

@JasonRuonanWang, something you can do here?

[JasonRuonanWang]

We have discussed this for a long time. This is because we added backward compatibility in the common compression headers in 2.8. Once we add the bytes in binary buffer to record the compressor version info, it will be always backward compatible. But there is no logically safe way to make a backward-compatible version and a non-backward-compatible version compatible. So @pnorbert and I thought, if we have to break it, then break it sooner than later. Break it when compression is still experimental, rather than break it when it's already production. If I were the original operator infrastructure designer, I would have added this from Day 1, so that it wouldn't break anything later on. But unfortunately I am not.

[eisenhauer]

I understand that there's nothing code-local to the operator, but we do have file version info in the file headers that we could use as a conditional. Any hope there? Or was the resolution simply that we wouldn't read old files?

[JasonRuonanWang]

I understand that there's nothing code-local to the operator, but we do have file version info in the file headers that we could use as a conditional. Any hope there? Or was the resolution simply that we wouldn't read old files?

What's in the file version info? Is it the ADIOS version or the BP format version? Maybe we can make it work for some specific cases, but I am not sure if it will work in general.

[pnorbert]

I am adding specific backward compatible hack for BP4 with BLOSC in #3275

[eisenhauer]

Norbert's fix in #3275 is merged into 2.8 and master branches and the 2.8.2 release has been cut. Closing this as fixed.