Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BP3 Index parsing fix: instead of relying on the reported length of p… #2153

Merged
merged 1 commit into from
Apr 22, 2020
Merged

BP3 Index parsing fix: instead of relying on the reported length of p… #2153

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 29 additions & 5 deletions source/adios2/toolkit/format/bp/bp3/BP3Deserializer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,15 @@ void BP3Deserializer::ParsePGIndex(const BufferSTL &bufferSTL,
std::unordered_set<uint32_t> stepsFound;
m_MetadataSet.StepsCount = 0;

while (localPosition < length)
/* Note: In ADIOS 1.x BP3 files, length is unreliable and is
* probably smaller than the actual length of the PG index. Let's use
* here the more reliable limit: the start of variable index - start of
* PG index (- the already parsed 16 bytes)
*/
const size_t pgIndexLength =
m_Minifooter.VarsIndexStart - m_Minifooter.PGIndexStart - 16;

while (localPosition < pgIndexLength)
{
ProcessGroupIndex index = ReadProcessGroupIndexHeader(
buffer, position, m_Minifooter.IsLittleEndian);
Expand Down Expand Up @@ -210,9 +218,17 @@ void BP3Deserializer::ParseVariablesIndex(const BufferSTL &bufferSTL,
const size_t startPosition = position;
size_t localPosition = 0;

/* Note: In ADIOS 1.x BP3 files, length is unreliable and is
* probably smaller than the actual length of the variable index. Let's use
* here the more reliable limit: the start of attribute index - start of
* variable index (- the already parsed 12 bytes)
*/
const size_t varIndexLength =
m_Minifooter.AttributesIndexStart - m_Minifooter.VarsIndexStart - 12;

if (m_Parameters.Threads == 1)
{
while (localPosition < length)
while (localPosition < varIndexLength)
{
lf_ReadElementIndex(engine, buffer, position);

Expand All @@ -231,7 +247,7 @@ void BP3Deserializer::ParseVariablesIndex(const BufferSTL &bufferSTL,

bool launched = false;

while (localPosition < length)
while (localPosition < varIndexLength)
{
// extract async positions
for (unsigned int t = 0; t < m_Parameters.Threads; ++t)
Expand All @@ -248,7 +264,7 @@ void BP3Deserializer::ParseVariablesIndex(const BufferSTL &bufferSTL,
asyncs[t].get();
}

if (localPosition <= length)
if (localPosition <= varIndexLength)
{
asyncs[t] = std::async(std::launch::async, lf_ReadElementIndex,
std::ref(engine), std::ref(buffer),
Expand Down Expand Up @@ -311,8 +327,16 @@ void BP3Deserializer::ParseAttributesIndex(const BufferSTL &bufferSTL,
const size_t startPosition = position;
size_t localPosition = 0;

/* Note: In ADIOS 1.x BP3 files, length is unreliable and is
* probably smaller than the actual length of the attribute index. Let's use
* here the more reliable limit: the end of index buffer - the size of the
* minifooter
*/
const size_t attrIndexLength =
buffer.size() - m_MetadataSet.MiniFooterSize - startPosition;

// Read sequentially
while (localPosition < length)
while (localPosition < attrIndexLength)
{
lf_ReadElementIndex(engine, buffer, position);
const size_t elementIndexSize =
Expand Down