Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix an issue when reading blocks in streaming mode #4332

Merged
merged 1 commit into from
Aug 30, 2024

Conversation

pnorbert
Copy link
Contributor

when some producers did not write any block to output.

Issue could be reproduced before this fix by running localArray example with multiple processes. v3 is not written by every process and in the third step, there is a "gap" meaning, rank N did not write, but rank N+1 did write a block. Then the following script would fail in reading the blocks from the third step.

#!/usr/bin/env python3
import numpy as np
import adios2

def process_var(f, varname):
    print("--------------------------")
    print(f"Process variable {varname}")
    v = f.inquire_variable(varname)
    # b = f.all_blocks_info(varname)
    for step in range(v.steps()):
        v.set_step_selection([step, 1])
        bs = f.engine.blocks_info(varname, step)
        for bid in range(len(bs)):   # could use here len(b[step])
            v.set_block_selection(bid)
            data = f.read(v)
            print(f"step {step} block {bid} count = {data.shape}")


with adios2.Stream("localArray.bp", "r") as f:
    for _ in f.steps():
        process_var(f, 'v3')

@pnorbert pnorbert requested a review from anagainaru August 30, 2024 15:22
@pnorbert pnorbert merged commit 75d801f into ornladios:master Aug 30, 2024
38 checks passed
@pnorbert pnorbert deleted the fix-read-blocks-bp5 branch August 30, 2024 17:37
@pnorbert pnorbert added this to the v2.10.2 milestone Oct 1, 2024
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 23, 2024
…rs did not write any block to output. (ornladios#4332)

(cherry picked from commit 75d801f)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 23, 2024
…rs did not write any block to output. (ornladios#4332)

(cherry picked from commit 75d801f)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 23, 2024
…rs did not write any block to output. (ornladios#4332)

(cherry picked from commit 75d801f)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 23, 2024
…rs did not write any block to output. (ornladios#4332)

(cherry picked from commit 75d801f)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 24, 2024
…rs did not write any block to output. (ornladios#4332)

(cherry picked from commit 75d801f)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 24, 2024
…rs did not write any block to output. (ornladios#4332)

(cherry picked from commit 75d801f)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 24, 2024
…rs did not write any block to output. (ornladios#4332)

(cherry picked from commit 75d801f)
vicentebolea added a commit that referenced this pull request Oct 26, 2024
* release_210: (33 commits)
  Bump version to v2.10.2
  Merge pull request #4379 from vicentebolea/increace-cmake-version-dep-blosc
  Fix missing int64_t definition (#4365)
  Merge pull request #4346 from anagainaru/memSelLayout
  Merge pull request #4343 from anagainaru/gpuSpan
  Compress the lossless part of MGARD data with Zstd. It was uncompressed until now. (#4339)
  Fix an issue when reading blocks in streaming mode, when some producers did not write any block to output. (#4332)
  Fix in python API:  (#4331)
  Merge pull request #4303 from anagainaru/build-frontier
  Merge pull request #4293 from anagainaru/check_parser_lexer
  Derived variables of type StoreData should not record the expression string (#4247)
  Update ReadMe.md (#4243)
  Switch from ROCM_VERSION_MAJOR to HIP_VERSION_MAJOR (#4222)
  Merge pull request #4207 from anagainaru/hide-symbols
  Patch for when both xrootd and derived variables are activated (#4200)
  Add defines for BP3, BP4 and BP5 (#4191)
  Overload NdCopy instead of adding a new argument for the 2.10 release
  Revert "Backport #3759 to Release 2.10 branch. (cherry-picked from master branch #4340) (#4341)"
  Revert "Merge pull request #4189 from pnorbert/campaign-s3-support"
  ci,python: enable python win2022 serial build
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants