Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: markdown-preprocess not multi-core safe with make #17322

Closed
Foxboron opened this issue Feb 1, 2023 · 2 comments · Fixed by #17331
Closed

[Bug]: markdown-preprocess not multi-core safe with make #17322

Foxboron opened this issue Feb 1, 2023 · 2 comments · Fixed by #17331
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@Foxboron
Copy link
Contributor

Foxboron commented Feb 1, 2023

Issue Description

When building podman with a lot of cores, like make -j64, it seems the markdown preprocessor is racey and doesn't like this.

The build will fail with

hack/markdown-preprocess
hack/markdown-preprocess
hack/markdown-preprocess
hack/markdown-preprocess
hack/markdown-preprocess
hack/markdown-preprocess
hack/markdown-preprocess
hack/markdown-preprocess
Traceback (most recent call last):
  File "/build/podman/src/podman/hack/markdown-preprocess", line 184, in <module>
Traceback (most recent call last):
  File "/build/podman/src/podman/hack/markdown-preprocess", line 184, in <module>
    main()
  File "/build/podman/src/podman/hack/markdown-preprocess", line 181, in main
    preprocessor.rewrite_optionfiles()
  File "/build/podman/src/podman/hack/markdown-preprocess", line 84, in rewrite_optionfiles
    main()
  File "/build/podman/src/podman/hack/markdown-preprocess", line 178, in main
    if not filecmp.cmp(optionfile, tmpfile):
  File "/usr/lib/python3.10/filecmp.py", line 53, in cmp
    s1 = _sig(os.stat(f1))
FileNotFoundError: [Errno 2] No such file or directory: 'options/latest.md'
    preprocessor.process(infile)
  File "/build/podman/src/podman/hack/markdown-preprocess", line 48, in process
    self.insert_file(fh_out, optionfile)
  File "/build/podman/src/podman/hack/markdown-preprocess", line 100, in insert_file
    with open(path, 'r', encoding='utf-8') as fh_included:
FileNotFoundError: [Errno 2] No such file or directory: 'options/latest.md'
Traceback (most recent call last):
  File "/build/podman/src/podman/hack/markdown-preprocess", line 184, in <module>
    main()
  File "/build/podman/src/podman/hack/markdown-preprocess", line 181, in main
    preprocessor.rewrite_optionfiles()
  File "/build/podman/src/podman/hack/markdown-preprocess", line 84, in rewrite_optionfiles
    if not filecmp.cmp(optionfile, tmpfile):
  File "/usr/lib/python3.10/filecmp.py", line 54, in cmp
    s2 = _sig(os.stat(f2))
FileNotFoundError: [Errno 2] No such file or directory: 'options/memory.md.tmp'
make: *** [Makefile:463: docs/source/markdown/podman-container-inspect.1.md] Error 1
make: *** Waiting for unfinished jobs....
make: *** [Makefile:463: docs/source/markdown/podman-diff.1.md] Error 1
make: *** [Makefile:463: docs/source/markdown/podman-auto-update.1.md] Error 1
make: Leaving directory '/build/podman/src/podman'

While forcing -j1 seems to work fine.

hack/markdown-preprocess
if [ ! -x "/usr/bin/go-md2man" ]; then \
	make -C test/tools build/go-md2man ; \
fi
mkdir -p docs/build/man
make: Leaving directory '/build/podman/src/podman'
make: Entering directory '/build/podman/src/podman'
if [ ! -x "/usr/bin/go-md2man" ]; then \
	make -C test/tools build/go-md2man ; \
fi
mkdir -p docs/build/man
(cd docs; ./dckrman.sh ./build/man/*.1)

I think it's related to this commit as this did not happen with 4.3.0.
c9c2f64

Steps to reproduce the issue

Steps to reproduce the issue

  1. make -j64

Describe the results you received

Failing build.

Describe the results you expected

A working build.

podman info output

podman v4.4.0

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

@Foxboron Foxboron added the kind/bug Categorizes issue or PR as related to a bug. label Feb 1, 2023
@p-linnane
Copy link

Hello 👋 . I'm a maintainer for the Homebrew project. We are seeing this trying to build podman v4.4.0. podman builds fine if we do not try to build docs. Relevant GitHub Actions run: https://github.com/Homebrew/homebrew-core/actions/runs/4071150551/jobs/7012750769

Relates to Homebrew/homebrew-core#122122

@vrothberg
Copy link
Member

Thanks for reaching!

@edsantiago PTAL

edsantiago added a commit to edsantiago/libpod that referenced this issue Feb 2, 2023
One of the tmpfiles was not uniquely named, resulting in
conflicts with parallel 'make docs'. Fix that.

A better solution would be to run only one instance of the
preprocess script, but the Makefile incantation for that
is beyond me. This approach is CPU-wasteful but good enough.

Fixes: containers#17322

Signed-off-by: Ed Santiago <[email protected]>
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/podman that referenced this issue Feb 3, 2023
One of the tmpfiles was not uniquely named, resulting in
conflicts with parallel 'make docs'. Fix that.

A better solution would be to run only one instance of the
preprocess script, but the Makefile incantation for that
is beyond me. This approach is CPU-wasteful but good enough.

Fixes: containers#17322

Signed-off-by: Ed Santiago <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 2, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants