Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-14514 pmdk: PMDK release 2.1.0 (w/o NDCTL enabled) #37

Merged
merged 75 commits into from
Aug 14, 2024
Merged

DAOS-14514 pmdk: PMDK release 2.1.0 (w/o NDCTL enabled) #37

merged 75 commits into from
Aug 14, 2024

Conversation

grom72
Copy link
Contributor

@grom72 grom72 commented May 13, 2024 •
edited
Loading

PMDK packages update release 2.1.0 (w/o NDCTL enabled).

NDCTL will be enabled in a separate PR (#38).

brianjmurrell
brianjmurrell reviewed May 13, 2024
View reviewed changes
debian/changelog Outdated Show resolved Hide resolved
brianjmurrell
brianjmurrell reviewed May 13, 2024
View reviewed changes
pmdk.spec Outdated Show resolved Hide resolved
brianjmurrell
brianjmurrell reviewed May 13, 2024
View reviewed changes
pmdk.spec Outdated
- Does not allow opening PMEMOBJ pool without unsafe shutdown counter (USC) if not explicitly disabled. (#5968)
- use PMEMOBJ_CONF="sds.at_create=0" to disable USC when working without PMem (emulated PMem, Docker, etc.).
- Drops support for building without libpthread (NO_LIBPTHREAD build-time define).- Reduces libpmemobj's stack usage below the 11kB threshold,
- This version is build w/o NDCTL support
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But

pmdk/pmdk.spec

Line 27 in efb8e66

%bcond_without ndctl
means this actually is built with NDCTL doesn't it? I.e. that line adds the --without ndctl option which means that

pmdk/pmdk.spec

Line 27 in efb8e66

%bcond_without ndctl
does not get used unless the --without ndctl option is actually used which should not get used for our builds.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is done in Makefile:
EXTERNAL_RPM_BUILD_OPTIONS := --without=ndctl

brianjmurrell
brianjmurrell reviewed May 13, 2024
View reviewed changes
debian/changelog Outdated

-- Tomasz Gromadzki <[email protected]> Fri, 10 May 2024 10:00:00 +0000

pmdk (2.0.1-1) stable; urgency=medium
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version-release in this entry is seems to be a duplicate of the entry above it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

brianjmurrell
brianjmurrell reviewed May 13, 2024
View reviewed changes
debian/rules Outdated
@@ -5,16 +5,24 @@ export DEB_BUILD_MAINT_OPTIONS = hardening=+all
DEB_HOST_MULTIARCH ?= $(shell dpkg-architecture -qDEB_HOST_MULTIARCH)
DEB_UPSTREAM_VERSION = $(shell dpkg-parsechangelog -S version | cut -d - -f 1 | cut -d : -f 2)

BUILD_ARGS = prefix=/usr libdir=/usr/lib/$(DEB_HOST_MULTIARCH) sysconfdir=/etc bashcompdir=/usr/share/bash-completion/completions NORPATH=1 NDCTL_ENABLE=n BUILD_EXAMPLES=n BUILD_BENCHMARKS=n
BUILD_ARGS = prefix=/usr libdir=/usr/lib/$(DEB_HOST_MULTIARCH) sysconfdir=/etc bashcompdir=/usr/share/bash-completion/completions NORPATH=1 BUILD_EXAMPLES=n BUILD_BENCHMARKS=n
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing NDCTL_ENABLE=n here seems to be in conflict with

pmdk/debian/changelog

Line 21 in 108170d

* This version is build w/o NDCTL support
.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@grom72 grom72 requested a review from janekmi May 16, 2024 05:57
grom72 added 24 commits May 16, 2024 08:55
@grom72
DAOS-14514 pmdk: update to version 2.0.1
dd76510 
Upgrade to version 2.0.1.
This version reduces libpmemobj's stack usage below the 11kB threshold.

This is the version that allows enabling NDCTL without
risk of stack over-usage in argobots ULT.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Enable NDCTL on the top of 2.0.1
6373932 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
PMDK release 2.1.0 w/o NDCTL enabled
371e45c 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Fix build release
81d79f6 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Small typo in changelog
56b4a9f 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Disable NDCTL in debian spec + few minor doc fixes
c00c735 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Fix changelog format
24f0f6e 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Fix source version to rc1
13027c4 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Extend symbols with new API calls
8ea96b5 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Add missing packaging files
cfd3ced 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Remove obsolete patch file
95b3826 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Build w/ NDCTL but w/o Valgrind (only for test purpose)
d0daaa0 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Typo in build parameter
a090f6c 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Skip tests until problem is resolved in PMDK
d44e770 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Skip tests until problem w/ test on nonpmem is resolved in PMDK
0e022cd 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Enable pmem test on non-PMem
9d663d0 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Enable pmem test on non-PMem"
804bd31 
This reverts commit 62b6cf4.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Skip tests until problem w/ test on nonpmem is resolved in PMDK"
a2906f6 
This reverts commit a81d59c.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Skip tests until problem is resolved in PMDK"
8251247 
This reverts commit fb67490.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Typo in build parameter"
d9ed87c 
This reverts commit 7c1f506.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Build w/ NDCTL but w/o Valgrind (only for test purpose)"
218bb4e 
This reverts commit 78ecb26.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Cleanup changelog
5561f34 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Cleanup Debian changelog
cc1ddf5 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Enable NDCTL on the top of 2.0.1"
c96f6c0 
This reverts commit d899795.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Fix libpmem2 remove command - 6th
02d75d1 
Skip-list: test_daos_rebuild_ec:DAOS-14982 test_dfuse_daos_build_wt_il:DAOS-16168 test_dfuse_daos_build_wt_pil4dfs:DAOS-16168 -> DAOS-16215 test_osa_offline_reintegration_without_checksum:DAOS-15608i
Skip-list-master: DaosCoreTest:DAOS-16008 test_ms_failover:DAOS-16103
Skip-list-release/2.6: test_daos_rebuild_simple:DAOS-15290 test_dfuse_daos_build_wb:DAOS-16215 test_dfuse_daos_build_wt:DAOS-16215 test_dfuse_daos_build_metadata:DAOS-16215 test_dfuse_daos_build_data:DAOS-16215 test_dfuse_daos_build_nocache:DAOS-16215 test_daos_rebuild_ec:DAOS-14982

Allow-unstable-test: true
Skip-func-hw-test-large: false

Priority: 2

Signed-off-by: Tomasz Gromadzki <[email protected]>
@brianjmurrell
Copy link
Contributor

You need to update packaging/ on one of your next commits:

image

It's actually probably time to make that fail dependency pipelines now.

grom72 added 15 commits August 6, 2024 16:44
@grom72
Fix libpmem2 remove command - 7th
4ccbb78 
Skip-list: test_daos_rebuild_ec:DAOS-14982 test_dfuse_daos_build_wt_il:DAOS-16168 test_dfuse_daos_build_wt_pil4dfs:DAOS-16168 -> DAOS-16215 test_osa_offline_reintegration_without_checksum:DAOS-15608i
Skip-list-master: DaosCoreTest:DAOS-16008 test_ms_failover:DAOS-16103
Skip-list-release/2.6: test_daos_rebuild_simple:DAOS-15290 test_dfuse_daos_build_wb:DAOS-16215 test_dfuse_daos_build_wt:DAOS-16215 test_dfuse_daos_build_metadata:DAOS-16215 test_dfuse_daos_build_data:DAOS-16215 test_dfuse_daos_build_nocache:DAOS-16215 test_daos_rebuild_ec:DAOS-14982

Allow-unstable-test: true
Skip-func-hw-test-large: false

Priority: 2

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Fix libpmem2 remove command - 8th
1644bad 
Skip-list: test_daos_rebuild_ec:DAOS-14982 test_dfuse_daos_build_wt_il:DAOS-16168 test_dfuse_daos_build_wt_pil4dfs:DAOS-16168 -> DAOS-16215 test_osa_offline_reintegration_without_checksum:DAOS-15608i
Skip-list-master: DaosCoreTest:DAOS-16008 test_ms_failover:DAOS-16103
Skip-list-release/2.6: test_daos_rebuild_simple:DAOS-15290 test_dfuse_daos_build_wb:DAOS-16215 test_dfuse_daos_build_wt:DAOS-16215 test_dfuse_daos_build_metadata:DAOS-16215 test_dfuse_daos_build_data:DAOS-16215 test_dfuse_daos_build_nocache:DAOS-16215 test_daos_rebuild_ec:DAOS-14982

Allow-unstable-test: true
Skip-func-hw-test-large: false

Priority: 2

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Fix libpmem2 remove command - 9th
29bda2a 
Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Update packaging
b1fbdc6 
Skip-list: test_daos_rebuild_ec:DAOS-14982 test_dfuse_daos_build_wt_il:DAOS-16168 test_dfuse_daos_build_wt_pil4dfs:DAOS-16168 -> DAOS-16215 test_osa_offline_reintegration_without_checksum:DAOS-15608i
Skip-list-master: DaosCoreTest:DAOS-16008 test_ms_failover:DAOS-16103
Skip-list-release/2.6: test_daos_rebuild_simple:DAOS-15290 test_dfuse_daos_build_wb:DAOS-16215 test_dfuse_daos_build_wt:DAOS-16215 test_dfuse_daos_build_metadata:DAOS-16215 test_dfuse_daos_build_data:DAOS-16215 test_dfuse_daos_build_nocache:DAOS-16215 test_daos_rebuild_ec:DAOS-14982

Allow-unstable-test: true
Skip-func-hw-test-large: false

Priority: 2

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Fix libpmem2 remove command - 9th"
8b5f254 
This reverts commit 29bda2a.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Fix libpmem2 remove command - 8th"
2dc2086 
This reverts commit 1644bad.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Fix libpmem2 remove command - 7th"
af1559c 
This reverts commit 4ccbb78.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Fix libpmem2 remove command - 6th"
00cfde3 
This reverts commit 02d75d1.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Fix libpmem2 remove command - 5th"
b8b560d 
This reverts commit 77fb688.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Fix libpmem2 remove command - 4th"
6d7dc3a 
This reverts commit 84e7c73.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Fix libpmem2 remove command - 3rd"
b51edbb 
This reverts commit 76102cc.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Fix libpmem2 remove command - 2nd"
4b86396 
This reverts commit 7c82538.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Fix libpmem2 remove command"
879c18b 
This reverts commit aaf63ab.

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Revert "Remove libpmem2 related files from installation."
c8bcbc1 
This reverts commit f36e93d.
@grom72
Update release date
dc01fbe 
Skip-list: test_daos_rebuild_ec:DAOS-14982 test_dfuse_daos_build_wt_il:DAOS-16168 test_dfuse_daos_build_wt_pil4dfs:DAOS-16168 -> DAOS-16215 test_osa_offline_reintegration_without_checksum:DAOS-15608i
Skip-list-master: DaosCoreTest:DAOS-16008 test_ms_failover:DAOS-16103
Skip-list-release/2.6: test_daos_rebuild_simple:DAOS-15290 test_dfuse_daos_build_wb:DAOS-16215 test_dfuse_daos_build_wt:DAOS-16215 test_dfuse_daos_build_metadata:DAOS-16215 test_dfuse_daos_build_data:DAOS-16215 test_dfuse_daos_build_nocache:DAOS-16215 test_daos_rebuild_ec:DAOS-14982

Allow-unstable-test: true
Skip-func-hw-test-large: false

Priority: 2

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72 grom72 requested a review from janekmi August 6, 2024 15:33
@grom72
Skip list update
536c4a6 
Skip-list:  test_osa_offline_reintegration_without_checksum:DAOS-15608
Skip-list-release/2.6: test_daos_rebuild_ec:DAOS-14982 test_dfuse_daos_build_wt_pil4dfs:DAOS-16215 test_daos_rebuild_simple:DAOS-15290 test_dfuse_daos_build_wb:DAOS-16215 test_dfuse_daos_build_wt:DAOS-16215 test_dfuse_daos_build_metadata:DAOS-16215 test_dfuse_daos_build_data:DAOS-16215 test_dfuse_daos_build_nocache:DAOS-16215
Allow-unstable-test: true
Skip-func-hw-test-large: false

Priority: 2

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72 grom72 requested a review from brianjmurrell August 6, 2024 17:39
@grom72
Update test skip list
48ceba0 
Skip-list-master:  test_ior_intercept_libioil:DAOS-16260  test_osa_offline_reintegration_without_checksum:DAOS-15608  test_daos_drain_simple:DAOS-15271  test_daos_rebuild_ec:DAOS-14982  test_osa_offline_reintegration_without_checksum:DAOS-14570
Skip-list-release/2.6: test_cart_rpc: DAOS-15989 test_daos_rebuild_ec:DAOS-14982 test_dfuse_daos_build_wt_pil4dfs:DAOS-16215 test_daos_rebuild_simple:DAOS-15290 test_dfuse_daos_build_wb:DAOS-16215 test_dfuse_daos_build_wt:DAOS-16215 test_dfuse_daos_build_metadata:DAOS-16215 test_dfuse_daos_build_data:DAOS-16215 test_dfuse_daos_build_nocache:DAOS-16215
Allow-unstable-test: true
Skip-func-hw-test-large: false

Priority: 2

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Fix typo in skip list.
51b0d61 
Skip-list-master: test_ior_intercept_libioil:DAOS-16260 test_osa_offline_reintegration_without_checksum:DAOS-15608 test_daos_drain_simple:DAOS-15271 test_daos_rebuild_ec:DAOS-14982 test_osa_offline_reintegration_without_checksum:DAOS-14570
Skip-list-release/2.6: test_cart_rpc:DAOS-15989 test_daos_rebuild_ec:DAOS-14982 test_dfuse_daos_build_wt_pil4dfs:DAOS-16215 test_daos_rebuild_simple:DAOS-15290 test_dfuse_daos_build_wb:DAOS-16215 test_dfuse_daos_build_wt:DAOS-16215 test_dfuse_daos_build_metadata:DAOS-16215 test_dfuse_daos_build_data:DAOS-16215 test_dfuse_daos_build_nocache:DAOS-16215
Allow-unstable-test: true
Skip-func-hw-test-large: false

Priority: 2

Signed-off-by: Tomasz Gromadzki <[email protected]>
@grom72
Copy link
Contributor Author

grom72 commented Aug 8, 2024

These are the best results that we can achieve with our CI in terms of stable (but not green) test results
https://build.hpdd.intel.com/job/daos-stack/job/pmdk/job/PR-37/60/
https://build.hpdd.intel.com/job/daos-stack/job/daos/job/ci-daos-stack-pmdk-PR-37-master/44/testReport/
https://build.hpdd.intel.com/job/daos-stack/job/daos/job/ci-daos-stack-pmdk-PR-37-release-2.6/32/testReport/

We shall consider that the master branch's failures are related to the MD on SSD, where PMem (end PMDK) shall not be utilized at all.

It looks like PMDK does not introduce any problem that is permanent. New test failures appear and disappear in harmony with the rest of the project work.

Therefore, I hereby support @janekmi 's request for this change to be accepted. 🙏

@grom72 grom72 requested a review from a team August 13, 2024 07:21
@brianjmurrell
Copy link
Contributor

We shall consider that the master branch's failures are related to the MD on SSD, where PMem (end PMDK) shall not be utilized at all.

It's a shame we are even wasting cluster resources testing PMDK with MD-on-SSD. But I talked to @phender and there is no existing mechanism to prevent that. We could add some code to pipeline-lib for dependency jobs to specify downstream testing stages to include/exclude so that we can enable skipping of MD-on-SSD for pmdk for example. Perhaps there are other situations that could benefit from this.

It would be good to open an SRE ticket to capture this new requirement.

@brianjmurrell brianjmurrell merged commit 16d3bc2 into master Aug 14, 2024
0 of 2 checks passed
@brianjmurrell brianjmurrell changed the title PMDK release 2.1.0 (w/o NDCTL enabled) DAOS-14514 pmdk: PMDK release 2.1.0 (w/o NDCTL enabled) Aug 14, 2024
@brianjmurrell brianjmurrell deleted the grom72/pmdk-2.1.0 branch August 14, 2024 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@janekmi janekmi janekmi approved these changes

@brianjmurrell brianjmurrell brianjmurrell approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@grom72 @janekmi @brianjmurrell @osalyk