mjmac/dstress #13552
Closed
mjmac/dstress #13552
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
mjmac
commented
Jan 2, 2024
mjmac
commented
- Add a Google-specific Dockerfile
- Fix the Dockerfile to use the local repository
- Add hashes to pip requirements.txt
- Add Dockerfiles for build & base images
- DAOSGCP-213 dfuse: Ignore ENOENT errors when mounting to a file descriptor
- DAOS-10942 utils: Ignore unsupported bits when modifying permissions
- DAOS-13194 gurt: environment APIs hook (DAOS-13194 gurt: environment APIs hook #12220)
- Placeholder presubmit script.
- Set executable script on google_ci/presubmit.sh
- Portable shebang for presubmit script.
- Build DAOS as part of GCP presubmits.
- Run unit tests as part of GCP presubmits
- Use getpass instead of os.getlogin for log files
- Publish test artifacts to prow GCS
- Coalesce JUnit results files.
- Update requirements.txt
- Add missing --require-hashes
- Add base_requirements.txt for upgrading pip
- Require hashes for presubmit requirements install.
- Add chmod to daos utility
- DAOS-14561 vos: Add garbage collection metrics (DAOS-14561 vos: Add garbage collection metrics #13244)
- DAOS-14727 pool: Clean orphaned NVMe blobs on create
- b/308835681 control: Support metadata outside of tmpfs
- DAOS-14484 cart: Implement per-context inflight queue
- DAOS-10028 client: Go bindings for Client API
- fix LDFLAGS
- PoolQuery() unified
- Remove dep on -ldaos from lib/daos (only -ldaos_common allowed)
- fixed up create path handling
- fix generateRankSet()
- Make pool/cont flags optional
- Add DFS bindings
- Add lib/dfs
- fix some shiz
- remove auth_sys hackaround
- fixups
- fix read
- Add dstress tool
- Misc fixes
- dstress udpates
- faster rand, per-proc mode
- blap
Change-Id: Ib97771a7cad4ac64256c19f57d8199402ca83162
Instead of cloning from GitHub, the Dockerfile now takes the source from the build context (like utils/docker/Dockerfile.el.8 does). This also enables to build the image from a local repository. Also added some clarifying comments. Change-Id: Ie1c605bb5ffb9a9e71d1f16ec7764bdef1d1e09a
This is the result of following go/pip-install-remediation. See details in b/287289147. Change-Id: I9455e59f808e4c0defbed9f2a10cd0be75b51f5d
Change-Id: If45a7459fd23587d761f06f02c8fa3507de723f3
…iptor libfuse supports opening /dev/fuse and passing the file descriptor as the mountpoint. In some cases, realpath may not work for these file descriptors, and so we should ignore ENOENT errors and instead check that we can get file descriptor attributes from the given path. Change-Id: I2e9aad0e11a4c6f27ec2c4b1aeb75fc651d2540d
The setuid, setgid, and sticky bit can cause fatal errors when the datamover tool sets file permissions after copying a file, since these are not supported by DFS. We can just ignore this bit when calling dfs_chmod. Change-Id: Ibf2b6d793f95dd59c902c8d847bc087fb479c5ea
In order to prevent known race to occur due to lack of locking in Glibc environment APIs (getenv()/[uns]setenv()/ putenv()/clearenv()), they have been overloaded and strengthened in Gurt with hooks now all using a common lock/mutex. Libgurt is the preferred place for this as it is the lowest layer in DAOS, so it will be the earliest to be loaded and will ensure the hook to be installed as early as possible and could prevent usage of LD_PRELOAD. This will address the main lack of multi-thread protection in the Glibc APIs but do not handle all unsafe use-cases (like the change/removal of an env var when its value address has already been grabbed by a previous getenv(), ...). Change-Id: I38cda09746ddb4e79f0297fee26c2a22e1cb881b Signed-off-by: Bruno Faccini <[email protected]>
Eventually we'll use this to build DAOS and run the unit tests, but for now it's just a placeholder to test the prow integration. Change-Id: I28d99b39218cab0a01a96da2e4f9b649bc6863ce
Change-Id: Ib16b334895357a81a951fbdeaa6042f812d2c794
Change-Id: I78be9af9508ae63edd47acb948255d7f78821d18
Change-Id: I38c3ed1e5712e7270b01e4f116b4b22150a0fb5a
Change-Id: I8904d0136edd0873de1d645ec13a10e4904b4124
Change-Id: Ic0eeee9df2f0ef29f3f3f047080fdce109af71bf
Change-Id: Ia58529b77ab1c19db2a728607eb5ced283c2e361
Prow does not handle large numbers of XML files well, and the results were not showing up in the Spyglass UI. This change coalesces the XML results files into a single file and renames it in such a way that Spyglass will pick it up.o Tested: Presubmits Bug: 300680522 Change-Id: I0b49c5e137066587bf76cfb18541350240b1a596
Updated by running `pip-compile requirements.in --generate-hashes --upgrade`, see b/299710336#comment5. Tested by running `google_build/local_build.sh`. Bug: b/299710336 Change-Id: Id7cf7602ffe866167b63c59d3bb8c43334ceac63
This should be a no-op because `--require-hashes` is implied if any package in the requirements file has `--hash` (see https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-require-hashes). Tested: `docker build . -f google_build/build_image/Dockerfile` Bug: b/299585572 Change-Id: I3bdf1d80a259bffd3bedeb91cf32620de4a6cf08
This enables running `pip install pip` with `--require-hashes`. Tested: `docker build . -f google_build/build_image/Dockerfile` Bug: b/299585572 Change-Id: I8d81fbddaed2bbefec909ba2dce21dba5afbdeea
Change-Id: Ieedd142f06e044917e3dc8b9828369421f03ca29
TESTED=https://paste.googleplex.com/6208972604833792 BUG=311738671 Change-Id: Ia6658d7c99c8d21c35d724b86fa2c1c48b41069f
* Add some metrics for garbage collection * Add total metrics for vos space, so we can infer free space * Add aggregation failure counter Also adds a small change to the Go builder for handling versions with a -suffix. Required-githooks: true Change-Id: I098ec9ddce846758f669fd8d7dba7a0a5b44ef2d Co-authored-by: Michael MacDonald <[email protected]> Change-Id: I97502b0503fbe0b5bc57d7b3cee514264b885fa3 Signed-off-by: Jeff Olivier <[email protected]> Signed-off-by: Michael MacDonald <[email protected]>
When creating a pool on a target, add a step to clean up any orphaned NVMe blobs. Improves support for reintegration of a rank that is missing the VOS metadata for a pool. Required-githooks: true Change-Id: I38a314f1e5f140f1d149246802e47385e1c00311 Signed-off-by: Michael MacDonald <[email protected]>
The upstream 2.4 release has support for storing engine metadata outside of tmpfs, but it is tied to the new MD-on-SSD feature preview. With some small adjustments to the code, we can enable external metadata without MD-on-SSD. Required-githooks: true Change-Id: If3e728a2db7a4994572bbe53c92654f2e9b01ee0 Signed-off-by: Michael MacDonald <[email protected]>
Backport of in-flight upstream PR #13202 - Add per-context quotas - Implement RPC inflight quota Required-githooks: true Change-Id: I3fdf77082d66d8009ee7099cc838fcff5da72d4b Signed-off-by: Alexander A Oganezov <[email protected]> Signed-off-by: Jeff Olivier <[email protected]>
Add Go bindings for DAOS Client API. This is an initial pass, and will be expanded incrementally. Adds the following API methods: Init() Fini() PoolConnect() PoolQuery() PoolDisconnect() ContainerCreate() ContainerDestroy() ContainerOpen() ContainerQuery() ContainerGetProperties() ContinaerSetProperties() ContainerClose() Change-Id: I373a5ce5c4fc64384b7923eb3fba449887fd8b1e Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: I8c767e731bc61b9d31aaa29bec0ee711794a742c Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: If191f3d5a49ba329cc201c5d5b5dc76114ab6e1d Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: I569d3a832096ac766daf4d216ff83dbba4f0913c Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: I72a4e1b11e180fb61b9147492d20cf0e70eac5ef Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Features: daos_cmd Change-Id: I6c8fea3dcb7886bb01959408bc289f058f2f8338 Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: Ib5df55578c5714f77a52b1b1063bcc621bf30d4d Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: I9e0c631cc3cd441e774a9d04a56736597c096656 Required-githooks: true
Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: I86968b936ebe6d61721592a005da518d56436423 Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: I53c3bb23eeede45988f43166b749e191b57d268f Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: Idc15a4e9f3f972a2ba81c0bc2f1fec2156b13727 Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: I05c8b25f484f74ea5023dd427d2c298aaeab14e3 Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: I8738f9666cf3c379e5360fc8108e2d5a07bc64df Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: Ie68723ad86f6c7698a84c35e70228d117409b37d Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: I3a2c6fb05085301409424f3b5f2fcc1e09cffc9b Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
Change-Id: I1968c2a5c1e1591cc193f50c3bc61c273dc9ae8c Required-githooks: true Signed-off-by: Michael MacDonald <[email protected]>
|
Bug-tracker data: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.