sync #1
Merged
sync #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…2234). Contributed by Uma Maheswara Rao G.
Now skips ITestS3AEncryptionWithDefaultS3Settings.testEncryptionOverRename when server side encryption is not set to sse:kms Contributed by Mukund Thakur
Contributed by Steve Loughran.
…MappingPlacementRule and AppNameMappingPlacementRule using the new mapping rules. Contributed by Gergely Pollak.
…uted by Gergely Pollak.
…ISSIONING state. Contributed by yehuanhuan
…recoverLease when there are no mounts configured (#2275) Contributed by Uma Maheswara Rao G.
…r. Contributed by Gergely Pollak.
…dAheadCounters (#2272) Contributed by: Mehakmeet Singh.
…nse failure occurs in ABFS (#2264) Contributed by Mehakmeet Singh
…ensure backwards compatibility. Contributed by Benjamin Teke.
…tract tests (#2286) Contributed by Steve Loughran. * Fixes AbstractContractSeekTest test to use readFully * Doesn't do this to AbstractContractUnbufferTest test as it changes the test too much. Instead just notes in the error that this may be transient The issue is that read(buffer) doesn't guarantee that the buffer is filled, only that it will read up to a point, and that may be just be the amount of data left in the TCP packet. readFully corrects for this, but using it in the unbuffer test runs the risk that what is tested for in terms of unbuffering doesn't actually get validated.
…om scheduler supports placement constraints. Contributed by Bilwa S T.
…tributed by Lisheng Sun. Co-authored-by: Chen Zhang <[email protected]> Signed-off-by: He Xiaoqiao <[email protected]> Reviewed-by: Ayush Saxena <[email protected]>
…n mapping rules. Contributed by Peter Bacsko
Adds the options to control the size of the per-output-stream threadpool when writing data through the abfs connector * fs.azure.write.max.concurrent.requests * fs.azure.write.max.requests.to.queue Contributed by Bilahari T H
…ble dir prefix matches the path of a snapshottable dir (#2295) Change-Id: Iefcbc14ca939b61f0c1befc9a94d686be1684ddc
#2280) This changes directory tree deletion so that only files are incrementally deleted from S3Guard after the objects are deleted; the directories are left alone until metadataStore.deleteSubtree(path) is invoke. This avoids directory tombstones being added above files/child directories, which stop the treewalk and delete phase from working. Also: * Callback to delete objects splits files and dirs so that any problems deleting the dirs doesn't trigger s3guard updates * New statistic to measure #of objects deleted, alongside request count. * Callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers to clarify behavior. * Test enhancements to replicate the failure and verify the fix Contributed by Steve Loughran
…Contributed by Zhenzhao Wang. Co-authored-by: Zhenzhao Wang <[email protected]>
Contributed by fanrui. Signed-off-by: Mingliang Liu <[email protected]> Signed-off-by: He Xiaoqiao <[email protected]>
Contributed by imbajin. Reviewed-by: leosunli <[email protected]> Signed-off-by: He Xiaoqiao <[email protected]>
…ithSync. Contributed by Fei Hui.
…rs() calls. Contributed by Samir Khan (samkhan).
…are both true. Contributed by Stephen O'Donnell
…has file. (#2298). Contributed by Uma Maheswara Rao G.
…ntributed by Daryn Sharp and Ahmed Hussein
…le loader imp through conf (#2389). Contributed by Junfan Zhang.
Followup to HADOOP-17379. Contributed by Jungtaek Lim.
… notify the given URL via a custom class. Contributed by Zoltan Erdmann
…lt to false (#2443). Contributed by Ahmed Hussein.
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license Contributed by Zhihua Deng.
…eption. Contributed by Hongbing Wang. Reviewed-by: He Xiaoqiao <[email protected]> Reviewed-by: Hui Fei <[email protected]>
Contributed by Steve Loughran.
…2469) Contributed by Zhihua Deng.
Contributed by Ayush Saxena and Steve Loughran Change-Id: I4bb76815bc1d11d1804dc67bafde68b6a995b974
This patch * fixes the inversion * adds a precondition check * if the commands are supplied inverted, swaps them with a warning. This is to stop breaking any tests written to cope with the existing behavior. Contributed by Steve Loughran
…l should support it. Contributed by zhuqi.
Contributed by Mukund Thakur.
…apper/reducers from config. Contributed by Peter Bacsko & Wangda Tan
…esourcemanager.scheduler.fair (#2485) Co-authored-by: akumar <[email protected]> Reviewed-by: Wei-Chiu Chuang <[email protected]> Reviewed-by: Akira Ajisaka <[email protected]>
Contributed by bilaharith.
…s. (#2396) This adds a semaphore to throttle the number of FileSystem instances which can be created simultaneously, set in "fs.creation.parallel.count". This is designed to reduce the impact of many threads in an application calling FileSystem.get() on a filesystem which takes time to instantiate -for example to an object where HTTPS connections are set up during initialization. Many threads trying to do this may create spurious delays by conflicting for access to synchronized blocks, when simply limiting the parallelism diminishes the conflict, so speeds up all threads trying to access the store. The default value, 64, is larger than is likely to deliver any speedup -but it does mean that there should be no adverse effects from the change. If a service appears to be blocking on all threads initializing connections to abfs, s3a or store, try a smaller (possibly significantly smaller) value. Contributed by Steve Loughran.
…trash provisioning/emptiness check (#2472)
Reviewed-by: Takanobu Asanuma <[email protected]>
…A ID is missing. This reverts commit ce5b3d7.
Reviewed-by: Takanobu Asanuma <[email protected]>
Contributed by Sneha Vijayarajan.
#2492) Contributed by Sneha Vijayarajan.
…2473). Contributed by Steve Loughran The addition of deprecated S3A configuration options in HADOOP-17318 triggered a reload of default (xml resource) configurations, which breaks tests which fail if there's a per-bucket setting inconsistent with test setup. Creating an S3AFS instance before creating the Configuration() instance for test runs gets that reload out the way before test setup takes place. Along with the fix, extra changes in the failing test suite to fail fast when marker policy isn't as expected, and to log FS state better. Rather than create and discard an instance, add a new static method to S3AFS and invoke it in test setup. This forces the load Change-Id: Id52b1c46912c6fedd2ae270e2b1eb2222a360329
…e tests (#2493) Follow-on to HADOOP-17323. Contributed by Mukund Thakur.
Contributed by Sneha Vijayarajan.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
sync