Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ant task to automatically convert to a simple shim layout #7561

Merged
Merged
Show file tree
Hide file tree
Changes from 126 commits
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
60f53e1
wip
gerashegalov Nov 30, 2022
ec1da64
311 compiles
gerashegalov Dec 1, 2022
bce0644
31X compiles
gerashegalov Dec 1, 2022
acdc350
320 compiles
gerashegalov Dec 1, 2022
23c1870
321 compiles
gerashegalov Dec 1, 2022
e56934f
321cdh compiles
gerashegalov Dec 1, 2022
72c1a1f
321db TBD
gerashegalov Dec 1, 2022
b07753c
322 compiles
gerashegalov Dec 1, 2022
e7f81c7
323 compiles
gerashegalov Dec 1, 2022
2db8372
330 compiles
gerashegalov Dec 1, 2022
9224daa
330cdh compiles
gerashegalov Dec 1, 2022
364c799
331 compiles
gerashegalov Dec 1, 2022
dfb28fb
340 compiles
gerashegalov Dec 1, 2022
acf1a1d
312db compiles
gerashegalov Dec 1, 2022
9f23a64
Merge remote-tracking branch 'origin/branch-23.02' into generateSymlinks
gerashegalov Dec 1, 2022
3895d8a
Markdown update
gerashegalov Dec 1, 2022
388a778
332 compiles
gerashegalov Dec 1, 2022
675dbc7
symlinks optin
gerashegalov Dec 3, 2022
34b7362
gen
gerashegalov Dec 4, 2022
8f04d3e
typo
gerashegalov Dec 4, 2022
c4069fa
new tests
gerashegalov Dec 4, 2022
af6d194
move to parent pom
gerashegalov Dec 5, 2022
1e80aef
Unify with tests
gerashegalov Dec 5, 2022
d0a0bab
scalastyle
gerashegalov Dec 5, 2022
1b7b953
add tests to validate
gerashegalov Dec 6, 2022
33a17a3
scalastyle
gerashegalov Dec 6, 2022
d9b3ba7
remove extra newline
gerashegalov Dec 6, 2022
4197e2a
Merge remote-tracking branch 'origin/branch-23.02' into generateSymlinks
gerashegalov Dec 7, 2022
5db6c19
shim list validation
gerashegalov Dec 7, 2022
4a8c556
JSON format and validation prototype
gerashegalov Dec 8, 2022
a91c2da
340
gerashegalov Dec 8, 2022
e2294dc
330cdh
gerashegalov Dec 8, 2022
ac8341a
wip
gerashegalov Dec 8, 2022
45f6f2c
Many shims fixed
gerashegalov Dec 8, 2022
53d8674
verify all shim files have a comment
gerashegalov Dec 8, 2022
8ef615e
Merge remote-tracking branch 'origin/branch-23.02' into generateSymlinks
gerashegalov Dec 8, 2022
5ce17c4
Validate all files regardless
gerashegalov Dec 8, 2022
353e801
fix 311 330
gerashegalov Dec 8, 2022
82ed248
ensure only strings are used for versions
gerashegalov Dec 8, 2022
a8aca67
Merge remote-tracking branch 'origin/branch-23.02' into generateSymlinks
gerashegalov Dec 9, 2022
9bdfe0c
Merge remote-tracking branch 'origin/branch-23.02' into generateSymlinks
gerashegalov Dec 9, 2022
18402ae
Remove databricks unit tests
gerashegalov Dec 9, 2022
0ae39f4
undo code changes
gerashegalov Dec 23, 2022
d5d4e34
Merge remote-tracking branch 'origin/branch-23.02' into generateSymli…
gerashegalov Dec 23, 2022
c0c859f
wip
gerashegalov Dec 29, 2022
6ffd4ef
Merge remote-tracking branch 'origin/branch-23.02' into generateSymli…
gerashegalov Dec 29, 2022
7b645b6
Add shimplify task
gerashegalov Jan 18, 2023
bd2b82c
Traverse shim file set id
gerashegalov Jan 18, 2023
e47afb8
comment works, but executed twice
gerashegalov Jan 18, 2023
d2951fd
Merge remote-tracking branch 'origin/branch-23.02' into generateSymli…
gerashegalov Jan 18, 2023
2b3cb95
revert copyrights
gerashegalov Jan 19, 2023
fbdb77f
undo tests/pom.xml
gerashegalov Jan 19, 2023
b77cce6
undo unnecessary changes
gerashegalov Jan 19, 2023
d1088ca
wip
gerashegalov Jan 19, 2023
71d8034
comment done
gerashegalov Jan 19, 2023
7986f58
Merge remote-tracking branch 'origin/branch-23.02' into generateSymli…
gerashegalov Jan 19, 2023
0a123c0
restore highest-basedir
gerashegalov Jan 19, 2023
689984d
Merge remote-tracking branch 'origin/branch-23.02' into generateSymli…
gerashegalov Jan 19, 2023
aaf0999
idempotent overwrite
gerashegalov Jan 19, 2023
7fe32f2
Refactoring
gerashegalov Jan 19, 2023
b43f26b
prepare for rename
gerashegalov Jan 19, 2023
762fa1d
fix if attribute
gerashegalov Jan 20, 2023
0ea2576
Git rename
gerashegalov Jan 20, 2023
4b8bf3f
configurable rename
gerashegalov Jan 20, 2023
d71ab02
wip template how to finish test dir conversion
gerashegalov Jan 20, 2023
a13922d
Merge remote-tracking branch 'origin/branch-23.02' into generateSymli…
gerashegalov Jan 20, 2023
864bf7f
minimize changes
gerashegalov Jan 20, 2023
cd2831a
add comments and compile
gerashegalov Jan 20, 2023
6ea4d3d
wip
gerashegalov Jan 21, 2023
91c439b
fix test sources typo
gerashegalov Jan 21, 2023
76bf270
configurable trace
gerashegalov Jan 21, 2023
5b34fd3
flake8 lint
gerashegalov Jan 22, 2023
f82a0ed
Add code to detect shims that have multiple unshared dirs
gerashegalov Jan 24, 2023
613a5a0
make symlynk the onluy option
gerashegalov Jan 24, 2023
981385c
Merge remote-tracking branch 'origin/branch-23.02' into generateSymli…
gerashegalov Jan 24, 2023
7bcdca8
add shimple source locations to create symlinks in
gerashegalov Jan 24, 2023
d4a8e36
src/main compile works
gerashegalov Jan 25, 2023
081bf8c
Refactor to process main and test for symlinks
gerashegalov Jan 25, 2023
4ed4a75
no need for fileset in shimplify.py
gerashegalov Jan 26, 2023
c1e7298
all.buildvers
gerashegalov Jan 26, 2023
ba5412f
shimplify.shims to restrict the set of shims being converted
gerashegalov Jan 26, 2023
392aea7
Merge remote-tracking branch 'origin/branch-23.02' into generateSymli…
gerashegalov Jan 26, 2023
f9d2344
review and sel.fail instead of raise
gerashegalov Jan 26, 2023
504b0b0
hide most variables
gerashegalov Jan 26, 2023
8598dcc
Merge remote-tracking branch 'origin/branch-23.02' into generateSymli…
gerashegalov Jan 28, 2023
4745147
shimplify.dirs for targetted detangling of particular dirs
gerashegalov Jan 28, 2023
370fcfd
Dedupe shims supplied via shimplify.dirs
gerashegalov Jan 28, 2023
57de093
Merge remote-tracking branch 'origin/branch-23.02' into generateSymli…
gerashegalov Jan 28, 2023
0ce5eb6
indentation lint
gerashegalov Jan 30, 2023
c1b5431
Merge remote-tracking branch 'origin/branch-23.04' into generateSymli…
gerashegalov Jan 30, 2023
9ae393a
WIP adding doc and automation for 'add shim 331 like 330'
gerashegalov Jan 31, 2023
0912855
missing params
gerashegalov Feb 1, 2023
9e17d9f
comment
gerashegalov Feb 1, 2023
1835c9a
doc ways to invoke
gerashegalov Feb 1, 2023
2eb1065
Bug fixes and asserts
gerashegalov Feb 1, 2023
eff36c2
Add shim X like Y
gerashegalov Feb 2, 2023
9cfaaa2
Automatically add to git and advise to
gerashegalov Feb 2, 2023
cd4316c
typos
gerashegalov Feb 2, 2023
22a1f30
Automatic fixing for import and package
gerashegalov Feb 2, 2023
cf3af17
s/coma/comma
gerashegalov Feb 3, 2023
29c06ac
double it
gerashegalov Feb 3, 2023
df4ad0b
Simplify package rename call path
gerashegalov Feb 3, 2023
f1ce508
Merge remote-tracking branch 'gerashegalov/generateSymlinks-just-buil…
gerashegalov Feb 3, 2023
92704af
Merge remote-tracking branch 'origin/branch-23.04' into generateSymli…
gerashegalov Feb 6, 2023
7504406
refactor dirs2bv global
gerashegalov Feb 7, 2023
677d081
Basic add.shim from new sources done
gerashegalov Feb 7, 2023
8b351bf
get rid of manual shim comment array conversion
gerashegalov Feb 8, 2023
26675ed
Separate write and readonly shimplify paths
gerashegalov Feb 8, 2023
d936f4a
doc checkpoint
gerashegalov Feb 10, 2023
c29742f
Update the doc and remove unused packages from shimplify
gerashegalov Feb 11, 2023
bf90b6e
Merge branch 'branch-23.04' into generateSymlinks-just-build-v1
gerashegalov Feb 11, 2023
1ad7669
Update docs/dev/shimplify.md
gerashegalov Feb 13, 2023
02a3491
Update docs/dev/shimplify.md
gerashegalov Feb 13, 2023
aae3e61
Update docs/dev/shimplify.md
gerashegalov Feb 13, 2023
08475f9
Merge branch 'generateSymlinks-just-build-v1' of github.com:gerashega…
gerashegalov Feb 13, 2023
0fe43ca
Undo relative links unitl gh-pages branch no longer includes dev docs.
gerashegalov Feb 13, 2023
b2697ca
symlink clarify
gerashegalov Feb 14, 2023
914fe4b
Apply suggestions from code review
gerashegalov Feb 14, 2023
e3a5ada
Add deprecation note to CONTRIBUTING.md
gerashegalov Feb 14, 2023
a86be53
Merge branch 'generateSymlinks-just-build-v1' of github.com:gerashega…
gerashegalov Feb 14, 2023
6405720
Merge remote-tracking branch 'origin/branch-23.04' into generateSymli…
gerashegalov Feb 16, 2023
7ca79a7
Undo whitespace change
gerashegalov Feb 16, 2023
7826a38
Merge remote-tracking branch 'origin/branch-23.04' into generateSymli…
gerashegalov Feb 16, 2023
3d46182
Support for -Dshimplify.remove.shim
gerashegalov Feb 17, 2023
a6c22d4
Update docs/dev/shimplify.md
gerashegalov Feb 17, 2023
39fe554
Update docs/dev/shimplify.md
gerashegalov Feb 17, 2023
0b18c7c
bug fixes
gerashegalov Feb 17, 2023
51100db
Apply suggestions from code review
gerashegalov Feb 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 27 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,22 +186,38 @@ The following acronyms may appear in directory names:
|cdh |Cloudera CDH|321cdh |Cloudera CDH Spark based on Apache Spark 3.2.1|

The version-specific directory names have one of the following forms / use cases:
- `src/main/312/scala` contains Scala source code for a single Spark version, 3.1.2 in this case
- `src/main/312+-apache/scala`contains Scala source code for *upstream* **Apache** Spark builds,

#### Version range directories

The following source directory system is deprecated. See below and [shimplify.md][1]

* `src/main/312/scala` contains Scala source code for a single Spark version, 3.1.2 in this case
* `src/main/312+-apache/scala`contains Scala source code for *upstream* **Apache** Spark builds,
only beginning with version Spark 3.1.2, and + signifies there is no upper version boundary
among the supported versions
- `src/main/311until320-all` contains code that applies to all shims between 3.1.1 *inclusive*,
* `src/main/311until320-all` contains code that applies to all shims between 3.1.1 *inclusive*,
3.2.0 *exclusive*
- `src/main/pre320-treenode` contains shims for the Catalyst `TreeNode` class before the
* `src/main/pre320-treenode` contains shims for the Catalyst `TreeNode` class before the
[children trait specialization in Apache Spark 3.2.0](https://issues.apache.org/jira/browse/SPARK-34906).
- `src/main/post320-treenode` contains shims for the Catalyst `TreeNode` class after the
* `src/main/post320-treenode` contains shims for the Catalyst `TreeNode` class after the
[children trait specialization in Apache Spark 3.2.0](https://issues.apache.org/jira/browse/SPARK-34906).

For each Spark shim, we use Ant path patterns to compute the property
`spark${buildver}.sources` in [sql-plugin/pom.xml](./sql-plugin/pom.xml) that is
picked up as additional source code roots. When possible path patterns are reused using
the conventions outlined in the pom.

#### Simplified version directory structure

Going forward new shim files should be added under:

* `src/main/spark${buildver}`, example: `src/main/spark330db`
* `src/test/spark${buildver}`, example: `src/test/spark340`

with a special shim descriptor as a Scala/Java comment. See [shimplify.md][1]

[1]: ./docs/dev/shimplify.md

### Setting up an Integrated Development Environment

Our project currently uses `build-helper-maven-plugin` for shimming against conflicting definitions of superclasses
Expand Down Expand Up @@ -238,7 +254,12 @@ Known Issues:

* There is a known issue that the test sources added via the `build-helper-maven-plugin` are not handled
[properly](https://youtrack.jetbrains.com/issue/IDEA-100532). The workaround is to `mark` the affected folders
such as `tests/src/test/320+-noncdh-nondb` manually as `Test Sources Root`
such as

* `tests/src/test/320+-noncdh-nondb`
* `tests/src/test/spark340`

manually as `Test Sources Root`

* There is a known issue where, even after selecting a different Maven profile in the Maven submenu,
the source folders from a previously selected profile may remain active. As a workaround,
Expand Down
Loading