Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare 0.7.0 release #176

Merged
merged 381 commits into from
Aug 10, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
381 commits
Select commit Hold shift + click to select a range
71f8239
fix readme
vovak Apr 29, 2021
b218627
Install build-essential
SpirinEgor Apr 29, 2021
7fd6c47
Return installing g++
SpirinEgor Apr 29, 2021
9ea602c
Specify version in gradle
SpirinEgor Apr 29, 2021
e22d375
specify version explicitly
vovak Apr 29, 2021
799ed4e
update version in README
vovak Apr 29, 2021
e9206ea
saved refactored label extractors
Furetur Apr 30, 2021
2bdcb77
Merge branch 'filters' into pipeline
Furetur Apr 30, 2021
c3dd34e
pipeline sketch
Furetur Apr 30, 2021
51277f5
Fix kotlin warnings
SpirinEgor Apr 30, 2021
f83ac9e
Use default properties for group and version
SpirinEgor Apr 30, 2021
ccc34d2
bug fixed and test added
illided May 1, 2021
3f74bc2
little renaming
illided May 1, 2021
d82c093
AntlrJavaScriptElementInfo added
illided May 2, 2021
948a0c9
utils refactoring
illided May 2, 2021
e4a0ab2
GumTreeJavaFunctionInfo added
illided May 3, 2021
658cc70
small refactoring
illided May 3, 2021
b501e8a
cast fixed
illided May 3, 2021
bf5aefa
toList statements removed
illided May 3, 2021
259f482
pipeline + project imported + pipeline frontend + storage config
Furetur May 3, 2021
a666f23
saved unfinished work
Furetur May 3, 2021
0d8f8dc
Create docker image for CI and easy local run
SpirinEgor May 3, 2021
afb6297
Add github CI workflow
SpirinEgor May 3, 2021
731976a
cli.sh check if docker image exist
SpirinEgor May 3, 2021
c12af50
Add information about docker and maven local into readme
SpirinEgor May 3, 2021
6435066
Update version
SpirinEgor May 3, 2021
f937c0d
Pipeline + config + GetPipeline
Furetur May 4, 2021
2b2120e
find enclosing moved higher
illided May 4, 2021
06a786d
fuzzy function info added
illided May 4, 2021
45d2403
Improved ParsingException.kt and logging
Furetur May 4, 2021
d9a1853
added documentation
Furetur May 4, 2021
d433900
rearranged rows in PipelineConfig.kt
Furetur May 4, 2021
717c558
removed the cli -_-
Furetur May 4, 2021
902ce18
gumtreefunctioninfo added
illided May 5, 2021
27b68ca
remove CircleCI badge
vovak May 6, 2021
e49a9dc
fix spelling of `astminer`
vovak May 6, 2021
b2e87a6
fix cli script: do not run gradle every time, change wording
vovak May 6, 2021
a6bbc54
fix readme: adjust order, fix wording, add links to sections
vovak May 6, 2021
21ac7fc
fix links in readme
vovak May 6, 2021
5474afc
resolved all compilation errors
Furetur May 6, 2021
16edca9
removed maxWordsNumber == -1 checks
Furetur May 6, 2021
5611d0e
Merge branch 'function-info' into pipeline
Furetur May 6, 2021
8d4d32c
added tests
Furetur May 6, 2021
ae28652
added 1 todo
Furetur May 7, 2021
2bddf86
cleaned up renaming MethodInfo* to FunctionInfo*
Furetur May 7, 2021
fb48197
Merge branch 'function-info' into function-info-implementation
Furetur May 7, 2021
e9ed485
Merge pull request #1 from Furetur/function-info-implementation
Furetur May 7, 2021
24fe066
removed unused import
Furetur May 7, 2021
51e83d6
removed redundant null-safety (?. and ?:)
Furetur May 7, 2021
55eec33
Merge branch 'parse-result-not-nullable-root' into function-info
Furetur May 7, 2021
bf5c822
removed redundant null-safety (?. and ?:)
Furetur May 7, 2021
184f512
Merge branch 'parse-result-not-nullable-root' into function-info
Furetur May 7, 2021
bf5be24
Merge branch 'function-info' into pipeline
Furetur May 7, 2021
e907e7f
added FilterConfigs, ProblemConfigs, StorageCreatorConfigs
Furetur May 7, 2021
46d73ca
Merge pull request #136 from JetBrains-Research/intergrate_new_ci_pip…
vovak May 9, 2021
dfaa9e9
added working cli command (kinda working)
Furetur May 9, 2021
a9d1411
Merge remote-tracking branch 'origin/intergrate_new_ci_pipeline' into…
Furetur May 11, 2021
2035f66
fixed dependencies after merge
Furetur May 11, 2021
732d937
Merge branch 'master-dev' of github.com:JetBrains-Research/astminer i…
SpirinEgor May 13, 2021
d2b2d79
Add python parser into docker image
SpirinEgor May 13, 2021
291371e
Check installed parsers before running tests
SpirinEgor May 13, 2021
52f2976
Merge branch 'master-dev-current' into parse-result-not-nullable-root
Furetur May 13, 2021
0ceba2c
fix group name of maven artifact
breandan May 14, 2021
e590849
Merge pull request #141 from breandan/patch-1
vovak May 14, 2021
04bece7
Merge pull request #140 from JetBrains-Research/master-dev-current
vovak May 14, 2021
ca7582c
Update minor version
SpirinEgor May 17, 2021
7a2eead
removed circleci config + moved log from log/log.txt to log.txt
Furetur May 18, 2021
6cd908a
better config + config examples
Furetur May 18, 2021
48cfbdb
renamed StorageCreator to StorageFactory
Furetur May 18, 2021
09dd5af
Merge branch 'JetBrains-Research:master-dev' into master-dev
illided May 18, 2021
3e02268
switched to yaml configs
Furetur May 18, 2021
a5c27c2
Merge pull request #135 from Furetur/parse-result-not-nullable-root
SpirinEgor May 18, 2021
2c3da19
Merge branch 'parse-result-not-nullable-root' into function-info
Furetur May 18, 2021
156c690
Merge branch 'JetBrains-Research:master-dev' into master-dev
illided May 18, 2021
45d085c
fixed problems after merge
Furetur May 18, 2021
545052c
gumtreefunctioninfo return type added
illided May 18, 2021
a07e440
keyword Constant added to the tests
illided May 18, 2021
00bc6c4
return type fix
illided May 18, 2021
45ee8e9
params possible fix
illided May 18, 2021
7053810
refactored pipeline and removed all redundant entities
Furetur May 19, 2021
82782d1
reduced filters and problems complexity
Furetur May 19, 2021
4fc5a0f
Merge branch 'master-dev' into pipeline
Furetur May 20, 2021
3fea695
fixed issues after merge except GumTreePythonMethodSplitterTest.kt
Furetur May 20, 2021
17f6edd
fixed naming issues and leftover code
Furetur May 20, 2021
24f30bb
renamed java method splitters to function splitters
Furetur May 20, 2021
42b205d
fixed remaining naming issues and warnings
Furetur May 20, 2021
3561075
fixed label extractor
Furetur May 20, 2021
47cd452
fixed Code2VecJavaMethods.kt
Furetur May 20, 2021
a0d7f6f
migrated old tests
Furetur May 20, 2021
8ded188
renamed stuff
Furetur May 20, 2021
deefe6d
removed several todos
Furetur May 20, 2021
96ec814
made the TODO more descriptive
Furetur May 20, 2021
c01501b
added pipeline documentation
Furetur May 20, 2021
11a3d51
Merge branch 'function-info' into pipeline
Furetur May 20, 2021
059ff07
converted parser types and file extensions into enums
Furetur May 20, 2021
19f27c4
Merge pull request #137 from Furetur/function-info
SpirinEgor May 20, 2021
fce1b97
Merge branch 'master-dev' into pipeline
Furetur May 20, 2021
d97b2fd
Merge branch 'JetBrains-Research:master-dev' into master-dev
illided May 20, 2021
0d6c4b7
Merge branch 'master-dev' into node_refactor
illided May 20, 2021
0bd26b9
conflicts resolved
illided May 20, 2021
b068af0
antlr util test fixed
illided May 20, 2021
f91a401
comment removed
illided May 21, 2021
6e1328a
post order iterator implemented properly
illided May 21, 2021
3993437
improved readability of a few lines of code
Furetur May 23, 2021
2e354d4
now token uses proper constant
illided May 23, 2021
e1ad4cf
Merge pull request #133 from illided/node_refactor
SpirinEgor May 23, 2021
7401a8e
Update minor version
SpirinEgor May 23, 2021
5bf7c73
reimplemented configs
Furetur May 24, 2021
dbe64fe
implemented FileNameExtractor
Furetur May 24, 2021
262e5cc
removed the deprecated function
Furetur May 24, 2021
f66bac6
Merge branch 'master-dev' into pipeline
Furetur May 24, 2021
4483cdb
resolved errors after conflicts
Furetur May 24, 2021
9664112
fixed config examples and logging
Furetur May 24, 2021
5a70bfd
changed code2vec config serial name
Furetur May 24, 2021
5fcede8
extracted logging logic in PipelineBranch.kt into filterNotSupported()
Furetur May 24, 2021
a9c0800
Replace token processor with automatic handling of node token
SpirinEgor May 24, 2021
6f56364
Fix splitting test, accurate matching of self calls
SpirinEgor May 24, 2021
c31ea83
Shield new line symbol
SpirinEgor May 24, 2021
43a88f7
Merge pull request #146 from JetBrains-Research/fix_normalization_pip…
SpirinEgor May 24, 2021
ef18aaa
Merge branch 'master-dev' into pipeline
Furetur May 25, 2021
42f80ba
resolved merge conflicts
Furetur May 25, 2021
73ad122
empty commit
Furetur May 25, 2021
6e1ea1c
Merge branch 'master-pipeline' into pipeline
Furetur May 25, 2021
4377d84
Add log file to gitignore
SpirinEgor May 25, 2021
633115c
Use jar execution as docker entry point
SpirinEgor May 25, 2021
ff82ca6
php lexer and parser added
illided May 28, 2021
bbfe5da
php parser adapter added
illided May 28, 2021
f942d7d
php keyword now being parsed correctly
illided May 28, 2021
1372789
commentary added
illided May 29, 2021
a2a1946
raw splitter and function info added
illided May 29, 2021
3bc1332
splitter and test for splitting added (no info being collected yet)
illided May 29, 2021
ced0e15
php lexer base added
illided May 29, 2021
880dc4b
Merge branch 'JetBrains-Research:master-dev' into php
illided May 29, 2021
9ef6b1b
more tests added
illided May 29, 2021
9de4756
parameters extracting added
illided May 29, 2021
1f04816
simple parameter bug fixed
illided May 29, 2021
9ae83d6
actual function info changed
illided May 29, 2021
cdc28c8
parameter extraction bug possible fix
illided May 30, 2021
e75d339
raw algorithm for enclosing extraction added
illided May 30, 2021
0b7efea
tests fixed
illided May 30, 2021
337309a
proper type generating in actual info json
illided May 30, 2021
dfe2bac
bug fixes (proper enclosing extraction and ampersand parameter extrac…
illided May 30, 2021
6931725
renamed to function splitter
illided May 30, 2021
1ac1ed6
factory added
illided May 30, 2021
dc673ac
replaces all whens in pipeline with polymorphic object creation
Furetur May 30, 2021
90b67e6
renamed Problem to LabelExtractor
Furetur May 30, 2021
649a63a
fixed TreeSizeFilterConfig
Furetur May 30, 2021
6718cc9
fixed docs
Furetur May 30, 2021
6bc35b7
fixed initialization of pipeline branches + small fixes
Furetur May 31, 2021
6dcac0c
warning about parser added
illided May 31, 2021
16da8e2
url corrected
illided May 31, 2021
28f8e64
Merge pull request #148 from illided/php
SpirinEgor May 31, 2021
1ecc663
moved all pipeline interfaces to PipelineModel.kt
Furetur Jun 1, 2021
2deb5ed
Merge pull request #143 from Furetur/pipeline
SpirinEgor Jun 1, 2021
4e68a4f
Update astminer dependencies
SpirinEgor Jun 1, 2021
c1e35b0
Merge pull request #150 from JetBrains-Research/Update_dependency_ver…
SpirinEgor Jun 1, 2021
3d99167
added JsonAstStorage.kt with tests, documentation and a config example
Furetur Jun 4, 2021
2ed8c57
added filePath to all function infos
Furetur Jun 4, 2021
b9ba0aa
changed postorder to preorder
Furetur Jun 4, 2021
936909a
renamed splitIntoFunctions to collectFunctionInfo
Furetur Jun 4, 2021
5b07420
reimplemented TreeEnumerator
Furetur Jun 5, 2021
62fbee7
Merge pull request #151 from Furetur/json-ast-storage
SpirinEgor Jun 8, 2021
22a5a3a
Revert "renamed splitIntoFunctions to collectFunctionInfo"
Furetur Jun 9, 2021
8dc5351
Merge pull request #152 from Furetur/functioninfo-filepath
SpirinEgor Jun 9, 2021
00deebc
Update to Java 11
SpirinEgor Jul 13, 2021
b636029
added possible fix and test
illided Jul 14, 2021
4f28c53
Merge pull request #159 from illided/weird_parameter_fix
SpirinEgor Jul 14, 2021
b6dc497
Merge branch 'master-dev' into pipeline_origin
illided Jul 14, 2021
7d6ce44
small merging fix
illided Jul 14, 2021
3ebb9fa
test fix
illided Jul 14, 2021
9b006a0
Merge pull request #160 from illided/pipeline
SpirinEgor Jul 14, 2021
e0f7a4c
Update kotlin
SpirinEgor Jul 21, 2021
ce2f04b
Merge branch 'master-dev' of github.com:JetBrains-Research/astminer i…
SpirinEgor Jul 21, 2021
35e4618
Update antlr php to use file path
SpirinEgor Jul 21, 2021
17ed0c0
Add detekt to astminer
SpirinEgor Jul 21, 2021
c45180b
Merge remote-tracking branch 'origin/pipeline' into pipeline
SpirinEgor Jul 21, 2021
6aab607
Remove multiple companions
SpirinEgor Jul 21, 2021
7a42375
Correct building command in github action
SpirinEgor Jul 21, 2021
e180ed6
Fix all detekt issues
SpirinEgor Jul 22, 2021
ae67a65
Use a temporary folder for storage tests
SpirinEgor Jul 22, 2021
af59468
Move examples to separate module
SpirinEgor Jul 22, 2021
ec9f455
Write docs for pipeline parts, rewrite README, optimise docker
SpirinEgor Jul 23, 2021
6bafcc8
Add code2seq storage
SpirinEgor Jul 26, 2021
884db1d
Fix tests
SpirinEgor Jul 26, 2021
6be0265
Implement filters for gumtree java parser
SpirinEgor Jul 26, 2021
478a8b2
Update paths to configs in github CI config
SpirinEgor Jul 26, 2021
46c1100
Remove ParseResult, rename parsing result factory
SpirinEgor Jul 27, 2021
5416072
Add preprocessing before creating parsing result
SpirinEgor Jul 27, 2021
db6e78b
Merge pull request #149 from JetBrains-Research/pipeline
SpirinEgor Jul 27, 2021
66b0a88
added possibility to save ast with paths
illided Jul 28, 2021
0e6c05c
Run CI on pull requests
SpirinEgor Jul 28, 2021
d7f2978
status bar prototype added
illided Jul 30, 2021
245dca0
Merge branch 'master-dev' into json_storage_path_storing
illided Aug 2, 2021
3177259
some performance improvement
illided Aug 2, 2021
30f1c19
calculation heavy part moved from critical section
illided Aug 2, 2021
23bdd3d
small improvements and rollback to normalized token
illided Aug 2, 2021
5a8482c
parameters list properties now is nullable
illided Aug 2, 2021
c86986d
progress bar close added
illided Aug 3, 2021
ca966bd
code style fixes
illided Aug 3, 2021
a74a732
Use astminer docker container for CI
SpirinEgor Aug 3, 2021
4a9755a
antlr parameters function properties now null on failed parse
illided Aug 3, 2021
ada15d4
Merge branch 'master-dev' into json_storage_path_storing
illided Aug 3, 2021
f8741c8
Merge pull request #166 from illided/json_storage_path_storing
SpirinEgor Aug 3, 2021
e71b170
same update on other parsers
illided Aug 3, 2021
7dee650
proper loggers added
illided Aug 3, 2021
0db4290
Merge branch 'master-dev' into nullable_properties
illided Aug 3, 2021
a8124bd
code style fixes
illided Aug 3, 2021
9e36101
json AST stress tests added
illided Aug 3, 2021
e4e44a3
code style issues fixed
illided Aug 3, 2021
9fdc80e
file distribution fix
illided Aug 4, 2021
05c9bc8
normalization moved to critical section again
illided Aug 4, 2021
306a6a3
lazy stayed in Node but look ahead calculation for antlr was added
illided Aug 4, 2021
a027ce3
code style fixes
illided Aug 4, 2021
c59f16c
gumtree exception wrappers
illided Aug 4, 2021
9f9c43d
edit README.md
vovak Aug 4, 2021
f789c7b
edit cli.sh
vovak Aug 4, 2021
171b16a
edit filters.md
vovak Aug 4, 2021
b3f5d78
edit label_extractors.md
vovak Aug 4, 2021
5452f07
edit parsers.md
vovak Aug 4, 2021
7090f27
edit storages.md
vovak Aug 4, 2021
f23862a
Merge pull request #172 from illided/nullable_properties
SpirinEgor Aug 5, 2021
1699388
path stress tests added
illided Aug 5, 2021
3bf52db
look ahead calculation removed because it had no effect
illided Aug 5, 2021
942ad49
unused import removed
illided Aug 5, 2021
455f33d
Update README.md
ElenaErratic Aug 5, 2021
c7397ee
Merge pull request #173 from JetBrains-Research/proofread
vovak Aug 5, 2021
dec1da8
code style fixes
illided Aug 5, 2021
d78a3fc
holdout prototype added
illided Aug 5, 2021
122cc1d
tests fixed
illided Aug 5, 2021
b8a9eac
code style fixes
illided Aug 5, 2021
8fbeb39
config and multiple tweaks added
illided Aug 5, 2021
08048e6
code style fixes
illided Aug 5, 2021
10cd055
performance config added
illided Aug 5, 2021
e2000da
some little improvements
illided Aug 6, 2021
6c8ba43
num of thread usage added to configs
illided Aug 6, 2021
56b6929
final new lines added
illided Aug 6, 2021
d5c1e49
little improvements in config and docs
illided Aug 6, 2021
7aefe8e
Merge pull request #171 from illided/status_bar
SpirinEgor Aug 6, 2021
968ad32
Merge branch 'master-dev' into dataset_structure
illided Aug 6, 2021
b7eb5f3
conflicts resolved and some refactoring done
illided Aug 6, 2021
175d7a2
code style issues fixed
illided Aug 6, 2021
2e30fcf
spacing around curly
illided Aug 6, 2021
f1400e1
exception type changed
illided Aug 6, 2021
a6b1ad5
unused import removed
illided Aug 6, 2021
de26ca0
small improvements and test fix
illided Aug 6, 2021
d30f321
unused import removed
illided Aug 6, 2021
d2344a4
Merge pull request #175 from illided/dataset_structure
SpirinEgor Aug 7, 2021
e87a904
Merge pull request #174 from illided/gumtree_error_recovery
SpirinEgor Aug 9, 2021
e8e98e7
Prepare new release
SpirinEgor Aug 10, 2021
2c4179d
Update version in README
SpirinEgor Aug 10, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 0 additions & 66 deletions .circleci/config.yml

This file was deleted.

77 changes: 77 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
name: Build with lint

on: [push, pull_request]

jobs:
setup:
runs-on: ubuntu-latest
container: voudy/astminer

steps:
- uses: actions/checkout@v2
- name: Cache Gradle dependencies
uses: actions/cache@v2
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
restore-keys: |
${{ runner.os }}-gradle-

build:
needs: setup
runs-on: ubuntu-latest
container: voudy/astminer

steps:
- uses: actions/checkout@v2

- name: Cache Gradle dependencies
uses: actions/cache@v2
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
restore-keys: |
${{ runner.os }}-gradle-

- name: Build astminer
run: ./gradlew build

- name: Upload Test Report
uses: actions/upload-artifact@v2
if: ${{ always() }}
with:
name: test-report
path: build/astminer/reports/tests/**/*

- name: Upload Detekt Report
uses: github/codeql-action/upload-sarif@v1
if: ${{ always() }}
with:
sarif_file: build/astminer/reports/detekt/detekt.sarif

run-on-configs:
needs: build
runs-on: ubuntu-latest
container: voudy/astminer

steps:
- uses: actions/checkout@v2

- name: Prepare shadowJar
run: ./gradlew shadowJar

- name: Run antlr_java_js_ast.yaml
run: java -jar build/shadow/astminer.jar configs/antlr_java_js_ast.yaml

- name: Run antlr_python_paths.yaml
run: java -jar build/shadow/astminer.jar configs/antlr_python_paths.yaml

- name: Run gumtree_java_ast.yaml
run: java -jar build/shadow/astminer.jar configs/gumtree_java_ast.yaml

- name: Run gumtree_java_function_names_paths.yaml
run: java -jar build/shadow/astminer.jar configs/gumtree_java_function_names_paths.yaml
12 changes: 10 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
*.iml
.idea/
.gradle/
examples/out/
.DS_Store

src/main/generated/
build/

examples_output/
output/
*.csv
log.txt

20 changes: 20 additions & 0 deletions .space.kts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
job("Release") {
startOn {
gitPush {
branchFilter {
+"refs/tags/*"
}
}
}

container(image="voudy/astminer") {
env["PUBLISH_USER"] = Secrets("publish_user")
env["PUBLISH_PASSWORD"] = Secrets("publish_password")

shellScript {
content = """
./gradlew test publish
"""
}
}
}
27 changes: 27 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
FROM ubuntu:20.04

LABEL desc="Docker container to run ASTMiner with all preinstalled requirements"

# Instal OpenJDK11
RUN apt-get update && apt-get install -y openjdk-11-jdk

# Install G++ (required for Fuzzy parser)
RUN apt-get update && apt-get install -y g++

# Install PythonParser for GumTree
ARG PYTHONPARSER_REPO=https://raw.githubusercontent.com/JetBrains-Research/pythonparser/master
RUN apt-get update && \
apt-get install -y python3.8 python3-pip git wget && \
mkdir pythonparser && \
cd pythonparser && \
wget $PYTHONPARSER_REPO/requirements.txt && \
wget $PYTHONPARSER_REPO/src/main/python/pythonparser/pythonparser_3.py -O pythonparser && \
pip3 install -r requirements.txt && \
chmod +x pythonparser
ENV PATH="/pythonparser:${PATH}"

# Copy astminer shadow jar
WORKDIR astminer
COPY ./build/shadow/astminer.jar .

ENTRYPOINT ["java", "-jar", "astminer.jar"]
Loading