Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support reading spark conf file from cli #1600

Merged
merged 4 commits into from
Apr 12, 2022

Conversation

tobegit3hub
Copy link
Collaborator

  • Rename the config of using SparkSQL
  • Support reading spark conf file in CLI with gflag parameter
  • Pass map of config for TaskManager APIs
  • Add template of spark_conf.ini

Now we can create the spark config file in client side and submit offline jobs with these configuration.

./bin/openmldb --zk_cluster=127.0.0.1:2181 --zk_root_path=/openmldb --role=sql_client --spark_conf=../release/conf/spark_conf.ini

@tobegit3hub tobegit3hub added the enhancement New feature or request label Apr 8, 2022
@tobegit3hub tobegit3hub self-assigned this Apr 8, 2022
@tobegit3hub tobegit3hub added this to the v0.5 milestone Apr 8, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Apr 8, 2022

SDK Test Report

  72 files    72 suites   6m 40s ⏱️
172 tests 169 ✔️ 3 💤 0
212 runs  209 ✔️ 3 💤 0

Results for commit eed1a17.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 8, 2022

Linux Test Report

       55 files       182 suites   43m 58s ⏱️
  8 201 tests   8 201 ✔️ 0 💤 0
12 117 runs  12 117 ✔️ 0 💤 0

Results for commit eed1a17.

♻️ This comment has been updated with latest results.

@codecov
Copy link

codecov bot commented Apr 8, 2022

Codecov Report

Merging #1600 (eed1a17) into main (912f022) will decrease coverage by 0.02%.
The diff coverage is 6.25%.

@@             Coverage Diff              @@
##               main    #1600      +/-   ##
============================================
- Coverage     67.22%   67.19%   -0.03%     
  Complexity      323      323              
============================================
  Files           600      600              
  Lines        114041   114078      +37     
  Branches        995      999       +4     
============================================
  Hits          76659    76659              
- Misses        37175    37212      +37     
  Partials        207      207              
Impacted Files Coverage Δ
...m/openmldb/taskmanager/spark/SparkJobManager.scala 0.00% <0.00%> (ø)
src/client/taskmanager_client.cc 0.00% <0.00%> (ø)
src/sdk/sql_cluster_router.cc 52.12% <0.00%> (-0.48%) ⬇️
src/sdk/sql_cluster_router.h 83.33% <ø> (ø)
...4paradigm/openmldb/batch/OpenmldbBatchConfig.scala 86.66% <100.00%> (ø)
...4paradigm/openmldb/batch/api/OpenmldbSession.scala 57.57% <100.00%> (ø)
src/zk/dist_lock.cc 81.81% <0.00%> (-1.52%) ⬇️
src/catalog/sdk_catalog.cc 74.48% <0.00%> (-1.05%) ⬇️
src/catalog/tablet_catalog.cc 68.56% <0.00%> (-0.90%) ⬇️
src/tablet/tablet_impl.cc 42.50% <0.00%> (-0.17%) ⬇️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77d37b4...eed1a17. Read the comment docs.

@zhanghaohit zhanghaohit self-requested a review April 12, 2022 06:45
@tobegit3hub tobegit3hub merged commit 8f9cd29 into 4paradigm:main Apr 12, 2022
@lumianph lumianph mentioned this pull request May 13, 2022
39 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants