Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDK-381 support for spark streaming cluster #289

Merged
merged 24 commits into from
Nov 4, 2019
Merged

SDK-381 support for spark streaming cluster #289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
b1a4ffc
SDK-335: GCP creds schema changes (#263)
tarrygo Feb 15, 2019
3631d0e
[SDK-338] Added option to enable Rubix (#265)
abhijithshankar93 Feb 21, 2019
62468d2
SDK-323: GCP changes for preemptible nodes (#266)
akaranjkar-qu Mar 1, 2019
756d517
fixes #175. utf-8 errors in Python 3 (#208)
mcarlsen Mar 19, 2019
8afa7fe
SDK-344: Enable use of port argument for DbTaps #267
chattarajoy Mar 19, 2019
c7b9cb6
Added exception instead of "pass" to avoid silently skipping of file …
Mar 19, 2019
ac95aaf
SDK-339 : Added comments on how to use List command. (#269)
lordozb Mar 26, 2019
70b075e
SDK-345: Add custom resource group and start-stop related params for …
akaranjkar-qu Apr 1, 2019
7fb3292
SDK-340 : Support for Airflow Version and Python Version (#270)
lordozb Apr 19, 2019
0db727c
SDK-350 MultiCloud Fix for Script Location Parameter (#272)
chattarajoy May 8, 2019
d1cccf4
Release Version 1.11.1
msumit May 14, 2019
838819e
SDK-XXX: Update readme
msumit May 14, 2019
6951fe2
SDK-352 : Add storage config params - block_volume_count and block_vo…
thispc May 21, 2019
f8e3548
SDK-354 Default to S3 Signature Version V4 (#273)
chattarajoy May 21, 2019
35fe334
Release Version 1.12.0
chattarajoy May 21, 2019
87989ce
SDK-109 Proper Error Message when results unavailable (#276)
lordozb Jun 19, 2019
915bdea
fix: dev: SDK-364: Oracle Multi AD changes required in qds-sdk
chattarajoy Jul 29, 2019
c5c035b
Update requests and urllib library versions (#280)
brickyard Jul 31, 2019
7661327
SDK-366 Drop Support for Python 2.6 (#281)
chattarajoy Jul 31, 2019
2d72f4b
Release Version 1.13.0
chattarajoy Jul 31, 2019
22880b2
DeepSource.io Integeration (#275)
sanketsaurav Aug 7, 2019
488e3e8
DeepSource Integeration for Commit checks and Test Coverage (#282)
chattarajoy Aug 10, 2019
4816274
spark streaming cluster support
saiyam1712 Oct 24, 2019
c5752dd
merge conflict resolved and added test cases for spark streaming cluster
saiyam1712 Nov 4, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion qds_sdk/engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ def engine_parser(argparser):
engine_group = argparser.add_argument_group("engine settings")
engine_group.add_argument("--flavour",
dest="flavour",
choices=["hadoop", "hadoop2", "presto", "spark", "hbase", "airflow", "deeplearning"],
choices=["hadoop", "hadoop2", "presto", "spark", "sparkstreaming", "hbase", "airflow", "deeplearning"],
default=None,
help="Set engine flavour")

Expand Down
19 changes: 18 additions & 1 deletion tests/test_clusterv2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -460,6 +460,23 @@ def test_spark_engine_config(self):
'custom_spark_config': 'spark-overrides'}},
'cluster_info': {'label': ['test_label'],}})

def test_sparkstreaming_engine_config(self):
with tempfile.NamedTemporaryFile() as temp:
temp.write("config.properties:\na=1\nb=2".encode("utf8"))
temp.flush()
sys.argv = ['qds.py', '--version', 'v2', 'cluster', 'create', '--label', 'test_label',
'--flavour', 'sparkstreaming', '--custom-spark-config', 'spark-overrides']
Qubole.cloud = None
print_command()
Connection._api_call = Mock(return_value={})
qds.main()
Connection._api_call.assert_called_with('POST', 'clusters',
{'engine_config':
{'flavour': 'sparkstreaming',
'spark_settings': {
'custom_spark_config': 'spark-overrides'}},
'cluster_info': {'label': ['test_label'],}})

def test_airflow_engine_config(self):
with tempfile.NamedTemporaryFile() as temp:
temp.write("config.properties:\na=1\nb=2".encode("utf8"))
Expand Down Expand Up @@ -787,7 +804,7 @@ def test_engine_config(self):
temp.write("a=1\nb=2".encode("utf8"))
temp.flush()
sys.argv = ['qds.py', '--version', 'v2', 'cluster', 'update', '123',
'--use-qubole-placement-policy', '--enable-rubix',
'--use-qubole-placement-policy', '--enable-rubix',
'--custom-hadoop-config',temp.name]
Qubole.cloud = None
print_command()
Expand Down