-
Notifications
You must be signed in to change notification settings - Fork 0
/
pySparkJobRequestWithPyEnv
43 lines (42 loc) · 1.19 KB
/
pySparkJobRequestWithPyEnv
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
```
{"descriptor":
"dataProduct:
id: urn:dmb:dp:dmName:dpName:dpMajorVrs
name: dpName
domain: dmName
environment: environment
version: dpVersion
dataProductOwner: dataProductOwner
specific: {}
component:
id: urn:dmb:cmp:dmName:dpName:dpMajorVrs:componentName
name: componentName
description: description
version: cmpVersion
specific:
cdeService: cdeService
cdeCluster: cdeCluster
jobName: jobName
pythonFile: s3://path/to/pythonFile.py
pythonVersion: python3
requirementsFile: s3://path/to/requirements.txt
pyMirror: pyMirror
jobConfig:
args: []
dependencies:
- s3://path/to/file.py
- s3://path/to/file.egg
- s3://path/to/file.zip
driverCores: 1
driverMemory: 2g
executorCores: 1
executorMemory: 2g
numExecutors: 4
logLevel: INFO
conf: {}
schedule:
cronExpression: '0 * * * *'
startDate: 2021-05-19T09:00:34Z
endDate: 2021-05-20T12:00:00Z
"}
```