Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(business-types): initial implementation of SIP-78 #18794

Merged
merged 167 commits into from
May 16, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
167 commits
Select commit Hold shift + click to select a range
de12c85
add BUSINESS_TYPE_ADDONS to config with example callback
villebro Sep 27, 2021
4e23585
Removing uneeded whitespace
cccs-RyanS Sep 27, 2021
8e608e8
[Work in progress] Modifying cidr function to allow for single ip an…
cccs-RyanS Sep 27, 2021
7eb030d
Added test REST endpoint, added some more ports
dparent1 Sep 28, 2021
4f67986
Moving code from config.py into the business api
dparent1 Sep 28, 2021
b4a9c83
Removing commented out bits.
dparent1 Sep 28, 2021
e9392fe
Adding fucntion dict back to the config
cccs-RyanS Sep 29, 2021
be9db34
Moving business_type endpoint to charts
cccs-RyanS Sep 29, 2021
ea58232
Adding schema for get endpoint
cccs-RyanS Sep 29, 2021
70759c5
Removing imports, updating docstring, fixing typo
dparent1 Sep 29, 2021
61d461c
Adding translation dict
cccs-RyanS Oct 1, 2021
ed5478c
Fixing ops
cccs-RyanS Oct 1, 2021
0e4cecc
Adding check for list
cccs-RyanS Oct 1, 2021
4069544
Modifying changes to add quotes where needed
dparent1 Oct 1, 2021
d8514e0
Merge branch 'adding-ports-ips' of github.com:superset-collab/superse…
dparent1 Oct 1, 2021
a2f72da
Adding in some code to call the filter config
dparent1 Oct 1, 2021
be44080
Porting Ryan's changes
Oct 21, 2021
59b83e3
Adding migration script (as per Ryan's PR)
Oct 21, 2021
02aaa3a
Fixing typo
Oct 21, 2021
59d7486
Prettier fixes
Nov 3, 2021
1b346c9
[CLDN-1043] Adding rough version of filter changes for business types
cccs-RyanS Nov 18, 2021
bb9410c
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
cccs-RyanS Nov 18, 2021
8ba841c
Merge branch 'master' into villebro/business-types-ui-2
villebro Nov 19, 2021
cf71c91
fix down migration
villebro Nov 19, 2021
d109ce0
Merge branch 'villebro/business-types-ui-2' into business-types-ui
Nov 22, 2021
7f63ff9
Fixing bugs after merge
Nov 22, 2021
7a32250
adding functionality to appy filters in back end
Nov 26, 2021
3587f95
Fixing linting issues
Nov 26, 2021
8c82103
Merge branch 'master' into business-types-ui
villebro Nov 29, 2021
e6d0de6
fix down revision
villebro Nov 29, 2021
b0102e2
Changing conversion callback to handle multiple values at once
Nov 29, 2021
d2ea94b
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Nov 29, 2021
0460616
Adding string representation of values
Nov 29, 2021
26e4ac6
Code cleanup plus fixing debouce to only be called once for each entry
Nov 30, 2021
0170e4c
Removing non needed logginh
Nov 30, 2021
5c4cb0e
Merge branch 'master' into business-types-ui
villebro Dec 2, 2021
263f792
Changing operator list to use sting values
Dec 4, 2021
1223f80
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Dec 4, 2021
a0a962d
Using text value operators
Dec 4, 2021
9b00f55
Removing clear operator call
Dec 4, 2021
b9e1d5c
Moving business type endpoints
Dec 7, 2021
6a7fe40
Merge branch 'master' into business-types-ui
villebro Dec 9, 2021
ee16c97
fix down revision
villebro Dec 9, 2021
d2024d2
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Dec 9, 2021
baa7042
Adding port functions
Dec 9, 2021
61e9a4d
Merge branch 'master' into business-types-ui
villebro Dec 22, 2021
2a74006
update migration
villebro Dec 22, 2021
ebe95e6
fix bad rebase and add ff
villebro Dec 22, 2021
4fa1d0d
implement validator
villebro Dec 22, 2021
84a1dd8
dont add invalid values to response
villebro Dec 22, 2021
753de93
[CLDN-1205] Added a new exception type for a business type translatio…
cccs-Dustin Jan 26, 2022
1afa7d6
[CLDN-1205] Added meaningful error message for port translation errors
cccs-Dustin Jan 27, 2022
8906b36
Removing status field from businesstype Response and adding in error …
Jan 27, 2022
4090439
[CLDN-1205] Added check to make sure the port business type is within…
Jan 27, 2022
b3eba02
[CLDN-1205] Fixed the if statement that checks to see if the string_v…
Jan 28, 2022
bce157b
[CLDN-1205] Fixed an error where it was trying to use string_value in…
Jan 28, 2022
9d86475
[CLDN-1207] Added unit tests for the cidr_func and port_translation_f…
Jan 28, 2022
d6937ff
[CLDN-1207] removed the assertRaises line as it does not work with th…
Jan 28, 2022
ecbc974
Merge branch 'feature/CLDN-1205' into feature/CLDN-1207
cccs-Dustin Jan 28, 2022
547abf5
[CLDN-1207] Added the skeleton of the test_cidr_translate_filter_func…
cccs-Dustin Jan 28, 2022
089f704
[CLDN-1207] Added the remainder of the back-end unit tests for the bu…
cccs-Dustin Jan 31, 2022
2bcebee
[CLDN-1207] Fixed the syntax error which caused the test_cidr_transla…
cccs-Dustin Jan 31, 2022
b17ad7d
[CLDN-1207] Removed the logging that was added for debugging purposes
cccs-Dustin Feb 1, 2022
0be8622
[CLDN-1207] Formatted the commands_tests.py file to make it nicer to …
cccs-Dustin Feb 1, 2022
dc475a8
[CLDN-1207] Fixed the code so that it conformed to the pylint require…
cccs-Dustin Feb 1, 2022
06b02c3
[CLDN-1207] Modified some of the docstrings so they made better use o…
cccs-Dustin Feb 1, 2022
0ce8cbc
[CLDN-1207] Added the beginnings of the unit tests for the
cccs-Dustin Feb 1, 2022
1fec9b1
[CLDN-1207] Added a comment to the top of the commands_tests.py file …
cccs-Dustin Feb 2, 2022
f7a1ce1
[CLDN-1207] Added a grammar fix to the comments describing how to run…
cccs-Dustin Feb 2, 2022
afa5e09
[CLDN-1207] Modified the description of the business_type API endpoin…
cccs-Dustin Feb 2, 2022
7182921
[CLDN-1207] Added further instructions on how to run the unit tests t…
cccs-Dustin Feb 2, 2022
8ee8e59
Merge branch 'master' into business-types-ui
villebro Feb 3, 2022
2e04139
add request validation
villebro Feb 3, 2022
2dd5358
disable request if business type missing
villebro Feb 3, 2022
3b9773d
[CLDN-1207] Unit tests for the business type API are now working, how…
Feb 4, 2022
08b2948
Removing businesss types deffinitons from config
Feb 7, 2022
16b427a
Adding unit tests for business types
Feb 7, 2022
16bdf2c
Adding select to only show valid business types
Feb 10, 2022
c1ecf9d
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Feb 10, 2022
77d9559
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Feb 10, 2022
03c6bbd
Fixed Enzyme tests
Feb 10, 2022
8a68770
Added scalfolding for selecting filter dropdown
Feb 10, 2022
550b3a6
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Feb 10, 2022
ac2cbde
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Feb 11, 2022
ba62deb
Adding intigration tests
Feb 11, 2022
318faa3
Merge branch 'master' into business-types-ui
villebro Feb 11, 2022
8177ada
fix revision
villebro Feb 11, 2022
447c168
fix typos and unnecessary requests
villebro Feb 11, 2022
d304d51
break out useBusinessTypes
villebro Feb 11, 2022
5a437db
Added front-end RTL unit tests for the business type API endpoint
cccs-Dustin Feb 11, 2022
9bd2c29
Added front-end RTL unit tests for the business type API endpoint
cccs-Dustin Feb 11, 2022
bbd044b
Fixed error from unit tests
cccs-Dustin Feb 11, 2022
a8cbdee
Added a unit test to ensure the operator list is updated after a busi…
cccs-Dustin Feb 11, 2022
cff1b57
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Feb 11, 2022
24496a8
Removing elect compoenet for business types
Feb 14, 2022
e590043
Adding feature flag and allowing saving when no business type present
Feb 14, 2022
e537b70
fixing useEffect hooks
Feb 14, 2022
39df89c
Adding feature flag to model
Feb 14, 2022
f78b946
Changing behavior such that an empty string returns a default response
Feb 14, 2022
821e4b9
add form validation
villebro Feb 14, 2022
42485d5
Merge branch 'master' into business-types-ui
villebro Feb 14, 2022
d4a7d7f
Modified comments in unit test as command to run test has changed
cccs-Dustin Feb 14, 2022
b84a9bf
Modified comments in unit test as filename to run test has changed
cccs-Dustin Feb 14, 2022
44ba940
Modified the api_tests.py file to conform to the linting requirements
cccs-Dustin Feb 14, 2022
301a231
Changed the name of one of the tests to reflect what the test is actu…
cccs-Dustin Feb 14, 2022
654dcc8
Added cypress back to the package.json
cccs-Dustin Feb 14, 2022
ec94565
Added informative comments
cccs-Dustin Feb 14, 2022
378e918
Updated comments in files as well as removed imports which were not b…
cccs-Dustin Feb 14, 2022
f37fc4e
Changes made by npm run prettier
cccs-Dustin Feb 14, 2022
6cffadf
Fixed spelling mistakes
cccs-Dustin Feb 14, 2022
abc0a67
Updated models.py to remove placeholder comments used in development
cccs-Dustin Feb 14, 2022
3796c72
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Feb 14, 2022
3783714
Added feature flag mocking in unit test
cccs-Dustin Feb 14, 2022
2e21d4e
Merge branch 'master' into business-types-ui
Feb 14, 2022
f93e09d
Fixing open api failure
Feb 15, 2022
ddd7e9e
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Feb 15, 2022
293878a
Fixing business types to pass unit tests
Feb 15, 2022
2dd6f01
Reverting unsafe connections back to false
Feb 15, 2022
93af873
Removing print statement
Feb 15, 2022
698036f
Adding business tpye to export test
Feb 15, 2022
6e27c23
setting default feature flag to false for business type
Feb 15, 2022
1cf6bdf
Reverting pre commit
Feb 15, 2022
b6dcbac
Reverting pre commit and running pre commit
Feb 15, 2022
96a4d33
Reverting pre commit and running pre commit
Feb 15, 2022
80ce50f
Fixing formatting
Feb 15, 2022
e1c7996
Adding license
Feb 15, 2022
43fe11a
Fixing Linting
Feb 15, 2022
0f374a6
Protecting api enpoints
Feb 15, 2022
119559a
updating model
Feb 15, 2022
133830b
Fixing code path when business type exists
Feb 15, 2022
0a8f095
Linting
Feb 15, 2022
c5302c9
Linting
Feb 15, 2022
768e162
Fixing linting
Feb 15, 2022
3c37967
Fixing spelling
Feb 15, 2022
a583fa2
Fixing schemas
Feb 15, 2022
d12bdb1
Fixing app import
Feb 15, 2022
91fbe57
fixing item render
Feb 15, 2022
fc41b7f
Added RTL test to make sure business type operator list is updated af…
cccs-Dustin Feb 15, 2022
827fe56
Fixing linting
Feb 15, 2022
73af3c9
Merge branch 'master' into business-types-ui
villebro Feb 25, 2022
853ae9c
fix migration
villebro Feb 25, 2022
09de7ee
Changing unit tests
Mar 1, 2022
53ea4f3
Merge branch 'business-types-ui' of github.com:superset-collab/supers…
Mar 1, 2022
6c70457
Merge branch 'master' into business-types-ui
Apr 5, 2022
e444630
Fixing import and DB migration after rebase
Apr 6, 2022
5d35adb
Renaming to advanced types
Apr 6, 2022
c26168d
Fixing Linting
Apr 6, 2022
f5393e0
More renaming
Apr 7, 2022
38fbe66
Removing uneeded change
Apr 7, 2022
6a9b1d5
Fixing linting and test errors
Apr 7, 2022
4ab7996
Removing unused imports
Apr 7, 2022
253771e
Merge branch 'master' into business-types-ui
Apr 8, 2022
17056bb
linting
Apr 8, 2022
0b7f3a6
Adding more detailed name for migration
Apr 8, 2022
ed9dfb9
Moving files to plugins
Apr 8, 2022
d29f00f
more renaming
Apr 8, 2022
96fc022
Fixing schema name
Apr 8, 2022
92387d0
Disabling feature flag that should not be enabled by default
Apr 8, 2022
027f793
Adding extra cehck
Apr 8, 2022
498d8c3
NameChange
Apr 8, 2022
1e6aef6
Merge branch 'master' into business-types-ui
Apr 20, 2022
255de4d
formatting
Apr 20, 2022
f68bbdc
Fixing equals check
Apr 21, 2022
e8c75ee
Merge branch 'master' into business-types-ui
Apr 27, 2022
3e5cc20
Merge branch 'master' into business-types-ui
May 13, 2022
897aebb
Moveing all advanced type classes and types to one file, and converti…
May 16, 2022
c23ad8e
Adding advanced type to test and fix linitng
May 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions docs/static/resources/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -3205,6 +3205,12 @@
"owners": {
"$ref": "#/components/schemas/DashboardRestApi.get_list.User2"
},
"advanced_data_type": {
"maxLength": 255,
"minLength": 1,
"nullable": true,
"type": "string"
},
"position_json": {
"nullable": true,
"type": "string"
Expand Down Expand Up @@ -4245,6 +4251,12 @@
"nullable": true,
"type": "string"
},
"advanced_data_type": {
"maxLength": 255,
"minLength": 1,
"nullable": true,
"type": "string"
},
"uuid": {
"format": "uuid",
"nullable": true,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ export enum FeatureFlag {
ALERT_REPORTS = 'ALERT_REPORTS',
CLIENT_CACHE = 'CLIENT_CACHE',
DYNAMIC_PLUGINS = 'DYNAMIC_PLUGINS',
ENABLE_ADVANCED_DATA_TYPES = 'ENABLE_ADVANCED_DATA_TYPES',
SCHEDULED_QUERIES = 'SCHEDULED_QUERIES',
SQL_VALIDATORS_BY_ENGINE = 'SQL_VALIDATORS_BY_ENGINE',
ESTIMATE_QUERY_COST = 'ESTIMATE_QUERY_COST',
Expand Down
249 changes: 178 additions & 71 deletions superset-frontend/src/components/Datasource/DatasourceEditor.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -173,22 +173,46 @@ function ColumnCollectionTable({
return (
<CollectionTable
collection={columns}
tableColumns={[
'column_name',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]}
sortColumns={[
'column_name',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]}
tableColumns={
isFeatureEnabled(FeatureFlag.ENABLE_ADVANCED_DATA_TYPES)
? [
'column_name',
'advanced_data_type',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]
: [
'column_name',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]
}
sortColumns={
isFeatureEnabled(FeatureFlag.ENABLE_ADVANCED_DATA_TYPES)
? [
'column_name',
'advanced_data_type',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]
: [
'column_name',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]
}
allowDeletes
allowAddItem={allowAddItem}
itemGenerator={itemGenerator}
Expand Down Expand Up @@ -243,6 +267,20 @@ function ColumnCollectionTable({
}
/>
)}
{isFeatureEnabled(FeatureFlag.ENABLE_ADVANCED_DATA_TYPES) ? (
<Field
fieldKey="advanced_data_type"
label={t('Advanced data type')}
control={
<TextControl
controlId="advanced_data_type"
placeholder={t('Advanced Data type')}
/>
}
/>
) : (
<></>
)}
<Field
fieldKey="python_date_format"
label={t('Datetime format')}
Expand Down Expand Up @@ -300,62 +338,131 @@ function ColumnCollectionTable({
</Fieldset>
</FormContainer>
}
columnLabels={{
column_name: t('Column'),
type: t('Data type'),
groupby: t('Is dimension'),
is_dttm: t('Is temporal'),
main_dttm_col: t('Default datetime'),
filterable: t('Is filterable'),
}}
columnLabels={
isFeatureEnabled(FeatureFlag.ENABLE_ADVANCED_DATA_TYPES)
? {
column_name: t('Column'),
advanced_data_type: t('Advanced data type'),
type: t('Data type'),
groupby: t('Is dimension'),
is_dttm: t('Is temporal'),
main_dttm_col: t('Default datetime'),
filterable: t('Is filterable'),
}
: {
column_name: t('Column'),
type: t('Data type'),
groupby: t('Is dimension'),
is_dttm: t('Is temporal'),
main_dttm_col: t('Default datetime'),
filterable: t('Is filterable'),
}
}
onChange={onColumnsChange}
itemRenderers={{
column_name: (v, onItemChange, _, record) =>
editableColumnName ? (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
<TextControl value={v} onChange={onItemChange} />
</StyledLabelWrapper>
) : (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
{v}
</StyledLabelWrapper>
),
main_dttm_col: (value, _onItemChange, _label, record) => {
const checked = datasource.main_dttm_col === record.column_name;
const disabled = !columns.find(
column => column.column_name === record.column_name,
).is_dttm;
return (
<Radio
data-test={`radio-default-dttm-${record.column_name}`}
checked={checked}
disabled={disabled}
onChange={() =>
onDatasourceChange({
...datasource,
main_dttm_col: record.column_name,
})
}
/>
);
},
type: d => (d ? <Label>{d}</Label> : null),
is_dttm: checkboxGenerator,
filterable: checkboxGenerator,
groupby: checkboxGenerator,
}}
itemRenderers={
isFeatureEnabled(FeatureFlag.ENABLE_ADVANCED_DATA_TYPES)
? {
column_name: (v, onItemChange, _, record) =>
editableColumnName ? (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
<EditableTitle
canEdit
title={v}
onSaveTitle={onItemChange}
/>
</StyledLabelWrapper>
) : (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
{v}
</StyledLabelWrapper>
),
main_dttm_col: (value, _onItemChange, _label, record) => {
const checked = datasource.main_dttm_col === record.column_name;
const disabled = !columns.find(
column => column.column_name === record.column_name,
).is_dttm;
return (
<Radio
data-test={`radio-default-dttm-${record.column_name}`}
checked={checked}
disabled={disabled}
onChange={() =>
onDatasourceChange({
...datasource,
main_dttm_col: record.column_name,
})
}
/>
);
},
type: d => (d ? <Label>{d}</Label> : null),
advanced_data_type: d => (
<Label onChange={onColumnsChange}>{d}</Label>
),
is_dttm: checkboxGenerator,
filterable: checkboxGenerator,
groupby: checkboxGenerator,
}
: {
column_name: (v, onItemChange, _, record) =>
editableColumnName ? (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
<TextControl value={v} onChange={onItemChange} />
</StyledLabelWrapper>
) : (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
{v}
</StyledLabelWrapper>
),
main_dttm_col: (value, _onItemChange, _label, record) => {
const checked = datasource.main_dttm_col === record.column_name;
const disabled = !columns.find(
column => column.column_name === record.column_name,
).is_dttm;
return (
<Radio
data-test={`radio-default-dttm-${record.column_name}`}
checked={checked}
disabled={disabled}
onChange={() =>
onDatasourceChange({
...datasource,
main_dttm_col: record.column_name,
})
}
/>
);
},
type: d => (d ? <Label>{d}</Label> : null),
is_dttm: checkboxGenerator,
filterable: checkboxGenerator,
groupby: checkboxGenerator,
}
}
/>
);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ export default class AdhocFilterEditPopover extends React.Component {
this.onMouseMove = this.onMouseMove.bind(this);
this.onMouseUp = this.onMouseUp.bind(this);
this.onAdhocFilterChange = this.onAdhocFilterChange.bind(this);
this.setSimpleTabIsValid = this.setSimpleTabIsValid.bind(this);
this.adjustHeight = this.adjustHeight.bind(this);
this.onTabChange = this.onTabChange.bind(this);

Expand All @@ -106,6 +107,7 @@ export default class AdhocFilterEditPopover extends React.Component {
width: POPOVER_INITIAL_WIDTH,
height: POPOVER_INITIAL_HEIGHT,
activeKey: this.props?.adhocFilter?.expressionType || 'SIMPLE',
isSimpleTabValid: true,
};

this.popoverContentRef = React.createRef();
Expand All @@ -124,6 +126,10 @@ export default class AdhocFilterEditPopover extends React.Component {
this.setState({ adhocFilter });
}

setSimpleTabIsValid(isValid) {
this.setState({ isSimpleTabValid: isValid });
}

onSave() {
this.props.onChange(this.state.adhocFilter);
this.props.onClose();
Expand Down Expand Up @@ -214,6 +220,7 @@ export default class AdhocFilterEditPopover extends React.Component {
onHeightChange={this.adjustHeight}
partitionColumn={partitionColumn}
popoverRef={this.popoverContentRef.current}
validHandler={this.setSimpleTabIsValid}
/>
</ErrorBoundary>
</Tabs.TabPane>
Expand Down Expand Up @@ -252,7 +259,7 @@ export default class AdhocFilterEditPopover extends React.Component {
</Button>
<Button
data-test="adhoc-filter-edit-popover-save-button"
disabled={!stateIsValid}
disabled={!stateIsValid || !this.state.isSimpleTabValid}
buttonStyle={
hasUnsavedChanges && stateIsValid ? 'primary' : 'default'
}
Expand Down
Loading