Merge branch 'master' of https://github.com/apache/superset into hugh…

…/migrate-estimate_query_cost-to-v1

.github/workflows/superset-python-unittest.yml

@@ -50,6 +50,8 @@ jobs:
             mkdir ${{ github.workspace }}/.temp
       - name: Python unit tests
         if: steps.check.outcome == 'failure'
+        env:
+          SUPERSET_TESTENV: true
         run: |
           pytest --durations-min=0.5 --cov-report= --cov=superset ./tests/common ./tests/unit_tests --cache-clear
       - name: Upload code coverage

UPDATING.md

@@ -24,6 +24,18 @@ assists people when migrating to a new version.
 
 ## Next
 
+- [22809](https://github.com/apache/superset/pull/22809): Migrated endpoint `/superset/sql_json` and `/superset/results/` to `/api/v1/sqllab/execute/` and `/api/v1/sqllab/results/` respectively. Corresponding permissions are `can sql_json on Superset` to `can execute on SQLLab`, `can results on Superset` to `can results on SQLLab`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22931](https://github.com/apache/superset/pull/22931): Migrated endpoint `/superset/get_or_create_table/` to `/api/v1/dataset/get_or_create/`. Corresponding permissions are `can get or create table on Superset` to `can get or create dataset on Dataset`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22882](https://github.com/apache/superset/pull/22882): Migrated endpoint `/superset/filter/<datasource_type>/<int:datasource_id>/<column>/` to `/api/v1/datasource/<datasource_type>/<datasource_id>/column/<column_name>/values/`. Corresponding permissions are `can filter on Superset` to `can get column values on Datasource`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22789](https://github.com/apache/superset/pull/22789): Migrated endpoint `/superset/recent_activity/<user_id>/` to `/api/v1/log/recent_activity/<user_id>/`. Corresponding permissions are `can recent activity on Superset` to `can recent activity on Log`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22913](https://github.com/apache/superset/pull/22913): Migrated endpoint `/superset/csv` to `/api/v1/sqllab/export/`. Corresponding permissions are `can csv on Superset` to `can export csv on SQLLab`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22496](https://github.com/apache/superset/pull/22496): Migrated endpoint `/superset/slice_json/<int:layer_id>` to `/api/v1/chart/<int:id>/data/`. Corresponding permissions are `can slice json on Superset` to `can read on Chart`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22496](https://github.com/apache/superset/pull/22496): Migrated endpoint `/superset/annotation_json/<int:layer_id>` to `/api/v1/chart/<int:id>/data/`. Corresponding permissions are `can annotation json on Superset` to `can read on Chart`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22624](https://github.com/apache/superset/pull/22624): Migrated endpoint `/superset/stop_query/` to `/api/v1/query/stop`. Corresponding permissions are `can stop query on Superset` to `can read on Query`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22579](https://github.com/apache/superset/pull/22579): Migrated endpoint `/superset/search_queries/` to `/api/v1/query/`. Corresponding permissions are `can search queries on Superset` to `can read on Query`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22501](https://github.com/apache/superset/pull/22501): Migrated endpoint `/superset/tables/<int:db_id>/<schema>/` to `/api/v1/database/<int:id>/tables/`. Corresponding permissions are `can tables on Superset` to `can read on Database`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [22611](https://github.com/apache/superset/pull/22611): Migrated endpoint `/superset/queries/` to `api/v1/query/updated_since`. Corresponding permissions are `can queries on Superset` to `can read on Query`. Make sure you add/replace the necessary permissions on any custom roles you may have.
+- [23186](https://github.com/apache/superset/pull/23186): Superset will refuse to start if a default `SECRET_KEY` is detected on a non Flask debug setting.
 - [22022](https://github.com/apache/superset/pull/22022): HTTP API endpoints `/superset/approve` and `/superset/request_access` have been deprecated and their HTTP methods were changed from GET to POST
 - [20606](https://github.com/apache/superset/pull/20606): When user clicks on chart title or "Edit chart" button in Dashboard page, Explore opens in the same tab. Clicking while holding cmd/ctrl opens Explore in a new tab. To bring back the old behaviour (always opening Explore in a new tab), flip feature flag `DASHBOARD_EDIT_CHART_IN_NEW_TAB` to `True`.
 - [20799](https://github.com/apache/superset/pull/20799): Presto and Trino engine will now display tracking URL for running queries in SQL Lab. If for some reason you don't want to show the tracking URL (for example, when your data warehouse hasn't enabled access for to Presto or Trino UI), update `TRACKING_URL_TRANSFORMER` in `config.py` to return `None`.

docker/.env-non-dev

@@ -42,6 +42,7 @@ REDIS_PORT=6379
 FLASK_ENV=production
 SUPERSET_ENV=production
 SUPERSET_LOAD_EXAMPLES=yes
+SUPERSET_SECRET_KEY=TEST_NON_DEV_SECRET
 CYPRESS_CONFIG=false
 SUPERSET_PORT=8088
 MAPBOX_API_KEY=''

docs/docs/installation/alerts-reports.mdx

@@ -193,15 +193,15 @@ creator if either is contained within the list of owners, otherwise the first ow
 will be used) and finally `THUMBNAIL_SELENIUM_USER`, set as follows:
 
 ```python
-from superset.reports.types import ReportScheduleExecutor
+from superset.tasks.types import ExecutorType
 
 ALERT_REPORTS_EXECUTE_AS = [
-    ReportScheduleExecutor.CREATOR_OWNER,
-    ReportScheduleExecutor.CREATOR,
-    ReportScheduleExecutor.MODIFIER_OWNER,
-    ReportScheduleExecutor.MODIFIER,
-    ReportScheduleExecutor.OWNER,
-    ReportScheduleExecutor.SELENIUM,
+    ExecutorType.CREATOR_OWNER,
+    ExecutorType.CREATOR,
+    ExecutorType.MODIFIER_OWNER,
+    ExecutorType.MODIFIER,
+    ExecutorType.OWNER,
+    ExecutorType.SELENIUM,
 ]
 ```
 

docs/docs/installation/configuring-superset.mdx

@@ -23,8 +23,8 @@ SUPERSET_WEBSERVER_PORT = 8088
 # Your App secret key will be used for securely signing the session cookie
 # and encrypting sensitive information on the database
 # Make sure you are changing this key for your deployment with a strong key.
-# You can generate a strong key using `openssl rand -base64 42`
-
+# You can generate a strong key using `openssl rand -base64 42`.
+# Alternatively you can set it with `SUPERSET_SECRET_KEY` environment variable.
 SECRET_KEY = 'YOUR_OWN_RANDOM_GENERATED_SECRET_KEY'
 
 # The SQLAlchemy connection string to your database backend

docs/docs/installation/installing-superset-from-scratch.mdx

@@ -138,6 +138,12 @@ superset load_examples
 # Create default roles and permissions
 superset init
 
+# Build javascript assets
+cd superset-frontend
+npm ci
+npm run build
+cd ..
+
 # To start a development web server on port 8088, use -p to bind to another port
 superset run -p 8088 --with-threads --reload --debugger
 ```

docs/docs/installation/sql-templating.mdx

@@ -30,7 +30,9 @@ made available in the Jinja context:
 For example, to add a time range to a virtual dataset, you can write the following:
 
 ```sql
-SELECT * from tbl where dttm_col > '{{ from_dttm }}' and dttm_col < '{{ to_dttm }}'
+SELECT *
+FROM tbl
+WHERE dttm_col > '{{ from_dttm }}' and dttm_col < '{{ to_dttm }}'
 ```
 
 You can also use [Jinja's logic](https://jinja.palletsprojects.com/en/2.11.x/templates/#tests)
@@ -64,6 +66,41 @@ JINJA_CONTEXT_ADDONS = {
 }
 ```
 
+Default values for jinja templates can be specified via `Parameters` menu in the SQL Lab user interface.
+In the UI you can assign a set of parameters as JSON
+
+```json
+{
+  "my_table": "foo"
+}
+```
+The parameters become available in your SQL (example: `SELECT * FROM {{ my_table }}` ) by using Jinja templating syntax.
+SQL Lab template parameters are stored with the dataset as `TEMPLATE PARAMETERS`.
+
+There is a special ``_filters`` parameter which can be used to test filters used in the jinja template.
+
+```json
+{
+  "_filters": [
+    {
+      "col": "action_type",
+      "op": "IN",
+      "val": ["sell", "buy"]
+    }
+  ]
+}
+```
+
+```sql
+SELECT action, count(*) as times
+FROM logs
+WHERE action in {{ filter_values('action_type'))|where_in }}
+GROUP BY action
+```
+
+Note ``_filters`` is not stored with the dataset. It's only used within the SQL Lab UI.
+
+
 Besides default Jinja templating, SQL lab also supports self-defined template processor by setting
 the `CUSTOM_TEMPLATE_PROCESSORS` in your superset configuration. The values in this dictionary
 overwrite the default Jinja template processors of the specified database engine. The example below
@@ -174,7 +211,7 @@ Here's a concrete example:
 
 - You write the following query in SQL Lab:
 
-  ```
+  ```sql
   SELECT count(*)
   FROM ORDERS
   WHERE country_code = '{{ url_param('countrycode') }}'
@@ -185,15 +222,15 @@ Here's a concrete example:
   and your coworker in the USA the following SQL Lab URL `www.example.com/superset/sqllab?countrycode=US`
 - For your coworker in Spain, the SQL Lab query will be rendered as:
 
-  ```
+  ```sql
   SELECT count(*)
   FROM ORDERS
   WHERE country_code = 'ES'
   ```
 
 - For your coworker in the USA, the SQL Lab query will be rendered as:
 
-  ```
+  ```sql
   SELECT count(*)
   FROM ORDERS
   WHERE country_code = 'US'
@@ -222,7 +259,7 @@ This is useful if:
 
 Here's a concrete example:
 
-```
+```sql
 SELECT action, count(*) as times
 FROM logs
 WHERE

docs/docs/security.mdx

@@ -148,7 +148,7 @@ a certain resource type or policy area. You can check possible directives
 
 It's extremely important to correclty configure a Content Security Policy when deploying Superset to
 prevent many types of attacks. For that matter, Superset provides the ` TALISMAN_CONFIG` key in `config.py`
-where admnistrators can define the policy. When running in production mode, Superset will check for the presence
+where administrators can define the policy. When running in production mode, Superset will check for the presence
 of a policy and if it's not able to find one, it will issue a warning with the security risks. For environments
 where CSP policies are defined outside of Superset using other software, administrators can disable
 the warning using the `CONTENT_SECURITY_POLICY_WARNING` key in `config.py`.

docs/docusaurus.config.js

@@ -247,7 +247,14 @@ const config = {
         darkTheme: darkCodeTheme,
       },
     }),
-  scripts: ['/script/matomo.js'],
+  scripts: [
+    '/script/matomo.js',
+    {
+      src:
+        'https://www.bugherd.com/sidebarv2.js?apikey=enilpiu7bgexxsnoqfjtxa',
+      async: true,
+    },
+  ],
 };
 
 module.exports = config;

docs/package.json

@@ -18,11 +18,11 @@
   "dependencies": {
     "@algolia/client-search": "^4.13.0",
     "@ant-design/icons": "^4.7.0",
-    "@docsearch/react": "^3.0.0",
-    "@docusaurus/core": "^2.0.0-beta.17",
-    "@docusaurus/plugin-client-redirects": "^2.0.0-beta.17",
-    "@docusaurus/plugin-google-gtag": "^2.0.0-beta.18",
-    "@docusaurus/preset-classic": "^2.0.0-beta.17",
+    "@docsearch/react": "^3.3.3",
+    "@docusaurus/core": "^2.3.1",
+    "@docusaurus/plugin-client-redirects": "^2.3.1",
+    "@docusaurus/plugin-google-gtag": "^2.3.1",
+    "@docusaurus/preset-classic": "^2.3.1",
     "@emotion/core": "^10.1.1",
     "@emotion/styled": "^10.0.27",
     "@mdx-js/react": "^1.6.22",
@@ -45,8 +45,8 @@
     "url-loader": "^4.1.1"
   },
   "devDependencies": {
-    "@docusaurus/module-type-aliases": "^2.0.0-beta.17",
-    "@tsconfig/docusaurus": "^1.0.4",
+    "@docusaurus/module-type-aliases": "^2.3.1",
+    "@tsconfig/docusaurus": "^1.0.6",
     "@types/react": "^17.0.42",
     "typescript": "^4.3.5",
     "webpack": "^5.61.0"

requirements/integration.txt

@@ -29,7 +29,7 @@ packaging==21.3
     #   tox
 pep517==0.11.0
     # via build
-pip-compile-multi==2.6.1
+pip-compile-multi==2.6.2
     # via -r integration.in
 pip-tools==6.8.0
     # via pip-compile-multi

setup.py

@@ -135,7 +135,7 @@ def get_git_sha() -> str:
             "sqlalchemy-bigquery>=1.5.0",
             "google-cloud-bigquery>=3.4.0",
         ],
-        "clickhouse": ["clickhouse-connect>=0.4.6, <0.5"],
+        "clickhouse": ["clickhouse-connect>=0.5.14, <1.0"],
         "cockroachdb": ["cockroachdb>=0.3.5, <0.4"],
         "cors": ["flask-cors>=2.0.0"],
         "crate": ["crate[sqlalchemy]>=0.26.0, <0.27"],

superset-embedded-sdk/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@superset-ui/embedded-sdk",
-  "version": "0.1.0-alpha.8",
+  "version": "0.1.0-alpha.9",
   "description": "SDK for embedding resources from Superset into your own application",
   "access": "public",
   "keywords": [

superset-embedded-sdk/src/index.ts

@@ -150,6 +150,7 @@ export async function embedDashboard({
       });
 
       iframe.src = `${supersetDomain}/embedded/${id}${dashboardConfig}${filterConfigUrlParams}`;
+      //@ts-ignore
       mountPoint.replaceChildren(iframe);
       log('placed the iframe')
     });
@@ -173,6 +174,7 @@ export async function embedDashboard({
 
   function unmount() {
     log('unmounting');
+    //@ts-ignore
     mountPoint.replaceChildren();
   }
 

superset-frontend/cypress-base/cypress/integration/dashboard/load.test.ts

@@ -42,7 +42,8 @@ describe('Dashboard load', () => {
     cy.get('#app-menu').should('not.exist');
   });
 
-  it('should send log data', () => {
+  // TODO flaky test. skipping to unblock CI
+  it.skip('should send log data', () => {
     interceptLog();
     cy.visit(WORLD_HEALTH_DASHBOARD);
     cy.wait('@logs', { timeout: 15000 });

superset-frontend/cypress-base/cypress/integration/dashboard/nativeFilters.test.ts

@@ -119,7 +119,6 @@ function prepareDashboardFilters(
     });
     if (dashboardId) {
       const jsonMetadata = {
-        show_native_filters: true,
         native_filter_configuration: allFilters,
         timed_refresh_immune_slices: [],
         expanded_slices: {},
@@ -379,10 +378,13 @@ describe('Horizontal FilterBar', () => {
       { name: 'test_12', column: 'year', datasetId: 2 },
     ]);
     setFilterBarOrientation('horizontal');
+    openMoreFilters();
+    applyNativeFilterValueWithIndex(8, testItems.filterDefaultValue);
+    cy.get(nativeFilters.applyFilter).click({ force: true });
     cy.getBySel('slice-header').within(() => {
-      cy.get('.filter-counts').click();
+      cy.get('.filter-counts').trigger('mouseover');
     });
-    cy.get('.filterStatusPopover').contains('test_8').click();
+    cy.get('.filterStatusPopover').contains('test_9').click();
     cy.getBySel('dropdown-content').should('be.visible');
     cy.get('.ant-select-focused').should('be.visible');
   });