fix(migrations): reorder skipped 1.4 migrations

[villebro]

SUMMARY

Two migrations (b92d69a6643c and 0ca9e5f1dacd) had been skipped in the 1.4 release, causing errors when upgrading from 1.4 to 1.5 or master. See the different down revisions of f9847149153d in 1.4 and master:

superset/superset/migrations/versions/f9847149153d_add_certifications_columns_to_slice.py

Lines 29 to 30 in 9fc5e09

	revision = "f9847149153d"
	down_revision = "32646df09c64"

superset/superset/migrations/versions/f9847149153d_add_certifications_columns_to_slice.py

Lines 29 to 30 in 156ac7d

	revision = "f9847149153d"
	down_revision = "0ca9e5f1dacd"

This places the skipped migrations in front of the first migration of 1.5, making sure the migrations are applied to the metadata.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

1. Install version 1.4 of Superset, run db migrations
2. Install 1.5.0rc1 or master branch, run db migrations and run superset init and see the error

ADDITIONAL INFORMATION

• Has associated issue: closes Database error when upgrading from 1.4.1 to master #19100
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #19543 (be4dd17) into master (156ac7d) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #19543      +/-   ##
==========================================
- Coverage   66.57%   66.54%   -0.03%     
==========================================
  Files        1682     1682              
  Lines       64321    64321              
  Branches     6561     6561              
==========================================
- Hits        42821    42804      -17     
- Misses      19799    19816      +17     
  Partials     1701     1701              

Flag	Coverage Δ
hive	52.71% <ø> (ø)
mysql	?
postgres	81.94% <ø> (?)
presto	52.56% <ø> (ø)
python	82.28% <ø> (-0.06%)	⬇️
sqlite	?
unit	47.21% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/common/utils/dataframe_utils.py	85.71% <0.00%> (-7.15%)	⬇️
superset/connectors/sqla/utils.py	89.13% <0.00%> (-5.44%)	⬇️
superset/db_engine_specs/sqlite.py	91.89% <0.00%> (-5.41%)	⬇️
superset/db_engine_specs/mysql.py	93.97% <0.00%> (-3.62%)	⬇️
superset/utils/celery.py	86.20% <0.00%> (-3.45%)	⬇️
superset/result_set.py	96.77% <0.00%> (-1.62%)	⬇️
superset/databases/commands/test_connection.py	98.57% <0.00%> (-1.43%)	⬇️
superset/utils/cache.py	73.58% <0.00%> (-0.95%)	⬇️
superset/models/core.py	88.36% <0.00%> (-0.72%)	⬇️
superset/db_engine_specs/base.py	87.85% <0.00%> (-0.50%)	⬇️
... and 5 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 156ac7d...be4dd17. Read the comment docs.

[zhaoyongjie]

LGTM! I have tested this PR in my local through superset db history, and this fix is expected.

before

after

[betodealmeida]

I'm running superset db upgrade and master right now, with a clean DB, and no migrations are applied:

% psql master
psql (14.2)
Type "help" for help.

master=# \dt
                List of relations
 Schema |          Name           | Type  | Owner
--------+-------------------------+-------+-------
 public | ab_permission           | table | beto
 public | ab_permission_view      | table | beto
 public | ab_permission_view_role | table | beto
 public | ab_register_user        | table | beto
 public | ab_role                 | table | beto
 public | ab_user                 | table | beto
 public | ab_user_role            | table | beto
 public | ab_view_menu            | table | beto
(8 rows)

master=#

Nothing happens on superset db upgrade:

% superset db upgrade
Loaded your LOCAL configuration at [/Users/beto/Projects/incubator-superset/superset_config.py]
--------------------------------------------------------------------------------
                                    WARNING
--------------------------------------------------------------------------------
A Default SECRET_KEY was detected, please use superset_config.py to override it.
Use a strong complex alphanumeric string and use a tool to help you generate
a sufficiently random sequence, ex: openssl rand -base64 42
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
logging was configured successfully
2022-04-06 12:24:03,523:INFO:superset.utils.logging_configurator:logging was configured successfully
2022-04-06 12:24:03,536:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
Falling back to the built-in cache, that stores data in the metadata database, for the followinng cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
2022-04-06 12:24:03,542:WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the followinng cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
Falling back to the built-in cache, that stores data in the metadata database, for the followinng cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
2022-04-06 12:24:03,548:WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the followinng cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.

Not sure if it's related to this PR.

[villebro]

@betodealmeida to double check, I pulled current master, wiped my metastore and ran superset db upgrade, and it ran successfully. Maybe you have a lingering old migration in the version directory? Which reminds me that for some reason alembic error messages don't appear to work anymore (it just fails silently), making debugging very difficult 🙁 (will try to look at why that's the case at some point)

[lzfxxx]

I'm running superset db upgrade and master right now, with a clean DB, and no migrations are applied:

% psql master
psql (14.2)
Type "help" for help.

master=# \dt
                List of relations
 Schema |          Name           | Type  | Owner
--------+-------------------------+-------+-------
 public | ab_permission           | table | beto
 public | ab_permission_view      | table | beto
 public | ab_permission_view_role | table | beto
 public | ab_register_user        | table | beto
 public | ab_role                 | table | beto
 public | ab_user                 | table | beto
 public | ab_user_role            | table | beto
 public | ab_view_menu            | table | beto
(8 rows)

master=#

Nothing happens on superset db upgrade:

% superset db upgrade
Loaded your LOCAL configuration at [/Users/beto/Projects/incubator-superset/superset_config.py]
--------------------------------------------------------------------------------
                                    WARNING
--------------------------------------------------------------------------------
A Default SECRET_KEY was detected, please use superset_config.py to override it.
Use a strong complex alphanumeric string and use a tool to help you generate
a sufficiently random sequence, ex: openssl rand -base64 42
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
logging was configured successfully
2022-04-06 12:24:03,523:INFO:superset.utils.logging_configurator:logging was configured successfully
2022-04-06 12:24:03,536:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
Falling back to the built-in cache, that stores data in the metadata database, for the followinng cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
2022-04-06 12:24:03,542:WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the followinng cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
Falling back to the built-in cache, that stores data in the metadata database, for the followinng cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
2022-04-06 12:24:03,548:WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the followinng cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.

Not sure if it's related to this PR.

@betodealmeida I'm facing the same problem, did you find the solution?

[artemonsh]

@lzfxxx Probably, there is an existing sqlite database. You can check its presence ls ~/.superset/, remove it rm ~/.superset/superset.db and try again