Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to get started with SQLite as Superset metadata database with custom superset_config.py #17551

Closed
sairamkrish opened this issue Nov 25, 2021 · 1 comment
Closed

Unable to get started with SQLite as Superset metadata database with custom superset_config.py #17551

sairamkrish opened this issue Nov 25, 2021 · 1 comment
Labels
#bug Bug report

Comments

@sairamkrish
Copy link

sairamkrish commented Nov 25, 2021 •
edited
Loading

Can't get started with Apache Superset + SQLite as metadata store + custom superset_config.py.

Environment

  • Apache superset version 1.3.2
  • superset_config.py --> this has only one override SQLALCHEMY_DATABASE_URI

Steps to reproduce

  • Without superset_config.py., if we use the default flow superset db upgrade, this issue is not happening. Sqlite db gets created at ~/.superset/superset.db
  • Create a custom superset_config.py with content
SQLALCHEMY_DATABASE_URI = 'sqlite:///superset.db'
  • Invoke PYTHONPATH=.:$PYTHONPATH superset db upgrade
  • we get following exception
(superset-experiment) ➜  superset-experiment git:(main) ✗ PYTHONPATH=.:$PYTHONPATH superset db upgrade
Loaded your LOCAL configuration at [/Users/sairam/superset-experiment/superset_config.py]
logging was configured successfully
2021-11-25 07:28:07,681:INFO:superset.utils.logging_configurator:logging was configured successfully
2021-11-25 07:28:07,695:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/flask_caching/__init__.py:201: UserWarning: Flask-Caching: CACHE_TYPE is set to null, caching is effectively disabled.
  warnings.warn(
No PIL installation found
2021-11-25 07:28:07,915:INFO:superset.utils.screenshots:No PIL installation found
WARNI [alembic.env] SQLite Database support for metadata databases will         be removed in a future version of Superset.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 12d55656cbca -> 2591d77e9831, user_id
Traceback (most recent call last):
  File "/Users/sairam/superset-experiment/.venv/bin/superset", line 8, in <module>
    sys.exit(superset())
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/flask_migrate/cli.py", line 149, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/flask_migrate/__init__.py", line 98, in wrapped
    f(*args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/flask_migrate/__init__.py", line 185, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/superset/migrations/env.py", line 124, in <module>
    run_migrations_online()
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/superset/migrations/env.py", line 116, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
    step.migration_fn(**kw)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/superset/migrations/versions/2591d77e9831_user_id.py", line 36, in upgrade
    batch_op.create_foreign_key("user_id", "ab_user", ["user_id"], ["id"])
  File "/Users/sairam/.pyenv/versions/3.8.1/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/alembic/operations/base.py", line 374, in batch_alter_table
    impl.flush()
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/alembic/operations/batch.py", line 118, in flush
    existing_table = Table(
  File "<string>", line 2, in __new__
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 139, in warned
    return fn(*args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 563, in __new__
    metadata._remove_table(name, schema)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 558, in __new__
    table._init(name, metadata, *args, **kw)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 647, in _init
    self._autoload(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 670, in _autoload
    autoload_with.run_callable(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1653, in run_callable
    return callable_(self, *args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 484, in reflecttable
    return insp.reflecttable(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 684, in reflecttable
    self._reflect_fk(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 868, in _reflect_fk
    sa_schema.Table(
  File "<string>", line 2, in __new__
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 139, in warned
    return fn(*args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 563, in __new__
    metadata._remove_table(name, schema)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 558, in __new__
    table._init(name, metadata, *args, **kw)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 647, in _init
    self._autoload(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 670, in _autoload
    autoload_with.run_callable(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1653, in run_callable
    return callable_(self, *args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 484, in reflecttable
    return insp.reflecttable(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 684, in reflecttable
    self._reflect_fk(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 868, in _reflect_fk
    sa_schema.Table(
  File "<string>", line 2, in __new__
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 139, in warned
    return fn(*args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 563, in __new__
    metadata._remove_table(name, schema)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 558, in __new__
    table._init(name, metadata, *args, **kw)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 647, in _init
    self._autoload(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 670, in _autoload
    autoload_with.run_callable(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1653, in run_callable
    return callable_(self, *args, **kwargs)
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 484, in reflecttable
    return insp.reflecttable(
  File "/Users/sairam/superset-experiment/.venv/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 678, in reflecttable
    raise exc.NoSuchTableError(table.name)
sqlalchemy.exc.NoSuchTableError: ab_user
@sairamkrish sairamkrish added the #bug Bug report label Nov 25, 2021
@sairamkrish sairamkrish mentioned this issue Nov 25, 2021
Closed
@sairamkrish sairamkrish changed the title Unable to get started with SQLite as Superset metadata database Unable to get started with SQLite as Superset metadata database with custom superset_config.py Nov 27, 2021
@betodealmeida
Copy link
Member

betodealmeida commented Dec 8, 2021 •
edited
Loading

@sairamkrish can you try setting PREVENT_UNSAFE_DB_CONNECTIONS to False in your superset_config.py? That should make it work. I'll take a look into this, I think there's value in Sqlite so I bring the issue up with the community.

@sergiimk sergiimk mentioned this issue Nov 3, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
#bug Bug report
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sairamkrish @betodealmeida @amitmiran137