UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4 #16733

zhangjiajie023 · 2017-04-12T07:57:07Z

zhangjiajie023
Apr 12, 2017

Make sure these boxes are checked before submitting your issue - thank you!

I have checked the superset logs for python stacktraces and included it here as text if any
I have reproduced the issue with at least the latest released version of superset
I have checked the issue tracker for the same issue and I haven't found one similar

Superset version

0.15.4

Expected results

change the metadata db from db2 to previous db1 can work

Actual results

after changing to db1, exists the encoding error

Steps to reproduce

add ?charset=utf8 in sqlalchemy_database_uri , and create database db1 character set utf8,
use db1 as the metadata db, works
add ?charset=utf8 in sqlalchemy_database_uri , and create database db2 character set utf8,
use db2 as the metadata db, works
change the sqlalchemy_database_uri from db2 to db1
view a slice or use SQL Lab, occur the encoding bug

python3.5 superset 0.15.4 mysql 5.7

File "/home/jiajie/project/superset/superset/superset/views.py", line 1588, in explore
    datasources = sorted(datasources, key=lambda ds: ds.full_name)
  File "/home/jiajie/project/superset/superset/superset/views.py", line 1588, in <lambda>
    datasources = sorted(datasources, key=lambda ds: ds.full_name)
  File "/home/jiajie/project/superset/superset/superset/models.py", line 1155, in full_name
    self.database, self.table_name, schema=self.schema)
......................
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/strategies.py", line 557, in _load_for_state
    return self._emit_lazyload(session, state, ident_key, passive)
  File "\<string\>", line 1, in \<lambda\>
    
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/strategies.py", line 603, in _emit_lazyload
    return loading.load_on_ident(q, ident_key)
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/loading.py", line 223, in load_on_ident
    return q.one()
  .......
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/util/compat.py", line 186, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/loading.py", line 75, in instances
    rows = [proc(row) for row in fetch]
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/loading.py", line 75, in <listcomp>
    rows = [proc(row) for row in fetch]
 ....................
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy_utils/types/encrypted.py", line 261, in process_result_value
    decrypted_value = self.engine.decrypt(value)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy_utils/types/encrypted.py", line 89, in decrypt
    decrypted = decrypted.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4: invalid continuation byte

xrmx · 2017-04-12T21:10:06Z

xrmx
Apr 12, 2017

Could you please replicate using utf8mb4 as character set instead of utf8?

0 replies

zhangjiajie023 · 2017-04-13T13:01:14Z

zhangjiajie023
Apr 13, 2017
Author

@xrmx ok, i'll try， tks. But still amazing

0 replies

kkalyan · 2017-04-13T14:14:29Z

kkalyan
Apr 13, 2017

this is a workaround you can use - convert the utf field to ascii using mysql expression in the edit dimension.

or include ?charset=utf8 in the SQLAlchemy URL

0 replies

timfeirg · 2017-05-08T15:43:22Z

timfeirg
May 8, 2017

~~Just upgraded superset to python 3.6 today, had the same problem, adding ?charset=utf8mb4 to SQLALCHEMY_DATABASE_URI and it's fixed~~

0 replies

xrmx · 2017-05-08T15:47:07Z

xrmx
May 8, 2017

@timfeirg does ?charset=utf8mb4 work too?

0 replies

xrmx · 2017-05-08T15:55:22Z

xrmx
May 8, 2017

@timfeirg sorry, typo :) i meant ?charset=utf8 without the mb4

0 replies

timfeirg · 2017-05-08T16:16:57Z

timfeirg
May 8, 2017

I'm sorry, I just tried both ?charset=utf8 and ?charset=utf8mb4 using superset under python3.6.1, and open up the example charts provided by superset, for example /superset/dashboard/misc_charts/, and the request ended up with the same UnicodeDecodeError as described in this issue.

If I switch back to python2.7, the example charts would work fine, with or without adding the extra charset configs.

this comment was a mistake, @xrmx

0 replies

xrmx · 2017-06-23T10:16:36Z

xrmx
Jun 23, 2017

I'm hitting this too when upgrading a sqlite installation from caravel to superset, both python3.4.

0 replies

xrmx · 2017-06-23T13:13:54Z

xrmx
Jun 23, 2017

@zhangjiajie023 @timfeirg Here this was caused by having in the config a SECRET_KEY different than the one used to encrypt the password.

0 replies

jerryhall · 2017-08-15T02:22:08Z

jerryhall
Aug 15, 2017

I've been down for some time. I checked the security key and am using same one as with 2.7.
I upgraded to 3.6 and tried appending the sqlalchemy to this:
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(DATA_DIR, 'superset.db?charset=utf8')
with no luck & I get this error message:
2017-08-15 02:19:26,671:ERROR:flask_appbuilder.security.sqla.manager:DB Creation and initialization failed: 'charset' is an invalid keyword argument for this function

My complete traceback below.

Any help would be appreciated!
Traceback (most recent call last): File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app response = self.full_dispatch_request() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request rv = self.handle_user_exception(e) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception reraise(exc_type, exc_value, tb) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise raise value File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/utils.py", line 574, in wraps return f(self, *args, **kwargs) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/views/core.py", line 1691, in dashboard if datasource and not self.datasource_access(datasource): File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/views/base.py", line 94, in datasource_access self.schema_access(datasource, user=user) or File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/views/base.py", line 87, in schema_access self.database_access(datasource.database, user=user) or File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 237, in __get__ return self.impl.get(instance_state(instance), dict_) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 584, in get value = self.callable_(state, passive) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 557, in _load_for_state return self._emit_lazyload(session, state, ident_key, passive) File "<string>", line 1, in <lambda> File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 603, in _emit_lazyload return loading.load_on_ident(q, ident_key) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 223, in load_on_ident return q.one() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2814, in one ret = self.one_or_none() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2784, in one_or_none ret = list(self) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 90, in instances util.raise_from_cause(err) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise raise value File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 75, in instances rows = [proc(row) for row in fetch] File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 75, in <listcomp> rows = [proc(row) for row in fetch] File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 437, in _instance loaded_instance, populate_existing, populators) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 498, in _populate_full dict_[key] = getter(row) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/sql/type_api.py", line 1122, in process return process_value(impl_processor(value), dialect) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy_utils/types/encrypted.py", line 261, in process_result_value decrypted_value = self.engine.decrypt(value) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy_utils/types/encrypted.py", line 89, in decrypt decrypted = decrypted.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 4: invalid continuation byte

0 replies

mistercrunch · 2017-08-17T19:49:56Z

mistercrunch
Aug 17, 2017
Collaborator

We don't recommend using sqlite in production. Also we don't build against python 3.6 (yet). Might be safer/easier to run py3.4 on against mysql

0 replies

xrmx · 2017-08-17T20:42:06Z

xrmx
Aug 17, 2017

@mistercrunch it's not an unicode issue, it is trying to decode encrypted data with the wrong key. I've opened a PR raising an intelligible error upstream but i forgot to update it.

0 replies

xrmx · 2017-08-17T20:45:03Z

xrmx
Aug 17, 2017

@jerryhall I don't think sqlite takes a charset option

0 replies

bmackattack · 2017-08-31T14:29:04Z

bmackattack
Aug 31, 2017

@xrmx any solutions on this? If not, can you tell me how to run this in python 2.7.

I believe I'm running into the same error.... Stack trace as follows:
Loaded your LOCAL configuration at [/usr/local/bin/superset_config.py] 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/superset/security.py", line 92, in get_or_create_main_db 2017-08-31T14:27:13Z Traceback (most recent call last): 2017-08-31T14:27:13Z manager.run() 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/flask_script/__init__.py", line 412, in run 2017-08-31T14:27:13Z result = self.handle(sys.argv[0], sys.argv[1:]) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/flask_script/__init__.py", line 383, in handle 2017-08-31T14:27:13Z res = handle(*args, **config) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/flask_script/commands.py", line 216, in __call__ 2017-08-31T14:27:13Z return self.run(*args, **kwargs) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/superset/cli.py", line 28, in init 2017-08-31T14:27:13Z security.sync_role_definitions() 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/superset/security.py", line 201, in sync_role_definitions 2017-08-31T14:27:13Z get_or_create_main_db() 2017-08-31T14:27:13Z File "/usr/local/bin/superset", line 15, in <module> 2017-08-31T14:27:13Z .filter_by(database_name='main') 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2755, in first 2017-08-31T14:27:13Z ret = list(self[0:1]) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2547, in __getitem__ 2017-08-31T14:27:13Z return list(res) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 90, in instances 2017-08-31T14:27:13Z util.raise_from_cause(err) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause 2017-08-31T14:27:13Z reraise(type(exception), exception, tb=exc_tb, cause=cause) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise 2017-08-31T14:27:13Z raise value 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 75, in instances 2017-08-31T14:27:13Z rows = [proc(row) for row in fetch] 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 75, in <listcomp> 2017-08-31T14:27:13Z loaded_instance, populate_existing, populators) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 437, in _instance 2017-08-31T14:27:13Z UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa1 in position 0: invalid start byte 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 498, in _populate_full 2017-08-31T14:27:13Z dict_[key] = getter(row) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/type_api.py", line 1122, in process 2017-08-31T14:27:13Z return process_value(impl_processor(value), dialect) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy_utils/types/encrypted.py", line 261, in process_result_value 2017-08-31T14:27:13Z decrypted_value = self.engine.decrypt(value) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy_utils/types/encrypted.py", line 89, in decrypt 2017-08-31T14:27:13Z decrypted = decrypted.decode('utf-8')

0 replies

xrmx · 2017-08-31T14:38:14Z

xrmx
Aug 31, 2017

@bmackattack if you are using the wrong key the only solution is to use the right one, you are the only one who knows it. My patch just makes the error explicit.

0 replies

jhult · 2021-09-17T20:26:52Z

jhult
Sep 17, 2021

Here is how I was able to fix this: #8538 (comment)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4 #16733

{{title}}

{{editor}}'s edit

{{editor}}'s edit

change the `sqlalchemy_database_uri` from db2 to db1

Replies: 16 comments

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4 #16733

Superset version

Expected results

Actual results

Steps to reproduce

change the sqlalchemy_database_uri from db2 to db1

python3.5 superset 0.15.4 mysql 5.7

Replies: 16 comments

zhangjiajie023 Apr 13, 2017 Author

mistercrunch Aug 17, 2017 Collaborator

change the `sqlalchemy_database_uri` from db2 to db1

zhangjiajie023
Apr 13, 2017
Author

mistercrunch
Aug 17, 2017
Collaborator