UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4 #2600

zhangjiajie023 · 2017-04-12T07:57:07Z

Make sure these boxes are checked before submitting your issue - thank you!

I have checked the superset logs for python stacktraces and included it here as text if any
I have reproduced the issue with at least the latest released version of superset
I have checked the issue tracker for the same issue and I haven't found one similar

Superset version

0.15.4

Expected results

change the metadata db from db2 to previous db1 can work

Actual results

after changing to db1, exists the encoding error

Steps to reproduce

add ?charset=utf8 in sqlalchemy_database_uri , and create database db1 character set utf8,
use db1 as the metadata db, works
add ?charset=utf8 in sqlalchemy_database_uri , and create database db2 character set utf8,
use db2 as the metadata db, works
change the sqlalchemy_database_uri from db2 to db1
view a slice or use SQL Lab, occur the encoding bug

python3.5 superset 0.15.4 mysql 5.7

File "/home/jiajie/project/superset/superset/superset/views.py", line 1588, in explore
    datasources = sorted(datasources, key=lambda ds: ds.full_name)
  File "/home/jiajie/project/superset/superset/superset/views.py", line 1588, in <lambda>
    datasources = sorted(datasources, key=lambda ds: ds.full_name)
  File "/home/jiajie/project/superset/superset/superset/models.py", line 1155, in full_name
    self.database, self.table_name, schema=self.schema)
......................
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/strategies.py", line 557, in _load_for_state
    return self._emit_lazyload(session, state, ident_key, passive)
  File "\<string\>", line 1, in \<lambda\>
    
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/strategies.py", line 603, in _emit_lazyload
    return loading.load_on_ident(q, ident_key)
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/loading.py", line 223, in load_on_ident
    return q.one()
  .......
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/util/compat.py", line 186, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/loading.py", line 75, in instances
    rows = [proc(row) for row in fetch]
  File "/usr/local/lib/python3.5/dist-packages/SQLAlchemy-1.1.5.dev0-py3.5-linux-x86_64.egg/sqlalchemy/orm/loading.py", line 75, in <listcomp>
    rows = [proc(row) for row in fetch]
 ....................
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy_utils/types/encrypted.py", line 261, in process_result_value
    decrypted_value = self.engine.decrypt(value)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy_utils/types/encrypted.py", line 89, in decrypt
    decrypted = decrypted.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4: invalid continuation byte

The text was updated successfully, but these errors were encountered:

xrmx · 2017-04-12T21:10:06Z

Could you please replicate using utf8mb4 as character set instead of utf8?

zhangjiajie023 · 2017-04-13T13:01:14Z

@xrmx ok, i'll try， tks. But still amazing

kkalyan · 2017-04-13T14:14:29Z

this is a workaround you can use - convert the utf field to ascii using mysql expression in the edit dimension.

or include ?charset=utf8 in the SQLAlchemy URL

timfeirg · 2017-05-08T15:43:22Z

~~Just upgraded superset to python 3.6 today, had the same problem, adding ?charset=utf8mb4 to SQLALCHEMY_DATABASE_URI and it's fixed~~

xrmx · 2017-05-08T15:47:07Z

@timfeirg does ?charset=utf8mb4 work too?

xrmx · 2017-05-08T15:55:22Z

@timfeirg sorry, typo :) i meant ?charset=utf8 without the mb4

timfeirg · 2017-05-08T16:16:57Z

I'm sorry, I just tried both ?charset=utf8 and ?charset=utf8mb4 using superset under python3.6.1, and open up the example charts provided by superset, for example /superset/dashboard/misc_charts/, and the request ended up with the same UnicodeDecodeError as described in this issue.

If I switch back to python2.7, the example charts would work fine, with or without adding the extra charset configs.

this comment was a mistake, @xrmx

xrmx · 2017-06-23T10:16:36Z

I'm hitting this too when upgrading a sqlite installation from caravel to superset, both python3.4.

xrmx · 2017-06-23T13:13:54Z

@zhangjiajie023 @timfeirg Here this was caused by having in the config a SECRET_KEY different than the one used to encrypt the password.

jerryhall · 2017-08-15T02:22:08Z

I've been down for some time. I checked the security key and am using same one as with 2.7.
I upgraded to 3.6 and tried appending the sqlalchemy to this:
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(DATA_DIR, 'superset.db?charset=utf8')
with no luck & I get this error message:
2017-08-15 02:19:26,671:ERROR:flask_appbuilder.security.sqla.manager:DB Creation and initialization failed: 'charset' is an invalid keyword argument for this function

My complete traceback below.

Any help would be appreciated!
Traceback (most recent call last): File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app response = self.full_dispatch_request() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request rv = self.handle_user_exception(e) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception reraise(exc_type, exc_value, tb) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise raise value File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/utils.py", line 574, in wraps return f(self, *args, **kwargs) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/views/core.py", line 1691, in dashboard if datasource and not self.datasource_access(datasource): File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/views/base.py", line 94, in datasource_access self.schema_access(datasource, user=user) or File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/superset/views/base.py", line 87, in schema_access self.database_access(datasource.database, user=user) or File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 237, in __get__ return self.impl.get(instance_state(instance), dict_) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 584, in get value = self.callable_(state, passive) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 557, in _load_for_state return self._emit_lazyload(session, state, ident_key, passive) File "<string>", line 1, in <lambda> File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 603, in _emit_lazyload return loading.load_on_ident(q, ident_key) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 223, in load_on_ident return q.one() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2814, in one ret = self.one_or_none() File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2784, in one_or_none ret = list(self) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 90, in instances util.raise_from_cause(err) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise raise value File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 75, in instances rows = [proc(row) for row in fetch] File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 75, in <listcomp> rows = [proc(row) for row in fetch] File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 437, in _instance loaded_instance, populate_existing, populators) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 498, in _populate_full dict_[key] = getter(row) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy/sql/type_api.py", line 1122, in process return process_value(impl_processor(value), dialect) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy_utils/types/encrypted.py", line 261, in process_result_value decrypted_value = self.engine.decrypt(value) File "/home/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/site-packages/sqlalchemy_utils/types/encrypted.py", line 89, in decrypt decrypted = decrypted.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 4: invalid continuation byte

mistercrunch · 2017-08-17T19:49:56Z

We don't recommend using sqlite in production. Also we don't build against python 3.6 (yet). Might be safer/easier to run py3.4 on against mysql

xrmx · 2017-08-17T20:42:06Z

@mistercrunch it's not an unicode issue, it is trying to decode encrypted data with the wrong key. I've opened a PR raising an intelligible error upstream but i forgot to update it.

xrmx · 2017-08-17T20:45:03Z

@jerryhall I don't think sqlite takes a charset option

bmackattack · 2017-08-31T14:29:04Z

@xrmx any solutions on this? If not, can you tell me how to run this in python 2.7.

I believe I'm running into the same error.... Stack trace as follows:
Loaded your LOCAL configuration at [/usr/local/bin/superset_config.py] 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/superset/security.py", line 92, in get_or_create_main_db 2017-08-31T14:27:13Z Traceback (most recent call last): 2017-08-31T14:27:13Z manager.run() 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/flask_script/__init__.py", line 412, in run 2017-08-31T14:27:13Z result = self.handle(sys.argv[0], sys.argv[1:]) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/flask_script/__init__.py", line 383, in handle 2017-08-31T14:27:13Z res = handle(*args, **config) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/flask_script/commands.py", line 216, in __call__ 2017-08-31T14:27:13Z return self.run(*args, **kwargs) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/superset/cli.py", line 28, in init 2017-08-31T14:27:13Z security.sync_role_definitions() 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/superset/security.py", line 201, in sync_role_definitions 2017-08-31T14:27:13Z get_or_create_main_db() 2017-08-31T14:27:13Z File "/usr/local/bin/superset", line 15, in <module> 2017-08-31T14:27:13Z .filter_by(database_name='main') 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2755, in first 2017-08-31T14:27:13Z ret = list(self[0:1]) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2547, in __getitem__ 2017-08-31T14:27:13Z return list(res) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 90, in instances 2017-08-31T14:27:13Z util.raise_from_cause(err) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause 2017-08-31T14:27:13Z reraise(type(exception), exception, tb=exc_tb, cause=cause) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise 2017-08-31T14:27:13Z raise value 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 75, in instances 2017-08-31T14:27:13Z rows = [proc(row) for row in fetch] 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 75, in <listcomp> 2017-08-31T14:27:13Z loaded_instance, populate_existing, populators) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 437, in _instance 2017-08-31T14:27:13Z UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa1 in position 0: invalid start byte 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 498, in _populate_full 2017-08-31T14:27:13Z dict_[key] = getter(row) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/type_api.py", line 1122, in process 2017-08-31T14:27:13Z return process_value(impl_processor(value), dialect) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy_utils/types/encrypted.py", line 261, in process_result_value 2017-08-31T14:27:13Z decrypted_value = self.engine.decrypt(value) 2017-08-31T14:27:13Z File "/usr/local/lib/python3.6/site-packages/sqlalchemy_utils/types/encrypted.py", line 89, in decrypt 2017-08-31T14:27:13Z decrypted = decrypted.decode('utf-8')

xrmx · 2017-08-31T14:38:14Z

@bmackattack if you are using the wrong key the only solution is to use the right one, you are the only one who knows it. My patch just makes the error explicit.

Now sqlalchemy-utils will make explicit that we are trying decoding the secret with the wrong key instead of a generic UnicodeDecodeError. Fix apache#2600

Now sqlalchemy-utils will make explicit that we are trying decoding the secret with the wrong key instead of a generic UnicodeDecodeError. Fix #2600

jhult · 2021-09-17T20:26:52Z

Here is how I was able to fix this: #8538 (comment)

zhangjiajie023 changed the title ~~[Mysql] UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4: invalid continuation byte~~ [Mysql] UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4 Apr 12, 2017

mistercrunch added the help wanted label Apr 12, 2017

xrmx changed the title ~~[Mysql] UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4~~ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4 Jun 23, 2017

xrmx mentioned this issue Jun 23, 2017

UnicodeDecodeError in sqlalchemy_utils.types.encrypted kvesteri/sqlalchemy-utils#275

Closed

xrmx mentioned this issue Jul 20, 2017

UTF-8 codec can't decode byte - Python 3.6 / Superset 1.8 #3166

Closed

mistercrunch added the unicode label Aug 17, 2017

xrmx added a commit to xrmx/superset that referenced this issue Sep 1, 2017

setup: Bump sqlalchemy-utils to 0.32.16

1162c3f

Now sqlalchemy-utils will make explicit that we are trying decoding the secret with the wrong key instead of a generic UnicodeDecodeError. Fix apache#2600

xrmx mentioned this issue Sep 1, 2017

setup: Bump sqlalchemy-utils to 0.32.16 #3405

Merged

mistercrunch closed this as completed in #3405 Sep 1, 2017

mistercrunch pushed a commit that referenced this issue Sep 1, 2017

setup: Bump sqlalchemy-utils to 0.32.16 (#3405)

3dfdde1

Now sqlalchemy-utils will make explicit that we are trying decoding the secret with the wrong key instead of a generic UnicodeDecodeError. Fix #2600

andor-pierdelacabeza mentioned this issue Oct 26, 2017

SECRET_KEY encrypts database secrets without warning, possibly breaking web interface if changed #3724

Closed

3 tasks

apache locked and limited conversation to collaborators Sep 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

This issue was moved to a discussion.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4 #2600

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4 #2600

zhangjiajie023 commented Apr 12, 2017 •

edited by xrmx

Loading

change the `sqlalchemy_database_uri` from db2 to db1

xrmx commented Apr 12, 2017

zhangjiajie023 commented Apr 13, 2017

kkalyan commented Apr 13, 2017 •

edited

Loading

timfeirg commented May 8, 2017 •

edited

Loading

xrmx commented May 8, 2017

xrmx commented May 8, 2017

timfeirg commented May 8, 2017 •

edited

Loading

xrmx commented Jun 23, 2017

xrmx commented Jun 23, 2017 •

edited

Loading

jerryhall commented Aug 15, 2017 •

edited

Loading

mistercrunch commented Aug 17, 2017

xrmx commented Aug 17, 2017

xrmx commented Aug 17, 2017

bmackattack commented Aug 31, 2017 •

edited

Loading

xrmx commented Aug 31, 2017

jhult commented Sep 17, 2021

This issue was moved to a discussion.

This issue was moved to a discussion.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4 #2600

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 4 #2600

Comments

zhangjiajie023 commented Apr 12, 2017 • edited by xrmx Loading

Superset version

Expected results

Actual results

Steps to reproduce

change the sqlalchemy_database_uri from db2 to db1

python3.5 superset 0.15.4 mysql 5.7

xrmx commented Apr 12, 2017

zhangjiajie023 commented Apr 13, 2017

kkalyan commented Apr 13, 2017 • edited Loading

timfeirg commented May 8, 2017 • edited Loading

xrmx commented May 8, 2017

xrmx commented May 8, 2017

timfeirg commented May 8, 2017 • edited Loading

xrmx commented Jun 23, 2017

xrmx commented Jun 23, 2017 • edited Loading

jerryhall commented Aug 15, 2017 • edited Loading

mistercrunch commented Aug 17, 2017

xrmx commented Aug 17, 2017

xrmx commented Aug 17, 2017

bmackattack commented Aug 31, 2017 • edited Loading

xrmx commented Aug 31, 2017

jhult commented Sep 17, 2021

This issue was moved to a discussion.

zhangjiajie023 commented Apr 12, 2017 •

edited by xrmx

Loading

change the `sqlalchemy_database_uri` from db2 to db1

kkalyan commented Apr 13, 2017 •

edited

Loading

timfeirg commented May 8, 2017 •

edited

Loading

timfeirg commented May 8, 2017 •

edited

Loading

xrmx commented Jun 23, 2017 •

edited

Loading

jerryhall commented Aug 15, 2017 •

edited

Loading

bmackattack commented Aug 31, 2017 •

edited

Loading