Skip to content

Commit

Permalink
reduce chunk size for countries table (#1279)
Browse files Browse the repository at this point in the history
One user reported the load of the "countries" table exceed
max_allowed_packet which in some configurations can be as low as 1MB.

Changing the chunk size from 500 to 50 has a small cost on the initial
load of the data (one additional second to the 17 taken previously)
while being more universally usable without changing the configuration
of the mysql server.
The new packet size is estimated to be about 500KB.

The committer has not checked other tables
  • Loading branch information
vivo75 authored and bkyryliuk committed Oct 7, 2016
1 parent 8a5f050 commit 94578cb
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion caravel/data/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ def load_world_bank_health_n_pop():
tbl_name,
db.engine,
if_exists='replace',
chunksize=500,
chunksize=50,
dtype={
'year': DateTime(),
'country_code': String(3),
Expand Down

0 comments on commit 94578cb

Please sign in to comment.