Query csv with group by very slow

[xbsura]

Describe the situation
import chdb
res=chdb.query('select count(*) cnt from file("/Users/xbsura/Downloads/organizations-2000000.csv", CSVWithNames) group by Name order by cnt desc', 'CSV')

wc -l /Users/xbsura/Downloads/organizations-2000000.csv
2000001 /Users/xbsura/Downloads/organizations-2000000.csv

head /Users/xbsura/Downloads/organizations-2000000.csv
Index,Organization Id,Name,Website,Country,Description,Founded,Industry,Number of employees
1,391dAA77fea9EC1,Daniel-Mcmahon,https://stuart-rios.biz/,Cambodia,Focused eco-centric help-desk,2013,Sports,1878
2,9FcCA4A23e6BcfA,"Mcdowell, Tate and Murray",http://jacobs.biz/,Guyana,Front-line real-time portal,2018,Legal Services,9743
3,DB23330238B7B3D,"Roberts, Carson and Trujillo",http://www.park.com/,Jordan,Innovative hybrid data-warehouse,1992,Hospitality,7537
4,bbf18835CFbEee7,"Poole, Jefferson and Merritt",http://hayden.com/,Cocos (Keeling) Islands,Extended regional Graphic Interface,1991,Food Production,9974

this sql need more than 1min to finish, and memory used is more than 100G

• Which ClickHouse server version to use
  res = chdb.query('select version()', 'Pretty'); print(res.data())
  ┏━━━━━━━━━━━┓
  ┃ version() ┃
  ┡━━━━━━━━━━━┩
  │ 22.12.1.1 │
  └───────────┘

• Queries to run that lead to slow performance
  select count(*) cnt from file("/Users/xbsura/Downloads/organizations-2000000.csv", CSVWithNames) group by Name order by cnt desc

Expected performance
200MB file, maybe less than 1 seconds is ok

[auxten]

Where can we download organizations-2000000.csv?

[xbsura]

https://media.githubusercontent.com/media/datablist/sample-csv-files/main/files/organizations/organizations-2000000.zip

download from here

[auxten]

This is a serious bug, any CSV data above 200MB with aggression can reproduce.
Will dig into this soon.

[lmangani]

Test seems to be passing with chdb 0.8.0 and libchdb 0.8.0 which includes #32 by @auxten

[lmangani]

@xbsura could you kindly retest and confirm the latest release fixes the reported issue? Thanks for your report!

[xbsura]

@xbsura could you kindly retest and confirm the latest release fixes the reported issue? Thanks for your report!

confirm fixed, thanks