Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DelimitedFiles readdlm is externally slow for Complex{Float64} #35721

Closed
yzcj105 opened this issue May 4, 2020 · 5 comments · Fixed by #35742
Closed

DelimitedFiles readdlm is externally slow for Complex{Float64} #35721

yzcj105 opened this issue May 4, 2020 · 5 comments · Fixed by #35742
Labels
performance Must go faster

Comments

@yzcj105
Copy link

yzcj105 commented May 4, 2020

 using DelimitedFiles
A = randn(Complex{Float64},300,30)
B = randn(Float64,300,30)
writedlm("A.csv",A,',')
writedlm("B.csv",B,',')
 @time readdlm("A.csv",',',Complex{Float64},'\n')
 10.645386 seconds (67.52 k allocations: 3.250 GiB, 2.68% gc time)
@time readdlm("B.csv",',',Float64,'\n')
  0.005304 seconds (18.00 k allocations: 878.875 KiB)

The readdlm for Complex{Float64} is much much slower and memory requiring compared to real numbers.
I've also compared the time consuming of pandas.read_csv in python, pandas is very fast every for complex numbers.

@yzcj105
Copy link
Author

yzcj105 commented May 4, 2020

For Complex 300*300 matrix, the result has never come out after waiting for minutes.

@yzcj105
Copy link
Author

yzcj105 commented May 4, 2020

alternative ways to read complex matrix from csv files
A is 300*300 matrix.

julia> @time df = CSV.read("A.csv";header=0,delim=',')
  0.006457 seconds (22.12 k allocations: 3.413 MiB)
# return 300×300 DataFrames.DataFrame. Omitted printing of 298 columns
....
....
 A = convert(Matrix,df) .|> x->parse(Complex{Float64},x)
# return. 300×300 Array{Complex{Float64},2}:

Which is much much faster compared to readdlm

@stevengj
Copy link
Member

stevengj commented May 7, 2020

Should be fixed by #35742.

@yzcj105
Copy link
Author

yzcj105 commented May 7, 2020

Thank you!

@yzcj105 yzcj105 closed this as completed May 7, 2020
@KristofferC
Copy link
Member

Let's not close until that merges (it will close automatically)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Must go faster
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants