ENH: DataFrame.append is slow when the DataFrame is huge

[junjunjunk]

Is your feature request related to a problem?

pandas.DataFrame.append is slow.
Users use DataFrame.append multiple times. And it's more pronounced when the data is huge.
That would take a long time to combine the data.
So some users are using DataFrame.from_dict which is faster than DataFrame.append. (check additional reference)

N = 10000
for i in range(N):
  df = df.append(other, ignore_index=True)

Describe the solution you'd like

I guess DataFrame.append is slow because there's a copy operation of DataFrame in somewhere.
But I haven't found it yet.
The solution will be to rewrite the process to be more efficient by not making copies.

Additional reference

https://stackoverflow.com/questions/27929472/improve-row-append-performance-on-pandas-dataframes

[mroeschke]

Of note, append may be deprecated in the future: #35407

We recommend using concat by first compiling a list of DataFrames to concat first.

[jreback]

there is a large warning in the docs about this