Memory problems in the case of large data volumes！！！

[higher2016]

elasticsearch-ingest-csv/src/main/java/info/johtani/elasticsearch/plugin/ingest/csv/CsvProcessor.java

Line 56 in 1be903f

CsvParser parser = new CsvParser(this.csvSettings);

[johtani]

Hi @higher2016
Good catch. One question. Do you have any issue on that?

[higher2016]

The problem we faced was very simple, so we use a simple way to fix it. We just use apache's StringUtil to cute the input string and get the string array. But this solution does not satisfy all situations. My advice is to avoid creating objects inside the methods. If you are interested our test data, we can communicate privately.

[johtani]

Thanks. Of course I can understand your suggestion and it make sense.
I'll make a simple benchmark on JVM heap size before and after. Please wait a moment :)

[higher2016]

Thanks. But now Japan is estimated to be at 1 am, pay attention to rest.

[johtani]

Ah, "a moment" that I meant is few days...

[johtani]

Create a parser instance at Constructor

[2019-09-04T18:51:11,957][INFO ][i.j.e.p.i.c.CsvProcessorTests] [testManyTimes] start
[2019-09-04T18:51:12,380][INFO ][i.j.e.p.i.c.CsvProcessorTests] [testManyTimes] end. Loop is 50000 times. Process Time is 422 ms

Create a parser instance in process method.

[2562-09-04T11:53:13,798][INFO ][i.j.e.p.i.c.CsvProcessorTests] [testManyTimes] start
[2562-09-04T11:53:21,188][INFO ][i.j.e.p.i.c.CsvProcessorTests] [testManyTimes] end. Loop is 50000 times. Process Time is 7389 ms

[higher2016]

I‘m no sure if CsvParser is thread safe. And one more question, why the way that Create a parser instance at Constructor use more time?

[johtani]

Oh, I mistook. that's opposite, so I updated the comment. I shouldn't do something in midnight :p

[johtani]

https://discuss.elastic.co/t/does-elasticsearch-ingest-processor-need-to-be-thread-safe/68084

and

uniVocity/univocity-parsers#268

[johtani]

I'm releasing to Sonatype, but Sonatype respond 502 or 504... please wait a release...

[johtani]

Fixed >= version 7.3.2

[johtani]

Hi @higher2016 ,
I released the plugin 7.3.2 that fixed this issue. If you are interested in, please try it.