Fix csv parsing function

[joshuali925]

Signed-off-by: Joshua Li [email protected]

Issue #, if available:

• [BUG] Missing fields in the csv report opensearch-project/reporting#43
• Missing fields in the generated file #363

Description of changes:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[zhongnansu]

Can we add test case for this?

[zhongnansu]

Why CI is not triggered for this PR

[joshuali925]

Can we add test case for this?

Added. Did we have memory issues with csv reports? I had to flatten the hits to take care of selecting columns like a.b.c from { 'a.b': { 'c': 'value' } }, this might increase memory consumption

[joshuali925]

https://github.com/joshuali925/kibana-reports/actions check CI here, i think github is having problem with PR triggers

[zhongnansu]

Can we add test case for this?

Added. Did we have memory issues with csv reports? I had to flatten the hits to take care of selecting columns like a.b.c from { 'a.b': { 'c': 'value' } }, this might increase memory consumption

maybe, like this issue described. opensearch-project/reporting#34

[joshuali925]

maybe, like this issue described. opensearch-project/dashboards-reports#34

Got it. I don't think the issue can be resolved by just optimize memory usage while generating csv. We'll have to introduce some kind of external storage for processing data. So this PR shouldn't affect that issue

[zhongnansu]

maybe, like this issue described. opensearch-project/dashboards-reports#34

Got it. I don't think the issue can be resolved by just optimize memory usage while generating csv. We'll have to introduce some kind of external storage for processing data. So this PR shouldn't affect that issue

I think a hard limit on csv file size may be another approach, such as 10 MB

[zhongnansu]

can we merge this in? @joshuali925

[joshuali925]

missing an approval, @davidcui-amzn could you take a look please?