uploading large files for data analysis

[Knochi]

Hi,

would it possible to open large files >10Mb for data analysis? The upload function seems not to work for this.
In pure python one would open a file and iterate through it line by line (loading only one line at a time to RAM).

with open("log.txt") as infile:
    for line in infile:
        print(line)

i want to open a microcontroller serial trace and visualize certain events on a timeline as a plot (timeline or matplotlib)

[rodja]

Thanks for reporting this issue. The current ui.upload implementation does not support such large files because the uploaded data is transferred to the python world via socket.io (eg. websocket) which only allows small data packages. We may be should change ui.upload to be based on Quasars Uploader where an upload url can be specified. Stay tuned.

[falkoschindler]

I wonder if we need some possibility to load local files. In a web application context this does not make too much sense. But when building a UI for data processing that runs locally anyway, we would only need to know the file path and the user could load the file line by line.

[rodja]

@Knochi where is the file saved initially? Do you want to use NiceGUI as a server where you can upload files for analysis? We often let NiceGUI directly run on the machine which creates the logs in the first place. Then you don't need an upload. Simply run a function with your example code from #220 (comment) on button click.

[Knochi]

Yes at the moment I‘m processing files on the same machine. So could be hard coded theoretically.
Would be nice to have the ability to select those files via the UI, even if they are on the same machine and in the future I can think of an analyzer service where users can upload their files.

[falkoschindler]

Ok. We will definitely improve the way files are transmitted from frontend to backend. This should allow you to work with pretty large files.

But opening files line by line is rather tricky - at least from an HTML/JavaScript/Quasar frontend. Theoretically, if we are on the same machine, we would only need to select a file and transmit the file path to Python where it can be opened. But for security reasons JavaScript does not get the path of a selected file, only its name.

An alternative would be to create a custom UI, e.g. presenting the user a list of files to choose from.