Enable support to verify SHA256 checksum when uploading gcode file

[shamal69]

Is your feature request related to a problem? Please describe

I have had multiple instances where valid gcode from my computer is uploaded to the RPi and is corrupted in transfer. This results in [!! Move out of range: xxxxxxx] failing the print.
The ability to verify a SHA256 checksum upon upload would be a great option and result in the ability to reupload invalid code prior to commencing printing.

Describe the solution you'd like

Ability to Paste into a field, or upload SHA256 checksum for the gcode file via the Fluidd interface and have this compared to the calculated value for the file prior to committing to virtual_sd_card storage.

Describe alternatives you've considered

Generating my own checksums from both the source machine and once the file has been uploaded to the RPi ~/gcode_files directory.

Additional information

This is a great interface, thank you very much for your efforts!

[matmen]

Corruption while uploading seems implausible, as the checksum used in the ethernet frames should prevent corrupted data from being accepted on the host. Are you sure the gcode files are correct and intended for the printer/in range? Can you supply a corrupted gcode and its working counterpart?

[pedrolamas]

I agree, sounds very unlikely to have corruption on upload... would make more sense if the Pi was overheating, overclocked, or if the SD card was malfunctioning/currupted!

[shamal69]

From my source machine:
xxx /G-Code Files % LL Bag_Holder.gcode
-rw-r--r--@ 1 uuuu gggg 18416713 31 May 16:10 Bag_Holder.gcode
xxx /G-Code Files % shasum -a 256 ./Bag_Holder.gcode
c752a7abc0ec6af11319cce539baf0c184e03c95494c741f5ac2c3d182b12811 ./Bag_Holder.gcode

From the RPi host:
pi@klipper:/gcode_files $ ll Bag_Holder.gcode
-rw-r--r-- 1 pi pi 18416713 May 31 16:12 Bag_Holder.gcode
pi@klipper:/gcode_files $ shasum -a 256 Bag_Holder.gcode
b5dfd12a2ff7f17e2819ba73365e0967a7a8e87dfe7eb3789140769b219b14f2 Bag_Holder.gcode

As you can see the Checksums are different, but not the file sizes.

[matmen]

Any chance you can upload both of these files to a file sharing host? I believe network transfer shouldn't be an issue here. Especially the errors in only move commands seem odd, I would expect other characters to get corrupted aswell.

[shamal69]

I agree, sounds very unlikely to have corruption on upload... would make more sense if the Pi was overheating, overclocked, or if the SD card was malfunctioning/currupted!

The Rpi (4B / 8GB) is in an Argon One Case with a USB 3.0 SSD attached. The issue is definitely not the SD card, but thanks for the question. No overclocking.

As for temps:
pi@klipper:~/gcode_files $ vcgencmd measure_temp
temp=41.3'C

???

[shamal69]

Any chance you can upload both of these files to a file sharing host? I believe network transfer shouldn't be an issue here. Especially the errors in only move commands seem odd, I would expect other characters to get corrupted aswell.

Here is the error ... would it be possible to get a gcode line number where the error occurred in the output at all?

16:12:41 $ SDCARD_PRINT_FILE FILENAME="Bag_Holder.gcode"
16:12:41 File opened:Bag_Holder.gcode Size:18416713
16:12:41 File selected
21:23:12 !! Move out of range: 55.855 99187.718 12.940 [18892.739]

As you can see, this was 5 hours later. I am just trying to get my printer (CR-10S Pro V2) up and running reliably.

This is the Sliced File link: https://1drv.ms/u/s!Alykd85u5rFRgtdjGHbk3AP8OYyKZQ?e=Pl66Ty

This is the uploaded file link: https://1drv.ms/u/s!Alykd85u5rFRgtdkAmGbXiv23IkfSg?e=Al1HX9

[pedrolamas]

The 2 files text is exactly the same, the only difference is probably on encoding, BOM, or something like that.

Having said that, the error is not an upload issue, but the fact that you are trying to move the toolhead outside of the bounds of the printer!

Apologies, I see now I compared the files incorrectly...

Here is the correct diff:

I can see clearly there are really odd issues here...

Can I ask how you uploaded these files?

[pedrolamas]

@shamal69 are you using any special module here? like the Preprocessor for Object Cancellation or something of sorts?

[shamal69]

@shamal69 are you using any special module here? like the Preprocessor for Object Cancellation or something of sorts?

I have no strange bits in the mix at all ... I uploaded the file using Fluidd to the web interface. My setup is simply a USB connected RPi4 8GB to a stock (ATMEGA 2560) Creality controller board (v2.4) in a CR-10s Pro v2.

Thanks for the reply and does that information help?

[matmen]

This is really odd.. Does it happen consistently or only sporadically? What if you delete the file and re-upload it (in that order, to try to get it to write to the same address)?
A couple of the blocks are multiples of 512b, so this makes me think the SSD might have gotten corrupted, can you run a SMART status on it (and maybe even fsck)?
I'm honestly not sure if this is worth implementing, as I haven't heard of anyone having problems with this yet..

[shamal69]

This is really odd.. Does it happen consistently or only sporadically? What if you delete the file and re-upload it (in that order, to try to get it to write to the same address)? A couple of the blocks are multiples of 512b, so this makes me think the SSD might have gotten corrupted, can you run a SMART status on it (and maybe even fsck)? I'm honestly not sure if this is worth implementing, as I haven't heard of anyone having problems with this yet..

It is random, hence my initial request to implement a CRC/HASH check for the upload within the web interface ...

The SSD is fine - here is the output of the SMART test:
pi@klipper:/gcode_files $ sudo smartctl -t short /dev/sda
smartctl 7.2 2020-12-30 r5155 [armv7l-linux-5.15.32-v7l+] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 2 minutes for test to complete.
Test will complete after Wed Jun 1 08:41:44 2022 AEST
Use smartctl -X to abort test.
pi@klipper:/gcode_files $ sudo smartctl -H /dev/sda
smartctl 7.2 2020-12-30 r5155 [armv7l-linux-5.15.32-v7l+] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

The SSD is fine. The printer, printed a larger file overnight with no issues. I assumed corruption either in the transfer process between the slicer machine and the RPi, or from the RPi to the Printer over the USB interface. Given that we have established the files are different between the slicer and RPi machine, I think that at least rules out the USB option (for the moment anyway) ...

So, how can I get my setup to reliably print using the Klipper architecture? That's is really all I am trying to achieve here. Thank you for the help.

[shamal69]

So I have run a number of test prints today through copying files over to the RPi using either SFTP or Samba ... no issues whatsoever, so it seems the upload via the web interface may be causing the issue.

Using latest versions of both Firefox and Microsoft Edge on Mac OS (Monterey 12.4) have produced failed print issues.

Hope this helps

[pedrolamas]

@shamal69 what type of Fluidd/Klipper setup do you have? Did you install everything with Kiauh, or using FluiddPi, or something else?

[shamal69]

@shamal69 what type of Fluidd/Klipper setup do you have? Did you install everything with Kiauh, or using FluiddPi, or something else?

@pedrolamas I have Fluidd/Klipper installed on RPi 4+ 4GB board in an Argon case with USB attached SSD. The printer is a completely stock CR-10s Pro V2 (Mega 2560 board) apart from Klipper. I have not patched the LCD yet - (trying to understand what Klipper build I need to support that and how to install it). No Octoprint, No Mainsail, just klipper and Fluidd, both installed via Kiauh. The only non "vanilla" package installed is mlocate from the RPi repository.

Klipper version is: v0.10.0-419-g4490a584
Fluidd version is: v1.18.1-1b4d6d6
Moonraker version: v0.7.1-572-gd53cf27

No updates currently available ... all up to date on the Settings Cog -> General screen.

[pedrolamas]

Thank you for providing all this information @shamal69, I have to admit I am quite stunned with the issue you are having... in a "normal" environment, this should not be happening as it can't be any data corruption given the existing multiple layers of system checks (on the TCP, ethernet frames, disk checks, etc.)

I tested uploading the files you sent and then downloading them again (using Edge Dev in Windows 11) and the contents matched 100%, so there was no corruption whatsoever.

The only way I can see this happening would be if there was a MIT ("man in the middle") kind of thing, a plugin on the browser or some sort of firewall/antivirus that is hijacking the connection and changing the data... I'm not a Mac user so this is not something I'm going even to start speculating.

The bottom line is that Fluidd is just a web interface, so there is little we can do in terms of generating hashes - this would have to be done on the server side, Moonraker.

[shamal69]

Hi @pedrolamas - thank you very much for your and the others feedback in response to this issue. It seems the problem is entirely with the G-Code generation from SuperSlicer on the Mac OS. I have swapped to SuperSlicer on a Windows 10 machine and have had no more issues ...