Comments (7)
One more question: is the checksum calculation done by POST /server/files/upload a heavy operation CPU wise (for the Moonraker server/host)? Or is this done in a streaming fashion while the upload is being processed (chunk by chunk) and therefore not of big impact?
The checksum is calculated in a streaming fashion as the file is received. Its always calculated, regardless of whether or not the form includes the checksum
argument, so there is no additional overhead. The checksum is calculated before the file is passed off to the metadata processor, so there it should always match the checksum calculated by the frontend before submitting the upload request.
from moonraker.
The upload endpoint accepts an optional SHA256 checksum in the form. The frontend should pre-calculate the hash and add it to the form. If the checksum doesn't match Moonraker will return a 422 error. This flow is generally superior over the requested flow (return a checksum after processsing), as prevents a file with a mismatched checksum from being written to the destination.
from moonraker.
Hi,
The original question above was not intended to implement upload error prevention, but instead as a mechanism to prevent having to upload a file which already exists on Moonraker.
The idea is to optimize having to send 400MB files. The OP originally asked in DM about whether an approach like rsync was possible, and I had to fall back to API based approach. In that case knowing a checksum can prevent the client (the FDM Monster server in this case) to upload a big file (if its not new by only 1 bit) and then proceed immediately to print it.
In the proposed situation the file has to be uploaded (with checksum multipart field) after which Moonraker will (most likely) never throw an error as the new file is probably fine. Therefore the (large) upload is not prevented and no optimization is achieved.
Our feature request might have been misinterpreted a bit ;)
from moonraker.
@iandouglas based on our latest conversation and change in requirements on my part, shall we close this issue?
from moonraker.
It looks like I did misinterpret the original request. While it sounds like you have reached an alternative solution, I thought it would be worth providing feedback on why I don't believe it would be a good idea to modify the return values for existing endpoints:
- Moonraker attempts to avoid processing complete gcode files. There are exceptions (outlined below), generally speaking Moonraker attempts to read out a minimal part of the file in an effort to extract metadata as fast as possible.
- Presuming Moonraker calculated a checksum for every gcode file, caching the checksum isn't necessarily reliable. A file could be modified after upload.
- As mentioned above, Moonraker does process complete uploads in two scenarios: UFP uploads and when "object processing" is enabled and detected as necessary. Checksums aren't useful in either of these scenarios, as the the result is guaranteed to be different from the original file. That said, popular slicers now support native Klipper objects, so the latter scenario is less likely to occur.
FWIW, I would be willing to consider a new endpoint that calculates a checksum for a file on request. This way we avoid the major pitfalls outlined above.
from moonraker.
- Moonraker attempts to avoid processing complete gcode files. There are exceptions (outlined below), generally speaking Moonraker attempts to read out a minimal part of the file in an effort to extract metadata as fast as possible.
And I completely agree with this from your point of view. I would do the same.
- Presuming Moonraker calculated a checksum for every gcode file, caching the checksum isn't necessarily reliable. A file could be modified after upload.
Agreed. If a file is changed on the root directory attached to Moonraker, that's up to the user to figure out (its simply out of my control). I hope to document this so the user is aware from the perspective of FDM Monster.
- As mentioned above, Moonraker does process complete uploads in two scenarios: UFP uploads and when "object processing" is enabled and detected as necessary. Checksums aren't useful in either of these scenarios, as the the result is guaranteed to be different from the original file. That said, popular slicers now support native Klipper objects, so the latter scenario is less likely to occur.
I see. Interesting to note.
FWIW, I would be willing to consider a new endpoint that calculates a checksum for a file on request. This way we avoid the major pitfalls outlined above.
It would be a nice thing. But let's get this question answered first:
One more question: is the checksum calculation done by POST /server/files/upload
a heavy operation CPU wise (for the Moonraker server/host)? Or is this done in a streaming fashion while the upload is being processed (chunk by chunk) and therefore not of big impact?
I want to be able to advise the users of FDM Monster appropriately whether checksums are wise for large files (400MB+)
from moonraker.
Perfect! I suggest that we keep this issue open with a new title "Add API endpoint to recalculate file hash" and that it gets low priority. For my project at this point the feature is an edge case, but it might become important after the next half year when moonraker is fully supported and out of beta in FDM Monster.
If things change on my end I will let you know.
from moonraker.
Related Issues (20)
- process uptime
- MQTT Sensors Not Receiving Data HOT 3
- Have an auto backup mechanism for the LMDb database
- update_manager: git repo validation is done on stale state, results in exceptions HOT 8
- Bug: Component [update_manager] failed post init HOT 3
- failed to detect branch when switching from dirty repo to new fork of plugin HOT 3
- Startup failed: NameError: name 'botocore' is not defined HOT 9
- Moonraker Error with some Filenames HOT 2
- Failed to load button HOT 5
- Fritzdect Power Outlet http
- Ensure time is synced before refreshing the update manager status. HOT 1
- Device Power on mqtt zigbee2mqtt payload HOT 7
- Power locked_while_printing: True asserting at startup when not printing HOT 5
- Cannot Retrieve File Metadata for Spoolman HOT 10
- Spoolman Proxy: Include total count header in response
- How to add Authenticator HOT 2
- unable get printer info from moonraker HOT 5
- Gcode-shell-command Error: Failed to restart KlipperScreen.service: Interactive authentication required. HOT 2
- Fluidd Moonraker connection to elegoo Neptune 4+ on MAC failed (Force Refresh) HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from moonraker.