Filament manager

[mateuszbrzezinski]

module name: filament manager

Component to manage filaments and track filament usage.

Signed-off-by: Mateusz Brzezinski [email protected]

[mateuszbrzezinski]

Endpoints examples:

List all spools

GET http://octopi.local:7125/filament/spool/list

List materials

GET http://octopi.local:7125/filament/materials

Add Spool

POST http://octopi.local:7125/filament/spool
Content-Type: application/json

{
"name": "test1",
"color_name": "red",
"color_code": "#FF0000",
"vendor": "eSUN",
"total_weight": 1000,
"used_weight": 0,
"diameter": 1.75,
"density": 1.24,
"material": "PLA",
"cost": 50.0
}

Update Spool

POST http://octopi.local:7125/filament/spool
Content-Type: application/json

{
"id": "00000C",
"active": true,
"used_weight": 123,
"diameter": 1.75
}

Get spool details

GET http://octopi.local:7125/filament/spool?id=000003

Delete spool

DELETE http://octopi.local:7125/filament/spool?id=00000E

Get spool set as active

GET http://octopi.local:7125/filament/active_spool

Set active spool

POST http://octopi.local:7125/filament/active_spool
Content-Type: application/json

{
"id": "000000"
}

[jarekwas]

looks good

[jordanruthe]

I don't have enough time to put all my thoughts out at this moment. By here are a few thoughts:

What about filament expended between jobs? It doesn't seem like this is tracking that and ideally a filament manager will track all actions, not just during a job.

It would be ideal to have a field for starting_weight or something where we could track the weight of the filament+spool so if I weigh it later I know exactly how much is left of filament.

I think having the ability to mark a spool as spent rather than having to delete it would also be nice. So you can track overall filament usage.

When I have time to do a better review I'm make some more comments.

[mateuszbrzezinski]

What about filament expended between jobs? It doesn't seem like this is tracking that and ideally a filament manager will track all actions, not just during a job.

@jordanruthe Not sure what you mean. Do you mean like manual filament extrude? Hmm not sure how to track that? If so these are rather small consumption, within the measurement error.

It would be ideal to have a field for starting_weight or something where we could track the weight of the filament+spool so if I weigh it later I know exactly how much is left of filament.

I can add spool weight field.

I think having the ability to mark a spool as spent rather than having to delete it would also be nice. So you can track overall filament usage.

Delete option is rather for different purpose, just to delete spool.
There is already active field, but may have sense to rename it to spent

When I have time to do a better review I'm make some more comments.

Thanks

[OSHW-Rico]

I like the idea with spool weight. In my opinion we should not account for every possible input weight configuration. Because it gets complicated if you want to manually update the weight, because you removed some filament or just entered the wrong values at the beginning. Take a look at the Octoprint Spool Manager, in my Opinion it is overkill how many input options there are. Some are calculated based on others, but still can be set manually. At also it isn't working as expected.

I think a spool creation should consist of selecting a Filament from a Database nothing more ideally. But if the filament is not in a database, i should be not more than Total Weight + Assumed Filament on Spool. Because that are the two things I can measure/know. Yes refill is a thing, where you could weight the filament on it's own, but you could also weight the spool before and after the refill is put on. But that's what I mean with it gets complicated with many input options.

[OSHW-Rico]

I think having the ability to mark a spool as spent rather than having to delete it would also be nice. So you can track overall filament usage.

Delete option is rather for different purpose, just to delete spool.
There is already active field, but may have sense to rename it to spent

You could create and endpoint to get only spools with more than x meters left. So you don't have to load all spools into the front-end to filter than for spool which can be activated. But it is still possible to list all spools if desired.

[mateuszbrzezinski]

You could create and endpoint to get only spools with more than x meters left. So you don't have to load all spools into the front-end to filter than for spool which can be activated. But it is still possible to list all spools if desired.

That might be good idea, actually when filament usage exceeds filament total weight, then it can be auto marked as active=false, and will add parameter show_inactive with default value false to spools list ednpoint

[Arksine]

If the history module isn't configured this call will raise an exception. Generally speaking I don't think that the spool manager should rely on the history module. If we need to add some spool data to job history we can do that, however it would probably be better for the history module to look up this module and request the current spool.

[mateuszbrzezinski]

Maybe spool_manager can publish event which can be consumed by history component? but anyway spool_manager depends on history component currently

[jordanruthe]

Maybe you can check if history == None and remove the metadata storage if the component isn't found.

A spool change notification would be good either way as it would notify front ends that they should update the active spool info (which I'm assuming will be shown).

I believe we should also allow for the case of multiple spools used during one print as changing colors/filaments mid-print seems to be a common thing (especially with the pause at layer feature in slicers). Which means you'd need to make a spools list in metadata rather than a spool dict. Spools would also have to be attached to jobs mid-print.

[Arksine]

I think in an ideal scenario the history module will look up spool_manager and add a reference to the spool if it has been configured. The spool_manager can emit a "spool_changed" event that that history can register. When a spool change is noted then history can add the current spool's reference to the ongoing job.

[Arksine]

I think rather than tracking a job's filament usage it would be more accurate to track the toolhead`s E position. We may need to register a remote method that allows Klipper macros to toggle spool tracking off/on during a spool change.

[jordanruthe]

I agree on the tracking the E position. Tracking the E position would also allow for outside of job spool tracking. I realize you previously have said that outside of job lengths should be lost in the noise or error, but if I expend 1m of filament doing some calibration/testing, that could be the difference between me being able to use the last of the spool or not, depending on a job size.

Agreed for the macros on the spool change.

[Arksine]

I left a few comments after an initial review. I'd like to get Jordan's feedback and make some decisions on what direction we want this to go before proceeding with a more detailed review. I think feedback from other client developers would be useful as well. Some things I would like to hammer out:

• Does spool_manager better describe this module's functionality than filament_manager?
• What fields can be calculated in the request? For example, used weight can be calculated from diameter, density and used length, so there is no need to store its value.
• What is the best way to store static spool data? Manufacturer, total weight, filament_diameter, density, etc. We could add a config file that contains spool templates and allow users to specify custom templates in moonraker.conf. The spool entries in the database could just hold a reference to the correct template. I think @OSHW-Rico makes a good point that we want adding a spool to be as painless as possible.
• I'm okay with keeping a "history" of spools, however I think we need to set a sane limit. Truthfully the same should done for the history module. Long term it is possible we could run into issues if we dont.

When working locally my commits look a lot like yours. However it is a good idea to squash them and create proper commit messages (as outlined in contributing.md) in a pull request, as it makes reviewing the code easier. Changes to any other modules (such as history.py) should be done in a separate commit.

[mateuszbrzezinski]

@Arksine

• Does spool_manager better describe this module's functionality than filament_manager?

yes, that makes sense, fixed that

• What fields can be calculated in the request? For example, used weight can be calculated from diameter, density and used length, so there is no need to store its value.

fixed that

• What is the best way to store static spool data? Manufacturer, total weight, filament_diameter, density, etc. We could add a config file that contains spool templates and allow users to specify custom templates in moonraker.conf. The spool entries in the database could just hold a reference to the correct template. I think @OSHW-Rico makes a good point that we want adding a spool to be as painless as possible.

not sure if that is good idea, what if you want to add new spool to db, but is already used a bit, or sometimes partial spools are available for purchase, so I rather would not do that on configuration level. We can make the default value for total_weight to 1000g, diameter to 1.75, and let user to use that default values or edit that, that should be enough.
Also adding new spool for new manufacturer, I don't feel like it should involve editing configuration file.
Although filament materials, that might be good idea to put it in configuration, that is rather finite/static set, rarely modified, but even then I would use that just as template, default value, but rather to let to edit that and store that as value parameter than just reference to configured material.
Similar for manufacturers, that is something dynamic, so when you add first filament, you have empty completion list, you need to add manufacturer by yourself, later we can provide available manufacturers in dropdown list in UI collecting existing manufacturers from existing spools, or add spool with new manufacturer, that would be more flexible.

• I'm okay with keeping a "history" of spools, however I think we need to set a sane limit. Truthfully the same should done for the history module. Long term it is possible we could run into issues if we dont.

that is to consideration, but you would rather not add 1000000 spools, rather like 20-100

When working locally my commits look a lot like yours. However it is a good idea to squash them and create proper commit messages (as outlined in contributing.md) in a pull request, as it makes reviewing the code easier. Changes to any other modules (such as history.py) should be done in a separate commit.

Definitely would squash commits on merging PR to master, but also squashed existing commits

For now I left changes to history module here, until we sort everything, otherwise that PR at least for now would be withiout context

Happy to get some comments from @jordanruthe

[Arksine]

not sure if that is good idea, what if you want to add new spool to db, but is already used a bit, or sometimes partial spools are available for purchase, so I rather would not do that on configuration level. We can make the default value for total_weight to 1000g, diameter to 1.75, and let user to use that default values or edit that, that should be enough.
Also adding new spool for new manufacturer, I don't feel like it should involve editing configuration file.
Although filament materials, that might be good idea to put it in configuration, that is rather finite/static set, rarely modified, but even then I would use that just as template, default value, but rather to let to edit that and store that as value parameter than just reference to configured material.
Similar for manufacturers, that is something dynamic, so when you add first filament, you have empty completion list, you need to add manufacturer by yourself, later we can provide available manufacturers in dropdown list in UI collecting existing manufacturers from existing spools, or add spool with new manufacturer, that would be more flexible.

I was thinking that specific manufacturers may produce materials with slightly different properties. For example, maybe Prusa PLA has a different density than ESun PLA. This information is static and could be templated through a config file. That said, I'm okay with just providing generic templates and allowing the user to add custom values.

that is to consideration, but you would rather not add 1000000 spools, rather like 20-100

I'm thinking that we max out on 1000 spool entries. I pushed a recent commit that sets a 10000 job limit for the history module. When (if?) the namespace reaches 10k entries it will pop off the oldest entry and add a new one.

[mateuszbrzezinski]

I was thinking that specific manufacturers may produce materials with slightly different properties. For example, maybe Prusa PLA has a different density than ESun PLA. This information is static and could be templated through a config file. That said, I'm okay with just providing generic templates and allowing the user to add custom values.

Could you elaborate a bit more, how would you want to implement that?
Can it be like it is now, predefined filaments in code, but export it to configuration file, then the filament name, and details are stored in spool data
Or you mean to create separate db namespace to store filaments, with dedicated endpoint to add/edit/delete?

I'm thinking that we max out on 1000 spool entries. I pushed a recent commit that sets a 10000 job limit for the history module. When (if?) the namespace reaches 10k entries it will pop off the oldest entry and add a new one.

That is a neat idea, in case of spools will be a bit more complex, as I guess we should pop off the oldest (last_used) inactive spool. But anyway that won't be a problem.

EDIT: eventually I took different approach, as spools are managed by user, so just in case limit is reached, endpoint will return error, so user has to delete spool by himself.

[marcandre83]

Thank you so much for developing this! If there is any possibility to, please add the option to use the filament database across several klipper instances in a local network.

[mateuszbrzezinski]

Thank you so much for developing this! If there is any possibility to, please add the option to use the filament database across several klipper instances in a local network.

that will be rather much more complicated, and I guess rather in another release, as currently Moonraker has only local db
Maybe @Arksine has some better view of that?

[OSHW-Rico]

Regarding the database for default values for spools. I thought more about a public github repo with all sort of default data to choose from. Everybody can do a PR with new Spool Values, Filamenttypes, Densitys, Netto/Brutto Weight of the spools etc. Maybe you could add your own private Repo via the Config file.

This could solve the MultiPrinter MultiClient issue and generate a big database (outside of Moonraker) which could even evolve into a bigger filament database as than just the data used by moonraker.

[Arksine]

Thank you so much for developing this! If there is any possibility to, please add the option to use the filament database across several klipper instances in a local network.

Moonraker isn't aware of other instances on the network. Clients could possibly sync spools across instances, however this can get complex as they must deal with new instances added, some instances not having the spool manager configured, some instances not reachable.

Regarding the database for default values for spools. I thought more about a public github repo with all sort of default data to choose from. Everybody can do a PR with new Spool Values, Filamenttypes, Densitys, Netto/Brutto Weight of the spools etc. Maybe you could add your own private Repo via the Config file.

Interesting idea, however I'm weary of having Moonraker rely on remote service. Some users have connection issues, it has already caused headaches with the update manager. In my view it would be better for clients to query such a service directly, they could then add these templates to the local instance when a user selects them.

With regard the current state of the module, I have a few comments:

• It only needs to emit an event when a new spool is selected. I don't think spool_added and spool_update events are necessary.

• We should remove the dependency on history. Instead it should subscribe to toolhead.position in a server:klippy_ready event handler. This subscription can be used to initialize a last_epos attribute, which it can use to calculate the difference in epos and accumulate filament used on the active spool. Unlike the other toolhead positions we shouldn't need to worry about its position being reset. We can save the active spool to the database when it is either deselected or in an on_exit() method in the SpoolManager class. If a component has defined on_exit() then moonraker will call it before it closes the database on shutdown/restart.

• For the configured templates, I was thinking about something like the following:

  [spool_manager template prusament_pla]
  material: PLA
  vendor: Prusa
  density:  1.24
  diameter: 1.75
  total_weight: 970
  spool_weight: (whatever it is)
  cost: 24.99
  

  A spool entry can store a reference to the template, the cost can be overridden. I don't think the spool entries need to store active, color_code, or comment, I'd prefer to streamline those entries and not store extraneous data. Of course an alternative is to have clients add this information to the database. If we go that route, we would need to store the templates separate from the spool entries.

• We will need client developers to chime in on the API. They can give specific guidance as to what they are looking for.

[Drachenkaetzchen]

• We will need client developers to chime in on the API.

I'm playing around with an RDM6300 RFID reader and tags with the intention to track which spool is inserted. My use cases are:

• Verify that the inserted spool matches the slicer's preset
• Check if the job about to be printed has enough filament left

I think all that should be possible once this PR is merged, however, a few things I noticed:

• The ID returned when adding a spool should be a GUID, and/or the ID should be settable by the user. This makes migration or merging of the database easier if a user decides to do so.

• I agree that the database should store as few fields as possible. The fields material, vendor and cost, color etc are not used for any calculation, and often they are not filled at all. Even though we developers wish that all fields are properly used, even I as a user am often lazy and might put "PLA eSun Green" in the name instead of the respective fields.

• However, I think there should be a way to store these values if needed. Not sure if that would work with the current database system in use, but a simple metadata field with JSON values could to the trick here. If that doesn't work, it's up to the person implementing additional features to create a separate database with all additional fields needed.

[campoyyano]

Hi everyone, any update on this ?? Is a very nice feature to have available for the klipper ecosystem

[locki-cz]

Unfortunatly still not merged to moonraker, not know where is problem ;((

[Tareku99]

It would be very nice to see this happen. What are the merge conflicts ?

[Digityle]

I too am interested in this project, would love to contribute. Have emailed @mateuszbrzezinski but no response, know very little python but am learning for this very project. I have a hx711 filament weight scale and RFID as @Drachenkaetzchen mentioned, managed through home assistant and would like very much to see this usable in Klipper. @feniarus any updates? Thx all...

[DerKnerd]

This feature is the main reason I cannot switch from Octoprint. I have three printers and I need to coordinate the filament use between them.

[jmccaull]

Since this seems like its been abandoned, I'm making an attempt at solving the last set of comments from @Arksine and creating a new PR.

@Arksine I'm somewhat learning python and how the framework functions as I do this so a push in the right direction would help. I'm trying to remove the dependency on history and plan on setting up a similar number of event handlers. From examples in other components, it appears to me like once I subscribe to the tool head, I'll need to check "server:status_update" events for the epos updates? Am I guaranteed to get an event every time the epos changes or could I miss some steps if a retraction happens?

Tracking the epos should also track usage outside of prints (right?) and my current plan is to update the usage every time a request is sent to get spool info, every time a job event happens and using an on_exit(). Are job events good to use like this or should I track printing status manually?
This could lead to a situation where usage since the last UI update causes the remaining amount to no longer be enough for the print when one is started and desired behavior might be to abort the print. Is this something we can/should handle here or would we expect the UI to request an update before every print is started?
Finally, will commands like G92 impact tracking this way? I'm not seeing a lot of info on this in the Klipper docs. Is there a max value for epos?

Thanks!

[DerKnerd]

@jmccaull Could you build it in a way so that external databases can be used. If you have multiple printers it is important to share the same DB between them.

[jmccaull]

@DerKnerd the "database" component used here is based on LMDB and from what I can gather only does local connections. It probably makes sense to get the use case working first for local, then figure out how to architect a distributed set up. There are a lot of problems to solve for when trying to set something like that up, maybe it make sense to set up some sort of filament server and have moonraker and front ends talk to that instead of directly to each other?

[DerKnerd]

@jmccaull I like the idea with a filament server. That would add more flexibility when you want to run multiple instances. I think the feature to run against an external database could be critical for many users.

[Jasontwd]

@Arksine , would you know a good macro for moonraker shutdown??
Im keen to have a shutdown with delay of 1800secs after the current printing has completed and hotends are cool.

eg
[gcode_macro SHUTDOWN]
Delay:??????

gcode:
{action_call_remote_method("shutdown_machine")}

something simple for the slicer to run in G code at end print command
cheers jason

[Digityle]

I also like the idea of a server, possibly even a public one with user spool collections (in my use case would be to track empty spool weights for scale tare, could get an average brand weight report from anonymous data sharing)... few years down the road who knows? got 200 grams of glow in the dark left over, anyone have a similar weight of blue to trade? etc... also @jmccaull I know very little of python but if u would like any testing r/e weight scale or rfid tagging I'd be glad to help! pretty handy with YAML and esp32 also...

[CrispyPyro]

Building something that can connect to https://www.3dprintlog.com/ might be an option. The Cura plugin can’t capture actual usage, and there’s nothing (yet)for Klipper.

[jmccaull]

Integrating with some sort of service sounds cool, but I was more meaning something you would run locally, probably on one of the machines already running moonraker. The point is that functionality and the interaction between different systems is beyond the scope of moonraker. You probably don't want all the other printers dependent on a single moonraker instance and you probably don't want them all dependent on something connected over the internet either. If anything the local filament server could reach out to some service over the internet and periodically sync data.

[pejotigrek]

if I may drop my few cents here...

while making/connecting to some kind of a centered filament database [preferably self-hosted] is a cool idea and could be as easy as just adding MySQL/MariaDB credentials to db that would keep those records, I think that most important think is to get filament manager working on a single, local installation first ;)

as not-a-programmer guy, who had a real pain moving from octoprint to klipper/moonraker because the whole semi-automatic filament manager was lost I am happy to see the discussion about the possibilities, but I am starting to be afraid that the main goal can be lost and I'll stuck with my google-sheets based filament control log ;)

[jmccaull]

Between vacation and life, I've not had much time to work on this. Digging through the code I'm pretty sure I've found where the status updates contain the subscribed objects, I'm just not sure how often these updates happen. I'm also having some issues getting pycharm and the tests working locally. If anyone wants to take this up, feel free, otherwise I'll get to it eventually.

Per the server and multi-printer functionality, the more I think about it the more I want to say they should be different components. They would share minimal functionality but the desired flow would probably be pretty different, ie you might need to "check out" spools before using them, then release them on change or shutdown, etc.

[jmccaull]

created #597 as a draft so far

[NonaSuomy]

Would there be a way to print QR codes for the spools and then scan them with the chamber camera feed to identify which roll you are loading quickly?

I tested this with

pi@mainsailos:~ $ wget http://IP/webcam/?action=snapshot -O qrcode.jpg | zbarimg qrcode.jpg
--2023-02-18 07:36:48--  http://IP/webcam/?action=snapshot
Connecting to IP:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [image/jpeg]
Saving to: ‘qrcode.jpg’

qrcode.jpg                        [ <=>                                            ]  28.89K  --.-KB/s    in 0.001s

2023-02-18 07:36:48 (24.7 MB/s) - ‘qrcode.jpg’ saved [29579]

QR-Code:
serial: 007
model: white
material: PLA
vendor: Prusa
density:  1.24
diameter: 1.75
total_weight: 970
spool_weight: 832
cost: 24.99
scanned 1 barcode symbols from 1 images in 0.18 seconds

Guess that works then you could poke this pr's spool code some how?

Is it possible to add this pr in it's current state to test this out?

Would like to then add a custom button to the mainsail webui where I could command it to run those processes while holding the spool to the cam feed then pull this code with the correct spool. As I’ve tested this below.

sudo apt install zbar-tools
cd /home/pi/klipper/klippy/extras/
wget https://raw.githubusercontent.com/th33xitus/kiauh/master/resources/gcode_shell_command.py

[gcode_shell_command qrcode_snapshot]
command: wget http://IP/webcam/?action=snapshot -O /home/pi/printer_data/gcodes/qrcode.jpg
verbose: True
timeout: 2.
[gcode_shell_command qrcode_scanner]
command: zbarimg /home/pi/printer_data/gcodes/qrcode.jpg
verbose: True
timeout: 2.
[gcode_macro QRCode_Scan]
gcode:
    RUN_SHELL_COMMAND CMD=qrcode_snapshot
    RUN_SHELL_COMMAND CMD=qrcode_scanner

I tested regular barcodes on most spools with this code above and enabling all barcodes in the apps arguments but for whatever reason maybe my camera isn’t clear enough for the library. It just says it can’t detect any code. QR codes almost always work no matter the weirdness of what it’s having with what it sees.

[Danabw]

Sorry to bother, but wondering where this stands...the lack of spool/filament management is the biggest (practically only) gap for me after leaving Octoprint for Klipper/Mainsail/Moonraker. Does this still have any likelihood of being implemented? I get folks' personal lives take priority, so please don't take this as a "hurry up, you owe me something" post. :)

[locki-cz]

Please try to make it first simple and add some features like qr code scan later? :)

[NonaSuomy]

It would be nice if there was a way to just read the btt smart filament sensors encoder wheel to understand consumption.

[Donkie]

As mentioned in this PR: #597 (comment) , I'm working on a separate web service to act as a database. Would be nice to get some feedback on the direction of this and whether @Arksine believes this is a good approach.

[Donkie]

Moonraker now supports interaction with Spoolman which fulfills the same goals, so I believe this PR could be closed.

https://moonraker.readthedocs.io/en/latest/configuration/#spoolman

[Arksine]

Thanks.