Troubleshooting

Start a Discussion or join our Discord for support and to help out.

First off, Machinaris (and Chia blockchains in general) are long-running processes, highly asynchronous in nature. As such, you must be patient when starting up a Machinaris container and immediately reporting a defect. Always wait at least 15 minutes after a startup, before re-checking for a concern. Thanks for your patience!

Here are some common issues encountered by Machinaris users. Solutions provided below.

Where are the logs?

The simplest way to get the logs is via the Machinaris WebUI:

1. Workers page, right-hand side, click on Server Log icon.
2. Blockchains page, choose your blockchain row (example: Chia), click View Log icon on right-side to see debug.log
3. Farming page, choose Worker by menu, click on blockchain such as Chia to see debug.log
4. Alerts page, choose Worker by menu, click on blockchain such as Chia to see the Chiadog log.

Log Disk Locations

If you have launched Machinaris and you've mounted an appdata folder like /home/USER/.machinaris, C:\Users\USER\.machinaris, /mnt/user/appdata/machinaris, etc, then look in that folder for logs. In particular, look in the machinaris/logs folder for files named webui.log, apisrv.log, and migration.log. These can have useful info. There are also logs at plotman/logs, mainnet/log, and chiadog/logs.

Finally, you can also view the main Docker output itself via:

docker logs machinaris

Internal Server Error

First, check the Web log for the error. If you can browse to /workers page, then find the Chia fullnode row, click on Web Log icon which pops open a log window. Then go to the page showing the Internal Server Error. Copy the error you find over to the Discord for support.

If the entire WebUI is not responding, then look in the webui.log and apisrv.log files on disk for an error and cause. Details above.

It's possible to restart the web server without restarting the entire Machinaris container, for example if you're actively plotting. To do so try:

docker exec -it machinaris bash
cd /machinaris
./scripts/start_machinaris.sh

Reset Machinaris Database

Machinaris stores recent status in local Sqlite databases, allowing fast page responses. Nothing in them is critical to keep, as they are re-populated within 10-15 minutes of Machinaris launch with most recent status. As such, there are times you may wish to remove them if SQL-related error is presenting. This will also ensure a clean SQL schema migration to latest. To do so, on Linux and MacOS try:

docker-compose stop
docker-compose rm -f
rm -f ~/.machinaris/machinaris/dbs/*.db
docker-compose pull
docker-compose up -d --force-recreate

On Windows, use Powershell from within your C:/Users/USERNAME/.machinaris folder.

On Unraid:

1. Stop the Machinaris container from Unraid Admin UI | Docker tab.
2. From the Unraid Console, delete all the database files with: rm -f /mnt/user/appdata/machinaris/machinaris/dbs/*.db
3. Start the Machinaris container from Unraid Admin UI | Docker tab.
4. Wait 10-15 minutes for data to re-populate, then browse to http://UNRAID_TOWER_NAME:8926

Reset Blockchain Config

Sometimes, even after multiple restarts of the Chia (or fork) container, the farming and/or wallet services simply do not start correctly. You may see repeated errors about "failing to connect" or "port not listening" in the logs outlined above. If you are certain that chia farm summary and/or chia wallet show is not good and not correcting itself, you can first try to delete the current Chia config.yaml and restart. The file for Chia is: ~/.machinaris/mainnet/config/config.yaml. The file for a Fork would be: ~/.machinaris-FORK/FORK/mainnet/config/config.yaml.

This will force a fresh (recent) configuration file to reset to the Chia/fork defaults. This can sometimes fix issues.

Reset Blockchain Database

If after a restart, and waiting 20-30 minutes, the Chia/fork farmer or wallet still will not work, you may need to stop the container and delete these databases:

1. ~/.machinaris/mainnet/db/ (or ~/.machinaris-FORK/FORK/mainnet/db/ for a fork)
2. ~/.machinaris/mainnet/wallet/ (or ~/.machinaris-FORK/FORK/mainnet/wallet/ for a fork)

Then restart the Machinaris container. In this case, Chia will begin syncing from zero again. This can take days, but has been known to fix problems with Chia. One workaround is to download a Chia DB checkpoint manually.

My Plots/Plotting/Key Files and Folders aren't found!

Machinaris runs as a Docker container. Think of this as a tiny virtual machine (VM) running on your host computer (Windows/Linux/Mac/etc). The Machinaris "VM" only knows about itself and it's filesystem. Using Docker volume mounts at container runtime, the Machinaris container has access to a handful of directories/folders on the host computer. However, the paths to a file within the Machinaris container and on the host filesystem ARE NOT THE SAME.

For example, when launching Machinaris on the Unraid host OS, the following host:container volume mapping is included:

- ~/.machinaris:/root/.chia

Let's break this down:

1. On the HOST filesystem, there is a folder named /home/USER/.machinaris
2. Inside the CONTAINER (aka VM), that same folder is mounted (read/write) at location: /root/.chia

If you create a file on the HOST filesystems named mnemonic.txt and put it in /home/USER/.machinaris, then programs running inside the CONTAINER will access this new file at /root/.chia/mnemonic.txt. It is the same file, but accessed at two different paths depending on whether you are at the HOST or CONTAINER level.

The same holds true for other paths such as completed plots and temporary plotting directories. Please understand the difference between a path on the HOST OS and in the Docker CONTAINER.

How can I use the commandline (CLI) inside the Machinaris container?

First you need to get a shell inside the container, for general Docker users, this is likely:

docker exec -it machinaris bash

For the fork containers, please use their name such as machinaris-flax, machinaris-hddcoin, etc in place of the main machinaris container name.

Some systems, such as Unraid, may offer a window which enters directly to the in-container shell.

Once there you can run commands against the chia, plotman, etc binaries.

Workers Are Missing

Please see the worker page for tips on troubleshooting connectivity between workers and the controller.

Performance Issues

Chia blockchains require a fair amount of resources to run, particularly when the fullnode is syncing. Multiply this by the number of fork fullnodes you are running on the same machine and you may have issues on your hardware. Add in a regular Chia "dust storm" or two and you may see a "Farming" status of "Not Synced". To troubleshoot these sync issues:

1. Ensure you are running Machinaris fullnodes (Chia & forks) with the /root/.chia volume mount (aka appdata) on a decent SSD drive. Do not use a HDD drive to hold the blockchains and other application data. This is too slow.
2. Then, stop all containers.
3. Start only the Machinaris container (aka the Chia fullnode). Run that for 24 hours with no other forks and verify that Chia is synced and stays synced. If you are unable to get Chia synced without any forks even running, possibly during a dust storm, try modifying these Chia settings.
4. Only if you have a synced Chia container, then start up a fork fullnode container and let it run until fully synced. Let it run another few hours to ensure the fork fullnode maintains its sync. Ensure the Chia fullnode does not lose sync when you are running the fork fullnode.
5. Slowly repeat the previous step, each time adding one more fork fullnode container. Ensure Chia and all other running forks maintain their sync.

Only add as many forks as your hardware can support. If you begin seeing sync issues, then stop recently added fork containers to free up resources, and get your primary blockchains syncing again.

Memory Limitations

If you run out of memory on the host when farming multiple blockchains, I recommend you stop farming those you feel are lower priority. Alternatively, you can limit memory at an individual container level also, docker run args for reference:

• --memory=4GB caps memory usage, if swap is available the same amount is made available (as a maximum), swapiness is same as host.
• --memory=4GB --memory-swap=10GB allows you to specify combined memory + swap (this example would give 6GB swap limit), swapiness is same as host.
• --memory=4GB --memory-swap=10GB --memory-swappiness=10 swapiness value can be set for the container, allowing you to be more or less aggressive with swap that host settings

For Unraid, these args go in Extra Parameters of the template) NOTE: being too aggressive with memory limits can lead to OOM kills of farming process