Adventures in Chia

With the continuing decline of the Chia price and the increase in the cost of everything, especially hard disk drives I decided to get out of Chia. I'll be selling off the hardware that I dedicated to it.

The chia farm is still running. Chia have released a 2.7.0 version that is needed for the upcoming soft-fork  to address the blockchain security threats from AI. The soft-fork will activate at block 8,655,000, around the 29th of April . Apart from applying a point release to bring all the harvesters up to date I also had to upgrade the chia client on each one due to the 2.7.0 release. As of writing this the farm consists of 8 harvesters and one full-node. It has 4253 plots totalling 420TiB. I am still trying to fill one of the harvesters after the TR-004 decided it didn't understand the disk configuration. Refer to my Sept 2025 "The farm took a hit" post for the details.  

I filled up the TR-004 I mentioned previously after having to wipe the file system. I have also got the last one refilling, weather permitting. I've installed Raspberry Pi OS based upon Debian Trixie on all the Rpi harvesters now. I managed to preserve the btrfs file system and reinstalled everything from scratch on them. Current setup: 2 x Raspberry Pi4 (4GB), two external USB 16TB HDD each running btrfs in single mode. 4 x Raspberry Pi5 (8GB), TR-004 with 4 x 16TB HDD each running btrfs in raid0. 1 x Ryzen 3600 with 10 x 20TB HDD running zfs in raidz2. 1 x Intel i3 with 7 x 8TB HDD running zfs in raidz1.   Tomorrow is forecast to be 33 degrees C so I'll be powering the farm down for a bit.

I went to replace what I thought was the loud TR-004 with a new one. After swapping drives around and powering it up the TR-004 decided it had one drive of 54TB and btrfs couldn't mount it as the other drives were missing. This is despite the dip switches on the back being set to JBOD mode. Pressing the Set button on the back didn't help. Changing the dip switches to individual mode didn't help. Swapping everything back into the original TR-004 didn't help either. I ended up wiping the array and reinstalling everything. I have been filling it up, currently we're up to 320 plots. Using 16TB drives they hold 560 plots with a bit of free space left over. I have it doing a defrag which has been going for 3 days now. It needs enough free space to do the defrag. Its online while doing the defrag meaning the harvester can still access the plots. You'll notice I said above what I thought was the loud TR-004. It turns out it wasn't the loud one. For a bit of context ...

I am currently filling out the last TR-004 which has 4 x 16TB drives in a raid0 configuration. Its point 8 on my July post. Its doing the last 60 plots which should take it up to 560 plots. One of TR-004's is making a loud humming noise which I think is the fan not working properly so I will look at replacing it. They have a 2 year warranty so I might RMA it once I have another one to put the drives is so there is minimal down time. I took all the drives out and reseated them in the sleds to try and troubleshoot it. The other three TR-004's only make hard disk noises. Chia's current price is 9.13 USD as I type this. At this point I could look at replacing the smallest drives in the farm which are 8TB drives (point 2 on my July post) with something larger or I could add another Raspberry Pi5, TR-004 and four new drives. Maybe the US SEC's "Project Crypto" will improve the price although most likely Bitcoin will be the currency that people invest in.  

  I currently have 4350 plots taking 430.5TiB spread between 8 harvesters. Current storage consists of: Main storage is a PC with 10 x 20TB drives running ZFS in raidz2 (1460 plots). Secondary storage is a PC with 7 x 8TB drives running ZFS in raidz1 (397 plots). Pi4 (4GB) plus 2 x 16TB USB drives running btrfs in single mode (280 plots). Pi4 (4GB) plus 2 x 16TB USB drives running btrfs in single mode (275 plots). Pi5 (8GB) plus TR004 with 4 x 16TB drives running bftrs in raid0 mode (560 plots). Pi5 (8GB) plus TR004 with 4 x 16TB drives running btrfs in raid0 mode (560 plots). Pi5 (8GB) plus TR004 with 4 x 16TB drives running btrfs in raid0 mode (560 plots). Pi5 (8GB) plus TR004 with 4 x 16TB drives running btrfs in raid0 mode (268 plots, filling up).  Full node is an Intel I5-11000 PC with a U.2 enterprise SSD for work space (currently on its 3rd) plus a couple of M.2 SSD (boot drive, spooling drive). It has 128GB of RAM and a 10GbE NIC. Even the enterprise SSD's wear out. A...

I currently have 2597 plots spread between 4 harvesters. Due to heat I have had the Chia farm off for multiple days which doesn't help with winning blocks or for filling up the 2nd TR-004. Current storage consists of: Main storage is a 10 x 20TB machine running ZFS file system, raidz2 (1460 plots) Secondary storage is 7 x 8TB machine running ZFS file system, raidz1 (397 plots) Pi5 (8GB) + TR004 with 4 x 16TB drives running bftrs file system, raid0 (560 plots) Pi5 (8GB) + TR004 with 4 x 16TB drives running btrfs file system, raid0 (slowly filling - 190 plots) I have a I5-11000 machine as the full node.   The good news is the  Pi5 + TR004 seems quite scalable, that is its fairly easy to add another one and once filled can switch it over to being a harvester. I'm seeing average latency on the Pi5 harvester of 400ms and max of 1.5 seconds, so it seems able to handle the random reads. While one could use a rack mount system with tons of drives I don't think the returns are enoug...

I finally filled up the primary storage server with 1450 plots. My next task is to replace the couple of Pi4's that have a pair of external hard disks with the Qnap TR-004 (4 bay) storage units. To that end I have ordered a couple of Pi5 (8GB). I will start by swapping one at a time so that I have as many plots online at a time. Each of the Pi4's have around 275 plots so I should be able to double that. My concern is the TR-004 connected to a Pi5 will be slow as the Pi only has USB 3.0 (5 Gbit) ports even though the TR-004 supports USB 3.2 gen 2 (10 Gbit) connections. I guess I will find out. If it works out one could scale the solution to have multiple Pi5/TR-004 running as a harvester only requiring two power points and 1 network connection each. Update January 2025 First Pi5/TR-004 filled up with plots and the second on its way. I got 560 plots on it with 4 x 16TB drives (95.3% used). I leave some space free. The slower USB 3 port doesn't seem to effect things much. I am...

As mentioned in my previous post I replaced all the drives in my primary storage server with larger ones. I am still trying to fill it up. Unfortunately the Bladebit plotter seems to lock up regularly and when I notice I have to shut down Chia, delete the temporary files off the SSD and the half-done plot that it was working on when it locked up. Top shows a zombie task and the plotter seems to have lost track of it. The 2nd WD 16TB drive that I returned under warranty has been rejected. They claimed it has a dent on the back of the drive, so likely shipping damage when I got it. I had 12 of these drives, now 11, so no point wasting any more money on the WD drives. I currently have two Qnap TR-004 units that hold 4 drives each, so 8 drives are enough. I need to fill the primary storage server before working on the TR-004's.

I bought 10 x 22TB Seagate HDD. Last week I bit the bullet and removed all the 16TB HDD from the primary storage server and put the 22TB drives in. I am now plotting away trying to fill it up. The Bladebit CPU plotter seems to lock up my full-node so I have to check on it once a day. The used 16TB drives are destined to go into a couple of QNAP TR004 units running in JBOD mode plugged into a Raspberry Pi5 running as a harvester. The TR004 can hold 4 drives. If this setup works out I will look at replacing the two Pi4 + external HDD's with Pi5 + TR004 units. I might have 3 of these as it seems a fairly inexpensive way of adding another harvester with 64TB of disk space. Speaking of the 16TB drives, I returned a 2nd WD 16TB HDD and picked up a replacement under warranty for the previous drive that I returned. It had been sitting at the computer shop for almost 3 months before I drove there to pickup the replacement. Thats 2 out of 14 WD drives that have failed so far. The lack of rel...

The TR-004 units arrived. The plan is to put four of the 16TB WD drives into each of the TR-004's. I am experimenting with running the TR-004's off a Pi5, but that may turn out to be too slow. I have a problem in that I have 10 x WD 16TB drives in my primary storage server and I had 4 spares. Two of those spares are now in one of TR-004's. One is off getting replaced under warranty and another spare seems to have died as well. I don't want to pull the drives from the primary storage server until I can generate more plots. Its currently too hot for that at the moment and you need to use an Nvidia GPU if you want compressed plots. I got 10 x Seagate 20TB drives to put into the primary storage server which will increase its capacity and free up those WD 16TB drives but the weather is holding me back.

Farming continues. All disks are full. Currently I have 200TiB of disk space occupied. I have ordered a pair of QNAP TR-004 "NAS expanders". These are 4 bay expansion boxes that connect via USB. They can do RAID or be a JBOD, controlled via dip switches on the back. My idea is to try the Raspberry Pi's running them but I might just plug them into the secondary storage server. I may have wasted my money on these, but I will have to see how they work out. The 5 bay ORICO expansion box I have tends to overheat despite having a fan in the back. I have stopped using it now. The drive bay doors have most of the holes covered and the PCB at the back where the drives plug into the SATA connectors is solid so it blocks all air flow that the fan produces. It also has an annoying habit of putting the drives to sleep after 12 minutes. Lastly they were rather slow but that is probably because the USB connection is 5 Gbit and has to be shared between 5 drives. QNAP have a TL-D400S (4 b...

Chia 2.1.1 released. All disks are full of uncompressed plots. I need to get a Nvidia GPU with 8GB (or more) vRAM and replot to get some compressed plots. Farm currently consists of: Primary Storage server (Ryzen 5600X, 128GB RAM, 10 x 16TB HDD in raidz2). Has 1060 plots. Secondary storage server (Core i3-8100U, 64GB RAM, 7 x 8TB HDD in raidz1). Has 397 plots. Pi storage server #1 (Pi4, 4GB RAM, 2 x 16TB HDD in btrfs single mode). Has 290 plots. Pi storage server #2 (Pi4, 4GB RAM, 2 x 16TB HDD in btrfs single mode). Has 290 plots. That gives 2037 plots taking 201.6TiB. I also have a plotting machine that is a full node and mainly for plotting. It doesn't have a discreet GPU at the moment. I could add some more USB connected HDD to the Pi's fairly easily but have to organise another power board for all the power adapters. I have some 10TB Seagate Expansion drives and a USB 3 hub available.

Shortly after Chia 2 was released they had a Chia 2.0.1 release which corrects an issue with computing invalid compressed plots due to passing incorrect parameters to the Bladebit plotter. I have filled all the disks at this point. I might get some more Seagate Expansion drives to plug into the Pi4's but I would rather create compressed plots if I could. Unfortunately I don't have a graphics card with 8GB of memory. The ones I have only have 6GB of video memory and I don't want to buy a GPU just for Chia plotting. Hopefully they we have an updated plotter out soon that I can use for compressed plots, in the mean time I am simply farming. My main storage server is currently run via an nfs share so its latency is worse than even the Raspberry Pi4's running a harvester. The full node runs a local harvester instance and accesses the plots over a network share. I used this to initially get the plots across but haven't yet switched it to run as a harvester.

Chia 2.0 was released on the 24th of August with support for compressed plots. Unfortunately you will need 416GB of free memory to compute them on CPU or you'll need a Nvidia GPU with CUDA support. Most desktop CPU's only support 128GB of memory so that means a server grade machine or a GPU. Hopefully a later release of the Bladebit plotter will support disk-based plots with compression. After upgrading my full node to 2.0 all my harvesters were unable to communicate so I had to upgrade  them to 2.0 as well. To get support for compressed plots you need to tell the harvester(s) the parallel_cpu count and the number of threads they can use. Unfortunately there is no configuration option to do this via the command line so you'll need to manually edit config.yaml if you don't have access to the GUI. Apart from the additional CPU usage expect an additional 500MB of memory use on each harvester. One of the new features in the new GUI is latency stats for each harvester. My P...

Time for an update. I have 1562 plots taking 154.6 TiB scattered across 4 machines. Primary storage server Ryzen 5600X with 128GB of ECC memory 10GbE network card 8 port SAS controller in HBA mode 10 x 16TB HDD Secondary storage server Intel i3-8100U with 64GB of ECC memory 10GbE network card 4 port SATA card 7 x 8TB HDD Pi storage servers x2 Raspberry Pi4 4GB 2 x 18TB HDD via its USB 3 ports I have filled the two Pi storage servers with 290 plots each. They have a pair of Seagate Expansion 18TB drives using the Pi's USB3 ports. I left around 500GB free space on them but after running a btrfs defrag they now have no free space. It should have released the space after each file. Regardless I am running Chia in harvester mode on each one. I could create a plot in 40 minutes but it takes another 20 minutes to get the file onto the Pi due to the 1 gigabit network port. I have filled out the secondary storage server. Its got 397 plots. At the moment I am filling out the primary storage...

I added a 2nd Raspberry Pi harvester into the mix. Its the same specs as the other one, that is a Pi4 4GB with a couple of 16TB Seagate Expansion drives plugged into the USB3 ports and running btrfs. With the drives in a "single" configuration I can get around 298 plots on them. Once I have it filled I will install the Chia software and run it as a harvester. To get the drives filled I transfer the completed plots onto it as a nfs drive. Once filled I stop running the nfs server on it and its just running the harvester. This allows one to have a low-power harvester. I could use the USB2 ports as well and get 4 drives. I'm not sure if/how it would effect the performance. I could use a USB hub as well but given the Pi has two USB3 ports this will do for the moment. I don't know how many drives a single Pi could support, but it works fine with just the two drives. I have over 1000 plots at the moment and the Chia GUI is telling me I am still 2 months off winning, so it l...

The main storage server I built which has 10 x WD Red 16TB drives is having disk issues. I had one spare drive I purchased at the time. After getting a bunch of disk errors recently I bought 3 more drives as spares. The first spare I put in refused to spin up. The second drive (brand new out of its anti-static bag) was giving lots of write errors. I'm now using the 3rd spare to replace one of the original disks that reported tons of errors. I can hear disks retrying but because they are so close to each other can't tell which one from the noise. The drives state on the packaging they have a 5 year warranty so I will be sending some back. Recently WD are in the media with people reporting that WD drives have been giving warnings for drives that are older than 3 years (26,280 hours) and need to be replaced.  My original drives have run hours in the 11,000 range. If a drive is working fine there is no reason to replace it based simply on its age. I will look at mixing the drive br...

The replacement PCIe to U2 adapter card arrived and I tried the original Intel P4510 in it. It wouldn't work so I put another P4510 on it and after a few teething troubles its been running since. The U2 drive is used for work space when plotting. Chia released a 1.8.0 client as an urgent security update so I upgraded to it. That broke the Mad Max plotter so I was only able to use the default plotter. They have since released 1.8.1 that fixed it. One nice thing is the 1.8.0 client now tells you when a new version is available. I have been furiously plotting as fast as my one and only plotter can go. I have filled my secondary storage server (7 x 8TB drives in a ZFS raidz config) and a Raspberry Pi (2 x 16TB external drives in a BTRFS single drive config). Both of these are now harvesters. I have brought up the main storage server (10 x 16TB drives in a ZFS raidz2 config giving 106TB available) and I am working on filling it up. There is no sign of compressed plots in the 1.8.1 clien...

While waiting for a replacement parts I thought I might cover the specs of the current farm. Full node Intel core i5-11400 with 128GB of DDR4-3200 memory. 10GbE network card. Samsung 970EVO 1TB as boot drive. U2 to PCIe adapter with Intel P4510 U2 SSD (1.8TB). Storage server 1 Ryzen 5600X with 128GB of DDR4-3200 ECC memory Nvidia GT710 graphics card. 8 port SAS controller. 10GbE network card. Samsung 970EVO 1TB as boot drive. 10 x WD Red Pro 16TB SATA HDD in a ZFS raidz2 array (two drive redundancy). Storage server 2 Intel core i3-8100 with 64GB of DDR4-2666 memory. 4 port SATA card. 10GbE network card. Samsung 970EVO 1TB as boot drive. 7 x WD Red Pro 8TB SATA HDD in a ZFS raidz array (one drive redundancy). Raspberry Pi storage server Raspberry Pi4 with 4GB of memory. Micro SD card as boot drive. 2 x Seagate Expansion 16TB HDD via USB3 ports in a btrfs single drive (no redundancy). I have ordered another pair of Seagate Expansion drives in order to create a 2nd Raspberry Pi storage s...