Adventures in Chia

I had to copy the plots off the 8TB disks so I could swap them out of the storage machine (aka harvester). That took 2 days, even with a 10GbE network connection between the two storage servers. I then removed the 8TB drives, put the 5 x 16TB drives in and copied the plots back. That took another 2 days with it doing 240MB/sec for most of that, which would mean the storage server simply can't write the data any quicker. After that I fired up the full node which told me 121 out of the 252 plots couldn't be opened. I ended up deleting them and I will have to plot fresh ones. Clearly they didn't survive being copied. It probably would have been quicker to upgrade the disks in the storage server and re-plot everything. At the moment I have put 4 of the 8TB drives into the external disk enclosure and I'm copying some of the remaining plots over to it. I ended up setting up a cron task to run every 10 minutes to write a small file to the disk enclosure in order to prevent the...

After filling up the disk space on the home made storage server there have been a couple of developments. External disk enclosure The external disk enclosure arrived. I have put the two spare 8TB HDD into the enclosure and spent a bit of time testing it. It presents the disks as individual drives, which is what I wanted. I am currently running them as a ZFS mirror. I had an issue where the drives spin down if you don't keep them busy. Trying to set the spin down time doesn't work. Fortunately Chia keeps checking the plots so that solves the issue as its checking every 10 seconds or so. I am now moving plots from the internal drives in the storage machine to the external enclosure. Its getting around 120MB/sec. While that is going I have started creating more plots. More drives on way I managed to get a good deal on new WD Red Pro 16TB drives, so I bought 5 of them. This changes my original plan. Now the plan is to swap out the 8TB drives that are in the storage server and put t...

I've had the plotter churning out tasks fairly consistently since the last post. I have now got the storage machine (or farmer) at 90% of its 29TB disk space. That is 254 plots worth. I'm running ZFS on the farmer so they don't recommend going over 90% for it to work.   I have ordered one of these external hard disk boxes that holds 4 drives. It doesn't support RAID functionality. It says it supports up to 16TB drives so that suggests to me that rather than presenting itself as a 4 drives like a JBOD would, it will appear as one logical disk. I will find out when it arrives next week. It connects to the host computer by USB 3.0. I was hoping for a JBOD type of device but the few (4 or 8 disk) models I found that claim to be JBOD are sold out. The idea for the external hard disk box is to allow me to copy the existing pool to the external disk box. I can then add the remaining 8TB drives into the farmer, destroy the existing storage pool of 6 drives and recreate it with ...

After sorting out the network issues things are now humming along. I need to get another 10GbE network switch so I can connect some of my other devices but at the moment the farmer and plotter are on 10GbE so moving plots only takes 3 minutes. The plotter is churning out plots and I have used over 50% of the disk space in the farmer. Current stats: 175 plots 17.322 TiB space used Estimated time to win is "a year" (based upon current Chia total netspace). I have two spare 8TB disks that I could put into the farmer but I am waiting for additional drive brackets which aren't due until August. The Meshify 2 case only had 6 drive brackets and I have used all of them. After getting the extra drives into the machine I need to rebuild the ZFS disk pool so I would have to move the plots somewhere else, delete the pool, create a new pool of 8 drives and copy the plots back.  I have decided not to spend any money on buying hard disks. All the disks I have used so far I already had. ...

I swapped the power supplies between the plotter and harvester. The harvester needed the extra SATA power connectors that the modular power supply had. I also installed a couple of 2TB Samsung SSD's in the harvester. The Meshify 2 case has a couple of 2.5" SSD brackets on the back of the motherboard tray. I then created a ZFS pool of the two SSD's with no redundancy, so it should have fast writes. At that point I did some testing of how long it takes to get a plot from the farmer to the harvester across the network. Test 1 10GbE network card in farmer. Dual 1GbE network card in harvester. Copying to SSD on the harvester with no redundancy (to rule ZFS out). The speed going from the farmer to the plotter was taking the same as before. Speed starts off at 240 MB/sec and drops to around 100MB/sec. Test 2 10GbE network cards in both farmer and harvester. Copying to SSD on the harvester again. The speed going from the farmer to the plotter starts around 500MB/sec and drops back...

Currently plotting away one plot at a time, using 4 threads (at least in phase 1). This is so as to not wear out the M.2 SSD so fast. I have a farmer or as Chia refers to it a "full node". I have a separate harvester machine (in a Meshify 2 case) that is basically a storage server. It has 6x WD Red 8TB drives at the moment with 60 plots taking around 5.9TiB. I have an M.2 to U.2 adapter on order, that I got on Amazon, but its not expected until next month. Must have sent it by sea.   I ordered one of these from one of my local PC shops in the hope it will turn up sooner. Its a PCIe card that you slot the U.2 drive onto it and then put the card into a PCIe x4 slot. Once this arrives I should be able to remove the M.2 SSD from the farmer and replace it with this and run multiple plots at a time. Its taking about 15 minutes to transfer a single plot from the farmer onto the harvester. I've setup a cron job to run every hour to pick up any completed plots from the local hard ...

I finally bit the bullet and decided to swap out the SAS 10k hard disk from my plotter and put a Sabrent Rocket 4 (1TB) NVMe SSD into it. While it has reasonable life expectancy I don't know how long the M.2 to U.2 adapter is going to take before it arrives. If I end up sacrificing a 1TB SSD then so be it. As expected the time to create a plot has dropped dramatically. Some timings running 1 plot at a time and allowing 4 threads: SAS 10k             Phase Secs Minutes Phase Secs Minutes 1 14213 236.88 1 14258 237.63 2 5896 98.27 2 5858 97.63 3 14040 234.00 3 15917 265.28 4 665 11.08 4 743 12.38 Total 34814 580.23     36776 612.93   1TB SSD             Phase Secs Minutes Ph...