End of 2023 update

Farming continues. All disks are full. Currently I have 200TiB of disk space occupied.

I have ordered a pair of QNAP TR-004 "NAS expanders". These are 4 bay expansion boxes that connect via USB. They can do RAID or be a JBOD, controlled via dip switches on the back. My idea is to try the Raspberry Pi's running them but I might just plug them into the secondary storage server. I may have wasted my money on these, but I will have to see how they work out.

The 5 bay ORICO expansion box I have tends to overheat despite having a fan in the back. I have stopped using it now. The drive bay doors have most of the holes covered and the PCB at the back where the drives plug into the SATA connectors is solid so it blocks all air flow that the fan produces. It also has an annoying habit of putting the drives to sleep after 12 minutes. Lastly they were rather slow but that is probably because the USB connection is 5 Gbit and has to be shared between 5 drives.

QNAP have a TL-D400S (4 bay) and TL-D800S (8 bay) unit that has a SATA controller on a PCIe card and connects via a SAS cable. This should alleviate the bandwidth issue between the box and the host computer. This won't work for me though as the Pi4 doesn't have PCIe slots and I don't have enough PCIe lanes on either of my x64 machines to plug the card into.

The Pi4's are working quite well as harvesters with external USB hard disks plugged in via the USB 3 ports on the Pi. According to the chia GUI they are responding with an average of 200ms and max of 674ms. Each one has a pair of 16TB Seagate Expansion drives giving 28.7TiB of usable space, which I have managed to cram 290 uncompressed plots on. I have them configured as a single disk using btrfs. As a low powered harvester these seem to work quite well.

Next years problem is how to get more storage online.


Comments

Post a Comment

Popular posts from this blog

That took a while

I had to copy the plots off the 8TB disks so I could swap them out of the storage machine (aka harvester). That took 2 days, even with a 10GbE network connection between the two storage servers. I then removed the 8TB drives, put the 5 x 16TB drives in and copied the plots back. That took another 2 days with it doing 240MB/sec for most of that, which would mean the storage server simply can't write the data any quicker. After that I fired up the full node which told me 121 out of the 252 plots couldn't be opened. I ended up deleting them and I will have to plot fresh ones. Clearly they didn't survive being copied. It probably would have been quicker to upgrade the disks in the storage server and re-plot everything. At the moment I have put 4 of the 8TB drives into the external disk enclosure and I'm copying some of the remaining plots over to it. I ended up setting up a cron task to run every 10 minutes to write a small file to the disk enclosure in order to prevent the
Read more

Storage server upgrade part 1

The first part of the upgrade was to the existing storage server. The existing server is in a Meshify 2 case and has 10 x 16TB HDD drives and a couple of SATA SSD's to use as log devices. The change was to bump the memory up to the maximum it supports (128GB). I tried around a year ago however the memory I got didn't work and the computer wouldn't even POST. This time using different memory modules it worked. While I was at it I disabled the log devices and removed the 2TB SATA SSD's. The log device is only used for writes. Seeing as the disks are full I don't write anything to the drives so there is no point in having a log device. I put a single 1TB SATA SSD in their place to use as a cache drive. I let the system run for a day so I could get some stats on how it was being used. It had a zero hit rate so I decided to disable it. On the subject of stats I seem to be getting around 75% hit rate on the memory cache (or ARC as its called under ZFS). A cache drive (or
Read more

22nd of June

Image
Currently plotting away one plot at a time, using 4 threads (at least in phase 1). This is so as to not wear out the M.2 SSD so fast. I have a farmer or as Chia refers to it a "full node". I have a separate harvester machine (in a Meshify 2 case) that is basically a storage server. It has 6x WD Red 8TB drives at the moment with 60 plots taking around 5.9TiB. I have an M.2 to U.2 adapter on order, that I got on Amazon, but its not expected until next month. Must have sent it by sea.   I ordered one of these from one of my local PC shops in the hope it will turn up sooner. Its a PCIe card that you slot the U.2 drive onto it and then put the card into a PCIe x4 slot. Once this arrives I should be able to remove the M.2 SSD from the farmer and replace it with this and run multiple plots at a time. Its taking about 15 minutes to transfer a single plot from the farmer onto the harvester. I've setup a cron job to run every hour to pick up any completed plots from the local hard
Read more