Update 13th of Aug 2023

Time for an update.

I have 1562 plots taking 154.6 TiB scattered across 4 machines.

Primary storage server
Ryzen 5600X with 128GB of ECC memory
10GbE network card
8 port SAS controller in HBA mode
10 x 16TB HDD

Secondary storage server
Intel i3-8100U with 64GB of ECC memory
10GbE network card
4 port SATA card
7 x 8TB HDD

Pi storage servers x2
Raspberry Pi4 4GB
2 x 18TB HDD via its USB 3 ports

I have filled the two Pi storage servers with 290 plots each. They have a pair of Seagate Expansion 18TB drives using the Pi's USB3 ports. I left around 500GB free space on them but after running a btrfs defrag they now have no free space. It should have released the space after each file. Regardless I am running Chia in harvester mode on each one. I could create a plot in 40 minutes but it takes another 20 minutes to get the file onto the Pi due to the 1 gigabit network port.

I have filled out the secondary storage server. Its got 397 plots.

At the moment I am filling out the primary storage server. It has 10 drives in a raidz2 array (so 2 drives for redundancy). Just as well as it has been giving disk errors. I've replaced one of the drives in it and expect I will need to replace another one that frequently gives errors. It will take some time to fill this machine as it has 110TB of usable space. I've got 50TB on it so far.

Once filled I will have to look at my options for getting more disk space. I think the simplest will be to add two more external drives to the Pi's using the USB 2 ports.


Comments

Post a Comment

Popular posts from this blog

That took a while

I had to copy the plots off the 8TB disks so I could swap them out of the storage machine (aka harvester). That took 2 days, even with a 10GbE network connection between the two storage servers. I then removed the 8TB drives, put the 5 x 16TB drives in and copied the plots back. That took another 2 days with it doing 240MB/sec for most of that, which would mean the storage server simply can't write the data any quicker. After that I fired up the full node which told me 121 out of the 252 plots couldn't be opened. I ended up deleting them and I will have to plot fresh ones. Clearly they didn't survive being copied. It probably would have been quicker to upgrade the disks in the storage server and re-plot everything. At the moment I have put 4 of the 8TB drives into the external disk enclosure and I'm copying some of the remaining plots over to it. I ended up setting up a cron task to run every 10 minutes to write a small file to the disk enclosure in order to prevent the
Read more

Storage server upgrade part 1

The first part of the upgrade was to the existing storage server. The existing server is in a Meshify 2 case and has 10 x 16TB HDD drives and a couple of SATA SSD's to use as log devices. The change was to bump the memory up to the maximum it supports (128GB). I tried around a year ago however the memory I got didn't work and the computer wouldn't even POST. This time using different memory modules it worked. While I was at it I disabled the log devices and removed the 2TB SATA SSD's. The log device is only used for writes. Seeing as the disks are full I don't write anything to the drives so there is no point in having a log device. I put a single 1TB SATA SSD in their place to use as a cache drive. I let the system run for a day so I could get some stats on how it was being used. It had a zero hit rate so I decided to disable it. On the subject of stats I seem to be getting around 75% hit rate on the memory cache (or ARC as its called under ZFS). A cache drive (or
Read more

22nd of June

Image
Currently plotting away one plot at a time, using 4 threads (at least in phase 1). This is so as to not wear out the M.2 SSD so fast. I have a farmer or as Chia refers to it a "full node". I have a separate harvester machine (in a Meshify 2 case) that is basically a storage server. It has 6x WD Red 8TB drives at the moment with 60 plots taking around 5.9TiB. I have an M.2 to U.2 adapter on order, that I got on Amazon, but its not expected until next month. Must have sent it by sea.   I ordered one of these from one of my local PC shops in the hope it will turn up sooner. Its a PCIe card that you slot the U.2 drive onto it and then put the card into a PCIe x4 slot. Once this arrives I should be able to remove the M.2 SSD from the farmer and replace it with this and run multiple plots at a time. Its taking about 15 minutes to transfer a single plot from the farmer onto the harvester. I've setup a cron job to run every hour to pick up any completed plots from the local hard
Read more