Disaster strikes

No sync
I went away for a week and powered everything off. When I got back I powered everything back up and the full node was having problems syncing. I left it for a week but it wasn't progressing at all. I rebooted it and so on and gave it another week but it refused to sync. At that point I decided to upgrade to the latest version. It didn't accept my 24 word mnemonic so I had no choice and had to start a new wallet. I started a new wallet but it refused to accept my existing plots as the keys didn't match, so I had to delete all of them. Kiss goodbye to the 22 XCH that I had.

ZFS errors
I have now started plotting and I am close to 200 or so plots. I was having some issues reported by my main storage server with ZFS indicating CRC and write errors. SMART stats didn't indicate the 16TB drives thought they had issues.

I suspect the SAS controller was overheating as it was very hot to touch. While it has a heatsink on the card there is no fan. I disconnected a couple of HDD from the SAS controller and plugged them into the motherboard SATA ports instead. I added a fan inside the case to blow air towards the SAS controller and 10GbE network card. In the mean time I have been writing the plots to the other storage server which has 8TB drives.

Compressed plots
In other news the Giga-Horse plotter has been able to create compressed plots. The official CHIA developers have acknowledged this and have said they will be adding official support for compressed plots in a future release.

There is a price to pay for having smaller plot files and that is the CPU overhead of making up for the missing data. The higher the compression level more CPU is needed. They have said that a Raspberry Pi harvester can handle compression level 1 (the lowest) but higher levels cause it issues. Space savings were estimated at 15% with the lowest compression level.

Harvesters
In other news I thought I would try setting the storage server I am currently trying to fill up (the one with 7 x 8TB drives) as a harvester. That is installing the CHIA software on it in harvester mode. That actually worked and had less network traffic as a result, when compared to just having the storage server as an NFS drives that the full-node can access.

One issue I noticed was the full-node would show as a Local harvester with the same number of plots as the remote harvester, that is they appear to be duplicated but it wasn't complaining about duplicated plots.

Hardware updates
I noticed the full-node would use all available memory when plotting, including all the swap. I updated the memory to its maximum (128GB) and it still did this. At one point it had 110GB as file cache and swap was fully used. I am now plotting with swap turned off. Under Linux you can turn swap on or off with the swapon (which is the default state) or the swapoff command.

Comments

Post a Comment

Popular posts from this blog

22nd of June

Image
Currently plotting away one plot at a time, using 4 threads (at least in phase 1). This is so as to not wear out the M.2 SSD so fast. I have a farmer or as Chia refers to it a "full node". I have a separate harvester machine (in a Meshify 2 case) that is basically a storage server. It has 6x WD Red 8TB drives at the moment with 60 plots taking around 5.9TiB. I have an M.2 to U.2 adapter on order, that I got on Amazon, but its not expected until next month. Must have sent it by sea.   I ordered one of these from one of my local PC shops in the hope it will turn up sooner. Its a PCIe card that you slot the U.2 drive onto it and then put the card into a PCIe x4 slot. Once this arrives I should be able to remove the M.2 SSD from the farmer and replace it with this and run multiple plots at a time. Its taking about 15 minutes to transfer a single plot from the farmer onto the harvester. I've setup a cron job to run every hour to pick up any completed plots from the local hard
Read more

More harvesting

I have 750 plots at the moment taking up 74TiB according to the the chia software. Storage server I purchased a pair of 32GB ECC memory modules to update the storage server. The machine wouldn't boot with them so I went back to using the pair of 16GB memory modules it had before. I purchased some more 16TB hard disks with a view to replacing the 8TB drives that are in the storage server. I am running ZFS and have 3 pools of drives: 5 x 16TB drives in Raidz (approx 32TB usable) 4 x 8TB drives in Raidz in an external enclosure (approx 24TB) 4 x 8TB drives (approx 24TB) The plan is to make a Raidz2 (two drive redundancy) of 10 x 16TB drives. This should yield approx 128TB usable space. The pool in the external enclosure can remain as is. I might upgrade it to larger drives at some point but don't have any plans at the moment. Due to the USB interface they are almost twice as slow as the drives inside the storage server, even though they are the same make and model 8TB drives. The
Read more

That took a while

I had to copy the plots off the 8TB disks so I could swap them out of the storage machine (aka harvester). That took 2 days, even with a 10GbE network connection between the two storage servers. I then removed the 8TB drives, put the 5 x 16TB drives in and copied the plots back. That took another 2 days with it doing 240MB/sec for most of that, which would mean the storage server simply can't write the data any quicker. After that I fired up the full node which told me 121 out of the 252 plots couldn't be opened. I ended up deleting them and I will have to plot fresh ones. Clearly they didn't survive being copied. It probably would have been quicker to upgrade the disks in the storage server and re-plot everything. At the moment I have put 4 of the 8TB drives into the external disk enclosure and I'm copying some of the remaining plots over to it. I ended up setting up a cron task to run every 10 minutes to write a small file to the disk enclosure in order to prevent the
Read more