Only 100MB/sec

I swapped the power supplies between the plotter and harvester. The harvester needed the extra SATA power connectors that the modular power supply had. I also installed a couple of 2TB Samsung SSD's in the harvester. The Meshify 2 case has a couple of 2.5" SSD brackets on the back of the motherboard tray. I then created a ZFS pool of the two SSD's with no redundancy, so it should have fast writes. At that point I did some testing of how long it takes to get a plot from the farmer to the harvester across the network.

Test 1
10GbE network card in farmer. Dual 1GbE network card in harvester. Copying to SSD on the harvester with no redundancy (to rule ZFS out). The speed going from the farmer to the plotter was taking the same as before. Speed starts off at 240 MB/sec and drops to around 100MB/sec.

Test 2
10GbE network cards in both farmer and harvester. Copying to SSD on the harvester again. The speed going from the farmer to the plotter starts around 500MB/sec and drops back to 100MB/sec as before. Clearly its not the network.

At this point I suspect its NFS and start some internet searching and other people have also had this issue well before Chia existed. There are some settings one can adjust which I need to try next. I'll be doing them later. I have some more plots being generated and have to wait until they have finished.


Update: 28th of June
After a lot of testing and swapping of network cards in the plotter its my fault. I had the plotter plugged into a 10GbE port on one of the 8+2 switches (8 gigabit plus 2 multi-gig ports). However its uplink port was plugged into the router which is only capable of gigabit. After plugging the uplink into the 10GbE switch its now able to go at high speed.

I have a 2.5GbE card in the farmer at the moment. I will swap the 10GbE card back in once it finishes the in progress plots. At gigabit speed it takes 15 minutes to move a plot onto the harvester. With 2.5GbE it takes around 7 and a half minutes.

While I was at it I also noticed the network cards had old firmware. I downloaded the firmware updater but it only runs under windows. I had to set up a machine with Windows just to flash the firmware. I've now flashed two of the cards. The third one I only got last week so its up to date.

Comments

Post a Comment

Popular posts from this blog

22nd of June

Image
Currently plotting away one plot at a time, using 4 threads (at least in phase 1). This is so as to not wear out the M.2 SSD so fast. I have a farmer or as Chia refers to it a "full node". I have a separate harvester machine (in a Meshify 2 case) that is basically a storage server. It has 6x WD Red 8TB drives at the moment with 60 plots taking around 5.9TiB. I have an M.2 to U.2 adapter on order, that I got on Amazon, but its not expected until next month. Must have sent it by sea.   I ordered one of these from one of my local PC shops in the hope it will turn up sooner. Its a PCIe card that you slot the U.2 drive onto it and then put the card into a PCIe x4 slot. Once this arrives I should be able to remove the M.2 SSD from the farmer and replace it with this and run multiple plots at a time. Its taking about 15 minutes to transfer a single plot from the farmer onto the harvester. I've setup a cron job to run every hour to pick up any completed plots from the local hard ...
Read more

More plotting

I'm still producing plots as I have free disk space. While that is ongoing I have rounded up all my old hard disks. I have repurposed one of my old machines and have it running Darricks Boot and Nuke (DBAN) to wipe the hard disks. I started with the 320GB drives and they're taking around 6 hours to wipe. They were made in 2007 and 2008. I'll be dropping these drives off at an ewaste (electronic waste) facility so they can be recycled. I'll be getting rid of all the 3.5" drives that are less than 4TB. I am still waiting on the drive brackets for the storage server. They were supposed to be in this week but have been delayed again. I saw the QNAP TL-R1200C-RP 12 bay 2U rack mount JBOD for sale. What is unusual about this is most rack mount JBOD's use SAS which means you need a HBA card with external SAS ports in your PC, but this one uses USB 3.2 to connect. The new network switch was supposed to be available "mid-August". The computer shop don't kn...
Read more

Storage server upgrade part 1

The first part of the upgrade was to the existing storage server. The existing server is in a Meshify 2 case and has 10 x 16TB HDD drives and a couple of SATA SSD's to use as log devices. The change was to bump the memory up to the maximum it supports (128GB). I tried around a year ago however the memory I got didn't work and the computer wouldn't even POST. This time using different memory modules it worked. While I was at it I disabled the log devices and removed the 2TB SATA SSD's. The log device is only used for writes. Seeing as the disks are full I don't write anything to the drives so there is no point in having a log device. I put a single 1TB SATA SSD in their place to use as a cache drive. I let the system run for a day so I could get some stats on how it was being used. It had a zero hit rate so I decided to disable it. On the subject of stats I seem to be getting around 75% hit rate on the memory cache (or ARC as its called under ZFS). A cache drive (or ...
Read more