17th of June

I finally bit the bullet and decided to swap out the SAS 10k hard disk from my plotter and put a Sabrent Rocket 4 (1TB) NVMe SSD into it. While it has reasonable life expectancy I don't know how long the M.2 to U.2 adapter is going to take before it arrives. If I end up sacrificing a 1TB SSD then so be it. As expected the time to create a plot has dropped dramatically. Some timings running 1 plot at a time and allowing 4 threads:

SAS 10k            
Phase Secs Minutes
Phase Secs Minutes
1 14213 236.88
1 14258 237.63
2 5896 98.27
2 5858 97.63
3 14040 234.00
3 15917 265.28
4 665 11.08
4 743 12.38
Total 34814 580.23     36776 612.93

 

1TB SSD            
Phase Secs Minutes
Phase Secs Minutes
1 4863 81.05
1 5003 83.38
2 1799 29.98
2 1897 31.62
3 4789 79.82
3 4733 78.88
4 290 4.83
4 287 4.78
Copy 572 9.53
Copy 569 9.48
Total 12313 205.22     12489 208.15

I got the Meshify 2 ready and went to copy my 20+ plots off the WD Red hard disk that is in the plotter, but before I did that I put a dual 1GbE network card into both machines. With the ports bonded together it will in theory transfer around 200MB/second. Well it didn't. It seems Debian Bullseye doesn't handle bonding network ports. I even tried adding a plain 1GbE NIC into it and bonding that with the built-in NIC and it still wouldn't work. I ended up putting a 10GbE NIC into the plotter and trying that on its own. Before anyone asks, I do have 10GbE switches and cat 6a cabling.

At that point I tried to copy the 2TB worth of plots across and while it started off at 200MB/sec it  dropped to about 100MB/sec. I checked the Meshify and its only using 1 CPU core to keep up so its not the CPU that is the bottleneck. It wanted 9 hours to get them across. I'm guessing the disks in the Meshify can't keep up, or it could be WD Red in the plotter can't sustain the read speed. The drive specs say they can do 210MB/sec (the drive internal speed). To help this I have ordered a few SATA SSD's. One is a 1TB drive for the plotter to store completed plots and the other two are 500GB drives for the Meshify to use for write caching.

Speaking of disk write speed the plotter logs how long it takes to copy the final file across. You see that figure in my second set of times. Unfortunately I didn't get them for the SAS hard disk. With the WD Red that is in the plotter its giving times of approximately 9.5 minutes just to save the completed plot to disk. I think that works out around 190MB/sec, which is close to its rated speed. I can see now why people use a SSD to store completed plots before copying them off to hard disk.

I managed to order some more drive mounts for the Meshify. One of my computer shops listed them for pre-order. The catch is they aren't available until August.


Update: 19th of June
I copied one plot from the WD Red over to one of the M.2 SSDs in the same machine. That managed to sustain a 200MB/sec write speed. I then copied it from the SSD across to the Meshify. The speed started off at 240MB/sec and then dropped down to 100MB/sec. It would seem the Meshify is the bottleneck.

I am now tossing up getting a couple of SATA SSD's to put into the Meshify as fast disk. The Samsung 4TB SATA SSD are rated at 530MB/sec. I can then move the files from it to the slower hard disks.

Another alternative is to put multiple M.2 SSD's on a card (eg ASUS Hyper Gen4 or StarTech PEX8M2E2). The ASUS requires a PCIe x16 slot and can hold up to 4x M.2 SSD's. The StarTech requires a PCIe x8 slot and can hold up to 2x M.2 SSD's. The ASUS Hyper/StarTech solution would be faster but costs more and M.2 drives seem to top out at 2TB at the moment.

Links:

ASUS Hyper Gen4

StarTech PEX8M2E2

Comments

Post a Comment

Popular posts from this blog

22nd of June

Image
Currently plotting away one plot at a time, using 4 threads (at least in phase 1). This is so as to not wear out the M.2 SSD so fast. I have a farmer or as Chia refers to it a "full node". I have a separate harvester machine (in a Meshify 2 case) that is basically a storage server. It has 6x WD Red 8TB drives at the moment with 60 plots taking around 5.9TiB. I have an M.2 to U.2 adapter on order, that I got on Amazon, but its not expected until next month. Must have sent it by sea.   I ordered one of these from one of my local PC shops in the hope it will turn up sooner. Its a PCIe card that you slot the U.2 drive onto it and then put the card into a PCIe x4 slot. Once this arrives I should be able to remove the M.2 SSD from the farmer and replace it with this and run multiple plots at a time. Its taking about 15 minutes to transfer a single plot from the farmer onto the harvester. I've setup a cron job to run every hour to pick up any completed plots from the local hard ...
Read more

More plotting

I'm still producing plots as I have free disk space. While that is ongoing I have rounded up all my old hard disks. I have repurposed one of my old machines and have it running Darricks Boot and Nuke (DBAN) to wipe the hard disks. I started with the 320GB drives and they're taking around 6 hours to wipe. They were made in 2007 and 2008. I'll be dropping these drives off at an ewaste (electronic waste) facility so they can be recycled. I'll be getting rid of all the 3.5" drives that are less than 4TB. I am still waiting on the drive brackets for the storage server. They were supposed to be in this week but have been delayed again. I saw the QNAP TL-R1200C-RP 12 bay 2U rack mount JBOD for sale. What is unusual about this is most rack mount JBOD's use SAS which means you need a HBA card with external SAS ports in your PC, but this one uses USB 3.2 to connect. The new network switch was supposed to be available "mid-August". The computer shop don't kn...
Read more

Storage server upgrade part 1

The first part of the upgrade was to the existing storage server. The existing server is in a Meshify 2 case and has 10 x 16TB HDD drives and a couple of SATA SSD's to use as log devices. The change was to bump the memory up to the maximum it supports (128GB). I tried around a year ago however the memory I got didn't work and the computer wouldn't even POST. This time using different memory modules it worked. While I was at it I disabled the log devices and removed the 2TB SATA SSD's. The log device is only used for writes. Seeing as the disks are full I don't write anything to the drives so there is no point in having a log device. I put a single 1TB SATA SSD in their place to use as a cache drive. I let the system run for a day so I could get some stats on how it was being used. It had a zero hit rate so I decided to disable it. On the subject of stats I seem to be getting around 75% hit rate on the memory cache (or ARC as its called under ZFS). A cache drive (or ...
Read more