Computers

Building the new storage server is a major pain in the neck. I had so many issues, problems, regrets I never had before. Just to buy the 6TB disks caused so much trouble even a young Jedi had to feel it from across the Galaxy.

So first I bought a 6TB used WD Gold disk for a very good price. It was used, so I bought only one, planned to buy more if the quality is ok. And it was, it was cheap, true, it arrived from the US real slow, but it worked perfectly. No issues found by the smartctl either. Except... except the seller immediately disabled my country, they are not selling to me any more. Maybe they realized the delivery to the EU is too expensive...

Then I bought one more, used, from the US. 6TB, used... Well, it was broken. It reported a lot of recoverable errors, the server immediately put a yellow exclamation mark beside the disk. That is not acceptable right? That's a disaster.

So I sent it back, I got my money back after 4 or 6 weeks. Well, it is still not on my credit card, but eBay says that's just the delay of the bank. Lot of work, weeks of waiting and at the end nothing to show for it.

Right... then I decided I will not buy from "over the ocean", I will not buy second hand, I will buy new from my country, from Hungary, from an "official" site. It was a bit expensive, but I bought 2 new 6TB to begin with. Toshiba, that was cheaper than the others, should work right?

Yes, they arrived in 2 days, they work, they nice, everything. Built a little 3x6TB RAID 5 virtual disk for test, works nice. Time to buy some more of the same... except the company removed these disks from their page. Maybe they sold the last 2 to me? Or they just read my blog and wanted to pull my leg? To see how far I will go? Guys, is this a test? Someone is watching me, right? Right?

Right?

Well, this is a follow-up. Some RAM modules arrived, I installed them and now checked the statistics about NUMA. And remember, I know very little about the subject, so take this with a grain of salt...

So I got the RAM from China to my house in 8 days, which is great. This is the second time I bought RAM modules from this guy (in fact this particular product was bought only by me), and I like the speed. I like the price too. And the RAM I got works perfectly.

There is something satisfying about buying RAM and installing them one small batch at a time... Anyway, I added the screenshot of the eBay product if someone is interested.

So the NUMA stat has indeed changed. Here is how it looked like before:

And here is how it looks like now. Better... much better...

It was some 15 years ago when I first heard about non uniform memory access. I spent the last 15 years not worrying about NUMA. Never checked, never worried, never cared.

So today I run some commands on some servers of mine to see what's going on. I wouldn't say I worry now, but there is certainly a new thing to think about.

So the first screenshot was made on a server that has 8 processors with memory integrated on all the processor cards. All the memory banks are populated. The stats are pretty straightforward, everything seems to be ok. Sun Microsystems, I like it...

The second has four processor slots, but only two of those populated and four, fully populated memory expansion cards. Well, it is showing only one memory area, right? I am not sure why... maybe the HW is built like that or the kernel doesn't handle it properly?

The third shows four processors, two populated memory expansion boards with RAM in them. There are also two unpopulated memory expansion boards. So it shows everything OK, but I know I should buy more RAM, right? The owner's manual also says it is better to have memory for every processor. Well, I bought some already (that's why there are two NUMA arrays already), the rest is on the way between China and the EU.

Oh, my, I hope when the ship arrives there will be two extra columns on this screen...

Just a quick little note about an experiment I made. You may find some deeper meaning of this, although it is just a reflection on a well known fact.

So I created a code that does evolution in a multi-threaded environment. 16 threads processing 16 independent populations of organisms. As you may expect some population did better than the other, some became better and better while some other evolved slower. Some were even stuck, produced very little progress. Evolutionary dead-end and whatnot...

Then I made a very little, but important change. Added a code that chooses an organism, a successful one, and moves it from its place to a random population from time to time. Just like that, sometimes a single successful organism moved to a new location, into a new population. Guess what, here is the graph, the whole species got significantly better, more evolved, getting better.

Exactly as I expected, of course, but it is good to see that the code works...

So, here are teenager servers, what can we expect of them? Are they any good? 

If you happen to buy some used servers you probably end up buying hardware that is about 10-15 years old. Anything older is usually seriously deprecated. When servers reach the ripe old age of 15-20 they are usually like old-timers in the car world. They are fun, they are nice, they are sure a great hobby, but they are not running that fast and there is a lot of maintenance involved.

So what can we expect from a 10-15 years old servers? Are these provide any value at all? Can they be seriously used, run for a real job? Well, let's see. Let's check on the disks first. Let's see what can we expect of the teenager servers from the storage point of view.

First, it is important to understand that servers are usually strong in the storage department, relatively stronger than desktops and laptops. Yes, compared to the PC the CPU in an enterprise server is more serious, the memory is better, but the difference between a server storage and a desktop storage is usually bigger. Yes, 10 years are 10 years, but still, an enterprise server is an enterprise server.

These servers, practically all of them have SAS drives, not SATA. SAS is similar to SATA, but it is is better. SATA drives are running on 7200 RPM or 5400 RPM while SAS drives are usually 10K or 15K. They might be spinning 3 times faster than the drive you used to. They also seem to have a better quality. I might be wrong, it is just a feeling, but they feel to be better. So the second thing is that server SAS drives are faster and more reliable.

The servers have SAS RAID cards, they are usually compatible with SATA too, you simple plug the SATA drive in and it works. You usually can't mix the SAS and SATA drives in one virtual disks, but you can mix them in one server, no problem there. SATA drives might be even cheaper to buy, especially the big ones so we might want to be open-minded about SATA too.

The third thing is the RAID card. All the servers have proper RAID cards and that is a big plus on their side. Handling a storage with a proper RAID card is a huge difference not just for the fault tolerance, but for the reliability and the speed too. When 6, 8 or even more disks are running together they are indeed more than the sum of the parts. Yes, there is the software RAID, but that's a joke, and yes, there are RAID cards in some other computers too, but the proper server RAID card is included in the enterprise servers and that's great news!

The fourth thing to remember is the hot-swappable disks. Replacing a disk while the server is running is not that important in a home IT system, sure, but hot-swap also means easy access. To install or replace a disk in a desktop might take half an hour, even more. You need to disconnect, mine the box from under the desktop, open it up, cables, screws... then you realize you don't have the cable, not enough power connectors... it is a very annoying work that you may have to do from time to time. Installing or replacing a disk may be as fast as 5 seconds when you are dealing with a server.

But really, there is a yellow light blinking, you notice it, you open the box of replacement disks, pick one out, left hand removes the old disk, right hand installs the new disk, wait for a few seconds, it is blinking. The RAID card already started to rebuild the data, let's have a coffee. Takes more time to write it down than doing it, you just need to keep some extra disks near your server.

And let's not forget that sometimes you want to move disks around even if the disks are not failing. You may want to install some extra storage check a disk that you bought, maybe archive something or whatever. Hot-swap disks are the best since sliced bread for sure!

And of course the money. If you don't need huge storage the price of the second hand 3,5" SAS disks is pretty low. You can buy 72, 146 GByte disks for next to nothing, especially if you buy it bulk. Buy them with caddies, buy a bunch of them, they are very reliable, you won't regret it. Yes, huge disks like 2TB, 6TB, those might fail, you need to be careful buying those. Also 2.5" disks might be a high risk, better not buying them second hand at all. But the smaller capacity 3.5" disks are great. You put 3-6 72-146.300GByte  disks into your server and they will not fail you... well, especially when you use a redundant RAID version. Disks without redundancy is like playing Russian roulette with a glock, you loose the first time you have a chance.

And even if one goes wrong what's the harm? For a price of a cheeseburger and a large coke you put in the replacement in under 15 seconds.

And of course I also have some bad experience, some things that I learned for a price... that I had to pay.

First, I do not buy second hand 2.5" disks. Multiple sources sent me many faulty drives of these, a lot of wasted time and money. I could try to get the money back, sometimes I do, but not always. Sometimes the disk fail after a few months, sometimes you don't even test them for weeks, sometimes I am just too lazy to send them back. Anyway, 2,5" disks have smaller area holding the same amount of data or something else... dunno, don't care. I am not buying them any more.

The other thing is that I never buy SSD second hand. Not ever. If that SSD was removed from a computer it was removed for a reason. Because it is broken, that's why.

And even when you buy a new disk you still might buy a used one, even one that was used for many-many years. Even if it looks like having the original packaging. Even if the SMART info says it is new. Sellers sometimes say the disk is refurbished, which is impossible, of course. Sometimes they say they checked it. Well, they don't. They delete the data from it, yes, that's true.

And by the way, this whole article sounds like a set of advises, here is one that really is and advice, really is important. Here is the best advice I heard in my life: