Computers

Just a quickie, about an upgrade I made yesterday. Well, it was not an upgrade like we do all the time, it was an upgrade involving some screwdrivers and some hotline maintenance. And by hot I mean 250V AC live wire, not the kind of hot you might enjoy. :)

I measure the power consumption of the servers at multiple places using metered PDUs and node-red, but I also have some offline power meters. I bought some cheap power meters from eBay years ago, they measure the voltage and the current. Well those are really what I needed, those were just what I had at hand.

Now I replaced the meters by a similarly cheap eBay meters that actually measure power consumption. I need to know how much I will have to pay at the end of the year for powering my servers, not measuring the wash machine, the water heater, the fridge and so on and so forth. The three boxes on the photos are holding the three phase, I replaced the meters one by one, so I did not even had to power down my server farm. All the servers have at least two power supplies, all of them are getting two different phases, so it is all redundant.

The new meters measure the voltage, the current, the time, the cos φ, the cosine of the angle between the voltage and the current. Well this last one is for the big boys who already finished high school and learned the delicate details of the alternating current, but if it is unknown for you don't be intimidated, it is suffice to say this is something worth measuring. Out of these the meters calculate and display the running sum for the energy consumption in kWh. There is a little push button to switch the display between the Watts and kilowatt-hours. Long press resets... great.

Now I am ready to do some experiments, to extrapolate how much money I actually spend... but wait, how accurate is this thing? And why is my cos φ so low when the load is low?

Building the new storage server is a major pain in the neck. I had so many issues, problems, regrets I never had before. Just to buy the 6TB disks caused so much trouble even a young Jedi had to feel it from across the Galaxy.

So first I bought a 6TB used WD Gold disk for a very good price. It was used, so I bought only one, planned to buy more if the quality is ok. And it was, it was cheap, true, it arrived from the US real slow, but it worked perfectly. No issues found by the smartctl either. Except... except the seller immediately disabled my country, they are not selling to me any more. Maybe they realized the delivery to the EU is too expensive...

Then I bought one more, used, from the US. 6TB, used... Well, it was broken. It reported a lot of recoverable errors, the server immediately put a yellow exclamation mark beside the disk. That is not acceptable right? That's a disaster.

So I sent it back, I got my money back after 4 or 6 weeks. Well, it is still not on my credit card, but eBay says that's just the delay of the bank. Lot of work, weeks of waiting and at the end nothing to show for it.

Right... then I decided I will not buy from "over the ocean", I will not buy second hand, I will buy new from my country, from Hungary, from an "official" site. It was a bit expensive, but I bought 2 new 6TB to begin with. Toshiba, that was cheaper than the others, should work right?

Yes, they arrived in 2 days, they work, they nice, everything. Built a little 3x6TB RAID 5 virtual disk for test, works nice. Time to buy some more of the same... except the company removed these disks from their page. Maybe they sold the last 2 to me? Or they just read my blog and wanted to pull my leg? To see how far I will go? Guys, is this a test? Someone is watching me, right? Right?

Right?

Well, this is a follow-up. Some RAM modules arrived, I installed them and now checked the statistics about NUMA. And remember, I know very little about the subject, so take this with a grain of salt...

So I got the RAM from China to my house in 8 days, which is great. This is the second time I bought RAM modules from this guy (in fact this particular product was bought only by me), and I like the speed. I like the price too. And the RAM I got works perfectly.

There is something satisfying about buying RAM and installing them one small batch at a time... Anyway, I added the screenshot of the eBay product if someone is interested.

So the NUMA stat has indeed changed. Here is how it looked like before:

And here is how it looks like now. Better... much better...

It was some 15 years ago when I first heard about non uniform memory access. I spent the last 15 years not worrying about NUMA. Never checked, never worried, never cared.

So today I run some commands on some servers of mine to see what's going on. I wouldn't say I worry now, but there is certainly a new thing to think about.

So the first screenshot was made on a server that has 8 processors with memory integrated on all the processor cards. All the memory banks are populated. The stats are pretty straightforward, everything seems to be ok. Sun Microsystems, I like it...

The second has four processor slots, but only two of those populated and four, fully populated memory expansion cards. Well, it is showing only one memory area, right? I am not sure why... maybe the HW is built like that or the kernel doesn't handle it properly?

The third shows four processors, two populated memory expansion boards with RAM in them. There are also two unpopulated memory expansion boards. So it shows everything OK, but I know I should buy more RAM, right? The owner's manual also says it is better to have memory for every processor. Well, I bought some already (that's why there are two NUMA arrays already), the rest is on the way between China and the EU.

Oh, my, I hope when the ship arrives there will be two extra columns on this screen...

Just a quick little note about an experiment I made. You may find some deeper meaning of this, although it is just a reflection on a well known fact.

So I created a code that does evolution in a multi-threaded environment. 16 threads processing 16 independent populations of organisms. As you may expect some population did better than the other, some became better and better while some other evolved slower. Some were even stuck, produced very little progress. Evolutionary dead-end and whatnot...

Then I made a very little, but important change. Added a code that chooses an organism, a successful one, and moves it from its place to a random population from time to time. Just like that, sometimes a single successful organism moved to a new location, into a new population. Guess what, here is the graph, the whole species got significantly better, more evolved, getting better.

Exactly as I expected, of course, but it is good to see that the code works...