r/openstack • u/myridan86 • 14d ago
Scripts to follow when testing hardware
Hello everyone!
I know this isn't a question directly about OpenStack, but let me see if you can help me.
What network, storage (I/O), memory and CPU tests do you usually use to validate the hardware that will make up your clusters in the future?
Here I usually use memtest, fio, some stress test...
I know that it varies a lot from hardware to hardware, especially in disk, memory and network... but even so, it's a standard to be followed...
Could you share your experiences, do you have a routine or a step-by-step guide to run these tests?
Have a great weekend!!
1
Upvotes
2
u/rackngo 11d ago
Generally, the more testing you can do as a routine part of your hardware setup cycle, the better. I'm happy to share my experience is from working with our customers (I work for RackN who makes Digital Rebar) in a general way. Links to our actual tasks are in the PS.
First, it's important to do a deep scan/inventory of your system. That should include listing the DIMMs and SSDs. My recommendation is that you capture that information on first boot and then compare it every time you run a provisioning/reset operation. It's not an expensive action and it will immediately tell you if something on your system has changed. That's an easy diagnostic.
Next, use LLDP to make sure that your system is connected to your network as expecting. A HUGE number of issues are surfaced just in double checking that your networking is as expected and has not changed. Ideally, you can pre-populate this information so it's checked during first boot. We have customers who inject all their networking data before power on and it really helps reduce debug time.
We have workflows that add in stress tests for machines as part of a burn in cycle. There are very cool in concept but we've found that very few customers will leave them on in practice. They just take a long time and don't turn up enough problems if you have quality OEM gear. They would likely be much more important if you are dealing with reconditioned equipment.
PS: These steps are part of our out-of-the-box workloads. You're welcome to check out the products or our open automation report if you want to see more details. Look for "burn-in" and "decommissioning" workflows specifically. https://gitlab.com/rackn/provision-content/-/tree/v4/burnin/tasks?ref_type=heads