IPMI (Intelligent Platform Management Interface) Protocol Explained

Today we have millions of servers globally, to be precise, 44 million. Google alone has around 2.5 million servers. We can’t even imagine how much computing power the Internet development enabled. However, while servers are getting more sophisticated with their performance, we never paid much attention to its remote accessibility. Let me clarify that, but before let me explain some definitions. Every decent server machine has remote access, which is called IPMI (Intelligent Platform Management Interface). A while ago (it is still being used), data centers had options for KVM over IP (that translates to Keyboard, Video, Mouse over IP address). Since most of the nowadays servers have IPMI integrated, KVM over IP became less needed.

Coming back to my concern about remote accessibility, many vendors still use outdated Java-based IPMI interfaces. Watching global market news, we see raising security concerns due to unsecured IPMI access. The problem will become much greater if most infrastructure providers do not take the necessary measures to make IPMI a secure environment.

Let’s roll back a bit and see how IPMI works. The concept of the IPMI is awesome, and we must consider that it helps to save a lot of time, not to mention the savings of remote-hands charges. It is so convenient that even a sysadmin can be located in a different part of the world and connect remotely to the server, I mean, it looks obvious today, but back in the days, it was something unreal.

The IPMI concept is pretty straightforward. Understanding its principles server has a separate network port that sits on the separate network layer and is connected directly to the BIOS. If we dig into the details, that would be the following:

BMC stands for a baseboard management controller that is connected to the other controllers in the server mainboard. In other words, it mirrors the BIOS and devices/controllers that are available on the server.

The new servers usually have a basic network interface controller (NIC), which has minimal port speed. A basic text console that sufficient to do the server’s basic setup or configuration (BIOS level). That is a cost-effective integrated option most of the server vendors provide. IPMI network management cards come as a separate module and are connected to the motherboard.

However, the most vulnerable part is the network, and in many cases, infrastructure service providers assign the public IP to it. This is where a potential intruder can scan the IPMI port and see if it is open or not (having IPMI on public IP address). Due to the mentioned fact that IPMI clients are very buggy and laggy, it is a challenge to secure the environment and, at the same time, make it stable. How dangerous the IPMI is if it is not isolated from the public network? Basically, it is how difficult your password is, but having in mind how advanced the brute-force botnets are, would help you for a short period of time. ComputerWorld published a great article back in 2013, based on the University of Michigan’s research on how unprotected and vulnerable the servers are, purely having unsecured IPMI. The problem persists for as long as the IPMI was introduced!

Has anything changed after so much time? Not really! Well, you can have a solution, which is a firewall, and try to protect the whole IPMI environment by firewalling it. However, this is not a quick fix, and the hardest part is the automation and actual management. We will get back to it shortly.

There is also a downside if the network will go down completely within the IP transit provider server will be unreachable by any time of connectivity, including IPMI. This is where old good KVM over IP can be handy. However, some of the data centers have it, and it is definitely an advantage, and it should be on your checklist choosing the data center.

Either way, if the infrastructure provider has assigned the public IP address to the IPMI interface, you should remove it or ask the infrastructure provider to do it as soon as possible due to the mentioned security issues. You definitely want to have an isolated IPMI environment.

The fun fact is that some of the server vendors rebrand IPMI that comes with their servers. Dell, for instance, calls their IPMI controller iDRAC (Integrated Dell Remote Access Controller). HP calls their IPMI – iLO (Integrated Lights Out). At the same time, Lenovo calls it IMM (Integrated Management Module) the same as IBM, which is obvious reasons.

However, the hardware makers release updates for their IPMI from time to time, but most updates are focused on stability, not much on security. I don’t blame them. Since the user (infrastructure provider) has the responsibility to take care of it, every server should be secured and properly managed.

Addressing security concerns, the first thing was to isolate the IPMI. After some hours of brainstorming, we came up with the idea of how to secure the IPMI. However, the grand idea was to have access based on time, and then the cards shuffle again. Let me explain.

We came up with a plan that had four phases. Each phase marks a certain security or stability issue. The ultimate goal to make Heficed’s IPMI environment secure and stable for thousands of servers we run around the globe.

Since the public IP was not a solution, we had an idea of introducing so-called Virtual Console Management (VCM) that would be between Terminal and IPMI. VCM is the core processor to manage the local IP pool and assign the server access for the tenant. After two hours, it would remove the session and would close the connection.

Having a secure environment was not enough since we had another challenge to make the IPMI stable. As many infrastructure providers would know, there are lots of issues with the IPMI protocols. As I have mentioned earlier, server vendors do not put much love into developing the IPMI. That was the biggest challenge. However, we had to re-write the whole access and re-used just a small bit of what integrated server management had.

When we had VCM deployed, we had to complete a task to remove the public IPs and introduce the local ones.

Apply the same system to all our existing locations. By the way, we have a Virtual Console name for the Heficed IPMI environment project. So if you would spot in the Terminal so-called Virtual Console, you know it is IPMI access.