The Network You Use When the Network Is Down: Designing Out-of-Band Management for iDRAC, iLO, Switches, and Console Servers

It is two in the morning and a server in the data center has stopped answering. Its website is down, its login prompt is gone, and ping gets nothing back. Somebody has to decide whether to drive forty minutes to press a power button, and that somebody will be standing in a cold aisle at three, holding a phone flashlight, wondering what the screen would have said if anyone could see it. The alternative costs nothing at that hour, because it was built months earlier: a second, quieter network that reaches the machine's own remote controls even when the machine itself is dead. This post is about building that quieter network and, in particular, giving it addresses of its own, separate from everything else you run.

Nearly every server sold in the last fifteen years contains a second, much smaller computer that stays on as long as the power cord is plugged in. Dell calls it the iDRAC (Integrated Dell Remote Access Controller), HPE (Hewlett Packard Enterprise) calls it iLO (Integrated Lights-Out), and the generic name is the BMC (Baseboard Management Controller), which most other vendors speak to through a standard called IPMI (Intelligent Platform Management Interface) or its modern successor, Redfish. Whatever the badge says, the job is the same. The BMC has its own network port and its own IP (Internet Protocol) address, and through it you can see the server's screen as if you were standing at a monitor, press its power button, attach an installer disc from your laptop, read the fans and temperatures, and update firmware, all while the main operating system is frozen, crashed, or not installed yet. Network switches carry the same idea in the form of a dedicated management port, separate from the ports that carry your traffic. And for the moments when even that fails, a console server (a box with a shelf of serial ports, sometimes called a terminal server) connects to the old-fashioned text console of each switch and router, the same connection you would get with a cable and a laptop in the closet.

The cast, and what each one does with the box switched off

server remote controller   Dell iDRAC, HPE iLO, generic IPMI     see the screen, power on and off, mount an installer, read sensors, flash firmware
switch management port     "mgmt0", "ma1", "management1"        log into the switch even when its data ports are dead
console server             Opengear, Lantronix, a Raspberry Pi   a serial cable to every switch and router, reachable from anywhere
smart PDU                  APC, Raritan, Eaton                   turn one specific outlet off and on again, from home

Reaching these controls through your regular network is called in-band management, and it works fine until the regular network is the problem. Out-of-band management (OOB) means reaching them through a path that does not depend on the production network at all: separate switches, separate cabling, and ideally a separate way out of the building. The distinction matters because the failures that hurt most are exactly the ones that take the ordinary path with them. A bad configuration pushed to the core switch cuts off every device behind it, including the switch's own in-band address. A firewall rule typed backwards blocks the administrators along with the attackers. A server that hangs during an update takes its in-band interface down with it, since that interface is run by the very operating system that just froze. In our worked IP plan for the fictional Meridian Logistics, every site got an in-band management VLAN (virtual local area network, a way of splitting one physical network into several isolated ones) for its switches and access points, plus a separate block, 10.64.128.0/17, reserved for out-of-band management inside the infrastructure space. This post fills in that reserved block.

Two facts about management interfaces drive the whole design. The first is that they are the most powerful devices on your network. Whoever controls a server's BMC controls the server completely: they can watch the screen, type at the console, boot from their own disc, and rewrite the firmware underneath the operating system, where no antivirus will ever look. The second is that they run some of the worst software you own. A BMC is a small embedded computer with a web interface, a firmware update cycle that most teams ignore for years, and a long public history of default passwords and remotely exploitable bugs. Put those two facts together and the requirements follow. Management interfaces need to be reachable by a few trusted people from a few trusted places, unreachable by everyone and everything else, and kept in address space that a stranger can identify at a glance, so that a rule like "nothing from the office may talk to 10.64.128.0/17" is short enough to type and easy enough to audit. Our guide to segmenting cameras, badge readers, and other IoT (Internet of Things) devices applies the same logic to the least trusted gear in the building, and the management network is the same idea pointed at the most privileged.

The address space for management belongs in the infrastructure block, outside every site's own allocation, for the same reason router loopbacks and point-to-point links live there: the management network is a single system that spans every site, and its rules, its monitoring, and its access controls are written once for all of it. Meridian's /17 holds about 32,000 addresses, which sounds absurd for a few hundred management ports until you carve it and discover that the structure consumes far more of it than the devices do. Give each site a /20, a block of 4,096 addresses, and inside it lay out the same handful of /24s in the same order everywhere. If the slash notation is unfamiliar, our visual guide to CIDR (Classless Inter-Domain Routing) explains it, and the short version is that a /24 holds 256 addresses.

Out-of-band management: a /20 per site, the same /24s in the same order

10.64.128.0/17                 Out-of-band management  (from the infrastructure block)
├── 10.64.128.0/20        Ashburn data center
│   ├── 10.64.128.0/24    Server BMCs (iDRAC, iLO, IPMI)
│   ├── 10.64.129.0/24    Switch and router management ports
│   ├── 10.64.130.0/24    Console servers, PDUs, environmental sensors
│   ├── 10.64.131.0/24    OOB gateway, jump host, cellular uplink
│   └── 10.64.132.0/24 to 10.64.143.0/24   Reserved
├── 10.64.144.0/20        Chicago headquarters (same layout)
├── 10.64.160.0/20        Dallas branch (same layout)
├── 10.64.176.0/20        Atlanta branch (same layout)
└── 10.64.192.0/18        Reserved (four more sites)

Giving each class of device its own subnet, where one big shared range would have been easier, pays off in three places. Firewall rules get shorter, because server BMCs, switch management ports, and console servers speak different protocols to different people. The server team reaches BMCs over HTTPS, the same encrypted web connection a browser uses, plus a remote-console port. The network team reaches switches over SSH (Secure Shell, the encrypted command-line login used for network gear), and the console server is the thing you use when SSH to the switch has failed. A subnet per class means the rule "server admins may reach 10.64.128.0/24" is complete and correct without a list of exceptions. Ownership gets cleaner, because the subnet's edge is the team's edge. And troubleshooting gets faster, because the third number of the address announces what kind of device you are looking at: anything at 10.64.129.x in Ashburn is a switch, anything at 10.64.128.x is a server's BMC, and the pattern repeats at every site with that site's own offset. If a branch is small enough that four /24s feel silly, keep the four subnets and shrink them, since our guide to right-sizing subnets makes the case that a /26 for a rack of switches is an honest size and a /24 is a habit. The layout is what matters, and the numbers are yours to change.

One decision at the hardware level decides whether any of this is real. Most BMCs can run in either of two modes. In dedicated mode, the BMC uses its own physical port on the back of the server, which you cable to the management switch. In shared mode, the BMC borrows one of the server's regular network ports and rides alongside production traffic, which saves a cable and a switch port, and means the BMC vanishes the moment that production link, that production switch, or that production VLAN has a bad day. Dell sells the dedicated port as part of a higher iDRAC license on some models, and HPE's iLO has an equivalent shared-port setting, so whichever brand you run, choose dedicated, cable it to the management switch, and write the choice into the build checklist. The management switch itself should be the least interesting device you own: a cheap, small, separately powered switch with almost no configuration, so that it never needs to be managed through the network it exists to protect. Plug it into its own PDU (power distribution unit, the smart power strip in the rack that lets you switch outlets remotely) where you can, and give that PDU an address in the console-and-power subnet, because a PDU that can power-cycle a hung switch from three states away is the second-best tool on this network after the console server.

The path out of the building deserves the same independence. A console server in a rack is useless if the only way to reach it is through the switch it is supposed to rescue. Larger shops give the management network its own small router with a cellular modem or a separate cheap internet line, so that an administrator at home on a VPN (virtual private network, an encrypted tunnel from a laptop into the office) can reach the OOB gateway while the main circuit is dark. Smaller shops get most of the benefit from a single console server with a built-in cellular modem, or from a jump host, a hardened little machine on the management network that is the only thing allowed to log into anything else. Whichever shape fits, the gateway subnet in the carve above is where those pieces live, and the test is simple: unplug the main internet line and confirm you can still reach a BMC. If you have never run that test, the out-of-band network is still a plan on paper.

Every address on the management network should be static, typed into the device by hand and recorded the same day. Management gear is the bootstrap layer of the network, the equipment that has to work before and without everything else, and our guide to DHCP, static, and reservations explains why that layer never asks a server for its address: the day you need the BMC most is the day the DHCP (Dynamic Host Configuration Protocol, the service that hands out addresses automatically) server may be one of the things that is down. Factory defaults make the case louder. Dell's iDRAC ships with a default address of 192.168.0.120, the same on every unit, so a rack of new servers powered up before anyone configures them is a rack of identical addresses fighting over the same management switch, the exact situation our guide to IP address conflicts describes. Set the static address as part of the rack-and-stack checklist, before the server is cabled to anything else, and record the address, the server it belongs to, the serial number (Dell calls it a service tag), and the BMC type in your IPAM (IP Address Management, the tool that tracks which addresses exist and what they are for) in the same step. An address that is set on the device and written down nowhere is the first thing the next person will have to rediscover, and our post on tracing an IP address back to a physical device describes what that rediscovery costs.

Names carry the rest of the burden. A BMC should be named for the server it lives in, with a suffix that says what it is, so that ash-db-01-idrac is unmistakably the remote controller for ash-db-01, and a switch's management port takes the switch's name with -mgmt on the end. Where you can, give the BMC the same last number as the server's production address, so that 10.64.128.11 and 10.16.50.11 are obviously the same machine seen from two sides. Our guide to naming conventions explains why a name should hold only the facts that die with the device, meaning its site, its role, and its number, while the model, the firmware version, the rack position, and the owner belong in description and custom fields you can change without renaming anything. Give every one of these names a DNS (Domain Name System, the service that turns names into addresses) entry, visible inside the company only, so nobody has to memorize that the third database server's controller is .23, and give them reverse entries too, so a management address that shows up in a log resolves to something a human can read. Keeping those records honest is its own discipline, covered in our post on keeping the IP inventory and DNS in sync, and the management network is where it matters most, because these are the names you will type at two in the morning.

One server and its neighbors, seen from both networks

ash-db-01              10.16.50.11     the server itself, on the Ashburn server network
ash-db-01-idrac        10.64.128.11    its BMC, same last number, on the Ashburn BMC subnet
ash-sw-core-01-mgmt    10.64.129.1     the core switch's dedicated management port
ash-con-01             10.64.130.5     console server, serial lines to every switch in row A
ash-pdu-a01            10.64.130.21    smart PDU in rack A01, remote outlet control
ash-oob-gw-01          10.64.131.1     the OOB gateway, with the cellular uplink

Every rule that protects this network names a source, a destination, and a verdict, and the whole set fits on one screen. Write it in plain English first, and the firewall configuration becomes a translation exercise.

The rules, in plain English

from  admin VPN and the jump host     to  every management subnet        ALLOW   HTTPS, SSH, the BMC console ports
from  the monitoring server          to  every management subnet        ALLOW   read-only health checks
from  management subnets             to  time, DNS, logging, firmware   ALLOW   the few services they need
from  management subnets             to  anything else                  DENY    a BMC never needs to browse
from  users, Wi-Fi, servers, cloud   to  management subnets             DENY    in-band stays in-band
from  the internet                   to  management subnets             DENY    and check it twice

Two of those lines deserve a word. The rule stopping management subnets from reaching out matters because a compromised BMC that cannot phone home is a much smaller problem than one that can, and the handful of services it genuinely needs are easy to list: a clock to set itself by (NTP, the Network Time Protocol), name resolution, a place to send its logs (syslog, the standard log-forwarding protocol), and a server to fetch firmware from. The rule blocking the internet sounds too obvious to write down, and a search engine for internet-connected devices will show you thousands of BMC login pages that somebody's firewall let through anyway, usually because a rule meant for one web server was written one number too wide. On the switches themselves, most vendors let you place the management port into its own VRF (Virtual Routing and Forwarding instance, a separate routing table inside one device), so that management traffic and production traffic never share a route even inside the same box. Our guide to VLANs and VRFs explains the machinery, and this is the rare case where a VRF earns its keep for isolation alone, with no overlapping addresses in sight.

The management network is small, changes rarely, and is read under the worst possible conditions, which makes its documentation worth more per line than any other part of the plan. In IPCraft, the /17 becomes a subnet under the Infrastructure folder, marked as a container so that it lends structure without pretending to hold devices, and each site's /20 and its four /24s nest underneath, tagged with the site's location and the management VLAN number. Every BMC, switch port, console server, and PDU becomes an address record with a hostname that follows the convention above and custom fields for the server it belongs to, the serial number, the BMC type, the firmware version, and the rack position. When a new site opens, the plus action on the /17 finds the next free /20 without anyone counting on their fingers, and our free subnet calculator checks the arithmetic if you would rather see it. And when you suspect the records have drifted, which on a network nobody looks at for months is a reasonable suspicion, log into the jump host, print its ARP (Address Resolution Protocol) table, the list of addresses it has recently seen on the wire, and paste it into scan reconcile: any BMC that was racked and never recorded shows up as a new address with a Dell or HPE vendor name beside it. That same technique, applied to an inherited network, is how you find the management interfaces the previous engineer configured and told no one about.

Replay the opening scene with the network in place. The alert names ash-db-03, the server is silent, and the administrator at home opens the VPN to the OOB gateway, which is up because it never depended on the main circuit. A DNS lookup turns ash-db-03-idrac into 10.64.128.23, the login page loads, and the virtual console shows a kernel panic frozen on the screen, which is the diagnosis that would otherwise have waited for a forty-minute drive. One click on the power menu reboots the machine, the database is back in six minutes, and the panic text is already pasted into the ticket for the morning. Three weeks later a mistyped command on ash-sw-core-01 cuts off the entire data center, including the switch's own management port, and the fix runs through ash-con-01 instead: SSH to the console server, pick serial line 3, and the switch's text console answers as if a laptop were plugged into it. Rolling back the configuration takes ninety seconds. Nobody drives anywhere.

The management network is the one part of the plan you build for a night you hope never comes, and it is the easiest to skip because it does nothing on ordinary days. A separate block of addresses, one subnet per class of device at each site, static assignments recorded on the day they are set, a rule set short enough to read, and a real test with the main line unplugged turn a room full of remote controls into something you can rely on. The same shape shrinks to a home lab, where a spare switch and a cheap smart plug do the job of a rack of console servers, and it grows to a fleet of data centers without changing. Draw it now, while everything is working, because the alternative is designing it at two in the morning, in a cold aisle, by flashlight.