How a one-cabinet hardware consolidation at Equinix DX1 in Dubai turned into a thirteen-hour Saturday, and what the cables taught us.
The downtime window opened at 10:00. Four minutes later two links on the core stack went dark, and nobody had touched anything yet.
The customer’s network administrator saw it from his console and called the port numbers out in the group chat. Igor Bryzgin, our head of delivery, checked them on the front panel and pushed one connector home. Both came back. The cause was a patch cable at the firewall with a broken locking clip, the kind where a light nudge drops the link. “Gentler, gentler,” wrote the customer’s infrastructure lead, and our engineers took a step back from the firewalls.
We noted it and got on with the day. It settled an argument nine hours later.
What the job was
A multi-billion dollar global company keeps its regional systems at Equinix DX1 in Dubai. The work was on-prem hardware consolidation: empty one cabinet completely and rebuild its contents in another cabinet in the same row, already contracted and powered and standing empty.
The list was a Check Point firewall cluster, two Huawei switch stacks, two Fibre Channel fabrics, a tape library, a backup server, an OceanStor controller and three disk shelves. Five ESXi hosts stayed where they were in the neighbouring cabinet and had to find all of it again afterwards.
The plan, and the people who picked it apart
Midhun Soumini surveyed the cabinet on 4 September, photographed every unit and checked what was actually in the rack against the as-built drawing. That comparison is the dull part of this kind of job and it is the part that pays.
The plan went to the customer on 9 September. One working day. Anything that existed as a redundant pair would move one unit at a time so its partner carried the service, and anything without a partner would be powered down for the day. On paper that put the whole outage into a single 90-minute window in the afternoon.
Then the customer’s engineers read it, which is what you want.
Their network administrator wrote back that four interfaces in the cable journal were unconfirmed. His console showed each port’s state, but not what was on the other end, and he did not want to find out live on the Saturday. So on the Friday Igor drove to the data centre, and the two of them went through the four together: Igor at the rack, the administrator on VPN.
Two were down, and that was normal. They were management and similar service ports, shut on the firewall side on purpose. But a few years of running changes leave a rack full of links like these, and nobody misses one until it is gone. So we checked them anyway and made sure each would go back where it belonged, even though the administrators had reported nothing wrong.
The other two were up, and nobody knew what they were. A ping identified the first one: the tape library. The second cable matched the journal at both ends. At 16:48 an updated cable journal went round, and four question marks came off the Saturday plan.
Both corrections went into v2. Neither would have been funny to discover at three in the afternoon on the Saturday.
Moving day
Shutdown ran from 10:00 to 11:00 in a fixed order: mail first, then the virtual machines, then vCenter, with the domain controller holding the FSMO roles last off and first on. The tape library has no remote power-off, so somebody pressed the button on the front. The backup server hung and got the same treatment.
At 11:16 the old cabinet went dark.
The first thing we carried across was a core switch, deliberately. The DAC cables that link the stack back to the cabinet next door are a fixed length, and if they didn’t reach, every assumption after that was wrong. They reached. The survey had said they would, but there is a difference between a measurement on paper and a cable in your hand.
Then the rest went in, bottom up.
At 14:14 Igor sent a photo of the half-built rack with a one-line caption: taking longer than planned. Not the mounting. The cables. Every cable that was now too short had to be swapped for a longer one, every label on the new cabinet had to be remade to match the old one, and a handful of cables turned out never to have been labelled at all.
The labels that did exist were not always right. The DAC cables between the two 40G switches were tagged 5 to 5 and 6 to 6. They are a stack, so they cross: 5 to 6, 6 to 5. “Wrongly labelled,” the network administrator wrote. “Which is why I keep saying it is dangerous to look at the tags,” his colleague added. We cabled it the way the cable journal said and replaced the labels.
The argument we lost, and should have
At 18:14 everything was mounted and connected, with the bundles still hanging loose. Igor asked to power the network on and dress the cables afterwards, once we knew nothing needed moving. It is the faster order, and it is how most people would do it.
The customer said no. Dressing cables means pulling on them, and pulling on cables in a live rack is exactly what had knocked two links over that morning. “You already saw this morning what that leads to.”
So we dressed the cables first, in a cabinet with no power on it, and powered up after. That cost an hour we would rather have spent going home. It was the right call, and it was theirs, not ours.
Switching it on, one box at a time
The power-on was staged, and slow on purpose. First firewall node. Wait. The customer checks LLDP and the port channels from his side, checks that the cluster address answers, then says go. Second firewall node. Wait. Sync comes up. First core switch, three minutes, second core switch, and the stack verified on the switch itself before anything else is touched.
Then the copper switches, the Fibre Channel fabrics, the tape library. At 20:01: “The network is up, you can bring the servers up.”
Storage next. Three disk shelves, a couple of minutes of even fan noise and green LEDs, then the controller, then the backup server, then the hosts. At 20:27 all five went green.
The cable under the brush panel
One of them didn’t, quite.
The storage engineer could see only one of the two links on the first host. The switch port said it was up. The host said it wasn’t there.
What followed took forty minutes and is the most useful part of the story. The network administrator shut the port down and brought it back up, over and over, so that Igor could watch which physical port blinked in a rack of identical connectors. That found the first fault. A patch cord labelled for the first host was actually the backup server’s, and it was sitting in port 10 instead of its own port 9. It had been mislabelled before the move, and we had faithfully carried the mislabel across. Moved back to port 9, the backup server returned.
The host’s own second link was still down. Igor traced the fibre by hand and found that during cable dressing it had been tucked under the brush panel and pinched out of sight. At 21:13 both links were up.
By 21:56 the domain controllers were back, Exchange was back, and someone reported a test email arriving from Gmail. That is the line you wait for.
The last fault was a Fibre Channel link to the backup storage showing amber. Polarity swap, new cable, old cable back again. Somewhere in the middle of it Igor was asked whether the array had spun up and had to answer that it is extremely loud in there and he honestly could not tell. The infrastructure lead suggested pointing a phone camera into the connector to look for the transmit light, which is a trick worth knowing. Midhun found the fault. The link came up at 23:11.
At 23:20 the storage engineer wrote: “Yes. That is it for today. Tape library and backup server next time. Thank you!!!”
“Perfect is the enemy of good. The main things are working.”
What we took away
Thirteen hours in the data centre against a nine-hour plan. The lifting was never the problem. The cabling was, and cabling is the part that is hardest to estimate from a survey photograph.
A label is a claim, and claims get checked. Three separate faults that day came from a tag that disagreed with reality, and in each case the cable journal and a live LLDP check settled it faster than the tag ever could have.
A staged power-on with someone watching from the far end turns one large question into thirty small ones, each with a yes or no answer. It is slower, and it is why we found a mislabelled patch cord at 20:32 that night instead of on Monday morning.
Customer engineers picking the plan apart is not an inconvenience either. The SAS correction alone would have cost an hour in the middle of the window.
And the job was not closed that night. The tape library and the backup server were not validated until the following week, backup ran on a temporary arrangement for a few more days, and Midhun pushed his leave back to finish them off. Worth saying out loud, because the version of this story where everything lights up green at 18:00 does not exist.
