How A Real Estate Agent Can Bring Down An Entire Office Network

(Or, when you’re doing network work, make sure you’re thorough.)

Warning: Geek Alert… long story after the jump.

So at 4:42 this evening, right as I’m getting ready to pack up and go home, I get a phone call from one of the other vacation rental offices I work at. “We can’t get logged in at all, it keeps booting us out.” The entire office is powered by a Citrix Metaframe server and a bunch of thin terminals, and they have to log in before they can do anything. If they can’t reach the server, they can’t work. And the office is open until 6:00 and needed to be accessible remotely in the evenings after hours.

I try to log into the machine remotely, and it lets me on, but then kicks me off. Log in again, lets me on for about five seconds, boots me off. So I hop in my car, drive on over there, and head upstairs to have a look at the server. It was due for a full reboot, so I just figured I’d reboot everything, including the network switches, and try it again. Upon reboot, the server tells me it can’t log into the domain. Every other machine on the network can get onto the domain fine, but this one can’t. So I change the network cable, still no dice. Even tried throwing in a new network card, still didn’t work getting onto the domain.

So I gave up on the domain, and logged into the machine locally, which worked fine, and the network card couldn’t get online at all. I ran “ipconfig” via the command line, and it didn’t have an IP address, for some reason (this server has a static IP x.x.x.10 address assigned to it). I go into the config for the network card, make sure everything’s still assigned there properly, and it is. So, for giggles, I change the IP address to have it collect it via DHCP — and it got online fine. I got on another machine on the network, and pinged the x.x.x.10 address, and it was getting responses — so something on my network was trying to use the same IP address and it was obviously causing problems. Time for a wild goose chase.

I knew all the thin terminals got their IP addresses via DHCP in the x.x.x.100+ range, so they weren’t the problem. I ran a scan and dig on that IP address so I could get a MAC address an hostname out of it. The hostname was HP2E8D03, and the MAC address was easy enough to match up to a port on my switch, but since the wires/jacks aren’t all labeled, it didn’t really help me. But looking at the hostname, I figured it was probably some HP device. I disconnected all that company’s HP devices that I could find that would be running off that switch and made sure that their IP addresses were correct. Yet I was still able to ping that stupid IP address. So I disconnected all the other stuff — HP or otherwise — for the company, and still could ping it.

Then I realized something. In the downstairs part of the building there were some real estate agents that are renting out some space. When we first set up the office for them, I got them their own seperate router, switch and ‘net connection and reconfigured all their jacks in their office so that they were physically on a different network than the rest of the building — but apparently I forgot one jack that was still on the vacation rental network. When I originally helped him setup his HP all-in-one inkjet, he was printing to it fine and the office was running to the outside world via the other ‘net connection, so I thought everything was fine. Well, he rearranged the office a bit and moved that printer somewhere to the one jack I didn’t rewire upstairs in the network closet (I don’t know how I missed it before — I didn’t even know the jack was there). That other jack was wired onto the vacation rental network, and his printer had a static IP of x.x.x.10 — the exact same IP as the Metaframe server. Once I rewired the jack upstairs and moved that printer off the network, the server had no problem at all getting onto the domain. Life could go on again.

What a mess…I want those couple hours back.