The last few days have been the kind of frustrating IT days you can have anywhere. It’s just that here I’m sleeping in a classroom, showering in a portable shower unit, and eating wilderness fireman food in between the continual IT hassles.
As has been observed countless times through history, this job would be OK if it weren’t for the customers. They have this expectation that things will work, and when they don’t they come find us and ask us to stop everything and fix our network. Of course, usually we have already noticed that things are broken, and have already stopped everything to work on the broken stuff instead of the new projects we were hoping to get done.
Two days running now, we have found in the morning that our link to Gulfport is not working. The radios have power, but the channels are getting noisy. The FCC Part 15 rules, which define how signals are supposed to coexist in the 900, 2400, and 5800 Mhz unlicensed bands are pretty clear. There are things you can do to be a good citizen, and things you can do in the face of noise to keep your stuff running. Overall the system works astonishingly well. But there’s a new player on the scene, Motorola Canopy. The way it has been described to me, the Canopy radio system is within the letter of the law, but still manages to wipe everyone else off the net by raising the noise level for everyone but itself. The key “innovation” that Canopy introduced is time syncronization among the radios in the network. So a group of radios in on the time sync plan can work together to ignore each other’s noise. Everyone else, like the off the shelf 802.11b, the Trango AP’s we are using, etc, sees all the noise. That’s not playing fair, but there is nothing in the Part 15 rules that preclude it, since the FCC did not envision radios using time synchronization to flood the channels.
Alas, the people who reportedly bought the Canopy system in Gulfport are FEMA. We aren’t in a very good bargaining position with them. They can play the “we’re the government, we’re doing important work, you’re just amatuers” card. All we can play is is the “you’re dicks, and you bought a bad system from an evil vendor” card. Their suit is trump.
So this morning we put into effect a backup plan to feed the critical parts of our network from a satellite feed here. We’ve shut down parts of our distribution network so that the satellite-provided bandwidth is limited to sites that have a legitimate need for it. Shelters will have to wait for Internet access until we can fix the link to Gulfport.
There’s another group doing work like us in Slidell, Louisiana. We are looking into conneting into them, which will work until someone brings up a Motorola Canopy system there and knocks us back off the air.
All this among frustrating attempts to improve the network at client sites. Hot tip: you can’t shoot 802.11b through the magnetic field of a 75 kvA generator, especially if the 100 foot path also includes two refer trucks with big diesel engines. Diesel engines, thankfully, don’t make electrical noise, but those big heavy engine blocks probably sucked up a bunch of our signal. We gave up (way too late) and ran a 600 foot Ethernet run. (At least I got to go in the girl’s showers to scope out a place to put an intermediate switch to regenerate the signal, should that have prevented the long run from linking right!)
Two interns from Mississippi State University have showed up. They will be doing 300 hours each of volunteer time to fulfill an internship requirement for graduation. They report to me. So far, I have used them a little for tech help, and a lot for gofer tasks. That’s the nature of being an intern. As I get to know them better, and they start picking stuff up the gofer stuff should let up a bit. I’ve assigned one a longer term project that he should learn a lot from (making a transparent caching proxy). I’m thinking about what to assign the other one… network management would have been a logical choice, but that was too pressing to wait for an intern to figure it out. The big picure plan for this network is for those of us that built it to clear out when the building is done and leave it in the hands of the interns, with backup support from us from afar. It’s an ambitious goal, but we’ll give it a try and see how it goes.
Leave a Reply