Frustrating couple of days

The last few days have been the kind of frustrating IT days you can have anywhere. It’s just that here I’m sleeping in a classroom, showering in a portable shower unit, and eating wilderness fireman food in between the continual IT hassles.

As has been observed countless times through history, this job would be OK if it weren’t for the customers. They have this expectation that things will work, and when they don’t they come find us and ask us to stop everything and fix our network. Of course, usually we have already noticed that things are broken, and have already stopped everything to work on the broken stuff instead of the new projects we were hoping to get done.

Two days running now, we have found in the morning that our link to Gulfport is not working. The radios have power, but the channels are getting noisy. The FCC Part 15 rules, which define how signals are supposed to coexist in the 900, 2400, and 5800 Mhz unlicensed bands are pretty clear. There are things you can do to be a good citizen, and things you can do in the face of noise to keep your stuff running. Overall the system works astonishingly well. But there’s a new player on the scene, Motorola Canopy. The way it has been described to me, the Canopy radio system is within the letter of the law, but still manages to wipe everyone else off the net by raising the noise level for everyone but itself. The key “innovation” that Canopy introduced is time syncronization among the radios in the network. So a group of radios in on the time sync plan can work together to ignore each other’s noise. Everyone else, like the off the shelf 802.11b, the Trango AP’s we are using, etc, sees all the noise. That’s not playing fair, but there is nothing in the Part 15 rules that preclude it, since the FCC did not envision radios using time synchronization to flood the channels.

Alas, the people who reportedly bought the Canopy system in Gulfport are FEMA. We aren’t in a very good bargaining position with them. They can play the “we’re the government, we’re doing important work, you’re just amatuers” card. All we can play is is the “you’re dicks, and you bought a bad system from an evil vendor” card. Their suit is trump.

So this morning we put into effect a backup plan to feed the critical parts of our network from a satellite feed here. We’ve shut down parts of our distribution network so that the satellite-provided bandwidth is limited to sites that have a legitimate need for it. Shelters will have to wait for Internet access until we can fix the link to Gulfport.

There’s another group doing work like us in Slidell, Louisiana. We are looking into conneting into them, which will work until someone brings up a Motorola Canopy system there and knocks us back off the air.

All this among frustrating attempts to improve the network at client sites. Hot tip: you can’t shoot 802.11b through the magnetic field of a 75 kvA generator, especially if the 100 foot path also includes two refer trucks with big diesel engines. Diesel engines, thankfully, don’t make electrical noise, but those big heavy engine blocks probably sucked up a bunch of our signal. We gave up (way too late) and ran a 600 foot Ethernet run. (At least I got to go in the girl’s showers to scope out a place to put an intermediate switch to regenerate the signal, should that have prevented the long run from linking right!)

Two interns from Mississippi State University have showed up. They will be doing 300 hours each of volunteer time to fulfill an internship requirement for graduation. They report to me. So far, I have used them a little for tech help, and a lot for gofer tasks. That’s the nature of being an intern. As I get to know them better, and they start picking stuff up the gofer stuff should let up a bit. I’ve assigned one a longer term project that he should learn a lot from (making a transparent caching proxy). I’m thinking about what to assign the other one… network management would have been a logical choice, but that was too pressing to wait for an intern to figure it out. The big picure plan for this network is for those of us that built it to clear out when the building is done and leave it in the hands of the interns, with backup support from us from afar. It’s an ambitious goal, but we’ll give it a try and see how it goes.

Life in the EOC

First, be sure to read the Radio Response website. I am the primary author there, so reading that is as good as reading this. Except there I have to be a bit more professional and circumspect.

A few days ago I moved to Bay St. Louis. The work was really heating up down here for IT generalists (as they charitably call people who aren’t allowed on towers…). The cynic would also point out that once the Bay St. Louis team’s lodgings were upgraded to include air conditioning I consented to go to Bay St. Louis.

Look, I’m a datacenter guy, not a tower monkey. I don’t DO hot.

It’s been nice working here at the EOC. There are interesting people to talk to, when you feel like it. There is hands on work to do out in the community. Today at the feeding center, we had several people come up to us and respond positively to the news that any day now there would be free internet and phone calls where they eat three meals a day. We picked that site because we knew that it would reduce the necessity of marketing; the food brings them in, the free long distance keeps them coming back.

My schedule here works like this… at 6:30 or so our team leader here (Joe) turns on the lights and I get up. I go to breakfast and a morning meeting at 7am to get an assignment for the day. I sometimes get partnered with someone, sometimes I work alone. Depending on the work, I head back to the EOC at lunch for a sack lunch. Today I ate on the run because the network was coming up around lunch and I was too excited to stop and eat. We work until sunset and eat back at the EOC. I spend the evening on “office work”, then shower and hit the sack. I’m working too many hours, but each hour is interesting, so I’ll keep at it until there’s a reason to change it.

It has been hot and humid, but the weather changed a bit as Rita came into the gulf. We started to get some breezes, and the air turned less humid. I think it might be because the wind is coming off the land instead of off the sea, but I haven’t taken the time to really check.

The National Guard folks are rotating out. Some are coming to replace them, but not as much. Earlier in the week, then it seemed like Rita might come this way, they were packing up pretty fast. That was a little worrying, except that they have so much more crap to move than we do. I can just get in my car and drive north. They have a day of packing to do first.

Today we got an infusion of fresh blood and brainpower from the folks at One of the guys wrote the wireless book I took to Guatemala with me. This place is really attracting a great crowd. Hopefully we can keep the volunteers coming. It will get pretty tough to keep making progress if it dwindles to just Mac and me!

There’s been no news on an FCC grant, or any other money. We have a dedicated guy working on it, but the wheels of government turn slowly. If you want to throw a hail mary pass, drop your representatives to congress a line and tell them Radio Response is the kind of people’s initiative that should be rewarded with modest support. We don’t want much, just our gas money paid for. We’re already getting food and shelter from the EOC (though we had to work hard to prove ourselves to get that). What we’ve done so far has an input of well under 1/10000 of 1 percent of the total amount budgeted for Katrina response. All we are asking for is our 1/10000 of a percent.

Superfast update

Super-fast update, since I am in the middle of stuff here, but also realized I haven’t written in a while.

I have visited Bay St. Louis. The destruction there is incredible. I have not seen the water front, just the wind and flood damage several hundred yards inland. Our team is kicking butt here, but it just takes so much longer to get things done than you can possibly imagine. It’s frustrating, but we won’t give up. It’s no one’s fault. We are now tied into the systems here so that we are no longer being actively prevented from working by the authorities. Now it’s simply a complex situation, and you have to work around problems one at a time.

I took a survey trip into New Orleans this weekend. I visited Algiers, where this is only wind damage, not flooding damage. I also visited the Central Business District of Nola, which seems to have not flooded either. After finishing my business there Sunday morning I headed down to the French Quarter to be a disaster tourist for a bit. I had set aside Sunday as my day off, and had no other work planned. Joel and I stopped in to this bar that never ever closes. There have been news reports about it. They were getting resupplied, so we helped carry ice upstairs to the ice machine (which, like the rest of the neighboorhood, has no electricity). After the restocking was complete, we had a couple screwdrivers. I have never been so happy to hand over $10 to a bartender as I was Sunday. It meant so much to me and to him to have a normal nusiness transaction in the middle of this mess.

I went “home” to Ponchatoula Sunday afternoon. I visited Miss Gerri, who has adopted the guys (and now, gal) of Radio Response. She lets us come over to her house for showers. She cooks for us too. I had a very relaxing afternoon at her place, and am totally recharged for a new week.

In other news, if you know anyone who will listen, tell them this: the mayor is making a huge mistake trying to get people in to the city too fast. I have been there and seen it. There’s nothing to do in the city. There’s no electricity, there’s no potable water, there’s. He’s making a huge mistake, endangering his residents, and doing it in the face of reasoned statistics and arguments from EPA, CDC, and Coast Guard. People who return too early will die unless they are completely self sufficient, as those of us working here are.

Something our group needs is remote helpers who can work the phone. If you’d like to volunteer, call the number on the Radio Response website. We need continue to line up experienced installers to replace the current group. We need to find places to install computer labs; in places we have bandwidth and equipment, but we don’t know where the people who need access to FEMA’s website are. None of this work requires someone on site.

First full day

Yesterday, I worked my first full day. So I’ve been at work 1.5 days. In that time, two new people have come in. I made a minimal system to get people up and running, and tried it out on the next two after me. I suspect that as I start working in the field it wil fall by the wayside, but at lesat two people know what new people should get told.

Yesterday night I grabbed the journalist hat and put it on and wrote our update for the day. Because I had been trapped in a desk job for the day, it was fun to interview people and find out what was happening in the field. Then to strip out the four letter words and tales of big-org incompetence. We intentionally went for a more aggressive tone in this one, in order to help get the word out that things aren’t as under control as the spokespersons tell the press, and that we have the equipment, tools, and talent to make it better.

Please take a look at our new website, including yesterday’s update by me.

I am building up quite a sleep deficit from the drive out, followed by there always being one more thing to do before bed (and an alarm clock set for the morning). I have Sunday scheduled as a day off. I plan to stick to a 6-day work week for the next few weeks to make sure I keep in good shape. Another guy here is of the slightly higher-strung variety and has been on the go for quite a while now. After coming back from Bay St. Louis he was really wiped out, both physically and emotionally. He took an all-day rest, and it made a huge difference.

I’m really excited about today. I will be going to Bay St. Louis to get checked in with the Emergency Operations Center there and get my ID to work in Harrison County, MS. After that (and some other errands people in my car need to do there, like shots) we will head in to Algiers, downtown. Our team is delivering some stuff to some community radio people who are running an internet lab for the commnuity down there. My job will be to explore how we can get them some serious bandwidth. Their current connection for an entire room of computers is a laptop with a cell phone card!

Day 1

My first day was half a day because I got here and got put to work after noon. I spent the first while getting the lay of the land, the people, etc. The Bay St. Louis team was getting loaded up to head south, so there was some work to do there. Once they left it calmed down quite a bit.

I identified two needs and a guy named Alecs and I self-organized into a little team to chip away at them. We need a single public website, and also a single internal place for a couple web apps people have thrown together with Ruby on Rails. We could also use a better “new volunteer arrives and gets to work” process. So those are the things I am working on.

I slept indoors on someone’s cot in the air conditioning. It was so cold I was glad I had my sleeping bag. The weather outside is 90% with 90% humidity. I went out at night and the air was in some kind of super-humid state such that it was literally hard to breathe. Every time I would take a deep breath, I’d feel like I wasn’t getting enough oxygen. I think the air was actually condensing inside my lungs! A thick fog hung in the air, like something out of a Poe novel.

This morning we woke up and realized the Internet connection here was sucking. So I started debugging it, trying to decide if it was a local problem or a remote one. The experienced wifi guys, instead of looking at their computers, walked outside and looked at the link. The next door neighbor, who we are getting Internet from wirelessly, had parked two semi trucks between the two antennas during the night! Looks like I have a lot to learn about debugging wireless networks…

Arrived in Ponchatoula

I arrived in Ponchatoula a little while ago. I can tell I am going to like working with these guys. The trick now is to find a job and start doing it. I think perhaps I will start the same way I did in Guatemala, by drawing a network diagram of the staging area here so that I know how things work (and so that the next guy knows too).

With Cliff in Dallas

I dropped in on Cliff McCarthy in Dallas for the night. Cliff and I enjoyed dinner and chatting, then a big shopping trip at Walmart. Came in way under budget on food and water, so either I am going to starve, or I was being pessimistic when I budgeted. Cliff reminded me that not only is everything bigger in Texas, but everything is cheaper too. Which might explain why everyone in Texas is bigger too!

I crossed two timezones in two days in my car. I’ve actually got jet-lag, and didn’t even get in a jet! But I was flying, since the traffic was light and the weather was perfect.

I will leave in the morning for Rayville, and if there is no work for me there, I will head south to Ponchatoula, LA where our group has another staging area. The group is starting to work in Bay Saint Louis MS now, so things are even more in flux than usual.

The umbrella group for our work is The team I am joining writes updates here, but they are a bit behind right now due to the upheaval of moving from Rayville to the Gulf Coast.


I am in Flagstaff right now. I will have dinner here, then hit the road to get another hour of progress eastward then bed down for the night. I will hit the road again at dawn.

Random data from the road: gas in Bakersfield was 3.35. The higest I saw was 3.65, on the border with CA and AZ. On the AZ side of the border the gas was $2.99. It is inconceiveable to me that within 10 miles the price changed by 66 cents!

Because the link to donation info got messed up in the last email, here it is again:

Change of Plans, again

I’ve gotten in touch with some great guys who are installing WiFi in shelters in Louisiana, and have decided to make a trip out there in my car to work with them.

Being outside the Palo Alto area will likely disrupt my attempts to get placed with Red Cross, so I plan to drop by the Red Cross office today and have them stop my application from going onwards. When I check in to the farm where I will be staying in LA, I will also check in with the Red Cross there and let them know that if/when it makes sense for me to transition from tech work to shelter work, I’m ready to do so.

This trip will be expensive, and I am on a tight budget, as I am still between jobs. I have set up a page here to answer the question some people have already asked me in person, “can I help with your trip?”.

I hope to drive out this afternoon and join the team Tuesday at midday.