Dell and the NSA

While I was reading this blog about how NSA’s bad-BIOS malware probably works, I was struck by a “coincidence”: Dell does significant amount of government contracting work. In fact, Ed Snowden worked for Dell at one point. NSA’s bad-BIOS targets the RAID cards in Dell servers.

Now, Dell servers are widely deployed. I’ve used them in several jobs, for example. So it’s not unreasonable that NSA would target them, to get the best bang for the buck. But it also seems possible that in order to achieve the things Dell’s executives promised to NSA executives in fancy sales calls, some Dell engineers would find themselves using what they know about Dell servers to write bad-BIOS malware to attack those very servers.

Which made me think about my company, Cisco. We publicly said we don’t put in backdoors. But we also have a big sales organization staffed with people with clearances who make special products for government organizations. It isn’t hard to imagine, especially with the revolving door between military, intelligence and defense contractors, that some of those people would find their allegiances split between intelligence people asking them for hints from the source code, and Cisco’s Code of Business Conduct.

As Bruce Schneier reminds us, once you start wondering if you can trust your suppliers, it is very hard to stop wondering.

Medium, what’s up with comments?, why do you require me to use Twitter or Facebook to comment? With all your respect for language, ideas, and design, is it really possible that you think people who choose not to use either of those services don’t have anything useful or interesting to add to your conversations?


My time-lapse camera in the attic is still working, though I resorted to adding an auto reboot once a week, because the Raspberry Pi is not acting too stable. And even then, sometimes it hangs. I blame the power supply. Because it’s always the power supply, right?

Anyway watching the sun’s track northward as spring advances has given me a much better instinctive feel for celestial mechanics. And that made me pay closer attention to the moon rise last month. After 40 years on this planet I just realized that the moon, being in the same orbital plane as the sun and earth, traces the same track as the sun. Whoa. That means my attic window is perfectly oriented to catch a nice time-lapse of the moonrise!

(Check out the planet riding with the moon up into the sky. It’s probably VEnus. Is there an astronomer in the audience that can tell me?)

Instead of putting the moon’s mechanics into my program, I took the wimpy way out and I built in the full 2014 tables for my location, which I got from USNO (Look! The US military does something other than killing people and breaking stuff. Go Navy!)

Here’s a nickle kid, get yourself a better TTY

You know you’ve been doing this too long when nothing in an article like this is new to you.

The first TTY I ever saw and (maybe) used was a TI, like this one. The librarian at my junior high borrowed it and used it to connect into Lexis-Nexis or something.

But the funny thing is, even in the late 80′s this was out of style. PCs were taking over and so I didn’t see a TTY again until university. There I saw plenty of DEC VT-100′s. And, our giant multi-processing machine from Sequent had an honest-to-god paper TTY attached to its /dev/console port. The sysadmins liked it that way so that they could see dumps on paper in the morning if the machine crashed at night. I learned about fsck that way, by watching one of them coax the filesystem back to health. (By the way, fsck is for babies. Real men fix filesystems with fsdb. Look it up.)

A few years later, I was the sysadmin for Fenris, a DEC RISCstation. This was one of the first machines using the MIPS processor, and it ran a funky Unix called Ultrix. It had a big monitor on top of it that you could use to do X11. So I figured, “whoo hoo, no more TTYs!”. But it turns out, the system had to be up and running to do windows.

Once, we needed to relink the kernel to change a tuning. Something didn’t go right. We had to fix it from /dev/console, in single user mode. The bios knew how to put characters up on the screen, but it couldn’t even emulate a VT100. It had line discipline and that was it.

To fix the computer, I had to learn the ed line editor from the printed manuals. But after reading a bit I realized immediately I knew how it worked. About 10 years before, I’d begged my mom to mail-order a word processor for my TI-99/4A called TEXTTIGER. It came on a shiny new cassette and I’d loaded it in. I was disappointed that it wasn’t WYSIWYG, but I learned how to use it’s line oriented editor. And I’d learned how to keep the content of the file in my head, and use search to move through the file instead of “goto line 1″, “next”, “next” etc.

This all likely explains why I am a vi guy today.

And why my beard is turning grey.

IPv6 in Mont-la-ville!

When I got my Raspberry Pi up and running, I reactivated my AICCU tunnel to But then I remembered that two years ago when I last touched IPv6, Swisscom was running a beta test to do IPv6 in the home.

So today I went looking to see if the test still existed and if I could join it. Why? Well, to be honest, it never even occurred to me that IPv6 to the home was in production. The lack of IPv6 uptake has become one of those “so sad it’s funny” things in our industry. But guess what?

It just worked.

Really, like totally, no questions asked. It. Just. Worked. I went to the Swisscom customer website, clicked on “turn on IPv6″, and it immediately told me my prefix was assigned as 2a02:120b:2c25:5940/60. In a few minutes, my home router had been reconfigured by Swisscom and it showed that IPv6 was turned on. Then I took a look at my computer, and it had auto-selected an address. I typed “ping6” and it worked.

So there you have it. If you are a Swisscom home DSL customer, you’ve got IPv6. Whoo hoo!

(You might need to ask Swisscom for a new router; they seem to not offer IPv6 if your router is older than about 6 months.)

Update: The router defaults to a strict IPv6 firewall, so if you want to run a server on a device, you need to login to the router and turn off the IPv6 firewall.

Live from Mont-la-ville

The last few days I’ve been working on a new home hacking project. The eventual plan is to create a panoramic time-lapse of sunrise as seen from my house each morning. We’ve got a wonderful view, and recording some of those beautiful morning colors as the sun comes up over the Alps should make them easier to appreciate — without setting my alarm for 6 AM!

We have little windows in our attic that look out over the view. And being so high up in the house, they have a commanding view over the trees in front of our house. So I knew I needed my camera mounted up there. That location, in turn, fixed some other variables. There’s power up there, but no ethernet, so the camera needs to be on wifi. (Yes, I’m kicking myself. I’m an idiot for not specifying an ethernet run up to there. But the good news is there’s a closet where I can run it, so doing it myself will be quick and easy.)

I got a camera for my Raspberry Pi. I’ve had bad experiences in the past with webcams due to cheap lenses. So when I saw a camera module with a CS mount lens included, I went for it. The specs of the Raspberry Pi camera are incredible. The fact that it is on a high speed bus direclty attached to the GPU is very interesting. It means that the GPU can accelerate video compression. So the little Raspberry Pi can still manage to stream 1080p video!

In fact, once I had it hooked up and mounted up in the window, I needed to focus it. I found some instructions on how to stream video from it. They worked flawlessly. Using VLC to watch the camera’s output, I got it aimed and focused. I loved having the stream so much, I left it running. Paste this URL into VLC to see the stream:  (IPv6 only)

Video streaming only works over IPv6, because I don’t want to fuss around with IPv4 NAT traversal settings. NATs are stupid; IPv6 is the answer. And IPv6 is getting easier and easier…

As for the time-lapses, I used Go to write a little program to that uses a library to calculate the time of sunrise and sunset (thank you Github user keep94!). My program wakes up a bit early and starts snapping images. Afterwards, it runs a script to make the time-lapse. It uses the technique shown in this post on the Ubuntu forums.

Each day’s results are currently posted here. (IPv6 only)

Looking forward to some better weather and some beautiful sunrises and sunsets!

Taking an MQTT server out for a few holes on the code golf course

A while ago at work, I needed to learn a little about MQTT. After reading the spec and finding this package that handles reading and writing the messages themselves, I decided to write a server.

The result is here:

After I finished, I wanted to know if my server was comparable to Mosquitto, so I wrote some load testing tools. I found that my Go server was generally comparable to Mosquitto, in that it had the same order of magnitude for transaction rate and latency (but it was strictly slower than Mosquitto; there is a cost to pay for the benefits Go gives you). As usual, Go programs compare quite favorably to C programs in terms of memory safety, overall length, and readability. It is also useful to consider that my implementation took only a couple of days to get finished. A C program with equivalent safety, correctness, and speed would take much longer to write. I’ve written them, I know.

Then I mentioned to my coworker Atila what I’d done. Knowing that Atila is incredibly competitive and a total D fanatic (and Go refusnik) I might have casually implied that D would have a hard time competing with my incredibly short, fast, correct, and beautiful Go version. Heh heh heh.


And so the code golf began. It only ended yesterday when Atila came back to me and begged me not to write any more benchmarks because he was tired of beating my sorry ass into the ground with his blazing fast D implementation. Fair ’nuff.

If I still have your attention…

Here are some interesting things from the experience and the code I wanted to point out.

It is often said that the solution to a problem in a domain specific language will be shorter and more correct than the solution in a general purpose language. Of course, the drawback is that you’ve got to write the DSL first, and then people who come along after you have to learn it. Implementing an MQTT server in Go really felt like writing in the correct domain specific language. It basically wrote itself, because channels and garbage collection meant that the hardest parts of a message passing server happen in the runtime instead of in the code.

Though Atila had a huge amount of fun tuning his D implementation, I didn’t do much at all to tune my implementation.

I used “go tool pprof” to fetch a CPU profile from it while it was under load from the pingtest program. But the resulting trace showed that it was spending most of its time jumping on and off of system threads. In other words the program is I/O bound, not CPU bound. It is making “too many” kernel calls, and thus speeding it up by looking for a hot spot or reducing garbage collection overhead isn’t interesting. The solution would have been to add some kind of buffering layer between the net.Conn and the message encoder/decoder. But the problem with an intervention like that is that you’d need to do it carefully so that the buffer would cut down on kernel/user context switches while there were very high transaction rates, but not introduce latency on messages when the transaction rate is low. And if the goal is massive scalability, with 1 million or more connections, each of which have a very low transaction rate, then this buffering technique wouldn’t help anything.

I did notice one nice optimization I was able to make when I was looking at the traffic with Wireshark. It is documented in this checkin. The lesson here is fewer calls to the kernel is not only a good idea for kernel/user context switch time, but may also save a significant amount of bandwidth.

As for Go techniques, I created something called a “receipt”, which lets you get synchronous feedback about a job sent in via a asynchronous mechanism. Take a look at how submitSync is used when we need to know when a certain message has departed. A receipt is a channel. The only two operations done on the channel are “make” and “close”. This makes it work like a condition variable. It flowed out of my fingers at the time; I don’t know if it is an idiom I’ll use again. It certainly didn’t give me any problems, so I guess I’ll keep it!

I don’t know or care where this code will go. If I receive pull requests, I’ll act on them. If someone needs to depend on this for a production-quality system, they might want to fork it and take long-term ownership of it. That’s OK with me too.