Go will make you a better programmer

The last line of Dave Cheny’s Gophercon 2015 India keynote is the best: “Go will make you a better programmer.”

It’s true. When I am programming Go, I never think, “OK, is this an OK shortcut to take here? Is this a play context? Is this a work context, but I can push off bounds checking on some other layer? Is this just a little local server, and I don’t care about hackers?”

These questions don’t exist, because the way Go and its standard library and its idioms work together, the right way — the easy way! — to write it is simply, clearly, and securely.

Go makes me a better programmer.

And it makes me increasingly intolerant of C and Python. This week alone at work I struggled with a crashing router from a missing C bounds check and late- and strangely-failing Python code from dynamic typing. Idiomatic Go code does not have those two failure modes, and my patience for them is waning fast as a result.

A Quick Go hack: lines-per-second

Today I wanted to compare how fast two builds were on two different machines. As a rough estimate, I just wanted to see how many lines per second were going into the log file. This is what I came up with:

package main

import (
        "bufio"
        "fmt"
        "os"
        "time"
)

func main() {
        scn := bufio.NewScanner(os.Stdin)

        lines := 0

        go func() {
                for {
                        fmt.Fprintf(os.Stderr, "%v lps\n", lines)
                        lines = 0
                        time.Sleep(time.Second)
                }
        }()

        for scn.Scan() {
                lines++
        }

}

This is a quick hack. It’s not production quality, because there is a data race on lines. I think if I had to fix that race up, I’d choose to use sync/atomic’s ops. This because I’d only need to hit the two places lines is touched concurrently. A channel-based solution would be bigger and messier, not in tune with the minimal nature of this little program.

The equivalent program in Python or Perl or whatever would have been as short and sweet. That’s not the point. The point is that it was so easy to do in Go, I just did it in go without really even considering an alternative.

Golang on the Geode processor

If you are trying to use Golang on a PC-Engines Alix board, you need to be careful that all the Go code you are using is compiled while the GO386 environment variable is set to 387. The Geode processor does not support the SSE instructions.

If you have Linux directly on the Alix, you’d not run into this because if GO386 is not set, then the compiler auto-detects the right thing. But if you are, for example, running OpenWRT on the device, and you are using cross-compilation to build on another device, you might run into this if some of your Go code (for example, the std lib) was compiled without GO386.

The symptom you’ve given yourself this problem is a traceback like this:

# ./sms.test
SIGILL: illegal instruction
PC=0x813db39

goroutine 1 [running, locked to thread]:
math.init·1()
/home/jra/go/src/math/pow10.go:34 +0x19 fp=0x1862df64 sp=0x1862df60
math.init()

The solution is that when you initialize cross compilation, you need to do it like this:


$ cd $HOME/go/src
$ GO386=387 GOARCH=386 ./make.bash --no-clean

And all of your compiles, like go test -c need to have the GO386 environment variable set as well, thus:


$ cd $GOROOT/src/github.com/jeffallen/linkio
$ GO386=387 GOARCH=386 go test -c

(If your code doesn’t use floating point, you might dodge a bullet if you forget to set GO386. But don’t say I didn’t warn you…)

Type safety saves the day again

Recently, I was writing some code to check the SHA256 of a buffer. Simple, right? All you have to do is take the hash you have, get the hash of the data, and compare them. But then you think, “oh, what a drag, I gotta compare two []byte values for equality, and == doesn’t work on them”.

And then you think, “oh, I’ll use reflection!” And now you have two problems.

package main

import (
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"reflect"
)

func main() {
	in := []byte("a test string")
	hashHex := "b830543dc5d1466110538736d35c37cc61d32076a69de65c42913dfbb1961f46"
	hash1, _ := hex.DecodeString(hashHex)
	hash2 := sha256.Sum256(in)
	fmt.Println("equal?", reflect.DeepEqual(hash1, hash2))
}

The code on play.golang.org.

When you run that, you find out that even though the hash is most assuredly right, reflect.DeepEqual returns “false”.

If you’ve just written this as new code, in the context of some other new code that gets and parses the hash, you might not go looking at that reflect.DeepEqual until you’ve checked and rechecked everything else. And by then you’ll be really annoyed.

You, the reader of this blog, can probably tell you’re being set up by now. And you are.

The thing about the reflect package is that it is all about runtime type checking. And the thing about runtime is that it happens later, when you, the programmer, and the compiler are not available to apply your respective unique talents. So bad things can happen, and the compiler can’t save you (and it goes without saying that you can’t save yourself, because you are a fallible human).

It turns out the type of hash2 is not []byte as I lead you to think. Go read the docs and you’ll see that it’s type is [32]byte.

While you are in the docs, you might notice bytes.Compare. It is the solution to our problem. Because it uses static types instead of interface{} like the reflect package, the compiler is going to be able to help you use it correctly. And when you try to use it:

package main

import (
	"bytes"
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"reflect"
)

func main() {
	in := []byte("a test string")
	hashHex := "b830543dc5d1466110538736d35c37cc61d32076a69de65c42913dfbb1961f46"
	hash1, _ := hex.DecodeString(hashHex)
	hash2 := sha256.Sum256(in)
	fmt.Println("equal?", reflect.DeepEqual(hash1, hash2))
	fmt.Println("equal?", bytes.Compare(hash1, hash2) == 0)
}

The code on play.golang.org.

This gives the helpful error message cannot use hash2 (type [32]byte) as type []byte in argument to bytes.Compare. Which, at a stroke, explains why reflect.DeepEqual is screwing up: the first check it is making is “are these the same type?” And the answer is no, so hash1 and hash2 are not equal. Even though they are.

In order to turn hash2 into a []byte so that it can be the second argument to bytes.Compare, you just need to take a full slice of it, changing it to hash2[:].

The final, working, hash compare is here.

Moonrise

My time-lapse camera in the attic is still working, though I resorted to adding an auto reboot once a week, because the Raspberry Pi is not acting too stable. And even then, sometimes it hangs. I blame the power supply. Because it’s always the power supply, right?

Anyway watching the sun’s track northward as spring advances has given me a much better instinctive feel for celestial mechanics. And that made me pay closer attention to the moon rise last month. After 40 years on this planet I just realized that the moon, being in the same orbital plane as the sun and earth, traces the same track as the sun. Whoa. That means my attic window is perfectly oriented to catch a nice time-lapse of the moonrise!

(Check out the planet riding with the moon up into the sky. It’s probably VEnus. Is there an astronomer in the audience that can tell me?)

Instead of putting the moon’s mechanics into my program, I took the wimpy way out and I built in the full 2014 tables for my location, which I got from USNO (Look! The US military does something other than killing people and breaking stuff. Go Navy!)

Live from Mont-la-ville

The last few days I’ve been working on a new home hacking project. The eventual plan is to create a panoramic time-lapse of sunrise as seen from my house each morning. We’ve got a wonderful view, and recording some of those beautiful morning colors as the sun comes up over the Alps should make them easier to appreciate — without setting my alarm for 6 AM!

We have little windows in our attic that look out over the view. And being so high up in the house, they have a commanding view over the trees in front of our house. So I knew I needed my camera mounted up there. That location, in turn, fixed some other variables. There’s power up there, but no ethernet, so the camera needs to be on wifi. (Yes, I’m kicking myself. I’m an idiot for not specifying an ethernet run up to there. But the good news is there’s a closet where I can run it, so doing it myself will be quick and easy.)

I got a camera for my Raspberry Pi. I’ve had bad experiences in the past with webcams due to cheap lenses. So when I saw a camera module with a CS mount lens included, I went for it. The specs of the Raspberry Pi camera are incredible. The fact that it is on a high speed bus direclty attached to the GPU is very interesting. It means that the GPU can accelerate video compression. So the little Raspberry Pi can still manage to stream 1080p video!

In fact, once I had it hooked up and mounted up in the window, I needed to focus it. I found some instructions on how to stream video from it. They worked flawlessly. Using VLC to watch the camera’s output, I got it aimed and focused. I loved having the stream so much, I left it running. Paste this URL into VLC to see the stream: http://pi.nella.org:8554  (IPv6 only; and it’s not up anymore anyway)

Video streaming only works over IPv6, because I don’t want to fuss around with IPv4 NAT traversal settings. NATs are stupid; IPv6 is the answer. And IPv6 is getting easier and easier…

As for the time-lapses, I used Go to write a little program to that uses a library to calculate the time of sunrise and sunset (thank you Github user keep94!). My program wakes up a bit early and starts snapping images. Afterwards, it runs a script to make the time-lapse. It uses the technique shown in this post on the Ubuntu forums.

Each day’s results are currently posted here. (IPv6 only)

Looking forward to some better weather and some beautiful sunrises and sunsets!

Taking an MQTT server out for a few holes on the code golf course

A while ago at work, I needed to learn a little about MQTT. After reading the spec and finding this package that handles reading and writing the messages themselves, I decided to write a server.

The result is here: https://github.com/jeffallen/mqtt.

After I finished, I wanted to know if my server was comparable to Mosquitto, so I wrote some load testing tools. I found that my Go server was generally comparable to Mosquitto, in that it had the same order of magnitude for transaction rate and latency (but it was strictly slower than Mosquitto; there is a cost to pay for the benefits Go gives you). As usual, Go programs compare quite favorably to C programs in terms of memory safety, overall length, and readability. It is also useful to consider that my implementation took only a couple of days to get finished. A C program with equivalent safety, correctness, and speed would take much longer to write. I’ve written them, I know.

Then I mentioned to my coworker Atila what I’d done. Knowing that Atila is incredibly competitive and a total D fanatic (and Go refusnik) I might have casually implied that D would have a hard time competing with my incredibly short, fast, correct, and beautiful Go version. Heh heh heh.

FORE!

And so the code golf began. It only ended yesterday when Atila came back to me and begged me not to write any more benchmarks because he was tired of beating my sorry ass into the ground with his blazing fast D implementation. Fair ’nuff.

If I still have your attention…

Here are some interesting things from the experience and the code I wanted to point out.

It is often said that the solution to a problem in a domain specific language will be shorter and more correct than the solution in a general purpose language. Of course, the drawback is that you’ve got to write the DSL first, and then people who come along after you have to learn it. Implementing an MQTT server in Go really felt like writing in the correct domain specific language. It basically wrote itself, because channels and garbage collection meant that the hardest parts of a message passing server happen in the runtime instead of in the code.

Though Atila had a huge amount of fun tuning his D implementation, I didn’t do much at all to tune my implementation.

I used “go tool pprof” to fetch a CPU profile from it while it was under load from the pingtest program. But the resulting trace showed that it was spending most of its time jumping on and off of system threads. In other words the program is I/O bound, not CPU bound. It is making “too many” kernel calls, and thus speeding it up by looking for a hot spot or reducing garbage collection overhead isn’t interesting. The solution would have been to add some kind of buffering layer between the net.Conn and the message encoder/decoder. But the problem with an intervention like that is that you’d need to do it carefully so that the buffer would cut down on kernel/user context switches while there were very high transaction rates, but not introduce latency on messages when the transaction rate is low. And if the goal is massive scalability, with 1 million or more connections, each of which have a very low transaction rate, then this buffering technique wouldn’t help anything.

I did notice one nice optimization I was able to make when I was looking at the traffic with Wireshark. It is documented in this checkin. The lesson here is fewer calls to the kernel is not only a good idea for kernel/user context switch time, but may also save a significant amount of bandwidth.

As for Go techniques, I created something called a “receipt”, which lets you get synchronous feedback about a job sent in via a asynchronous mechanism. Take a look at how submitSync is used when we need to know when a certain message has departed. A receipt is a channel. The only two operations done on the channel are “make” and “close”. This makes it work like a condition variable. It flowed out of my fingers at the time; I don’t know if it is an idiom I’ll use again. It certainly didn’t give me any problems, so I guess I’ll keep it!

I don’t know or care where this code will go. If I receive pull requests, I’ll act on them. If someone needs to depend on this for a production-quality system, they might want to fork it and take long-term ownership of it. That’s OK with me too.

Qz for Mobile Phones

I love Quartz. however, I find that the best time/place for me to read it is on the train during my commute. And Quartz’s overly clever and overly dynamic website hates my phone’s browser. So I made Mobile Quartz to help me read it. The front page has the current top articles from Quartz. Click the links to see the article.

I used this as a chance to write some Go on Google Appspot code, and it was really a pleasure.

Zero Downtime upgrades of TCP servers in Go

A recent post on the golang-nuts mailing list mentioned that Nginx can upgrade on the fly without ever stopping listening to it’s listen socket. The trick is to unset close-on-exec on the listen socket, then fork/exec a new copy of the server (on the upgraded binary) with an argument to tell it to use the inherited file descriptor instead of calling socket() and listen(s).

I wanted to see if I could achieve the same thing with Go, and what changes would be necessary to the standard libraries to make this possible. I got it working, without changing the standard library, so I wanted to explain what I did here.

The code for this post is here.

There are several interesting things I’d like to point out in this program. Twice, I used the pattern of “implement an interface in order to intercept method calls”. This is an important pattern in Go that I don’t think is widely understood and documented.

When I started thinking about this job I knew one of my problems would be to dig down inside of http.(*Server).Serve in order to get it to stop calling Accept() when the old server should shutdown gracefully. The problem is that there’s no hooks in there; the only way out of the loop (“accept, start go routine to process, do it again”) is for Accept to return an error. But if you think of Accept as a system call, you might think, “I can’t get in there and inject an error”. But Accept() is not a system call: it’s a method in the interface net.Listener. Which means that if you make your own object which satisfies net.Listener, you can pass that in to http.(*Server).Serve and do what you want in Accept().

The first time I read about embedding types in structures I got lost and confused. And when I tried it, I had pointers in the mix and I had lots of unexplained nil pointer errors. This time, I read it again and it made sense. Type embedding is essential when you want to interpose one of the methods of an interface. It lets you inherit all the implementations of the underlying object except for the one that you redefine. Take a look at stoppableListener in upgradable.go. The net.Listener interface requires three methods including Accept, Close, and Addr. But I only defined one of those, Accept(). How is it that stoppableListener still implements net.Listener? Because the other two methods “show through” from where they were embedded in it. Only Accept() has a more precise definition. When I wrote Accept(), then I needed to figure out how to talk to the underlying object, in order to pass on the Accept() call. The trick here is to understand that embedding a type creates a new field in your structure with the unqualified name of the type. So I can refer to the net.Listener inside of stoppableListener sl as sl.Listener, and I can call the underlying Accept() as sl.Listener.Accept().

Next I started wondering how to handle the “stopped” error code from Serve(). Immediately exiting with os.Exit(0) isn’t right, because there can still be go routines servicing HTTP clients. We need some way to know when each client is done. Interposition to the rescue again, since we can wrap up the net.Conn returned by Accept() and use it to decrement a count of the number of currently running connections. This technique of interposing the net.Conn object could have other interesting uses. For example, by trapping the Read() or Write() calls, you could enforce rate limiting on the connection, without needing the protocol implementation to know anything about it. You could even do other zany things like implement opportunistic encryption, again without the protocol knowing it was happening.

Once I knew that I would be able to stop my server gracefully, I needed to figure out how to start the new one on the correct file descriptor. Rog Peppe pointed me to the net.FileListener object, which one can construct from an *os.File (which you can make from a raw file descriptor in an int using os.NewFile).

The final problem is that net always sets the close-on-exec flag on file descriptors for sockets it opens. So I needed to turn that off on the listen socket, so that the file descriptor would still be valid in the new process. Unfortunately syscall.CloseOnExec does not take a boolean to tell it what you want (do close or don’t close). So instead, I pulled out the stuff from syscall that I needed and put it directly in upgradable.go. Not pretty, but nicer than hacking the standard library. (This is a nice thing about Go: there’s almost always a way to do what you want, even if you have to hack around just a bit.)

I tested that it works manually (using command-line GETs from another window). I also tested it under load using http_load. It is really cool to be able to set up a load test for 20 seconds and get 3937 fetches/sec, then do the test again, adding in a few “GET http://localhost:8000/upgrade” from another window and still getting 3880 fetches/sec, even as I replaced the running binary a few times during the load test!