Golang Challenge 2 comments

I’ve just finished evaluating 40 of the 105 entries to the Golang Challenge #2. The organizer, Satish, asked me to write up my thoughts.

The main similarity I noticed in the entries was not enough testing. The vast majority of the entries used the tests provided in with the challenge unmodified. Taking the given tests without thinking critically about them lead people to make a number of critical mistakes, over and over again. The majority of the entries I graded passed the tests, but would not have stood up to production use. Of the 40 I graded, only 2 or 3 would have received a “ship it” from me in my day job. Those were (not by chance) the ones with extra tests beyond the ones provided in the challenge.

A key part of growing into a mature programmer is to understand that the problem as it arrives in your “todo basket” is rarely correctly specified. This challenge is no exception. The challenge asked participants to “make the given tests pass”, but didn’t mention if you could or should write extra tests. The bottom line is that you can always write extra tests, and by doing so, you will discover where the spec is weak, or wrong. You will also decide for yourself where the “edge” of the task is; thinking about tests you can and should write will help fix the size of the job at hand.

What I’m describing here is test-driven development. Go’s tools make it easy to stay on the golden path of TDD, you just need to decide that you believe in its benefits, and then teach yourself the discipline to do it. In TDD, the tests push the implementation away from the hacky thing that seems to work towards the robust thing that actually does work, that lives alongside of proof that it works (the tests) and that will keep working even as the world changes around it (because you’ll fix the code when the tests start failing).

Two deficiencies in the given tests were that they did not test legitimate network behaviors like short reads, and they gave the impression that the protocol should be one-shot (i.e. set up a TCP connection, exchange public keys, echo one message and close). Given that the problem statement asked for an io.Reader and and io.Writer that encrypted data, it stands to reason that a good solution would be able to handle more than one message, since io.Reader/io.Writer are not one-shot interfaces.

A common problem in TCP servers is short reads. Just because the sender put 1000 bytes into the TCP stream via one write system call does not mean you are guaranteed to receive those same 1000 bytes in a matching read call. To get that behavior, you’d need a reliable transaction protocol like SCTP. With TCP, all you are guaranteed is that the same bytes will arrive at the far end, in the same order.

To see where a short read can come from, we need to look at it one packet at a time. Imagine what happens if the sender gives the kernel 2000 bytes, then sleeps on a the read of a reply. If the MTU on the link is 1500 bytes, the TCP implementation might send out one packet with about 1500 bytes in it, and a second with 500 (the “about” is hand waving for the TCP/IP overhead we’re ignoring here). Now imagine that the second packet is lost, and the replacement is lost as well. On the third try it gets through. On the receiving end, what we see is 1000 bytes which arrive, followed by a pause and 500 more. If the pause is long enough, the kernel will decide to stop making the application wait around for the first 1000 bytes and pass them up. The reader, blocked in the recv() system call, will get unblocked and receive the 1000 bytes. The next time it blocks on recv(), it will get the enxt 500 bytes. This is one of many explanations for how you can get a short read. There’s no guarantees, and code that expects there to be is wrong and will fail in real life (but not in your fast, non-lossy, testing environment; in particular, if you test via localhost, you’re very unlikely to ever see short reads).

But short reads could some from someplace much simpler… from the tests. Consider how your challenge entry would have behaved if it was reading from an io.Reader that intentionally returned only one byte at a time.

Given the reality of short reads, it is up to the application to handle framing the bytes into messages. The majority of entries did not take this into consideration. The solution is that each message needs to be sent with it’s length. And when SecureReader reads from the underlying io.Reader, it needs to use something like ioutil.ReadAll to make certain all the bytes arrive, no matter how many calls to the underlying io.Reader it takes to get them all. But then that opens up the question of what to do when the connection hangs? Do you timeout? How can you cause ioutil.ReadAll to return early?

In Go, different pieces of the language and the standard library can be composed to make something better than any of them alone. A great way to read frames from the network is to prepend the frame with it’s length. You read the length using encoding/binary.Read, make an io.LimitedReader which is limited to the length, then give that LimitedReader to ioutil.ReadAll. You can handle timeouts by checking that if the underlying io.Reader satisfies an interface with SetReadDeadline in it (as the various connection types in package net do), and then using it to set a deadline. After the deadline passes, the underlying io.Reader will return a timeout error, which the LimitedReader will return to ioutil.ReadAll, which will cause it to return early, telling you that the read timed out.

The other thing I noticed is that even in a language designed for simplicity, there are a myriad of ways to do things too difficultly in Go. In other words, go is not WTF-proof. *

The only solution to this problem is that you need to read lots and lots of code (way, way more than your write!). Read the highest quality code you can find. Code from the Go core team is a very good choice (the standard library itself and side projects by Andrew, Russ, and Brad). You can also read the reviews of potential new code for the Go core (read the golang-dev mailing list to find out where the reviews are). By watching other people get their screw ups corrected, you can avoid making them in the first place! It is much easier on the ego too, to watch other people get corrected for a mistake you would have made. ๐Ÿ™‚

* In order to not single people out here in public I’m not going to talk about the specific WTF’s I saw. But if you’d be willing to have your code publicly mocked by me, drop me a line with a pointer to your challenge solution. If there’s a WTF in there, I’ll post about it and you’ll be infamous, and we can all have a good laugh… ๐Ÿ™‚

Building Go 1.4 when the linker doesn’t know about build-id

Today at work, on a Redhat 5.5 machine, I tried to build Go 1.4.

This happened:


$ cd go1.4/src
$ ./all.bash
...snip...
# runtime/cgo
/usr/bin/ld: unrecognized option '--build-id=none'
/usr/bin/ld: use the --help option for usage information
collect2: ld returned 1 exit status

The solution is to retry without the “–build-id=none” option:

diff --git a/src/cmd/go/build.go b/src/cmd/go/build.go
index ad03239..ca45217 100644
--- a/src/cmd/go/build.go
+++ b/src/cmd/go/build.go
@@ -2436,13 +2436,21 @@ func (b *builder) cgo(p *Package, cgoExe, obj string, pcCFLAGS, pcLDFLAGS, cgofi
 	// --build-id=none.  So that is what we do, but only on
 	// systems likely to support it, which is to say, systems that
 	// normally use gold or the GNU linker.
+	retryWithoutBuildId := false
 	switch goos {
 	case "android", "dragonfly", "linux", "netbsd":
 		ldflags = append(ldflags, "-Wl,--build-id=none")
+		retryWithoutBuildId = true
 	}
 
 	if err := b.gccld(p, ofile, ldflags, gccObjs); err != nil {
-		return nil, nil, err
+		if retryWithoutBuildId {
+			ldflags = ldflags[0:len(ldflags)-1]
+			err = b.gccld(p, ofile, ldflags, gccObjs)
+		}
+		if err != nil {
+			return nil, nil, err
+		}
 	}
 
 	// NOTE(rsc): The importObj is a 5c/6c/8c object and on Windows

Just in case someone else is looking for it… ๐Ÿ™‚

Go does not officially support RHEL before version 6, but it does seem to sort of work. This post explains why it doesn’t really work.

Go will make you a better programmer

The last line of Dave Cheny’s Gophercon 2015 India keynote is the best: “Go will make you a better programmer.”

It’s true. When I am programming Go, I never think, “OK, is this an OK shortcut to take here? Is this a play context? Is this a work context, but I can push off bounds checking on some other layer? Is this just a little local server, and I don’t care about hackers?”

These questions don’t exist, because the way Go and its standard library and its idioms work together, the right way — the easy way! — to write it is simply, clearly, and securely.

Go makes me a better programmer.

And it makes me increasingly intolerant of C and Python. This week alone at work I struggled with a crashing router from a missing C bounds check and late- and strangely-failing Python code from dynamic typing. Idiomatic Go code does not have those two failure modes, and my patience for them is waning fast as a result.

A Quick Go hack: lines-per-second

Today I wanted to compare how fast two builds were on two different machines. As a rough estimate, I just wanted to see how many lines per second were going into the log file. This is what I came up with:

package main

import (
        "bufio"
        "fmt"
        "os"
        "time"
)

func main() {
        scn := bufio.NewScanner(os.Stdin)

        lines := 0

        go func() {
                for {
                        fmt.Fprintf(os.Stderr, "%v lps\n", lines)
                        lines = 0
                        time.Sleep(time.Second)
                }
        }()

        for scn.Scan() {
                lines++
        }

}

This is a quick hack. It’s not production quality, because there is a data race on lines. I think if I had to fix that race up, I’d choose to use sync/atomic’s ops. This because I’d only need to hit the two places lines is touched concurrently. A channel-based solution would be bigger and messier, not in tune with the minimal nature of this little program.

The equivalent program in Python or Perl or whatever would have been as short and sweet. That’s not the point. The point is that it was so easy to do in Go, I just did it in go without really even considering an alternative.

Golang on the Geode processor

If you are trying to use Golang on a PC-Engines Alix board, you need to be careful that all the Go code you are using is compiled while the GO386 environment variable is set to 387. The Geode processor does not support the SSE instructions.

If you have Linux directly on the Alix, you’d not run into this because if GO386 is not set, then the compiler auto-detects the right thing. But if you are, for example, running OpenWRT on the device, and you are using cross-compilation to build on another device, you might run into this if some of your Go code (for example, the std lib) was compiled without GO386.

The symptom you’ve given yourself this problem is a traceback like this:

# ./sms.test
SIGILL: illegal instruction
PC=0x813db39

goroutine 1 [running, locked to thread]:
math.initยท1()
/home/jra/go/src/math/pow10.go:34 +0x19 fp=0x1862df64 sp=0x1862df60
math.init()

The solution is that when you initialize cross compilation, you need to do it like this:


$ cd $HOME/go/src
$ GO386=387 GOARCH=386 ./make.bash --no-clean

And all of your compiles, like go test -c need to have the GO386 environment variable set as well, thus:


$ cd $GOROOT/src/github.com/jeffallen/linkio
$ GO386=387 GOARCH=386 go test -c

(If your code doesn’t use floating point, you might dodge a bullet if you forget to set GO386. But don’t say I didn’t warn you…)

Type safety saves the day again

Recently, I was writing some code to check the SHA256 of a buffer. Simple, right? All you have to do is take the hash you have, get the hash of the data, and compare them. But then you think, “oh, what a drag, I gotta compare two []byte values for equality, and == doesn’t work on them”.

And then you think, “oh, I’ll use reflection!” And now you have two problems.

package main

import (
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"reflect"
)

func main() {
	in := []byte("a test string")
	hashHex := "b830543dc5d1466110538736d35c37cc61d32076a69de65c42913dfbb1961f46"
	hash1, _ := hex.DecodeString(hashHex)
	hash2 := sha256.Sum256(in)
	fmt.Println("equal?", reflect.DeepEqual(hash1, hash2))
}

The code on play.golang.org.

When you run that, you find out that even though the hash is most assuredly right, reflect.DeepEqual returns “false”.

If you’ve just written this as new code, in the context of some other new code that gets and parses the hash, you might not go looking at that reflect.DeepEqual until you’ve checked and rechecked everything else. And by then you’ll be really annoyed.

You, the reader of this blog, can probably tell you’re being set up by now. And you are.

The thing about the reflect package is that it is all about runtime type checking. And the thing about runtime is that it happens later, when you, the programmer, and the compiler are not available to apply your respective unique talents. So bad things can happen, and the compiler can’t save you (and it goes without saying that you can’t save yourself, because you are a fallible human).

It turns out the type of hash2 is not []byte as I lead you to think. Go read the docs and you’ll see that it’s type is [32]byte.

While you are in the docs, you might notice bytes.Compare. It is the solution to our problem. Because it uses static types instead of interface{} like the reflect package, the compiler is going to be able to help you use it correctly. And when you try to use it:

package main

import (
	"bytes"
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"reflect"
)

func main() {
	in := []byte("a test string")
	hashHex := "b830543dc5d1466110538736d35c37cc61d32076a69de65c42913dfbb1961f46"
	hash1, _ := hex.DecodeString(hashHex)
	hash2 := sha256.Sum256(in)
	fmt.Println("equal?", reflect.DeepEqual(hash1, hash2))
	fmt.Println("equal?", bytes.Compare(hash1, hash2) == 0)
}

The code on play.golang.org.

This gives the helpful error message cannot use hash2 (type [32]byte) as type []byte in argument to bytes.Compare. Which, at a stroke, explains why reflect.DeepEqual is screwing up: the first check it is making is “are these the same type?” And the answer is no, so hash1 and hash2 are not equal. Even though they are.

In order to turn hash2 into a []byte so that it can be the second argument to bytes.Compare, you just need to take a full slice of it, changing it to hash2[:].

The final, working, hash compare is here.

Moonrise

My time-lapse camera in the attic is still working, though I resorted to adding an auto reboot once a week, because the Raspberry Pi is not acting too stable. And even then, sometimes it hangs. I blame the power supply. Because it’s always the power supply, right?

Anyway watching the sun’s track northward as spring advances has given me a much better instinctive feel for celestial mechanics. And that made me pay closer attention to the moon rise last month. After 40 years on this planet I just realized that the moon, being in the same orbital plane as the sun and earth, traces the same track as the sun. Whoa. That means my attic window is perfectly oriented to catch a nice time-lapse of the moonrise!

(Check out the planet riding with the moon up into the sky. It’s probably VEnus. Is there an astronomer in the audience that can tell me?)

Instead of putting the moon’s mechanics into my program, I took the wimpy way out and I built in the full 2014 tables for my location, which I got from USNO (Look! The US military does something other than killing people and breaking stuff. Go Navy!)

Live from Mont-la-ville

The last few days I’ve been working on a new home hacking project. The eventual plan is to create a panoramic time-lapse of sunrise as seen from my house each morning. We’ve got a wonderful view, and recording some of those beautiful morning colors as the sun comes up over the Alps should make them easier to appreciate — without setting my alarm for 6 AM!

We have little windows in our attic that look out over the view. And being so high up in the house, they have a commanding view over the trees in front of our house. So I knew I needed my camera mounted up there. That location, in turn, fixed some other variables. There’s power up there, but no ethernet, so the camera needs to be on wifi. (Yes, I’m kicking myself. I’m an idiot for not specifying an ethernet run up to there. But the good news is there’s a closet where I can run it, so doing it myself will be quick and easy.)

I got a camera for my Raspberry Pi. I’ve had bad experiences in the past with webcams due to cheap lenses. So when I saw a camera module with a CS mount lens included, I went for it. The specs of the Raspberry Pi camera are incredible. The fact that it is on a high speed bus direclty attached to the GPU is very interesting. It means that the GPU can accelerate video compression. So the little Raspberry Pi can still manage to stream 1080p video!

In fact, once I had it hooked up and mounted up in the window, I needed to focus it. I found some instructions on how to stream video from it. They worked flawlessly. Using VLC to watch the camera’s output, I got it aimed and focused. I loved having the stream so much, I left it running. Paste this URL into VLC to see the stream: http://pi.nella.org:8554ย  (IPv6 only; and it’s not up anymore anyway)

Video streaming only works over IPv6, because I don’t want to fuss around with IPv4 NAT traversal settings. NATs are stupid; IPv6 is the answer. And IPv6 is getting easier and easier…

As for the time-lapses, I used Go to write a little program to that uses a library to calculate the time of sunrise and sunset (thank you Github user keep94!). My program wakes up a bit early and starts snapping images. Afterwards, it runs a script to make the time-lapse. It uses the technique shown in this post on the Ubuntu forums.

Each day’s results are currently posted here. (IPv6 only)

Looking forward to some better weather and some beautiful sunrises and sunsets!