Building Go 1.4 when the linker doesn’t know about build-id

Today at work, on a Redhat 5.5 machine, I tried to build Go 1.4.

This happened:


$ cd go1.4/src
$ ./all.bash
...snip...
# runtime/cgo
/usr/bin/ld: unrecognized option '--build-id=none'
/usr/bin/ld: use the --help option for usage information
collect2: ld returned 1 exit status

The solution is to retry without the “–build-id=none” option:

diff --git a/src/cmd/go/build.go b/src/cmd/go/build.go
index ad03239..ca45217 100644
--- a/src/cmd/go/build.go
+++ b/src/cmd/go/build.go
@@ -2436,13 +2436,21 @@ func (b *builder) cgo(p *Package, cgoExe, obj string, pcCFLAGS, pcLDFLAGS, cgofi
 	// --build-id=none.  So that is what we do, but only on
 	// systems likely to support it, which is to say, systems that
 	// normally use gold or the GNU linker.
+	retryWithoutBuildId := false
 	switch goos {
 	case "android", "dragonfly", "linux", "netbsd":
 		ldflags = append(ldflags, "-Wl,--build-id=none")
+		retryWithoutBuildId = true
 	}
 
 	if err := b.gccld(p, ofile, ldflags, gccObjs); err != nil {
-		return nil, nil, err
+		if retryWithoutBuildId {
+			ldflags = ldflags[0:len(ldflags)-1]
+			err = b.gccld(p, ofile, ldflags, gccObjs)
+		}
+		if err != nil {
+			return nil, nil, err
+		}
 	}
 
 	// NOTE(rsc): The importObj is a 5c/6c/8c object and on Windows

Just in case someone else is looking for it… 🙂

Go does not officially support RHEL before version 6, but it does seem to sort of work. This post explains why it doesn’t really work.

Go will make you a better programmer

The last line of Dave Cheny’s Gophercon 2015 India keynote is the best: “Go will make you a better programmer.”

It’s true. When I am programming Go, I never think, “OK, is this an OK shortcut to take here? Is this a play context? Is this a work context, but I can push off bounds checking on some other layer? Is this just a little local server, and I don’t care about hackers?”

These questions don’t exist, because the way Go and its standard library and its idioms work together, the right way — the easy way! — to write it is simply, clearly, and securely.

Go makes me a better programmer.

And it makes me increasingly intolerant of C and Python. This week alone at work I struggled with a crashing router from a missing C bounds check and late- and strangely-failing Python code from dynamic typing. Idiomatic Go code does not have those two failure modes, and my patience for them is waning fast as a result.

A Quick Go hack: lines-per-second

Today I wanted to compare how fast two builds were on two different machines. As a rough estimate, I just wanted to see how many lines per second were going into the log file. This is what I came up with:

package main

import (
        "bufio"
        "fmt"
        "os"
        "time"
)

func main() {
        scn := bufio.NewScanner(os.Stdin)

        lines := 0

        go func() {
                for {
                        fmt.Fprintf(os.Stderr, "%v lps\n", lines)
                        lines = 0
                        time.Sleep(time.Second)
                }
        }()

        for scn.Scan() {
                lines++
        }

}

This is a quick hack. It’s not production quality, because there is a data race on lines. I think if I had to fix that race up, I’d choose to use sync/atomic’s ops. This because I’d only need to hit the two places lines is touched concurrently. A channel-based solution would be bigger and messier, not in tune with the minimal nature of this little program.

The equivalent program in Python or Perl or whatever would have been as short and sweet. That’s not the point. The point is that it was so easy to do in Go, I just did it in go without really even considering an alternative.

Golang on the Geode processor

If you are trying to use Golang on a PC-Engines Alix board, you need to be careful that all the Go code you are using is compiled while the GO386 environment variable is set to 387. The Geode processor does not support the SSE instructions.

If you have Linux directly on the Alix, you’d not run into this because if GO386 is not set, then the compiler auto-detects the right thing. But if you are, for example, running OpenWRT on the device, and you are using cross-compilation to build on another device, you might run into this if some of your Go code (for example, the std lib) was compiled without GO386.

The symptom you’ve given yourself this problem is a traceback like this:

# ./sms.test
SIGILL: illegal instruction
PC=0x813db39

goroutine 1 [running, locked to thread]:
math.init·1()
/home/jra/go/src/math/pow10.go:34 +0x19 fp=0x1862df64 sp=0x1862df60
math.init()

The solution is that when you initialize cross compilation, you need to do it like this:


$ cd $HOME/go/src
$ GO386=387 GOARCH=386 ./make.bash --no-clean

And all of your compiles, like go test -c need to have the GO386 environment variable set as well, thus:


$ cd $GOROOT/src/github.com/jeffallen/linkio
$ GO386=387 GOARCH=386 go test -c

(If your code doesn’t use floating point, you might dodge a bullet if you forget to set GO386. But don’t say I didn’t warn you…)

Type safety saves the day again

Recently, I was writing some code to check the SHA256 of a buffer. Simple, right? All you have to do is take the hash you have, get the hash of the data, and compare them. But then you think, “oh, what a drag, I gotta compare two []byte values for equality, and == doesn’t work on them”.

And then you think, “oh, I’ll use reflection!” And now you have two problems.

package main

import (
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"reflect"
)

func main() {
	in := []byte("a test string")
	hashHex := "b830543dc5d1466110538736d35c37cc61d32076a69de65c42913dfbb1961f46"
	hash1, _ := hex.DecodeString(hashHex)
	hash2 := sha256.Sum256(in)
	fmt.Println("equal?", reflect.DeepEqual(hash1, hash2))
}

The code on play.golang.org.

When you run that, you find out that even though the hash is most assuredly right, reflect.DeepEqual returns “false”.

If you’ve just written this as new code, in the context of some other new code that gets and parses the hash, you might not go looking at that reflect.DeepEqual until you’ve checked and rechecked everything else. And by then you’ll be really annoyed.

You, the reader of this blog, can probably tell you’re being set up by now. And you are.

The thing about the reflect package is that it is all about runtime type checking. And the thing about runtime is that it happens later, when you, the programmer, and the compiler are not available to apply your respective unique talents. So bad things can happen, and the compiler can’t save you (and it goes without saying that you can’t save yourself, because you are a fallible human).

It turns out the type of hash2 is not []byte as I lead you to think. Go read the docs and you’ll see that it’s type is [32]byte.

While you are in the docs, you might notice bytes.Compare. It is the solution to our problem. Because it uses static types instead of interface{} like the reflect package, the compiler is going to be able to help you use it correctly. And when you try to use it:

package main

import (
	"bytes"
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"reflect"
)

func main() {
	in := []byte("a test string")
	hashHex := "b830543dc5d1466110538736d35c37cc61d32076a69de65c42913dfbb1961f46"
	hash1, _ := hex.DecodeString(hashHex)
	hash2 := sha256.Sum256(in)
	fmt.Println("equal?", reflect.DeepEqual(hash1, hash2))
	fmt.Println("equal?", bytes.Compare(hash1, hash2) == 0)
}

The code on play.golang.org.

This gives the helpful error message cannot use hash2 (type [32]byte) as type []byte in argument to bytes.Compare. Which, at a stroke, explains why reflect.DeepEqual is screwing up: the first check it is making is “are these the same type?” And the answer is no, so hash1 and hash2 are not equal. Even though they are.

In order to turn hash2 into a []byte so that it can be the second argument to bytes.Compare, you just need to take a full slice of it, changing it to hash2[:].

The final, working, hash compare is here.

Moonrise

My time-lapse camera in the attic is still working, though I resorted to adding an auto reboot once a week, because the Raspberry Pi is not acting too stable. And even then, sometimes it hangs. I blame the power supply. Because it’s always the power supply, right?

Anyway watching the sun’s track northward as spring advances has given me a much better instinctive feel for celestial mechanics. And that made me pay closer attention to the moon rise last month. After 40 years on this planet I just realized that the moon, being in the same orbital plane as the sun and earth, traces the same track as the sun. Whoa. That means my attic window is perfectly oriented to catch a nice time-lapse of the moonrise!

(Check out the planet riding with the moon up into the sky. It’s probably VEnus. Is there an astronomer in the audience that can tell me?)

Instead of putting the moon’s mechanics into my program, I took the wimpy way out and I built in the full 2014 tables for my location, which I got from USNO (Look! The US military does something other than killing people and breaking stuff. Go Navy!)

Live from Mont-la-ville

The last few days I’ve been working on a new home hacking project. The eventual plan is to create a panoramic time-lapse of sunrise as seen from my house each morning. We’ve got a wonderful view, and recording some of those beautiful morning colors as the sun comes up over the Alps should make them easier to appreciate — without setting my alarm for 6 AM!

We have little windows in our attic that look out over the view. And being so high up in the house, they have a commanding view over the trees in front of our house. So I knew I needed my camera mounted up there. That location, in turn, fixed some other variables. There’s power up there, but no ethernet, so the camera needs to be on wifi. (Yes, I’m kicking myself. I’m an idiot for not specifying an ethernet run up to there. But the good news is there’s a closet where I can run it, so doing it myself will be quick and easy.)

I got a camera for my Raspberry Pi. I’ve had bad experiences in the past with webcams due to cheap lenses. So when I saw a camera module with a CS mount lens included, I went for it. The specs of the Raspberry Pi camera are incredible. The fact that it is on a high speed bus direclty attached to the GPU is very interesting. It means that the GPU can accelerate video compression. So the little Raspberry Pi can still manage to stream 1080p video!

In fact, once I had it hooked up and mounted up in the window, I needed to focus it. I found some instructions on how to stream video from it. They worked flawlessly. Using VLC to watch the camera’s output, I got it aimed and focused. I loved having the stream so much, I left it running. Paste this URL into VLC to see the stream: http://pi.nella.org:8554  (IPv6 only; and it’s not up anymore anyway)

Video streaming only works over IPv6, because I don’t want to fuss around with IPv4 NAT traversal settings. NATs are stupid; IPv6 is the answer. And IPv6 is getting easier and easier…

As for the time-lapses, I used Go to write a little program to that uses a library to calculate the time of sunrise and sunset (thank you Github user keep94!). My program wakes up a bit early and starts snapping images. Afterwards, it runs a script to make the time-lapse. It uses the technique shown in this post on the Ubuntu forums.

Each day’s results are currently posted here. (IPv6 only)

Looking forward to some better weather and some beautiful sunrises and sunsets!

Taking an MQTT server out for a few holes on the code golf course

A while ago at work, I needed to learn a little about MQTT. After reading the spec and finding this package that handles reading and writing the messages themselves, I decided to write a server.

The result is here: https://github.com/jeffallen/mqtt.

After I finished, I wanted to know if my server was comparable to Mosquitto, so I wrote some load testing tools. I found that my Go server was generally comparable to Mosquitto, in that it had the same order of magnitude for transaction rate and latency (but it was strictly slower than Mosquitto; there is a cost to pay for the benefits Go gives you). As usual, Go programs compare quite favorably to C programs in terms of memory safety, overall length, and readability. It is also useful to consider that my implementation took only a couple of days to get finished. A C program with equivalent safety, correctness, and speed would take much longer to write. I’ve written them, I know.

Then I mentioned to my coworker Atila what I’d done. Knowing that Atila is incredibly competitive and a total D fanatic (and Go refusnik) I might have casually implied that D would have a hard time competing with my incredibly short, fast, correct, and beautiful Go version. Heh heh heh.

FORE!

And so the code golf began. It only ended yesterday when Atila came back to me and begged me not to write any more benchmarks because he was tired of beating my sorry ass into the ground with his blazing fast D implementation. Fair ’nuff.

If I still have your attention…

Here are some interesting things from the experience and the code I wanted to point out.

It is often said that the solution to a problem in a domain specific language will be shorter and more correct than the solution in a general purpose language. Of course, the drawback is that you’ve got to write the DSL first, and then people who come along after you have to learn it. Implementing an MQTT server in Go really felt like writing in the correct domain specific language. It basically wrote itself, because channels and garbage collection meant that the hardest parts of a message passing server happen in the runtime instead of in the code.

Though Atila had a huge amount of fun tuning his D implementation, I didn’t do much at all to tune my implementation.

I used “go tool pprof” to fetch a CPU profile from it while it was under load from the pingtest program. But the resulting trace showed that it was spending most of its time jumping on and off of system threads. In other words the program is I/O bound, not CPU bound. It is making “too many” kernel calls, and thus speeding it up by looking for a hot spot or reducing garbage collection overhead isn’t interesting. The solution would have been to add some kind of buffering layer between the net.Conn and the message encoder/decoder. But the problem with an intervention like that is that you’d need to do it carefully so that the buffer would cut down on kernel/user context switches while there were very high transaction rates, but not introduce latency on messages when the transaction rate is low. And if the goal is massive scalability, with 1 million or more connections, each of which have a very low transaction rate, then this buffering technique wouldn’t help anything.

I did notice one nice optimization I was able to make when I was looking at the traffic with Wireshark. It is documented in this checkin. The lesson here is fewer calls to the kernel is not only a good idea for kernel/user context switch time, but may also save a significant amount of bandwidth.

As for Go techniques, I created something called a “receipt”, which lets you get synchronous feedback about a job sent in via a asynchronous mechanism. Take a look at how submitSync is used when we need to know when a certain message has departed. A receipt is a channel. The only two operations done on the channel are “make” and “close”. This makes it work like a condition variable. It flowed out of my fingers at the time; I don’t know if it is an idiom I’ll use again. It certainly didn’t give me any problems, so I guess I’ll keep it!

I don’t know or care where this code will go. If I receive pull requests, I’ll act on them. If someone needs to depend on this for a production-quality system, they might want to fork it and take long-term ownership of it. That’s OK with me too.

Qz for Mobile Phones

I love Quartz. however, I find that the best time/place for me to read it is on the train during my commute. And Quartz’s overly clever and overly dynamic website hates my phone’s browser. So I made Mobile Quartz to help me read it. The front page has the current top articles from Quartz. Click the links to see the article.

I used this as a chance to write some Go on Google Appspot code, and it was really a pleasure.