Protecting private keys in Go

Today I was looking at Upspin and thinking about private keys. I asked myself, “what would it take to make sure that there was one single copy of the private key in RAM, and that Go and the OS worked together to make sure it never went onto disk?”

(Some other people have talked about this too.)

It turns out the answer to that question is much harder than expected, but doable and I’ll try to do it.

So, the last part first. How do you ask the kernel to help you make sure that a given piece of RAM never ends up on disk? There’s a hint here. I have not looked to see if this technique has been picked up by OpenSSL or BoringSSL. The way Akamai hooked it in to ASN.1 parsing ugly and gross and we won’t be doing that, but the kernel system calls they are using are interesting.

mlock and madvise work at page granularity. Data structures exposed in Go programs are at byte granularity, so we’ll need some kind of technique to convince Go to put our data on a page all to itself. I’m thinking that make([]byte, 0, pagesize) will probably do what we want. Once the underlying page is marked correctly, we return []byte to the caller. It is their responsibility to (1) never reallocate the underlying storage (i.e. only write into it via copy, never append) and (2) make sure that all of the places in the code where the private key is used do not make copies off of this secure page.

Then we need to go hunting, looking for places that the Go runtime is going to make copies of our private key. This is easy but error-prone work (at least for a Go veteran, who can spot when a copy is happening), and it makes me wonder if undertaking a task like this in Rust would be easier; maybe you’d be able to mark the private key immutable and implement the Copy trait as panic, then see what broke? Although I think you would still need to audit the code, because it would be possible to copy out the bytes in a typesafe way, I think.

Here’s a list of places I’ve already noticed I’d need to work on:

  • in Upspin’s factotum.go, the private key comes off disk and goes into memory. That’s in a []byte, so we’d be able to put it into our safe []byte, no problem. However it is converted to a string and then back into a []byte, which would have to be avoided. No biggie, and also some nice cleanup because stores the private key as a string (unused) and also an ecdsaKeypair.
  • The ecdsa.PrivateKey ends up with a pointer in it to a big.Int, which has the private key in it. It gets there via UnmarshalText. But UnmarshalText converts to a string too, so it has to get fixed. It seems likely that UnmarshalText should be using Fscanf to read directly from the []byte it receives, since big.Int implements fmt.Scanner.
  • While constructing the big.Int to hold the private key, there is a risk of leaving parts, or maybe all of the private key around in memory. This is because the final length of the underlying “words” array where the private key is stored is not known ahead of time to the math/big library. However, we do know how long the private key is going to be, so we can store a decoy private key into the big.Int to get it to stretch out it’s words slice, and then scan the real private key in, so that the word slice underlying array is never reallocated while reading the private key. However, there does not appear to be a way to tell big.Int where it should put the []Word. Due to type safety, it would require unsafe to do that. But it can be done in package factotum before the first call to Unmarshal.
  • The implementation of big.Int.Bytes() is used from time to time, and it causes a copy of the private key to be made. It does not allow us to pass in “please write the bytes here” argument in order to make sure that it is copying the private bytes exactly where we want them to go (i.e. to the protected page). At first glance, this would appear to require a new API, which is too bad.
  • The factotum package does not expose the private key, so it cannot be accessed outside of this package. That makes auditing usage of the key easy to make sure that it stays private. Thank you Upspin hackers!

So, in summary, this is probably doable, except that controlling the backing store for the big.Int to be private requires unsafe-ness.

My first ever Rust program!

Here, for posterity, is my first ever Rust program. It checks the key log on the Upspin Key Server.

extern crate sha2;
extern crate reqwest;

use std::io::Read;
use sha2::{Sha256, Digest};

fn main() {
    let mut resp = match reqwest::get("") {
        Ok(resp) => resp,
        Err(e) => panic!(e),

    let mut content = String::new();
    match resp.read_to_string(&mut content) {
        Ok(_) => {}
        Err(e) => panic!(e),

    let mut hasher = Sha256::default();

    let mut ct = 0;
    let mut last_hash = "".to_string();
    for line in content.lines() {
        if line.starts_with("SHA256:") {
            let mut fields = line.split(":");

            // skip first token
            match {
                Some(_) => {}
                _ => {
                    println!("Bad SHA256 line: {}", line);

            last_hash =;
            let expected = string_to_u8_array(&last_hash);
            let output = hasher.result();
            assert_eq!(output.as_slice(), expected.as_slice());
        } else {
            hasher = Sha256::default();
            let newline = "\n".as_bytes();
            if last_hash != "" {

        ct += 1;
        println!("Line {}", ct);

use std::u8;
fn string_to_u8_array(hex: &String) -> Vec<u8> {

    // Make vector of bytes from octets
    let mut bytes = Vec::new();
    for i in 0..(hex.len() / 2) {
        let res = u8::from_str_radix(&hex[2 * i..2 * i + 2], 16);
        match res {
            Ok(v) => bytes.push(v),
            Err(e) => {
                println!("Problem with hex: {}", e);
                return bytes;
    return bytes;

I found myself sprinkling mut’s and unpack()’s here and there like the mutation unpacking fairy, hoping something would work. I don’t think that how you are supposed to do it, but we’ll see.

People give Go a bad time about the size of it’s compiled files, so I expected this little Rust program to be much smaller. But not so:

$ ls -lh ../target/{debug,release}/upspin
-rwxrwxr-x 2 jra jra  23M Jul 21 15:12 ../target/debug/upspin
-rwxrwxr-x 2 jra jra 5.1M Jul 21 15:14 ../target/release/upspin

People also say, “Go is fat because it is statically linked”. Pure Rust code is also self contained in the program’s ELF file. But the 5.1 meg release image does not count several quite large shared libraries that it depends on:

$ ldd ../target/release/upspin =>  (0x00007fff319c2000) => /lib/x86_64-linux-gnu/ (0x00007f3d7a6cc000) => /lib/x86_64-linux-gnu/ (0x00007f3d7a288000) => /lib/x86_64-linux-gnu/ (0x00007f3d7a083000) => /lib/x86_64-linux-gnu/ (0x00007f3d79e7b000) => /lib/x86_64-linux-gnu/ (0x00007f3d79c5e000) => /lib/x86_64-linux-gnu/ (0x00007f3d79a47000) => /lib/x86_64-linux-gnu/ (0x00007f3d7967d000)
	/lib64/ (0x00005555d33b8000)

From a code safety point of view, I find it really troubling that even if we work so hard to make the compiler happy to prove our code is secure, we are still depending on libc and the platform’s OpenSSL. I remember the first time I started digging down into Go and realized that it wasn’t even linking libc, and it gave me a fantastic feeling of finally cutting the cord to the old, dangerous way of programming. With Rust (and hyper) we’re right back to depending on C again. Yuck.

Finally, the incredible speedup from debug to release images was surprising:

$ time ../target/debug/upspin > /dev/null

real	0m3.631s
user	0m0.888s
sys	0m0.020s
$ time ../target/release/upspin > /dev/null

real	0m2.961s
user	0m0.068s
sys	0m0.016s

Look at user, not real. Real is dominated by waiting on the far-end server. Real is more timing the TLS decryption, the parsing and the hashing.

Read it and weep

I searched for “how do I make an HTTP request in Rust?”. I’m a newbie, we do things like that. Don’t judge.

I found this. I still don’t know how, because the answer marked correct refers to a library that a comment from 2016 informs me is no longer supported. There’s also a helpful comment pointing me at cURL, which is written in C, which is the opposite of safe.

It does appear that the right answer, in 2017, is hyper.

I do not have warm fuzzies from this experience.

Update: Headache-inducing head slap from this (answer with 1 vote: “It is now 2017, and rustc-serialize is now deprecated as well…”), which happened 30 minutes after the above.

A Go programmer continues to learn Rust

I went looking for the equivalent of goimports and didn’t find it. Sad.

I wanted to use std::fmt to do the same thing as sprintf or fmt.Sprintf. I got stuck on “expected &str, found struct `std::string::String`”. I found a blog posting trying to explain it but I don’t understand enough yet to understand it. What I do understand is that it is highly suspicious that a language has two incompatible types for string. WTF, Rust? I’m trying to write a program here and you want me to sprinkle magic to_string()s in it? How about you STFU and let me be a human, and you be the computer doing the boring crap for me?

So back to basics, how about search for “rust format a string”. Top 3 hits are 3 different pages from the Rust reference manual, with 3 different import paths for apparently different ways to format strings? Well, just try one. Compiler says add a &. Fixed! So “give stuff to macros format!, get back a std::string::String, borrow a reference to it (which somehow magically does some type conversion between the 2 kinds of strings?), and give that to expect”. Right, got it.

I want to detect if nothing was read from io::stdin().read_line(). Hmm, how about `if guess.trim() == “”`? Bingo! Maybe this isn’t so hard after all!

Ugh, so many semi-colons. Bro, do you even semi-colon?

Aiee, the tutorial ran out and now I’m into a boring reference book. Let’s switch to this.

What’s #[]? Is that a macro? A compiler directive? A pre-processor directive? Tutorials that don’t give links to fundamental docs are annoying. I’m not a child…

Well, let’s go see what kind of crypto libraries are available. Hmm. Curious, the TLS connector for Hyper uses Rust Native TLS, which uses OpenSSL. Rust: fast, modern, and safe. Unless you are doing security critical work, then we hand that off to C programmers from 1990, because they never made any mistakes. Doh.

So, search for a pure Rust TLS, and I find rustls. Which uses ring for crypto. Which… wait for it… is made from Rust and C, with the C copied from OpenSSL.

Come on guys, this is not very impressive so far.

However, ring uses something called “untrusted” to parse attacker-controlled input more safely. That’s really interesting.

A Go programmer’s first day with Rust

Where is the tutorial? The first Google hit gives a redirect to a page explaining that I should read the book. The first page of the book explains that I should read the Second Edition unless I want to go deep, then I should later read the First Edition also (and presumably ignore the things that had been changed later in the Second Edition?)

OK, let’s do this thing! Get rust-mode installed in Emacs, and get it to call rustfmt on save, because Go taught me that the tab button is from 1960 and I don’t use it anymore.

Search for “why is rustc hanging while compiling rustfmt”. Find closed issue from someone who did the same thing. He says, “in the 6 minutes it took me to make this issue, it finally finished compiling”. I go back to my compilation window and find the same thing. Why is Rust’s compiler so slow?

OK, let’s do this thing! Wait, why is rustfmt making different format than the tutorial? Try some stuff, find out rustfmt wants me to upgrade my compiler to nightly, tell it, “No, my friend, that’s not going to happen. I rode that train with Go and it made me crazy.” Give up and accept that rustfmt is fighting the tutorial to teach me different formats for Rust. I miss gofmt.

Play with the example code in the tutorial, make syntax errors on purpose. The compiler error messages are… trying to be helpful, but as a result are too big to fit in my window. Rust developers must have bigger monitors than me. It is surprisingly difficult to say the minimal helpful thing. “I would have written a shorter error message, but I didn’t have time.”

Read about Results and how they encapsulate the return value. This is really elegant and pretty.

Tried making some code with and without mut and I think I’ve got it more or less. I understand why this is important, but I’m still pretty suspicious that it should be my job as a programmer to keep track of this. Wonder if there should be a rustfmt plugin where it adds in the mut’s that I’m not interested in keeping track of myself?

I would rate this 60 minute session of Rusting as about a 3 out of 10 on the “that was fun and I’d like to learn more of that” scale. Let’s see how the next hour goes…

Python keyword “finally” trumps “return”

Here is a piece of Python that I did not expect to surprise me:

def a():
                raise 1
                return 1
                print "exception!"
                return 2
                return 3

print a()

In both Python 2.7 and Python 3, that prints:


It seems that in Python, return is just a casual suggestion to the interpreter, and finally gets to decide, “Nope! Your return is canceled and I will do my return instead!”

Now, to be fair, in Go a defer can change the return value of a function as well. But the defer keyword literally does what it says on the tin, it defers some work until after the return, so you can reason about why it might be changing the return value.