Going in the Deep End

Thoughts after using Go on a few projects.

We have been Wading into Go on some projects recently. In fact we have been using it on small and throw away projects for a while. We first used Go in anger to manage transfering and updating ~500,000 unique files (~1TB total) from an EBS volume to S3.
It was my first code in Go so it isn’t pretty. I am also not sure what the likelihood of it working now are as it used a fork of goamz. The fork was absorbed into central (hah) fork of goamz, but YMMV. The take away is that Go did made dealing with a massive number of files during a large scale migration practicable and I would definitely choose it again.
Missing Go SDK for AWSNB: A first class AWS SDK for Go would be awesome. This is definitely the missing tooth in a smile.
During that same project migrated a large number of vanity URL redirects. As part of the move there was a rule; if a redirect hasn’t been reviewed in more than 2 years, get rid of it. We had no way to know when rules had last been reviewed. They were stored as apache rules across a dozen servers. So the order was given that redirects had to have a “last reviewed” date. We used Go again to build an in memory redirect server with custom “Last reviewed” headers.
Most recently we have been using Go to write the backend API for an app powered by angularjs on the client side. This our first project which leverages GAE and is expected to have sufficient complexity and lifespan to warrant first class testing. The rest of this post discusses what warts we’ve seen when we got up close and how we have worked around them.

Testing

Testing with Go is a pleasant experience. Go’s standard library ships with a testing package that should feel familiar to most programmers. It is admittedly missing some convenience items like assertions, but that does not have much impact. Many coming from dynamic languages might think this is an ugly feature. However, it is easy enough to include a few of your own.
I have been including the three following functions for testing. I stole them from@benbjohnson’s article. Well really from his Github repo. The only change I made was to make equals use “want”/”got” instead of “act”/”exp” and to change the argument signature and logging order to match Go’s conventions. Here are those three functions: assert, equals, and ok.
assert fails with msg if condition is false.
func assert(t testing.TB, condition bool, msg string, v ...interface{}) {
    if !condition {
        _, file, line, _ := runtime.Caller(1)
        fmt.Printf("33[31m%s:%d: "+msg+"33[39mnn", append([]interface{}{filepath.Base(file), line}, v...)...)
        tb.FailNow()
    }
}
ok fails if an err is not nil.
func ok(t testing.TB, err error) {
    if err != nil {
        _, file, line, _ := runtime.Caller(1)
        fmt.Printf("33[31m%s:%d: unexpected error: %s33[39mnn", filepath.Base(file), line, err.Error())
        tb.FailNow()
    }
}
equals fails if got is not equal to want.
func equals(t testing.TB, got, want interface{}) {
    if !reflect.DeepEqual(got, want) {
        _, file, line, _ := runtime.Caller(1)
        fmt.Printf("33[31m%s:%d:nntgot: %#vnntwant: %#v33[39mnn", filepath.Base(file), line, got, want)
        tb.FailNow()
    }
}
I use equals and ok far and away more often than assert. This makes tests very easy to reason about, i.e.:
func TestUnmarshalXML(t *testing.T) {
    r := XMLWrap{}
    err := xml.Unmarshal([]byte(xmlData), &r)
    ok(t, err)
    equals(t, r.RootID, "anID")
    equals(t, r.RootValue, "aValue")
}   
Or in this table driven test:
func TestReadConfig(t *testing.T) {
    testValues := []struct {
        key  string
        want interface{}
    }{
        {"a_string", "This is a string."},
        {"a_int", 123},
        {"a_float64", 123.456},
    }

    for _, tv := range testValues {
        got := Config[tv.key]
        equals(t, got, tv.want)
    }
}
Go also has two nice http test helpers hidden in net/http/httptest, ResponseRecorder and Server.
ResponseRecorder provides an http.ResponseWriter which can be used to test an http handler or middleware. Following is an example testing a JSON not found handler. One passes in a ResponseRecorder to the function and verify what was written to it. Again equals and ok make it easy to reason about what is happening.
func TestNotFound(t *testing.T) {
        r, err := http.NewRequest("DELETE", "http://pkg.test/testuri", nil)
        ok(t, err)
        w := httptest.NewRecorder()
        NotFound(w, r)
        equals(t, w.Header().Get("Content-Type"), "application/json")
        equals(t, w.Code, 404)
        equals(t, w.Body.String(), "{"Status":"Not Found","StatusCode":404,"RequestMethod":"DELETE","RequestURI":""}")
    }
httptest.Server is also handy for integration tests. Take the following example which tests that a possible API client handles an HTTP error and doesn’t try to parse a non existent response. In this example we use the assert helper function rather than equals:
func TestAClientCallError(t *testing.T){
    ts := httptest.NewServer(
        http.HandlerFunc(func(w http.ResponseWriter, r  *http.Request) {
            w.Header().Set("Server", "golang httptest")
            w.WriteHeader(http.StatusInternalServerError)
            return
        }))
    defer ts.Close()

    client, err := NewClient("SecretKey")
    ok(t, err)
    wrap := &XMLWrap{}
    err = client.APICall("some_data", ts.URL, wrap)
    assert(t, strings.Contains(err.Error(), "500 Internal Server Error"), "Unexpected response received.")
}

Google AppEngine

Our current project targets Google App Engine (GAE) as its deployment platform. We decided to use GAE to eliminate the need to focus on which persistence and caching technologies to use, how to manage centralized logging, or how to scale. We could focus solely on our application. For the most part this has worked out well. GAE has first class Go support and has been a cinch to use. When we got up close and personal, however, we did notice another wart. This more with GAE than Go.
huge warts

The wart I couldn’t stop looking at was appengine.Context. In GAE you cannot use a vanilla http.Client. You must use the GAE provided transport from urlfetch.

import (
    "fmt"
    "net/http"

    "appengine"
    "appengine/urlfetch"
)   

func handler(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)
    client := urlfetch.Client(c)
    resp, err := client.Get("http://example.com/api/call")
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    fmt.Fprintf(w, "API call returned status %v", resp.Status)
}
You can see that this creates a context from the incoming http.Request, which then is used to generate the http.Client. This means that you have to use an appengine.Context in your tests. Which, in turn, means starting dev_appserver.py — a python powered web framework. This isn’t particularly difficult:
import (
    "testing"

    "appengine/aetest"
)

func TestAPICall(t *testing.T) {
    c, err := aetest.NewContext(nil)
    if err != nil {
        t.Fatal(err)
    }
    defer c.Close()

    client := urlfetch.Client(c)
    ...             
}
However, adding aetest.Context added ~3 seconds per test (on my notebook) — presumably to start up and shut down dev_appserver. That is untenable and a major hurdle to progress.
We looked for ways to work around this and found a couple prospects. The first and most desirable, but not yet out of preview, would be to run our app in managed VMs on GAE. The second would be to use Go interfaces to allow us to avoid usingaetest.Context in our tests.

Running Managed VMs in GAE

Running our app in a managed VM on GAE would be ideal as we could use a generic http.Client. However, the service is still in preview mode.
There are some differences with standard GAE deployment that are worth considering. None of these would be show stoppers for us, if not for the alpha/preview status. Even with the status it was worth taking a look to see if we could use vanilla ahttp.Client.
To get started you have to sign up to create a managed VM GAE project.
Once you get an email back letting you know that you are all set you can get you app uploaded. I’ll show a quick app I did to test using an http.Client without using the GAE urlfetch service. First you have to install appengine to your $GOPATH.
go get google.golang.org/appengine
Then we can put together an appropriate app.yaml and app.go file and upload them. app.yaml doesn’t vary much from the traditional one.
# app.yaml
application: test-vm-http2
version: 1
api_version: go1
runtime: go
    # New knob
vm: true

manual_scaling:
  instances: 1
# New knob
vm_settings:
  machine_type: n1-standard-1

- url: /.*
  script: _go_app
See configuring managed VMs for more detailed information. Our very basic app will get html content from a remote source and fill in our response with it.
package hellovm

import (
    "fmt"
    "io/ioutil"
    "net/http"
)

func init() {
    http.HandleFunc("/", handle)
}

func handle(w http.ResponseWriter, r *http.Request) {
    if r.URL.Path != "/" {
        http.NotFound(w, r)
        return
    }

    resp, err := http.Get("http://example.com/")
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    w.Header().Set("Content-Type", "text/html")
    fmt.Fprintf(w, "%s", body)
}
Viola! It serves without appengine.Context and works with a vanilla http.Client. You can see it if it is running when you read this. I don’t intend to run it forever as it costs real money for the smallest instance available as a managed VM.
NB: While it works without appengine.Context that isn’t entirely desirable. It would still be required to access the Logging service and other appengine services such as the Datastore, Memcache, and Users. The important part is that it wouldn’t be necessary to test API client calls any longer. Alas, it is still in alpha. Lastly, there is no support for running a local dev server for apps targeting a managed VM deployment (or maybe that is coming,,too (To access this link you will need to sign up for the VM tested truster program or request access.)).
That means we need to work around it another way…with an interface.

Testing via an Interface

To show how we use an interface lets first cover how we get the context into each of our handlers without having to repetitively call c := appengine.NewContext(r). What we do instead is make our handlers take the context as an argument and wrap them with an http.HandlerFunc us to serve up our handlers which don’t match the ServMux.HandleFunc signature.
func init() {
    http.Handle("/foo", aeContext(handler))
    http.Handle("/another", aeContext(handlers.Another))
}

func handler(c appengine.Context, w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, world! %s", config.Config["tms_api_key"])
}

// aeContext wraps handlers & injects the appengine.Context. This is
// primarily to allow testing of handlers directly.
type aeContext func(appengine.Context, http.ResponseWriter, *http.Request)

// ServeHTTP allows aeContext to serve our handlers
func (fn aeContext) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)
    fn(c, w, r)
}
The above snippet wraps our handlers with aeContext which is a HandlerFunc and implements ServHTTP appropriately. Our ServeHTTP method above creates a context from the incoming request and passes it in to our handlers. This is fine but means in order to test our handlers we have to create an aetest.Context to test them and that starts dev_appserver. Boo!
We can hide the detail of the appengine.Context by defining an interface and implementing a “real” and a “test” version of that interface. This has been discussed a few places before. So our “Contexter” interface looks like this.
// Contexter provides a way to use context. This exists primarily to allow for
// testing without depending on aetest.Context. aetest.Context starts dev_appserver,
// which makes tests take *forever*.

// Contexter has the following methods:

// GetHTTPClient, which returns an appengine-compatible HTTP client for a 
// real use and a testing one for the Dummy context

// Criticalf and Errorf, which wrap the appengine logging interface in a real
// implementation and do nothing in the testing context
type Contexter interface {
    GetHTTPClient() *http.Client
    // NB: We collect the other methods here to record what we need to implement. 
    // Anything we don't can just be an empty method. This ins't stricly 
    // necessary.
    Criticalf(format string, args ...interface{})
    Errorf(format string, args ...interface{})
    ...
}
Now that we have this we can change our HandlerFunc and ServeHTTP method to look like this.
...
    func handler(c appengine.Context, w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, world! %s", config.Config["tms_api_key"])
}

// aeContext wraps handlers & injects the appengine.Context. This is
// primarily to allow testing of handlers directly.
type aeContext func(Contexter, http.ResponseWriter, *http.Request)

// ServeHTTP allows aeContext to serve our handlers
func (fn aeContext) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    c := NewContext(r)
    fn(c, w, r)
}
...
Now we don’t have to provide a concrete Context, we only need to provide something that implements the Contexter interface. In the above example that is a “constructor”,  NewContext, that returns something that implements Contexter, which looks like this.
// Context is a Contexter which wraps a real appengine.Context for
// testing purposes (i.e., to avoid launching an instance of dev_appserver.py).
type Context struct {
    // This is the real appengine.Context.
    c appengine.Context
}

// NewContext returns a new Context instance for production use.
func NewContext(r *http.Request) Contexter {
    c := appengine.NewContext(r)
    return Context{c: c}
}

// GetHTTPClient returns an appengine-compatible http.Client.
func (c Context) GetHTTPClient() *http.Client {
    return urlfetch.Client(c.c)
}

// Errorf logs to the underlying appengine.Context
func (c Context) Errorf(format string, args ...interface{}) {
    c.c.Errorf(format, args...)
}

// Criticalf logs to the underlying appengine.Context
func (c Context) Criticalf(format string, args ...interface{}) {
    c.c.Criticalf(format, args...)
}
Above, our real Context constructor, NewContext, gets a real appengine.Context and creates a Context with it. The logging methods log to the real Logging service. This works great and now allows us to pass in a dummy context when testing. Here is what our DummyContext looks like.
type DummyContext struct {
}

// GetHTTPClient returns a dependency-free http.Client.
func (c DummyContext) GetHTTPClient() *http.Client {
    return &http.Client{}
}

// errorf logs to the underlying appengine.Context
func (c DummyContext) Errorf(format string, args ...interface{}) {
}

// criticalf logs to the underlying appengine.Context
func (c DummyContext) Criticalf(format string, args ...interface{}) {
}
Above, our DummyContext uses a vanilla http.Client and just eats logs. So we can test against an httptest.Server as noted earlier in the “Testing” section rather than going through dev_appserver.
func TestErrch(t *testing.T) {
    ts := httptest.NewServer(
        http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            w.Header().Set("Server", "golang httptest")
            w.WriteHeader(http.StatusInternalServerError)
            return
        }))
    defer ts.Close()

    ctxr := vtest.DummyContext{}
    client, err := NewClient(ctxr, "SecretKey")
    ok(t, err)
    wr := &XMLWrap{}
    err = client.Call("someData", ts.URL, wr)
    assert(t, strings.Contains(err.Error(), "500 Internal Server Error"), "Unexpected response received.")
}
Setting up and tearing down an httptest.Server still takes ~1 second (on my notebook) for each instance, but that is still an a lot faster than setup/teardown of dev_appserver. We will also look at setting up an httptest.Server with an http.ServeMux that handles all the test cases so we only need to setup one httptest.Server. Hopefully that will bring test times back down to a level where developers can do TDD.

Conclusion

While Go may have some warts of its own, they aren’t any uglier than warts in other languages. As we work out how to do things the Go way, it doesn’t always feel “right”, but that is because change is hard. Constructive feedback about any content in this post is very welcome. Please tell us what you think…