Monday, July 28, 2014

GoConvey for Gophers

GoConvey is a testing utility for Go built around the standard testing package and go tools. If you're not familiar with GoConvey yet, take a look, maybe try it out, or watch the intro video before finishing this article.

Mike (@mdwhatcott) and I (@mholt6) made GoConvey to improve our workflow and to help us write clearer and more deliberate tests.

I wanted to point out a few things that may not be clear or obvious about GoConvey. Then I'll discuss why it is or isn't a good choice for you and your project.

"GoConvey" means a few things.

In your editor, it's a testing framework (namely the Convey and So functions).

In the terminal, goconvey is an HTTP server that auto-runs tests.

At http://localhost:8080, it's a graphical interface for using go test and viewing results.

In your compiled binaries, it's nothing. (More on this later.)

GoConvey is loosely coupled.

That means you can use the DSL without the goconvey server. And vice-versa.

Many developers use the web UI to auto-run and visualize standard Go tests that don't use GoConvey's special functions. GoConvey can run tests built with other testing frameworks as long as they also use go test.

By using GoConvey, you're not automatically building another barrier to entry: you're automatically building and running Go tests.

On the flip-side, it's totally normal to just keep running go test from the command line manually while using GoConvey's syntax to structure tests. (If that's what you prefer.)

GoConvey runs go test.

Look! Nothing up my sleeve. All goconvey really does is run go test. Yeah, it uses the -cover flag and stuff, but all the output and data it generates comes from the go command.

go test runs GoConvey.

GoConvey tests are in _test.go files and inside Test functions with the testing.T type, just like regular tests.

It's a good idea to use regular tests when you don't want the BDD structure or when it becomes unnatural to describe your tests in the BDD style. You can use both kinds of tests in the same package and file.

GoConvey is not a CI tool any more than go test is.

Sometimes I see people refer to GoConvey as a CI (continuous integration) utility. Maybe they define that differently than I do, but the only thing "continuous" about GoConvey is the auto-test feature and the only "integration" features it has are with go test (and maybe with the file system, if you count polling the project directory for changes).

If it works for you in a CI environment, great! (Would you tell me how?) You can write tests using the DSL and run those in CI jobs, but beyond that, the web UI is interactive and I don't think the goconvey server that powers it is useful in automated scripts...

Test dependencies are not compiled into your programs.

Except for running go test, your Go programs are not built with test files and dependencies. This means that using a testing framework like GoConvey has no technical effects on your compiled program. The file size and function of your resulting binary is unchanged.

Even go-getting any package does not download its test dependencies. You need the -t flag for that.

Test dependencies are thus very deliberately obtained and are only needed by your library's contributors/hackers. Most people will never need to concern themselves with your testing framework or assertions package of choice.

GoConvey documents code beyond godoc.

Godoc is Go's convenient, built-in documentation system. It reads comments directly inline with your code to produce package, function, and variable documentation; examples; known bugs; and more.

Far from replacing godoc, GoConvey's DSL complements it. While godoc documents how to use a package, Convey tests document and enforce package functionality and behavior.

For example, suppose you wrote a bowling game. The godoc would probably explain how to use functions like NewGame, Score, and Roll, along with what they do. This is useful for users of the package, but leaves a lot to be assumed by any developers that want to start hacking away on it. To fix this, you could make your godoc much more verbose and describe how the package is supposed to behave, but that's just noise for users who don't need to know about the internals, and developers still don't have any assertions that prove your program works like you say. You could write regular tests, but then you hope the comments stay true to the code.

This is where behavioral tests come in. The GoConvey tests for the bowling game make it clear exactly what the program should do normally and in edge cases; plus, it asserts correct behavior in context of the test case. The tests actually convey intent and become that missing documentation.

See, by using godoc along with descriptive tests, you've now sufficiently documented your code for both users and developers.

Deciding when and how to use GoConvey

Should everyone use GoConvey for everything?


GoConvey is all about workflow and intent. It's not for everyone in every situation. I don't even use it all the time.

It does some things very well, but other things not so much.

Does well:

  • Documents your code with descriptive, structured tests
  • Auto-runs tests
  • Assertions
  • Reports results in real-time in the browser or via desktop notifications
  • Integrates with go test and standard testing package
  • Stubs out test code for you

Doesn't do well:

  • Automated environments (regular go test works fine though)
  • Run outside of $GOPATH
  • Save battery life
  • Race detection
  • Truly randomized test execution order
  • Idiomatic Go (I'm referring to the DSL)

There’s no magic formula that I know of to determine whether you should use GoConvey one way or another. But here are a few ideas to try:

  • Execute the goconvey command and open the web UI on your existing Go tests. See how it feels.
  • If you are starting a new project, check out the examples folder and try a new way of writing tests using GoConvey's DSL (careful, you might like it). Make sure the GoConvey server is running so you get instant feedback after every save. (Or take a look at the tests for Go-AWS-Auth for a real use case.)
  • If you don't like the nested BDD structure, try using a single level of Convey instead. This way your tests can still be somewhat descriptive and you get the benefit of the many So assertions. (For example, you can still run table-driven tests using a for loop inside a single Convey level.)
  • Use Go's dot-import for the convey package so you can make it "one of your own," so to speak. Way more convenient.

One of our hopes for GoConvey is to make writing tests more of a pleasure than any old task on your to-do list. Watching the coverage bars go up, up, up and seeing that orange "FAIL" turn into a big green "PASS" is seriously motivating.

(okay, really; done with this meme now)