Test-Driven

by Alex Chaffee

alexch @ gmail.com

Intended Audience

• Developers or QA Engineers
• Familiar with Unit Testing (optional)
• Want more detail on Automated Testing in general
• Want to know the case for Test-Driven Development
• Want to know style tips and gotchas for Testing and TDD

Part I: Basic Techniques

Red, Green, Refactor

• First, write a test and watch it fail
  • Make sure it's failing for the right reason!
• Next, write just enough code to make it pass
  • Enjoy the green!
• Finally, look at your code (and your test!) and see if you can clean it up
  • change names
  • remove duplication
  • extract methods

Make it green, then make it clean

Addicted to green

• You get a little rush every time a new test passes
• Steady, incremental feeling of progress
• Don't write code all day without knowing if it works

Blueprint for a Single Test

• GIVEN (set up preconditions)
• WHEN (call production code)
• THEN (check the results)

Assert

• The heart of a unit test

• An assertion is a claim about the code

  • Failed assertion -> code is incorrect
  • Passing assertion -> code is correct

• Example:

  Set set = new MySet();
  set.add("ice cream");
  assertTrue(set.contains("ice cream"));
  

• In RSpec, "assert" is called "should" or "expect"

• In Jasmine/Jest, "assert" is called "expect...to"

One Step At A Time

• Don't be over-ambitious
• Each test -- especially each new test -- should add one brick to the wall of knowledge
• Pick tests (features) in order of growth

The Null Test

• A great first test to write
• Input and output are trivial
• Helps you develop skeleton
• Helps you think about interface

Test List

• Before you begin, make a TODO list
• Write down a bunch of operations
• For each operation, list the null test and some others
• Also put down refactorings that come to mind
• Why not write all the tests in code first?
  • Could box you in
  • Interferes with red-green-refactor

Fake it till you make it

• It's okay to hardcode answers for the first few tests
• After a while, your tests will force your code to be more correct
• If you refactor, you will almost always come up with a more elegant solution than a big switch statement

Assert First

• When you get stuck on a test, try starting with the assertion(s) and then work your way backwards to the setup

• Start with the assert

  assertTrue(set.contains("cake"));
  

• Then add the code above it

  Set set = new MySet();
  set.add("cake");
  

• Helps focus on goal

Fake It 'Til You Make It

• Start with hardcoded results and wait until later tests to force them to become real

Obvious Implementation

• aka Don't be stupid
• If you really, really, honestly know the "right" way to implement it, then write it that way
• But keep track of how many times your "obvious implementation" was broken or untested
  • Edge cases, off-by-one errors, null handling... all deserve tests and often the Obvious Implementation is not covered

Interlude: The Blank Page

• Let's test-drive a utility class

Part II: Testing Philosophy

Automated Testing Layers

• Unit
• Integration
• Acceptance
• QA
• UI
• Performance
• Monitoring

A Good Test Is...

• Automated
• Isolated
• Repeatable
• Fast
• Easy to write and maintain
• Focused
• Easy to read
• Robust (opposite: Brittle)

Tests are "Executable Specifications"

• Someone should be able to understand your class by reading the tests

"Any fool can write code that a computer can understand. Good programmers write code that humans can understand." – Martin Fowler

Why do you test?

Why do you test?

• Design
• Prevent bugs
• Regress bugs ("bug repellant")
• Discover bugs
• Localize defects
• Understand design
• Document (or specify) design
• Improve design
• Support refactorings
• Enable experimentation and change

When do you test?

When do you test?

• Before checkin
• After update
• Before deploy
• While coding
• In the background

When do you test?

• All the time

Why test first?

• Gets tests written
• Easier than retrofitting tests onto an existing system
• Guarantees 100% test coverage
  • Note: not the same as 100% functional coverage, or a guarantee that your code does what it's supposed to

Why test first? (cont.)

• Reduces scope of production code
  • less scope -> less work
• Encourages better design
  • usable interfaces (since the tests are a working client)
  • more useful methods and fewer useless ones
  • Implementation matches design
• Guarantees testability

Can't I write tests later?

• In practice, you never have time after the code is written
• but under TDD, you always have the time before
  • Go figure :-)

How can you write tests for code that doesn't exist?

Some tricks to get started:

• Think of tests as examples or specs. Think of a thing the program should do, then write a test for just that one thing.
• Test the null case -- what should happen when you call the function or construct the object with no parameters?
• Write functioning code inside the test suite, then extract that code to a function or class, then move that function (or class) into the production codebase

"If you can't write a test, then you don't know what the code should do. And what business do you have writing code in the first place when you can't say what it's supposed to do?" - Rob Mee

Spike to Learn

If you don't know what test to write, then start with a spike.

A "spike" is an experiment, a proof of concept, a playground.

Code until you feel confident that you understand the general shape of the solution.

Then put your spike aside and write it again test-first.

Unit Testing Is Hard...

• It forces you to really understand the requirements
• It forces you to really understand the code
• It forces you to really understand the tests
• It forces you to create code that is truly reusable and modular and testable
  • "put your money where your mouth is"
• These forces drive you to keep your code and your tests simple and easy to understand

...but it makes your life easier

Test-Driving Is Slower At First

• Need to spend time on infrastructure, fixtures, getting comfortable with TDD
• Business case for TDD: sustainable velocity
  • for feature velocity, stabilty > early oomph
• Famous Graph

TDD vs TDD

• Test-Driven Development
  • Good old-fashioned coding, now with tests!
  • Much of the design is already specified before you start
• Test-Driven Design
  • Free your mind and the code will follow
  • Refactor at will, listen to what the tests are telling you

Quite a lot of overlap, but worth keeping difference in mind

Test for "essential complexity"

• Not too big, not too small
• Same concept as high coherence, low coupling

Tests Are An Extension of Code

• Every time you write code, you write tests that exercise it
• That means that if you change the code, and the tests break, you must either
  • Change the tests to match the new spec
  • Change the code to meet the old spec
• Do not remove the failing tests
  • Unless they no longer apply to the new code's design or API
  • Do not work around the failing tests
• Test code is not "wasted" or "extra" -- tests are first-class citizens

Meszaros' Principles of Test Automation

• Write the tests first
• Design for testability
• Use the front door first
• Communicate intent
• Don't modify the SUT
• Keep tests independent
• Isolate the SUT
• Minimize test overlap
• Minimize untestable code
• Keep test logic out of production code
• Verify one condition per test
• Test separate concerns separately
• Ensure commensurate effort and responsibility

Part III: Advanced Techniques

What to test?

• Simple Rule

Test everything that could possibly break

• Depends on definitions of "everything" and "possibly" (and "break")
• Corollary: don't test things that couldn't possibly break
  • e.g. Getters and Setters
  • Unless you think they could fail!
  • Better safe than sorry; test what you don't trust

How much to test?

• Personal judgement, skill, experience
• Usually, you start by testing too little, then you let a bug through
• Then you start testing a lot more, then you gradually test less and less, until you let a bug through
• Then you start testing too much again :-)
• Eventually you reach homeostasis

Triangulate To Abstraction

• aka "Fake it till you make it"
• Make the code abstract only when you have two or more examples

Step one:

function testSum() {
  assertEquals(4, plus(3,1));
}
plus(x, y) {
  return 4;
}

Step two:

function testSum() {
  assertEquals(4, plus(3,1));
  assertEquals(5, plus(3,2));
}
function plus(x, y) {
  return x + y;
}

Full Range Testing

Positive Tests

• exercise normal conditions ("sunny day scenarios")
• E.g. Verify that after adding an element to a set, that element exists in the set

Negative Tests

• Exercise failure conditions ("rainy day scenarios")
• E.g. verify that trying to remove an element from an empty set throws an exception

Boundary Conditions

• Exercise the limits of the system ("cloudy day")
• E.g. adding the maximum number of elements to a set
• E.g. test 0, -1, maximum, max+1

Descriptive Test Naming

• instead of SetTest.testEmpty
• how about SetTest.testShouldHaveSizeZeroWhenEmpty
• or EmptySetTest.testHasSizeZero

nested "describe" blocks can help too...(see later slide)

Should Statements

• Assertion messages can be confusing

  • (double negatives are not uncomplicated)

• Example: assertTrue("set is empty", set.isEmpty());

  • Does FAILURE: set is empty mean
  • "the set must be empty, and it's not" or
  • "the set is empty, and that's a problem"
  • ?

• Solution: should statements

  • assertTrue("set should be empty", set.isEmpty())

• or even better:

  • assertTrue("a newly-created set should be empty", set.isEmpty())

Nested Describe Blocks

describe('Set', ()=> {
  let set;
  describe('when first created', ()=> {
    beforeEach(()=> {
      set = new Set();
    });

    it('should exist', ()=>{
      expect(set).toBeTruthy();
    })

    it('should be empty', ()=> {
      expect(set.size()).toBe(0);
    });
  });
});

Output:

  Set
    when first created
      ✓ should exist
      ✓ should be empty

Test-Only Methods

• Philosophy: a test is a valid client of an object
• Therefore don't be ashamed of adding a method to an object for the sake of making a test easier to write
• Used -> Useful
• Tests are examples of use

Refactoring Test Code

• Do spend time refactoring your tests
• It'll pay off later, when writing new tests or extending/debugging old ones
• Refactor for readability, not necessarily for removing all duplication
  • Different priorities than for production code
  • MOIST not DRY

Refactoring Test Code - How?

• Extract methods
• Shorter lines
• Break up long tests (scenario tests) into several short tests (feature tests)
• RSpec
  • use magic keywords like let and subject and context
  • see EggsOnBread's RSpec best practices and tips

Evident Data

Increase test readability by clarifying your input and output values

assertEquals(86400, new Day().getSeconds())

vs.

assertEquals(60 * 60 * 24, new Day().getSeconds())

vs.

secondsPerMinute = 60
minutesPerHour = 60
hoursPerDay = 24
assertEquals(secondsPerMinute * minutesPerHour * hoursPerDay,
  new Day().getSeconds())

Matrix Tests

• Problem: several axes of variability, combinatorial explosion
• Solution: Loop through a matrix of data in your test, call a "check" function on each row
• In dynamic languages like Ruby and JavaScript you can loop outside a function definition, producing one actual test per iteration

%w(a e i o u).each do |letter|
  it "#{letter} is a vowel" do
    assertTrue(letter.vowel?)
  end
end

Characterization Tests

• aka "Golden Data Tests"
• Grab the complete output of a routine, put it into the test
• Not amenable to test-driven development
• Effective for large or risky refactorings
• Quite brittle, so often thrown away after the refactoring is done

How to Test Exceptions?

public void testUnknownCountry() {
  try {
    currencyConverter.getRate("Snozistan");
    fail("Should have thrown an exception for unknown country");
  } catch (UnknownCountryException e) {
    // ok
  }
}

The empty catch block is fine here, since here an exception is a success, not a failure to be handled.

Jasmine has a built-in way to test exceptions:

expect( function(){ parser.parse(bogus); } )
    .toThrow(new Error("Parsing is not possible"));

Characterization Tests

• aka "Golden Data Tests"
• Grab the complete output of a routine, put it into the test
• Not amenable to test-driven development
• Effective for large or risky refactorings
• Quite brittle; often thrown away after the refactoring is done

Pair Programming

• A pair's job is to keep you focused
  • "Wait, let's write a test first."
  • "Wait, let's refactor first."
  • "Wait, let's discuss this."
  • "Can I drive?"

Ping-Pong Pairing

• One pair writes a test
• The other pair makes it pass and writes the next test
• Repeat
• Good way to get out of a rut, or cure a keyboard hog

Regression Test

"Regression tests are tests that you would have written originally." - Kent Beck

• When a bug is reported, the first step is to write a (failing) test that reproduces the bug
• Fix the bug by writing code until the test passes
• Verify the bug in the running application
• Check in the bugfix test and code
• Now it's always run – instant regression test!
• If your regression test is high-level (e.g. Selenium), you probably want a failing unit test too

Do Over

• Often the best thing to do is throw away your work and start again

Leave One For Tomorrow

• At the end of the day, write a failing test and leave it there for tomorrow
• Based on writer's trick: start a sentence and leave it unfinished

The Need For Speed

• Tests are only valuable if they're run all the time
• If they're slow, people will not want to run them all the time
• So keep them fast!

• Difficult quest, but worth it

• Don't get stuck in molasses!

  • Refactor your code to be easier to write fast tests on
  • Replace slow tests with (one or more) fast tests that cover the same area

• Corey Haines has some great tips for keeping Rails tests fast

Continuous Integration

• Any time all the tests are green, you can check in
• Run all the tests all the time
• Don't check in until all tests pass
• If you broke "someone else's" tests, you are responsible for fixing "their" code
• Remember, they are in the room, so go get them if you need help

Retrofitting

Q: What to do when you have an existing untested codebase?

A: Start small!

• Write one test, make it pass, check it in
• Write tests for all new code
• Write tests for all new bugs

• Write tests before attempting refactoring

• Usually easier to write characterization tests (UI/integration/Selenium)

  • But don't fall into the slow test trap

Fixtures and Factories

• A natural progression of refactoring your test data
  • literals
  • constants
  • local variables
  • instance variables (defined in setup / before blocks)
  • creation methods
  • external fixture files
  • (used by Rails by default)
  • parameterized creation methods or objects
  • ("factories" or "object mothers")

Test Doubles (Mock Objects)

A Test Double replaces the "real" instance of an object used by the production code with something suitable for the currently running test, but with the same interface.

• Stubs
• Mocks
• Fakes
• Spies
• Saboteurs
• Shunts

Test Doubles in general are often called Mock Objects, but there's also a specific technical type of double called a Mock.

Test Doubles

• Stubs
  • Hard-coded values
• Mocks
  • Pre-programmed with expectations
  • Fail-fast
  • Test Doubles in general are often called Mock Objects, so be careful about terminology
• Fakes
  • Can store values across calls, but don't really do what the live object would do
  • E.g. in-memory database

Test Doubles (cont.)

• Spies
  • Remember what methods were called with what values
  • Tests can inspect these lists after code returns
• Saboteurs
  • Blow up in ways that would be difficult to make happen for real
  • To test what would happen when, e.g., the database goes away, or the disk is full
• Shunts
  • The test itself declares methods or classes implementing the above, and passes in a pointer to itself

Mock Clock

A very useful test double

In ruby:

@fake_time = Time.now
Time.stub(:now) { @fake_time }

In Jasmine (built in, see the docs for more details):

  it("causes a timeout to be called synchronously", function() {
    let timerCallback = jasmine.createSpy("timerCallback");

    setTimeout(function() {
      timerCallback();
    }, 100);

    expect(timerCallback).not.toHaveBeenCalled();

    jasmine.clock().tick(101);

    expect(timerCallback).toHaveBeenCalled();
  });

Complete Construction

• a way of designing your objects to be more isolated and more testable

  • (a form of Dependency Injection aka Inversion of Control)

• Pass in dependencies to the constructor

  • (or, if necessary, to setters)

• An object under test will receive references to all external services

• Allows tests to inject Test Doubles at will

• Forces objects to be isolated

BDD (specs)

• Changes the language of tests to emphasize that they're specifications or examples
• Replaces "assert" with "should"

Outside-in vs. Inside-out

• Matter of preference
• Both are useful at times

Inside-out

• Start with domain objects
• Next layer of tests

Outside-in

• Start with customer story or user interface
• Makes you think like a user
• Tests capture these requirements
• Lower layers implemented with

Outside-in design, inside-out development

• Write a bunch of UI-level tests
• Leave them there while you test-drive inside-out

Part IV: Q&A

Thanks!