CSCI 5030: Unit Testing


Unit testing is a central part of modern software testing and Test-Driven Development (TDD). Every language has some kind of unit testing support. Some are separate libraries or frameworks (such as JUnit or Pytest), while others are integrated into the language or in an IDE (such as with Microsoft Visual Studio). Many of the concepts and IDEs are similar, though different language capabilities (objects, exceptions, etc.) result in different unit testing capabilities.

This exercise is written in Python using pytest, but you are free to use any language or test suite you prefer. These exercises as written should be doable on hopper.slu.edu without any effort on your part.

In this exercise you will:

  1. Explore basic unit testing using logical assertions
  2. Learn the difference between dummies, stubs, mocks, spies,and fakes
  3. Write a short piece of software using Test-Driven Development

Exercises

  1. Before we start you should create a new directory and move into it. pytest by default will recursively search directories visible from the current directory when it looks for tests. Running the following exercises from your default home directory may take a long time or could result in unwanted behavior.

  2. First, we will explore the basic unit testing interface provided by pytest. Imagine you are writing a software library, meaning you are authoring a collection of functions, but you are not trying to write a finished executable. If you're like me, traditionally you test your software by running it in a traditional executable and printing lots of input/output pairs. This works, but unit testing with pytest is a better solution: pytest lets you run tests independent of any particular finished program, and it assesses correctness for you. Moreover, a good set of unit tests then become a regression test suite- if you ever refactor your code you can re-run your unit tests and verify that the overall behavior has not changed.

    Start by creating a file called myLibrary.py and writing a trivial function inside. For example, the contents of your file might be:

    def myFunction(x):
    	return x + 1;
    

    Remember: Correct indentation in Python is non-optional. The function definition should start at the beginning of a line and the return statement should have a tab in front of it.

  3. Second, let's test that your function runs the way you expect. You can invoke your code directly from the python interpreter. To do so, run the python interpreter with the "python" command from the terminal. Then, you will need to import your code with "import myLibrary" and finally you can run it with (for example) "myLibrary.myFunction(5)". In total, your output might look something like this:

    [username@hopper directory]$python
    
    Python 2.7.5 (default, Aug  7 2019, 00:51:29) 
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    
    >>> import myLibrary
    
    >>> myLibrary.myFunction(5)
    6
    
    >>> 
    

    Make sure your code runs as expected. You will have to press CTRL-D to quit the Python interpreter.

  4. Now let's use pytest to create a test suite for this function. There are several ways you can specify tests to be run, but for now we will separate all our tests into another file. However, be warned that there are some considerations when constructing test suites for large or complicated projects. See the following documentation on pytest test discovery for full details.

    Create a file called test_example.py. The "test_" tells pytest that this file contains a unit test. Then, we can create a unit test by defining a function within that file that also has the "test_" preface. For example, we can create a unit test assertion with the following:

    import myLibrary
    
    def test_case1():
            assert myLibrary.myFunction(2) == 3
    

    Then, since both the file and the test within the file should be automatically discoverable by pytest, we can run the unit test just by typing "pytest" at the terminal. Do so now, and the test should run successfully.

  5. Of course, the goal of unit testing is to catch failing tests. Let's create a test that is designed to fail. Add the following to your test_example.py:

    def test_case2():
            assert myLibrary.myFunction(5) == 7
    

    Since both test cases are prefaced with "test_" we can execute the entire test suite just by typing "pytest" at the terminal.

    [username@hopper directory]$pytest
    ============================= test session starts ==============================
    platform linux -- Python 3.6.8, pytest-5.1.2, py-1.8.0, pluggy-0.13.0
    rootdir: /student/username/directory
    collected 1 item                                                               
    
    test_example.py F                                                         [100%]
    =================================== FAILURES ===================================
    
    __________________________________ test_case1 __________________________________
    
        def test_case2():
    >       assert myLibrary.myFunction(5) == 7
    E    assert 6 == 7
    E     +  where 6 = <function myFunction at 0x7f8779561e18>(5)
    E     +    where <function myFunction at 0x7f8779561e18> = myLibrary.myFunction
    
    test_example.py:7: AssertionError
    ============================== 1 failed in 0.41s ===============================
    
    [username@hopper directory]$
    

  6. Our test failed, as expected. Note that pytest is trying to be as helpful as possible. The failure message indicates the test case that failed as well as the values in the assertion, and even gives us a stack trace of where the value "6" came from above.

    Go ahead and fix the failing test, and confirm that you can run pytest without errors.

  7. Of course, the file myLibrary.py can have more than one function and the file test_example.py can have more unit tests. Write a few functions and try out various logical assertions ( ==, !=, <, >, etc).

  8. Now we will introduce a series of test objects called "test doubles." These are made just for the purpose of testing, and their purpose is to stand in for other software components. Sometimes software units are interdependent on each other and there's no getting away from it, but in general, you want to minimize your test's dependence on other pieces of software.

    Why? First, that other software may not be written yet, especially if you are doing TDD. Second, there's no guarantee that other software is correct. Suppose that software A depends on software B to function, and software A's tests are failing. Where does the error come from? At first glance it's not obvious if the error stems from A, or from B, or from the interface between them. However, if the software B is actually a fake test double, then you can rule it out as a source of error. Moreover, you can do system integration tests later whose sole purpose is to test A with B, so it's just fine if the unit tests for A don't really test A with B.

    The test doubles we will look at are called dummies, stubs, spies, mocks, and fakes. The first four are related to each other, and each builds upon the others (in the order given above). Using the simplest possible test double makes your tests easier to write, and makes it obvious exactly what is and is not tested.

    We saw this article about these test doubles earlier that describes each of these in a simple, conversational style. Go ahead and read it, if you have not already.

  9. To discuss test doubles we will write a series of unit tests for car.py, a simple class for representing your everyday automobile. You can download car.py to the current directory of your Linux terminal with the following command:

    wget https://cs.slu.edu/~dferry/courses/csci5030/notes/car.py --no-check-certificate

    Or, you can access it directly with this link. Do so, and take a moment to read over the code. Hopefully nothing there should be terribly surprising.

  10. Now, let's write our first unit test. In particular, let's assert that a newly constructed car object should always have its distance counter set to zero. To do so we can use a line like:

    assert myCarInstance.distance == 0

    Easy to say, but we can't actually run this test yet. Why? First we need to call the constructor for the car class, and the constructor expects an engine object and a fuel_tank object. We don't have those objects yet... so what are we to do? We could give up on testing car until we actually have implementations for engine and fuel_tank, but that's a bad solution. This is exactly the situation for a test double, and in particular we need a dummy double.

    A dummy double is used whenever we need to pass an argument, but we know that the argument is never actually used. That's the case here. If you look at the code for car.py you can see that the distance attribute is set in the constructor, and engine and fuel_tank objects are not actually used.

    Try to write a unit test using dummy objects. If you get stumped, or when you're done, you can see my solution.

  11. Two remarks about my solution to the previous problem. First, the structure of the empty class doesn't really matter. The dummy is not used by definition. We don't even need a constructor, since our dummy objects will automatically inherit the default constructor.

    However, the second point is that we don't actually even need dummy classes to begin with, but only because we're using Python. Python is a very flexible language, and in particular there are no strong data types. The result is that Python doesn't really care what we pass to the car constructor... we can pass anything we want and Python will just throw us an error if we try to use it incorrectly. However, since engine and fuel_tank are never used, they can't be used incorrectly either. I would still suggest the first solution because it's only a few extra keystrokes and it makes it obvious what you're doing, but you could have gotten away with a solution that looks like this

    In a more strongly typed language, such as Java, you'd always have to create a proper dummy object.

  12. Now, let's think about testing the computeRange() function. Now mere dummy objects are no longer enough because we need to access the fuel_tank.size and engine.MPG attributes. However, the actual values returned can be chosen by you, the tester. This is the purpose of a stub test double.

    Create a stub test double for the engine and fuel_tank classes by giving them the appropriate attributes. Then, write a unit test or two for the computeRange() function. My solution is here when you're done.

    In particular, note that an empty class is no longer acceptable for testing this new function. If you try to use the dummy classes from before you'll get errors about accessing the size and MPG attributes.

  13. Third, now we would like to create a spy test double. Unlike the previous two, the purpose of the spy is to inspect the state of the system. Now- why is this useful? Consider the fillGasTank() function. This is a difficult function to test, because all that happens is that we call a method in the fuel_tank class. From the point of view of the person testing car.py, the fillGasTank() function is essentially a black box. All we really can do is verify that fuel_tank.refill() is called. Hence, we use a spy test double to verify exactly that.

    Now, we could implement our own spy similar to what is shown in The Little Mocker, but Python includes support for creating spies and other mock objects in the unittest module. This module is an alternative to pytest in many respects, and while pytest is perhaps more intuitive than unittest, the unittest module includes some indispensible functionality that is not provided by pytest. One of these features is the ability to create and use mock objects.

    In particular, you can import the unittest.mock module, and take a brief look at the documentation here. The basic workflow is to create an instance of the class you want, and then create mock methods using the unittest.mock module. Then, the module automatically logs if and how the mocked methods are called, and allows you to use a set of assert statements to confirm the behavior.

    You can see my implementation of a spy object here . If you run this test case as-is, you should get an assertion error at line 19 because the call to fillGasTank() is commented out. Once you uncomment that line the test case should run successfully.

  14. Another method to spy on is the call to fuel_tank.subtract() in the drive() function. One feature of the unittest.mock module is that you can spy on what specific values a function is called with. Use the documentation above to write a set of stubs and spies that verifies the value passed to fuel_tank.subtract(). In particular, if miles = 60 and engine.MPG = 30, then the value passed to fuel_tank.subtract() should be 2.

    Make sure that your code passes correctly, and then fails if you change some of the numbers involved. My solution is here.

  15. Surprise! The previous exercise had you create a true mock object. The spy we created first merely verified that the function fuel_tank.refill() was called. That spy looked at actions happening in the system, but it was oblivious to the actual behavior of the system. This last exercise created a mock object that expects a certain behavior... given a certain initial state (attribute values) and an input (drive(60)) the mock object knew that fuel_tank.subtract() should be called with value 2. Note that you don't ever call the subtract() function yourself, but the spy can still test it.

  16. The last test double we will look at today are fakes. These are so-called because they imitate the system under test, but they're not the system. Another name for a fake might be a simulation. A fake simulates elements of the system under test, with the goal that whatever simulation is provided is accurate enough to be informative without requiring prohibitive development time itself.

    The last method in car.py computes the total weight of the vehicle dependent on how much gas is in the fuel tank. Suppose we want to validate that driving our vehicle does in fact consume gas, and that it should weigh less after calling the drive() function than it did before. To do so we need some kind of logic implementing the computeWeight() function. We don't necessarily need the full or completely accurate implementation of the fuel_tank class, we just need something approximate. We need a fake.

    Write a fake fuel_tank that includes an implementation of the subtract() and computeWeight() functions. Then, verify that the weight after driving is less than the weight that was before driving. You can see my solution here. If you modify the drive(60) line to drive a negative number of miles then you can induce a test failure to prove to yourself that the test works. (And that we can magically grow gasoline just by driving in reverse. I guess we should have had a unit test for that, huh?)

  17. Now, armed with your knowledge of unit testing in Python, I'd like you to engage in test-driven development (TDD). As we said in class, in TDD you:

    1. Write down a list of features
    2. Pick one feature
    3. Write down a set of tests that verify/accept that feature
    4. Write a set of test cases, and just enough code for your program to run
    5. Run the test cases and verify that they fail
    6. Then, write just enough code to make all tests pass
    7. Go back to part 2, unless we have no more features

  18. Create a file called shapes.py. Then, create a class called rectangle. The constructor should take two arguments: the length and the width of the rectangle. You should implement functions called computeArea() and computePerimeter(). Develop this code using TDD. Remember that writing good test cases in TDD is tantamount to requirements analysis in traditional software development. Make sure to consider edge cases such as what happens if a user passes negative numbers for length or width, (or both), or if they pass non-numeric data such as None or a string. What happens if the user passes zeros to your constructor- does the code behave how you'd like, or were you surprised?

  19. If you would like, repeat the above exercise for triangles, circles, cylinders, etc.