The Purpose of Unit Testing

Creating tests for your code creates a methodology of proving your code works. Tests define precisely the intent of a routine. Even more important, testing code offers a safety net for both development and maintenance.

Testing forms the modern way to create solid code. An entire corpus of work supports the benefits of test driven development.

As you write code, you test it. By “test” I mean write code in the form of unit tests. When someone asks if you can prove your code works, you can run the tests and prove it. More importantly, you use the test code to prove it to yourself.

An old saying is “No test? Then, by definition, the code is broken.”

I do not mean just run the test through a debugger session! However, you can use a debugger to get the code working initially. After that initial debugger session, write a unit test!

(Enough of the emphasis. No one likes to listen to a rant.)

When you have a formally written test, you have an automatic debugger.

Unit testing often adds more features to routines as testing digs down into each part of code. A better program emerges. Areas difficult or tricky to test get refactored. Code gets re-written for ease of comprehensibility. Variables and routines get renamed. Overall logic crystallizes. Testing frequently detects problems early.

The target of unit testing targets the quality of code in each function. By designing and scripting quality tests, detailed documentation exists to prove program work-ability and correctness.

Decent tests demonstrate program correctness. Therefore, tests prove more valuable than external documentation.

Intro to unittest

Unittest provides a framework for unit testing python code. Unittest comes with your python download – nothing else is required to use and execute unit tests on your python code.

While unittest supports a wide variety of operations, we will use only a minimum set of these to ensure our code works properly. Constructing and running with the descriptions herein demonstrates a small subset of unittest that meets the needs of most users.

Each of the two files involved have command line options. We will test that each of the options in each of the files works as expected.

Setup for Testing

The testing code for both log_server and log_client resides in server_client_test.py. Splitting them into a test file for each could work, but the similarities in the logic for both remove some duplicated code.

The very first line begins as:

#!/usr/bin/env python

Developers familiar with this understand env runs the python program found in the environment. Put more simply, env finds the python program and executes it.

This technique permits python to live in different places on different systems. Some systems place python in /usr/local/bin/python and others in /usr/bin/python or /usr/bin64/python, ….

The imports come next:

import sys
import unittest
import log_server
import log_client

The sys module gets used in many (most?) python modules. Our code uses this in a reporting function.

We require the unittest module to create our own unit tests. The unittest framework subclasses unittest with out own test code.

Next the default names of log_server and log_client get coded into a single variable each:

# Names of client and server python scripts.
LOG_SERVER_NAME = './log_server.py'
LOG_CLIENT_NAME = './log_client.py'

Placing the names of these two programs in a single place allows changing the names once and only once. If the exact names were scattered all over the code, changing the names would become tedious and error-prone. The DRY principle of “Don’t Repeat Yourself” is a pillar of clean coding practice.

function_name() and FCN_FMT: Because this test code output multiple lines to the console, printing the test name assists in identifying potential errors and interpreting output.

str_to_argv(astr): When testing calls to log_server and log_client, using a string very close to what gets used on the command line really enhances readability. But, our lower level functions want something that looks like sys.argv, a list of strings. As an example, when invoking the server, examine how a console command becomes a list internally:

./log_server.py --log=x.log --port=5555

BUT, internally python turns the command line into a list:

python
>>> from server_client_test import str_to_argv  # Just for this example
>>> argv = str_to_argv('./log_server.py --log=x.log --port=5555')
>>> argv
['./log_server.py', '--log=x.log', '--port=5555']

This behavior mimics the conversion of command line input to sys.argv lists.

Class LogServerCmdLineTest

A unit test consists of a class derived from unittest.TestCase and uses one or more methods to test each aspect of the external code.

This condensed listing illustrates testing an invalid option in the command line (the “–Xlog” is invalid):

class LogServerCmdLineTest(unittest.TestCase):
    def test_server_invalid_option(self):
        print(FCN_FMT % function_name())
        argv = str_to_argv('%s --Xlog=log.log --log_append=False --port=5555' %
                LOG_SERVER_NAME)
        with self.assertRaises(SystemExit) as cm:
            log_server.process_cmd_line(argv[1:])
        self.assertEqual(cm.exception.code, 1)

The class groups multiple methods into a single testing area. Only the first method, test_server_invalid_option() gets listed here.

Notice the “test” in front of all these names. The unittest framework uses this to detect and automatically run this method. This naming convention informs the test runner about which methods represents tests.

For my own preference, I want each function tested to list the method name. Thus the print of function_name() with the FCN_FMT. Your preferences may vary and you may wish to omit the step.

The Logic of a log_server Test

The internal representation akin to sys.argv uses str_to_argv() the convert a much more readable command line string to a list. A deliberate error in “–Xlog=log.log” expects log_server to exit with an error condition.

This test simulates running the log_server from the command line.

In log_server, the command line gets processed by log_server.process_cmd_line(). The list of command line options gets passed as sys.argv[1:]. This strips the leading option, which is “./log_server.py”, from the list and passed the rest of the list. In case you are unfamiliar with python slicing, This is what happens:

python
from server_client_test import str_to_argv
>>> argv = str_to_argv('./log_server..py --Xlog=log.log --log-append=False')
>>> argv
['./log_server..py', '--Xlog=log.log', '--log-append=False']
>>> argv[1:]
['--Xlog=log.log', '--log-append=False']

The initial item in the list disappears leaving the rest of the list intact.

Now comes an interesting part. When log_server attempts to make sense of the –Xlog option, it cannot because it is illegal.

log_server responds to an illegal run-time option by exiting. Exactly what we want. BUT – If log server exits, how can we continue to test?

A bit of unittest framework magic does the trick:

        with self.assertRaises(SystemExit) as err:
            log_server.process_cmd_line(argv[1:])
        self.assertEqual(err.exception.code, 1)

The first 2 lines capture that log_server sys.exit(1) in the “err” variable.

Capturing an error condition from log_server serves the important purpose of detecting error conditions on the command line. Specifically, know the difference between a normal log_server exit and a log_server dying due to an error condition make it easier to build bash scripts and other integrations much easier.

While we expect log_server to run “forever” in normal operating circumstances, detecting the reason it exited allows us to take proper remedial actions.

For our unit test, this exit code get placed in err.exception.code and is code 1.

The unittest framework checks this with the line of:

        self.assertEqual(err.exception.code, 1)

This technique of creating an invalid command line option, calling log_server, capturing the error condition and testing the exit code gets repeated over and over for all the options.

Testing Alias Names for Options

Using alternate keywords for the same purpose can be tested as well. --log_append and --log-append map to the same, identical handling internally. This looks like:

    def test_server_log_append(self):
        print(FCN_FMT % function_name())

        argv = str_to_argv('%s --log-append=true')
        params = log_server.process_cmd_line(argv)
        self.assertEqual(True, params['log_append'])

        argv = str_to_argv('%s --log_append=true')
        params = log_server.process_cmd_line(argv)
        self.assertEqual(True, params['log_append'])

The return value of log_server.process_cmd_line(argv) contains a dictionary of keyword/values where the keywords are the option name and value is the value passed.

The internal keyword for log_append in the returned params variable is “log_append”. The chosen underscore version is pythonic.

In log_server.py the logic that handles this logic is:

        if opt in ['--log-append', '--log_append']:
            params['log_append'] = True if arg.lower() == 'true' else False
            continue

‘opt’ for our example would be one of the two values in the list. The exact value is ‘–log-append’ because that is what we used in the first test.

To determine if append to a file or not has been requested, a simple condition determines the outcome. The actual value gets lower cased to allow some flexibility of input. Notice the condition only checks for ‘true’. Any string other than ‘true’, case irrelevant, becomes False.

Interested readers may easily tighten the test to validate a ‘false’ value as well.

Testing Opening a File

test_server_open_log_file passes a variety of illegal filenames to verify that log_server properly exits. For example, writing to ‘/’, the root directory, should only be allowed if the user is root. Writing to a directory instead of a file is just plain wrong. Likewise, writing to a system file such as /var/log/messages and not root is illegal.

However, writing to perfectly valid areas such as /tmp/ABCDEF passes with ease. (Ignore the crazy /tmp filename!). Everyone has permission to write to /tmp. Users can also write to their own directories.

Test All Options Changed

log_server uses process_cmd_line() with a params dictionary of default values while processing the command line. Testing that each of these options properly changes should be part of this test as well.

The test all options logic consists of recording a non-default value for each options. Then a command line string gets built containing these non-default value. The string become an argv and the command line get parsed with process_cmd_line() as before.

Each modified parameter gets tested that must reflect the changes noted in the variables. For example, setting “port = 12345” and parsing results in params[‘port’] == 12345.

Conclusions and Future Work

Some of you may have astutely noticed that actually running log_server and log_client was not part of this testing.

Integration tests involves running multiple programs and testing their results.

Yes! Integration testing needs to wrap itself around our system. But integration testing can be tricky. It involves spawning processes, noting exit code, sending messages that result in log files, reading and comparing these log files to some standard, etc.

These simpler unit test are enough for now.