Purpose of this document
------------------------
This document contains information about the project's testing facilities.

In a nutshell, the following facilities exist:
- automated unit test suite using Python's unittest module; this test suite can
  be run on the command line or from within the Eclipse IDE
- automated end-to-end test suite that can be run on the command line
- manual tests by running mkroesti on the command line


Run unit tests from the command line
------------------------------------
The following command runs all automated tests from the command line, from
within the git working tree:

  ./setup.py test

To run tests from only one module (test.algorithm.py in this example):

  setup.py test --suite=tests.test_algorithm


Run unit tests from within Eclipse
----------------------------------
There are two types of run configuration that can be used to run mkroesti's
automated test suite from within Eclipse:

- the "Python unittest" configuration type, which runs the tests directly from
  the "tests" package
- The "Python Run" configuration type, which is using setup.py to run the tests

Set up a run configuration of type "Python unittest":
- PyDev must be installed for the following to work
- Right-click on the folder that contains your tests and select
  "Run as... -> Python unittest"
- Configure the resulting run configuration as follows:
  - name = mkroesti tests
  - project = mkroesti
  - main module = ${workspace_loc:mkroesti/src/packages/tests} (or any other folder that contains tests)
  - on the "Arguments" tab: working directory = ${workspace_loc:mkroesti/src/packages}

Set up a run configuration of type "Python Run":
- PyDev must be installed for the following to work
- Open "Run configurations" dialog
- Double-click "Python Run" to create a new configuration
- Configure the resulting run configuration as follows:
  - name = mkroesti tests (setup.py)
  - project = mkroesti
  - main module = ${workspace_loc:mkroesti/setup.py}
  - on the "Arguments" tab: program arguments = "test" 


Run end-to-end test suite from the command line
-----------------------------------------------
The end-to-end test suite is a shell script that exercises the mkroesti
front-end. The goals of this test suite are:
- Have a black-box oriented end-to-end test suite (as opposed to the more
  detailed, white-box oriented unit tests)
- Exercises *all* algorithms available in the current environment
- Exercise *all* input methods

To run the test suite from within a git working tree, do the following:

  cd src/scripts
  PYTHONPATH=../packages ./mkroesti-test

Documentation of the test suite:
- command line usage: run the script using --help
- test data: read the file packages/tests/testdata/README

Note: Unfortunately, the test of the interactive input method sometimes fails.
The problem is that, for some unknown reason, the expect utility sometimes fails
to properly run the script that exercises the interactive input method. For the
moment, the only workaround for this is to re-run "mkroesti-test" and hope for
the best.


Run mkroesti from the command line
----------------------------------
The mkroesti shell script can be run on the command line to execute a manual
end-to-end test, e.g. to compare the script's output with predefined hashes.

To run mkroesti from within a git working tree, do the following:

  cd src/scripts
  PYTHONPATH=../packages ./mkroesti --batch foo


Writing new unit tests
----------------------
New test modules must be placed into the "tests" package. If the module is
intended to be run by setup.py, it must also be added to the callable

  tests.__init__.py.allTests()

Last but not least, each module should contain the following block of code so
that it can be run standalone:

  if __name__ == "__main__":
      unittest.main()

