Testing Framework

The Mars Test Framework (MTF) is a suite of tools for testing Mars program code, and the Mars compiler itself.

We ultimately wish to test three things:

  • Compilation (correct and incorrect code, errors and warnings),
  • Compiler-inferred properties, and
  • Runtime correctness (system and unit tests on Mars programs)

We currently test the first and third of these.

Test suite

A test suite is simply a collection of .mar files (Mars source code files) and .mtc files (Mars Test Case). MTC files are optional, and correspond to a .mar file of the same name.

A “test case” is a single .mar file and possibly a .mtc file of the same name. Test cases are always just a single .mar file (though they can import other modules, eg. to test importing).

MTC file format

MTC files are YAML text files which provide instructions about the test case, controlling the behaviour of the testing system.

At the top-level is a dict mapping section names to dicts.

Valid sections are: compile, run (both optional).

eg:

compile:
    (compile section)
run:
    (run section)

Inside each section is a dict specific to that section (see sections below).

Note that Unicode characters in MTC files are encoded with UTF-8 before being used, unless otherwise noted.

Compilation testing

Compilation testing involves compiling a .mar file just to check whether compilation succeeds or fails, and check that appropriate warnings and errors are produced. This sort of test does not involve running the code at all.

The most basic test case is a single .mar file with nothing special about it. Such a case will be compiled and will pass if it compiles successfully without warnings, and fail if it does not compile, or there are warnings.

If the case has a .mtc file with a compile section, that section is consulted for further information (which can affect the runtime and unit tests as well). Its dictionary has the following attributes:

  • outcome: succeed | fail (succeed is default). This dictates what the outcome of compilation should be (assuming a correct implementation) – succeed means the program should be accepted, and fail means the program should be rejected.
  • errors: string data – the expected output of stderr from the compiler. If omitted, expects stderr to be empty.
  • expect: pass | fail | compile_error (pass is default). This dictates whether the current implementation is correct – pass means it behaves correctly (compiles correctly or fails to compile as required, depending on outcome), fail means it doesn’t behave correctly when the program should be rejected (compiles when it shouldn’t, or gives the wrong compiler error message), and compile_error means it doesn’t compile when it should.
  • environ: Dict mapping environment strings to values. If supplied, will be appended to the environment (replacing existing values), but existing variables will not be removed. The environment will be used for all runtime and unit tests for the given module.

Hence, the default value for compile is as follows:

compile:
    outcome: succeed
    errors: ""
    expect: pass
    environ: {}

The test framework will only report unexpected outcomes. This means that passing cases are not normally reported, but a passing case which is expected to fail will be reported, so you know you have fixed something. Use expect: fail or expect: compile_error if you have a test case which you know fails but intend to fix.

Runtime testing

Runtime testing is executing functions and checking their output. It implies that compilation succeeds (if both compilation testing and runtime testing are specified, compilation must not expect failure).

Compilation is not performed with the -m option, so the program does not do anything by default.

Runtime testing requires an .mtc file. The run section is consulted for information on how to do the test.

It is a list of individual cases, which are tested separately. Each case has a dictionary.

The case’s dictionary has the following attributes:

  • call: Mars expression to be evaluated after loading the module.

    Required. (Typically used to call a function to be tested).

  • stdin: Input passed to stdin (may be text or binary). If omitted, empty.

  • stdout: Expected stdout. If omitted, is not tested.

  • stderr: Expected stdout. If omitted, expect empty.

  • expect: pass | fail Will only report unexpected outcomes. (If expecting fail, then “pass” will be reported. Use this if you have a test case which you know fails but intend to fix) (pass is default).

Note

The test framework has no way to distinguish between when a program is computing and when it is waiting for input. Therefore, if a test case waits for input and none is supplied, MTF will wait forever. Be sure to supply a line of stdin for each line expected by the program.

Unit testing

The third kind of test is a “unit test”, which tests individual functions inside Mars files. These are usually written to verify the correctness of Mars code (eg. library code), rather than the compiler.

A module (unittest) is supplied in the Mars standard library, which can be imported by test suites and called. It doesn’t produce nice output - instead, it dumps machine-readable output to stdout, which is expected to be processed by a wrapper.

The Mars Test Framework is such a wrapper. It can be passed a Mars file and will automatically execute any function beginning with “test_”, and filter the output into nice human-readable output. This does not require an .mtc file.

Note that like compile and runtime testing, you are able to specify that a case is expected to fail (but is intended to be fixed), using begin_case_pending.

See unittest for more information.

Example MTC file

compile:
    outcome: succeed
    errors: |
        Warning: Some warning message
    environ: {"SOME_VAR": "some value"}
run:
    - call: foo(2)
      stdin: |
          forty two
      stdout: |
          expected stdout text
          perhaps across several lines
    - call: foo(1)
      stdin: "forty one\n"
      stderr: "Incorrect\n"

Table Of Contents

Previous topic

Type dictionaries

This Page