Testing

sultanc test builds a program's tests and runs each one in its own process. A test is a function marked @@test (@@اختبار in Arabic). The command reports every test as passed or failed, says why each failure happened, and ends with a summary and an exit status a script can check.

import std.debug.assert;
import std.error.failure;

def double(x: int): int { return x * 2; }

@@test
def doubles(): void {
    assert(double(2) == 4);
}

@@test
def doubles_twenty(): result int, Error {
    if (double(20) != 40) { return result.err(error_new("double(20) is not 40")); };
    return result.ok(0);
}

sultanc test src/main.sn

PASS doubles
PASS doubles_twenty

test result: ok. passed: 2; failed: 0; filtered out: 0

Declaring a test#

@@test and @@اختبار are two spellings of one built-in definition-time annotation, the same in either language and in any file. It marks an ordinary function, with a body:

• it takes no parameters;
• it returns nothing (void / فراغ), or result int, Error (نتيجة عدد, فشل, the error type of std.error.failure), the result main may return. A test that returns result.ok(...) passes whatever the value; a test that returns result.err(e) fails, and the run shows the error's text. (A result void, Error has no value to return, so the canonical result carries an int.)

Any module of the root package may declare tests: the entry file and the files of its package that it imports. The tests of dependency packages (from --package-graph) are not run, as Cargo runs a crate's own tests and never its dependencies'. A dependency's tests run when that package is the root. A test is found by its declaration, after the program is parsed and checked, not by searching text. Its name is the name as written; a test declared with an English name is listed with it, one declared with an Arabic name with that. A function with other names (def adds alias يجمع) has each of them as a spelling: --filter and --test match any spelling, and the reports use the first.

The compiler rejects a test it cannot run before anything is built, and reports each one at its name:

sultanc --explain <code> explains each.

One validation applies these rules, over the frontend's Name and Type facts (compiler/test_mode/validation.sn). sultanc test rejects with it before anything is lowered, and an editor's analysis (a tooling request of the frontend, as the language server makes) reports the same diagnostics at the test's name while it is written, among its other diagnostics. A build or check reports none of them.

Ordinary builds#

Outside sultanc test, the annotation has no effect. A build or sultanc run of the same program runs its main and never a test, and a test function is emitted only when the program itself calls it. A program whose tests live next to its code builds as before.

test and اختبار name the built-in annotation: a definition-time decorator of the program with either name is not called through @@test.

A program's definition-time decorators run in test mode as in a build: the entry the compiler gives the tests first runs every registration a build's entry runs before main's body, in the same order, compiled and with --interpret. A counter a @@register decorator increments is 1 in a test as in main.

Running tests#

sultanc test <file> builds one executable through the same pipeline as any build (the same frontend, MSIR, optimizer and backend). The compiler gives it an entry of its own, synthesized in MSIR from the discovered tests, in place of the program's main. The command then runs that executable once per test, with the test's number as its argument, so a test that panics, fails an assertion, traps, exits or crashes cannot stop the others or see their state. Each test's standard output and standard error are captured and shown under its report when it fails.

A failure says what ended the test:

FAIL assertion_fails: exit status 1
    --> tests/main.sn:19:5
    note: the test is declared at tests/main.sn:18:5
    error[SULTANC-RUNTIME-PROGRAM-STOPPED]: the program stopped — An assertion failed.
FAIL returns_error: returned an error
    --> tests/main.sn:28:5
    double(20) is 40, expected 41
FAIL divides_by_zero: terminated by signal 4
    --> tests/main.sn:35:5

• returned an error: the test returned result.err(e); the text below is e (error_describe).
• exit status N: the test ended the process. A panic and a failed assertion end it with status 1 and write why (the program's runtime-failure frame, and the panic's message) to standard error, which is shown below. Another status is the test's own (exit(7)).
• terminated by signal N: the process was stopped by a signal (a trap, a fault).
• could not be started: the executable could not be run.

The --> line is where the test failed when it says so: a failed assertion reports the place of the assert call (below), and the test's declaration follows as a note. Otherwise (a panic, a returned error, an exit status, a signal) it is where the test is declared.

Where an assertion failed: @@track_caller#

assert / تأكد and assert_with_message / تأكد_برسالة (std.debug.assert) print the place of the call that failed under the failure frame, --> <file>:<line>:<column>, and sultanc test shows it on its --> line. They are marked @@track_caller (@@يتتبع_المستدعي), as Rust's #[track_caller], and any function can be:

import std.debug.assert;
import std.debug.location;

@@track_caller
def assert_positive(value: int, location: source_location): void {
    assert(value > 0);
}

• The marked function's last parameter is a source_location (موقع_مصدر, std.debug.location): a reference to the file's path, the 1-based line and column, a plain value. Callers do not pass it: every use of the function has its type without that parameter, and the compiler passes the place of each call (source_location_text, source_location_file, source_location_line, source_location_column read it).
• Inside another marked function, a call passes the location that function received, so assert in assert_positive above reports the line that called assert_positive. A plain helper is not marked: an assert in it reports its own line.
• A function value of a marked function (let f: fn(bool): void = assert;) passes the place of the function's definition, as Rust's function-pointer shim does.
• Passing the location yourself is an error (SULTANC-TRACK-CALLER-EXPLICIT-LOCATION); so is marking a function that is generic, external, has compile-time parameters or has no source_location last parameter (SULTANC-TRACK-CALLER-SIGNATURE).

The location is an ordinary constant argument (MSIR lowering builds it, compiler/msir/lowering/caller_location.sn; the Type pass removes the parameter from the function's uses, compiler/frontend/types/semantic_pipeline.sn), so the native and LLVM backends and the interpreters need nothing of their own.

The summary counts the tests that passed, failed and were left out by --filter, --test and --tests-in. The words are those of --lang, like every message of the compiler; the program's own failure frames follow the program's language, as for any program.

Options#

-o and --emit do not apply: the executable is temporary and removed after the run.

Other targets#

A test executable is never run on a host that cannot run it. With --target naming another target, the tests are built (which checks them and the program for that target) and not run:

note: the tests were built, not run: this host cannot run programs for the target aarch64-linux-gnu

--list still lists them. The status is 0 when the build succeeded.

Exit status#

With --test-index the status is the test's own: 0 passed, 1 a panic or a failed assertion, 3 a returned error, any other the test's exit status, and 128 + N for signal N.

How it works#

• Discovery (compiler/test_mode/discovery.sn) reads the annotation where the frontend does (compiler/frontend/annotations/definition_decorators.sn recognizes it and removes it before the type phase): each module of the checked program that contains @@ is parsed again by the same parser for the same target, and each declaration marked @@test is recorded with its name and span.
• The harness (compiler/test_mode/harness.sn) matches every discovered test to the MSIR function lowered from the same declaration and validates its signature against the MSIR types of std.testing.harness. It then adds the entry: for each test i, if test_selected(i) { [r :=] test_i(); return 0 or finish_test_result(r) }, then return 2. No SultanC text is generated. The program's own main is not called.
• std.testing.harness (library/std/testing/harness.sn) is the runtime: it reads the test number from the program's first argument (std.process.argument) and prints a returned error. It joins the program only in test mode.
• The driver (compiler/driver/process_driver.sn) runs each test with std.process.command and captures its output. On Linux that layer starts and waits for processes with raw system calls (no C library); on other systems it uses the system's own process interface, as std does everywhere.