Daniel Shahaf d5a4a6e195 tests: Make $expected_mismatch skip the cardinality check, rather
than consider it an expected failure.

With this change, if $expected_region_highlight and $region_highlight
coincidentally have the same number of elements, the test won't be considered
to fail.

This is useful in conjunction with the next commit, q.v..

At this time, no tests set $expected_mismatch explicitly.  However, the
commit after next (this commit's grandchild) will add a test that will
set $expected_mismatch implicitly, using the functionality in the next
commit (this commit's child).
2019-11-10 11:48:40 +00:00

110 lines
3.4 KiB
Markdown

zsh-syntax-highlighting / tests
===============================
Utility scripts for testing zsh-syntax-highlighting highlighters.
The tests harness expects the highlighter directory to contain a `test-data`
directory with test data files.
See the [main highlighter](../highlighters/main/test-data) for examples.
Each test should define the string `$BUFFER` that is to be highlighted and the
array parameter `$expected_region_highlight`.
The value of that parameter is a list of strings of the form `"$i $j $style"`.
or `"$i $j $style $todo"`.
Each string specifies the highlighting that `$BUFFER[$i,$j]` should have;
that is, `$i` and `$j` specify a range, 1-indexed, inclusive of both endpoints.
`$style` is a key of `$ZSH_HIGHLIGHT_STYLES`.
If `$todo` exists, the test point is marked as TODO (the failure of that test
point will not fail the test), and `$todo` is used as the explanation.
If a test sets `$skip_test` to a non-empty string, the test will be skipped
with the provided string as the reason.
If a test sets `unsorted=1` the order of highlights in `$expected_region_highlight`
need not match the order in `$region_highlight`.
Normally, tests fail if `$expected_region_highlight` and `$region_highlight`
have different numbers of elements. Tests may set `$expected_mismatch` to an
explanation string (like `$todo`) to avoid this and skip the cardinality check.
**Note**: `$region_highlight` uses the same `"$i $j $style"` syntax but
interprets the indexes differently.
**Note**: Tests are run with `setopt NOUNSET WARN_CREATE_GLOBAL`, so any
variables the test creates must be declared local.
**Isolation**: Each test is run in a separate subshell, so any variables,
aliases, functions, etc., it defines will be visible to the tested code (that
computes `$region_highlight`), but will not affect subsequent tests. The
current working directory of tests is set to a newly-created empty directory,
which is automatically cleaned up after the test exits. For example:
```zsh
setopt PATH_DIRS
mkdir -p foo/bar
touch foo/bar/testing-issue-228
chmod +x foo/bar/testing-issue-228
path+=( "$PWD"/foo )
BUFFER='bar/testing-issue-228'
expected_region_highlight=(
"1 21 command" # bar/testing-issue-228
)
```
Writing new tests
-----------------
An experimental tool is available to generate test files:
```zsh
zsh -f tests/generate.zsh 'ls -x' acme newfile
```
This generates a `highlighters/acme/test-data/newfile.zsh` test file based on
the current highlighting of the given `$BUFFER` (in this case, `ls -x`).
_This tool is experimental._ Its interface may change. In particular it may
grow ways to set `$PREBUFFER` to inject free-form code into the generated file.
Highlighting test
-----------------
[`test-highlighting.zsh`](tests/test-highlighting.zsh) tests the correctness of
the highlighting. Usage:
```zsh
zsh test-highlighting.zsh <HIGHLIGHTER NAME>
```
All tests may be run with
```zsh
make test
```
which will run all highlighting tests and report results in [TAP format][TAP].
By default, the results of all tests will be printed; to show only "interesting"
results (tests that failed but were expected to succeed, or vice-versa), run
`make quiet-test` (or `make test QUIET=y`).
[TAP]: http://testanything.org/
Performance test
----------------
[`test-perfs.zsh`](tests/test-perfs.zsh) measures the time spent doing the
highlighting. Usage:
```zsh
zsh test-perfs.zsh <HIGHLIGHTER NAME>
```
All tests may be run with
```zsh
make perf
```