Add docs/testsuite

docs/Makefile

@@ -1,5 +1,5 @@
 
-all: hacking-howto.html debugging.html userguide.html ipc.html multi-monitor.html wsbar.html refcard.pdf
+all: hacking-howto.html debugging.html userguide.html ipc.html multi-monitor.html wsbar.html refcard.pdf testsuite.html
 
 hacking-howto.html: hacking-howto
 	asciidoc -a toc -n $<
@@ -10,6 +10,9 @@ debugging.html: debugging
 userguide.html: userguide
 	asciidoc -a toc -n $<
 
+testsuite.html: testsuite
+	asciidoc -a toc -n $<
+
 ipc.html: ipc
 	asciidoc -a toc -n $<
 

docs/testsuite

@@ -0,0 +1,446 @@
+i3 testsuite
+============
+Michael Stapelberg <michael+i3@stapelberg.de>
+September 2011
+
+This document explains how the i3 testsuite works, how to use it and how to
+extend it. It is targeted at developers who not necessarily have been doing
+testing before or have not been testing in Perl before. In general, the
+testsuite is not of interest for end users.
+
+
+== Introduction
+
+The i3 testsuite is a collection of files which contain testcases for various
+i3 features. Some of them test if a certain workflow works correctly (moving
+windows, focus behaviour, …). Others are regression tests and contain code
+which previously made i3 crash or lead to unexpected behaviour. They then check
+if i3 still runs (meaning it did not crash) and if it handled everything
+correctly.
+
+The goal of having these tests is to automatically find problems and to
+automatically get a feel for whether a change in the source code breaks any
+existing feature. After every modification of the i3 sourcecode, the developer
+should run the full testsuite. If one of the tests does not pass (but fails),
+the corresponding problem should be fixed (or, in some cases, the testcase has
+to be modified). For every bugreport, a testcase should be written to test the
+correct behaviour. Initially, it will fail, but after fixing the bug, it will
+pass. This ensures (or increases the chance) that bugs which have been fixed
+once will never be found again.
+
+Also, when implementing a new feature, a testcase might be a good way to be
+able to easily test if the feature is working correctly. Many developers will
+test manually if everything works. Having a testcase not only helps you with
+that, but it will also be useful for every future change.
+
+== Implementation
+
+For several reasons, the i3 testsuite has been implemented in Perl:
+
+1. Perl has a long tradition of testing. Every popular/bigger Perl module which
+   you can find on CPAN will not only come with documentation, but also with
+   tests. Therefore, the available infrastructure for tests is comprehensive.
+   See for example the excellent http://search.cpan.org/perldoc?Test::More
+   and the referenced http://search.cpan.org/perldoc?Test::Tutorial.
+
+2. Perl is widely available and has a well-working package infrastructure.
+3. The author is familiar with Perl :).
+
+Please do not start programming language flamewars at this point.
+
+=== Mechanisms
+
+==== Script: complete-run
+
+The testcases are run by a script called +complete-run.pl+. It runs all
+testcases by default, but you can be more specific and let it only run one or
+more testcases. Also, it takes care of starting up a separate instance of i3
+with an appropriate configuration file and creates a folder for each run
+containing the appropriate i3 logfile for each testcase.  The latest folder can
+always be found under the symlink +latest/+. It is recommended that you run the
+tests on one or more separate X server instances (you can only start one window
+manager per X session), for example using the provided Xdummy script.
++complete-run.pl+ takes one or more X11 display specifications and parallelizes
+the testcases appropriately:
+
+.Example invocation of complete-run.pl+
+---------------------------------------
+$ cd ~/i3/testcases
+
+# start two dummy X11 instances in the background
+$ ./Xdummy :1 &
+$ ./Xdummy :2 &
+
+$ ./complete-run.pl -d :1,:2
+# output omitted because it is very long
+All tests successful.
+Files=78, Tests=734, 27 wallclock secs ( 0.38 usr  0.48 sys + 17.65 cusr  3.21 csys = 21.72 CPU)
+Result: PASS
+
+$ ./complete-run.pl -d :1 t/04-floating.t
+[:3] i3 startup: took 0.07s, status = 1
+[:3] Running t/04-floating.t with logfile testsuite-2011-09-24-16-06-04-4.0.2-226-g1eb011a/i3-log-for-04-floating.t
+[:3] t/04-floating.t finished
+[:3] killing i3
+output for t/04-floating.t:
+ok 1 - use X11::XCB::Window;
+ok 2 - The object isa X11::XCB::Window
+ok 3 - Window is mapped
+ok 4 - i3 raised the width to 75
+ok 5 - i3 raised the height to 50
+ok 6 - i3 did not map it to (0x0)
+ok 7 - The object isa X11::XCB::Window
+ok 8 - i3 let the width at 80
+ok 9 - i3 let the height at 90
+ok 10 - i3 mapped it to x=1
+ok 11 - i3 mapped it to y=18
+ok 12 - The object isa X11::XCB::Window
+ok 13 - i3 let the width at 80
+ok 14 - i3 let the height at 90
+1..14
+
+All tests successful.
+Files=1, Tests=14,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.19 cusr  0.03 csys =  0.23 CPU)
+Result: PASS
+
+$ less latest/i3-log-for-04-floating.t
+----------------------------------------
+
+==== IPC interface
+
+The testsuite makes extensive use of the IPC (Inter-Process Communication)
+interface which i3 provides. It is used for the startup process of i3, for
+terminating it cleanly and (most importantly) for modifying and getting the
+current state (layout tree).
+
+See [http://i3wm.org/docs/ipc.html] for documentation on the IPC interface.
+
+==== X11::XCB
+
+In order to open new windows, change attributes, get events, etc., the
+testsuite uses X11::XCB, a new (and quite specific to i3 at the moment) Perl
+module which uses the XCB protocol description to generate Perl bindings to
+X11. They work in a very similar way to libxcb (which i3 uses) and provide
+relatively high-level interfaces (objects such as +X11::XCB::Window+) aswell as
+access to the low-level interface, which is very useful when testing a window
+manager.
+
+=== Filesystem structure
+
+In the git root of i3, the testcases live in the folder +testcases+. This
+folder contains the +complete-run.pl+ and +Xdummy+ scripts and a base
+configuration file which will be used for the tests. The different testcases
+themselve can be found in the conventionally named subfolder +t+:
+
+.Filesystem structure
+--------------------------------------------
+├── testcases
+│   ├── complete-run.pl
+│   ├── i3-test.config
+│   ├── t
+│   │   ├── 00-load.t
+│   │   ├── 01-tile.t
+│   │   ├── 02-fullscreen.t
+│   │   ├── ...
+│   │   ├── omitted for brevity
+│   │   ├── ...
+│   │   ├── 74-regress-focus-toggle.t
+│   │   └── lib
+│   │       └── i3test.pm
+│   └── Xdummy
+--------------------------------------------
+
+== Anatomy of a testcase
+
+Learning by example is definitely a good strategy when you are wondering how to
+write a testcase. Let's take +t/11-goto.t+ as an easy example and go through it
+step by step:
+
+.t/11-goto.t: Boilerplate
+----------------------
+#!perl
+# vim:ts=4:sw=4:expandtab
+
+use i3test;
+use File::Temp;
+
+my $x = X11::XCB::Connection->new;
+-----------------------
+
+This is what we call boilerplate. It exists at the top of every test file (to
+some extent). The first line is the shebang, which specifies that this file is
+a Perl script. The second line contains VIM specific settings on how to
+edit/format this file (use spaces instead of tabs, indent using 4 spaces).
+Afterwards, the +i3test+ module is used. This module contains i3 testsuite
+specific functions which you are strongly encouraged to use. They make writing
+testcases a lot easier and will make it easier for other people to read your
+tests.
+
+The next line uses the +File::Temp+ module. This is specific to this testcase,
+because it needs to generate a temporary name during the test. Many testcases
+use only the +i3test+ module.
+
+The last line opens a connection to X11. You might or might not need this in
+your testcase, depending on whether you are going to open windows (etc.) or
+only use i3 commands.
+
+.t/11-goto.t: Setup
+----------------------
+my $tmp = fresh_workspace;
+
+cmd 'split h';
+----------------------
+
+The first line calls i3test's +fresh_workspace+ function which looks for a
+currently unused workspace, switches to it, and returns its name. The variable
++$tmp+ will end up having a value such as +"/tmp/87kBVcHbA9"+. Note that this
+is not (necessarily) a valid path, it's just a random workspace name.
+
+So, now that we are on a new workspace, we ensure that the workspace uses
+horizontal orientation by issuing the +split h+ command (see the i3 User's
+Guide for a list of commands). This is not strictly necessary, but good style.
+In general, the +cmd+ function executes the specified i3 command by using the
+IPC interface and returns once i3 acknowledged the command.
+
+.t/11-goto.t: Setup
+----------------------
+#####################################################################
+# Create two windows and make sure focus switching works
+#####################################################################
+
+my $top = open_window($x);
+my $mid = open_window($x);
+my $bottom = open_window($x);
+----------------------
+
+In every major section of a testcase, you should put a comment like the one
+above. This makes it immediately clear how the file is structured.
+
+The +open_window+ function opens a standard window, which will then be put into
+tiling mode by i3. If you want a floating window, use the
++open_floating_window+ function. These functions accept the same parameters as
++X11::XCB::Window->new+, see the i3test documentation at TODO.
+
+.t/11-goto.t: Helper function
+----------------------
+#
+# Returns the input focus after sending the given command to i3 via IPC
+# end sleeping for half a second to make sure i3 reacted
+#
+sub focus_after {
+    my $msg = shift;
+
+    cmd $msg;
+    sync_with_i3 $x;
+    return $x->input_focus;
+}
+----------------------
+
+This section defines a helper function which will be used over and over in this
+testcase. If you have code which gets executed more than once or twice
+(depending on the length of your test, use your best judgement), please put it
+in a function. Tests should be short, concise and clear.
+
+The +focus_after+ function executes a command and returns the X11 focus after
+the command was executed. The +sync_with_i3+ command makes sure that i3 could
+push its state to X11. See <<i3_sync>> to learn how this works exactly.
+
+.t/11-goto.t: Test assumptions
+----------------------
+$focus = $x->input_focus;
+is($focus, $bottom->id, "Latest window focused");
+
+$focus = focus_after('focus left');
+is($focus, $mid->id, "Middle window focused");
+----------------------
+
+Now, we run the first two real tests. They use +Test::More+'s +is+ function,
+which compares two values and prints the differences if they are not the same.
+After the arguments, we supply a short comment to indicate what we are testing
+here. This makes it vastly more easy for the developer to spot which testcase
+is the problem in case one fails.
+
+The first test checks that the most recently opened window is focused.
+Afterwards, the command +focus left+ is issued and it is verified that the
+middle window now has focus.
+
+Note that this is not a comprehensive test of the +focus+ command -- we would
+have to test wrapping, focus when using a more complex layout, focusing the
+parent/child containers, etc. But that is not the point of this testcase.
+Instead, we just want to know if +$x->input_focus+ corresponds with what we are
+expecting. If not, something is completely wrong with the test environment and
+this trivial test will fail.
+
+.t/11-goto.t: Test that the feature does not work (yet)
+----------------------
+#####################################################################
+# Now goto a mark which does not exist
+#####################################################################
+
+my $random_mark = mktemp('mark.XXXXXX');
+
+$focus = focus_after(qq|[con_mark="$random_mark"] focus|);
+is($focus, $mid->id, "focus unchanged");
+----------------------
+
+In this new major section, a random mark (mark is an identifier for a window,
+see "VIM-like marks" in the i3 User’s Guide) will be generated. Afterwards, we
+test that trying to focus that mark will not do anything. This is important: Do
+not only test that using a feature has the expected outcome, but also test that
+using it without properly initializing it does no harm. This command could for
+example have changed focus anyways (a bug) or crash i3 (obviously a bug).
+
+.t/11-goto.t: Test that the feature does work
+----------------------
+cmd "mark $random_mark";
+
+$focus = focus_after('focus left');
+is($focus, $top->id, "Top window focused");
+
+$focus = focus_after(qq|[con_mark="$random_mark"] focus|);
+is($focus, $mid->id, "goto worked");
+----------------------
+
+Remember: Focus was on the middle window (we verified that earlier in "Test
+assumptions"). We now mark the middle window with our randomly generated mark.
+Afterwards, we switch focus away from the middle window to be able to tell if
+focusing it via its mark will work. If it does work (next test), the goto
+command works.
+
+.t/11-goto.t: Test corner case
+----------------------
+# check that we can specify multiple criteria
+
+$focus = focus_after('focus left');
+is($focus, $top->id, "Top window focused");
+
+$focus = focus_after(qq|[con_mark="$random_mark" con_mark="$random_mark"] focus|);
+is($focus, $mid->id, "goto worked");
+----------------------
+
+Now we test the same feature, but specifying the mark twice in the command.
+This should have no effect, but let’s be sure: test it and see if things go
+wrong.
+
+.t/11-goto.t: Test second code path
+----------------------
+#####################################################################
+# Check whether the focus command will switch to a different
+# workspace if necessary
+#####################################################################
+
+my $tmp2 = fresh_workspace;
+
+is(focused_ws(), $tmp2, 'tmp2 now focused');
+
+cmd qq|[con_mark="$random_mark"] focus|;
+
+is(focused_ws(), $tmp, 'tmp now focused');
+----------------------
+
+This part of the test checks that focusing windows by mark works across
+workspaces. It uses i3test's +focused_ws+ function to get the current
+workspace.
+
+.t/11-goto.t: Test second code path
+----------------------
+done_testing;
+----------------------
+
+The end of every testcase has to contain the +done_testing+ line. This tells
++complete-run.pl+ that the test was finished successfully. If it does not
+occur, the test might have crashed during execution -- some of the reasons why
+that could happen are bugs in the used modules, bugs in the testcase itself or
+an i3 crash resulting in the testcase being unable to communicate with i3 via
+IPC anymore.
+
+[[i3_sync]]
+== Appendix A: The i3 sync protocol
+
+Consider the following situation: You open two windows in your testcase, then
+you use +focus left+ and want to verify that the X11 focus has been updated
+properly. Sounds simple, right? Let’s assume you use this straight-forward
+implementation:
+
+.Racey focus testcase
+-----------
+my $left = open_window($x);
+my $right = open_window($x);
+cmd 'focus left';
+is($x->input_focus, $left->id, 'left window focused');
+----------
+
+However, the test fails. Sometimes. Apparantly, there is a race condition in
+your test. If you think about it, this is because you are using two different
+pieces of software: You tell i3 to update focus, i3 confirms that, and then you
+ask X11 to give you the current focus. There is a certain time i3 needs to
+update the X11 state. If the testcase gets CPU time before X11 processed i3's
+requests, the test will fail.
+
+image::i3-sync.png["Diagram of the race condition", title="Diagram of the race condition"]
+
+One way to "solve" this would be to add +sleep 0.5;+ after the +cmd+ call.
+After 0.5 seconds it should be safe to assume that focus has been updated,
+right?
+
+In practice, this usually works. However, it has several problems:
+
+1. This is obviously not a clean solution, but a workaround. Ugly.
+2. On very slow machines, this might not work. Unlikely, but in different
+   situations (a delay to wait for i3 to startup) the necessary time is much
+   harder to guess, even for fast machines.
+3. This *wastes a lot of time*. Usually, your computer is much faster than 0.5s
+   to update the status. However, sometimes, it might take 0.4s, so we can’t
+   make it +sleep 0.1+.
+
+To illustrate how grave the problem with wasting time actually is: Before
+removing all sleeps from the testsuite, a typical run using 4 separate X
+servers took around 50 seconds on my machine. After removing all the sleeps,
+we achieved times of about 25 seconds. This is very significant and influences
+the way you think about tests -- the faster they are, the more likely you are
+to check whether everything still works quite often (which you should).
+
+What I am trying to say is: This adds up quickly and makes the test suite less
+robust.
+
+The real solution for this problem is a mechanism which I call "the i3 sync
+protocol". The idea is to send a request (which does not modify state) via X11
+to i3 which will then be answered. Due to the request's position in the event
+queue (*after* all previous events), you can be sure that by the time you
+receive the reply, all other events have been dealt with by i3 (and, more
+importantly, X11).
+
+image::i3-sync-working.png["Diagram of the i3 sync solution", title="Diagram of the i3 sync solution"]
+
+=== Implementation details
+
+The client which wants to sync with i3 initiates the protocol by sending a
+ClientMessage to the X11 root window:
+
+.Send ClientMessage
+-------------------
+# Generate a ClientMessage, see xcb_client_message_t
+my $msg = pack "CCSLLLLLLL",
+    CLIENT_MESSAGE, # response_type
+    32,     # format
+    0,      # sequence
+    $root,  # destination window
+    $x->atom(name => 'I3_SYNC')->id,
+
+    $_sync_window->id,    # data[0]: our own window id
+    $myrnd, # data[1]: a random value to identify the request
+    0,
+    0,
+    0;
+
+# Send it to the root window -- since i3 uses the SubstructureRedirect
+# event mask, it will get the ClientMessage.
+$x->send_event(0, $root, EVENT_MASK_SUBSTRUCTURE_REDIRECT, $msg);
+-------------------
+
+i3 will then reply with the same ClientMessage, sent to the window specified in
++data[0]+. In the reply, +data[0]+ and +data[1]+ are exactly the same as in the
+request. You should use a random value in +data[1]+ and check that you received
+the same one when getting the reply.
+
+== Appendix B: Socket activation