Merge topic 'doc-test-RunCMake'

0dbf2c24bf Tests/RunCMake: Document how to write multi-step test cases
61a08b0856 Tests/RunCMake: Document commands/variables available to RunCMakeTest scripts
aebf6b8b29 Tests/RunCMake: Clarify documentation of platform-specific output matching
c2f8a6729f Tests/RunCMake: Rename "sub-test" terminology to "case"
0add7a802f Tests/RunCMake: Organize README as multiple sections
aff4dc47c4 Tests/RunCMake: Cross-reference README from main Tests/README

Acked-by: Kitware Robot <[email protected]>
Merge-request: !9548

Tests/README.rst

@@ -29,3 +29,7 @@ are organized as follows.
   Tests that run CMake and/or other tools while precisely checking
   their return code and stdout/stderr content.  Useful for testing
   error cases and diagnostic output.
+
+  See `RunCMake/README.rst`_.
+
+.. _`RunCMake/README.rst`: RunCMake/README.rst

Tests/RunCMake/README.rst

@@ -1,33 +1,49 @@
-This directory contains tests that run CMake to configure a project
-but do not actually build anything.  To add a test:
+CMake Tests/RunCMake Directory
+******************************
+
+This directory contains tests that run CMake and/or other tools while
+precisely checking their return code and stdout/stderr content.
+The RunCMake infrastructure is useful for testing error cases and
+diagnostic output.
+
+See also `../README.rst`_ and the `CMake Source Code Guide`_.
+
+.. _`../README.rst`: ../README.rst
+.. _`CMake Source Code Guide`: ../../Help/dev/source.rst
+.. _`CMakeLists.txt`: CMakeLists.txt
+
+Adding a Test
+=============
+
+To add a test:
 
 1. Add a subdirectory named for the test, say ``<Test>/``.
 
-2. In ``./CMakeLists.txt`` call ``add_RunCMake_test`` and pass the
+2. In `CMakeLists.txt`_ call ``add_RunCMake_test`` and pass the
    test directory name ``<Test>``.
 
 3. Create script ``<Test>/RunCMakeTest.cmake`` in the directory containing::
 
     include(RunCMake)
-    run_cmake(SubTest1)
+    run_cmake(Case1)
     ...
-    run_cmake(SubTestN)
+    run_cmake(CaseN)
 
-   where ``SubTest1`` through ``SubTestN`` are sub-test names each
-   corresponding to an independent CMake run and project configuration.
+   where ``Case1`` through ``CaseN`` are case names each corresponding to
+   an independent CMake run and project configuration.
 
    One may also add calls of the form::
 
-    run_cmake_command(SubTestI ${CMAKE_COMMAND} ...)
+    run_cmake_command(CaseI ${CMAKE_COMMAND} ...)
 
    to fully customize the test case command-line.
 
    Alternatively, if the test is to cover running ``ctest -S`` then use::
 
     include(RunCTest)
-    run_ctest(SubTest1)
+    run_ctest(Case1)
     ...
-    run_ctest(SubTestN)
+    run_ctest(CaseN)
 
    and create ``test.cmake.in``, ``CTestConfig.cmake.in``, and
    ``CMakeLists.txt.in`` files to be configured for each case.
@@ -50,40 +66,195 @@ but do not actually build anything.  To add a test:
 
    where ``${RunCMake_TEST}`` is literal.  A value for ``RunCMake_TEST``
    will be passed to CMake by the ``run_cmake`` macro when running each
-   sub-test.
+   case.
 
-5. Create a ``<Test>/<SubTest>.cmake`` file for each sub-test named
+5. Create a ``<Test>/<case>.cmake`` file for each case named
    above containing the actual test code.  Optionally create files
    containing expected test results:
 
-   ``<SubTest>-result.txt``
+   ``<case>-result.txt``
     Regex matching expected process result, if not ``0``
-   ``<SubTest>-stdout.txt``
+   ``<case>-stdout.txt``
     Regex matching expected stdout content
-   ``<SubTest>-stderr.txt``
+   ``<case>-stderr.txt``
     Regex matching expected stderr content, if not ``^$``
-   ``<SubTest>-check.cmake``
+   ``<case>-check.cmake``
     Custom result check.
 
-  Note that when a specific platform expects differing stdout or stderr that
-  can be done by adding a platform specific output file. These follow the
-  naming convention of:
-   ``<SubTest>-stdout-<platform_lower_case>.txt``
-   ``<SubTest>-stderr-<platform_lower_case>.txt``
+   To specify platform-specific matches, create files of the form
+   ``<case>-{stdout,stderr}-<platform_lower_case>.txt``.
 
    Note that trailing newlines will be stripped from actual and expected
    test output before matching against the stdout and stderr expressions.
-   The code in ``<SubTest>-check.cmake`` may use variables
-
-   ``RunCMake_TEST_SOURCE_DIR``
-    Top of test source tree
-   ``RunCMake_TEST_BINARY_DIR``
-    Top of test binary tree
-
-   and an failure must store a message in ``RunCMake_TEST_FAILED``.
+   The code in ``<case>-check.cmake`` may use the `RunCMake Variables`_.
+   On failure the script must store a message in ``RunCMake_TEST_FAILED``.
    The check script may optionally set ``RunCMake_TEST_FAILURE_MESSAGE``
    with additional text to be included in the message if the test fails.
 
+RunCMake Commands
+=================
+
+A ``RunCMakeTest.cmake`` script, after ``include(RunCMake)``, may use
+the following commands.
+
+``run_cmake(<case>)``
+  Run CMake or another command and check expected results described by
+  ``<case>-{result,stdout,stderr}.txt`` and ``<case>-check.cmake``.
+  The command is executed by a call of the form::
+
+    execute_process(
+      COMMAND ${RunCMake_TEST_COMMAND} ${RunCMake_TEST_OPTIONS}
+      WORKING_DIRECTORY "${RunCMake_TEST_COMMAND_WORKING_DIRECTORY}"
+      [TIMEOUT "${RunCMake_TEST_TIMEOUT}"]
+      ...
+      )
+
+  Behavior may be customized by setting `RunCMake Variables`_ before
+  the call.
+
+``run_cmake_command(<case> <command> <args>...)``
+  Sets ``RunCMake_TEST_COMMAND`` to ``<command>;<args>...``
+  and calls ``run_cmake(<case>)``.
+
+  This is useful to run an arbitrary command.
+
+``run_cmake_script(<case> <args>...)``
+  Sets ``RunCMake_TEST_COMMAND`` to
+  ``${CMAKE_COMMAND};<args>...;-P;${RunCMake_SOURCE_DIR}/<case>.cmake``
+  and calls ``run_cmake(<case>)``.
+
+  This is useful to run CMake in script mode without configuring a project.
+
+``run_cmake_with_options(<case> <opts>...)``
+  Sets ``RunCMake_TEST_OPTIONS`` to ``<opts>...``
+  and calls ``run_cmake(<case>)``.
+
+``run_cmake_with_raw_args(<case> "<args>")``
+  Calls ``run_cmake(<case>)`` with the underlying ``execute_process()``
+  call extended with the content of ``<args>`` treated as literal source
+  code of CMake language command arguments::
+
+    execute_process(
+      COMMAND ${RunCMake_TEST_COMMAND} ${RunCMake_TEST_OPTIONS} <args>
+      ...
+      )
+
+  This is useful to pass arguments to the test command that cannot be
+  encoded in CMake language ``;``-separated lists.
+
+RunCMake Variables
+==================
+
+The behavior of `RunCMake Commands`_ such as ``run_cmake()`` may be
+customized by setting the following variables before a call.
+
+``RunCMake_GENERATOR``
+  CMake generator to use when configuring projects.
+  This provided to ``RunCMakeTest.cmake`` scripts automatically
+  when they are executed, based on the CMake generator used to
+  configure the test suite.
+
+  For some generators, additional variables are also provided:
+
+  ``RunCMake_GENERATOR_PLATFORM``
+    Specifies the ``CMAKE_GENERATOR_PLATFORM``.
+
+  ``RunCMake_GENERATOR_TOOLSET``
+    Specifies the ``CMAKE_GENERATOR_TOOLSET``.
+
+  ``RunCMake_GENERATOR_INSTANCE``
+    Specifies the ``CMAKE_GENERATOR_INSTANCE``.
+
+``RunCMake_GENERATOR_IS_MULTI_CONFIG``
+  Boolean value indicating whether ``${RunCMake_GENERATOR}`` is a
+  multi-config generator.
+  This provided to ``RunCMakeTest.cmake`` scripts automatically
+  when they are executed, based on the CMake generator used to
+  configure the test suite.
+
+``RunCMake_SOURCE_DIR``
+  Absolute path to the ``Tests/RunCMake/<Test>`` directory in
+  the CMake source tree.  This provided to ``RunCMakeTest.cmake``
+  scripts automatically when they are executed.
+
+``RunCMake_BINARY_DIR``
+  Absolute path to the ``Tests/RunCMake/<Test>`` directory in
+  the CMake binary tree.  This provided to ``RunCMakeTest.cmake``
+  scripts automatically when they are executed.
+
+``RunCMake_TEST_SOURCE_DIR``
+  Absolute path to the individual test case's source tree.
+  If not set, defaults to ``${RunCMake_SOURCE_DIR}``.
+
+``RunCMake_TEST_BINARY_DIR``
+  Absolute path to the individual test case's binary tree.
+  If not set, defaults to ``${RunCMake_BINARY_DIR}/<case>-build``.
+
+``RunCMake_TEST_NO_CLEAN``
+  Boolean value indicating whether ``run_cmake(<case>)`` should remove the
+  ``${RunCMake_TEST_BINARY_DIR}`` directory before running the test case.
+  If not set, or if set to a false value, the directory is removed.
+
+  This is useful to run `Multi-Step Test Cases`_.
+
+``RunCMake_TEST_COMMAND``
+  The command for ``run_cmake(<case>)`` to execute.
+  If not set, defaults to running CMake to generate a project::
+
+    ${CMAKE_COMMAND} ${RunCMake_TEST_SOURCE_DIR} \
+      -G ${RunCMake_GENERATOR} ... -DRunCMake_TEST=<case>
+
+``RunCMake_TEST_COMMAND_WORKING_DIRECTORY``
+  The working directory in which ``run_cmake(<case>)`` to execute its command.
+  If not set, defaults to ``${RunCMake_TEST_BINARY_DIR}``.
+
+``RunCMake_TEST_OPTIONS``
+  Additional command-line options for ``run_cmake(<case>)`` to pass to
+  CMake when configuring a project with a default ``RunCMake_TEST_COMMAND``.
+  If not set, defaults to empty.
+  If ``RunCMake_TEST_COMMAND`` is set, ``RunCMake_TEST_OPTIONS`` is forced
+  to empty.
+
+``RunCMake_TEST_OUTPUT_MERGE``
+  Boolean value indicating whether ``run_cmake(<case>)`` should redirect
+  the test process's ``stderr`` into its ``stdout``.
+
+``RunCMake_TEST_TIMEOUT``
+  Specify a timeout, in seconds, for ``run_cmake(<case>)`` to pass to its
+  underlying ``execute_process()`` call using the ``TIMEOUT`` option.
+
+Multi-Step Test Cases
+=====================
+
+Normally each ``run_cmake(<case>)`` call corresponds to one standalone
+test case with its own build tree.  However, some test cases may require
+multiple steps to be performed in a single build tree.  This can be
+achieved as follows::
+
+  block()
+    set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/example-build)
+    run_cmake(example)
+    set(RunCMake_TEST_NO_CLEAN 1)
+    set(RunCMake_TEST_OUTPUT_MERGE 1)
+    run_cmake_command(example-build ${CMAKE_COMMAND} --build . --config Debug)
+  endblock()
+
+In this example, ``block() ... endblock()`` is used to isolate the
+variable settings from later cases.  A single build tree is used for
+all cases inside the block.  The first step cleans the build tree and
+runs CMake to configure the case's project.  The second step runs
+``cmake --build`` to drive the generated build system and merges the
+build tool's ``stderr`` into its ``stdout``.  Note that each call uses
+a unique case name so that expected results can be expressed individually.
+
+Running a Test
+==============
+
+Each call to ``add_RunCMake_test(Example)`` in `CMakeLists.txt`_ creates
+a test named ``RunCMake.Example`` that may be run with ``ctest``::
+
+  $ ctest -R "^RunCMake\.Example$"
+
 To speed up local testing, you can choose to run only a subset of
 ``run_cmake()`` tests in a ``RunCMakeTest.cmake`` script by using the
 ``RunCMake_TEST_FILTER`` environment variable. If this variable is set,
@@ -92,7 +263,7 @@ match the regular expression are not run. For example::
 
   $ RunCMake_TEST_FILTER="^example" ctest -R '^RunCMake\.Example$'
 
-This will only run subtests in ``RunCMake.Example`` that start with
+This will only run cases in ``RunCMake.Example`` that start with
 ``example``.
 
 To speed up the process of creating a new ``RunCMake`` test, you can run a