Help/dev: Add a CMake Testing Guide

Document how developers working on CMake itself may run the test suite
locally.

Fixes: #26001

CONTRIBUTING.rst

@@ -26,7 +26,8 @@ To contribute patches:
 #. Fork the upstream `CMake Repository`_ into a personal account.
 #. Run `Utilities/SetupForDevelopment.sh`_ for local git configuration.
 #. See `Building CMake`_ for building CMake locally.
-#. See the `CMake Source Code Guide`_ for coding guidelines.
+#. See the `CMake Source Code Guide`_ for coding guidelines
+   and the `CMake Testing Guide`_ for testing instructions.
 #. Create a topic branch named suitably for your work.
    Base all new work on the upstream ``master`` branch.
    Base work on the upstream ``release`` branch only if it fixes a
@@ -49,6 +50,7 @@ The merge request will enter the `CMake Review Process`_ for consideration.
 .. _`Utilities/SetupForDevelopment.sh`: Utilities/SetupForDevelopment.sh
 .. _`Building CMake`: README.rst#building-cmake
 .. _`CMake Source Code Guide`: Help/dev/source.rst
+.. _`CMake Testing Guide`: Help/dev/testing.rst
 .. _`commit messages`: Help/dev/review.rst#commit-messages
 .. _`CMake Review Process`: Help/dev/review.rst
 

Help/dev/README.rst

@@ -38,10 +38,12 @@ CMake developer documentation is provided by the following documents:
 
 * The `CMake Source Code Guide`_.
 * The `CMake Documentation Guide`_.
+* The `CMake Testing Guide`_.
 * The `CMake Experimental Features Guide`_.
 
 .. _`CMake Source Code Guide`: source.rst
 .. _`CMake Documentation Guide`: documentation.rst
+.. _`CMake Testing Guide`: testing.rst
 .. _`CMake Experimental Features Guide`: experimental.rst
 
 Maintainer Documentation

Help/dev/integration-testing.rst

@@ -4,7 +4,10 @@ CMake Integration Testing
 The following documents how to run integration testing builds.
 See documentation on `CMake Development`_ for more information.
 
+See the `CMake Testing Guide`_ for running the test suite locally.
+
 .. _`CMake Development`: README.rst
+.. _`CMake Testing Guide`: testing.rst
 
 CMake Dashboard Scripts
 =======================

Help/dev/source.rst

@@ -310,6 +310,7 @@ The CMake source tree is organized as follows.
 
 * ``Tests/``:
   The test suite.  See `Tests/README.rst`_.
+  To run the tests, see the `CMake Testing Guide`_.
 
 * ``Utilities/``:
   Scripts, third-party source code.
@@ -331,5 +332,6 @@ The CMake source tree is organized as follows.
     See `Utilities/Release/README.rst`_.
 
 .. _`CMake Documentation Guide`: documentation.rst
+.. _`CMake Testing Guide`: testing.rst
 .. _`Tests/README.rst`: ../../Tests/README.rst
 .. _`Utilities/Release/README.rst`: ../../Utilities/Release/README.rst

Help/dev/testing.rst

@@ -1,8 +1,111 @@
 CMake Testing Guide
 *******************
 
-See `CMake Integration Testing`_ for running integration testing builds.
+The following is a guide to the CMake test suite for developers.
 See documentation on `CMake Development`_ for more information.
 
-.. _`CMake Integration Testing`: integration-testing.rst
+See `CMake Integration Testing`_ for running integration testing builds.
+
+See `Tests/README.rst`_ for the test suite layout in the source tree.
+
 .. _`CMake Development`: README.rst
+.. _`CMake Integration Testing`: integration-testing.rst
+.. _`Tests/README.rst`: ../../Tests/README.rst
+
+Running Tests in the Build Tree
+===============================
+
+After `Building CMake`_, one may run the test suite in the build tree
+using `ctest(1)`_:
+
+* With a single-configuration CMake generator, such as ``Ninja``
+  or ``Unix Makefiles``, one may simply run ``ctest``:
+
+  .. code-block:: console
+
+    $ ctest
+
+* With a multi-configuration CMake generator, such as
+  ``Ninja Multi-Config``, ``Visual Studio``, or ``Xcode``,
+  one must tell ``ctest`` which configuration to test
+  by passing the ``-C <config>`` option:
+
+  .. code-block:: console
+
+    $ ctest -C Debug
+
+Some useful `ctest(1)`_ options include:
+
+``-N``
+  List test names without running them.
+
+``-V``
+  Show verbose output from each test.
+
+``-j <N>``
+  Run to run up to ``N`` tests concurrently.
+
+``-R <regex>``
+  Select tests for which the regular expression matches a substring
+  of their name.
+
+Cleaning Test Build Trees
+-------------------------
+
+Many CMake tests create their own test project build trees underneath
+the ``Tests/`` directory at the top of the CMake build tree.  These
+build trees are left behind after testing completes in order to
+facilitate manual investigation of results.  Many of the tests do *not*
+clean their build trees if they are run again, with the exception of
+tests using the `RunCMake`_ infrastructure.
+
+In order to clear test build trees, drive the ``test_clean`` custom target
+in the CMake build tree:
+
+.. code-block:: console
+
+  $ cmake --build . --target test_clean
+
+This removes the ``Tests/`` subdirectories created by individual tests
+so they will use a fresh directory next time they run.
+
+.. _`Building CMake`: ../../README.rst#building-cmake
+.. _`ctest(1)`: https://cmake.org/cmake/help/latest/manual/ctest.1.html
+.. _`RunCMake`: ../../Tests/RunCMake/README.rst
+
+Running Tests with a Different Generator
+========================================
+
+After `Building CMake`_ with one CMake generator, one may configure the
+test suite using a different generator in a separate build tree, without
+building CMake itself again, by defining ``CMake_TEST_EXTERNAL_CMAKE``
+to be the absolute path to the ``bin`` directory containing the ``cmake``,
+``ctest``, and ``cpack`` executables.
+
+For example, after building CMake with the ``Ninja`` generator:
+
+.. code-block:: console
+
+  $ cmake -B build-ninja -G Ninja -DCMAKE_BUILD_TYPE=Debug
+  $ cmake --build build-ninja
+
+one may configure a second build tree to drive tests with the
+``Ninja Multi-Config`` generator:
+
+.. code-block:: console
+
+  $ cmake -B build-nmc-tests -G "Ninja Multi-Config" \
+    -DCMake_TEST_EXTERNAL_CMAKE="$PWD/build-ninja/bin"
+  $ cmake --build build-nmc-tests --config Release
+
+The second build tree does not build CMake itself, but does configure
+the test suite and build test binaries.  One may then run tests normally:
+
+.. code-block:: console
+
+  $ cd build-nmc-tests
+  $ ctest -C Release
+
+Note that the configuration with which one drives tests in the second
+build tree is independent of the configuration with which CMake was
+built in the first.

README.rst

@@ -59,7 +59,11 @@ To build the documentation, install `Sphinx`_ and configure CMake with
 "man" builder.  Add ``-DSPHINX_EXECUTABLE=/path/to/sphinx-build`` if the
 tool is not found automatically.
 
+To run the test suite, run ``ctest`` in the CMake build directory after
+building.  See the `CMake Testing Guide`_ for details.
+
 .. _`Sphinx`: https://sphinx-doc.org
+.. _`CMake Testing Guide`: Help/dev/testing.rst
 
 Building CMake from Scratch
 ---------------------------

Tests/README.rst

@@ -2,8 +2,9 @@ CMake Tests Directory
 *********************
 
 This directory contains the CMake test suite.
-See also the `CMake Source Code Guide`_.
+See also the `CMake Testing Guide`_ and the `CMake Source Code Guide`_.
 
+.. _`CMake Testing Guide`: ../Help/dev/testing.rst
 .. _`CMake Source Code Guide`: ../Help/dev/source.rst
 
 Many tests exist as immediate subdirectories, but some tests

Tests/RunCMake/README.rst

@@ -6,9 +6,11 @@ precisely checking their return code and stdout/stderr content.
 The RunCMake infrastructure is useful for testing error cases and
 diagnostic output.
 
-See also `../README.rst`_ and the `CMake Source Code Guide`_.
+See also `../README.rst`_, the `CMake Testing Guide`_,
+and the `CMake Source Code Guide`_.
 
 .. _`../README.rst`: ../README.rst
+.. _`CMake Testing Guide`: ../../Help/dev/testing.rst
 .. _`CMake Source Code Guide`: ../../Help/dev/source.rst
 .. _`CMakeLists.txt`: CMakeLists.txt