instrumentation: Store CDash settings in query files

Adds new `cdashSubmit` and `cdashVerbose` options to allow enabling
instrumentation in CDash submissions using query files or the
`cmake_instrumentation` command.

Fixes: #26783, #26727

Help/manual/cmake-instrumentation.7.rst

@@ -100,7 +100,7 @@ Enabling Instrumentation for CDash Submissions
 You can enable instrumentation when using CTest in :ref:`Dashboard Client`
 mode by setting the :envvar:`CTEST_USE_INSTRUMENTATION` environment variable
 to the current UUID for the ``CMAKE_EXPERIMENTAL_INSTRUMENTATION`` feature.
-Doing so automatically enables the ``dynamicSystemInformation`` query.
+Doing so automatically enables the ``dynamicSystemInformation`` option.
 
 The following table shows how each type of instrumented command gets mapped
 to a corresponding type of CTest XML file.
@@ -125,6 +125,9 @@ By default the command line reported to CDash is truncated at the first space.
 You can instead choose to report the full command line (including arguments)
 by setting :envvar:`CTEST_USE_VERBOSE_INSTRUMENTATION` to 1.
 
+Alternatively, you can use the `v1 Query Files`_ to enable instrumentation for
+CDash using the ``cdashSubmit`` and ``cdashVerbose`` options.
+
 .. _`cmake-instrumentation API v1`:
 
 API v1
@@ -146,8 +149,9 @@ subdirectories:
 
 ``query/generated/``
   Holds query files generated by a CMake project with the
-  :command:`cmake_instrumentation` command. These files are owned by CMake and
-  are deleted and regenerated automatically during the CMake configure step.
+  :command:`cmake_instrumentation` command or the
+  :envvar:`CTEST_USE_INSTRUMENTATION` variable. These files are owned by CMake
+  and are deleted and regenerated automatically during the CMake configure step.
 
 ``data/``
   Holds instrumentation data collected on the project. CMake owns all data
@@ -193,8 +197,9 @@ key is required, but all other fields are optional.
   * ``postTest``
 
 ``options``
-  A list of strings specifying additional optional data to collect during
-  instrumentation. Elements in this list should be one of the following:
+  A list of strings used to enable certain optional behavior, including the
+  collection of certain additional data. Elements in this list should be one of
+  the following:
 
     ``staticSystemInformation``
       Enables collection of the static information about the host machine CMake
@@ -207,13 +212,26 @@ key is required, but all other fields are optional.
       generated by CMake, and includes information from immediately before and
       after the command is executed.
 
+    ``cdashSubmit``
+      Enables including instrumentation data in CDash. This does not
+      automatically enable ``dynamicSystemInformation``, but is otherwise
+      equivalent to having the :envvar:`CTEST_USE_INSTRUMENTATION` environment
+      variable enabled.
+
+    ``cdashVerbose``
+      Enables including the full untruncated commands in data submitted to
+      CDash. Equivalent to having the
+      :envvar:`CTEST_USE_VERBOSE_INSTRUMENTATION` environment variable enabled.
+
 The ``callbacks`` listed will be invoked during the specified hooks
 *at a minimum*. When there are multiple query files, the ``callbacks``,
 ``hooks`` and ``options`` between them will be merged. Therefore, if any query
 file includes any ``hooks``, every ``callback`` across all query files will be
 executed at every ``hook`` across all query files. Additionally, if any query
-file includes any optional ``options``, the optional query data will be present
-in all data files.
+file requests optional data using the ``options`` field, any related data will
+be present in all snippet files. User written ``callbacks`` should be able to
+handle the presence of this optional data, since it may be requested by an
+unrelated query.
 
 Example:
 
@@ -231,7 +249,8 @@ Example:
     ],
     "options": [
       "staticSystemInformation",
-      "dynamicSystemInformation"
+      "dynamicSystemInformation",
+      "cdashSubmit"
     ]
   }
 
@@ -240,12 +259,14 @@ invocation, an index file ``index-<timestamp>.json`` will be generated in
 ``<build>/.cmake/instrumentation/v1/data`` containing a list of data snippet
 files created since the previous indexing. The commands
 ``/usr/bin/python callback.py index-<timestamp>.json`` and
-``/usr/bin/cmake -P callback.cmake arg index-<timestamp>.json`` will be executed in
-that order. The index file will contain the ``staticSystemInformation`` data and
-each snippet file listed in the index will contain the
+``/usr/bin/cmake -P callback.cmake arg index-<timestamp>.json`` will be executed
+in that order. The index file will contain the ``staticSystemInformation`` data
+and each snippet file listed in the index will contain the
 ``dynamicSystemInformation`` data. Once both callbacks have completed, the index
 file and all snippet files listed by it will be deleted from the project build
-tree.
+tree. The instrumentation data will be present in the XML files submitted to
+CDash, but with truncated command strings because ``cdashVerbose`` was not
+enabled.
 
 .. _`cmake-instrumentation Data v1`:
 

Source/cmCTest.cxx

@@ -114,7 +114,6 @@ struct cmCTest::Private
   bool UseHTTP10 = false;
   bool PrintLabels = false;
   bool Failover = false;
-  bool UseVerboseInstrumentation = false;
   cmJSONState parseState;
 
   bool FlushTestProgressLine = false;
@@ -320,10 +319,6 @@ cmCTest::cmCTest()
     this->Impl->TestProgressOutput = !cmIsOff(envValue);
   }
   envValue.clear();
-  if (cmSystemTools::GetEnv("CTEST_USE_VERBOSE_INSTRUMENTATION", envValue)) {
-    this->Impl->UseVerboseInstrumentation = !cmIsOff(envValue);
-  }
-  envValue.clear();
 
   this->Impl->Parts[PartStart].SetName("Start");
   this->Impl->Parts[PartUpdate].SetName("Update");
@@ -3629,11 +3624,6 @@ cmInstrumentation& cmCTest::GetInstrumentation()
   return *this->Impl->Instrumentation;
 }
 
-bool cmCTest::GetUseVerboseInstrumentation() const
-{
-  return this->Impl->UseVerboseInstrumentation;
-}
-
 void cmCTest::ConvertInstrumentationSnippetsToXML(cmXMLWriter& xml,
                                                   std::string const& subdir)
 {
@@ -3662,6 +3652,8 @@ void cmCTest::ConvertInstrumentationSnippetsToXML(cmXMLWriter& xml,
 bool cmCTest::ConvertInstrumentationJSONFileToXML(std::string const& fpath,
                                                   cmXMLWriter& xml)
 {
+  bool verboseCommands = this->GetInstrumentation().HasOption(
+    cmInstrumentationQuery::Option::CDashVerbose);
   Json::Value root;
   this->Impl->parseState = cmJSONState(fpath, &root);
   if (!this->Impl->parseState.errors.empty()) {
@@ -3710,7 +3702,7 @@ bool cmCTest::ConvertInstrumentationJSONFileToXML(std::string const& fpath,
       }
       // Truncate the full command line if verbose instrumentation
       // was not requested.
-      if (key == "command" && !this->GetUseVerboseInstrumentation()) {
+      if (key == "command" && !verboseCommands) {
         std::string command_str = root[key].asString();
         std::string truncated = command_str.substr(0, command_str.find(' '));
         if (command_str != truncated) {

Source/cmCTest.h

@@ -436,7 +436,6 @@ public:
   std::vector<std::string> GetCommandLineHttpHeaders() const;
 
   cmInstrumentation& GetInstrumentation();
-  bool GetUseVerboseInstrumentation() const;
 
 private:
   int GenerateNotesFile(cmake* cm, std::string const& files);

Source/cmInstrumentation.cxx

@@ -29,6 +29,53 @@
 
 using LoadQueriesAfter = cmInstrumentation::LoadQueriesAfter;
 
+std::map<std::string, std::string> cmInstrumentation::cdashSnippetsMap = {
+  {
+    "configure",
+    "configure",
+  },
+  {
+    "generate",
+    "configure",
+  },
+  {
+    "compile",
+    "build",
+  },
+  {
+    "link",
+    "build",
+  },
+  {
+    "custom",
+    "build",
+  },
+  {
+    "build",
+    "skip",
+  },
+  {
+    "cmakeBuild",
+    "build",
+  },
+  {
+    "cmakeInstall",
+    "build",
+  },
+  {
+    "install",
+    "build",
+  },
+  {
+    "ctest",
+    "build",
+  },
+  {
+    "test",
+    "test",
+  }
+};
+
 cmInstrumentation::cmInstrumentation(std::string const& binary_dir,
                                      LoadQueriesAfter loadQueries)
 {
@@ -38,6 +85,7 @@ cmInstrumentation::cmInstrumentation(std::string const& binary_dir,
   this->binaryDir = binary_dir;
   this->timingDirv1 =
     cmStrCat(this->binaryDir, "/.cmake/instrumentation-", uuid, "/v1");
+  this->cdashDir = cmStrCat(this->timingDirv1, "/cdash");
   if (cm::optional<std::string> configDir =
         cmSystemTools::GetCMakeConfigDirectory()) {
     this->userTimingDirv1 =
@@ -60,7 +108,10 @@ void cmInstrumentation::LoadQueries()
     this->hasQuery = this->hasQuery ||
       this->ReadJSONQueries(cmStrCat(this->userTimingDirv1, "/query"));
   }
+}
 
+void cmInstrumentation::CheckCDashVariable()
+{
   std::string envVal;
   if (cmSystemTools::GetEnv("CTEST_USE_INSTRUMENTATION", envVal) &&
       !cmIsOff(envVal)) {
@@ -69,63 +120,23 @@ void cmInstrumentation::LoadQueries()
                                  cmExperimental::Feature::Instrumentation)
                                  .Uuid;
       if (envVal == uuid) {
+        std::set<cmInstrumentationQuery::Option> options_ = {
+          cmInstrumentationQuery::Option::CDashSubmit,
+          cmInstrumentationQuery::Option::DynamicSystemInformation
+        };
+        if (cmSystemTools::GetEnv("CTEST_USE_VERBOSE_INSTRUMENTATION",
+                                  envVal) &&
+            !cmIsOff(envVal)) {
+          options_.insert(cmInstrumentationQuery::Option::CDashVerbose);
+        }
+        for (auto const& option : options_) {
+          this->AddOption(option);
+        }
+        std::set<cmInstrumentationQuery::Hook> hooks_ = {
+          cmInstrumentationQuery::Hook::PrepareForCDash
+        };
         this->AddHook(cmInstrumentationQuery::Hook::PrepareForCDash);
-        this->AddOption(
-          cmInstrumentationQuery::Option::DynamicSystemInformation);
-        this->cdashDir = cmStrCat(this->timingDirv1, "/cdash");
-        cmSystemTools::MakeDirectory(this->cdashDir);
-        cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/configure"));
-        cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/build"));
-        cmSystemTools::MakeDirectory(
-          cmStrCat(this->cdashDir, "/build/commands"));
-        cmSystemTools::MakeDirectory(
-          cmStrCat(this->cdashDir, "/build/targets"));
-        cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/test"));
-        this->cdashSnippetsMap = { {
-                                     "configure",
-                                     "configure",
-                                   },
-                                   {
-                                     "generate",
-                                     "configure",
-                                   },
-                                   {
-                                     "compile",
-                                     "build",
-                                   },
-                                   {
-                                     "link",
-                                     "build",
-                                   },
-                                   {
-                                     "custom",
-                                     "build",
-                                   },
-                                   {
-                                     "build",
-                                     "skip",
-                                   },
-                                   {
-                                     "cmakeBuild",
-                                     "build",
-                                   },
-                                   {
-                                     "cmakeInstall",
-                                     "build",
-                                   },
-                                   {
-                                     "install",
-                                     "build",
-                                   },
-                                   {
-                                     "ctest",
-                                     "build",
-                                   },
-                                   {
-                                     "test",
-                                     "test",
-                                   } };
-        this->hasQuery = true;
+        this->WriteJSONQuery(options_, hooks_, {});
       }
     }
   }
@@ -192,13 +203,9 @@ void cmInstrumentation::WriteJSONQuery(
   for (auto const& callback : callbacks_) {
     root["callbacks"].append(cmInstrumentation::GetCommandStr(callback));
   }
-  cmsys::Directory d;
-  int n = 0;
-  if (d.Load(cmStrCat(this->timingDirv1, "/query/generated"))) {
-    n = (int)d.GetNumberOfFiles() - 2; // Don't count '.' or '..'
-  }
-  this->WriteInstrumentationJson(root, "query/generated",
-                                 cmStrCat("query-", n, ".json"));
+  this->WriteInstrumentationJson(
+    root, "query/generated",
+    cmStrCat("query-", this->writtenJsonQueries++, ".json"));
 }
 
 void cmInstrumentation::ClearGeneratedQueries()
@@ -306,7 +313,7 @@ int cmInstrumentation::CollectTimingData(cmInstrumentationQuery::Hook hook)
   }
 
   // Special case for CDash collation
-  if (this->HasHook(cmInstrumentationQuery::Hook::PrepareForCDash)) {
+  if (this->HasOption(cmInstrumentationQuery::Option::CDashSubmit)) {
     this->PrepareDataForCDash(directory, index_path);
   }
 
@@ -676,6 +683,13 @@ std::string const& cmInstrumentation::GetCDashDir()
 void cmInstrumentation::PrepareDataForCDash(std::string const& data_dir,
                                             std::string const& index_path)
 {
+  cmSystemTools::MakeDirectory(this->cdashDir);
+  cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/configure"));
+  cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/build"));
+  cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/build/commands"));
+  cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/build/targets"));
+  cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/test"));
+
   Json::Value root;
   std::string error_msg;
   cmJSONState parseState = cmJSONState(index_path, &root);

Source/cmInstrumentation.h

@@ -33,6 +33,7 @@ public:
   cmInstrumentation(std::string const& binary_dir,
                     LoadQueriesAfter loadQueries = LoadQueriesAfter::Yes);
   void LoadQueries();
+  void CheckCDashVariable();
   int InstrumentCommand(
     std::string command_type, std::vector<std::string> const& command,
     std::function<int()> const& callback,
@@ -91,7 +92,7 @@ private:
   std::set<cmInstrumentationQuery::Hook> hooks;
   std::vector<std::string> callbacks;
   std::vector<std::string> queryFiles;
-  std::map<std::string, std::string> cdashSnippetsMap;
+  static std::map<std::string, std::string> cdashSnippetsMap;
   Json::Value preTestStats;
   std::string errorMsg;
   bool hasQuery = false;
@@ -101,4 +102,5 @@ private:
   std::unique_ptr<cmsys::SystemInformation> systemInformation;
   cmsys::SystemInformation& GetSystemInformation();
 #endif
+  int writtenJsonQueries = 0;
 };

Source/cmInstrumentationQuery.cxx

@@ -16,7 +16,8 @@
 #include "cmStringAlgorithms.h"
 
 std::vector<std::string> const cmInstrumentationQuery::OptionString{
-  "staticSystemInformation", "dynamicSystemInformation"
+  "staticSystemInformation", "dynamicSystemInformation", "cdashSubmit",
+  "cdashVerbose"
 };
 std::vector<std::string> const cmInstrumentationQuery::HookString{
   "postGenerate",  "preBuild",        "postBuild",

Source/cmInstrumentationQuery.h

@@ -15,7 +15,9 @@ public:
   enum Option
   {
     StaticSystemInformation,
-    DynamicSystemInformation
+    DynamicSystemInformation,
+    CDashSubmit,
+    CDashVerbose
   };
   static std::vector<std::string> const OptionString;
 

Source/cmake.cxx

@@ -2644,6 +2644,7 @@ int cmake::ActualConfigure()
       cmStrCat(this->GetHomeOutputDirectory(), "/CMakeFiles"_s),
       this->FileAPI->GetConfigureLogVersions());
     this->Instrumentation->LoadQueries();
+    this->Instrumentation->CheckCDashVariable();
   }
 #endif
 

Tests/RunCMake/Instrumentation/query/cmake-command.cmake

@@ -8,6 +8,7 @@
     API_VERSION 1
     DATA_VERSION 1
     HOOKS postGenerate
+    OPTIONS cdashSubmit cdashVerbose
     CALLBACK ${CMAKE_COMMAND} -E echo callback1
   )
   # Query 2

Tests/RunCMake/Instrumentation/query/generated/query-1.json.in

@@ -7,6 +7,9 @@
   [
     "postGenerate"
   ],
-  "options" : [],
+  "options" : [
+    "cdashSubmit",
+    "cdashVerbose"
+  ],
   "version" : 1
 }

Tests/RunCMake/ctest_instrumentation/InstrumentationInCTestXML-check.cmake

@@ -1,3 +1,12 @@
+set(timingDir "${RunCMake_TEST_BINARY_DIR}/.cmake/instrumentation-a37d1069-1972-4901-b9c9-f194aaf2b6e0/v1")
+file(READ "${timingDir}/query/generated/query-0.json" jsonData)
+string(JSON options GET "${jsonData}" options)
+if (options MATCHES cdashVerbose AND NOT ${RunCMake_USE_VERBOSE_INSTRUMENTATION})
+  set(RunCMake_TEST_FAILED "cdashVerbose option not found in generated query despite environment variable")
+elseif (NOT options MATCHES cdashVerbose AND ${RunCMake_USE_VERBOSE_INSTRUMENTATION})
+  set(RunCMake_TEST_FAILED "cdashVerbose option found in generated query despite environment variable")
+endif()
+
 foreach(xml_type Configure Build Test)
   file(GLOB xml_file "${RunCMake_TEST_BINARY_DIR}/Testing/*/${xml_type}.xml")
   if(xml_file)
@@ -29,6 +38,12 @@ foreach(xml_type Configure Build Test)
       if(NOT xml_content MATCHES "<CmakeBuild")
         set(RunCMake_TEST_FAILED "<CmakeBuild> element not found in Build.xml")
       endif()
+      if(NOT RunCMake_USE_VERBOSE_INSTRUMENTATION AND NOT xml_content MATCHES "(truncated)")
+        set(RunCMake_TEST_FAILED "Commands not truncated despite cdashVerbose option")
+      endif()
+      if(verbose AND xml_content MATCHES "(truncated)")
+        set(RunCMake_TEST_FAILED "Commands truncated despite cdashVerbose option")
+      endif()
     endif()
   else()
     set(RunCMake_TEST_FAILED "${xml_type}.xml not found")
@@ -37,7 +52,7 @@ endforeach()
 
 foreach(dir_to_check "configure" "test" "build/targets" "build/commands")
   file(GLOB leftover_cdash_snippets
-    "${RunCMake_TEST_BINARY_DIR}/.cmake/instrumentation-a37d1069-1972-4901-b9c9-f194aaf2b6e0/v1/cdash/${dir_to_check}/*")
+    "${timingDir}/cdash/${dir_to_check}/*")
   if(leftover_cdash_snippets)
     set(RunCMake_TEST_FAILED "Leftover snippets found in cdash dir: ${leftover_cdash_snippets}")
   endif()

Tests/RunCMake/ctest_instrumentation/RunCMakeTest.cmake

@@ -1,6 +1,13 @@
 include(RunCTest)
 
-function(run_InstrumentationInCTestXML USE_INSTRUMENTATION)
+function(run_InstrumentationInCTestXML CASE_NAME USE_INSTRUMENTATION USE_VERBOSE_INSTRUMENTATION)
+  if(USE_VERBOSE_INSTRUMENTATION)
+    set(ENV{CTEST_USE_VERBOSE_INSTRUMENTATION} "1")
+    set(RunCMake_USE_VERBOSE_INSTRUMENTATION TRUE)
+  else()
+    set(ENV{CTEST_USE_VERBOSE_INSTRUMENTATION} "0")
+    set(RunCMake_USE_VERBOSE_INSTRUMENTATION FALSE)
+  endif()
   if(USE_INSTRUMENTATION)
     set(ENV{CTEST_USE_INSTRUMENTATION} "1")
     set(ENV{CTEST_EXPERIMENTAL_INSTRUMENTATION} "a37d1069-1972-4901-b9c9-f194aaf2b6e0")
@@ -18,5 +25,6 @@ function(run_InstrumentationInCTestXML USE_INSTRUMENTATION)
   unset(RunCMake_USE_LAUNCHERS)
   unset(RunCMake_USE_INSTRUMENTATION)
 endfunction()
-run_InstrumentationInCTestXML(ON)
-run_InstrumentationInCTestXML(OFF)
+run_InstrumentationInCTestXML(InstrumentationInCTestXML ON OFF)
+run_InstrumentationInCTestXML(VerboseInstrumentationInCTestXML ON ON)
+run_InstrumentationInCTestXML(NoInstrumentationInCTestXML OFF OFF)