find_library: Allow custom lib suffix be used as find path

Add a new `CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX` variable to allow use
of a custom suffix on `lib` directory names.  This is a more general
option than that added by commit v3.7.0-rc1~504^2 (Teach find_library
and find_package to search lib32 paths, 2016-06-10).  It allows the find
path to be more deterministic on custom setups.

See discussion in #10287 and #15994.

Help/command/find_library.rst

@@ -49,6 +49,13 @@ path to the framework ``<fullPath>/A.framework``.  When a full path to a
 framework is used as a library, CMake will use a ``-framework A``, and a
 ``-F<fullPath>`` to link the framework to the target.
 
+If the :variable:`CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX` variable is set all
+search paths will be tested as normal, with the suffix appended, and with
+all matches of ``lib/`` replaced with
+``lib${CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX}/``.  This variable overrides
+the :prop_gbl:`FIND_LIBRARY_USE_LIB32_PATHS`
+and :prop_gbl:`FIND_LIBRARY_USE_LIB64_PATHS` global properties.
+
 If the :prop_gbl:`FIND_LIBRARY_USE_LIB32_PATHS` global property is set
 all search paths will be tested as normal, with ``32/`` appended, and
 with all matches of ``lib/`` replaced with ``lib32/``.  This property is

Help/manual/cmake-variables.7.rst

@@ -130,6 +130,7 @@ Variables that Change Behavior
    /variable/CMAKE_SYSROOT
    /variable/CMAKE_FIND_APPBUNDLE
    /variable/CMAKE_FIND_FRAMEWORK
+   /variable/CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX
    /variable/CMAKE_FIND_LIBRARY_PREFIXES
    /variable/CMAKE_FIND_LIBRARY_SUFFIXES
    /variable/CMAKE_FIND_NO_INSTALL_PREFIX

Help/prop_gbl/FIND_LIBRARY_USE_LIB32_PATHS.rst

@@ -8,3 +8,5 @@ Whether the :command:`find_library` command should automatically search
 :command:`find_library` command should automatically search the ``lib32``
 variant of directories called ``lib`` in the search path when building 32-bit
 binaries.
+
+See also the :variable:`CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX` variable.

Help/prop_gbl/FIND_LIBRARY_USE_LIB64_PATHS.rst

@@ -8,3 +8,5 @@ FIND_LIBRARY_USE_LIB64_PATHS is a boolean specifying whether the
 :command:`find_library` command should automatically search the lib64
 variant of directories called lib in the search path when building
 64-bit binaries.
+
+See also the :variable:`CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX` variable.

Help/release/dev/find_library-custom-lib-suffix.rst

@@ -0,0 +1,6 @@
+find_library-custom-lib-suffix
+------------------------------
+
+* A :variable:`CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX` variable was added to
+  tell the :command:`find_library` command to search in a ``lib<suffix>``
+  directory before each ``lib`` directory that would normally be searched.

Help/variable/CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX.rst

@@ -0,0 +1,11 @@
+CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX
+------------------------------------
+
+Specify a ``<suffix>`` to tell the :command:`find_library` command to
+search in a ``lib<suffix>`` directory before each ``lib`` directory that
+would normally be searched.
+
+This overrides the behavior of related global properties:
+
+* :prop_gbl:`FIND_LIBRARY_USE_LIB32_PATHS`
+* :prop_gbl:`FIND_LIBRARY_USE_LIB64_PATHS`

Source/cmFindLibraryCommand.cxx

@@ -43,20 +43,22 @@ bool cmFindLibraryCommand::InitialPass(std::vector<std::string> const& argsIn,
     return true;
   }
 
-  if (this->Makefile->GetState()->GetGlobalPropertyAsBool(
-        "FIND_LIBRARY_USE_LIB32_PATHS")) {
-    // add special 32 bit paths if this is a 32 bit compile.
-    if (this->Makefile->PlatformIs32Bit()) {
-      this->AddArchitecturePaths("32");
-    }
+  // add custom lib<qual> paths instead of using fixed lib32 or lib64
+  if (const char* customLib = this->Makefile->GetDefinition(
+        "CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX")) {
+    this->AddArchitecturePaths(customLib);
   }
-
-  if (this->Makefile->GetState()->GetGlobalPropertyAsBool(
-        "FIND_LIBRARY_USE_LIB64_PATHS")) {
-    // add special 64 bit paths if this is a 64 bit compile.
-    if (this->Makefile->PlatformIs64Bit()) {
-      this->AddArchitecturePaths("64");
-    }
+  // add special 32 bit paths if this is a 32 bit compile.
+  else if (this->Makefile->PlatformIs32Bit() &&
+           this->Makefile->GetState()->GetGlobalPropertyAsBool(
+             "FIND_LIBRARY_USE_LIB32_PATHS")) {
+    this->AddArchitecturePaths("32");
+  }
+  // add special 64 bit paths if this is a 64 bit compile.
+  else if (this->Makefile->PlatformIs64Bit() &&
+           this->Makefile->GetState()->GetGlobalPropertyAsBool(
+             "FIND_LIBRARY_USE_LIB64_PATHS")) {
+    this->AddArchitecturePaths("64");
   }
 
   std::string library = this->FindLibrary();

Tests/CMakeOnly/find_library/CMakeLists.txt

@@ -24,7 +24,7 @@ endmacro()
 macro(test_find_library_subst expected)
   get_filename_component(dir ${expected} PATH)
   get_filename_component(name ${expected} NAME)
-  string(REGEX REPLACE "lib/?64" "lib" dir "${dir}")
+  string(REGEX REPLACE "lib/?[36X][24Y][Z]*" "lib" dir "${dir}")
   test_find_library(", searched as ${dir}" "${expected}"
     NAMES ${name}
     PATHS ${CMAKE_CURRENT_SOURCE_DIR}/${dir}
@@ -79,3 +79,16 @@ test_find_library("" A/libtestA.a
   NAMES testB testA NAMES_PER_DIR
   PATHS ${CMAKE_CURRENT_SOURCE_DIR}/A ${CMAKE_CURRENT_SOURCE_DIR}/B
   )
+
+set(CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX "XYZ")
+foreach(libXYZ
+    lib/XYZ/libtest1.a
+    lib/A/libXYZ/libtest2.a
+    lib/libtest3.a
+    libXYZ/A/lib/libtest4.a
+    libXYZ/A/libXYZ/libtest5.a
+    libXYZ/A/libtest6.a
+    libXYZ/libtest7.a
+    )
+  test_find_library_subst(${libXYZ})
+endforeach()