4 VTK uses the [ExternalData][] CMake module to handle the
data management
for
5 its test suite.
Test data is only downloaded when a test which requires it is
6 enabled and it is cached so that every build does not need to redownload the
9 To facilitate
this workflow, there are a number of CMake functions available in
10 order to indicate that test
data is required.
16 get_filename_component(_vtkModuleTesting_dir
"${CMAKE_CURRENT_LIST_FILE}" DIRECTORY)
21 Data may be downloaded manually
using this function:
27 This will download
data inside of the input
data directory
for the modules
28 being built at that
time (see the `TEST_INPUT_DATA_DIRECTORY` argument of
31 For supported `PATHSPEC` syntax, see the
32 [associated
documentation][ExternalData pathspecs] in `ExternalData`. These
33 arguments are already wrapped in the `DATA{}` syntax and are assumed to be
34 relative paths from the input
data directory.
36 [ExternalData pathspecs]: TODO
40 foreach (arg IN LISTS ARGN)
41 if (IS_ABSOLUTE
"${arg}")
46 "DATA{${_vtk_build_TEST_INPUT_DATA_DIRECTORY}/${arg}}")
50 ExternalData_Expand_Arguments("${_vtk_build_TEST_DATA_TARGET}
" _ ${data_args})
54 ## Creating test executables
56 This function creates an executable from the list of sources passed to it. It
57 is automatically linked to the module the tests are intended for as well as any
58 declared test dependencies of the module.
61 vtk_module_test_executable(<NAME> <SOURCE>...)
64 This function is not usually used directly, but instead through the other
65 convenience functions.
67 function (vtk_module_test_executable name)
68 add_executable("${
name}
" ${ARGN})
69 get_property(test_depends GLOBAL
70 PROPERTY "_vtk_module_${_vtk_build_test}_test_depends
")
71 get_property(test_optional_depends GLOBAL
72 PROPERTY "_vtk_module_${_vtk_build_test}_test_optional_depends
")
73 set(optional_depends_flags)
74 foreach (test_optional_depend IN LISTS test_optional_depends)
75 if (TARGET "${test_optional_depend}
")
76 list(APPEND test_depends
77 "${test_optional_depend}
")
79 string(REPLACE "::
" "_
" safe_test_optional_depend "${test_optional_depend}
")
80 list(APPEND optional_depends_flags
81 "VTK_MODULE_ENABLE_${safe_test_optional_depend}=$<TARGET_EXISTS:${test_optional_depend}>
")
84 if (_vtk_build_UTILITY_TARGET)
85 target_link_libraries("${
name}
"
87 "${_vtk_build_UTILITY_TARGET}
")
90 target_link_libraries("${
name}
"
94 target_compile_definitions("${
name}
"
96 ${optional_depends_flags})
100 MODULES "${_vtk_build_test}
"
107 Test names default to using the basename of the filename which contains the
108 test. Two tests may share the same file by prefixing with a custom name for the
111 The two parsed syntaxes are:
113 - `CustomTestName,TestFile`
116 Note that `TestFile` should already have had its extension stripped (usually
117 done by `_vtk_test_parse_args`).
119 In general, the name of a test will be `<EXENAME>-<TESTNAME>`, however, by
120 setting `vtk_test_prefix`, the test name will instead be
121 `<EXENAME>-<PREFIX><TESTNAME>`.
125 This function parses the name from a testspec. The calling scope has
126 `test_name`, `test_arg`, and `test_file` variables set in it.
129 _vtk_test_parse_name(<TESTSPEC>)
132 function (_vtk_test_parse_name name ext)
133 if (name AND name MATCHES "^([^,]*),(.*)$
")
134 set(test_name "${CMAKE_MATCH_1}
")
135 set(test_file "${CMAKE_MATCH_2}
")
137 # Strip the extension from the test name.
138 string(REPLACE ".${ext}
" "" test_name "${
name}
")
139 set(test_name "${test_name}
")
140 set(test_file "${
name}
")
143 string(REPLACE ".${ext}
" "" test_arg "${test_file}
")
145 set(test_name "${test_name}
" PARENT_SCOPE)
146 set(test_file "${test_file}
" PARENT_SCOPE)
147 set(test_arg "${test_arg}
" PARENT_SCOPE)
151 ## Test function arguments
153 Each test is specified using one of the two following syntaxes
155 - `<NAME>.<SOURCE_EXT>`
156 - `<NAME>.<SOURCE_EXT>,<OPTIONS>`
158 Where `NAME` is a valid test name. If present, the specified `OPTIONS` are only
159 for the associated test. The expected extension is specified by the associated
164 Given a list of valid "options
", this function will parse out a the following
167 - `args`: Unrecognized arguments. These should be interpreted as arguments
168 that should be passed on the command line to all tests in this parse group.
169 - `options`: Options specified globally (for all tests in this group).
170 - `names`: A list containing all named tests. These should be parsed by
171 `_vtk_test_parse_name`.
172 - `_<NAME>_options`: Options specific to a certain test.
175 _vtk_test_parse_args(<OPTIONS> <SOURCE_EXT> <ARG>...)
178 In order to be recognized as a source file, the `SOURCE_EXT` must be used.
179 Without it, all non-option arguments are placed into `args`. Each test is
180 parsed out matching these:
182 function (_vtk_test_parse_args options source_ext)
187 foreach (arg IN LISTS ARGN)
189 foreach (option IN LISTS options)
190 if (arg STREQUAL option)
191 list(APPEND global_options "${option}
")
198 elseif (source_ext AND arg MATCHES "^([^.]*\\.${source_ext}),?(.*)$
")
199 set(name "${CMAKE_MATCH_1}
")
200 string(REPLACE ",
" ";
" "_${
name}_options
" "${CMAKE_MATCH_2}
")
201 list(APPEND names "${
name}
")
203 list(APPEND args "${arg}
")
207 foreach (name IN LISTS names)
208 set("_${
name}_options
" "${_${
name}_options}
"
211 set(options "${global_options}
"
220 For handling global option settings, this function sets variables in the
221 calling scoped named `<PREFIX><OPTION>` to either `0` or `1` if the option is
222 present in the remaining argument list.
225 _vtk_test_set_options(<OPTIONS> <PREFIX> <ARG>...)
228 Additionally, a non-`0` default for a given option may be specified by a
229 variable with the same name as the option and specifying a prefix for the
232 function (_vtk_test_set_options options prefix)
233 foreach (option IN LISTS options)
236 set(default "${${option}}
")
238 set("${prefix}${option}
" "${
default}
"
241 foreach (option IN LISTS ARGN)
242 set("${prefix}${option}
" 1
247 # If set, use the maximum number of processors for tests. Otherwise, just use 1
248 # processor by default.
249 set(VTK_MPI_NUMPROCS "2
" CACHE STRING
250 "Number of processors available to run parallel tests.
")
251 # Hide the variable if we don't have `MPIEXEC_EXECUTABLE` anyways.
252 if (MPIEXEC_EXECUTABLE)
253 set(_vtk_mpi_max_numprocs_type STRING)
255 set(_vtk_mpi_max_numprocs_type INTERNAL)
257 set_property(CACHE VTK_MPI_NUMPROCS
259 TYPE "${_vtk_mpi_max_numprocs_type}
")
264 This function declares C++ tests. Source files are required to use the `cxx`
268 vtk_add_test_cxx(<EXENAME> <VARNAME> <ARG>...)
271 Each argument should be either an option, a test specification, or it is passed
272 as flags to all tests declared in the group. The list of tests is set in the
273 `<VARNAME>` variable in the calling scope.
277 - `NO_DATA`: The test does not need to know the test input data directory. If
278 it does, it is passed on the command line via the `-D` flag.
279 - `NO_VALID`: The test does not have a valid baseline image. If it does, the
280 baseline is assumed to be in `../Data/Baseline/<NAME>.png` relative to the
281 current source directory. If alternate baseline images are required,
282 `<NAME>` may be suffixed by `_1`, `_2`, etc. The valid image is passed via
284 - `NO_OUTPUT`: The test does not need to write out any data to the
285 filesystem. If it does, a directory which may be written to is passed via
288 Additional flags may be passed to tests using the `${_vtk_build_test}_ARGS`
289 variable or the `<NAME>_ARGS` variable.
291 function (vtk_add_test_cxx exename _tests)
296 _vtk_test_parse_args("${cxx_options}
" "cxx
" ${ARGN})
297 _vtk_test_set_options("${cxx_options}
" "" ${options})
304 "instance(s)? still around
"
307 "DartMeasurement
name=.ImageNotFound
")
310 # Insufficient graphics resources.
311 "Attempt to use a texture buffer exceeding your hardware
's limits")
313 foreach (name IN LISTS names)
314 _vtk_test_set_options("${cxx_options}" "local_" ${_${name}_options})
315 _vtk_test_parse_name("${name}" "cxx")
318 if (NOT local_NO_DATA)
319 set(_D -D "${_vtk_build_TEST_OUTPUT_DATA_DIRECTORY}")
323 if (NOT local_NO_OUTPUT)
324 set(_T -T "${_vtk_build_TEST_OUTPUT_DIRECTORY}")
328 if (NOT local_NO_VALID)
329 set(_V -V "DATA{${CMAKE_CURRENT_SOURCE_DIR}/../Data/Baseline/${test_name}.png,:}")
333 VTK_SERIAL_TESTS_USE_MPIEXEC)
334 set(_vtk_test_cxx_pre_args
335 "${MPIEXEC_EXECUTABLE}"
336 "${MPIEXEC_NUMPROC_FLAG}" "1"
340 ExternalData_add_test("${_vtk_build_TEST_DATA_TARGET}"
341 NAME "${_vtk_build_test}Cxx-${vtk_test_prefix}${test_name}"
342 COMMAND "${_vtk_test_cxx_pre_args}" "$<TARGET_FILE:${exename}>"
345 ${${_vtk_build_test}_ARGS}
348 set_tests_properties("${_vtk_build_test}Cxx-${vtk_test_prefix}${test_name}"
350 LABELS "${_vtk_build_test_labels}"
351 FAIL_REGULAR_EXPRESSION "${_vtk_fail_regex}"
352 SKIP_REGULAR_EXPRESSION "${_vtk_skip_regex}"
353 # This must match VTK_SKIP_RETURN_CODE in vtkTesting.h
357 if (_vtk_testing_ld_preload)
358 set_property(TEST "${_vtk_build_test}Cxx-${vtk_test_prefix}${test_name}" APPEND
360 ENVIRONMENT "LD_PRELOAD=${_vtk_testing_ld_preload}")
363 list(APPEND ${_tests} "${test_file}")
366 set("${_tests}" ${${_tests}} PARENT_SCOPE)
372 This function declares C++ tests which should be run under an MPI environment.
373 Source files are required to use the `cxx` extension.
376 vtk_add_test_mpi(<EXENAME> <VARNAME> <ARG>...)
379 Each argument should be either an option, a test specification, or it is passed
380 as flags to all tests declared in the group. The list of tests is set in the
381 `<VARNAME>` variable in the calling scope.
386 - `NO_VALID`: The test does not have a valid baseline image. If it does, the
387 baseline is assumed to be in `../Data/Baseline/<NAME>.png` relative to the
388 current source directory. If alternate baseline images are required,
389 `<NAME>` may be suffixed by `_1`, `_2`, etc. The valid image is passed via
392 Each test is run using the number of processors specified by the following
393 variables (using the first one which is set):
396 - `<EXENAME>_NUMPROCS`
397 - `VTK_MPI_NUMPROCS` (defaults to `2`)
399 Additional flags may be passed to tests using the `${_vtk_build_test}_ARGS`
400 variable or the `<NAME>_ARGS` variable.
402 function (vtk_add_test_mpi exename _tests)
407 _vtk_test_parse_args("${mpi_options}" "cxx" ${ARGN})
408 _vtk_test_set_options("${mpi_options}" "" ${options})
410 set(_vtk_fail_regex "(\n|^)ERROR: " "ERR\\|" "instance(s)? still around")
413 # Insufficient graphics resources.
414 "Attempt to use a texture buffer exceeding your hardware's limits
")
416 set(default_numprocs ${VTK_MPI_NUMPROCS})
417 if (${exename}_NUMPROCS)
418 set(default_numprocs ${${exename}_NUMPROCS})
421 foreach (name IN LISTS names)
422 _vtk_test_set_options("${mpi_options}
" "local_
" ${_${name}_options})
423 _vtk_test_parse_name(${name} "cxx
")
428 if (local_TESTING_DATA)
429 set(_D -D "${_vtk_build_TEST_OUTPUT_DATA_DIRECTORY}
")
430 set(_T -T "${_vtk_build_TEST_OUTPUT_DIRECTORY}
")
432 if (NOT local_NO_VALID)
433 set(_V -V "DATA{${CMAKE_CURRENT_SOURCE_DIR}/../Data/Baseline/${test_name}.png,:}
")
437 set(numprocs ${default_numprocs})
438 if (${test_name}_NUMPROCS)
439 set(numprocs "${${test_name}_NUMPROCS}
")
442 ExternalData_add_test("${_vtk_build_TEST_DATA_TARGET}
"
443 NAME "${_vtk_build_test}Cxx-MPI-${vtk_test_prefix}${test_name}
"
444 COMMAND "${MPIEXEC_EXECUTABLE}
"
445 "${MPIEXEC_NUMPROC_FLAG}
" "${numprocs}
"
447 "$<TARGET_FILE:${exename}>
"
451 ${${_vtk_build_test}_ARGS}
453 ${MPIEXEC_POSTFLAGS})
454 set_tests_properties("${_vtk_build_test}Cxx-MPI-${vtk_test_prefix}${test_name}
"
456 LABELS "${_vtk_build_test_labels}
"
457 PROCESSORS "${numprocs}
"
458 FAIL_REGULAR_EXPRESSION "${_vtk_fail_regex}
"
459 SKIP_REGULAR_EXPRESSION "${_vtk_skip_regex}
"
460 # This must match VTK_SKIP_RETURN_CODE in vtkTesting.h"
464 if (_vtk_testing_ld_preload)
465 set_property(TEST
"${_vtk_build_test}Cxx-MPI-${vtk_test_prefix}${test_name}" APPEND
467 ENVIRONMENT
"LD_PRELOAD=${_vtk_testing_ld_preload}")
470 set_property(TEST "${_vtk_build_test}Cxx-MPI-${vtk_test_prefix}${test_name}
" APPEND
472 REQUIRED_FILES "$<TARGET_FILE:${exename}>
")
473 list(APPEND ${_tests} "${test_file}
")
476 set(${_tests} ${${_tests}} PARENT_SCOPE)
480 ### C++ test executable
483 vtk_test_cxx_executable(<EXENAME> <VARNAME> [RENDERING_FACTORY] [<SRC>...])
486 Creates an executable named `EXENAME` which contains the tests listed in the
487 variable named in the `VARNAME` argument. The `EXENAME` must match the
488 `EXENAME` passed to the test declarations when building the list of tests.
490 If `RENDERING_FACTORY` is provided, VTK's rendering factories are initialized
493 By default, VTK's rendering tests enable FP exceptions to find floating point
494 errors in debug builds. If `DISABLE_FLOATING_POINT_EXCEPTIONS` is provided,
495 FP exceptions are not enabled for the test. This is useful when testing against
496 external libraries to ignore exceptions in third-party code.
498 Any additional arguments are added as additional sources for the executable.
500 function (vtk_test_cxx_executable exename _tests)
503 DISABLE_FLOATING_POINT_EXCEPTIONS
505 _vtk_test_parse_args("${exe_options}
" "" ${ARGN})
506 _vtk_test_set_options("${exe_options}
" "" ${options})
509 # No tests -> no need for an executable.
513 if (RENDERING_FACTORY)
514 include("${_vtkModuleTesting_dir}/vtkTestingRenderingDriver.cmake
")
515 set(test_driver vtkTestingObjectFactory.h)
517 include("${_vtkModuleTesting_dir}/vtkTestingDriver.cmake
")
518 set(test_driver vtkTestDriver.h)
521 set(extra_sources ${args})
523 create_test_sourcelist(test_sources "${exename}.cxx
" ${${_tests}}
524 EXTRA_INCLUDE "${test_driver}
")
527 vtk_module_test_executable("${exename}
" ${test_sources} ${extra_sources})
529 message(FATAL_ERROR "_vtk_build_test is not set!
")
534 MPI executables used to have their own test executable function. This is no
535 longer necessary and is deprecated. Instead, `vtk_test_cxx_executable` should
538 function (vtk_test_mpi_executable exename _tests)
542 vtk_test_cxx_executable("${exename}
" "${_tests}
" ${ARGN})
548 This function declares Python tests. Test files are required to use the `py`
552 vtk_add_test_python(<EXENAME> <VARNAME> <ARG>...)
557 If the `_vtk_testing_python_exe` variable is not set, the `vtkpython` binary is
558 used by default. Additional arguments may be passed in this variable as well.
570 Each argument should be either an option, a test specification, or it is passed
571 as flags to all tests declared in the group. The list of tests is set in the
572 `<VARNAME>` variable in the calling scope.
576 - `NO_DATA`: The test does not need to know the test input data directory. If
577 it does, it is passed on the command line via the `-D` flag.
578 - `NO_OUTPUT`: The test does not need to write out any data to the
579 filesystem. If it does, a directory which may be written to is passed via
581 - `NO_VALID`: The test does not have a valid baseline image. If it does, the
582 baseline is assumed to be in `../Data/Baseline/<NAME>.png` relative to the
583 current source directory. If alternate baseline images are required,
584 `<NAME>` may be suffixed by `_1`, `_2`, etc. The valid image is passed via
586 - `NO_RT`: If `NO_RT` is specified, `-B` is passed instead of `-V`, only
587 providing a baseline dir, assuming `NO_VALID` is not specified.
588 - `DIRECT_DATA` : If `DIRECT_DATA` is specified, the baseline path will be provided
589 as is, without the use of ExternalData_add_test.
590 - `JUST_VALID`: Only applies when both `NO_VALID` and `NO_RT` are not
591 present. If it is not specified, `-A` is passed with path to the directory
592 of the `vtkTclTest2Py` Python package and the test is run via the
593 `rtImageTest.py` script. Note that this currently only works when building
594 against a VTK build tree; the VTK install tree does not include this script
595 or its associated Python package.
597 Additional flags may be passed to tests using the `${_vtk_build_test}_ARGS`
598 variable or the `<NAME>_ARGS` variable.
600 Note that the `vtkTclTest2Py` support will eventually be removed. It is a
601 legacy of the conversion of many tests from Tcl to Python.
603 function (vtk_add_test_python)
604 if (NOT _vtk_testing_python_exe)
605 set(_vtk_testing_python_exe "$<TARGET_FILE:VTK::vtkpython>
")
615 _vtk_test_parse_args("${python_options}
" "py
" ${ARGN})
616 _vtk_test_set_options("${python_options}
" "" ${options})
618 set(_vtk_fail_regex "(\n|^)ERROR:
" "ERR\\|
" "instance(s)? still around
")
621 # Insufficient graphics resources.
622 "Attempt to use a texture buffer exceeding your hardware
's limits")
624 foreach (name IN LISTS names)
625 _vtk_test_set_options("${python_options}" "local_" ${_${name}_options})
626 _vtk_test_parse_name(${name} "py")
629 if (NOT local_NO_DATA)
630 set(_D -D "${_vtk_build_TEST_OUTPUT_DATA_DIRECTORY}")
637 if (NOT local_NO_VALID)
639 if (local_DIRECT_DATA)
640 set(_B -B "${CMAKE_CURRENT_SOURCE_DIR}/Data/Baseline/")
642 set(_B -B "DATA{${CMAKE_CURRENT_SOURCE_DIR}/../Data/Baseline/,REGEX:${test_name}(-.*)?(_[0-9]+)?.png}")
645 if (local_DIRECT_DATA)
646 set(_V -V "${CMAKE_CURRENT_SOURCE_DIR}/Data/Baseline/${test_name}.png")
648 set(_V -V "DATA{${CMAKE_CURRENT_SOURCE_DIR}/../Data/Baseline/${test_name}.png,:}")
650 if (NOT local_JUST_VALID)
651 # TODO: This should be fixed to also work from an installed VTK.
652 set(rtImageTest "${VTK_SOURCE_DIR}/Utilities/vtkTclTest2Py/rtImageTest.py")
653 set(_A -A "${VTK_SOURCE_DIR}/Utilities/vtkTclTest2Py")
659 if (NOT local_NO_OUTPUT)
660 set(_T -T "${_vtk_build_TEST_OUTPUT_DIRECTORY}")
663 if (NOT _vtk_build_TEST_FILE_DIRECTORY)
664 set(_vtk_build_TEST_FILE_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
668 VTK_SERIAL_TESTS_USE_MPIEXEC AND
669 NOT DEFINED _vtk_test_python_pre_args)
670 set(_vtk_test_python_pre_args
671 "${MPIEXEC_EXECUTABLE}"
672 "${MPIEXEC_NUMPROC_FLAG}" "1"
675 set(testArgs NAME "${_vtk_build_test}Python${_vtk_test_python_suffix}-${vtk_test_prefix}${test_name}"
676 COMMAND ${_vtk_test_python_pre_args}
677 "${_vtk_testing_python_exe}" ${_vtk_test_python_args} --enable-bt
679 "${_vtk_build_TEST_FILE_DIRECTORY}/${test_file}"
681 ${${_vtk_build_test}_ARGS}
683 ${_D} ${_B} ${_T} ${_V} ${_A})
685 if (local_DIRECT_DATA)
686 add_test(${testArgs})
688 ExternalData_add_test("${_vtk_build_TEST_DATA_TARGET}" ${testArgs})
691 if (_vtk_testing_ld_preload)
692 set_property(TEST "${_vtk_build_test}Python${_vtk_test_python_suffix}-${vtk_test_prefix}${test_name}"
695 ENVIRONMENT "LD_PRELOAD=${_vtk_testing_ld_preload}")
698 set_tests_properties("${_vtk_build_test}Python${_vtk_test_python_suffix}-${vtk_test_prefix}${test_name}"
700 LABELS "${_vtk_build_test_labels}"
701 FAIL_REGULAR_EXPRESSION "${_vtk_fail_regex}"
702 SKIP_REGULAR_EXPRESSION "${_vtk_skip_regex}"
703 # This must match the skip() function in vtk/test/Testing.py"
708 set_tests_properties("${_vtk_build_test}Python${_vtk_test_python_suffix}-${vtk_test_prefix}${test_name}"
710 PROCESSORS "${numprocs}")
718 A small wrapper around `vtk_add_test_python` which adds support for running
719 MPI-aware tests written in Python.
721 The `$<module library name>_NUMPROCS` variable may be used to use a non-default
722 number of processors for a test.
724 This forces running with the `pvtkpython` executable.
726 function (vtk_add_test_python_mpi)
727 set(_vtk_test_python_suffix "-MPI")
729 set(numprocs "${VTK_MPI_NUMPROCS}")
730 _vtk_module_get_module_property("${_vtk_build_test}"
731 PROPERTY "library_name"
732 VARIABLE _vtk_test_python_library_name)
733 if (${_vtk_test_python_library_name}_NUMPROCS)
734 set(numprocs "${${_vtk_test_python_library_name}_NUMPROCS}")
737 set(_vtk_test_python_pre_args
738 "${MPIEXEC_EXECUTABLE}"
739 "${MPIEXEC_NUMPROC_FLAG}" "${numprocs}"
742 if (NOT _vtk_testing_python_exe)
743 set(_vtk_testing_python_exe "$<TARGET_FILE:VTK::pvtkpython>")
745 vtk_add_test_python(${ARGN})
function vtk_module_build()
Build modules and kits.
int Test(int argc, char *argv[], const char *dfile, const InitializationCallback &initCallback)
function vtk_test_cxx_executable(exename, _tests)
.md
function vtk_test_mpi_executable(exename, _tests)
.md INTERNAL MPI executables used to have their own test executable function.
function vtk_module_test_data()
.md