Skip to main content
Version: 4.x

Testing strategy

The H3 developers try to ensure the robustness and reliability of the H3 library. Tools used to do this include defensive code, unit tests with coverage reporting, and fuzzing.

The H3 library may despite these efforts behave unexpectedly; in these cases the developers welcome feedback and contributions.

Unit testing

Github Actions are used to run the H3 test suite for every commit. A variety of configurations, described below, are tested.

Coverage information is collected in Coveralls. Because of the self-contained nature of the H3 library, we seek to have as close to 100% code coverage as possible.

Operating systemCompilerBuild typeProcessor architectureSpecial notes
Linux (Ubuntu)ClangDebug, Releasex64clang-format-14 is used to ensure all code is consistently formatted
LinuxClangDebugx64An additional copy of the job runs with Valgrind
LinuxClangDebugx64An additional copy of the job runs with coverage reporting, and excerising the H3_PREFIX mechanism.
LinuxgccDebug, Releasex64
Mac OSApple ClangDebug, Releasex64
WindowsMSVCDebug, Releasex64Static library
WindowsMSVCDebug, Releasex86Static library
WindowsMSVCDebug, Releasex64Dynamic library; testing is not run in this configuration
WindowsMSVCDebug, Releasex86Dynamic library; testing is not run in this configuration

Defensive code

H3 uses preprocessor macros borrowed from SQLite's testing methodology to include defensive code in the library. Defensive code is code that handles error conditions for which there are no known test cases to demonstrate it. The lack of known test cases means that without the macros, the defensive cases could inappropriately reduce coverage metrics, disincentivizing including them. The macros behave differently, depending on the build configuration:

  • Under release builds of the library (CMAKE_BUILD_TYPE=Release), the defensive code is included without modification. These branches are intended to be very simple (usually only returning an error code and possibly freeing some resources) and to be visually inspectable.

  • Under debug builds of the library (CMAKE_BUILD_TYPE=Debug), the defensive code is included and assert calls are included if the defensive code is invoked. Any unit test or fuzzer which can demonstrate the defensive code is actually reached will trigger a test failure and the developers can be alerted to cover the defensive code in unit tests.

  • Under coverage builds of the library (CMAKE_BUILD_TYPE=Debug ENABLE_COVERAGE=ON), the defensive code is not included by replacing its condition with a constant value. The compiler removes the defensive code and it is not counted in coverage metrics. This is intended only for determining test coverage of the library.

Benchmarks

H3 uses benchmarks to offer a comparison of the library's performance between revisions.

Benchmarks are automatically run on Linux x64 with Clang and GCC compilers for each commit in Github Actions.

Fuzzers

H3 uses fuzzers to find novel inputs that crash or result in other undefined behavior.

On each commit, CI is triggered to run OSS-Fuzz for H3. OSS-Fuzz regularly runs fuzzers against the latest development version of H3 and reports newly discovered issues to the H3 core maintainers.