How GoogleTest Floating-Point Comparison Assertions Handle Precision

GoogleTest's EXPECT_FLOAT_EQ and EXPECT_DOUBLE_EQ macros use a ULP-based (Units in the Last Place) comparison with a hard-coded tolerance of 4 ULPs to determine floating-point equality, while EXPECT_NEAR uses absolute error thresholds.

The google/googletest framework provides specialized assertions for floating-point comparison that go beyond simple equality checks. These assertions implement sophisticated ULP-based precision handling to account for the inherent representational limitations of IEEE-754 floating-point arithmetic. Understanding how EXPECT_FLOAT_EQ, EXPECT_DOUBLE_EQ, and EXPECT_NEAR evaluate numeric closeness is essential for writing robust C++ unit tests that handle rounding errors correctly.

ULP-Based Comparison Architecture

Macro Definitions in gtest.h

The entry points for floating-point comparison are defined in googletest/include/gtest/gtest.h. The EXPECT_FLOAT_EQ macro expands to a predicate format call to CmpHelperFloatingPointEQ templated on float, while EXPECT_DOUBLE_EQ uses the same helper with double.

#define EXPECT_FLOAT_EQ(val1, val2) \
    EXPECT_PRED_FORMAT2(::testing::internal::CmpHelperFloatingPointEQ<float>, \
                        val1, val2)

This indirection allows the framework to capture both the expression strings and the actual values for failure reporting.

The CmpHelperFloatingPointEQ Helper

The template function CmpHelperFloatingPointEQ resides in googletest/include/gtest/gtest.h and serves as the orchestration layer. It constructs FloatingPoint<RawType> wrappers around the input values and delegates the equality decision to the AlmostEquals method.

template <typename RawType>
AssertionResult CmpHelperFloatingPointEQ(const char* lhs_expression,
                                         const char* rhs_expression,
                                         RawType lhs_value, RawType rhs_value) {
  const FloatingPoint<RawType> lhs(lhs_value), rhs(rhs_value);
  if (lhs.AlmostEquals(rhs)) return AssertionSuccess();
  // ... failure formatting logic
}

The FloatingPoint Template and AlmostEquals Logic

The core comparison algorithm lives in googletest/include/gtest/internal/gtest-internal.h within the FloatingPoint<RawType> class. The key method AlmostEquals first rejects any comparison involving NaN values, then calculates the distance between the two numbers in sign-and-magnitude representation.

bool AlmostEquals(const FloatingPoint& rhs) const {
  if (is_nan() || rhs.is_nan()) return false;
  return DistanceBetweenSignAndMagnitudeNumbers(bits_, rhs.bits_) <= kMaxUlps;
}

This approach measures the exact number of representable floating-point values between two numbers, providing a scale-invariant tolerance that automatically adjusts to the magnitude of the compared values.

Default Tolerance and Special Cases

The 4 ULP Default (kMaxUlps)

The tolerance threshold is controlled by static const uint32_t kMaxUlps = 4 defined in the FloatingPoint class within gtest-internal.h. This hard-coded constant means two values are considered equal if they differ by at most four representable floating-point steps. The choice of 4 ULPs balances practical tolerance requirements with the IEEE-754 guarantee that a single operation’s rounding error is bounded by 0.5 ULP.

Handling Zero, Infinity, and NaN

The implementation contains specific logic for IEEE-754 special cases:

• Signed Zero: +0.0 and -0.0 have identical bit patterns after the sign-bit mask is applied during distance calculation, resulting in 0 ULP distance and successful comparison.
• Infinity: The distance calculation saturates at the exponent-bit mask, causing sufficiently large finite values to compare equal to infinity.
• NaN: Any comparison involving a NaN value immediately returns false, causing the assertion to fail regardless of the other operand.

Absolute Error Alternative with EXPECT_NEAR

Unlike the ULP-based macros, EXPECT_NEAR implements absolute error comparison. This macro expands to call DoubleNearPredFormat, which computes std::abs(v1 - v2) <= abs_error. This mechanism does not use the FloatingPoint template or ULP arithmetic; instead, it relies on a user-provided absolute threshold that remains constant regardless of the values' magnitudes.

// Simplified logic
EXPECT_NEAR(val1, val2, abs_error);
// Checks: std::abs(val1 - val2) <= abs_error

Use EXPECT_NEAR when you need deterministic error bounds (e.g., ensuring calculated values are within 0.001 of expected), rather than the relative tolerance provided by ULP comparisons.

Practical Usage Examples

The following examples demonstrate the precision boundaries and special case handling in google/googletest:

#include <gtest/gtest.h>
#include <cmath>

TEST(FloatingPointDemo, FloatEqPrecision) {
  // Within 4 ULPs → assertion succeeds
  EXPECT_FLOAT_EQ(1.0000001f, 1.0000002f);

  // Beyond 4 ULPs → assertion fails
  EXPECT_NONFATAL_FAILURE(EXPECT_FLOAT_EQ(1.0f, 1.0001f), "1.0001");
}

TEST(FloatingPointDemo, SpecialValues) {
  // Signed zero comparison succeeds
  EXPECT_FLOAT_EQ(+0.0f, -0.0f);

  // NaN comparisons always fail, even NaN == NaN
  EXPECT_NONFATAL_FAILURE(EXPECT_FLOAT_EQ(NAN, NAN), "NaN");
}

TEST(FloatingPointDemo, DoublePrecision) {
  // Same ULP logic applies to double precision
  EXPECT_DOUBLE_EQ(1.0000000001, 1.0000000002);
}

TEST(FloatingPointDemo, AbsoluteError) {
  // User-specified absolute tolerance (0.00002)
  EXPECT_NEAR(3.14159, 3.14160, 0.00002);  // succeeds
  
  // Failure when difference exceeds tolerance
  EXPECT_NONFATAL_FAILURE(EXPECT_NEAR(3.14159, 3.14170, 0.00002), "3.14170");
}

Summary

• ULP-based tolerance: EXPECT_FLOAT_EQ and EXPECT_DOUBLE_EQ consider values equal if they are within 4 ULPs (Units in the Last Place) of each other.
• Implementation path: Macros expand to CmpHelperFloatingPointEQ, which uses the FloatingPoint<RawType>::AlmostEquals method in gtest-internal.h.
• Special value handling: The framework correctly treats +0.0 and -0.0 as equal, rejects all NaN comparisons, and handles infinity through saturated distance calculations.
• Alternative mechanism: EXPECT_NEAR provides absolute error comparison via std::abs(v1 - v2) <= abs_error, bypassing ULP arithmetic entirely.

Frequently Asked Questions

What is the default tolerance for EXPECT_FLOAT_EQ in GoogleTest?

The default tolerance is 4 ULPs (Units in the Last Place). This is defined by the constant static const uint32_t kMaxUlps = 4 in the FloatingPoint template class located in googletest/include/gtest/internal/gtest-internal.h. Two floating-point numbers are considered equal if the distance between their sign-and-magnitude representations is less than or equal to 4.

How does GoogleTest handle NaN values in floating-point comparisons?

Comparisons involving NaN always fail. In the AlmostEquals method of the FloatingPoint class, the implementation checks if (is_nan() || rhs.is_nan()) return false; before performing the ULP distance calculation. Consequently, EXPECT_FLOAT_EQ(NAN, NAN) will fail, as will any assertion comparing a NaN to a valid number.

Why does EXPECT_FLOAT_EQ treat +0.0 and -0.0 as equal?

IEEE-754 specifies that +0.0 and -0.0 compare as equal, and GoogleTest honors this by masking the sign bit before calculating the ULP distance. The DistanceBetweenSignAndMagnitudeNumbers function treats these values as having identical bit patterns, resulting in a distance of 0 ULPs, which satisfies the <= kMaxUlps condition.

When should I use EXPECT_NEAR instead of EXPECT_FLOAT_EQ?

Use EXPECT_NEAR when you require an absolute error tolerance that does not scale with the magnitude of the numbers being compared. For example, when testing physical measurements where the error must be below 0.01 units regardless of whether the value is 1.0 or 1,000,000. Use EXPECT_FLOAT_EQ when you need a relative tolerance that accounts for floating-point precision limitations across different scales, which is implemented via the 4 ULP mechanism.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}