How to Use PerformanceHelper in Ocaramba: Measure and Report Test Execution Times

PerformanceHelper in the Ocaramba framework lets you time test operations using StartMeasure() and StopMeasure(), then aggregates results by browser and scenario for automated CI reporting via PrintPerformanceResultsHelper.

The accenture/ocaramba repository provides a dedicated utility for capturing and analyzing test performance metrics without cluttering your test code with manual logging. By integrating directly with the DriverContext, PerformanceHelper automatically tracks elapsed milliseconds across different browsers and generates statistics suitable for TeamCity and AppVeyor dashboards.

Core Components of the Performance Measurement System

The implementation relies on three primary types located in the OcarambaLite project:

• PerformanceHelper (/OcarambaLite/Helpers/PerformanceHelper.cs): Manages a Stopwatch instance, maintains a collection of SavedTimes objects, and computes aggregated statistics including average duration and 90th percentile per scenario.
• SavedTimes (/OcarambaLite/Types/SavedTimes.cs): A simple DTO storing the scenario name, browser identifier (read from BaseConfiguration.TestBrowser), and measured duration in milliseconds.
• PrintPerformanceResultsHelper (/OcarambaLite/Helpers/PrintPerformanceResultsHelper.cs): Formats aggregated data into CI-specific service messages for TeamCity and AppVeyor, typically invoked during test teardown.

Each DriverContext instance initializes a single PerformanceHelper accessible through the PerformanceMeasures property, ensuring consistent measurement scope across test fixtures.

How to Measure Performance in Your Tests

Starting and Stopping Measurements

To capture timing data, invoke StartMeasure() before the operation and StopMeasure(string title) immediately after. The title parameter should uniquely identify the step, often combining the test name with a descriptive suffix.

[Test]
public void LoadingMainPage()
{
    // Start the timer
    this.DriverContext.PerformanceMeasures.StartMeasure();
    
    // Execute the operation to benchmark
    this.Driver.Navigate().GoToUrl("https://example.com");
    
    // Stop and record with a descriptive identifier
    this.DriverContext.PerformanceMeasures.StopMeasure(
        TestContext.CurrentContext.Test.Name + "LoadingMainPage");
}

This pattern is demonstrated in the NUnit test suite at Ocaramba.Tests.NUnit/Tests/PerformanceTestsNUnit.cs (lines 39-45), where measurements wrap navigation and page load operations.

Accessing Aggregated Results Programmatically

For custom reporting or verification, access the AllGroupedDurationsMilliseconds property, which returns a collection of AverageGroupedTimes objects grouped by scenario and browser:

var aggregated = this.DriverContext.PerformanceMeasures.AllGroupedDurationsMilliseconds;

foreach (var item in aggregated)
{
    Console.WriteLine(
        $"{item.StepName} ({item.Browser}) – " +
        $"avg: {item.AverageDuration} ms, " +
        $"p90: {item.Percentile90} ms");
}

The aggregation logic, implemented in PerformanceHelper.cs (lines 68-88), orders measurements by duration, groups them by an anonymous key combining scenario and browser, and calculates both the arithmetic mean and the 90th percentile using Math.Round() and index-based percentile extraction.

Reporting Results to CI Servers

TeamCity and AppVeyor Integration

The PrintPerformanceResultsHelper class provides static methods to output performance data in formats recognized by CI servers. Call these methods in your teardown or fixture cleanup to publish metrics automatically:

[TearDown]
public void ReportPerformance()
{
    // TeamCity service messages
    PrintPerformanceResultsHelper.PrintAverageDurationMillisecondsInTeamcity(
        this.DriverContext.PerformanceMeasures);
    PrintPerformanceResultsHelper.PrintPercentiles90DurationMillisecondsinTeamcity(
        this.DriverContext.PerformanceMeasures);
    
    // AppVeyor commands (optional)
    PrintPerformanceResultsHelper.PrintAverageDurationMillisecondsInAppVeyor(
        this.DriverContext.PerformanceMeasures);
    PrintPerformanceResultsHelper.PrintPercentiles90DurationMillisecondsInAppVeyor(
        this.DriverContext.PerformanceMeasures);
}

These calls are automatically executed in the common base class Ocaramba.Tests.NUnit/ProjectTestBase.cs (lines 91-97), ensuring consistent reporting across all derived test classes without requiring repetitive boilerplate.

Understanding the Aggregation Logic

When StopMeasure completes, the helper creates a SavedTimes instance containing the scenario title, current browser from BaseConfiguration.TestBrowser, and elapsed milliseconds. These records accumulate in the internal loadTimeList.

The AllGroupedDurationsMilliseconds property processes this list through the following steps:

1. Ordering: Sorts all measurements by duration.
2. Grouping: Executes a GroupBy operation on an anonymous key combining scenario name and browser.
3. Calculation: For each group, computes:
   • AverageDuration: Math.Round(savedTimeses.Average(dur => dur.Duration))
   • Percentile90: The element at index ceil(count * 0.9) - 1 after sorting
4. Sorting: Returns the final collection ordered by StepName for consistent reporting.

This approach ensures you receive browser-specific performance baselines, critical for identifying cross-browser performance regressions in Selenium test suites.

Summary

• Access PerformanceHelper through DriverContext.PerformanceMeasures, which provides a shared instance for each test context.
• Capture timings by wrapping code blocks with StartMeasure() and StopMeasure(title) to record scenario-specific durations.
• Retrieve statistics via AllGroupedDurationsMilliseconds, which returns averaged data including 90th percentile calculations per browser.
• Automate reporting using PrintPerformanceResultsHelper static methods to emit TeamCity service messages or AppVeyor commands during teardown.
• Locate source code in OcarambaLite/Helpers/PerformanceHelper.cs and OcarambaLite/Types/SavedTimes.cs for customization or extension.

Frequently Asked Questions

How do I access PerformanceHelper in my test class?

Access the helper through the PerformanceMeasures property of your DriverContext instance. In typical Ocaramba implementations, test classes inherit from a base class that exposes this.DriverContext, allowing you to call this.DriverContext.PerformanceMeasures.StartMeasure() directly without manual instantiation.

What metrics does PerformanceHelper calculate?

PerformanceHelper calculates two primary metrics for each scenario-browser combination: AverageDuration (arithmetic mean rounded to the nearest millisecond) and Percentile90 (the value below which 90% of observations fall). These metrics help identify both typical performance and tail latency in your test executions.

Can I use PerformanceHelper with test frameworks other than NUnit?

Yes. While the examples in Ocaramba.Tests.NUnit demonstrate NUnit integration, the core classes in OcarambaLite have no NUnit dependencies. You can instantiate and use PerformanceHelper directly in MSTest, xUnit, or SpecFlow projects, provided you call the measurement methods and PrintPerformanceResultsHelper from the appropriate lifecycle hooks (such as [TestCleanup] or [AfterScenario]).

Where are the performance results stored before reporting?

Raw timing data accumulates in an internal List<SavedTimes> within the PerformanceHelper instance. Each SavedTimes object stores the scenario identifier, browser name, and duration. This list persists for the lifetime of the DriverContext, allowing you to collect multiple measurements across different test methods before aggregating and reporting them during fixture teardown.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}