How to Perform Benchmark Testing of Models on Different Hardware in the Gallery App

The Gallery app performs on-device benchmark testing using a Litert-LM integration that measures init latency, prefill throughput, and decode tokens per second, storing results in a DataStore for cross-hardware comparison.

The google-ai-edge/gallery repository provides a complete Android implementation for benchmark testing of models on different hardware. This open-source app leverages the Litert-LM library to execute performance benchmarks directly on-device, enabling developers to compare LLM inference speeds across CPU, GPU, and NNAPI configurations without external dependencies.

Architecture of the On-Device Benchmarking System

The benchmark testing implementation follows a Model-View-ViewModel (MVVM) architecture that isolates performance measurement logic from UI presentation.

UI Layer Components

The user interface resides in three primary Kotlin files under Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/benchmark/:

• BenchmarkScreen.kt: Hosts the model selector and results list view.
• BenchmarkConfigDialog.kt: Presents the configuration dialog for warm-up and benchmark iteration counts (lines 79-87).
• BenchmarkResultsViewer.kt: Renders saved benchmark runs and handles baseline comparison logic (lines 156-162, 441-500).

Data and Execution Layer

Behind the scenes, BenchmarkViewModel.kt orchestrates the benchmark execution. This ViewModel creates temporary cache directories under the app's cache folder, invokes the Litert-LM benchmark() function, and persists aggregated results to a DataStore<BenchmarkResults> defined in di/AppModule.kt.

Executing a Benchmark Test

The workflow for benchmark testing begins when a user taps "Run benchmark" from the chat panel.

First, BenchmarkConfigDialog collects configuration parameters:

val warmUpIterations = remember { mutableStateOf(2) }
val benchmarkIterations = remember { mutableStateOf(5) }

Button(
    onClick = {
        onBenchmarkClicked(
            message = selectedMessage,
            warmUpIterations = warmUpIterations.value,
            benchmarkIterations = benchmarkIterations.value
        )
    }
) { Text(stringResource(R.string.run_benchmark)) }

The BenchmarkViewModel.runBenchmark() function then handles the actual execution (lines 124-251 in BenchmarkViewModel.kt):

fun runBenchmark(
    model: Model,
    warmUpIterations: Int,
    benchmarkIterations: Int,
    onComplete: (BenchmarkResult) -> Unit
) {
    val timestamp = System.currentTimeMillis()
    val cacheDir = File(appContext.cacheDir, "benchmark_$timestamp")
    cacheDir.mkdirs()
    Log.d(TAG, "Using benchmark cache dir: ${cacheDir.path}")

    // Execute Litert-LM benchmark for each iteration
    repeat(benchmarkIterations) {
        val info = benchmark(
            modelPath = model.path,
            cacheDir = cacheDir.path,
            warmUpIterations = warmUpIterations,
        )
        // Collect per-iteration metrics
        prefillSpeeds.add(info.lastPrefillTokensPerSecond)
        decodeSpeeds.add(info.lastDecodeTokensPerSecond)
        timesToFirstToken.add(info.timeToFirstTokenInSecond)
    }

    // Compute averages and persist to DataStore
    val result = BenchmarkResult.newBuilder()
        .setAvgPrefillTokensPerSecond(prefillSpeeds.average())
        .setAvgDecodeTokensPerSecond(decodeSpeeds.average())
        .setAvgTimeToFirstToken(timesToFirstToken.average())
        .build()
    benchmarkResultsStore.updateData { it + result }
    cacheDir.deleteRecursively()
    onComplete(result)
}

Key Performance Metrics for Hardware Evaluation

The Litert-LM library (com.google.ai.edge.litertlm.benchmark) extracts four critical metrics that vary significantly across different hardware:

1. initTimeInSecond: Model initialization latency.
2. lastPrefillTokensPerSecond: Prompt processing throughput.
3. lastDecodeTokensPerSecond: Token generation speed during inference.
4. timeToFirstTokenInSecond: End-to-end latency before the first response token.

Because these benchmarks execute on-device using the hardware's specific CPU, GPU, or NNAPI delegates, results automatically reflect the exact silicon capabilities—whether running on Snapdragon 8 Gen 2, Tensor G3, or other chipsets.

Comparing Results Across Different Devices

After completing benchmark testing on multiple devices, the Benchmark Results screen enables direct comparison through the baseline selection feature.

In BenchmarkResultsViewer.kt, users tap a Baseline chip on any result (e.g., a Pixel 7 run) to establish a reference point. The UI then calculates percentage differences for all other runs, displaying metrics like "+23% faster decode on Galaxy S23" relative to the selected baseline.

For offline analysis, the Export CSV button downloads a spreadsheet containing all numeric metrics from the benchmark_results.pb DataStore, allowing detailed comparison of model performance across different hardware configurations in external tools.

Summary

• The google-ai-edge/gallery app implements benchmark testing through a dedicated MVVM stack in Android/src/app/src/main/java/com/google/ai/edge/gallery/ui/benchmark/.
• BenchmarkViewModel.kt orchestrates Litert-LM execution, creating temporary cache directories and aggregating metrics into BenchmarkResult protobufs.
• Four core metrics—init latency, prefill throughput, decode throughput, and time-to-first-token—capture hardware-specific performance characteristics.
• Results persist in a DataStore (benchmark_results.pb) and support baseline comparison through BenchmarkResultsViewer.kt.
• All benchmarks run on-device, ensuring measurements reflect actual hardware capabilities rather than server-side approximations.

Frequently Asked Questions

What hardware delegates does the Gallery app benchmark support?

The Litert-LM library automatically utilizes available hardware delegates including CPU, GPU, and NNAPI. The benchmark results reflect whichever delegate the model utilizes on the specific device, allowing you to compare NNAPI performance on Pixel devices against GPU acceleration on Snapdragon platforms.

How many iterations should I configure for accurate benchmark testing?

The BenchmarkConfigDialog defaults to 2 warm-up iterations and 5 benchmark iterations. For stable results across different hardware, use at least 3-5 warm-up passes to stabilize thermals and caches, followed by 5-10 measurement iterations. The ViewModel computes averages across all iterations to minimize variance.

Where are benchmark results stored on the device?

Results serialize to benchmark_results.pb via a DataStore provider defined in di/AppModule.kt. This protobuf file persists in the app's private storage and survives app restarts, enabling longitudinal comparison of model performance as you test different hardware configurations.

Can I export benchmark data to compare across phones?

Yes. BenchmarkResultsViewer.kt implements an Export CSV feature that writes all metrics—including avgPrefillTokensPerSecond, avgDecodeTokensPerSecond, and avgTimeToFirstToken—to a spreadsheet. This facilitates offline analysis when evaluating which hardware best fits specific LLM performance requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}