Benchmarking in C#: Measuring Performance with Confidence

As developers, we all want our C# applications to run faster. But here’s a truth that even experienced engineers sometimes forget: without proper measurement, performance tuning is just guesswork. Benchmarking is the tool that gives us confidence in our choices, separating “it feels faster” from “it is faster.”

In this article, we’ll take a deep dive into benchmarking in C#, exploring not just the basics but also the nuances that matter when you’re working at scale. Whether you’re building low-latency APIs, high-throughput services, or computationally heavy algorithms, understanding how to measure performance accurately is a core engineering skill.

The Problem with Naïve Benchmarking

Most developers start with something like this:

var sw = Stopwatch.StartNew();
MyMethod();
sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds);

While Stopwatch is great for quick checks, it’s not reliable for precise benchmarking. Why?

• JIT compilation: The first time a method is called, the JIT compiler kicks in. That time gets included in your measurement unless you warm up the code first.
• CPU and OS noise: Background tasks, context switches, and turbo boost can introduce variability between runs.
• Garbage Collection: If GC happens during your measurement, the results will be skewed.
• Microsecond-level timing: Stopwatch isn’t granular enough for extremely fast code paths.

In short: Stopwatch is useful for diagnostics, not for scientific benchmarking.

CLR, JIT, and Why Warmup Matters

Benchmarking in C# is complicated by the nature of the CLR:

• JIT optimizations: The Just-In-Time compiler may inline methods, unroll loops, or eliminate “dead” code paths if it thinks the results aren’t used.
• Tiered compilation: .NET Core uses tiered JIT, where methods are first compiled with quick optimizations, then recompiled later with heavier optimizations.
• Garbage collection pauses: GC can pause execution mid-run. Without multiple iterations, your measurement might just reflect unlucky timing.

This is why serious benchmarking frameworks like BenchmarkDotNet perform warmup runs before collecting data, ensuring JIT overhead doesn’t distort your results.

BenchmarkDotNet: The Professional Way

BenchmarkDotNet has become the industry standard for .NET performance benchmarking. It doesn’t just time your methods-it manages warmups, multiple iterations, statistical analysis, and even environment isolation to ensure reliable results.

Basic Example: String Concatenation

Let’s revisit string concatenation vs StringBuilder:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Text;

[MemoryDiagnoser]
public class StringBenchmarks
{
    [Benchmark]
    public string ConcatStrings()
    {
        string result = "";
        for (int i = 0; i < 1000; i++)
            result += i;
        return result;
    }

    [Benchmark]
    public string UseStringBuilder()
    {
        var sb = new StringBuilder();
        for (int i = 0; i < 1000; i++)
            sb.Append(i);
        return sb.ToString();
    }
}

public class Program
{
    public static void Main(string[] args) =>
        BenchmarkRunner.Run();
}

The output includes execution time, error margins, standard deviation, and memory allocations for each method-far more than a Stopwatch can provide.

Real-World Benchmarking Examples

Example 1: List vs HashSet Lookup

This is a classic performance comparison. Let’s benchmark lookup speed:

public class LookupBenchmarks
{
    private List list;
    private HashSet set;

    [GlobalSetup]
    public void Setup()
    {
        list = Enumerable.Range(0, 10000).ToList();
        set = new HashSet(list);
    }

    [Benchmark]
    public bool ListContains() => list.Contains(9999);

    [Benchmark]
    public bool HashSetContains() => set.Contains(9999);
}

You’ll see that HashSet.Contains is significantly faster-especially as the list grows.

Example 2: Async vs Sync I/O

Let’s compare reading a file synchronously vs asynchronously:

public class FileReadBenchmarks
{
    private const string Path = "sample.txt";

    [Benchmark]
    public string ReadSync() => File.ReadAllText(Path);

    [Benchmark]
    public async Task ReadAsync() => await File.ReadAllTextAsync(Path);
}

This helps you decide whether async I/O is worth it for your workload. Spoiler: it usually is, especially under load.

Example 3: Sorting Algorithms

Let’s compare Array.Sort vs OrderBy:

public class SortingBenchmarks
{
    private int[] data;

    [Params(100, 1000, 10000)]
    public int N;

    [GlobalSetup]
    public void Setup() =>
        data = Enumerable.Range(0, N).Reverse().ToArray();

    [Benchmark(Baseline = true)]
    public void ArraySort() => Array.Sort(data);

    [Benchmark]
    public void LinqOrderBy() => data.OrderBy(x => x).ToArray();
}

This shows how algorithm choice affects performance as input size grows.

Example 4: Span vs Array Slicing

Let’s compare slicing an array using Span<T> vs copying with Array.Copy:

public class SliceBenchmarks
{
    private int[] data;

    [GlobalSetup]
    public void Setup() =>
        data = Enumerable.Range(0, 1000).ToArray();

    [Benchmark]
    public int[] ArrayCopy()
    {
        var slice = new int[100];
        Array.Copy(data, 100, slice, 0, 100);
        return slice;
    }

    [Benchmark]
    public Span SpanSlice()
    {
        Span span = data;
        return span.Slice(100, 100);
    }
}

Span avoids allocations and is faster for many scenarios.

Interpreting Results Correctly

Raw numbers are only half the story. You need to understand:

• Mean and Median: Average execution times. Median is less sensitive to outliers.
• StdDev and Error: Variability in results. High values suggest noise or instability.
• Confidence intervals: Are the differences statistically significant, or within margin of error?
• Allocations: Fewer allocations often mean less GC pressure, which can matter more than raw execution speed.

Common Mistakes Even Pros Make

• Running benchmarks in Debug mode (always use Release)
• Benchmarking code paths that the JIT optimizes away
• Ignoring memory traffic in allocation-heavy workloads
• Running benchmarks on laptops with power-saving modes enabled
• Not repeating benchmarks enough times to account for noise

Bringing It Into the Real World

Benchmarking should inform-not replace-profiling. Profiling tells you where the bottlenecks are, and benchmarking tells you which approach is faster. Together, they form a complete performance toolkit.

For production systems, you can even integrate BenchmarkDotNet into CI pipelines, generating reports to ensure performance regressions are caught early.

Summary

Benchmarking in C# is about moving from intuition to evidence. By understanding the CLR’s behavior, accounting for JIT and GC effects, and using tools like BenchmarkDotNet, you can measure with confidence. For senior engineers, the value isn’t just knowing which method is faster-it’s knowing why, and being able to trust the data behind your decisions.

At the end of the day, the fastest code is the code you can prove is fast. With solid benchmarking practices, you’ll be able to optimize your applications with clarity, precision, and confidence.