BenchmarkDotNet v0.11.3

Highlights

This release is focused mainly on bug fixes that were affecting user experience. But don't worry, we have some new features too!

  • Diagnosers
    • ConcurrencyVisualizerProfiler (allows profiling benchmarks on Windows and exporting the data to a trace file which can be opened with Concurrency Visualizer)
  • Command-line:
    • --stopOnFirstError: Stops the benchmarks execution on first error. #947
    • --statisticalTest: Performs a Mann–Whitney Statistical Test for identifying regressions and improvements. #960
  • Bug fixes:
    • Dry mode doesn't work because of the ZeroMeasurementHelper #943
    • MannWhitneyTest fails when comparing statistics of different sample size #948 and #950
    • Improve the dynamic loading of Diagnostics package #955
    • BenchmarkRunner.RunUrl throws NRE when Config is not provided #961
    • Don't require the users to do manual installation of TraceEvent when using Diagnostics package #962
    • Stop benchmark after closing application + Flush log after stopping benchmark #963

Diagnosers

ConcurrencyVisualizerProfiler

ConcurrencyVisualizerProfiler allows to profile the benchmarked .NET code on Windows and exports the data to a CVTrace file which can be opened with Concurrency Visualizer.

ConcurrencyVisualizerProfiler uses EtwProfiler to get a .etl file which still can be opened with PerfView or Windows Performance Analyzer. The difference is that it also enables all Task and Thread related ETW Providers and exports a simple xml which can be opened with Visual Studio if you install Concurrency Visualizer plugin

open trace

Utilization

Threads

Command-line

In this release, we have some new command-line arguments!

--stopOnFirstError: Stops the benchmarks execution on first error

When provided, BenchmarkDotNet is going to stop the benchmarks execution on first error.

  • #947 Add option to stop running when the first benchmark fails (by @wojtpl2)

--statisticalTest: Statistical Test

To perform a Mann–Whitney U Test and display the results in a dedicated column you need to provide the Threshold via --statisticalTest. Examples: 5%, 10ms, 100ns, 1s.

Example: run Mann–Whitney U test with relative ratio of 1% for all benchmarks for .NET 4.6 (base), .NET Core 2.0 and .NET Core 2.1.

class Program
{
    static void Main(string[] args) => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
}

public class MySample
{
    [Benchmark]
    public void Sleep()
    {
#if NETFRAMEWORK
        Thread.Sleep(50);
#elif NETCOREAPP2_0
        Thread.Sleep(45);
#elif NETCOREAPP2_1
        Thread.Sleep(55);
#endif
    }
    
    [Benchmark]
    public void Same() => Thread.Sleep(50);
}

dotnet run -c Release -f netcoreapp2.1 --filter * --runtimes net46 netcoreapp2.0 netcoreapp2.1 --statisticalTest 1%

Note: .NET 4.6 will be our baseline because it was provided as first on the runtimes list.

Method Runtime Toolchain Mean Error StdDev Ratio MannWhitney(1%)
Sleep Clr net46 50.51 ms 0.1833 ms 0.1714 ms 1.00 Base
Sleep Core netcoreapp2.0 45.53 ms 0.1262 ms 0.1181 ms 0.90 Faster
Sleep Core netcoreapp2.1 55.50 ms 0.1217 ms 0.1138 ms 1.10 Slower
Same Clr net46 50.47 ms 0.1795 ms 0.1679 ms 1.00 Base
Same Core netcoreapp2.0 50.55 ms 0.1873 ms 0.1752 ms 1.00 Same
Same Core netcoreapp2.1 50.55 ms 0.2162 ms 0.2022 ms 1.00 Same

Milestone details

In the v0.11.3 scope, 10 issues were resolved and 10 pull requests were merged. This release includes 26 commits by 6 contributors.

Resolved issues (10)

  • #870 Error after adding OperationsPerInvoke (assignee: @AndreyAkinshin)
  • #885 Closing application dot't stop benchmark (assignee: @WojciechNagorski)
  • #933 Investigate hanging SingleBenchmarkCanBeExecutedForMultipleRuntimes test (assignee: @adamsitnik)
  • #939 We need an option to stop running when the first benchmark fails. (assignee: @WojciechNagorski)
  • #943 Dry mode doesn't work because of the ZeroMeasurementHelper (assignee: @AndreyAkinshin)
  • #948 BenchmarkDotNet.Mathematics.StatisticalTesting.MannWhitneyTest.PValueForSmallN(int n, int m, double u) (assignee: @AndreyAkinshin)
  • #950 MannWhitneyTest fails when comparing statistics of different sample size (assignee: @AndreyAkinshin)
  • #955 Improve the dynamic loading of Diagnostics package (assignee: @WojciechNagorski)
  • #961 BenchmarkRunner.RunUrl throws NRE when Config is not provided
  • #964 Concurrency Visualizer Profiler (assignee: @adamsitnik)

Merged pull requests (10)

Commits (26)

Contributors (6)

Thank you very much!

Additional details

Date: November 20, 2018

Milestone: v0.11.3 (List of commits)

NuGet Packages: