Class WhatifFilter

An alternative to BenchmarkDotNet's --list command line option that prints a mock summary of all available benchmarks according to the current effective BenchmarkDotNet configuration.

A configuration is the interplay of console arguments, global and local configs, and attributes. The WhatifFilter can be controlled either programmatically via the Enabled property, or by using the --whatif option (short: -w) on the command line and passing the console arguments through PreparseConsoleArguments().

If enabled, the filter will collect all benchmark cases created by BenchmarkRunner, BenchmarkSwitcher, or BenchmarkConverter and suppress their execution.

The properties FilteredBenchmarkCases and FilteredBenchmarkRunInfos provide access to the collected benchmarks, while PrintAsSummaries() outputs them to the console or another logger.

Inheritance

Object

WhatifFilter

Implements

IFilter

Namespace: Mawosoft.Extensions.BenchmarkDotNet

Assembly: Mawosoft.Extensions.BenchmarkDotNet.dll

Syntax

public class WhatifFilter : Object

Examples

public static void Main(string[] args)
{
    // Create the WhatifFilter and let it process the --whatif argument if one exists.
    WhatifFilter whatifFilter = new();
    args = whatifFilter.PreparseConsoleArguments(args);

    // Create a global confing and add the filter (DefaultConfig used for simplicity).
    ManualConfig config = DefaultConfig.Instance.AddFilter(whatifFilter);

    // Run the benchmarks as you would normally do. If the filter was enabled,
    // execution has been supressed and the returned array will be empty.
    Summary[] summaries = BenchmarkRunner.Run(typeof(Program).Assembly, config, args);

    // Check the filter and print the What-if summaries
    if (whatifFilter.Enabled)
    {
        whatifFilter.PrintAsSummaries(ConsoleLogger.Default);
        whatifFilter.Clear(dispose: true);
    }
}

 

Sample Output

• --whatif
• --list flat
• --list tree

// * What If Summary *

Console arguments: --runtimes net5.0 netcoreapp31 net48 --whatif

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19042.1237 (20H2/October2020Update), VM=VirtualBox
AMD Ryzen 5 3400G with Radeon Vega Graphics, 1 CPU, 4 logical and 4 physical cores
.NET SDK=5.0.401
  [Host] : .NET 5.0.10 (5.0.1021.41214), X64 RyuJIT


// Benchmarks1

IterationCount=3  LaunchCount=1  WarmupCount=3

|  Method |        Job |            Runtime |    Toolchain |     Mean | Error | Ratio |
|-------- |----------- |------------------- |------------- |---------:|------:|------:|
| Method1 | Job-PFBCDR |           .NET 5.0 |       net5.0 | 102.0 ms |    NA |  1.00 |
| Method2 | Job-PFBCDR |           .NET 5.0 |       net5.0 | 105.0 ms |    NA |  1.03 |
| Method3 | Job-PFBCDR |           .NET 5.0 |       net5.0 | 108.0 ms |    NA |  1.06 |
| Method1 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 | 100.0 ms |    NA |  0.98 |
| Method2 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 | 103.0 ms |    NA |  1.01 |
| Method3 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 | 106.0 ms |    NA |  1.04 |
| Method1 | Job-LLNYBD | .NET Framework 4.8 |        net48 | 101.0 ms |    NA |  0.99 |
| Method2 | Job-LLNYBD | .NET Framework 4.8 |        net48 | 104.0 ms |    NA |  1.02 |
| Method3 | Job-LLNYBD | .NET Framework 4.8 |        net48 | 107.0 ms |    NA |  1.05 |

// Benchmarks2

IterationCount=3  LaunchCount=1  WarmupCount=3

|  Method |        Job |            Runtime |    Toolchain | Prop1 | next |     Mean | Error | Ratio |
|-------- |----------- |------------------- |------------- |------ |----- |---------:|------:|------:|
| Method3 | Job-PFBCDR |           .NET 5.0 |       net5.0 | False |   44 | 125.0 ms |    NA |  1.00 |
| Method3 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 | False |   44 | 121.0 ms |    NA |  0.97 |
| Method3 | Job-LLNYBD | .NET Framework 4.8 |        net48 | False |   44 | 123.0 ms |    NA |  0.98 |
|         |            |                    |              |       |      |          |       |       |
| Method1 | Job-PFBCDR |           .NET 5.0 |       net5.0 | False |    ? | 113.0 ms |    NA |  1.00 |
| Method1 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 | False |    ? | 109.0 ms |    NA |  0.96 |
| Method1 | Job-LLNYBD | .NET Framework 4.8 |        net48 | False |    ? | 111.0 ms |    NA |  0.98 |
|         |            |                    |              |       |      |          |       |       |
| Method2 | Job-PFBCDR |           .NET 5.0 |       net5.0 | False |    ? | 119.0 ms |    NA |  1.00 |
| Method2 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 | False |    ? | 115.0 ms |    NA |  0.97 |
| Method2 | Job-LLNYBD | .NET Framework 4.8 |        net48 | False |    ? | 117.0 ms |    NA |  0.98 |
|         |            |                    |              |       |      |          |       |       |
| Method3 | Job-PFBCDR |           .NET 5.0 |       net5.0 |  True |   44 | 126.0 ms |    NA |  1.00 |
| Method3 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 |  True |   44 | 122.0 ms |    NA |  0.97 |
| Method3 | Job-LLNYBD | .NET Framework 4.8 |        net48 |  True |   44 | 124.0 ms |    NA |  0.98 |
|         |            |                    |              |       |      |          |       |       |
| Method1 | Job-PFBCDR |           .NET 5.0 |       net5.0 |  True |    ? | 114.0 ms |    NA |  1.00 |
| Method1 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 |  True |    ? | 110.0 ms |    NA |  0.96 |
| Method1 | Job-LLNYBD | .NET Framework 4.8 |        net48 |  True |    ? | 112.0 ms |    NA |  0.98 |
|         |            |                    |              |       |      |          |       |       |
| Method2 | Job-PFBCDR |           .NET 5.0 |       net5.0 |  True |    ? | 120.0 ms |    NA |  1.00 |
| Method2 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 |  True |    ? | 116.0 ms |    NA |  0.97 |
| Method2 | Job-LLNYBD | .NET Framework 4.8 |        net48 |  True |    ? | 118.0 ms |    NA |  0.98 |

// Benchmarks3

LaunchCount=1

|  Method |        Job |            Runtime |    Toolchain | IterationCount | RunStrategy | UnrollFactor | WarmupCount |     Mean | Error | Ratio |
|-------- |----------- |------------------- |------------- |--------------- |------------ |------------- |------------ |---------:|------:|------:|
| Method1 | Job-PFBCDR |           .NET 5.0 |       net5.0 |              3 |     Default |           16 |           3 | 129.0 ms |    NA |  1.00 |
| Method1 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 |              3 |     Default |           16 |           3 | 127.0 ms |    NA |  0.98 |
| Method1 | Job-LLNYBD | .NET Framework 4.8 |        net48 |              3 |     Default |           16 |           3 | 128.0 ms |    NA |  0.99 |
| Method1 |        Dry |           .NET 5.0 |      Default |              1 |   ColdStart |            1 |           1 | 130.0 ms |    NA |  1.01 |
|         |            |                    |              |                |             |              |             |          |       |       |
| Method2 | Job-PFBCDR |           .NET 5.0 |       net5.0 |              3 |     Default |           16 |           3 | 133.0 ms |    NA |  1.00 |
| Method2 | Job-GTPATI |      .NET Core 3.1 | netcoreapp31 |              3 |     Default |           16 |           3 | 131.0 ms |    NA |  0.98 |
| Method2 | Job-LLNYBD | .NET Framework 4.8 |        net48 |              3 |     Default |           16 |           3 | 132.0 ms |    NA |  0.99 |
| Method2 |        Dry |           .NET 5.0 |      Default |              1 |   ColdStart |            1 |           1 | 134.0 ms |    NA |  1.01 |

Constructors

WhatifFilter()

Declaration

public WhatifFilter()

Properties

Enabled

Gets or sets the filter's Enabled state.

Declaration

public bool Enabled { get; set; }

Property Value

Type	Description
Boolean	If true, the filter will collect all benchmark cases created by BenchmarkRunner or related classes, and suppress their execution. If false, the filter is disabled and will not have any effect.

FilteredBenchmarkCases

Gets an enumerable collection of the filtered, individual benchmark cases.

Declaration

public IEnumerable<BenchmarkCase> FilteredBenchmarkCases { get; }

Property Value

Type	Description
IEnumerable<BenchmarkCase>	The collection of the filtered, individual benchmark cases.

FilteredBenchmarkRunInfos

Gets an enumerable collection of the filtered benchmark cases grouped by containing type.

Declaration

public IEnumerable<BenchmarkRunInfo> FilteredBenchmarkRunInfos { get; }

Property Value

Type	Description
IEnumerable<BenchmarkRunInfo>	The collection of the filtered, grouped benchmark cases.

Methods

Clear(Boolean)

Clears the list of collected benchmark cases and optionally disposes them.

Declaration

public void Clear(bool dispose)

Parameters

Type	Name	Description
Boolean	dispose	true to dispose all benchmarks, false to only remove them from the list without disposal.

Predicate(BenchmarkCase)

IFilter implementation.

Declaration

public bool Predicate(BenchmarkCase benchmarkCase)

Parameters

Type	Name	Description
BenchmarkCase	benchmarkCase

Returns

Type	Description
Boolean

PreparseConsoleArguments(String[])

Preparses the console arguments for the option --whatif (short: -w) and automatically enables the filter if the option is present.

Declaration

public string[] PreparseConsoleArguments(string[] args)

Parameters

Type	Name	Description
String[]	args	The array of console arguments as passed to the Main method of the application.

Returns

Type	Description
String[]	The passed in array of console arguments with the --whatif option removed.

PrintAsSummaries(ILogger)

Prints the collected benchmark cases as summaries to the given logger.

Declaration

public void PrintAsSummaries(ILogger logger)

Parameters

Type	Name	Description
ILogger	logger	The logger to use for printing the summaries.

Implements

IFilter

See Also

What-If Filter Sample on GitHub