Using Benchmarks

Codeflash is able the determine the impact of an optimization on predefined benchmarks, when used in benchmark mode.

Benchmark mode is an easy way for users to define workflows that are performance-critical and need to be optimized. For example, if a user has an important function that requires minimal latency, the user can define a benchmark for that function. Codeflash will then calculate the impact (if any) of any optimization on the performance of that function.

Using Codeflash in Benchmark Mode

Create a benchmarks root

Create a directory for benchmarks. This directory must be a sub directory of your tests directory.

In your pyproject.toml, add the path to the 'benchmarks-root' section.

[tool.codeflash]
# All paths are relative to this pyproject.toml's directory.
module-root = "inference"
tests-root = "tests"
test-framework = "pytest"
benchmarks-root = "tests/benchmarks" # add your benchmarks root dir here
ignore-paths = []
formatter-cmds = ["disabled"]

Define your benchmarks

Currently, Codeflash only supports benchmarks written as pytest-benchmarks. Check out the pytest-benchmark documentation for more information on syntax.

For example:
```
from core.bubble_sort import sorter

def test_sort(benchmark):
    result = benchmark(sorter, list(reversed(range(500))))
    assert result == list(range(500))
```
Note that these benchmarks should be defined in such a way that they don't take a long time to run.

The pytest-benchmark format is simply used as an interface. The plugin is actually not used - Codeflash will run these benchmarks with its own pytest plugin
Run Codeflash

Run Codeflash with the --benchmark flag. Note that benchmark mode cannot be used with --all.
```
codeflash --file test_file.py --benchmark
```
If you did not define your benchmarks-root in your pyproject.toml, you can do:
```
codeflash --file test_file.py --benchmark --benchmarks-root path/to/benchmarks
```
Run Codeflash in CI

Benchmark mode is best used together with Codeflash as a Github Action. This way, with every PR, you will know the impact of Codeflash's optimizations on your benchmarks.

Use codeflash init for an easy way to set up Codeflash as a Github Action (with the option to enable benchmark mode).

Otherwise, you can run the following command in your Codeflash GitHub Action:
```
codeflash --benchmark
```

How it works

Codeflash identifies benchmarks in the benchmarks-root directory.
The benchmarks are run so that runtime statistics and information can be recorded.
Replay tests are generated so the performance of optimization candidates on the exact inputs used in the benchmarks can be measured.
If an optimization candidate is verified to be correct, the speedup of the optimization is calculated for each benchmark.
Codeflash then reports the impact of the optimization on each benchmark.

Using Codeflash with benchmarks is a great way to find optimizations that really matter.

Codeflash is actively working on this feature and will be adding new capabilities in the near future!

Using Codeflash in Benchmark Mode​

How it works​

Using Codeflash in Benchmark Mode

How it works