flaky 3.8.1

Marking tests flaky

To mark a test as flaky, simply import flaky and decorate the test with @flaky:

from flaky import flaky

@flaky
def test_something_that_usually_passes(self):
    value_to_double = 21
    result = get_result_from_flaky_doubler(value_to_double)
    self.assertEqual(result, value_to_double * 2, 'Result doubled incorrectly.')

By default, flaky will retry a failing test once, but that behavior can be overridden by passing values to the flaky decorator. It accepts two parameters: max_runs, and min_passes; flaky will run tests up to max_runs times, until it has succeeded min_passes times. Once a test passes min_passes times, it’s considered a success; once it has been run max_runs times without passing min_passes times, it’s considered a failure.

@flaky(max_runs=3, min_passes=2)
def test_something_that_usually_passes(self):
    """This test must pass twice, and it can be run up to three times."""
    value_to_double = 21
    result = get_result_from_flaky_doubler(value_to_double)
    self.assertEqual(result, value_to_double * 2, 'Result doubled incorrectly.')

@flaky
class TestMultipliers(TestCase):
    def test_flaky_doubler(self):
        value_to_double = 21
        result = get_result_from_flaky_doubler(value_to_double)
        self.assertEqual(result, value_to_double * 2, 'Result doubled incorrectly.')

    @flaky(max_runs=3)
    def test_flaky_tripler(self):
        value_to_triple = 14
        result = get_result_from_flaky_tripler(value_to_triple)
        self.assertEqual(result, value_to_triple * 3, 'Result tripled incorrectly.')

Don’t rerun certain types of failures

Depending on your tests, some failures are obviously not due to flakiness. Instead of rerunning after those failures, you can specify a filter function that can tell flaky to fail the test right away.

def is_not_crash(err, *args):
    return not issubclass(err[0], ProductCrashedError)

@flaky
def test_something():
    raise ProductCrashedError

@flaky(rerun_filter=is_not_crash)
def test_something_else():
    raise ProductCrashedError

Flaky will run test_something twice, but will only run test_something_else once.

It can also be used to incur a delay between test retries:

import time

def delay_rerun(*args):
    time.sleep(1)
    return True

@flaky(rerun_filter=delay_rerun)
def test_something_else():
    ...

Activating the plugin

With pytest, flaky will automatically run. It can, however be disabled via the command line:

pytest -p no:flaky

Command line arguments

Setup

Create a virtual environment and install packages -

mkvirtualenv flaky
pip install -r requirements-dev.txt

Testing

Run all tests using -

tox

The tox tests include code style checks via pycodestyle and pylint.