Running Pytest Tests Against Multiple Backends?

I've built a series of tests (using pytest) for a codebase interacting with the Github API (both making calls to it, and receiving webhooks). Currently, these tests run against a s

Solution 1:

There are several ways to approach this. The one I would go for it by default run your mocked tests and then when a command line flag is present test against both the mock and the real version.

So first the easier part, adding a command line option:

defpytest_addoption(parser):
    parser.addoption('--github', action='store_true',
                 help='Also test against real github')

Now this is available via the pytestconfig fixture as pytestconfig.getoption('github'), often also indirectly available, e.g. via the request fixture as request.config.getoption('github').

Now you need to use this parametrize any test which needs to interact with the github API so that they get run both with the mock and with the real instance. Without knowing your code it sounds like a good point would be the Werkzeug client: make this into a fixture and then it can be parameterized to return both a real client or the test client you mention:

@pytest.fixturedefwerkzeug_client(werkzeug_client_type):
    if werkzeug_client_type == 'real':
        return create_the_real_client()
    else:
        return create_the_mock_client()

defpytest_generate_tests(metafunc):
    if'werkzeug_client_type'in metafunc.fixturenames:
        types = ['mock']
        if metafunc.config.getoption('github'):
            types.append('real')
        metafunc.parametrize('werkzeug_client_type', types)

Now if you write your test as:

deftest_foo(werkzeug_client):
     assert werkzeug_client.whatever()

You will get one test normally and two tests when invoked with pytest --github.

(Be aware hooks must be in conftest.py files while fixtures can be anywhere. Be extra aware that the pytest_addoption hook should really only be used in the toplevel conftest file to avoid you from confusion about when the hook is used by pytest and when not. So you should put all this code in a toplevel conftest.py file really.)