AI Tools Compared

Best AI Tools for Writing Pytest Fixtures

Writing good pytest fixtures requires understanding scope, teardown, parametrize interactions, and dependency injection patterns. AI tools vary significantly in how well they handle these — some generate fixtures that work but have subtle scope bugs; others produce boilerplate that’s hard to extend.

This guide tests Claude, Copilot, and Cursor against real fixture-writing tasks.


The Test Cases

Three scenarios that distinguish good from mediocre fixture generation:

  1. Database fixture with proper teardown — requires understanding transaction rollback vs truncate
  2. Factory fixture with faker integration — requires knowing factory_boy patterns
  3. Parametrized fixture with indirect — a common point of confusion even for experienced developers

Test 1: Database Fixture with Proper Teardown

Prompt:

Write a pytest fixture for a PostgreSQL database that:
- Creates a test database at session scope
- Provides a connection at function scope with transaction rollback after each test
- Works with psycopg2
- Handles cleanup even if tests fail

Claude’s output:

import pytest
import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT

TEST_DB_NAME = "test_myapp"
ADMIN_DSN = "postgresql://postgres:password@localhost/postgres"


@pytest.fixture(scope="session")
def test_database():
    """Create test database once per session, drop after."""
    admin_conn = psycopg2.connect(ADMIN_DSN)
    admin_conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
    cursor = admin_conn.cursor()

    cursor.execute(f"DROP DATABASE IF EXISTS {TEST_DB_NAME}")
    cursor.execute(f"CREATE DATABASE {TEST_DB_NAME}")
    cursor.close()
    admin_conn.close()

    yield f"postgresql://postgres:password@localhost/{TEST_DB_NAME}"

    # Teardown: drop the test database
    admin_conn = psycopg2.connect(ADMIN_DSN)
    admin_conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
    cursor = admin_conn.cursor()
    cursor.execute(f"DROP DATABASE IF EXISTS {TEST_DB_NAME}")
    cursor.close()
    admin_conn.close()


@pytest.fixture(scope="function")
def db_connection(test_database):
    """Provide a connection with automatic transaction rollback."""
    conn = psycopg2.connect(test_database)
    conn.autocommit = False

    yield conn

    # Always rollback, even on test failure
    conn.rollback()
    conn.close()


@pytest.fixture(scope="function")
def db_cursor(db_connection):
    """Convenience fixture providing a cursor."""
    cursor = db_connection.cursor()
    yield cursor
    cursor.close()

Copilot’s output (inline in VS Code): Similar structure but it used conn.commit() in the teardown instead of conn.rollback(), which would persist test data between tests — a subtle but critical bug. It also didn’t set autocommit = False explicitly.

Cursor generated the same structure as Claude but added a note recommending SQLAlchemy instead of raw psycopg2, which was unsolicited and not wrong, just off-topic.

Winner: Claude — correct teardown semantics on the first try.


Test 2: Factory Fixture with Faker

Prompt:

Write pytest fixtures using factory_boy and faker to create:
- A User model factory with realistic data
- An Order factory that references User
- A fixture that creates a batch of 10 orders for the same user
import factory
import faker as faker_module
from factory.faker import Faker
import pytest

from myapp.models import User, Order

fake = faker_module.Faker()


class UserFactory(factory.Factory):
    class Meta:
        model = User

    id = factory.Sequence(lambda n: n + 1)
    email = factory.LazyAttribute(lambda o: f"user{o.id}@example.com")
    username = Faker("user_name")
    first_name = Faker("first_name")
    last_name = Faker("last_name")
    created_at = Faker("date_time_this_year")


class OrderFactory(factory.Factory):
    class Meta:
        model = Order

    id = factory.Sequence(lambda n: n + 100)
    user = factory.SubFactory(UserFactory)
    total_cents = factory.LazyFunction(lambda: fake.random_int(min=100, max=100000))
    status = factory.Iterator(["pending", "processing", "completed", "cancelled"])
    created_at = Faker("date_time_this_month")


@pytest.fixture
def user():
    """Create a single test user."""
    return UserFactory()


@pytest.fixture
def user_with_orders():
    """Create a user with 10 orders."""
    test_user = UserFactory()
    orders = OrderFactory.create_batch(10, user=test_user)
    return test_user, orders


@pytest.fixture(params=[1, 5, 10])
def user_with_variable_orders(request):
    """Parametrized fixture: user with variable order count."""
    test_user = UserFactory()
    orders = OrderFactory.create_batch(request.param, user=test_user)
    return test_user, orders

All three tools generated correct factory_boy code. The differences were in the create_batch fixture — both GPT-4 and Copilot used factory.build_batch (which doesn’t call save()) when the prompt didn’t specify ORM usage. Claude used create_batch which calls save() — correct if you’re using Django ORM or SQLAlchemy with factory_boy’s ORM support.

For a pure Python model without ORM, you’d want build_batch. Claude would have been wrong there — but for the most common use case (Django/SQLAlchemy), it’s correct.


Test 3: Parametrized Fixtures with indirect

This is where most AI tools struggle. The indirect parameter to @pytest.mark.parametrize lets you pass values through a fixture before the test sees them.

Prompt:

Write a pytest fixture and test that uses indirect parametrize to test
an API client with different authentication states: anonymous, user, admin.
The fixture should set up the auth headers based on the parameter.

Claude:

import pytest
import httpx


AUTH_STATES = {
    "anonymous": {},
    "user": {"Authorization": "Bearer user-token-123"},
    "admin": {"Authorization": "Bearer admin-token-456"},
}


@pytest.fixture
def auth_client(request):
    """Client fixture that configures auth based on indirect parameter."""
    auth_state = request.param  # receives value from parametrize
    headers = AUTH_STATES[auth_state]

    with httpx.Client(
        base_url="http://localhost:8000",
        headers=headers,
    ) as client:
        yield client


@pytest.mark.parametrize(
    "auth_client,expected_status",
    [
        ("anonymous", 401),
        ("user", 200),
        ("admin", 200),
    ],
    indirect=["auth_client"],  # only auth_client goes through the fixture
)
def test_protected_endpoint(auth_client, expected_status):
    response = auth_client.get("/api/protected")
    assert response.status_code == expected_status


@pytest.mark.parametrize(
    "auth_client,expected_status",
    [
        ("anonymous", 403),
        ("user", 403),
        ("admin", 200),
    ],
    indirect=["auth_client"],
)
def test_admin_only_endpoint(auth_client, expected_status):
    response = auth_client.get("/api/admin/users")
    assert response.status_code == expected_status

Copilot generated indirect=True instead of indirect=["auth_client"], which would try to pass expected_status through a fixture too — causing a fixture-not-found error at collection time.

Cursor got indirect right but generated request.params (plural) instead of request.param, a subtle typo that causes an AttributeError at runtime.

Winner for this task: Claude — only tool that got both indirect syntax and request.param correct.


Comparison Table

Fixture Pattern Claude Copilot Cursor
Session/function scope Correct Correct Correct
Transaction rollback Correct teardown Bug: commits instead Correct
factory_boy create vs build Context-aware create_batch always Correct
indirect parametrize Correct indirect=True bug request.params bug
Async fixtures (asyncio) Strong Decent Strong
conftest.py organization Good suggestions Inline only Explicit conftest

When to Use Each Tool

Claude for complex fixture hierarchies and fixtures with non-obvious pytest semantics. Particularly strong at explaining why a fixture is structured a certain way.

Copilot for quick, simple fixtures when you’re already in the flow of writing application code. Fast and usually correct for straightforward cases.

Cursor for fixtures inside a larger codebase where the AI can read your existing models. Its codebase-awareness helps it generate factories that match your actual model fields.


Prompting Tips

Always specify:

Write a pytest fixture with function scope that creates a PostgreSQL
test transaction using SQLAlchemy 2.0 async session. Include teardown
via rollback. The test session should be nested inside a transaction
that rolls back after each test.

The more specific the prompt, the less time you spend fixing scope and teardown issues.



Built by theluckystrike — More at zovo.one