Our review

Sets up and runs integration tests for dbt adapters using Testcontainers against real database engines in Docker containers.

Strengths

Verifies real database compatibility with SQL dialects and connection edge cases.
Isolates each test session in a clean container, ensuring reproducibility and automatic cleanup.
Uses specific image tags for consistent test results.
Optimizes performance with session or module scoped pytest fixtures.

Limitations

Requires Docker and a container runtime environment.
Container startup can slow down test execution.
Limited to database images supported by Testcontainers.

When to use it

When testing dbt adapters against real database engines to catch SQL dialect and connection issues.

When not to use it

If unit tests are sufficient or Docker is not available in the test environment.

Examples

Integration test for a new Postgres adapter

Set up integration tests for a new Postgres dbt adapter using Testcontainers, including a pytest fixture that starts a Postgres container and a test that verifies table creation and data insertion.

Filesystem test with DuckDB

Write a pytest fixture that starts a DuckDB container for testing filesystem-dependent operations in a dbt adapter, and add a test that checks that reading and writing CSV files works correctly.

BigQuery adapter integration test

Create an integration test for a BigQuery adapter using Testcontainers, testing partition extraction and query execution against a real BigQuery emulator or container.

name: dbt-helpers-integration-testcontainers description: Set up and run integration tests using Testcontainers. Use when testing warehouse adapters against real database engines or verifying filesystem-dependent operations.

dbt-helpers-integration-testcontainers

Purpose

To verify that our adapters work correctly against real infrastructure, ensuring that SQL dialect nuances, connection edge cases, and filesystem behaviors are correctly handled.

Core Principles

Isolation: Each test session should have its own clean container instance.
Reproducibility: Use specific image tags (e.g., duckdb:latest, postgres:15) to ensure consistent results.
Efficiency: Use pytest fixtures with session or module scope where appropriate to minimize container startup overhead.

Patterns

1. Database Fixture

@pytest.fixture(scope="session")
def duckdb_container():
    with DuckDBContainer("duckdb/duckdb:latest") as duckdb:
        yield duckdb

@pytest.fixture
def catalog_client(duckdb_container):
    return DuckDBCatalogClient(connection_url=duckdb_container.get_connection_url())

2. Cleanup

Always ensure containers are stopped and resources are cleaned up using context managers or yield fixtures.

Instructions

When writing integration tests

Locate tests in src/<package>/tests/integration/.
Use testcontainers-python for managing Docker instances.
Verify the adapter's behavior against the real engine (e.g., check that BQ partition info is correctly extracted).

When implementing a new warehouse adapter

Create a corresponding integration test in the plugin package.
Test both "happy path" and common failure modes (e.g., table not found, permission denied).

dbt Integration Testing with Testcontainers

Recommended for