contract-tester

Description: API contract validation, breaking change detection, and consumer-driven contract testing

Category: API Quality Assurance

Complexity: Medium-High (schema validation + contract compatibility)

Purpose

Validate API implementations against CTR (Contract) documents, detect breaking changes, and ensure provider-consumer compatibility. Prevents API contract violations before deployment.

Capabilities

CTR YAML Schema Validation

Validate YAML structure against schema
Check required fields completeness
Verify data type correctness
Validate enum values
Check format specifications (email, URL, UUID)

OpenAPI/AsyncAPI Compliance

Convert CTR to OpenAPI 3.0/3.1 spec
Validate against OpenAPI schema
AsyncAPI validation for event-driven APIs
JSON Schema validation
GraphQL schema validation

Contract Versioning

Semantic versioning validation (MAJOR.MINOR.PATCH)
Version compatibility checking
Deprecation warnings
Migration path validation

Breaking Change Detection

Removed endpoints/operations
Changed request/response schemas
Removed required fields
Changed field types
Stricter validation rules
Changed authentication requirements

Mock Server Generation

Generate mock API server from CTR
Realistic response generation
Configurable response delays
Error scenario simulation
State management for stateful APIs

Consumer-Driven Contract Testing

Pact contract testing
Provider verification
Consumer contract generation
Bidirectional contract testing
Contract evolution tracking

Contract Compatibility Matrix

Provider version compatibility
Consumer version support
Deprecation timeline tracking
Migration requirements

API Documentation Generation

Generate API documentation from CTR
Interactive API explorer
Code examples in multiple languages
Authentication guides

Contract Testing Workflow

graph TD A[CTR Document] --> B[Parse YAML] B --> C{Valid YAML?} C -->|No| D[Report Parse Error] C -->|Yes| E[Schema Validation]

E --> F{Schema Valid?}
F -->|No| G[Report Schema Errors]
F -->|Yes| H[Version Check]

H --> I{Version Valid?}
I -->|No| J[Report Version Error]
I -->|Yes| K[Breaking Change Detection]

K --> L{Previous Version Exists?}
L -->|No| M[First Version]
L -->|Yes| N[Compare Versions]

N --> O{Breaking Changes?}
O -->|Yes| P[Report Breaking Changes]
O -->|No| Q[Compatible Change]

M --> R[Generate Mock Server]
Q --> R
P --> R

R --> S[Run Contract Tests]
S --> T{Tests Pass?}
T -->|No| U[Report Test Failures]
T -->|Yes| V[Generate Documentation]

V --> W[Validation Complete]

D --> W
G --> W
J --> W
U --> W

Usage Instructions

Validate CTR Document

contract-tester validate --file ctrs/CTR-USER-V1.yaml

Output:

=== Contract Validation Report: CTR-USER-V1 ===

Status: PASS

Schema Validation: ✓

Service name: user-service
Version: 1.0.0
Protocol: REST
Endpoints: 5
Models: 3

OpenAPI Compliance: ✓

Valid OpenAPI 3.1 specification
All schemas properly defined
Authentication specified (Bearer JWT)

Version Check: ✓

Semantic versioning: 1.0.0
No previous version (initial release)

Breaking Changes: N/A (initial release)

Warnings (1):

Endpoint DELETE /users/{id} lacks rate limiting specification → Recommendation: Add rate limiting to prevent abuse

Summary: Contract is valid and ready for implementation

Detect Breaking Changes

contract-tester diff
--previous ctrs/CTR-USER-V1.yaml
--current ctrs/CTR-USER-V2.yaml

Output:

=== Breaking Change Analysis: V1 → V2 ===

BREAKING CHANGES (3):

[CRITICAL] Removed endpoint
- DELETE /users/{id}
- Impact: Consumers using this endpoint will fail
- Migration: Use DELETE /users/{id}/deactivate instead
[CRITICAL] Changed response type
- Endpoint: GET /users/{id}
- Field: created_at
- V1: string (format: date-time)
- V2: integer (unix timestamp)
- Impact: Date parsing will fail in consumers
- Migration: Update consumers to handle unix timestamp
[MAJOR] Added required field
- Endpoint: POST /users
- Field: email_verified (boolean, required)
- Impact: Existing POST requests missing this field will fail
- Migration: Update all POST requests to include email_verified

NON-BREAKING CHANGES (2):

[MINOR] Added optional field
- Endpoint: GET /users/{id}
- Field: last_login (string, optional)
- Impact: None (backward compatible)
[PATCH] Fixed typo in description
- Model: User
- Field: username
- Impact: Documentation only

RECOMMENDATION: Version should be 2.0.0 (MAJOR bump due to breaking changes) Current version: 1.1.0 ❌ Suggested version: 2.0.0 ✓

Generate Mock Server

contract-tester mock
--file ctrs/CTR-USER-V1.yaml
--port 8080
--response-delay 100ms

Mock server features:

Realistic data generation based on schemas
Configurable response delays
Error scenario injection
Request validation
Response logging

Test against mock server

curl http://localhost:8080/api/users/123

Response

{ "user_id": "123e4567-e89b-12d3-a456-426614174000", "username": "john_doe", "email": "john.doe@example.com", "created_at": "2025-01-15T10:30:00Z", "status": "active" }

Run Consumer Contract Tests

contract-tester test-consumer
--contract ctrs/CTR-USER-V1.yaml
--consumer user-frontend
--provider user-service
--output reports/contract-tests/

Generated Pact test:

import pytest from pact import Consumer, Provider, Like, EachLike, Term

@pytest.fixture def pact(): return Consumer('user-frontend').has_pact_with( Provider('user-service'), pact_dir='./pacts' )

def test_get_user_contract(pact): """Verify GET /users/{id} contract""" expected_response = { 'user_id': Like('123e4567-e89b-12d3-a456-426614174000'), 'username': Like('john_doe'), 'email': Term(r'.+@.+..+', 'john.doe@example.com'), 'created_at': Like('2025-01-15T10:30:00Z'), 'status': Term(r'active|inactive|suspended', 'active') }

(pact
 .given('user with id 123 exists')
 .upon_receiving('a request for user 123')
 .with_request('GET', '/api/users/123')
 .will_respond_with(200, body=expected_response))

with pact:
    result = user_api.get_user('123')
    assert result.status_code == 200
    assert 'user_id' in result.json()

Verify Provider Implementation

contract-tester test-provider
--contract ctrs/CTR-USER-V1.yaml
--provider-url http://localhost:8000
--output reports/provider-verification/

CTR Document Structure

Complete CTR Example

ctrs/CTR-USER-V1.yaml

contract_id: CTR-USER-V1 version: 1.0.0 service_name: user-service protocol: REST base_path: /api

metadata: description: User management API contract owner: backend-team consumers: - user-frontend - admin-dashboard traceability: requirements: - REQ-USER-001 - REQ-USER-002 specifications: - SPEC-API-V1

authentication: type: bearer scheme: JWT token_location: header header_name: Authorization

endpoints:

operation_id: get_user method: GET path: /users/{id} summary: Retrieve user by ID parameters:
- name: id in: path required: true schema: type: string format: uuid responses: 200: description: User found content: application/json: schema: $ref: '#/components/schemas/User' 404: description: User not found content: application/json: schema: $ref: '#/components/schemas/Error' 401: description: Unauthorized rate_limit: requests: 100 window: 60s
operation_id: create_user method: POST path: /users summary: Create new user request_body: required: true content: application/json: schema: $ref: '#/components/schemas/CreateUserRequest' responses: 201: description: User created content: application/json: schema: $ref: '#/components/schemas/User' 400: description: Invalid request content: application/json: schema: $ref: '#/components/schemas/Error' 409: description: User already exists
operation_id: update_user method: PUT path: /users/{id} summary: Update user parameters:
- name: id in: path required: true schema: type: string format: uuid request_body: required: true content: application/json: schema: $ref: '#/components/schemas/UpdateUserRequest' responses: 200: description: User updated content: application/json: schema: $ref: '#/components/schemas/User' 404: description: User not found
operation_id: delete_user method: DELETE path: /users/{id} summary: Delete user parameters:
- name: id in: path required: true schema: type: string format: uuid responses: 204: description: User deleted 404: description: User not found
operation_id: list_users method: GET path: /users summary: List all users parameters:
- name: page in: query schema: type: integer minimum: 1 default: 1
- name: limit in: query schema: type: integer minimum: 1 maximum: 100 default: 20 responses: 200: description: User list content: application/json: schema: type: object properties: users: type: array items: $ref: '#/components/schemas/User' pagination: $ref: '#/components/schemas/Pagination'

components: schemas: User: type: object required: - user_id - username - email - created_at - status properties: user_id: type: string format: uuid description: Unique user identifier username: type: string minLength: 3 maxLength: 20 pattern: '^[a-zA-Z0-9_]+$' description: Unique username email: type: string format: email description: User email address full_name: type: string maxLength: 100 description: User's full name created_at: type: string format: date-time description: Account creation timestamp updated_at: type: string format: date-time description: Last update timestamp status: type: string enum: [active, inactive, suspended] description: Account status

CreateUserRequest:
  type: object
  required:
    - username
    - email
    - password
  properties:
    username:
      type: string
      minLength: 3
      maxLength: 20
      pattern: '^[a-zA-Z0-9_]+$'
    email:
      type: string
      format: email
    password:
      type: string
      minLength: 8
      maxLength: 128
    full_name:
      type: string
      maxLength: 100

UpdateUserRequest:
  type: object
  properties:
    email:
      type: string
      format: email
    full_name:
      type: string
      maxLength: 100
    status:
      type: string
      enum: [active, inactive, suspended]

Error:
  type: object
  required:
    - error_code
    - message
  properties:
    error_code:
      type: string
      description: Machine-readable error code
    message:
      type: string
      description: Human-readable error message
    details:
      type: object
      description: Additional error details

Pagination:
  type: object
  required:
    - page
    - limit
    - total
  properties:
    page:
      type: integer
      description: Current page number
    limit:
      type: integer
      description: Items per page
    total:
      type: integer
      description: Total number of items
    has_next:
      type: boolean
      description: Whether there are more pages

Breaking Change Rules

authentication: type: bearer scheme: JWT

username: minLength: 3 # Less strict

PATCH Version Changes (x.y.Z)

Documentation Updates
Bug Fixes in Descriptions
Typo Corrections
Example Updates

Contract Compatibility Matrix

Provider Version Compatibility:

V1.0.0 consumers can use:

V1.0.0, V1.0.1, V1.1.0, V1.2.0 providers ✓
V2.0.0 providers ❌ (breaking changes)

V1.1.0 consumers can use:

V1.1.0, V1.1.1, V1.2.0 providers ✓
V1.0.0 providers ⚠️ (missing optional features)
V2.0.0 providers ❌ (breaking changes)

V2.0.0 consumers can use:

V2.0.0, V2.0.1, V2.1.0 providers ✓
V1.x.x providers ❌ (incompatible)

Tool Access

Required tools:

Read : Read CTR YAML files
Write : Generate test files and documentation
Bash : Execute validation and testing tools
Grep : Search for contract patterns

Required libraries:

pyyaml: YAML parsing
jsonschema: Schema validation
openapi-spec-validator: OpenAPI validation
pact-python: Consumer-driven contract testing
faker: Mock data generation

Integration Points

With doc-flow

Validate CTR documents during generation
Ensure traceability to requirements
Update documentation on contract changes

With test-automation

Generate contract tests automatically
Integrate with test suites
Track contract test coverage

With code-review

Validate API implementation against contracts
Detect contract violations in code
Suggest contract-compliant fixes

With devops-flow

Contract validation in CI/CD pipelines
Block deployments with breaking changes
Generate API gateway configurations

Best Practices

Semantic versioning: Follow MAJOR.MINOR.PATCH strictly
Backward compatibility: Prefer additive changes
Deprecation notices: Warn before removing features
Consumer feedback: Involve consumers in contract changes
Documentation: Clear migration guides for breaking changes
Testing: Test all contract changes before release
Versioning strategy: Support N-1 versions minimum
Change log: Maintain comprehensive contract changelog

Success Criteria

100% contract validation coverage
Zero undetected breaking changes
<5 minute contract validation time
100% consumer-provider compatibility in same MAJOR version
All breaking changes documented with migration paths
Mock servers available for all contracts
All CTR documents include complete Document Control sections with:
Project metadata (name, version, date, owner, preparer, status)
Document Revision History table tracking all contract changes

Notes

Contract tests run in CI/CD before deployment
Breaking changes require MAJOR version bump
Provider must maintain backward compatibility within MAJOR version
Consumers should handle minor version differences gracefully
Contract changes trigger automatic notification to consumers

contract-tester

Safety Notice

Copy this and send it to your AI assistant to learn

Test against mock server

Response

ctrs/CTR-USER-V1.yaml

V1

V2 (BREAKING)

Endpoint removed

V1

V2 (BREAKING)

V1

V2 (BREAKING)

V1

V2 (BREAKING)

V1

V2 (BREAKING)

V1.0.0

V1.1.0 (Non-breaking)

V1.0.0

V1.1.0 (Non-breaking)

V1.0.0

V1.1.0 (Non-breaking)

Source Transparency

Related Skills

n8n

google-adk

doc-prd

mermaid-gen