Hybrid Cloud Test Generation

This skill generates tests for Sentry's hybrid cloud architecture. It covers RPC services, API gateway proxying, outbox patterns, and endpoint silo decorators.

Critical Constraints

ALWAYS use factory methods (self.create_user() , self.create_organization() ) — never Model.objects.create() .

NEVER wrap factory method calls in assume_test_silo_mode or assume_test_silo_mode_of . Factories are silo-aware and handle silo mode internally. Only use silo mode context managers for direct ORM queries (Model.objects.get/filter/count/exists/delete ).

ALWAYS use pytest -style assertions (assert x == y ) — never self.assertEqual() .

ALWAYS add tests to existing test files rather than creating new ones, unless no file exists for that module.

For cross-silo ORM access: use assume_test_silo_mode_of(Model) when accessing a single model (auto-detects silo). Use assume_test_silo_mode(SiloMode.X) when the block covers multiple models or non-model operations.

Use TestCase for most tests, including those using outbox_runner() . Only use TransactionTestCase when tests need real committed transactions (threading, concurrency, multi-process scenarios).

NEVER use from future import annotations in test files that deal with RPC models.

Step 1: Identify Test Category

Determine which category of HC test to generate based on the user's request:

Signal Category Go To

RPC service, service method, serialization round-trip, dispatch RPC Service Tests Step 3

API gateway, proxy, middleware, forwarding API Gateway Tests Step 4

Outbox, cross-silo message, ControlOutbox, RegionOutbox, outbox drain Outbox Pattern Tests Step 5

API endpoint with silo decorator, endpoint test, permission check Endpoint Silo Tests Step 6

If the signal is ambiguous, ask the user to clarify which category.

Step 2: Gather Context

Before generating any test:

Read the source module being tested. Determine its silo mode by checking for @cell_silo_endpoint , @control_silo_endpoint , local_mode = SiloMode.X , or @cell_silo_model /@control_silo_model decorators.

Find the existing test file using the mirror path convention:

• src/sentry/foo/bar.py → tests/sentry/foo/test_bar.py

• src/sentry/foo/services/bar/service.py → tests/sentry/foo/services/test_bar.py

• src/sentry/foo/services/bar/impl.py → tests/sentry/foo/services/test_bar.py

Read the existing test file to understand what's already tested, what base classes are used, and what patterns are established.

Read source method signatures to understand parameters, return types, and which RPC models are involved.

Step 3: Generate RPC Service Tests

Load references/rpc-service-tests.md for complete templates and patterns.

RPC service tests must cover:

• Silo compatibility: @all_silo_test ensures the service works across all silo modes

• Serialization round-trip: dispatch_to_local_service verifies args/return survive serialization

• Field accuracy: Field-by-field comparison of RPC model against ORM object

• Error handling: Not-found returns, disabled methods, remote exception wrapping

• Cross-silo effects: outbox_runner()

• assume_test_silo_mode for propagation checks

Quick Reference — Decorator & Base Class

Scenario Decorator Base Class

Standard RPC service @all_silo_test

TestCase

RPC with named regions @all_silo_test(regions=create_test_regions("us"))

TestCase

RPC with member mapping assertions @all_silo_test

TestCase, HybridCloudTestMixin

Step 4: Generate API Gateway Tests

Load references/api-gateway-tests.md for complete templates and patterns.

API gateway tests verify that requests to control-silo endpoints are correctly proxied to the appropriate region. They must cover:

• Proxy pass-through: Requests forwarded with correct params, headers, body

• Query parameter forwarding: Multi-value params preserved

• Error proxying: Upstream errors forwarded correctly

• Streaming responses: close_streaming_response() for reading proxied response body

Quick Reference — Decorator & Base Class

Scenario Decorator Base Class

Standard gateway test @control_silo_test(regions=[ApiGatewayTestCase.REGION], include_monolith_run=True)

ApiGatewayTestCase

Step 5: Generate Outbox Pattern Tests

Load references/outbox-tests.md for complete templates and patterns.

Outbox tests verify that cross-silo messages are created, drained, and produce the expected side effects. They must cover:

• Outbox creation: Verify correct outbox records with outbox_context(flush=False)

• Outbox processing: outbox_runner() drains pending messages

• Cross-silo side effects: assume_test_silo_mode_of(Model) to check replica/mapping state

• Idempotency: Draining the same shard twice produces no duplicates

Quick Reference — Decorator & Base Class

Scenario Decorator Base Class

Control outbox test @control_silo_test

TestCase

Region outbox test @region_silo_test

TestCase

Outbox with threading/concurrency (none) TransactionTestCase

Step 6: Generate Endpoint Silo Tests

Load references/endpoint-silo-tests.md for complete templates and patterns.

Endpoint silo tests verify that API endpoints work correctly under their declared silo mode. They must cover:

• Correct silo decorator: Match endpoint → test decorator

• Cross-silo data setup: Create data using factory methods (no silo wrapper needed)

• Permission checks: Verify 401/403 for unauthorized access

• Response accuracy: Verify response body matches expected data

Quick Reference — Decorator Mapping

Endpoint Decorator Test Decorator

@cell_silo_endpoint

@region_silo_test

@control_silo_endpoint

@control_silo_test

@control_silo_endpoint (with proxy) @control_silo_test(regions=create_test_regions("us"))

No decorator (monolith-only) @no_silo_test

Step 7: Validate

Before presenting the generated test, verify against this checklist:

• Correct silo decorator on test class

• assume_test_silo_mode_of(Model) for single-model ORM access; assume_test_silo_mode(SiloMode.X) for multi-model/non-model ORM blocks

• Factory methods (self.create_* ) are NEVER wrapped in assume_test_silo_mode

• Factory methods used — never Model.objects.create()

• pytest -style assertions only (assert x == y )

• Correct base class (TestCase for most tests; TransactionTestCase only for threading/concurrency)

• Imports are correct and minimal

• Test file at correct mirror path

• Test methods have descriptive names (test_<action>_<scenario> )

• Run command: pytest -svv --reuse-db tests/sentry/path/to/test_file.py

Key Imports Quick Reference

Silo decorators

from sentry.testutils.silo import ( all_silo_test, control_silo_test, region_silo_test, no_silo_test, assume_test_silo_mode, assume_test_silo_mode_of, create_test_regions, )

Base classes

from sentry.testutils.cases import TestCase, TransactionTestCase, APITestCase

Cross-silo utilities

from sentry.testutils.outbox import outbox_runner from sentry.testutils.hybrid_cloud import HybridCloudTestMixin from sentry.silo.base import SiloMode

RPC testing

from sentry.hybridcloud.rpc.service import dispatch_to_local_service

API gateway testing

from sentry.testutils.helpers.apigateway import ApiGatewayTestCase, verify_request_params

Outbox models

from sentry.hybridcloud.models.outbox import ControlOutbox, RegionOutbox, outbox_context from sentry.hybridcloud.outbox.category import OutboxCategory, OutboxScope

Context Manager Quick Reference

Use ONLY for direct ORM queries — never for factory calls

assume_test_silo_mode(SiloMode.CONTROL) # Switch to control silo for ORM access assume_test_silo_mode(SiloMode.CELL) # Switch to cell silo for ORM access assume_test_silo_mode_of(ModelClass) # Switch to silo matching model's silo mode

outbox_runner() # Drain all pending outboxes on exit outbox_context(flush=False) # Create outboxes without flushing override_regions(regions) # Override active region config override_settings(SILO_MODE=SiloMode.X) # Override Django settings override_options({"key": value}) # Override Sentry options