RealityKit Development Guide

Purpose: Build 3D content, AR experiences, and spatial computing apps using RealityKit's Entity-Component-System architecture iOS Version: iOS 13+ (base), iOS 18+ (RealityView on iOS), visionOS 1.0+ Xcode: Xcode 15+

When to Use This Skill

Use this skill when:

• Building any 3D experience (AR, games, visualization, spatial computing)

• Creating SwiftUI apps with 3D content (RealityView, Model3D)

• Implementing AR with anchors (world, image, face, body tracking)

• Working with Entity-Component-System (ECS) architecture

• Setting up physics, collisions, or spatial interactions

• Building multiplayer or shared AR experiences

• Migrating from SceneKit to RealityKit

• Targeting visionOS

Do NOT use this skill for:

• SceneKit maintenance (use axiom-scenekit )

• 2D games (use axiom-spritekit )

• Metal shader programming (use axiom-metal-migration-ref )

• Pure GPU compute (use Metal directly)

1. Mental Model: ECS vs Scene Graph

Scene Graph (SceneKit)

In SceneKit, nodes own their properties. A node IS a renderable, collidable, animated thing.

Entity-Component-System (RealityKit)

In RealityKit, entities are empty containers. Components add data. Systems process that data.

Entity (identity + hierarchy) ├── TransformComponent (position, rotation, scale) ├── ModelComponent (mesh + materials) ├── CollisionComponent (collision shapes) ├── PhysicsBodyComponent (mass, mode) └── [YourCustomComponent] (game-specific data)

System (processes entities with specific components each frame)

Why ECS matters:

• Composition over inheritance: Combine any components on any entity

• Data-oriented: Systems process arrays of components efficiently

• Decoupled logic: Systems don't know about each other

• Testable: Components are pure data, Systems are pure logic

The ECS Mental Shift

Scene Graph Thinking ECS Thinking

"The player node moves" "The movement system processes entities with MovementComponent"

"Add a method to the node subclass" "Add a component, create a system"

"Override update(_:) in the node" "Register a System that queries for components"

"The node knows its health" "HealthComponent holds data, DamageSystem processes it"

2. Entity Hierarchy

Creating Entities

// Empty entity let entity = Entity() entity.name = "player"

// Entity with components let entity = Entity() entity.components[ModelComponent.self] = ModelComponent( mesh: .generateBox(size: 0.1), materials: [SimpleMaterial(color: .blue, isMetallic: false)] )

// ModelEntity convenience (has ModelComponent built in) let box = ModelEntity( mesh: .generateBox(size: 0.1), materials: [SimpleMaterial(color: .red, isMetallic: true)] )

Hierarchy Management

// Parent-child parent.addChild(child) child.removeFromParent()

// Find entities let found = root.findEntity(named: "player")

// Enumerate for child in entity.children { // Process children }

// Clone let clone = entity.clone(recursive: true)

Transform

// Local transform (relative to parent) entity.position = SIMD3<Float>(0, 1, 0) entity.orientation = simd_quatf(angle: .pi / 4, axis: SIMD3(0, 1, 0)) entity.scale = SIMD3<Float>(repeating: 2.0)

// World-space queries let worldPos = entity.position(relativeTo: nil) let worldTransform = entity.transform(relativeTo: nil)

// Set world-space transform entity.setPosition(SIMD3(1, 0, 0), relativeTo: nil)

// Look at a point entity.look(at: targetPosition, from: entity.position, relativeTo: nil)

3. Components

Built-in Components

Component Purpose

Transform

Position, rotation, scale

ModelComponent

Mesh geometry + materials

CollisionComponent

Collision shapes for physics and interaction

PhysicsBodyComponent

Mass, physics mode (dynamic/static/kinematic)

PhysicsMotionComponent

Linear and angular velocity

AnchoringComponent

AR anchor attachment

SynchronizationComponent

Multiplayer sync

PerspectiveCameraComponent

Camera settings

DirectionalLightComponent

Directional light

PointLightComponent

Point light

SpotLightComponent

Spot light

CharacterControllerComponent

Character physics controller

AudioMixGroupsComponent

Audio mixing

SpatialAudioComponent

3D positional audio

AmbientAudioComponent

Non-positional audio

ChannelAudioComponent

Multi-channel audio

OpacityComponent

Entity transparency

GroundingShadowComponent

Contact shadow

InputTargetComponent

Gesture input (visionOS)

HoverEffectComponent

Hover highlight (visionOS)

AccessibilityComponent

VoiceOver support

Custom Components

struct HealthComponent: Component { var current: Int var maximum: Int

var percentage: Float {
    Float(current) / Float(maximum)
}

}

// Register before use (typically in app init) HealthComponent.registerComponent()

// Attach to entity entity.components[HealthComponent.self] = HealthComponent(current: 100, maximum: 100)

// Read if let health = entity.components[HealthComponent.self] { print(health.current) }

// Modify entity.components[HealthComponent.self]?.current -= 10

Component Lifecycle

Components are value types (structs). When you read a component, modify it, and write it back, you're replacing the entire component:

// Read-modify-write pattern var health = entity.components[HealthComponent.self]! health.current -= damage entity.components[HealthComponent.self] = health

Anti-pattern: Holding a reference to a component and expecting mutations to propagate. Components are copied on read.

4. Systems

System Protocol

struct DamageSystem: System { // Define which components this system needs static let query = EntityQuery(where: .has(HealthComponent.self))

init(scene: RealityKit.Scene) {
    // One-time setup
}

func update(context: SceneUpdateContext) {
    for entity in context.entities(matching: Self.query,
                                    updatingSystemWhen: .rendering) {
        var health = entity.components[HealthComponent.self]!
        if health.current <= 0 {
            entity.removeFromParent()
        }
    }
}

}

// Register system DamageSystem.registerSystem()

System Best Practices

• One responsibility per system: MovementSystem, DamageSystem, RenderingSystem — not GameLogicSystem

• Query filtering: Use precise queries to avoid processing irrelevant entities

• Order matters: Systems run in registration order. Register dependencies first.

• Avoid storing entity references: Query each frame instead. Entity references can become stale.

Event Handling

// Subscribe to collision events scene.subscribe(to: CollisionEvents.Began.self) { event in let entityA = event.entityA let entityB = event.entityB // Handle collision }

// Subscribe to scene update scene.subscribe(to: SceneEvents.Update.self) { event in let deltaTime = event.deltaTime // Per-frame logic }

5. SwiftUI Integration

RealityView (iOS 18+, visionOS 1.0+)

struct ContentView: View { var body: some View { RealityView { content in // make closure — called once let box = ModelEntity( mesh: .generateBox(size: 0.1), materials: [SimpleMaterial(color: .blue, isMetallic: false)] ) content.add(box)

    } update: { content in
        // update closure — called when SwiftUI state changes
    }
}

}

RealityView with Camera (iOS)

On iOS, RealityView provides a camera content parameter for configuring the AR or virtual camera:

RealityView { content, attachments in // Load 3D content if let model = try? await ModelEntity(named: "scene") { content.add(model) } }

Loading Content Asynchronously

RealityView { content in // Load from bundle if let entity = try? await Entity(named: "MyScene", in: .main) { content.add(entity) }

// Load from URL
if let entity = try? await Entity(contentsOf: modelURL) {
    content.add(entity)
}

}

Model3D (Simple Display)

// Simple 3D model display (no interaction) Model3D(named: "toy_robot") { model in model .resizable() .scaledToFit() } placeholder: { ProgressView() }

SwiftUI Attachments (visionOS)

RealityView { content, attachments in let entity = ModelEntity(mesh: .generateSphere(radius: 0.1)) content.add(entity)

if let label = attachments.entity(for: "priceTag") {
    label.position = SIMD3(0, 0.15, 0)
    entity.addChild(label)
}

} attachments: { Attachment(id: "priceTag") { Text("$9.99") .padding() .glassBackgroundEffect() } }

State Binding Pattern

struct GameView: View { @State private var score = 0

var body: some View {
    VStack {
        Text("Score: \(score)")

        RealityView { content in
            let scene = try! await Entity(named: "GameScene")
            content.add(scene)
        } update: { content in
            // React to state changes
            // Note: update is called when SwiftUI state changes,
            // not every frame. Use Systems for per-frame logic.
        }
    }
}

}

6. AR on iOS

AnchorEntity

// Horizontal plane let anchor = AnchorEntity(.plane(.horizontal, classification: .table, minimumBounds: SIMD2(0.2, 0.2)))

// Vertical plane let anchor = AnchorEntity(.plane(.vertical, classification: .wall, minimumBounds: SIMD2(0.5, 0.5)))

// World position let anchor = AnchorEntity(world: SIMD3<Float>(0, 0, -1))

// Image anchor let anchor = AnchorEntity(.image(group: "AR Resources", name: "poster"))

// Face anchor (front camera) let anchor = AnchorEntity(.face)

// Body anchor let anchor = AnchorEntity(.body)

SpatialTrackingSession (iOS 18+)

let session = SpatialTrackingSession() let configuration = SpatialTrackingSession.Configuration(tracking: [.plane, .object]) let result = await session.run(configuration)

if let notSupported = result { // Handle unsupported tracking on this device for denied in notSupported.deniedTrackingModes { print("Not supported: (denied)") } }

AR Best Practices

• Anchor entities to detected surfaces rather than world positions for stability

• Use plane classification (.table , .floor , .wall ) to place content appropriately

• Start with horizontal plane detection — it's the most reliable

• Test on real devices; simulator AR is limited

• Provide visual feedback during surface detection (coaching overlay)

7. Interaction

ManipulationComponent (iOS, visionOS)

// Enable drag, rotate, scale gestures entity.components[ManipulationComponent.self] = ManipulationComponent( allowedModes: .all // .translate, .rotate, .scale )

// Also requires CollisionComponent for hit testing entity.generateCollisionShapes(recursive: true)

InputTargetComponent (visionOS)

// Required for visionOS gesture input entity.components[InputTargetComponent.self] = InputTargetComponent() entity.components[CollisionComponent.self] = CollisionComponent( shapes: [.generateBox(size: SIMD3(0.1, 0.1, 0.1))] )

Gesture Integration with SwiftUI

RealityView { content in let entity = ModelEntity(mesh: .generateBox(size: 0.1)) entity.generateCollisionShapes(recursive: true) entity.components.set(InputTargetComponent()) content.add(entity) } .gesture( TapGesture() .targetedToAnyEntity() .onEnded { value in let tappedEntity = value.entity // Handle tap } ) .gesture( DragGesture() .targetedToAnyEntity() .onChanged { value in value.entity.position = value.convert(value.location3D, from: .local, to: .scene) } )

Hit Testing

// Ray-cast from screen point if let result = arView.raycast(from: screenPoint, allowing: .estimatedPlane, alignment: .horizontal).first { let worldPosition = result.worldTransform.columns.3 // Place entity at worldPosition }

8. Materials and Rendering

Material Types

Material Purpose Customization

SimpleMaterial

Solid color or texture Color, metallic, roughness

PhysicallyBasedMaterial

Full PBR All PBR maps (base color, normal, metallic, roughness, AO, emissive)

UnlitMaterial

No lighting response Color or texture, always fully lit

OcclusionMaterial

Invisible but occludes AR content hiding behind real objects

VideoMaterial

Video playback on surface AVPlayer-driven

ShaderGraphMaterial

Custom shader graph Reality Composer Pro

CustomMaterial

Metal shader functions Full Metal control

PhysicallyBasedMaterial

var material = PhysicallyBasedMaterial() material.baseColor = .init(tint: .white, texture: .init(try! .load(named: "albedo"))) material.metallic = .init(floatLiteral: 0.0) material.roughness = .init(floatLiteral: 0.5) material.normal = .init(texture: .init(try! .load(named: "normal"))) material.ambientOcclusion = .init(texture: .init(try! .load(named: "ao"))) material.emissiveColor = .init(color: .blue) material.emissiveIntensity = 2.0

let entity = ModelEntity( mesh: .generateSphere(radius: 0.1), materials: [material] )

OcclusionMaterial (AR)

// Invisible plane that hides 3D content behind it let occluder = ModelEntity( mesh: .generatePlane(width: 1, depth: 1), materials: [OcclusionMaterial()] ) occluder.position = SIMD3(0, 0, 0) anchor.addChild(occluder)

Environment Lighting

// Image-based lighting if let resource = try? await EnvironmentResource(named: "studio_lighting") { // Apply via RealityView content }

9. Physics and Collision

Collision Shapes

// Generate from mesh (accurate but expensive) entity.generateCollisionShapes(recursive: true)

// Manual shapes (prefer for performance) entity.components[CollisionComponent.self] = CollisionComponent( shapes: [ .generateBox(size: SIMD3(0.1, 0.2, 0.1)), // Box .generateSphere(radius: 0.1), // Sphere .generateCapsule(height: 0.3, radius: 0.05) // Capsule ] )

Physics Body

// Dynamic — physics simulation controls movement entity.components[PhysicsBodyComponent.self] = PhysicsBodyComponent( massProperties: .init(mass: 1.0), material: .generate(staticFriction: 0.5, dynamicFriction: 0.3, restitution: 0.4), mode: .dynamic )

// Static — immovable collision surface ground.components[PhysicsBodyComponent.self] = PhysicsBodyComponent( mode: .static )

// Kinematic — code-controlled, participates in collisions platform.components[PhysicsBodyComponent.self] = PhysicsBodyComponent( mode: .kinematic )

Collision Groups and Filters

// Define groups let playerGroup = CollisionGroup(rawValue: 1 << 0) let enemyGroup = CollisionGroup(rawValue: 1 << 1) let bulletGroup = CollisionGroup(rawValue: 1 << 2)

// Filter: player collides with enemies and bullets entity.components[CollisionComponent.self] = CollisionComponent( shapes: [.generateSphere(radius: 0.1)], filter: CollisionFilter( group: playerGroup, mask: enemyGroup | bulletGroup ) )

Collision Events

// Subscribe in RealityView make closure or System scene.subscribe(to: CollisionEvents.Began.self, on: playerEntity) { event in let otherEntity = event.entityA == playerEntity ? event.entityB : event.entityA handleCollision(with: otherEntity) }

Applying Forces

if var motion = entity.components[PhysicsMotionComponent.self] { motion.linearVelocity = SIMD3(0, 5, 0) // Impulse up entity.components[PhysicsMotionComponent.self] = motion }

10. Animation

Transform Animation

// Animate to position over duration entity.move( to: Transform( scale: SIMD3(repeating: 1.5), rotation: simd_quatf(angle: .pi, axis: SIMD3(0, 1, 0)), translation: SIMD3(0, 2, 0) ), relativeTo: entity.parent, duration: 2.0, timingFunction: .easeInOut )

Playing USD Animations

if let entity = try? await Entity(named: "character") { // Play all available animations for animation in entity.availableAnimations { entity.playAnimation(animation.repeat()) } }

Animation Playback Control

let controller = entity.playAnimation(animation) controller.pause() controller.resume() controller.speed = 2.0 // 2x playback speed controller.blendFactor = 0.5 // Blend with current state

11. Audio

Spatial Audio

// Load audio resource let resource = try! AudioFileResource.load(named: "engine.wav", configuration: .init(shouldLoop: true))

// Create entity with spatial audio let audioEntity = Entity() audioEntity.components[SpatialAudioComponent.self] = SpatialAudioComponent() let controller = audioEntity.playAudio(resource)

// Position the audio source in 3D space audioEntity.position = SIMD3(2, 0, -1)

Ambient Audio

entity.components[AmbientAudioComponent.self] = AmbientAudioComponent() entity.playAudio(backgroundMusic)

12. Performance

Entity Count

• Under 100 entities: No concerns

• 100-1000 entities: Monitor with RealityKit debugger

• 1000+ entities: Use instancing and LOD strategies

Instancing

// Share mesh and material across many entities let sharedMesh = MeshResource.generateSphere(radius: 0.01) let sharedMaterial = SimpleMaterial(color: .white, isMetallic: false)

for i in 0..<1000 { let entity = ModelEntity(mesh: sharedMesh, materials: [sharedMaterial]) entity.position = randomPosition() parent.addChild(entity) }

RealityKit automatically batches entities with identical mesh and material resources.

Component Churn

Anti-pattern: Creating and replacing components every frame.

// BAD — component allocation every frame func update(context: SceneUpdateContext) { for entity in context.entities(matching: query, updatingSystemWhen: .rendering) { entity.components[ModelComponent.self] = ModelComponent( mesh: .generateBox(size: 0.1), materials: [newMaterial] // New allocation every frame ) } }

// GOOD — modify existing component func update(context: SceneUpdateContext) { for entity in context.entities(matching: query, updatingSystemWhen: .rendering) { // Only update when actually needed if needsUpdate { var model = entity.components[ModelComponent.self]! model.materials = [cachedMaterial] entity.components[ModelComponent.self] = model } } }

Collision Shape Optimization

• Use simple shapes (box, sphere, capsule) instead of mesh-based collision

• generateCollisionShapes(recursive: true) is convenient but expensive

• For static geometry, generate shapes once during setup

Profiling

Use Xcode's RealityKit debugger:

• Entity Inspector: View entity hierarchy and components

• Statistics Overlay: Entity count, draw calls, triangle count

• Physics Visualization: Show collision shapes

13. Multiplayer

Synchronization Basics

// Components sync automatically if they conform to Codable struct ScoreComponent: Component, Codable { var points: Int }

// SynchronizationComponent controls what syncs entity.components[SynchronizationComponent.self] = SynchronizationComponent()

MultipeerConnectivityService

let service = try MultipeerConnectivityService(session: mcSession) // Entities with SynchronizationComponent auto-sync across peers

Ownership

• Only the owner of an entity can modify it

• Request ownership before modifying shared entities

• Non-Codable component data does not sync

14. Anti-Patterns

Anti-Pattern 1: UIKit-Style Thinking in ECS

Time cost: Hours of frustration from fighting the architecture

// BAD — subclassing Entity for behavior class PlayerEntity: Entity { func takeDamage(_ amount: Int) { /* logic in entity */ } }

// GOOD — component holds data, system has logic struct HealthComponent: Component { var hp: Int } struct DamageSystem: System { static let query = EntityQuery(where: .has(HealthComponent.self)) func update(context: SceneUpdateContext) { // Process damage here } }

Anti-Pattern 2: Monolithic Entities

Time cost: Untestable, inflexible architecture

Don't put all game logic in one entity type. Split into components that can be mixed and matched.

Anti-Pattern 3: Frame-Based Updates Without Systems

Time cost: Missed frame updates, inconsistent behavior

// BAD — timer-based updates Timer.scheduledTimer(withTimeInterval: 1/60, repeats: true) { _ in entity.position.x += 0.01 }

// GOOD — System update struct MovementSystem: System { static let query = EntityQuery(where: .has(VelocityComponent.self)) func update(context: SceneUpdateContext) { for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) { let velocity = entity.components[VelocityComponent.self]! entity.position += velocity.value * Float(context.deltaTime) } } }

Anti-Pattern 4: Not Generating Collision Shapes for Interactive Entities

Time cost: 15-30 min debugging "why taps don't work"

Gestures require CollisionComponent . If an entity has InputTargetComponent (visionOS) or ManipulationComponent but no CollisionComponent , gestures will never fire.

Anti-Pattern 5: Storing Entity References in Systems

Time cost: Crashes from stale references

// BAD — entity might be removed between frames struct BadSystem: System { var playerEntity: Entity? // Stale reference risk

func update(context: SceneUpdateContext) {
    playerEntity?.position.x += 0.1  // May crash
}

}

// GOOD — query each frame struct GoodSystem: System { static let query = EntityQuery(where: .has(PlayerComponent.self))

func update(context: SceneUpdateContext) {
    for entity in context.entities(matching: Self.query,
                                    updatingSystemWhen: .rendering) {
        entity.position.x += Float(context.deltaTime)
    }
}

}

15. Code Review Checklist

• Custom components registered via registerComponent() before use

• Systems registered via registerSystem() before scene loads

• Components are value types (structs), not classes

• Read-modify-write pattern used for component updates

• Interactive entities have CollisionComponent

• visionOS interactive entities have InputTargetComponent

• Collision shapes are simple (box/sphere/capsule) where possible

• No entity references stored across frames in Systems

• Mesh and material resources shared across identical entities

• Component updates only occur when values actually change

• USD/USDZ format used for 3D assets (not .scn)

• Async loading used for all model/scene loading

• [weak self] in closure-based subscriptions if retaining view/controller

16. Pressure Scenarios

Scenario 1: "ECS Is Overkill for Our Simple App"

Pressure: Team wants to avoid learning ECS, just needs one 3D model displayed

Wrong approach: Skip ECS, jam all logic into RealityView closures.

Correct approach: Even simple apps benefit from ECS. A single ModelEntity in a RealityView is already using ECS — you're just not adding custom components yet. Start simple, add components as complexity grows.

Push-back template: "We're already using ECS — Entity and ModelComponent. The pattern scales. Adding a custom component when we need behavior is one struct definition, not an architecture change."

Scenario 2: "Just Use SceneKit, We Know It"

Pressure: Team has SceneKit experience, RealityKit is unfamiliar

Wrong approach: Build new features in SceneKit.

Correct approach: SceneKit is soft-deprecated. New features won't be added. Invest in RealityKit now — the ECS concepts transfer to other game engines (Unity, Unreal, Bevy) if needed.

Push-back template: "SceneKit is in maintenance mode — no new features, only security patches. Every line of SceneKit we write is migration debt. RealityKit's concepts (Entity, Component, System) are industry-standard ECS."

Scenario 3: "Make It Work Without Collision Shapes"

Pressure: Deadline, collision shape setup seems complex

Wrong approach: Skip collision shapes, use position-based proximity detection.

Correct approach: entity.generateCollisionShapes(recursive: true) takes one line. Without it, gestures won't work and physics won't collide. The "shortcut" creates more debugging time than it saves.

Push-back template: "Collision shapes are required for gestures and physics. It's one line: entity.generateCollisionShapes(recursive: true) . Skipping it means gestures silently fail — a harder bug to diagnose."

Resources

WWDC: 2019-603, 2019-605, 2021-10074, 2022-10074, 2023-10080, 2023-10081, 2024-10103, 2024-10153

Docs: /realitykit, /realitykit/entity, /realitykit/realityview, /realitykit/modelentity, /realitykit/anchorentity, /realitykit/component

Skills: axiom-realitykit-ref, axiom-realitykit-diag, axiom-scenekit, axiom-scenekit-ref