Senior Computer Vision Engineer

Production computer vision engineering skill for object detection, image segmentation, and visual AI system deployment.

Table of Contents

• Quick Start

• Core Expertise

• Tech Stack

• Workflow 1: Object Detection Pipeline

• Workflow 2: Model Optimization and Deployment

• Workflow 3: Custom Dataset Preparation

• Architecture Selection Guide

• Reference Documentation

• Common Commands

Quick Start

Generate training configuration for YOLO or Faster R-CNN

python scripts/vision_model_trainer.py models/ --task detection --arch yolov8

Analyze model for optimization opportunities (quantization, pruning)

python scripts/inference_optimizer.py model.pt --target onnx --benchmark

Build dataset pipeline with augmentations

python scripts/dataset_pipeline_builder.py images/ --format coco --augment

Core Expertise

This skill provides guidance on:

• Object Detection: YOLO family (v5-v11), Faster R-CNN, DETR, RT-DETR

• Instance Segmentation: Mask R-CNN, YOLACT, SOLOv2

• Semantic Segmentation: DeepLabV3+, SegFormer, SAM (Segment Anything)

• Image Classification: ResNet, EfficientNet, Vision Transformers (ViT, DeiT)

• Video Analysis: Object tracking (ByteTrack, SORT), action recognition

• 3D Vision: Depth estimation, point cloud processing, NeRF

• Production Deployment: ONNX, TensorRT, OpenVINO, CoreML

Tech Stack

Category Technologies

Frameworks PyTorch, torchvision, timm

Detection Ultralytics (YOLO), Detectron2, MMDetection

Segmentation segment-anything, mmsegmentation

Optimization ONNX, TensorRT, OpenVINO, torch.compile

Image Processing OpenCV, Pillow, albumentations

Annotation CVAT, Label Studio, Roboflow

Experiment Tracking MLflow, Weights & Biases

Serving Triton Inference Server, TorchServe

Workflow 1: Object Detection Pipeline

Use this workflow when building an object detection system from scratch.

Step 1: Define Detection Requirements

Analyze the detection task requirements:

Detection Requirements Analysis:

• Target objects: [list specific classes to detect]
• Real-time requirement: [yes/no, target FPS]
• Accuracy priority: [speed vs accuracy trade-off]
• Deployment target: [cloud GPU, edge device, mobile]
• Dataset size: [number of images, annotations per class]

Step 2: Select Detection Architecture

Choose architecture based on requirements:

Requirement Recommended Architecture Why

Real-time (>30 FPS) YOLOv8/v11, RT-DETR Single-stage, optimized for speed

High accuracy Faster R-CNN, DINO Two-stage, better localization

Small objects YOLO + SAHI, Faster R-CNN + FPN Multi-scale detection

Edge deployment YOLOv8n, MobileNetV3-SSD Lightweight architectures

Transformer-based DETR, DINO, RT-DETR End-to-end, no NMS required

Step 3: Prepare Dataset

Convert annotations to required format:

COCO format (recommended)

python scripts/dataset_pipeline_builder.py data/images/
--annotations data/labels/
--format coco
--split 0.8 0.1 0.1
--output data/coco/

Verify dataset

python -c "from pycocotools.coco import COCO; coco = COCO('data/coco/train.json'); print(f'Images: {len(coco.imgs)}, Categories: {len(coco.cats)}')"

Step 4: Configure Training

Generate training configuration:

For Ultralytics YOLO

python scripts/vision_model_trainer.py data/coco/
--task detection
--arch yolov8m
--epochs 100
--batch 16
--imgsz 640
--output configs/

For Detectron2

python scripts/vision_model_trainer.py data/coco/
--task detection
--arch faster_rcnn_R_50_FPN
--framework detectron2
--output configs/

Step 5: Train and Validate

Ultralytics training

yolo detect train data=data.yaml model=yolov8m.pt epochs=100 imgsz=640

Detectron2 training

python train_net.py --config-file configs/faster_rcnn.yaml --num-gpus 1

Validate on test set

yolo detect val model=runs/detect/train/weights/best.pt data=data.yaml

Step 6: Evaluate Results

Key metrics to analyze:

Metric Target Description

mAP@50

0.7 Mean Average Precision at IoU 0.5

mAP@50:95

0.5 COCO primary metric

Precision

0.8 Low false positives

Recall

0.8 Low missed detections

Inference time <33ms For 30 FPS real-time

Workflow 2: Model Optimization and Deployment

Use this workflow when preparing a trained model for production deployment.

Step 1: Benchmark Baseline Performance

Measure current model performance

python scripts/inference_optimizer.py model.pt
--benchmark
--input-size 640 640
--batch-sizes 1 4 8 16
--warmup 10
--iterations 100

Expected output:

Baseline Performance (PyTorch FP32):

• Batch 1: 45.2ms (22.1 FPS)
• Batch 4: 89.4ms (44.7 FPS)
• Batch 8: 165.3ms (48.4 FPS)
• Memory: 2.1 GB
• Parameters: 25.9M

Step 2: Select Optimization Strategy

Deployment Target Optimization Path

NVIDIA GPU (cloud) PyTorch → ONNX → TensorRT FP16

NVIDIA GPU (edge) PyTorch → TensorRT INT8

Intel CPU PyTorch → ONNX → OpenVINO

Apple Silicon PyTorch → CoreML

Generic CPU PyTorch → ONNX Runtime

Mobile PyTorch → TFLite or ONNX Mobile

Step 3: Export to ONNX

Export with dynamic batch size

python scripts/inference_optimizer.py model.pt
--export onnx
--input-size 640 640
--dynamic-batch
--simplify
--output model.onnx

Verify ONNX model

python -c "import onnx; model = onnx.load('model.onnx'); onnx.checker.check_model(model); print('ONNX model valid')"

Step 4: Apply Quantization (Optional)

For INT8 quantization with calibration:

Generate calibration dataset

python scripts/inference_optimizer.py model.onnx
--quantize int8
--calibration-data data/calibration/
--calibration-samples 500
--output model_int8.onnx

Quantization impact analysis:

Precision Size Speed Accuracy Drop

FP32 100% 1x 0%

FP16 50% 1.5-2x <0.5%

INT8 25% 2-4x 1-3%

Step 5: Convert to Target Runtime

TensorRT (NVIDIA GPU)

trtexec --onnx=model.onnx --saveEngine=model.engine --fp16

OpenVINO (Intel)

mo --input_model model.onnx --output_dir openvino/

CoreML (Apple)

python -c "import coremltools as ct; model = ct.convert('model.onnx'); model.save('model.mlpackage')"

Step 6: Benchmark Optimized Model

python scripts/inference_optimizer.py model.engine
--benchmark
--runtime tensorrt
--compare model.pt

Expected speedup:

Optimization Results:

• Original (PyTorch FP32): 45.2ms
• Optimized (TensorRT FP16): 12.8ms
• Speedup: 3.5x
• Accuracy change: -0.3% mAP

Workflow 3: Custom Dataset Preparation

Use this workflow when preparing a computer vision dataset for training.

Step 1: Audit Raw Data

Analyze image dataset

python scripts/dataset_pipeline_builder.py data/raw/
--analyze
--output analysis/

Analysis report includes:

Dataset Analysis:

• Total images: 5,234
• Image sizes: 640x480 to 4096x3072 (variable)
• Formats: JPEG (4,891), PNG (343)
• Corrupted: 12 files
• Duplicates: 45 pairs

Annotation Analysis:

• Format detected: Pascal VOC XML
• Total annotations: 28,456
• Classes: 5 (car, person, bicycle, dog, cat)
• Distribution: car (12,340), person (8,234), bicycle (3,456), dog (2,890), cat (1,536)
• Empty images: 234

Step 2: Clean and Validate

Remove corrupted and duplicate images

python scripts/dataset_pipeline_builder.py data/raw/
--clean
--remove-corrupted
--remove-duplicates
--output data/cleaned/

Step 3: Convert Annotation Format

Convert VOC to COCO format

python scripts/dataset_pipeline_builder.py data/cleaned/
--annotations data/annotations/
--input-format voc
--output-format coco
--output data/coco/

Supported format conversions:

From To

Pascal VOC XML COCO JSON

YOLO TXT COCO JSON

COCO JSON YOLO TXT

LabelMe JSON COCO JSON

CVAT XML COCO JSON

Step 4: Apply Augmentations

Generate augmentation config

python scripts/dataset_pipeline_builder.py data/coco/
--augment
--aug-config configs/augmentation.yaml
--output data/augmented/

Recommended augmentations for detection:

configs/augmentation.yaml

augmentations: geometric: - horizontal_flip: { p: 0.5 } - vertical_flip: { p: 0.1 } # Only if orientation invariant - rotate: { limit: 15, p: 0.3 } - scale: { scale_limit: 0.2, p: 0.5 }

color: - brightness_contrast: { brightness_limit: 0.2, contrast_limit: 0.2, p: 0.5 } - hue_saturation: { hue_shift_limit: 20, sat_shift_limit: 30, p: 0.3 } - blur: { blur_limit: 3, p: 0.1 }

advanced: - mosaic: { p: 0.5 } # YOLO-style mosaic - mixup: { p: 0.1 } # Image mixing - cutout: { num_holes: 8, max_h_size: 32, max_w_size: 32, p: 0.3 }

Step 5: Create Train/Val/Test Splits

python scripts/dataset_pipeline_builder.py data/augmented/
--split 0.8 0.1 0.1
--stratify
--seed 42
--output data/final/

Split strategy guidelines:

Dataset Size Train Val Test

<1,000 images 70% 15% 15%

1,000-10,000 80% 10% 10%

10,000 90% 5% 5%

Step 6: Generate Dataset Configuration

For Ultralytics YOLO

python scripts/dataset_pipeline_builder.py data/final/
--generate-config yolo
--output data.yaml

For Detectron2

python scripts/dataset_pipeline_builder.py data/final/
--generate-config detectron2
--output detectron2_config.py

Architecture Selection Guide

Object Detection Architectures

Architecture Speed Accuracy Best For

YOLOv8n 1.2ms 37.3 mAP Edge, mobile, real-time

YOLOv8s 2.1ms 44.9 mAP Balanced speed/accuracy

YOLOv8m 4.2ms 50.2 mAP General purpose

YOLOv8l 6.8ms 52.9 mAP High accuracy

YOLOv8x 10.1ms 53.9 mAP Maximum accuracy

RT-DETR-L 5.3ms 53.0 mAP Transformer, no NMS

Faster R-CNN R50 46ms 40.2 mAP Two-stage, high quality

DINO-4scale 85ms 49.0 mAP SOTA transformer

Segmentation Architectures

Architecture Type Speed Best For

YOLOv8-seg Instance 4.5ms Real-time instance seg

Mask R-CNN Instance 67ms High-quality masks

SAM Promptable 50ms Zero-shot segmentation

DeepLabV3+ Semantic 25ms Scene parsing

SegFormer Semantic 15ms Efficient semantic seg

CNN vs Vision Transformer Trade-offs

Aspect CNN (YOLO, R-CNN) ViT (DETR, DINO)

Training data needed 1K-10K images 10K-100K+ images

Training time Fast Slow (needs more epochs)

Inference speed Faster Slower

Small objects Good with FPN Needs multi-scale

Global context Limited Excellent

Positional encoding Implicit Explicit

Reference Documentation

1. Computer Vision Architectures

See references/computer_vision_architectures.md for:

• CNN backbone architectures (ResNet, EfficientNet, ConvNeXt)

• Vision Transformer variants (ViT, DeiT, Swin)

• Detection heads (anchor-based vs anchor-free)

• Feature Pyramid Networks (FPN, BiFPN, PANet)

• Neck architectures for multi-scale detection

2. Object Detection Optimization

See references/object_detection_optimization.md for:

• Non-Maximum Suppression variants (NMS, Soft-NMS, DIoU-NMS)

• Anchor optimization and anchor-free alternatives

• Loss function design (focal loss, GIoU, CIoU, DIoU)

• Training strategies (warmup, cosine annealing, EMA)

• Data augmentation for detection (mosaic, mixup, copy-paste)

3. Production Vision Systems

See references/production_vision_systems.md for:

• ONNX export and optimization

• TensorRT deployment pipeline

• Batch inference optimization

• Edge device deployment (Jetson, Intel NCS)

• Model serving with Triton

• Video processing pipelines

Common Commands

Ultralytics YOLO

Training

yolo detect train data=coco.yaml model=yolov8m.pt epochs=100 imgsz=640

Validation

yolo detect val model=best.pt data=coco.yaml

Inference

yolo detect predict model=best.pt source=images/ save=True

Export

yolo export model=best.pt format=onnx simplify=True dynamic=True

Detectron2

Training

python train_net.py --config-file configs/COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml
--num-gpus 1 OUTPUT_DIR ./output

Evaluation

python train_net.py --config-file configs/faster_rcnn.yaml --eval-only
MODEL.WEIGHTS output/model_final.pth

Inference

python demo.py --config-file configs/faster_rcnn.yaml
--input images/*.jpg --output results/
--opts MODEL.WEIGHTS output/model_final.pth

MMDetection

Training

python tools/train.py configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py

Testing

python tools/test.py configs/faster_rcnn.py checkpoints/latest.pth --eval bbox

Inference

python demo/image_demo.py demo.jpg configs/faster_rcnn.py checkpoints/latest.pth

Model Optimization

ONNX export and simplify

python -c "import torch; model = torch.load('model.pt'); torch.onnx.export(model, torch.randn(1,3,640,640), 'model.onnx', opset_version=17)" python -m onnxsim model.onnx model_sim.onnx

TensorRT conversion

trtexec --onnx=model.onnx --saveEngine=model.engine --fp16 --workspace=4096

Benchmark

trtexec --loadEngine=model.engine --batch=1 --iterations=1000 --avgRuns=100

Performance Targets

Metric Real-time High Accuracy Edge

FPS

30 10 15

mAP@50

0.6 0.8 0.5

Latency P99 <50ms <150ms <100ms

GPU Memory <4GB <8GB <2GB

Model Size <50MB <200MB <20MB

Resources

• Architecture Guide: references/computer_vision_architectures.md

• Optimization Guide: references/object_detection_optimization.md

• Deployment Guide: references/production_vision_systems.md

• Scripts: scripts/ directory for automation tools