Portfolio Optimization

Overview

This skill provides guidance for implementing high-performance portfolio optimization algorithms using Python C extensions. It covers the workflow for creating C extensions that interface with NumPy arrays, proper verification strategies, and common pitfalls to avoid when optimizing numerical computations.

When to Apply This Skill

Apply this skill when:

• Implementing portfolio risk calculations (variance, volatility, Sharpe ratio)

• Optimizing matrix-vector operations for large asset portfolios

• Creating C extensions for Python numerical code

• Performance requirements specify speedup ratios (e.g., >= 1.2x)

• Working with covariance matrices and portfolio weights

Recommended Workflow

Phase 1: Codebase Understanding

Before writing any code:

• Read all relevant source files completely - Understand the baseline implementation, data structures, and expected interfaces

• Identify the mathematical operations - Common operations include:

• Matrix-vector multiplication (covariance matrix times weights)

• Dot products (weights times returns)

• Square root operations (for volatility from variance)

• Understand the test suite - Know what correctness tolerances are expected (e.g., 1e-10) and what performance benchmarks must be met

• Document the input/output contracts - Array shapes, data types (typically float64), and return value specifications

Phase 2: Implementation Planning

Consider these factors before implementation:

Why C provides speedup:

• Eliminates Python interpreter overhead

• Enables direct memory access without bounds checking

• Allows compiler optimizations (vectorization, loop unrolling)

• Reduces temporary array allocations

Design decisions to make:

• Whether to use NumPy C API for zero-copy array access

• Memory layout assumptions (C-contiguous vs Fortran-contiguous)

• Error handling strategy for type mismatches and dimension errors

Potential algorithmic optimizations:

• Cache-friendly memory access patterns (row-major iteration for C arrays)

• SIMD vectorization opportunities

• Minimizing Python-to-C data conversion overhead

Phase 3: C Extension Implementation

When implementing the C extension:

Include proper headers:

• Python.h (must be first)

• numpy/arrayobject.h for NumPy array access

Initialize NumPy in the module init function:

• Call import_array() to initialize NumPy C API

Use NumPy C API for array access:

• PyArray_DATA() for getting data pointer

• PyArray_DIM() for dimensions

• PyArray_STRIDE() for memory strides

• Check PyArray_IS_C_CONTIGUOUS() for memory layout

Implement robust error handling:

• Validate array dimensions match expected shapes

• Check data types (expect NPY_FLOAT64 for double precision)

• Handle non-contiguous arrays (either reject or handle strides)

• Set appropriate Python exceptions on error

Phase 4: Python Wrapper Implementation

Create a Python module that:

• Imports the C extension module

• Provides a clean interface matching the baseline API

• Handles any necessary array preparation (ensuring contiguity)

• Documents the interface clearly

Phase 5: Verification Strategy

Critical: Verify every change completely

After editing files, re-read them - Confirm edits were applied correctly, especially for multi-line changes

Test incrementally:

• Build the C extension first and verify it compiles

• Test individual functions before running full benchmarks

• Use small test cases for correctness verification before scaling up

Correctness verification:

• Compare outputs against baseline implementation

• Use appropriate numerical tolerances (typically 1e-10 for double precision)

• Test with known inputs where expected outputs can be calculated manually

Performance verification:

• Run benchmarks with representative data sizes

• Verify speedup meets requirements across different portfolio sizes

• Test edge cases: small portfolios (n=1, n=10), large portfolios (n=5000+)

Edge Cases to Handle

Ensure the implementation addresses:

• Empty portfolios (n=0) - Return appropriate default or error

• Single-asset portfolios (n=1) - Degenerate case for covariance

• Dimension mismatches - Weights vector length vs covariance matrix dimensions

• Invalid inputs:

• Non-square covariance matrices

• NaN or infinity values in inputs

• Negative variance (mathematically invalid)

• Memory considerations:

• Non-contiguous NumPy arrays

• Memory allocation failures in C code

• Large portfolios that may stress memory

Common Pitfalls to Avoid

Code Completeness

• Never truncate code in edit operations - always provide complete implementations

• Verify file contents after editing to confirm changes applied correctly

• Document all design choices explicitly

Testing Approach

• Avoid going directly from implementation to full benchmark testing

• Test each function individually before integration testing

• Do not rely solely on "tests pass" for validation - understand why they pass

C Extension Specific

• Always check NumPy array types before accessing data

• Handle reference counting properly to avoid memory leaks

• Initialize NumPy API with import_array() in module init

• Use PyErr_SetString() to set exceptions on errors

Performance Validation

• Verify speedup is consistent across different input sizes

• Profile if further optimizations might be needed

• Consider the overhead of Python-to-C transitions for small inputs

Build and Test Commands

Typical workflow commands:

Build the C extension

python setup.py build_ext --inplace

Run correctness tests

python -c "from portfolio_optimized import *; # test calls"

Run benchmark

python benchmark.py

Run full test suite

pytest test_portfolio.py -v

Verification Checklist

Before considering the task complete:

• All source files read and understood

• C extension compiles without warnings

• Individual functions tested for correctness

• Numerical results match baseline within tolerance

• Performance meets speedup requirements

• Edge cases explicitly tested or handled

• Error handling implemented for invalid inputs

• File contents verified after all edits

• No memory leaks in C code (proper reference counting)