Feature Engineer

Overview

Feature engineering often makes the difference between mediocre and excellent ML models. This skill transforms raw data into model-ready features through systematic data quality assessment, feature creation, selection, and transformation—all integrated with SpecWeave's increment workflow.

The Feature Engineering Pipeline

Phase 1: Data Quality Assessment

Before creating features, understand your data:

from specweave import DataQualityReport

Automated data quality check

report = DataQualityReport(df, increment="0042")

Generates:

- Missing value analysis

- Outlier detection

- Data type validation

- Distribution analysis

- Correlation matrix

- Duplicate detection

Quality Report Output:

Data Quality Report

Dataset Overview

• Rows: 100,000
• Columns: 45
• Memory: 34.2 MB

Missing Values

Outliers Detected

• transaction_amount: 234 outliers (>3 std dev)
• user_age: 12 outliers (<18 or >100)

Data Type Issues

• user_id: Stored as float, should be int
• date_joined: Stored as string, should be datetime

Recommendations

1. Impute email/phone or create "missing" indicator features
2. Cap/remove outliers in transaction_amount
3. Convert data types for efficiency

Phase 2: Feature Creation

Create features from domain knowledge:

from specweave import FeatureCreator

creator = FeatureCreator(df, increment="0042")

Temporal features (from datetime)

creator.add_temporal_features( date_column="purchase_date", features=["hour", "day_of_week", "month", "is_weekend", "is_holiday"] )

Aggregation features (user behavior)

creator.add_aggregation_features( group_by="user_id", target="purchase_amount", aggs=["mean", "std", "count", "min", "max"] )

Creates: user_purchase_amount_mean, user_purchase_amount_std, etc.

Interaction features

creator.add_interaction_features( features=[("age", "income"), ("clicks", "impressions")], operations=["multiply", "divide", "subtract"] )

Creates: age_x_income, clicks_per_impression, etc.

Ratio features

creator.add_ratio_features([ ("revenue", "cost"), ("conversions", "visits") ])

Creates: revenue_to_cost_ratio, conversion_rate

Binning (discretization)

creator.add_binned_features( column="age", bins=[0, 18, 25, 35, 50, 65, 100], labels=["child", "young_adult", "adult", "middle_aged", "senior", "elderly"] )

Text features (from text columns)

creator.add_text_features( column="product_description", features=["length", "word_count", "unique_words", "sentiment"] )

Generate all features

df_enriched = creator.generate()

Auto-documents in increment folder

creator.save_feature_definitions( path=".specweave/increments/0042.../features/feature_definitions.yaml" )

Feature Definitions (auto-generated):

.specweave/increments/0042.../features/feature_definitions.yaml

features:

• name: purchase_hour type: temporal source: purchase_date description: Hour of purchase (0-23)

• name: user_purchase_amount_mean type: aggregation source: purchase_amount group_by: user_id description: Average purchase amount per user

• name: age_x_income type: interaction sources: [age, income] operation: multiply description: Product of age and income

• name: conversion_rate type: ratio sources: [conversions, visits] description: Conversion rate (conversions / visits)

Phase 3: Feature Selection

Reduce dimensionality, improve performance:

from specweave import FeatureSelector

selector = FeatureSelector(X_train, y_train, increment="0042")

Method 1: Correlation-based (remove redundant features)

selector.remove_correlated_features(threshold=0.95)

Removes features with >95% correlation

Method 2: Variance-based (remove constant features)

selector.remove_low_variance_features(threshold=0.01)

Removes features with <1% variance

Method 3: Statistical tests

selector.select_by_statistical_test(k=50)

SelectKBest with chi2/f_classif

Method 4: Model-based (tree importance)

selector.select_by_model_importance( model=RandomForestClassifier(), threshold=0.01 )

Removes features with <1% importance

Method 5: Recursive Feature Elimination

selector.select_by_rfe( model=LogisticRegression(), n_features=30 )

Get selected features

selected_features = selector.get_selected_features()

Generate selection report

selector.generate_report()

Feature Selection Report:

Feature Selection Report

Original Features: 125

Selected Features: 35 (72% reduction)

Selection Process

1. Removed 12 correlated features (>95% correlation)
2. Removed 8 low-variance features
3. Statistical test: Selected top 50 (chi-squared)
4. Model importance: Removed 15 low-importance features (<1%)

Top 10 Features (by importance)

1. user_purchase_amount_mean (0.18)
2. days_since_last_purchase (0.12)
3. total_purchases (0.10)
4. age_x_income (0.08)
5. conversion_rate (0.07) ...

Removed Features

• user_id_hash (constant)
• temp_feature_1 (99% correlated with temp_feature_2)
• random_noise (0% importance) ...

Phase 4: Feature Transformation

Scale, normalize, encode for model compatibility:

from specweave import FeatureTransformer

transformer = FeatureTransformer(increment="0042")

Numerical transformations

transformer.add_numerical_transformer( columns=["age", "income", "purchase_amount"], method="standard_scaler" # Or: min_max, robust, quantile )

Categorical encoding

transformer.add_categorical_encoder( columns=["country", "device_type", "product_category"], method="onehot", # Or: label, target, binary handle_unknown="ignore" )

Ordinal encoding (for ordered categories)

transformer.add_ordinal_encoder( column="education", order=["high_school", "bachelors", "masters", "phd"] )

Log transformation (for skewed distributions)

transformer.add_log_transform( columns=["transaction_amount", "page_views"], method="log1p" # log(1 + x) to handle zeros )

Box-Cox transformation (for normalization)

transformer.add_power_transform( columns=["revenue", "engagement_score"], method="box-cox" )

Custom transformation

def clip_outliers(x): return np.clip(x, x.quantile(0.01), x.quantile(0.99))

transformer.add_custom_transformer( columns=["outlier_prone_feature"], func=clip_outliers )

Fit and transform

X_train_transformed = transformer.fit_transform(X_train) X_test_transformed = transformer.transform(X_test)

Save transformer pipeline

transformer.save( path=".specweave/increments/0042.../features/transformer.pkl" )

Phase 5: Feature Validation

Ensure features are production-ready:

from specweave import FeatureValidator

validator = FeatureValidator( X_train, X_test, increment="0042" )

Check for data leakage

leakage_report = validator.check_data_leakage()

Detects: perfectly correlated features, future data in training

Check for distribution drift

drift_report = validator.check_distribution_drift()

Compares train vs test distributions

Check for missing values after transformation

missing_report = validator.check_missing_values()

Check for infinite/NaN values

invalid_report = validator.check_invalid_values()

Generate validation report

validator.generate_report()

Validation Report:

Feature Validation Report

Data Leakage: ✅ PASS

No perfect correlations detected between train and test.

Distribution Drift: ⚠️ WARNING

Features with significant drift (KS test p < 0.05):

• user_age: p=0.023 (minor drift)
• device_type: p=0.001 (major drift)

Recommendation: Check if test data is from different time period.

Missing Values: ✅ PASS

No missing values after transformation.

Invalid Values: ✅ PASS

No infinite or NaN values detected.

Overall: READY FOR TRAINING

2 warnings, 0 critical issues.

Integration with SpecWeave

Automatic Feature Documentation

All feature engineering steps logged to increment

with track_experiment("feature-engineering-v1", increment="0042") as exp: # Create features df_enriched = creator.generate()

# Select features
selected = selector.select()

# Transform features
X_transformed = transformer.fit_transform(X)

# Validate
validation = validator.validate()

# Auto-logs:
exp.log_param("original_features", 125)
exp.log_param("created_features", 45)
exp.log_param("selected_features", 35)
exp.log_metric("feature_reduction", 0.72)
exp.save_artifact("feature_definitions.yaml")
exp.save_artifact("transformer.pkl")
exp.save_artifact("validation_report.md")

Living Docs Integration

After completing feature engineering:

/sw:sync-docs update

Updates:

<!-- .specweave/docs/internal/architecture/feature-engineering.md -->

Recommendation Model Features (Increment 0042)

Feature Engineering Pipeline

1. Data Quality: 100K rows, 45 columns
2. Created: 45 new features (temporal, aggregation, interaction)
3. Selected: 35 features (72% reduction via importance + RFE)
4. Transformed: StandardScaler for numerical, OneHot for categorical

Key Features

• user_purchase_amount_mean: Average user spend (top feature, 18% importance)
• days_since_last_purchase: Recency indicator (12% importance)
• age_x_income: Interaction feature (8% importance)

Feature Store

All features documented in: .specweave/increments/0042.../features/

• feature_definitions.yaml: Feature catalog
• transformer.pkl: Production transformation pipeline
• validation_report.md: Quality checks

Best Practices

1. Document Feature Rationale

Bad: Create features without explanation

df["feature_1"] = df["col_a"] * df["col_b"]

Good: Document why features were created

creator.add_interaction_feature( sources=["age", "income"], operation="multiply", rationale="High-income older users have different behavior patterns" )

2. Handle Missing Values Systematically

Options for missing values:

1. Imputation (mean, median, mode)

creator.impute_missing(column="age", strategy="median")

2. Indicator features (flag missing as signal)

creator.add_missing_indicator(column="email")

Creates: email_missing (0/1)

3. Forward/backward fill (for time series)

creator.fill_missing(column="sensor_reading", method="ffill")

4. Model-based imputation

creator.impute_with_model(column="income", model=RandomForestRegressor())

3. Avoid Data Leakage

❌ WRONG: Fit on all data (includes test set!)

scaler.fit(X) X_train = scaler.transform(X_train) X_test = scaler.transform(X_test)

✅ CORRECT: Fit only on train, transform both

scaler.fit(X_train) X_train = scaler.transform(X_train) X_test = scaler.transform(X_test)

SpecWeave's transformer enforces this pattern

transformer.fit_transform(X_train) # Fits transformer.transform(X_test) # Only transforms

4. Version Feature Engineering Pipeline

Version features with increment

transformer.save( path=".specweave/increments/0042.../features/transformer-v1.pkl", metadata={ "version": "v1", "features": selected_features, "transformations": ["standard_scaler", "onehot"] } )

Load specific version for reproducibility

transformer_v1 = FeatureTransformer.load( ".specweave/increments/0042.../features/transformer-v1.pkl" )

5. Test Feature Engineering on New Data

Before deploying, test on held-out data

X_production_sample = load_production_data()

try: X_transformed = transformer.transform(X_production_sample) except Exception as e: raise FeatureEngineeringError(f"Failed on production data: {e}")

Check for unexpected values

validator = FeatureValidator(X_train, X_production_sample) validation_report = validator.validate()

if validation_report["status"] == "CRITICAL": raise FeatureEngineeringError("Feature engineering failed validation")

Common Feature Engineering Patterns

Pattern 1: RFM (Recency, Frequency, Monetary)

For e-commerce / customer analytics

creator.add_rfm_features( user_id="user_id", transaction_date="purchase_date", transaction_amount="purchase_amount" )

Creates:

- recency: days since last purchase

- frequency: total purchases

- monetary: total spend

Pattern 2: Rolling Window Aggregations

For time series

creator.add_rolling_features( column="daily_sales", windows=[7, 14, 30], aggs=["mean", "std", "min", "max"] )

Creates: daily_sales_7day_mean, daily_sales_7day_std, etc.

Pattern 3: Target Encoding (Categorical → Numerical)

Encode categorical as target mean (careful: can leak!)

creator.add_target_encoding( column="product_category", target="purchase_amount", cv_folds=5 # Cross-validation to prevent leakage )

Creates: product_category_target_encoded

Pattern 4: Polynomial Features

For non-linear relationships

creator.add_polynomial_features( columns=["age", "income"], degree=2, interaction_only=True )

Creates: age^2, income^2, age*income

Commands

Generate feature engineering pipeline for increment

/ml:engineer-features 0042

Validate features before training

/ml:validate-features 0042

Generate feature importance report

/ml:feature-importance 0042

Integration with Other Skills

• ml-pipeline-orchestrator: Task 2 is "Feature Engineering" (uses this skill)

• experiment-tracker: Logs all feature engineering experiments

• model-evaluator: Uses feature importance from models

• ml-deployment-helper: Packages feature transformer for production

Summary

Feature engineering is 70% of ML success. This skill ensures:

• ✅ Systematic approach (quality → create → select → transform → validate)

• ✅ No data leakage (train/test separation enforced)

• ✅ Production-ready (versioned, validated, documented)

• ✅ Reproducible (all steps tracked in increment)

• ✅ Traceable (feature definitions in living docs)

Good features make mediocre models great. Great features make mediocre models excellent.