GrepAI Storage with GOB

This skill covers using GOB (Go Binary) as the storage backend for GrepAI, the default and simplest option.

When to Use This Skill

• Single developer projects

• Small to medium codebases

• Simple setup without external dependencies

• Local development environments

What is GOB Storage?

GOB is Go's native binary serialization format. GrepAI uses it to store:

• Vector embeddings

• File metadata

• Chunk information

Everything is stored in a single local file.

Advantages

Benefit Description

🚀 Simple No external services needed

⚡ Fast setup Works immediately

📁 Portable Single file, easy to backup

💰 Free No infrastructure costs

🔒 Private Data stays local

Limitations

Limitation Description

📏 Scalability Not ideal for very large codebases

👤 Single user No concurrent access

🔄 No sharing Can't share index across machines

💾 Memory Loads into RAM for searches

Configuration

Default Configuration

GOB is the default backend. Minimal config:

.grepai/config.yaml

store: backend: gob

Explicit Configuration

store: backend: gob

Index stored in .grepai/index.gob (automatic)

Storage Location

GOB storage creates files in your project's .grepai/ directory:

.grepai/ ├── config.yaml # Configuration ├── index.gob # Vector embeddings └── symbols.gob # Symbol index for trace

File Sizes

Approximate .grepai/index.gob sizes:

Codebase Files Chunks Index Size

Small 100 500 ~5 MB

Medium 1,000 5,000 ~50 MB

Large 10,000 50,000 ~500 MB

Operations

Creating the Index

Initialize project

grepai init

Start indexing (creates index.gob)

grepai watch

Checking Index Status

grepai status

Output:

Index: .grepai/index.gob

Files: 245

Chunks: 1,234

Size: 12.5 MB

Last updated: 2025-01-28 10:30:00

Backing Up the Index

Simple file copy

cp .grepai/index.gob .grepai/index.gob.backup

Clearing the Index

Delete and re-index

rm .grepai/index.gob grepai watch

Moving to a New Machine

Copy entire .grepai directory

cp -r .grepai /path/to/new/location/

Note: Only works if using same embedding model

Performance Considerations

Memory Usage

GOB loads the entire index into RAM for searches:

Index Size RAM Usage

10 MB ~20 MB

50 MB ~100 MB

500 MB ~1 GB

Search Speed

GOB provides fast searches for typical codebases:

Codebase Size Search Time

Small (100 files) <50ms

Medium (1K files) <200ms

Large (10K files) <1s

When to Upgrade

Consider PostgreSQL or Qdrant when:

• Index exceeds 1 GB

• Need concurrent access

• Want to share index across team

• Codebase has 50K+ files

.gitignore Configuration

Add .grepai/ to your .gitignore :

GrepAI (machine-specific index)

.grepai/

Why: The index is machine-specific because:

• Contains binary embeddings

• Tied to the embedding model used

• Each machine should generate its own

Sharing Index (Not Recommended)

While you can copy the index file, it's not recommended because:

• Must use identical embedding model

• File paths are absolute

• Different machines may have different code versions

Better approach: Each developer runs their own grepai watch .

Migrating to Other Backends

To PostgreSQL

• Update config:

store: backend: postgres postgres: dsn: postgres://user:pass@localhost:5432/grepai

• Re-index:

rm .grepai/index.gob grepai watch

To Qdrant

• Update config:

store: backend: qdrant qdrant: endpoint: localhost port: 6334

• Re-index:

rm .grepai/index.gob grepai watch

Common Issues

❌ Problem: Index file too large ✅ Solution: Add more ignore patterns or migrate to PostgreSQL/Qdrant

❌ Problem: Slow searches on large codebase ✅ Solution: Migrate to Qdrant for better performance

❌ Problem: Corrupted index ✅ Solution: Delete and re-index:

rm .grepai/index.gob .grepai/symbols.gob grepai watch

❌ Problem: "Index not found" error ✅ Solution: Run grepai watch to create the index

Best Practices

• Use for small/medium projects: Up to ~10K files

• Add to .gitignore: Don't commit the index

• Backup before major changes: Copy index.gob before experiments

• Re-index after model changes: If you change embedding models

• Monitor file size: Migrate if index exceeds 1GB

Output Format

GOB storage status:

✅ GOB Storage Configured

Backend: GOB (local file) Index: .grepai/index.gob Size: 12.5 MB

Contents:

• Files: 245
• Chunks: 1,234
• Vectors: 1,234 × 768 dimensions

Performance:

• Search latency: <100ms
• Memory usage: ~25 MB