Keboola Configuration Knowledge

Provide expertise on Keboola project structure, configuration formats, and best practices for managing data pipelines.

Project Structure

A Keboola project pulled locally has this structure:

project-root/ ├── .keboola/ │ └── manifest.json # Project metadata and branch info ├── .env.local # API token (never commit) ├── .env.dist # Template for .env.local ├── .gitignore └── [branch-name]/ # One directory per branch └── [component-id]/ # e.g., keboola.snowflake-transformation └── [config-name]/ # Configuration directory ├── config.json # Main configuration ├── meta.json # Metadata (name, description) └── rows/ # Configuration rows (if applicable)

Configuration Files

manifest.json

Located in .keboola/manifest.json , contains:

• Project ID and API host

• Branch information

• Sorting and naming conventions

config.json

The main configuration file for each component. Structure varies by component type but typically includes:

• parameters

• Component-specific settings

• storage

• Input/output table mappings

• processors

• Pre/post processing steps

meta.json

Metadata about the configuration:

{ "name": "Configuration Name", "description": "What this configuration does", "isDisabled": false }

Component Types

Transformations

SQL or Python/R transformations for data processing.

Snowflake Transformation (keboola.snowflake-transformation ):

{ "parameters": { "blocks": [ { "name": "Block Name", "codes": [ { "name": "Script Name", "script": ["SELECT * FROM table"] } ] } ] }, "storage": { "input": { "tables": [ { "source": "in.c-bucket.table", "destination": "table" } ] }, "output": { "tables": [ { "source": "output_table", "destination": "out.c-bucket.result" } ] } } }

Extractors

Components that pull data from external sources (databases, APIs, files).

Common extractors:

• keboola.ex-db-snowflake

• Snowflake extractor

• keboola.ex-google-analytics-v4

• Google Analytics

• keboola.ex-generic-v2

• Generic HTTP API extractor

Writers

Components that push data to external destinations.

Common writers:

• keboola.wr-db-snowflake

• Snowflake writer

• keboola.wr-google-sheets

• Google Sheets writer

Orchestrations

Workflow definitions that run multiple configurations in sequence.

Located in keboola.orchestrator/ with:

• Task definitions

• Dependencies

• Scheduling

Best Practices

When Editing Configurations

• Always run kbc diff before and after changes

• Validate JSON syntax before pushing

• Use kbc validate to check configuration validity

• Keep descriptions updated in meta.json

Storage Mappings

• Input tables: Map source tables to working names

• Output tables: Map result tables to destination buckets

• Use consistent naming conventions

Transformations

• Break complex logic into multiple blocks

• Use meaningful names for blocks and scripts

• Document SQL with comments

• Test locally when possible

Common Tasks

Add a New Input Table to Transformation

In config.json , add to storage.input.tables :

{ "source": "in.c-bucket.new_table", "destination": "new_table", "columns": [] // Empty = all columns }

Add Output Table

In config.json , add to storage.output.tables :

{ "source": "result_table", "destination": "out.c-bucket.result", "primary_key": ["id"] }

Modify SQL Script

Edit the script array in the relevant block/code section. Each array element is a SQL statement.

Troubleshooting

Invalid Configuration

• Check JSON syntax (missing commas, brackets)

• Verify table names exist in storage

• Check column names in mappings

Push Conflicts

• Pull latest changes first

• Merge conflicts manually in config files

• Push again after resolution

Missing Tables

• Ensure input tables exist in Keboola Storage

• Check bucket permissions

• Verify table names match exactly (case-sensitive)