dataknobs-config¶

A modular configuration system with composable settings, environment variable overrides, and cross-references.

Overview¶

The dataknobs-config package provides a flexible and powerful configuration management system designed for complex applications. It supports:

Modular configurations organized by type
Atomic configuration units with validation
Cross-references between configurations
Environment variable overrides with bash-compatible naming
Path resolution for relative paths
Object construction from configurations
Global and type-specific settings
Environment-aware configuration for portable configs across dev/staging/prod
Late-binding resolution of environment variables at instantiation time
Logical resource references that map to environment-specific implementations

Installation¶

pip install dataknobs-config

Quick Start¶

from dataknobs_config import Config

# Load configuration from a YAML file
config = Config("config.yaml")

# Get a specific configuration
db_config = config.get("database", "production")

# Get all configurations of a type
all_databases = config.get_all("database")

# Use with environment overrides
# export DATAKNOBS_DATABASE__PRODUCTION__HOST=new-host.com
config = Config("config.yaml", use_env=True)

Key Features¶

1. Modular Design¶

Organize configurations by type, with each type containing multiple atomic configurations:

database:
  - name: primary
    host: localhost
    port: 5432
  - name: replica
    host: localhost
    port: 5433

cache:
  - name: redis
    host: localhost
    port: 6379

2. Cross-References¶

Link configurations using the xref: syntax:

app:
  - name: web
    database: "xref:database[primary]"
    cache: "xref:cache[redis]"

3. Environment Overrides¶

Override any configuration value using environment variables:

export DATAKNOBS_DATABASE__PRIMARY__HOST=prod.db.com
export DATAKNOBS_CACHE__REDIS__PORT=6380

4. Path Resolution¶

Automatically resolve relative paths based on global or type-specific roots:

settings:
  global_root: /etc/myapp
  path_resolution_attributes: ["config_file", "data_dir"]

service:
  - name: api
    config_file: configs/api.yaml  # Resolved to /etc/myapp/configs/api.yaml

5. Object Construction¶

Build objects directly from configurations:

logger:
  - name: main
    class: logging.Logger
    level: INFO

logger = config.build_object("xref:logger[main]")

6. Environment-Aware Configuration¶

Deploy the same app config across different environments with logical resource references:

# Portable app config
bot:
  database:
    $resource: conversations
    type: databases

from dataknobs_config import EnvironmentAwareConfig

# Auto-detects environment (dev/staging/prod)
config = EnvironmentAwareConfig.load_app("my-bot")

# Late-bind resources at instantiation time
resolved = config.resolve_for_build()

See the Environment-Aware Configuration Guide for details.

Configuration File Format¶

Configuration files can be in YAML or JSON format:

# config.yaml
settings:
  global_root: /app
  path_resolution_attributes: ["path", "dir"]

database:
  - name: main
    host: localhost
    port: 5432

cache:
  - name: redis
    host: localhost
    port: 6379

app:
  - name: web
    database: "xref:database[main]"
    cache: "xref:cache[redis]"

API Overview¶

Config Class¶

The main configuration management class:

Config(*sources, use_env=True) - Initialize with one or more sources
get(type_name, name_or_index) - Get a specific configuration
get_all(type_name) - Get all configurations of a type
set(type_name, name_or_index, value) - Set a configuration value
resolve_reference(ref) - Resolve a string reference
build_object(ref) - Build an object from configuration
to_dict() - Export as dictionary
to_file(path) - Save to file

Examples¶

Basic Usage¶

from dataknobs_config import Config

# Load from file
config = Config("config.yaml")

# Access configurations
db = config.get("database", "primary")
print(f"Connecting to {db['host']}:{db['port']}")

# Update configuration
config.set("database", "primary", {"host": "new-host.com", "port": 5432})

# Save changes
config.to_file("updated_config.yaml")

Using References¶

config = Config({
    "database": [{"name": "main", "host": "localhost"}],
    "app": [{"name": "web", "db": "xref:database[main]"}]
})

app_config = config.get("app", "web")
# app_config["db"] will contain the resolved database configuration

Environment Overrides¶

import os
os.environ["DATAKNOBS_DATABASE__MAIN__HOST"] = "prod.db.com"

config = Config("config.yaml", use_env=True)
db = config.get("database", "main")
print(db["host"])  # "prod.db.com"

Best Practices¶

Use meaningful type names that reflect the configuration's purpose
Keep atomic configurations small and focused on a single concern
Use references to avoid duplication and maintain consistency
Document your settings section for team members
Use environment overrides for deployment-specific values
Validate configurations early in your application startup