Guides

​Schema Migration Guide

Outpost stores tenant, destination, event, and delivery data using configurable storage backends. You can choose from various options including Redis, Postgres, ClickHouse, and more, depending on your infrastructure and requirements.

When upgrading Outpost, we handle data migrations automatically during startup whenever possible. However, some migrations require manual intervention due to their complexity or impact. For these cases, we provide the outpost migrate tool.

​Migration Tool

The migration tool is delivered as part of the outpost CLI. Given that Outpost is primarily distributed as a Docker image, running the migration tool via Docker is the easiest approach. This guide uses Docker for all examples.

Ensure you're using the correct version by specifying the Docker tag:

# Use a specific version tag
docker run --rm hookdeck/outpost:v1.2.3 --version

# Or use latest (verify the version before running migrations)
docker run --rm hookdeck/outpost:latest --version
bash

​Migration Workflow

When Outpost starts up, it can automatically check and run migrations using migrate init --current --log-format=json. However, for production environments with critical data, manual migration using the steps below is recommended.

​Execute a Migration

​Step 1: Plan the Migration

Review what changes will be made without applying them:

docker run --rm hookdeck/outpost migrate plan
bash

​Step 2: Apply the Migration

Execute the migration (creates new structures, preserves old ones):

# Apply the next pending migration
docker run --rm -it hookdeck/outpost migrate apply

# Or apply all pending migrations in sequence
docker run --rm -it hookdeck/outpost migrate apply --all
bash

The --all flag runs each pending migration in version order, automatically verifying each one before proceeding to the next. If any migration fails verification, the process stops immediately.

​Step 3: Verify the Migration

After applying the migration, verify that data was migrated correctly:

docker run --rm hookdeck/outpost migrate verify
bash

This performs spot-checks on a sample of migrated data to ensure integrity. Additionally, test your application to ensure it works correctly with the migrated data:

• Run your test suite
• Verify critical workflows
• Monitor for any errors

​Step 4: Cleanup Old Data

Once verified, remove the old data structures:

docker run --rm -it hookdeck/outpost migrate cleanup
bash

​Safety Features

​Migration Locks

Migrations use distributed locks to prevent concurrent execution:

• Only one migration can run at a time across all instances
• Locks expire after 1 hour to prevent deadlocks
• Force-unlock available if a migration process crashes using:

docker run --rm -it hookdeck/outpost migrate unlock --yes
bash

​Non-Destructive Application

The Apply phase creates new data structures without removing old ones:

• Allows rollback by pointing application back to old structures
• Provides time for thorough testing
• Cleanup is an explicit separate step

​Automatic Verification

Cleanup operations require successful verification by default:

• Prevents accidental data loss
• Can be overridden with --force flag (use with caution)

​Running in Private Environments

Redis is often deployed in private networks (VPCs, internal subnets) that aren't accessible from your local machine. The migration tool needs direct network access to Redis, so you'll need to run it from within the same network.

​Common Approaches

Run from an existing host in the network:

If you have SSH access to a host that can reach Redis (e.g., your application server), you can run the migration there:

# SSH into a host with Redis access
ssh your-server

# Run migration using Docker
docker run --rm -it \
  -e REDIS_HOST=redis.internal \
  -e REDIS_PASSWORD=... \
  hookdeck/outpost:v0.12.0 migrate apply --all
bash

Kubernetes:

Run the migration as a one-off pod in the same namespace/cluster:

kubectl run outpost-migrate --rm -it --restart=Never \
  --image=hookdeck/outpost:v0.12.0 \
  --env="REDIS_HOST=redis.internal" \
  --env="REDIS_PASSWORD=..." \
  -- migrate apply --all
bash

Port forwarding:

If direct access isn't possible, you can forward the Redis port to your local machine:

# Example: SSH tunnel
ssh -L 6379:redis.internal:6379 bastion-host

# Then run migration locally
docker run --rm -it \
  -e REDIS_HOST=host.docker.internal \
  -e REDIS_PORT=6379 \
  hookdeck/outpost:v0.12.0 migrate apply --all
bash

​Configuration

The outpost CLI tool uses the same configuration as Outpost itself. Configuration can be provided through environment variables, a configuration file (using --config flag), or CLI flags (e.g., --redis-host, --redis-password).

For example, common Redis configurations:

# Environment variables
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=your-password
REDIS_DATABASE=0
REDIS_CLUSTER_ENABLED=true
bash

# Configuration file
redis:
  host: localhost
  port: 6379
  password: your-password
  cluster_enabled: false
yaml

For a complete list of configuration options, see the Configuration Reference.

​Troubleshooting

​Migration Locked

If a migration fails and leaves a lock:

# View lock status
docker run --rm hookdeck/outpost migrate status

# Force unlock (ensure no migration is actually running)
docker run --rm -it hookdeck/outpost migrate unlock --yes
bash

​Failed Migration

If a migration fails during application:

1. Check the error logs for specific issues
2. The migration can be safely retried (idempotent)
3. Partial progress is preserved between attempts

​Rollback Before Cleanup

If issues are discovered after applying but before cleanup:

1. Stop using the new Outpost version
2. Deploy the previous version (will use old data structures)
3. Debug and resolve issues before retrying

---