Operations Guide
Best practices for running, maintaining, and troubleshooting BobaMixer in production environments.
Production Installation
System Requirements
- OS: Linux (recommended), macOS, Windows (WSL)
- Go: 1.22+ (if building from source)
- SQLite: 3.x
- Disk Space: 50MB + database growth (estimate ~1KB per API call)
- Memory: 50MB typical, 200MB peak
- Git: Optional, for hooks integration
Installation Methods
Using Homebrew (Recommended)
bash
brew tap royisme/tap
brew install bobamixer
# Verify installation
boba version
boba doctorManual Installation
bash
# Download latest release
wget https://github.com/royisme/BobaMixer/releases/latest/download/bobamixer_Linux_x86_64.tar.gz
# Extract
tar -xzf bobamixer_Linux_x86_64.tar.gz
# Install
sudo mv boba /usr/local/bin/
sudo chmod +x /usr/local/bin/boba
# Verify
boba versionFrom Source
bash
git clone https://github.com/royisme/BobaMixer.git
cd BobaMixer
make build
sudo make install
# Or
go install github.com/royisme/bobamixer/cmd/boba@latestInitial Setup
bash
# Create configuration directory
mkdir -p ~/.boba/logs
chmod 700 ~/.boba
# Generate default configs
boba doctor
# Secure secrets file
chmod 600 ~/.boba/secrets.yaml
# Verify setup
boba doctorConfiguration Management
Directory Structure
~/.boba/
├── profiles.yaml # Profile definitions
├── routes.yaml # Routing rules
├── pricing.yaml # Model pricing
├── secrets.yaml # API keys (0600)
├── usage.db # SQLite database
├── logs/ # Application logs
│ └── boba.log
├── git-hooks/ # Git hooks
└── pricing.cache.json # Cached pricing (auto-generated)Version Control
Track configurations (except secrets):
bash
cd ~/.boba
git init
git add profiles.yaml routes.yaml pricing.yaml
cat > .gitignore << EOF
secrets.yaml
*.db
*.db-*
logs/
pricing.cache.json
EOF
git commit -m "Initial BobaMixer configuration"
# Push to private repo
git remote add origin git@github.com:youruser/boba-config.git
git push -u origin mainShared Configuration
For team environments:
bash
# Shared configs in team repo
git clone git@github.com:team/boba-shared-config.git ~/boba-shared
# Symlink shared configs
ln -sf ~/boba-shared/profiles.yaml ~/.boba/profiles.yaml
ln -sf ~/boba-shared/routes.yaml ~/.boba/routes.yaml
ln -sf ~/boba-shared/pricing.yaml ~/.boba/pricing.yaml
# Keep individual secrets
cp ~/.boba/secrets.yaml.example ~/.boba/secrets.yaml
chmod 600 ~/.boba/secrets.yaml
# Edit with your personal API keysDatabase Management
Database Location
~/.boba/usage.dbBackup Strategy
Automated Daily Backup
bash
# Create backup script
cat > /usr/local/bin/backup-boba.sh << 'EOF'
#!/bin/bash
BACKUP_DIR=~/backups/boba
mkdir -p $BACKUP_DIR
# Hot backup (safe during use)
sqlite3 ~/.boba/usage.db ".backup $BACKUP_DIR/usage-$(date +%Y%m%d).db"
# Compress old backups
find $BACKUP_DIR -name "usage-*.db" -mtime +7 -exec gzip {} \;
# Delete backups older than 90 days
find $BACKUP_DIR -name "usage-*.db.gz" -mtime +90 -delete
echo "Backup completed: $BACKUP_DIR/usage-$(date +%Y%m%d).db"
EOF
chmod +x /usr/local/bin/backup-boba.sh
# Schedule daily backup (add to crontab)
crontab -e
# Add line:
0 2 * * * /usr/local/bin/backup-boba.shManual Backup
bash
# Hot backup (safe during use)
sqlite3 ~/.boba/usage.db ".backup /tmp/usage-backup.db"
# Or simple copy (only when boba is not running)
pkill -f boba
cp ~/.boba/usage.db ~/.boba/usage.db.backupRestore from Backup
bash
# Stop all boba processes
pkill -f boba
# Restore
cp /path/to/backup.db ~/.boba/usage.db
# Verify integrity
sqlite3 ~/.boba/usage.db "PRAGMA integrity_check;"
# Test
boba stats --todayDatabase Maintenance
Weekly Maintenance Script
bash
cat > /usr/local/bin/boba-maintenance.sh << 'EOF'
#!/bin/bash
DB=~/.boba/usage.db
echo "BobaMixer Database Maintenance"
echo "==============================="
# Check integrity
echo "Checking integrity..."
if ! sqlite3 $DB "PRAGMA integrity_check;" | grep -q "ok"; then
echo "ERROR: Database integrity check failed!"
exit 1
fi
echo "✓ Integrity OK"
# Vacuum
echo "Vacuuming database..."
sqlite3 $DB "VACUUM;"
echo "✓ Vacuum complete"
# Analyze
echo "Analyzing database..."
sqlite3 $DB "ANALYZE;"
echo "✓ Analyze complete"
# Show size
echo "Database size: $(du -h $DB | cut -f1)"
# Show record count
echo "Total records: $(sqlite3 $DB 'SELECT COUNT(*) FROM usage_records;')"
echo "Maintenance complete!"
EOF
chmod +x /usr/local/bin/boba-maintenance.sh
# Schedule weekly (add to crontab)
crontab -e
# Add line:
0 3 * * 0 /usr/local/bin/boba-maintenance.shVacuum Database
bash
sqlite3 ~/.boba/usage.db "VACUUM;"Check Integrity
bash
sqlite3 ~/.boba/usage.db "PRAGMA integrity_check;"Analyze for Query Optimization
bash
sqlite3 ~/.boba/usage.db "ANALYZE;"View Database Size
bash
du -h ~/.boba/usage.dbData Cleanup
Purge Old Records
bash
# Delete records older than 90 days
sqlite3 ~/.boba/usage.db "DELETE FROM usage_records WHERE ts < strftime('%s', 'now', '-90 days');"
# Reclaim space
sqlite3 ~/.boba/usage.db "VACUUM;"Archive Before Purging
bash
# Export to CSV
sqlite3 -header -csv ~/.boba/usage.db \
"SELECT * FROM usage_records WHERE ts < strftime('%s', 'now', '-90 days');" \
> ~/.boba/archive-$(date +%Y%m%d).csv
# Compress archive
gzip ~/.boba/archive-$(date +%Y%m%d).csv
# Then purge
sqlite3 ~/.boba/usage.db "DELETE FROM usage_records WHERE ts < strftime('%s', 'now', '-90 days');"
sqlite3 ~/.boba/usage.db "VACUUM;"Automated Cleanup Script
bash
cat > /usr/local/bin/boba-cleanup.sh << 'EOF'
#!/bin/bash
DB=~/.boba/usage.db
ARCHIVE_DIR=~/.boba/archives
DAYS_TO_KEEP=90
mkdir -p $ARCHIVE_DIR
# Archive old records
echo "Archiving records older than $DAYS_TO_KEEP days..."
sqlite3 -header -csv $DB \
"SELECT * FROM usage_records WHERE ts < strftime('%s', 'now', '-$DAYS_TO_KEEP days');" \
> $ARCHIVE_DIR/archive-$(date +%Y%m%d).csv
if [ -s $ARCHIVE_DIR/archive-$(date +%Y%m%d).csv ]; then
# Compress archive
gzip $ARCHIVE_DIR/archive-$(date +%Y%m%d).csv
# Delete from database
echo "Purging old records..."
sqlite3 $DB "DELETE FROM usage_records WHERE ts < strftime('%s', 'now', '-$DAYS_TO_KEEP days');"
# Vacuum
echo "Vacuuming database..."
sqlite3 $DB "VACUUM;"
echo "Cleanup complete. Archived to: $ARCHIVE_DIR/archive-$(date +%Y%m%d).csv.gz"
else
echo "No records to archive."
rm $ARCHIVE_DIR/archive-$(date +%Y%m%d).csv
fi
EOF
chmod +x /usr/local/bin/boba-cleanup.sh
# Schedule monthly (add to crontab)
crontab -e
# Add line:
0 4 1 * * /usr/local/bin/boba-cleanup.shLogging
Log Location
~/.boba/logs/boba.logLog Levels
Set via environment variable:
bash
export BOBA_LOG_LEVEL=debug # trace|debug|info|warn|errorLog Rotation
Using logrotate (Linux)
bash
# Create logrotate config
sudo cat > /etc/logrotate.d/bobamixer << EOF
/home/*/.boba/logs/boba.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
create 0644 $USER $USER
}
EOFManual Script
bash
cat > /usr/local/bin/rotate-boba-logs.sh << 'EOF'
#!/bin/bash
LOG_DIR=~/.boba/logs
LOG_FILE=$LOG_DIR/boba.log
if [ -f "$LOG_FILE" ]; then
# Rotate
mv $LOG_FILE $LOG_FILE.$(date +%Y%m%d)
# Compress old logs
find $LOG_DIR -name "boba.log.*" ! -name "*.gz" -mtime +1 -exec gzip {} \;
# Delete logs older than 30 days
find $LOG_DIR -name "boba.log.*.gz" -mtime +30 -delete
# Create new log file
touch $LOG_FILE
echo "Log rotation complete"
fi
EOF
chmod +x /usr/local/bin/rotate-boba-logs.sh
# Schedule daily
crontab -e
# Add:
0 0 * * * /usr/local/bin/rotate-boba-logs.shViewing Logs
bash
# Tail logs
tail -f ~/.boba/logs/boba.log
# Last 100 lines
tail -n 100 ~/.boba/logs/boba.log
# Search for errors
grep -i error ~/.boba/logs/boba.log
# View with timestamps
cat ~/.boba/logs/boba.log | grep "$(date +%Y-%m-%d)"Monitoring
Health Checks
bash
# Basic health check
boba doctor
# Detailed check
boba doctor --verbose
# Exit code check (for monitoring systems)
if boba doctor > /dev/null 2>&1; then
echo "OK"
else
echo "CRITICAL"
exit 2
fiUsage Monitoring Script
bash
cat > /usr/local/bin/boba-monitor.sh << 'EOF'
#!/bin/bash
# Thresholds
DAILY_BUDGET=50.00
WARN_THRESHOLD=75 # percent
# Get today's spending
CURRENT=$(boba stats --today --format json | jq '.total_cost_usd')
# Calculate percentage
PERCENT=$(echo "scale=2; $CURRENT / $DAILY_BUDGET * 100" | bc)
# Check threshold
if (( $(echo "$PERCENT > $WARN_THRESHOLD" | bc -l) )); then
echo "WARNING: Daily spending at ${PERCENT}% ($CURRENT / $DAILY_BUDGET)"
# Send alert (example: email)
echo "BobaMixer spending alert: ${PERCENT}% of daily budget" | \
mail -s "BobaMixer Budget Alert" admin@example.com
fi
EOF
chmod +x /usr/local/bin/boba-monitor.sh
# Run every hour
crontab -e
# Add:
0 * * * * /usr/local/bin/boba-monitor.shIntegration with Monitoring Systems
Prometheus Metrics Export
bash
# Export metrics to file
cat > /usr/local/bin/boba-export-metrics.sh << 'EOF'
#!/bin/bash
METRICS_FILE=/var/lib/prometheus/node-exporter/bobamixer.prom
cat > $METRICS_FILE << METRICS
# HELP bobamixer_daily_cost Daily cost in USD
# TYPE bobamixer_daily_cost gauge
bobamixer_daily_cost $(boba stats --today --format json | jq '.total_cost_usd')
# HELP bobamixer_daily_requests Daily request count
# TYPE bobamixer_daily_requests counter
bobamixer_daily_requests $(boba stats --today --format json | jq '.total_requests')
# HELP bobamixer_daily_tokens Daily token count
# TYPE bobamixer_daily_tokens counter
bobamixer_daily_tokens $(boba stats --today --format json | jq '.total_tokens')
METRICS
EOF
chmod +x /usr/local/bin/boba-export-metrics.sh
# Update every 5 minutes
crontab -e
# Add:
*/5 * * * * /usr/local/bin/boba-export-metrics.shUpgrading
Pre-Upgrade Checklist
bash
# 1. Backup configuration
cp -r ~/.boba ~/.boba.backup.$(date +%Y%m%d)
# 2. Backup database
sqlite3 ~/.boba/usage.db ".backup ~/.boba/usage.db.backup.$(date +%Y%m%d)"
# 3. Check current version
boba version
# 4. Review changelog
# https://github.com/royisme/BobaMixer/releasesUpgrade Process
Using Homebrew
bash
# Update tap
brew update
# Upgrade BobaMixer
brew upgrade bobamixer
# Verify new version
boba version
# Run health check
boba doctorManual Upgrade
bash
# Backup current binary
sudo cp /usr/local/bin/boba /usr/local/bin/boba.old
# Download new version
wget https://github.com/royisme/BobaMixer/releases/latest/download/bobamixer_Linux_x86_64.tar.gz
# Extract and install
tar -xzf bobamixer_Linux_x86_64.tar.gz
sudo mv boba /usr/local/bin/
sudo chmod +x /usr/local/bin/boba
# Verify
boba version
# Test
boba doctor
# If issues, rollback
# sudo mv /usr/local/bin/boba.old /usr/local/bin/bobaPost-Upgrade
bash
# Run health check
boba doctor
# Check for config updates needed
boba doctor --verbose
# Test functionality
boba stats --today
boba route test "test"
# Review and apply suggestions
boba actionDatabase Migrations
BobaMixer automatically migrates the database schema when needed.
If manual migration is required:
bash
# Backup first
sqlite3 ~/.boba/usage.db ".backup ~/.boba/usage.db.pre-migration"
# Run migration (if provided)
sqlite3 ~/.boba/usage.db < migration.sql
# Verify
sqlite3 ~/.boba/usage.db "PRAGMA integrity_check;"Performance Tuning
Database Performance
bash
# Enable WAL mode (better concurrency)
sqlite3 ~/.boba/usage.db "PRAGMA journal_mode=WAL;"
# Set cache size (in KB, default ~2MB)
sqlite3 ~/.boba/usage.db "PRAGMA cache_size=-8000;" # 8MB
# Set page size (before data is added)
sqlite3 ~/.boba/usage.db "PRAGMA page_size=4096;"Disk Space Management
bash
# Check usage
du -sh ~/.boba
# Breakdown by component
du -sh ~/.boba/*
# Find large log files
find ~/.boba/logs -type f -size +10M
# Cleanup recommendations
boba-cleanup.sh # Run cleanup scriptSecurity
File Permissions
bash
# Verify permissions
ls -la ~/.boba
# Correct permissions
chmod 700 ~/.boba
chmod 600 ~/.boba/secrets.yaml
chmod 644 ~/.boba/profiles.yaml
chmod 644 ~/.boba/routes.yaml
chmod 644 ~/.boba/pricing.yamlAPI Key Rotation
bash
# 1. Add new key to secrets.yaml
vi ~/.boba/secrets.yaml
# Add: anthropic_key_new: sk-ant-new-key
# 2. Test with new key
# Update one profile temporarily
sed -i 's/anthropic_key/anthropic_key_new/g' ~/.boba/profiles.yaml
# 3. Test
boba doctor
# 4. If successful, update all profiles
# 5. Remove old key from secrets.yaml
# 6. Verify
boba doctorSecrets Backup
bash
# Encrypted backup of secrets
gpg --symmetric --cipher-algo AES256 ~/.boba/secrets.yaml
# This creates ~/.boba/secrets.yaml.gpg
# Restore:
gpg --decrypt ~/.boba/secrets.yaml.gpg > ~/.boba/secrets.yaml
chmod 600 ~/.boba/secrets.yamlTroubleshooting
Database Locked Error
bash
# Find processes using database
lsof ~/.boba/usage.db
# Kill if necessary
pkill -f boba
# If persists, check for stale locks
rm ~/.boba/usage.db-shm ~/.boba/usage.db-wal
# Restart
boba doctorHigh Disk Usage
bash
# Check database size
du -h ~/.boba/usage.db
# Check log size
du -h ~/.boba/logs/
# Cleanup
boba-cleanup.sh
rotate-boba-logs.sh
# Vacuum database
sqlite3 ~/.boba/usage.db "VACUUM;"Performance Issues
bash
# Analyze database
sqlite3 ~/.boba/usage.db "ANALYZE;"
# Check for missing indexes
sqlite3 ~/.boba/usage.db ".schema"
# Enable WAL mode
sqlite3 ~/.boba/usage.db "PRAGMA journal_mode=WAL;"
# Increase cache
sqlite3 ~/.boba/usage.db "PRAGMA cache_size=-10000;"Best Practices
1. Regular Backups
- Daily automated database backups
- Weekly configuration backups
- Monthly verification of backup restoration
2. Monitoring
- Set up health checks
- Monitor disk usage
- Track API spending
- Alert on budget thresholds
3. Maintenance
- Weekly database maintenance
- Monthly cleanup of old records
- Quarterly review of configurations
- Regular log rotation
4. Security
- Maintain correct file permissions
- Rotate API keys periodically
- Never commit secrets to version control
- Use encrypted backups for secrets
5. Documentation
- Document custom configurations
- Track changes to routing rules
- Maintain runbooks for common issues
- Share knowledge with team
Production Checklist
- [ ] BobaMixer installed and verified
- [ ] Configuration files created and validated
- [ ] Secrets file secured (0600 permissions)
- [ ] Database backup scheduled
- [ ] Log rotation configured
- [ ] Monitoring set up
- [ ] Cleanup scripts scheduled
- [ ] Team has access to documentation
- [ ] Runbooks created for common issues
- [ ] Upgrade process documented
Next Steps
- Troubleshooting - Common issues and solutions
- CLI Reference - Command documentation
- Configuration - Setup guide