# Whisper AI Backend - Deployment Guide

## ⚠️ Critical Requirement: FFmpeg 4.x

**IMPORTANT**: This application requires **FFmpeg 4.x** (not 5.x or 6.x) because PyAV 11.0.0 (used by faster-whisper) is not compatible with newer FFmpeg versions.

- ✅ **FFmpeg 4.4.x** - Works perfectly
- ❌ **FFmpeg 5.x / 6.x / 7.x** - Will cause `subprocess-exited-with-error` during `pip install`

See **Step 5** for detailed FFmpeg 4 installation instructions.

---

## Overview

You **can deploy without Nginx** for internal use, but **Nginx is recommended** for production. Here are both options:

---

## Option 1: Simple Deployment (Without Nginx)

✅ **Best for**: Internal servers, development, or direct API access  
⚠️ **Not recommended for**: Public-facing production sites

### Step-by-Step Deployment

#### 1. Connect to Your Server

```bash
ssh user@your-server-ip
```

#### 2. Install System Dependencies

```bash
# Update system
sudo apt update && sudo apt upgrade -y

# Install Python 3.11
sudo apt install python3.11 python3.11-venv python3-pip -y

# Install Redis
sudo apt install redis-server -y

# Install pkg-config (required for PyAV compilation)
sudo apt install pkg-config -y

# Install FFmpeg 4.x (PyAV 11.0.0 requires FFmpeg 4.x, not 5.x+)
# Check Ubuntu version first
lsb_release -a

# For Ubuntu 20.04 (has FFmpeg 4.x by default)
sudo apt install ffmpeg -y

# For Ubuntu 22.04+ (has FFmpeg 5.x+, need to downgrade)
# Option 1: Install from Ubuntu 20.04 repo
sudo add-apt-repository ppa:savoury1/ffmpeg4 -y
sudo apt update
sudo apt install ffmpeg -y

# Option 2: Build from source (if PPA doesn't work)
# See detailed instructions below in "FFmpeg 4 Installation" section

# Install Supervisor (for process management)
sudo apt install supervisor -y

# Verify installations
python3.11 --version
redis-cli --version
ffmpeg -version
```

#### 3. Create Application Directory

```bash
# Create app directory
sudo mkdir -p /opt/whisper-server
sudo chown $USER:$USER /opt/whisper-server
cd /opt/whisper-server
```

#### 4. Upload Your Code

**Option A: Using Git (Recommended)**

```bash
# If you have a git repo
git clone https://github.com/yourusername/whisper-server.git .

# Or initialize and push
git init
git remote add origin https://github.com/yourusername/whisper-server.git
git pull origin main
```

**Option B: Using SCP from your local machine**

```bash
# From your local machine
scp -r /Users/metta/Documents/sandbox/whisper-server/* user@your-server-ip:/opt/whisper-server/
```

**Option C: Using rsync (Best for updates)**

```bash
# From your local machine
rsync -avz --exclude '.venv' --exclude '__pycache__' \
  /Users/metta/Documents/sandbox/whisper-server/ \
  user@your-server-ip:/opt/whisper-server/
```

#### 5. FFmpeg 4 Installation (Important!)

**Why FFmpeg 4?** PyAV 11.0.0 (required by faster-whisper 1.0.1) only works with FFmpeg 4.x, not 5.x+

**Check your FFmpeg version:**

```bash
ffmpeg -version
```

**If you have FFmpeg 5.x or 6.x, you need to downgrade:**

**Method 1: Using PPA (Easiest)**

```bash
# Remove existing FFmpeg
sudo apt remove ffmpeg -y
sudo apt autoremove -y

# Add FFmpeg 4 PPA
sudo add-apt-repository ppa:savoury1/ffmpeg4 -y
sudo apt update

# Install FFmpeg 4
sudo apt install ffmpeg -y

# Verify version (should show 4.x)
ffmpeg -version
```

**Method 2: Install from Ubuntu 20.04 packages (if PPA fails)**

```bash
# Download FFmpeg 4 packages
cd /tmp
wget http://archive.ubuntu.com/ubuntu/pool/universe/f/ffmpeg/libavcodec58_4.4.2-0ubuntu0.22.04.1_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/f/ffmpeg/libavformat58_4.4.2-0ubuntu0.22.04.1_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/f/ffmpeg/libavutil56_4.4.2-0ubuntu0.22.04.1_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/f/ffmpeg/ffmpeg_4.4.2-0ubuntu0.22.04.1_amd64.deb

# Install packages
sudo dpkg -i *.deb
sudo apt-get install -f -y

# Verify
ffmpeg -version
```

**Method 3: Compile from source (most reliable but slower)**

```bash
# Install build dependencies
sudo apt install -y build-essential yasm cmake libtool libc6 libc6-dev \
  unzip wget libnuma1 libnuma-dev libx264-dev libx265-dev

# Download FFmpeg 4.4
cd /tmp
wget https://ffmpeg.org/releases/ffmpeg-4.4.4.tar.bz2
tar -xjf ffmpeg-4.4.4.tar.bz2
cd ffmpeg-4.4.4

# Configure and build
./configure --enable-gpl --enable-libx264 --enable-libx265
make -j$(nproc)
sudo make install
sudo ldconfig

# Verify
ffmpeg -version
```

#### 6. Setup Python Environment

```bash
cd /opt/whisper-server

# Create virtual environment
python3.11 -m venv .venv

# Activate virtual environment
source .venv/bin/activate

# Install dependencies
pip install --upgrade pip
pip install -r requirements.txt
```

#### 7. Configure Redis

```bash
# Edit Redis config for persistence
sudo nano /etc/redis/redis.conf

# Find and set these values:
# maxmemory 2gb
# maxmemory-policy allkeys-lru
# save 900 1
# save 300 10

# Restart Redis
sudo systemctl restart redis-server
sudo systemctl enable redis-server

# Verify Redis is running
redis-cli ping  # Should return "PONG"
```

#### 8. Test the Application

```bash
# Activate venv
source .venv/bin/activate

# Test FastAPI server
python main.py &
# Should see: "Uvicorn running on http://0.0.0.0:8000"

# Test Celery worker (in another terminal)
celery -A celery_app worker --loglevel=info --concurrency=8 -Q whisper

# Test health endpoint
curl http://localhost:8000/health

# Stop test processes
pkill -f "python main.py"
pkill -f "celery.*worker"
```

#### 9. Configure Firewall

```bash
# Allow SSH (if not already)
sudo ufw allow 22/tcp

# Allow API port
sudo ufw allow 8000/tcp

# Enable firewall
sudo ufw enable
sudo ufw status
```

#### 10. Setup Supervisor (Process Management)

Create supervisor config:

```bash
sudo nano /etc/supervisor/conf.d/whisper.conf
```

Paste this configuration:

```ini
[program:whisper_api]
command=/opt/whisper-server/.venv/bin/python main.py
directory=/opt/whisper-server
user=YOUR_USERNAME
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
stderr_logfile=/var/log/whisper_api.err.log
stdout_logfile=/var/log/whisper_api.out.log
environment=PATH="/opt/whisper-server/.venv/bin:%(ENV_PATH)s"

[program:whisper_worker]
command=/opt/whisper-server/.venv/bin/celery -A celery_app worker --loglevel=info --concurrency=8 -Q whisper
directory=/opt/whisper-server
user=YOUR_USERNAME
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
stderr_logfile=/var/log/whisper_worker.err.log
stdout_logfile=/var/log/whisper_worker.out.log
environment=PATH="/opt/whisper-server/.venv/bin:%(ENV_PATH)s"
```

**Replace `YOUR_USERNAME` with your actual username!**

Update supervisor:

```bash
# Reload supervisor configuration
sudo supervisorctl reread
sudo supervisorctl update

# Start services
sudo supervisorctl start whisper_api
sudo supervisorctl start whisper_worker

# Check status
sudo supervisorctl status

# View logs
sudo tail -f /var/log/whisper_api.out.log
sudo tail -f /var/log/whisper_worker.out.log
```

#### 11. Verify Deployment

```bash
# Test from server
curl http://localhost:8000/health

# Test from your local machine
curl http://your-server-ip:8000/health

# Test transcription
curl -X POST http://your-server-ip:8000/transcribe \
  -F "file=@test.m4a"
```

---

## Option 2: Production Deployment (With Nginx)

✅ **Best for**: Public-facing production  
✅ **Benefits**: SSL/HTTPS, better performance, security, load balancing

### Additional Steps After Basic Setup

#### 1. Install Nginx

```bash
sudo apt install nginx -y
```

#### 2. Configure Nginx

```bash
sudo nano /etc/nginx/sites-available/whisper
```

Paste this configuration:

```nginx
# HTTP configuration
server {
    listen 80;
    server_name your-domain.com www.your-domain.com;  # Change this!

    # Increase timeouts for long audio processing
    proxy_connect_timeout 300;
    proxy_send_timeout 300;
    proxy_read_timeout 300;
    send_timeout 300;

    # Increase max upload size (adjust as needed)
    client_max_body_size 100M;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # CORS headers (if not handled by FastAPI)
        add_header 'Access-Control-Allow-Origin' '*' always;
        add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
        add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range' always;
    }
}
```

Enable the site:

```bash
# Enable site
sudo ln -s /etc/nginx/sites-available/whisper /etc/nginx/sites-enabled/

# Remove default site
sudo rm /etc/nginx/sites-enabled/default

# Test configuration
sudo nginx -t

# Restart Nginx
sudo systemctl restart nginx
sudo systemctl enable nginx
```

#### 3. Configure Firewall for Nginx

```bash
# Close direct API access
sudo ufw delete allow 8000/tcp

# Allow HTTP and HTTPS
sudo ufw allow 'Nginx Full'
sudo ufw status
```

#### 4. Setup SSL with Let's Encrypt (Optional but Recommended)

```bash
# Install Certbot
sudo apt install certbot python3-certbot-nginx -y

# Get SSL certificate (replace with your domain)
sudo certbot --nginx -d your-domain.com -d www.your-domain.com

# Test auto-renewal
sudo certbot renew --dry-run
```

#### 5. Update FastAPI CORS for Production

Edit `/opt/whisper-server/main.py`:

```python
# Change from
allow_origins=["*"]

# To
allow_origins=[
    "https://your-frontend-domain.com",
    "https://www.your-frontend-domain.com"
]
```

Restart services:

```bash
sudo supervisorctl restart whisper_api
```

---

## Management Commands

### Supervisor Commands

```bash
# Check status
sudo supervisorctl status

# Start services
sudo supervisorctl start whisper_api
sudo supervisorctl start whisper_worker
sudo supervisorctl start all

# Stop services
sudo supervisorctl stop whisper_api
sudo supervisorctl stop whisper_worker
sudo supervisorctl stop all

# Restart services
sudo supervisorctl restart whisper_api
sudo supervisorctl restart whisper_worker

# View logs
sudo tail -f /var/log/whisper_api.out.log
sudo tail -f /var/log/whisper_worker.err.log
```

### Update Application

```bash
# Pull latest code
cd /opt/whisper-server
git pull origin main

# Or upload new files via rsync
# (from local machine)
rsync -avz --exclude '.venv' --exclude '__pycache__' \
  /Users/metta/Documents/sandbox/whisper-server/ \
  user@your-server-ip:/opt/whisper-server/

# Restart services
sudo supervisorctl restart all
```

### Monitor Performance

```bash
# Check CPU and memory
htop

# Check Redis
redis-cli info stats

# Check disk space
df -h

# Check logs in real-time
sudo tail -f /var/log/whisper_worker.out.log
```

---

## Troubleshooting

### Service won't start

```bash
# Check supervisor logs
sudo supervisorctl tail whisper_api stderr
sudo supervisorctl tail whisper_worker stderr

# Check if ports are in use
sudo lsof -i :8000

# Check Python path
which python3.11
/opt/whisper-server/.venv/bin/python --version
```

### Out of memory

```bash
# Check memory usage
free -h

# Reduce Celery workers in supervisor config
# Change --concurrency=8 to --concurrency=4
sudo nano /etc/supervisor/conf.d/whisper.conf
sudo supervisorctl restart whisper_worker
```

### Redis issues

```bash
# Restart Redis
sudo systemctl restart redis-server

# Check Redis logs
sudo tail -f /var/log/redis/redis-server.log

# Clear Redis cache
redis-cli FLUSHDB
```

### FFmpeg / PyAV issues

**Error: `pkg-config is required for building PyAV`**

```bash
sudo apt install pkg-config -y
```

**Error: `error: subprocess-exited-with-error` during pip install**

This usually means FFmpeg version mismatch. PyAV 11.0.0 needs FFmpeg 4.x:

```bash
# Check FFmpeg version
ffmpeg -version

# If it shows 5.x or 6.x, downgrade to 4.x
sudo apt remove ffmpeg -y
sudo add-apt-repository ppa:savoury1/ffmpeg4 -y
sudo apt update
sudo apt install ffmpeg -y

# Reinstall PyAV
source .venv/bin/activate
pip uninstall av -y
pip install --no-cache-dir faster-whisper
```

**Error: `cannot find -lavcodec` or similar linking errors**

```bash
# Install FFmpeg development libraries
sudo apt install libavcodec-dev libavformat-dev libavutil-dev libswscale-dev -y

# Make sure pkg-config can find them
pkg-config --modversion libavcodec
# Should output version like 58.x.x (for FFmpeg 4)
```

**On macOS (for local development)**

```bash
# Install FFmpeg 4 via Homebrew
brew install ffmpeg@4

# Set environment variables for pip install
export PKG_CONFIG_PATH="/opt/homebrew/opt/ffmpeg@4/lib/pkgconfig"
export LDFLAGS="-L/opt/homebrew/opt/ffmpeg@4/lib"
export CPPFLAGS="-I/opt/homebrew/opt/ffmpeg@4/include"

# Install requirements
pip install -r requirements.txt
```

### Nginx issues

```bash
# Check Nginx status
sudo systemctl status nginx

# Check Nginx error logs
sudo tail -f /var/log/nginx/error.log

# Test configuration
sudo nginx -t

# Restart Nginx
sudo systemctl restart nginx
```

---

## Security Best Practices

1. **Change default Redis port** or bind to localhost only
2. **Use environment variables** for sensitive config
3. **Enable UFW firewall**
4. **Keep system updated**: `sudo apt update && sudo apt upgrade`
5. **Use SSL/HTTPS** in production
6. **Restrict CORS origins** to your frontend domain
7. **Set up log rotation** for supervisor logs
8. **Use strong SSH keys** instead of passwords
9. **Disable root SSH login**
10. **Set up monitoring** (optional: Prometheus + Grafana)

---

## Environment Variables (Optional)

Create `.env` file:

```bash
nano /opt/whisper-server/.env
```

Add:

```bash
REDIS_URL=redis://localhost:6379/3
CELERY_CONCURRENCY=8
ALLOWED_ORIGINS=https://your-frontend.com
ENVIRONMENT=production
```

Update code to use environment variables:

```python
import os
from dotenv import load_dotenv

load_dotenv()

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/3")
```

Install python-dotenv:

```bash
pip install python-dotenv
```

---

## Cost Optimization

**Without Nginx**: Free (if you have the server)
**With Nginx**: Free (Nginx is free, Let's Encrypt SSL is free)

**Recommended Server Specs**:

- **Minimum**: 2 CPU cores, 4GB RAM, 20GB storage
- **Recommended**: 4+ CPU cores, 8GB RAM, 50GB storage
- **Your Setup**: 6 cores, 64GB RAM (excellent, can handle heavy load!)

---

## Summary

### Simple Deployment (No Nginx)

✅ Faster to setup (30 minutes)
✅ Direct API access
⚠️ No SSL/HTTPS
⚠️ Less secure for public internet

### Production Deployment (With Nginx)

✅ SSL/HTTPS support
✅ Better security
✅ Better performance
✅ Load balancing ready
⏱️ Takes 1 hour to setup

**Recommendation**: Start simple, add Nginx later when going to production!