1. Log in to your Okta Admin Console
2. Navigate to Applications → Applications → Create App Integration
3. Select OIDC - OpenID Connect as the sign-in method
4. Select Web Application as the application type
5. Click Next

Set the following values:

• App integration name: Nexus
• Grant type: Authorization Code (default)
• Sign-in redirect URI: https://<your-domain>/auth/sso/oidc/callback
• Sign-out redirect URI: Leave empty
• Controlled access: Select which users/groups should have access

Click Save.

From the app's General tab, copy:

• Client ID
• Client Secret

Issuer URL: Go to Security → API → Authorization Servers and copy the Issuer URI.
Default: https://dev-12345678.okta.com/oauth2/default

To map Okta groups to roles, add a groups claim:

Option A — Via App Settings:

1. In the Okta app, go to Sign On → OpenID Connect ID Token
2. Under Groups claim type, select Filter
3. Claim name: groups, Filter: Matches regex .*

Option B — Via Authorization Server:

1. Go to Security → API → Authorization Servers → default → Claims
2. Add Claim: name groups, include in ID Token + Access Token, value type Groups, filter regex .*

1. Click "+ Add Provider" at the top of this page
2. Set Display Name to Okta
3. Set Protocol to OpenID Connect (OIDC)
4. Paste the Issuer URL, Client ID, and Client Secret
5. Set Scopes to openid profile email groups
6. Choose a Default Role and optionally add Group → Role Mappings
7. Check Enable this provider, then Save

1. Click the Test button on the provider card above
2. If successful, the OIDC discovery document was fetched
3. Open the login page — a login button should appear for the provider
4. Click it to verify the full authentication flow

• invalid_client — Double-check Client ID and Secret. Regenerate in Okta if needed.
• redirect_uri_mismatch — Sign-in redirect URI must match exactly: https://<your-domain>/auth/sso/oidc/callback
• No groups in token — Verify the groups claim is configured (Step 4) and the groups scope is included.
• SSO button missing — Make sure the provider is enabled.
• BASE_URL mismatch — The server's BASE_URL env var must match your domain exactly.

• Primary runs all services normally and uploads SQLite database snapshots + a heartbeat file to S3 every 15 seconds
• Backup polls the primary's health endpoint every 3 seconds. It pre-downloads DB snapshots to stay warm
• After 3 consecutive health check failures, the backup checks the S3 heartbeat. If stale (>30s), it promotes itself
• Promotion downloads the latest DBs, reinitializes them, reassigns the Elastic IP, and starts all services (~1 second)
• Clients (overlays, dashboards, automation) reconnect automatically via the Elastic IP
• Split-brain prevention: when the old primary recovers, it reads the S3 heartbeat. If another instance owns it, it starts as backup

1. Elastic IP — Allocate one in EC2 → Elastic IPs. Associate it with the primary instance. Note the eipalloc-xxx allocation ID
2. S3 Bucket — Create a bucket in the same region (e.g. mlnexus-backup). No public access needed
3. IAM Role — Create an instance role with S3 read/write (s3:GetObject, s3:PutObject, s3:ListBucket) and EC2 EIP permissions (ec2:DescribeAddresses, ec2:AssociateAddress, ec2:DisassociateAddress). Attach to both instances

1. Launch a second EC2 instance (same type as primary, same VPC/subnet)
2. Install Node.js 20: curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - && sudo apt install -y nodejs
3. Install nginx, PM2, and clone the repo: sudo apt install -y nginx && sudo npm install -g pm2 && git clone <repo> ~/nexus && cd ~/nexus && npm install
4. Copy the nginx config and SSL certs from the primary
5. Copy the .env from the primary, then change the failover vars (see Step 3)

Add to the primary's .env:

FAILOVER_MODE=primary
FAILOVER_S3_BUCKET=mlnexus-backup
FAILOVER_PARTNER_URL=http://<backup-private-ip>:8080/health
FAILOVER_ELASTIC_IP_ALLOC_ID=eipalloc-xxxxxxxxx

Add to the backup's .env:

FAILOVER_MODE=backup
FAILOVER_S3_BUCKET=mlnexus-backup
FAILOVER_PARTNER_URL=http://<primary-private-ip>:8080/health
FAILOVER_ELASTIC_IP_ALLOC_ID=eipalloc-xxxxxxxxx

Both instances need port 8080 TCP open from the partner's private IP (for health check polling). Add inbound rules:

• Primary SG: allow TCP 8080 from <backup-private-ip>/32
• Backup SG: allow TCP 8080 from <primary-private-ip>/32
• Both: 80/443 TCP from 0.0.0.0/0 (nginx), 22 TCP from your IP (SSH)

1. Start the primary first: pm2 restart all. Wait for "Sync complete" in the logs
2. Then start the backup: pm2 restart all. It should log "Starting in BACKUP mode"
3. Return to this page — the status tiles above should show Role: PRIMARY, Partner: Healthy
4. Test failover: stop the primary (pm2 stop all). Within ~9 seconds the backup should promote and the site stays live

• Partner shows "Down" — Check that port 8080 is open between instances. Test with: curl http://<partner-ip>:8080/health
• Backup keeps trying to promote — The primary isn't running or hasn't written a heartbeat to S3 yet. Start the primary first
• EIP reassignment fails — Check that the IAM role has ec2:AssociateAddress and ec2:DisassociateAddress permissions
• SSH disconnects after failover — The EIP moved. Clear the old host key: ssh-keygen -R <eip>, then reconnect
• Both instances in backup mode — Promote one manually via the button above or API: POST /api/failover/promote