PHASE 15 Prod · ~5 hours

Production Engineering

The bridge from "works on Hardhat" to "survives a Monday morning with 50k users and a flaky RPC". This phase is pure DevOps with Web3 quirks.

1. The deploy checklist

• [ ] Contracts audited (or at least: Slither clean + 90%+ tests + fuzzing).
• [ ] Deployment is a script in Git (no "I ran it from my laptop").
• [ ] Constructor args committed; contract verified on Etherscan/Basescan.
• [ ] Owner is a multisig (Gnosis Safe) or timelock, not a single EOA.
• [ ] Emergency pause tested.
• [ ] Frontend reads contract addresses from config per chain, not hard-coded.
• [ ] Indexer has backfill strategy and reorg safety.
• [ ] Monitoring & alerts wired for stuck txs, RPC errors, event gap.

2. RPC strategy — your biggest single dependency

3. Key management

• User wallets — not your problem; MetaMask / WalletConnect / RainbowKit.
• Hot keys (indexer relayer, bots) — KMS (AWS KMS, GCP KMS). Never raw keys in env vars in prod. Use @aws-sdk/client-kms + ethers Signer adapter.
• Admin keys — Gnosis Safe multisig with threshold N-of-M. Add a timelock (e.g., 48h) so users can react to malicious proposals.

// Gnosis Safe + Timelock upgrade flow
Safe → schedule(timelock, contract.upgradeTo(newImpl))
        │ 48h pass, users can exit
        ▼
Safe → execute(timelock, contract.upgradeTo(newImpl))

4. Upgradeability — only if you need it

• Transparent proxy (EIP-1967) — classic, simple.
• UUPS — upgrade logic in the implementation. Cheaper to deploy, risk of locking yourself out.
• Diamond (EIP-2535) — multi-facet; powerful, complex.
• Don't upgrade — simplest and safest. Deploy v2 and migrate.

5. CI/CD

# .github/workflows/contracts.yml
name: contracts
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci
      - run: npx hardhat test
      - run: npx hardhat coverage
      - name: Slither
        uses: crytic/slither-action@v0.4.0
        with: { fail-on: medium }
      - name: Gas report
        run: REPORT_GAS=true npx hardhat test

Ship a deploy workflow that is gated on manual approval + tag, and writes deployments/<chain>.json back to the repo as a PR.

6. Observability

7. Docker + infra shape

For 10–50k DAU, a single VPS + managed DB + Alchemy free tier is enough. Scale by adding worker replicas and a pull-oracle pattern for price fetch.

8. Deploying the frontend

• Static build (Vite) → Vercel / Cloudflare Pages. No secrets ship client-side.
• Config per chain in public/config.json; don't rebuild per env.
• Set Content-Security-Policy; forbid inline scripts except vetted.
• Pin RPC endpoints in your own backend; don't expose API keys to the browser.

9. Monitoring with OpenZeppelin Defender / Tenderly

• Defender Sentinels — condition-based alerts on events (e.g., "paging if withdraw > $100k").
• Tenderly — tx simulator + alerts + Web3 Actions (serverless reactors). Great for incident forensics.
• Forta — federated monitoring network; detect protocol anomalies with custom bots.

10. Incident response

• Pause switch runbook: who calls it, how many sigs, how to communicate.
• Public post-mortem template (within 72h of major incident).
• Bug bounty on Immunefi with clear scope and severity payouts.
• "War room" Slack channel; dedicated status page.

11. Project

Quiz