Lightning, Operations & Production Capstone
Lightning adds a second state machine on top of Bitcoin: channel funding, commitment updates, routing, liquidity, backups, and on-chain recovery. Production work means operating all of that deliberately.
Goal — run a test Lightning integration against a controlled Bitcoin environment, then ship either the EVM or Bitcoin capstone with observable, recoverable operations.
1. Lightning primitives
| Primitive | Operational meaning |
|---|---|
| Channel funding | A 2-of-2 on-chain output anchors off-chain commitments; it is not a custodial database balance. |
| Commitment transaction | Each state update replaces the economic ownership view; old-state publication is punished/recovered through protocol mechanisms. |
| HTLC | Conditional payment using a hash preimage and timelock; it drives routing and atomicity. |
| Invoice / offer | Payment request and amount/expiry metadata; validate network, amount, expiry, and destination policy. |
| Liquidity | Inbound and outbound capacity are distinct; a channel can be open and still fail a payment. |
| Watchtower / backup | Availability and recovery are security properties; monitor channels and rehearse restoration. |
Read the relevant BOLTs and choose a stack intentionally: LND, Core Lightning, or LDK. Their API and operational tradeoffs differ. Use a test environment first; do not expose an unauthenticated node RPC or assume a hosted provider removes custody/liquidity risk.
1b. LDK & watchtowers
- LDK — embed Lightning in Rust/mobile apps; you own persistence, networking, and signer boundaries. Read
labs/phase-26-lightning/ldk/README.mdfor integration path. - Watchtowers — encrypted channel backups; tower publishes penalty tx if counterparty broadcasts revoked state. Configure on regtest; document in
labs/phase-26-lightning/watchtower/RUNBOOK.md. - Liquidity swaps — submarine swaps and loop-style tools change inbound capacity; treat as custodial/trust tradeoffs until verified.
2. Lightning service boundaries
- Invoice creation: server creates a bounded, expiring invoice and records an idempotency key; client never fabricates a paid state.
- Settlement: trust the node’s settled/failed event plus durable reconciliation, not a browser redirect.
- Withdrawal: apply limits, velocity checks, destination policy, and two-person review for treasury paths.
- Reconciliation: reconcile invoices, channel state, on-chain wallet balance, and ledger entries on a schedule.
- Failure handling: surface failed/expired/pending payments; do not retry blindly with a new invoice or duplicate idempotency key.
3. Production operations checklist
| System | Monitor | Runbook trigger |
|---|---|---|
| RPC / node | block height lag, peer health, RPC latency/errors, disk and index state | provider/node lag exceeds policy |
| Indexer / relayer | cursor lag, reorg rollback, queue depth, nonce age, fee replacement | lag grows or a nonce is stuck |
| Contracts | admin calls, pause events, invariant/event anomalies, oracle deviation, cap utilization | unexpected privileged/economic event |
| Bridge / messages | sent → delivered → executed state, retries, rate-limit use, supply cap | message exceeds SLA or state diverges |
| Lightning | channel availability, inbound/outbound liquidity, payment success, force-close/watchtower alerts | liquidity or channel-security threshold breached |
Use two independent data sources for critical state. Keep secrets in a KMS/HSM or a dedicated signer; use a Safe/timelock for EVM administration. Run incident drills: pause/disable, contain, reconcile, communicate, recover, and write the postmortem.
4. Practical lab — Lightning payment service
Deliverable
- Run two Lightning nodes on a controlled test network backed by regtest/signet. Open and fund a channel, make a payment, and observe invoice/payment/channel events.
- Build a small API with
POST /invoices(idempotency key) and a webhook/event consumer that marks an order paid only after durable settlement reconciliation. - Simulate an expired invoice, failed route, restart, duplicate webhook, channel liquidity shortage, and a force-close/recovery procedure. Each scenario must have an automated test or written runbook.
- Expose dashboards for payment success rate, pending/failed invoices, channel liquidity, on-chain confirmations, and reconciliation mismatches. Alert on an actionable threshold.
5. Final capstone — choose a production boundary
| Track | Build | Non-negotiable evidence |
|---|---|---|
| EVM protocol | Guarded vault/lending integration on an L2 | Foundry fuzz/invariants/fork tests, verified deploy, Safe/timelock, monitoring, threat model |
| EVM infrastructure | Reorg-safe indexer + relayer + operations dashboard | backfill/reorg tests, provider failover, nonce runbook, lag/error alerts, load test |
| Bitcoin wallet | Descriptor + PSBT treasury workflow | regtest integration tests, fee/RBF/CPFP tests, multisig recovery drill, signer boundary |
| Lightning service | Invoice settlement service with controlled node integration | idempotent ledger, reconciliation, failure drills, channel/liquidity monitoring |
Gate — testnets and local networks prove mechanics, not security or economic safety. Do not use real funds until an independent reviewer has evaluated the code, keys, operations, and threat model. Capstone rubric: docs/capstone-rubric.html.
Track note — Lightning capstone evidence continues in Phase 26 if you have not yet published a reproducible repo.