Symptoms

• Grafana: webhook_received_total{event_type="*", dedup_skipped="false"} AND HTTP 401 rate from /webhooks/github > 10/min sustained
• Sentry: cluster of "Invalid signature" responses from webhook handler
• GitHub App "Recent deliveries" page shows a spike of 401 responses

Severity & escalation

• PAGE 24/7 — potential security incident (attempted webhook forgery OR our secret leak)
• Ack window: 15 min
• Escalate immediately if pattern indicates external attack (distributed IPs, scripted requests)
• Engineering lead + security review

Immediate actions (< 5 min)

1. Tail webhook traffic:

   cd apps/api && npx wrangler tail --config wrangler.toml --format=pretty | grep webhooks
2. Check source distribution: Sentry → group by client IP / user agent. GitHub webhooks come from known IPs (https://api.github.com/meta (opens in a new tab) → hooks)
3. Check GitHub App settings: https://github.com/settings/apps/arno-dev-vadimpianov/advanced (opens in a new tab) → recent deliveries
   • If all recent deliveries 401 (including legitimate ones) → our secret is broken / mismatched
   • If only some are 401 → external scanner/attacker
4. Compare secret hash: wrangler secret list --config wrangler.toml | grep WEBHOOK shows secret name (not value). Verify it matches GitHub App webhook secret in dashboard

Diagnosis (5-20 min)

Branch A: Our secret mismatch (legitimate webhooks are 401-ing)

• Cause: webhook secret rotated in GitHub App settings but not updated in Workers (or vice versa)
• Risk: all incoming webhooks rejected → drift status not updated → component_md_raw stale
• Recovery: re-sync secret (see below)

Branch B: External attacker (random IPs, legitimate webhooks still ok)

• Cause: scanner probing endpoints, OR scripted attempt to forge webhook
• Risk: low (HMAC verification working as designed, attacker will not get payload acceptance)
• Mitigation:
  • Cloudflare WAF rule: block requests to /webhooks/github without User-Agent: GitHub-Hookshot
  • Rate limit per IP in CF dashboard
  • Log + monitor; don't escalate if no successful forgery

Branch C: Our secret leaked

• Indicator: 200 responses to forged webhooks (we accept them as valid) → check pushed_by_us KV for unexpected SHAs
• Verify: review git history for accidental secret commits, scan logs for the secret value
• Recovery: rotate immediately, audit Worker writes triggered between leak time and rotation

Recovery

Verification

• webhook_received_total{status_code="200"} recovers to baseline
• 401 rate < 0.5/min sustained
• GitHub App "Recent deliveries" — recent ones are 200
• Test webhook: trigger ping from App settings → must return 200

Aftermath

• Post-mortem trigger: any signature failure due to our config OR ANY leak suspicion
• Document: cause, blast radius, rotation timeline
• Add to quarterly secret rotation schedule if leak occurred

Known false positives

• GitHub App settings edit — briefly disables webhook → recent deliveries show 401 temporarily during rotation
• Webhook redelivery from old App config — if webhook URL was recently changed, old deliveries fail. Do not PAGE.