Runbooks
Runbook: URL-import SSRF guard rejection spike

Symptoms

  • Grafana alert: rate(url_import_ssrf_rejections_total[5m]) > 10/min
  • Sentry: SSRFError exceptions spike > baseline 7-day avg
  • API logs: many 400 responses from extractor with reason safe_fetch_failed

Severity & escalation

  • INVESTIGATE (not PAGE — SSRF rejections are the correct defensive behavior)
  • Ack window: 4 hours business hours, next-day off-hours
  • Escalate if sustained > 1 hour OR pattern indicates infrastructure compromise

Immediate actions (< 5 min)

  1. Check rejection reasons distribution:

    # Query Sentry / log aggregator for SSRFError reasons
    # Common reasons: scheme_not_allowed, no_safe_ips, dns_resolve_failed
  2. Sample failing URLs (anonymized):

    • If all private IP ranges → normal attack/scan from external
    • If valid domains → DNS issue OR config drift
  3. Cross-reference with user_ids:

    • Single user spamming? → anti-abuse (see user_id rate limit)
    • Distributed across users? → external scan against extraction endpoint

Diagnosis (5-20 min)

Branch A: All rejections to private IP ranges

This is normal SSRF guard behavior. Causes:

  • External scan attempting SSRF against ARNO
  • User entered URL that resolves to 127.0.0.1 (typo, dev URL)
  • DNS rebinding attempt

Action: check whether rate limits on the user_id are already in place. If a single user — possibly an anti-abuse review.

Branch B: Rejections for valid public domains

Possible causes:

  • DNS resolver issues (Cloudflare DNS proxy outage?)
  • BLOCKED_NETWORKS list misconfigured (e.g. accidentally added public range)
  • IPv6 false positives (new IPv6 prefix not in block list correctly)

Action:

  1. Verify resolution from Worker context:
    wrangler tail | grep "dns_resolve_failed"
  2. Compare with public DNS resolution:
    dig +short example.com  # whatever URL failing
  3. Check BLOCKED_NETWORKS list in packages/url-import-extractor/src/safe-fetch.ts — verify a public range was not recently added

Branch C: dns_resolve_failed bursts

Indicates external DNS layer issue:

  • Cloudflare DNS outage
  • Network egress from Workers blocked
  • Specific TLD not resolvable

Action: check Cloudflare status page. If CF issue — wait, document outage.

Recovery

If valid SSRF protection (Branch A):

  • No action — system working as designed
  • Optionally: send anti-abuse notification to offending user
  • Verify rate limits on user_id are tight enough

If false positives (Branch B):

  • Hotfix BLOCKED_NETWORKS list in safe-fetch.ts
  • Deploy backend update
  • Re-test failing URLs

If DNS layer issue (Branch C):

  • Cannot fix at ARNO layer — wait for CF/upstream resolution
  • Communicate to users (status page) if sustained

Aftermath

  • Post-mortem if sustained > 30min OR caused customer-visible failures
  • Backfill SSRFError metrics in Grafana if missed
  • Tune alert threshold if too noisy (currently 10/min — may need a higher floor for high-traffic periods)

Known false positives

  • Tranco-listed domain with regional CDN: DNS resolves to a geo-specific IP that occasionally hits filtered range. Document specific case if recurring.
  • IPv6 addresses in shared CGNAT (carrier-grade NAT) — may look like private. Verify prefix correctly in BLOCKED_NETWORKS.

References