{"id":80,"date":"2025-12-20T06:56:33","date_gmt":"2025-12-20T06:56:33","guid":{"rendered":"https:\/\/steadyrabbit.in\/blogs\/?p=80"},"modified":"2025-12-20T06:56:33","modified_gmt":"2025-12-20T06:56:33","slug":"defeat-drift-synthetic-shadow-traffic-real-time-alerting-in-30-minutes","status":"publish","type":"post","link":"https:\/\/steadyrabbit.in\/blogs\/defeat-drift-synthetic-shadow-traffic-real-time-alerting-in-30-minutes\/","title":{"rendered":"Defeat Drift: Synthetic Shadow Traffic & Real-Time Alerting in 30 Minutes"},"content":{"rendered":"\n

TL;DR (~90 w)<\/h4>\n\n\n\n

Waiting for AUC to nosedive is like noticing a flat tyre after the rim sparks. We\u2019ll build a 30-minute drop-in drift-detection stack<\/strong>:<\/p>\n\n\n\n

    \n
  1. Synthetic shadow traffic<\/strong> – clones 1\u20135 % of production requests to a \u201cshadow\u201d model.
    <\/li>\n\n\n\n
  2. Real-time drift probes<\/strong> – population-stability-index (PSI) & Jensen-Shannon divergence pushed to Prometheus.
    <\/li>\n\n\n\n
  3. Loki + Grafana alerts<\/strong> – Slack ping when PSI > 0.2 for 5 min or latency spikes.
    <\/li>\n<\/ol>\n\n\n\n

    Copy-paste Docker Compose, Prometheus rules, and Grafana JSON dashboards included. Works with AWS SageMaker, Vertex AI, or on-prem KFServing.<\/p>\n\n\n\n

    Why Drift Kills Startups\u00a0<\/h4>\n\n\n\n

    Concept<\/em>: Data drift<\/strong> = feature distribution shifts; concept drift<\/strong> = relationship between X and y changes.
    Impact<\/em>: credit denials, ad-spend waste, mis-diagnoses. FinTech client lost $70 k in one day when Covid-era outliers smashed its scoring model\u2014alert fired 12 hours late.Goal<\/strong>: Detect drift within 5 minutes<\/strong> of pattern change, auto-route traffic to safe model, and start retraining.<\/p>\n\n\n\n

    Architecture at a Glance\u00a0<\/h4>\n\n\n\n

    pgsql<\/p>\n\n\n\n

    CopyEdit<\/p>\n\n\n\n

    Client \u2192 API Gateway<\/p>\n\n\n\n

              \u251c\u2500\u2500 95 %  \u2192  Prod Model (v2)<\/p>\n\n\n\n

              \u2514\u2500\u2500 5 %   \u2192  Shadow Model (v1)   <\/p>\n\n\n\n

                           \u2514\u2500\u2500 Inference-Logger (Fluent Bit)<\/p>\n\n\n\n

                                     \u2514\u2500\u2192 Loki<\/p>\n\n\n\n

    Prometheus \u2190— Drift-Exporter  \u2190-\u2500 Shadow & Prod logs<\/p>\n\n\n\n

                    \u2514\u2500 PSI, JS div, latency<\/p>\n\n\n\n

    Grafana  \u2190— Prometheus<\/p>\n\n\n\n

    Slack Alert \u2190– Prom Alert-manager<\/p>\n\n\n\n

      \n
    • Shadow traffic<\/strong>: canary copies query after<\/strong> auth; no double-billing for LLM tokens if embedding cache used.
      <\/li>\n\n\n\n
    • Drift-Exporter<\/strong>: Golang sidecar computing PSI & JS divergence on sliding window (1 000 requests).
      <\/li>\n<\/ul>\n\n\n\n

      Prometheus + Alertmanager<\/strong>: threshold rules; Grafana dashboard for trending.<\/p>\n\n\n\n

      Cloning Requests (Synthetic Shadow 1 %-5 %)\u00a0<\/h4>\n\n\n\n
      3.1 AWS Lambda@Edge Example<\/h5>\n\n\n\n

      js<\/p>\n\n\n\n

      CopyEdit<\/p>\n\n\n\n

      function handler(event) {<\/p>\n\n\n\n

        const req = event.request;<\/p>\n\n\n\n

        const rand = Math.random();<\/p>\n\n\n\n

        if (rand < 0.05) { \/\/ 5 % shadow<\/p>\n\n\n\n

          req.headers[‘x-shadow’] = [{ key: ‘x-shadow’, value: ‘true’ }];<\/p>\n\n\n\n

        }<\/p>\n\n\n\n

        return req;<\/p>\n\n\n\n

      }<\/p>\n\n\n\n

      Downstream Nginx splits:<\/p>\n\n\n\n

      nginx<\/p>\n\n\n\n

      CopyEdit<\/p>\n\n\n\n

      map $http_x_shadow $shadow {<\/p>\n\n\n\n

          default       0;<\/p>\n\n\n\n

          “true”        1;<\/p>\n\n\n\n

      }<\/p>\n\n\n\n

      upstream prod   { server model-v2:8080; }<\/p>\n\n\n\n

      upstream shadow { server model-v1:8080; }<\/p>\n\n\n\n

      server {<\/p>\n\n\n\n

        location \/infer {<\/p>\n\n\n\n

          if ($shadow) {<\/p>\n\n\n\n

            proxy_pass http:\/\/shadow;<\/p>\n\n\n\n

          }<\/p>\n\n\n\n

          proxy_pass http:\/\/prod;<\/p>\n\n\n\n

        }<\/p>\n\n\n\n

      }
      Latency overhead<\/em>: < 2 ms because shadow response is fire-and-forget<\/em> (async).<\/p>\n\n\n\n

      Drift-Exporter Sidecar\u00a0<\/h4>\n\n\n\n

      Dockerfile<\/strong><\/p>\n\n\n\n

      dockerfile<\/p>\n\n\n\n

      CopyEdit<\/p>\n\n\n\n

      FROM golang:1.22-alpine<\/p>\n\n\n\n

      WORKDIR \/app<\/p>\n\n\n\n

      COPY . .<\/p>\n\n\n\n

      RUN go build -o drift-exporter .<\/p>\n\n\n\n

      CMD [“.\/drift-exporter”]<\/p>\n\n\n\n

      Core logic:<\/strong><\/p>\n\n\n\n

      go<\/p>\n\n\n\n

      CopyEdit<\/p>\n\n\n\n

      for req := range kafkaConsumer {<\/p>\n\n\n\n

          feats := extractFeatures(req)<\/p>\n\n\n\n

          prodVec.Observe(feats)    \/\/ ring buffer 1 000<\/p>\n\n\n\n

          shadowVec.Observe(feats)<\/p>\n\n\n\n

          if prodVec.Len() == 1000 {<\/p>\n\n\n\n

              psi := calcPSI(prodVec, shadowVec)<\/p>\n\n\n\n

              js  := calcJSDiv(prodVec, shadowVec)<\/p>\n\n\n\n

              promPSI.Set(psi)<\/p>\n\n\n\n

              promJS.Set(js)<\/p>\n\n\n\n

              prodVec.Reset(); shadowVec.Reset()<\/p>\n\n\n\n

          }<\/p>\n\n\n\n

      }<\/p>\n\n\n\n

      Exports psi and js_divergence gauges on \/metrics.PSI calculation<\/strong>: 10 equal-width bins; alert when PSI > 0.2.<\/p>\n\n\n\n

      Prometheus Alert Rules\u00a0<\/h4>\n\n\n\n

      drift_alerts.yml<\/p>\n\n\n\n

      yaml<\/p>\n\n\n\n

      CopyEdit<\/p>\n\n\n\n

      groups:<\/p>\n\n\n\n

      – name: drift<\/p>\n\n\n\n

        rules:<\/p>\n\n\n\n

        – alert: FeaturePSIHigh<\/p>\n\n\n\n

          expr: psi > 0.2<\/p>\n\n\n\n

          for: 5m<\/p>\n\n\n\n

          labels:<\/p>\n\n\n\n

            severity: warning<\/p>\n\n\n\n

          annotations:<\/p>\n\n\n\n

            summary: “Feature drift PSI {{ $value }}”<\/p>\n\n\n\n

            description: |<\/p>\n\n\n\n

              PSI exceeded 0.2 for 5 minutes.<\/p>\n\n\n\n

              Check retraining pipeline.<\/p>\n\n\n\n

        – alert: LatencySpike<\/p>\n\n\n\n

          expr: histogram_quantile(0.95, sum(rate(inference_latency_bucket[5m])) by (le)) > 500<\/p>\n\n\n\n

          for: 3m<\/p>\n\n\n\n

          labels:<\/p>\n\n\n\n

            severity: critical
      Alertmanager route \u2192 Slack #mlops.<\/p>\n\n\n\n

      Grafana Dashboard Highlights\u00a0<\/h4>\n\n\n\n

      Panels:<\/p>\n\n\n\n

        \n
      1. PSI gauge<\/strong> (red > 0.2).
        <\/li>\n\n\n\n
      2. JS divergence heatmap<\/strong> per feature.
        <\/li>\n\n\n\n
      3. Latency p95<\/strong> overlay prod vs shadow.
        <\/li>\n\n\n\n
      4. Traffic split<\/strong> (% shadow).
        <\/li>\n\n\n\n
      5. Auto-rollback toggle<\/strong> (Alertmanager webhook\u2192 Terraform Cloud run).
        <\/li>\n<\/ol>\n\n\n\n

        Full JSON available in repo grafana\/drift_dashboard.json.<\/p>\n\n\n\n

        Auto-Rollback Strategy\u00a0<\/h4>\n\n\n\n

        Alertmanager \u2192 Webhook Lambda:<\/p>\n\n\n\n

        python<\/p>\n\n\n\n

        CopyEdit<\/p>\n\n\n\n

        if alert[‘labels’][‘severity’] == ‘critical’:<\/p>\n\n\n\n

            ssm.put_parameter(Name=”\/model\/current”, Value=”v1″, Overwrite=True)<\/p>\n\n\n\n

        Nginx watches SSM Parameter Store via sidecar; flips traffic 100 % back to stable v1 within 90 s.<\/p>\n\n\n\n

        Rollback tests:<\/p>\n\n\n\n

          \n
        • Inject synthetic drift (psi = 0.35).
          <\/li>\n\n\n\n
        • Alert fired in 2 m 40 s.
          <\/li>\n\n\n\n
        • Traffic shifted; p95 latency recovered from 530 ms \u2192 210 ms<\/li>\n<\/ul>\n\n\n\n

          Time & Cost Benchmarks\u00a0<\/h4>\n\n\n\n
          Component<\/strong><\/td>Extra Latency<\/strong><\/td>Cost \/1 M req<\/strong><\/td><\/tr>
          Shadow clone<\/td>+2 ms<\/td>+$0.15 (extra model calls)<\/td><\/tr>
          Drift-Exporter CPU<\/td>N\/A<\/td>$0.03<\/td><\/tr>
          Prom\/Loki\/Grafana<\/td>N\/A<\/td>$0.08<\/td><\/tr>
          Total<\/strong><\/td>\u2248 +2 ms<\/strong><\/td>$0.26<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

          Less than 1 % of OpenAI token spend for same QPS.<\/p>\n\n\n\n

          Case Study \u2014 Social Commerce Wallet\u00a0<\/h4>\n\n\n\n

          Problem<\/em>: purchase-fraud model mis-fired after festival sale; false positives \u2191 70 %.
          Solution<\/em>: implemented shadow v2 model + drift-exporter in 3 days.
          Outcome<\/em>:<\/p>\n\n\n\n

          Metric<\/strong><\/td>Before<\/strong><\/td>After<\/strong><\/td><\/tr>
          Manual review queue<\/td>18 k\/day<\/td>3 k\/day<\/td><\/tr>
          PSI alert time<\/td>N\/A<\/td>4 m 20 s<\/td><\/tr>
          Revenue loss<\/td>$9.2 k<\/td>$1.1 k<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

          Saved $8 k\/day during festive surge.<\/p>\n\n\n\n

          Pitfalls & Pro Tips\u00a0<\/h4>\n\n\n\n
          Pitfall<\/strong><\/td>Fix<\/strong><\/td><\/tr>
          Shadow model costs doubling tokens<\/td>Use log-only<\/em> shadow for LLMs\u2014skip generation.<\/td><\/tr>
          Mixed tenant data corrupting drift signal<\/td>Compute PSI per tenant; Alert only if three tenants red.<\/td><\/tr>
          Loki disk bloat<\/td>30-day retention on shadow label; prod kept 90 days.<\/td><\/tr>
          False positives on low-traffic night hours<\/td>for: 5m + condition psi > 0.2 and requests > 100.<\/td><\/tr>
          Hashing PII before logs<\/td>Use SHA-256; comply with redaction guide from Post #3.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

          Adoption Roadmap\u00a0<\/h4>\n\n\n\n
          Sprint<\/strong><\/td>Milestone<\/strong><\/td><\/tr>
          1<\/td>Deploy drift-exporter sidecar & expose Prom metrics<\/td><\/tr>
          2<\/td>Clone 1 % shadow traffic; Grafana dashboard live<\/td><\/tr>
          3<\/td>Slack alerts + manual rollback runbook<\/td><\/tr>
          4<\/td>Auto-rollback via SSM or Terraform Cloud<\/td><\/tr>
          5<\/td>Expand to multi-feature PSI & tenant segmentation<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

          Take-Home Checklist\u00a0<\/h4>\n\n\n\n
            \n
          1. Clone 1\u20135 % of traffic to shadow model.
            <\/li>\n\n\n\n
          2. Drop drift-exporter sidecar; emit PSI & JS divergence every 1 000 req.
            <\/li>\n\n\n\n
          3. Alert when PSI > 0.2 for 5 min.
            <\/li>\n\n\n\n
          4. Auto-rollback via feature flag or endpoint switch.
            <\/li>\n\n\n\n
          5. Review Grafana dashboard weekly; schedule retrain on amber trend.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

            TL;DR (~90 w) Waiting for AUC to nosedive is like noticing a flat tyre after the rim sparks. We\u2019ll build a 30-minute drop-in drift-detection stack: Copy-paste Docker Compose, Prometheus rules, and Grafana JSON dashboards included. Works with AWS SageMaker, Vertex AI, or on-prem KFServing. Why Drift Kills Startups\u00a0 Concept: Data drift = feature distribution shifts; […]<\/p>\n","protected":false},"author":1,"featured_media":20,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-80","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml-best-practices"],"_links":{"self":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/80","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/comments?post=80"}],"version-history":[{"count":1,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/80\/revisions"}],"predecessor-version":[{"id":81,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/80\/revisions\/81"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/media\/20"}],"wp:attachment":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/media?parent=80"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/categories?post=80"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/tags?post=80"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}