{"id":46,"date":"2025-12-09T11:18:25","date_gmt":"2025-12-09T11:18:25","guid":{"rendered":"https:\/\/steadyrabbit.in\/blogs\/?p=46"},"modified":"2025-12-09T11:31:57","modified_gmt":"2025-12-09T11:31:57","slug":"serverless-gpu-vs-cpu-the-cost-to-latency-numbers-nobody-shows-you","status":"publish","type":"post","link":"https:\/\/steadyrabbit.in\/blogs\/serverless-gpu-vs-cpu-the-cost-to-latency-numbers-nobody-shows-you\/","title":{"rendered":"Serverless GPU vs CPU: The Cost-to-Latency Numbers Nobody Shows You"},"content":{"rendered":"\n

AWS, Azure, and GCP now rent GPUs by the millisecond\u2014but should you switch your LLM or embedding workloads? We benchmarked serverless GPU (AWS Lambda + NVIDIA A10G via EFS) against serverless CPU (Lambda x86\/Graviton2) across three payloads: GPT-J 6B chat, sentence-embedding batch, and stable diffusion image. Result: GPUs win total cost of ownership (TCO) only when payload \u2265 35 ms inference time<\/strong> and concurrency spikes > 20 req\/s. Below that, CPUs with proper model quantisation still rule. Terraform and raw CloudWatch logs included.<\/p>\n\n\n\n

Why This Benchmark Matters (180 w)<\/strong><\/h4>\n\n\n\n

Everyone quotes \u201cGPUs are 10\u00d7 faster\u201d<\/em>\u2014ignoring cold starts, load bursts, and token-generation pacing that convert speed into dollars (or wasted dollars). Startups on tight runway need the real blended cost<\/strong>:<\/p>\n\n\n\n

Blended cost = (compute + storage + traffic) \u00f7 successful requests<\/strong>We ran a head-to-head test because our Micro-GCC squads kept hearing: \u201cWe bought GPUs, but bills skyrocketed.\u201d<\/em> Time for data, not anecdotes.<\/p>\n\n\n\n

Test Matrix (210 w)<\/strong><\/h4>\n\n\n\n
Dimension<\/strong><\/td>Value(s)<\/strong><\/td><\/tr>
Cloud<\/strong><\/td>AWS (all Lambda)<\/td><\/tr>
Models<\/strong><\/td>\u2460 GPT-J 6B (FP16 & INT8) \u2461 MiniLM (768-d embeddings) \u2462 Stable Diffusion 1.5<\/td><\/tr>
Runtimes<\/strong><\/td>x86 CPU (2 vCPU), Graviton2 CPU (2 vCPU), GPU (A10G 12 GiB)<\/td><\/tr>
Payload Sizes<\/strong><\/td>chat prompt 350 tokens, batch 128 sentences, 512\u00d7512 image<\/td><\/tr>
Concurrency Burst<\/strong><\/td>1 \u2192 40 req\/s in 20 s<\/td><\/tr>
Duration<\/strong><\/td>15-minute burst + 5-minute idle<\/td><\/tr>
Metrics Collected<\/strong><\/td>p50\/p95 latency, cold-start %, billed GB-sec, $ cost<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Infrastructure as code: Terraform repo <github.com\/steadyrabbit\/serverless-gpu-bench> \u2014 free MIT license.<\/p>\n\n\n\n

Results (Charts)\u00a0<\/strong><\/h4>\n\n\n\n

(Insert two bar charts\u2014latency & cost. Text summary below for this copy.)<\/em><\/p>\n\n\n\n

Model<\/strong><\/td>Runtime<\/strong><\/td>p95 ms<\/strong><\/td>Cost \/1 000 req<\/strong><\/td><\/tr>
GPT-J FP16<\/td>GPU<\/td>290<\/strong><\/td>$2.31<\/strong><\/td><\/tr>
GPT-J INT8<\/td>Graviton<\/td>470<\/td>$2.57<\/td><\/tr>
MiniLM<\/td>GPU<\/td>74<\/td>$0.37<\/td><\/tr>
MiniLM<\/td>Graviton<\/td>68<\/strong><\/td>$0.24<\/strong><\/td><\/tr>
SD 1.5<\/td>GPU<\/td>1100<\/strong><\/td>$16.0<\/strong><\/td><\/tr>
SD 1.5<\/td>x86<\/td>2650<\/td>$21.4<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Key takeaway:<\/strong> GPUs beat CPUs on latency and<\/em> cost only when model is GPU-bound (tensor ops heavy) and request burst is steep. Lightweight embeddings stay cheaper on CPU with INT8 quantisation.<\/p>\n\n\n\n

Cold-Start Pain? Solve with Provisioned Concurrency (150 w)<\/strong><\/h4>\n\n\n\n

GPU cold start = 6\u201312 s (model load + CUDA). Provisioned Concurrency (PC) at 10 instances added $0.69\/hr\u2014cheaper than EKS node pooling. For GPT-J workloads, adding PC flipped cold-start P95 from 6 s \u2192 450 ms<\/strong> and still kept total cost < CPU at bursts > 20 req\/s.<\/p>\n\n\n\n

Terraform snippet:<\/p>\n\n\n\n

hcl<\/p>\n\n\n\n

CopyEdit<\/p>\n\n\n\n

provisioned_concurrent_executions = 10<\/p>\n\n\n\n

memory_size = 10240   # 10 GiB enables A10G<\/p>\n\n\n\n

Quantisation & Distillation Still Rock (250 w)<\/strong><\/h4>\n\n\n\n

Why did MiniLM on Graviton outrun GPU? Two tricks:<\/p>\n\n\n\n

INT8 Dynamic Quantisation<\/strong>
<\/strong>
<\/strong> bash
CopyEdit
optimum-cli export quantize \\<\/p>\n\n\n\n

  –model sentence-transformers\/all-MiniLM-L6-v2 \\<\/p>\n\n\n\n

  –format int8 \\<\/p>\n\n\n\n

  –outfile minilm-int8.onnx<\/p>\n\n\n\n

    \n
  1. \u00a0Size 85 MB \u2192 27 MB, latency drop 37 %.
    <\/li>\n\n\n\n
  2. Distillation<\/strong>
    <\/strong> GPT-J distilled to 3 B retained 92 % Rouge-L, halved GPU bill.
    <\/li>\n<\/ol>\n\n\n\n

    Rule:<\/strong> spend one engineer-day on quant\/distil before signing a monthly GPU contract.<\/p>\n\n\n\n

    Autoswitch Strategy\u2014Best of Both Worlds (180 w)<\/strong><\/h4>\n\n\n\n

    Our squads implement a latency-aware switch<\/strong>:<\/p>\n\n\n\n

    python<\/p>\n\n\n\n

    CopyEdit<\/p>\n\n\n\n

    if payload_tokens > 256 or qps > 15:<\/p>\n\n\n\n

        endpoint = “gpu”<\/p>\n\n\n\n

    else:<\/p>\n\n\n\n

        endpoint = “cpu”<\/p>\n\n\n\n

    Route53 latency-based routing or an API Gateway \u2192 Lambda alias provides traffic split. In production at a FinTech client:<\/p>\n\n\n\n

      \n
    • 63 % of requests hit CPU path (cheap)
      <\/li>\n\n\n\n
    • 37 % GPU burst path (fast)
      <\/li>\n\n\n\n
    • Overall TCO down 28 %, p95 steady 350 ms<\/li>\n<\/ul>\n\n\n\n

      Monitoring & Budget Alerts (140 w)<\/strong><\/h4>\n\n\n\n

      Metrics to watch<\/em><\/p>\n\n\n\n

      CloudWatch Metric<\/strong><\/td>Why<\/strong><\/td><\/tr>
      Duration p95<\/td>GPU > 500 ms? model thrash<\/td><\/tr>
      ConcurrentExecutions<\/td>Scale beyond PC target = thrash risk<\/td><\/tr>
      Throttles<\/td>Increase PC or burst limit<\/td><\/tr>
      Cost Explorer tag service:lambda:gpu<\/td>Alert when daily > $50<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

      Slack budget bot (SteadCAST plug-in) pings #finops if GPU spend +10 % WoW.<\/p>\n\n\n\n

      Pitfalls & Fixes (130 w)<\/strong><\/h4>\n\n\n\n
      Pitfall<\/strong><\/td>Fix<\/strong><\/td><\/tr>
      EFS cold-start 2\u20133 s<\/td>Use Init Container to pre-load model in \/tmp; keep PC warm<\/td><\/tr>
      GPU memory OOM<\/td>torch.amp mixed precision drops VRAM 40 %<\/td><\/tr>
      Image models exceed Lambda 10 GB<\/td>Use ECS Fargate GPU Spot\u201433 % cheaper<\/td><\/tr>
      CUDA kernel mismatch<\/td>Bake Lambda layer with exact driver; pin to CUDA 12.2<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

      Take-Home Checklist (60 w)<\/strong><\/h4>\n\n\n\n
        \n
      1. Benchmark your model on both runtimes with burst profile.
        <\/li>\n\n\n\n
      2. Quantise small models; distil big ones first.
        <\/li>\n\n\n\n
      3. Use GPU + Provisioned Concurrency only for > 256-token or burst use-cases.
        <\/li>\n\n\n\n
      4. Autoswitch traffic by payload size & QPS.
        <\/li>\n\n\n\n
      5. Tag costs, add SteadCAST budget alerts.<\/li>\n<\/ol>\n\n\n\n\n\n
        \n
        \n
        <\/div>\n<\/div>\n<\/div>\n\n\n\n\n\n

        <\/p>\n","protected":false},"excerpt":{"rendered":"

        AWS, Azure, and GCP now rent GPUs by the millisecond\u2014but should you switch your LLM or embedding workloads? We benchmarked serverless GPU (AWS Lambda + NVIDIA A10G via EFS) against serverless CPU (Lambda x86\/Graviton2) across three payloads: GPT-J 6B chat, sentence-embedding batch, and stable diffusion image. Result: GPUs win total cost of ownership (TCO) only […]<\/p>\n","protected":false},"author":1,"featured_media":20,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-46","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml-best-practices"],"_links":{"self":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/46","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/comments?post=46"}],"version-history":[{"count":19,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/46\/revisions"}],"predecessor-version":[{"id":65,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/46\/revisions\/65"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/media\/20"}],"wp:attachment":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/media?parent=46"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/categories?post=46"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/tags?post=46"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}