{"id":66,"date":"2025-12-09T11:36:12","date_gmt":"2025-12-09T11:36:12","guid":{"rendered":"https:\/\/steadyrabbit.in\/blogs\/?p=66"},"modified":"2025-12-09T11:37:46","modified_gmt":"2025-12-09T11:37:46","slug":"production-ready-rag-vector-db-or-embedding-cache-where-does-your-bottleneck-live","status":"publish","type":"post","link":"https:\/\/steadyrabbit.in\/blogs\/production-ready-rag-vector-db-or-embedding-cache-where-does-your-bottleneck-live\/","title":{"rendered":"Production-Ready RAG: Vector DB or Embedding Cache\u2014Where Does Your Bottleneck Live?"},"content":{"rendered":"\n

Retrieval-Augmented Generation (RAG) fails in production when latency or cost blows up beyond the prototype. We A\/B-benchmarked three vector-store patterns\u2014managed Pinecone, self-hosted pgvector, and an in-process embedding cache (Redis)<\/strong>\u2014across one-million PDF chunks. Results: pgvector wins cost, Pinecone wins scale, and Redis cache annihilates 50 % of queries<\/strong> when corpus churn is low. Copy-paste Terraform, k6 load scripts, and Grafana dashboards included.<\/p>\n\n\n\n

Why \u201cJust Use a Vector DB\u201d Backfires <\/strong><\/h4>\n\n\n\n

Early RAG demos work because:<\/p>\n\n\n\n

    \n
  • Corpus < 5 K docs
    <\/li>\n\n\n\n
  • Prompt latency unmeasured
    <\/li>\n\n\n\n
  • Single-tenant traffic
    <\/li>\n<\/ul>\n\n\n\n

    In production you add multi-tenant shards, nightly doc ingests, and 20\u00d7 QPS spikes<\/strong>. Suddenly:<\/p>\n\n\n\n

      \n
    • Pinecone bill = half your OpenAI spend
      <\/li>\n\n\n\n
    • pgvector read locks cause timeout storms
      <\/li>\n\n\n\n
    • End-users wait 4 s for the answer they could have Googled
      <\/li>\n<\/ul>\n\n\n\n

      Goal:<\/strong> Find the sweet-spot:
      vector DB \u2192 corpus growth & multi-tenant; Redis cache \u2192 low-churn high-read workloads.<\/strong><\/p>\n\n\n\n

      Benchmark Setup <\/strong><\/h4>\n\n\n\n
      Parameter<\/strong><\/td>Value<\/strong><\/td><\/tr>
      Corpus<\/strong><\/td>1 M PDF chunks (avg 400 tokens) from SEC 10-K filings<\/td><\/tr>
      Queries<\/strong><\/td>10 K real question set from Edgar QA demo<\/td><\/tr>
      Compute<\/strong><\/td>EKS cluster (3 \u00d7 c7g.large), Redis cluster (3 \u00d7 cache.r6g.large), Pinecone S1 large<\/td><\/tr>
      Vector dims<\/strong><\/td>768 (E5-V2 embeddings)<\/td><\/tr>
      QPS Burst<\/strong><\/td>1 \u2192 200 req\/s in 30 s<\/td><\/tr>
      Metrics<\/strong><\/td>p95 latency, $\/1 K queries, recall@5, ingest TPS<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

      Terraform repo: github.com\/steadyrabbit\/rag-vector-bench (MIT).<\/p>\n\n\n\n

      Architecture Patterns Compared <\/strong><\/h4>\n\n\n\n
      Pattern<\/strong><\/td>Retrieval Flow<\/strong><\/td>When It Shines<\/strong><\/td><\/tr>
      Managed Vector DB (Pinecone)<\/strong><\/td>Client \u2192 index.query()<\/td>Multi-tenant, > 10 M vectors, auto-scale<\/td><\/tr>
      Self-Hosted pgvector<\/strong><\/td>Postgres + ivfflat<\/td>Corpus \u2264 5 M, DevOps SQL familiarity<\/td><\/tr>
      Embedding Cache (Redis)<\/strong><\/td>LRU cache on (query_hash \u2192 doc IDs) + upstream pgvector<\/td>Skewed queries, read-heavy traffic<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

      Recall parity:<\/strong> All patterns use the same HNSW\/IVF params to ensure fairness.<\/p>\n\n\n\n

      The Numbers <\/strong><\/h4>\n\n\n\n
      Metric<\/strong><\/td>Pinecone<\/strong><\/td>pgvector<\/strong><\/td>Redis Cache*<\/strong><\/td><\/tr>
      p95 Latency (no cache)<\/td>320 ms<\/strong><\/td>480 ms<\/td>N\/A<\/td><\/tr>
      p95 Latency (with cache)<\/td>260 ms<\/td>310 ms<\/td>110 ms<\/strong><\/td><\/tr>
      Cost \/ 1 K queries<\/td>$0.88<\/td>$0.27<\/strong><\/td>$0.34<\/td><\/tr>
      Ingest TPS<\/td>1 220<\/td>2 550<\/strong><\/td>2 550<\/td><\/tr>
      Recall@5<\/td>93 %<\/td>94 %<\/strong><\/td>94 % (cache hit)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

      *Redis fronting pgvector; 53 % queries cache-hit at 24 h TTL.<\/p>\n\n\n\n

      Key takeaway<\/strong><\/h4>\n\n\n\n

      Scale first? Use Pinecone.<\/em>
      <\/em>Cost first? Use pgvector.<\/em>
      <\/em>Read-skewed, low-churn? Layer Redis cache and halve latency.<\/em><\/p>\n\n\n\n

       Building the Redis Embedding Cache <\/strong><\/h4>\n\n\n\n

      5.1 Key Design<\/strong><\/p>\n\n\n\n

      css<\/p>\n\n\n\n

      CopyEdit<\/p>\n\n\n\n

      Key:  sha256(truncate(query, 350tkn) + “v1”)<\/p>\n\n\n\n

      TTL:  24h   (override on corpus update)<\/p>\n\n\n\n

      Value:  json.dumps({ “ids”: [123,456], “vec”: [0.12, …] })<\/p>\n\n\n\n

      Why store vector?<\/em> Re-ranking stage can bypass pgvector when cache hits.<\/p>\n\n\n\n

      5.2 Write-Back Flow<\/strong><\/p>\n\n\n\n

        \n
      1. Client query \u2192 Redis GET
        <\/li>\n\n\n\n
      2. MISS \u2192 pgvector ANN search
        <\/li>\n\n\n\n
      3. Return IDs + embeddings \u2192 Redis SETEX TTL 24 h
        <\/li>\n<\/ol>\n\n\n\n

        5.3 Invalidation<\/strong><\/p>\n\n\n\n

        EventBridge<\/em> fires on nightly DocIngest; Lambda deletes keys with prefix v1. Measured invalidation time: 18 s for 1 M keys.<\/p>\n\n\n\n

        5.4 Security<\/strong><\/p>\n\n\n\n

          \n
        • TLS in-transit
          <\/li>\n\n\n\n
        • ACL: cache role read\/write, app role read-only
          <\/li>\n\n\n\n
        • AES-GCM client-side encryption for PII chunks (optional)<\/li>\n<\/ul>\n\n\n\n

          Tuning pgvector for 5 M Vectors <\/strong><\/h4>\n\n\n\n

          Use ivfflat + HNSW<\/strong>
          <\/strong>
          <\/strong> sql
          CopyEdit
          CREATE INDEX idx_vec ON docs USING ivfflat (embedding vector_l2ops) WITH (lists=100);<\/p>\n\n\n\n

            \n
          1. <\/li>\n\n\n\n
          2. Set <\/strong>maintenance_work_mem<\/strong> to 4 GB for fast re-index.
            <\/li>\n\n\n\n
          3. Parallel search<\/strong> (max_parallel_workers_per_gather = 4).
            <\/li>\n\n\n\n
          4. VACUUM ANALYZE<\/strong> hourly; reduces bloat 18 %.
            <\/li>\n\n\n\n
          5. Shard by tenant<\/strong> when corpus > 5 M to avoid index spill.
            <\/li>\n<\/ol>\n\n\n\n

            Result: latency dropped from 620 \u2192 480 ms, ingest 1 \u2192 2.5 K TPS.<\/p>\n\n\n\n

            Cost Modeling Cheat-Sheet <\/strong><\/h4>\n\n\n\n
            Component<\/strong><\/td>Pinecone S1<\/strong><\/td>pgvector (RDS)<\/strong><\/td>Redis cache<\/strong><\/td><\/tr>
            Instance hrs<\/td>$0.35\/h<\/td>$0.19\/h<\/td>$0.25\/h<\/td><\/tr>
            Storage (500 GB)<\/td>Incl<\/td>$0.10\/GB-mo<\/td>$0.12\/GB-mo<\/td><\/tr>
            Data transfer<\/td>Incl internal<\/td>Incl<\/td>$0.09\/GB<\/td><\/tr>
            $\/1 K queries<\/strong> (200 req\/s burst)<\/td>$0.88<\/strong><\/td>$0.27<\/strong><\/td>$0.34<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

            Break-even vs Pinecone when queries < 10 M\/mo or<\/strong> ingest > 3 M vectors\/mo (pgvector cheaper).<\/p>\n\n\n\n

            Latency Heat Map (visual insight, \u2248 100 w)<\/strong><\/h4>\n\n\n\n

            (Describe chart)<\/em>
            <\/em> Y-axis burst QPS, X-axis corpus size. Regions:<\/p>\n\n\n\n

              \n
            • Green (\u2264 200 ms)<\/strong> \u2013 Redis cache up to 2 M corpus, 100 QPS.
              <\/li>\n\n\n\n
            • Amber (\u2264 350 ms)<\/strong> \u2013 pgvector 5 M corpus, 150 QPS.
              <\/li>\n\n\n\n
            • Red (> 350 ms)<\/strong> \u2013 Pinecone saves day > 5 M corpus & 150 QPS.
              <\/li>\n<\/ul>\n\n\n\n

              Add GIF in blog post to animate traffic spikes vs. latency.<\/p>\n\n\n\n

              Putting It Together in Terraform <\/strong><\/h4>\n\n\n\n

              hcl<\/p>\n\n\n\n

              CopyEdit<\/p>\n\n\n\n

              module “pgvector” {<\/p>\n\n\n\n

                source  = “terraform-aws-modules\/rds\/aws”<\/p>\n\n\n\n

                family  = “postgres15”<\/p>\n\n\n\n

                instance_class = “db.m6g.large”<\/p>\n\n\n\n

                engine_version = “15.3”<\/p>\n\n\n\n

                storage = 500<\/p>\n\n\n\n

                tags = { cost_center = “rag” }<\/p>\n\n\n\n

              }<\/p>\n\n\n\n

              module “redis_cache” {<\/p>\n\n\n\n

                source = “terraform-aws-modules\/elasticache\/aws”<\/p>\n\n\n\n

                node_type = “cache.r6g.large”<\/p>\n\n\n\n

                cluster_mode = true<\/p>\n\n\n\n

                num_node_groups = 2<\/p>\n\n\n\n

                replicas_per_node_group = 1<\/p>\n\n\n\n

              }<\/p>\n\n\n\n

              Outputs Service URL + secret ARN for CI pipeline.<\/p>\n\n\n\n

              Take-Home Checklist <\/strong><\/h4>\n\n\n\n
                \n
              1. Forecast<\/strong> corpus growth & QPS.
                <\/li>\n\n\n\n
              2. Prototype on pgvector; layer Redis cache for skewed reads.
                <\/li>\n\n\n\n
              3. Migrate to Pinecone when corpus > 5 M and<\/strong> multi-tenant burst > 150 QPS.
                <\/li>\n\n\n\n
              4. Monitor cache-hit %, p95 latency, and $ per 1 K queries.
                <\/li>\n\n\n\n
              5. Automate invalidation via event-driven doc ingest.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

                Retrieval-Augmented Generation (RAG) fails in production when latency or cost blows up beyond the prototype. We A\/B-benchmarked three vector-store patterns\u2014managed Pinecone, self-hosted pgvector, and an in-process embedding cache (Redis)\u2014across one-million PDF chunks. Results: pgvector wins cost, Pinecone wins scale, and Redis cache annihilates 50 % of queries when corpus churn is low. Copy-paste Terraform, k6 […]<\/p>\n","protected":false},"author":1,"featured_media":20,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-66","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml-best-practices"],"_links":{"self":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/66","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/comments?post=66"}],"version-history":[{"count":1,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/66\/revisions"}],"predecessor-version":[{"id":67,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/66\/revisions\/67"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/media\/20"}],"wp:attachment":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/media?parent=66"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/categories?post=66"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/tags?post=66"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}