{"id":77,"date":"2025-12-20T06:52:05","date_gmt":"2025-12-20T06:52:05","guid":{"rendered":"https:\/\/steadyrabbit.in\/blogs\/?p=77"},"modified":"2025-12-20T06:52:50","modified_gmt":"2025-12-20T06:52:50","slug":"from-notebook-to-prod-in-one-commit-mlflow-feature-store-ci-cd","status":"publish","type":"post","link":"https:\/\/steadyrabbit.in\/blogs\/from-notebook-to-prod-in-one-commit-mlflow-feature-store-ci-cd\/","title":{"rendered":"From Notebook to Prod in One Commit: MLflow + Feature Store CI\/CD"},"content":{"rendered":"\n

TL;DR (\u2248 85 words)<\/h4>\n\n\n\n

Notebooks rot, hand-clicked MLflow UI tags drift, and \u201cone-off\u201d Bash deploys become snowflakes. We\u2019ll turn that mess into a Git-ops pipeline<\/strong> where a single pull-request:<\/p>\n\n\n\n

    \n
  1. Runs unit tests on notebooks with Papermill.
    <\/li>\n\n\n\n
  2. Registers the model in MLflow.
    <\/li>\n\n\n\n
  3. Creates or updates features in a Feast store.
    <\/li>\n\n\n\n
  4. Deploys the model to a live endpoint (SageMaker, Vertex, or on-prem) via GitHub Actions.
    <\/li>\n<\/ol>\n\n\n\n

    End-to-end latency: \u2248 12 minutes<\/strong>; rollback in < 90 seconds. All YAML, Terraform, and Grafana dashboards included.<\/p>\n\n\n\n

    Why \u201cThrow It Over the Wall\u201d Still Rules ML\u00a0<\/h4>\n\n\n\n

    Most data-science orgs still:<\/p>\n\n\n\n

      \n
    • Export a model.pkl, ping MLOps, and wait days for infra tickets.
      <\/li>\n\n\n\n
    • Copy\/paste feature code between Airflow DAGs\u2014guaranteed drift.
      <\/li>\n\n\n\n
    • Discover too late that the \u201cdev\u201d feature uses log1p(x) while prod uses log10(x)\u2014hello, shadow drift.
      <\/li>\n<\/ul>\n\n\n\n

      The solution is Shift-Left MLOps<\/strong>: turn model, feature definitions, deployment infra, and monitoring into code committed with the PR<\/strong>. If the code changes, the pipeline enforces:<\/p>\n\n\n\n

        \n
      • Tests pass
        <\/li>\n\n\n\n
      • Model and features register
        <\/li>\n\n\n\n
      • Endpoint rolls forward\u2014or reverts
        <\/li>\n<\/ul>\n\n\n\n

        No ticket queues. No stale notebooks. Predictable releases.<\/p>\n\n\n\n

        Reference Stack\u00a0<\/h4>\n\n\n\n
        Layer<\/strong><\/td>Tool<\/strong><\/td>Why<\/strong><\/td><\/tr>
        Version Control<\/strong><\/td>GitHub \u2013 trunk-based<\/td>PR triggers Action<\/td><\/tr>
        Model Tracking<\/strong><\/td>MLflow 2.9<\/td>REST API + model lineage<\/td><\/tr>
        Feature Store<\/strong><\/td>Feast 0.37<\/td>Decouples online vs offline<\/td><\/tr>
        Training Notebook<\/strong><\/td>Jupyter + Papermill<\/td>Parameterised tests<\/td><\/tr>
        CI\/CD<\/strong><\/td>GitHub Actions + Terraform Cloud<\/td>Git-ops, rollback<\/td><\/tr>
        Serving<\/strong><\/td>AWS SageMaker Endpoint<\/td>Blue\/green & A\/B<\/td><\/tr>
        Monitoring<\/strong><\/td>Prometheus \/ Grafana<\/td>Drift & latency alerts<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

        (Swap SageMaker for Vertex AI or On-Prem KFServing by changing one Terraform module.)<\/em><\/p>\n\n\n\n

        Notebook Unit Tests\u00a0<\/h4>\n\n\n\n
        3.1 Papermill Parameter Test<\/h5>\n\n\n\n

        train.ipynb cell:<\/p>\n\n\n\n

        python<\/p>\n\n\n\n

        CopyEdit<\/p>\n\n\n\n

        epochs = int(params.get(“EPOCHS”, 10))<\/p>\n\n\n\n

        assert 1 <= epochs <= 100, “Epochs out of range”<\/p>\n\n\n\n

        GitHub Action step:<\/p>\n\n\n\n

        yaml<\/p>\n\n\n\n

        CopyEdit<\/p>\n\n\n\n

        – name: Execute notebook tests<\/p>\n\n\n\n

          run: |<\/p>\n\n\n\n

            papermill train.ipynb output.ipynb -p EPOCHS 5<\/p>\n\n\n\n

            papermill evaluate.ipynb output_eval.ipynb<\/p>\n\n\n\n

        Failures break the PR early\u2014before expensive GPUs spin.Median runtime:<\/em> 2 minutes on t3.large runner.<\/p>\n\n\n\n

        Model + Feature Registration\u00a0<\/h4>\n\n\n\n
        4.1 MLflow Registration<\/h5>\n\n\n\n

        bash<\/p>\n\n\n\n

        CopyEdit<\/p>\n\n\n\n

        mlflow models register \\<\/p>\n\n\n\n

          –model-uri “runs:\/${RUN_ID}\/model” \\<\/p>\n\n\n\n

          –name “credit_risk_classifier”<\/p>\n\n\n\n

        Action parses run ID from Papermill output. Tag model with Git SHA and dataset hash for lineage.<\/p>\n\n\n\n

        4.2 Feast Apply in CI<\/h5>\n\n\n\n

        features\/credit_risk.py<\/p>\n\n\n\n

        python<\/p>\n\n\n\n

        CopyEdit<\/p>\n\n\n\n

        from feast import FeatureView, Entity, Field<\/p>\n\n\n\n

        from feast.types import Float32, Int64<\/p>\n\n\n\n

        customer = Entity(name=”customer_id”, value_type=Int64)<\/p>\n\n\n\n

        credit_features = FeatureView(<\/p>\n\n\n\n

            name=”credit_features”,<\/p>\n\n\n\n

            entities=[customer],<\/p>\n\n\n\n

            ttl=86400,<\/p>\n\n\n\n

            schema=[<\/p>\n\n\n\n

                Field(name=”avg_balance_30d”, dtype=Float32),<\/p>\n\n\n\n

                Field(name=”max_txn_amt_30d”, dtype=Float32),<\/p>\n\n\n\n

            ],<\/p>\n\n\n\n

            online=True,<\/p>\n\n\n\n

        )<\/p>\n\n\n\n

        Action step:<\/p>\n\n\n\n

        yaml<\/p>\n\n\n\n

        CopyEdit<\/p>\n\n\n\n

        – name: Feast apply<\/p>\n\n\n\n

          run: feast apply<\/p>\n\n\n\n

        Feast bumps version if schema changes, ensuring online\/offline parity.<\/p>\n\n\n\n

        Automated Deployment via Terraform & GitHub Actions\u00a0<\/h4>\n\n\n\n

        yaml<\/p>\n\n\n\n

        CopyEdit<\/p>\n\n\n\n

        env:<\/p>\n\n\n\n

          MODEL_NAME: credit_risk_classifier<\/p>\n\n\n\n

          MODEL_STAGE: Staging<\/p>\n\n\n\n

        jobs:<\/p>\n\n\n\n

          deploy:<\/p>\n\n\n\n

            needs: [test, register]<\/p>\n\n\n\n

            runs-on: ubuntu-latest<\/p>\n\n\n\n

            steps:<\/p>\n\n\n\n

              – uses: actions\/checkout@v4<\/p>\n\n\n\n

              – name: Terraform plan<\/p>\n\n\n\n

                run: terraform -chdir=infra plan -input=false<\/p>\n\n\n\n

              – name: Terraform apply<\/p>\n\n\n\n

                run: terraform -chdir=infra apply -auto-approve -input=false<\/p>\n\n\n\n

        infra\/main.tf:<\/p>\n\n\n\n

        hcl<\/p>\n\n\n\n

        CopyEdit<\/p>\n\n\n\n

        module “sagemaker_model” {<\/p>\n\n\n\n

          source     = “terraform-aws-modules\/sagemaker\/aws\/\/modules\/model”<\/p>\n\n\n\n

          name       = var.model_name<\/p>\n\n\n\n

          primary_container = {<\/p>\n\n\n\n

            image = “763104351884.dkr.ecr.us-east-1.amazonaws.com\/xgboost:1.5-1”<\/p>\n\n\n\n

            model_data_url = “${var.model_s3_path}”<\/p>\n\n\n\n

          }<\/p>\n\n\n\n

        }<\/p>\n\n\n\n

        module “sagemaker_endpoint” {<\/p>\n\n\n\n

          source = “terraform-aws-modules\/sagemaker\/aws\/\/modules\/endpoint”<\/p>\n\n\n\n

          name   = “${var.model_name}-ep”<\/p>\n\n\n\n

          variant_weight = 0.10   # blue\/green\u201410 % to new model<\/p>\n\n\n\n

        }<\/p>\n\n\n\n

        On successful health probes (p95 latency < 300 ms & 0 \u2265 error_rate < 1 %), traffic shifts to 100 %; else Terraform rollbacks.Median deploy time:<\/em> 6 minutes (model S3 pull largest slice).<\/p>\n\n\n\n

        Monitoring & Drift Alerts\u00a0<\/h4>\n\n\n\n

        Prometheus agent on endpoint emits:<\/p>\n\n\n\n

          \n
        • inference_latency_ms
          <\/li>\n\n\n\n
        • prediction_drift_psi (population stability index)
          <\/li>\n\n\n\n
        • feature_null_ratio
          <\/li>\n<\/ul>\n\n\n\n

          Grafana threshold panel:<\/p>\n\n\n\n

            \n
          • Red light if prediction_drift_psi > 0.2 for 3 consecutive minutes.
            <\/li>\n<\/ul>\n\n\n\n

            Slack alert to #mlops-critical; Terraform -target=endpoint rollback plan fires if red > 10 min.<\/p>\n\n\n\n

            Time & Cost Benchmarks\u00a0<\/h4>\n\n\n\n
            Stage<\/strong><\/td>Median Time<\/strong><\/td>AWS Cost \/ run<\/strong><\/td><\/tr>
            Notebook tests<\/td>2 m<\/td>$0.02<\/td><\/tr>
            Build + push model<\/td>3 m<\/td>$0.05<\/td><\/tr>
            Feature apply<\/td>1 m<\/td>$0.004<\/td><\/tr>
            Terraform deploy<\/td>6 m<\/td>$0.12<\/td><\/tr>
            Total<\/strong><\/td>12 m<\/strong><\/td>$0.19<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

            Rollback (blue\/green revert) costs $0.03 & takes 80 s.<\/p>\n\n\n\n

            Real-World Impact (FinTech Credit-Risk Model)\u00a0<\/h4>\n\n\n\n

            Before pipeline<\/em>: Model refresh every 3 months, drift alerts manual, rollbacks 4 hours.
            After CI\/CD<\/em>:<\/p>\n\n\n\n

              \n
            • Weekly model refresh \u2192 credit-risk AUC +4 pp.
              <\/li>\n\n\n\n
            • p95 latency held steady at 220 ms.
              <\/li>\n\n\n\n
            • Rollback tested live \u2014 production revert in 80 s, zero user impact.
              <\/li>\n\n\n\n
            • Compliance audit passed 1st try\u2014full lineage via MLflow tags.<\/li>\n<\/ul>\n\n\n\n

              Pitfalls & Pro Tips\u00a0<\/h4>\n\n\n\n
              Pitfall<\/strong><\/td>Fix<\/strong><\/td><\/tr>
              \u201cMLflow UI tag drift\u201d<\/td>Enforce tags via mlflow.set_tags() inside notebook; fail Action if missing.<\/td><\/tr>
              Feast online\/offline skew<\/td>Schedule hourly feast materialize-incremental $(date +%s)<\/td><\/tr>
              Terraform apply timeout 30 m<\/td>Use EFS-backed model; warm container shortcut.<\/td><\/tr>
              Feature DAG race<\/td>Serialize Airflow tasks that mutate same entity using task-level mutex.<\/td><\/tr>
              CI bill shock<\/td>Self-host GitHub runner spot fleet; cost drops 60 %.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

              Adoption Roadmap\u00a0<\/h4>\n\n\n\n
              Sprint<\/strong><\/td>Milestone<\/strong><\/td><\/tr>
              1<\/td>Add Papermill tests + MLflow tracking<\/td><\/tr>
              2<\/td>Feast offline + online stores, feast apply in CI<\/td><\/tr>
              3<\/td>Terraform SageMaker blue\/green deploy<\/td><\/tr>
              4<\/td>Prometheus drift metrics + auto-rollback<\/td><\/tr>
              5<\/td>Merge notebooks into repo trunk; freeze ad-hoc JupyterHub<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
              \n\n\n\n

              Take-Home Checklist\u00a0<\/h4>\n\n\n\n
                \n
              1. Parameter-test notebooks with Papermill.
                <\/li>\n\n\n\n
              2. Register models and<\/strong> features in the same PR.
                <\/li>\n\n\n\n
              3. Deploy via Terraform blue\/green; rollback on health fail.
                <\/li>\n\n\n\n
              4. Monitor drift & latency in Grafana; auto-alert Slack.
                <\/li>\n\n\n\n
              5. Audit lineage with MLflow tags & Git SHA.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

                TL;DR (\u2248 85 words) Notebooks rot, hand-clicked MLflow UI tags drift, and \u201cone-off\u201d Bash deploys become snowflakes. We\u2019ll turn that mess into a Git-ops pipeline where a single pull-request: End-to-end latency: \u2248 12 minutes; rollback in < 90 seconds. All YAML, Terraform, and Grafana dashboards included. Why \u201cThrow It Over the Wall\u201d Still Rules ML\u00a0 […]<\/p>\n","protected":false},"author":1,"featured_media":20,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-77","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml-best-practices"],"_links":{"self":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/77","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/comments?post=77"}],"version-history":[{"count":2,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/77\/revisions"}],"predecessor-version":[{"id":79,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/posts\/77\/revisions\/79"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/media\/20"}],"wp:attachment":[{"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/media?parent=77"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/categories?post=77"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/steadyrabbit.in\/blogs\/wp-json\/wp\/v2\/tags?post=77"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}