TL;DR
- Treat the service repo as a product: it should own its Dockerfile, CI workflow, Helm chart, and runtime contract.
- Build immutable images in GitHub Actions, publish them to GHCR with
GITHUB_TOKEN, and use Buildx cache withtype=ghato keep builds fast. - If you use ArgoCD, keep Git as the source of truth. A registry push alone is not a GitOps deployment strategy unless you add image automation that writes back to Git.
- Use Helm to package the service, External Secrets Operator to materialize secrets, and IRSA to grant AWS permissions without long-lived AWS keys in pods.
- Use
startupProbeandreadinessProbedeliberately for slow-booting AI services that load models, warm connections, or run migrations. - Sign release artifacts and publish provenance so your pipeline is not only fast, but also auditable.
Modern deployment guides often stop too early. They explain Kubernetes, or Docker, or GitHub Actions, but not the handoff between them. In practice, that handoff is where most production friction lives.
This article focuses on the service-repository layer of deployment for an AI-powered SaaS backend: the part that starts with application code and ends with a healthy pod receiving traffic. It complements the infrastructure-first angle of Production GitOps: Terraform, Helm, and ArgoCD for Node.js APIs and Agno Agents by zooming in on one deployable service.
The example is inspired by a fictional AI SaaS platform, but the operational guidance below was reviewed against official documentation on March 24, 2026.
The Service We Are Deploying
Assume one backend service in a broader platform:
pulse-api/
├── app/
│ ├── main.py
│ ├── agents/
│ ├── api/
│ ├── db/
│ └── config.py
├── charts/pulse-api/
├── .github/workflows/
├── Dockerfile
├── compose.yaml
├── pyproject.toml
└── scripts/entrypoint.sh
This service owns four things:
- The application code.
- The container image definition.
- The deployment manifest contract.
- The automation that turns a commit into a releasable artifact.
That ownership boundary matters. When the same repo defines the runtime image, the health endpoints, and the Helm values schema, you reduce the number of places a deployment can silently drift.
1. Build a Container for Repeatability, Not Just for Local Success
For an AI SaaS backend, the container has to do more than “run Python.” It needs predictable dependency resolution, safe filesystem behavior, and a startup path that matches production realities.
A good service Dockerfile usually optimizes for three things:
- Stable tooling versions.
- Efficient build layers.
- A non-root runtime.
FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:0.6.6 /uv /uvx /bin/
WORKDIR /app
ENV PYTHONPATH=/app
RUN apt-get update && apt-get install -y --no-install-recommends \
git curl libcairo2 libpango-1.0-0 && \
rm -rf /var/lib/apt/lists/*
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev
COPY . .
RUN groupadd -r pulse && useradd -r -g pulse -d /app pulse && \
chown -R pulse:pulse /app
USER pulse
EXPOSE 8000
ENTRYPOINT ["bash", "/app/scripts/entrypoint.sh"]
CMD ["uv", "run", "python", "app/main.py"]
The big ideas are simple:
- Copy dependency manifests before application code so dependency installs stay cached longer.
- Pin build tooling versions instead of depending on
latest. - Drop root privileges in the image so Kubernetes can enforce
runAsNonRoot.
For AI services specifically, I also like keeping the entrypoint tiny and explicit. If you need to materialize a private key, convert an env var into a file, or create a runtime-only directory, do it in one auditable place:
#!/bin/bash
set -e
if [ -n "$SNOWFLAKE_PRIVATE_KEY" ]; then
mkdir -p "$(dirname "$SNOWFLAKE_PRIVATE_KEY_PATH")"
printf '%s\n' "$SNOWFLAKE_PRIVATE_KEY" > "$SNOWFLAKE_PRIVATE_KEY_PATH"
chmod 600 "$SNOWFLAKE_PRIVATE_KEY_PATH"
unset SNOWFLAKE_PRIVATE_KEY
fi
exec "$@"
That kind of script is boring, and boring is exactly what you want in production startup code.
2. CI Should Produce Immutable Artifacts
A service repo usually needs three automation lanes:
pull_request: prove the branch builds and optionally publish a preview image.main: publish the integration image used by development or staging.tag: publish a release image and related release metadata.
GitHub’s own docs show the standard GHCR pattern: log in with docker/login-action, use ${{ secrets.GITHUB_TOKEN }}, and grant packages: write on the job. That is cleaner than introducing a personal access token for same-repo publication.
name: Build Main
on:
push:
branches: [main]
permissions:
contents: read
packages: write
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/build-push-action@v6
with:
context: .
push: true
tags: ghcr.io/acme/pulse-api:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
Two details are worth preserving from the original draft:
- Buildx cache with
type=ghais a pragmatic default on GitHub-hosted runners. - Single-platform builds are fine when your cluster only runs
linux/amd64.
But there is one improvement I would make immediately:
Do not let long-lived mutable tags such as development or latest become your core deployment identity. Publish them if they help humans, but let your deployment system converge on an immutable tag or digest.
Why? Because immutable image references make rollbacks, debugging, provenance, and incident review much easier. GitHub’s container registry docs also explicitly document pulling by digest when you want an exact image.
3. Helm Packages the Runtime Contract
Helm’s job is not “Kubernetes templating because YAML is annoying.” Its real job is to define the contract between the service and each environment.
Typical chart structure:
charts/pulse-api/
├── Chart.yaml
├── values.yaml
└── templates/
├── deployment.yaml
├── service.yaml
├── ingress.yaml
├── serviceaccount.yaml
├── externalsecret.yaml
├── pdb.yaml
└── hpa.yaml
The Deployment template should make a few things obvious:
- Which image runs.
- Which non-secret env vars are configured in Git.
- Which secrets are injected at runtime.
- Which probes determine startup and readiness.
- Which security settings the pod must satisfy.
containers:
- name: pulse-api
image: {{ include "pulse-api.image" . }}
env:
- name: PORT
value: {{ .Values.service.port | quote }}
envFrom:
- secretRef:
name: {{ include "pulse-api.fullname" . }}
startupProbe:
httpGet:
path: /health
port: http
failureThreshold: 30
periodSeconds: 5
readinessProbe:
httpGet:
path: /health
port: http
periodSeconds: 10
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
Helm also makes it straightforward to publish charts as OCI artifacts. The official Helm docs support helm push to an oci:// registry reference, with chart name and tag inferred from the packaged chart metadata.
If your team likes a release model where the application image and chart move together, OCI-backed charts are a clean fit.
4. The GitOps Handoff: ArgoCD Should Track Git
This is the most important audit correction I made while adapting the original draft.
It is common to describe a flow like this:
CI builds image -> image lands in registry -> ArgoCD notices -> cluster updates
That is not the baseline GitOps model.
ArgoCD’s own documentation is clear: automated sync happens when ArgoCD detects a difference between the desired manifests in Git and the live cluster state. The intended pattern is that CI changes Git, and ArgoCD reconciles from there.
So the cleaner flow is:
commit code
-> CI builds image
-> CI writes immutable image tag or digest into Git-tracked manifests
-> ArgoCD detects Git change
-> ArgoCD syncs cluster
If you want registry-driven image promotion, add a dedicated layer such as Argo CD Image Updater and configure it to write changes back to Git. Its docs explicitly support a git write-back method for Helm and Kustomize applications.
That distinction matters because it preserves the main GitOps benefit: anyone can inspect Git and know exactly what the cluster is supposed to run.
5. Secrets: Keep Discovery Dynamic, Access Least-Privilege
For a modern AI SaaS backend, secrets often include:
- database credentials
- provider API keys
- OAuth client secrets
- observability tokens
- private keys
The pattern I like here is:
source of truth -> cloud secret manager -> External Secrets Operator -> Kubernetes Secret -> pod env
External Secrets Operator supports spec.dataFrom.find to fetch multiple secrets into a single Kubernetes Secret, and its rewrite support can merge JSON payloads with the Extract strategy. That makes tag-based discovery genuinely useful for service-oriented secret grouping.
Example:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
spec:
refreshInterval: 10m
secretStoreRef:
name: pulse-api
kind: SecretStore
target:
name: pulse-api
dataFrom:
- find:
tags:
service-pulse-api: enabled
rewrite:
- merge:
strategy: Extract
into: ""
Why this pattern scales:
- The deployment template does not need a new line for every secret.
- Secret rotation happens outside the application image.
- The operator can refresh the Kubernetes Secret periodically.
But do not stop at secret injection. Your pod also needs AWS permissions for the resources it calls. On EKS, IRSA is still a strong default for this. AWS documents the model clearly: associate an IAM role with a Kubernetes service account so pods can obtain scoped AWS credentials without baking long-lived credentials into the container.
That gives you a much better security posture than shipping AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY around your environment.
6. Startup, Health, and Migrations Need Different Rules
AI services are often slow starters for legitimate reasons:
- database pool initialization
- cache warmup
- model or embedding client bootstrapping
- integration checks
- schema migrations
Kubernetes distinguishes these concerns for a reason:
startupProbeprotects slow initial boot.readinessProbegates traffic.livenessProbedetects a stuck process later in the container lifecycle.
The Kubernetes docs explicitly recommend using startupProbe with failureThreshold * periodSeconds long enough to cover worst-case startup time. They also make clear that a failed readiness probe removes the pod from Service traffic without killing it.
For AI backends, that usually means:
- generous startup windows
- strict readiness checks
- conservative liveness probes
What About Running Migrations on Startup?
This is where practical engineering and architectural purity usually collide.
Running migrations at app startup is attractive because it reduces operational moving parts. Ship the image, boot the container, let it converge the schema, start serving traffic.
That can be completely reasonable for a small team and a low-chaos deployment model.
But it comes with tradeoffs:
- multiple replicas may race to migrate
- long migrations stretch startup time
- rollout safety becomes tightly coupled to schema change behavior
My rule of thumb is:
- Early stage: startup migrations can be acceptable if the migration surface is small and well understood.
- Later stage: move to a dedicated migration job or enforce a database advisory lock so one actor owns schema change execution.
The source draft was directionally right to call this a simplicity tradeoff. I would just avoid presenting it as the universal default.
7. Release Hardening: Signing and Provenance
A mature deployment pipeline should answer two questions:
- What artifact did we deploy?
- Can we prove how it was built?
For the first, sign release artifacts. Sigstore’s Cosign supports keyless container signing using OIDC-backed ephemeral credentials, including GitHub identities.
For the second, publish provenance. The SLSA project documents GitHub Actions builders and generators for producing signed provenance for container workflows, and GitHub also documents artifact attestations for build provenance.
That means a production-grade release flow can look like this:
git tag v1.5.0
-> build image
-> push image
-> package chart
-> sign image
-> publish provenance / attestation
-> update deployment manifests
-> ArgoCD sync
You do not need all of that on day one. But if you expect enterprise customers, regulated environments, or even just a future security questionnaire, provenance becomes a lot easier to add early than retroactively.
8. A Practical End-to-End Flow
Here is the version of the pipeline I would actually recommend for most teams:
Developer merges PR
-> GitHub Actions builds image
-> Push image to GHCR with immutable tag
-> CI updates Helm values or Argo CD Image Updater writes back to Git
-> ArgoCD detects Git change
-> External Secrets Operator refreshes referenced secrets as needed
-> New pod starts
-> startupProbe gives it time to initialize
-> readinessProbe flips healthy
-> Service sends traffic
That gives you:
- reproducible images
- visible deployment intent in Git
- clean separation between config and secrets
- safer cloud access through service-account identity
- fewer “why did this pod restart during rollout?” mysteries
9. What I Would Keep, and What I Would Change
The original source draft had strong instincts. I would absolutely keep these ideas:
- one service repo owning its app, image, chart, and workflows
- non-root containers by default
- pinned tooling versions
- tag-based secret discovery
- a dedicated release workflow for promoted builds
But after auditing it against the surrounding project and current docs, I would change these points:
- Present ArgoCD as Git-driven, not registry-driven by default.
- Prefer immutable image tags or digests over relying on floating tags as the deployment identity.
- Treat startup migrations as a conscious convenience tradeoff, not a forever architecture.
- Frame Cosign and provenance as artifact-level release hardening, not just “extra steps for later.”
That turns the guide from a good fictional story into a stronger production pattern.
Sources
Verified on March 24, 2026 against primary documentation:
- GitHub Docs: Publishing Docker images with GitHub Actions
- GitHub Docs: Working with the Container registry
- Docker Docs: GitHub Actions cache backend
- Helm Docs: Use OCI-based registries
- Argo CD Docs: Automated Sync Policy
- Argo CD Image Updater Docs: Update methods
- External Secrets Operator: Fetching information from multiple secrets
- External Secrets Operator: Rewriting keys from
dataFrom - External Secrets Operator API:
ExternalSecret - Kubernetes Docs: Configure liveness, readiness, and startup probes
- Amazon EKS Docs: IAM roles for service accounts
- Sigstore Docs: Signing containers with Cosign
- SLSA: General availability of SLSA 3 Container Generator for GitHub Actions
- GitHub Docs: Artifact attestations and build provenance
This article uses a fictional company and repository names to make the deployment story concrete. The specific architecture examples are illustrative, while the platform and workflow guidance above is grounded in the linked documentation.