Build for the Leak: A Secret-Rotation Stack for AI Apps

TL;DR

• A leaked value works until something ends it: a revocation, a lease expiry, or a rotation that lands first. In one 2026 incident, an administrative key for an internal LiteLLM platform was committed to a public repository on 2026-05-27, flagged for revocation, and still worked when used on 2026-10-10, 136 days later (LiteLLM postmortem).
• Remove standing secrets first, meaning values that stay valid until someone revokes them. GitHub jobs get short-lived OIDC tokens, revocable secrets stay out of Railway variables, and Terraform reads secrets as ephemeral values so they never reach state.
• Then rotate what remains, choosing a mechanism per secret: OIDC, a dynamic lease, dual-phase rotation, or a static value with a cap and an expiry.
• Three things decide the design and are unverified: whether Infisical can reach a private Postgres for leases, whether the app drops a revoked lease within one poll, and whether gateways need Advanced or Enterprise. The drill at the end tests the first two.
• Plan gates set the cost. Dynamic secrets need Infisical Advanced, approvals need Infisical Enterprise, and required reviewers on a private repository need GitHub Enterprise.

Thesis ([INFERENCE], editorial judgment): fewer standing secrets beat faster rotation. Falsifier: if a revoked lease keeps working past one poll, leases bound exposure no better than a rotation schedule, and this design is wrong.

We’ll work through one concrete stack: a Railway app with a private Postgres, OpenRouter keys, an Auth0 login, and an Agno AgentOS service, deployed from GitHub Actions. Infisical holds the values; Terraform holds the topology. You’ll leave with a mechanism per secret, a CI workflow and Terraform config to adapt, and a drill that can prove the design wrong. For secrets inside an agent’s process, read Keep Secrets Out of the Agent. For database grants and row-level security, read Preparing Databases for Secure AI Agents.

A leaked value works until something ends it

A rotation schedule helps only if it fires before the attacker acts, and nobody knows when that is. In each incident below, a standing value sat where an intruder could reach it and stayed valid until someone ended it.

• Railway, October 2026. Railway said that at most 1.85% of workspaces, about 100,000, had some environment variables potentially compromised. It told customers who signed up before 2026-10-21 to rotate their stored keys, and it advised removing public network access to databases (Railway preliminary report, 2026-10-22, primary).
• LiteLLM, 2026. A configuration file committed on 2026-05-27 to a public repository contained an administrative key for an internal LiteLLM Agent Platform deployment. The team flagged the key for revocation. On 2026-10-10 the attacker used it, and the platform returned a GitHub CLI token belonging to an organization owner (LiteLLM postmortem, primary).
• Stolen GitHub tokens. Tokens with write access read workflow secrets. Wiz’s advice: required reviewers, and OIDC trust limited to protected branches or environments (Wiz, primary; reported by The Hacker News, 2026-10-26, secondary).

Where each secret sits

The stack at a glance

flowchart LR
  GHA["GitHub Actions job<br/>environment: production"] -- "OIDC token" --> INF["Infisical"]
  TF["Terraform"] -- "topology; ephemeral reads" --> INF
  INF -- "push config, no revocable secrets" --> RW["Railway variables"]
  INF -. "via relay; unverified (drill step 2)" .-> GW["Gateway in the Railway environment"]
  GW -- "TCP 5432" --> PG[("Postgres, no public port")]
  APP["App service"] -- "bootstrap credential" --> INF
  APP -- "lease or rotated credential" --> PG

CI with no stored secret

The job asks GitHub for an OIDC token and sends it to Infisical. Infisical checks the token’s subject, audience, and claims, then returns an access token (Infisical GitHub OIDC). Nothing long-lived sits in GitHub. Three settings make that true.

Set the subject to the environment, in the form your repository uses. A job that names production produces repo:<org>/<repo>:environment:production. Repositories created after July 15, 2026 use an immutable form that adds owner and repository IDs, repo:<org>@<org-id>/<repo>@<repo-id>:environment:production (GitHub OIDC reference). Use hardcoded values, not globs, and read the real sub from a test run with actions-oidc-debugger before you save.

Turn the subject into a gate. A subject for an environment job does not include the branch, so any branch a stolen token can push gets the same subject. Add protection rules to the environment. Required reviewers allow up to six reviewers and need one approval; turn on “prevent self-reviews” and turn off administrator bypass (GitHub environments). Restrict deployment branches to protected branches, which Pro and Team support for private repositories. On the Infisical identity, set Claims, for example a ref claim for refs/heads/main.

Remove the defaults. New identities start with Universal Auth. Remove that configuration, and its client secrets, after you add OIDC (Infisical GitHub OIDC). Then set the access-token TTL and max TTL to the length of a job. The field table on that page says 30 days and a note says 7,200 seconds, so set both explicitly.

Required reviewers have a plan limit. On GitHub Free, Pro, or Team, they work only on public repositories, so a private repository needs Enterprise.

on:
  workflow_dispatch:                       # use your own release trigger
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production                # sets the OIDC subject and the protection rules
    permissions:
      id-token: write                      # lets the job request an OIDC token
      contents: read
    steps:
      - uses: actions/checkout@<full-commit-sha-of-v5>             # v5
      - name: Fetch deploy secrets from Infisical
        uses: Infisical/secrets-action@<full-commit-sha-of-v1.0.18>  # v1.0.18
        with:
          method: "oidc"
          identity-id: ${{ vars.INFISICAL_IDENTITY_ID }}
          domain: "https://app.infisical.com"
          project-slug: "<project-slug>"
          env-slug: "prod"
          secret-path: "/deploy"
          export-type: "env"

Pin both actions to full commit SHAs, unless your repository uses workflow dependency lockfiles, which GitHub recommends first and which supersede SHA pinning (GitHub secure use). A tag can be moved. The secrets-action README lists the inputs and OIDC setup (secrets-action).

Runtime: keep revocable secrets out of Railway variables

The rule is simple: a secret you may need to revoke fast should not depend on a deploy to change.

• Railway applies a variable change to a running service only through a deploy. By default, a change redeploys the service. The API field skipDeploys turns that off, and Railway’s guide says to trigger a deploy yourself when a rotation must take effect now (Railway guide).
• Each service keeps one bootstrap credential. Railway staff said in April 2026 that running services have no workload identity OIDC (Railway Station, 2026-04-29). That credential is a standing secret [INFERENCE]. Infisical’s bootstrap example pairs a client secret capped at one use, or a short TTL, with a periodic access token the workload keeps renewing. That token has no maximum lifetime, and any break in renewal, including a restart, needs a new client secret. If you set a client-secret TTL, add a credential expiry alert (Universal Auth).
• Sealed variables are hidden from Railway’s API and railway run (Railway guide). They don’t hide the value from code running in the service [INFERENCE].
• Keep the database private. A TCP proxy exposes a service’s port to the internet (Railway TCP proxy). Delete any proxy on the database service and confirm no public domain remains. Private networking reaches services only within the same project environment (Railway private networking).
• Keep server secrets out of static builds. Astro replaces environment values at build time (Astro environment variables).
• For the agent service, use JWT mode. AgentOS uses a shared OS_SECURITY_KEY when no JWT source is configured. With Auth0 tokens, set the JWT environment variables and authorization=True, then remove the shared key. A valid security key can also mint service-account tokens (Agno AgentOS security).

Postgres: rotation, leases, and the route

Two mechanisms, with different costs.

• Rotation alternates two users, so one password stays valid while the other changes (Infisical rotation overview). That overlap is the catch: under dual-phase rotation, the inactive credential stays valid until the next cycle, so a leaked password can survive two intervals. If you suspect a leak, revoke first; don’t wait for the schedule. The rotation role needs CREATEROLE (PostgreSQL connection). Use CREATEROLE, not SUPERUSER. Grant each managed user to the rotation role with WITH ADMIN OPTION, as the same page shows, and enable Platform Managed Credentials so Infisical sets the connection’s password. CREATEROLE can create and alter roles, so treat that role as privileged.
• Leases create a new account per lease, named from a username template, with a TTL (dynamic secrets). Deleting a lease revokes its credentials at the provider (Infisical CLI: dynamic secrets). Dynamic secrets are an Advanced feature (Infisical pricing).

This is the part we couldn’t close: the route Infisical takes to a Postgres with no public port, which rotation and leases both need. Infisical’s gateway runs inside your network and connects out in relay mode (gateway overview). The gateway deployment page says a database reached through an app connection, a dynamic secret, or a rotation target uses the gateway as its network path (gateway deployment). None of the Postgres pages we read shows how to pick a gateway. The drill below tests that.

Terraform: topology in code, values as ephemeral reads

Terraform writes any value you put in configuration into state and plan files. The sensitive argument only hides it in CLI output (Terraform: manage sensitive data). Since Terraform 1.10, ephemeral values and resources never reach state or plan files. Since 1.11, write-only arguments do the same for managed resources, when the provider supports them.

The Infisical provider manages topology: infisical_project, infisical_identity, and infisical_project_role (project, identity, project role). Custom roles need Advanced on the pricing page or Enterprise in the RBAC doc, so check the tier before you rely on them. The provider reads values through the ephemeral infisical_secret resource (secret docs).

terraform {
  required_version = ">= 1.10"
  required_providers {
    infisical = {
      source = "infisical/infisical"
    }
  }
}

variable "infisical_client_secret" {
  type      = string
  sensitive = true
  ephemeral = true               # never written to state or plan
}

provider "infisical" {
  host = "https://app.infisical.com"
  auth = {
    universal = {
      client_id     = "<machine-identity-client-id>"
      client_secret = var.infisical_client_secret
    }
  }
}

ephemeral "infisical_secret" "postgres_password" {
  workspace_id = "<project-id>"
  env_slug     = "prod"
  folder_path  = "/db"
  name         = "POSTGRES_PASSWORD"
}

locals {
  db_password = ephemeral.infisical_secret.postgres_password.value
}

The value can flow only into write-only arguments or other ephemeral contexts. Anywhere else, such as a plain argument of a managed resource, Terraform rejects the plan with an error, so the value never reaches state (ephemeral block).

Two caveats follow.

• The identity’s universal-auth client-secret resource returns client_secret as a Sensitive, read-only attribute (provider docs). Terraform keeps that in state, and anyone who can read state can read the secret. Create that secret outside Terraform, or keep state in an encrypted backend with access controls (Terraform: manage sensitive data).
• The community Railway provider marks variable values Sensitive, not write-only (Railway provider docs). Keep Railway values in Infisical’s sync, not in Terraform state.

Plans and cost, as of 2026-11-02

Monthly cost is roughly identities × $40 + max(0, brokered calls − 5,000) ÷ 1,000 × $2, on Advanced with annual billing. A brokered call is one where the Agent Vault proxy attaches a real credential before forwarding it (pricing). Example: 6 service identities and 200,000 brokered calls a month come to 6 × $40 + 195 × $2 = $630, before people, and before any Enterprise price if gateways need it.

A drill that can fail

1. Change a non-secret test variable through the Railway API with skipDeploys: true. The running container should keep the old value until a deploy. That confirms a variable change is not live on its own (Railway guide).
2. Issue one Postgres lease for the app and run one rotation through the gateway. This tests the route. Agent Vault does not apply here, because it attaches credentials only to HTTP/1.1 requests (Agent Vault proxies).
3. Delete the lease. The app should stop using it within one Infisical Agent poll, five minutes by default (Infisical Agent), and it should pick up a replacement without a deploy.
4. Revoke one per-app OpenRouter key, then issue a replacement. Agent Vault polls every 60 seconds by default, so the proxy should use the replacement within one poll, with no deploy (Agent Vault proxies).

Pass: the app recovers with no deploy, and revoked credentials stop working within one poll. Fail: the app keeps the revoked credential, or needs a deploy to see the new value. A failure at step 2 means the runtime design doesn’t work yet, because the route is missing. A failure at step 3 or 4 means the thesis is wrong for that secret.

What is still unverified

• Whether a Postgres app connection or dynamic secret can select a gateway. No Postgres page shows a selector. Step 2 tests it.
• Whether the app drops a revoked lease and picks up a replacement within one poll. Step 3 tests it.
• Whether an Agent Vault session picks up a replaced OpenRouter key within one poll. Step 4 tests it.
• The gateway tier: Advanced on the pricing page, Enterprise in the gateway overview.
• Whether Auth0 rotation can be zero-downtime for your app. Auth0 says either the old or the new secret may be accepted for up to 30 seconds, which is a propagation window, not a guaranteed overlap. Disable auto-rotation, deploy the new secret, and roll it in a window.

Sources

• LiteLLM: the attack chain behind the v1.107.0 release: primary postmortem; events dated 2026-05-27 and 2026-10-10.
• Railway: preliminary report, October security incident: 2026-10-22, primary.
• Railway: sync secrets from an external manager: updated 2026-10-29, primary.
• Railway Station: workload OIDC question: 2026-04-29, staff reply.
• Railway: TCP proxy and private networking: undated.
• Wiz: Gheist: 2026-10-25, primary; The Hacker News: 2026-10-26, secondary.
• GitHub: OpenID Connect reference, deployments and environments, and secure use: undated.
• actions/checkout v5 and Infisical secrets-action v1.0.18: runtime and inputs checked; undated.
• Infisical: GitHub OIDC auth, secrets-action, Universal Auth: undated.
• Infisical: rotation overview, Auth0 rotation, PostgreSQL connection, PostgreSQL dynamic secrets, dynamic secrets CLI: undated.
• Infisical: gateway overview and gateway deployment: undated.
• Infisical: Agent and Agent Vault proxies: undated.
• Infisical: pricing: accessed 2026-11-02; prices and plan gates change.
• Infisical: change policies, RBAC, audit logs: undated.
• Infisical Terraform provider: project, identity, project role, client secret, and ephemeral secret docs; undated.
• Auth0: rotate client secrets and Private Key JWT: undated.
• OpenRouter: Security Center: 2026-09-28; management API keys: undated.
• Terraform: ephemeral block and manage sensitive data: undated.
• Community Railway Terraform provider: variable: undated.
• Agno: AgentOS security and Astro: environment variables: undated.