LiteLLM supply-chain leak: how I checked my own home gateway before the news cycle caught up

TL;DR - On Tuesday Kevin Beaumont posted about the largest AI supply-chain breach of 2026: a 40-minute poisoned-publish window on PyPI in March put credential-scraping code into LiteLLM 1.82.7 and 1.82.8. 195 TB of secrets, 2,500+ organisations, 434,000 CI/CD pipelines. I run LiteLLM at home as my personal AI gateway. Here is the audit I ran this morning against my own Postgres request logs, what I found, and the call I made on whether to rotate six provider keys. The TL;DR for IT-pros who run any AI proxy themselves: check when your image was built, check the requester IPs in your gateway logs, and rotate keys by revoking at the provider, not by editing a secrets file.

If you run any AI infrastructure in production - gateway, proxy, eval rig, training estate, agent harness - this is the morning to check your request logs. The same instinct that says "audit before you trust the cleanup is complete" applies at home too, where the logs were probably already being written; you just hadn't looked.

Get my Personal Security Quick-Start Guide - the practical handbook for the controls that actually change your day. 193 pages, no jargon.

Plus: Join 158+ Australians getting one 5-minute security briefing every Friday.

Get The Free Guide →

The bus, Phil, Beaumont, and a typo that started my morning

This morning I was on the bus on the way to work when a friend in Discord (hi Phil) dropped a Kevin Beaumont post. Beaumont is a researcher I trust; when he posts, I read. The post talked about a thing called "LiveLLM" - terabytes of credentials scraped from thousands of orgs, attacker was a group of teenagers called TeamPCP, root cause was orgs adopting GenAI without proper security.

I had a couple of questions immediately. The first was spelling. "LiveLLM" is not a package I had ever installed. "LiteLLM" is. Phil's link included the Hudson Rock blog URL, and the slug said litellm-hack-impacts-thousands-of-global-enterprises. The screenshot Beaumont attached was captioned "Microsoft" - someone at Microsoft had apparently been running the compromised package on a workstation that was leaking @microsoft.com credentials. So the typo was either Beaumont's autocorrect or his coffee.

The second question was: do I run this thing?

Yes, I run LiteLLM at home

I have a self-hosted stack at home. It is mostly the kind of thing an IT-pro with too many hobbies and not enough restraint ends up building. There is a Postgres database, a Gitea instance, a handful of AI agent processes, and a single OpenAI-compatible gateway in front of all of them. The gateway is LiteLLM. It runs as a container called devops-litellm-gateway-1, image ghcr.io/berriai/litellm:main-stable, bound to loopback only - not the public internet.

Why do I run it? Because LiteLLM gives me four things I want and I do not want to write them myself:

API aggregation. One local endpoint, every model I have a key for. OpenRouter, ClinePass, MiniMax, OpenCode Go, Moonshot Kimi, ZAI GLM, plus a few local instances on the gaming rig. The agent code asks for openai/MiniMax-M3 or openai/kimi-k3 and does not know or care which provider that resolves to. If a provider changes their slug, my agent code does not change.
A single local key for the whole stack. The agents hold the LiteLLM gateway key. The real provider keys live in the gateway's process environment, mounted from docker secrets. If an agent is compromised, the blast radius is "the gateway", not "six upstream provider accounts".
Cost and token tracking, for free. Every request goes through LiteLLM_SpendLogs in Postgres. I can see exactly how many tokens each agent consumed last Tuesday, and what the same call would have cost on PAYG against OpenAI or Anthropic. That visibility is the difference between "the agents feel expensive" and "I know the agents cost me $77 on 7 August and $42 on 8 August and I can correlate that to what they were doing."
Fallbacks. If Kimi K3 is down, route to GLM-5.2. If GLM is rate-limited, fall through to DeepSeek. If everything cloud-side is rate-limited, fall through to a local qwen3.5 7B on the gaming rig. The gateway handles the failover. My agent code does not need retry loops.

If you are an IT-pro who builds anything on top of LLMs, this is the pattern you want. If you have not seen IndyDevDan's work on the Super Simple Software Factory, it is the gold standard of AI developer workflows and the LiteLLM-gateway pattern is one of his. Worth your evening.

So yes, I run LiteLLM. And this morning, from the bus, I had to figure out whether I had been part of the breach.

What the breach actually was

By the time I got to a laptop, the picture had filled in. Two security firms, CloudSEK and Hudson Rock, had published on Tuesday and Wednesday. Ars Technica had a write-up by Dan Goodin. The mechanics were:

A supply-chain attack on LiteLLM 1.82.7 and 1.82.8, published to PyPI for a 40-minute window in March 2026.
The malicious versions ran a payload that read process memory of every machine they were installed on, scraped the contents, and exfiltrated it through an attacker-controlled channel.
What got scraped: cloud keys (AWS, Azure, GCP), repo tokens (GitHub and GitLab PATs), SSH keys, Kubernetes service-account tokens, package-publishing credentials, AI-provider keys, Slack signing secrets, Salesforce client secrets, database passwords. Everything in process env at the moment the malicious LiteLLM ran.
Scale: about 2,500 organisations confirmed by CloudSEK, 434,000 CI/CD pipelines had credentials in the dump. The 195 TB figure comes from Hudson Rock, which pulled the dataset to analyse.
The compromised LiteLLM was downstream of an earlier supply-chain attack that hit Trivy, the popular vulnerability scanner. Same campaign also compromised KICS and the Telnyx Python SDK. This was a broad offensive against developer tooling, not a single-package hit.
Threat actor: TeamPCP, a group CloudSEK and others describe as a ramshackle but capable gang largely made up of teenagers. Researchers have largely corroborated the claim.

The victim list reads like a list of the biggest companies on earth. Microsoft, Amazon, Cisco, Samsung, Salesforce, Nvidia, Siemens, Airbus, Deutsche Bahn, Vodafone, John Deere, the London Stock Exchange, Thomson Reuters, FedEx, Deloitte, X/Twitter, Zscaler, Epic Games, HP, Roku. CloudSEK said the credentials often could not be matched back to an org - a siriusxm.com email address turned out to be inside AdsWizz, a SiriusXM subsidiary, not the broadcaster itself. Many CI/CD pipelines are configured generically; the dumped variables included active database passwords with no identifying organisation context at all.

Beaumont's verdict, with the part that mattered most to me in bold: "[The data is] a massive supply chain breach due to poor AI security - not because AI is the threat, but teens can run circles around orgs obsessed with rushing out AI and poor DevOps security."

The follow-through detail that made it worse. After the public disclosures, Beaumont tested one of the affected orgs that claimed "we rotated everything, it's a nothingburger". He looked up their responsible-disclosure policy, which allowed trying credentials, so he tried them. Almost every one worked. Trivy developers, separately, had rotated an automation token over a 20-day window without fully revoking it, giving the attackers a three-week period to force-push malicious code to downstream builds. "Rotated" was doing a lot of work in that sentence.

The audit I ran against my own gateway

So. Do I have to care? Two questions:

Was my container running a poisoned version when the malicious code was active on PyPI?
If yes, has anything since connected to my gateway from an IP that is not me?

The first one I could answer from the data I already had on disk.

$ docker inspect devops-litellm-gateway-1 --format '{{.Created}}'
2026-08-09T14:25:59Z

$ docker exec devops-litellm-gateway-1 /app/.venv/bin/litellm --version
LiteLLM: Current Version = 1.92.0

The container was created on 9 August. The image it runs was built on 12 July, well after the 40-minute poisoned-publish window in March. And the version is 1.92.0, not 1.82.7 or 1.82.8 - so even if the image had been built earlier, I would not have been on a known-bad version.

I also checked the host filesystem. pip list, pipx list, find / -name litellm: no live pip install litellm had ever run on this machine. The only host-side traces were ddtrace's vendored observability code (unrelated) and our own ~/Projects/devops/litellm/ config repo (also unrelated, that is where our docker-compose.yml and gateway config live).

So question one: no, my running binary was never one of the poisoned wheels, and I never resolved the bad versions from PyPI. The poisoned PyPI window in March closed about four months before my first LiteLLM footprint appeared on disk.

Question two was the one that mattered more. Even if the malicious code never ran in my container, the credentials in its environment could be brute-tested by anyone who had a copy of the 195 TB dump and a list of LiteLLM-gateway-shaped endpoints to try them against. (My gateway is loopback-only, so a remote brute test is not the attack vector I am worried about, but the principle stands.) What I actually wanted to know was: has anyone used my gateway key from an IP that is not me?

LiteLLM logs every request to LiteLLM_SpendLogs. The columns I cared about were requester_ip_address, api_key, user, and startTime. The queries:

SELECT min("startTime"), max("startTime"), count(*),
       count(DISTINCT "user"), count(DISTINCT "api_key")
FROM "LiteLLM_SpendLogs";

SELECT "requester_ip_address", count(*), min("startTime"), max("startTime")
FROM "LiteLLM_SpendLogs"
WHERE "requester_ip_address" IS NOT NULL AND "requester_ip_address" != ''
GROUP BY "requester_ip_address"
ORDER BY count(*) DESC;

The answer was clean. Across 16,947 requests spanning 2026-07-17 to 2026-08-13 (28 days, matching the gateway's actual uptime), the requester-IP column held exactly two values: the Docker bridge gateway that every container-to-host connection arrives through (the agent processes, the daily-report.py script, and the pifactory dispatcher - 16,759 of the 16,947 requests), and direct loopback from an early batch of test requests before the gateway was properly wired into the stack (60 requests). No other IPs. No Tor exits. No cloud-provider egress. No 203.0.113.x test ranges. No unfamiliar subnets.

The api_key column told the same story. 13 distinct key prefixes in the log, but 16,821 of 16,947 requests (99.3%) used the same gateway key, and every one of those was from one of the two local IPs above. The other 12 keys were test artefacts: fake, $LITELLM_KEY (a literal unexpanded shell variable from an early test), None (internal health checks), a handful of hash-prefixed sub-keys I had minted when validating the gateway-key rotation flow, and one Â«redacted:lit… marker that appears to be a PII-redaction tag from the daily-report.py callback. No unknown sub-keys. If someone had minted themselves a key off my gateway key and was using my providers on my balance, they would have shown up in the log with their own IP. They did not.

Spend trend across the 28 days matched the work I had been doing. The curve rises from a single test request on 17 July to ~3,000 requests a day at full production load in early August, then drops to a partial 5-request day today as I write this. No anomalous spikes, no days with thousands of requests from a model I do not use, no overnight bursts.

So question two: no, no unknown IP has ever talked to my gateway, no unknown key has ever been used against it, and the spend curve matches my actual usage. If I had been owned, the log would have shown it. It did not.

What I did, and what I did not do

I did not rotate the keys. Let me explain why, because I want to be honest about the reasoning rather than perform the rotation as a ritual.

The case for rotating is the Beaumont follow-up. The 195 TB dump exists. TeamPCP exists. There are other groups with copies. Even if I was not on a known-bad version, even if my gateway has never been touched by an unknown IP, the safest posture is "the dump is in the wild, treat every key you hold as potentially in the dump, rotate everything."

The case for not rotating, on the morning of 13 August 2026, for this specific incident:

The poisoned publish window was in March. The first LiteLLM artifact on my disk is from 17 July. The first container run was 9 August. The 195 TB dump contains keys that were in process memory during the 40-minute window, which means credentials for machines that were running LiteLLM in March. None of my machines were running LiteLLM in March. The keys I hold now were created after the dump was assembled. They cannot be in it.
The audit showed no unknown requester IP. No unknown sub-key. No anomalous spend curve. The single strongest indicator of compromise - "someone is using my gateway key from somewhere that is not me" - is absent, not "we cannot tell". The Postgres logs are deterministic about this.
Rotating is not free. Six upstream provider dashboards, six revocation flows, six new key generations, six secret-file updates, six container restarts, six smoke tests to confirm the new keys resolve. Real work. And the work itself does not reduce risk if the audit shows zero risk to begin with - it just spends time that could go into something higher-value, like image-digest pinning (which I will do, see below) or moving the gateway off long-lived env-var secrets (which is the structural fix, also below).
Beaumont's "rotated but still owned" pattern was about orgs that rotated at the file level but did not actually revoke at the provider. If I were going to rotate, I would do it properly: revoke at the provider dashboard, not edit a secrets file. But the audit said I was not in the dump, so the right call was to do the cheaper thing first - confirm the audit - and only rotate if the audit had shown something.

I do want to be clear about the asymmetry. If my audit had shown a single unknown requester IP, I would have rotated immediately, regardless of the "we were not online in March" argument. I am not making the case that the March date-line makes rotation unnecessary for everyone. It makes it unnecessary for me, given that I had a deterministic log to check. If you ran LiteLLM in March and never audited your request logs, you should rotate today. The Beaumont follow-up pattern - "rotated, still owned" - is the case for rotating when you do not have a clean audit, not when you do.

I am also not making the case that the architectural problem is small. It is not. LiteLLM is the proxy that holds every provider key in process memory so that the agent code does not have to. That is the design. It is also the design's vulnerability class. The structural fix is below.

What I am considering anyway: the structural fixes

Three things worth considering because the audit told me the gateway is fine but the design tells me the gateway is one mistake away from not being fine.

1. Pin the image to a digest, not a tag. Right now docker-compose.yml pulls ghcr.io/berriai/litellm:main-stable. That is a moving target. If BerriAI rebuilds the image tomorrow for any reason - a security fix, a feature regression, a supply-chain incident of their own - I get the new binary without knowing it. The image only changes when I reboot the host anyway, which is about once a week, so the chance of catching a silent binary swap between reboots is real but small. Pinning to a digest means I see exactly which version is running and I'm the one choosing to upgrade.

2. Network-egress default-deny for the container. Today the LiteLLM container can reach any IP outbound. That is how it calls the upstream providers. It is also how a memory-scraping payload would phone home. An option here is moving the container to an egress allowlist: only the upstream-provider API endpoints plus my own Postgres and my own host. If a future payload tries to exfiltrate to an unfamiliar IP, the connection dies at the network layer and an alert fires. The provider endpoints are a finite list; allowlisting them is straightforward.

3. Move the gateway off long-lived env-var secrets. This is the structural one. LiteLLM reads OPENAI_API_KEY, ANTHROPIC_API_KEY, etc. from process environment because the upstream-provider SDKs want them there. That puts every key into process memory, which is exactly what the LiteLLM supply-chain attack scraped. The architectural fix is workload-identity federation - the proxy authenticates to each provider as itself, gets a short-lived token at request time, never holds a long-lived key in memory. Most providers do not support this yet. For the ones that do (AWS Bedrock, Google Vertex), the move is straightforward. For the ones that do not (OpenAI, Anthropic, OpenRouter, the rest), the option is putting the keys in a sidecar process that the proxy calls over a local socket, so the memory exposure is one process, not the whole gateway. Same class of fix as "use Azure Managed Identity instead of a connection string".

None of those three changes touch the gateway key, the provider keys, or my gateway's existing trust posture. They reduce the surface for the next incident.

What to do if your audit is not as clean as mine

If you ran LiteLLM during the 40-minute poisoned window in March, or if you cannot deterministically prove you did not, or if you are running any AI proxy that holds long-lived provider secrets in process env, here is the order I would do things in.

Within the hour:

Identify the versions of LiteLLM (or any other AI proxy) you actually ran in March. The poisoned versions are 1.82.7 and 1.82.8. Anything 1.83.0+ is clean of this specific payload, though later versions may carry different issues.
Pull the request logs from your proxy. Most AI gateways log requester IPs. Look for any IP that is not yours, any sub-key you did not mint, any spend curve you cannot explain. If the audit is dirty, rotate. If the audit is clean, you can defer rotation to a scheduled cycle.
If you are going to rotate, rotate at the provider dashboard, not in the secrets file. The Beaumont "we rotated but it still works" pattern is the failure mode of file-level rotation. Revoke the key at the provider, generate a new one, then update the secrets file.

Within the week:

Pin your gateway image to a digest.
Default-deny egress from the gateway container; allowlist the upstream provider endpoints.
Separate the AI-proxy process from the CI/CD secret plane. If you have a build pipeline that holds long-lived credentials, the AI proxy is the wrong place for them.
Subscribe to the GitHub Security Advisory feed for your AI proxy. BerriAI has 12 published advisories on LiteLLM this year (auth bypasses, SQL injection, sandbox escapes). Most are not catastrophic, but they tell you what classes of issue the project considers normal.

Longer term:

Plan for the architecture where the AI proxy does not hold long-lived secrets. Workload-identity federation is the right answer; where it is not available, a sidecar secret-broker process is the next-best answer.
Plan for the version of this incident where the attacker is not "teens with PyPI publish access" but "nation-state with a long-term foothold in a maintainer's laptop." The defence is the same - dig the secrets out of process memory - but the threat model is harder.

The lesson I keep coming back to

If you run any AI infrastructure in production - gateway, proxy, eval rig, training estate, agent harness - the unit of compromise is not "did the AI vendor have a bad week". The unit of compromise is "what credentials are sitting in process memory of the AI process at the moment an attacker gets read access to that process". Today the answer came from a 40-minute poisoned-publish window. Tomorrow it will be a different vector - a maintainer laptop, a CI/CD pipeline, a model provider's own breach.

The mitigation is the same in every case. Short-lived credentials. Workload identity. Egress allowlists. Image digest pinning. Auditable request logs that let you answer "is anyone using my gateway from somewhere they should not be" in seconds, not days.

I run LiteLLM at home because the engineering trade-offs make sense for me. I will keep running it. But the audit I ran this morning was the audit I should have been able to run six months ago - the logs were already there, I just had to write the query. If you run any AI proxy yourself, write the audit query now, before you need it. It is the difference between "probably fine" and "I checked the Postgres and I know".

Frequently Asked Questions

Was the breach limited to LiteLLM? No. The same campaign also compromised the Trivy vulnerability scanner, KICS (Keeping Infrastructure as Code Secure), and the Telnyx Python SDK. The LiteLLM compromise was downstream of the Trivy one. CloudSEK's 36-month retrospective, "The Scanner Was the Weapon," covers the broader pattern of precision attacks on DevSecOps tooling.

Were Microsoft, AWS, Cisco, and the other named orgs all running the poisoned LiteLLM? High-confidence yes for the named orgs, per CloudSEK's analysis of the 195 TB dump. Many credentials could not be matched back to a specific org because CI/CD pipelines are configured generically, so the actual count of affected orgs is higher than the named list suggests.

Did TeamPCP actually do it? Researchers have largely corroborated the claim. TeamPCP took public credit. Beaumont and others have confirmed the data is real by cross-checking victim organisations.

How does this compare to ChainDrop (the August 2026 npm worm)? Same general class - supply-chain attack on developer tooling, credential theft, lateral movement. Different vector - ChainDrop abused stolen npm publish tokens to inject preinstall lifecycle hooks; LiteLLM supply-chain injected a memory-scraper into the package itself. ChainDrop was self-propagating; LiteLLM is not. The defence principles overlap heavily (disable lifecycle scripts, audit credentials, segment the build plane) but the LiteLLM case is specifically about secrets-in-process-memory, which ChainDrop is not.

Should I rotate my provider keys? If you ran LiteLLM 1.82.7 or 1.82.8 during the 40-minute poisoned window in March, yes, today. If you can prove via gateway logs that you did not (or that no unknown IP has ever talked to your gateway), you can defer to a scheduled rotation cycle. If you cannot prove either way, rotate. The cost of rotation is real but bounded; the cost of "rotated but still owned" is much larger.

Why did Beaumont write "LiveLLM"? Best guess is autocorrect. The Hudson Rock blog URL in the same post says litellm-hack; the screenshot is captioned "Microsoft" (a victim's leak dashboard); every other write-up uses LiteLLM; no product called "LiveLLM" exists. Treat it as a typo and move on.

LiteLLM supply-chain leak: how I checked my own home gateway before the news cycle caught up

The bus, Phil, Beaumont, and a typo that started my morning

Yes, I run LiteLLM at home

What the breach actually was

The audit I ran against my own gateway

What I did, and what I did not do

What I am considering anyway: the structural fixes

What to do if your audit is not as clean as mine

The lesson I keep coming back to

Frequently Asked Questions

Further Reading

You might also like

AI Offense Is Now Automated. Your Defense Isn't.

Linux bridge STP vulnerability: the KVM and Docker patch

RoVoblast: a one-click Atlassian AI attack to check today