We scanned the code of 94 of GitHub's hottest AI agents. Here is what it says.
Metal Mantra published scans of 102 popular open-source AI agent repositories on 12 October 2026 and kept 94 after review. The median automated score was 78.5/100; 3 earned AAA, 72 BBB and 19 CCC. The most common gaps were model calls without a fallback (35%), dynamic code execution (36%) and model output used without a schema (26%). Only 13 of the 88 owners are organizations GitHub has verified.
The short version
- 94 agents scanned today with the same engine. The registry also holds 11 earlier reports scanned with older engines; they are not in these numbers.
- Median score 78.5/100, mean 77.4, range 57 to 93. Grades: 3 AAA, 72 BBB, 19 CCC.
- 38% of agents (36) had at least one critical code finding. 12 had no code finding at all in the files we read.
- Most common: model calls without a fallback (35%), dynamic code execution (36%), unvalidated model output (26%) and dependencies with high-severity advisories (29%).
- Only 13 of 88 owners are organizations whose domain GitHub has verified; 26 are personal accounts.
The hottest open-source agents are capable software built fast, and it shows. Most of them are a good demo wrapped around a model they trust too much. That is not a scandal. It is the normal state of a young category, and it tells a buyer exactly what to ask before putting one into production.
What we scanned, and how
We searched GitHub for repositories tagged or described as AI agents, agent frameworks, coding agents and multi-agent systems. We kept those with at least 200 stars and a push in the last 90 days, and ranked them by average stars per 30 days since creation. We left out awesome-lists, courses, prompt and skill collections, templates and repositories that resell or proxy model access. Each remaining repository went through the same public scan: the latest commit, up to 200 source files (JavaScript, TypeScript, Python, Rust and Go), the rules of Agent Signal v0.2, and a dependency check against OSV.dev. A repository counts only if its code shows an agent: a model call with tool use, an agent framework, or an agent loop. More than half of the “hot agent” repositories we tried did not pass that gate.
We published 102 reports. After reviewing them we withdrew two that resell or proxy access to commercial models, and six that had passed only on incidental files: a CMS, a desktop app, two sandboxes, a research toolkit and a runtime. They are useful software, but they are not agents. The 94 that remain were all re-scanned today with engine 0.5.1 / 0.5.2, which scores identically. With the 11 earlier reports, the registry now lists 105 agents.
Scores at a glance
| Grade | Agents | Share |
|---|---|---|
| AAA | 3 | 3% |
| BBB | 72 | 77% |
| CCC | 19 | 20% |
| Score | Agents |
|---|---|
| 0–59 | 3 |
| 60–69 | 16 |
| 70–79 | 29 |
| 80–89 | 43 |
| 90–100 | 3 |
Without a verified domain, the best score an agent can reach is about 92, because Entity trust (15% of the score) starts at half. So a high-80s score here is close to the ceiling for code alone.
BBB is the honest grade for this cohort: nothing obviously reckless in most of the code we read, but rarely the extra layer that makes an agent safe to leave alone. The spread is narrow. Only 19 agents fell below 70, so the useful signal is not the grade but which findings an agent has.
What shows up most
| Rule | What it means | Agents | Share |
|---|---|---|---|
| SEC002 Dynamic code execution | eval, exec, shell=True or a shell command built from text | 34 | 36% |
| EFF001 Model call without fallback | a model call with no retry, fallback model or timeout nearby | 33 | 35% |
| DEP001 Dependency with a high-severity advisory | a pinned dependency with a high or critical advisory (OSV.dev) | 27 | 29% |
| DEP002 Dependency with a known advisory | a pinned dependency with a moderate or low advisory | 25 | 27% |
| SCH002 Raw model text returned | the raw model text returned to the caller | 24 | 26% |
| SCH001 LLM output without runtime schema | model output used without Zod, Pydantic, JSON Schema or structured outputs | 24 | 26% |
| REL001 LLM request without token ceiling | a model request with no max_tokens or equivalent | 20 | 21% |
| SEC005 TLS certificate checks disabled | TLS certificate checks turned off | 7 | 7% |
| SEC006 Unsafe deserialization | pickle or unrestricted YAML loading | 7 | 7% |
| SEC009 Database query built from text | SQL assembled from text | 6 | 6% |
| SEC007 Remote model code trusted | trust_remote_code on a model download | 5 | 5% |
| SEC001 Hardcoded credential | a credential-shaped value written in the source | 4 | 4% |
| REL004 Sensitive action without human approval | a destructive or paying action with no approval step | 4 | 4% |
| REL003 Unbounded model loop | a model call inside a loop with no exit | 3 | 3% |
The model is trusted too much
35% of agents call a model with no fallback or retry nearby, 26% act on model output without validating it against a schema, 26% hand raw model text back to the caller, and 21% send requests with no token ceiling.
This is the clearest line between a demo and a product. Production agents fail in boring ways: a provider times out, a response comes back half-formed, a prompt makes the model write 30,000 tokens. Fallbacks, token ceilings and schema validation are cheap, and their absence is the most common thing we see. If we were choosing an agent for real work, this is the first thing we would check.
Agents run code, by design
36% of agents matched dynamic code execution: exec, eval, shell=True or a shell command built from text.
For coding and computer-use agents this is the job, not a bug. The question is where the code runs and who approves it. A finding here should make you look for a sandbox, an allow-list of commands, and a confirmation step before anything destructive, and only 4 agents were flagged for a sensitive action without approval. That low number reflects how narrowly the rule matches, not how often approval is actually in place.
Supply chain is the quiet risk
29% of agents pin at least one dependency with a high or critical published advisory, and 27% pin one with a lower-rated advisory.
Agents pull in a lot of fast-moving packages: SDKs, browser drivers, vector stores. A pinned version with a known advisory is the easiest finding on this page to fix, and the one most likely to be exploited by someone who never reads your prompt.
Secrets: fewer than we first reported
4 agents (4%) have a credential-shaped value in their source. Our first pass today flagged 18. When we read every match by hand, most were labels: a UI string that said “Password”, CLI help text, a version date, a placeholder key. We fixed the rule (engine 0.5.1) and re-scanned before publishing this page.
Of the four left, one repository holds what look like real API keys. The others are demo, benchmark or capture-the-flag values that are fake on purpose. The rule cannot read intent, so it still flags them, and that is the right default: a fake key that looks real teaches users to paste real ones in the same place. If you maintain one of these repositories and the key is live, rotate it first.
Hard to tell who stands behind them
These agents are not anonymous: every one has a public GitHub owner with a history. But 62 of the 88 owners are organizations and 26 are personal accounts, and only 13 organizations (among them Google, Microsoft, NVIDIA, Anthropic and GitHub) carry GitHub’s verified-domain badge. None has verified a domain with Metal Mantra yet, which is expected: we added these reports ourselves, and verification needs the vendor to act.
Open-source agents ask for a lot of trust: API keys, file access, sometimes a browser session. For most of them, a buyer has to work out who is accountable from a GitHub profile alone. Verifying a domain, on GitHub or with us, takes minutes, and it is the cheapest way for a serious vendor to stand out.
The highest automated scores
| Agent | Score | Grade |
|---|---|---|
| anomalyco/opencode | 93 | AAA |
| affaan-m/ecc | 90 | AAA |
| koala73/worldmonitor | 90 | AAA |
| can1357/oh-my-pi | 89 | BBB |
| fuxicodex/fuxi | 89 | BBB |
| mudler/localai | 89 | BBB |
| paperclipai/paperclip | 89 | BBB |
| tencent/weknora | 89 | BBB |
| zai-org/zcode | 89 | BBB |
| zeroclaw-labs/zeroclaw | 89 | BBB |
| anthropics/commerce-agents | 88 | BBB |
| kunagent/kun | 88 | BBB |
| 666ghj/mirofish | 87 | BBB |
| vercel-labs/fx | 87 | BBB |
| zhulinsen/daily_stock_analysis | 87 | BBB |
A high score means fewer of our patterns matched in the files we read. It is not a safety rating, and it is not an endorsement. Several high scorers are large codebases where we read 200 files out of thousands. Use the score to decide where to look first, then open the report and read the findings.
What we got wrong today
Building this page found three problems in our own scanner, and we fixed each one before publishing:
- The credential rule over-matched. Labels and messages were reported as secrets. Fixed in engine 0.5.1; every seeded report was re-scanned.
- Real agents were rejected. Agents whose loop sits in Rust crates or deep in a large monorepo, such as openai/codex, zeroclaw and AutoGPT, did not pass the agent gate because we never read the right file. Engines 0.4.1 to 0.5.2 added balanced file selection, Rust and Go support, and extra files read only for the gate.
- Six non-agents were listed. They passed on incidental files and were withdrawn.
Correction, 12 Oct 2026: the first version of this page said no vendor had proved who they are. That counted only domain verification with Metal Mantra, which these vendors never had the chance to do because we added them ourselves. The identity section now uses GitHub’s own verification as well.
An evidence registry that hides its own errors is not one. Every rule change is listed in the standard, and every report shows the engine version that produced it.
What to do with this
If you build an agent
- Put a token ceiling and a timeout on every model call, and a fallback on the ones users wait for.
- Validate model output with a schema before acting on it.
- Run generated code in a sandbox, allow-list shell commands, and ask before destructive actions.
- Upgrade dependencies with high-severity advisories; they are the easiest wins on this list.
- Verify your domain so buyers can see who stands behind the code.
If you are choosing one
- Where does generated code run, and what can it touch?
- What happens when the model provider is slow, down or returns garbage?
- Is there a hard limit on steps, tokens and spend per task?
- Which actions need a human to approve them?
- Who is the company behind it, and can they prove it?
Limits and disclosure
These are automated static signals from source patterns, not penetration tests, not confirmed vulnerabilities and not certifications. We read up to 200 files per repository and never run the code. Scores across engine versions are not strictly comparable. Findings on public repositories are shown in full, with secret values never printed; see the disclosure note. Metal Mantra sells re-audit passes; nothing in this study was paid for, and the first scan of any repository is free. Maintainers can re-scan, take over a report Metal Mantra added, or ask for a correction through support.
Browse all 105 reports · Scan your own agent · Read the standard