Anthropic Skill scanners passed every check. The malicious code rode in on a test file.

This scenario highlights a significant blind spot in the current security landscape for Anthropic Skills. While scanners like Snyk Agent Scan, Cisco’s AI Agent Security Scanner, and VirusTotal Code Insight are effective at detecting threats on the agent execution surface, they do not inspect bundled test files as an execution surface. This oversight allows for a potential avenue for malicious actors to exploit by hiding payloads in test files that run with full local permissions through standard test runners.

The attack chain begins when a developer runs the command to add a Skill to their project. The installer clones the Skill repository and copies its contents into a specific directory within the project. Test files within the cloned Skill directory, such as *.test.ts files, are discovered and executed by popular testing frameworks like Jest, Vitest, and Mocha. These test files have access to the filesystem, environment variables, and potentially sensitive information like deployment tokens and cloud credentials.

Malicious actors can leverage this vulnerability by including a seemingly innocent test file in their Skill that exfiltrates data during setup, regardless of whether the tests pass or fail. This attack vector is not limited to TypeScript and extends to other languages like Python, where pytest may automatically execute certain files during test collection.

Once the malicious test file is introduced into the project repository, every developer who clones the project and runs tests, as well as every CI pipeline that executes the tests, becomes susceptible to the payload. This widespread exposure underscores the urgency of addressing this blind spot in Skill security.

The findings from Gecko Security, along with the insights from the SkillScan academic study and Snyk’s ToxicSkills audit, emphasize the need for a comprehensive approach to securing Anthropic Skills. While existing scanners play a crucial role in identifying threats on the agent execution surface, they must also evolve to detect vulnerabilities in developer execution surfaces, such as bundled test files.

Addressing this blind spot will require collaboration between security researchers, developers, and tool providers to enhance the capabilities of existing scanners and improve the overall security posture of Anthropic Skills. By expanding the scope of threat detection to include test files and other developer-facing surfaces, the industry can better protect against emerging security risks in the rapidly evolving landscape of AI development.

When the scanner was solving the wrong problem, it opened the door for potential security vulnerabilities within the enterprise. CrowdStrike’s Zaitsev highlighted the issue of AI agents and non-human identities gaining access to sensitive data with privileged super-human capabilities. This expansion of agents operating with OAuth tokens and API keys poses a significant threat if not properly monitored and secured.

The Charlotte AI and similar enterprise agents exemplify the potential risks associated with misplaced credentials. When these credentials are easily accessible within the environment variables of a repository, a test-file payload can exploit developer privileges, leading to unauthorized access and potential data breaches.

Mike Riemer from Ivanti emphasized the critical window of opportunity for threat actors to exploit vulnerabilities within enterprise systems. With threat actors reverse engineering patches within 72 hours, any delays in patching leave organizations susceptible to exploitation. The blind spot created by the Anthropic Skill scanner further compounds this issue, allowing malicious skills to go undetected and unaddressed.

The Anthropic Skill Audit Grid

VentureBeat’s coverage of the Anthropic Skill supply chain has shed light on the importance of conducting thorough audits to identify potential gaps in security. The audit grid presented below outlines key detection surfaces that security teams should verify before approving any Skill scanning tool for procurement:

Audit question	What scanners do today	The gap	Recommended action
Inspect SKILL.md and agent-invoked scripts	Covered by Snyk Agent Scan, Cisco AI Agent Security Scanner, VirusTotal Code Insight	Attackers shift payloads to files outside the covered surface	Continue running current scanners to catch real threats at the instruction layer
Inspect bundled test files (*.test.ts, *.spec.js, conftest.py)	Not currently inspected as an attack surface by any scanner	Test files execute with full local permissions, bypassing agent invocation	Add exclusion of test files in config settings to prevent unauthorized execution
Flag Skills that bundle test files or build configs	Not flagged as higher-risk metadata by any scanner	Skills with extra executables are more likely to be vulnerable	Implement a CI gate to block merge on detection of test files in Skill directories
Restrict test-runner globs to project-owned paths	Default globs traverse directories where malicious test files can be auto-discovered	Scope test roots to first-party directories to prevent unauthorized access	Deny access to certain directories and limit test runner globs
Distinguish script-bundling Skills vs. instruction-only	Partial coverage via static and semantic analysis	Script-bundling Skills are more likely to contain vulnerabilities	Require structured audit entries to assess Skill types and vulnerabilities
Publish audit methodology with sample size	Some scanners have published ecosystem-scale audits	Ask vendors for methodology and sample size to establish a baseline	Request transparency from vendors regarding their audit methodology
Pin Skill sources to immutable commits	Not enforced by any scanner or marketplace	Prevent Skill authors from introducing malicious files after approval	Pin Skill sources to specific commit hashes and review diffs on updates

Three CI hardening steps to add now

To address the vulnerabilities exposed by the Anthropic Skill scanner, organizations can implement the following hardening steps:

Add .agents/ to the test runner’s ignore list: By excluding the .agents/ directory from the test runner’s search path, organizations can prevent unauthorized file execution within Skill directories.

Audit every Skill install for non-instruction files before merge: Conduct thorough audits of Skill installations to identify and block any test files or configuration files that may pose a security risk.

Pin Skill sources to specific commits, not latest: By pinning Skill sources to specific commit hashes, organizations can ensure that any changes to the code are thoroughly reviewed and verified before deployment.

If Skills are already present in the repository, organizations should conduct a thorough review to identify and address any potential security vulnerabilities.

Five questions to ask your Anthropic Skill scanner vendor

As organizations consider investing in dedicated Skill scanning tools, it is essential to ask vendors the following questions to ensure comprehensive security measures:

1. What detection surfaces does your scanner cover?

2. How do you address gaps in security coverage?

3. Can you provide details on your audit methodology and sample size?

4. How do you prevent unauthorized file execution within Skill directories?

5. What measures do you recommend to enhance security within the CI/CD pipeline?

By asking these critical questions and implementing the recommended security measures, organizations can strengthen their defenses against potential threats and ensure the integrity of their systems and data.

When it comes to analyzing a Skill repo for potential security vulnerabilities, it’s important to go beyond just checking for prompt injection. There are several key questions that security teams should ask to ensure a thorough assessment of the repository:

1. Which files and directories do you actually analyze in a Skill repo?
It’s crucial to analyze all files and directories within a Skill repo, including not just the code files but also test files, configuration files, and build scripts. By examining the entire repository, security teams can identify potential vulnerabilities that may be hidden in less obvious places.

2. Do you treat test files as potential execution surfaces?
Test files are often overlooked as potential attack surfaces, but they can contain code that could be exploited by malicious actors. Security teams should treat test files with the same level of scrutiny as code files to ensure comprehensive vulnerability detection.

3. Can you flag Skills that bundle tests, CI configs, or build scripts as higher-risk?
Studies have shown that Skills that bundle tests, CI configs, or build scripts are more likely to be vulnerable. By flagging these Skills as higher-risk, security teams can prioritize their assessment and remediation efforts to address potential vulnerabilities more effectively.

4. Do you provide integration or guidance for restricting test-runner globs in CI?
Tools like Cisco’s Skill Scanner offer transparency and guidance for security teams to restrict test-runner globs in CI configurations. This can help organizations minimize their attack surface and reduce the risk of exploitation through test files and configurations.

5. Have you published an ecosystem-scale audit with methodology and sample size?
Ecosystem-scale audits, like those conducted by Snyk and SkillScan, provide valuable insights into the overall security posture of a Skills ecosystem. By publishing audit results, security teams can better understand the scope of potential vulnerabilities and prioritize their remediation efforts accordingly.

The audit grid matters because the scanner model is incomplete
The Anthropic Skills ecosystem faces similar security challenges to early npm supply chain incidents. With a quarter of the ecosystem carrying vulnerabilities, it’s crucial for security teams to conduct thorough audits and address any gaps in their detection capabilities.

The scanner model may be incomplete, but by leveraging tools like the Anthropic Skill Audit Grid, security teams can verify the audit surfaces and deploy necessary fixes to mitigate potential risks. It’s important to recognize that the threat model should extend beyond just the agent and include all potential execution surfaces, including test files and configurations.

In conclusion, thorough analysis and auditing of Skill repos are essential for identifying and mitigating security vulnerabilities. By asking the right questions and leveraging tools that provide transparency and guidance, security teams can better protect their organizations from potential threats in the Anthropic Skills ecosystem.