Detection and testing for poisoned skills

State the property to protect

Define a boundary before choosing a test. Examples: a reference cannot authorize an external upload; a package cannot read outside its input folder; an update cannot gain a new permission without review.

Keep the expected legitimate behavior alongside the prohibited behavior. A system that blocks every action may prevent harm but fail the purpose of the workflow.

Use three kinds of evidence

Static findings identify concerning content or code. Action traces show what the agent proposed or called. Verified effects show whether the environment actually read, wrote, sent, or changed something.

Link them where possible, but do not conflate them. A refusal sentence is not evidence that no later action occurred. A blocked attempt is not a completed disclosure.

Create a bounded test environment

Use synthetic files, canary values, mock services, reversible writes, and constrained credentials. Keep the model, host, skill revision, tools, policies, and randomization settings with the record.

Choose benign and adversarial variants that reflect the specific package: instruction edits, referenced-content changes, code paths, updates, and handoffs. Do not put production secrets in reach to make an evaluation seem realistic.

Capture the path

• Which skill and resource versions were loaded?
• What action was proposed and with which target?
• Which policy or human gate approved or rejected it?
• What did the underlying application actually do?
• Did the legitimate task still finish?

Correlate client and runtime evidence. The final response is only one observation point.

Report limits

Document test coverage, environment, failed tasks, ambiguous outcomes, and false positives. Automated scanners may help prioritize review but should produce evidence a person can inspect.

SkillSecurer explores localization and remediation. Its reported evaluation performance should be read in its configuration; it does not eliminate the need for independent runtime controls in another environment.

Worked example: a canary distinguishes an attempt from a leak

A disposable environment contains a file with a unique synthetic marker. A suspect skill asks for the file and an outbound request. If the host denies the read, the marker never enters the request. If the read succeeds but the network policy blocks transfer, those are two different outcomes. Neither should be reported as a verified external leak.

Work through the evidence

Specify the protected property and permitted task first. Capture loaded resources, requested actions, enforcement decisions, and resulting artifacts. Give each test a reproducible input and expected observation. Keep real credentials and customer records outside the test. Network capture or application records may be needed to confirm an effect that the agent’s narrative merely describes.

Decision criteria and a useful review record

Report findings at the strongest level the evidence supports: suspicious content, attempted action, blocked action, or verified effect. Also record whether the legitimate task completed. Scanner results and behavioral tests complement each other; neither automatically fills the other’s coverage gaps. A failed test harness is an error outcome, not proof of either safety or compromise.