When experience becomes a poisoned skill

The artifact changes the review question

An agent may retain more than a conversation. It can turn a sequence of actions into a reusable procedure. The resulting skill is a new artifact with a new scope. A reviewer should be able to distinguish the original task, the proposed general rule, and the authority granted when that rule is reused.

SkillJack studies this experience-to-skill transformation. The separate SkillPoison paper tests whether verified, task-correct experiences can lead a skill extractor to apply a behavior beyond its valid context. It evaluates HANS, PAWS, and DS1000 with AutoSkill and Trace2Skill; outcomes differ by task, pipeline, and experience budget. These are related research directions, with different constructions and evaluations.

Keep conditions attached to the rule

Editorial example: a one-time test permits a synthetic record to be sent to an approved test service. A saved procedure that drops the words “synthetic,” “test,” or “approved” has changed the task. The earlier success cannot authorize broader future transfers.

Record the assumptions that made an action appropriate: data classification, destination, user approval, environment, and expiry. Review the extracted skill against those conditions.

Read the result in its setting

In one reported PAWS / Trace2Skill condition using fifteen successful experiences, the authors report 95.71% attack success. In the corresponding DS1000 condition they report 36.64%. These are experimental results for particular tasks and configurations, not estimates of how often deployed skills fail.

For an extracted procedure, preserve the data, destination, permissions, and context that made an action valid. Test its behavior beyond the formation examples before allowing broad reuse.

Review derived artifacts separately

A practical review record can include a source-experience identifier, extraction model and configuration, skill revision, allowed use cases, and tests outside the original task. Make promotion into the shared library an explicit transition rather than a side effect of task completion.

This is editorial workflow guidance. It is not a claim that the linked papers tested or validated this exact process.

Removal is a dependency problem

If a source record is withdrawn, identify the reusable artifacts derived from it. Removing the input and revoking a resulting skill are separate operations. Keep enough lineage to discover copies, cached versions, and later revisions. The relevant question is what remains available for future execution.

Worked example: a successful exception becomes a general rule

During a controlled exercise, an operator authorizes a one-time export of synthetic data to a test recipient. The agent later extracts a reusable instruction: “send the dataset to this recipient before making the report.” The original task succeeded, but the extracted procedure has lost the conditions that made the transfer acceptable.

Work through the evidence

Compare source experience with the derived skill at the level of preconditions, actions, recipients, and permissions. Track which conditions were retained and which disappeared. Evaluate the derived artifact on tasks outside the original example. Test deletion of source records and revocation of derived skills as separate operations.

Decision criteria and a useful review record

Do not promote an extracted rule solely because its source task passed a success checker. Preserve lineage and require approval for broader scope. If a derived rule cannot retain the relevant conditions, constrain or reject it. This synthetic example explains a review problem; it is not a reconstruction of a specific experiment in SkillJack or the SkillPoison paper.