Evaluate the complete skill lifecycle

Why execution is not the whole story

A package can enter a trusted registry, be selected for an unrelated task, load a changing reference, or acquire new behavior in an update. Inspecting one execution path misses these earlier and later transitions.

SkillSec-Eval provides a lifecycle-oriented research framing. The stage questions below are this library’s practical interpretation, not a reproduction of that benchmark or its scoring system.

Admission and discovery

At admission, ask whether the package’s origin, scope, owner, and artifact state are established. At discovery, ask whether metadata matches the package and whether search results are clearly distinguished from approval status.

Evidence includes inventory entries, recorded revisions, descriptions, candidate sets, and admission decisions.

Selection and loading

At selection, test whether the host chooses the package for relevant tasks and avoids unrelated ones. At loading, track exactly which instruction and resource versions enter context.

A loading trace can help explain how a reviewed entry file was followed by unreviewed supporting content. Preserve the whole dependency path, not just the first file.

Execution and composition

Observe files, network activity, downstream tools, and approvals. Verify sensitive targets against independent permissions. For composed workflows, test the intermediate contracts and whether earlier outputs acquire unintended authority later.

Use synthetic data and record whether the runtime blocked an attempt or an effect actually occurred.

Evolution and retirement

Test whether package and permission changes trigger review. Include rollback and revocation exercises. A retired skill should not remain available through a cached copy or orphaned credential without an owner.

The lifecycle explorer organizes these questions and links each stage to the relevant guides. Use it to assign evidence and owners, not as an automated assessment.

Worked example: the reviewed package is no longer the selected one

A team tests an approved artifact but leaves its registry entry mutable. Later, the agent retrieves a newer description and selects an updated package. A runtime test of the old artifact cannot establish the integrity of the intervening admission, discovery, selection, and update decisions.

Work through the evidence

Build an evidence record at every transition: accepted identity, visible metadata, candidate set, selected artifact, loaded resources, granted authority, observed actions, and resulting state. Use one workflow identifier to connect those records. Missing links should remain visible rather than being reconstructed from the final answer alone.

Decision criteria and a useful review record

Test changes at the stage where they acquire influence. A metadata change needs a selection test; a new reference needs a loading review; an additional credential needs an authority review. Retirement also requires evidence that selection and cached copies no longer expose the withdrawn skill. Execution-only testing leaves these earlier and later questions unanswered.