Weekly signal

This week brought three tightly connected signals that matter for human–agent trust in agentic AI: a mechanistic fragility finding that undercuts assumptions about where safety lives inside LLMs; specification and platform hardening around agent outputs and repo-side automation; and vendor tooling that raises visibility and containment options for enterprise agents.

What changed

1. Unit 42 (Palo Alto Networks) published "Perturbation Probing," a diagnostic showing that safety/refusal behavior can be concentrated in an extremely small set of feed‑forward neurons in tested LLMs — in one experiment ~50 neurons controlled a refusal template — meaning a small, local change can materially alter safety behavior. The paper and Unit 42 blog recommend adding mechanistic diagnostics and defense‑in‑depth guardrails before deployment.

2. The GitHub Agentic Work (GH‑AW) Safe‑Outputs MCP Gateway spec received an update (version notes published Aug 24) tightening validation for pre‑created PR branches, normalizing/stripping undeclared agent‑supplied fields, and requiring explicit allowlists for agent‑supplied comment IDs — concrete changes to reduce ways an agent can escalate or inject privileged metadata into repo workflows. These are immediate, operational rules for running agents against repositories.

3. Platform and tooling updates in late August increased both agent transparency and enterprise controls: GitHub’s August Copilot wave (agent plugins, shared sessions, and client controls) standardizes portable agent plugins and adds UI/attribution for agent skills; Copilot code‑review already reports when comments came from agent skills/MCP context. Microsoft’s August security update calls out agent‑containment playbooks and new visibility features (Agent 365 / Defender exposure tooling) to inventory and constrain agent‑initiated actions. Together these moves push provenance, audit trails, and runtime containment into mainstream workflows.

What to do with it

• Treat model alignment claims as brittle: add a mechanistic fragility check (perturbation probing or equivalent) to pre‑deployment audits and require external runtime guardrails (filters, prompt‑detectors, policy sandboxes).
• Harden your repo agent pipeline: adopt safe‑outputs/MCP validation patterns (strip undeclared fields, validate PR refs and comment IDs, use staged mode) and require explicit trust gates for pre‑created branches. Implement attribution and human‑in‑the‑loop checkpoints for agent PRs.
• Inventory and govern agents: use platform agent registries (e.g., Agent 365‑style inventories), require identity and least‑privilege for agents, and feed agent telemetry into your SIEM/SOAR for runtime monitoring. Prioritize OS/endpoint discovery for local agents and network‑level containment.
• Operational checklist: add a pre‑deploy fragility diagnostic, enforce MCP safe‑outputs validation, require visible attribution for agent outputs in code reviews, and mandate an incident playbook for agent misbehavior.