Executive summary

Co‑pilot is the stabilization phase between HITL and Full Autonomy. The system executes the happy path end‑to‑end while humans focus on exceptions, policy changes and continuous KB updates. We harden drift detection, formalise SLAs, enforce RBAC and respect data‑residency constraints. Graduation to Full Auto only happens when quality, cost and reliability stay within SLOs for multiple consecutive weeks.

1) Objectives & scope

• Keep humans on exceptions and policy-sensitive actions; automate the happy path.

• Maintain living knowledge (KB) so prompts, tools and policies stay current.

• Detect model/data drift early and revert routes safely when needed.

• Operate under explicit SLAs/SLOs with RBAC and data‑residency controls.

2) Operating model — co‑pilot pattern

• Agents execute; humans handle exceptions and approve policy changes.

• Continuous KB updates: new facts, templates, reference decisions and red‑flags.

• Evaluators and trace data provide real‑time quality/latency/cost visibility.

• Canary routes for changes; instant rollback via feature flags.

3) Exceptions — classes & handling

Exception class

Trigger

Routing

Owner

Resolution target

Quality risk

Evaluator < 0.80 or policy watchlist

Exception queue

Senior operator

Same business day

Ambiguity

Low confidence / missing context

Request clarification

Operator

Within 4 business hours

Safety/compliance

PII/PHI/regulatory flag

Dual control + redaction

Security + Operator

Same day

Cost anomaly

>$ cost/item or route drift

Pause route; switch to cheaper model

Ops lead

1 hour

Customer impact

External send to customer

Approval required

Supervisor

2 hours

4) Knowledge base (KB) updates — living system

• Sources: operator rationales, new policies, product changes, resolved exceptions and post‑mortems.

• Mechanics: KB as code (versioned), PR reviews, auto‑deploy to retrieval index; stale entries flagged.

• Validation: evaluator deltas before/after KB update; rollback if regression exceeds threshold.

5) Drift detection — don’t trust, verify

Drift type

Signal

Detector

Mitigation

Data drift

Input distribution shift

Embedding distance / KS‑tests

Refresh KB; retrain; tighten prompts

Model drift

Quality/latency degradation

Evaluator trends; p95 latency

Reroute to stable model; rollback

Cost drift

Tokens/item ↑, cache ↓

Telemetry + anomaly rules

Compress prompts; adjust cache; cap tokens

Policy drift

More violations

Red‑team & policy evaluators

Revert policy; add tests; training

6) SLAs & SLOs — what we promise

Metric

SLO (target)

SLA (guarantee)

Notes

Quality (evaluator median)

≥ 0.85

≥ 0.80

Task‑specific evaluators; sample ≥5%

Latency (p95)

≤ 3.0s

≤ 3.5s

Per step, at business hours

Exception rate

≤ 15%

≤ 20%

Eligible workflows only

Cost per item

−15–25% vs baseline

Within budget cap

Telemetry per route

Uptime (workflow)

≥ 99.5%

≥ 99.0%

Excludes planned maintenance

7) RBAC & data residency — access by design

• RBAC: least privilege roles (Reader, Operator, Supervisor, Engineer, Security, Finance).

• Approvals: dual control for irreversible/external actions.

• Data residency: deploy in requested region/VPC; pin storage and backups; geo‑fencing on logs.

• PII handling: masking/redaction; purpose limitation; retention policy; audit access.

8) Roles & RACI

Role

Responsibilities

R

A

C

I

Operator

Resolve exceptions; annotate KB gaps

R

C

I

Supervisor

Approvals; escalations; coaching

A

C

I

MLE/Engineer

Routes; evaluators; detectors; rollbacks

R

C

I

Security/Legal

Policy updates; audits; DPA

A

R

I

Ops Lead

SLAs/SLOs; staffing; incidents

R

A

C

I

Finance

Budget caps; anomaly reviews

A

C

I

9) Runbooks — day/week/month

Daily: review exceptions; sample 5% outputs; check evaluator/latency/cost dashboards; triage alerts.

Weekly: KB grooming; retrain window (if flagged); threshold tuning; policy review; change log.

Monthly: DR drill; audit sample; budget rebase; red‑team rotation; data‑residency verification.

10) KPIs — graduation gates toward Full Auto

KPI

Definition

Target (2 consecutive weeks)

Notes

Exception rate

% items needing review

≤ 12–15%

Eligible scope

First‑pass yield

% correct w/o edits

≥ 90–92%

Random sample ≥5%

Evaluator median

Task quality

≥ 0.86

Stable IQR

Cost per item

All‑in

−20% vs baseline

Within cap, no anomalies

MTTR incidents

Mean time to recovery

< 2 hours

Sev2 and below

11) Tooling & integration

• Observability: tracing, metrics and logs wired to dashboards; alerting to Slack/Teams.

• KB as code: versioned in Git; CI pushes to retrieval indexes; review required.

• CI/CD: tests in pipeline; canary routes; feature flags; rollback in seconds.

• Optional: Kubernetes (K8s) deployment for autoscaling and isolation when required.

12) Graduation to 04 — Full Autonomy

Requirements: meet KPI targets for 2–4 consecutive weeks; no Sev1 incidents; budget adherence for 3 weeks; KB hygiene score ≥ 90%; completed operator training on new policies.

Documentation: updated runbooks, change logs, and audit samples attached to the Go/No‑Go record.

13) Risks & mitigations

• Stale KB → weekly grooming; ownership per domain; auto‑flags on low hit rates.

• Silent drift → detectors with alerts; freeze promotion on signal; roll back routes.

• Approval fatigue → smarter thresholds; batch approvals; rotating reviewers.

• Privacy gaps → periodic audits; synthetic data for labs; strict residency checks.

14) FAQ

Q: How is Co‑pilot different from HITL?
A: HITL requires routine approvals; Co‑pilot automates the happy path while humans manage exceptions, policies and KB upkeep.

Q: Do we need Kubernetes for Co‑pilot?
A: Not required; we support serverless and VM deployments. K8s is optional for scale/isolation.

Q: How often do we update the KB?
A: Minimum weekly, plus on‑event after policy/product changes.

Book a 30’ Co‑pilot readiness review

Email: contact@smartonsteroids.com — we’ll review your HITL metrics, tune thresholds and plan Co‑pilot rollout.

© 2025 Smart On Steroids — AI Automation Studio → Platform