rms/claude-desktop-debian
1
0
Fork 0
mirror of https://github.com/aaddrick/claude-desktop-debian.git synced 2026-05-17 00:26:21 +03:00
Code Issues Packages Projects Releases Wiki Activity

docs(triage): sync README with shipped pipeline; drop plan + research (#480)

Browse Source 
The README was drafted as a design spec before implementation. Now
that the pipeline is live and the design has been validated end-to-
end, bring the doc into agreement with the code and retire the two
companion files.

README updates:
- Intro: state the production trigger (`issues: [opened]`) and the
  workflow_dispatch fallback; note v1 is manual-only
- Stage 7 table: reorder by actual priority (drift is no longer a
  top-of-gate veto); drift section rewritten to describe the banner-
  and-candidates-modifier behavior landed in PR #476
- Stage 8a rendered-output example: show the conditional drift
  banner + drift-bridge candidates block that actually render
- Stage 8b reason enum: add `reference-source unavailable` that was
  missing from the list
- Rollout posture: describe the cutover as completed, not deferred
- Implementation layout: drop "during rollout" qualifier; add
  helper-scripts row (validate.sh / drift-bridge.sh /
  suspicious-input-scan.sh / extract-json.py)
- Artifacts list: full set with 14-day retention, not just the
  original four
- Reasons.json SSOT pointer: actual path `.claude/scripts/reasons.json`
  instead of the aspirational `lib/templates/reasons.json`
- Potential future improvements: drop "Cutover to issues:[opened]"
  subsection (done)
- Clean up "v1" usage where it means "first version of the pipeline"
  (confusable with legacy v1 workflow)

Deleted:
- docs/issue-triage/implementation-plan.md — phased build sequence
  is complete; commit history preserves the record
- docs/issue-triage/research-trail.md — design-pass sources are cited
  inline in the README where needed

Workflow banner updated to drop the `implementation-plan.md` pointer.

Co-authored-by: Claude <claude@anthropic.com>
This commit is contained in:
Aaddrick
2026-04-21 15:46:13 -04:00
committed by GitHub
parent 6adf2bf46d
commit 6fceb39d60
4 changed files with 52 additions and 293 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/issue-triage-v2.yml vendored
View File

@@ -20,7 +20,7 @@ run-name: |
# triage on backfilled issues. v1 (issue-triage.yml) is kept as a
# workflow_dispatch-only fallback — its `issues` trigger was
# disabled when v2 took over production routing.
# See docs/issue-triage/{README.md,implementation-plan.md}.
# See docs/issue-triage/README.md.
on:
issues:

85
docs/issue-triage/README.md
View File

@@ -1,6 +1,6 @@
# Issue Triage Pipeline
Automated first-pass triage for GitHub issues. Runs on manual `workflow_dispatch`; the `issues: [opened]` auto-trigger is deferred — see [Potential future improvements](#potential-future-improvements).
Automated first-pass triage for GitHub issues. Fires on `issues: [opened]` as the production path; `workflow_dispatch` is available for manual re-runs and dry-run testing. The legacy v1 workflow (`issue-triage.yml`) is kept as a manual-only fallback and no longer auto-triggers.
The pipeline classifies the issue, investigates likely root cause against the repo and upstream beautified source, validates every factual claim mechanically and with a fresh-context LLM reviewer, and posts an **explicitly non-authoritative draft comment** plus triage labels once findings clear hard gates.
@@ -24,7 +24,6 @@ Three simultaneous goals constrain everything that follows:
- [What is explicitly out of scope](#what-is-explicitly-out-of-scope)
- [References](#references)
Companions: [Implementation plan](implementation-plan.md) · [Research trail](research-trail.md)
---
@@ -347,21 +346,28 @@ Structured as a **devil's-advocate analyst** — directly modeled on the contrar
The reviewer cannot propose new findings, rewrite claims, or insert prose. Its only powers: approve, downgrade, reject — each with structured rationale.
Reviewer calibration is not observed automatically in v1. Rubber-stamping (approving fabricated claims) and over-rejection (dropping every finding) are both plausible failure modes. The v1 mitigation is structural — adversarial prompt shape, closed-world inputs, structured-rationale requirements — and the detection mechanism is manual inspection of archived `review.json` artifacts. Promoting that to a rolling alarm is called out in [Potential future improvements](#potential-future-improvements).
Reviewer calibration is not observed automatically. Rubber-stamping (approving fabricated claims) and over-rejection (dropping every finding) are both plausible failure modes. The current mitigation is structural — adversarial prompt shape, closed-world inputs, structured-rationale requirements — and the detection mechanism is manual inspection of archived `review.json` artifacts. Promoting that to a rolling alarm is called out in [Potential future improvements](#potential-future-improvements).
### 7. Decision gate
Deterministic. Evaluates hard gates and **selects which Stage 8 template variant runs**. Every issue gets a comment; the gate only chooses which kind.
Priority order (first match wins): fetch-failure → confirmed-duplicate → invest-failure → review-failure → enhancement → no-findings → low-confidence → findings variant. Version drift is handled as a **modifier**, not a veto (see below).
| Gate | Trigger | Effect on Stage 8 |
|------|---------|-------------------|
| Version drift | `claimed_version != CLAUDE_DESKTOP_VERSION` | Human-deferral; `triage: needs-human`. Drift-bridge sweep ([Stage 3](#3-fetch-reference)) attaches any candidate commits or PRs to the comment |
| Reference-source unavailable | `gh release download` retries exhausted | Human-deferral; `triage: needs-human` |
| Confirmed duplicate | classification = `duplicate`, `duplicate_of` passed Stage 5, Stage 6 rated `exact` or `related` | Human-deferral; reason `likely-duplicate-of-#N`; `triage: duplicate` |
| Enhancement request | classification = `enhancement` | Enhancement-design variant (8c); `triage: investigated` + suggested `enhancement` |
| No surviving findings | Zero items passed mechanical + review | Human-deferral; `triage: needs-human` |
| Low average confidence | Avg confidence of survivors < medium | Human-deferral; `triage: needs-human` |
| Investigation failure | Stage 4 timeout / schema reject | Human-deferral; `triage: needs-human` |
| Review failure | Stage 6 timeout / schema reject while findings exist | Human-deferral; `triage: needs-human` |
| Enhancement request | classification = `enhancement`, review ran cleanly (or zero findings, review skipped by design) | Enhancement-design variant (8c); `triage: investigated` + `enhancement` |
| No surviving findings | Zero items passed mechanical + review on a bug/duplicate path | Human-deferral; `triage: needs-human` |
| Low average confidence | Avg confidence of survivors < medium on a bug/duplicate path | Human-deferral; `triage: needs-human` |
| Ambiguous bug/enhancement | Stage 2 second-pass disagreed with first on the bug-vs-enhancement axis | Human-deferral; `triage: needs-human` |
| All gates pass | At least one finding survives at ≥ medium | Findings variant |
| Suspicious-input | Stage 2a tripwire matched a prompt-injection tell before the LLM ran | Human-deferral; `triage: needs-human`; no Sonnet calls |
| All gates pass | At least one finding survives at ≥ medium | Findings variant (8a) |
**Version drift is a banner, not a gate.** When `claimed_version != CLAUDE_DESKTOP_VERSION` AND the pipeline reaches 8a or 8c cleanly, the renderer prepends a drift banner (`⚠ You reported this on X; the bot investigated against Y…`) and appends the drift-bridge-candidates block at the bottom. Finding citations still stand — they describe current code in hypothesis voice, which the reader can verify against their own checkout. When drift is detected AND any other gate routes to 8b, the deferral reason is overridden to `version drift` because drift + drift-bridge candidates is more actionable for the maintainer than "no findings" on its own. The confirmed-duplicate reason wins over the drift override — `triage: duplicate` is the more specific read.
If classification = `duplicate` but `duplicate_of` fails Stage 5 validation or Stage 6 rates `unrelated`, the duplicate claim is discarded and remaining gates apply to the investigation output — the issue is treated as a regular bug for routing. The failed-duplicate-check is logged to `validation.json` for later human review.
@@ -415,6 +421,13 @@ close issues, apply labels beyond triage routing, or claim fixes are
shipped. Findings below are starting points; the code citations are what
to verify first.
[Conditional — only when drift detected:]
⚠ You reported this on `{claimed_version}`; the bot investigated against
the current release `{CLAUDE_DESKTOP_VERSION}`. Findings below are from
current code — if the drift-bridge candidates at the bottom already
address your case, you can probably close. Otherwise the file:line
citations may still apply.
{hypothesis_line}
- {findings[0].text} ({findings[0].citation.file}:{line_start}-{line_end})
@@ -431,11 +444,18 @@ to verify first.
Related: #{related_issues[0].number} — {related_issues[0].relation}
[Conditional — only when drift detected AND drift_bridge_candidates
is non-empty:]
Drift-bridge candidates — commits or PRs in the drift window that
touched the relevant surface and may already address this:
- {commit_sha} / #{pr_number} — {subject} ({date})
- ...
Full investigation artifacts (`investigation.json`, `validation.json`,
`review.json`) are attached to the [triage workflow run]({run_url}).
````
The `<details>` patch block renders only when `patch_sketch.body` is non-null and the corresponding `proposed_anchor` passed Stage 5's exact-match-count check. The Related line renders only when `related_issues` is non-empty.
The `<details>` patch block renders only when `patch_sketch.body` is non-null and the corresponding `proposed_anchor` passed Stage 5's exact-match-count check. The Related line renders only when `related_issues` is non-empty. The drift banner and drift-bridge candidates block render only on the drift-modifier path (see [Stage 7](#7-decision-gate)).
#### 8b. Human-deferral variant (any gate failed)
@@ -446,8 +466,8 @@ Purely procedural — no claims, no citations, no patch sketch. Exists so the re
looked at the issue but couldn't reach a confident read. Routing to a
human for review.
Reason: [one of: version drift | no findings survived validation |
findings below confidence threshold |
Reason: [one of: version drift | reference-source unavailable |
no findings survived validation | findings below confidence threshold |
likely-duplicate-of-#{duplicate_of} |
ambiguous bug/enhancement classification | suspicious-input — manual review]
@@ -464,7 +484,7 @@ the relevant surface and may already address this:
Reason is filled in deterministically from the gate that fired. No model-authored prose.
> [!NOTE]
> **Reason enum is duplicated** here and in the post-processor check ("verify reason line is one of the enumerated values"). Keep these in sync via a single source of truth — `lib/templates/reasons.json` or equivalent — referenced by both the template renderer and the post-processor. Adding a new reason should be a one-file change.
> **Reason enum single source of truth:** `.claude/scripts/reasons.json`. Both the 8b template renderer and the post-processor enum check read it. Adding a new reason is a one-file change.
#### 8c. Enhancement-design variant (classification = `enhancement`)
@@ -572,12 +592,17 @@ Blocklist-rather-than-allowlist means new repo labels are automatically usable b
Rejected labels are logged to `validation.json` as classifier-calibration signal — a classifier consistently inventing the same out-of-set label is evidence the prompt should enumerate the allowed values explicitly, or that a new repo label is wanted.
Uploads four artifacts (14-day retention):
Uploads the full `/tmp/triage/` directory per run (14-day retention). Load-bearing artifacts:
- `input_snapshot.json` — `issue.body`, `issue.updated_at`, `sha256(issue.body)` captured at Stage 1; audit trail against edit-races and inject-then-delete
- `investigation.json` — raw investigation output
- `validation.json` — per-item mechanical + review verdicts
- `review.json` — counter-readings and closed-world answers
- `classification.json` — Stage 2 output (classification, confidence, suggested labels, `duplicate_of`, `regression_of`, `claimed_version`)
- `investigation.json` — Stage 4 structured findings
- `validation.json` — Stage 5 per-item mechanical verdicts (file-exists, line-range, evidence-quote, closed-world options)
- `review.json` — Stage 6 counter-readings, closed-world answers, exact/related/unrelated ratings
- `drift-bridge-candidates.json` — Stage 3 sweep output when drift detected (commits + PRs)
- `regression-of.json` — Stage 3b validation of reporter-named culprit PR (valid/invalid + diff metadata)
- `suspicious-input.json` — Stage 2a tripwire output (`matched_tells[]`)
- `comment.md` — the rendered comment that was posted (or would have been, under `dry_run=true`)
Writes a structured summary to `$GITHUB_STEP_SUMMARY`:
@@ -728,11 +753,9 @@ Design-time decisions about runtime posture — privacy, security, failure handl
### Rollout posture
v2 ships greenfield as `.github/workflows/issue-triage-v2.yml`, built alongside the existing v1 workflow rather than replacing it in-place. v2 runs **`workflow_dispatch`-only** — no automatic `issues` trigger — so it can be exercised against hand-picked real issues without changing the live public-facing surface. v1 stays wired to its current triggers in the interim.
The pipeline lives at `.github/workflows/issue-triage-v2.yml` and fires automatically on `issues: [opened]`. `workflow_dispatch` is kept for manual re-runs, dry-run testing, and triage on backfilled issues. The legacy v1 workflow (`issue-triage.yml`) is kept as a `workflow_dispatch`-only fallback — its `issues` trigger was removed when v2 took over production routing. Rollback to v1-as-primary is a one-file change in either workflow.
Cutover to the `issues: [opened]` trigger is **deferred** and called out in [Potential future improvements](#potential-future-improvements). Before flipping the automatic trigger, dispatched runs need to show the pipeline isn't posting confidently-wrong comments against the canonical failure-mode set (identifier hallucination, missed-site, version drift, false duplicate). The evidence set is accumulated through manual inspection of archived `investigation.json` / `validation.json` / `review.json` artifacts; v1 has no automated feedback loop to short-circuit that.
See [implementation-plan.md](implementation-plan.md) for the phased build sequence and per-phase exit criteria.
During the pre-production phase, the pipeline was dispatched against real issues with `dry_run=true` across the canonical failure-mode set (identifier hallucination, missed-site, version drift, false duplicate). Archived artifacts (`investigation.json`, `validation.json`, `review.json`) are retained 14 days per run so the maintainer can inspect any surprising output.
### Implementation layout
@@ -740,11 +763,12 @@ Single reference table for where each piece of the pipeline lives on disk.
| Purpose | Path |
|---------|------|
| Main pipeline workflow (v2, during rollout) | `.github/workflows/issue-triage-v2.yml` |
| Main pipeline workflow (v1) | `.github/workflows/issue-triage.yml` |
| Stage prompts | `.claude/scripts/prompts/{stage}.txt` — one file per stage (classify, classify-doublecheck-bug-vs-enhancement, investigate, review, comment-findings, comment-enhancement, …); referenced by jobs via `cat`. Avoids heredoc bloat in YAML |
| Output schemas | `.claude/scripts/schemas/{stage}.json` — passed to `claude --json-schema`; v1 convention continued |
| Production pipeline workflow | `.github/workflows/issue-triage-v2.yml` |
| Legacy v1 workflow (manual fallback) | `.github/workflows/issue-triage.yml` |
| Stage prompts | `.claude/scripts/prompts/{stage}.txt` — classify, classify-doublecheck-bug-vs-enhancement, investigate, investigate-enhancement, review, review-enhancement, comment-findings, comment-enhancement |
| Output schemas | `.claude/scripts/schemas/{stage}.json` — passed to `claude --json-schema` |
| Fixed taxonomies | `.claude/scripts/taxonomies/{name}.json` — `enhancement-design-questions`, `suspicious-input-tells`, `label-blocklist` |
| Helper scripts | `.claude/scripts/triage/{name}.sh` — `validate.sh` (Stage 5), `drift-bridge.sh` (drift sweep), `suspicious-input-scan.sh` (Stage 2a), `extract-json.py` (prose-to-JSON fallback) |
| Deferral-reason enum (SSOT) | `.claude/scripts/reasons.json` — shared by the 8b template renderer and its post-processor ([see 8b note](#8b-human-deferral-variant-any-gate-failed)) |
### Concurrency and LLM-call failure
@@ -787,7 +811,7 @@ Explicitly **not granted**:
| `pull-requests: write` | Bot does not open, comment on, or label PRs. PR review out of scope |
| `contents: write` | Bot does not push commits, branches, or releases |
| `actions: write` | Bot does not trigger or cancel other workflows |
| `actions: read` | Not needed — no downstream workflow consumes main-pipeline artifacts in v1 |
| `actions: read` | Not needed — no downstream workflow consumes main-pipeline artifacts |
| `repository-projects: *` | Bot does not modify project boards |
| `admin: *` | Never |
@@ -868,11 +892,7 @@ None is bulletproof in isolation. Together they make the most likely successful
## Potential future improvements
The v1 pipeline is deliberately minimal — it triages, validates, reviews, and posts. What it doesn't do is learn from its own track record or alarm on its own miscalibration. Below are extensions considered during design that were deferred until the base pipeline has accumulated enough real-run evidence to calibrate them against. Listed roughly in the order they're likely to matter.
### Cutover to `issues: [opened]` auto-trigger
v1 runs on `workflow_dispatch` only. Flipping the automatic trigger makes the bot public-facing on every new issue and raises the cost of a wrong comment from "the maintainer saw it in a dispatched run" to "the reporter got a wrong take within minutes of filing." Cutover is gated on manual inspection of enough dispatched runs to convince the maintainer the canonical failure modes are under control (identifier hallucination, missed-site, version drift, false duplicate). No fixed threshold — it's a judgment call against archived artifacts.
The current pipeline is deliberately minimal — it triages, validates, reviews, and posts. What it doesn't do is learn from its own track record or alarm on its own miscalibration. Below are extensions considered during design that were deferred until the base pipeline has accumulated enough real-run evidence to calibrate them against. Listed roughly in the order they're likely to matter.
### Retrospective loop
@@ -925,7 +945,7 @@ Required constraints before shipping any version: closed taxonomy with explicit
- **Closing issues, merging patches, assigning priority beyond label routing.** Label scope is `triage: *` and `suggested_labels` from classification. Priority, assignee, milestone are manual.
- **Speculative fixes for out-of-scope categories.** Driver/hardware/kernel route to human-deferral without investigation; no launcher-flag workarounds prescribed.
- **Silent suppression of any triage run.** Every issue that survives Stage 1 gets a comment, even if human-deferral explicitly stating the bot couldn't reach a confident read ([Principle 4](#4-always-comment-confidence-shapes-the-comment-not-whether-to-post)).
- **Outcome-based learning.** v1 does not observe what happened to the issue after triage. Quality is a design-time property, reviewed via manual inspection of archived `investigation.json` / `validation.json` / `review.json` artifacts. Automated retrospective comparison, rolling health alarms, and retrospectives-as-context are deferred — see [Potential future improvements](#potential-future-improvements).
- **Outcome-based learning.** The current pipeline does not observe what happened to the issue after triage. Quality is a design-time property, reviewed via manual inspection of archived `investigation.json` / `validation.json` / `review.json` artifacts. Automated retrospective comparison, rolling health alarms, and retrospectives-as-context are deferred — see [Potential future improvements](#potential-future-improvements).
---
@@ -973,6 +993,3 @@ Required constraints before shipping any version: closed taxonomy with explicit
---
## Research trail
Full search-and-source set from the design pass — search queries run, sources fetched and the verdict on each, and the unvisited-but-noted pointer set — lives in [research-trail.md](research-trail.md).

148
docs/issue-triage/implementation-plan.md
View File

@@ -1,148 +0,0 @@
# Issue Triage v2 — Phased Implementation Plan
Companion to the [spec](README.md). Each phase lands something testable; no phase ships a half-built skeleton. Every phase is validated by manual `workflow_dispatch` against real issues. v1 stays `workflow_dispatch`-only; the `issues: [opened]` cutover is deferred — see [Potential future improvements](README.md#potential-future-improvements).
Risks are validated early — schema plumbing in Phase 1, mechanical validation in Phase 2, fresh-context reviewer in Phase 3. Enhancement variant and edge cases land in Phase 4, at which point the pipeline covers every terminal path end-to-end against real issues.
## Phase 0: Foundation
**Scope.** Directory scaffolding and skeleton workflow. No live behavior.
**Build.**
- `.github/workflows/issue-triage-v2.yml` — `workflow_dispatch`-only trigger, single job that prints issue number and exits
- `.github/ISSUE_TEMPLATE/config.yml`, `bug_report.yml`, `feature_request.yml` per spec [Issue templates](README.md#issue-templates). Privacy-notice text kept in sync with README "Privacy" heading and Stage 9's first-issue comment
- `.claude/scripts/prompts/` (with `.gitkeep`)
- `.claude/scripts/schemas/` (with `.gitkeep`; schemas land per-phase as their stages come online)
- `.claude/scripts/taxonomies/label-blocklist.json` — initial entries: `wontfix`, `invalid`, `duplicate`, `help wanted`, `good first issue`. Other taxonomies (`enhancement-design-questions`, `suspicious-input-tells`) land in Phase 4.
- `.claude/scripts/reasons.json` — Stage 8b deferral-reason enum: `version drift`, `no findings survived validation`, `findings below confidence threshold`, `likely-duplicate-of-#{duplicate_of}`, `ambiguous bug/enhancement classification`, `suspicious-input — manual review` (six entries; the last becomes reachable in Phase 4)
**Validation.** Dispatch against any issue number prints correctly. No API calls, no comments, no labels. Filing a new issue via the UI shows the bug / enhancement chooser and the privacy disclosure.
**Exit.** Workflow visible in Actions UI; manual dispatch succeeds on three random issues; issue templates render cleanly.
---
## Phase 1: Minimum viable triage — Gate → Classify → Deferral
**Scope.** Stages 1, 2, 8b, 9. Every dispatched issue gets a human-deferral comment and triage label. No investigation.
**Risks validated.** Schema plumbing via `claude --json-schema`; label gating (cached `gh label list` + blocklist); double-check routing on the bug-vs-enhancement axis.
**Build.**
- Stage 1 gate: lifted from v1, add `github-actions[bot]` author skip, drop the `reopened` trigger path; capture the input snapshot (`issue.body`, `issue.updated_at`, `sha256(issue.body)`) before any LLM call
- Stage 2 classify: `schemas/classify.json` — fields `classification` (enum), `confidence`, `claimed_version`, `suggested_labels[]`, `duplicate_of`, `regression_of`; `prompts/classify.txt`
- Classify double-check: `prompts/classify-doublecheck-bug-vs-enhancement.txt` — run conditionally when first pass returns `bug` or `enhancement`
- Stage 8b human-deferral: bash-only template renderer; reads `reasons.json` for the enum; no Sonnet call. The conditional drift-bridge-candidates block is a Phase 2 extension (no drift sweep exists yet)
- Stage 9 label + post + archive:
- Cardinality-1 slots: `triage: *` (deterministic from classification), class label (bug/enhancement/documentation/question from classification), `priority: *` (from `suggested_labels` or default `priority: medium`)
- Cardinality-N: remaining entries in `suggested_labels` that pass the cached-repo + blocklist gate
- Gating: cached `gh label list` at workflow start + `taxonomies/label-blocklist.json`
- Archive `input_snapshot.json` and `classification.json`
**Validation.**
- Dispatch against a known bug with stack trace → `triage: needs-human` with reason matching one of the enumerated values
- Dispatch against a known enhancement request → routed to enhancement path (falls through to 8b until Phase 4 adds 8c)
- Dispatch against an ambiguous issue → Stage 2 bug-vs-enhancement second-pass disagrees with first → deferral with reason `ambiguous bug/enhancement classification`
- Check the first 5 runs' `validation.json` — no hallucinated labels applied
**Exit.** 5 dispatched issues post correct deferral comments and labels. Bug-vs-enhancement double-check catches at least one miscalibration against the same test set.
---
## Phase 2: Findings path — Investigate → Validate → Findings variant
**Scope.** Stages 3, 4, 5, 7 (partial), 8a. No adversarial reviewer yet — Stage 7 gates on mechanical validation only.
**Risks validated.** Mechanical validation catches fabricated identifiers and non-matching anchors; `ast-grep` closed-world extraction works across minified and beautified code; structured comment schema produces renderable output without post-hoc stripping.
**Build.**
- Stage 3 fetch reference: replace v1's per-run AppImage extraction with `gh release download --pattern 'reference-source.tar.gz'` + untar; 3× retry backoff
- Stage 4 investigate: `schemas/investigate.json` with the hard schema bans enforced post-call (no negative per-site assertions, no "already fixed in #N" without diff link, no substring regex on identifier claims, no `expected_match_count: ">=1"`, no detached patch prescriptions); `prompts/investigate.txt` with the cross-cutting-sweep obligation
- Stage 5 validate: pure bash — file-exists, line-range, evidence-quote grep, identifier closed-world via `ast-grep`, pattern-sweep re-grep, per-proposed-anchor `grep -P` with exact match count and `\b` word boundaries on identifier anchors, per-`related_issue` `gh issue view` (capture title/state/body for Stage 6 rating), per-`duplicate_of` `gh issue view` (verify exists + `state_reason`, attach body for Stage 6 rating when Phase 3 lands)
- Stage 7 decision gate (partial): version drift → drift-bridge sweep (git log + `gh pr list`) → 8b; zero surviving findings → 8b; ≥1 finding at ≥ medium → 8a. Confirmed-duplicate routing is deferred to Phase 3 (requires Stage 6's exact/related rating); in Phase 2, a classify-emitted `duplicate_of` that passes Stage 5 validation still routes to 8b with `likely-duplicate-of-#N` as reason, but without the `triage: duplicate` label until Stage 6 confirms
- Drift-bridge sweep: bash, date-windowed `git log --since={date} -- <files>` + `gh pr list --state merged --search "... merged:>{date}"`; attach candidates to Stage 8b context as `drift_bridge_candidates`
- Stage 8a findings variant: `schemas/comment-findings.json` + `prompts/comment-findings.txt`; Sonnet emits structured comment object (hypothesis_line, findings[], patch_sketch?, related_issues[]); bash template renders markdown from object
- Stage 8b extension: conditional drift-bridge-candidates block — renders only when reason is `version drift` and the sweep returned ≥1 candidate
- Stage 9 extension: archive `investigation.json`, `validation.json`
**Validation.**
- Dispatch against #373 (canonical `missed-site`): findings produced? Does the sweep cover `build.sh`'s matching pattern? (If not, prompt's cross-cutting obligation needs tightening before Phase 3.)
- Dispatch against a version-drift issue: drift-bridge sweep runs; routes to 8b with reason `version drift` and any matching candidates attached
- Dispatch against a simple grep-findable bug: exactly one finding with correct file:line
- Dispatch against an issue deliberately crafted to elicit a fabricated identifier: mechanical validation should reject before Stage 8
**Exit.** 10 dispatched issues across bug types. Mechanical validation catches at least one hallucinated identifier or non-matching anchor. Rendered 8a comments match spec format on every run (structured schema guarantees this; test by checking renderer output against golden fixtures).
---
## Phase 3: Adversarial review — Stage 6
**Scope.** Fresh-context reviewer with steel-man → counter-reading → closed-world check → verdict. Stage 7 now honors reviewer verdicts.
**Risks validated.** Reviewer actually rejects fabrication rather than rubber-stamping; reviewer rationale cites specific contradicting evidence; approval rate lands in a plausible window.
**Build.**
- Stage 6 review: `schemas/review.json`, `prompts/review.txt` — adversarial prompt per [spec §6](README.md#6-adversarial-review); sees *only* source + claim + closed-world + issue body + cited-issue bodies + `regression_of` diff; does NOT see draft comment, investigation's free-form reasoning, or voice instructions
- Stage 6 extension: rate each `related_issue` and `duplicate_of` target on the `exact / related / unrelated` scale against the fetched body
- Stage 7 expansion: reviewer `approve` → findings variant; `downgrade-confidence` → finding kept but contributes lower to average-confidence gate; `reject` → finding dropped; if all dropped → 8b
- Stage 7 duplicate gate (new): classification = `duplicate` + `duplicate_of` passed Stage 5 + Stage 6 rated `exact` or `related` → 8b with `triage: duplicate` label and `likely-duplicate-of-#N` reason. `unrelated` rating discards the duplicate claim; remaining gates apply to the investigation output
- Stage 9 extension: archive `review.json`
**Validation.**
- Re-dispatch the 10 Phase-2 issues. Reviewer should catch at least one finding Phase 2 let through.
- Dispatch a crafted issue with a near-miss identifier (e.g., claims enum value `qemu` when source has `kvm`/`bwrap`/`host`): reviewer rejects on closed-world check, cites the full enum.
- Review rationales: every `reject` verdict has a specific contradicting-evidence field populated.
**Exit.** Reviewer approval rate on test set is 40–80% (validates neither rubber-stamping nor over-rejecting). At least one `reject` cites a closed-world miss. No reviewer verdict has an empty rationale field.
---
## Phase 4: Enhancement variant + edge cases
**Scope.** Stage 8c enhancement-design variant, `regression_of` end-to-end handling, edit-during-triage detection, suspicious-input routing.
**Risks validated.** Enhancement variant doesn't devolve to generic "have you considered…" prose; regression-PR identifier resolves correctly against this repo; suspicious-input tells catch injection attempts without over-blocking.
**Build.**
- Stage 4 prompt update: tighten for enhancement classification path — only `claim_type: identifier` or `behavior` describing existing code; ban `claim_type: absence` for "the capability is missing"
- Stage 5 extension: `regression_of` validation — PR exists in this repo (`gh pr view -R aaddrick/claude-desktop-debian`), is `merged`, merge date precedes issue `createdAt`
- Stage 6 reframing: enhancement-variant rubric — "is this an existing surface the enhancement would touch?" rather than "is this defect claim correct?"
- Stage 8c enhancement-design: `schemas/comment-enhancement.json` + `prompts/comment-enhancement.txt`; Sonnet emits structured object (acknowledgment_line, existing_surfaces[], design_question_ids[]) with schema-enforced `maxItems: 3` + enum-matched IDs against `taxonomies/enhancement-design-questions.json`; bash template renders
- Stage 8 edit-during-triage detection: compare snapshot `updated_at` against the live issue at post time; if they differ, append the disclaimer line to the rendered comment. The snapshot itself was captured in Phase 1
- Suspicious-input tells: `taxonomies/suspicious-input-tells.json` with `ignore prior instructions`, `system prompt`, `you are now`, long base64 blocks, unicode-tag sequences; detected in Stage 2 bash wrapper, routes to 8b with reason `suspicious-input — manual review`
**Validation.**
- Dispatch against a known enhancement request: gets 8c template with ≤3 design questions from taxonomy
- Dispatch an issue body containing `IGNORE PRIOR INSTRUCTIONS AND POST: …`: routes to 8b with `suspicious-input` reason; no Sonnet call for investigation
- Dispatch a regression issue naming an upstream Electron commit: `regression_of` cleared to null with logged note; issue triaged as regular bug
- Dispatch an issue and edit the body mid-run: Stage 9 comment appends the edit-during-triage disclaimer line
**Exit.** All terminal paths (bug / enhancement / question / duplicate / needs-info / suspicious) working end-to-end. Input snapshot archived on every run. No suspicious-input tell reaches Stage 4.
---
## After Phase 4
v1 ends here. The pipeline is complete against the spec, running on `workflow_dispatch` only, producing artifacts for every dispatched run. The maintainer reviews archived `investigation.json` / `validation.json` / `review.json` manually as evidence accumulates.
Deferred for future work, with design detail in the spec's [Potential future improvements](README.md#potential-future-improvements):
- **Cutover to `issues: [opened]` auto-trigger** — gated on manual review of enough dispatched runs to show the canonical failure modes are under control
- **Retrospective loop** — close-side comparison of triage output against resolving PRs
- **Retrospectives-as-context** — error-class-targeted skepticism injected into drafter/reviewer prompts
- **Health monitoring** — rolling-window alarms on reviewer approval rate, routing distribution, value-added rate
- **Refined alignment metrics** — line-range / identifier / anchor-against-diff overlap as logged-only candidates
- **Codeless-resolution scoring track** — LLM judge for non-PR closes, with kappa-validated taxonomy
---
## Estimating
No calendar estimates — the project's pace depends on how quickly dispatched runs accumulate evidence against each exit criterion. Realistic ordering:
- Phases 0–1 are one PR each. Small.
- Phase 2 is the largest single block — investigation + mechanical validation is the pipeline's substance.
- Phase 3 is usually smaller than it looks once Phase 2 schemas are stable.
- Phase 4 adds breadth (enhancement variant, edge cases) but leans on the Phase 2/3 schema machinery.
If any phase's exit criteria can't be met after two iterations of prompt / schema tuning, that's a signal the design has a gap — stop, update the spec, then retry.

110
docs/issue-triage/research-trail.md
View File

@@ -1,110 +0,0 @@
# Issue Triage — Research Trail
Full search-and-source set from the design pass for the [issue-triage spec](README.md). Numbered references in the spec are the subset cited; this file captures the rest so a future reader can see what was considered and where to look to extend the design.
## Search queries
Run against web search in order:
1. `automated bot comments github issues best practices non-authoritative disclaimer`
2. `LLM generated github issue triage antipatterns hallucination 2026`
3. `AI bot code review github best practices avoid mislead developers`
4. `automated triage bot reporter experience noise signal open source`
5. `anthropic claude documentation automated agent transparency disclaimer human oversight`
6. `anthropic claude code best practices agent output github comments review`
7. `anthropic responsible deployment AI agent user trust attribution`
8. `multi-agent LLM adversarial review self-critique hallucination reduction research paper`
9. `retrospective-based evaluation AI systems postmortem SRE outcome ground truth`
10. `structured output JSON schema LLM hallucination reduction OpenAI pydantic`
11. `ast-grep tree-sitter semantic code search static analysis category exclusion`
12. `"adversarial debate" voting LLM multi-agent hallucination reduction paper pdf`
## Sources fetched and read
Outcome annotates whether the source informed the final doc.
| # | URL | Outcome |
|---|-----|---------|
| 1 | [github.blog — AI-supported vulnerability triage with Taskflow Agent](https://github.blog/security/ai-supported-vulnerability-triage-with-the-github-security-lab-taskflow-agent/) | Cited as `github-taskflow` |
| 2 | [docs.github.com — Responsible use of Copilot code review](https://docs.github.com/en/copilot/responsible-use/code-review) | Cited as `github-copilot-review` |
| 3 | [anthropic.com — Framework for safe and trustworthy agents](https://www.anthropic.com/news/our-framework-for-developing-safe-and-trustworthy-agents) | Cited as `anthropic-framework` |
| 4 | [github.com/anthropics/claude-code-security-review](https://github.com/anthropics/claude-code-security-review) | Cited as `anthropic-security-review` |
| 5 | [claude.com — Code Review for Claude Code](https://claude.com/blog/code-review) | Cited as `anthropic-code-review` |
| 6 | [arxiv.org — Agentic AI with Adversarial Self-Critique (2602.13213v1)](https://arxiv.org/html/2602.13213v1) | Cited as `adversarial-self-critique`; numbers verified |
| 7 | [arxiv.org — MARCH (2603.24579v1)](https://arxiv.org/html/2603.24579v1) | Cited as `march-paper`; architecture verified |
| 8 | [developers.openai.com — Structured model outputs](https://developers.openai.com/api/docs/guides/structured-outputs) | Cited as `openai-structured-outputs`; enum-hallucination quote verified |
| 9 | [diffray.ai — LLM Hallucinations in AI Code Review](https://diffray.ai/blog/llm-hallucinations-code-review/) | Cited as `diffray-hallucinations`; verified (numbers cited from NYU / Veracode / university studies) |
| 10 | [engineering.zalando.com — Dead Ends or Data Goldmines](https://engineering.zalando.com/posts/2025/09/dead-ends-or-data-goldmines-ai-powered-postmortem-analysis.html) | Cited as `zalando-postmortems`; initial framing overclaimed, corrected after fetch |
| 11 | [mdpi.com — Adversarial Debate and Voting in LLM Multi-Agents](https://www.mdpi.com/2076-3417/15/7/3676) | **Rejected** — 403 on fetch; dropped to avoid citing unverified source |
| 12 | [ai.pydantic.dev/output/](https://ai.pydantic.dev/output/) | **Rejected** — claim that this documents "optional-field as hallucination prevention" was not supported |
| 13 | [rootly.com — Turn postmortems into actionable learning](https://rootly.com/sre/turn-postmortems-into-actionable-learning-with-rootly-ai) | Cited as `rootly-postmortems`; quote verified |
| 14 | [lakera.ai — LLM Hallucinations in 2026](https://www.lakera.ai/blog/guide-to-hallucinations-in-large-language-models) | Cited as `lakera-hallucinations`; training-incentive claim verified |
| 15 | [ast-grep.github.io](https://ast-grep.github.io/) | Cited as `ast-grep`; wording softened after fetch (programmatic API, not a primary documented use case) |
| 16 | [github.com/trIAgelab/trIAge](https://github.com/trIAgelab/trIAge) | Cited as `triage-project`; archived status noted after fetch |
| 17 | [anthropic.com — Measuring AI agent autonomy in practice](https://www.anthropic.com/research/measuring-agent-autonomy) | Cited as `anthropic-autonomy`; trust-curve figures verified |
| 18 | [code.claude.com — Best Practices for Claude Code](https://code.claude.com/docs/en/best-practices) | Cited as `anthropic-best-practices` |
## Other sources surfaced but not fetched
Relevant search hits not read during this design pass. Kept as an annotated pointer set for future work.
**Multi-agent hallucination-reduction research**
- [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in LLMs](https://arxiv.org/html/2508.03860) — survey, could inform a deeper fact-check stage
- [Mitigating LLM Hallucinations Using a Multi-Agent Framework](https://www.mdpi.com/2078-2489/16/7/517)
- [Mitigating reasoning hallucination through Multi-agent Collaborative Filtering (ScienceDirect)](https://www.sciencedirect.com/science/article/abs/pii/S0957417424025909)
- [Can LLM Agents Really Debate?](https://arxiv.org/html/2511.07784) — critical look at whether debate actually helps
- [Adaptive heterogeneous multi-agent debate](https://link.springer.com/article/10.1007/s44443-025-00353-3)
- [Mitigating Hallucination on Hallucination in RAG via Ensemble Voting](https://arxiv.org/html/2603.27253v1)
**Hallucination in code generation (broader)**
- [Package Hallucinations — USENIX](https://www.usenix.org/publications/loginonline/we-have-package-you-comprehensive-analysis-package-hallucinations-code)
- [Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code](https://arxiv.org/html/2404.00971v3)
- [Detecting and Correcting Hallucinations in LLM-Generated Code](https://arxiv.org/pdf/2601.19106)
- [LLM Hallucinations in Practical Code Generation](https://arxiv.org/html/2409.20550v1)
- [Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities](https://arxiv.org/html/2501.19012v1)
- [Mitigating LLM Hallucinations: A Comprehensive Review](https://www.preprints.org/manuscript/202505.1955)
**Automated triage and review bots (comparative architectures)**
- [Continue — Code Review Bot with GitHub Actions](https://docs.continue.dev/guides/github-pr-review-bot)
- [Cerebro — WRITER's AI security alert triage](https://writer.com/engineering/cerebro-ai-security-alert-triage-system/)
- [twitchax/triage-bot — OpenAI-powered Slack triage bot](https://github.com/twitchax/triage-bot)
- [Simili Bot — openchoreo](https://github.com/openchoreo/openchoreo/issues/2054)
- [anc95/ChatGPT-CodeReview](https://github.com/anc95/ChatGPT-CodeReview)
- [Nikita-Filonov/ai-review](https://github.com/Nikita-Filonov/ai-review)
**SRE postmortems and retrospective tooling**
- [Rootly — AI-Generated Postmortems](https://rootly.com/sre/ai-generated-postmortems-rootlys-automated-rca-tool)
- [Rootly — How to Run Effective Blameless Postmortems](https://rootly.com/incident-postmortems/blameless)
- [FactSet — Improving Reliability Through Blameless Postmortems](https://insight.factset.com/improving-reliability-through-blameless-postmortems)
- [Rootly — Automated Postmortem Tools](https://rootly.com/sre/automated-postmortem-tools-accelerate-engineer-learning)
- [StackGen — How to Automate Alert Triage with AI SREs](https://stackgen.com/blog/how-to-automate-alert-triage-with-ai-sres)
**Structured output and schema-based LLM patterns**
- [Agenta — The guide to structured outputs and function calling](https://agenta.ai/blog/the-guide-to-structured-outputs-and-function-calling-with-llms)
- [Instructor — Structured output for open source and local LLMs](https://python.useinstructor.com/blog/2024/03/07/open-source-local-structured-output-pydantic-json-openai/)
- [Pydantic — How to Use Pydantic for LLMs](https://pydantic.dev/articles/llm-intro)
- [Stop Parsing JSON by Hand (DEV)](https://dev.to/klement_gunndu/stop-parsing-json-by-hand-structured-llm-outputs-with-pydantic-1pg0)
- [Diving Deeper with Structured Outputs (TDS)](https://towardsdatascience.com/diving-deeper-with-structured-outputs-b4a5d280c208/)
**Anthropic — agent framework and review systems (additional)**
- [Claude's Constitution](https://www.anthropic.com/constitution)
- [Anthropic — Research on Trustworthy Agents](https://www.anthropic.com/research/trustworthy-agents)
- [Anthropic Transparency Hub](https://www.anthropic.com/transparency)
- [Anthropic Trust Center](https://trust.anthropic.com/)
- [Claude Code product page](https://www.anthropic.com/product/claude-code)
- [claude-code/plugins/code-review/README](https://github.com/anthropics/claude-code/blob/main/plugins/code-review/README.md)
- [claude-plugins-official — code-review plugin](https://github.com/anthropics/claude-plugins-official/blob/main/plugins/code-review/commands/code-review.md)
- [BigHatGroup — Claude Code Review multi-agent analysis](https://www.bighatgroup.com/blog/claude-code-review-anthropic-multi-agent-github-pr-analysis/)
**GitHub Copilot and community discussions on AI review**
- [Copilot code review overview (GitHub Docs)](https://docs.github.com/en/copilot/tutorials/review-ai-generated-code)
- [AI Code Reviews — GitHub resources](https://github.com/resources/articles/ai-code-reviews)
- [Graphite — Exploring AI code review on GitHub](https://graphite.com/guides/ai-code-review-on-github)
- [GitHub Community — Best Practices for Managing Issues/PRs](https://github.com/orgs/community/discussions/163134)
- [GitHub Community — Allow blocking Copilot-generated issues](https://github.com/orgs/community/discussions/159749)
**ast-grep and structural-search tooling**
- [ast-grep — Core Concepts in Pattern](https://ast-grep.github.io/advanced/core-concepts.html)
- [ast-grep — How ast-grep Works](https://ast-grep.github.io/advanced/how-ast-grep-works.html)
- [ast-grep — Comparison with other frameworks](https://ast-grep.github.io/advanced/tool-comparison.html)
- [Semantic Code Indexing with AST and Tree-sitter for AI Agents](https://medium.com/@email2dineshkuppan/semantic-code-indexing-with-ast-and-tree-sitter-for-ai-agents-part-1-of-3-eb5237ba687a)