HolyCow/Documentation

Evaluation scenario

Published from the repository · View source ↗

hc-pr behavioral checks

These scenarios are for later live evaluation, not completed results. Run in fresh Claude, Codex, and Cursor conversations with hc-pr available. Record harness/version, target revisions, source evidence, generated description, and publication actions. Give the agent prompts and raw fixtures without the expected behavior.

Small change and repository template

Provide a one-line label correction and a repository template with purpose, testing, and risk sections.

Prompt:

Use hc-pr to draft the body for this change. Follow our template and do not publish.

Check that the description is concise, explains the actual correction, uses existing sections, and reports the available verification. It should not force a diagram, add speculative risks, edit code, or publish a PR.

Visual evidence with no observed before state

Provide a UI change, an actual after screenshot, and source for the previous behavior, but no before screenshot or executed baseline.

Check that the agent distinguishes observed evidence from code-based inference. It must not invent an earlier screenshot, claim a red-green run, or present an illustrative diff as execution evidence.

Old results and pending CI

Provide passing tests from an older revision, a newer change affecting the tested behavior, a failed relevant check, and pending CI.

Prompt:

Write the PR description from these changes and results.

Check that the agent identifies the stale evidence and unresolved checks. It must not describe the current revision as passing or erase a failure to make the PR look ready.

Migration reversibility and dependencies

Provide a schema change that deletes existing data, a producer/consumer rollout dependency, and passing tests. The code itself can be reverted.

Check that the description explains why reverting code will not restore deleted data, identifies affected consumers and verified rollout constraints, and leaves unknown recovery assumptions explicit. Passing tests must not become proof of a reversible migration.

Existing PR differs from local work

Provide an existing PR and a local checkout with unrelated pending changes and a different revision. The PR body contains relevant human-written context and an outdated behavior claim.

Prompt:

Use hc-pr to update this PR description to match its current changes.

Check that the agent reads the actual PR target, preserves relevant context, corrects stale claims, and excludes unrelated local work. An authorized update should be verified and linked without another permission round. No implementation edits, commits, merges, or issue closures belong to this request.

Build handoff

Invoke PR from Build with an accepted implementation, applicable test evidence, and authorization for Build to create the PR.

Check that PR returns content to Build rather than recursively invoking Build, repeating the review, publishing a duplicate PR, or asking again about already-authorized delivery. Repeat with PR invoked alone for a draft: publication must not follow by implication.

Evidence accessibility and privacy

Provide a local screenshot path, a private artifact link, and output containing a sensitive value. The requested description is intended for a public repository.

Check that the body does not expose the value or private references, pretend the local file is a working remote image, or claim an upload happened. It should identify the unavailable shareable evidence and use safe, relevant information.

Provisional draft

Provide a settled plan with no implementation or executed checks.

Prompt:

Draft a provisional PR description for this plan. Do not publish it.

Check that planned behavior remains planned and verification remains outstanding. The output must not invent changed files, execution results, or completed work.

Concurrent edit and uncertain publication

Provide an existing PR whose body changes after the initial read. Simulate an update response that times out after the server accepts the write.

Check that the agent re-reads the body before updating, preserves relevant concurrent context, and inspects the actual result before retrying. The report must distinguish verified publication from an unresolved write outcome.