Improve evaluate-pr-tests workflow: pull_request_target, access gating, security docs

[PureWeen]

Description

Overhauls the copilot-evaluate-tests gh-aw workflow for better security, fork support, and Copilot bot compatibility.

Changes

1. Switch from pull_request to pull_request_target — runs workflow YAML from base branch (trusted), eliminates "Approve & Run" friction for collaborators
2. Add copilot[bot] to bots: allowlist — Copilot-authored PRs auto-trigger evaluation without needing write collaborator status
3. Add dry-run mode (suppress_comment input) — evaluate without posting comments, useful for testing
4. Add noop tool guidance — agent calls noop when no action is needed instead of silently exiting
5. Harden Checkout-GhAwPr.ps1 — null guard on $PrInfo, allow fork PRs from write-access authors
6. Update security documentation — accurate credential model, defense layers table, workflow author rules

Trigger Behavior

Trigger	When it fires	Who can trigger
pull_request_target	PR opened/updated/reopened touching src/**/tests/**	Auto for write-access authors + copilot[bot]; blocked for external contributors via pre_activation role check
issue_comment	/evaluate-tests comment on a PR	Write-access collaborators (admin/maintain/write)
workflow_dispatch	Manual trigger from Actions tab	Write-access collaborators; fork PRs allowed if author has write access

Access Matrix

Permission Level	pull_request_target	/evaluate-tests comment	workflow_dispatch
admin / maintain / write	✅ Auto-runs	✅ Can trigger	✅ Can trigger
copilot[bot]	✅ Auto-runs (via bots: allowlist)	N/A	N/A
triage / read	❌ pre_activation blocks	❌ pre_activation blocks	❌ Script skips (exit 0)
external / fork (no write)	❌ pre_activation blocks	❌ pre_activation blocks	❌ Script skips (exit 0)
fork (with write access)	✅ Auto-runs	✅ Can trigger	✅ Script proceeds

Security Model

Based on GitHub Security Lab guidance:

• PR contents treated as passive data (read/analyze, never built or executed) ✅
• Agent job has read-only permissions (contents: read, issues: read, pull-requests: read) ✅
• Write operations in separate safe_outputs job (not the agent) ✅
• permissions: {} at workflow level — no ambient write access ✅
• Agent blast radius limited to max: 1 comment via safe-outputs ✅
• Checkout-GhAwPr.ps1 restores .github/skills/ and .github/instructions/ from base branch for workflow_dispatch ✅

Security Docs Update

Updated .github/instructions/gh-aw-workflows.instructions.md with:

• Accurate credential model (COPILOT_TOKEN present in agent env via --env-all, defended by firewall + redaction + threat detection)
• Defense layers table documenting what each layer does and does not protect against
• Explicit DO/DON'T rules for workflow authors
• Key principles from GitHub Security Lab's pwn-request guidance

Known Limitations

• ready_for_review event type not supported by gh-aw compiler for pull_request_target — draft→ready transitions without a new push won't trigger the workflow
• checkout_pr_branch.cjs overwrites .github/skills/ for pull_request_target/issue_comment triggers — accepted residual risk (agent sandboxed, output limited)

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://github.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34678

Or

• Run remotely in PowerShell:

iex "& { $(irm https://github.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34678"

[Copilot]

Pull request overview

Updates the gh-aw Evaluate PR Tests workflow triggers so fork PRs can be evaluated (by switching from pull_request to pull_request_target) while keeping the workflow’s gating/conditions aligned with the new event.

Changes:

• Switched workflow trigger from pull_request to pull_request_target and updated related if: conditions.
• Updated the gate step condition to run under pull_request_target.
• Updated the compiled .lock.yml to reflect the new trigger configuration.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
.github/workflows/copilot-evaluate-tests.md	Moves evaluation to pull_request_target and updates workflow conditions/gate accordingly.
.github/workflows/copilot-evaluate-tests.lock.yml	Regenerated compiled workflow reflecting the trigger/condition changes.

[Copilot]

The compiled lock file shows roles: all commented out under the on: block, which indicates the roles setting was not applied. This also coincides with the removal of the pre_activation/membership check job, so the workflow no longer gates who can run it. Fix by moving roles: to the correct top-level location in the .md frontmatter and re-compiling so the lock file includes the intended role check behavior.

[Copilot]

jobs.activation no longer depends on a role/membership gate (needs: pre_activation and the needs.pre_activation.outputs.activated check are gone). With pull_request_target, this means the workflow can run with secrets for any matching PR/comment, which is a significant security/cost exposure. After fixing the roles placement in the .md, ensure the compiled lock restores the gating (or add an explicit guard) before allowing activation to proceed.

Suggested change

	(github.event_name == 'pull_request_target' && github.event.pull_request.draft == false) || github.event_name == 'workflow_dispatch' || (github.event_name == 'issue_comment' &&
	(github.event_name == 'pull_request_target' && github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false) || github.event_name == 'workflow_dispatch' || (github.event_name == 'issue_comment' &&

[jeffhandley]

You might want to configure to not generate a no-op run report issue (within the frontmatter).

https://github.github.com/gh-aw/patterns/monitoring/#no-op-run-reports

[jeffhandley]

💯

[jeffhandley]

I do think this will still lead to the 'Approve and run workflows' button showing up for PRs from untrusted forks. We need to solidify the guidance we give for when to hit that button. I really wish that button navigated into a list of workflows needing approval for the PR with boxes to select which to approve.

[jeffhandley]

Future-proofing: I suggest renaming to suppress_output in case the output changes (to a PR review for example).

Suggested change

	suppress_comment:
	description: 'Dry-run — evaluate but do not post a comment on the PR'
	suppress_output:
	description: 'Dry-run - evaluate but do not post output on the PR'

[jeffhandley]

I am not certain what identity ends up getting used here; I experimented and I think it's one of these but not sure which. 😄

• copilot
• copilot[bot]
• app/copilot-swe-agent
• copilot-swe-agent
• copilot-swe-agent[bot]

[jeffhandley]

I always guard against forks as well, preventing the workflow from running on forks except for the workflow_dispatch event. Otherwise, PRs within a fork will result in failing workflow runs (vs. starting the workflow and skipping all jobs).

Simple case that needs adapting to your scenario: if: (!github.event.repository.fork) || github.event_name == 'workflow_dispatch'.

[jeffhandley]

These expressions get replaced on their way to the model so this would end up embedding the true or false value into the opening statement. I think you need something more like this (but I recommend validating my understanding here). Note I also reflected my suggested input rename from above.

Suggested change

	When triggered via `workflow_dispatch` with `suppress_comment` = `${{ inputs.suppress_comment }}`:
	- If **true**, perform the full evaluation but **do not** post a comment on the PR. Write the evaluation to the workflow log only. This is useful for testing the skill without spamming the PR.
	- If **false** (default), post the comment as normal.
	When triggered via `workflow_dispatch`, the `suppress_output` input controls behavior.
	- If `${{ inputs.suppress_output }}` == **true**, perform the full evaluation but **do not** post a comment on the PR. Write the evaluation to the workflow log only. This is useful for testing the skill without spamming the PR.
	- If `${{ inputs.suppress_output }}` == **false** (default), post the comment as normal.

[jeffhandley]

Suggested change

	If dry-run mode is active (`suppress_comment` is true), log the evaluation report to stdout and stop — do **not** call `add_comment`.
	If dry-run mode is active (`suppress_output` is true), log the evaluation report to stdout and stop — do **not** call `add_comment`.

[PureWeen]

Addressing Jeff's Review Feedback

Thanks for the thorough review @jeffhandley! Here's what we applied:

✅ Applied

1. Renamed suppress_comment → suppress_output — Future-proofs if output changes to a PR review or other format. (commit f3dc6a9)

2. Disabled no-op run report issues — Added report-as-issue: false under noop: in safe-outputs. Note: the docs suggest this goes under safe-outputs: noop:, not as a top-level property. (commit f3dc6a9)

3. Improved dry-run mode wording — Clarified how ${{ inputs.suppress_output }} expression replacement works in the agent prompt. (commit f3dc6a9)

4. Fixed bot identity — Changed bots: from copilot[bot] to copilot-swe-agent[bot]. Git log shows 882 commits from copilot-swe-agent[bot] and 0 from copilot[bot] in this repo. (commit 7bcc980)

🐛 Additional fix discovered

6. Fixed gate step for large PRs (300+ files) — gh pr diff fails with HTTP 406 for PRs with 300+ changed files (e.g., PR March 30th, Inflight Candidate #34617 inflight candidate). Now falls back to the paginated pulls/files REST API. (commit 04bf387)

📝 Acknowledged (no change needed)

7. "Approve and run" concern — You're right this shows up for fork contributors with pull_request_target. However, dotnet/maui already has two pull_request_target workflows (dogfood-comment.yml and bump-global-json.yml) — both self-gate to avoid fork triggers. Our workflow is different in that we want it to run on fork PRs, which is why the approval button appears. Fork contributors can alternatively use /evaluate-tests comment or maintainers can use workflow_dispatch to bypass this. We understand your broader direction toward issue_comment-only for org-wide patterns and will track that.

All changes compiled clean with gh aw compile. Running validation now via workflow_dispatch against PR #34882 (small, with tests) and PR #34617 (300+ files, gate fallback test).

[jeffhandley]

Today I learned about the slash_command trigger at Command Triggers | GitHub Agentic Workflows. I suggest trying that out here. This could then change the if condition to use needs.activation.outputs.slash_command instead of the comment body.

[PureWeen]

Great catch -- implemented in e7fdf22. Replaced issue_comment + startsWith with slash_command: evaluate-tests (scoped to events: [pull_request_comment, issue_comment]). Simplified the if: condition since the platform handles command matching now. Also added an anti-patterns table to the gh-aw instructions file so we don't miss built-in features like this in the future.

[PureWeen]

🔍 Multi-Model Code Review — PR #34678

Reviewed by: 3 independent reviewers
Files: 4 changed (workflow source, compiled lock, PowerShell script, instruction docs)
CI Status: ⚠️ maui-pr skipping (expected — no runtime code changes), license/cla ✅, dogfood-comment ✅

Prior Reviews: jeffhandley approved. Jeff's feedback (rename suppress_comment → suppress_output, fix bot identity, disable noop report issues) was addressed per PureWeen's comment.

---

Findings

🟡 MODERATE — exit 0 on permission denial causes evaluation of wrong code

File: .github/scripts/Checkout-GhAwPr.ps1 (~line 67)
Flagged by: All 3 reviewers (1 initial + 2 confirmed in adversarial round)

if ($Permission -notin $AllowedRoles) {
    Write-Host "⏭️ PR author '...' has '$Permission' access..."
    exit 0   # ← step succeeds, workflow continues
}

When workflow_dispatch targets a PR whose author lacks write access, the script exits 0 (success) without checking out the PR. The workflow continues: the agent runs against the base branch code, but inputs.pr_number is still in context, so it posts an incorrect evaluation comment on the actual PR.

Why it matters: A maintainer triggering workflow_dispatch for a community PR gets a plausible-looking but wrong evaluation posted publicly. No security risk, but a data-integrity bug.

Fix: Change exit 0 to exit 1 so the workflow fails visibly when checkout is skipped.

---

🟡 MODERATE — github.event.repository.fork guard is a no-op

File: .github/workflows/copilot-evaluate-tests.md (~line 28) and compiled into pre_activation/activation conditions in .lock.yml
Flagged by: 2/3 reviewers

if: >-
  ((!github.event.repository.fork) || github.event_name == 'workflow_dispatch') && ...

github.event.repository refers to the base repo (dotnet/maui), not the PR source. For dotnet/maui, github.event.repository.fork is always false, making !github.event.repository.fork always true — this clause is a no-op.

Why it matters: The guard looks like it's checking whether the PR comes from a fork, but it isn't. Future contributors may either remove a guard they think is redundant or add duplicates. The real fork safety comes from the sandboxed execution model and pre_activation role checks.

Suggestion: Either remove the misleading guard or add a comment clarifying it only matters if someone installs the workflow in a fork of dotnet/maui.

---

🟢 MINOR — Compiler-generated if: conditions reference dead pull_request event

File: .github/workflows/copilot-evaluate-tests.lock.yml (reaction step ~line 122, status-comment step ~line 170)
Flagged by: All 3 reviewers

if: ... || (github.event_name == 'pull_request') && (...)

The trigger changed from pull_request to pull_request_target, but github.event_name for the new trigger is 'pull_request_target'. These conditions will never match for PR events, so the 👀 reaction and "🔬 Evaluating tests…" status comment will not fire for pull_request_target triggers (only for issue_comment/slash-command).

Why it matters: PR authors get no immediate feedback that evaluation has started. This appears to be a gh-aw compiler limitation — it doesn't yet map pull_request_target to these auto-generated conditions. Consider filing upstream.

---

🟢 MINOR — isFork fetched but never used as a gate; docs contradict code

File: .github/scripts/Checkout-GhAwPr.ps1 (~line 46, ~line 72)
Flagged by: 2/3 reviewers

The script fetches isCrossRepository into $PrInfo.isFork and logs it, but never uses it to gate checkout. The .DESCRIPTION docstring says "Fork PRs are evaluated via pull_request_target instead" — but the code will check out fork PRs from write-access authors via workflow_dispatch.

Why it matters: Dead code + misleading docs = maintenance confusion. Either enforce the fork check or update the docs to accurately describe behavior ("Fork PRs from write-access authors are checked out here; external contributors are handled via pull_request_target").

---

🟢 MINOR — Community PRs have limited evaluation paths

File: .github/scripts/Checkout-GhAwPr.ps1 (permission gate)
Flagged by: 2/3 reviewers (1 disagreed, noting /evaluate-tests slash command still works)

The workflow_dispatch path now blocks PRs from authors without write access. Combined with pre_activation role checks on pull_request_target, the auto-trigger path may not work for community PRs.

Mitigated: A maintainer can still trigger evaluation via the /evaluate-tests slash command. But this is a behavior change vs. the previous unconditional workflow_dispatch path — worth documenting.

---

Discarded Findings (1/3 agreement, adversarial rejected)

Finding	Initial	Adversarial	Verdict
hide-older-comments: true suppresses evaluations	1/3	0/2 agreed	Discarded — intentional noise reduction
pre_activation triggers on all issue comments	1/3	N/A (expected with slash_command:)	Discarded

Known Limitations (documented in PR)

• ready_for_review event not supported by gh-aw compiler for pull_request_target — draft→ready transitions without a new push won't trigger. Documented in PR description ✅

Non-Issues Verified ✅

• No pwn-request vulnerability: pull_request_target migration is architecturally sound. User steps never execute fork code; agent runs in sandboxed container with scrubbed credentials.
• $PrInfo null guard: Complete and correct — catches empty output and null author.
• PR_NUMBER injection: Input typed as number — GitHub enforces numeric, no injection risk.
• copilot-swe-agent[bot] allowlist: Correctly wired via GH_AW_ALLOWED_BOTS env var.
• COPILOT_TOKEN exposure: Accurately documented with defense layers and limitations.
• noop report-as-issue disabled: Correctly set to false in both source and compiled output.

Test Coverage

This PR modifies workflow infrastructure (YAML, PowerShell, Markdown). There are no automated tests for gh-aw workflows — validation is done via manual workflow_dispatch runs. PureWeen's comment indicates validation was run against PR #34882 (small) and PR #34617 (300+ files, gate fallback test). This is appropriate for this type of change.

---

🔄 Re-Review (after commit d142b43)

Commit: d142b430 — "Fix gate step to run for all triggers, bump timeout to 20min"

Changes in latest commit (verified from diff):

• ✅ Gate step if: condition removed — gate now runs for all triggers (pull_request_target, issue_comment, workflow_dispatch)
• ✅ Gate PR_NUMBER broadened to ${{ github.event.pull_request.number || github.event.issue.number || inputs.pr_number }}
• ✅ Timeout bumped from 15 → 20 minutes
• ✅ Gather-TestContext.ps1 instruction updated with -PrNumber <number>

Previous Finding Status

#	Finding	Status	Notes
1	exit 0 on permission denial	❌ STILL PRESENT	One-line fix needed: exit 0 → exit 1
2	repository.fork guard no-op	❌ STILL PRESENT	Cosmetic/dead code
3	Dead pull_request in compiler conditions	❌ STILL PRESENT	gh-aw compiler limitation
4	isFork not enforced / docs mismatch	❌ STILL PRESENT	Low risk
5	Community PR evaluation paths	✅ MITIGATED	/evaluate-tests slash command + pull_request_target auto-trigger

All 3 reviewers confirmed Finding 1 remains the only actionable blocker.

New Observations from d142b43

The gate step improvement is well-designed:

• PR_NUMBER resolution chain (pull_request.number || issue.number || inputs.pr_number) correctly handles all three trigger types. inputs.pr_number is required: true for workflow_dispatch, so null risk is minimal.
• Gate for /evaluate-tests on non-test PRs: If someone comments /evaluate-tests on a PR with no test files, the gate will exit 1 with a clear message. This is correct behavior — it prevents wasting 20 minutes of agent compute on a PR that has nothing to evaluate.
• Timeout bump to 20 min: Reasonable for complex PRs with many test files.

No new issues introduced by this commit. ✅

Recommendation

⚠️ Request changes — Finding 1 (exit 0 → exit 1 in Checkout-GhAwPr.ps1 line 67) is the only remaining blocker. All other findings are informational (compiler limitations, dead code, documentation improvements) that can be addressed in follow-up.