Skip to content

fix: redact GitHub token from git clone error messages in eval engine#87

Open
gn00295120 wants to merge 2 commits intoanthropics:mainfrom
gn00295120:fix/token-leak-in-clone-error
Open

fix: redact GitHub token from git clone error messages in eval engine#87
gn00295120 wants to merge 2 commits intoanthropics:mainfrom
gn00295120:fix/token-leak-in-clone-error

Conversation

@gn00295120
Copy link
Copy Markdown

Summary

When git clone fails in the eval engine, git may echo the clone URL in stderr. Since the GitHub token is embedded in the URL (https://<TOKEN>@github.com/...), the token appears in error messages that are logged and saved to the JSON result file.

Problem

clone_url = f"https://{self.github_token}@github.com/{repo_name}.git"
# ...
except subprocess.CalledProcessError as e:
    error_msg = f"Failed to clone repository: {e.stderr.decode()}"
    # ↑ stderr contains: "fatal: repository 'https://ghp_SECRET@github.com/...' not found"
    self.log(error_msg)           # printed to stdout/stderr
    return False, "", error_msg   # saved to EvalResult → JSON file

In CI/CD environments, this means the token is exposed in:

  • Workflow logs visible to all repository collaborators
  • JSON result files written to disk (and potentially uploaded as artifacts)
  • For public repos, workflow logs are publicly visible

Before fix:

Failed to clone repository: fatal: repository 'https://ghp_SuperSecretToken@github.com/private/repo.git/' not found

After fix:

Failed to clone repository: fatal: repository 'https://[REDACTED]@github.com/private/repo.git/' not found

Changes

  • Redact self.github_token from stderr before including it in error messages
  • No change to the clone mechanism itself (token-in-URL is the standard pattern)

Test plan

  • POC confirms token appears in error message before fix
  • POC confirms token is redacted after fix
  • Normal clone path unchanged (redaction only applies to error handling)

When git clone fails, git may echo the clone URL in stderr. Since
the GitHub token is embedded in the URL, it appears in the error
message which is logged and saved to the JSON result file.

In CI/CD environments this means the token is exposed in workflow
logs visible to all repository collaborators (or publicly for
public repos).

Redact the token from stderr before including it in error messages.
Copilot AI review requested due to automatic review settings March 23, 2026 15:15
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR mitigates accidental leakage of embedded GitHub tokens in eval-engine logs/results by redacting credentials from git clone failure output before it’s logged and persisted.

Changes:

  • Capture git clone stderr defensively (handle missing stderr).
  • Redact self.github_token from the captured stderr prior to constructing/logging the error message.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread claudecode/evals/eval_engine.py Outdated
Comment thread claudecode/evals/eval_engine.py Outdated
Comment thread claudecode/evals/eval_engine.py Outdated
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@JasonLePage
Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants