From 76fa16444c850dd58d8a33af606e97e10c483202 Mon Sep 17 00:00:00 2001 From: CYJiang Date: Thu, 13 Aug 2026 11:12:53 +0800 Subject: [PATCH 1/2] docs(skill-upper): add yaml field comments --- skills/skill-upper/SKILL.md | 4 + skills/skill-upper/assets/case.yaml.tmpl | 27 ++++-- skills/skill-upper/assets/eval.yaml.tmpl | 50 +++++++--- ...-context-generates-english-only-cases.yaml | 2 +- .../evals/cases/scaffold-new-evals.yaml | 5 +- .../assert-english-only-generated-cases.sh | 96 ++++++++++++++++++- 6 files changed, 161 insertions(+), 23 deletions(-) diff --git a/skills/skill-upper/SKILL.md b/skills/skill-upper/SKILL.md index d4875600..d4abc262 100644 --- a/skills/skill-upper/SKILL.md +++ b/skills/skill-upper/SKILL.md @@ -30,7 +30,10 @@ When creating or editing `eval.yaml`, `case.yaml`, grading scripts, README snipp - If the target Skill itself is written in Chinese but the user asks in English, translate the Skill's functional intent into English test prompts and assertions instead of copying Chinese prose from the target Skill or templates. - In an English context, deterministic keywords in `rule_based` cases, including `expect.must_contain` and `judge.success.output_contains`, must also be English keywords. Translate terms such as `资源泄漏`, `关闭`, and `异常处理` into `resource leak`, `close`, and `exception handling`; do not write bilingual parentheticals like `"资源" (resources)`. - Keep technical identifiers unchanged, such as `schema_version`, `environment.type`, `engine.name`, `rule_based`, `agent_judge`, `script_path`, file paths, and commands. +- Generated YAML comments must use field-leading comments. Keep each comment short: one line for field meaning, plus one line for options only when useful. +- When listing options in comments, keep enum values unchanged, such as `none | opensandbox | docker` and `rule_based | agent_judge | script`. - Treat `assets/*.tmpl` as structural references only. Rewrite placeholder prose and comments into the current output language; in an English context, translate or remove every Chinese comment and Chinese placeholder before writing generated files. +- `skill-up import` uses the CLI conversion path and does not preserve template comments; do not promise commented YAML for import-generated files. - In an English context, after generating all files but BEFORE submitting the final reply, you **MUST perform a CJK self-check**: open every `evals/cases/*.yaml` and `evals/eval.yaml` and scan for CJK characters (Unicode ranges `\u4e00-\u9fff\u3400-\u4dbf\uf900-\ufaff\u3000-\u303f\uff00-\uffef`), including but not limited to `title`, `description`, `input.prompt`, `expect` keywords, `judge.criteria`, and YAML comments. If any CJK character is found, **replace it with an equivalent English expression before finishing the task**. This step is mandatory and must not be skipped. ## What is skill-up @@ -117,6 +120,7 @@ Precedence (low → high): embedded empty defaults < user config < project `.ski - Copy `assets/case.yaml.tmpl` to `/evals/cases/.yaml`. Adapt language per "Language Rules for Generated Artifacts". In an English context, it is **prohibited** to copy Chinese placeholder text from the templates into generated files — all prose must be rewritten in English. The Chinese in the templates is for structural reference only, not to be carried over. +Preserve short field-leading comments in generated YAML. In Chinese context, rewrite those comments into Chinese while keeping field names and enum values in English. Selection guidelines: diff --git a/skills/skill-upper/assets/case.yaml.tmpl b/skills/skill-upper/assets/case.yaml.tmpl index c8b225eb..077c5c6b 100644 --- a/skills/skill-upper/assets/case.yaml.tmpl +++ b/skills/skill-upper/assets/case.yaml.tmpl @@ -1,28 +1,34 @@ -id: basic # Must match the filename +# Stable case ID used in reports. Keep it the same as the filename without .yaml. +id: basic +# Human-readable summary shown in case lists and reports. title: Brief description of what this case validates +# Optional background explaining the behavior or regression this case covers. description: | - More detailed background (optional). + More detailed background. input: + # User prompt sent to the Agent for a single-turn case. prompt: | Write the actual prompt to send to the Agent here. Use a tone close to real users; avoid being too mechanical. # For multi-turn conversations, use turns instead of prompt: # input: +# # Ordered conversation turns; use this instead of prompt for multi-turn cases. # turns: # - role: user # content: "First request" # post_condition: # must_contain_any: ["keywordA", "keywordB"] -# on_fail: skip_remaining # or fail +# on_fail: skip_remaining # capture: # - variable: var_name # pattern: "(?Ppattern)" # - role: user # content: "Follow-up using {{var_name}}" -# context: # Optional: initialize workspace +# Optional workspace setup copied or created before the Agent runs. +# context: # repo_fixture: fixtures/repos/sample-project # git: # init: true @@ -32,11 +38,13 @@ input: # "src/main.py": | # def hello(): ... +# Per-case limits that override values from eval.yaml cases.defaults. # constraints: # timeout_seconds: 180 # max_turns: 8 -expect: # Zero-cost gate checks; judge is skipped if these fail +# Cheap pre-judge checks; when these fail, the judge block is skipped. +expect: must_contain: - "Required keyword" must_not_contain: @@ -46,7 +54,10 @@ expect: # Zero-cost gate checks; judge is skipped i # - "output.json" judge: - type: rule_based # rule_based | agent_judge | script + # Grading strategy used to decide whether this case passes. + # Options: rule_based | agent_judge | script. + type: rule_based + # Assertions that must pass when judge.type is rule_based. success: - output_contains: all: ["keyword1", "keyword2"] @@ -54,7 +65,7 @@ judge: # - output_contains: # any: ["LGTM", "No changes needed"] -# Example for agent_judge (use when LLM semantic evaluation is needed): +# Example for agent_judge, useful when semantic evaluation is needed: # judge: # type: agent_judge # model: anthropic/claude-sonnet-4-6 @@ -63,7 +74,7 @@ judge: # - "Does not fabricate non-existent fields" # pass_threshold: 0.7 -# Per-turn assertions (use with input.turns): +# Per-turn assertions, used with input.turns: # judge: # type: rule_based # success: diff --git a/skills/skill-upper/assets/eval.yaml.tmpl b/skills/skill-upper/assets/eval.yaml.tmpl index 8b76eafd..5f488a51 100644 --- a/skills/skill-upper/assets/eval.yaml.tmpl +++ b/skills/skill-upper/assets/eval.yaml.tmpl @@ -1,31 +1,45 @@ +# Evaluation config schema version used by skill-up. Do not change this value. schema_version: v1alpha1 environment: - type: none # none | opensandbox - # kwargs: # For opensandbox: base_url, extensions, etc. See references/eval-yaml.md + # Runtime environment used for each case; use none when no sandbox is needed. + # Options: none | opensandbox | docker. + type: none + # Extra runtime settings for opensandbox or other selected environments. + # kwargs: # base_url: https://agent-sandbox.example.com + # Container image used by sandboxed environments when a custom image is needed. # image: registry.example.com/your-org/sandbox-base:latest + # Workspace mount path inside the execution environment. # workspace_mount: /workspace + # Environment variables injected before setup steps and agent execution. # env: # TZ: UTC + # Commands to prepare the workspace before each case starts. # setup_steps: # - run: git config --global user.name "eval" skills: + # Skills installed into the Agent Engine before running cases. + # source options: local_path. - source: local_path - path: . # Directory of the Skill under test + # Path to the Skill under test, relative to this Skill root unless absolute. + path: . # - source: local_path - # path: ../dependency_skill # Path to other SKILL.md directory + # path: ../dependency_skill engine: - name: claude_code # claude_code | codex | qodercli (also compatible with qoder-cli) - # model is optional; omit to use engine's default model + # Agent Engine used to execute prompts and produce outputs for grading. + # Options: claude_code | codex | qodercli. + name: claude_code + # Optional model override; omit this block to use the engine default model. # model: # provider: anthropic # name: claude-sonnet-4-6 # base_url: "" - # kwargs: # agent-specific switches; see references/eval-yaml.md - # bypass_sandbox: "true" # codex: skip its own process sandbox (host kernel lacks Landlock) + # Agent-specific switches passed through to the selected engine. + # kwargs: + # bypass_sandbox: "true" # mcp: # servers: @@ -35,27 +49,41 @@ engine: # config_ref: evals/fixtures/mcp/github.json cases: + # Case YAML files included in this eval suite, relative to the Skill root. files: - evals/cases/basic.yaml + # Default constraints inherited by every case unless the case overrides them. defaults: timeout_seconds: 120 max_turns: 5 - # expect: # Optional checks inherited by every case + # Default checks inherited by every case unless the case overrides them. + # expect: # exit_code: 0 # must_not_contain: # - "TODO" - # parallelism: 4 # Stateless cases can run in parallel; or use skill-up run --parallelism N to override + # Maximum number of cases to run at the same time when cases are stateless. + # parallelism: 4 # benchmark: # enabled: true +# Cheap pre-judge checks; when these fail, the configured judge is skipped. +# expect: +# must_contain: +# - "Required keyword" + +# Default judge used when a case does not define its own judge block. # judge: +# # Grading strategy for case results. +# # Options: rule_based | agent_judge | script. # type: agent_judge # model: anthropic/claude-sonnet-4-6 # context: -# profile: standard # standard = file refs for transcript/diff; minimal = omit them +# profile: standard # criteria: # - "The response satisfies the case-specific success criteria." report: + # Report formats written after each run. + # Options: json | junit | html | grading | benchmark. formats: [json] diff --git a/skills/skill-upper/evals/cases/english-context-generates-english-only-cases.yaml b/skills/skill-upper/evals/cases/english-context-generates-english-only-cases.yaml index 61d99b30..5ee783d5 100644 --- a/skills/skill-upper/evals/cases/english-context-generates-english-only-cases.yaml +++ b/skills/skill-upper/evals/cases/english-context-generates-english-only-cases.yaml @@ -3,7 +3,7 @@ title: Respond in English and generate English-only case files in English contex description: | Verifies that skill-upper follows an English user prompt by replying in English and by generating eval case YAML files that do not contain Chinese - or CJK characters. + or CJK characters, while preserving short field comments near key YAML fields. context: repo_fixture: evals/fixtures/skills/sample-no-evals diff --git a/skills/skill-upper/evals/cases/scaffold-new-evals.yaml b/skills/skill-upper/evals/cases/scaffold-new-evals.yaml index af76e41a..08cf1a0c 100644 --- a/skills/skill-upper/evals/cases/scaffold-new-evals.yaml +++ b/skills/skill-upper/evals/cases/scaffold-new-evals.yaml @@ -26,7 +26,10 @@ judge: model: anthropic/claude-sonnet-4-6 criteria: - "生成了 evals/eval.yaml 文件，且包含 schema_version、environment、engine、cases 等必要字段" + - "evals/eval.yaml 中 schema_version、environment.type、engine.name、cases.files、judge 等关键字段附近有简短中文字段注释" - "在 evals/cases/ 下生成了至少一个 case yaml 文件，且包含 id、title、input.prompt、judge 字段" + - "case yaml 中 id、input.prompt、expect、judge.type 等关键字段附近有简短中文字段注释" + - "YAML 字段名和枚举值保持英文，例如 schema_version、environment.type、rule_based、agent_judge、script" - "case 的 prompt 与 code-review-helper Skill 的功能相关（代码审查），而非无意义的占位符" - "eval.yaml 中 cases.files 列表正确引用了生成的 case 文件路径" - pass_threshold: 0.75 + pass_threshold: 1.0 diff --git a/skills/skill-upper/evals/fixtures/scripts/assert-english-only-generated-cases.sh b/skills/skill-upper/evals/fixtures/scripts/assert-english-only-generated-cases.sh index 47b02c0c..f27965f9 100755 --- a/skills/skill-upper/evals/fixtures/scripts/assert-english-only-generated-cases.sh +++ b/skills/skill-upper/evals/fixtures/scripts/assert-english-only-generated-cases.sh @@ -25,18 +25,110 @@ case_files = sorted(case_dir.glob("*.yaml")) if case_dir.exists() else [] if not case_files: failures.append("No generated eval case YAML files were found under evals/cases/.") -for path in case_files: +generated_yaml_files = [] +eval_yaml = pathlib.Path("evals/eval.yaml") +if not eval_yaml.exists(): + failures.append("Generated evals/eval.yaml was not found.") +else: + generated_yaml_files.append(eval_yaml) +generated_yaml_files.extend(case_files) + +for path in generated_yaml_files: text = path.read_text(encoding="utf-8") match = cjk_pattern.search(text) if match: line_no = text[: match.start()].count("\n") + 1 failures.append(f"{path}:{line_no} contains Chinese/CJK characters.") +def file_lines(path): + return path.read_text(encoding="utf-8").splitlines() + +def is_explanatory_comment(line): + stripped = line.lstrip() + if not stripped.startswith("#"): + return False + content = stripped[1:].strip() + if not content: + return False + if content.startswith("-"): + return False + return not re.match(r"^[A-Za-z0-9_.-]+:\s*", content) + +def has_adjacent_explanatory_comment(lines, index, stop_index=0): + cursor = index - 1 + found = False + while cursor >= stop_index: + line = lines[cursor] + if not line.strip(): + return found + if not line.lstrip().startswith("#"): + return found + found = is_explanatory_comment(line) or found + cursor -= 1 + return found + +def has_comment_before(lines, field_line): + for index, line in enumerate(lines): + if line.lstrip().startswith("#"): + continue + if line.lstrip().startswith(field_line): + return has_adjacent_explanatory_comment(lines, index) + return False + +def has_real_field(lines, field_line): + return any( + not line.lstrip().startswith("#") and line.lstrip().startswith(field_line) + for line in lines + ) + +def has_nested_comment_before(lines, parent, child): + for index, line in enumerate(lines): + if line.lstrip().startswith("#"): + continue + if line.startswith(parent): + for child_index in range(index + 1, min(len(lines), index + 8)): + if lines[child_index].lstrip().startswith("#"): + continue + if lines[child_index].startswith(child): + return has_adjacent_explanatory_comment(lines, child_index, index + 1) + return False + +if eval_yaml.exists(): + lines = file_lines(eval_yaml) + eval_checks = [ + ("schema_version", has_comment_before(lines, "schema_version:")), + ("environment.type", has_nested_comment_before(lines, "environment:", " type:")), + ("engine.name", has_nested_comment_before(lines, "engine:", " name:")), + ("cases.files", has_nested_comment_before(lines, "cases:", " files:")), + ("report.formats", has_nested_comment_before(lines, "report:", " formats:")), + ] + if has_real_field(lines, "expect:"): + eval_checks.append(("expect", has_comment_before(lines, "expect:"))) + if has_real_field(lines, "judge:"): + eval_checks.append(("judge", has_comment_before(lines, "judge:"))) + for field, passed in eval_checks: + if not passed: + failures.append(f"evals/eval.yaml is missing a field-leading comment for {field}.") + +for path in case_files: + lines = file_lines(path) + case_checks = [ + ("id", has_comment_before(lines, "id:")), + ("title", has_comment_before(lines, "title:")), + ("input.prompt", has_nested_comment_before(lines, "input:", " prompt:")), + ("judge.type", has_nested_comment_before(lines, "judge:", " type:")), + ] + if has_real_field(lines, "expect:"): + case_checks.append(("expect", has_comment_before(lines, "expect:"))) + for field, passed in case_checks: + if not passed: + failures.append(f"{path} is missing a field-leading comment for {field}.") + if failures: print("English-only language check failed:") for failure in failures: print(f"- {failure}") sys.exit(1) -print("PASS: final response is English-like, and generated eval case YAML files contain no Chinese/CJK characters.") +print("PASS: final response is English-like, generated YAML is English-only, and key fields have comments.") PY From eaaf8066ea4077ada3c956e4944d784633de99cf Mon Sep 17 00:00:00 2001 From: CYJiang <86391540+googs1025@users.noreply.github.com> Date: Mon, 17 Aug 2026 12:29:20 +0800 Subject: [PATCH 2/2] docs(skill-upper): align eval template option comments --- skills/skill-upper/assets/eval.yaml.tmpl | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/skills/skill-upper/assets/eval.yaml.tmpl b/skills/skill-upper/assets/eval.yaml.tmpl index 5f488a51..52ec5226 100644 --- a/skills/skill-upper/assets/eval.yaml.tmpl +++ b/skills/skill-upper/assets/eval.yaml.tmpl @@ -30,7 +30,7 @@ skills: engine: # Agent Engine used to execute prompts and produce outputs for grading. - # Options: claude_code | codex | qodercli. + # Built-in options: claude_code | codex | qodercli | qwen_code. name: claude_code # Optional model override; omit this block to use the engine default model. # model: @@ -85,5 +85,6 @@ cases: report: # Report formats written after each run. - # Options: json | junit | html | grading | benchmark. + # Options: json | junit | html. + # grading.json and benchmark artifacts are generated automatically. formats: [json]