Skip to content

docs(build-agent): annotate-trace is agent-callable on-platform via the annotate_trace op #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mmabrouk wants to merge 1 commit into
base: main
Choose a base branch
from
+15 −6
Open

docs(build-agent): annotate-trace is agent-callable on-platform via the annotate_trace op #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions skills/build-agent/references/annotate-trace.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,15 @@ human at the specific run the note is about.

## The current limitation

There is no agent-callable platform operation for this yet, so this is an experimenter and
demo tool. A self-reflecting **agent** cannot call it autonomously today — you run it yourself,
from your terminal, about a run you just tested. Do not build an agent whose instructions tell
it to annotate its own traces; that path does not exist yet.
This script drives the public HTTP API from outside a run: you supply an `application_id`
and a note yourself, from your terminal, about a run you already tested. That is still why it
lives in `scripts/extras/` — there is no public-API equivalent bound to "your own current
trace," so it stays an experimenter and demo tool for out-of-run annotation.

That said, an agent built and run **on** the Agenta platform (with the default build kit) can
self-annotate at runtime: it has an `annotate_trace` platform op that targets its own current
trace and span automatically. The agent supplies only `references.evaluator.slug` (the
annotation category, e.g. `self_reflection`) and `data.outputs` (the scores, labels, or notes);
a new slug auto-creates a simple evaluator. Write instructions that call that op when you want
an agent to grade itself as part of a run — reach for this script only when you, a human, want
to attach a note from outside a run.
5 changes: 3 additions & 2 deletions skills/build-agent/scripts/extras/annotate-trace.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@
# traces (wraps POST /api/annotations/).
#
# Usage: extras/annotate-trace.sh <application_id> "<note>"
# NOTE: there is no agent-callable platform-op for this yet, so this is an experimenter/demo
# tool. A self-reflecting AGENT cannot call it autonomously today (see references/annotate-trace.md).
# NOTE: this drives the public HTTP API from outside a run, so it stays an experimenter/demo
# tool for human use. On-platform agents (default build kit) can self-annotate at runtime via
# the `annotate_trace` platform op instead (see references/annotate-trace.md).
source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh"
APP_ID="${1:?usage: extras/annotate-trace.sh <application_id> \"<note>\"}"
NOTE="${2:?note required}"
Expand Down