Skip to content

Capture cpu profile for keynote bench in nightly job #5163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
joshua-spacetime wants to merge 1 commit into
base: master
Choose a base branch
from
+175 −2
Open

Capture cpu profile for keynote bench in nightly job #5163

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 58 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,9 @@ jobs:
timeout-minutes: 60
env:
CARGO_TARGET_DIR: ${{ github.workspace }}/target
CARGO_PROFILE_RELEASE_DEBUG: line-tables-only
KEYNOTE_PROFILE_DIR: ${{ github.workspace }}/keynote-profile
STDB_V8_FLAGS: "--perf-basic-prof --perf-basic-prof-only-functions --interpreted-frames-native-stack"
steps:
- name: Find Git ref
env:
Expand Down Expand Up @@ -303,7 +306,14 @@ jobs:
prefix-key: v1

- name: Build keynote benchmark binaries
run: cargo build --release -p spacetimedb-cli -p spacetimedb-standalone
run: >
cargo
--config 'build.rustflags = ["-C", "force-frame-pointers=yes"]'
build
--release
-p spacetimedb-cli
-p spacetimedb-standalone
--features spacetimedb-standalone/perfmap

# Node 24 is the current Active LTS line.
- name: Set up Node.js
Expand All @@ -319,8 +329,54 @@ jobs:
run: pnpm build
working-directory: crates/bindings-typescript

- name: Install samply
Comment thread
cloutiertyler marked this conversation as resolved.
env:
CARGO_PROFILE_RELEASE_DEBUG: "false"
run: |
if ! command -v samply >/dev/null 2>&1; then
cargo install --locked --version 0.13.1 samply
fi
samply --version

- name: Run keynote-2 benchmark regression check
run: cargo ci keynote-bench
id: keynote_bench
run: |
mkdir -p "$KEYNOTE_PROFILE_DIR"
set +e
samply record \
Copy link
Copy Markdown
Contributor Author

@joshua-spacetime joshua-spacetime Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: Need to set /proc/sys/kernel/perf_event_paranoid to 1 in the runner in order to give samply access to kernel-level perf events.

Copy link
Copy Markdown
Contributor

@cloutiertyler cloutiertyler Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that a requirement for this to work?

Copy link
Copy Markdown
Contributor Author

@joshua-spacetime joshua-spacetime Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To capture a cpu profile, yes.

Copy link
Copy Markdown
Contributor

@cloutiertyler cloutiertyler Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps this TODO should be a code comment. Otherwise we might forget and perhaps the DevOps team won't realize.

Copy link
Copy Markdown
Contributor Author

@joshua-spacetime joshua-spacetime Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the confusion. These TODOs are for me. I'll get them resolved before merging.

--save-only \
--unstable-presymbolicate \
--profile-name keynote-bench \
-o "$KEYNOTE_PROFILE_DIR/profile.json.gz" \
-- cargo ci keynote-bench
status=$?
ls -lh "$KEYNOTE_PROFILE_DIR" || true
exit "$status"

- name: Upload keynote benchmark profile
if: failure() && steps.keynote_bench.outcome == 'failure'
uses: actions/upload-artifact@v4
with:
name: keynote-bench-profile
path: ${{ env.KEYNOTE_PROFILE_DIR }}
if-no-files-found: warn
retention-days: 14

- name: Report keynote benchmark profile instructions
if: failure() && steps.keynote_bench.outcome == 'failure'
run: |
{
echo "## Keynote benchmark profile"
echo
echo "The keynote benchmark failed after recording a Samply/Firefox profile."
echo
echo "1. Open this run's Summary page: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "2. Download the artifact named \`keynote-bench-profile\`."
echo "3. Open <https://profiler.firefox.com/> and load \`profile.json.gz\`."
echo "4. Compare hot frames against a recent nightly \`keynote-bench-nightly-profile\` artifact."
} >> "$GITHUB_STEP_SUMMARY"

echo "::error title=Keynote benchmark profile uploaded::Download the keynote-bench-profile artifact from this run and open profile.json.gz in https://profiler.firefox.com/."

lints:
name: Lints
Expand Down
117 changes: 117 additions & 0 deletions .github/workflows/keynote-bench-nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
name: Keynote Bench Nightly

on:
schedule:
# Nightly at 3 AM UTC.
- cron: '0 3 * * *'
workflow_dispatch:
inputs:
ref:
description: "Git ref to benchmark"
required: false
default: master

permissions:
contents: read

concurrency:
group: keynote-bench-nightly
cancel-in-progress: true

jobs:
keynote_bench:
name: Keynote Bench
runs-on: spacetimedb-benchmark-runner
timeout-minutes: 60
env:
CARGO_TARGET_DIR: ${{ github.workspace }}/target
CARGO_PROFILE_RELEASE_DEBUG: line-tables-only
KEYNOTE_PROFILE_DIR: ${{ github.workspace }}/keynote-profile
STDB_V8_FLAGS: "--perf-basic-prof --perf-basic-prof-only-functions --interpreted-frames-native-stack"
steps:
- name: Checkout sources
uses: actions/checkout@v4
with:
ref: ${{ github.event.inputs.ref || 'master' }}

- uses: dsherret/rust-toolchain-file@v1
- name: Set default rust toolchain
run: rustup default $(rustup show active-toolchain | cut -d' ' -f1)

- name: Cache Rust dependencies
uses: Swatinem/rust-cache@v2
with:
workspaces: ${{ github.workspace }}
shared-key: spacetimedb
save-if: false
prefix-key: v1

- name: Build keynote benchmark binaries
run: >
cargo
--config 'build.rustflags = ["-C", "force-frame-pointers=yes"]'
build
--release
-p spacetimedb-cli
-p spacetimedb-standalone
--features spacetimedb-standalone/perfmap
# Node 24 is the current Active LTS line.
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: 24

- uses: ./.github/actions/setup-pnpm
with:
run_install: true

- name: Build TypeScript SDK
run: pnpm build
working-directory: crates/bindings-typescript

- name: Install samply
env:
CARGO_PROFILE_RELEASE_DEBUG: "false"
run: |
if ! command -v samply >/dev/null 2>&1; then
cargo install --locked --version 0.13.1 samply
fi
samply --version
- name: Run keynote-2 benchmark regression check
id: keynote_bench
run: |
mkdir -p "$KEYNOTE_PROFILE_DIR"
samply record \
--save-only \
--unstable-presymbolicate \
--profile-name keynote-bench-nightly \
-o "$KEYNOTE_PROFILE_DIR/profile.json.gz" \
-- cargo ci keynote-bench
ls -lh "$KEYNOTE_PROFILE_DIR" || true
- name: Upload keynote benchmark profile
if: always()
uses: actions/upload-artifact@v4
with:
name: keynote-bench-nightly-profile
path: ${{ env.KEYNOTE_PROFILE_DIR }}
if-no-files-found: warn
retention-days: 90

- name: Report keynote benchmark profile instructions
if: failure() && steps.keynote_bench.outcome == 'failure'
run: |
{
echo "## Keynote benchmark profile"
echo
echo "The nightly keynote benchmark failed after recording a Samply/Firefox profile."
echo
echo "1. Open this run's Summary page: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "2. Download the artifact named \`keynote-bench-nightly-profile\`."
echo "3. Open <https://profiler.firefox.com/> and load \`profile.json.gz\`."
echo "4. Compare hot frames against earlier nightly \`keynote-bench-nightly-profile\` artifacts."
} >> "$GITHUB_STEP_SUMMARY"
echo "::error title=Keynote benchmark profile uploaded::Download the keynote-bench-nightly-profile artifact from this run and open profile.json.gz in https://profiler.firefox.com/."
Loading