backend-usage.md

AgentPay Backend Usage

What This Guide Covers

This guide is for developers and operators who need to build, run, test, and inspect the backend nodes in this repo.

It covers three common workflows:

1. Run a focused end-to-end test to validate code changes.
2. Bring up a local multi-OSP environment manually.
3. Start a backend node directly with your own profile and storage settings.

Prerequisites

You need the following on a development machine:

• Go
• geth for local Ethereum-based tests
• CockroachDB only if you want the shared-SQL mode used in the manual examples

Toolchain note:

• The module target in go.mod is Go 1.24.
• On some macOS amd64 hosts, the local go1.25.5 toolchain fails cgo linking for this repo and even trivial import "C" programs with duplicate runtime/cgo symbols.
• If you hit that linker signature, set a known-good toolchain for the current shell before building or testing:

export GOTOOLCHAIN=go1.24.9

For the manual scripts, set:

export AGENTPAY=$PWD
export AGENTPAY_MANUAL_ROOT=${AGENTPAY_MANUAL_ROOT:-/tmp/celer_manual_test}

Useful assets already in the repo:

• Example profile schema: test/manual/sample_profile.json
• Test keystores: testing/env/keystore
• Runtime config examples: testing/profile and test/manual/rt_config.json
• Operational troubleshooting guide: docs/backend-troubleshooting.md

Build the Binaries

If your host is affected by the macOS amd64 go1.25.5 cgo linker issue noted above, export GOTOOLCHAIN=go1.24.9 first and then run the commands below unchanged.

SQLite-backed local builds also require cgo. If you build with CGO_ENABLED=0, the binary may still compile, but local -storedir startup now fails immediately with a clear sqlite3 requires cgo error instead of dying later during store setup. Keep cgo enabled unless you are intentionally using -storesql only.

From the repo root:

mkdir -p ./bin
go build -o ./bin/server ./server
go build -o ./bin/osp-cli ./tools/osp-cli

Optional entry points you may also care about:

• go build ./webapi/cmd
• go build ./webproxy/cmd

Fastest Validation: Focused E2E Test

If you are changing the backend and want the shortest realistic validation loop, start with the end-to-end tests in test/e2e.

On an affected macOS amd64 host, export GOTOOLCHAIN=go1.24.9 in the same shell before running the go test commands in this section.

From the repo root:

go test ./test/e2e -run '^TestE2E$/^e2e-grp2$/^sendCondPayWithErc20$'

If you are already inside test/e2e, the shorter form also works:

go test -run '^TestE2E$/^e2e-grp2$/^sendCondPayWithErc20$'

What this test setup does for you automatically:

• Starts a local geth-based chain
• Builds the required binaries into a temp output directory
• Deploys contracts and funds test accounts
• Generates runtime profiles under /tmp/celer_e2e_*
• Registers the OSP router used by the tests
• Starts the default OSP/backend process

Important files behind that workflow:

• test/e2e/e2e_setup_test.go
• test/e2e/e2e_test.go
• test/e2e/constants.go

The default single-network e2e flow does not provision the extra networks required by the cross-net suite. Run cross-net explicitly with:

go test ./test/e2e -run '^TestE2ECrossNet$' -args -multinet

Useful debugging behavior:

• Successful runs delete the temp directory.
• Failed runs keep it and print a -reuse path so you can rerun without rebuilding or redeploying.

Example:

go test ./test/e2e -reuse /tmp/celer_e2e_1712960000/ -run '^TestE2E$/^e2e-grp2$/^sendCondPayWithErc20$'

If you are specifically changing the pay-centric OSP WebAPI listener, use the focused listener test:

go test ./test/e2e -run '^TestOSPWebApi$/^ospWebApiPaySubset$' -count=1

Broader Test Matrix

For a wider validation sweep, the old CI flow maps reasonably well to the following current-package commands.

As above, on an affected macOS amd64 host, export GOTOOLCHAIN=go1.24.9 before running these commands.

Prerequisites beyond Go:

• geth for e2e suites
• sqlite3 CLI for storage-related test helpers and inspection flows

Legacy CI-style unit/package sweep:

go test ./storage ./celersdk ./common/cobj ./dispatchers ./lrucache ./rpc ./rtconfig ./metrics ./route ./utils/bar

Recommended validation tiers:

• Payment-path regression check:

go test ./test/e2e -run '^TestE2E$/^e2e-grp2$/^sendCondPayWithErc20$'

• Cross-net routing check:

go test ./test/e2e -run '^TestE2ECrossNet$' -args -multinet

• Manual multi-OSP smoke flow: use test/manual/README.md or run AGENTPAY=$PWD ./test/manual/smoke.sh

The full go test ./test/e2e package includes broader multi-OSP and specialized integration suites in addition to the core single-network flow. Use the targeted commands above when you want predictable validation for a specific area.

Manual Multi-OSP Workflow

The best operator-oriented walkthrough already in the repo is test/manual/README.md. The steps below summarize it and point to the files that matter.

1. Prepare the environment

export AGENTPAY=$PWD
go build -o ./osp-cli ./tools/osp-cli
cd test/manual

2. Start the local chain and generate profiles

./setup.sh

This does more than just start geth. It also:

• deploys the ledger, resolver, registry, wallet, and ERC20 contracts
• funds test accounts
• writes OSP profiles under $AGENTPAY_MANUAL_ROOT/profile/

See test/manual/setup.go and test/manual/sample_profile.json.

3. Fund and register OSPs

From the repo root or from test/manual with the built CLI available:

./osp-cli -profile $AGENTPAY_MANUAL_ROOT/profile/o1_profile.json \
  -ks $AGENTPAY/testing/env/keystore/osp1.json \
  -wrapdeposit -amount 10000 -register -nopassword

./osp-cli -profile $AGENTPAY_MANUAL_ROOT/profile/o2_profile.json \
  -ks $AGENTPAY/testing/env/keystore/osp2.json \
  -wrapdeposit -amount 10000 -register -nopassword

This is required if you want route-controller behavior that depends on on-chain router registration.

4. Start OSP nodes

SQLite-backed example:

./run_osp.sh 1
./run_osp.sh 2

For localhost manual runs, test/manual/run_osp.sh defaults AGENTPAY_INSECURE_TLS=1 so inter-OSP dials work with the built-in self-signed localhost certificate.

CockroachDB-backed example:

./cockroachdb.sh start
./cockroachdb.sh 1
./cockroachdb.sh 2
./run_osp.sh 1_crdb
./run_osp.sh 2_crdb

See test/manual/run_osp.sh for the exact flags passed to the server.

5. Connect OSPs and exercise the payment path

Register an inter-OSP stream:

./osp-cli -adminhostport localhost:8190 \
  -registerstream \
  -peer 00290a43e5b2b151d530845b2d5a818240bc7c70 \
  -peerhostport localhost:10002

Open an OSP-to-OSP channel:

./osp-cli -adminhostport localhost:8190 \
  -openchannel \
  -peer 00290a43e5b2b151d530845b2d5a818240bc7c70 \
  -selfdeposit 10 \
  -peerdeposit 10

Send an off-chain payment:

./osp-cli -adminhostport localhost:8190 \
  -sendtoken \
  -receiver 00290a43e5b2b151d530845b2d5a818240bc7c70 \
  -amount 0.01

Inspect state:

• off-chain DB queries with tools/osp-cli/README.md
• on-chain queries with the CLI's -onchainview options

Running a Backend Node Directly

You do not need the helper scripts if you already have a profile and keys.

On an affected macOS amd64 host, export GOTOOLCHAIN=go1.24.9 before running the server binary or go run commands in this section.

Example command from the repo root:

go run ./server/server.go \
  -profile $AGENTPAY_MANUAL_ROOT/profile/o1_profile.json \
  -ks ./testing/env/keystore/osp1.json \
  -port 10001 \
  -adminrpc localhost:11001 \
  -adminweb localhost:8190 \
  -svrname o1 \
  -storedir $AGENTPAY_MANUAL_ROOT/store \
  -rtc ./test/manual/rt_config.json \
  -nopassword

Optional phase-1 seller-OSP WebAPI listener for a same-host caller:

-webapigrpc 127.0.0.1:12000

If this process will dial localhost peers using the built-in localhost certificate, prefix the command with AGENTPAY_INSECURE_TLS=1 unless you are using test/manual/run_osp.sh, which already does that for local manual runs.

For a CockroachDB-backed node, replace -storedir with -storesql:

-storesql 'postgresql://celer_test_o1@localhost:26257/celer_test_o1?sslmode=disable'

Configuration Files

Profile JSON

The profile schema is defined in common/profile.go. The main sections are:

• Ethereum: RPC gateway, chain id, block timing, and contract addresses
• Osp: this node's gRPC host and ETH address
• Sgn: SGN-related endpoints and contract address

Example: test/manual/sample_profile.json

Runtime config JSON

The runtime config file passed by -rtc is separate from the profile. It controls operational values such as:

• min/max payment timeouts
• refill thresholds and refill amounts
• deposit polling and batching
• OSP-to-OSP open-channel limits

Examples:

• testing/profile/rt_config.json
• testing/profile/rt_config_multiosp.json
• test/manual/rt_config.json

Deadline and timeout units

The on-chain contracts use block.timestamp (unix seconds) for every challenge window, dispute timeout, and deadline (disputeTimeout, settleFinalizedTime, withdrawDeadline, openDeadline, resolveDeadline, resolveTimeout, migrationDeadline, plus the RouterRegistry register/refresh value). Off-chain code follows the same unit, so all of the following are seconds:

• profile DisputeTimeout (challenge window applied to each opened channel)
• rtconfig min_dispute_timeout / max_dispute_timeout / max_payment_timeout
• per-token rtconfig min_deadline_delta / max_deadline_delta (open-channel policy)
• config.OpenChannelTimeout, CooperativeWithdrawTimeout, PayResolveTimeout, AdminSendTokenTimeout, TcbTimeoutSeconds
• env-var safe-margin knobs (AGENTPAY_RECV_SAFE_MARGIN_S, AGENTPAY_SEND_SAFE_MARGIN_S, AGENTPAY_WITHDRAW_SAFE_MARGIN_S, default 60 each)

When tuning rtconfig for a new chain, retune in seconds — not blocks. There is no implicit blocks-per-second multiplier in the off-chain code.

Test environment overrides

The e2e test harness sets AGENTPAY_*_SAFE_MARGIN_S=5 in TestMain so the timeout-and-sweep flow runs in seconds instead of minutes. Production deployments should leave these unset (default 60).

Server Flags That Matter Most

Flag	Meaning
-profile	Chain, contract, and OSP profile
-ks	Main keystore for signing and transactions
-depositks	Optional separate deposit signer
-storedir	Local SQLite storage root
-storesql	Shared SQL store URL
-port	Main gRPC endpoint for clients and peers
-adminrpc	Admin gRPC endpoint
-adminweb	Admin HTTP endpoint that serves /admin/ and /metrics
-webapigrpc	Optional pay-centric OSP WebAPI gRPC endpoint for localhost/private same-host callers
-selfrpc	Second gRPC endpoint used in multi-server mode
-rtc	Runtime config file
-isosp	Whether to run with OSP/service-node behavior
-loc	Whether this process listens to on-chain logs
-tlscert, -tlskey, -tlsclient	TLS customization

Only one of -storedir and -storesql should be set.

The phase-1 OSP WebAPI listener intentionally runs without TLS transport credentials, so bind it only to 127.0.0.1:<port> or another private interface used by a colocated same-host caller. It is not intended as a public network-facing API in this cut.

The OSP WebAPI listener is pay-centric only in phase 1. Channel-scoped methods such as GetBalance, Deposit, and CooperativeWithdraw still belong on the admin surface and return codes.Unimplemented on the OSP WebAPI listener.

Deployment Modes

Single-server mode

This is the default and easiest setup:

• one process owns its peers directly
• storage is local or at least logically local to that process
• forwarding never leaves the process boundary

This is what most e2e tests and the simple manual SQLite workflow use.

Multi-server mode

This mode is enabled when the profile or flags provide both shared SQL storage and SelfRPC.

In that mode:

• multiple server processes share storage
• the process exposes the MultiServer gRPC service
• a message may be forwarded to another server if the target client is connected there

The implementation lives in cnode/multiserver.go.

Admin and Operator Interfaces

In practice, most operational control happens through the admin surface exposed by server/server.go:

• admin gRPC server on -adminrpc
• HTTP gateway on -adminweb, mounted under /admin/
• Prometheus metrics on /metrics

If you also enable -webapigrpc, treat that as a local integration surface rather than an operator surface. It is useful for seller-OSP workflows that already speak rpc.WebApiClient, but Admin remains the control plane for channel-scoped operations in phase 1.

The normal operator tool for that surface is tools/osp-cli.

Common admin actions:

• register a peer stream
• open an OSP-to-OSP channel
• send a payment
• start and query deposits
• inspect off-chain and on-chain state

Full command reference: tools/osp-cli/README.md

Embedding a Client

If you are using this backend from application code instead of running only OSP nodes, there are two relevant entry points:

• client/celer_client.go for direct Go integration
• celersdk/api.go for the higher-level SDK interface

The standard client flow is:

1. Create the client with a profile and keystore.
2. Register a stream to the server OSP.
3. Open or instantiate a channel.
4. Deposit, withdraw, and send payments through the SDK/client APIs.

These clients still use the same backend protocol pipeline and storage model described in the implementation guide.

WebAPI Notes

• WebApi.SendToken is the explicit alias for sending a payment without caller-specified app conditions.
• WebApi.SendConditionalPayment remains the lower-level payment API when you want to attach app-level conditions, or when you want to pass an empty conditions list explicitly.
• Even with empty conditions, the runtime may still prepend an internal hash-lock condition for non-direct pays.
• Public WebApi.Deposit and WebApi.CooperativeWithdraw are blocking calls that return after the transaction is mined.
• WebApi.DepositNonBlocking and WebApi.CooperativeWithdrawNonBlocking start jobs that can be tracked with MonitorDepositJob and MonitorCooperativeWithdrawJob on the public surface.
• The matching InternalWebApi non-blocking variants remain available for internal callers.

Practical Notes

• The e2e tests set AGENTPAY_INSECURE_TLS=1 so localhost clients can talk to the server's built-in localhost certificate without CA setup.
• OSP routing behavior only becomes meaningful after the OSP is registered in the on-chain RouterRegistry.
• The server starts periodic OSP cleanup that clears expired or on-chain-resolved payments with peer OSPs.
• rtconfig is operationally important. Payment timeout, refill, and deposit behavior are not hardcoded solely in Go constants.

Suggested Reading Path

If you are new to the repo, this order works well:

1. docs/backend-implementation.md
2. test/manual/README.md
3. tools/osp-cli/README.md
4. docs/backend-troubleshooting.md
5. test/e2e for executable examples