Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
a77f75a
Add UTS test specs for LiveObjects path-based API (~330 tests)
paddybyers May 13, 2026
3de4a4f
Delegate proxy port assignment to uts-proxy in all test specs
paddybyers May 14, 2026
2fba05e
Update UTS test specs to match LiveObjects path-based API spec (a397e34)
paddybyers May 27, 2026
86e9636
UTS: add missing assertions
paddybyers Jun 5, 2026
e57d340
UTS: correct sandbox endpoint domain name
paddybyers Jun 5, 2026
add9398
UTS: remove spurious Map.clear() test
paddybyers Jun 5, 2026
3ad5a41
UTS: remove unnecessary test
paddybyers Jun 5, 2026
e77bdb8
UTS: delete integration GC test — duplicates unit tests
paddybyers Jun 5, 2026
eaa6983
UTS: add Protocol Variants to LiveObjects integration tests
paddybyers Jun 5, 2026
99421b7
UTS: fix root.increment() → root.get("score").increment()
paddybyers Jun 5, 2026
8f52447
UTS: fix InstanceSubscriptionEvent assertions in live_object_subscribe
paddybyers Jun 5, 2026
0b15af8
UTS: move appliedOnAckSerials/bufferedObjectOperations to RealtimeObject
paddybyers Jun 5, 2026
f7ace4c
UTS: fix RTPO19b subscribe-on-detached test
paddybyers Jun 5, 2026
d7d75e7
UTS: rewrite RTO20f to use observable behaviour
paddybyers Jun 5, 2026
e3614cd
UTS: rewrite RTO5c9 re-sync test to use observable behaviour
paddybyers Jun 5, 2026
6bcf593
UTS: add OBJECT, OBJECT_SYNC, ANNOTATION to proxy action numbers table
paddybyers Jun 5, 2026
051bb95
fix(uts/objects): align REST fixture provisioning with V2 objects API
sacOO7 Jun 26, 2026
9883347
fix(uts): align provision_objects_via_rest host with the nonprod sandbox
sacOO7 Jun 26, 2026
356b57f
fix(uts): route objects integration clients to the nonprod sandbox
sacOO7 Jun 26, 2026
4eb5e0d
Removed non-required `objects_batch_spec.md` from the liveobjects uts…
sacOO7 Jun 26, 2026
e10eefb
Added missing endpoint and autoConnect to objects_faults spec
sacOO7 Jun 26, 2026
61f16f8
Merge pull request #497 from ably/fix/uts-liveobjects-rest-provisioning
sacOO7 Jun 29, 2026
36db239
fix(uts): correct objects proxy specs and harden proxy rule/timeout g…
sacOO7 Jul 4, 2026
ffcd875
docs(uts): move proxy infrastructure spec to uts/docs/proxy.md
sacOO7 Jul 4, 2026
61b5afd
Added missing objects_gc_test.md for garbage collecting tombstoned ob…
sacOO7 Jul 6, 2026
f64479c
Fixed relative paths pointing to `proxy.md`
sacOO7 Jul 8, 2026
a1ac5dd
chore(uts): reconcile LiveObjects (objects) test specs with main spec
sacOO7 Jun 30, 2026
e0fd856
fix(uts): sort-safe ack_serial and serial-ordering fixes for objects …
sacOO7 Jul 7, 2026
7fd3e17
fix(uts/objects): fix double-count & stale-serial fixtures, add seria…
sacOO7 Jul 9, 2026
f894640
fix(uts/objects): wire-valid createOps via builder auto-fill; accurat…
sacOO7 Jul 10, 2026
0a531c7
fix(uts/objects): review fixes + naming/behavior audit corrections
sacOO7 Jul 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions uts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,13 @@ uts/
│ │ ├── connection/ # RTN — connection management
│ │ └── presence/ # RTP — realtime presence
│ └── integration/ # Realtime integration tests
│ ├── helpers/
│ │ └── proxy.md # Proxy infrastructure spec
│ ├── proxy/ # Proxy-based fault injection tests
│ └── *.md # Direct sandbox tests
├── docs/ # Guides and reference
│ ├── writing-test-specs.md # How to write UTS specs
│ ├── writing-derived-tests.md # How to translate specs into SDK tests
│ ├── integration-testing.md # Integration testing policy
│ ├── proxy.md # Proxy infrastructure spec (cross-module)
│ └── completion-status.md # Spec coverage matrix
└── README.md # This file
```
Expand Down Expand Up @@ -101,4 +100,4 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo

The programmable proxy for integration testing lives in a separate repository: [ably/uts-proxy](https://github.com/ably/uts-proxy). It sits between the SDK and the Ably sandbox, transparently forwarding traffic while allowing rule-based fault injection.

See `realtime/integration/helpers/proxy.md` for the proxy infrastructure specification used by test specs in this repository.
See `uts/docs/proxy.md` for the proxy infrastructure specification used by test specs in this repository.
15 changes: 11 additions & 4 deletions uts/docs/integration-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,6 @@ realtime/
presence/
presence_lifecycle_test.md
...
helpers/
proxy.md # Proxy infrastructure spec
proxy/ # Proxy-based tests (sandbox + proxy)
connection_open_failures.md
connection_resume.md
Expand Down Expand Up @@ -151,13 +149,12 @@ AFTER ALL TESTS:

### Proxy Setup (integration-proxy only)

Proxy tests additionally set up a proxy session per test or group of tests. See `realtime/integration/helpers/proxy.md` for the proxy infrastructure API.
Proxy tests additionally set up a proxy session per test or group of tests. See `uts/docs/proxy.md` for the proxy infrastructure API.

```pseudo
BEFORE EACH TEST:
session = create_proxy_session(
endpoint: "nonprod:sandbox",
Comment thread
paddybyers marked this conversation as resolved.
port: allocated_port,
rules: [ ...initial rules... ]
)

Expand Down Expand Up @@ -228,6 +225,12 @@ Guidelines:

The goal is: every await in the test is bounded, and the suite timeout is generous enough that it only fires if something truly unexpected happens. When a test fails, the error should say *what* timed out, not just "suite timeout exceeded."

**All timeouts are wall-clock time.** Every `WITH timeout`, `poll_until` and `WAIT` in an
integration spec measures real elapsed time — the test is waiting on a real server over a real
network. Derived tests in frameworks that virtualise time by default must run these waits
against the real clock; see the *Integration timeouts are wall-clock* section in
`writing-derived-tests.md` for the failure mode and recommended helpers.

### Avoiding Flaky Tests

- Use polling with timeouts instead of fixed waits (see `README.md` polling conventions)
Expand Down Expand Up @@ -296,6 +299,10 @@ The following integration test specs are annotated with `## Protocol Variants`:
- `realtime/integration/mutable_messages_test.md`
- `realtime/integration/delta_decoding_test.md`

**LiveObjects:**
- `objects/integration/objects_lifecycle_test.md`
- `objects/integration/objects_sync_test.md`

## Writing Proxy Tests

The proxy mediates between the SDK and the real Ably server. It is not a mock server. Tests should be written to rely on actual server responses as much as possible, with the proxy intervening only where necessary to create the specific fault or error condition under test.
Expand Down
19 changes: 16 additions & 3 deletions uts/realtime/integration/helpers/proxy.md → uts/docs/proxy.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ Proxy integration tests use this to verify fault-handling behaviour against the
# 1. Create a proxy session with rules
session = create_proxy_session(
endpoint: "nonprod:sandbox",
Comment thread
paddybyers marked this conversation as resolved.
port: allocated_port,
rules: [ ...rules... ]
)

Expand Down Expand Up @@ -58,7 +57,7 @@ session.close()
interface ProxySession:
session_id: String
proxy_host: String # Always "localhost"
proxy_port: Int # Assigned from port pool
proxy_port: Int # Auto-assigned by proxy, or explicit if specified

add_rules(rules: List<Rule>, position?: "append"|"prepend")
trigger_action(action: ActionRequest)
Expand All @@ -67,7 +66,7 @@ interface ProxySession:

function create_proxy_session(
endpoint: String, # e.g. "nonprod:sandbox" → resolves to sandbox.realtime.ably-nonprod.net
port: Int,
port?: Int, # Optional; proxy auto-assigns a free port if omitted
rules?: List<Rule>,
timeoutMs?: Int # Session auto-cleanup timeout (default 30000)
): ProxySession
Expand Down Expand Up @@ -113,6 +112,14 @@ Rules are evaluated in order. First matching rule wins. Unmatched traffic passes

**`count`**: 1-based occurrence counter. `count: 2` matches only the 2nd occurrence.

**`action`** (frame matches): must be a **string** — either a protocol action *name*
(e.g. `"ATTACHED"`) or a *numeric string* (e.g. `"20"`). A JSON number is rejected at
session creation with HTTP 400 (`cannot unmarshal number into Go struct field
MatchConfig.match.action of type string`). Note: uts-proxy v0.3.0 resolves action
*names* only up to `AUTH` (17) — an unresolvable name (e.g. `"OBJECT_SYNC"`) makes the
rule silently never match, so use numeric strings for `OBJECT` (`"19"`),
`OBJECT_SYNC` (`"20"`) and `ANNOTATION` (`"21"`) until the proxy's name table is extended.

### Actions

```json
Expand Down Expand Up @@ -203,6 +210,9 @@ ASSERT attach_frames.length == 1
| MESSAGE | 15 | Both |
| SYNC | 16 | Server → Client |
| AUTH | 17 | Client → Server |
| OBJECT | 19 | Both |
| OBJECT_SYNC | 20 | Server → Client |
| ANNOTATION | 21 | Both |

## SDK ClientOptions for Proxy Tests

Expand Down Expand Up @@ -231,3 +241,6 @@ ClientOptions(
7. Timeouts are generous (10-30s) since real network is involved
8. Each test file provisions a sandbox app in `BEFORE ALL TESTS` and cleans up in `AFTER ALL TESTS`
9. Each test creates its own proxy session and cleans it up after
10. All `WITH timeout` / `poll_until` / `WAIT` durations are **wall-clock (real) time** — see
the *Integration timeouts are wall-clock* section in `writing-derived-tests.md` for
the virtual-time trap in derived tests
21 changes: 21 additions & 0 deletions uts/docs/writing-derived-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,27 @@ const timer = setTimeout(() => reject(new Error('Timed out')), 5000);
connection.once('connected', () => { clearTimeout(timer); resolve(); });
```

### Integration timeouts are wall-clock (beware virtual-time frameworks)

The rule above is inverted for **integration and proxy tests**: every `WITH timeout`,
`poll_until` and `WAIT` in an integration spec is **wall-clock (real) time**, because the test
is waiting on a real server or proxy over a real network.

This is a trap in test frameworks that virtualise time by default. For example,
kotlinx-coroutines' `runTest` runs the test body on a virtual clock: a bare `withTimeout(15s)`
wrapping a real network await measures *virtual* time, which fast-forwards the moment the test
coroutine idles — the timeout fires almost instantly, long before the server can respond, with
a misleading "Timed out after 15s" error. The same applies to a bare `delay()`, which skips
instead of waiting.

Derived integration tests in such frameworks must run their waits against the real clock —
e.g. dispatch onto a real-thread dispatcher before applying the timeout
(`withContext(Dispatchers.Default.limitedParallelism(1)) { withTimeout(...) { ... } }` in
Kotlin), or use the framework's escape hatch for real time. Define shared helpers
(`awaitState`, `pollUntil`, `withRealTimeout`, ...) that encapsulate this once, and use them for
every wait in integration test bodies. Unit tests are unaffected — there, fake/virtual timers
remain the preferred mechanism.

### Cleanup with afterEach

Always restore mocks in `afterEach`, not just at the end of each test. If a test throws before its cleanup code, the next test inherits dirty state. Use the SDK's mock restoration mechanism (e.g. `restoreAll()`) in an `afterEach` hook.
Expand Down
13 changes: 7 additions & 6 deletions uts/docs/writing-test-specs.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ This guide provides comprehensive guidance for writing portable test specificati
- Run against Ably Sandbox through a programmable proxy ([ably/uts-proxy](https://github.com/ably/uts-proxy))
- Proxy transparently forwards traffic but can inject faults via rules
- Use for testing fault behaviour: connection failures, token renewal under errors, heartbeat starvation, channel error injection
- See `realtime/integration/helpers/proxy.md` for the full proxy infrastructure spec
- See `uts/docs/proxy.md` for the full proxy infrastructure spec

## Test IDs

Expand Down Expand Up @@ -296,7 +296,7 @@ mock_ws.active_connection.simulate_disconnect()

## Proxy Integration Tests

For detailed proxy infrastructure documentation, see `realtime/integration/helpers/proxy.md`.
For detailed proxy infrastructure documentation, see `uts/docs/proxy.md`.

### When to Use Proxy Tests

Expand All @@ -317,7 +317,7 @@ Spec points: `RTN14a`, `RTN14b`, ...
Proxy integration test against Ably Sandbox endpoint

## Proxy Infrastructure
See `realtime/integration/helpers/proxy.md` for proxy infrastructure specification.
See `uts/docs/proxy.md` for proxy infrastructure specification.

## Corresponding Unit Tests
- `realtime/unit/connection/connection_failures_test.md` — RTN15a, RTN15b
Expand All @@ -344,7 +344,6 @@ Tests that [behaviour] when the proxy injects [fault].
```pseudo
session = create_proxy_session(
target: TargetConfig(realtimeHost: "sandbox.realtime.ably-nonprod.net", restHost: "sandbox.realtime.ably-nonprod.net"),
port: allocated_port,
rules: [{
"match": { ... },
"action": { ... },
Expand Down Expand Up @@ -588,6 +587,10 @@ This means implementations should:
- Otherwise wait for state change events with timeout
- Fail if timeout expires

In **integration and proxy** specs these timeouts are **wall-clock time** (the test waits on a
real server — see the *Timeout Strategy* section in `docs/integration-testing.md`); in unit specs they may be
realised with fake/virtual timers.

## Timer Mocking

Tests verifying timeout behavior should use timer mocking where practical to avoid slow tests.
Expand Down Expand Up @@ -1026,8 +1029,6 @@ realtime/
connection_open_failures_test.md
...
integration/
helpers/
proxy.md # Proxy infrastructure spec
proxy/
connection_open_failures.md # RTN14 tests via proxy
connection_resume.md # RTN15 tests via proxy
Expand Down
Loading
Loading