diff --git a/.github/images/benchmark_ping.png b/.github/images/benchmark_ping.png new file mode 100644 index 000000000..dcd083d59 Binary files /dev/null and b/.github/images/benchmark_ping.png differ diff --git a/CHANGELOG.md b/CHANGELOG.md index e1c5ac9d2..402088a91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,87 @@ All notable changes to this project will be documented in this file. This format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +#### [v3.3.0](https://github.com/ergo-services/ergo/releases/tag/v1.999.330) 2026-xx-xx [tag version v1.999.330] #### + +* Added **pointer type support** in EDF - `*int`, `*string`, `[]*T`, `map[K]*V`, pointer struct fields. Nil state preserved. Nested pointers (`**T`) not supported. Max encoding depth limit (100) prevents stack overflow on deeply nested structures. See [Network Transparency](https://docs.ergo.services/networking/network-transparency) documentation +* Fixed logger to preserve Behavior name when process registers name +* Added **process lifecycle counters** to `gen.NodeInfo` - `ProcessesSpawned`, `ProcessesSpawnFailed`, `ProcessesTerminated` for cumulative statistics +* Added **mailbox latency measurement** (build with `-tags=latency`). `QueueMPSC.Latency()` returns the age of the oldest message in the queue (nanoseconds), -1 if disabled. `ProcessMailbox.Latency()` returns the max across all four queues. Added `MailboxLatency` field to `ProcessShortInfo` and latency fields to `MailboxQueues` in `ProcessInfo`. See [Debugging](https://docs.ergo.services/advanced/debugging) documentation +* Added **`Node.ProcessRangeShortInfo`** for efficient callback-based iteration over all processes with their current state. See [Metrics actor](https://docs.ergo.services/extra-library/actors/metrics) for Prometheus integration +* Added **per-event metrics** - `EventInfo` now includes `MessagesPublished`, `MessagesLocalSent`, `MessagesRemoteSent` counters. Added `Node.EventInfo` and `Node.EventRangeInfo` for querying event statistics. Added `EventsPublished`, `EventsReceived`, `EventsLocalSent`, `EventsRemoteSent` to `NodeInfo`. `EventsPublished` counts only local producer publishes, `EventsReceived` counts events arriving from remote nodes +* Added **process init time measurement** - `InitTime` field in `ProcessShortInfo` and `ProcessInfo` records the time spent in `ProcessInit` callback (nanoseconds) +* Fixed **message counters for meta processes** - meta process traffic now propagates to parent process counters, making `ProcessRangeShortInfo` aggregates balanced +* Fixed **self-send message counter** - `messagesOut` now incremented for self-sends +* Fixed **simultaneous connect dead loop** - two nodes dialing each other at the same time no longer cause infinite retry loops. Deterministic connection IDs and Erlang-style collision detection (`EnableSimultaneousConnect` flag) ensure exactly one connection per pair. Fixed related connection leaks +* Fixed **silent data loss on connection pool write failure** - a transient write error could permanently break a pool item's write path without detection, causing all subsequent messages to be silently dropped while the connection appeared healthy +* Added **software keepalive** for inter-node connections. Application-level heartbeat detects silent failures that TCP keepalive cannot: stuck processes, broken flushers, goroutine starvation. Each side advertises its period during handshake (8 bits in `NetworkFlags`); receiver uses peer's period for timeout. Enabled by default (15s period, 3 misses, 45s timeout). Configure via `NetworkFlags.EnableSoftwareKeepAlive` and `NetworkOptions.SoftwareKeepAliveMisses`. See [Network Stack](https://docs.ergo.services/networking/network-stack#software-keepalive) documentation +* Added **handshake deadline** (5s) to prevent hung handshakes from blocking connection goroutines indefinitely +* Added **message fragmentation** for large messages. Messages exceeding the fragment size (default 65000 bytes) are automatically split for transmission and reassembled on the receiving side. Works with compression, important delivery, and all message types. With `KeepNetworkOrder` disabled, fragments are distributed across all TCP connections in the pool for maximum throughput. Both nodes must enable `EnableFragmentation` flag (enabled by default). Configure via `NetworkOptions.FragmentSize`, `FragmentTimeout`, `MaxFragmentAssemblies`. See [Network Stack](https://docs.ergo.services/networking/network-stack#message-fragmentation) documentation +* Fixed **important delivery use-after-release** - reference ID for acknowledgment was read from buffer after it was returned to the pool, causing corrupted ACK responses under load. Affected `SendImportant` for PID, ProcessID, and Alias targets + +#### [v3.2.0](https://github.com/ergo-services/ergo/releases/tag/v1.999.320) 2026-02-04 [tag version v1.999.320] #### + +* Introduced **mTLS support** - new `gen.CertAuthManager` interface for mutual TLS with CA pool management (`ClientCAs`, `RootCAs`, `ClientAuth`, `ServerName`). See [Mutual TLS](https://docs.ergo.services/networking/mutual-tls) documentation +* Introduced **NAT support** - new `RouteHost` and `RoutePort` options in `gen.AcceptorOptions` for nodes behind NAT or load balancers. See [Behind the NAT](https://docs.ergo.services/networking/behind-the-nat) documentation +* Introduced **spawn time control** - `InitTimeout` option in `gen.ProcessOptions` limits `ProcessInit` duration for both local and remote spawn. Remote spawn and application processes limited to max 15 seconds. See [Process](https://docs.ergo.services/basics/process) documentation +* Introduced **zip-bomb protection** - decompression size limits to prevent memory exhaustion attacks +* Added `gen.Ref` methods for request timeout tracking. See [Generic Types](https://docs.ergo.services/basics/generic-types#gen.ref): + - `Deadline` - returns deadline timestamp stored in reference + - `IsAlive` - checks if reference is still valid (deadline not exceeded) +* Added `gen.Node` methods. See [Node](https://docs.ergo.services/basics/node) documentation: + - `ProcessPID` / `ProcessName` - resolve process PID by name and vice versa + - `Call`, `CallWithTimeout`, `CallWithPriority`, `CallImportant`, `CallPID`, `CallProcessID`, `CallAlias` - synchronous requests from Node interface + - `Inspect` / `InspectMeta` - inspect processes and meta processes + - `MakeRefWithDeadline` - create reference with embedded deadline +* Added `gen.RemoteNode.ApplicationInfo` - query application information from remote nodes. See [Remote Start Application](https://docs.ergo.services/networking/remote-start-application) documentation +* Added `gen.Process` methods. See [Process](https://docs.ergo.services/basics/process) documentation: + - `SendWithPriorityAfter` - delayed send with priority + - `SendExitAfter` / `SendExitMetaAfter` - delayed exit signals + - `SendResponseImportant` / `SendResponseErrorImportant` - important delivery for responses +* Added `gen.Meta` methods. See [Meta Process](https://docs.ergo.services/basics/meta-process) documentation: + - `SendResponse` / `SendResponseError` - respond to requests from meta process + - `SendPriority` / `SetSendPriority` - message priority control + - `Compression` / `SetCompression` - compression settings + - `EnvDefault` - get environment variable with default value +* Added `gen.ApplicationSpec` / `gen.ApplicationInfo` fields: + - `Tags` - labels for instance selection (blue/green, canary, maintenance). See [Tags for Instance Selection](https://docs.ergo.services/basics/application#tags-for-instance-selection) + - `Map` - logical role to process name mapping. See [Process Role Mapping](https://docs.ergo.services/basics/application#process-role-mapping) +* Added **HandleInspect** implementations for all supervisor types (OFO, ARFO, SOFO) +* Fixed **LinkChild** in `RemoteNode.Spawn` / `RemoteNode.SpawnRegister` +* Fixed **args persistence** for Simple One For One supervisor - child processes now restart with their original spawn arguments +* Fixed **critical bug**: terminate signals (Link/Monitor exits) were incorrectly rejected due to wrong incarnation validation in network layer. Thanks to [@qjpcpu](https://github.com/qjpcpu) for reporting [#248](https://github.com/ergo-services/ergo/issues/248) +* Completely reworked internal **Target Manager** (`node/tm/`) - improved architecture for process, event, and node target management with comprehensive test coverage +* Completely reworked internal **Pub/Sub** mechanism - improved reliability and performance +* Improved **ProcessInit state** - more `gen.Process` methods now available during initialization: + - `Link*`, `Unlink*`, `Monitor*`, `Demonitor*` + - `Call*`, `Inspect`, `InspectMeta` + - `RegisterName`, `UnregisterName`, `RegisterEvent`, `UnregisterEvent` + - `SendResponse*`, `SendResponseError*` + - `CreateAlias`, `DeleteAlias` +* Introduced **shutdown timeout** - `ShutdownTimeout` option in `gen.NodeOptions` (default 3 minutes). During graceful shutdown, pending processes are logged every 5 seconds with state and queue info. After timeout, node force exits with error code 1. See [Node](https://docs.ergo.services/basics/node) documentation +* Added **pprof labels** for actor and meta process goroutines (with `--tags pprof`) - each process goroutine is labeled with its PID, each meta process with its Alias, making it easy to identify stuck processes in pprof output +* Improved API documentation - comprehensive godoc comments for all public interfaces +* **Documentation rewritten** - complete documentation now included in the repository (`docs/`) and available at [docs.ergo.services](https://docs.ergo.services) +* New documentation articles: + - [Project Structure](https://docs.ergo.services/basics/project-structure) - organizing projects with message isolation levels, deployment patterns, and evolution strategies + - [Building a Cluster](https://docs.ergo.services/advanced/building-a-cluster) - step-by-step guide to distributed systems with service discovery, load balancing, and failover + - [Message Versioning](https://docs.ergo.services/advanced/message-versioning) - evolving message contracts in distributed clusters with explicit versioning strategies + - [Handle Sync](https://docs.ergo.services/advanced/handle-sync) - synchronous message handling patterns + - [Important Delivery](https://docs.ergo.services/advanced/important-delivery) - guaranteed delivery mechanism + - [Pub/Sub Internals](https://docs.ergo.services/advanced/pub-sub-internals) - event system architecture + - [Debugging](https://docs.ergo.services/advanced/debugging) - build tags, pprof integration, troubleshooting stuck processes + +* **Extra Library - Actors** (https://github.com/ergo-services/actor): + - Introduced **Leader** actor - distributed leader election with Raft-inspired consensus algorithm. Features: term-based disambiguation, automatic failover, split-brain prevention through majority quorum, dynamic peer discovery. See [documentation](https://docs.ergo.services/extra-library/actors/leader) + - Introduced **Metrics** actor - Prometheus metrics exporter that collects node/network telemetry via HTTP endpoint. Features: automatic collection of node metrics (uptime, processes, memory), network metrics per remote node, extensible for custom metrics. See [documentation](https://docs.ergo.services/extra-library/actors/metrics) + +* **Extra Library - Meta Processes** (https://github.com/ergo-services/meta): + - Introduced **SSE** (Server-Sent Events) meta-process - unidirectional server-to-client streaming over HTTP. Features: server handler for accepting connections, client connection for external SSE endpoints, full SSE spec support (event types, IDs, retry hints, multi-line data), process pool with round-robin load balancing, Last-Event-ID for reconnection. See [documentation](https://docs.ergo.services/extra-library/meta-processes/sse) + +* **Benchmarks** (https://github.com/ergo-services/benchmarks): + - Introduced **Distributed Pub/Sub** benchmark - demonstrates event delivery to 1,000,000 subscribers across 10 nodes. Achieves 2.9M msg/sec delivery rate with only 10 network messages (one per consumer node) instead of 1M + + #### [v3.1.0](https://github.com/ergo-services/ergo/releases/tag/v1.999.310) 2025-09-04 [tag version v1.999.310] #### **New Features** diff --git a/README.md b/README.md index 4a9d938b5..349d5960f 100644 --- a/README.md +++ b/README.md @@ -1,65 +1,157 @@

Ergo Framework

[![Gitbook Documentation](https://img.shields.io/badge/GitBook-Documentation-f37f40?style=plastic&logo=gitbook&logoColor=white&style=flat)](https://docs.ergo.services) -[![GoDoc](https://pkg.go.dev/badge/ergo-services/ergo)](https://pkg.go.dev/ergo.services/ergo) [![MIT license](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://opensource.org/licenses/MIT) [![Telegram Community](https://img.shields.io/badge/Telegram-ergo__services-229ed9?style=flat&logo=telegram&logoColor=white)](https://t.me/ergo_services) [![Reddit](https://img.shields.io/badge/Reddit-r/ergo__services-ff4500?style=plastic&logo=reddit&logoColor=white&style=flat)](https://reddit.com/r/ergo_services) +[![ergo.cloud](https://img.shields.io/badge/ergo.cloud-Cloud_Overlay_Network-00b4d8?style=flat&logo=icloud&logoColor=white)](https://ergo.cloud) -The Ergo Framework is an implementation of ideas, technologies, and design patterns from the Erlang world in the Go programming language. It is based on the actor model, network transparency, and a set of ready-to-use components for development. This significantly simplifies the creation of complex and distributed solutions while maintaining a high level of reliability and performance. +**Actor model for Go. Build distributed systems without the distributed systems headache.** -### Features ### +Goroutines and channels work great until your system grows. Then come the mutexes, the race conditions, the service discovery configs, the retry logic, the connection pool management. Ergo replaces all of that with one model: isolated processes that communicate through messages, supervised automatically, addressable across any cluster. -1. **Actor Model**: isolated processes communicate through message passing, handling messages sequentially in their own mailbox with four priority queues. Processes support both asynchronous messaging and synchronous request-response patterns, enabling flexible communication while maintaining the actor model guarantees. +Inspired by Erlang/OTP. Zero external dependencies. Pure Go. -2. **Network Transparency**: actors interact the same way whether local or remote. The framework uses custom serialization and protocol for [efficient](https://github.com/ergo-services/benchmarks) distributed communication with connection pooling, compression, and type caching, making network location transparent to application code. +### The core idea in 30 seconds ### -3. **Supervision Trees**: hierarchical fault recovery where supervisors monitor child processes and apply restart strategies when failures occur. Supports multiple supervision types (One For One, All For One, Rest For One, Simple One For One) and restart strategies (Transient, Temporary, Permanent) for building self-healing systems. +```go +type Counter struct { + act.Actor + count int +} -4. **Meta Processes**: bridge blocking I/O with the actor model through dedicated meta processes that handle TCP, UDP, Port, and Web protocols. Meta processes run blocking operations without affecting regular actor message processing. +type MessageInc struct{} -5. **Distributed Systems**: service discovery through embedded or external registrars (etcd, Saturn), distributed publish/subscribe events with token-based authorization and buffering, remote process spawning with factory-based permissions, and remote application orchestration across nodes. +func (c *Counter) HandleMessage(from gen.PID, msg any) error { + switch msg.(type) { + case MessageInc: + // safe without locks even with thousands of concurrent senders: + // messages are processed one at a time + c.count++ + c.Log().Info("count: %d", c.count) + } + return nil +} -6. **Ready-to-use Components**: core framework includes Actor, Supervisor, Pool, and WebWorker actors plus TCP, UDP, Port, and Web meta processes. Extra library provides Leader, Metrics actors, WebSocket, SSE meta processes, Observer application, Colored and Rotate loggers, Erlang protocol support. +func factory_Counter() gen.ProcessBehavior { return &Counter{} } -7. **Flexibility**: customize network stack components, certificate management, compression and message priorities, logging, distributed events, and meta processes. Framework supports mTLS, NAT traversal, important delivery for guaranteed messaging, and Cron-based scheduling. +// Start a node and spawn the actor +node, _ := ergo.StartNode("mynode@localhost", gen.NodeOptions{}) +pid, _ := node.Spawn(factory_Counter, gen.ProcessOptions{}) -Examples demonstrating the framework's capabilities are available in the [examples repository](https://github.com/ergo-services/examples). +// Same API whether local or on another continent +node.Send(pid, MessageInc{}) +node.Send(pid, MessageInc{}) +``` -### Benchmarks ### +No locks. No race conditions. Sequential message handling is the guarantee. -On a 64-core processor, Ergo Framework demonstrates a performance of **over 21 million messages per second locally** and **nearly 5 million messages per second over the network**. +### Why not just goroutines + channels? ### -![image](https://github.com/user-attachments/assets/813900ad-ff79-4cc7-b1e6-396c5781acb1) +| | Goroutines + channels | Ergo | +|---|---|---| +| Shared state | You manage with mutexes | No shared state by design | +| Failure recovery | Manual | Supervision trees restart automatically | +| Cross-node messaging | Build it yourself | Same API, transparent | +| Service discovery | External tool needed | Built in | +| Race conditions | Possible | Impossible within a process | -Available benchmarks can be found in the [benchmarks repository](https://github.com/ergo-services/benchmarks). +### What you can build ### -* Messaging performance (local, network) +**Real-time backends.** Each WebSocket connection becomes an addressable actor. Any node in your cluster can push to any specific client. No pub/sub intermediaries. -* Memory consumption per process (demonstrates framework memory footprint) +**IoT platforms.** One actor per device. Thousands of devices per node. Supervisors restart failed device actors automatically. -* Serialization performance comparison: EDF vs Protobuf vs Gob +**Multi-agent AI systems.** Each agent is an isolated actor with a mailbox. Crash isolation, supervision, distributed addressability, and a built-in [MCP server](https://docs.ergo.services/extra-library/applications/mcp) that exposes your running cluster to any AI assistant (Claude Code, Cursor, and other MCP-compatible clients). See [AI Agents](https://docs.ergo.services/ai-agents) for patterns and diagnostics. -* Distributed Pub/Sub (event delivery to 1,000,000 subscribers across 10 nodes) +**Financial and event-driven systems.** Four priority queues per mailbox, guaranteed delivery, no dropped messages. -### Observer ### -To inspect the node, network stack, running applications, and processes, you can use the [`observer`](https://github.com/ergo-services/tools/) tool +**Distributed Pub/Sub across the cluster.** Producer registers an event once; any process on any node subscribes. The framework delivers one network message per node, not per subscriber. 1M subscribers across 10 nodes cost 10 network messages, not 1M. - +```go +// Producer on any node +token, _ := producer.RegisterEvent("prices", gen.EventOptions{}) +producer.SendEvent("prices", token, PriceUpdate{Asset: "BTC", Price: 95000}) -To install the Observer tool, you need to have the Go compiler version 1.20 or higher. Run the following command: +// Subscriber on any other node, identical API +process.MonitorEvent(gen.Event{Name: "prices", Node: "producer@host"}) +func (s *Sub) HandleEvent(event gen.MessageEvent) error { + fmt.Println(event.Message.(PriceUpdate)) + return nil +} ``` -$ go install ergo.tools/observer@latest + +### Performance ### + +On a 64-core processor: + +* **21M+ messages/second** locally +* **~5.5M messages/second** over the network +* **Distributed Pub/Sub**: 2.9M msg/sec delivery to 1,000,000 subscribers across 10 nodes + +Lock-free queues. Processes sleep when idle. No CPU wasted. + +![image](.github/images/benchmark_ping.png) + +Full benchmarks: [benchmarks repository](https://github.com/ergo-services/benchmarks). + +### Observer ### + +Observer is a real-time web UI for monitoring and inspecting Ergo nodes. It provides live visibility into every layer of the system: + +- **Processes** - full process list with state, mailbox depth, latency, running time, wakeups, and uptime. Click any process to inspect its supervision tree, links, monitors, aliases, environment, and internal actor state +- **Applications** - running applications with their process trees, modes, and uptime +- **Network** - cluster topology, per-node connection details, traffic counters, and protocol info +- **Events** - registered events with producer, subscriber counts, and publication statistics +- **Logs** - live log stream with level filtering across the cluster +- **Profiler** - goroutine dump with grouping and stack traces, heap profile with allocation breakdown, and GC pressure charts + + + +Add Observer to your node as an application: + +```go +import "ergo.services/application/observer" + +options.Applications = []gen.ApplicationBehavior{ + observer.CreateApp(observer.Options{}), +} ``` -You can also embed the [Observer application](https://docs.ergo.services/extra-library/applications/observer) into your node. To see it in action, see the [demo example](https://github.com/ergo-services/examples/tree/master/demo). For more information, visit the [Observer documentation](https://docs.ergo.services/tools/observer). +To see it in action with a fully loaded cluster, see the [observability example](https://github.com/ergo-services/examples/tree/master/observability). For more information, visit the [Observer documentation](https://docs.ergo.services/extra-library/applications/observer). + +### Features ### + +1. **Actor Model:** isolated processes communicate through message passing, handling messages sequentially with four priority queues. Supports asynchronous messaging and synchronous request-response, with per-process [mailbox latency measurement](https://docs.ergo.services/advanced/debugging#mailbox-latency) (`-tags=latency`) for production diagnostics. + +2. **Network Transparency:** actors interact the same way whether local or remote. Uses EDF (Ergo Data Format), a custom binary serialization with type caching, pointer support, and [message versioning](https://docs.ergo.services/advanced/message-versioning) for seamless upgrades. Includes connection pooling, compression, [message fragmentation](https://docs.ergo.services/networking/network-stack#message-fragmentation), and [application-level keepalive](https://docs.ergo.services/networking/network-stack#software-keepalive) for silent failure detection. + +3. **Supervision Trees:** hierarchical fault recovery where supervisors monitor child processes and apply configurable restart strategies. Supports One For One, All For One, Rest For One, and Simple One For One supervision types with Transient, Temporary, and Permanent restart policies. + +4. **Meta Processes:** bridge blocking I/O with the actor model through dedicated meta processes handling [TCP](https://docs.ergo.services/meta-processes/tcp), [UDP](https://docs.ergo.services/meta-processes/udp), [Port](https://docs.ergo.services/meta-processes/port), [Web](https://docs.ergo.services/meta-processes/web), [WebSocket](https://docs.ergo.services/extra-library/meta-processes/websocket), and [SSE](https://docs.ergo.services/extra-library/meta-processes/sse) protocols without affecting regular actor message processing. + +5. **Distributed Systems:** service discovery via embedded or external registrars ([etcd](https://docs.ergo.services/extra-library/registrars/etcd-client), [Saturn](https://docs.ergo.services/extra-library/registrars/saturn-client)), distributed [publish/subscribe events](https://docs.ergo.services/basics/events) with token-based authorization and buffering, [remote process spawning](https://docs.ergo.services/networking/remote-spawn-process) with factory-based permissions, [remote application orchestration](https://docs.ergo.services/networking/remote-start-application) across nodes, and Raft-based [leader election](https://docs.ergo.services/extra-library/actors/leader) without external dependencies for coordinating exclusive work across cluster replicas. + +6. **Observability:** real-time cluster inspection via the [Observer](https://docs.ergo.services/extra-library/applications/observer) web UI, native [distributed tracing](https://docs.ergo.services/advanced/distributed-tracing) that follows message chains across nodes with automatic propagation (exportable to OTLP backends like Grafana Tempo or Jaeger via [Pulse](https://docs.ergo.services/extra-library/applications/pulse)), and production metrics via [Radar](https://docs.ergo.services/extra-library/applications/radar) with a ready-to-use Grafana dashboard covering process lifecycle, mailbox pressure, network traffic, and event fanout. The extensible [Metrics](https://docs.ergo.services/extra-library/actors/metrics) actor adds custom Prometheus collectors alongside built-in node telemetry. + +7. **AI-Native:** built-in [MCP server](https://docs.ergo.services/extra-library/applications/mcp) exposes the full cluster to AI agents (Claude, Cursor, and any MCP-compatible client). Inspect processes, query events, capture goroutine dumps, stream logs, and run real-time samplers through natural language, turning any AI assistant into an interactive SRE for your Ergo cluster. + +8. **Cloud Native:** built-in Kubernetes health probes (liveness, readiness, startup) via the [Health](https://docs.ergo.services/extra-library/actors/health) actor, [Prometheus](https://docs.ergo.services/extra-library/actors/metrics) metrics endpoint, and [mTLS](https://docs.ergo.services/networking/mutual-tls) support for zero-trust deployments. + +9. **Ready-to-use Components:** core framework includes Actor, Supervisor, Pool, and WebWorker actors plus TCP, UDP, Port, WebSocket, SSE, and Web meta processes. Extra library provides [Leader](https://docs.ergo.services/extra-library/actors/leader), [Metrics](https://docs.ergo.services/extra-library/actors/metrics), and [Health](https://docs.ergo.services/extra-library/actors/health) actors, [Observer](https://docs.ergo.services/extra-library/applications/observer) and [Radar](https://docs.ergo.services/extra-library/applications/radar) applications, [Colored](https://docs.ergo.services/extra-library/loggers/colored) and [Rotate](https://docs.ergo.services/extra-library/loggers/rotate) loggers. +10. **Erlang Interoperability:** native support for the [Erlang distribution protocol](https://docs.ergo.services/extra-library/network-protocols/erlang) enables heterogeneous clusters where Ergo (Go) and Erlang/Elixir nodes participate as equal peers. Send messages, spawn processes, and set up links and monitors across language boundaries without any proxies or bridges. +11. **Flexibility:** customize network stack, certificate management ([mTLS](https://docs.ergo.services/networking/mutual-tls), [NAT traversal](https://docs.ergo.services/networking/behind-the-nat)), compression and message priorities, [Cron-based scheduling](https://docs.ergo.services/basics/cron), [important delivery](https://docs.ergo.services/advanced/important-delivery) for guaranteed messaging, and logging. The [`ergo`](https://docs.ergo.services/tools/ergo) CLI tool generates project scaffolding, actors, supervisors, and message types from the command line. + +Examples demonstrating the framework's capabilities are available in the [examples repository](https://github.com/ergo-services/examples). + +Questions and answers: [FAQ](https://docs.ergo.services/faq). ### Quick start ### -For a quick start, use the [`ergo`](https://docs.ergo.services/tools/ergo) tool — a command-line utility designed to simplify the process of generating boilerplate code for your project based on the Ergo Framework. With this tool, you can rapidly create a complete project structure, including applications, actors, supervisors, network components, and more. It offers a set of arguments that allow you to customize the project according to specific requirements, ensuring it is ready for immediate development. +The [`ergo`](https://docs.ergo.services/tools/ergo) CLI generates project scaffolding for you: applications, actors, supervisors, message types. The output is a complete, runnable project structure. Add components incrementally as your service grows. To install use the following command: @@ -67,135 +159,85 @@ To install use the following command: $ go install ergo.tools/ergo@latest ``` -Now, you can create your project with just one command. Here is example: - -Supervision tree +Create a project and start adding components: ``` - mynode - ├─ myapp - │ │ - │ └─ mysup - │ │ - │ └─ myactor - ├─ myweb - └─ myactor2 +$ ergo init MyNode github.com/myorg/mynode +$ cd mynode +$ ergo add supervisor MyNodeApp:MySup +$ ergo add actor MySup:MyActor +$ go run ./cmd ``` -To generate project for this design use the following command: +The generated project is ready to run immediately. Add more components as your service grows: ``` -$ ergo -init MyNode \ - -with-app MyApp \ - -with-sup MyApp:MySup \ - -with-actor MySup:MyActor \ - -with-web MyWeb \ - -with-actor MyActor2 \ - -with-observer +$ ergo add actor --pool MySup:MyPool +$ ergo add app BackgroundApp +$ ergo add message MessageConnect --field ID:gen.Alias --field Addr:string ``` -as a result you will get generated project: +For the full command reference, see the [ergo tool documentation](https://docs.ergo.services/tools/ergo). -``` - mynode - ├── apps - │ └── myapp - │ ├── myactor.go - │ ├── myapp.go - │ └── mysup.go - ├── cmd - │ ├── myactor2.go - │ ├── mynode.go - │ ├── myweb.go - │ └── myweb_worker.go - ├── go.mod - ├── go.sum - └── README.md -``` +### Claude Code integration ### + +Pre-built agents and skills for [Claude Code](https://claude.com/claude-code) turn any Claude session into an Ergo-aware collaborator. Two paired toolkits shipped in the [ergo-services/claude](https://github.com/ergo-services/claude) repository: + +- **framework** - designing and implementing actor systems. An architect agent (DDD bounded contexts, supervision trees, cluster topology, load analysis) plus a skill with progressive-disclosure references covering actors, supervision, messages, applications, pool, meta processes, node configuration, EDF, cluster, unit testing, and every extension library. + +- **devops** - diagnosing running clusters via the built-in [MCP application](https://docs.ergo.services/extra-library/applications/mcp). An SRE agent that runs hypothesis-driven investigations (observe, hypothesize, test, confirm) plus a skill with the full 48-tool catalog, counters reference, 10 diagnostic playbooks, active/passive sampler recipes, and build-tag awareness. -to try it: +Install as a Claude Code plugin (one-shot, updates managed by Claude Code): ``` -$ cd mynode -$ go run ./cmd +/plugin marketplace add ergo-services/claude +/plugin install ergo@ergo-services ``` -Since we included Observer application, open http://localhost:9911 to inspect your node and running processes. +After install, invoke the skills as `/ergo:framework` or `/ergo:devops`. Agents pick themselves up from trigger phrases ("design ergo application", "why is it slow", "check cluster health", etc.). -### Erlang support ### +### ergo.cloud ### -Starting from version 3.0.0, support for the Erlang network stack has been moved to a [separate module](https://github.com/ergo-services/proto). Version 3.0 was distributed under the BSL 1.1 license, but starting from version 3.1 it is available under the MIT license. Detailed information is available in the [Erlang protocol documentation](https://docs.ergo.services/extra-library/network-protocols/erlang). +[ergo.cloud](https://ergo.cloud) is an overlay network that connects Ergo nodes across AWS, GCP, Azure, and bare metal into a single transparent cluster without VPNs, proxies, or tunnels. End-to-end encrypted. Currently available via [waitlist](https://ergo.cloud). ### Requirements ### -* Go 1.20.x and above +* Go 1.21.x and above ### Changelog ### Fully detailed changelog see in the [ChangeLog](CHANGELOG.md) file. -#### [v3.2.0](https://github.com/ergo-services/ergo/releases/tag/v1.999.320) 2026-02-04 [tag version v1.999.320] #### - -* Introduced **mTLS support** - new `gen.CertAuthManager` interface for mutual TLS with CA pool management (`ClientCAs`, `RootCAs`, `ClientAuth`, `ServerName`). See [Mutual TLS](https://docs.ergo.services/networking/mutual-tls) documentation -* Introduced **NAT support** - new `RouteHost` and `RoutePort` options in `gen.AcceptorOptions` for nodes behind NAT or load balancers. See [Behind the NAT](https://docs.ergo.services/networking/behind-the-nat) documentation -* Introduced **spawn time control** - `InitTimeout` option in `gen.ProcessOptions` limits `ProcessInit` duration for both local and remote spawn. Remote spawn and application processes limited to max 15 seconds. See [Process](https://docs.ergo.services/basics/process) documentation -* Introduced **zip-bomb protection** - decompression size limits to prevent memory exhaustion attacks -* Added `gen.Ref` methods for request timeout tracking. See [Generic Types](https://docs.ergo.services/basics/generic-types#gen.ref): - - `Deadline` - returns deadline timestamp stored in reference - - `IsAlive` - checks if reference is still valid (deadline not exceeded) -* Added `gen.Node` methods. See [Node](https://docs.ergo.services/basics/node) documentation: - - `ProcessPID` / `ProcessName` - resolve process PID by name and vice versa - - `Call`, `CallWithTimeout`, `CallWithPriority`, `CallImportant`, `CallPID`, `CallProcessID`, `CallAlias` - synchronous requests from Node interface - - `Inspect` / `InspectMeta` - inspect processes and meta processes - - `MakeRefWithDeadline` - create reference with embedded deadline -* Added `gen.RemoteNode.ApplicationInfo` - query application information from remote nodes. See [Remote Start Application](https://docs.ergo.services/networking/remote-start-application) documentation -* Added `gen.Process` methods. See [Process](https://docs.ergo.services/basics/process) documentation: - - `SendWithPriorityAfter` - delayed send with priority - - `SendExitAfter` / `SendExitMetaAfter` - delayed exit signals - - `SendResponseImportant` / `SendResponseErrorImportant` - important delivery for responses -* Added `gen.Meta` methods. See [Meta Process](https://docs.ergo.services/basics/meta-process) documentation: - - `SendResponse` / `SendResponseError` - respond to requests from meta process - - `SendPriority` / `SetSendPriority` - message priority control - - `Compression` / `SetCompression` - compression settings - - `EnvDefault` - get environment variable with default value -* Added `gen.ApplicationSpec` / `gen.ApplicationInfo` fields: - - `Tags` - labels for instance selection (blue/green, canary, maintenance). See [Tags for Instance Selection](https://docs.ergo.services/basics/application#tags-for-instance-selection) - - `Map` - logical role to process name mapping. See [Process Role Mapping](https://docs.ergo.services/basics/application#process-role-mapping) -* Added **HandleInspect** implementations for all supervisor types (OFO, ARFO, SOFO) -* Fixed **LinkChild** in `RemoteNode.Spawn` / `RemoteNode.SpawnRegister` -* Fixed **args persistence** for Simple One For One supervisor - child processes now restart with their original spawn arguments -* Fixed **critical bug**: terminate signals (Link/Monitor exits) were incorrectly rejected due to wrong incarnation validation in network layer. Thanks to [@qjpcpu](https://github.com/qjpcpu) for reporting [#248](https://github.com/ergo-services/ergo/issues/248) -* Completely reworked internal **Target Manager** (`node/tm/`) - improved architecture for process, event, and node target management with comprehensive test coverage -* Completely reworked internal **Pub/Sub** mechanism - improved reliability and performance -* Improved **ProcessInit state** - more `gen.Process` methods now available during initialization: - - `Link*`, `Unlink*`, `Monitor*`, `Demonitor*` - - `Call*`, `Inspect`, `InspectMeta` - - `RegisterName`, `UnregisterName`, `RegisterEvent`, `UnregisterEvent` - - `SendResponse*`, `SendResponseError*` - - `CreateAlias`, `DeleteAlias` -* Introduced **shutdown timeout** - `ShutdownTimeout` option in `gen.NodeOptions` (default 3 minutes). During graceful shutdown, pending processes are logged every 5 seconds with state and queue info. After timeout, node force exits with error code 1. See [Node](https://docs.ergo.services/basics/node) documentation -* Added **pprof labels** for actor and meta process goroutines (with `--tags pprof`) - each process goroutine is labeled with its PID, each meta process with its Alias, making it easy to identify stuck processes in pprof output -* Improved API documentation - comprehensive godoc comments for all public interfaces -* **Documentation rewritten** - complete documentation now included in the repository (`docs/`) and available at [docs.ergo.services](https://docs.ergo.services) -* New documentation articles: - - [Project Structure](https://docs.ergo.services/basics/project-structure) - organizing projects with message isolation levels, deployment patterns, and evolution strategies - - [Building a Cluster](https://docs.ergo.services/advanced/building-a-cluster) - step-by-step guide to distributed systems with service discovery, load balancing, and failover - - [Message Versioning](https://docs.ergo.services/advanced/message-versioning) - evolving message contracts in distributed clusters with explicit versioning strategies - - [Handle Sync](https://docs.ergo.services/advanced/handle-sync) - synchronous message handling patterns - - [Important Delivery](https://docs.ergo.services/advanced/important-delivery) - guaranteed delivery mechanism - - [Pub/Sub Internals](https://docs.ergo.services/advanced/pub-sub-internals) - event system architecture - - [Debugging](https://docs.ergo.services/advanced/debugging) - build tags, pprof integration, troubleshooting stuck processes - -* **Extra Library - Actors** (https://github.com/ergo-services/actor): - - Introduced **Leader** actor - distributed leader election with Raft-inspired consensus algorithm. Features: term-based disambiguation, automatic failover, split-brain prevention through majority quorum, dynamic peer discovery. See [documentation](https://docs.ergo.services/extra-library/actors/leader) - - Introduced **Metrics** actor - Prometheus metrics exporter that collects node/network telemetry via HTTP endpoint. Features: automatic collection of node metrics (uptime, processes, memory), network metrics per remote node, extensible for custom metrics. See [documentation](https://docs.ergo.services/extra-library/actors/metrics) - -* **Extra Library - Meta Processes** (https://github.com/ergo-services/meta): - - Introduced **SSE** (Server-Sent Events) meta-process - unidirectional server-to-client streaming over HTTP. Features: server handler for accepting connections, client connection for external SSE endpoints, full SSE spec support (event types, IDs, retry hints, multi-line data), process pool with round-robin load balancing, Last-Event-ID for reconnection. See [documentation](https://docs.ergo.services/extra-library/meta-processes/sse) - -* **Benchmarks** (https://github.com/ergo-services/benchmarks): - - Introduced **Distributed Pub/Sub** benchmark - demonstrates event delivery to 1,000,000 subscribers across 10 nodes. Achieves 2.9M msg/sec delivery rate with only 10 network messages (one per consumer node) instead of 1M +#### [v3.3.0](https://github.com/ergo-services/ergo/releases/tag/v1.999.330) 2026-xx-xx [tag version v1.999.330] #### + +* Added **pointer type support** in EDF - `*int`, `*string`, `[]*T`, `map[K]*V`, pointer struct fields. Nil state preserved. Nested pointers (`**T`) not supported. Max encoding depth limit (100) prevents stack overflow on deeply nested structures. See [Network Transparency](https://docs.ergo.services/networking/network-transparency) documentation +* Added **per-type encode/decode statistics** (build with `-tags=typestats`). Tracks count of root-level operations and decompressed wire-byte volume per registered EDF type. Available via `Network().RegisteredTypes()` and the Observer Types panel. Helps identify heavy message types and decide where to enable compression. Overhead approximately 2-3% on encode/decode throughput +* Fixed logger to preserve Behavior name when process registers name + +* Added **process lifecycle counters** to `NodeInfo` - `ProcessesSpawned`, `ProcessesSpawnFailed`, `ProcessesTerminated` for cumulative statistics + +* Added **mailbox latency measurement** (build with `-tags=latency`). `QueueMPSC.Latency()` returns the age of the oldest message in the queue (nanoseconds), -1 if disabled. `ProcessMailbox.Latency()` returns the max across all four queues. Added `MailboxLatency` field to `ProcessShortInfo` and `MailboxQueues` latency fields to `ProcessInfo`. Added `Node.ProcessRangeShortInfo` for efficient iteration over all processes. See [actor/metrics](https://github.com/ergo-services/actor-metrics) for Prometheus integration with histogram, top-N, and Grafana dashboard + +* Added **per-event metrics** - `EventInfo` now includes `MessagesPublished`, `MessagesLocalSent`, `MessagesRemoteSent` counters. Added `Node.EventInfo` and `Node.EventRangeInfo` for querying event statistics. Added `EventsPublished`, `EventsReceived`, `EventsLocalSent`, `EventsRemoteSent` to `NodeInfo`. `EventsPublished` counts only local producer publishes, `EventsReceived` counts events arriving from remote nodes + +* Added **process init time measurement** - `InitTime` field in `ProcessShortInfo` and `ProcessInfo` records the time spent in `ProcessInit` callback (nanoseconds). Enables detection of slow process initialization + +* Fixed **message counters for meta processes** - meta process traffic now propagates to parent process counters (`messagesIn`/`messagesOut`), making `ProcessRangeShortInfo` aggregates balanced. Meta process own counters preserved for meta-level observability + +* Fixed **self-send message counter** - `messagesOut` now incremented for self-sends (process sending to itself), consistent with other send paths + +* Fixed **simultaneous connect dead loop** - two nodes dialing each other at the same time no longer cause infinite retry loops. Deterministic connection IDs and Erlang-style collision detection (`EnableSimultaneousConnect` flag) ensure exactly one connection per pair. Fixed related connection leaks + +* Fixed **silent data loss on connection pool write failure** - a transient write error could permanently break a pool item's write path without detection, causing all subsequent messages to be silently dropped while the connection appeared healthy + +* Added **software keepalive** for inter-node connections - application-level heartbeat that detects silent failures invisible to TCP keepalive. Enabled by default (15s period, 3 misses, 45s timeout). See [Network Stack](https://docs.ergo.services/networking/network-stack#software-keepalive) documentation + +* Added **handshake deadline** (5s) to prevent hung handshakes from blocking connection goroutines indefinitely + +* Added **message fragmentation** for large messages. Messages exceeding the fragment size (default 65000 bytes) are automatically split and reassembled. With `KeepNetworkOrder` disabled, fragments use all TCP connections for maximum throughput. See [Network Stack](https://docs.ergo.services/networking/network-stack#message-fragmentation) documentation + +* Fixed **important delivery use-after-release** - reference ID read from buffer after pool release, causing corrupted ACK responses ### Development and debugging ### @@ -220,7 +262,11 @@ This helps identify stuck processes during shutdown by matching PIDs/Aliases fro To disable panic recovery use `--tags norecover`. -To enable trace logging level for the internals (node, network,...) use `--tags trace` and set the log level `gen.LogLevelTrace` for your node. +To enable mailbox latency measurement use `--tags latency`. This adds a monotonic timestamp to every message pushed into the MPSC queue, allowing `QueueMPSC.Latency()` and `ProcessMailbox.Latency()` to report the age of the oldest unprocessed message. Overhead is approximately 10-25% on micro-benchmarks (LOCAL 1-1 scenario). Without the tag, `Latency()` returns -1 and there is zero overhead. + +To enable per-type encode/decode statistics use `--tags typestats`. This tracks the count of root-level encode/decode operations and decompressed wire-byte volume per registered EDF type, exposed via `Network().RegisteredTypes()` and visible in the Observer Types panel. Helps identify which message types dominate network traffic and which processes would benefit from compression. Overhead is approximately 2-3% on encode/decode throughput. Without the tag, counters remain zero and there is zero overhead. + +To enable trace logging level for the internals (node, network,...) use `--tags verbose` and set the log level `gen.LogLevelTrace` for your node. For detailed debugging techniques, troubleshooting scenarios, and best practices, see the [Debugging](https://docs.ergo.services/advanced/debugging) documentation. diff --git a/act/actor.go b/act/actor.go index ee4599df5..dea127a75 100644 --- a/act/actor.go +++ b/act/actor.go @@ -4,7 +4,7 @@ import ( "fmt" "reflect" "runtime" - "strings" + "time" "ergo.services/ergo/gen" "ergo.services/ergo/lib" @@ -45,6 +45,9 @@ type ActorBehavior interface { // this event using gen.Process.LinkEvent or gen.Process.MonitorEvent HandleEvent(message gen.MessageEvent) error + // HandleSpan invoked on a tracing span if this process was added as a tracing exporter. + HandleSpan(message gen.TracingSpan) error + // HandleInspect invoked on the request made with gen.Process.Inspect(...) HandleInspect(from gen.PID, item ...string) map[string]string } @@ -102,8 +105,7 @@ func (a *Actor) ProcessInit(process gen.Process, args ...any) (rr error) { var ok bool if a.behavior, ok = process.Behavior().(ActorBehavior); ok == false { - unknown := strings.TrimPrefix(reflect.TypeOf(process.Behavior()).String(), "*") - return fmt.Errorf("ProcessInit: not an ActorBehavior %s", unknown) + return fmt.Errorf("ProcessInit: not an ActorBehavior %s", process.BehaviorName()) } if lib.Recover() { @@ -125,6 +127,7 @@ func (a *Actor) ProcessInit(process gen.Process, args ...any) (rr error) { func (a *Actor) ProcessRun() (rr error) { var message *gen.MailboxMessage + var savedTracing gen.Tracing if lib.Recover() { defer func() { @@ -186,6 +189,13 @@ func (a *Actor) ProcessRun() (rr error) { retry: switch message.Type { case gen.MailboxMessageTypeRegular: + // activate tracing context from the incoming message + messageHasTracing := message.Tracing.ID != [2]uint64{} + if messageHasTracing { + savedTracing = a.PropagatingTrace() + a.SetPropagatingTrace(message.Tracing) + } + var reason error if a.split { @@ -202,10 +212,28 @@ func (a *Actor) ProcessRun() (rr error) { } if reason != nil { + if messageHasTracing { + a.sendSpanProcessed(message, gen.TracingKindSend, reason.Error()) + } return reason } + if messageHasTracing { + a.sendSpanProcessed(message, gen.TracingKindSend, "") + // restore tracing only if handler didn't change it + if a.PropagatingTrace().ID == message.Tracing.ID { + a.SetPropagatingTrace(savedTracing) + } + } + case gen.MailboxMessageTypeRequest: + // activate tracing context from the incoming message + messageHasTracing := message.Tracing.ID != [2]uint64{} + if messageHasTracing { + savedTracing = a.PropagatingTrace() + a.SetPropagatingTrace(message.Tracing) + } + var reason error var result any @@ -223,21 +251,40 @@ func (a *Actor) ProcessRun() (rr error) { } if reason != nil { - // if reason is "normal" and we got response - send it before termination if reason == gen.TerminateReasonNormal && result != nil { + if messageHasTracing { + a.sendSpanProcessed(message, gen.TracingKindRequest, "") + } a.SendResponse(message.From, message.Ref, result) + return reason + } + if messageHasTracing { + a.sendSpanProcessed(message, gen.TracingKindRequest, reason.Error()) } return reason } if result == nil { - // async handling of sync request. response could be sent - // later, even by the other process + // async handling — emit Processed for tracing chain completeness + if messageHasTracing { + a.sendSpanProcessed(message, gen.TracingKindRequest, "") + if a.PropagatingTrace().ID == message.Tracing.ID { + a.SetPropagatingTrace(savedTracing) + } + } continue } + if messageHasTracing { + a.sendSpanProcessed(message, gen.TracingKindRequest, "") + } + a.SendResponse(message.From, message.Ref, result) + if messageHasTracing && a.PropagatingTrace().ID == message.Tracing.ID { + a.SetPropagatingTrace(savedTracing) + } + case gen.MailboxMessageTypeEvent: if reason := a.behavior.HandleEvent(message.Message.(gen.MessageEvent)); reason != nil { return reason @@ -289,6 +336,11 @@ func (a *Actor) ProcessRun() (rr error) { case gen.MailboxMessageTypeInspect: result := a.behavior.HandleInspect(message.From, message.Message.([]string)...) a.SendResponse(message.From, message.Ref, result) + + case gen.MailboxMessageTypeSpan: + if reason := a.behavior.HandleSpan(message.Message.(gen.TracingSpan)); reason != nil { + return reason + } } } @@ -330,6 +382,11 @@ func (a *Actor) HandleEvent(message gen.MessageEvent) error { return nil } +func (a *Actor) HandleSpan(message gen.TracingSpan) error { + a.Log().Warning("Actor.HandleSpan: unhandled tracing span %#v", message) + return nil +} + func (a *Actor) Terminate(reason error) {} func (a *Actor) HandleMessageName(name gen.Atom, from gen.PID, message any) error { @@ -351,3 +408,26 @@ func (a *Actor) HandleCallAlias(alias gen.Alias, from gen.PID, ref gen.Ref, requ a.Log().Warning("Actor.HandleCallAlias %s: unhandled request from %s", alias, from) return nil, nil } + +func (a *Actor) sendSpanProcessed(message *gen.MailboxMessage, kind gen.TracingKind, errStr string) { + var msgType string + if message.Message != nil { + msgType = reflect.TypeOf(message.Message).String() + } + a.SendTracingSpan(gen.TracingSpan{ + TraceID: message.Tracing.ID, + SpanID: message.Tracing.SpanID, + Point: gen.TracingPointProcessed, + Kind: kind, + Timestamp: time.Now().UnixNano(), + Node: a.Node().Name(), + From: message.From, + To: a.PID(), + Ref: message.Ref, + Behavior: a.BehaviorName(), + Message: msgType, + Error: errStr, + Attributes: a.TracingAttributes(), + }) + a.ClearTracingSpanAttributes() +} diff --git a/act/pool.go b/act/pool.go index 38b2ff40f..6a7ec73da 100644 --- a/act/pool.go +++ b/act/pool.go @@ -4,7 +4,7 @@ import ( "fmt" "reflect" "runtime" - "strings" + "time" "ergo.services/ergo/gen" "ergo.services/ergo/lib" @@ -104,8 +104,7 @@ func (p *Pool) ProcessInit(process gen.Process, args ...any) (rr error) { var ok bool if p.behavior, ok = process.Behavior().(PoolBehavior); ok == false { - unknown := strings.TrimPrefix(reflect.TypeOf(process.Behavior()).String(), "*") - return fmt.Errorf("ProcessInit: not a PoolBehavior %s", unknown) + return fmt.Errorf("ProcessInit: not a PoolBehavior %s", process.BehaviorName()) } p.Process = process p.mailbox = process.Mailbox() @@ -197,7 +196,7 @@ func (p *Pool) ProcessRun() (rr error) { // got new regular message. handle it message = msg.(*gen.MailboxMessage) if message.Type < gen.MailboxMessageTypeExit { - // MailboxMessageTypeRegular, MailboxMessageTypeRequest, MailboxMessageTypeEvent + // MailboxMessageTypeRegular, MailboxMessageTypeRequest, MailboxMessageTypeEvent, MailboxMessageTypeSpan p.forward(message) // it shouldn't be "released" back to the pool message = nil @@ -217,32 +216,57 @@ func (p *Pool) ProcessRun() (rr error) { switch message.Type { case gen.MailboxMessageTypeRegular: + messageHasTracing := message.Tracing.ID != [2]uint64{} + if messageHasTracing { + p.SetPropagatingTrace(message.Tracing) + } + if reason := p.behavior.HandleMessage(message.From, message.Message); reason != nil { + p.sendSpanProcessed(message, gen.TracingKindSend, reason.Error()) return reason } + p.sendSpanProcessed(message, gen.TracingKindSend, "") + + if messageHasTracing { + p.SetPropagatingTrace(gen.Tracing{}) + } case gen.MailboxMessageTypeRequest: + messageHasTracing := message.Tracing.ID != [2]uint64{} + if messageHasTracing { + p.SetPropagatingTrace(message.Tracing) + } + var reason error var result any result, reason = p.behavior.HandleCall(message.From, message.Ref, message.Message) if reason != nil { - // if reason is "normal" and we got response - send it before termination if reason == gen.TerminateReasonNormal && result != nil { + p.sendSpanProcessed(message, gen.TracingKindRequest, "") p.SendResponse(message.From, message.Ref, result) + } else { + p.sendSpanProcessed(message, gen.TracingKindRequest, reason.Error()) } return reason } if result == nil { - // async handling of sync request. response could be sent - // later, even by the other process + p.sendSpanProcessed(message, gen.TracingKindRequest, "") + if messageHasTracing { + p.SetPropagatingTrace(gen.Tracing{}) + } continue } + p.sendSpanProcessed(message, gen.TracingKindRequest, "") p.SendResponse(message.From, message.Ref, result) + if messageHasTracing { + p.SetPropagatingTrace(gen.Tracing{}) + } + case gen.MailboxMessageTypeEvent: if reason := p.behavior.HandleEvent(message.Message.(gen.MessageEvent)); reason != nil { return reason @@ -272,6 +296,7 @@ func (p *Pool) ProcessRun() (rr error) { case gen.MailboxMessageTypeInspect: result := p.behavior.HandleInspect(message.From, message.Message.([]string)...) p.SendResponse(message.From, message.Ref, result) + } } @@ -298,6 +323,31 @@ func (p *Pool) HandleEvent(message gen.MessageEvent) error { p.Log().Warning("Pool.HandleEvent: unhandled event message %#v", message) return nil } +func (p *Pool) sendSpanProcessed(message *gen.MailboxMessage, kind gen.TracingKind, errStr string) { + if message.Tracing.ID == [2]uint64{} { + return + } + var msgType string + if message.Message != nil { + msgType = reflect.TypeOf(message.Message).String() + } + p.SendTracingSpan(gen.TracingSpan{ + TraceID: message.Tracing.ID, + SpanID: message.Tracing.SpanID, + Point: gen.TracingPointProcessed, + Kind: kind, + Timestamp: time.Now().UnixNano(), + Node: p.Node().Name(), + From: message.From, + To: p.PID(), + Ref: message.Ref, + Message: msgType, + Error: errStr, + Attributes: p.TracingAttributes(), + }) + p.ClearTracingSpanAttributes() +} + func (p *Pool) HandleInspect(from gen.PID, item ...string) map[string]string { return map[string]string{ "pool_size": fmt.Sprintf("%d", p.options.PoolSize), diff --git a/act/supervisor.go b/act/supervisor.go index c06de4f80..97f934370 100644 --- a/act/supervisor.go +++ b/act/supervisor.go @@ -5,7 +5,6 @@ import ( "reflect" "runtime" "sort" - "strings" "time" "ergo.services/ergo/gen" @@ -254,8 +253,7 @@ func (s *Supervisor) ProcessInit(process gen.Process, args ...any) (rr error) { var ok bool if s.behavior, ok = process.Behavior().(SupervisorBehavior); ok == false { - unknown := strings.TrimPrefix(reflect.TypeOf(process.Behavior()).String(), "*") - return fmt.Errorf("ProcessInit: not a SupervisorBehavior %s", unknown) + return fmt.Errorf("ProcessInit: not a SupervisorBehavior %s", process.BehaviorName()) } s.Process = process @@ -410,6 +408,11 @@ func (s *Supervisor) ProcessRun() (rr error) { switch message.Type { case gen.MailboxMessageTypeRegular: + messageHasTracing := message.Tracing.ID != [2]uint64{} + if messageHasTracing { + s.SetPropagatingTrace(message.Tracing) + } + var reason error if s.handleChild { switch m := message.Message.(type) { @@ -425,14 +428,33 @@ func (s *Supervisor) ProcessRun() (rr error) { } if reason != nil { + s.sendSpanProcessed(message, gen.TracingKindSend, reason.Error()) action := s.sup.childTerminated(s.Name(), s.PID(), reason) if err := s.handleAction(action); err != nil { return err } + } else { + s.sendSpanProcessed(message, gen.TracingKindSend, "") + } + + if messageHasTracing { + s.SetPropagatingTrace(gen.Tracing{}) } case gen.MailboxMessageTypeRequest: + messageHasTracing := message.Tracing.ID != [2]uint64{} + if messageHasTracing { + s.SetPropagatingTrace(message.Tracing) + } + result, reason := s.behavior.HandleCall(message.From, message.Ref, message.Message) + + if reason != nil { + s.sendSpanProcessed(message, gen.TracingKindRequest, reason.Error()) + } else { + s.sendSpanProcessed(message, gen.TracingKindRequest, "") + } + if result != nil { s.SendResponse(message.From, message.Ref, result) } @@ -443,6 +465,10 @@ func (s *Supervisor) ProcessRun() (rr error) { } } + if messageHasTracing { + s.SetPropagatingTrace(gen.Tracing{}) + } + case gen.MailboxMessageTypeEvent: if reason := s.behavior.HandleEvent(message.Message.(gen.MessageEvent)); reason != nil { return reason @@ -497,6 +523,9 @@ func (s *Supervisor) ProcessRun() (rr error) { case gen.MailboxMessageTypeInspect: result := s.behavior.HandleInspect(message.From, message.Message.([]string)...) s.SendResponse(message.From, message.Ref, result) + + case gen.MailboxMessageTypeSpan: + panic("supervisor process can not be a tracing exporter") } } } @@ -752,3 +781,29 @@ func sortSupChild(c []supChild) []SupervisorChild { } return children } + +func (s *Supervisor) sendSpanProcessed(message *gen.MailboxMessage, kind gen.TracingKind, errStr string) { + if message.Tracing.ID == [2]uint64{} { + return + } + var msgType string + if message.Message != nil { + msgType = reflect.TypeOf(message.Message).String() + } + s.SendTracingSpan(gen.TracingSpan{ + TraceID: message.Tracing.ID, + SpanID: message.Tracing.SpanID, + Point: gen.TracingPointProcessed, + Kind: kind, + Timestamp: time.Now().UnixNano(), + Node: s.Node().Name(), + From: message.From, + To: s.PID(), + Ref: message.Ref, + Behavior: s.BehaviorName(), + Message: msgType, + Error: errStr, + Attributes: s.TracingAttributes(), + }) + s.ClearTracingSpanAttributes() +} diff --git a/act/web_worker.go b/act/web_worker.go index 11637029d..b92b9dd26 100644 --- a/act/web_worker.go +++ b/act/web_worker.go @@ -5,7 +5,7 @@ import ( "net/http" "reflect" "runtime" - "strings" + "time" "ergo.services/ergo/gen" "ergo.services/ergo/lib" @@ -67,8 +67,7 @@ type WebWorker struct { func (w *WebWorker) ProcessInit(process gen.Process, args ...any) (rr error) { var ok bool if w.behavior, ok = process.Behavior().(WebWorkerBehavior); ok == false { - unknown := strings.TrimPrefix(reflect.TypeOf(process.Behavior()).String(), "*") - return fmt.Errorf("ProcessInit: not a WebWorkerBehavior %s", unknown) + return fmt.Errorf("ProcessInit: not a WebWorkerBehavior %s", process.BehaviorName()) } w.Process = process w.mailbox = process.Mailbox() @@ -89,6 +88,7 @@ func (w *WebWorker) ProcessInit(process gen.Process, args ...any) (rr error) { func (w *WebWorker) ProcessRun() (rr error) { var message *gen.MailboxMessage + var savedTracing gen.Tracing if lib.Recover() { defer func() { @@ -145,6 +145,12 @@ func (w *WebWorker) ProcessRun() (rr error) { switch message.Type { case gen.MailboxMessageTypeRegular: + messageHasTracing := message.Tracing.ID != [2]uint64{} + if messageHasTracing { + savedTracing = w.PropagatingTrace() + w.SetPropagatingTrace(message.Tracing) + } + if r, ok := message.Message.(meta.MessageWebRequest); ok { var reason error switch r.Request.Method { @@ -169,36 +175,61 @@ func (w *WebWorker) ProcessRun() (rr error) { } r.Done() if reason != nil { + w.sendSpanProcessed(message, gen.TracingKindSend, r.Request.Method+" "+r.Request.RequestURI, reason.Error()) return reason } + w.sendSpanProcessed(message, gen.TracingKindSend, r.Request.Method+" "+r.Request.RequestURI, "") + if messageHasTracing && w.PropagatingTrace().ID == message.Tracing.ID { + w.SetPropagatingTrace(savedTracing) + } continue } if reason := w.behavior.HandleMessage(message.From, message.Message); reason != nil { + w.sendSpanProcessed(message, gen.TracingKindSend, reflectMsgType(message.Message), reason.Error()) return reason } + w.sendSpanProcessed(message, gen.TracingKindSend, reflectMsgType(message.Message), "") + if messageHasTracing && w.PropagatingTrace().ID == message.Tracing.ID { + w.SetPropagatingTrace(savedTracing) + } case gen.MailboxMessageTypeRequest: + messageHasTracing := message.Tracing.ID != [2]uint64{} + if messageHasTracing { + savedTracing = w.PropagatingTrace() + w.SetPropagatingTrace(message.Tracing) + } + var reason error var result any result, reason = w.behavior.HandleCall(message.From, message.Ref, message.Message) if reason != nil { - // if reason is "normal" and we got response - send it before termination if reason == gen.TerminateReasonNormal && result != nil { + w.sendSpanProcessed(message, gen.TracingKindRequest, reflectMsgType(message.Message), "") w.SendResponse(message.From, message.Ref, result) + return reason } + w.sendSpanProcessed(message, gen.TracingKindRequest, reflectMsgType(message.Message), reason.Error()) return reason } if result == nil { - // async handling of sync request. response could be sent - // later, even by the other process + w.sendSpanProcessed(message, gen.TracingKindRequest, reflectMsgType(message.Message), "") + if messageHasTracing && w.PropagatingTrace().ID == message.Tracing.ID { + w.SetPropagatingTrace(savedTracing) + } continue } + w.sendSpanProcessed(message, gen.TracingKindRequest, reflectMsgType(message.Message), "") + w.SendResponse(message.From, message.Ref, result) + if messageHasTracing && w.PropagatingTrace().ID == message.Tracing.ID { + w.SetPropagatingTrace(savedTracing) + } case gen.MailboxMessageTypeEvent: if reason := w.behavior.HandleEvent(message.Message.(gen.MessageEvent)); reason != nil { @@ -229,6 +260,9 @@ func (w *WebWorker) ProcessRun() (rr error) { case gen.MailboxMessageTypeInspect: result := w.behavior.HandleInspect(message.From, message.Message.([]string)...) w.SendResponse(message.From, message.Ref, result) + + case gen.MailboxMessageTypeSpan: + panic("web worker process can not be a tracing exporter") } } @@ -295,3 +329,32 @@ func (w *WebWorker) HandleOptions(from gen.PID, writer http.ResponseWriter, requ http.Error(writer, "unhandled request", http.StatusNotImplemented) return nil } + +func reflectMsgType(msg any) string { + if msg == nil { + return "" + } + return reflect.TypeOf(msg).String() +} + +func (w *WebWorker) sendSpanProcessed(message *gen.MailboxMessage, kind gen.TracingKind, msgType string, errStr string) { + if message.Tracing.ID == [2]uint64{} { + return + } + w.SendTracingSpan(gen.TracingSpan{ + TraceID: message.Tracing.ID, + SpanID: message.Tracing.SpanID, + Point: gen.TracingPointProcessed, + Kind: kind, + Timestamp: time.Now().UnixNano(), + Node: w.Node().Name(), + From: message.From, + To: w.PID(), + Ref: message.Ref, + Behavior: w.BehaviorName(), + Message: msgType, + Error: errStr, + Attributes: w.TracingAttributes(), + }) + w.ClearTracingSpanAttributes() +} diff --git a/app/system/app.go b/app/system/app.go index 9d712d55f..7a32cfa49 100644 --- a/app/system/app.go +++ b/app/system/app.go @@ -1,6 +1,9 @@ package system import ( + "fmt" + + "ergo.services/ergo/app/system/inspect" "ergo.services/ergo/gen" ) @@ -15,6 +18,9 @@ type systemApp struct { } func (sa *systemApp) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { + if err := inspect.RegisterTypes(node.Network()); err != nil { + return gen.ApplicationSpec{}, fmt.Errorf("inspect types: %w", err) + } return gen.ApplicationSpec{ Name: Name, Description: "System Application", diff --git a/app/system/inspect/application_list.go b/app/system/inspect/application_list.go index 42cb5967a..cdcb91b08 100644 --- a/app/system/inspect/application_list.go +++ b/app/system/inspect/application_list.go @@ -16,21 +16,36 @@ type application_list struct { token gen.Ref generating bool + loopID uint64 event gen.Atom } func (ial *application_list) Init(args ...any) error { ial.Log().SetLogger("default") ial.Log().Debug("application list inspector started") - // RegisterEvent is not allowed here - ial.Send(ial.PID(), register{}) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, // keep the last event + } + evname := gen.Atom(fmt.Sprintf("%s", inspectApplicationList)) + token, err := ial.RegisterEvent(evname, eopts) + if err != nil { + ial.Log().Error("unable to register event: %s", err) + return err + } + ial.Log().Info("registered event %s", evname) + ial.event = evname + ial.token = token + ial.SendAfter(ial.PID(), shutdown{}, inspectApplicationListIdlePeriod) + return nil } func (ial *application_list) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { case generate: - if ial.generating == false { + if m.id != ial.loopID || ial.generating == false { ial.Log().Debug("generating canceled") break // cancelled } @@ -60,7 +75,7 @@ func (ial *application_list) HandleMessage(from gen.PID, message any) error { return gen.TerminateReasonNormal } - ial.SendAfter(ial.PID(), generate{}, inspectApplicationListPeriod) + ial.SendAfter(ial.PID(), generate{id: ial.loopID}, inspectApplicationListPeriod) case requestInspect: response := ResponseInspectApplicationList{ @@ -72,23 +87,6 @@ func (ial *application_list) HandleMessage(from gen.PID, message any) error { ial.SendResponse(m.pid, m.ref, response) ial.Log().Debug("sent response for the inspect application list request to: %s", m.pid) - case register: - eopts := gen.EventOptions{ - Notify: true, - Buffer: 1, // keep the last event - } - evname := gen.Atom(fmt.Sprintf("%s", inspectApplicationList)) - token, err := ial.RegisterEvent(evname, eopts) - if err != nil { - ial.Log().Error("unable to register event: %s", err) - return err - } - ial.Log().Info("registered event %s", evname) - ial.event = evname - - ial.token = token - ial.SendAfter(ial.PID(), shutdown{}, inspectApplicationListIdlePeriod) - case shutdown: if ial.generating { ial.Log().Debug("ignore shutdown. generating is active") @@ -98,7 +96,8 @@ func (ial *application_list) HandleMessage(from gen.PID, message any) error { case gen.MessageEventStart: // got first subscriber ial.Log().Debug("got first subscriber. start generating events...") - ial.Send(ial.PID(), generate{}) + ial.loopID++ + ial.Send(ial.PID(), generate{id: ial.loopID}) ial.generating = true case gen.MessageEventStop: // no subscribers diff --git a/app/system/inspect/application_tree.go b/app/system/inspect/application_tree.go index 68a032c37..71ea803a0 100644 --- a/app/system/inspect/application_tree.go +++ b/app/system/inspect/application_tree.go @@ -18,6 +18,7 @@ type application_tree struct { application gen.Atom limit int generating bool + loopID uint64 event gen.Atom } @@ -26,15 +27,30 @@ func (iat *application_tree) Init(args ...any) error { iat.limit = args[1].(int) iat.Log().SetLogger("default") iat.Log().Debug("application tree inspector started for %s with limit %d", iat.application, iat.limit) - // RegisterEvent is not allowed here - iat.Send(iat.PID(), register{}) + iat.SetCompression(true) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, // keep the last event + } + evname := gen.Atom(fmt.Sprintf("%s_%s_%d", inspectApplicationTree, iat.application, iat.limit)) + token, err := iat.RegisterEvent(evname, eopts) + if err != nil { + iat.Log().Error("unable to register event: %s", err) + return err + } + iat.Log().Info("registered event %s", evname) + iat.event = evname + iat.token = token + iat.SendAfter(iat.PID(), shutdown{}, inspectApplicationTreeIdlePeriod) + return nil } func (iat *application_tree) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { case generate: - if iat.generating == false { + if m.id != iat.loopID || iat.generating == false { iat.Log().Debug("generating canceled") break // cancelled } @@ -56,7 +72,7 @@ func (iat *application_tree) HandleMessage(from gen.PID, message any) error { return gen.TerminateReasonNormal } - iat.SendAfter(iat.PID(), generate{}, inspectApplicationTreePeriod) + iat.SendAfter(iat.PID(), generate{id: iat.loopID}, inspectApplicationTreePeriod) case requestInspect: response := ResponseInspectApplicationTree{ @@ -68,23 +84,6 @@ func (iat *application_tree) HandleMessage(from gen.PID, message any) error { iat.SendResponse(m.pid, m.ref, response) iat.Log().Debug("sent response for the inspect application tree request to: %s", m.pid) - case register: - eopts := gen.EventOptions{ - Notify: true, - Buffer: 1, // keep the last event - } - evname := gen.Atom(fmt.Sprintf("%s_%s_%d", inspectApplicationTree, iat.application, iat.limit)) - token, err := iat.RegisterEvent(evname, eopts) - if err != nil { - iat.Log().Error("unable to register event: %s", err) - return err - } - iat.Log().Info("registered event %s", evname) - iat.event = evname - - iat.token = token - iat.SendAfter(iat.PID(), shutdown{}, inspectApplicationTreeIdlePeriod) - case shutdown: if iat.generating { iat.Log().Debug("ignore shutdown. generating is active") @@ -94,7 +93,8 @@ func (iat *application_tree) HandleMessage(from gen.PID, message any) error { case gen.MessageEventStart: // got first subscriber iat.Log().Debug("got first subscriber. start generating events...") - iat.Send(iat.PID(), generate{}) + iat.loopID++ + iat.Send(iat.PID(), generate{id: iat.loopID}) iat.generating = true case gen.MessageEventStop: // no subscribers diff --git a/app/system/inspect/connection.go b/app/system/inspect/connection.go index 85393fdd1..60e7777fc 100644 --- a/app/system/inspect/connection.go +++ b/app/system/inspect/connection.go @@ -17,6 +17,7 @@ type connection struct { event gen.Atom generating bool + loopID uint64 remote gen.Atom } @@ -24,15 +25,29 @@ func (ic *connection) Init(args ...any) error { ic.remote = args[0].(gen.Atom) ic.Log().SetLogger("default") ic.Log().Debug("connection inspector started") - // RegisterEvent is not allowed here - ic.Send(ic.PID(), register{}) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, // keep the last event + } + evname := gen.Atom(fmt.Sprintf("%s_%s", inspectConnection, ic.remote)) + token, err := ic.RegisterEvent(evname, eopts) + if err != nil { + ic.Log().Error("unable to register connection event: %s", err) + return err + } + ic.Log().Info("registered event %s", inspectNetwork) + ic.event = evname + ic.token = token + ic.SendAfter(ic.PID(), shutdown{}, inspectNetworkIdlePeriod) + return nil } func (ic *connection) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { case generate: - if ic.generating == false { + if m.id != ic.loopID || ic.generating == false { ic.Log().Debug("generating canceled") break // cancelled } @@ -60,7 +75,7 @@ func (ic *connection) HandleMessage(from gen.PID, message any) error { if ev.Disconnected { return gen.TerminateReasonNormal } - ic.SendAfter(ic.PID(), generate{}, inspectNetworkPeriod) + ic.SendAfter(ic.PID(), generate{id: ic.loopID}, inspectNetworkPeriod) case requestInspect: response := ResponseInspectConnection{ @@ -80,23 +95,6 @@ func (ic *connection) HandleMessage(from gen.PID, message any) error { return gen.TerminateReasonNormal } - case register: - eopts := gen.EventOptions{ - Notify: true, - Buffer: 1, // keep the last event - } - evname := gen.Atom(fmt.Sprintf("%s_%s", inspectConnection, ic.remote)) - token, err := ic.RegisterEvent(evname, eopts) - if err != nil { - ic.Log().Error("unable to register connection event: %s", err) - return err - } - ic.Log().Info("registered event %s", inspectNetwork) - ic.event = evname - - ic.token = token - ic.SendAfter(ic.PID(), shutdown{}, inspectNetworkIdlePeriod) - case shutdown: if ic.generating { ic.Log().Debug("ignore shutdown. generating is active") @@ -106,7 +104,8 @@ func (ic *connection) HandleMessage(from gen.PID, message any) error { case gen.MessageEventStart: // got first subscriber ic.Log().Debug("got first subscriber. start generating events...") - ic.Send(ic.PID(), generate{}) + ic.loopID++ + ic.Send(ic.PID(), generate{id: ic.loopID}) ic.generating = true case gen.MessageEventStop: // no subscribers diff --git a/app/system/inspect/connection_list.go b/app/system/inspect/connection_list.go new file mode 100644 index 000000000..63b455384 --- /dev/null +++ b/app/system/inspect/connection_list.go @@ -0,0 +1,140 @@ +package inspect + +import ( + "fmt" + "slices" + "strings" + + "ergo.services/ergo/act" + "ergo.services/ergo/gen" +) + +func factory_connection_list() gen.ProcessBehavior { + return &connection_list{} +} + +type connection_list struct { + act.Actor + token gen.Ref + + name string + limit int + hash string + generating bool + loopID uint64 + event gen.Atom +} + +func (icl *connection_list) Init(args ...any) error { + icl.name = args[0].(string) + icl.limit = args[1].(int) + icl.hash = args[2].(string) + + icl.Log().SetLogger("default") + icl.Log().Debug("connection list inspector started. name=%q limit=%d", icl.name, icl.limit) + icl.SetCompression(true) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, + } + icl.event = gen.Atom(fmt.Sprintf("%s_%s", inspectConnectionList, icl.hash)) + token, err := icl.RegisterEvent(icl.event, eopts) + if err != nil { + icl.Log().Error("unable to register event: %s", err) + return err + } + icl.Log().Info("registered event %s", icl.event) + icl.token = token + icl.SendAfter(icl.PID(), shutdown{}, inspectConnectionListIdlePeriod) + + return nil +} + +func (icl *connection_list) HandleMessage(from gen.PID, message any) error { + switch m := message.(type) { + case generate: + if m.id != icl.loopID || icl.generating == false { + break + } + + networkInfo, err := icl.Node().Network().Info() + if err != nil { + return err + } + + nameLower := strings.ToLower(icl.name) + var connections []gen.RemoteNodeInfo + + // sort node names for stable output + slices.Sort(networkInfo.Nodes) + + for _, n := range networkInfo.Nodes { + if nameLower != "" { + if strings.Contains(strings.ToLower(string(n)), nameLower) == false { + continue + } + } + + remote, rerr := icl.Node().Network().Node(n) + if rerr != nil { + continue + } + + connections = append(connections, remote.Info()) + + if icl.limit > 0 && len(connections) >= icl.limit { + break + } + } + + ev := MessageInspectConnectionList{ + Node: icl.Node().Name(), + Connections: connections, + } + + if err := icl.SendEvent(icl.event, icl.token, ev); err != nil { + icl.Log().Error("unable to send event %q: %s", icl.event, err) + return gen.TerminateReasonNormal + } + + icl.SendAfter(icl.PID(), generate{id: icl.loopID}, inspectConnectionListPeriod) + + case requestInspect: + response := ResponseInspectConnectionList{ + Event: gen.Event{ + Name: icl.event, + Node: icl.Node().Name(), + }, + } + icl.SendResponse(m.pid, m.ref, response) + + case shutdown: + if icl.generating { + break + } + return gen.TerminateReasonNormal + + case gen.MessageEventStart: + icl.Log().Debug("got first subscriber. start generating events...") + icl.loopID++ + icl.Send(icl.PID(), generate{id: icl.loopID}) + icl.generating = true + + case gen.MessageEventStop: + icl.Log().Debug("no subscribers. stop generating") + if icl.generating { + icl.generating = false + icl.SendAfter(icl.PID(), shutdown{}, inspectConnectionListIdlePeriod) + } + + default: + icl.Log().Error("unknown message (ignored) %#v", message) + } + + return nil +} + +func (icl *connection_list) Terminate(reason error) { + icl.Log().Debug("connection list inspector terminated: %s", reason) +} diff --git a/app/system/inspect/event_list.go b/app/system/inspect/event_list.go new file mode 100644 index 000000000..76da4952a --- /dev/null +++ b/app/system/inspect/event_list.go @@ -0,0 +1,157 @@ +package inspect + +import ( + "fmt" + "strings" + + "ergo.services/ergo/act" + "ergo.services/ergo/gen" +) + +func factory_event_list() gen.ProcessBehavior { + return &event_list{} +} + +type event_list struct { + act.Actor + token gen.Ref + + timestamp int64 + name string + notify int + buffered int + open int + minSubscribers int64 + limit int + hash string + + generating bool + loopID uint64 + event gen.Atom +} + +func (iel *event_list) Init(args ...any) error { + iel.timestamp = args[0].(int64) + iel.name = args[1].(string) + iel.notify = args[2].(int) + iel.buffered = args[3].(int) + iel.open = args[4].(int) + iel.minSubscribers = args[5].(int64) + iel.limit = args[6].(int) + iel.hash = args[7].(string) + + iel.Log().SetLogger("default") + iel.Log().Debug("event list inspector started. timestamp=%d name=%q notify=%d buffered=%d open=%d minSubs=%d limit=%d", + iel.timestamp, iel.name, iel.notify, iel.buffered, iel.open, iel.minSubscribers, iel.limit) + iel.SetCompression(true) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, + } + iel.event = gen.Atom(fmt.Sprintf("%s_%s", inspectEventList, iel.hash)) + token, err := iel.RegisterEvent(iel.event, eopts) + if err != nil { + iel.Log().Error("unable to register event: %s", err) + return err + } + iel.Log().Info("registered event %s", iel.event) + iel.token = token + iel.SendAfter(iel.PID(), shutdown{}, inspectEventListIdlePeriod) + + return nil +} + +func (iel *event_list) HandleMessage(from gen.PID, message any) error { + switch m := message.(type) { + case generate: + if m.id != iel.loopID || iel.generating == false { + iel.Log().Debug("generating canceled") + break + } + iel.Log().Debug("generating event") + + events, _ := iel.Node().EventListInfo(iel.timestamp, iel.limit, iel.filterEvent) + + ev := MessageInspectEventList{ + Node: iel.Node().Name(), + Events: events, + } + + if err := iel.SendEvent(iel.event, iel.token, ev); err != nil { + iel.Log().Error("unable to send event %q: %s", iel.event, err) + return gen.TerminateReasonNormal + } + + iel.SendAfter(iel.PID(), generate{id: iel.loopID}, inspectEventListPeriod) + + case requestInspect: + response := ResponseInspectEventList{ + Event: gen.Event{ + Name: iel.event, + Node: iel.Node().Name(), + }, + } + iel.SendResponse(m.pid, m.ref, response) + iel.Log().Debug("sent response for the inspect event list request to: %s", m.pid) + + case shutdown: + if iel.generating { + iel.Log().Debug("ignore shutdown. generating is active") + break + } + return gen.TerminateReasonNormal + + case gen.MessageEventStart: + iel.Log().Debug("got first subscriber. start generating events...") + iel.loopID++ + iel.Send(iel.PID(), generate{id: iel.loopID}) + iel.generating = true + + case gen.MessageEventStop: + iel.Log().Debug("no subscribers. stop generating") + if iel.generating { + iel.generating = false + iel.SendAfter(iel.PID(), shutdown{}, inspectEventListIdlePeriod) + } + + default: + iel.Log().Error("unknown message (ignored) %#v", message) + } + + return nil +} + +func (iel *event_list) filterEvent(info gen.EventInfo) bool { + if iel.name != "" { + if strings.Contains(strings.ToLower(string(info.Event.Name)), strings.ToLower(iel.name)) == false { + return false + } + } + if iel.notify == 1 && info.Notify == false { + return false + } + if iel.notify == -1 && info.Notify == true { + return false + } + if iel.buffered == 1 && info.BufferSize == 0 { + return false + } + if iel.buffered == -1 && info.BufferSize > 0 { + return false + } + if iel.open == 1 && info.Open == false { + return false + } + if iel.open == -1 && info.Open == true { + return false + } + if iel.minSubscribers > 0 && info.Subscribers < iel.minSubscribers { + return false + } + return true +} + +func (iel *event_list) Terminate(reason error) { + iel.Log().Debug("event list inspector terminated: %s", reason) +} diff --git a/app/system/inspect/goroutines.go b/app/system/inspect/goroutines.go new file mode 100644 index 000000000..f427204f2 --- /dev/null +++ b/app/system/inspect/goroutines.go @@ -0,0 +1,169 @@ +package inspect + +import ( + "runtime" + "sort" + "strconv" + "strings" +) + +func captureGoroutines(req RequestDoGoroutines) ResponseDoGoroutines { + buf := make([]byte, 1<<20) + for { + n := runtime.Stack(buf, true) + if n < len(buf) { + buf = buf[:n] + break + } + buf = make([]byte, len(buf)*2) + } + + blocks := strings.Split(string(buf), "\n\n") + + stackFilter := strings.ToLower(req.Stack) + stateFilter := strings.ToLower(req.State) + + type parsed struct { + id int + state string + waitSec int64 + frames string + top string + bottom string + full string + } + + var matched []parsed + total := 0 + + for _, block := range blocks { + block = strings.TrimSpace(block) + if strings.HasPrefix(block, "goroutine ") == false { + continue + } + total++ + + id, state, waitSec := parseHeader(block) + + if stateFilter != "" && strings.ToLower(state) != stateFilter { + continue + } + if req.MinWait > 0 && waitSec < req.MinWait { + continue + } + if stackFilter != "" && strings.Contains(strings.ToLower(block), stackFilter) == false { + continue + } + + funcs := parseFuncLines(block) + top := "" + bottom := "" + if len(funcs) > 0 { + top = funcs[0] + bottom = funcs[len(funcs)-1] + } + + matched = append(matched, parsed{ + id: id, + state: state, + waitSec: waitSec, + frames: state + "|" + strings.Join(funcs, "|"), + top: top, + bottom: bottom, + full: block, + }) + } + + // group by identical stack + groupMap := make(map[string]*GoroutineGroup) + var order []string + + for _, p := range matched { + g, ok := groupMap[p.frames] + if ok == false { + g = &GoroutineGroup{State: p.state, WaitSec: p.waitSec, Current: p.top, Origin: p.bottom, Stack: p.full} + groupMap[p.frames] = g + order = append(order, p.frames) + } + g.Count++ + g.IDs = append(g.IDs, p.id) + } + + groups := make([]GoroutineGroup, 0, len(order)) + for _, key := range order { + groups = append(groups, *groupMap[key]) + } + + sort.Slice(groups, func(i, j int) bool { + return groups[i].Count > groups[j].Count + }) + + return ResponseDoGoroutines{ + Groups: groups, + Total: total, + Filtered: len(matched), + } +} + +func parseHeader(block string) (id int, state string, waitSec int64) { + lines := strings.SplitN(block, "\n", 2) + header := lines[0] + + rest := header[len("goroutine "):] + spaceIdx := strings.IndexByte(rest, ' ') + if spaceIdx < 0 { + return + } + id, _ = strconv.Atoi(rest[:spaceIdx]) + + open := strings.IndexByte(rest, '[') + close := strings.IndexByte(rest, ']') + if open < 0 || close <= open { + return + } + + stateStr := rest[open+1 : close] + parts := strings.SplitN(stateStr, ",", 2) + state = strings.TrimSpace(parts[0]) + + if len(parts) > 1 { + waitSec = parseWaitDuration(strings.TrimSpace(parts[1])) + } + return +} + +func parseWaitDuration(s string) int64 { + // "5 minutes", "847 minutes", "2 hours" + parts := strings.Fields(s) + if len(parts) < 2 { + return 0 + } + n, err := strconv.ParseInt(parts[0], 10, 64) + if err != nil { + return 0 + } + unit := strings.TrimSuffix(parts[1], "s") // "minute" from "minutes" + switch unit { + case "minute": + return n * 60 + case "hour": + return n * 3600 + case "second": + return n + } + return 0 +} + +func parseFuncLines(block string) []string { + lines := strings.Split(block, "\n") + var funcs []string + for i := 1; i < len(lines); i++ { + if len(lines[i]) > 0 && lines[i][0] != '\t' { + f := strings.TrimSpace(lines[i]) + if f != "" { + funcs = append(funcs, f) + } + } + } + return funcs +} diff --git a/app/system/inspect/heap.go b/app/system/inspect/heap.go new file mode 100644 index 000000000..818ce4ef2 --- /dev/null +++ b/app/system/inspect/heap.go @@ -0,0 +1,62 @@ +package inspect + +import ( + "runtime" + "sort" +) + +func captureHeapProfile(req RequestDoHeapProfile) ResponseDoHeapProfile { + // force up-to-date stats + runtime.GC() + + var p []runtime.MemProfileRecord + n, _ := runtime.MemProfile(nil, true) + p = make([]runtime.MemProfileRecord, n) + runtime.MemProfile(p, true) + + var records []HeapRecord + var totalInuse, totalAlloc, totalObjects int64 + + for _, r := range p { + inuse := r.InUseBytes() + if req.MinBytes > 0 && inuse < req.MinBytes { + continue + } + + frames := runtime.CallersFrames(r.Stack()) + var stack []string + for { + frame, more := frames.Next() + if frame.Function != "" { + stack = append(stack, frame.Function) + } + if more == false { + break + } + } + + rec := HeapRecord{ + InuseBytes: inuse, + InuseObjects: r.InUseObjects(), + AllocBytes: r.AllocBytes, + AllocObjects: r.AllocObjects, + Stack: stack, + } + records = append(records, rec) + + totalInuse += inuse + totalAlloc += r.AllocBytes + totalObjects += r.InUseObjects() + } + + sort.Slice(records, func(i, j int) bool { + return records[i].InuseBytes > records[j].InuseBytes + }) + + return ResponseDoHeapProfile{ + Records: records, + TotalInuse: totalInuse, + TotalAlloc: totalAlloc, + TotalObjects: totalObjects, + } +} diff --git a/app/system/inspect/heap_inspector.go b/app/system/inspect/heap_inspector.go new file mode 100644 index 000000000..f9ecc2941 --- /dev/null +++ b/app/system/inspect/heap_inspector.go @@ -0,0 +1,201 @@ +package inspect + +import ( + "fmt" + "runtime" + "sort" + "strings" + + "ergo.services/ergo/act" + "ergo.services/ergo/gen" +) + +func factory_heap() gen.ProcessBehavior { + return &heap_inspector{} +} + +type heap_inspector struct { + act.Actor + token gen.Ref + + limit int + name string + + generating bool + loopID uint64 + event gen.Atom +} + +func (h *heap_inspector) Init(args ...any) error { + h.limit = args[0].(int) + h.name = args[1].(string) + + h.Log().SetLogger("default") + h.SetCompression(true) + + eopts := gen.EventOptions{Notify: true, Buffer: 1} + hash := filterHash(h.name, "", "", "", 0, h.limit) + h.event = gen.Atom(fmt.Sprintf("%s_%s", inspectHeap, hash)) + token, err := h.RegisterEvent(h.event, eopts) + if err != nil { + h.Log().Error("unable to register event: %s", err) + return err + } + h.token = token + h.SendAfter(h.PID(), shutdown{}, inspectHeapIdlePeriod) + + return nil +} + +func (h *heap_inspector) HandleMessage(from gen.PID, message any) error { + switch m := message.(type) { + case generate: + if m.id != h.loopID || h.generating == false { + break + } + + records, totalAlloc, totalFree := h.captureTop() + + var totalInuse, totalObjects int64 + for _, r := range records { + totalInuse += r.InuseBytes + totalObjects += r.InuseObjects + } + + var ms runtime.MemStats + runtime.ReadMemStats(&ms) + + ev := MessageInspectHeap{ + Node: h.Node().Name(), + Records: records, + TotalInuse: totalInuse, + TotalObjects: totalObjects, + TotalAlloc: totalAlloc, + TotalFree: totalFree, + GCCPUFraction: ms.GCCPUFraction, + } + + if err := h.SendEvent(h.event, h.token, ev); err != nil { + h.Log().Error("unable to send event %q: %s", h.event, err) + return gen.TerminateReasonNormal + } + + h.SendAfter(h.PID(), generate{id: h.loopID}, inspectHeapPeriod) + + case requestInspect: + response := ResponseInspectHeap{ + Event: gen.Event{ + Name: h.event, + Node: h.Node().Name(), + }, + } + h.SendResponse(m.pid, m.ref, response) + + case shutdown: + if h.generating { + break + } + return gen.TerminateReasonNormal + + case gen.MessageEventStart: + h.loopID++ + h.Send(h.PID(), generate{id: h.loopID}) + h.generating = true + + case gen.MessageEventStop: + if h.generating { + h.generating = false + h.SendAfter(h.PID(), shutdown{}, inspectHeapIdlePeriod) + } + } + + return nil +} + +func (h *heap_inspector) Terminate(reason error) {} + +func (h *heap_inspector) captureTop() ([]HeapRecord, int64, int64) { + var p []runtime.MemProfileRecord + n, _ := runtime.MemProfile(nil, false) // false = include freed records + p = make([]runtime.MemProfileRecord, n) + runtime.MemProfile(p, false) + + nameLower := strings.ToLower(h.name) + + type entry struct { + inuse int64 + objects int64 + alloc int64 + allocN int64 + freeN int64 + stack []string + } + + var entries []entry + var totalAlloc, totalFree int64 + + for _, r := range p { + totalAlloc += r.AllocObjects + totalFree += r.FreeObjects + + inuse := r.InUseBytes() + if inuse <= 0 { + continue + } + + frames := runtime.CallersFrames(r.Stack()) + var stack []string + for { + frame, more := frames.Next() + if frame.Function != "" { + stack = append(stack, frame.Function) + } + if more == false { + break + } + } + + if nameLower != "" { + matched := false + for _, f := range stack { + if strings.Contains(strings.ToLower(f), nameLower) { + matched = true + break + } + } + if matched == false { + continue + } + } + + entries = append(entries, entry{ + inuse: inuse, + objects: r.InUseObjects(), + alloc: r.AllocBytes, + allocN: r.AllocObjects, + freeN: r.FreeObjects, + stack: stack, + }) + } + + sort.Slice(entries, func(i, j int) bool { + return entries[i].inuse > entries[j].inuse + }) + + if len(entries) > h.limit { + entries = entries[:h.limit] + } + + records := make([]HeapRecord, len(entries)) + for i, e := range entries { + records[i] = HeapRecord{ + InuseBytes: e.inuse, + InuseObjects: e.objects, + AllocBytes: e.alloc, + AllocObjects: e.allocN, + FreeObjects: e.freeN, + Stack: e.stack, + } + } + return records, totalAlloc, totalFree +} diff --git a/app/system/inspect/inspect.go b/app/system/inspect/inspect.go index f85eb7b08..30d1eb6c9 100644 --- a/app/system/inspect/inspect.go +++ b/app/system/inspect/inspect.go @@ -48,6 +48,8 @@ const ( inspectLog = "inspect_log" inspectLogIdlePeriod = 10 * time.Second + inspectTracing = "inspect_tracing" + inspectApplicationList = "inspect_application_list" inspectApplicationListPeriod = time.Second inspectApplicationListIdlePeriod = 5 * time.Second @@ -55,6 +57,22 @@ const ( inspectApplicationTree = "inspect_application_tree" inspectApplicationTreePeriod = time.Second inspectApplicationTreeIdlePeriod = 5 * time.Second + + inspectEventList = "inspect_event_list" + inspectEventListPeriod = time.Second + inspectEventListIdlePeriod = 5 * time.Second + + inspectProcessRange = "inspect_process_range" + inspectProcessRangePeriod = time.Second + inspectProcessRangeIdlePeriod = 5 * time.Second + + inspectConnectionList = "inspect_connection_list" + inspectConnectionListPeriod = time.Second + inspectConnectionListIdlePeriod = 5 * time.Second + + inspectHeap = "inspect_heap" + inspectHeapPeriod = time.Second + inspectHeapIdlePeriod = 5 * time.Second ) var ( @@ -68,9 +86,67 @@ var ( ) func Factory() gen.ProcessBehavior { + return &inspectPool{} +} + +// RegisterTypes registers all inspector wire-format types with the given network. +// Called by the system application during Load, after the network stack is up. +func RegisterTypes(network gen.Network) error { + types := []any{ + RequestInspectNode{}, ResponseInspectNode{}, MessageInspectNode{}, + RequestInspectNetwork{}, ResponseInspectNetwork{}, MessageInspectNetwork{}, + RequestInspectConnection{}, ResponseInspectConnection{}, MessageInspectConnection{}, + RequestInspectConnectionList{}, ResponseInspectConnectionList{}, MessageInspectConnectionList{}, + RequestInspectProcessList{}, ResponseInspectProcessList{}, MessageInspectProcessList{}, + RequestInspectProcessRange{}, ResponseInspectProcessRange{}, + RequestInspectEventList{}, ResponseInspectEventList{}, MessageInspectEventList{}, + RequestInspectLog{}, ResponseInspectLog{}, InspectLogEntry{}, MessageInspectLog{}, + RequestInspectProcess{}, ResponseInspectProcess{}, MessageInspectProcess{}, + RequestInspectProcessState{}, ResponseInspectProcessState{}, MessageInspectProcessState{}, + RequestInspectMeta{}, ResponseInspectMeta{}, MessageInspectMeta{}, + RequestInspectMetaState{}, ResponseInspectMetaState{}, MessageInspectMetaState{}, + RequestInspectApplicationList{}, ResponseInspectApplicationList{}, MessageInspectApplicationList{}, + RequestInspectApplicationTree{}, ResponseInspectApplicationTree{}, MessageInspectApplicationTree{}, + RequestInspectHeap{}, ResponseInspectHeap{}, MessageInspectHeap{}, + RequestInspectTracing{}, ResponseInspectTracing{}, MessageInspectTracing{}, + + RequestDoSend{}, ResponseDoSend{}, + RequestDoSendMeta{}, ResponseDoSendMeta{}, + RequestDoSendExit{}, ResponseDoSendExit{}, + RequestDoSendExitMeta{}, ResponseDoSendExitMeta{}, + RequestDoKill{}, ResponseDoKill{}, + RequestDoSetLogLevel{}, RequestDoSetProcessLogLevel{}, RequestDoSetMetaLogLevel{}, ResponseDoSetLogLevel{}, + RequestDoSetProcessSendPriority{}, RequestDoSetProcessCompression{}, + RequestDoSetProcessCompressionType{}, RequestDoSetProcessCompressionLevel{}, + RequestDoSetProcessCompressionThreshold{}, RequestDoSetProcessKeepNetworkOrder{}, + RequestDoSetProcessImportantDelivery{}, RequestDoSetMetaSendPriority{}, ResponseDoSet{}, + RequestDoSetNodeTracingSampler{}, RequestDoSetProcessTracingSampler{}, + RequestDoAppStart{}, ResponseDoAppStart{}, + RequestDoAppStop{}, ResponseDoAppStop{}, + RequestDoAppUnload{}, ResponseDoAppUnload{}, + RequestDoInspect{}, ResponseDoInspect{}, + RequestDoGoroutines{}, GoroutineGroup{}, ResponseDoGoroutines{}, + RequestDoHeapProfile{}, HeapRecord{}, ResponseDoHeapProfile{}, + RequestDoTypes{}, ResponseDoTypes{}, + } + return network.RegisterTypes(types) +} + +func workerFactory() gen.ProcessBehavior { return &inspect{} } +type inspectPool struct { + act.Pool +} + +func (p *inspectPool) Init(args ...any) (act.PoolOptions, error) { + return act.PoolOptions{ + PoolSize: 5, + WorkerFactory: workerFactory, + }, nil +} + type inspect struct { act.Actor } @@ -82,11 +158,13 @@ type requestInspect struct { type register struct{} type shutdown struct{} -type generate struct{} +type generate struct{ id uint64 } +type flushLog struct{ id uint64 } func (i *inspect) Init(args ...any) error { i.Log().SetLogger("default") i.Log().Debug("%s started", i.Name()) + i.SetCompression(true) return nil } @@ -146,14 +224,16 @@ func (i *inspect) HandleCall(from gen.PID, ref gen.Ref, request any) (any, error opts := gen.ProcessOptions{ LinkParent: true, } - if r.Start < 1000 { + if r.Start >= 0 && r.Start < 1000 { r.Start = 1000 } if r.Limit < 1 { r.Limit = 1000 } - pname := gen.Atom(fmt.Sprintf("%s_%d_%d", inspectProcessList, r.Start, r.Start+r.Limit-1)) - _, err := i.SpawnRegister(pname, factory_process_list, opts, r.Start, r.Limit) + hash := filterHash(r.Name, r.Behavior, r.Application, r.State, r.MinMailbox, r.Limit) + pname := gen.Atom(fmt.Sprintf("%s_%d_%s", inspectProcessList, r.Start, hash)) + _, err := i.SpawnRegister(pname, factory_process_list, opts, + r.Start, r.Limit, r.Name, r.Behavior, r.Application, r.State, r.MinMailbox) if err != nil && err != gen.ErrTaken { return err, nil } @@ -165,6 +245,27 @@ func (i *inspect) HandleCall(from gen.PID, ref gen.Ref, request any) (any, error i.Send(pname, forward) return nil, nil // no reply + case RequestInspectProcessRange: + opts := gen.ProcessOptions{ + LinkParent: true, + } + if r.Limit < 1 { + r.Limit = 10000 + } + hash := filterHash(r.Name, r.Behavior, r.Application, r.State, r.MinMailbox, r.Limit) + pname := gen.Atom(fmt.Sprintf("%s_%s", inspectProcessRange, hash)) + _, err := i.SpawnRegister(pname, factory_process_range, opts, + r.Name, r.Behavior, r.Application, r.State, r.MinMailbox, r.Limit, hash) + if err != nil && err != gen.ErrTaken { + return err, nil + } + forward := requestInspect{ + pid: from, + ref: ref, + } + i.Send(pname, forward) + return nil, nil // no reply + case RequestInspectProcess: opts := gen.ProcessOptions{ LinkParent: true, @@ -240,36 +341,28 @@ func (i *inspect) HandleCall(from gen.PID, ref gen.Ref, request any) (any, error // try to spawn node inspector process opts := gen.ProcessOptions{ LinkParent: true, + Compression: gen.Compression{ + Enable: true, + Type: gen.CompressionTypeGZIP, + Level: gen.CompressionBestSpeed, + }, } - name := "diwep" levels := r.Levels if len(r.Levels) > 0 { - b := []byte{} - sort.Slice(r.Levels, func(i, j int) bool { - return r.Levels[i] < r.Levels[j] - }) - for i := range r.Levels { - switch r.Levels[i] { - case gen.LogLevelDebug: - b = append(b, 'd') - case gen.LogLevelInfo: - b = append(b, 'i') - case gen.LogLevelWarning: - b = append(b, 'w') - case gen.LogLevelError: - b = append(b, 'e') - case gen.LogLevelPanic: - b = append(b, 'p') - } - } - name = string(b) + sort.Slice(levels, func(i, j int) bool { return levels[i] < levels[j] }) } else { levels = inspectLogFilter } - pname := gen.Atom(fmt.Sprintf("%s_%s", inspectLog, name)) - _, err := i.SpawnRegister(pname, factory_log, opts, levels) + limit := r.Limit + if limit < 1 { + limit = 500 + } + + hash := fmt.Sprintf("%x", hashStr(fmt.Sprintf("%v|%d|%s|%v", levels, limit, r.MessagePattern, r.MessageExclude))) + pname := gen.Atom(fmt.Sprintf("%s_%s", inspectLog, hash)) + _, err := i.SpawnRegister(pname, factory_log, opts, levels, limit, r.MessagePattern, r.MessageExclude) if err != nil && err != gen.ErrTaken { return err, nil } @@ -281,6 +374,75 @@ func (i *inspect) HandleCall(from gen.PID, ref gen.Ref, request any) (any, error i.Send(pname, forward) return nil, nil // no reply + case RequestInspectTracing: + opts := gen.ProcessOptions{ + LinkParent: true, + Compression: gen.Compression{ + Enable: true, + Type: gen.CompressionTypeGZIP, + Level: gen.CompressionBestSpeed, + }, + } + + limit := r.Limit + if limit < 1 { + limit = 500 + } + + hash := fmt.Sprintf("%x", hashStr(fmt.Sprintf("%v|%d|%d|%d|%s|%v", r.Flags, limit, r.Kinds, r.Points, r.MessagePattern, r.MessageExclude))) + pname := gen.Atom(fmt.Sprintf("%s_%s", inspectTracing, hash)) + _, err := i.SpawnRegister(pname, factory_tracing, opts, r.Flags, limit, r.Kinds, r.Points, r.MessagePattern, r.MessageExclude) + if err != nil && err != gen.ErrTaken { + return err, nil + } + forward := requestInspect{ + pid: from, + ref: ref, + } + i.Send(pname, forward) + return nil, nil + + case RequestInspectEventList: + opts := gen.ProcessOptions{ + LinkParent: true, + } + if r.Limit < 1 { + r.Limit = 500 + } + hash := eventListHash(r.Timestamp, r.Name, r.Notify, r.Buffered, r.Open, r.MinSubscribers, r.Limit) + pname := gen.Atom(fmt.Sprintf("%s_%s", inspectEventList, hash)) + _, err := i.SpawnRegister(pname, factory_event_list, opts, + r.Timestamp, r.Name, r.Notify, r.Buffered, r.Open, r.MinSubscribers, r.Limit, hash) + if err != nil && err != gen.ErrTaken { + return err, nil + } + forward := requestInspect{ + pid: from, + ref: ref, + } + i.Send(pname, forward) + return nil, nil + + case RequestInspectConnectionList: + opts := gen.ProcessOptions{ + LinkParent: true, + } + if r.Limit < 1 { + r.Limit = 100 + } + hash := connectionListHash(r.Name, r.Limit) + pname := gen.Atom(fmt.Sprintf("%s_%s", inspectConnectionList, hash)) + _, err := i.SpawnRegister(pname, factory_connection_list, opts, r.Name, r.Limit, hash) + if err != nil && err != gen.ErrTaken { + return err, nil + } + forward := requestInspect{ + pid: from, + ref: ref, + } + i.Send(pname, forward) + return nil, nil + case RequestInspectApplicationList: opts := gen.ProcessOptions{ LinkParent: true, @@ -317,6 +479,21 @@ func (i *inspect) HandleCall(from gen.PID, ref gen.Ref, request any) (any, error i.Send(pname, forward) return nil, nil // no reply + case RequestInspectHeap: + opts := gen.ProcessOptions{LinkParent: true} + if r.Limit < 1 { + r.Limit = 100 + } + hash := filterHash(r.Name, "", "", "", 0, r.Limit) + pname := gen.Atom(fmt.Sprintf("%s_%s", inspectHeap, hash)) + _, err := i.SpawnRegister(pname, factory_heap, opts, r.Limit, r.Name) + if err != nil && err != gen.ErrTaken { + return err, nil + } + forward := requestInspect{pid: from, ref: ref} + i.Send(pname, forward) + return nil, nil + // do commands case RequestDoSend: @@ -355,19 +532,111 @@ func (i *inspect) HandleCall(from gen.PID, ref gen.Ref, request any) (any, error } return response, nil - case RequestDoSetLogLevelProcess: + case RequestDoSetNodeTracingSampler: + sampler := makeSampler(r.Type, r.Rate, r.Limit) + return ResponseDoSet{Error: i.Node().SetTracingSampler(sampler)}, nil + + case RequestDoSetProcessTracingSampler: + sampler := makeSampler(r.Type, r.Rate, r.Limit) + return ResponseDoSet{Error: i.Node().SetProcessTracingSampler(r.PID, sampler)}, nil + + case RequestDoSetProcessLogLevel: response := ResponseDoSetLogLevel{ - Error: i.Node().SetLogLevelProcess(r.PID, r.Level), + Error: i.Node().SetProcessLogLevel(r.PID, r.Level), } return response, nil - case RequestDoSetLogLevelMeta: + case RequestDoSetMetaLogLevel: response := ResponseDoSetLogLevel{ - Error: i.Node().SetLogLevelMeta(r.Meta, r.Level), + Error: i.Node().SetMetaLogLevel(r.Meta, r.Level), } return response, nil + + // process settings + + case RequestDoSetProcessSendPriority: + return ResponseDoSet{Error: i.Node().SetProcessSendPriority(r.PID, r.Priority)}, nil + + case RequestDoSetProcessCompression: + return ResponseDoSet{Error: i.Node().SetProcessCompression(r.PID, r.Enabled)}, nil + + case RequestDoSetProcessCompressionType: + return ResponseDoSet{Error: i.Node().SetProcessCompressionType(r.PID, r.Type)}, nil + + case RequestDoSetProcessCompressionLevel: + return ResponseDoSet{Error: i.Node().SetProcessCompressionLevel(r.PID, r.Level)}, nil + + case RequestDoSetProcessCompressionThreshold: + return ResponseDoSet{Error: i.Node().SetProcessCompressionThreshold(r.PID, r.Threshold)}, nil + + case RequestDoSetProcessKeepNetworkOrder: + return ResponseDoSet{Error: i.Node().SetProcessKeepNetworkOrder(r.PID, r.Order)}, nil + + case RequestDoSetProcessImportantDelivery: + return ResponseDoSet{Error: i.Node().SetProcessImportantDelivery(r.PID, r.Important)}, nil + + // meta settings + + case RequestDoSetMetaSendPriority: + return ResponseDoSet{Error: i.Node().SetMetaSendPriority(r.Meta, r.Priority)}, nil + + // app lifecycle + + case RequestDoAppStart: + opts := gen.ApplicationOptions{} + var err error + switch r.Mode { + case gen.ApplicationModeTemporary: + err = i.Node().ApplicationStartTemporary(r.Name, opts) + case gen.ApplicationModeTransient: + err = i.Node().ApplicationStartTransient(r.Name, opts) + case gen.ApplicationModePermanent: + err = i.Node().ApplicationStartPermanent(r.Name, opts) + default: + err = i.Node().ApplicationStart(r.Name, opts) + } + return ResponseDoAppStart{Error: err}, nil + + case RequestDoAppStop: + var err error + if r.Force { + err = i.Node().ApplicationStopForce(r.Name) + } else { + err = i.Node().ApplicationStop(r.Name) + } + return ResponseDoAppStop{Error: err}, nil + + case RequestDoAppUnload: + return ResponseDoAppUnload{Error: i.Node().ApplicationUnload(r.Name)}, nil + + // one-shot inspect + + case RequestDoInspect: + state, err := i.Inspect(r.PID) + return ResponseDoInspect{State: state, Error: err}, nil + + case RequestDoGoroutines: + return captureGoroutines(r), nil + + case RequestDoHeapProfile: + return captureHeapProfile(r), nil + + case RequestDoTypes: + return ResponseDoTypes{Types: i.Node().Network().RegisteredTypes()}, nil } i.Log().Error("unsupported request: %#v", request) return gen.ErrUnsupported, nil } + +func makeSampler(typ string, rate float64, limit int) gen.TracingSampler { + switch typ { + case "always": + return gen.TracingSamplerAlways + case "ratio": + return gen.TracingSamplerRatio(rate) + case "rate_limit": + return gen.TracingSamplerRateLimit(limit) + } + return gen.TracingSamplerDisable +} diff --git a/app/system/inspect/log.go b/app/system/inspect/log.go index 1ffb873f5..a568daec3 100644 --- a/app/system/inspect/log.go +++ b/app/system/inspect/log.go @@ -2,6 +2,8 @@ package inspect import ( "fmt" + "strings" + "time" "ergo.services/ergo/act" "ergo.services/ergo/gen" @@ -16,16 +18,47 @@ type log struct { token gen.Ref event gen.Atom - levels []gen.LogLevel - generating bool + levels []gen.LogLevel + limit int + messagePattern string // lower-cased for fast matching + messageExclude bool + generating bool + loopID uint64 + + // ring buffer + ring []InspectLogEntry + pos int + full bool + received int64 } +const logFlushInterval = time.Second + func (il *log) Init(args ...any) error { il.levels = args[0].([]gen.LogLevel) + il.limit = args[1].(int) + if len(args) > 3 { + il.messagePattern = strings.ToLower(args[2].(string)) + il.messageExclude = args[3].(bool) + } + il.ring = make([]InspectLogEntry, il.limit) il.Log().SetLogger("default") - il.Log().Debug("log inspector started") - // RegisterEvent is not allowed here - il.Send(il.PID(), register{}) + il.Log().Debug("log inspector started (limit: %d)", il.limit) + il.SetCompression(true) + + eopts := gen.EventOptions{ + Notify: true, + } + evname := gen.Atom(fmt.Sprintf("%s_%s", string(il.Name()), il.PID())) + token, err := il.RegisterEvent(evname, eopts) + if err != nil { + return err + } + + il.event = evname + il.token = token + il.SendAfter(il.PID(), shutdown{}, inspectLogIdlePeriod) + return nil } @@ -34,6 +67,49 @@ func (il *log) Init(args ...any) error { func (il *log) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { + case flushLog: + if m.id != il.loopID || il.generating == false { + break + } + if il.received == 0 { + il.SendAfter(il.PID(), flushLog{id: il.loopID}, logFlushInterval) + break + } + + // collect entries from ring buffer in correct order + var entries []InspectLogEntry + if il.full { + // ring wrapped: oldest at pos, newest at pos-1 + entries = make([]InspectLogEntry, il.limit) + copy(entries, il.ring[il.pos:]) + copy(entries[il.limit-il.pos:], il.ring[:il.pos]) + } else { + entries = make([]InspectLogEntry, il.pos) + copy(entries, il.ring[:il.pos]) + } + + suppressed := il.received - int64(len(entries)) + if suppressed < 0 { + suppressed = 0 + } + + ev := MessageInspectLog{ + Node: il.Node().Name(), + Entries: entries, + Suppressed: suppressed, + } + + // reset ring + il.pos = 0 + il.full = false + il.received = 0 + + if err := il.SendEvent(il.event, il.token, ev); err != nil { + return gen.TerminateReasonNormal + } + + il.SendAfter(il.PID(), flushLog{id: il.loopID}, logFlushInterval) + case requestInspect: response := ResponseInspectLog{ Event: gen.Event{ @@ -43,39 +119,26 @@ func (il *log) HandleMessage(from gen.PID, message any) error { } il.SendResponse(m.pid, m.ref, response) - case register: - eopts := gen.EventOptions{ - Notify: true, - } - evname := gen.Atom(fmt.Sprintf("%s_%s", string(il.Name()), il.PID())) - token, err := il.RegisterEvent(evname, eopts) - if err != nil { - return err - } - - il.event = evname - il.token = token - il.SendAfter(il.PID(), shutdown{}, inspectLogIdlePeriod) - case shutdown: if il.generating { - break // ignore. + break // ignore } return gen.TerminateReasonNormal case gen.MessageEventStart: // got first subscriber - // register this process as a logger il.Log().Debug("add this process as a logger") il.Node().LoggerAddPID(il.PID(), il.PID().String(), il.levels...) - // we cant use Log() method while this process registered as a logger + il.loopID++ il.generating = true + il.SendAfter(il.PID(), flushLog{id: il.loopID}, logFlushInterval) case gen.MessageEventStop: // no subscribers - // unregister this process as a logger il.Node().LoggerDeletePID(il.PID()) - // now we can use Log() method il.Log().Debug("removed this process as a logger") il.generating = false + il.pos = 0 + il.full = false + il.received = 0 il.SendAfter(il.PID(), shutdown{}, inspectLogIdlePeriod) } @@ -83,67 +146,52 @@ func (il *log) HandleMessage(from gen.PID, message any) error { } func (il *log) HandleLog(message gen.MessageLog) error { + msg := fmt.Sprintf(message.Format, message.Args...) + + if il.messagePattern != "" { + contains := strings.Contains(strings.ToLower(msg), il.messagePattern) + if il.messageExclude == contains { + return nil + } + } + + entry := InspectLogEntry{ + Timestamp: message.Time.UnixNano(), + Level: message.Level, + Message: msg, + Fields: message.Fields, + } + switch m := message.Source.(type) { case gen.MessageLogNode: - // handle message - ev := MessageInspectLogNode{ - Node: m.Node, - Creation: m.Creation, - Timestamp: message.Time.UnixNano(), - Level: message.Level, - Message: fmt.Sprintf(message.Format, message.Args...), - } - if err := il.SendEvent(il.event, il.token, ev); err != nil { - return gen.TerminateReasonNormal - } + entry.Source = "node" + entry.Creation = m.Creation case gen.MessageLogProcess: - // handle message - ev := MessageInspectLogProcess{ - Node: m.Node, - Name: m.Name, - PID: m.PID, - Timestamp: message.Time.UnixNano(), - Level: message.Level, - Message: fmt.Sprintf(message.Format, message.Args...), - } - if err := il.SendEvent(il.event, il.token, ev); err != nil { - return gen.TerminateReasonNormal - } - + entry.Source = "process" + entry.Name = m.Name + entry.PID = m.PID + entry.Behavior = m.Behavior case gen.MessageLogMeta: - // handle message - ev := MessageInspectLogMeta{ - Node: m.Node, - Parent: m.Parent, - Meta: m.Meta, - Timestamp: message.Time.UnixNano(), - Level: message.Level, - Message: fmt.Sprintf(message.Format, message.Args...), - } - - if err := il.SendEvent(il.event, il.token, ev); err != nil { - return gen.TerminateReasonNormal - } + entry.Source = "meta" + entry.Parent = m.Parent + entry.Meta = m.Meta + entry.Behavior = m.Behavior case gen.MessageLogNetwork: - ev := MessageInspectLogNetwork{ - Node: m.Node, - Peer: m.Peer, - Timestamp: message.Time.UnixNano(), - Level: message.Level, - Message: fmt.Sprintf(message.Format, message.Args...), - } - if err := il.SendEvent(il.event, il.token, ev); err != nil { - return gen.TerminateReasonNormal - } + entry.Source = "network" + entry.Peer = gen.Atom(m.Peer.CRC32()) } - // ignore any other log messages - // TODO should we handle them? + + il.ring[il.pos] = entry + il.pos++ + if il.pos >= il.limit { + il.pos = 0 + il.full = true + } + il.received++ + return nil } func (il *log) Terminate(reason error) { - // since this process is already unregistered - // it is also unregistered as a logger - // so we can use Log() here il.Log().Debug("log inspector terminated: %s", reason) } diff --git a/app/system/inspect/message.go b/app/system/inspect/message.go index 547c6b3e9..c51042703 100644 --- a/app/system/inspect/message.go +++ b/app/system/inspect/message.go @@ -4,13 +4,15 @@ import "ergo.services/ergo/gen" type RequestInspectNode struct{} type ResponseInspectNode struct { - CRC32 string - Event gen.Event - OS string - Arch string - Cores int - Version gen.Version - Creation int64 + CRC32 string + Event gen.Event + OS string + Arch string + Cores int + Timezone string + GoVersion string + Version gen.Version + Creation int64 } type MessageInspectNode struct { @@ -48,11 +50,31 @@ type MessageInspectConnection struct { Info gen.RemoteNodeInfo } +// connection list (scoped) + +type RequestInspectConnectionList struct { + Limit int + Name string +} +type ResponseInspectConnectionList struct { + Event gen.Event +} + +type MessageInspectConnectionList struct { + Node gen.Atom + Connections []gen.RemoteNodeInfo +} + // process list type RequestInspectProcessList struct { - Start int - Limit int + Start int + Limit int + Name string + Behavior string + Application string + State string + MinMailbox uint64 } type ResponseInspectProcessList struct { Event gen.Event @@ -66,44 +88,34 @@ type MessageInspectProcessList struct { // node logs type RequestInspectLog struct { - Levels []gen.LogLevel + Levels []gen.LogLevel + Limit int + MessagePattern string + MessageExclude bool } type ResponseInspectLog struct { Event gen.Event } -type MessageInspectLogNode struct { - Node gen.Atom - Creation int64 - Timestamp int64 - Level gen.LogLevel - Message string -} - -type MessageInspectLogProcess struct { - Node gen.Atom +type InspectLogEntry struct { + Source string // "node", "process", "network", "meta" Name gen.Atom PID gen.PID - Timestamp int64 - Level gen.LogLevel - Message string -} - -type MessageInspectLogNetwork struct { - Node gen.Atom + Behavior string Peer gen.Atom - Timestamp int64 - Level gen.LogLevel - Message string -} - -type MessageInspectLogMeta struct { - Node gen.Atom Parent gen.PID Meta gen.Alias + Creation int64 Timestamp int64 Level gen.LogLevel Message string + Fields []gen.LogField +} + +type MessageInspectLog struct { + Node gen.Atom + Entries []InspectLogEntry + Suppressed int64 } // process @@ -116,9 +128,8 @@ type ResponseInspectProcess struct { } type MessageInspectProcess struct { - Node gen.Atom - Info gen.ProcessInfo - Terminated bool + Node gen.Atom + Info gen.ProcessInfo } // process state @@ -145,9 +156,8 @@ type ResponseInspectMeta struct { } type MessageInspectMeta struct { - Node gen.Atom - Info gen.MetaInfo - Terminated bool + Node gen.Atom + Info gen.MetaInfo } // meta state @@ -221,18 +231,235 @@ type ResponseDoSetLogLevel struct { Error error } +// do set tracing sampler and flags (node-level) + +type RequestDoSetNodeTracingSampler struct { + Type string // "always", "disable", "ratio", "rate_limit" + Rate float64 // for ratio + Limit int // for rate_limit +} + +type RequestDoSetProcessTracingSampler struct { + PID gen.PID + Type string + Rate float64 + Limit int +} + + // process -type RequestDoSetLogLevelProcess struct { +type RequestDoSetProcessLogLevel struct { PID gen.PID Level gen.LogLevel } // meta -type RequestDoSetLogLevelMeta struct { +type RequestDoSetMetaLogLevel struct { Meta gen.Alias Level gen.LogLevel } +// do set process settings + +type RequestDoSetProcessSendPriority struct { + PID gen.PID + Priority gen.MessagePriority +} + +type RequestDoSetProcessCompression struct { + PID gen.PID + Enabled bool +} + +type RequestDoSetProcessCompressionType struct { + PID gen.PID + Type gen.CompressionType +} + +type RequestDoSetProcessCompressionLevel struct { + PID gen.PID + Level gen.CompressionLevel +} + +type RequestDoSetProcessCompressionThreshold struct { + PID gen.PID + Threshold int +} + +type RequestDoSetProcessKeepNetworkOrder struct { + PID gen.PID + Order bool +} + +type RequestDoSetProcessImportantDelivery struct { + PID gen.PID + Important bool +} + +// do set meta settings + +type RequestDoSetMetaSendPriority struct { + Meta gen.Alias + Priority gen.MessagePriority +} + +// generic response for do-set operations +type ResponseDoSet struct { + Error error +} + +// do app lifecycle + +type RequestDoAppStart struct { + Name gen.Atom + Mode gen.ApplicationMode +} +type ResponseDoAppStart struct { + Error error +} + +type RequestDoAppStop struct { + Name gen.Atom + Force bool +} +type ResponseDoAppStop struct { + Error error +} + +type RequestDoAppUnload struct { + Name gen.Atom +} +type ResponseDoAppUnload struct { + Error error +} + +// do one-shot inspect + +type RequestDoInspect struct { + PID gen.PID +} +type ResponseDoInspect struct { + State map[string]string + Error error +} + +// goroutine dump + +type RequestDoGoroutines struct { + Stack string // substring match in stack text + State string // exact state match (running, chan receive, etc.) + MinWait int64 // minimum wait duration in seconds (0 = any) +} + +type GoroutineInfo struct { + ID int + State string + Wait string + Frames []string + FullText string +} + +type GoroutineGroup struct { + Count int + State string + WaitSec int64 + Origin string + Current string + Stack string + IDs []int +} + +type ResponseDoGoroutines struct { + Groups []GoroutineGroup + Total int + Filtered int + Error error +} + +// heap profile + +type RequestDoHeapProfile struct { + MinBytes int64 +} + +type HeapRecord struct { + InuseBytes int64 + InuseObjects int64 + AllocBytes int64 + AllocObjects int64 + FreeObjects int64 + Stack []string +} + +type HeapStats struct { + TotalInuse int64 + TotalObjects int64 + TotalAlloc int64 + TotalFree int64 +} + +type ResponseDoHeapProfile struct { + Records []HeapRecord + TotalInuse int64 + TotalAlloc int64 + TotalObjects int64 + Error error +} + +// heap inspector (event-based) + +type RequestInspectHeap struct { + Limit int + Name string +} +type ResponseInspectHeap struct { + Event gen.Event +} + +type MessageInspectHeap struct { + Node gen.Atom + Records []HeapRecord + TotalInuse int64 + TotalObjects int64 + TotalAlloc int64 + TotalFree int64 + GCCPUFraction float64 +} + +// process range (full scan with filters) + +type RequestInspectProcessRange struct { + Name string + Behavior string + Application string + State string + MinMailbox uint64 + Limit int +} +type ResponseInspectProcessRange struct { + Event gen.Event +} + +// event list + +type RequestInspectEventList struct { + Timestamp int64 // 0=oldest first, -1=newest first, >0=from this unix nanos + Limit int + Name string + Notify int // 0=any, 1=yes, -1=no + Buffered int // 0=any, 1=yes, -1=no + Open int // 0=any, 1=yes, -1=no + MinSubscribers int64 +} +type ResponseInspectEventList struct { + Event gen.Event +} + +type MessageInspectEventList struct { + Node gen.Atom + Events []gen.EventInfo +} + // application list type RequestInspectApplicationList struct{} @@ -260,3 +487,33 @@ type MessageInspectApplicationTree struct { Application gen.Atom Processes []gen.ProcessShortInfo } + +// tracing + +type RequestInspectTracing struct { + Flags gen.TracingFlags + Limit int + Kinds uint32 // bitmask: 1=send, 2=request, 4=response, 8=spawn, 16=terminate + Points uint32 // bitmask: 1=sent, 2=delivered, 4=processed + MessagePattern string + MessageExclude bool +} + +type ResponseInspectTracing struct { + Event gen.Event +} + +type MessageInspectTracing struct { + Node gen.Atom + Spans []gen.TracingSpan + Suppressed int64 +} + +// types + +type RequestDoTypes struct{} + +type ResponseDoTypes struct { + Types []gen.RegisteredTypeInfo + Error error +} diff --git a/app/system/inspect/meta.go b/app/system/inspect/meta.go index 7e7dc8e1d..f9c1c378d 100644 --- a/app/system/inspect/meta.go +++ b/app/system/inspect/meta.go @@ -17,6 +17,7 @@ type meta struct { event gen.Atom generating bool + loopID uint64 meta gen.Alias } @@ -24,22 +25,39 @@ func (im *meta) Init(args ...any) error { im.meta = args[0].(gen.Alias) im.Log().SetLogger("default") im.Log().Debug("meta process inspector started. pid %s", im.meta) - // RegisterEvent is not allowed here - im.Send(im.PID(), register{}) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, // keep the last event + } + evname := gen.Atom(fmt.Sprintf("%s_%s", inspectMeta, im.meta)) + token, err := im.RegisterEvent(evname, eopts) + if err != nil { + im.Log().Error("unable to register meta process event: %s", err) + return err + } + im.Log().Info("registered event %s", evname) + im.event = evname + im.token = token + im.SendAfter(im.PID(), shutdown{}, inspectMetaIdlePeriod) + return nil } func (im *meta) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { case generate: - if im.generating == false { + if m.id != im.loopID || im.generating == false { im.Log().Debug("generating canceled") break // cancelled } im.Log().Debug("generating event") ev := MessageInspectMeta{ - Node: im.Node().Name(), - Terminated: true, + Node: im.Node().Name(), + Info: gen.MetaInfo{ + ID: im.meta, + State: gen.MetaStateTerminated, + }, } info, err := im.MetaInfo(im.meta) @@ -56,12 +74,11 @@ func (im *meta) HandleMessage(from gen.PID, message any) error { default: im.Log().Error("unable to inspect meta process %s: %s", im.meta, err) // will try next time - im.SendAfter(im.PID(), generate{}, inspectMetaPeriod) + im.SendAfter(im.PID(), generate{id: im.loopID}, inspectMetaPeriod) return nil } - ev.Terminated = false ev.Info = info if err := im.SendEvent(im.event, im.token, ev); err != nil { @@ -69,7 +86,7 @@ func (im *meta) HandleMessage(from gen.PID, message any) error { return gen.TerminateReasonNormal } - im.SendAfter(im.PID(), generate{}, inspectMetaPeriod) + im.SendAfter(im.PID(), generate{id: im.loopID}, inspectMetaPeriod) case requestInspect: response := ResponseInspectMeta{ @@ -81,23 +98,6 @@ func (im *meta) HandleMessage(from gen.PID, message any) error { im.SendResponse(m.pid, m.ref, response) im.Log().Debug("sent response for the inspect meta request to: %s", m.pid) - case register: - eopts := gen.EventOptions{ - Notify: true, - Buffer: 1, // keep the last event - } - evname := gen.Atom(fmt.Sprintf("%s_%s", inspectMeta, im.meta)) - token, err := im.RegisterEvent(evname, eopts) - if err != nil { - im.Log().Error("unable to register meta process event: %s", err) - return err - } - im.Log().Info("registered event %s", evname) - im.event = evname - - im.token = token - im.SendAfter(im.PID(), shutdown{}, inspectMetaIdlePeriod) - case shutdown: if im.generating { im.Log().Debug("ignore shutdown. generating is active") @@ -107,7 +107,8 @@ func (im *meta) HandleMessage(from gen.PID, message any) error { case gen.MessageEventStart: // got first subscriber im.Log().Debug("got first subscriber. start generating events...") - im.Send(im.PID(), generate{}) + im.loopID++ + im.Send(im.PID(), generate{id: im.loopID}) im.generating = true case gen.MessageEventStop: // no subscribers diff --git a/app/system/inspect/meta_state.go b/app/system/inspect/meta_state.go index c27c8e6c2..31787de81 100644 --- a/app/system/inspect/meta_state.go +++ b/app/system/inspect/meta_state.go @@ -17,6 +17,7 @@ type meta_state struct { event gen.Atom generating bool + loopID uint64 meta gen.Alias } @@ -24,15 +25,29 @@ func (ims *meta_state) Init(args ...any) error { ims.meta = args[0].(gen.Alias) ims.Log().SetLogger("default") ims.Log().Debug("meta state inspector started. id %s", ims.meta) - // RegisterEvent is not allowed here - ims.Send(ims.PID(), register{}) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, // keep the last event + } + evname := gen.Atom(fmt.Sprintf("%s_%s", inspectMetaState, ims.meta)) + token, err := ims.RegisterEvent(evname, eopts) + if err != nil { + ims.Log().Error("unable to register meta state event: %s", err) + return err + } + ims.Log().Info("registered event %s", evname) + ims.event = evname + ims.token = token + ims.SendAfter(ims.PID(), shutdown{}, inspectMetaStateIdlePeriod) + return nil } func (ims *meta_state) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { case generate: - if ims.generating == false { + if m.id != ims.loopID || ims.generating == false { ims.Log().Debug("generating canceled") break // cancelled } @@ -44,7 +59,7 @@ func (ims *meta_state) HandleMessage(from gen.PID, message any) error { } ims.Log().Error("unable to inspect meta state %s: %s", ims.meta, err) // will try next time - ims.SendAfter(ims.PID(), generate{}, inspectMetaStatePeriod) + ims.SendAfter(ims.PID(), generate{id: ims.loopID}, inspectMetaStatePeriod) return nil } if state == nil { @@ -62,7 +77,7 @@ func (ims *meta_state) HandleMessage(from gen.PID, message any) error { return gen.TerminateReasonNormal } - ims.SendAfter(ims.PID(), generate{}, inspectMetaStatePeriod) + ims.SendAfter(ims.PID(), generate{id: ims.loopID}, inspectMetaStatePeriod) case requestInspect: response := ResponseInspectMetaState{ @@ -74,23 +89,6 @@ func (ims *meta_state) HandleMessage(from gen.PID, message any) error { ims.SendResponse(m.pid, m.ref, response) ims.Log().Debug("sent response for the inspect meta state %s request to: %s", ims.meta, m.pid) - case register: - eopts := gen.EventOptions{ - Notify: true, - Buffer: 1, // keep the last event - } - evname := gen.Atom(fmt.Sprintf("%s_%s", inspectMetaState, ims.meta)) - token, err := ims.RegisterEvent(evname, eopts) - if err != nil { - ims.Log().Error("unable to register meta state event: %s", err) - return err - } - ims.Log().Info("registered event %s", evname) - ims.event = evname - - ims.token = token - ims.SendAfter(ims.PID(), shutdown{}, inspectMetaStateIdlePeriod) - case shutdown: if ims.generating { ims.Log().Debug("ignore shutdown. generating is active") @@ -100,7 +98,8 @@ func (ims *meta_state) HandleMessage(from gen.PID, message any) error { case gen.MessageEventStart: // got first subscriber ims.Log().Debug("got first subscriber. start generating events...") - ims.Send(ims.PID(), generate{}) + ims.loopID++ + ims.Send(ims.PID(), generate{id: ims.loopID}) ims.generating = true case gen.MessageEventStop: // no subscribers diff --git a/app/system/inspect/network.go b/app/system/inspect/network.go index 4a054662d..919766ef3 100644 --- a/app/system/inspect/network.go +++ b/app/system/inspect/network.go @@ -15,20 +15,33 @@ type network struct { token gen.Ref generating bool + loopID uint64 } func (in *network) Init(args ...any) error { in.Log().SetLogger("default") in.Log().Debug("network inspector started") - // RegisterEvent is not allowed here - in.Send(in.PID(), register{}) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, // keep the last event + } + token, err := in.RegisterEvent(inspectNetwork, eopts) + if err != nil { + in.Log().Error("unable to register network event: %s", err) + return err + } + in.Log().Info("registered event %s", inspectNetwork) + in.token = token + in.SendAfter(in.PID(), shutdown{}, inspectNetworkIdlePeriod) + return nil } func (in *network) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { case generate: - if in.generating == false { + if m.id != in.loopID || in.generating == false { in.Log().Debug("generating canceled") break // cancelled } @@ -47,7 +60,7 @@ func (in *network) HandleMessage(from gen.PID, message any) error { return gen.TerminateReasonNormal } - in.SendAfter(in.PID(), generate{}, inspectNetworkPeriod) + in.SendAfter(in.PID(), generate{id: in.loopID}, inspectNetworkPeriod) case requestInspect: info, err := in.Node().Network().Info() @@ -62,21 +75,6 @@ func (in *network) HandleMessage(from gen.PID, message any) error { in.SendResponse(m.pid, m.ref, response) in.Log().Debug("sent response for the inspect network request to: %s", m.pid) - case register: - eopts := gen.EventOptions{ - Notify: true, - Buffer: 1, // keep the last event - } - token, err := in.RegisterEvent(inspectNetwork, eopts) - if err != nil { - in.Log().Error("unable to register network event: %s", err) - return err - } - in.Log().Info("registered event %s", inspectNetwork) - - in.token = token - in.SendAfter(in.PID(), shutdown{}, inspectNetworkIdlePeriod) - case shutdown: if in.generating { in.Log().Debug("ignore shutdown. generating is active") @@ -86,7 +84,8 @@ func (in *network) HandleMessage(from gen.PID, message any) error { case gen.MessageEventStart: // got first subscriber in.Log().Debug("got first subscriber. start generating events...") - in.Send(in.PID(), generate{}) + in.loopID++ + in.Send(in.PID(), generate{id: in.loopID}) in.generating = true case gen.MessageEventStop: // no subscribers diff --git a/app/system/inspect/node.go b/app/system/inspect/node.go index e7c275fc7..a41698091 100644 --- a/app/system/inspect/node.go +++ b/app/system/inspect/node.go @@ -5,6 +5,7 @@ import ( "fmt" "runtime" "slices" + "time" "ergo.services/ergo/act" "ergo.services/ergo/gen" @@ -19,20 +20,33 @@ type node struct { token gen.Ref generating bool + loopID uint64 } func (in *node) Init(args ...any) error { in.Log().SetLogger("default") in.Log().Debug("node inspector started") - // RegisterEvent is not allowed here - in.Send(in.PID(), register{}) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, // keep the last event + } + token, err := in.RegisterEvent(inspectNode, eopts) + if err != nil { + in.Log().Error("unable to register event: %s", err) + return err + } + in.Log().Info("registered event %s", inspectNode) + in.token = token + in.SendAfter(in.PID(), shutdown{}, inspectNodeIdlePeriod) + return nil } func (in *node) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { case generate: - if in.generating == false { + if m.id != in.loopID || in.generating == false { in.Log().Debug("generating canceled") break // cancelled } @@ -60,7 +74,7 @@ func (in *node) HandleMessage(from gen.PID, message any) error { return gen.TerminateReasonNormal } - in.SendAfter(in.PID(), generate{}, inspectNodePeriod) + in.SendAfter(in.PID(), generate{id: in.loopID}, inspectNodePeriod) case requestInspect: response := ResponseInspectNode{ @@ -69,9 +83,19 @@ func (in *node) HandleMessage(from gen.PID, message any) error { Node: in.Node().Name(), }, - Arch: runtime.GOARCH, - OS: runtime.GOOS, - Cores: runtime.NumCPU(), + Arch: runtime.GOARCH, + OS: runtime.GOOS, + Cores: runtime.NumCPU(), + GoVersion: runtime.Version(), + Timezone: func() string { + now := time.Now() + name, _ := now.Zone() + loc := now.Location().String() + if loc == "Local" { + return name // e.g. "MSK", "CET" + } + return loc // e.g. "Europe/Moscow" + }(), Version: in.Node().Version(), Creation: in.Node().Creation(), CRC32: in.Node().Name().CRC32(), @@ -79,21 +103,6 @@ func (in *node) HandleMessage(from gen.PID, message any) error { in.SendResponse(m.pid, m.ref, response) in.Log().Debug("sent response for the inspect node request to: %s", m.pid) - case register: - eopts := gen.EventOptions{ - Notify: true, - Buffer: 1, // keep the last event - } - token, err := in.RegisterEvent(inspectNode, eopts) - if err != nil { - in.Log().Error("unable to register event: %s", err) - return err - } - in.Log().Info("registered event %s", inspectNode) - - in.token = token - in.SendAfter(in.PID(), shutdown{}, inspectNodeIdlePeriod) - case shutdown: if in.generating { in.Log().Debug("ignore shutdown. generating is active") @@ -103,14 +112,14 @@ func (in *node) HandleMessage(from gen.PID, message any) error { case gen.MessageEventStart: // got first subscriber in.Log().Debug("got first subscriber. start generating events...") - in.Send(in.PID(), generate{}) + in.loopID++ + in.Send(in.PID(), generate{id: in.loopID}) in.generating = true case gen.MessageEventStop: // no subscribers in.Log().Debug("no subscribers. stop generating") if in.generating { in.generating = false - // wait 10 seconds and terminate this process in.SendAfter(in.PID(), shutdown{}, inspectNodeIdlePeriod) } diff --git a/app/system/inspect/process.go b/app/system/inspect/process.go index 9b94092b8..c0fdb524b 100644 --- a/app/system/inspect/process.go +++ b/app/system/inspect/process.go @@ -18,29 +18,47 @@ type process struct { event gen.Atom pid gen.PID generating bool + loopID uint64 } func (ip *process) Init(args ...any) error { ip.pid = args[0].(gen.PID) ip.Log().SetLogger("default") ip.Log().Debug("process inspector started. pid %s", ip.pid) - // RegisterEvent is not allowed here - ip.Send(ip.PID(), register{}) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, // keep the last event + } + evname := gen.Atom(fmt.Sprintf("%s_%s", inspectProcess, ip.pid)) + token, err := ip.RegisterEvent(evname, eopts) + if err != nil { + ip.Log().Error("unable to register event: %s", err) + return err + } + ip.Log().Info("registered event %s", evname) + ip.event = evname + ip.token = token + ip.SendAfter(ip.PID(), shutdown{}, inspectProcessIdlePeriod) + return nil } func (ip *process) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { case generate: - if ip.generating == false { + if m.id != ip.loopID || ip.generating == false { ip.Log().Debug("generating canceled") break // cancelled } ip.Log().Debug("generating event") ev := MessageInspectProcess{ - Node: ip.Node().Name(), - Terminated: true, + Node: ip.Node().Name(), + Info: gen.ProcessInfo{ + PID: ip.pid, + State: gen.ProcessStateTerminated, + }, } info, err := ip.Node().ProcessInfo(ip.pid) @@ -59,7 +77,7 @@ func (ip *process) HandleMessage(from gen.PID, message any) error { default: ip.Log().Error("unable to inspect process %s: %s", ip.pid, err) // will try next time (seems to be busy) - ip.SendAfter(ip.PID(), generate{}, inspectProcessPeriod) + ip.SendAfter(ip.PID(), generate{id: ip.loopID}, inspectProcessPeriod) return nil } @@ -67,7 +85,6 @@ func (ip *process) HandleMessage(from gen.PID, message any) error { info.Env[k] = fmt.Sprintf("%#v", v) } - ev.Terminated = false ev.Info = info if err := ip.SendEvent(ip.event, ip.token, ev); err != nil { @@ -75,7 +92,7 @@ func (ip *process) HandleMessage(from gen.PID, message any) error { return gen.TerminateReasonNormal } - ip.SendAfter(ip.PID(), generate{}, inspectProcessPeriod) + ip.SendAfter(ip.PID(), generate{id: ip.loopID}, inspectProcessPeriod) case requestInspect: response := ResponseInspectProcess{ @@ -87,23 +104,6 @@ func (ip *process) HandleMessage(from gen.PID, message any) error { ip.SendResponse(m.pid, m.ref, response) ip.Log().Debug("sent response for the inspect process request to: %s", m.pid) - case register: - eopts := gen.EventOptions{ - Notify: true, - Buffer: 1, // keep the last event - } - evname := gen.Atom(fmt.Sprintf("%s_%s", inspectProcess, ip.pid)) - token, err := ip.RegisterEvent(evname, eopts) - if err != nil { - ip.Log().Error("unable to register event: %s", err) - return err - } - ip.Log().Info("registered event %s", evname) - ip.event = evname - - ip.token = token - ip.SendAfter(ip.PID(), shutdown{}, inspectProcessIdlePeriod) - case shutdown: if ip.generating { ip.Log().Debug("ignore shutdown. generating is active") @@ -113,7 +113,8 @@ func (ip *process) HandleMessage(from gen.PID, message any) error { case gen.MessageEventStart: // got first subscriber ip.Log().Debug("got first subscriber. start generating events...") - ip.Send(ip.PID(), generate{}) + ip.loopID++ + ip.Send(ip.PID(), generate{id: ip.loopID}) ip.generating = true case gen.MessageEventStop: // no subscribers diff --git a/app/system/inspect/process_list.go b/app/system/inspect/process_list.go index 9380337bd..e8c2d4c67 100644 --- a/app/system/inspect/process_list.go +++ b/app/system/inspect/process_list.go @@ -3,6 +3,7 @@ package inspect import ( "fmt" "slices" + "strings" "ergo.services/ergo/act" "ergo.services/ergo/gen" @@ -16,31 +17,66 @@ type process_list struct { act.Actor token gen.Ref - start int - limit int + start int + limit int + name string + behavior string + application string + state string + minMailbox uint64 + generating bool + loopID uint64 event gen.Atom } func (ipl *process_list) Init(args ...any) error { ipl.start = args[0].(int) ipl.limit = args[1].(int) + ipl.name = args[2].(string) + ipl.behavior = args[3].(string) + ipl.application = args[4].(string) + ipl.state = args[5].(string) + ipl.minMailbox = args[6].(uint64) + ipl.Log().SetLogger("default") ipl.Log().Debug("process list inspector started. %d...%d", ipl.start, ipl.start+ipl.limit-1) - // RegisterEvent is not allowed here - ipl.Send(ipl.PID(), register{}) + ipl.SetCompression(true) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, + } + hash := filterHash(ipl.name, ipl.behavior, ipl.application, ipl.state, ipl.minMailbox, ipl.limit) + evname := gen.Atom(fmt.Sprintf("%s_%d_%s", inspectProcessList, ipl.start, hash)) + token, err := ipl.RegisterEvent(evname, eopts) + if err != nil { + ipl.Log().Error("unable to register event: %s", err) + return err + } + ipl.Log().Info("registered event %s", evname) + ipl.event = evname + ipl.token = token + ipl.SendAfter(ipl.PID(), shutdown{}, inspectProcessListIdlePeriod) + return nil } + func (ipl *process_list) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { case generate: - if ipl.generating == false { + if m.id != ipl.loopID || ipl.generating == false { ipl.Log().Debug("generating canceled") - break // cancelled + break } ipl.Log().Debug("generating event") - list, err := ipl.Node().ProcessListShortInfo(ipl.start, ipl.limit) + var filter []func(gen.ProcessShortInfo) bool + if ipl.hasFilters() { + filter = append(filter, ipl.matchFilter) + } + + list, err := ipl.Node().ProcessListShortInfo(ipl.start, ipl.limit, filter...) if err != nil { return err } @@ -59,7 +95,7 @@ func (ipl *process_list) HandleMessage(from gen.PID, message any) error { return gen.TerminateReasonNormal } - ipl.SendAfter(ipl.PID(), generate{}, inspectProcessListPeriod) + ipl.SendAfter(ipl.PID(), generate{id: ipl.loopID}, inspectProcessListPeriod) case requestInspect: response := ResponseInspectProcessList{ @@ -71,36 +107,20 @@ func (ipl *process_list) HandleMessage(from gen.PID, message any) error { ipl.SendResponse(m.pid, m.ref, response) ipl.Log().Debug("sent response for the inspect process list request to: %s", m.pid) - case register: - eopts := gen.EventOptions{ - Notify: true, - Buffer: 1, // keep the last event - } - evname := gen.Atom(fmt.Sprintf("%s_%d_%d", inspectProcessList, ipl.start, ipl.start+ipl.limit-1)) - token, err := ipl.RegisterEvent(evname, eopts) - if err != nil { - ipl.Log().Error("unable to register event: %s", err) - return err - } - ipl.Log().Info("registered event %s", evname) - ipl.event = evname - - ipl.token = token - ipl.SendAfter(ipl.PID(), shutdown{}, inspectProcessListIdlePeriod) - case shutdown: if ipl.generating { ipl.Log().Debug("ignore shutdown. generating is active") - break // ignore. + break } return gen.TerminateReasonNormal - case gen.MessageEventStart: // got first subscriber + case gen.MessageEventStart: ipl.Log().Debug("got first subscriber. start generating events...") - ipl.Send(ipl.PID(), generate{}) + ipl.loopID++ + ipl.Send(ipl.PID(), generate{id: ipl.loopID}) ipl.generating = true - case gen.MessageEventStop: // no subscribers + case gen.MessageEventStop: ipl.Log().Debug("no subscribers. stop generating") if ipl.generating { ipl.generating = false @@ -117,3 +137,26 @@ func (ipl *process_list) HandleMessage(from gen.PID, message any) error { func (ipl *process_list) Terminate(reason error) { ipl.Log().Debug("process list inspector terminated: %s", reason) } + +func (ipl *process_list) hasFilters() bool { + return ipl.name != "" || ipl.behavior != "" || ipl.application != "" || ipl.state != "" || ipl.minMailbox > 0 +} + +func (ipl *process_list) matchFilter(info gen.ProcessShortInfo) bool { + if ipl.name != "" && strings.Contains(strings.ToLower(string(info.Name)), strings.ToLower(ipl.name)) == false { + return false + } + if ipl.behavior != "" && strings.Contains(strings.ToLower(info.Behavior), strings.ToLower(ipl.behavior)) == false { + return false + } + if ipl.application != "" && strings.Contains(strings.ToLower(string(info.Application)), strings.ToLower(ipl.application)) == false { + return false + } + if ipl.state != "" && strings.EqualFold(info.State.String(), ipl.state) == false { + return false + } + if ipl.minMailbox > 0 && info.MessagesMailbox < ipl.minMailbox { + return false + } + return true +} diff --git a/app/system/inspect/process_range.go b/app/system/inspect/process_range.go new file mode 100644 index 000000000..9fbe27bac --- /dev/null +++ b/app/system/inspect/process_range.go @@ -0,0 +1,191 @@ +package inspect + +import ( + "fmt" + "slices" + "strings" + + "ergo.services/ergo/act" + "ergo.services/ergo/gen" +) + +func factory_process_range() gen.ProcessBehavior { + return &process_range{} +} + +type process_range struct { + act.Actor + token gen.Ref + + name string + behavior string + application string + state string + minMailbox uint64 + limit int + hash string + + generating bool + loopID uint64 + event gen.Atom +} + +func (ipr *process_range) Init(args ...any) error { + ipr.name = args[0].(string) + ipr.behavior = args[1].(string) + ipr.application = args[2].(string) + ipr.state = args[3].(string) + ipr.minMailbox = args[4].(uint64) + ipr.limit = args[5].(int) + ipr.hash = args[6].(string) + + ipr.Log().SetLogger("default") + ipr.Log().Debug("process range inspector started. name=%q behavior=%q app=%q state=%q mailbox>=%d limit=%d", + ipr.name, ipr.behavior, ipr.application, ipr.state, ipr.minMailbox, ipr.limit) + ipr.SetCompression(true) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, + } + ipr.event = gen.Atom(fmt.Sprintf("%s_%s", inspectProcessRange, ipr.hash)) + token, err := ipr.RegisterEvent(ipr.event, eopts) + if err != nil { + ipr.Log().Error("unable to register event: %s", err) + return err + } + ipr.Log().Info("registered event %s", ipr.event) + ipr.token = token + ipr.SendAfter(ipr.PID(), shutdown{}, inspectProcessRangeIdlePeriod) + + return nil +} + +func (ipr *process_range) HandleMessage(from gen.PID, message any) error { + switch m := message.(type) { + case generate: + if m.id != ipr.loopID || ipr.generating == false { + break + } + + var list []gen.ProcessShortInfo + nameLower := strings.ToLower(ipr.name) + behaviorLower := strings.ToLower(ipr.behavior) + appLower := strings.ToLower(ipr.application) + + ipr.Node().ProcessRangeShortInfo(func(info gen.ProcessShortInfo) bool { + // apply filters + if nameLower != "" { + if strings.Contains(strings.ToLower(string(info.Name)), nameLower) == false { + return true // skip, continue + } + } + if behaviorLower != "" { + if strings.Contains(strings.ToLower(info.Behavior), behaviorLower) == false { + return true + } + } + if appLower != "" { + if strings.Contains(strings.ToLower(string(info.Application)), appLower) == false { + return true + } + } + if ipr.state != "" { + if strings.EqualFold(info.State.String(), ipr.state) == false { + return true + } + } + if ipr.minMailbox > 0 { + if info.MessagesMailbox < ipr.minMailbox { + return true + } + } + + list = append(list, info) + + if ipr.limit > 0 && len(list) >= ipr.limit { + return false // stop iteration + } + return true + }) + + slices.SortStableFunc(list, func(a, b gen.ProcessShortInfo) int { + return int(a.PID.ID - b.PID.ID) + }) + + // reuse MessageInspectProcessList, same payload format + ev := MessageInspectProcessList{ + Node: ipr.Node().Name(), + Processes: list, + } + + if err := ipr.SendEvent(ipr.event, ipr.token, ev); err != nil { + ipr.Log().Error("unable to send event %q: %s", ipr.event, err) + return gen.TerminateReasonNormal + } + + ipr.SendAfter(ipr.PID(), generate{id: ipr.loopID}, inspectProcessRangePeriod) + + case requestInspect: + response := ResponseInspectProcessRange{ + Event: gen.Event{ + Name: ipr.event, + Node: ipr.Node().Name(), + }, + } + ipr.SendResponse(m.pid, m.ref, response) + + case shutdown: + if ipr.generating { + break + } + return gen.TerminateReasonNormal + + case gen.MessageEventStart: + ipr.Log().Debug("got first subscriber. start generating events...") + ipr.loopID++ + ipr.Send(ipr.PID(), generate{id: ipr.loopID}) + ipr.generating = true + + case gen.MessageEventStop: + ipr.Log().Debug("no subscribers. stop generating") + if ipr.generating { + ipr.generating = false + ipr.SendAfter(ipr.PID(), shutdown{}, inspectProcessRangeIdlePeriod) + } + + default: + ipr.Log().Error("unknown message (ignored) %#v", message) + } + + return nil +} + +func (ipr *process_range) Terminate(reason error) { + ipr.Log().Debug("process range inspector terminated: %s", reason) +} + +// filterHash builds a short deterministic suffix from filter fields +func filterHash(name, behavior, application, state string, minMailbox uint64, limit int) string { + return fmt.Sprintf("%x", hashStr(fmt.Sprintf("%s|%s|%s|%s|%d|%d", + name, behavior, application, state, minMailbox, limit))) +} + +// eventListHash builds a short deterministic suffix from event list filter fields +func eventListHash(timestamp int64, name string, notify, buffered, open int, minSubscribers int64, limit int) string { + return fmt.Sprintf("%x", hashStr(fmt.Sprintf("%d|%s|%d|%d|%d|%d|%d", + timestamp, name, notify, buffered, open, minSubscribers, limit))) +} + +func connectionListHash(name string, limit int) string { + return fmt.Sprintf("%x", hashStr(fmt.Sprintf("%s|%d", name, limit))) +} + +func hashStr(s string) uint32 { + h := uint32(2166136261) + for i := 0; i < len(s); i++ { + h ^= uint32(s[i]) + h *= 16777619 + } + return h +} diff --git a/app/system/inspect/process_state.go b/app/system/inspect/process_state.go index 4c3badad9..655ac511b 100644 --- a/app/system/inspect/process_state.go +++ b/app/system/inspect/process_state.go @@ -17,6 +17,7 @@ type process_state struct { event gen.Atom generating bool + loopID uint64 pid gen.PID } @@ -24,15 +25,29 @@ func (ips *process_state) Init(args ...any) error { ips.pid = args[0].(gen.PID) ips.Log().SetLogger("default") ips.Log().Debug("process state inspector started. pid %s", ips.pid) - // RegisterEvent is not allowed here - ips.Send(ips.PID(), register{}) + + eopts := gen.EventOptions{ + Notify: true, + Buffer: 1, // keep the last event + } + evname := gen.Atom(fmt.Sprintf("%s_%s", inspectProcessState, ips.pid)) + token, err := ips.RegisterEvent(evname, eopts) + if err != nil { + ips.Log().Error("unable to register process state event: %s", err) + return err + } + ips.Log().Info("registered event %s", evname) + ips.event = evname + ips.token = token + ips.SendAfter(ips.PID(), shutdown{}, inspectProcessStateIdlePeriod) + return nil } func (ips *process_state) HandleMessage(from gen.PID, message any) error { switch m := message.(type) { case generate: - if ips.generating == false { + if m.id != ips.loopID || ips.generating == false { ips.Log().Debug("generating canceled") break // cancelled } @@ -44,7 +59,7 @@ func (ips *process_state) HandleMessage(from gen.PID, message any) error { } ips.Log().Error("unable to inspect process state %s: %s", ips.pid, err) // will try next time - ips.SendAfter(ips.PID(), generate{}, inspectProcessStatePeriod) + ips.SendAfter(ips.PID(), generate{id: ips.loopID}, inspectProcessStatePeriod) return nil } @@ -59,7 +74,7 @@ func (ips *process_state) HandleMessage(from gen.PID, message any) error { return gen.TerminateReasonNormal } - ips.SendAfter(ips.PID(), generate{}, inspectProcessStatePeriod) + ips.SendAfter(ips.PID(), generate{id: ips.loopID}, inspectProcessStatePeriod) case requestInspect: response := ResponseInspectProcessState{ @@ -71,23 +86,6 @@ func (ips *process_state) HandleMessage(from gen.PID, message any) error { ips.SendResponse(m.pid, m.ref, response) ips.Log().Debug("sent response for the inspect process state %s request to: %s", ips.pid, m.pid) - case register: - eopts := gen.EventOptions{ - Notify: true, - Buffer: 1, // keep the last event - } - evname := gen.Atom(fmt.Sprintf("%s_%s", inspectProcessState, ips.pid)) - token, err := ips.RegisterEvent(evname, eopts) - if err != nil { - ips.Log().Error("unable to register process state event: %s", err) - return err - } - ips.Log().Info("registered event %s", evname) - ips.event = evname - - ips.token = token - ips.SendAfter(ips.PID(), shutdown{}, inspectProcessStateIdlePeriod) - case shutdown: if ips.generating { ips.Log().Debug("ignore shutdown. generating is active") @@ -97,7 +95,8 @@ func (ips *process_state) HandleMessage(from gen.PID, message any) error { case gen.MessageEventStart: // got first subscriber ips.Log().Debug("got first subscriber. start generating events...") - ips.Send(ips.PID(), generate{}) + ips.loopID++ + ips.Send(ips.PID(), generate{id: ips.loopID}) ips.generating = true case gen.MessageEventStop: // no subscribers diff --git a/app/system/inspect/tracing.go b/app/system/inspect/tracing.go new file mode 100644 index 000000000..085c9e516 --- /dev/null +++ b/app/system/inspect/tracing.go @@ -0,0 +1,189 @@ +package inspect + +import ( + "fmt" + "strings" + "time" + + "ergo.services/ergo/act" + "ergo.services/ergo/gen" +) + +func factory_tracing() gen.ProcessBehavior { + return &tracing{} +} + +type tracing struct { + act.Actor + token gen.Ref + event gen.Atom + + flags gen.TracingFlags + limit int + kinds uint32 + points uint32 + messagePattern string + messageExclude bool + generating bool + loopID uint64 + + // ring buffer + ring []gen.TracingSpan + pos int + full bool + received int64 +} + +type flushTracing struct{ id uint64 } + +const tracingFlushInterval = time.Second +const inspectTracingIdlePeriod = 10 * time.Second + +func (it *tracing) Init(args ...any) error { + it.flags = args[0].(gen.TracingFlags) + it.limit = args[1].(int) + if it.limit < 1 { + it.limit = 500 + } + it.kinds = args[2].(uint32) + it.points = args[3].(uint32) + it.messagePattern = args[4].(string) + it.messageExclude = args[5].(bool) + it.ring = make([]gen.TracingSpan, it.limit) + it.Log().Debug("tracing inspector started (limit: %d)", it.limit) + it.SetCompression(true) + + eopts := gen.EventOptions{ + Notify: true, + } + evname := gen.Atom(fmt.Sprintf("%s_%s", string(it.Name()), it.PID())) + token, err := it.RegisterEvent(evname, eopts) + if err != nil { + return err + } + + it.event = evname + it.token = token + it.SendAfter(it.PID(), shutdown{}, inspectTracingIdlePeriod) + + return nil +} + +func (it *tracing) HandleMessage(from gen.PID, message any) error { + switch m := message.(type) { + case flushTracing: + if m.id != it.loopID || it.generating == false { + break + } + if it.received == 0 { + it.SendAfter(it.PID(), flushTracing{id: it.loopID}, tracingFlushInterval) + break + } + + var spans []gen.TracingSpan + if it.full { + spans = make([]gen.TracingSpan, it.limit) + copy(spans, it.ring[it.pos:]) + copy(spans[it.limit-it.pos:], it.ring[:it.pos]) + } else { + spans = make([]gen.TracingSpan, it.pos) + copy(spans, it.ring[:it.pos]) + } + + suppressed := it.received - int64(len(spans)) + if suppressed < 0 { + suppressed = 0 + } + + ev := MessageInspectTracing{ + Node: it.Node().Name(), + Spans: spans, + Suppressed: suppressed, + } + + it.pos = 0 + it.full = false + it.received = 0 + + if err := it.SendEvent(it.event, it.token, ev); err != nil { + return gen.TerminateReasonNormal + } + + it.SendAfter(it.PID(), flushTracing{id: it.loopID}, tracingFlushInterval) + + case requestInspect: + response := ResponseInspectTracing{ + Event: gen.Event{ + Name: it.event, + Node: it.Node().Name(), + }, + } + it.SendResponse(m.pid, m.ref, response) + + case shutdown: + if it.generating { + break + } + return gen.TerminateReasonNormal + + case gen.MessageEventStart: + it.Log().Debug("registering as tracing exporter") + it.Node().TracingExporterAddPID(it.PID(), it.PID().String(), it.flags) + it.loopID++ + it.generating = true + it.SendAfter(it.PID(), flushTracing{id: it.loopID}, tracingFlushInterval) + + case gen.MessageEventStop: + it.Node().TracingExporterDeletePID(it.PID()) + it.Log().Debug("removed as tracing exporter") + it.generating = false + it.pos = 0 + it.full = false + it.received = 0 + it.SendAfter(it.PID(), shutdown{}, inspectTracingIdlePeriod) + } + + return nil +} + +func (it *tracing) HandleSpan(span gen.TracingSpan) error { + // kind filter: bitmask 1=send, 2=request, 4=response, 8=spawn, 16=terminate + if it.kinds != 0 && it.kinds != 31 { + kindBit := uint32(1) << (uint32(span.Kind) - 1) + if it.kinds&kindBit == 0 { + return nil + } + } + + // point filter: bitmask 1=sent, 2=delivered, 4=processed + if it.points != 0 && it.points != 7 { + pointBit := uint32(1) << (uint32(span.Point) - 1) + if it.points&pointBit == 0 { + return nil + } + } + + if it.messagePattern != "" { + match := strings.Contains(span.Message, it.messagePattern) || + strings.Contains(span.Error, it.messagePattern) + if it.messageExclude == true && match == true { + return nil + } + if it.messageExclude == false && match == false { + return nil + } + } + + it.ring[it.pos] = span + it.pos++ + if it.pos >= it.limit { + it.pos = 0 + it.full = true + } + it.received++ + return nil +} + +func (it *tracing) Terminate(reason error) { + it.Node().TracingExporterDeletePID(it.PID()) +} diff --git a/app/system/metrics.go b/app/system/metrics.go deleted file mode 100644 index 9736367d2..000000000 --- a/app/system/metrics.go +++ /dev/null @@ -1,205 +0,0 @@ -package system - -import ( - "bytes" - "crypto/aes" - "crypto/cipher" - "crypto/rand" - "crypto/rsa" - "crypto/sha256" - "crypto/x509" - "encoding/base64" - "encoding/binary" - "fmt" - "io" - "net" - "runtime" - "strconv" - "strings" - "time" - - "ergo.services/ergo/act" - "ergo.services/ergo/gen" - "ergo.services/ergo/lib" - "ergo.services/ergo/net/edf" -) - -const ( - period time.Duration = time.Second * 300 - - DISABLE_METRICS gen.Env = "disable_metrics" -) - -type MessageMetrics struct { - Name gen.Atom - Creation int64 - Uptime int64 - Arch string - OS string - NumCPU int - GoVersion string - Version string - ErgoVersion string - Commercial string -} - -func factory_metrics() gen.ProcessBehavior { - return &metrics{} -} - -type doSendMetrics struct{} - -type metrics struct { - act.Actor - cancelSend gen.CancelFunc - key []byte - block cipher.Block -} - -func (m *metrics) Init(args ...any) error { - if err := edf.RegisterTypeOf(MessageMetrics{}); err != nil { - if err != gen.ErrTaken { - return err - } - } - - if _, disabled := m.Env(DISABLE_METRICS); disabled { - if comm := m.Node().Commercial(); len(comm) == 0 { - m.Log().Trace("metrics disabled") - return nil - } - m.Log().Trace("a commercial package is used. enforce sending metrics") - } - - m.key = []byte(lib.RandomString(32)) - b, err := aes.NewCipher(m.key) - if err != nil { - return nil - } - m.block = b - - m.Log().Trace("scheduled sending metrics in %v", period) - m.cancelSend, _ = m.SendAfter(m.PID(), doSendMetrics{}, period) - return nil -} - -func (m *metrics) HandleMessage(from gen.PID, message any) error { - - switch message.(type) { - case doSendMetrics: - m.send() - m.Log().Trace("scheduled sending metrics in %v", period) - m.cancelSend, _ = m.SendAfter(m.PID(), doSendMetrics{}, period) - - default: - m.Log().Trace("received unknown message: %#v", message) - } - return nil -} - -func (m *metrics) Terminate(reason error) { - if m.cancelSend == nil { - return - } - m.cancelSend() -} - -func (m *metrics) send() { - var msrv = "metrics.ergo.services" - - values, err := net.LookupTXT(msrv) - if err != nil || len(values) == 0 { - m.Log().Trace("lookup TXT record in %s failed or returned empty result", msrv) - return - } - v, err := base64.StdEncoding.DecodeString(values[0]) - if err != nil { - return - } - - pk, err := x509.ParsePKCS1PublicKey([]byte(v)) - if err != nil { - m.Log().Trace("unable to parse public key (TXT record in %s)", msrv) - return - } - - _, srv, err := net.LookupSRV("data", "mt1", msrv) - if err != nil || len(srv) == 0 { - m.Log().Trace("unable to resolve SRV record: %s", err) - return - } - - dsn := net.JoinHostPort(strings.TrimSuffix(srv[0].Target, "."), - strconv.Itoa(int(srv[0].Port))) - c, err := net.Dial("udp", dsn) - if err != nil { - m.Log().Trace("unable to dial the host %s: %s", dsn, err) - return - } - defer c.Close() - - msg := MessageMetrics{ - Name: m.Node().Name(), - Creation: m.Node().Creation(), - Uptime: m.Node().Uptime(), - Arch: runtime.GOARCH, - OS: runtime.GOOS, - NumCPU: runtime.NumCPU(), - GoVersion: runtime.Version(), - Version: m.Node().Version().String(), - ErgoVersion: m.Node().FrameworkVersion().String(), - Commercial: fmt.Sprintf("%v", m.Node().Commercial()), - } - - buf := lib.TakeBuffer() - defer lib.ReleaseBuffer(buf) - - hash := sha256.New() - cipher, err := rsa.EncryptOAEP(hash, rand.Reader, pk, m.key, nil) - if err != nil { - m.Log().Trace("unable to encrypt metrics message: %s (len: %d)", err, buf.Len()) - return - } - - // 2 (magic: 1144) + 2 (length) + len(cipher) - buf.Allocate(4) - buf.Append(cipher) - binary.BigEndian.PutUint16(buf.B[0:2], uint16(1144)) - binary.BigEndian.PutUint16(buf.B[2:4], uint16(len(cipher))) - - // encrypt payload and append to the buf - payload := lib.TakeBuffer() - defer lib.ReleaseBuffer(payload) - if err := edf.Encode(msg, payload, edf.Options{}); err != nil { - m.Log().Trace("unable to encode metrics message: %s", err) - return - } - - x := encrypt(payload.B, m.block) - if x == nil { - return - } - buf.Append(x) - - if _, err := c.Write(buf.B); err != nil { - m.Log().Trace("unable to send metrics: %s", err) - } - m.Log().Trace("sent metrics to %s", dsn) -} - -func encrypt(data []byte, block cipher.Block) []byte { - l := len(data) - padding := aes.BlockSize - l%aes.BlockSize - padtext := bytes.Repeat([]byte{byte(padding)}, padding) - data = append(data, padtext...) - l = len(data) - - x := make([]byte, aes.BlockSize+l) - iv := x[:aes.BlockSize] - if _, err := io.ReadFull(rand.Reader, iv); err != nil { - return nil - } - cfb := cipher.NewCFBEncrypter(block, iv) - cfb.XORKeyStream(x[aes.BlockSize:], data) - return x -} diff --git a/app/system/sup.go b/app/system/sup.go index 7254d190b..1fae1de87 100644 --- a/app/system/sup.go +++ b/app/system/sup.go @@ -19,10 +19,6 @@ func (s *sup) Init(args ...any) (act.SupervisorSpec, error) { spec := act.SupervisorSpec{ Type: act.SupervisorTypeOneForOne, Children: []act.SupervisorChildSpec{ - { - Factory: factory_metrics, - Name: "system_metrics", - }, { Factory: inspect.Factory, Name: inspect.Name, diff --git a/docs/.gitbook/assets/Screenshot from 2023-08-18 16-57-50.png b/docs/.gitbook/assets/Screenshot from 2023-08-18 16-57-50.png deleted file mode 100644 index 2e61f679f..000000000 Binary files a/docs/.gitbook/assets/Screenshot from 2023-08-18 16-57-50.png and /dev/null differ diff --git a/docs/.gitbook/assets/Screenshot from 2023-08-30 23-00-56.png b/docs/.gitbook/assets/Screenshot from 2023-08-30 23-00-56.png deleted file mode 100644 index 3aa8dd240..000000000 Binary files a/docs/.gitbook/assets/Screenshot from 2023-08-30 23-00-56.png and /dev/null differ diff --git a/docs/.gitbook/assets/Screenshot from 2023-08-30 23-07-52.png b/docs/.gitbook/assets/Screenshot from 2023-08-30 23-07-52.png deleted file mode 100644 index 478212071..000000000 Binary files a/docs/.gitbook/assets/Screenshot from 2023-08-30 23-07-52.png and /dev/null differ diff --git a/docs/.gitbook/assets/image (1).png b/docs/.gitbook/assets/image (1).png deleted file mode 100644 index 113a50770..000000000 Binary files a/docs/.gitbook/assets/image (1).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (10).png b/docs/.gitbook/assets/image (10).png deleted file mode 100644 index 2e61f679f..000000000 Binary files a/docs/.gitbook/assets/image (10).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (11).png b/docs/.gitbook/assets/image (11).png deleted file mode 100644 index baeb2a358..000000000 Binary files a/docs/.gitbook/assets/image (11).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (12).png b/docs/.gitbook/assets/image (12).png deleted file mode 100644 index 292cc7164..000000000 Binary files a/docs/.gitbook/assets/image (12).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (14).png b/docs/.gitbook/assets/image (14).png deleted file mode 100644 index d33192cf7..000000000 Binary files a/docs/.gitbook/assets/image (14).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (15).png b/docs/.gitbook/assets/image (15).png deleted file mode 100644 index d33192cf7..000000000 Binary files a/docs/.gitbook/assets/image (15).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (16).png b/docs/.gitbook/assets/image (16).png deleted file mode 100644 index d33192cf7..000000000 Binary files a/docs/.gitbook/assets/image (16).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (17).png b/docs/.gitbook/assets/image (17).png deleted file mode 100644 index 6b08bb983..000000000 Binary files a/docs/.gitbook/assets/image (17).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (18).png b/docs/.gitbook/assets/image (18).png deleted file mode 100644 index 0d9c81ea6..000000000 Binary files a/docs/.gitbook/assets/image (18).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (19).png b/docs/.gitbook/assets/image (19).png deleted file mode 100644 index 8e28afc0d..000000000 Binary files a/docs/.gitbook/assets/image (19).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (2).png b/docs/.gitbook/assets/image (2).png deleted file mode 100644 index 6d96f5c9f..000000000 Binary files a/docs/.gitbook/assets/image (2).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (20).png b/docs/.gitbook/assets/image (20).png deleted file mode 100644 index 1c2c457ce..000000000 Binary files a/docs/.gitbook/assets/image (20).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (21).png b/docs/.gitbook/assets/image (21).png deleted file mode 100644 index 839386179..000000000 Binary files a/docs/.gitbook/assets/image (21).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (22).png b/docs/.gitbook/assets/image (22).png deleted file mode 100644 index cfdc5af2a..000000000 Binary files a/docs/.gitbook/assets/image (22).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (23).png b/docs/.gitbook/assets/image (23).png deleted file mode 100644 index 19f8b888c..000000000 Binary files a/docs/.gitbook/assets/image (23).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (24).png b/docs/.gitbook/assets/image (24).png deleted file mode 100644 index afe15170d..000000000 Binary files a/docs/.gitbook/assets/image (24).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (25).png b/docs/.gitbook/assets/image (25).png deleted file mode 100644 index 47d23e60d..000000000 Binary files a/docs/.gitbook/assets/image (25).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (26).png b/docs/.gitbook/assets/image (26).png deleted file mode 100644 index e925c9365..000000000 Binary files a/docs/.gitbook/assets/image (26).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (27).png b/docs/.gitbook/assets/image (27).png deleted file mode 100644 index eb4bcaeb9..000000000 Binary files a/docs/.gitbook/assets/image (27).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (28).png b/docs/.gitbook/assets/image (28).png deleted file mode 100644 index 42b85ddf1..000000000 Binary files a/docs/.gitbook/assets/image (28).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (29).png b/docs/.gitbook/assets/image (29).png deleted file mode 100644 index 449a7e94b..000000000 Binary files a/docs/.gitbook/assets/image (29).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (3).png b/docs/.gitbook/assets/image (3).png deleted file mode 100644 index 22f1c6bb3..000000000 Binary files a/docs/.gitbook/assets/image (3).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (30).png b/docs/.gitbook/assets/image (30).png deleted file mode 100644 index c30066eb4..000000000 Binary files a/docs/.gitbook/assets/image (30).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (31).png b/docs/.gitbook/assets/image (31).png deleted file mode 100644 index cb50d3426..000000000 Binary files a/docs/.gitbook/assets/image (31).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (32).png b/docs/.gitbook/assets/image (32).png deleted file mode 100644 index d9abc8822..000000000 Binary files a/docs/.gitbook/assets/image (32).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (33).png b/docs/.gitbook/assets/image (33).png deleted file mode 100644 index dc8690c1b..000000000 Binary files a/docs/.gitbook/assets/image (33).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (34).png b/docs/.gitbook/assets/image (34).png deleted file mode 100644 index 8de8a9144..000000000 Binary files a/docs/.gitbook/assets/image (34).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (35).png b/docs/.gitbook/assets/image (35).png deleted file mode 100644 index 8a2d435d0..000000000 Binary files a/docs/.gitbook/assets/image (35).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (36).png b/docs/.gitbook/assets/image (36).png deleted file mode 100644 index 8a2d435d0..000000000 Binary files a/docs/.gitbook/assets/image (36).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (37).png b/docs/.gitbook/assets/image (37).png deleted file mode 100644 index 138a90e57..000000000 Binary files a/docs/.gitbook/assets/image (37).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (38).png b/docs/.gitbook/assets/image (38).png deleted file mode 100644 index d90445547..000000000 Binary files a/docs/.gitbook/assets/image (38).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (39).png b/docs/.gitbook/assets/image (39).png deleted file mode 100644 index 7da04f4e8..000000000 Binary files a/docs/.gitbook/assets/image (39).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (4).png b/docs/.gitbook/assets/image (4).png deleted file mode 100644 index 73f534e64..000000000 Binary files a/docs/.gitbook/assets/image (4).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (5).png b/docs/.gitbook/assets/image (5).png deleted file mode 100644 index aaaaca99c..000000000 Binary files a/docs/.gitbook/assets/image (5).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (6).png b/docs/.gitbook/assets/image (6).png deleted file mode 100644 index aaaaca99c..000000000 Binary files a/docs/.gitbook/assets/image (6).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (7).png b/docs/.gitbook/assets/image (7).png deleted file mode 100644 index 93ab08cb4..000000000 Binary files a/docs/.gitbook/assets/image (7).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (8).png b/docs/.gitbook/assets/image (8).png deleted file mode 100644 index 33ea6bf57..000000000 Binary files a/docs/.gitbook/assets/image (8).png and /dev/null differ diff --git a/docs/.gitbook/assets/image (9).png b/docs/.gitbook/assets/image (9).png deleted file mode 100644 index 642d00047..000000000 Binary files a/docs/.gitbook/assets/image (9).png and /dev/null differ diff --git a/docs/.gitbook/assets/observer.png b/docs/.gitbook/assets/observer.png new file mode 100644 index 000000000..903ac0561 Binary files /dev/null and b/docs/.gitbook/assets/observer.png differ diff --git a/docs/README.md b/docs/README.md index 6b3a38e08..9aa1ae096 100644 --- a/docs/README.md +++ b/docs/README.md @@ -46,7 +46,7 @@ Benchmarks measuring message passing, network communication, and serialization p ## Zero Dependencies -The framework uses only the Go standard library. No external dependencies means no version conflicts, no supply chain vulnerabilities, no surprise breaking changes from third-party packages. The requirement is just Go 1.20 or higher. +The framework uses only the Go standard library. No external dependencies means no version conflicts, no supply chain vulnerabilities, no surprise breaking changes from third-party packages. The requirement is just Go 1.21 or higher. This isn't ideological purity. It's practical stability. The framework's behavior depends only on Go itself. Updates are predictable. Supply chain is simple. The code you write today will compile and run the same way years from now, assuming Go maintains backward compatibility (which it does). diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index ec15d0c42..fad927bdb 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -1,6 +1,8 @@ # Table of contents * [Overview](README.md) +* [FAQ](faq.md) +* [AI Agents](ai-agents.md) ## Basics @@ -56,14 +58,20 @@ * [Message Versioning](advanced/message-versioning.md) * [Building a Cluster](advanced/building-a-cluster.md) * [Debugging](advanced/debugging.md) +* [Distributed Tracing](advanced/distributed-tracing.md) +* [Inspecting With Observer](advanced/observer.md) ## extra library * [Actors](extra-library/actors/README.md) + * [Health](extra-library/actors/health.md) * [Leader](extra-library/actors/leader.md) * [Metrics](extra-library/actors/metrics.md) * [Applications](extra-library/applications/README.md) * [Observer](extra-library/applications/observer.md) + * [MCP](extra-library/applications/mcp.md) + * [Radar](extra-library/applications/radar.md) + * [Pulse](extra-library/applications/pulse.md) * [Meta-Processes](extra-library/meta-processes/README.md) * [WebSocket](extra-library/meta-processes/websocket.md) * [SSE](extra-library/meta-processes/sse.md) @@ -79,5 +87,4 @@ ## Tools * [Boilerplate Code Generation](tools/ergo.md) -* [Inspecting With Observer](tools/observer.md) * [Saturn - Central Registrar](tools/saturn.md) diff --git a/docs/actors/actor.md b/docs/actors/actor.md index e9535562f..4dabc9f12 100644 --- a/docs/actors/actor.md +++ b/docs/actors/actor.md @@ -382,8 +382,8 @@ Log messages have the lowest priority. They're processed after Urgent, System, a If your actor subscribed to an event (via `LinkEvent` or `MonitorEvent`), it receives event messages: ```go -func (w *Worker) HandleEvent(message gen.MessageEvent) error { - switch message.Name { +func (w *Worker) HandleEvent(event gen.MessageEvent) error { + switch event.Event.Name { case "config_updated": w.reloadConfig() case "cache_invalidated": diff --git a/docs/actors/supervisor.md b/docs/actors/supervisor.md index 189b4c908..304bca114 100644 --- a/docs/actors/supervisor.md +++ b/docs/actors/supervisor.md @@ -80,7 +80,7 @@ func createSupervisorFactory() gen.ProcessBehavior { pid, err := node.Spawn(createSupervisorFactory, gen.ProcessOptions{}) ``` -The supervisor spawns all children during `Init` (except Simple One For One, which starts with zero children). Each child is linked bidirectionally to the supervisor (`LinkChild` and `LinkParent` set automatically). If a child terminates, the supervisor receives an exit signal and applies the restart strategy. +The supervisor spawns all children during `Init` (except Simple One For One, which starts with zero children). Each child is connected to the supervisor with a pair of unidirectional links (`LinkChild` and `LinkParent` set automatically). If a child terminates, the supervisor receives an exit signal and applies the restart strategy. Children are started sequentially in declaration order. If any child's spawn fails (the factory's `ProcessInit` returns an error), the supervisor terminates immediately with that error. This ensures the supervision tree is fully initialized or not at all - no partial states. diff --git a/docs/advanced/building-a-cluster.md b/docs/advanced/building-a-cluster.md index cd294ff96..955af0a12 100644 --- a/docs/advanced/building-a-cluster.md +++ b/docs/advanced/building-a-cluster.md @@ -360,8 +360,8 @@ func (c *Coordinator) Init(args ...any) error { return nil } -func (c *Coordinator) HandleEvent(ev gen.MessageEvent) error { - switch msg := ev.Message.(type) { +func (c *Coordinator) HandleEvent(event gen.MessageEvent) error { + switch msg := event.Message.(type) { case etcd.EventApplicationStarted: if msg.Name == "worker" { @@ -825,8 +825,8 @@ cacheEnabled := config["cache.enabled"].(bool) // true React to config changes in real-time: ```go -func (a *App) HandleEvent(ev gen.MessageEvent) error { - switch msg := ev.Message.(type) { +func (a *App) HandleEvent(event gen.MessageEvent) error { + switch msg := event.Message.(type) { case etcd.EventConfigUpdate: a.Log().Info("config changed: %s = %v", msg.Item, msg.Value) @@ -937,8 +937,8 @@ func (c *Coordinator) HandleBecomeFollower(leader gen.PID) error { return nil } -func (c *Coordinator) HandleEvent(ev gen.MessageEvent) error { - switch ev.Message.(type) { +func (c *Coordinator) HandleEvent(event gen.MessageEvent) error { + switch event.Message.(type) { case etcd.EventApplicationStarted, etcd.EventApplicationStopped: c.refreshWorkers() } diff --git a/docs/advanced/debugging.md b/docs/advanced/debugging.md index 036737a6a..61b967cb2 100644 --- a/docs/advanced/debugging.md +++ b/docs/advanced/debugging.md @@ -47,12 +47,12 @@ With `norecover`, panics propagate normally, providing full stack traces and all - Tracking down type assertion failures - Understanding the call sequence leading to a panic -### The `trace` Tag +### The `verbose` Tag -The `trace` tag enables verbose logging of framework internals: +The `verbose` tag enables verbose logging of framework internals: ```bash -go run --tags trace ./cmd +go run --tags verbose ./cmd ``` This produces detailed output about: @@ -72,6 +72,57 @@ options := gen.NodeOptions{ } ``` +### The `latency` Tag + +The `latency` tag enables mailbox latency measurement for all processes: + +```bash +go run --tags latency ./cmd +``` + +This activates: + +- **Monotonic timestamp** on every message pushed into the MPSC queue +- **`QueueMPSC.Latency()`** returns the age (in nanoseconds) of the oldest unprocessed message in the queue +- **`ProcessMailbox.Latency()`** returns the maximum latency across all four mailbox queues (Main, System, Urgent, Log) +- **`MailboxLatency` field** in `ProcessShortInfo` for per-process latency snapshots +- **`Node.ProcessRangeShortInfo()`** for efficient iteration over all processes with their latency data + +Without the tag, `Latency()` returns -1 (disabled) and there is zero runtime overhead: no timestamps are recorded, no atomic operations are added to the message path. + +The overhead with the tag enabled is approximately 10-25% on micro-benchmarks (LOCAL 1-1 scenario with a single producer and consumer exchanging messages). In real applications with many processes, the overhead is lower because the cost is amortized across concurrent operations. + +Latency measurement answers the question "how long has the oldest message been sitting in this process's mailbox?" A high value means the process is not keeping up with incoming messages: it is either overloaded, stuck in a long-running callback, or blocked. This is particularly useful for: + +- Identifying backpressure in actor pipelines +- Detecting stuck processes before they cause cascading failures +- Finding hotspot processes in large clusters + +For cluster-wide observability with Prometheus and Grafana, see the [Metrics actor](../extra-library/actors/metrics.md) which integrates latency data into distribution, top-N, and per-node panels when built with the `latency` tag. + +### The `typestats` Tag + +The `typestats` tag enables per-type encode/decode statistics: + +```bash +go run --tags typestats ./cmd +``` + +This activates: + +- **Encoded/Decoded counts** per registered EDF type for root-level operations (calls at the message boundary, not nested fields) +- **EncodedBytes/DecodedBytes** measured as decompressed wire size, pre-compression on encode and post-decompression on decode, including the type-prefix header +- **`Stats.Enabled` flag** in `gen.RegisteredTypeInfo` set to `true` to signal counters are active +- Counters visible via **`Network().RegisteredTypes()`** API and the **Observer Types panel** + +Without the tag, counters remain zero, `Stats.Enabled` is `false`, and there is zero runtime overhead. Encode and decode go through pass-through wrappers that the Go inliner reduces to direct calls. + +The overhead with the tag enabled is approximately 2-3% on encode/decode throughput, from two `atomic.AddInt64` operations per root call. + +A counter increments only when a value of that type is the message itself, the top of an `Encode` or `Decode` call. Built-in primitives like `gen.PID`, `gen.Atom`, `gen.Ref` typically appear as fields inside other messages, so their bytes contribute to the parent message's byte total, not to their own counters. Encoded and Decoded are independent: a node may receive some types only and send others only. + +Use case: identify message types that dominate network traffic. The average byte size per operation (`EncodedBytes / Encoded`) indicates whether a type is a candidate for compression at the producer process. Types with a high average are strong candidates for compressing at the source; types with a low average are not worth the framing overhead. + ### Combining Tags Tags can be combined for comprehensive debugging: @@ -80,7 +131,19 @@ Tags can be combined for comprehensive debugging: go run --tags "pprof,norecover,trace" ./cmd ``` -This enables all debugging features simultaneously. Use this combination when investigating complex issues that span multiple subsystems. +or with latency measurement: + +```bash +go run --tags "pprof,latency" ./cmd +``` + +or with type statistics: + +```bash +go run --tags "pprof,latency,typestats" ./cmd +``` + +This enables all specified features simultaneously. Use combinations when investigating complex issues that span multiple subsystems. ## Profiler Integration @@ -347,9 +410,10 @@ Observer runs at `http://localhost:9911` by default when included in your node. Debugging actor systems requires tools that bridge the gap between logical actors and runtime goroutines. Ergo Framework provides this bridge through: -- **Build tags** that enable profiling and diagnostics without production overhead +- **Build tags** that enable profiling, diagnostics, and latency measurement without production overhead - **Goroutine labels** that link runtime goroutines to their actor (PID) and meta process (Alias) identities - **Shutdown diagnostics** that identify processes preventing clean termination - **Observer integration** for visual inspection of running systems Combined with Go's standard profiling tools, these capabilities enable effective debugging of even complex distributed systems. + diff --git a/docs/advanced/distributed-tracing.md b/docs/advanced/distributed-tracing.md new file mode 100644 index 000000000..d5542aaba --- /dev/null +++ b/docs/advanced/distributed-tracing.md @@ -0,0 +1,955 @@ +--- +description: Distributed tracing across actor message chains +--- + +# Distributed Tracing + +In a distributed actor system, a single user request can touch dozens of processes across multiple nodes. Messages hop from actor to actor, crossing network boundaries invisibly. When something goes wrong (latency spikes, a message seems to disappear, an error surfaces three hops away from its cause) you need to follow the message trail across the entire cluster. + +Traditional logging shows you individual perspectives. Process A logged a send at 10:00:00.001, process B logged a receive at 10:00:00.003. Connecting these fragments manually, with hundreds of messages per second, is impractical. Tracing solves this by giving the framework itself the job of tracking messages end-to-end. + +## What Is a Trace + +A trace is an identity that follows a chain of causally related messages. When a process sends a message and the framework decides to track it, a 128-bit trace ID is generated and attached to that message. From that moment, the trace identity travels with every message in the chain. When the recipient handles the message and sends new messages of its own, those messages carry the same trace ID. When those recipients send further messages, the identity continues. The trace follows the causal chain across processes and nodes until the chain ends. + +This is fundamentally different from HTTP tracing. In HTTP, a request enters a service, the service calls other services, and eventually a response comes back. The trace follows a request-response tree with clear boundaries. In an actor system, there are no such boundaries. A message arrives, the handler sends three async messages to different processes, each of those handlers sends more messages, and the chain branches and spreads across the cluster. There's no single "response" that marks the end. The trace ends when the last handler in the chain finishes without sending more traced messages. + + +```mermaid +sequenceDiagram + box rgb(200,220,255) Node X + participant A as Process A + end + box rgb(200,255,220) Node Y + participant B as Process B + end + box rgb(255,230,200) Node Z + participant C as Process C + participant D as Process D + end + + Note over A: New trace starts (TraceID=abc) + + A->>B: Send(Order) + rect rgb(245,245,245) + Note over A,B: TraceID=abc travels with the message + end + + activate B + Note over B: Handling Order... + + B->>C: Send(ReserveStock) + B->>D: Send(CreateInvoice) + deactivate B + + activate C + Note over C: Handling ReserveStock... + deactivate C + + activate D + Note over D: Handling CreateInvoice... + deactivate D +``` + +Process B never opted into tracing. Neither did C or D. The trace reached them because the message carried it. This is the key property: you configure tracing on entry-point processes, and the trace propagates through the entire downstream chain automatically. + +### The Lifecycle of a Trace + +A trace goes through three phases: + +**Birth.** A process handles a message and calls `Send`, `Call`, or `SendResponse`. The framework checks: is there an active trace from the incoming message being handled? If yes, the outgoing message inherits it. If no, the framework asks the process's sampler: "should we start a new trace?" If the sampler says yes, a new trace ID is generated. If it says no, the message goes out untraced. The sampler is covered in the Enabling Tracing section below. + +**Propagation.** The trace identity travels with the message. When the recipient's handler runs, the framework stores the trace as the "propagating context" for the duration of that handler. Every `Send`, `Call`, or `SendResponse` during the handler inherits the trace identity. When the handler returns, the context is restored. If the handler sends messages to five different processes, all five messages carry the same trace identity. Each recipient propagates it further in the same way. + +**End.** A trace has no explicit end and no timeout. It ends naturally when the last handler in the chain finishes processing and sends no further messages. A trace that spans a 30-second `Call` timeout will simply have a 30-second gap between observations. The trace identity is a value in the message, not a timer. + +### Observation Points + +As a trace flows through the system, the framework records observations at three points for each message: + +**Sent.** Recorded when the message leaves the sender. This is the sender's perspective: who sent what, to whom, and when. + +**Delivered.** Recorded when the message enters the recipient's mailbox. The recipient hasn't started processing yet, the message is queued. + +**Processed.** Recorded when the recipient's handler returns. If the handler returned an error, the observation captures it. + +```mermaid +sequenceDiagram + box rgb(200,220,255) Sender + participant A as Sender + end + box rgb(200,255,220) Recipient + participant B as Recipient + end + + A->>B: message + Note right of A: Sent + Note left of B: Delivered + activate B + Note over B: Handler runs... + deactivate B + Note left of B: Processed +``` + +These three points are not the trace itself. They are what gets recorded as the trace passes through. One message produces up to three observations. A trace spanning five messages across three nodes produces up to fifteen observations. Together, these observations reconstruct the complete message flow. + +The timing gaps between observations tell you where time is spent: + +| Gap | What It Tells You | +|-----|-------------------| +| Sent to Delivered | Network latency (remote) or scheduling delay (local) | +| Delivered to Processed | Mailbox wait time + handler execution time | +| Sent to Processed | Total end-to-end latency for this message | + +For local messages, Sent and Delivered happen nearly simultaneously. For remote messages, the gap is the network transit time. This makes tracing particularly valuable in distributed systems: you can see exactly how much time is spent in transit versus in processing. + +Each observation carries context: which node emitted it, the sender and recipient identities, the message type name, the actor behavior type, a timestamp, and any custom attributes. Together, the observations for a single trace form a tree that you can visualize as a waterfall in tools like Grafana Tempo or the Observer UI. + +### Why Three Points, Not Two + +HTTP tracing typically records two points per span: the start and end of a service call. Actor tracing needs three because messages go through a mailbox. In HTTP, when service A calls service B, B starts processing immediately. In an actor system, when A sends to B, the message enters B's mailbox and waits. B might be busy handling a previous message. The wait time can be significant under load. + +Without the Delivered point, you'd see Sent at time T and Processed at T+50ms, but you wouldn't know whether the 50ms was network latency, mailbox wait, or handler execution. With Delivered, you know: Sent to Delivered was 2ms (network), Delivered to Processed was 48ms (the message sat in the mailbox for 40ms and the handler took 8ms). This distinction is critical for diagnosing performance issues. + +### What Gets Traced + +All message kinds that go through the framework's routing: + +| Kind | Description | Observations | +|------|-------------|--------------| +| Send | Asynchronous message (`Send`) | Sent, Delivered, Processed | +| Request | Synchronous call (`Call`) | Sent, Delivered, Processed | +| Response | Return value from `HandleCall` | Sent, Delivered | +| Spawn | Process creation | Sent, Processed | +| Terminate | Process termination | Processed | + +Response has no Processed because the response delivery completes the Call. There's no separate handler on the caller side. Spawn has no Delivered because it's not a mailbox delivery. Terminate has only Processed because it's an internal lifecycle event, not a message between two processes. + +### What Doesn't Get Traced + +Exit signals (`SendExit`) do not carry trace context. These are control-plane operations outside of message chains. + +Events (`SendEvent`) also do not carry trace context. An event with a thousand subscribers would generate thousands of trace observations from a single publish, creating a storm that overwhelms exporters and backends. If you need to trace event-driven flows, trace the messages that your event handlers send in response to receiving events. + +Delayed messages (`SendAfter`) do not carry trace context. A delayed message is a scheduled future action, not a continuation of the current processing chain. By the time it fires, the original handler has long finished. This prevents periodic self-tick patterns from creating infinite traces. Each tick is an independent starting point for the sampler. See the Delayed Messages section for details. + +## Enabling Tracing + +By default, no processes create traces. You enable tracing by setting a sampler that decides whether to start a new trace for each outgoing message. + +```go +func (a *OrderProcessor) Init(args ...any) error { + a.SetTracingSampler(gen.TracingSamplerAlways) + return nil +} +``` + +Four sampler types are available: + +```go +gen.TracingSamplerDisable // never start traces (default) +gen.TracingSamplerAlways // trace every outgoing message +gen.TracingSamplerRatio(0.01) // trace 1% of messages +gen.TracingSamplerRateLimit(100) // at most 100 new traces per second +``` + +The sampler is only consulted when there is no active trace. If a process is already handling a traced message, every outgoing message inherits the trace regardless of the sampler. This means you can set a sampler on a single entry-point process and the trace will follow the entire message chain automatically. + +`TracingSamplerRatio(0.1)` traces approximately 10% of messages. `TracingSamplerRateLimit(100)` allows at most 100 new traces per second. During traffic spikes the effective sampling rate drops, during quiet periods more messages are traced. + +The sampler is set during `Init()` but only starts working when the process begins handling messages. Messages sent during `Init()` itself, including periodic ticks set up with `SendAfter`, are not traced. This is because `Init()` is a setup phase, not message processing. The sampler becomes active starting from the first `HandleMessage` or `HandleCall` invocation. + +### Setting Samplers at Runtime + +You can change a process's sampler without restarting it: + +```go +node.SetProcessTracingSampler(pid, gen.TracingSamplerAlways) +``` + +The node itself has a sampler for messages sent via `node.Send()` and `node.Call()`: + +```go +node.SetTracingSampler(gen.TracingSamplerRatio(0.01)) +``` + +Process samplers and the node sampler are independent. + +### Custom Samplers + +If the built-in samplers don't fit your needs, implement the `gen.TracingSampler` interface: + +```go +type TracingSampler interface { + Sample() bool + String() string +} +``` + +`Sample()` is called for each outgoing message that doesn't already carry a trace. Return `true` to start a new trace. `String()` provides a human-readable description shown in Observer and inspection APIs. + +## Tracing in Practice: Send + +The simplest traced scenario: process A handles a message and sends to process B on the same node. + +```go +func (a *gateway) Init(args ...any) error { + a.SetTracingSampler(gen.TracingSamplerAlways) + a.SetTracingAttribute("service", "gateway") + return nil +} + +func (a *gateway) HandleMessage(from gen.PID, message any) error { + req := message.(IncomingRequest) + a.Send(processorPID, ProcessOrder{ID: req.OrderID}) + return nil +} +``` + +When `a.Send()` executes, the sampler decides to start a new trace. The framework generates a trace identity shared by all observations for this message. Three observations are recorded: + +1. **Sent** on the sender's node, capturing: sender PID, receiver PID, message type `main.ProcessOrder`, behavior `gateway`, the custom attribute `service=gateway`. + +2. **Delivered** on the same node (it's local), capturing: the same message identity, the receiver's behavior name, the receiver's permanent attributes. + +3. **Processed** after the receiver's `HandleMessage` returns, capturing: whether the handler succeeded or returned an error, plus any one-shot attributes the receiver set during handling. + +The receiver didn't set a sampler. It didn't need to. The trace arrived with the message and the observations were recorded automatically. + +### Remote Send + +When process A on node X sends to process B on node Y, the trace crosses the network: + +```mermaid +sequenceDiagram + box rgb(200,220,255) Node X + participant A as Process A + end + box rgb(200,255,220) Node Y + participant B as Process B + end + + A->>B: Send(ProcessOrder) + Note right of A: Sent (on node X) + + rect rgb(245,245,245) + Note over A,B: network transit + end + + Note left of B: Delivered (on node Y) + activate B + Note over B: Handler runs... + deactivate B + Note left of B: Processed (on node Y) +``` + +Sent is recorded on node X, but Delivered and Processed are recorded on node Y. The framework preserves the message's identity across the network, so all three observations can be correlated even though they were emitted on different nodes. + +The gap between Sent and Delivered now represents real network latency. If you see a 50ms gap, that's 50ms of network transit. + +## Tracing in Practice: Message Chains + +The real power of tracing appears when messages form chains. Process A sends to B, and B sends to C and D while handling A's message. All hops share the same trace. + +```go +func (p *processor) HandleMessage(from gen.PID, message any) error { + order := message.(ProcessOrder) + p.SetTracingSpanAttribute("order_id", order.ID) + + p.Send(warehousePID, ReserveStock{OrderID: order.ID}) + p.Send(billingPID, CreateInvoice{OrderID: order.ID}) + return nil +} +``` + +```mermaid +sequenceDiagram + box rgb(200,220,255) Gateway Node + participant GW as gateway + end + box rgb(200,255,220) Worker Node + participant P as processor + end + box rgb(255,230,200) Service Node + participant W as warehouse + participant B as billing + end + + Note over GW: Sampler starts trace + + GW->>P: ProcessOrder + Note right of GW: Sent + Note left of P: Delivered + + activate P + Note over P: Handler runs + P->>W: ReserveStock + P->>B: CreateInvoice + deactivate P + Note left of P: Processed + + activate W + Note left of W: Delivered + Note over W: Handler runs + deactivate W + Note left of W: Processed + + activate B + Note left of B: Delivered + Note over B: Handler runs + deactivate B + Note left of B: Processed +``` + +The gateway started the trace. The processor inherited it from the incoming message. The warehouse and billing processes also inherited it. Five messages, three nodes, one trace. + +The propagation is automatic. During a handler, the framework stores the incoming message's trace context. Every `Send`, `Call`, or `SendResponse` during that handler carries the trace forward. When the handler returns, the context is restored to whatever it was before. + +The trace captures causality: the processor's messages to warehouse and billing were sent **because of** the gateway's message to the processor. This creates a tree of messages that represents the complete processing flow for the original request. + +## Tracing in Practice: Call and Response + +Synchronous calls create two traced message flows within the same trace: the request going out and the response coming back. + +```go +func (c *client) HandleMessage(from gen.PID, message any) error { + to := gen.ProcessID{Name: "inventory", Node: "warehouse@host"} + result, err := c.Call(to, CheckStockRequest{SKU: "WIDGET-42"}) + if err != nil { + c.Log().Warning("stock check failed: %s", err) + return nil + } + resp := result.(CheckStockResponse) + c.Log().Info("stock level: %d", resp.Available) + return nil +} + +func (inv *inventory) HandleCall(from gen.PID, ref gen.Ref, request any) (any, error) { + req := request.(CheckStockRequest) + level := inv.checkWarehouse(req.SKU) + return CheckStockResponse{Available: level}, nil +} +``` + +```mermaid +sequenceDiagram + box rgb(200,220,255) Node A + participant C as client + end + box rgb(200,255,220) Node B + participant I as inventory + end + + C->>I: Call(CheckStockRequest) + Note right of C: Sent (request) + + rect rgb(245,245,245) + Note over C,I: network + end + + Note left of I: Delivered (request) + activate I + Note over I: HandleCall runs + deactivate I + + I->>C: CheckStockResponse + Note right of I: Sent (response) + Note left of I: Processed (request) + + rect rgb(245,245,245) + Note over C,I: network + end + + Note left of C: Delivered (response) +``` + +The request and the response are separate messages, each with their own observations. They share a call reference (`gen.Ref`) that links them, so tools like Tempo and Observer can pair request and response even when multiple concurrent calls are in flight. + +If the inventory process sends additional messages during `HandleCall` (for example, querying a database actor), those messages are also part of the same trace, linked causally to the incoming request. + +### Forward Pattern + +In the actor model, a process handling a synchronous request can forward it to another process instead of responding directly. The relay wraps the original caller's identity and reference into the forwarded message, and the final recipient responds straight to the original caller: + +```go +func (r *relay) HandleCall(from gen.PID, ref gen.Ref, request any) (any, error) { + req := request.(Request) + target := gen.ProcessID{Name: "backend", Node: "target@node"} + + r.Send(target, MessageForward{ + OriginalFrom: from, + OriginalRef: ref, + Payload: req, + }) + return nil, nil // no direct response -- backend will respond to the original caller +} + +func (b *backend) HandleMessage(from gen.PID, message any) error { + fwd := message.(MessageForward) + result := b.process(fwd.Payload) + // process message + b.SendResponse(fwd.OriginalFrom, fwd.OriginalRef, result) + return nil +} +``` + +The trace follows the entire chain: A's call to the relay, the relay's forward to the backend, and the backend's response to A. Three messages, potentially three nodes, one trace. The response skips the relay entirely, and the trace captures this topology accurately. + +### Async Response and Trace Context + +When `HandleCall` returns `nil, nil` (async response), the process stores the caller's identity and reference to respond later. Between the request handler and the eventual response, other messages may arrive. The response will happen in a different handler invocation, potentially with a different trace context. + +If you need the response to be in the same trace as the original request, save the trace context alongside the caller identity: + +```go +type pendingCall struct { + From gen.PID + Ref gen.Ref + Tracing gen.Tracing +} + +func (s *service) HandleCall(from gen.PID, ref gen.Ref, request any) (any, error) { + s.pending = pendingCall{ + From: from, + Ref: ref, + Tracing: s.PropagatingTrace(), + } + return nil, nil +} + +func (s *service) HandleMessage(from gen.PID, message any) error { + // some event triggers the response + saved := s.PropagatingTrace() + s.SetPropagatingTrace(s.pending.Tracing) + s.SendResponse(s.pending.From, s.pending.Ref, result) + s.SetPropagatingTrace(saved) + return nil +} +``` + +`PropagatingTrace()` returns the current trace context. In `HandleCall`, this is the request's trace. Saving it and restoring before `SendResponse` ensures the response carries the original request's trace, regardless of which trace context the current handler is working with. + +The save-restore pattern is important: `SetPropagatingTrace` changes the trace context for all subsequent operations in the handler. If you don't restore the previous context, the modified trace will leak beyond the current handler into all subsequent handler invocations. Every message the process sends from that point on will carry the leaked trace until another traced message arrives and resets it. Always save before, always restore after. + +## Custom Attributes + +Traces show message flow. Custom attributes add business context that makes traces searchable and meaningful. + +Attributes describe the place where a message was sent, delivered, or processed. They are part of the observation record, not part of the trace context. Over the network, only the trace ID and span ID travel with the message, just enough to link observations into a chain. Attributes stay local to the node that emitted the observation. This keeps the network overhead minimal and lets each process describe its own context independently. + +### Permanent Attributes + +Set on a process, attached to every observation from that process for its entire lifetime: + +```go +func (a *PaymentService) Init(args ...any) error { + a.SetTracingSampler(gen.TracingSamplerRatio(0.01)) + a.SetTracingAttribute("service", "payment") + a.SetTracingAttribute("version", "2.1") + a.SetTracingAttribute("region", "eu-west") + return nil +} +``` + +When a message passes through this process, its attributes appear on every observation where the process is a participant. If another process sends a message to PaymentService, the Delivered and Processed observations carry `service=payment, version=2.1, region=eu-west`. When PaymentService sends a message to someone else, the Sent observation carries the same attributes. The attributes describe the location in the system where the observation was recorded. + +Setting an attribute with a key that already exists overwrites the value. Remove with `RemoveTracingAttribute(key)`. + +### Node-Level Attributes + +The node has its own permanent attributes, independent from process attributes: + +```go +node.SetTracingAttribute("env", "production") +node.SetTracingAttribute("cluster", "payments-eu") +``` + +Same mechanics as process attributes: set, overwrite, or remove at any time. + +### One-Shot Span Attributes + +Set during message handling, scoped to a single handler invocation: + +```go +func (a *OrderProcessor) HandleMessage(from gen.PID, message any) error { + order := message.(Order) + a.SetTracingSpanAttribute("order_id", order.ID) + a.SetTracingSpanAttribute("customer", order.CustomerID) + a.SetTracingSpanAttribute("amount", fmt.Sprintf("%.2f", order.Total)) + + a.Send(warehousePID, ReserveStock{OrderID: order.ID}) + a.Send(billingPID, CreateInvoice{OrderID: order.ID}) + return nil +} +``` + +One-shot attributes appear on the observations emitted during this handler invocation: the Processed observation for the incoming message, and the Sent observations for outgoing messages. When the handler returns, one-shot attributes are cleared automatically. The next handler invocation starts with a clean slate. + +If a one-shot attribute has the same key as a permanent attribute, the one-shot value takes priority for that handler invocation. The permanent attribute is not modified. + +### Where Attributes Appear + +Different observations carry different attributes: + +| Observation | Attributes | +|-------------|-----------| +| Sent | Sender's permanent + one-shot attributes | +| Delivered | Receiver's permanent attributes | +| Processed | Receiver's permanent + one-shot attributes | + +This means: the sender decides what context to attach at send time. The receiver's permanent identity (service name, version) appears on its Delivered and Processed observations. The receiver can add handler-specific context (order ID, customer) that appears on its Processed observation and on any Sent observations during that handler. + +### Searching by Attributes + +In Grafana Tempo or the Observer UI, search by any attribute value. If one observation in a trace has `order_id=ORD-456`, searching for it returns the complete trace, all observations across all nodes in the chain. You don't need the same attribute on every observation. + +This makes attributes a powerful debugging tool. Set `order_id` on the entry-point process, and you can find the complete processing trace for any order by searching for its ID. + +The `ergo.` prefix is reserved for framework-generated attributes (`ergo.node`, `ergo.from`, `ergo.behavior`). Attempts to set attributes with this prefix are silently ignored. + +## Delayed Messages + +`SendAfter` does not carry trace context. This is a deliberate design choice: a delayed message is a future action, not a continuation of the current processing chain. + +Consider a common pattern, a process that does periodic work via a self-tick: + +```go +func (w *worker) Init(args ...any) error { + w.SetTracingSampler(gen.TracingSamplerAlways) + w.SendAfter(w.PID(), messageTick{}, 3*time.Second) + return nil +} + +func (w *worker) HandleMessage(from gen.PID, message any) error { + switch message.(type) { + case messageTick: + w.Send(targetPID, DoWork{}) + w.SendAfter(w.PID(), messageTick{}, 3*time.Second) + } + return nil +} +``` + +Each tick arrives as an untraced message. The sampler on the worker decides independently for each `Send(targetPID, DoWork{})` whether to create a trace. The `SendAfter` at the end schedules the next tick without trace context, breaking the chain and ensuring the next tick starts fresh. + +If `SendAfter` inherited the trace, the first tick that happened to be traced would create an infinite trace: tick carries trace, handler sends traced tick, next handler sends traced tick, forever. A process running for days would accumulate millions of observations in a single trace. Decoupling `SendAfter` from the trace context prevents this. + +The same applies to `SendAfter` to other processes. If you need a delayed message to carry trace context, send it through a regular `Send` to an intermediary that schedules the delay, or store the trace context and restore it when the delayed action triggers (the same pattern as Async Response above). + +### Self-Send and Trace Propagation + +`Send` to self behaves like `Send` to any other process. The message carries the current trace context. This is consistent and enables patterns like async `HandleCall` where a process sends work to itself and responds later within the same trace. + +For periodic self-loops, use `SendAfter` which does not carry trace context. This is the natural choice for tick patterns since `SendAfter` provides the timing control that loops need. Each tick starts fresh, and the sampler decides independently whether to trace it. + +If your actor uses `Send` to itself for a finite internal sequence (state machine, batch processing), the internal steps will appear in the trace. For a three-step state machine triggered by a traced message, this adds six extra observations. This is proportional to the work done and finite, not a concern in practice. + +## Lifecycle Events: Spawn and Terminate + +When a process spawns a child during a traced handler, the spawn itself is part of the trace. + +```go +func (m *manager) HandleMessage(from gen.PID, message any) error { + task := message.(NewTask) + + pid, err := m.Spawn(workerFactory, gen.ProcessOptions{}, task.Config) + if err != nil { + return err + } + + m.Send(pid, BeginWork{TaskID: task.ID}) + return nil +} +``` + +The framework records two observations for the spawn: + +**Sent.** Emitted before the child's `Init()` runs. This is "spawn initiated." + +**Processed.** Emitted after `Init()` returns. If `Init()` returned an error, the error is recorded in this observation's Error field. + +The gap between Sent and Processed is the `Init()` execution time. If a spawn is slow, you'll see it in the trace. + +After `Init()` completes, the child process starts with a clean slate, no inherited trace context. Messages the child sends during `Init()` are not traced. The child's sampler decides whether to trace its own outgoing messages starting from the first `HandleMessage` or `HandleCall`. The `Send(pid, BeginWork{})` in the example above carries the parent's trace (it's a regular `Send` during the parent's traced handler), so the child receives and processes it within the parent's trace. + +### Terminate + +A terminate observation is recorded when a process terminates while handling a traced message. If the handler returns an error that causes the process to exit, the framework records the termination reason in the same trace as the message that caused the crash. This gives you the complete picture in one trace: the message arrived, the handler failed, the process terminated. + +Processes that terminate between handler invocations (normal shutdown, supervisor stop, `node.Kill`) do not generate a terminate observation. Normal lifecycle events don't produce tracing noise. + +## Exporters + +Observations go nowhere by themselves. To see them, you register one or more tracing exporters on the node. This works similarly to loggers: a node can have multiple loggers, each receiving log messages according to its own level filter. A node can have multiple tracing exporters, each receiving observations according to its own flags. + +The framework emits all observations unconditionally for traced messages. Each exporter declares which types of observations it wants to receive, and the framework delivers only those. One exporter might receive everything for a waterfall UI, while another on the same node receives only Sent observations for counting outgoing messages. + +### Exporter Flags + +When you register an exporter, you specify which observations it should receive: + +```go +gen.TracingFlagSend // Sent observations +gen.TracingFlagReceive // Delivered and Processed observations +gen.TracingFlagProcs // Spawn and Terminate lifecycle events +``` + +Combine with bitwise OR: + +```go +// receive everything +flags := gen.TracingFlagSend | gen.TracingFlagReceive | gen.TracingFlagProcs + +// only message delivery observations +flags := gen.TracingFlagReceive +``` + +### Two Kinds of Exporters + +**Process-based.** An actor process that receives observations in its mailbox. Use this when the exporter needs actor capabilities: batching with timers, sending over the network, accessing node services. This is how Observer and Pulse work internally. + +```go +node.TracingExporterAddPID(pid, "my-exporter", + gen.TracingFlagSend | gen.TracingFlagReceive | gen.TracingFlagProcs) +``` + +The process implements `HandleSpan(gen.TracingSpan)` to process each observation. If the process's mailbox is full, observations are silently dropped. Ensure the exporter can keep up with the observation rate. + +**Behavior-based.** A simple implementation of the `gen.TracingBehavior` interface. `HandleSpan` is called synchronously when an observation is emitted. Use this for lightweight exporters that don't need actor capabilities. + +```go +type TracingBehavior interface { + HandleSpan(TracingSpan) + Terminate() +} +``` + +```go +node.TracingExporterAdd("counter", &spanCounter{}, + gen.TracingFlagSend | gen.TracingFlagReceive) +``` + +Keep `HandleSpan` fast. It blocks delivery to the next exporter in the chain. + +### Registering Exporters + +At node startup: + +```go +options := gen.NodeOptions{ + Tracing: gen.TracingOptions{ + Exporters: []gen.TracingExporter{ + { + Name: "my-exporter", + Exporter: &myExporter{}, + Flags: gen.TracingFlagSend | gen.TracingFlagReceive, + }, + }, + }, +} +``` + +At runtime: + +```go +node.TracingExporterAdd("counter", &spanCounter{}, gen.TracingFlagSend) +node.TracingExporterAddPID(pid, "observer", gen.TracingFlagSend | gen.TracingFlagReceive | gen.TracingFlagProcs) +``` + +Each exporter has a unique name. Attempting to register a name that's already taken returns `gen.ErrTaken`. A process can only be registered as one exporter. A second attempt returns `gen.ErrNotAllowed`. + +### Removing Exporters + +```go +names := node.TracingExporters() // list registered exporter names +node.TracingExporterDelete("name") // remove by name +node.TracingExporterDeletePID(pid) // remove by PID +``` + +Removing a behavior-based exporter calls its `Terminate()` method. Exporters can be added and removed at any time while the node is running. + +## Observer and Pulse + +Two ready-made exporters are available out of the box. + +[Observer](observer.md) provides real-time tracing visualization directly in the web UI. It connects to a specific node and shows traces passing through that node, useful for live debugging and runtime sampler control. Since Observer sees only one node at a time, traces that span multiple nodes will appear partial. See [Inspecting With Observer](observer.md) for details. + +[Pulse](../extra-library/applications/pulse.md) exports traces to an OTLP-compatible backend (Grafana Tempo, Jaeger). Each node runs its own Pulse instance, sending observations to a shared collector. The backend assembles complete cross-cluster traces from all nodes, so you can see the full message chain end-to-end. See the [Pulse documentation](../extra-library/applications/pulse.md) for setup and configuration. + +## Production Patterns + +### Sampling at the Edge + +In production, you rarely want to trace everything. Set a ratio sampler on your entry-point processes and let propagation handle the rest: + +```go +func (gw *APIGateway) Init(args ...any) error { + gw.SetTracingSampler(gen.TracingSamplerRatio(0.01)) + gw.SetTracingAttribute("service", "api-gateway") + return nil +} +``` + +One percent of requests are traced end-to-end across the entire cluster. The other 99% have near-zero overhead: one `Sample()` call returning `false`. + +Downstream processes don't need samplers. They inherit traces from incoming messages. This means adding tracing to a complex system requires changes only at the entry points. + +### Rate Limiting Under Load + +When traffic volume varies, `TracingSamplerRateLimit` provides a steady flow of traces regardless of load: + +```go +gw.SetTracingSampler(gen.TracingSamplerRateLimit(50)) +``` + +This creates at most 50 new traces per second. During a traffic spike, the effective sampling rate drops. During quiet periods, more messages are traced. + +This is useful when your tracing backend or exporters have throughput limits. You get consistent trace volume without overwhelming the pipeline. + +### Debugging a Specific Process + +Something is wrong with a particular process. Enable full tracing on it without restarting: + +```go +node.SetProcessTracingSampler(problemPID, gen.TracingSamplerAlways) +``` + +Or through the Observer UI: open the process, go to Config, set the sampler to "always". Every message this process handles and every message it sends will be traced. When you're done investigating, set it back to "disable." + +Because trace propagation is automatic, you'll see not just this process's messages but the entire downstream chain. If the process calls a remote service, you'll see the round-trip. If it spawns workers, you'll see the spawn and the workers' activity. + +### Finding Specific Requests + +A customer reports a problem with order ORD-789. You need to see what happened: + +```go +func (a *OrderProcessor) HandleMessage(from gen.PID, message any) error { + order := message.(Order) + a.SetTracingSpanAttribute("order_id", order.ID) + // ... process the order + return nil +} +``` + +In Grafana Tempo, search for `order_id=ORD-789`. The complete trace appears: every message in the processing chain, across every node, with timing at every hop. You can see where the latency was, which service returned an error, and what happened next. + +This requires that the entry-point process was tracing when order ORD-789 came through. With 1% sampling, you won't have traces for every request. For critical flows where you always need traces, use `TracingSamplerAlways` on the entry-point process or a higher ratio. + +### Temporary Tracing for Incident Response + +During an incident, you need more visibility. Increase sampling temporarily: + +```go +// before: 1% sampling +node.SetProcessTracingSampler(gatewayPID, gen.TracingSamplerRatio(0.01)) + +// during incident: trace everything +node.SetProcessTracingSampler(gatewayPID, gen.TracingSamplerAlways) + +// after resolution: back to normal +node.SetProcessTracingSampler(gatewayPID, gen.TracingSamplerRatio(0.01)) +``` + +You can do this through the Observer UI without any code changes: open the process, change the sampler in the Config tab, investigate, and set it back. + +## Understanding Trace Trees + +As traces propagate through message chains, they form trees. Understanding the tree structure helps when reading traces in Tempo or Observer. + +### Linear Chain + +The simplest tree: A sends to B, B sends to C, C sends to D. + +```mermaid +sequenceDiagram + box rgb(200,220,255) + participant A + end + box rgb(200,255,220) + participant B + end + box rgb(255,230,200) + participant C + end + box rgb(230,220,255) + participant D + end + + A->>B: Send + activate B + Note over B: handles... + B->>C: Send + deactivate B + activate C + Note over C: handles... + C->>D: Send + deactivate C + activate D + Note over D: handles... + deactivate D +``` + +Each message is a child of the message that caused it. In a waterfall view, you see a staircase pattern: each hop starts when the previous handler runs. + +### Fan-Out + +One handler sends to multiple recipients: + +```mermaid +sequenceDiagram + box rgb(200,220,255) + participant A + end + box rgb(200,255,220) + participant B + end + box rgb(255,230,200) + participant C + participant D + participant E + end + + A->>B: Send + activate B + Note over B: handles... + B->>C: ReserveStock + B->>D: CreateInvoice + B->>E: SendNotification + deactivate B +``` + +B's handler sends three messages. All three are children of B's incoming message. In a waterfall view, the three sends appear at roughly the same timestamp, fanning out from B's processing. + +### Fan-Out with Call + +B calls C synchronously, then uses the result to send to D: + +```go +func (b *processor) HandleMessage(from gen.PID, message any) error { + result, err := b.Call(validatorPID, ValidateRequest{...}) + if err != nil { + return err + } + b.Send(executorPID, ExecuteRequest{Validated: result}) + return nil +} +``` + +```mermaid +sequenceDiagram + box rgb(200,220,255) + participant A + end + box rgb(200,255,220) + participant B + end + box rgb(255,230,200) + participant C as C (Validator) + end + box rgb(230,220,255) + participant D as D (Executor) + end + + A->>B: Send + activate B + Note over B: handles... + B->>C: Call(Validate) + activate C + C->>B: Response + deactivate C + B->>D: Send(Execute) + deactivate B +``` + +In the waterfall, you see B waiting for C's response before sending to D. The gap between the response arriving and D's Sent observation shows B's processing time between the call return and the next send. + +### Deep Chains Across Nodes + +In a microservice-style architecture with many nodes, traces can span many hops: + +```mermaid +sequenceDiagram + box rgb(200,220,255) Edge + participant GW as Gateway + end + box rgb(200,255,220) Auth + participant Auth + end + box rgb(255,230,200) Orders + participant Ord as OrderService + end + box rgb(230,220,255) Stock + participant Inv as Inventory + participant WH as Warehouse + end + + GW->>Auth: Call + activate Auth + Auth->>GW: Response + deactivate Auth + GW->>Ord: Send + activate Ord + Ord->>Inv: Call + activate Inv + Inv->>WH: Call + activate WH + WH->>Inv: Response + deactivate WH + Inv->>Ord: Response + deactivate Inv + deactivate Ord +``` + +Each arrow is a message with up to three observation points. The complete trace might have 15-20 observations across 5 nodes. In Tempo's waterfall view, you see exactly where time is spent: if the warehouse is slow, the gap between its Delivered and Processed observations will be large. + +## Writing a Custom Exporter + +For specialized needs beyond Pulse and Observer, you can write your own exporter. Here's an example that counts observations by kind: + +```go +type traceCounter struct { + sends int64 + requests int64 + responses int64 +} + +func (tc *traceCounter) HandleSpan(span gen.TracingSpan) { + switch span.Kind { + case gen.TracingKindSend: + atomic.AddInt64(&tc.sends, 1) + case gen.TracingKindRequest: + atomic.AddInt64(&tc.requests, 1) + case gen.TracingKindResponse: + atomic.AddInt64(&tc.responses, 1) + } +} + +func (tc *traceCounter) Terminate() {} +``` + +Register it at node startup: + +```go +options := gen.NodeOptions{ + Tracing: gen.TracingOptions{ + Exporters: []gen.TracingExporter{ + { + Name: "counter", + Exporter: &traceCounter{}, + Flags: gen.TracingFlagSend, + }, + }, + }, +} +``` + +Or register at runtime: + +```go +node.TracingExporterAdd("counter", &traceCounter{}, + gen.TracingFlagSend) +``` + +The flags on the exporter determine which observations it receives. The counter above only gets Sent observations (because of `TracingFlagSend`). To also receive Delivered and Processed, add `gen.TracingFlagReceive`. + +For more complex exporters that need actor capabilities (sending messages, using timers, accessing the network), register a process as an exporter with `TracingExporterAddPID` and implement `HandleSpan` in your actor. This is how Pulse works: a pool of actor processes that batch observations and flush them over HTTP. diff --git a/docs/advanced/images/observer/connection.png b/docs/advanced/images/observer/connection.png new file mode 100644 index 000000000..735e4802a Binary files /dev/null and b/docs/advanced/images/observer/connection.png differ diff --git a/docs/advanced/images/observer/events.png b/docs/advanced/images/observer/events.png new file mode 100644 index 000000000..97e68c5a3 Binary files /dev/null and b/docs/advanced/images/observer/events.png differ diff --git a/docs/advanced/images/observer/log.png b/docs/advanced/images/observer/log.png new file mode 100644 index 000000000..94cf5a09b Binary files /dev/null and b/docs/advanced/images/observer/log.png differ diff --git a/docs/advanced/images/observer/network.png b/docs/advanced/images/observer/network.png new file mode 100644 index 000000000..864113dda Binary files /dev/null and b/docs/advanced/images/observer/network.png differ diff --git a/docs/advanced/images/observer/process_info.png b/docs/advanced/images/observer/process_info.png new file mode 100644 index 000000000..2d3a3b652 Binary files /dev/null and b/docs/advanced/images/observer/process_info.png differ diff --git a/docs/advanced/images/observer/processes.png b/docs/advanced/images/observer/processes.png new file mode 100644 index 000000000..b3750d9a0 Binary files /dev/null and b/docs/advanced/images/observer/processes.png differ diff --git a/docs/advanced/images/observer/profiler.png b/docs/advanced/images/observer/profiler.png new file mode 100644 index 000000000..b9d0f811e Binary files /dev/null and b/docs/advanced/images/observer/profiler.png differ diff --git a/docs/advanced/images/observer/tracing.png b/docs/advanced/images/observer/tracing.png new file mode 100644 index 000000000..7dda3f571 Binary files /dev/null and b/docs/advanced/images/observer/tracing.png differ diff --git a/docs/advanced/message-versioning.md b/docs/advanced/message-versioning.md index defab15e2..9b4c1834f 100644 --- a/docs/advanced/message-versioning.md +++ b/docs/advanced/message-versioning.md @@ -43,23 +43,22 @@ func (a *Actor) HandleMessage(from gen.PID, message any) error { } ``` -All message types must be registered with EDF before connection establishment: +All message types must be registered with the network stack before connection establishment. Register them from your application's `Load(node)` callback: ```go -func init() { - types := []any{ +func (a *MyApp) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { + err := node.Network().RegisterTypes([]any{ OrderCreatedV1{}, OrderCreatedV2{}, + }) + if err != nil { + return gen.ApplicationSpec{}, err } - for _, t := range types { - if err := edf.RegisterTypeOf(t); err != nil && err != gen.ErrTaken { - panic(err) - } - } + return gen.ApplicationSpec{ /* ... */ }, nil } ``` -For details on EDF and type registration, see [Network Transparency](../networking/network-transparency.md). +For details on the type registry and the legacy `edf.RegisterTypeOf` API, see [Network Transparency](../networking/network-transparency.md). ## Versioning Strategies @@ -352,45 +351,41 @@ company.com/ ### Registration Helper -All message types must be registered with EDF before connection establishment - during handshake, nodes exchange their registered type lists which become the encoding dictionaries. Registration typically happens in `init()` functions before node startup. There are two approaches: centralized registration in the shared module or manual registration in each client. +All message types must be registered with the network stack before connection establishment. During handshake, nodes exchange their registered type lists which become the encoding dictionaries. Registration happens from an application's `Load(node)` callback, which runs after the network stack is initialized but before any traffic. There are two approaches: a centralized helper exported by the shared module, or manual registration per client. -**Centralized registration** uses `init()` to register all types when the package is imported: +**Centralized helper** exposes a single function that the consumer's application calls from `Load`: ```go // events/register.go package events -import ( - "ergo.services/ergo/gen" - "ergo.services/ergo/net/edf" -) +import "ergo.services/ergo/gen" -func init() { - types := []any{ +func RegisterTypes(network gen.Network) error { + return network.RegisterTypes([]any{ OrderCreatedV1{}, OrderCreatedV2{}, PaymentReceivedV1{}, - } - for _, t := range types { - if err := edf.RegisterTypeOf(t); err != nil && err != gen.ErrTaken { - panic(err) - } - } + }) } ``` -When clients import the package to use message types, `init()` runs automatically at program startup and registers all types: +Each consumer calls it from its application: ```go import "company.com/events" -// Using events.OrderCreatedV1 means the package is imported, -// init() has already run, types are registered +func (a *OrderService) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { + if err := events.RegisterTypes(node.Network()); err != nil { + return gen.ApplicationSpec{}, err + } + return gen.ApplicationSpec{ /* ... */ }, nil +} ``` -No risk of forgetting a type. +The shared `events` module owns the canonical list of types. Consumers register them all without having to enumerate each type, so there is no risk of forgetting one. `RegisterTypes` accepts a slice in any order and resolves nested-type dependencies internally. -**Manual registration** means each client registers only the types it uses. This gives more control but introduces risk: a missing registration is only detected at runtime - `"no encoder for type"` when sending, `"unknown reg type for decoding"` when receiving. For most projects, centralized registration is simpler and safer. Choose based on your needs. +**Manual registration** means each client registers only the types it uses. This gives more control but introduces risk: a missing registration is only detected at runtime, surfacing as `"no encoder for type"` when sending or `"unknown reg type for decoding"` when receiving. For most projects, centralized registration is simpler and safer. Choose based on your needs. For message isolation patterns within a single codebase, see [Project Structure](../basics/project-structure.md). @@ -635,11 +630,11 @@ type OrderV2 struct { **Forgetting to register new types** ```go -// Type exists but not registered - encoding fails at runtime +// Type exists but not registered. Encoding fails at runtime. type OrderV3 struct { ... } -// Must register before node starts -edf.RegisterTypeOf(OrderV3{}) +// Register from your application's Load callback before any traffic. +node.Network().RegisterType(OrderV3{}) ``` **Long coexistence periods** @@ -648,7 +643,7 @@ Supporting V1 for months creates maintenance burden. Set clear deprecation deadl **Registering after connection established** -Types must be registered before node starts. Dynamic registration requires connection cycling. +Types must be registered before connections are formed. Dynamic registration requires connection cycling. ## Summary diff --git a/docs/advanced/observer.md b/docs/advanced/observer.md new file mode 100644 index 000000000..dd2226b44 --- /dev/null +++ b/docs/advanced/observer.md @@ -0,0 +1,259 @@ +--- +description: Real-time inspection and management of Ergo nodes +--- + +# Inspecting With Observer + +This page walks through each page of the Observer web interface. For installation and configuration, see [Observer Application](../extra-library/applications/observer.md). + +The sidebar contains a node selector listing all nodes discovered through the registrar. Select a different node and Observer switches to showing that node's data. You deploy Observer on one node and monitor the entire cluster from a single browser tab. + +To try Observer with a live cluster: + +```bash +git clone https://github.com/ergo-services/examples +cd examples/observability +make up +``` + +This starts a multi-node cluster with Observer, tracing, health probes, Prometheus metrics, and Grafana dashboards. Open `http://localhost:9911` for Observer, `http://localhost:8888/dashboards` for Grafana. + +## Dashboard + +
Observer Dashboard
+ +The dashboard is the landing page with summary cards, real-time charts, and node-wide counters. Two controls let you manage the node directly from here: the log level dropdown changes the node-level severity threshold (see [Logging](../basics/logging.md)), and the [tracing](distributed-tracing.md) sampler dropdown controls whether the node starts new traces for messages sent via `node.Send()` and `node.Call()`. Both take effect immediately. + +The applications page lets you manage the lifecycle of applications running on the node: start in a selected [mode](../basics/application.md#application-modes), stop, or unload. When something goes wrong at the application level, this is where you act. But most investigation happens one level deeper, at individual processes. + +## Processes + +The processes page is where you spend most of your time when investigating issues. + +Every process on the node appears in a table that updates every second. The columns cover identification (PID, name, behavior, application), messaging (messages in/out, mailbox depth, latency), and lifecycle (running time, init time, wakeups, uptime, state). This is enough to answer most diagnostic questions without opening individual process details. + +When message counts change between updates, a green delta indicator appears next to the number. A "+42" next to Messages In tells you this process received 42 messages in the last second. The mailbox column changes color as the queue grows, making overloaded processes visually obvious in a list of thousands. The state column shows how long the process has been in its current state. A process stuck in "running" for 30 seconds is probably blocked inside a handler. + +All columns are sortable. Clicking Messages In sorts by busiest processes. Clicking Mailbox puts the most backlogged processes at the top. Clicking Running Time reveals which processes spend the most time executing handlers. + +Click any PID to open a floating detail window. + +### Scope + +The table does not show all processes at once. What you see is controlled by the Scope panel, which determines what the server sends to the browser. + +The scope works in two modes. In the default mode, you choose a window into the process ID space: "first 500" returns the 500 oldest processes (lowest PIDs), "last 500" returns the 500 newest, and entering a specific PID starts the window from that point. The node scans only the requested range and applies filters within it. This is fast even on nodes with tens of thousands of processes because the node never iterates beyond the requested window. + +The "All" mode switches to a full scan: the node iterates all processes, applies filters during iteration, and returns up to 10,000 matches. This mode requires at least one filter to be active to prevent the browser from receiving an unmanageable amount of data. + +Filters narrow results by name, behavior type, application, state, or minimum mailbox depth. In windowed mode, filters reduce the result count within the window. In All mode, filters are applied during the scan so only matching processes are counted toward the limit. + +Active filters appear as removable chips in the toolbar, and the scope label shows a compact summary like `first 500` or `last 100 . name:"worker"`. A separate search field adds client-side regex filtering on top of the server results for quick ad-hoc lookups without changing the scope. + +
+Processes page with scope panel + +
Processes page
+ +**Mailbox.** Total messages across all four mailbox queues (Main, System, Urgent, Log). Changes color as the queue grows: yellow for moderate, red for deep backlog. + +**Latency.** Time between a message entering the mailbox and the process starting to handle it. High latency means the process has a backlog and incoming messages are waiting. Requires the `latency` build tag to be enabled (see [Debugging](debugging.md)). + +**Running Time.** Total time spent inside handler callbacks (HandleMessage, HandleCall). High running time relative to uptime means the process spends most of its life executing handlers, whether due to computation or blocking I/O. + +**Init Time.** Time spent in the `Init` callback during startup. Highlighted red if over one second. Keep initialization fast: spawn has a timeout, and under a supervisor a slow Init blocks the restart of sibling processes. + +**Wakeups.** How many times the process was activated to handle messages. Each activation processes one batch from the mailbox. A high wakeup count with low message counts can indicate many small deliveries. + +
+ +## Process Details + +Floating detail windows are the primary tool for investigating individual processes. Multiple windows can be open simultaneously. They persist when you switch between pages, so you can keep a problematic process open while you check logs or traces elsewhere. + +The overview tab shows two real-time charts. The messages chart tracks incoming and outgoing message rates over the last 60 seconds, with a toggle between rate and cumulative views. The mailbox chart tracks the four queue depths: Main, System, Urgent, and Log. Below the charts, cards show running time, init time, and uptime. If the init time is suspiciously long, you know the process took a while to start. If the running time is high relative to uptime, the process is spending most of its life inside handlers rather than waiting for messages. The parent and leader processes appear as clickable links that open their own windows. + +The relations tab reveals the process's connections: aliases it has registered, meta processes it owns, events it has created, and its links and monitors grouped by type. This is valuable when you need to understand the supervision tree or figure out which processes will be affected if this one terminates. + +The inspect tab shows the output of the process's `HandleInspect` callback as key-value pairs. If your actor implements this method, it can expose internal state: queue lengths, cache sizes, connection counts, or any application-specific metrics. Auto-refresh polls the process once per second. + +### Managing a Process + +The config tab lets you change settings that take effect immediately. You can raise the log level to get more verbose output from a specific process, enable compression for network messages, change the tracing sampler for targeted diagnostics, or adjust message priority and delivery guarantees. The environment variables section is available if the node has `ExposeEnvInfo` enabled in its security settings. + +Three action buttons let you interact with the process. Send Message opens a dialog with a text field; the message is sent as a string value to the process. Send Exit sends an exit signal with a configurable reason. Kill forcefully terminates the process. These actions are disabled for system processes. + +
+Process detail window + +
Process detail window
+ +
+ +## Events + +The events page works like the processes page: it shows only what the scope defines, not the full list. + +Each row includes the event name, the producer process, registration time, subscriber count, and message statistics. Delta indicators highlight which events are actively publishing. The default sort is by registration time, newest first. + +The Scope panel controls which events the server returns. The From control chooses between First (oldest registered) and Last (newest registered). The node iterates events in registration order and stops after collecting the requested number of matches. Filters narrow by name, notify mode, buffered mode, and minimum subscriber count, and are applied during iteration so only matching events count toward the limit. + +Three toggle buttons in the toolbar control how the Registered column displays timestamps: 24h/12h clock format, raw millisecond timestamps for precise correlation, and an optional date prefix. These settings are shared with the Log and Tracing pages. + +
+Events page + +
Events page
+ +**Published.** Total number of times PublishEvent was called by the producer. Each call increments this counter once regardless of how many subscribers receive the message. + +**Local Sent.** Total messages delivered to local subscribers. If one publish reaches 5 local subscribers, this increments by 5. + +**Remote Sent.** Total messages sent to remote nodes. Counted per remote node, not per subscriber. If a remote node has 10 subscribers, this increments by 1 because the framework uses [shared subscriptions](pub-sub-internals.md#network-optimization-shared-subscriptions) to send one message per node. + +**Fanout.** Ratio of Local Sent to Published. Shows the average number of local deliveries per publish. A fanout of 3.0 means each publish reaches about 3 local subscribers. + +**Buffer.** Current messages in the event's ring buffer / buffer capacity. [Buffered events](pub-sub-internals.md#buffered-events-partial-optimization) retain recent messages so that new subscribers receive catch-up data. Yellow highlight if the buffer has pending messages. + +**Notify.** Whether the producer receives [notifications](pub-sub-internals.md#producer-notifications) (`MessageEventStart`/`MessageEventStop`) when the first subscriber arrives or the last subscriber leaves. + +
+ +## Network + +The network page shows how the node connects to the rest of the cluster. + +The top section displays network configuration: mode, max message size, handshake and protocol versions, and negotiated flags. The registrar section shows the service discovery backend with its capabilities. + +The acceptors section lists network listeners with their addresses, TLS configuration, and per-acceptor flags. + +Below the acceptors, the page splits into three tabs. + +The **Connections** tab is the default view. Four real-time charts show aggregate traffic across all connections: messages per second (in/out), bytes per second (in/out), compression operations per second (sent/received), and fragmentation operations per second (sent/received). A connection list table with its own scope controls shows all connections with delta indicators for message and byte counts. Click a row to open a floating window with detailed connection statistics. + +The **Routes** tab shows configured static routes and proxy routes side by side. Static routes are user-defined patterns that tell the node where to dial when a name matches; proxy routes describe how to reach nodes via an intermediate proxy. + +The **Types** tab is a one-shot view of the wire-format type registry. Each row shows registration ID, owning proto (the protocol version that registered the type), kind, MinSize (wire size of a zero-value), and canonical name. Click a row to expand its inferred schema (Go-syntax shape, multi-line for structs). Two filters at the top of the panel narrow the list by name and by schema content (useful for finding all types containing a specific field). The Refresh button re-fetches the registry; the panel does not subscribe to live updates because the registry rarely changes after node startup. + +When the node is built with `-tags=typestats`, four additional columns appear: **Encoded** and **Decoded** (operation counts), **Bytes Out** and **Bytes In** (decompressed wire-byte totals with average per operation). Counters reflect only root encode/decode at the message boundary; bytes folded inside other messages are accounted to the parent type. See [The typestats Tag](debugging.md#the-typestats-tag) for what gets counted and how to use the averages to pick compression candidates. + +The cluster nodes section shows all nodes known through the registrar or active connections, giving you a picture of the cluster topology. + +
+Network page + +
Network page
+ +**Node.** Contains several elements: a direction arrow, the node name, a CRC32 badge, and a TLS badge. The blue arrow (up-right) means the connection was initiated by this node (outgoing). The green arrow (down-left) means the connection was accepted from the remote node (incoming). The badge shows "TLS" if the connection uses TLS or "Plain" if it does not. + +**Node Uptime / Connection Uptime.** Node uptime is how long the remote node has been running. Connection uptime is how long this specific connection has been active. If the connection was recently re-established after a network issue, connection uptime will be shorter than node uptime. + +**Pool.** Number of TCP connections in the ENP protocol pool for this logical connection. Higher pool size allows more parallel message delivery. + +**Reconnections.** How many times the connection was re-established. Non-zero values are highlighted in red. Frequent reconnections may indicate network instability. + +**Clock Skew.** Measured difference between the local and remote node clocks. Used by the tracing waterfall to compensate for clock drift when displaying cross-node traces. + +
+ +### Connection Details + +Clicking a connection row opens a floating window with full connection information. + +At the top, four metric cards show messages and bytes in each direction. The identity section shows node and connection uptimes, framework and protocol versions, max message size, and negotiated network flags as colored pills (Remote Spawn, Fragmentation, Important Delivery, etc.). Each flag shows green if both nodes agreed to enable it. + +Below the identity section, the pool size and reconnection counter are shown. For outgoing connections, the Pool DSN lists the addresses of TCP connections in the pool. + +Two real-time charts track messages per second and bytes per second in each direction. If the connection carries proxy traffic, a third chart shows transit throughput. + +The compression section shows how many messages were compressed and decompressed, the compression ratio, and total bytes saved. The fragmentation section shows fragment counts and reassembly timeouts. These sections help diagnose whether compression and fragmentation are working efficiently or causing overhead. + +A "Switch observer to this node" button lets you start inspecting the remote node directly. + +
+Connection detail window + +
Connection detail window
+ +
+ +## Log + +The log page captures log messages in real time from every source on the node: processes, meta processes, the node itself, and the network stack. + +Each log entry shows a timestamp, severity level (color-coded badge), source, registered name, behavior type, and message text. The source column identifies where the message came from: a process PID, meta-process alias, node CRC, or network peer, each with its own color. The rich source toggle adds a type icon and makes the source clickable, opening a floating window for the process, meta-process, or network connection that generated the message. Long messages (over 200 characters or containing newlines) are truncated to three lines and expandable with a click. If the log entry carries structured fields, they appear below the message as key=value pairs. + +The Scope panel controls what the server captures. Level toggle buttons let you enable or disable each severity independently. This is server-side filtering: disabling debug means the server stops collecting debug messages entirely, reducing overhead on the node. Additional filters match against source, behavior, field names/values, and message text, with an exclude mode to filter out noise. The limit controls the ring buffer size. + +The Play/Pause button stops log capture without disconnecting. When you spot something interesting, pause and read through existing entries without new messages pushing them away. + +When the server drops messages because the ring buffer is full, a suppressed count indicator appears as a yellow alert in the toolbar. If you see this frequently, increase the limit in the scope panel. + +
+Log page + +
Log page
+ +
+ +## Profiler + +The profiler page has two tabs and a GC Pressure section that is always visible at the top. The key difference between the tabs: the Heap tab updates continuously via a live subscription, while the Goroutines tab captures snapshots on demand when you press the Capture button. + +The GC Pressure section shows four real-time charts: allocation rate (objects per second), dead rate (objects collected per second), live ratio (percentage of allocated objects still alive), and GC CPU fraction (percentage of CPU spent in garbage collection). These help you spot memory pressure trends before they become problems. + +### Heap + +The Heap tab updates continuously and shows allocation records sorted by in-use bytes. Each record shows in-use bytes, in-use objects, total allocated bytes, total allocated objects, and the function name (the first non-runtime function in the allocation stack). Expanding a record reveals the full stack trace. A scope panel filters by function name and limits how many records the server returns. A Pause button freezes the current data so you can examine it without updates overwriting what you are reading. + +Use the heap view when memory grows unexpectedly. The allocation stack traces show exactly which code paths are responsible. If a single function dominates the in-use bytes, that is your starting point. + +### Goroutines + +The Goroutines tab captures snapshots on demand. Press the Capture button to take a goroutine dump. The dump groups goroutines by their call stack: if 500 goroutines are all blocked on the same channel receive, they appear as one group with count 500. Each group shows the count, state (running, IO wait, chan receive, select, sleep, semacquire), wait duration (color-coded: green under 60s, yellow under 5 minutes, red above), and two function names: Origin (where the goroutine was spawned) and Current (where it is now). Expanding a group reveals the full stack trace and goroutine IDs. A scope panel filters the server-side capture by stack content, state, and minimum wait time. A search field filters the captured results client-side. + +This is how you diagnose deadlocks and blocking. Filter by state to isolate goroutines stuck in "chan receive". Search by package name to find goroutines from specific actors. A large group with a long wait time in a state that should be transient usually points directly at the problem. + +
+Profiler + +
Profiler
+ +
+ +## Tracing + +The tracing page shows distributed traces. Traces are collected continuously while Observer is connected, so data is already available when you navigate here. For background on how tracing works, see [Distributed Tracing](distributed-tracing.md). + +Because Observer connects to one node at a time, it shows only the observations emitted on that node. For complete cross-cluster traces, use [Pulse](../extra-library/applications/pulse.md) with Grafana Tempo or Jaeger. + +### Trace List + +Traces are sorted newest first. Each row shows a copyable trace ID, timestamp, root process PID with the root message type, an error icon if any span recorded an error, the span count, a duration bar showing this trace's duration as a proportion of the longest trace in the current scope buffer (red if any span recorded an error), and the total duration. + +The search field filters across trace ID, root process, root message, root node, and within spans across span ID, from, to, message text, and attribute keys and values. The Pause button stops the page from accepting new traces until resumed. The Clear button removes all collected traces. + +### Waterfall + +Click a trace row to expand its waterfall. The waterfall groups all observation points (Sent, Delivered, Processed) for the same message into a single row and arranges rows in a tree by parent-child relationships, with indentation showing the causal chain. + +Each row shows a color-coded kind badge (SEND in blue, CALL in violet, RESP in green, SPAWN in amber, TERM in red), the sender and receiver PIDs with their behavior types, the message type, and a timeline bar. The bar renders two phases: a lighter segment for transit time (Sent to Delivered) and a full segment for processing time (Delivered to Processed). Three dot markers show the observation points: blue for Sent, green for Delivered, orange for Processed. + +Hovering over the bar shows a tooltip with the node name at each point and the transit and processing durations. The duration column shows both the total and the breakdown. For cross-node spans (where Sent and Delivered happen on different nodes), the transit time calculation subtracts the measured clock skew between the nodes to show a more accurate transit duration. + +Local process PIDs in the waterfall are clickable and open detail windows. + +Click a span row to expand its detail panel. The panel has two columns: the left shows span fields (trace ID, span ID, kind, which points are present, behavior, from, to, call reference, message, node names, error) and the right shows custom attributes merged from all observation points. All values are copyable. + +Expanded traces persist when switching to other pages and back. + +### Scope + +The Scope panel has toggle buttons for span kinds (SEND, CALL, RESP, SPAWN, TERM) and observation points (Sent, Delivered, Processed). Disabled items appear with strikethrough. A message pattern filter matches against message type and error text, with an exclude toggle that inverts the match. The buffer limit controls how many traces are kept. Active scope filters appear as removable pills below the toolbar with a "Clear all" link. + +
+Tracing page with waterfall + +
Tracing page with waterfall
+ +
diff --git a/docs/advanced/pub-sub-internals.md b/docs/advanced/pub-sub-internals.md index 770b995aa..00d212229 100644 --- a/docs/advanced/pub-sub-internals.md +++ b/docs/advanced/pub-sub-internals.md @@ -442,7 +442,7 @@ process.SendEvent("market.prices", token, update) **What subscribers see:** ```go -func (c *Consumer) HandleEvent(message gen.MessageEvent) error { +func (c *Consumer) HandleEvent(event gen.MessageEvent) error { // Event arrives in your mailbox // Same timing whether you're the only subscriber or one of thousands // Same timing whether producer is local or remote @@ -724,6 +724,8 @@ func (p *Producer) HandleMessage(from gen.PID, message any) error { You only receive notifications when crossing the zero threshold. The notifications answer: "is anyone listening?" - not "how many are listening?" +Node-level events do not produce these notifications. The producer of a node-level event is the node core, which does not consume `MessageEventStart` or `MessageEventStop` messages. + ### Practical Use Case: On-Demand Data Production ```go diff --git a/docs/ai-agents.md b/docs/ai-agents.md new file mode 100644 index 000000000..63b90dd08 --- /dev/null +++ b/docs/ai-agents.md @@ -0,0 +1,229 @@ +--- +description: Build, run, and diagnose multi-agent AI systems on Ergo Framework +--- + +# AI Agents + +Modern AI systems are multi-agent by nature. A research agent delegates to an analysis agent. A planner coordinates with executors. A conversation manager spawns short-lived task agents. Moving from a demo with a handful of agents to a production deployment with hundreds or thousands surfaces the same problems that distributed systems have solved for decades: crash isolation, supervision, cross-node coordination, observability at scale. + +Ergo was built for telecom workloads where these requirements are baseline. AI agents have the same profile: many concurrent isolated workers with fault tolerance, coordination, and real-time behavior. This page shows how to use Ergo as runtime for your agents and as a live diagnostic surface for the running system. + +## Why Ergo fits AI agents + +Four problems appear as soon as you move AI agents out of a notebook: + +**Agent crashes.** One stuck LLM call or panicking tool handler takes down the whole process. Everything running in that process dies with it. + +**Coordination.** Agents need to talk to each other. Without a framework this becomes a web of channels, shared state, and custom routing code. + +**Observability.** You can't see what's happening inside a running agent system. Mailbox depth, per-agent CPU, which agents are waiting on which external calls, where cascade failures originate. + +**Scaling.** Distributing agents across nodes requires rethinking addressing, message delivery, and failure semantics. + +Ergo addresses all four: isolated processes with supervision, named event streams for coordination, a built-in MCP diagnostic surface, and network-transparent PIDs. The design choices were made for telecom-class distributed systems. The fit to AI workloads is incidental and exact. + +## Your agent as an actor + +An AI agent in Ergo is just an actor: a process with private state and a mailbox, handling messages sequentially. + +```go +type ResearchAgent struct { + act.Actor + notes []string +} + +type MessageResearchTask struct { + Query string + ReplyTo gen.PID +} + +func (a *ResearchAgent) HandleMessage(from gen.PID, msg any) error { + switch m := msg.(type) { + case MessageResearchTask: + result := callLLM(m.Query) // blocking call, isolated per agent + a.notes = append(a.notes, result) + a.Send(m.ReplyTo, result) + } + return nil +} + +func factory_ResearchAgent() gen.ProcessBehavior { return &ResearchAgent{} } +``` + +What you get automatically: + +- **Crash isolation.** A panicking LLM call or tool handler terminates only this actor. See [Process](basics/process.md). +- **Supervision.** Put the agent under a supervisor and it restarts on failure with your chosen strategy. See [Supervisor](actors/supervisor.md). +- **Distributed addressability.** Each agent has a PID that works across nodes. See [Remote Spawn Process](networking/remote-spawn-process.md). +- **Event-based coordination.** Agents publish to and subscribe to named event streams, fanning out one network message per node instead of one per subscriber. See [Events](basics/events.md). +- **Live diagnostics.** Expose the running system to any AI assistant via the [MCP application](extra-library/applications/mcp.md). + +The actor's private state (`notes` in the example) is safe without any synchronization. Messages arrive one at a time. The actor never shares memory with anyone. + +## Multi-agent architecture patterns + +### Agent Pool + +Run N identical worker agents and distribute incoming tasks across them. Ideal for stateless agents that process requests in parallel (LLM calls, embedding lookups, tool invocations). + +```go +type AgentPool struct { + act.Pool +} + +func (p *AgentPool) Init(args ...any) (act.PoolOptions, error) { + return act.PoolOptions{ + PoolSize: 10, + WorkerFactory: factory_ResearchAgent, + }, nil +} + +func factory_AgentPool() gen.ProcessBehavior { return &AgentPool{} } + +// Spawn the pool +poolPID, _ := node.Spawn(factory_AgentPool, gen.ProcessOptions{}) + +// Send tasks. The pool forwards to an available worker automatically. +node.Send(poolPID, MessageResearchTask{Query: "Summarize Q3 report"}) +``` + +Pool size and worker mailbox size together form a natural rate limit: at most `PoolSize × WorkerMailboxSize` tasks in flight. See [Pool](actors/pool.md). + +### Agent Pipeline + +Chain agents by sending from one stage to the next. Each stage runs under a supervisor. Failure in any stage is isolated and restarted. + +```go +// ResearchAgent forwards its result to AnalysisAgent +func (a *ResearchAgent) HandleMessage(from gen.PID, msg any) error { + switch m := msg.(type) { + case MessageResearchTask: + findings := a.research(m.Query) + a.Send(a.analysisPID, MessageAnalyze{Findings: findings, ReplyTo: m.ReplyTo}) + } + return nil +} +``` + +If `AnalysisAgent` crashes, the supervisor restarts it without affecting the other stages. Pipelines compose naturally with pools: a stage can be a single actor or a pool of identical workers. + +### Distributed Agent Cluster + +Spawn agents on specific nodes and address them with the same API as local agents. + +```go +// Register the factory on the target node. Security: only named factories +// can be spawned remotely. +network.EnableSpawn("research-agent", factory_ResearchAgent) + +// From any other node, get a handle and spawn +remote, _ := node.Network().GetNode("worker@otherhost") +pid, _ := remote.Spawn("research-agent", gen.ProcessOptions{}) + +// Send works identically whether pid is local or remote +node.Send(pid, MessageResearchTask{Query: "..."}) +``` + +See [Remote Spawn Process](networking/remote-spawn-process.md) for the security model and application-level inheritance. + +### Event-Driven Coordination + +Agents communicate through named event streams. One producer, any number of subscribers on any nodes. + +```go +// Producer: research agent publishes findings +token, _ := producer.RegisterEvent("research.findings", gen.EventOptions{}) +producer.SendEvent("research.findings", token, Finding{Topic: "market-trends"}) + +// Subscribers on any nodes +process.MonitorEvent(gen.Event{Name: "research.findings", Node: "research@host"}) +``` + +The framework delivers one network message per subscriber node regardless of how many subscribers that node has. 1M subscribers across 10 nodes cost 10 network messages, not 1M. See [Events](basics/events.md) and [Pub/Sub Internals](advanced/pub-sub-internals.md). + +## Live diagnostics for AI systems + +AI agents are nondeterministic. Behavior depends on prompts, external API latency, model temperature, and tool responses. Predefined metrics cover known failure modes, but the interesting failures are the ones you didn't anticipate. + +Add the [MCP application](extra-library/applications/mcp.md) to your node: + +```go +import "ergo.services/application/mcp" + +node, _ := ergo.StartNode("mynode@localhost", gen.NodeOptions{ + Applications: []gen.ApplicationBehavior{ + mcp.CreateApp(mcp.Options{Port: 9922}), + }, +}) +``` + +Connect Claude Code (or any MCP-compatible client): + +``` +claude mcp add --transport http ergo http://localhost:9922/mcp +``` + +Now you describe a symptom in plain English and the AI runs a diagnostic sequence against the live system: + +``` +You: "Why is the order processing agent slow?" + +AI: Checking process list sorted by mailbox... + -> order_processor has 847 queued messages (normal: <10) + Inspecting order_processor upstream dependencies... + -> payment_validator is processing 1 message per 3.2 seconds + Checking payment_validator CPU profile... + -> 73% time in external_api.Call(). The payment API is the bottleneck. +``` + +The MCP application exposes 48 diagnostic tools covering process inspection, profiling, cluster visibility, and samplers (continuous data collection into ring buffers for trend analysis). One entry point node gives access to every node in the cluster. Other nodes run MCP in agent mode without exposing an HTTP port. + +For the full toolkit, cluster proxy mechanics, sampler recipes, profiling options, and Claude Code plugin configuration, see [MCP](extra-library/applications/mcp.md). + +## Getting started + +``` +# Install the project generator +go install ergo.tools/ergo@latest + +# Create a project +ergo init AgentNode github.com/myorg/agentnode +cd agentnode + +# Add components +ergo add supervisor AgentNodeApp:AgentSup +ergo add actor AgentSup:ResearchAgent +ergo add actor AgentSup:AnalysisAgent + +# Run +go run ./cmd +``` + +Add MCP to the generated node setup: + +```go +import "ergo.services/application/mcp" + +options.Applications = []gen.ApplicationBehavior{ + agentnodeapp.CreateApp(), + mcp.CreateApp(mcp.Options{Port: 9922}), +} +``` + +Connect your AI assistant and start investigating: + +``` +claude mcp add --transport http ergo http://localhost:9922/mcp +``` + +## Cloud-connected agents + +Running agents across AWS, GCP, Azure, or bare metal is supported via [ergo.cloud](https://ergo.cloud), a managed overlay network that connects nodes without VPNs, proxies, or tunnels. End-to-end encrypted. Currently available via waitlist. + +## Next steps + +- [Process](basics/process.md) for the actor lifecycle +- [Supervisor](actors/supervisor.md) for restart strategies +- [Events](basics/events.md) for pub/sub coordination +- [MCP](extra-library/applications/mcp.md) for live diagnostics and AI-driven investigation +- [Examples](https://github.com/ergo-services/examples) for working reference projects diff --git a/docs/basics/events.md b/docs/basics/events.md index b51b89670..ef0dd758d 100644 --- a/docs/basics/events.md +++ b/docs/basics/events.md @@ -21,12 +21,33 @@ token, err := process.RegisterEvent("price_update", gen.EventOptions{ }) ``` -The `Notify` option controls whether the producer receives notifications about subscriber changes. When enabled, the producer receives `gen.MessageEventStart` when the first subscriber appears and `gen.MessageEventStop` when the last subscriber leaves. This allows the producer to start or stop expensive operations based on demand. If nobody's watching the price feed, why fetch prices? +The `Notify` option controls whether the producer receives notifications about subscriber changes. When enabled, the producer receives `gen.MessageEventStart` when the first subscriber appears and `gen.MessageEventStop` when the last subscriber leaves. This allows the producer to start or stop expensive operations based on demand. If nobody's watching the price feed, why fetch prices? This option is ignored for events registered at the node level, since the node core does not consume such messages. The `Buffer` option specifies how many recent events to keep. When a new subscriber joins, it receives the buffered events as a catch-up mechanism. Set this to zero if events are only relevant at the moment they're published. Set it to a reasonable number if new subscribers should see recent history. Events are identified by name and node. The combination must be unique. Two processes on the same node can't register events with the same name. But processes on different nodes can register events with the same name - they're different events. +## Open Events + +*Introduced in v3.3.0.* + +By default, only the token holder can publish to an event. This prevents unauthorized processes from publishing events they don't own. For events that represent an internal node-wide bus, this protection is sometimes more friction than benefit. You end up distributing the token across multiple processes, or through environment variables, just so known participants can publish. + +The `Open` option disables the token check on publish. + +```go +token, _ := process.RegisterEvent("app.events", gen.EventOptions{ + Open: true, + Buffer: 50, +}) +``` + +Any local process can now publish to this event by name, regardless of the token value. The owner check on `UnregisterEvent` is unaffected. Only the registering process (or the node, for node-level events) can unregister. + +Consider a bus inside the node where application events land: "order created", "user signed up", "payment received". They come from different modules, and subscribers (notifier, analytics, search indexer) pick up whichever ones matter. Nobody owns the bus. Handing a shared token to every emitter is plumbing that protects nothing. + +Open events trade the typo and bug protection that the token provides for simpler distribution. A process can accidentally publish to an event it was never supposed to touch. Use this option when the event is deliberately a shared bus and the token ceremony adds no real security in your context. + ## Publishing Events Publishing an event sends it to all current subscribers. @@ -75,6 +96,45 @@ The producer can explicitly unregister an event with `UnregisterEvent`. This tri If a subscriber terminates or unsubscribes (via `UnlinkEvent` or `DemonitorEvent`), the producer doesn't receive notification unless `Notify` was enabled. With `Notify`, the producer receives `gen.MessageEventStop` when the last subscriber leaves. +## Node-Level Events + +All examples so far registered events from a process. That process is the producer, and the event exists only as long as the process runs. When the process terminates, the event is unregistered and subscribers receive termination notifications. If another process later registers the same event name, subscribers must subscribe again. + +Some events belong to the node itself, not to any particular process. Application events, health signals, internal buses. You want these events to exist for the entire lifetime of the node, regardless of which process currently publishes. The `gen.Node` interface provides `RegisterEvent` for this. + +```go +token, err := node.RegisterEvent("notifications", gen.EventOptions{ + Open: true, + Buffer: 100, +}) +``` + +The event's producer is the node core. It survives any publisher process coming and going. A process that restarts continues publishing to the same event after restart. Subscribers are not affected. + +### Race on Subscription + +*Introduced in v3.3.0.* + +There is a timing problem with process-registered events. If a subscriber's `Init()` tries to `LinkEvent` before the producer process has called `RegisterEvent`, the link fails with `gen.ErrEventUnknown`. The subscriber then needs retry logic or some other coordination mechanism. + +The `NodeOptions.Events` field registers node-level events before any application is started. + +```go +options := gen.NodeOptions{ + Events: []gen.NodeEventSpec{ + {Name: "notifications", Buffer: 100}, + {Name: "audit", Buffer: 10}, + }, + Applications: []gen.ApplicationBehavior{...}, +} + +node, err := ergo.StartNode("mynode@localhost", options) +``` + +Events declared here are registered as open events with the node as producer. By the time the first application starts, these events already exist. Any process can subscribe from `Init()` without a race. Any process can publish by name. + +If your event requires the token check and you only want a specific process to publish, register it imperatively via `node.RegisterEvent(..., gen.EventOptions{Open: false})` and distribute the token through environment variables or process arguments. + ## Network Transparency Events work across nodes seamlessly. A producer on node A can publish events that subscribers on nodes B, C, and D receive. The framework handles the network distribution. @@ -104,6 +164,38 @@ Each `gen.MessageEvent` contains: Subscribers receive these wrapped messages and extract the application data. The wrapping provides context: which event this came from, when it was published, allowing subscribers to handle events from multiple sources or correlate timing. +## Event Statistics + +Each registered event tracks per-event counters: how many messages were published, how many were delivered to local subscribers, and how many were sent to remote nodes. These counters are available through `Node.EventInfo` and `Node.EventRangeInfo`. + +To query a specific event: + +```go +info, err := node.EventInfo(gen.Event{Name: "price_update", Node: "node@host"}) +// info.MessagesPublished - total messages published to this event +// info.MessagesLocalSent - messages delivered to local subscribers +// info.MessagesRemoteSent - messages sent to remote subscriber nodes +// info.Subscribers - current subscriber count +``` + +To iterate over all registered events on the node: + +```go +node.EventRangeInfo(func(info gen.EventInfo) bool { + fmt.Printf("event %s: published %d, local %d, remote %d\n", + info.Event.Name, + info.MessagesPublished, + info.MessagesLocalSent, + info.MessagesRemoteSent, + ) + return true // continue iteration +}) +``` + +Node-level aggregate counters are also available in `gen.NodeInfo` via `node.Info()`: `EventsPublished` (local producer publishes), `EventsReceived` (events arriving from remote nodes), `EventsLocalSent`, and `EventsRemoteSent`. + +The [Metrics actor](../extra-library/actors/metrics.md) automatically exports these counters as Prometheus metrics, along with per-event top-N breakdowns by subscribers, published, local deliveries, and remote sent. It also tracks event utilization state: whether events are actively used, waiting on demand, or idle. + ## Practical Patterns Events fit several common scenarios. diff --git a/docs/basics/logging.md b/docs/basics/logging.md index fd0355c64..8eeb47a08 100644 --- a/docs/basics/logging.md +++ b/docs/basics/logging.md @@ -36,7 +36,7 @@ The framework provides six severity levels, ordered from most to least verbose: `gen.LogLevelError` - Errors that prevent specific operations but don't crash the system. Failed requests, unavailable resources, validation failures. -`gen.LogLevelPanic` - Critical errors requiring immediate attention. Despite the name, logging at this level doesn't trigger a panic - it's just the highest severity marker. +`gen.LogLevelPanic` - Recovered panics inside actor callbacks. The highest severity marker. Setting a level creates a threshold. Set a process to `gen.LogLevelWarning` and it logs warnings, errors, and panics, but suppresses info, debug, and trace. Each level implicitly includes all higher severity levels. @@ -48,6 +48,8 @@ Two special levels control behavior rather than representing severity: Trace deserves special mention. It's so verbose that enabling it accidentally could flood storage. You can't enable it dynamically via `SetLevel`. It must be set at startup through `gen.NodeOptions.Log.Level` or `gen.ProcessOptions.LogLevel`. This restriction prevents operational mistakes. +Panic also deserves explanation. The framework recovers Go panics that occur inside actor callbacks and logs them at this level. A nil pointer dereference in `HandleMessage`, a failed type assertion in `HandleCall`, an index out of bounds in `Init` are all structural problems in actor code, not operational failures. Logging them at Panic level separates them from the business and technical errors you log at Error level. The framework catches these so your node keeps running, but the Panic log entry tells you something in your code needs fixing. Note that Go's standard library `log.Panic()` actually triggers a panic, while Ergo's `Log().Panic()` simply logs at the Panic severity level without panicking. If you are building actors with `act.Actor`, `act.Supervisor`, or `act.Pool`, you won't need to log at this level yourself. The framework handles it. It becomes relevant only if you implement an actor directly through the `gen.ProcessBehavior` interface and want to recover panics in your own processing loop. + The node starts at `gen.LogLevelInfo`. Processes inherit this unless their spawn options specify otherwise. After startup, you can adjust a process's level dynamically with `SetLevel`, allowing surgical verbosity changes during debugging. ## Identifying Log Sources @@ -282,10 +284,10 @@ Different processes often need different verbosity. Most processes log at Info. ```go // Debugging a specific process -node.SetLogLevelProcess(suspiciousPID, gen.LogLevelDebug) +node.SetProcessLogLevel(suspiciousPID, gen.LogLevelDebug) // Later, restore normal level -node.SetLogLevelProcess(suspiciousPID, gen.LogLevelInfo) +node.SetProcessLogLevel(suspiciousPID, gen.LogLevelInfo) ``` For processes generating high-volume logs, route them to a dedicated logger using a hidden logger. A trading engine logging every order would overwhelm general logs: diff --git a/docs/basics/node.md b/docs/basics/node.md index 25fa36cee..1dccd24fa 100644 --- a/docs/basics/node.md +++ b/docs/basics/node.md @@ -10,13 +10,13 @@ When you start a node, you're launching a complete system with several subsystem ## What a Node Provides -**Process Management** - The node tracks every process running on it. When you spawn a process, the node assigns it a unique PID, registers it in the process table, and manages its lifecycle. When a process terminates, the node cleans up its resources and notifies any processes that were linked or monitoring it. +**Process Management** - The node tracks every process running on it. When you spawn a process, the node assigns it a unique PID, registers it in the process table, and manages its lifecycle. When a process terminates, the node cleans up its resources and notifies any processes that were linked or monitoring it. The node provides `ProcessRangeShortInfo` for efficient iteration over all processes with their current state, including mailbox latency when built with `-tags=latency`. **Message Routing** - When a process sends a message, the node figures out where it needs to go. Local process? Route it directly to the mailbox. Remote process? Establish a network connection if needed and send it there. The sender doesn't need to know these details. **Network Stack** - The node handles all network communication. It discovers other nodes, establishes connections, encodes messages, and manages the complexity of distributed communication. This is what makes network transparency possible. -**Pub/Sub System** - Links, monitors, and events all work through a publisher/subscriber mechanism in the node core. When a process terminates or an event fires, the node knows who's subscribed and delivers the notifications. +**Pub/Sub System** - Links, monitors, and events all work through a publisher/subscriber mechanism in the node core. When a process terminates or an event fires, the node knows who's subscribed and delivers the notifications. The node provides `EventInfo` to query statistics for a specific event and `EventRangeInfo` for callback-based iteration over all registered events with their per-event counters (messages published, local/remote deliveries). **Logging** - Every log message goes through the node, which fans it out to registered loggers. This centralized logging makes it easy to capture, filter, and route log output. diff --git a/docs/basics/process.md b/docs/basics/process.md index acc65dcbe..c9688ad10 100644 --- a/docs/basics/process.md +++ b/docs/basics/process.md @@ -8,6 +8,8 @@ A process is an actor - a lightweight entity that handles messages sequentially Every process has a mailbox where incoming messages wait to be processed. The mailbox contains four queues with different priorities: Urgent for critical system messages, System for framework control, Main for regular application messages, and Log for logging. When the process wakes up to handle messages, it processes them in priority order, taking from Urgent first, then System, then Main, and finally Log. +When built with `-tags=latency`, each queue tracks the age of its oldest unprocessed message. `ProcessMailbox.Latency()` returns the maximum latency across all four queues in nanoseconds, or -1 if the tag is not enabled. This helps identify processes that are falling behind on message processing. See [Debugging](../advanced/debugging.md) for details. + The process runs only when it has messages to handle. When the mailbox is empty, the process sleeps, consuming no CPU. When a message arrives, the process wakes, handles the message, and sleeps again if nothing else is waiting. This efficiency is why you can have thousands of processes in a single application. ## Identifying Processes diff --git a/docs/basics/project-structure.md b/docs/basics/project-structure.md index 310a9d3c1..92bb5718e 100644 --- a/docs/basics/project-structure.md +++ b/docs/basics/project-structure.md @@ -269,7 +269,8 @@ package types import ( "time" - "ergo.services/ergo/net/edf" + + "ergo.services/ergo/gen" ) // Events published by the orders application @@ -285,14 +286,16 @@ type OrderCompleted struct { CompletedAt time.Time } -func init() { - // Register for network serialization - edf.RegisterTypeOf(OrderCreated{}) - edf.RegisterTypeOf(OrderCompleted{}) +// Helper that consumers call from their application's Load() callback. +func RegisterTypes(network gen.Network) error { + return network.RegisterTypes([]any{ + OrderCreated{}, + OrderCompleted{}, + }) } ``` -Both `apps/orders` and `apps/shipping` can import `types` without importing each other. This breaks the circular dependency while maintaining strong typing. +Both `apps/orders` and `apps/shipping` can import `types` and call `types.RegisterTypes(node.Network())` from their `Load(node)` callbacks. This breaks the circular dependency while maintaining strong typing. ### Shared Libraries (`lib/`) @@ -476,8 +479,6 @@ Messages that form public contracts between applications across the cluster. // types/commands.go package types -import "ergo.services/ergo/net/edf" - // EXPORTED type, EXPORTED fields // CAN be referenced by any package // CAN be serialized @@ -493,10 +494,29 @@ type TaskResult struct { Output []byte Error string } +``` + +Each consuming application registers the shared types from its `Load` callback: + +```go +// apps/worker/app.go +package worker -func init() { - edf.RegisterTypeOf(ProcessTask{}) - edf.RegisterTypeOf(TaskResult{}) +import ( + "ergo.services/ergo/gen" + + "myapp/types" +) + +func (a *Worker) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { + err := node.Network().RegisterTypes([]any{ + types.ProcessTask{}, + types.TaskResult{}, + }) + if err != nil { + return gen.ApplicationSpec{}, err + } + return gen.ApplicationSpec{ /* ... */ }, nil } ``` @@ -718,8 +738,8 @@ func (l *Listener) Init(args ...any) error { return nil } -func (l *Listener) HandleEvent(ev gen.MessageEvent) error { - switch e := ev.Message.(type) { +func (l *Listener) HandleEvent(event gen.MessageEvent) error { + switch e := event.Message.(type) { case types.OrderCompleted: l.createShipment(e.OrderID) } @@ -1134,7 +1154,7 @@ apps/ - Default to Level 4 for everything - Mix isolation levels arbitrarily - Use `any` or `interface{}` for messages -- Include pointers in network messages +- Include pointers to external resources (connections, files) in network messages ### Dependencies diff --git a/docs/extra-library/actors/README.md b/docs/extra-library/actors/README.md index b92ead6f2..0db998c11 100644 --- a/docs/extra-library/actors/README.md +++ b/docs/extra-library/actors/README.md @@ -2,6 +2,12 @@ An extra library of actors implementations not included in the standard Ergo Framework library. This library contains packages with a narrow specialization. It also includes packages with external dependencies, as Ergo Framework adheres to a "zero dependency" policy. +## [Health](health.md) + +Kubernetes health probe actor that serves `/health/live`, `/health/ready`, and `/health/startup` endpoints. Actors register named signals with probe type and optional heartbeat timeout. When a signal goes down, the corresponding probe endpoint returns 503. + +**Use cases:** Kubernetes liveness/readiness/startup probes, container orchestration integration, dependency health tracking, graceful degradation. + ## [Leader](leader.md) Distributed leader election actor implementing Raft-inspired consensus algorithm. Provides coordination primitives for building systems that require single leader selection across a cluster. diff --git a/docs/extra-library/actors/health.md b/docs/extra-library/actors/health.md new file mode 100644 index 000000000..d9fbaa254 --- /dev/null +++ b/docs/extra-library/actors/health.md @@ -0,0 +1,466 @@ +# Health + +The health actor provides Kubernetes-compatible health probe endpoints for Ergo applications. Instead of each application building its own HTTP health check logic, the health actor centralizes probe management into a single process that serves `/health/live`, `/health/ready`, and `/health/startup` endpoints. + +Actors register named signals with the health actor, optionally sending periodic heartbeats. The health actor aggregates signal states and serves HTTP responses that Kubernetes (or any other orchestrator) can use to determine whether to restart a pod, route traffic to it, or wait for it to finish starting. + +## The Problem + +Kubernetes uses three types of probes to manage pod lifecycle: + +**Liveness:** Is the application alive? A failing liveness probe causes Kubernetes to restart the pod. Use this for detecting deadlocks, infinite loops, or corrupted state that prevents the application from functioning. + +**Readiness:** Can the application serve traffic? A failing readiness probe removes the pod from service endpoints. Use this for temporary conditions like database connection loss, cache warming, or downstream dependency outages where restarting would not help. + +**Startup:** Has the application finished initializing? A failing startup probe prevents liveness and readiness checks from running. Use this for slow-starting applications that need time to load data, run migrations, or establish connections before health checks begin. + +In traditional applications, you implement these probes as HTTP handlers that check internal state. In actor systems, the "state" is distributed across many processes. A database connection actor, a cache warmer, and a message queue consumer each know their own status, but no single actor knows the overall health. + +The health actor solves this by accepting signal registrations from any actor in the system. Each actor reports its own status, and the health actor aggregates these signals into per-probe HTTP responses. + +## How It Works + +The health actor follows a registration and heartbeat pattern: + +1. **Actors register signals:** Each actor that contributes to health sends a `RegisterRequest` to the health actor (synchronous Call), specifying a signal name, which probes it affects, and an optional heartbeat timeout. The Call returns after the signal is registered, preventing race conditions with subsequent heartbeats. + +2. **The health actor monitors registrants:** When a signal is registered, the health actor monitors the registering process. If that process terminates, all its signals are automatically marked as down. + +3. **Actors send heartbeats:** For signals with a timeout, the registering actor periodically sends `MessageHeartbeat`. If the heartbeat interval exceeds the timeout, the health actor marks the signal as down. + +4. **HTTP handlers read atomic state:** The HTTP handlers read pre-built JSON responses from atomic values. The actor goroutine rebuilds these atomic values after every state change. No mutexes or channels are involved in serving HTTP requests. + +```mermaid +sequenceDiagram + participant DB as DB Actor + participant H as Health Actor + participant K as Kubernetes + + DB->>H: RegisterRequest{Signal: "db", Probe: Liveness|Readiness, Timeout: 5s} + H->>DB: RegisterResponse{} + Note over H: Monitor DB Actor
Signal "db" = up + + loop Every 2 seconds + DB->>H: MessageHeartbeat{Signal: "db"} + end + + K->>H: GET /health/live + H->>K: 200 {"status":"healthy","signals":[{"signal":"db","status":"up","timeout":"5s"}]} + + Note over DB: Process crashes + Note over H: MessageDownPID received
Signal "db" = down + + K->>H: GET /health/live + H->>K: 503 {"status":"unhealthy","signals":[{"signal":"db","status":"down","timeout":"5s"}]} + + Note over K: Restart pod +``` + +## ActorBehavior Interface + +The health actor extends `gen.ProcessBehavior` with a specialized interface: + +```go +type ActorBehavior interface { + gen.ProcessBehavior + + Init(args ...any) (Options, error) + HandleMessage(from gen.PID, message any) error + HandleCall(from gen.PID, ref gen.Ref, message any) (any, error) + HandleInspect(from gen.PID, item ...string) map[string]string + HandleSignalDown(signal gen.Atom) error + HandleSignalUp(signal gen.Atom) error + Terminate(reason error) +} +``` + +All callbacks have default (no-op) implementations. You only override what you need. + +`HandleSignalDown` is called when a signal transitions from up to down, due to heartbeat timeout, process termination, or explicit `MessageSignalDown`. Use this for alerting, logging, or triggering recovery actions. + +`HandleSignalUp` is called when a signal transitions from down to up, via heartbeat recovery or explicit `MessageSignalUp`. Use this to log recovery events or update external systems. + +## Basic Usage + +Spawn the health actor and register it with a name so other actors can find it: + +```go +package main + +import ( + "ergo.services/actor/health" + "ergo.services/ergo" + "ergo.services/ergo/gen" +) + +func main() { + node, _ := ergo.StartNode("mynode@localhost", gen.NodeOptions{}) + defer node.Stop() + + node.SpawnRegister(gen.Atom("health"), health.Factory, gen.ProcessOptions{}, + health.Options{Port: 8080}) + + // Endpoints: + // http://localhost:8080/health/live + // http://localhost:8080/health/ready + // http://localhost:8080/health/startup + node.Wait() +} +``` + +Default configuration: +- **Host**: `localhost` +- **Port**: `3000` +- **Path**: `/health` +- **CheckInterval**: `1 second` + +With no signals registered, all three endpoints return 200 with `{"status":"healthy"}`. This means a freshly started health actor does not block deployment. Signals opt in to health checking; only registered signals can cause a probe to fail. + +## Configuration + +```go +options := health.Options{ + Host: "0.0.0.0", // Listen on all interfaces + Port: 8080, // HTTP port + Path: "/health", // Path prefix (default: "/health") + CheckInterval: 2 * time.Second, // Heartbeat check interval +} +``` + +**Host** determines which network interface the HTTP server binds to. Use `"0.0.0.0"` for production/containerized environments. + +**Port** should not conflict with other services on the same pod. + +**Path** sets the prefix for health endpoints. Endpoints are registered as `Path+"/live"`, `Path+"/ready"`, `Path+"/startup"`. Change this when the default conflicts with your routing or when deploying behind a reverse proxy. For example, with `Path: "/k8s"` the endpoints become `/k8s/live`, `/k8s/ready`, `/k8s/startup`. + +**CheckInterval** controls how frequently the actor checks for expired heartbeats. The actor sends itself a timer message at this interval and iterates over all signals with a non-zero timeout, marking expired ones as down. Shorter intervals detect failures faster but increase message processing overhead. For most applications, 1-2 seconds provides a good balance. + +**Mux** accepts an external `*http.ServeMux`. When provided, the health actor registers its handlers on this mux and skips starting its own HTTP server. This is useful when you want to serve health endpoints alongside other HTTP handlers on a single port, for example, combining with the [Metrics](metrics.md) actor. + +```go +mux := http.NewServeMux() + +healthOpts := health.Options{Mux: mux} +node.SpawnRegister("health", health.Factory, gen.ProcessOptions{}, healthOpts) + +metricsOpts := metrics.Options{Mux: mux} +node.Spawn(metrics.Factory, gen.ProcessOptions{}, metricsOpts) + +// Serve the shared mux yourself +``` + +When `Mux` is set, `Host` and `Port` are ignored. + +## Signal Registration + +### Probe Types + +Each signal specifies which probes it affects using a bitmask: + +```go +const ( + ProbeLiveness Probe = 1 << iota // 1 -- /health/live + ProbeReadiness // 2 -- /health/ready + ProbeStartup // 4 -- /health/startup +) +``` + +Combine probes with bitwise OR. A database connection that affects both liveness and readiness: + +```go +health.Register(w, gen.Atom("health"), "db", + health.ProbeLiveness|health.ProbeReadiness, 5*time.Second) +``` + +A migration signal that only affects startup: + +```go +health.Register(w, gen.Atom("health"), "migrations", + health.ProbeStartup, 0) +``` + +When `Probe` is 0, it defaults to `ProbeLiveness`. + +### Helper Functions + +The package provides convenience functions: + +```go +// Register a signal (sync Call -- blocks until registered) +health.Register(process, to, signal, probe, timeout) + +// Remove a signal (sync Call -- blocks until removed) +health.Unregister(process, to, signal) + +// Send heartbeat (async Send) +health.Heartbeat(process, to, signal) + +// Manual control (async Send) +health.SignalUp(process, to, signal) +health.SignalDown(process, to, signal) +``` + +`Register` and `Unregister` use synchronous Call to confirm the operation completed. This prevents race conditions where a heartbeat or status update arrives before the signal is registered. All other helpers use async Send. + +The `to` parameter accepts anything that identifies a process: a `gen.Atom` name, `gen.PID`, `gen.ProcessID`, or `gen.Alias`. + +### Message Types + +If you prefer sending messages directly instead of using helpers: + +| Message | Type | Description | +|---------|------|-------------| +| `RegisterRequest` / `RegisterResponse` | sync (Call) | Register a signal. Fields: `Signal gen.Atom`, `Probe Probe`, `Timeout time.Duration` | +| `UnregisterRequest` / `UnregisterResponse` | sync (Call) | Remove a signal. Fields: `Signal gen.Atom` | +| `MessageHeartbeat` | async (Send) | Update heartbeat timestamp. Fields: `Signal gen.Atom` | +| `MessageSignalUp` | async (Send) | Mark a signal as up. Fields: `Signal gen.Atom` | +| `MessageSignalDown` | async (Send) | Mark a signal as down. Fields: `Signal gen.Atom` | + +All types are registered with EDF for network transparency. Actors on remote nodes can register signals with a health actor on any node in the cluster. + +## Heartbeat Pattern + +The heartbeat pattern is the primary mechanism for detecting failures in long-running dependencies. The actor that owns a resource (database, external API, message queue) knows best whether the resource is healthy. It registers a signal with a timeout and sends periodic heartbeats as long as the resource is available. + +```go +type DBWorker struct { + act.Actor + heartbeatTimer gen.CancelFunc +} + +type messageHeartbeatTick struct{} + +func (w *DBWorker) Init(args ...any) error { + // Register with 5-second heartbeat timeout + health.Register(w, gen.Atom("health"), "db", + health.ProbeLiveness|health.ProbeReadiness, 5*time.Second) + + // Send heartbeat every 2 seconds (well within the 5s timeout) + w.heartbeatTimer, _ = w.SendAfter(w.PID(), messageHeartbeatTick{}, 2*time.Second) + return nil +} + +func (w *DBWorker) HandleMessage(from gen.PID, message any) error { + switch message.(type) { + case messageHeartbeatTick: + health.Heartbeat(w, gen.Atom("health"), "db") + w.heartbeatTimer, _ = w.SendAfter(w.PID(), messageHeartbeatTick{}, 2*time.Second) + } + return nil +} + +func (w *DBWorker) Terminate(reason error) { + if w.heartbeatTimer != nil { + w.heartbeatTimer() + } +} +``` + +Choose the heartbeat interval to be at least 2x shorter than the timeout. This provides one missed heartbeat as a safety margin before the signal is marked as down. + +When the actor crashes, the health actor receives a `gen.MessageDownPID` (because it monitors the registrant) and marks all signals from that process as down. Heartbeat timeout is a secondary detection mechanism for situations where the process is alive but the resource it manages is not, for example, a database connection pool actor that is running but has lost all connections. + +## HTTP Endpoints + +| Path | Probe | Default (no signals) | +|------|-------|---------------------| +| `{Path}/live` | ProbeLiveness | 200 healthy | +| `{Path}/ready` | ProbeReadiness | 200 healthy | +| `{Path}/startup` | ProbeStartup | 200 healthy | + +Each endpoint evaluates only signals registered for that specific probe. A signal registered for `ProbeLiveness` only does not affect `/health/ready` or `/health/startup`. + +**200 OK:** all signals for this probe are up, or no signals are registered. + +**503 Service Unavailable:** at least one signal for this probe is down. + +### Response Format + +Healthy response with signals: + +```json +{"status":"healthy","signals":[{"signal":"db","status":"up","timeout":"5s"}]} +``` + +Unhealthy response: + +```json +{"status":"unhealthy","signals":[{"signal":"db","status":"down","timeout":"5s"},{"signal":"cache","status":"up"}]} +``` + +Healthy response with no signals (probe has no registered signals): + +```json +{"status":"healthy"} +``` + +The `timeout` field appears only for signals that have a heartbeat timeout configured. Signals without timeout omit this field. + +## Failure Detection + +The health actor detects failures through three mechanisms: + +### Process Termination + +When a process that registered signals terminates (normally or abnormally), the health actor receives `gen.MessageDownPID` through its monitor. All signals from that process are immediately marked as down. This is the fastest and most reliable detection mechanism. + +### Heartbeat Timeout + +For signals with a non-zero timeout, the health actor periodically checks whether the last heartbeat was received within the timeout window. If a heartbeat is overdue, the signal is marked as down and `HandleSignalDown` is called. + +Heartbeat timeout catches situations where the process is alive but the resource it monitors is unavailable. The process continues to run (so no `MessageDownPID` arrives) but stops sending heartbeats because the resource check fails. + +### Manual Control + +Actors can explicitly report status changes using `MessageSignalUp` and `MessageSignalDown`. Use this when you can detect failures immediately without waiting for a timeout, for example, catching a database connection error in a callback and immediately marking the signal as down, then marking it up again when the connection is re-established. + +## Extending with Custom Behavior + +Embed `health.Actor` in your own struct to add custom behavior: + +```go +type MyHealth struct { + health.Actor +} + +func MyHealthFactory() gen.ProcessBehavior { + return &MyHealth{} +} + +func (h *MyHealth) Init(args ...any) (health.Options, error) { + return health.Options{Port: 8080}, nil +} + +func (h *MyHealth) HandleSignalDown(signal gen.Atom) error { + h.Log().Error("signal went down: %s", signal) + // Alert external monitoring, update metrics, trigger recovery + return nil +} + +func (h *MyHealth) HandleSignalUp(signal gen.Atom) error { + h.Log().Info("signal recovered: %s", signal) + return nil +} +``` + +Override `HandleMessage` to handle application-specific messages alongside health management. The health actor dispatches its own types internally (`RegisterRequest`/`UnregisterRequest` via HandleCall, `MessageHeartbeat`/`MessageSignalUp`/`MessageSignalDown` via HandleMessage); only unrecognized messages are forwarded to your callbacks. + +## Kubernetes Configuration + +Configure Kubernetes probes to point to the health actor's endpoints: + +```yaml +apiVersion: v1 +kind: Pod +spec: + containers: + - name: myapp + livenessProbe: + httpGet: + path: /health/live + port: 3000 + initialDelaySeconds: 5 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /health/ready + port: 3000 + initialDelaySeconds: 5 + periodSeconds: 10 + startupProbe: + httpGet: + path: /health/startup + port: 3000 + failureThreshold: 30 + periodSeconds: 2 +``` + +Adjust `initialDelaySeconds` based on how long your application takes to start and register signals. The startup probe with `failureThreshold: 30` and `periodSeconds: 2` gives the application 60 seconds to complete initialization before Kubernetes considers it failed. + +## Common Patterns + +### Database Health + +Register liveness and readiness signals with heartbeat: + +```go +func (w *DBWorker) Init(args ...any) error { + health.Register(w, gen.Atom("health"), "postgres", + health.ProbeLiveness|health.ProbeReadiness, 10*time.Second) + + w.scheduleHeartbeat() + return nil +} + +func (w *DBWorker) HandleMessage(from gen.PID, message any) error { + switch message.(type) { + case messageHeartbeat: + if w.db.Ping() == nil { + health.Heartbeat(w, gen.Atom("health"), "postgres") + } + w.scheduleHeartbeat() + } + return nil +} +``` + +If `db.Ping()` fails, no heartbeat is sent, and the signal times out. The health actor marks it as down, causing Kubernetes to remove the pod from service endpoints (readiness) and eventually restart it (liveness). + +### Startup Gate + +Use the startup probe for slow initialization: + +```go +func (w *Migrator) Init(args ...any) error { + health.Register(w, gen.Atom("health"), "migrations", + health.ProbeStartup, 0) // No timeout -- manual control + + w.Send(w.PID(), messageRunMigrations{}) + return nil +} + +func (w *Migrator) HandleMessage(from gen.PID, message any) error { + switch message.(type) { + case messageRunMigrations: + if err := w.runMigrations(); err != nil { + health.SignalDown(w, gen.Atom("health"), "migrations") + return err + } + health.SignalUp(w, gen.Atom("health"), "migrations") + // Unregister since startup is complete + health.Unregister(w, gen.Atom("health"), "migrations") + } + return nil +} +``` + +While migrations run, the startup probe returns 503, preventing Kubernetes from running liveness and readiness checks. Once migrations complete, the signal is unregistered and the startup probe returns 200. + +### Temporary Degradation + +Use readiness-only signals for recoverable issues: + +```go +func (w *CacheWorker) HandleMessage(from gen.PID, message any) error { + switch msg := message.(type) { + case CacheConnectionLost: + health.SignalDown(w, gen.Atom("health"), "cache") + // Pod removed from service but not restarted + + case CacheConnectionRestored: + health.SignalUp(w, gen.Atom("health"), "cache") + // Pod added back to service + } + return nil +} +``` + +Register the signal for `ProbeReadiness` only. The pod stops receiving traffic during the outage but is not restarted, since the cache connection will likely recover on its own. + +## Observer Integration + +The health actor integrates with Observer via `HandleInspect()`. Inspecting the health actor shows the endpoint URL, signal count, check interval, and current status of each registered signal. + +## Radar Application + +If your node needs both health probes and Prometheus metrics, consider the [Radar](../applications/radar.md) application. It runs the health actor and metrics actor together on a single HTTP port and provides helper functions so your actors don't need to import either package directly. diff --git a/docs/extra-library/actors/metrics.md b/docs/extra-library/actors/metrics.md index aa1771534..ece1b1c33 100644 --- a/docs/extra-library/actors/metrics.md +++ b/docs/extra-library/actors/metrics.md @@ -1,29 +1,15 @@ # Metrics -The metrics actor provides observability for Ergo applications by collecting and exposing runtime statistics in Prometheus format. Instead of manually instrumenting your code with counters and gauges scattered throughout, the metrics actor centralizes telemetry into a single process that exposes an HTTP endpoint for Prometheus to scrape. +The metrics actor collects runtime statistics from an Ergo node and exposes them as a Prometheus HTTP endpoint. It runs as a regular process: spawn it, and it starts serving `/metrics` with node, network, process, and event telemetry. -This approach separates monitoring concerns from application logic. Your actors focus on business functionality while the metrics actor handles collection, aggregation, and exposure of operational data. Prometheus or compatible monitoring systems poll the `/metrics` endpoint periodically, building time-series data for alerting and visualization. +For application-specific metrics (request rates, business counters), you extend the actor with custom Prometheus collectors. ## Why Monitor Actors -Actor systems present unique monitoring challenges. Traditional thread-based applications have predictable resource usage patterns - you monitor thread pools, request queues, and database connections. Actor systems are more dynamic - processes spawn and terminate constantly, messages flow asynchronously through mailboxes, and work distribution depends on supervision trees and message routing. - -The metrics actor addresses this by tracking: - -**Process metrics** - How many processes exist, how many are running vs. idle vs. zombie. This reveals whether your node is under load or experiencing process leaks. - -**Memory metrics** - Heap allocation and actual memory used. Actor systems can accumulate small allocations across thousands of processes. Memory metrics help identify whether garbage collection keeps pace with allocation. - -**Network metrics** - For distributed Ergo clusters, tracking bytes and messages flowing between nodes reveals network bottlenecks, routing inefficiencies, or failing connections. - -**Application metrics** - How many applications are loaded and running. Applications failing to start or terminating unexpectedly appear in these counts. - -These base metrics provide system-level visibility. For application-specific metrics (request rates, business transactions, custom counters), you extend the metrics actor with your own Prometheus collectors. +Actor systems are dynamic. Processes spawn and terminate constantly, messages flow through mailboxes asynchronously, and load depends on message routing and supervision trees. Traditional monitoring (thread pools, request queues) does not capture this. The metrics actor tracks process lifecycle, mailbox pressure, message throughput, event fanout, network traffic, and delivery errors, giving visibility into what the actor runtime is actually doing. ## ActorBehavior Interface -The metrics actor extends `gen.ProcessBehavior` with a specialized interface: - ```go type ActorBehavior interface { gen.ProcessBehavior @@ -40,32 +26,16 @@ type ActorBehavior interface { } ``` -Only `Init()` is required - register your custom metrics and return options; all other callbacks have default implementations you can override as needed. - -You have two main patterns: - -**Periodic collection** - Implement `CollectMetrics()` to query state at intervals. Use when metrics reflect current state from other actors or external sources. - -**Event-driven updates** - Implement `HandleMessage()` or `HandleEvent()` to update metrics when events occur. Use when your application produces natural event streams or publishes events. - -## How It Works +Only `Init()` is required. All other callbacks have default implementations. -When you spawn the metrics actor: +Two patterns for custom metrics: -1. **HTTP endpoint starts** at the configured host and port. The `/metrics` endpoint immediately serves Prometheus-formatted data. +**Periodic collection:** implement `CollectMetrics()` to query state at intervals. Use when metrics reflect current state from other actors or external sources. -2. **Base metrics collect automatically**. Node information (processes, memory, CPU) and network statistics (connected nodes, message rates) update at the configured interval. - -3. **Custom metrics update** via `CollectMetrics()` callback or `HandleMessage()` processing, depending on your implementation. - -4. **Prometheus scrapes** the `/metrics` endpoint and receives current values for all registered collectors (base + custom). - -The actor handles HTTP serving and registry management. You focus on defining metrics and updating their values. +**Event-driven updates:** implement `HandleMessage()` or `HandleEvent()` to update metrics as events occur. Use when your application produces natural event streams. ## Basic Usage -Spawn the metrics actor like any other process: - ```go package main @@ -79,7 +49,6 @@ func main() { node, _ := ergo.StartNode("mynode@localhost", gen.NodeOptions{}) defer node.Stop() - // Spawn metrics actor with defaults node.Spawn(metrics.Factory, gen.ProcessOptions{}, metrics.Options{}) // Metrics available at http://localhost:3000/metrics @@ -90,79 +59,125 @@ func main() { Default configuration: - **Host**: `localhost` - **Port**: `3000` +- **Path**: `/metrics` - **CollectInterval**: `10 seconds` - -The HTTP endpoint starts automatically during initialization. The first metrics collection happens immediately, and subsequent collections run at the configured interval. +- **TopN**: `50` ## Configuration -Customize the HTTP endpoint and collection frequency: - ```go options := metrics.Options{ Host: "0.0.0.0", // Listen on all interfaces - Port: 9090, // Prometheus default port - CollectInterval: 5 * time.Second, // Collect every 5 seconds + Port: 9090, // HTTP port + Path: "/metrics", // HTTP path + CollectInterval: 5 * time.Second, // Collection frequency + TopN: 50, // Top-N entries per metric group } node.Spawn(metrics.Factory, gen.ProcessOptions{}, options) ``` -**Host** determines which network interface the HTTP server binds to. Use `"localhost"` to restrict access to local connections only (development, testing). Use `"0.0.0.0"` to accept connections from any interface (production, containerized environments). +**Host** determines which interface the HTTP server binds to. Use `"localhost"` for development, `"0.0.0.0"` for production/containers. + +**Port** should not conflict with other services. Prometheus conventionally uses `9090`, Observer UI defaults to `9911`. + +**TopN** controls how many top entries are tracked for each metric group (mailbox depth, utilization, latency for processes; subscribers, published, deliveries for events). Higher values increase Prometheus cardinality. -**Port** should not conflict with other services. Prometheus conventionally uses `9090`, but many Ergo applications use that for other purposes. Choose a port that doesn't collide with your application's HTTP servers, Observer UI (default `9911`), or other metrics exporters. +**CollectInterval** controls how frequently the actor queries node statistics. Collecting more frequently than your Prometheus scrape interval wastes resources. -**CollectInterval** controls how frequently the actor queries node statistics. Shorter intervals provide more granular time-series data but increase CPU usage for collection. Longer intervals reduce overhead but miss short-lived spikes. For most applications, 10-15 seconds balances responsiveness with resource usage. Prometheus typically scrapes every 15-60 seconds, so collecting more frequently than your scrape interval wastes resources. +**Mux** accepts an external `*http.ServeMux`. The metrics actor registers its handler on this mux and skips starting its own HTTP server. Useful for serving metrics alongside other handlers on a single port: + +```go +mux := http.NewServeMux() + +metricsOpts := metrics.Options{ + Mux: mux, + CollectInterval: 5 * time.Second, +} +node.Spawn(metrics.Factory, gen.ProcessOptions{}, metricsOpts) + +healthOpts := health.Options{Mux: mux} +node.SpawnRegister("health", health.Factory, gen.ProcessOptions{}, healthOpts) +``` + +When `Mux` is set, `Host` and `Port` are ignored. ## Base Metrics -The metrics actor automatically exposes these Prometheus metrics without any configuration: +The actor automatically collects metrics without any configuration. All metrics carry a `node` label identifying the source node. ### Node Metrics -| Metric | Type | Description | -|--------|------|-------------| -| `ergo_node_uptime_seconds` | Gauge | Time since node started. Useful for detecting node restarts and calculating availability. | -| `ergo_processes_total` | Gauge | Total number of processes including running, idle, and zombie. High counts suggest process leaks or inefficient cleanup. | -| `ergo_processes_running` | Gauge | Processes actively handling messages. Low relative to total suggests most processes are idle (good) or blocked (bad - investigate what they're waiting for). | -| `ergo_processes_zombie` | Gauge | Processes terminated but not yet fully cleaned up. These should be transient. Persistent zombies indicate bugs in termination handling. | -| `ergo_memory_used_bytes` | Gauge | Total memory obtained from OS (uses `runtime.MemStats.Sys`). | -| `ergo_memory_alloc_bytes` | Gauge | Bytes of allocated heap objects (uses `runtime.MemStats.Alloc`). | -| `ergo_cpu_user_seconds` | Gauge | CPU time spent executing user code. Increases as the node does work. Rate of change indicates CPU utilization. | -| `ergo_cpu_system_seconds` | Gauge | CPU time spent in kernel (system calls). High system time relative to user time suggests I/O bottlenecks or excessive syscalls. | -| `ergo_applications_total` | Gauge | Number of applications loaded. Should match your expected count. Unexpected changes indicate applications starting or stopping. | -| `ergo_applications_running` | Gauge | Applications currently active. Compare to total to identify stopped or failed applications. | -| `ergo_registered_names_total` | Gauge | Processes registered with atom names. High counts suggest heavy use of named processes for routing. | -| `ergo_registered_aliases_total` | Gauge | Total number of registered aliases. Includes aliases created by processes via `CreateAlias()` and aliases identifying meta-processes. | -| `ergo_registered_events_total` | Gauge | Event subscriptions active in the node. High counts indicate extensive pub/sub usage. | +Uptime, process counts (total, running, zombie), spawn/termination counters, memory (OS used, runtime allocated), CPU time (user, system), application counts, registered names/aliases/events, event publish/receive/delivery counters, and Send/Call delivery error counters (local and remote). + +Delivery errors are split by type: `ergo_send_errors_local_total` and `ergo_call_errors_local_total` count failures where the target process is unknown, terminated, or has a full mailbox. `ergo_send_errors_remote_total` and `ergo_call_errors_remote_total` count connection failures to remote nodes. + +### Log Metrics + +Log message count by level (`trace`, `debug`, `info`, `warning`, `error`, `panic`). Counted once before fan-out to loggers. ### Network Metrics -| Metric | Type | Labels | Description | -|--------|------|--------|-------------| -| `ergo_connected_nodes_total` | Gauge | - | Number of remote nodes connected. For distributed systems, this should match your expected cluster size. | -| `ergo_remote_node_uptime_seconds` | Gauge | `node` | Uptime of each connected remote node. Resets when the remote node restarts. | -| `ergo_remote_messages_in_total` | Gauge | `node` | Messages received from each remote node. Rate indicates traffic volume. | -| `ergo_remote_messages_out_total` | Gauge | `node` | Messages sent to each remote node. Asymmetric in/out rates may reveal routing issues. | -| `ergo_remote_bytes_in_total` | Gauge | `node` | Bytes received from each remote node. Disproportionate bytes-to-messages ratio suggests large messages or inefficient serialization. | -| `ergo_remote_bytes_out_total` | Gauge | `node` | Bytes sent to each remote node. Monitors network bandwidth usage per peer. | +Connected node count, per-node uptime, message and byte rates (in/out per remote node), cumulative connections established/lost, and per-acceptor handshake error count. Fragmentation metrics per remote node: fragments sent/received, fragmented messages sent/reassembled, assembly timeouts. Compression metrics per remote node: compressed messages sent, bytes before/after compression, decompressed messages received, bytes before/after decompression. Compression ratio (`original / compressed`) reveals whether compression is effective for each connection. + +### Mailbox Latency Metrics + +Requires building with `-tags=latency`. Measures how long the oldest message has been waiting in each process's mailbox. Provides distribution across ranges (1ms to 60s+), max latency, and top-N processes by latency. -Network metrics use labels (`node="..."`) to separate per-node data. This creates multiple time series - one per connected node. Prometheus queries can aggregate across labels or filter to specific nodes. +### Mailbox Depth Metrics + +Always active. Counts messages queued in each process's mailbox. Distribution across ranges (1 to 10K+), max depth, and top-N processes by depth. Complementary to latency: depth is "how many messages are waiting", latency is "how long the oldest has been waiting". + +### Process Metrics + +Always active. Includes: + +- **Utilization:** ratio of callback running time to uptime. Distribution, max, and top-N. +- **Init time:** ProcessInit duration. Max and top-N. +- **Throughput:** messages in/out per process (top-N) and node-level aggregates. +- **Wakeups and drains:** wakeup count and drain ratio (messages processed per wakeup). Drain ratio distinguishes between slow callbacks (drain ~1) and high-throughput batching (drain ~100) at the same utilization level. +- **Liveness:** detects processes stuck in blocking calls. Computed as `RunningTime / (Uptime * MailboxLatency)`. A healthy process has RunningTime growing with activity (high score). A process blocked in a mutex, channel, or IO has RunningTime frozen while uptime and latency keep growing (score drops over time). Zombie processes are excluded (detected separately). Bottom-N surfaces the most stuck processes. Requires `-tags=latency`. + +### Event Metrics + +Always active. Per-event subscriber count, publish/delivery counts, and utilization state (`active`, `on_demand`, `idle`, `no_subscribers`, `no_publishing`). See [Events](../../basics/events.md) for the pub/sub model and [Pub/Sub Internals](../../advanced/pub-sub-internals.md) for the shared subscription optimization that affects delivery counters. + +For the complete list of metric names, types, labels, and descriptions, see the [metrics actor README](https://github.com/ergo-services/actor). ## Custom Metrics -Extend the metrics actor by embedding `metrics.Actor`. You register custom Prometheus collectors in `Init()` and update them via `CollectMetrics()` or `HandleMessage()`. +All custom metrics automatically receive a `node` const label. Do not include `"node"` in your variable label names. -### Approach 1: Periodic Collection +### Helper Functions -Implement `CollectMetrics()` to poll state at regular intervals: +Any actor on the same node can register and update custom metrics without importing `prometheus` or embedding the metrics actor: + +```go +// Register metrics (sync Call, returns error) +metrics.RegisterGauge(w, "metrics_actor", "db_connections", "Active connections", []string{"pool"}) +metrics.RegisterCounter(w, "metrics_actor", "cache_ops", "Cache operations", []string{"op"}) +metrics.RegisterHistogram(w, "metrics_actor", "request_seconds", "Latency", []string{"path"}, nil) + +// Update metrics (async Send) +metrics.GaugeSet(w, "metrics_actor", "db_connections", 42, []string{"primary"}) +metrics.CounterAdd(w, "metrics_actor", "cache_ops", 1, []string{"hit"}) +metrics.HistogramObserve(w, "metrics_actor", "request_seconds", 0.023, []string{"/api"}) + +// Remove a metric (async Send) +metrics.Unregister(w, "metrics_actor", "db_connections") +``` + +When the registering process terminates, the metrics actor automatically unregisters all metrics it owned. + +### Embedding metrics.Actor + +For direct access to the Prometheus registry or periodic collection via `CollectMetrics`: ```go type AppMetrics struct { metrics.Actor - activeUsers prometheus.Gauge - queueDepth prometheus.Gauge + activeUsers prometheus.Gauge } func (m *AppMetrics) Init(args ...any) (metrics.Options, error) { @@ -171,12 +186,7 @@ func (m *AppMetrics) Init(args ...any) (metrics.Options, error) { Help: "Current number of active users", }) - m.queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{ - Name: "myapp_queue_depth", - Help: "Current queue depth", - }) - - m.Registry().MustRegister(m.activeUsers, m.queueDepth) + m.Registry().MustRegister(m.activeUsers) return metrics.Options{ Port: 9090, @@ -185,104 +195,79 @@ func (m *AppMetrics) Init(args ...any) (metrics.Options, error) { } func (m *AppMetrics) CollectMetrics() error { - // Called every CollectInterval - // Query other processes for current state - count, err := m.Call(userService, getActiveUsersMessage{}) if err != nil { m.Log().Warning("failed to get user count: %s", err) - return nil // Non-fatal, continue + return nil } m.activeUsers.Set(float64(count.(int))) - - depth, _ := m.Call(queueService, getDepthMessage{}) - m.queueDepth.Set(float64(depth.(int))) - return nil } ``` -Use this when metrics reflect state you need to query - current values from other actors, computed aggregates, external API calls. - -### Approach 2: Event-Driven Updates - -Update metrics immediately when events occur: +For event-driven updates, implement `HandleMessage()` instead of `CollectMetrics()`: ```go -type AppMetrics struct { - metrics.Actor - - requestsTotal prometheus.Counter - requestLatency prometheus.Histogram -} - -func (m *AppMetrics) Init(args ...any) (metrics.Options, error) { - m.requestsTotal = prometheus.NewCounter(prometheus.CounterOpts{ - Name: "myapp_requests_total", - Help: "Total requests processed", - }) - - m.requestLatency = prometheus.NewHistogram(prometheus.HistogramOpts{ - Name: "myapp_request_duration_seconds", - Help: "Request latency distribution", - Buckets: prometheus.DefBuckets, - }) - - m.Registry().MustRegister(m.requestsTotal, m.requestLatency) - - return metrics.Options{Port: 9090}, nil -} - func (m *AppMetrics) HandleMessage(from gen.PID, message any) error { switch msg := message.(type) { case requestCompletedMessage: m.requestsTotal.Inc() m.requestLatency.Observe(msg.duration.Seconds()) - case errorOccurredMessage: - m.errorsTotal.Inc() } return nil } ``` -Application actors send events to the metrics actor: +### Custom Top-N Metrics + +Top-N metrics track the N highest (or lowest) values observed during each collection cycle. Unlike gauges or counters, a top-N metric accumulates observations and periodically flushes only the top entries to Prometheus as a GaugeVec. This is useful when you want to identify the most active, slowest, or largest items out of many, without creating a separate time series for each one. + +Each top-N metric is managed by a dedicated actor spawned under a SimpleOneForOne supervisor. Registration creates this actor; observations are sent to it asynchronously. On each flush interval the actor writes the current top-N entries to Prometheus and resets for the next cycle. ```go -// In your request handler actor -func (h *RequestHandler) HandleMessage(from gen.PID, message any) error { - switch msg := message.(type) { - case ProcessRequest: - start := time.Now() - // ... process request ... - elapsed := time.Since(start) - - // Send metrics event - h.Send(metricsPID, requestCompletedMessage{duration: elapsed}) - } - return nil -} +// Register a top-N metric (sync Call, returns error) +// TopNMax keeps the N largest values; TopNMin keeps the N smallest +metrics.RegisterTopN(w, "topn_supervisor_name", "slowest_queries", "Slowest DB queries", + 10, metrics.TopNMax, []string{"query", "table"}) + +// Observe values (async Send) +metrics.TopNObserve(w, gen.Atom("radar_topn_slowest_queries"), 0.250, []string{"SELECT ...", "users"}) +metrics.TopNObserve(w, gen.Atom("radar_topn_slowest_queries"), 1.100, []string{"JOIN ...", "orders"}) ``` -Use this when your application naturally produces events. Metrics update in real-time without polling. +The `to` parameter in `RegisterTopN` is the name of the supervisor managing top-N actors. The `to` parameter in `TopNObserve` is the actor name, by convention `"radar_topn_" + metricName`. -## Metric Types +Ordering modes: -Prometheus defines four metric types, each suited for different use cases: +- `metrics.TopNMax`: keeps the N largest values (e.g., slowest queries, busiest actors, highest memory usage) +- `metrics.TopNMin`: keeps the N smallest values (e.g., lowest latency, least active processes) -**Counter** - Monotonically increasing value. Use for events that accumulate (requests processed, errors occurred, bytes sent). Counters never decrease except on process restart. Prometheus queries typically use `rate()` to calculate per-second rates or `increase()` for total change over a time window. +When the process that registered a top-N metric terminates, the actor automatically cleans up and unregisters its GaugeVec from Prometheus. -**Gauge** - Value that can go up or down. Use for current state (active connections, queue depth, memory usage, CPU utilization). Gauges represent snapshots. Prometheus queries can graph them directly or use functions like `avg_over_time()` to smooth spikes. +When used through the [Radar](../applications/radar.md) application, the supervisor is already wired in and you use `radar.RegisterTopN` / `radar.TopNObserve` helpers instead. -**Histogram** - Observations bucketed into configurable ranges. Use for latency or size distributions. Histograms let you calculate percentiles (p50, p95, p99) in Prometheus queries. They're more resource-intensive than gauges because they maintain multiple buckets per metric. +### Shared Mode -**Summary** - Similar to histogram but calculates quantiles client-side. Use when you need precise quantiles but can't predict bucket boundaries. Summaries are more expensive than histograms because they track exact quantiles, not approximations. +A single metrics actor processes messages sequentially. Under high throughput, its mailbox becomes a bottleneck. Shared mode lets multiple metrics actor instances share the same Prometheus registry: -For most use cases, counters and gauges suffice. Use histograms when you need latency percentiles. Avoid summaries unless you have specific reasons - histograms are more flexible for Prometheus queries. +```go +shared := metrics.NewShared() +// Primary actor: owns HTTP endpoint and base metrics +primaryOpts := metrics.Options{ + Port: 9090, + Shared: shared, +} -## Integration with Prometheus +// Worker actors: handle custom metric updates only +workerOpts := metrics.Options{ + Shared: shared, +} +``` + +The primary actor starts the HTTP server and collects base metrics. Workers only process custom metric messages. All actors write to the same registry through the shared object. Works well with `act.Pool` for automatic load distribution. -Configure Prometheus to scrape the metrics endpoint: +## Integration with Prometheus ```yaml scrape_configs: @@ -291,37 +276,27 @@ scrape_configs: - targets: - 'localhost:3000' - 'node1.example.com:3000' - - 'node2.example.com:3000' scrape_interval: 15s ``` -Prometheus fetches `/metrics` every 15 seconds, parses the text format, and stores time-series data. You can then query, alert, and visualize metrics using Prometheus queries or Grafana dashboards. +For dynamic discovery in Kubernetes, use Prometheus service discovery instead of static targets. -For dynamic discovery in Kubernetes or cloud environments, use Prometheus service discovery instead of static targets. The metrics actor itself doesn't need to know about Prometheus - it just exposes an HTTP endpoint. +## Grafana Dashboard -## Observer Integration +The metrics package includes a pre-built Grafana dashboard (`ergo-cluster.json`) for monitoring Ergo clusters. + +Import it in Grafana: Dashboards > Import > upload `ergo-cluster.json` > select your Prometheus data source. The `$node` dropdown at the top filters all panels by selected nodes. -The metrics actor includes built-in Observer support via `HandleInspect()`. When you inspect it in Observer UI (http://localhost:9911), you see: +The dashboard is organized top-down: Summary row at the top for cluster health at a glance, then Mailbox Latency and Depth for backpressure analysis, then collapsed rows for Events, Process Activity, Processes, Resources, Logging, and Network. The Network row includes compression overview (ratio, rate, percentage), per-node compression ratio, fragmentation rates (cluster and per-node), connectivity strength, and connection events. Each row focuses on a specific aspect of cluster behavior and can be expanded when investigating issues. -- Total number of registered metrics -- HTTP endpoint URL for Prometheus scraping -- Collection interval -- Current values for all metrics (base + custom) +For detailed panel descriptions, see the [metrics actor README](https://github.com/ergo-services/actor). -This works automatically for custom metrics - register them in `Init()` and they appear in Observer alongside base metrics. +## Observer Integration -If you need custom inspection behavior, override `HandleInspect()` in your implementation: +The metrics actor integrates with Observer via `HandleInspect()`. Inspecting the process shows total metric count, HTTP endpoint, collection interval, and current values for all metrics. -```go -func (m *AppMetrics) HandleInspect(from gen.PID, item ...string) map[string]string { - result := make(map[string]string) - - // Custom inspection logic - result["status"] = "healthy" - result["custom_info"] = "some value" - - return result -} -``` +When embedding `metrics.Actor` and overriding `HandleInspect()`, your keys are merged on top of base inspection data. + +## Radar Application -For detailed configuration options, see the `metrics.Options` struct and `ActorBehavior` interface in the package. For examples of custom metrics, see the [example directory](https://github.com/ergo-services/actor/tree/main/metrics/example). +If your node needs both Prometheus metrics and Kubernetes health probes, consider the [Radar](../applications/radar.md) application. It runs the metrics actor and [Health](health.md) actor together on a single HTTP port. diff --git a/docs/extra-library/applications/mcp.md b/docs/extra-library/applications/mcp.md new file mode 100644 index 000000000..7c0e35349 --- /dev/null +++ b/docs/extra-library/applications/mcp.md @@ -0,0 +1,285 @@ +--- +description: AI-powered diagnostics for running Ergo nodes via Model Context Protocol +--- + +# MCP + +Diagnosing a distributed actor system is hard. The problem isn't a lack of data - it's knowing what to look for. A node has hundreds of processes, dozens of connections, thousands of events flowing between them. Something is slow, but where? A process is stuck, but why? Memory is growing, but what's holding it? + +Traditional monitoring collects predefined metrics at fixed intervals. You decide upfront what matters, build dashboards, and then interpret the data when something breaks. This works for known failure modes. It doesn't work when the failure is something you haven't anticipated - and in distributed systems, the interesting failures are always unanticipated. + +MCP takes a different approach. Instead of predefined metrics, it exposes the full diagnostic surface of the node - processes, applications, events, network, profiling, runtime - as tools that an AI agent can call on demand. The agent decides what to inspect based on the symptom you describe. It runs diagnostic sequences, correlates findings across tools, narrows down root causes, and explains what it found. You describe the problem in words; the agent finds the answer in data. + +The real power comes from combination. The agent can see your source code, inspect the live cluster via MCP, and query your log storage - all in the same conversation. It reads the actor implementation to understand intent, checks runtime state to see what actually happens, and correlates with error logs to see the history. Together these eliminate guesswork in a way no single tool can. + +The application runs as a regular Ergo sidecar. Add it to your node's application list, and every process, connection, and event becomes inspectable - without restarting, redeploying, or attaching a debugger. + +## Two Deployment Modes + +MCP has two modes: entry point and agent. + +An entry point node runs an HTTP listener that accepts MCP protocol requests. This is the node your AI client connects to. An agent node has no HTTP listener at all - it's invisible from outside the cluster. But it runs the same diagnostic tools internally, and any entry point can reach it through cluster proxy. + +In practice, you deploy one entry point and make everything else an agent: + +```go +import ( + "ergo.services/ergo" + "ergo.services/application/mcp" + "ergo.services/ergo/gen" +) + +func main() { + node, _ := ergo.StartNode("example@localhost", gen.NodeOptions{ + Applications: []gen.ApplicationBehavior{ + // Entry point - the one HTTP endpoint for the entire cluster + mcp.CreateApp(mcp.Options{Port: 9922}), + }, + }) + node.Wait() +} +``` + +On every other node, the same application with no port: + +```go +// Agent mode - no HTTP, but fully diagnosable via cluster proxy +mcp.CreateApp(mcp.Options{}) +``` + +The AI client connects to `http://entry-point:9922/mcp` and reaches any node in the cluster through that single endpoint. + +## Configuration + +```go +mcp.Options{ + Host: "localhost", // Listen address + Port: 9922, // HTTP port (0 = agent mode) + Token: "secret", // Bearer token (empty = no auth) + ReadOnly: false, // Disable action tools + AllowedTools: nil, // Tool whitelist (nil = all) + PoolSize: 5, // Worker processes + CertManager: nil, // TLS certificate manager + LogLevel: gen.LogLevelInfo, +} +``` + +**Port** controls the deployment mode. A non-zero value starts an HTTP listener - this is an entry point. Zero means agent mode: no listener, accessible only via cluster proxy from another node that has an entry point. + +**Token** enables Bearer token authentication. When set, every HTTP request must include `Authorization: Bearer `. When empty, no authentication is required. Agent mode nodes don't need a token - they're accessed through the Ergo inter-node protocol, which has its own authentication via handshake cookies. + +**ReadOnly** disables tools that modify state: `send_message`, `call_process`, `send_exit`, `process_kill`. Everything else - inspection, profiling, sampling - remains available. Use this on production nodes where you want full visibility without the ability to interfere. + +**AllowedTools** restricts the tool set to a whitelist. When set, only the named tools are available. This is finer-grained than ReadOnly - you can, for example, allow `send_message` but not `process_kill`. When nil, all tools are enabled (respecting ReadOnly). + +## Connecting a Client + +### Claude Code + +```bash +claude mcp add --transport http ergo http://localhost:9922/mcp + +# Available from any directory (user scope) +claude mcp add --transport http ergo --scope user http://localhost:9922/mcp + +# With authentication +claude mcp add --transport http ergo http://localhost:9922/mcp \ + -H "Authorization: Bearer my-secret-token" +``` + +To allow all MCP tools without per-call permission prompts, add to `.claude/settings.json`: + +```json +{ + "permissions": { + "allow": ["mcp__ergo"] + } +} +``` + +### Other Clients + +The application implements MCP protocol version `2025-06-18` over HTTP. Any MCP-compatible client can connect by sending JSON-RPC 2.0 POST requests to `http://:/mcp`. + +## How Cluster Proxy Works + +Every tool accepts a `node` parameter. When specified, the entry point node forwards the request to the target node via native Ergo inter-node protocol - not HTTP. The target node's MCP worker executes the tool locally and returns the result through the same path. + +This works because of network transparency. The entry point calls `gen.ProcessID{Name: "mcp", Node: targetNode}` - the framework establishes a connection if needed, routes the request, and delivers the response. You never need to explicitly connect to a node before querying it. If the registrar knows about the target node, the connection happens automatically. + +The `timeout` parameter (default 30 seconds, max 120) controls how long the entry point waits for a remote response. Most tools respond in milliseconds. But CPU profiling collects data for a requested duration before responding, and goroutine dumps on large nodes take time to serialize. For these, pass a higher timeout. + +If a remote tool call fails with "remote call failed", it usually means the target node doesn't have the MCP application running. All proxy calls require an MCP pool process on the target node - agent mode is sufficient, but the application must be loaded and started. + +## Profiling Remote Nodes + +Profiling tools generate large output. A goroutine dump from a node with 500 goroutines can be megabytes of text. A heap profile with hundreds of allocation sites isn't much smaller. Push all of that through the proxy chain - remote node, entry point, HTTP, JSON-RPC - and you hit timeouts or transport limits. + +The solution is server-side filtering. All profiling tools accept `filter` and `exclude` parameters that reduce the output before it leaves the remote node. Instead of transferring 500 goroutine stacks and searching locally, you tell the remote node to return only the stacks that match: + +``` +pprof_goroutines node=backend@host debug=1 filter="orderHandler" limit=20 +``` + +The response header preserves the full picture: `goroutine profile: total 500, matched 3, showing 3`. You know the node has 500 goroutines, but only 3 matched your filter, and all 3 were returned. The agent can refine the filter, broaden it, or switch to a different angle - each query is cheap because the heavy lifting happens on the remote node. + +### CPU Profiling + +The `pprof_cpu` tool collects a CPU profile for a given duration and returns the top functions by CPU usage: + +``` +pprof_cpu node=backend@host duration=5 exclude="runtime" limit=15 timeout=30 +``` + +The node samples CPU activity for 5 seconds, aggregates by function, filters out Go runtime internals, and returns the top 15 application functions with flat and cumulative percentages. The `timeout` should be higher than `duration` to account for collection and transfer time. + +### Heap Profiling + +The `pprof_heap` tool shows the top memory allocators with two columns: `inuse` (live objects currently in memory) and `alloc` (cumulative allocations over the node's lifetime). A function with low `inuse` but high `alloc` is churning memory - allocating and releasing rapidly, putting pressure on the garbage collector. + +``` +pprof_heap node=backend@host filter="myapp" limit=20 +``` + +### Goroutine Analysis + +The `pprof_goroutines` tool has two modes. Without `pid`, it returns all goroutines on the node - use `filter` and `exclude` to narrow down. With `pid`, it returns the stack trace of a specific process's goroutine (requires `-tags=pprof`). + +Debug level controls the output format: `debug=1` groups goroutines by identical stack (compact summary with counts), `debug=2` shows individual goroutine traces with state and wait duration. + +A sleeping process parks its goroutine - it won't appear in the dump. To catch it, use an active sampler that polls until the process wakes up: + +``` +sample_start tool=pprof_goroutines arguments={"pid":""} interval_ms=300 count=1 max_errors=0 +``` + +The sampler ignores the "goroutine not found" error (`max_errors=0`) and keeps polling every 300ms until it catches the process in a non-sleep state. + +## Samplers + +Snapshots show one moment. Trends show the story. Samplers bridge this gap by collecting data into ring buffers that the agent reads incrementally. + +### Active Samplers + +An active sampler periodically calls any MCP tool and stores the results. It's a generic periodic executor - any tool with any arguments can be sampled. + +``` +sample_start tool=process_list arguments={"sort_by":"mailbox","limit":10} interval_ms=5000 duration_sec=300 +``` + +This calls `process_list` every 5 seconds for 5 minutes, storing each result in a ring buffer. The agent reads with `sample_read sampler_id=` to get all buffered entries, or `sample_read sampler_id= since=5` to get only entries newer than sequence 5. + +The `max_errors` parameter controls error tolerance. The default (0) means ignore all errors and keep retrying - useful for polling rare conditions. A non-zero value stops the sampler after that many consecutive failures. + +### Passive Samplers + +A passive sampler listens for events instead of polling. It captures log messages and event publications as they happen: + +``` +sample_listen log_levels=["warning","error"] duration_sec=120 +sample_listen event=order_events duration_sec=60 +sample_listen log_levels=["error"] event=order_events duration_sec=120 +``` + +Log capture and event subscription can be combined in a single sampler. + +### Linger + +Every sampler has a `linger_sec` parameter (default 30). After the sampler completes - duration expires, count reached, or max errors exceeded - it stays alive for this many additional seconds so the agent can retrieve the collected data. Without linger, a sampler that runs for 10 seconds would terminate before the agent gets a chance to read the results. + +The `sample_list` tool shows sampler status: `running`, `completed, lingering 25s`, or `completed`. The `sample_stop` tool terminates a sampler immediately, bypassing the linger period. + +### What to Sample + +| Goal | Sampler | +|------|---------| +| Mailbox pressure trend | `sample_start tool=process_list arguments={"sort_by":"mailbox","limit":10}` | +| Memory and GC trend | `sample_start tool=runtime_stats interval_ms=5000` | +| Error storm detection | `sample_listen log_levels=["error","panic"]` | +| Event traffic monitoring | `sample_listen event=` | +| Network health trend | `sample_start tool=network_nodes interval_ms=30000` | +| CPU hotspot sampling | `sample_start tool=pprof_goroutines arguments={"debug":1,"filter":"ProcessRun","exclude":"toolPprof","limit":20} interval_ms=500` | + +## Typed Messages + +When `ReadOnly` is not set, the agent can send messages to processes and make synchronous calls using the EDF type registry. This isn't raw JSON injection - the framework constructs real Go structs from the type information. + +If your application registers a type: + +```go +type StatusRequest struct { + Verbose bool +} + +func (a *MyApp) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { + if err := node.Network().RegisterType(StatusRequest{}); err != nil { + return gen.ApplicationSpec{}, err + } + return gen.ApplicationSpec{ /* ... */ }, nil +} +``` + +The agent discovers it with `message_types`, inspects its fields with `message_type_info`, and sends it with `call_process`. The process receives a real `StatusRequest{Verbose: true}` in its `HandleCall`, not a map or raw bytes. + +This makes interactive debugging possible: the agent can call any process with any registered request type, inspect the response, and reason about the behavior. + +## Network Diagnostics + +The `network_ping` tool sends a request through the full network path - flusher, TCP connection, remote MCP worker, response - and measures the round-trip time. This is an end-to-end health check, not a TCP-level ping. If the flusher is broken, the connection pool is degraded, or the remote node is overloaded, the ping will reflect it. + +``` +network_ping name=backend@host +→ ping backend@host: rtt 0.42ms +``` + +For deeper connection analysis, `network_node_info` shows per-connection statistics: messages in/out, bytes in/out, pool size, pool DSN (which side dialed), and a `Reconnections` counter that tracks how many times pool items have reconnected. A non-zero reconnection count indicates connection instability. + +When investigating connection problems, always check both sides: + +``` +network_node_info node=A name=B # A's view of the connection to B +network_node_info node=B name=A # B's view of the connection to A +``` + +Asymmetry between the two sides - one sees thousands of messages out while the other sees one message in - indicates data loss at the connection level. + +## Build Tags + +Two build tags enable additional diagnostic capabilities. Both add a small amount of overhead and should be enabled in staging and production builds where diagnostics matter. + +**`-tags=pprof`** enables the Go profiler and labels actor goroutines with their process PID. The labels appear in goroutine dumps as `{"pid":""}` for actors and `{"meta":"Alias#...", "role":"reader"}` for meta processes. The `pprof_goroutines` tool with `pid` parameter uses these labels to extract a specific actor's stack trace. Without this tag, the `pid` parameter returns an error. + +This tag also starts a pprof HTTP endpoint at `localhost:9009/debug/pprof/` (configurable via `PPROF_HOST` and `PPROF_PORT` environment variables) for use with `go tool pprof`. + +**`-tags=latency`** enables mailbox latency measurement. Each mailbox queue tracks the age of its oldest unprocessed message. The `process_list` tool gains `min_mailbox_latency_ms` filter and `mailbox_latency` sort field. Without this tag, latency fields return -1. + +## Relationship to Metrics Actor + +The [Metrics](../actors/metrics.md) actor collects predefined metrics into Prometheus format for scraping. MCP reads from the same underlying data sources - `ProcessRangeShortInfo`, `NodeInfo`, `EventRangeInfo` - but exposes them interactively. + +Active samplers can replicate any Prometheus metric: `sample_start tool=process_list arguments={"sort_by":"mailbox","limit":10}` is equivalent to `ergo_mailbox_depth_top`. The difference is that MCP samplers are on-demand and agent-driven, while Prometheus metrics are always-on and scraper-driven. + +Use the metrics actor for long-term trends, alerting, and Grafana dashboards. Use MCP for interactive investigation when alerts fire or when you need to explore something unexpected. + +## Agent and Skill for Claude Code + +A ready-to-use diagnostic agent and skill are available at [github.com/ergo-services/claude](https://github.com/ergo-services/claude). The agent contains playbooks for common scenarios: performance bottlenecks, process leaks, restart loops, zombie processes, memory growth, network issues, event system problems, goroutine investigation, and cluster health checks. Trigger it by describing a symptom - "why is it slow", "check the cluster", "find the process leak" - and it runs the appropriate diagnostic sequence. + +Install as a Claude Code plugin: + +```bash +/plugin marketplace add ergo-services/claude +/plugin install ergo@ergo-services +``` + +Or symlink into `~/.claude/` for local development: + +```bash +cd ergo.services/claude +ln -sf $(pwd)/agents/devops.md ~/.claude/agents/ +ln -sf $(pwd)/skills/devops ~/.claude/skills/ +``` + +## Full Tool Reference + +The complete list of 48 tools with parameters and descriptions is in the [MCP application README](https://github.com/ergo-services/application/blob/master/mcp/README.md). diff --git a/docs/extra-library/applications/observer.md b/docs/extra-library/applications/observer.md index 797e29b77..04a7e8074 100644 --- a/docs/extra-library/applications/observer.md +++ b/docs/extra-library/applications/observer.md @@ -1,6 +1,14 @@ +--- +description: Real-time web UI for monitoring and inspecting Ergo nodes +--- + # Observer -The Application _Observer_ provides a convenient web interface to view node status, network activity, and running processes in the node built with Ergo Framework. Additionally, it allows you to inspect the internal state of processes or meta-processes. The application is can also be used as a standalone tool Observer. For more details, see the section [Inspecting With Observer](../../tools/observer.md). You can add the _Observer_ application to your node during startup by including it in the node's startup options: +Observer is a web application that embeds into your node and provides real-time visibility into the running system. It uses Server-Sent Events (SSE) for live push updates to the browser. + +For a detailed description of the UI and all available views, see [Inspecting With Observer](../../advanced/observer.md). + +## Adding Observer to a node 
import (
 	"ergo.services/ergo"
@@ -9,21 +17,95 @@ The Application _Observer_ provides a convenient web interface to view node stat
 )
 
 func main() {
-	opt := gen.NodeOptions{
-		Applications: []gen.ApplicationBehavior {
+	options := gen.NodeOptions{
+		Applications: []gen.ApplicationBehavior{
 			observer.CreateApp(observer.Options{}),
-		}
+		},
 	}
-	node, err := ergo.StartNode("example@localhost", opt)
-	if err != nil {
+	node, err := ergo.StartNode("mynode@localhost", options)
+	if err != nil {
 		panic(err)
 	}
 	node.Wait()
 }
-The function `observer.CreateApp` takes `observer.Options` as an argument, allowing you to configure the _Observer_ application. You can set: +Open `http://localhost:9911` in your browser. + +## Options + +`observer.CreateApp` accepts `observer.Options`: + +* **Host**: interface to listen on. Default: `localhost`. +* **Port**: HTTP port. Default: `9911`. +* **PoolSize**: number of worker processes handling requests. Default: `10`. +* **LogLevel**: log level for Observer's own processes. Default: `gen.LogLevelInfo`. + +## What Observer shows + +### Node + +General node information: name, version, OS, architecture, CPU cores, timezone, uptime, memory usage, process count, goroutine count. Memory graph updates live over the last 60 seconds. Node-level log level can be changed directly from this view. + +### Processes + +Full process list with per-process metrics: state, mailbox depth, message latency, running time, wakeup count, uptime. Supports filtering by name pattern, behavior type, application, state, and minimum mailbox depth. + +Clicking a process opens its detail view: supervision tree position, links, monitors, registered names, aliases, environment variables, and the internal state returned by `HandleInspect`. + +Any actor that implements `HandleInspect` exposes its state as a live-updating key-value panel in the browser: + +```go +func (a *MyActor) HandleInspect(from gen.PID, item ...string) map[string]string { + return map[string]string{ + "connections": fmt.Sprintf("%d", a.connCount), + "last_error": a.lastError, + } +} +``` + +### Meta-processes + +Meta-processes (TCP servers, WebSocket handlers, SSE handlers, Port processes, and others) with their state, type, and parent process. + +### Applications + +All loaded applications with state (loaded, running, stopping), mode, uptime, and their process groups. Full application process tree on click. Applications can be started, stopped, and unloaded from this view. + +### Network + +Network stack details: mode, acceptors, protocol and handshake versions, registrar. Below the top stat cards and acceptors, three tabs are available: + +* **Connections** lists all active remote node connections with traffic counters and sortable columns. +* **Routes** shows configured static routes and proxy routes. +* **Types** shows the wire-format type registry (one entry per proto): registration ID, name, kind, MinSize (zero-value wire size), and the inferred schema. Two filters narrow the list by name and by schema content. The data is captured on demand via the Refresh button. With `-tags=typestats`, additional columns show per-type encode/decode counts and decompressed wire-byte totals (see [Debugging: typestats Tag](../../advanced/debugging.md#the-typestats-tag)). + +### Events + +All registered events: producer PID, subscriber count, buffered flag, and publication statistics. Filter by name, notification mode, buffered mode, and minimum subscriber count. + +### Logs + +Live log stream from the observed node. Filter by level (debug, info, warning, error, panic). + +### Profiler + +**Goroutines** - dump of all goroutines with stack traces, grouping, and filtering by state and minimum wait time. + +**Heap** - allocation profile showing top call sites by bytes. Filter by minimum allocation size. + +Both are available without restarting the node or enabling any special build flags. + +## Actions + +Observer is not read-only. From process and meta-process views you can: + +* **Send a message** to a process or meta-process +* **Send an exit signal** with a custom reason +* **Kill** a process +* **Change log level** for the node, a specific process, or a specific meta-process +* **Adjust per-process network settings**: send priority, message ordering (`KeepNetworkOrder`), important delivery, compression type/level/threshold + +## Inspecting the whole cluster -* **Port**: The port number for the web server (default: `9911` if not specified). -* **Host**: The interface name (default: `localhost`). -* **LogLevel**: The logging level for the Observer application (useful for debugging). The default is `gen.LogLevelInfo` +Observer communicates with the `system` application, which is started automatically on every Ergo node. Because of this, a single Observer instance can switch to any node in the cluster and inspect it without deploying anything extra to that node. Use the node selector in the UI to connect to any cluster node, via the registrar if configured, or by entering the host, port, and cookie explicitly. diff --git a/docs/extra-library/applications/pulse.md b/docs/extra-library/applications/pulse.md new file mode 100644 index 000000000..a80ab55d5 --- /dev/null +++ b/docs/extra-library/applications/pulse.md @@ -0,0 +1,260 @@ +# Pulse + +Tracing in Ergo Framework records observations locally on each node. To see the complete picture of a trace spanning multiple nodes, you need to send those observations to an external system that assembles them. Pulse exports tracing observations to any OTLP-compatible backend (Grafana Tempo, Jaeger, OpenTelemetry Collector) over HTTP. + +Pulse runs as an application on your node. It registers itself as a tracing exporter, receives observations from the framework, batches them, and periodically flushes them to the configured collector. Each node in your cluster runs its own Pulse instance pointing to the same collector, and the backend assembles cross-node traces automatically. + +## Adding to Your Node + +```go +import ( + "ergo.services/application/pulse" + "ergo.services/ergo" + "ergo.services/ergo/gen" +) + +func main() { + node, err := ergo.StartNode("mynode@localhost", gen.NodeOptions{ + Applications: []gen.ApplicationBehavior{ + pulse.CreateApp(pulse.Options{ + URL: "http://tempo:4318/v1/traces", + }), + }, + }) + if err != nil { + panic(err) + } + node.Wait() +} +``` + +With this configuration, Pulse sends observations to `http://tempo:4318/v1/traces` using protobuf encoding. The node name (`mynode@localhost`) is used as the OTLP resource `service.name`, so the backend groups observations by node. + +## Configuration + +```go +pulse.Options{ + URL: "http://tempo:4318/v1/traces", // full collector URL + Headers: map[string]string{ // custom HTTP headers + "Authorization": "Bearer ", + }, + BatchSize: 512, // flush after N observations + FlushInterval: 5 * time.Second, // max time between flushes + PoolSize: 3, // number of export workers + ExportTimeout: 10 * time.Second, // HTTP request timeout + Flags: gen.TracingFlagSend | // which observations to receive + gen.TracingFlagReceive | + gen.TracingFlagProcs, +} +``` + +| Option | Default | Description | +|--------|---------|-------------| +| URL | `http://localhost:4318/v1/traces` | Full OTLP/HTTP collector URL. | +| Headers | none | Custom HTTP headers sent with every export request. Use for authentication tokens or routing headers. | +| BatchSize | `512` | Maximum number of observations in a batch. When the batch reaches this size, it is flushed immediately. | +| FlushInterval | `5s` | Maximum time between flushes. Even if the batch is not full, it is flushed after this interval. | +| PoolSize | `3` | Number of export workers. Each worker maintains its own HTTP client and batch buffer. Increase if your observation rate exceeds what three workers can export. | +| ExportTimeout | `10s` | HTTP request timeout per flush. If the collector doesn't respond within this time, the flush fails and the error is logged. | +| Flags | Send + Receive + Procs | Which observation types Pulse receives. By default, Pulse receives everything. Set a subset to reduce volume, for example `TracingFlagSend` to export only Sent observations. | + +## How It Works + +Pulse starts a pool of worker actors. The pool registers itself as a process-based tracing exporter on the node. When the framework emits an observation matching the configured flags, it delivers the observation to the pool, which distributes it to a worker. + +Each worker maintains a batch buffer. Observations accumulate until either the batch reaches `BatchSize` or `FlushInterval` elapses, whichever comes first. On flush, the worker converts the batch to OTLP protobuf format and sends it via HTTP POST to the collector. + +Each worker has its own HTTP client with persistent connections. Workers operate independently. If one worker's flush is slow (waiting on the network), others continue batching and flushing. This provides throughput resilience under variable network conditions. + +If a flush fails (network error, collector down, non-2xx response), the error is logged and the worker continues with the next batch. Observations from the failed batch are lost. This is a deliberate trade-off: retrying failed batches would introduce unbounded memory growth and backpressure that could affect the node's primary workload. + +On shutdown, each worker flushes any remaining observations before terminating. + +## OTLP Span Mapping + +Each Ergo observation becomes one OTLP span. The mapping is deterministic. Any node can compute the OTLP span ID for any observation without coordination. + +### Span ID Encoding + +The OTLP span ID encodes both the Ergo span ID and the observation point: + +``` +OTLP SpanID = ErgoSpanID << 2 | Point +``` + +Where Point is: Sent=1, Delivered=2, Processed=3. + +This means the three observations for a single message (Sent, Delivered, Processed) have related but distinct OTLP span IDs. Given any one, you can compute the other two. + +### Parent-Child Relationships + +| Observation | OTLP Parent | Meaning | +|-------------|-------------|---------| +| Sent (with parent) | Processed of causing message | "sent because of processing that message" | +| Sent (root) | none | first message in trace | +| Delivered | Sent of same message | "delivered after sent" | +| Processed | Sent of same message | "processed after sent" | +| Terminate.Processed | Processed of parent context | "process terminated" (no Sent for Terminate) | + +Sent is the anchor for each message. Delivered and Processed are its children at the same level. Response spans nest under Request.Processed, forming a natural call hierarchy: + +``` +Req.Sent +├── Req.Delivered +└── Req.Processed + └── Resp.Sent + └── Resp.Delivered +``` + +### Span Attributes + +Every OTLP span includes framework attributes prefixed with `ergo.`: + +- `ergo.node` : node where the observation was recorded +- `ergo.from` : sender process identity +- `ergo.to` : recipient identity +- `ergo.kind` : Send, Request, Response, Spawn, or Terminate +- `ergo.point` : Sent, Delivered, or Processed +- `ergo.behavior` : actor behavior type name +- `ergo.message` : message type name +- `ergo.ref` : call reference (for Request/Response correlation) + +Custom attributes set by the process via `SetTracingAttribute` and `SetTracingSpanAttribute` are included as additional OTLP span attributes. + +### Span Name + +The OTLP span name is formatted as: + +``` +{behavior} {kind}.{point} {message} +``` + +For example: `OrderProcessor Send.Sent main.ReserveStock`. + +### Span Kind Mapping + +The OTLP SpanKind depends on both the Ergo kind and the observation point: + +| Ergo Kind + Point | OTLP SpanKind | +|-------------------|---------------| +| Send.Sent | PRODUCER | +| Send.Delivered | CONSUMER | +| Send.Processed | CONSUMER | +| Request.Sent | CLIENT | +| Request.Delivered | SERVER | +| Request.Processed | SERVER | +| Response.Sent | SERVER | +| Response.Delivered | CLIENT | +| Response.Processed | SERVER | +| Spawn | INTERNAL | +| Terminate | INTERNAL | + +The Sent side of a message gets the initiator kind (CLIENT/PRODUCER), while the Delivered/Processed side gets the handler kind (SERVER/CONSUMER). For Response, the roles are inverted: Sent is SERVER (handler sending back), Delivered is CLIENT (caller receiving the answer). + +## Reading Traces in Grafana + +OTLP was designed for request-response services where a span represents a unit of work with a start and end time. Ergo's actor model is different: messages are instantaneous events (sent, delivered, processed), not duration-based operations. Pulse maps each event to a zero-duration OTLP span placed at the exact timestamp when the event occurred. + +In trace visualization tools (Grafana, Jaeger, Zipkin), these appear as dots on a timeline rather than bars. This is expected. The horizontal distance between dots shows actual timing, and the tree structure shows causality. + +### Call (Request/Response) + +``` +Time ─────────────────────────────────────────────────────────► + +Node A ● ● + Req.Sent Resp.Delivered + (CLIENT) (CLIENT) + +Node B ● ● ● + Req.Delivered Req.Processed Resp.Sent + (SERVER) (SERVER) (SERVER) + + ├── network ──┤── handling ──┤ ├── network ──┤ +``` + +- Req.Sent to Req.Delivered = network latency from A to B +- Req.Delivered to Req.Processed = time B spent handling the request +- Req.Processed to Resp.Sent = response creation time +- Resp.Sent to Resp.Delivered = network latency from B back to A + +### Send (async) + +``` +Time ──────────────────────────────────────► + +Node A ● + Send.Sent + (PRODUCER) + +Node B ● ● + Send.Delivered Send.Processed + (CONSUMER) (CONSUMER) + + ├── network ──┤── handling ──┤ +``` + +### Forward (multi-hop) + +``` +Time ──────────────────────────────────────────────────────────────────────► + +Node A ● ● + Req.Sent Resp.Delivered + +Node B ● ● ● + Req.Delivered Req.Processed + Fwd.Sent + +Node C ● ● ● + Fwd.Delivered Fwd.Processed + Resp.Sent + + ├── network ──┤─ handling ─┤── network ──┤─ handling ─┤── network ──┤ +``` + +For duration-based visualization with timing bars, use the Observer web UI which renders Ergo traces natively. + +## Inspecting Workers + +Each Pulse worker exposes statistics through the standard inspection mechanism. In the Observer process list, find the Pulse worker processes and inspect them to see: + +- `spans_received` : total observations received by this worker +- `spans_exported` : total observations successfully exported +- `export_errors` : total failed flush attempts +- `batch_size` : current batch length + +These counters help diagnose export problems: if `export_errors` is growing, the collector may be unreachable or overloaded. + +## Grafana Dashboard + +Pulse includes a ready-to-use Grafana dashboard for trace search. Import `grafana-tracing.json` from the Pulse module into your Grafana instance. During import, Grafana will ask you to select a Tempo datasource. + +The dashboard provides a TraceQL filter for searching traces by node, behavior, message type, or any span attribute. Results include columns for service name, ergo.kind, ergo.behavior, and ergo.message. Click any Trace ID to open the full waterfall view. + +## Grafana Tempo Setup + +A minimal Tempo configuration for local development: + +```yaml +# tempo.yaml +server: + http_listen_port: 3200 + +distributor: + receivers: + otlp: + protocols: + http: + endpoint: "0.0.0.0:4318" + +storage: + trace: + backend: local + local: + path: /var/tempo/traces + wal: + path: /var/tempo/wal +``` + +Point Pulse at `tempo:4318` with `Insecure: true`. In Grafana, add Tempo as a data source (`http://tempo:3200`) and use the Explore view to search for traces by trace ID or attributes. diff --git a/docs/extra-library/applications/radar.md b/docs/extra-library/applications/radar.md new file mode 100644 index 000000000..f0a8fe74c --- /dev/null +++ b/docs/extra-library/applications/radar.md @@ -0,0 +1,368 @@ +# Radar + +Running an Ergo node in production typically requires two things: health probes for Kubernetes and a Prometheus metrics endpoint. Setting them up separately means two HTTP servers on two ports, two actor packages to import, and the same wiring code repeated on every node. + +Radar bundles both into a single application on one HTTP port. Internally it runs a [Health](../actors/health.md) actor for probe endpoints, a [Metrics](../actors/metrics.md) actor for base Ergo telemetry, and a pool of metrics workers for custom metric updates, all behind a shared mux served by one HTTP server. Actors interact with Radar through helper functions in the `radar` package without importing the underlying packages or knowing the internal actor names. + +## Adding to Your Node + +```go +import ( + "ergo.services/application/radar" + "ergo.services/ergo" + "ergo.services/ergo/gen" +) + +func main() { + node, _ := ergo.StartNode("mynode@localhost", gen.NodeOptions{ + Applications: []gen.ApplicationBehavior{ + radar.CreateApp(radar.Options{Port: 9090}), + }, + }) + + // Health: http://localhost:9090/health/live + // http://localhost:9090/health/ready + // http://localhost:9090/health/startup + // Metrics: http://localhost:9090/metrics + + node.Wait() +} +``` + +With no signals registered, all three health endpoints return 200 with `{"status":"healthy"}`. The metrics endpoint immediately serves base Ergo metrics. No additional configuration is required for a working production setup. + +## Configuration + +```go +radar.Options{ + Host: "0.0.0.0", + Port: 9090, + HealthPath: "/health", + MetricsPath: "/metrics", + HealthCheckInterval: 2 * time.Second, + MetricsCollectInterval: 15 * time.Second, + MetricsTopN: 100, + MetricsPoolSize: 5, +} +``` + +**Host** determines which network interface the HTTP server binds to. Default is `"localhost"`. Use `"0.0.0.0"` for containerized environments where probes and scraping come from outside the pod. + +**Port** sets the single HTTP port for all endpoints. Default is `9090`. Choose a port that does not conflict with your application's own listeners. + +**HealthPath** sets the URL prefix for health probe endpoints. Default is `"/health"`. The actual endpoints become `HealthPath+"/live"`, `HealthPath+"/ready"`, `HealthPath+"/startup"`. Change this when deploying behind a reverse proxy that expects a different path prefix. + +**MetricsPath** sets the URL path for the Prometheus scrape target. Default is `"/metrics"`. + +**HealthCheckInterval** controls how often the health actor checks for expired heartbeats. Default is 1 second. Shorter intervals detect failures faster but increase internal message traffic. For most applications, 1-2 seconds provides a good balance. + +**MetricsCollectInterval** sets how often base Ergo metrics are collected (processes, memory, CPU, network, events). Default is 10 seconds. Align this with your Prometheus scrape interval; collecting more frequently than Prometheus scrapes wastes CPU; collecting less frequently means Prometheus may see stale values. + +**MetricsTopN** limits the number of entries in per-process and per-event top-N metrics tables. Default is 50. Increase this for large nodes with thousands of processes where you need broader visibility into the tail. The collection cost scales linearly with TopN. + +**MetricsPoolSize** sets the number of worker actors in the custom metrics pool. Default is 3. Under normal load, a single worker is sufficient. Increase this if many actors send frequent metric updates and you observe the metrics mailbox growing. + +## Health Probes + +Actors register signals with Radar, specifying which probes the signal affects and an optional heartbeat timeout. The health actor monitors the registering process; if it terminates, all its signals are automatically marked as down. + +### Registering a Signal + +```go +func (w *DBWorker) Init(args ...any) error { + radar.RegisterService(w, "postgres", + radar.ProbeLiveness|radar.ProbeReadiness, 10*time.Second) + + w.scheduleHeartbeat() + return nil +} + +func (w *DBWorker) HandleMessage(from gen.PID, message any) error { + switch message.(type) { + case messageHeartbeat: + radar.Heartbeat(w, "postgres") + w.scheduleHeartbeat() + } + return nil +} + +func (w *DBWorker) scheduleHeartbeat() { + w.cancelHeartbeat, _ = w.SendAfter(w.PID(), messageHeartbeat{}, 3*time.Second) +} +``` + +The signal `"postgres"` participates in both liveness and readiness probes. If the heartbeat stops arriving (timeout expires) or the process terminates, Kubernetes receives a 503 on both `/health/live` and `/health/ready`. + +### Probe Types + +| Constant | Endpoint | +|----------|----------| +| `radar.ProbeLiveness` | `/health/live` | +| `radar.ProbeReadiness` | `/health/ready` | +| `radar.ProbeStartup` | `/health/startup` | + +Combine with bitwise OR. A signal registered for `ProbeLiveness|ProbeReadiness` affects both endpoints independently. + +### Manual Signal Control + +When you can detect failures immediately without waiting for a timeout: + +```go +case CacheConnectionLost: + radar.ServiceDown(w, "cache") + +case CacheConnectionRestored: + radar.ServiceUp(w, "cache") +``` + +### Helper Functions + +```go +radar.RegisterService(process, signal, probe, timeout) // sync Call +radar.UnregisterService(process, signal) // sync Call +radar.Heartbeat(process, signal) // async Send +radar.ServiceUp(process, signal) // async Send +radar.ServiceDown(process, signal) // async Send +``` + +`RegisterService` and `UnregisterService` are synchronous calls that return an error on failure. `Heartbeat`, `ServiceUp`, and `ServiceDown` are asynchronous sends (fire-and-forget). + +For a detailed explanation of the heartbeat model, failure detection mechanisms, and the HTTP response format, see the [Health](../actors/health.md) actor documentation. + +## Custom Metrics + +Actors register Prometheus metric collectors and update them through Radar's helper functions. The underlying metrics actor manages the Prometheus registry and HTTP exposition. Registration is synchronous, updates are asynchronous. + +All custom metrics automatically receive a `node` const label set to the node name. Do not include `"node"` in your variable label names; it will cause a "duplicate label names" registration error. + +### Registering Metrics + +```go +func (w *APIHandler) Init(args ...any) error { + radar.RegisterGauge(w, "active_connections", + "Number of active client connections", []string{"protocol"}) + + radar.RegisterCounter(w, "requests_total", + "Total HTTP requests processed", []string{"method", "status"}) + + radar.RegisterHistogram(w, "request_duration_seconds", + "Request latency distribution", []string{"method"}, + []float64{0.01, 0.05, 0.1, 0.5, 1.0, 5.0}) + + return nil +} +``` + +The `labels` parameter defines the label names for the metric. When updating, you provide label values in the same order. Pass `nil` for metrics without labels. The `buckets` parameter in `RegisterHistogram` defines histogram bucket boundaries; pass `nil` for Prometheus default buckets. + +### Updating Metrics + +```go +func (w *APIHandler) HandleMessage(from gen.PID, message any) error { + switch msg := message.(type) { + case RequestCompleted: + radar.CounterAdd(w, "requests_total", 1, + []string{msg.Method, msg.StatusCode}) + radar.HistogramObserve(w, "request_duration_seconds", + msg.Duration.Seconds(), []string{msg.Method}) + case ConnectionChange: + radar.GaugeSet(w, "active_connections", + float64(msg.Count), []string{msg.Protocol}) + } + return nil +} +``` + +Updates are distributed across the worker pool. Under high throughput, multiple actors can send updates concurrently without contending on a single actor's mailbox. + +### Automatic Cleanup + +When a process that registered metrics terminates, all its metrics are automatically unregistered from the Prometheus registry. No explicit cleanup is needed. To remove a metric while the process is still running, use `radar.UnregisterMetric(process, name)`. + +### Helper Functions + +```go +// Registration (sync Call, returns error) +radar.RegisterGauge(process, name, help, labels) +radar.RegisterCounter(process, name, help, labels) +radar.RegisterHistogram(process, name, help, labels, buckets) +radar.UnregisterMetric(process, name) + +// Updates (async Send, fire-and-forget) +radar.GaugeSet(process, name, value, labels) +radar.GaugeAdd(process, name, value, labels) +radar.CounterAdd(process, name, value, labels) +radar.HistogramObserve(process, name, value, labels) +``` + +For a detailed explanation of metric types, the Grafana dashboard, and advanced usage (embedding, shared mode), see the [Metrics](../actors/metrics.md) actor documentation. + +## Top-N Metrics + +Top-N metrics track the N highest (or lowest) values observed during each collection cycle and flush them to Prometheus as a GaugeVec. This is useful when you want to identify outliers (slowest queries, busiest workers, largest payloads) without creating a time series per item. + +### Registering and Observing + +```go +func (w *QueryTracker) Init(args ...any) error { + // Keep the 10 slowest queries each cycle + radar.RegisterTopN(w, "slowest_queries", "Slowest DB queries", + 10, radar.TopNMax, []string{"query", "table"}) + return nil +} + +func (w *QueryTracker) HandleMessage(from gen.PID, message any) error { + switch msg := message.(type) { + case queryCompleted: + radar.TopNObserve(w, "slowest_queries", msg.Duration.Seconds(), + []string{msg.SQL, msg.Table}) + } + return nil +} +``` + +Registration is synchronous (returns error). Observations are asynchronous (fire-and-forget). Each top-N metric is managed by a dedicated actor that accumulates observations and flushes the top entries to Prometheus on the same interval as base metrics collection. + +### Ordering Modes + +- `radar.TopNMax`: keeps the N largest values (e.g., slowest queries, busiest actors, highest memory) +- `radar.TopNMin`: keeps the N smallest values (e.g., lowest latency, least active processes) + +### Automatic Cleanup + +When the process that registered a top-N metric terminates, the metric actor cleans up and unregisters from Prometheus. No explicit teardown needed. + +### Helper Functions + +```go +// Registration (sync Call, returns error) +radar.RegisterTopN(process, name, help, topN, order, labels) + +// Observation (async Send, fire-and-forget) +radar.TopNObserve(process, name, value, labels) +``` + +## Common Patterns + +### Database Connection Pool + +An actor that manages a connection pool reports both health and metrics through Radar: + +```go +func (w *DBPool) Init(args ...any) error { + // Health: liveness + readiness with heartbeat + radar.RegisterService(w, "db_pool", + radar.ProbeLiveness|radar.ProbeReadiness, 10*time.Second) + + // Metrics: connection pool gauge + radar.RegisterGauge(w, "db_pool_connections", + "Database connection pool size", []string{"state"}) + + w.scheduleCheck() + return nil +} + +func (w *DBPool) HandleMessage(from gen.PID, message any) error { + switch message.(type) { + case messageCheck: + if w.pool.Ping() == nil { + radar.Heartbeat(w, "db_pool") + } + radar.GaugeSet(w, "db_pool_connections", + float64(w.pool.ActiveCount()), []string{"active"}) + radar.GaugeSet(w, "db_pool_connections", + float64(w.pool.IdleCount()), []string{"idle"}) + + w.scheduleCheck() + } + return nil +} +``` + +A single periodic check updates both the health signal and connection pool metrics. If the database becomes unreachable, the heartbeat stops and Kubernetes removes the pod from service. The metrics endpoint continues to show the last known pool state until the pod restarts. + +### Startup Gate with Progress + +An actor that runs migrations uses the startup probe to prevent premature traffic, and reports progress via a gauge: + +```go +func (w *Migrator) Init(args ...any) error { + radar.RegisterService(w, "migrations", radar.ProbeStartup, 0) + radar.RegisterGauge(w, "migrations_pending", + "Number of pending migrations", nil) + + w.Send(w.PID(), messageRunMigrations{}) + return nil +} + +func (w *Migrator) HandleMessage(from gen.PID, message any) error { + switch message.(type) { + case messageRunMigrations: + pending := w.countPending() + radar.GaugeSet(w, "migrations_pending", float64(pending), nil) + + if err := w.runNext(); err != nil { + return err + } + + if w.countPending() > 0 { + w.Send(w.PID(), messageRunMigrations{}) + return nil + } + + // All done -- mark startup complete + radar.GaugeSet(w, "migrations_pending", 0, nil) + radar.ServiceUp(w, "migrations") + radar.UnregisterService(w, "migrations") + } + return nil +} +``` + +While migrations run, the startup probe returns 503, Kubernetes waits, and Prometheus shows the remaining migration count. Once complete, the startup signal is released and liveness/readiness probes take over. + +## Kubernetes Configuration + +Configure Kubernetes probes and Prometheus scraping to point at the same port: + +```yaml +apiVersion: v1 +kind: Pod +spec: + containers: + - name: myapp + livenessProbe: + httpGet: + path: /health/live + port: 9090 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /health/ready + port: 9090 + periodSeconds: 10 + startupProbe: + httpGet: + path: /health/startup + port: 9090 + failureThreshold: 30 + periodSeconds: 2 +``` + +Prometheus scrape configuration: + +```yaml +scrape_configs: + - job_name: 'ergo' + static_configs: + - targets: ['localhost:9090'] + scrape_interval: 15s +``` + +Align `scrape_interval` with `MetricsCollectInterval` in Radar options. The default collect interval is 10 seconds; scraping more frequently than the collect interval returns identical data. + +## Relationship to Health and Metrics Actors + +Radar uses [Health](../actors/health.md) and [Metrics](../actors/metrics.md) actors internally. The helper functions in the `radar` package delegate to these actors by their internal registered names. If you need capabilities beyond what the helpers expose (embedding the metrics actor for direct Prometheus registry access, custom health actor behavior with `HandleSignalDown` callbacks, or shared mux with additional HTTP handlers), use the underlying actors directly. + +Radar is designed for the common case: production nodes that need standard health probes and Prometheus metrics with minimal setup. For advanced scenarios, the building blocks are available as separate packages. diff --git a/docs/extra-library/network-protocols/erlang.md b/docs/extra-library/network-protocols/erlang.md index 1af0d6b18..c1f58767d 100644 --- a/docs/extra-library/network-protocols/erlang.md +++ b/docs/extra-library/network-protocols/erlang.md @@ -33,6 +33,8 @@ To use this package, include `ergo.services/proto/erlang23/handshake`. The `ergo.services/proto/erlang/dist` package implements the `gen.NetworkProto` and `gen.Connection` interfaces. To create it, use the `dist.Create` function and provide `dist.Options` as an argument, where you can specify the `FragmentationUnit` size in bytes. This value is used for fragmenting large messages. The default size is set to `65000` bytes. +The Erlang DIST proto deliberately does **not** implement `gen.TypeRegistry`, because the Erlang external term format (ETF) carries primitives, atoms, lists, tuples, and binaries directly on the wire without a separate type-registration step. In a multi-proto setup, calls to `node.Network().RegisterType` skip the Erlang proto and register only in TypeRegistry-capable protos like the default ENP/EDF stack. Use `etf.RegisterTypeOf` (described below) to teach the Erlang decoder how to map incoming tuples or atoms to your Go types. + To use this package, include `ergo.services/proto/erlang/dist`. ### ETF data format diff --git a/docs/faq.md b/docs/faq.md new file mode 100644 index 000000000..949cf28fd --- /dev/null +++ b/docs/faq.md @@ -0,0 +1,236 @@ +--- +description: Answers to the questions developers and AI assistants ask most often +--- + +# FAQ + +## General + +### What is Ergo Framework? + +Ergo is an open-source Go framework for building concurrent and distributed systems using the actor model. It brings Erlang/OTP design patterns, including isolated processes, supervision trees, and network-transparent messaging, to Go with zero external dependencies. + +### Is Ergo production-ready? + +Yes. Ergo is used in production systems. It supports [mTLS](networking/mutual-tls.md), [NAT traversal](networking/behind-the-nat.md), graceful shutdown, panic recovery with stack traces, and has a comprehensive test suite. The framework has been in active development since 2019. + +### What license is Ergo distributed under? + +MIT License. Free to use in commercial projects without restrictions. + +### What Go version is required? + +Go 1.21 or higher. No other dependencies. + +## Actor Model + +### What is the actor model and why use it in Go? + +The actor model is a concurrency paradigm where independent units (actors, also called processes) communicate exclusively through message passing. Each actor has private state and processes messages one at a time. No shared memory, no mutexes, no race conditions. + +Go's goroutines and channels are powerful but don't enforce isolation. Goroutines can share memory, which requires manual synchronization. Ergo enforces the actor model guarantees: isolated state, message-only communication, and sequential processing per actor. See [Actor Model](basics/actor-model.md). + +### How is an Ergo process different from a goroutine? + +| | Goroutine | Ergo Process | +|---|---|---| +| Identity | No stable address | Has PID, addressable locally and remotely | +| State | Can share memory | Strictly private | +| Failure recovery | Manual | Automatic via supervision | +| Cross-node messaging | Not built in | Same API, transparent | +| Race conditions | Possible | Impossible within a process | + +See [Process](basics/process.md) for details. + +### How many processes can run on a single node? + +Thousands to hundreds of thousands. Processes sleep when idle and consume no CPU. Memory footprint per process is minimal, comparable to a goroutine plus a small mailbox struct. + +### Can actors communicate synchronously? + +Yes. Ergo supports both async (`Send`) and sync (`Call`) patterns. `Call` blocks the calling process until a response arrives or a timeout occurs, while maintaining full actor model guarantees. See [Handling Sync Requests](advanced/handle-sync.md). + +## Fault Tolerance + +### What happens when an actor crashes? + +Its supervisor detects the failure and applies a restart strategy: + +- **One-For-One**: restart only the failed child +- **All-For-One**: restart all children when one fails +- **Rest-For-One**: restart the failed child and all children started after it +- **Simple-One-For-One**: identical children spawned dynamically at runtime, restart failed ones + +Supervision trees are hierarchical. A failed subtree is isolated and recovered without affecting the rest of the system. See [Supervision Tree](basics/supervision-tree.md) and [Supervisor](actors/supervisor.md). + +### Do I need to write retry logic? + +No. Supervision handles process recovery automatically. For message delivery, use the [Important Delivery](advanced/important-delivery.md) flag for guaranteed delivery semantics. The sender receives an immediate error if the target doesn't exist, rather than a timeout. + +### What happens if a remote node disconnects? + +All processes that were monitoring or linked to processes on the disconnected node receive a notification (`MessageDownNode` or exit signal). Your actors handle this notification and decide how to respond: retry, failover, or graceful degradation. See [Links and Monitors](basics/links-and-monitors.md). + +## Distributed Systems + +### How do nodes find each other? + +Through a registrar. Each node runs a minimal built-in registrar by default. Nodes on the same host discover each other automatically via localhost. For production clusters across multiple hosts, configure an external registrar: + +- **etcd**: distributed key-value store, widely used +- **Saturn**: Ergo's own central registrar, purpose-built for Ergo clusters + +See [Service Discovering](networking/service-discovering.md). + +### Do I need Kubernetes or a service mesh? + +No. Ergo eliminates the integration tax of traditional microservice architectures. No HTTP or gRPC endpoints to define between services, no sidecar proxies, no API gateways for internal routing. Process-to-process communication is direct through the framework's network layer. + +Ergo does support Kubernetes for deployment. The [Health](extra-library/actors/health.md) actor provides liveness, readiness, and startup health probes, and the [Metrics](extra-library/actors/metrics.md) actor provides Prometheus metrics on a single port. + +### How does Ergo handle network partitions? + +The [Leader](extra-library/actors/leader.md) actor uses a Raft-inspired consensus algorithm with majority quorum to prevent split-brain scenarios. When a partition occurs, only the partition with a majority of nodes continues to elect a leader. Minority partitions stop processing leader-dependent operations until connectivity is restored. + +### Can I run Ergo nodes across different clouds? + +Yes. [ergo.cloud](https://ergo.cloud) is a managed overlay network that connects Ergo nodes across AWS, GCP, Azure, and bare metal into one transparent cluster without VPNs, proxies, or tunnels. End-to-end encrypted. Currently available via waitlist. + +## Pub/Sub + +### How does distributed Pub/Sub work in Ergo? + +A producer process registers a named event. Any process on any node subscribes using `LinkEvent` or `MonitorEvent`. The framework delivers messages to all subscribers transparently across the cluster. + +```go +// Producer +token, _ := producer.RegisterEvent("market.prices", gen.EventOptions{}) +producer.SendEvent("market.prices", token, PriceUpdate{Asset: "BTC", Price: 95000}) + +// Subscriber on any node +process.MonitorEvent(gen.Event{Name: "market.prices", Node: "producer@host"}) + +// Event messages arrive in HandleEvent +func (s *Sub) HandleEvent(event gen.MessageEvent) error { + update := event.Message.(PriceUpdate) + // handle update + return nil +} + +// Producer termination or event unregister arrives in HandleMessage as MessageDownEvent +func (s *Sub) HandleMessage(from gen.PID, msg any) error { + switch msg.(type) { + case gen.MessageDownEvent: + // producer terminated or unregistered + } + return nil +} +``` + +See [Events](basics/events.md). + +### How does Ergo Pub/Sub scale? + +The framework uses fan-out at the consumer node level, not per subscriber. One network message is sent per remote node regardless of how many subscribers that node has. Local delivery then fans out within the node. + +Result: 2.9M messages/second delivery rate to 1,000,000 subscribers across 10 nodes using only 10 network messages, not 1,000,000. See [Pub/Sub Internals](advanced/pub-sub-internals.md). + +### What's the difference between Links, Monitors, and Events? + +All three use the same underlying pub/sub mechanism internally. All three are unidirectional: the notification flows from the target to the watcher, not the other way around. Note this differs from Erlang, where links are bidirectional. + +- **Link**: when the target terminates, the watcher receives an exit signal on its Urgent queue. The default behavior is to terminate the watcher. Actors can enable exit trapping to receive the signal as a `gen.MessageExit*` message and decide how to react. +- **Monitor**: when the target terminates, the watcher receives a `gen.MessageDown*` notification on its System queue. The watcher continues running. +- **Event**: the watcher subscribes to a named stream of messages published by a producer. The producer terminating also delivers a notification (exit signal for link-based subscriptions, down message for monitor-based). + +See [Links and Monitors](basics/links-and-monitors.md) and [Pub/Sub Internals](advanced/pub-sub-internals.md). + +## Performance + +### How fast is Ergo? + +- 21M+ messages/second locally on a 64-core processor +- ~5.5M messages/second over the network +- EDF serialization: up to 47% faster encoding than Protobuf, 6 to 14 times faster than Gob +- Distributed Pub/Sub: 2.9M msg/sec to 1M subscribers across 10 nodes + +Full benchmarks: [benchmarks repository](https://github.com/ergo-services/benchmarks). + +### How does Ergo serialization compare to Protobuf? + +Ergo uses EDF (Ergo Data Format) with type caching. For repeated message types, type metadata is cached after the first transmission. Subsequent messages of the same type skip type information entirely. This makes EDF significantly faster than Protobuf for encoding and decoding in high-throughput scenarios. + +## Observability + +### Does Ergo support distributed tracing? + +Yes. Ergo has native distributed tracing that follows message chains across processes and nodes. When a traced process sends a message, the trace identity travels with the message and propagates automatically through the entire downstream chain of handlers. You configure tracing on entry-point processes. Downstream actors need no instrumentation. + +Traces can be viewed directly in Observer as waterfall diagrams or exported to OTLP-compatible backends (Grafana Tempo, Jaeger, OpenTelemetry Collector) via the [Pulse application](extra-library/applications/pulse.md). See [Distributed Tracing](advanced/distributed-tracing.md) for details. + +### How do I inspect a running node? + +Run the [Observer](extra-library/applications/observer.md) web UI for live visibility into processes, applications, network connections, events, logs, tracing waterfalls, and heap profiles. For AI-driven investigation, use the [MCP application](extra-library/applications/mcp.md) to expose the running system to Claude Code, Cursor, or any MCP-compatible client. For continuous metrics, the [Radar](extra-library/applications/radar.md) application provides a Prometheus endpoint with a ready-to-use Grafana dashboard. + +## Integration + +### Can Ergo nodes talk to Erlang/Elixir nodes? + +Yes. Ergo supports the full Erlang network stack: EPMD, ETF (External Term Format), and DIST protocol. You can build hybrid Go/Erlang clusters where Ergo nodes and BEAM nodes coexist and communicate natively. See [Erlang protocol](extra-library/network-protocols/erlang.md). + +### Does Ergo work with Prometheus and Grafana? + +Yes. The [Metrics](extra-library/actors/metrics.md) actor exports node and network telemetry via a Prometheus HTTP endpoint. A ready-to-use Grafana dashboard is provided via [Radar](extra-library/applications/radar.md). + +### Does Ergo support WebSockets and SSE? + +Yes, via [Meta Processes](basics/meta-process.md). Each [WebSocket](extra-library/meta-processes/websocket.md) or [SSE](extra-library/meta-processes/sse.md) connection becomes an independent meta-process with a stable identifier (`gen.Alias`). Any actor anywhere in the cluster can send messages directly to a specific client connection. No routing intermediaries needed. This enables real-time push from any cluster node to any specific connected client. + +### Can I use Ergo with standard Go HTTP libraries? + +Yes. Ergo's [Web](meta-processes/web.md) meta-process integrates with standard `net/http`. You use any Go router (stdlib ServeMux, gorilla/mux, chi, echo) and any HTTP middleware. Actors are an implementation detail invisible to the HTTP layer. + +## AI and MCP + +### Can Ergo be used for AI agent infrastructure? + +Yes, and it is particularly well-suited. Each AI agent runs as an isolated process with a mailbox. No shared state between agents, no race conditions. Supervisor trees restart stuck or crashed agents automatically. Multiple agents coordinate through message passing. Agents distribute transparently across cluster nodes as load grows. See [AI Agents](ai-agents.md) for patterns and diagnostics. + +### What is MCP support in Ergo? + +Ergo has built-in support for the Model Context Protocol (MCP), an emerging standard for AI tool integration. The [MCP application](extra-library/applications/mcp.md) exposes the running cluster to AI assistants (Claude Code, Cursor, and any MCP-compatible client) as a set of diagnostic tools. The AI inspects processes, queries events, captures goroutine dumps, reads logs, and runs samplers through natural language. + +Two deployment modes: + +- **Entry point**: the node runs an HTTP listener that accepts MCP requests. This is the node your AI client connects to. +- **Agent**: no HTTP listener. Accessible via cluster proxy from the entry point node. Use this for internal nodes that should be inspectable without exposing an HTTP port. + +## Getting Started + +### How do I create my first Ergo project? + +``` +# Install the project generator +go install ergo.tools/ergo@latest + +# Create a project +ergo init MyNode github.com/myorg/mynode +cd mynode + +# Add components +ergo add supervisor MyNodeApp:MySup +ergo add actor MySup:MyWorker + +# Run +go run ./cmd +``` + +See [ergo tool documentation](tools/ergo.md) for the full command reference. + +### Where can I get help? + +- [Documentation](https://docs.ergo.services) +- [Examples](https://github.com/ergo-services/examples) +- [Telegram community](https://t.me/ergo_services) +- [GitHub Discussions](https://github.com/ergo-services/ergo/discussions) +- Commercial support: support@ergo.services diff --git a/docs/networking/network-stack.md b/docs/networking/network-stack.md index cd9558956..27a96677c 100644 --- a/docs/networking/network-stack.md +++ b/docs/networking/network-stack.md @@ -103,9 +103,37 @@ After handshake, the accepting node tells the dialing node to create a connectio The dialing node opens additional TCP connections using a shortened join handshake (skips full authentication since the first connection already authenticated). These connections join the pool, forming a single logical connection with multiple physical TCP links. -Multiple connections enable parallel message delivery. Each message goes to a connection based on the sender's identity (derived from sender PID). Messages from the same sender always use the same connection, preserving order. Messages from different senders use different connections, enabling parallelism. +Multiple connections enable parallel message delivery. Each message goes to a connection based on the sender's identity, and the receiving side creates multiple receive queues per TCP connection for concurrent processing. This two-level mechanism (sender-side link selection and receiver-side queue routing) preserves per-sender message ordering while enabling parallelism across different senders. For details on how ordering works, including the `KeepNetworkOrder` flag and when to disable it, see [Message Ordering](network-transparency.md#message-ordering). -The receiving side creates 4 receive queues per TCP connection. A 3-connection pool has 12 receive queues processing messages concurrently. This parallel processing improves throughput while preserving per-sender message ordering. +### Software Keepalive + +*Introduced in v3.3.0.* + +TCP keepalive operates at the OS level - it detects hard network failures like unplugged cables or crashed hosts. But it can't detect application-level problems: a stuck process that stopped reading from a connection, a flusher that failed silently, a goroutine that never got scheduled. The connection looks alive to TCP while no useful data flows. + +Software keepalive works at the protocol level. When a connection pool item has nothing to send, its flusher periodically writes a small keepalive packet. The receiving side expects these packets and sets a read deadline based on the sender's advertised period. If nothing arrives - no real messages and no keepalive packets - the deadline fires and the connection is terminated. + +Each side advertises its keepalive period during handshake. This allows asymmetric configuration: a node in a reliable datacenter might send keepalive every 15 seconds, while a node on an unstable network might send every 5 seconds. The receiver calculates its deadline from the sender's period, not its own. + +```go +node, err := ergo.StartNode("myapp@localhost", gen.NodeOptions{ + Network: gen.NetworkOptions{ + Flags: gen.NetworkFlags{ + // ... other flags ... + EnableSoftwareKeepAlive: 15, // send keepalive every 15 seconds when idle + }, + SoftwareKeepAliveMisses: 3, // tolerate 3 missed keepalives before disconnect + }, +}) +``` + +The timeout calculation uses the remote node's period, not the local one. If the remote node advertises a 15-second period and you configure 3 misses, the connection is considered dead after 45 seconds of silence. Real messages reset the deadline just like keepalive packets do - on a busy connection, keepalive is never sent because regular traffic keeps the deadline from expiring. + +When a keepalive timeout fires on any pool item, the entire connection is terminated - not just the affected TCP link. A single unresponsive link is strong evidence that the whole network path to the remote node is down. This triggers the standard cleanup flow: monitors receive `MessageDown`, links receive `MessageExit`, and the connection is removed from the node's connection map. + +Software keepalive is enabled by default (15-second period, 3 misses, 45-second timeout). Set `EnableSoftwareKeepAlive` to 0 to disable it. Acceptors and routes can override the misses count; zero inherits from `NetworkOptions`. + +Both sides must have keepalive enabled for the feature to activate. If either side advertises period 0, the connection falls back to TCP-only keepalive with infinite read deadline - neither side sends keepalive packets and neither side sets read deadlines. This means a single node with keepalive disabled in a cluster removes protection for all its connections, not just its own. During a rolling upgrade from older nodes (which don't support the feature) to newer ones, connections between old and new nodes will not have software keepalive until both sides are upgraded. ## Message Encoding and Transmission @@ -115,7 +143,7 @@ Once a connection exists, messages flow through encoding and framing. EDF is a binary encoding specifically designed for the framework's communication patterns. It's type-aware - each value is prefixed with a type tag (e.g., `0x95` for int64, `0xaa` for PID, `0x9d` for slice). The decoder reads the tag and knows what follows. -Framework types like `gen.PID` and `gen.Ref` have optimized encodings. Structs are encoded field-by-field in declaration order (no field names on the wire). Custom types must be registered on both sides - registration happens during `init()`, and during handshake nodes exchange their type lists to agree on encoding. +Framework types like `gen.PID` and `gen.Ref` have optimized encodings. Structs are encoded field-by-field in declaration order (no field names on the wire). Custom types must be registered on both sides via `node.Network().RegisterType` (typically from an application's `Load` callback). During handshake, nodes exchange their type lists to agree on encoding. Compression is automatic. If a message exceeds the compression threshold (default 1024 bytes), it's compressed using GZIP, ZLIB, or LZW. The protocol frame indicates compression, so the receiver decompresses before decoding. @@ -129,6 +157,39 @@ The order byte preserves message ordering per sender. Messages from the same sen For details on protocol framing, order bytes, receive queue distribution, and the exact byte layout, see [Network Transparency](network-transparency.md). +### Message Fragmentation + +*Introduced in v3.3.0.* + +When a message exceeds the fragment size threshold (default 65000 bytes), the framework splits it into smaller pieces for transmission and reassembles them on the receiving side. This happens after compression; if a compressed message is still too large, it gets fragmented. From your code's perspective, nothing changes. You send a large message, and it arrives intact. + +Fragmentation works with all message types: regular sends, important delivery, calls, and events. It composes with compression: a message can be compressed first, then fragmented, and on the receiving side defragmented and then decompressed. + +When [`KeepNetworkOrder`](network-transparency.md#message-ordering) is disabled for a process, the framework distributes fragments across all TCP connections in the pool, using the full bandwidth of the connection. This is useful for transferring large payloads where throughput matters more than ordering. When `KeepNetworkOrder` is enabled (the default), all fragments travel through a single TCP connection to preserve message ordering for that sender. + +Both nodes must have `EnableFragmentation` in their network flags. If either side doesn't support it, large messages are sent as-is (subject to `MaxMessageSize` limits). During handshake, nodes exchange their fragmentation capability, and the feature activates only when both sides agree. + +`MaxMessageSize` is a logical limit on the EDF-encoded message, checked before compression and fragmentation. On the receiving side, the framework tracks the accumulated size of received fragments and rejects the assembly if it exceeds the limit. + +```go +node, err := ergo.StartNode("myapp@localhost", gen.NodeOptions{ + Network: gen.NetworkOptions{ + Flags: gen.NetworkFlags{ + EnableFragmentation: true, // default: true + }, + FragmentSize: 65000, // bytes per fragment, 0 = default + FragmentTimeout: 30, // seconds, assembly timeout, 0 = default + MaxFragmentAssemblies: 1000, // max concurrent assemblies, 0 = default + }, +}) +``` + +`FragmentSize` controls at what point messages get split. This is a sender-side setting; the receiver reassembles whatever arrives regardless of the sender's fragment size. Two nodes can use different fragment sizes. + +`FragmentTimeout` sets how long the receiver waits for all fragments before discarding an incomplete assembly. If a sender crashes mid-message or a connection drops, partial assemblies are cleaned up after this timeout. + +`MaxFragmentAssemblies` limits how many messages can be simultaneously reassembled per connection, protecting against memory exhaustion from many concurrent large messages. + ## Network Transparency in Practice Network transparency means remote operations look like local operations. You send to a PID without checking if it's local or remote. You establish links and monitors the same way regardless of location. The framework handles discovery, encoding, and transmission automatically. @@ -158,7 +219,12 @@ node, err := ergo.StartNode("myapp@localhost", gen.NodeOptions{ EnableRemoteSpawn: true, EnableRemoteApplicationStart: true, EnableImportantDelivery: true, + EnableFragmentation: true, // default: true + EnableSoftwareKeepAlive: 15, // seconds, 0 to disable }, + SoftwareKeepAliveMisses: 3, // tolerate 3 missed keepalives + FragmentSize: 65000, // 0 = default + FragmentTimeout: 30, // seconds, 0 = default Acceptors: []gen.AcceptorOptions{ { Port: 15000, @@ -176,13 +242,13 @@ node, err := ergo.StartNode("myapp@localhost", gen.NodeOptions{ **MaxMessageSize** - Maximum incoming message size. Protects against memory exhaustion. Default unlimited (fine for trusted clusters). -**Flags** - Control capabilities. Remote nodes learn your flags during handshake and can only use features you've enabled. `EnableRemoteSpawn` allows spawning (with explicit permission per process). `EnableImportantDelivery` enables delivery confirmation. +**Flags** - Control capabilities. Remote nodes learn your flags during handshake and can only use features you've enabled. `EnableRemoteSpawn` allows spawning (with explicit permission per process). `EnableImportantDelivery` enables delivery confirmation. `EnableFragmentation` enables message fragmentation for large messages (both sides must enable). `EnableSoftwareKeepAlive` sets the keepalive period in seconds (see [Software Keepalive](#software-keepalive)). **Acceptors** - Define listeners for incoming connections. Multiple acceptors on different ports are supported. Each can have its own cookie, TLS, and protocol. ## Custom Network Stacks -The framework provides three extension points: +The framework provides four extension points: **gen.NetworkHandshake** - Control connection establishment and authentication. Implement this to change how nodes authenticate or how connection pools are created. @@ -190,6 +256,8 @@ The framework provides three extension points: **gen.Connection** - The actual connection handling. Implement this for custom framing, routing, or error handling. +**gen.TypeRegistry** - Optional capability that proto implementations may declare to expose a wire-format type registry. The default ENP/EDF stack implements it. The Erlang distribution proto does not, since the Erlang external term format is schemaless on the wire. When a node has multiple protos configured, `node.Network().RegisterType` distributes registration to every TypeRegistry-capable proto strictly: any per-proto failure fails the call. Protos that do not implement TypeRegistry are skipped silently. + You can register multiple handshakes and protos, allowing one node to support multiple protocol stacks simultaneously: ```go diff --git a/docs/networking/network-transparency.md b/docs/networking/network-transparency.md index bdcd61faa..979dd36cc 100644 --- a/docs/networking/network-transparency.md +++ b/docs/networking/network-transparency.md @@ -156,8 +156,11 @@ type Order struct { Items []string } -func init() { - edf.RegisterTypeOf(Order{}) // Analyzed once, functions built +func (a *MyApp) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { + if err := node.Network().RegisterType(Order{}); err != nil { + return gen.ApplicationSpec{}, err + } + return gen.ApplicationSpec{ /* ... */ }, nil } // Later, during message sending: @@ -166,7 +169,7 @@ process.Send(to, Order{ID: 42, Items: []string{"item1"}}) // Uses pre-built enc This approach delivers Protocol Buffers-class performance without `.proto` files or `protoc` code generation. -Registration happens at runtime - no build step, no generated files. You call `edf.RegisterTypeOf()` in your `init()` function, and EDF builds the optimized encoders. Framework types like `gen.PID`, `gen.Ref`, and `gen.Event` have native support with specialized encodings. During node handshake, both sides exchange their registered type lists and negotiate short numeric IDs, turning a full type name into 3 bytes on the wire. Field names aren't encoded - only field values in declaration order. +Registration happens at runtime - no build step, no generated files. You call `node.Network().RegisterType()` from your application's `Load()` callback, and the framework builds the optimized encoders. Framework types like `gen.PID`, `gen.Ref`, and `gen.Event` have native support with specialized encodings. During node handshake, both sides exchange their registered type lists and negotiate short numeric IDs, turning a full type name into 3 bytes on the wire. Field names aren't encoded - only field values in declaration order. Performance benchmarks (see `benchmarks/serial/`) show encoding is 50-100% faster than Protocol Buffers, while decoding is 20-60% slower. The encoding advantage comes from the specialized functions built during registration. @@ -201,9 +204,9 @@ These limits are enforced during encoding. If you attempt to encode a 70,000 byt ## Type Registration Requirements -For custom types to cross the network, both sending and receiving nodes must register them. Registration tells EDF how to encode and decode the type, and creates a numeric ID that's shared during handshake for efficient encoding. +For custom types to cross the network, both sending and receiving nodes must register them. Registration tells the active wire-format proto how to encode and decode the type, and creates a numeric ID that's shared during handshake for efficient encoding. -Register types during initialization: +Register types from your application's `Load()` callback: ```go type Order struct { @@ -211,11 +214,34 @@ type Order struct { Items []string } -func init() { - edf.RegisterTypeOf(Order{}) +func (a *MyApp) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { + if err := node.Network().RegisterType(Order{}); err != nil { + return gen.ApplicationSpec{}, err + } + return gen.ApplicationSpec{ /* ... */ }, nil } ``` +`Network().RegisterType` distributes registration across every active wire-format proto (e.g., the default ENP/EDF stack). If your node has multiple wire-format protocols configured (for example, a legacy ENP and a newer one running side by side), one call registers in all of them. The call fails if any proto rejects the type. Wire-format consistency is enforced strictly to prevent silent split-brain registries. + +For batch registration of multiple types, use `RegisterTypes` (see the **Nested types** subsection below for the dependency-resolution behavior): + +```go +func (a *MyApp) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { + err := node.Network().RegisterTypes([]any{ + Order{}, + Customer{}, + Address{}, + }) + if err != nil { + return gen.ApplicationSpec{}, err + } + return gen.ApplicationSpec{ /* ... */ }, nil +} +``` + +`RegisterTypes` resolves inter-type dependencies internally. You can list types in any order, and the framework figures out the correct registration sequence. + ### Registration Requirements **Only exported fields** - Structs must have all fields exported (starting with uppercase). This is by design: exported fields define your actor's contract. When actors communicate - locally or across the network - they exchange messages according to explicit contracts. Unexported fields are implementation details, internal state that shouldn't cross actor boundaries. If registration encounters unexported fields, it fails with `"struct Order has unexported field(s)"`. @@ -227,18 +253,21 @@ type Order struct { } ``` -**No pointer types** - EDF rejects pointer types and structs containing pointer fields. This is by design: pointers are a local memory optimization and shouldn't be part of network contracts. A `*Database` field is meaningless to a remote actor - it can't dereference your memory address. Pointers express local sharing semantics that don't translate across address spaces. +**Pointer types** - Starting from version 3.3, EDF supports pointer types. Pointers can be `nil` or point to a value, and this state is preserved during encoding/decoding. Nested pointers (`**int`) are not supported. ```go +var discount *float64 // nil or value +var prices []*int // slice with nil elements +var cache map[string]*Config // map with nil values + type Order struct { - ID int64 - Cache *OrderCache // Registration fails - pointer is local optimization + Priority *int // optional field } ``` -For distributed references, use framework types designed for remote access: `gen.PID` (process reference), `gen.Alias` (named reference), `gen.Ref` (call reference). These work across nodes and provide location-independent semantics. +Note that pointers to external resources like `*Database` or `*Connection` are meaningless to a remote actor - it cannot dereference your memory address. Use pointers for optional value semantics, not for sharing local resources. For distributed references, use framework types: `gen.PID`, `gen.Alias`, `gen.Ref`. -**Nested types must be registered first** - If your type contains other custom types, register the inner types before the outer type: +**Nested types** - If your type contains other custom types, the inner types must be registered before the outer type. Use `RegisterTypes` (batch) which resolves dependency order automatically: ```go type Address struct { @@ -251,13 +280,18 @@ type Person struct { Address Address } -func init() { - edf.RegisterTypeOf(Address{}) // register child first - edf.RegisterTypeOf(Person{}) // then parent +func (a *MyApp) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { + // Order in the slice doesn't matter. The framework registers + // inner types first and retries until everything resolves. + err := node.Network().RegisterTypes([]any{Person{}, Address{}}) + if err != nil { + return gen.ApplicationSpec{}, err + } + return gen.ApplicationSpec{ /* ... */ }, nil } ``` -The order matters because registration builds the encoding schema by examining fields. When registering `Person`, EDF sees the `Address` field. If `Address` isn't registered yet, registration fails with `"type Address must be registered first"`. If `Address` is already registered, EDF references its schema, creating an efficient nested encoding. +If you call `RegisterType` (singular) on `Person` before `Address`, registration fails with `"type Address must be registered first"`. With `RegisterTypes`, the framework iteratively retries pending types whose dependencies become available. Only types that genuinely cannot be resolved produce an error. Registration builds the encoding schema by examining fields; once `Address` is registered, registering `Person` references its schema for efficient nested encoding. ### Custom Marshaling for Special Cases @@ -317,9 +351,14 @@ var ( ErrOutOfStock = errors.New("out of stock") ) -func init() { - edf.RegisterError(ErrInvalidOrder) - edf.RegisterError(ErrOutOfStock) +func (a *MyApp) Load(node gen.Node, args ...any) (gen.ApplicationSpec, error) { + if err := node.Network().RegisterError(ErrInvalidOrder); err != nil { + return gen.ApplicationSpec{}, err + } + if err := node.Network().RegisterError(ErrOutOfStock); err != nil { + return gen.ApplicationSpec{}, err + } + return gen.ApplicationSpec{ /* ... */ }, nil } ``` @@ -333,17 +372,32 @@ Type registration must happen before connection establishment. During handshake, If you register a type after a connection is established, that type isn't in the dictionary. Attempting to send a value of that type fails - the encoder can't find it in the shared schema. The only way to use the newly registered type is to disconnect and reconnect, forcing a new handshake that includes the type. -This is why registration typically happens in `init()` functions. The registration runs before `main()`, which runs before node startup, which runs before any connections are established. By the time connections form, all types are registered. +The recommended place to register types is the application's `Load(node)` callback. Applications are loaded after the network stack is initialized but before any outgoing or incoming traffic, so all types end up in the handshake dictionaries. An application owns its message types and registers them itself, keeping registration co-located with the code that defines the types. For dynamic type registration (registering types based on runtime configuration or plugin loading), you have limited options: -**Register before node start** - Load your configuration, determine which types you need, register them all, then start the node. This works but requires knowing all types upfront. +**Register before any traffic** - Load your configuration, determine which types you need, register them in your application's `Load()` callback. This works but requires knowing all types upfront for the application. -**Coordinate reconnection** - Register the new type, disconnect existing connections to nodes that need the type, wait for reconnection with new handshake. This is complex and causes temporary communication loss. +**Coordinate reconnection** - Register the new type via `node.Network().RegisterType`, disconnect existing connections to nodes that need the type, wait for reconnection with new handshake. This is complex and causes temporary communication loss. **Use custom marshaling** - Implement `edf.Marshaler`/`Unmarshaler` or `encoding.BinaryMarshaler`/`Unmarshaler`. These don't require pre-registration - they work immediately. The tradeoff is you write the encoding logic yourself. -Most applications register types statically in `init()` and avoid these complications. +Most applications register types statically from `Load()` and avoid these complications. + +## Legacy Registration API + +Earlier versions of the framework exposed registration as package-level functions on `ergo.services/ergo/net/edf`: + +```go +// Deprecated. Use node.Network().RegisterType / RegisterError / RegisterAtom instead. +edf.RegisterTypeOf(Order{}) +edf.RegisterError(ErrInvalidOrder) +edf.RegisterAtom("my_atom") +``` + +These functions remain for backward compatibility but are **deprecated**. They write directly into the EDF package state, bypassing the `gen.Network` abstraction. In a multi-proto setup (more than one wire-format proto registered on the node), they only register in EDF, and other protos won't see the type. The new `Network` API distributes registration to every active wire-format proto strictly. + +Prefer `node.Network().RegisterType` / `RegisterTypes` / `RegisterError` / `RegisterAtom` from your application's `Load()` callback. The legacy package-level functions emit a one-time deprecation warning when called from user code. ## Compression @@ -395,6 +449,8 @@ The caches are bidirectional - both nodes maintain the same mappings. During enc This caching is automatic. You don't manage the cache or invalidate entries. The framework handles it. You just benefit from smaller messages. +To measure how much each registered type actually contributes to network traffic and to identify candidates for compression, build the node with `-tags=typestats`. This enables per-type encode/decode counters and wire-byte totals exposed via `Network().RegisteredTypes()` and visible in the Observer Types panel. Counters increment only on root operations (a type sent or received as a message in its own right); bytes embedded inside other messages are accounted to the parent type. The cost is approximately 2-3% on encode/decode throughput; without the tag there is zero overhead. See [The typestats Tag](../advanced/debugging.md#the-typestats-tag) for details. + ## Important Delivery Network transparency breaks down when dealing with failures. Sending to a local process that doesn't exist returns an error immediately - the framework checks the process table and sees the PID isn't registered. Sending to a remote process that doesn't exist returns... nothing. The message is encoded, sent to the remote node, and the remote node silently drops it because there's no recipient. Your code doesn't know the process was missing. @@ -427,6 +483,71 @@ The cost is latency. Normal `Send` returns immediately - it queues the message a For detailed exploration of Important Delivery patterns, reliability guarantees, and protocols like RR-2PC and FR-2PC, see [Important Delivery](../advanced/important-delivery.md). +## Message Ordering + +Messages sent from process A to process B arrive in sending order. This is a per-sender FIFO guarantee; it applies to each sender independently, not globally across all senders. The guarantee is enabled by default for every process. + +### KeepNetworkOrder Flag + +Message ordering is controlled by a per-process flag called `KeepNetworkOrder`, which defaults to `true`. You can change it using `SetKeepNetworkOrder(bool)` during `Init` or at any point while the process is running. The flag applies to all outgoing messages from that process: `Send`, `Call`, `SendResponse`, and `SendEvent`. There is no per-message override; ordering is all-or-nothing for a given sender. + +### How It Works: Sender Side + +With ordering enabled, all messages from a process go through the same TCP link in the connection pool. The link is selected deterministically: `sender.ID % 255 % pool_size`. Since TCP guarantees FIFO delivery within a single connection, messages arrive at the remote node in exactly the order they were sent. + +With ordering disabled, messages are distributed round-robin across all pool links. This spreads the load for maximum throughput, but the arrival order across different TCP connections is no longer deterministic. + +### How It Works: Receiver Side + +Each message carries an **order byte** in the protocol header (byte 6 of the ENP frame). When ordering is enabled, the order byte is derived from the recipient's identity: +- For `gen.PID` recipients: `to.ID % 255` +- For `gen.Alias` recipients: `to.ID[1] % 255` + +The receiving node routes messages to receive queues based on this byte: `order_byte % queue_count`. Messages destined for the same recipient land in the same queue and are decoded sequentially, preserving order. + +When ordering is disabled, the order byte is zero. Messages distribute round-robin across receive queues, enabling parallel decoding at the cost of non-deterministic arrival order. + +### Two-Level Guarantee + +The ordering mechanism works at two levels: + +1. **Sender side:** pins messages to one TCP link, preserving send order in the TCP stream +2. **Receiver side:** pins messages to one decode queue, preserving decode order + +Together they ensure end-to-end FIFO from sender to recipient. The sender side prevents reordering during transmission; the receiver side prevents reordering during decoding and dispatch. + +### Special Cases + +Some system messages have fixed ordering semantics regardless of the `KeepNetworkOrder` flag: + +| Operation | Ordering | Notes | +|-----------------|---------------|---------------------------------------------| +| `SendExit` | Always ordered| No `KeepNetworkOrder` check, always uses sender-derived order byte | +| `SendTerminate` | Always unordered | Order byte is always 0 | +| Link/Monitor | Always ordered| System operations that must arrive in sequence | + +These are internal system messages where ordering behavior is fixed by the protocol, not configurable by the process. + +### When to Disable Ordering + +Processes that don't need ordering benefit from disabling it. When `KeepNetworkOrder` is `false`, messages spread across all TCP links in the pool and all receive queues on the remote side. This increases parallelism on both ends: more connections are utilized for sending, and more goroutines participate in decoding. + +Good candidates for disabling ordering: +- **Stateless workers** that process each request independently +- **Fan-out producers** that distribute work to many recipients +- **High-throughput event emitters** where each event is self-contained + +The tradeoff is straightforward: message arrival order becomes non-deterministic. If your process logic doesn't depend on message order, disabling ordering gives you better throughput. + +```go +func (w *Worker) Init(args ...any) error { + // This worker processes requests independently, + // ordering doesn't matter + w.SetKeepNetworkOrder(false) + return nil +} +``` + ## Protocol Frame Structure EDF-encoded messages are wrapped in ENP (Ergo Network Protocol) frames for transmission over TCP. @@ -445,9 +566,7 @@ For PID messages, the frame contains: - Recipient PID (8 bytes) - EDF-encoded message payload -The **order byte** (byte 6) preserves message ordering per sender. It's calculated as `senderPID.ID % 255`, ensuring messages from the same sender have the same order value. This guarantees sequential processing on the receiving side even if messages arrive on different TCP connections in the pool. Messages from different senders have different order values, enabling parallel processing. - -When the receiving node reads a frame from TCP, it extracts the order byte and routes the frame to the appropriate receive queue. The connection creates **4 receive queues per TCP connection** in the pool. So a 3-connection pool has 12 receive queues total. Frames are distributed to queues based on `order_byte % queue_count`. Each queue is processed by a dedicated goroutine that decodes frames and delivers messages to recipients. This parallel processing improves throughput while preserving per-sender ordering. +The **order byte** (byte 6) controls message ordering and receive queue routing. For details on how the order byte is calculated and how it interacts with the connection pool and receive queues, see [Message Ordering](#message-ordering) above. ## Limits of Transparency @@ -461,7 +580,7 @@ Network transparency is powerful but not magical. The network has physical prope **Partial failures** - In a distributed system, some nodes can fail while others continue working. A local system either works entirely or crashes entirely. A distributed system can be partially operational - some nodes reachable, others not. This partial failure is the hardest aspect of distributed systems. The framework can't hide it entirely. -**Ordering** - Message ordering is preserved per-sender within a connection. Messages from process A to process B arrive in the order sent. But messages from different senders can interleave arbitrarily. And if a connection drops and reconnects, messages sent during disconnection are lost or delayed. Don't assume global ordering across the cluster. +**Ordering** - Message ordering is preserved per-sender, not globally. Messages from process A to process B arrive in sending order, but messages from different senders can interleave arbitrarily. If a connection drops and reconnects, messages sent during disconnection are lost or delayed. Don't assume global ordering across the cluster. See [Message Ordering](#message-ordering) for how the ordering mechanism works and when to disable it. Network transparency makes distributed programming feel local. But distributed programming has fundamental differences from local programming. The transparency is a tool that simplifies common cases - it doesn't eliminate the need to think about distributed system challenges. @@ -481,6 +600,6 @@ Understanding network transparency helps you design better distributed systems. **Leverage compression** - Enable compression for processes that send large messages. The CPU cost of compression is usually worth the network bandwidth savings. But don't compress tiny messages - the overhead exceeds the benefit. -**Register types early** - Do all type registration in `init()` functions before the node starts. Avoid dynamic type registration that requires connection cycling. Static registration is simpler and more reliable. +**Register types early** - Do all type registration from your application's `Load(node)` callback so types are in the registry before any traffic. Avoid dynamic type registration that requires connection cycling. Static registration is simpler and more reliable. For details on how the network stack implements transparency, see [Network Stack](network-stack.md). For understanding how nodes discover each other, see [Service Discovery](service-discovering.md). diff --git a/docs/networking/remote-spawn-process.md b/docs/networking/remote-spawn-process.md index fbdc6a199..aeb6087b3 100644 --- a/docs/networking/remote-spawn-process.md +++ b/docs/networking/remote-spawn-process.md @@ -222,7 +222,7 @@ node, err := ergo.StartNode("myapp@localhost", gen.NodeOptions{ Now when you use `process.RemoteSpawn`, the remote process receives a copy of the calling process's environment. The remote node reads these values and sets them on the spawned process. -**Important:** Environment variable values must be EDF-serializable. Strings, numbers, booleans work fine. Custom types require registration via `edf.RegisterTypeOf`. If an environment variable contains a non-serializable value (e.g., a channel, function, or unregistered struct), the remote spawn fails entirely with an error like `"no encoder for type "`. The framework doesn't skip problematic variables - any non-serializable value causes the entire spawn request to fail. +**Important:** Environment variable values must be EDF-serializable. Strings, numbers, booleans work fine. Custom types require registration via `node.Network().RegisterType` (see [Network Transparency](network-transparency.md) for details on the type registry; the legacy `edf.RegisterTypeOf` still works but is deprecated). If an environment variable contains a non-serializable value (e.g., a channel, function, or unregistered struct), the remote spawn fails entirely with an error like `"no encoder for type "`. The framework doesn't skip problematic variables: any non-serializable value causes the entire spawn request to fail. Environment inheritance only works with `process.RemoteSpawn`. Using `RemoteNode.Spawn` doesn't inherit environment because there's no calling process - it's a node-level operation. diff --git a/docs/networking/remote-start-application.md b/docs/networking/remote-start-application.md index ebeedd84a..1adbccf7a 100644 --- a/docs/networking/remote-start-application.md +++ b/docs/networking/remote-start-application.md @@ -190,7 +190,7 @@ node, err := ergo.StartNode("scheduler@localhost", gen.NodeOptions{ Now when you start an application remotely, the application's processes receive a copy of the requesting node's core environment. This enables configuration propagation - your scheduler node has configuration in its environment, and applications started remotely inherit it. -**Important:** Environment variable values must be EDF-serializable. Strings, numbers, booleans work fine. Custom types require registration via `edf.RegisterTypeOf`. If an environment variable contains a non-serializable value (e.g., a channel, function, or unregistered struct), the remote application start fails entirely with an error like `"no encoder for type "`. The framework doesn't skip problematic variables - any non-serializable value causes the entire start request to fail. +**Important:** Environment variable values must be EDF-serializable. Strings, numbers, booleans work fine. Custom types require registration via `node.Network().RegisterType` (see [Network Transparency](network-transparency.md) for details on the type registry; the legacy `edf.RegisterTypeOf` still works but is deprecated). If an environment variable contains a non-serializable value (e.g., a channel, function, or unregistered struct), the remote application start fails entirely with an error like `"no encoder for type "`. The framework doesn't skip problematic variables: any non-serializable value causes the entire start request to fail. ## How It Works diff --git a/docs/networking/service-discovering.md b/docs/networking/service-discovering.md index 25ca81078..b0f5d829c 100644 --- a/docs/networking/service-discovering.md +++ b/docs/networking/service-discovering.md @@ -250,8 +250,8 @@ if err != nil { process.LinkEvent(event) // In your HandleEvent callback (etcd example): -func (w *Worker) HandleEvent(message gen.MessageEvent) error { - switch ev := message.Message.(type) { +func (w *Worker) HandleEvent(event gen.MessageEvent) error { + switch ev := event.Message.(type) { case etcd.EventConfigUpdate: // Configuration item changed diff --git a/docs/tools/ergo.md b/docs/tools/ergo.md index 411c46ffb..81cddc45d 100644 --- a/docs/tools/ergo.md +++ b/docs/tools/ergo.md @@ -1,127 +1,384 @@ -# Boilerplate Code Generation - -The `ergo` tool allows you to generate the structure and source code for a project based on the Ergo Framework. To install it, use the following command: - -`go install ergo.tools/ergo@latest` - -Alternatively, you can build it from the source code available at [https://github.com/ergo-services/tools](https://github.com/ergo-services/tools). - -When using `ergo` tool, you need to follow the specific template for providing arguments: - -`Parent:Actor{param1:value1,param2:value2...}` - -* **Parent** can be a _supervisor_ (specified earlier with `-with-sup`) or an _application_ (specified earlier with `-with-app`). -* **Actor** can be an _actor_ (added earlier with `-with-actor`) or a _supervisor_ (specified earlier with `-with-sup`). - -This structured approach ensures the proper hierarchy and parameters are defined for your _actors_ and _supervisors_ - -### Available Arguments and Parameters : - -* **`-init `**: a required argument that sets the name of the node for your service. Available parameters: - * **`tls`**: enables encryption for network connections (a self-signed certificate will be used). - * **`module`**: allows you to specify the module name for the `go.mod` file. -* **`-path `**: specifies the path for the code of the generated project. -* **`-with-actor `**: adds an actor (based on `act.Actor`). -* **`-with-app `**: adds an application. Available parameters: - * **`mode`**: specifies the application's [start mode](../basics/application.md#application-startup-modes) (`temp` - Temporary, `perm` - Permanent, `trans` - Transient). The default mode is `trans`.\ - Example: `-with-app MyApp{mode:perm}` -* **`-with-sup `**: adds a supervisor (based on `act.Supervisor`). Available parameters: - * **`type`**: specifies the [type of supervisor](../actors/supervisor.md#supervisor-types) (`ofo` - One For One, `sofo` - Simple One For One, `afo` - All For One, `rfo` - Rest For One). The default type is `ofo`. - * **`strategy`**: specifies the [restart strategy](../actors/supervisor.md#restart-strategy) for the supervisor (`temp` - Temporary, `perm` - Permanent, `trans` - Transient). The default strategy is `trans`. -* **`-with-pool `**: adds a process pool actor (based on `act.Pool`). Available parameters: - * **`size`**: Specifies the number of worker processes in the pool. By default, 3 processes are started. -* **`-with-web `**: adds a Web server (based on `act.Pool` and `act.WebHandler`). Available parameters: - * **`host`**: specifies the hostname for the Web server. - * **`port`**: specifies the port number for the Web server. The default is `9090`. - * **`tls`**: enables encryption for the Web server using the node's `CertManager`. -* **`-with-tcp `**: adds a TCP server actor (based on `act.Actor` and `meta.TCPServer` meta-process). Available parameters: - * **`host`**: specifies the hostname for the TCP server. - * **`port`**: specifies the port number for the TCP server. The default is `7654`. - * **`tls`**: enables encryption for the TCP server using the node's `CertManager`. -* **`-with-udp `**: adds a UDP server actor (based on `act.Pool` , `meta.UDPServer` and `act.Actor` as worker processes). Available parameters: - * **`host`**: specifies the hostname for the UDP server. - * **`port`**: specifies the port number for the UDP server. The default is `7654`. -* **`-with-msg `**: adds a message type for network interactions. -* **`-with-logger `**: adds a logger from the extended library. Available loggers: [colored](../extra-library/loggers/colored.md), [rotate](../extra-library/loggers/rotate.md) -* **`-with-observer`**: adds the [Observer application](../extra-library/applications/observer.md). - -### Example - -For clarity, let's use all available arguments for `ergo` in the following example: - -
$ ergo -path /tmp/project \
-      -init demo{tls} \
-      -with-app MyApp \
-      -with-actor MyApp:MyActorInApp \
-      -with-sup MyApp:MySup \
-      -with-actor MySup:MyActorInSup \
-      -with-tcp "MySup:MyTCP{port:12345,tls}" \
-      -with-udp MySup:MyUDP{port:54321} \
-      -with-pool MySup:MyPool{size:4} \
-      -with-web "MyWeb{port:8888,tls}" \
-      -with-msg MyMsg1 \
-      -with-msg MyMsg2 \
-      -with-logger colored \
-      -with-logger rotate \
-      -with-observer
-      
-Generating project "/tmp/project/demo"...
-   generating "/tmp/project/demo/apps/myapp/myactorinapp.go"
-   generating "/tmp/project/demo/apps/myapp/myactorinsup.go"
-   generating "/tmp/project/demo/cmd/myweb.go"
-   generating "/tmp/project/demo/cmd/myweb_worker.go"
-   generating "/tmp/project/demo/apps/myapp/mytcp.go"
-   generating "/tmp/project/demo/apps/myapp/myudp.go"
-   generating "/tmp/project/demo/apps/myapp/myudp_worker.go"
-   generating "/tmp/project/demo/apps/myapp/mypool.go"
-   generating "/tmp/project/demo/apps/myapp/mypool_worker.go"
-   generating "/tmp/project/demo/apps/myapp/mysup.go"
-   generating "/tmp/project/demo/apps/myapp/myapp.go"
-   generating "/tmp/project/demo/types.go"
-   generating "/tmp/project/demo/cmd/demo.go"
-   generating "/tmp/project/demo/README.md"
-   generating "/tmp/project/demo/go.mod"
-   generating "/tmp/project/demo/go.sum"
-
-Successfully completed.
-
- -Pay attention to the values of the `-with-tcp` and `-with-web` arguments — they are enclosed in double quotes. If an argument has multiple parameters, they are separated by commas without spaces. However, since commas are argument delimiters for the shell interpreter, we enclose the entire value of the argument in double quotes to ensure the shell correctly processes the parameters. - -In our example, we specified two loggers: `colored` and `rotate`. This allows for colored log messages in the standard output as well as logging to files with log rotation functionality. In this case, the default logger is disabled to prevent duplicate log messages from appearing on the standard output. - -Additionally, we included the `observer` application. By default, this interface is accessible at `http://localhost:9911`. - -As a result of the generation process, we have a well-structured project source code that is ready for execution: - -``` - demo -├── apps -│ └── myapp -│ ├── myactorinapp.go -│ ├── myactorinsup.go -│ ├── myapp.go -│ ├── mypool.go -│ ├── mypool_worker.go -│ ├── mysup.go -│ ├── mytcp.go -│ ├── myudp.go -│ └── myudp_worker.go -├── cmd -│ ├── demo.go -│ ├── myweb.go -│ └── myweb_worker.go -├── go.mod -├── go.sum -├── README.md -└── types.go -``` - -The generated code is ready for compilation and execution: - -
- -Since this example includes the [observer application](../extra-library/applications/observer.md), you can open `http://localhost:9911` in your browser to access the web interface for [inspecting the node](observer.md) and its running processes. +--- +description: Boilerplate Generator for Ergo Framework Projects +--- +# ergo +The `ergo` tool generates the initial structure and source code for Ergo Framework +projects. Instead of writing actor definitions, supervisor specs, and application +boilerplate by hand, you describe what you want and the tool writes it for you. +The generated code is plain Go that you own and can modify freely. The tool +understands which parts are structural wiring (regenerated as the project grows) +and which parts are your business logic (never touched again). + +## Installation + +Requires Go 1.21 or higher. + +``` +go install ergo.tools/ergo@latest +``` + +## Quick Start + +Three commands to get a running Ergo node: + +```bash +ergo init MyNode github.com/myorg/mynode +cd mynode +go run ./cmd +``` + +The node starts immediately with an application, a supervisor and an actor, +all wired together. + +## How It Works + +The tool maintains `ergo.yaml` in the project root. This file describes the +supervision tree. Every `ergo add` command updates this file and regenerates +the affected code. + +Each component produces two files: + +| File | Owned by | Regenerated | Contains | +|------|----------|-------------|---------| +| `name_gen.go` | tool | on every `ergo add` or `ergo generate` | factory, Init spec, Load group | +| `name.go` | you | never | Tune, handlers, Start, Terminate | + +User-owned files provide hooks that the generated code calls. The pattern is +consistent across all component types: + +| File | Hook | Purpose | +|------|------|---------| +| `mysup.go` | `Tune(spec, args) (SupervisorSpec, error)` | adjust supervisor spec before start | +| `myapp.go` | `Tune(node, spec, args) (ApplicationSpec, error)` | adjust application spec before start | +| `messages.go` | `extraMessages() []any` | register custom EDF message types | +| `cmd/main.go` | `extraApps() []ApplicationBehavior` | add external applications | + +## Commands + +### ergo init + +``` +ergo init +``` + +Creates a new project. The directory name is derived from the last segment of +the module path. Generates `ergo.yaml`, all boilerplate, `go.mod`, and runs +`go mod tidy`. + +```bash +ergo init MyNode github.com/myorg/mynode +ergo init Gateway github.com/acme/api-gateway +``` + +The default project has one application, one supervisor and one actor, enough +to verify everything works before adding real components. + +### ergo add actor + +``` +ergo add actor [--pool] <[Parent:]Name> +``` + +Adds an actor. `Parent` is the name of an existing supervisor or application. +Without a parent the actor is added to `node.processes` and spawned directly +by the node at startup. + +`--pool` generates a pool actor with a companion worker type. A pool distributes +incoming messages across a fixed set of workers and restarts them on failure. + +```bash +ergo add actor MySup:MyActor +ergo add actor --pool MySup:RequestPool +ergo add actor StandaloneActor +``` + +### ergo add supervisor + +``` +ergo add supervisor [--type ] [--strategy ] <[Parent:]Name> +``` + +Adds a supervisor. `Parent` is an existing application or supervisor. + +`--type` controls which children are restarted when one fails: + +| Type | Behavior | +|------|---------| +| `one_for_one` (default) | only the failed child | +| `all_for_one` | all children | +| `rest_for_one` | the failed child and all children started after it | +| `simple_one_for_one` | children spawned dynamically at runtime via `AddChild`, no static children list | + +`--strategy` controls when a child is restarted: + +| Strategy | Behavior | +|----------|---------| +| `transient` (default) | only on abnormal exit | +| `permanent` | always | +| `temporary` | never | + +```bash +ergo add supervisor MyApp:WorkerSup +ergo add supervisor MyApp:CriticalSup --type all_for_one --strategy permanent +ergo add supervisor WorkerSup:SubSup --type rest_for_one +``` + +### ergo add app + +``` +ergo add app [--mode ] +``` + +Adds an application. `--mode` declares what happens when the application stops: + +| Mode | Behavior | +|------|---------| +| `transient` (default) | node stops on abnormal exit | +| `permanent` | node always stops when app exits | +| `temporary` | node ignores the exit | + +```bash +ergo add app MyApp +ergo add app BackgroundApp --mode temporary +ergo add app CriticalApp --mode permanent +``` + +### ergo add message + +``` +ergo add message --field name:type [--field name:type ...] +``` + +Adds an EDF message type. Field types can be standard Go types (`string`, `int`, +`bool`, `[]byte`) or framework types (`gen.Alias`, `gen.PID`, `gen.Ref`). + +Generated struct definitions and EDF registration go into `messages_gen.go`, +which is always regenerated when the message list changes. + +```bash +ergo add message MessageConnect --field ID:gen.Alias --field Addr:string +ergo add message MessageData --field ID:gen.Alias --field Payload:"[]byte" +``` + +If a message type has fields of other custom types, add the inner type first. +EDF requires nested types to be registered before the types that reference them: + +```bash +ergo add message MessageAddress --field City:string --field Street:string +ergo add message MessageUser --field Name:string --field Address:MessageAddress +``` + +Both nodes must register the same types with identical field definitions. The +registration order between nodes does not need to match; nodes negotiate numeric +type IDs during handshake. + +For detailed coverage of EDF, type constraints, and custom marshaling, see +[Network Transparency](../networking/network-transparency.md#edf-ergo-data-format). + +### ergo generate + +``` +ergo generate [ergo.yaml] +``` + +Regenerates all `*_gen.go` files from `ergo.yaml`. Your `.go` files are never +overwritten. Searches for `ergo.yaml` in the current directory and its parents. + +```bash +ergo generate +ergo generate /path/to/ergo.yaml +``` + +## Project Structure + +After `ergo init MyNode github.com/myorg/mynode`: + +``` +mynode/ + ergo.yaml project definition + go.mod + go.sum + messages_gen.go EDF struct definitions + registration (generated) + messages.go extraMessages() hook for custom types (yours) + apps/ + mynodeapp/ + mynodeapp_gen.go CreateApp, Load with Group (generated) + mynodeapp.go Tune, Start, Terminate (yours) + mynodesup_gen.go factory, Init with SupervisorSpec (generated) + mynodesup.go Tune, HandleMessage (yours) + mynodeactor_gen.go factory (generated) + mynodeactor.go Init, HandleMessage, HandleCall (yours) + cmd/ + main_gen.go node startup, application list (generated) + main.go extraApps() hook (yours) + README.md +``` + +The `README.md` is regenerated on every `ergo add` or `ergo generate` and shows +the current supervision tree. + +## ergo.yaml Reference + +```yaml +node: + name: MyNode + module: github.com/myorg/mynode + host: localhost + + network: + tls: false + cookie: "" # empty means auto-generated on every start + + loggers: # colored, rotate + - colored + + apps: + + # User-defined application + - name: MyApp + mode: transient + children: + - sup: MySup + type: one_for_one + strategy: transient + intensity: 2 # max restarts within period + period: 5 # seconds + children: + - actor: MyActor + - actor: MyPool + pool: true + + # Known applications from the ergo.services ecosystem + - observer + - mcp + - radar + + processes: # spawned directly by node, no application + - actor: StandaloneActor + + messages: + - name: MessageConnect + fields: + - ID: gen.Alias + - Addr: string +``` + +Known loggers: `colored` ([docs](../extra-library/loggers/colored.md)), +`rotate` ([docs](../extra-library/loggers/rotate.md)). + +Known applications: `observer` ([docs](../extra-library/applications/observer.md)), +`mcp` ([docs](../extra-library/applications/mcp.md)), +`radar` ([docs](../extra-library/applications/radar.md)). + +## Customizing Generated Code + +### Supervisor + +`mynodesup.go` contains `Tune`, called from the generated `Init`. The generated +`Init` builds `SupervisorSpec` from `ergo.yaml` and passes it to `Tune`. Override +restart parameters or add dynamic children here: + +```go +func (sup *MySup) Tune(spec act.SupervisorSpec, args ...any) (act.SupervisorSpec, error) { + spec.Restart.Intensity = 10 + spec.Restart.Period = 30 + return spec, nil +} +``` + +Do not replace `spec.Children` in `Tune` unless you have a specific reason. +The children list is populated from `ergo.yaml` by the generated `Init`. + +### Application + +`mynodeapp.go` contains `Tune`, called from the generated `Load`. The `Group` +in `Load` is populated from `ergo.yaml`. Use `Tune` to set metadata, environment +variables or dependencies: + +```go +func (app *MyApp) Tune(node gen.Node, spec gen.ApplicationSpec, args ...any) (gen.ApplicationSpec, error) { + spec.Description = "main application" + spec.Version = gen.Version{Release: "1.0.0"} + spec.Env = map[gen.Env]any{ + "DB_HOST": "localhost", + "DB_PORT": 5432, + } + spec.Depends.Applications = []gen.Atom{"config"} + return spec, nil +} +``` + +### Custom EDF Message Types + +`messages.go` contains `extraMessages()`, called from the generated `init()`. +Add custom types that are not declared in `ergo.yaml`: + +```go +func extraMessages() []any { + return []any{ + MyCustomMessage{}, + AnotherMessage{}, + } +} +``` + +For types with unexported fields or special encoding needs, implement +`edf.Marshaler`/`Unmarshaler` or `encoding.BinaryMarshaler`/`Unmarshaler` +in a separate file. See [Network Transparency](../networking/network-transparency.md#edf-ergo-data-format). + +### External Applications + +Some applications cannot be described in `ergo.yaml` because their constructor +requires runtime arguments. Add them in `cmd/main.go`, which is never regenerated: + +```go +func extraApps() []gen.ApplicationBehavior { + return []gen.ApplicationBehavior{ + thirdparty.New(thirdparty.Options{ + DSN: os.Getenv("DATABASE_URL"), + Port: 8080, + }), + } +} +``` + +## Typical Workflow + +```bash +# 1. Create the project +ergo init OrderService github.com/acme/orders +cd orders + +# 2. Verify it runs +go run ./cmd + +# 3. Add the supervision tree incrementally +ergo add supervisor MyOrderServiceApp:ApiSup +ergo add actor ApiSup:HttpHandler +ergo add actor --pool ApiSup:RequestPool +ergo add supervisor MyOrderServiceApp:WorkerSup --type all_for_one +ergo add actor WorkerSup:OrderProcessor +ergo add actor WorkerSup:PaymentActor + +# 4. Add network message types +ergo add message MessageOrderCreated --field OrderID:string --field Total:int +ergo add message MessageOrderPaid --field OrderID:string + +# 5. Implement logic in .go files +# 6. Run, observe, iterate +go run ./cmd +``` + +Each `ergo add` updates `ergo.yaml`, regenerates `*_gen.go` files, and leaves +your `.go` files untouched. + +## What's Next + +- [Observer](observer.md): web UI for inspecting running nodes and processes +- [Actors](../actors): actor types, supervision and messaging patterns +- [Applications](../basics/application.md): application lifecycle and modes +- [Pool](../actors/pool.md): distributing work across worker processes +- [Network Transparency](../networking/network-transparency.md): EDF serialization and distributed messaging diff --git a/docs/tools/observer.md b/docs/tools/observer.md deleted file mode 100644 index adc17a86b..000000000 --- a/docs/tools/observer.md +++ /dev/null @@ -1,171 +0,0 @@ -# Inspecting With Observer - -### Installation and starting - -To install the `observer` tool, you need to have Golang compiler version 1.20 or higher. Run the following command: - -``` -$ go install ergo.services/tools/observer@latest -``` - -Available arguments for starting `observer`: - -* **`-help`**: displays information about the available arguments. -* **`-version`**: prints the current version of the Observer tool. -* **`-host`**: specifies the interface name for the Web server to run on (default: `"localhost"`). -* **`-port`**: defines the port number for the Web server (default: `9911`). -* **`-cookie`**: sets the default cookie value used for connecting to other nodes. - -
- -If you are running o`bserver` on a server for continuous operation, it is recommended to use the environment variable `COOKIE` instead of the `-cookie` argument. Using sensitive data in command-line arguments is insecure. - -After starting `observer`, it initially has no connections to other nodes, so you will be prompted to specify the node you want to connect to. - -
- -Once you establish a connection with a remote node, the _Observer application_ main page will open, displaying information about that node. - -If you have integrated the _Observer application_ into your node, upon opening the _Observer_ page, you will immediately land on the main page showing information about the node where the _Observer application_ was launched. - -### Info (main page) - -
- -On this tab, you will find general information about the node and the ability to manage its logging level. Changing the logging level only affects the node itself and any newly started processes, but it does not impact processes that are already running. - -Graphs provide real-time information over the last 60 seconds, including the total number of processes, the number of processes in the running state, and memory usage data. Memory usage is divided into **used**, which indicates how much memory was reserved from the operating system, and **allocated**, which shows how much of that reserved memory is currently being used by the Golang runtime. - -In addition to these details, you can view information about the available loggers on the node and their respective logging levels. For more details, refer to the [Logging](../basics/logging.md) section. Environment variables will also be displayed here, but only if the `ExposeEnvInfo` option was enabled in the `gen.NodeOptions.Security` settings when the inspected node was started. - -### Network (main page) - -
- -The **Network tab** displays information about the node's network stack. - -The **Mode** indicates how the network stack was started (_enabled_, _hidden_, or _disabled_). - -The **Registrar** section shows the properties of the registrar in use, including its capabilities. _Embedded Server_ indicates whether the registrar is running in server mode, while the _Server_ field shows the address and port number of the registrar with which the node is registered. - -Additionally, the tab provides information about the default _handshake_ and _protocol versions_ used for outgoing connections. - -The **Flags** section lists the set of flags that define the functionality available to remote nodes. - -The **Acceptors** section lists the node's acceptors, with detailed information available for each. This list will be empty if the network stack is running in hidden mode. - -Since the node can work with multiple network stacks simultaneously, some acceptors may have different _registrar_ parameters and _handshake_/_protocol_ versions. For an example of simultaneous usage of the Erlang and Ergo Framework network stacks, refer to the [Erlang](../extra-library/network-protocols/erlang.md) section. - -
- -The **Connected Nodes** section displays a list of active connections with remote nodes. For each connection, you can view detailed information, including the version of the handshake used when the connection was established and the protocol currently in use. The **Flags** section shows which features are available to the node when interacting with the remote node. - -
- -Since the ENP protocol supports a pool of TCP connections within a single network connection, you will find information about the **Pool Size** (the number of TCP connections). The **Pool DSN** field will be empty if this is an incoming connection for the node or if the protocol does not support TCP connection pooling. - -Graphs provide a summary of the number of received/sent messages and network traffic over the last 60 seconds, offering a quick overview of communication activity and data flow. - -### Process list (main page) - -
- -On the **Processes List** tab, you can view general information about the processes running on the node. The number of processes displayed is controlled by the **Start from** and **Limit** parameters. - -By default, the list is sorted by the process identifier. However, you can choose different sorting options: - -* **Top Running**: displays processes that have spent the most time in the _running_ state. -* **Top Messaging**: sorts processes by the number of sent/received messages in descending order. -* **Top Mailbox**: helps identify processes with the highest number of messages in their mailbox, which can be an indication that the process is struggling to handle the load efficiently. - -For each process, you can view brief information: - -
- -The **Behavior** field shows the type of object that the process represents. - -**Application** field indicates the application to which the process belongs. This property is inherited from the parent, so all processes started within an application and their child processes will share the same value. - -**Mailbox Messages** displays the total number of messages across all queues in the process's mailbox. - -**Running Time** shows the total time the process has spent in the _running_ state, which occurs when the process is actively handling messages from its queue. - -By clicking on the process identifier, you will be directed to a page with more detailed information about that specific process. - -### Log (main page) - -
- -All log messages from the node, processes, network stack, or meta-processes are displayed here. When you connect to the Observer via a browser, the Observer's backend sends a request to the inspector to start a _log process_ with specified logging levels (this _log process_ is visible on the main **Info** tab). - -When you change the set of logging levels, the Observer's backend requests the start of a new _log process_ (the old _log process_ will automatically terminate). - -To reduce the load on the browser, the number of displayed log messages is limited, but you can adjust this by setting the desired number in the **Last** field. - -The **Play/Pause** button allows you to stop or resume the _log process_, which is useful if you want to halt the flow of log messages and focus on examining the already received logs in more detail. - -### Process information - -
- -This page displays detailed information about the process, including its state, uptime, and other key metrics. - -The **fallback parameters** specify which process will receive redirected messages in case the current process's mailbox becomes full. However, if the **Mailbox Size** is unlimited, these fallback parameters are ignored. - -The **Message Priority** field shows the priority level used for messages sent by this process. - -**Keep Network Order** is a parameter applied only to messages sent over the network. It ensures that all messages sent by this process to a remote process are delivered in the same order as they were sent. This parameter is enabled by default, but it can be disabled in certain cases to improve performance. - -The **Important Delivery** setting indicates whether the important flag is enabled for messages sent to remote nodes. Enabling this option forces the remote node to send an acknowledgment confirming that the message was successfully delivered to the recipient's mailbox. - -The **Compression** parameters allow you to enable message compression for network transmissions and define the compression settings. - -Graphs on this page help you assess the load on the process, displaying data over the last 60 seconds. - -Additionally, you can find detailed information about any **aliases**, **links**, and **monitors** created by this process, as well as any registered **events** and started **meta-processes**. - -The list of environment variables is displayed only if the `ExposeEnvInfo` option was enabled in the node's `gen.NodeOptions.Security` settings. - -
- -Additionally, on this page, you can send a message to the process, send an _exit signal_, or even forcibly stop the process using the _kill_ command. These options are available in the context menu. - -
- -### Inspect (process page) - -
- -If the _behavior_ of this process implements the `HandleInspect` method, the response from the process to the inspect request will be displayed here. The Observer sends these requests once per second while you are on this tab. - -In the example screenshot above, you can see the inspection of a process based on `act.Pool`. Upon receiving the inspect request, it returns information about the pool of processes and metrics such as the number of messages processed. - -### Log (process page) - -
- -The **Log** tab on the process information page displays a list of log messages generated by the specific process. - -Please note that since the Observer uses a single stream for logging, any changes to the logging levels will also affect the content of the **Log** tab on the main page. - -### Meta-process information - -
- -On this page, you'll find detailed information about the meta-process, along with graphs showing data for the last 60 seconds related to incoming/outgoing messages and the number of messages in its mailbox. The meta-process has only two message queues: _main_ and _system_. - -You can also send a message to the meta-process or issue an _exit signal_. However, it is not possible to forcibly stop the meta-process using the _kill_ command. - -
- -### Inspect (meta-process page) - -
- -If the meta-process's _behavior_ implements the `HandleInspect` method, the response from the meta-process to the inspect request will be displayed on this tab. The Observer sends this request once per second while you are on the tab. - -### Log (meta-process page) - -
- -On the **Log** tab of the meta-process, you will see log messages generated by that specific meta-process. Changing the logging levels will also affect the content of the **Log** tab on the main page. diff --git a/gen/default.go b/gen/default.go index 1ff3091f9..21607b353 100644 --- a/gen/default.go +++ b/gen/default.go @@ -25,14 +25,24 @@ var ( DefaultTCPBufferSize int = 65535 DefaultPort uint16 = 11144 + DefaultHandshakeTimeout time.Duration = 5 * time.Second + DefaultSoftwareKeepAliveMisses int = 3 + DefaultFragmentSize int = 65000 + DefaultFragmentTimeout time.Duration = 30 * time.Second + DefaultMaxFragmentAssemblies int = 1000 + DefaultNetworkFlags = NetworkFlags{ Enable: true, EnableRemoteSpawn: true, EnableRemoteApplicationStart: true, - EnableFragmentation: false, + EnableFragmentation: true, EnableProxyTransit: false, EnableProxyAccept: true, EnableImportantDelivery: true, + EnableSimultaneousConnect: true, + EnableClockSkew: true, + EnableTracing: true, + EnableSoftwareKeepAlive: 15, // seconds } DefaultNetworkProxyFlags = NetworkProxyFlags{ diff --git a/gen/mailbox.go b/gen/mailbox.go index 0bf8f47f0..fdfc9fa96 100644 --- a/gen/mailbox.go +++ b/gen/mailbox.go @@ -10,8 +10,10 @@ const ( MailboxMessageTypeRegular MailboxMessageType = 0 MailboxMessageTypeRequest MailboxMessageType = 1 MailboxMessageTypeEvent MailboxMessageType = 2 - MailboxMessageTypeExit MailboxMessageType = 3 - MailboxMessageTypeInspect MailboxMessageType = 4 // for the observer's purposes + MailboxMessageTypeSpan MailboxMessageType = 3 + + MailboxMessageTypeExit MailboxMessageType = 10 + MailboxMessageTypeInspect MailboxMessageType = 11 ) type MailboxMessage struct { @@ -20,6 +22,7 @@ type MailboxMessage struct { Type MailboxMessageType Target any Message any + Tracing Tracing } var ( @@ -42,5 +45,6 @@ func ReleaseMailboxMessage(m *MailboxMessage) { m.Type = 0 m.From = emptyPID m.Ref = emptyRef + m.Tracing = Tracing{} mbm.Put(m) } diff --git a/gen/network.go b/gen/network.go index 3d665191b..7c53ac6c6 100644 --- a/gen/network.go +++ b/gen/network.go @@ -5,6 +5,7 @@ import ( "fmt" "io" "net" + "reflect" ) // Network interface provides distributed communication and node connectivity management. @@ -123,6 +124,49 @@ type Network interface { // Mode returns the current network mode (Enabled, Hidden, or Disabled). Mode() NetworkMode + + // Protos returns all registered network protocols. + Protos() []NetworkProto + + // RegisterType registers a Go type with every proto that implements TypeRegistry. + // Strict: returns error if any TypeRegistry-capable proto fails. + // Returns ErrUnsupported if no proto implements TypeRegistry. + RegisterType(v any) error + + // RegisterTypes registers multiple types as a batch. Each proto resolves + // inter-type dependencies internally; order of input is irrelevant. + // Strict aggregation: any per-proto failure fails the call. + RegisterTypes(types []any) error + + // RegisterError registers a sentinel error for wire transport. + // Strict aggregation across TypeRegistry-capable protos. + RegisterError(e error) error + + // RegisterAtom registers an atom for the wire-format atom cache. + // Strict aggregation across TypeRegistry-capable protos. + RegisterAtom(a Atom) error + + // RegisteredTypes aggregates entries from every TypeRegistry-capable proto. + // One Go type may appear once per proto; entries carry Proto field set. + RegisteredTypes() []RegisteredTypeInfo + + // LookupType resolves a registered type name to its reflect.Type via the + // active wire-format protos. Returns the first match across protos. + // Accepts either the canonical name ("#pkgpath/Type") or a short name ("Type"). + LookupType(name string) (reflect.Type, bool) +} + +// TypeRegistry is implemented by NetworkProto implementations that have +// a wire-format type registry (e.g., EDF). Implementations using a +// schemaless / fixed wire format (e.g., Erlang external term) need not +// implement this interface. +type TypeRegistry interface { + RegisterType(v any) error + RegisterTypes(types []any) error + RegisterError(e error) error + RegisterAtom(a Atom) error + RegisteredTypes() []RegisteredTypeInfo + LookupType(name string) (reflect.Type, bool) } // RemoteNode interface represents a connection to a remote Ergo node. @@ -332,6 +376,13 @@ type NetworkOptions struct { // Can be replaced with Erlang protocol for Erlang/OTP compatibility. Proto NetworkProto + // SoftwareKeepAliveMisses sets how many consecutive keepalives from a remote node can be missed + // before the connection is considered dead. The remote node advertises its keepalive period + // during handshake; this value controls how patient we are waiting for them. + // Timeout = RemotePeriod * Misses. Zero uses DefaultSoftwareKeepAliveMisses. + // Acceptors and routes inherit this unless overridden. + SoftwareKeepAliveMisses int + // Acceptors configures listeners for incoming connections. // Node can have multiple acceptors on different ports/interfaces. // Empty means no acceptors (same as NetworkModeHidden). @@ -354,9 +405,20 @@ type NetworkOptions struct { // Controls how proxy connections are routed through this node. ProxyTransit ProxyTransitOptions - // TODO - // FragmentationUnit chunck size in bytes - //FragmentationUnit int + // FragmentSize sets the maximum fragment packet size in bytes. + // Messages larger than this are split into fragments for transmission. + // Zero uses DefaultFragmentSize (65000). Sender-local, no negotiation needed. + FragmentSize int + + // FragmentTimeout sets the maximum time in seconds to wait for all fragments of a message. + // Incomplete assemblies are cleaned up after this duration. + // Zero uses DefaultFragmentTimeout (30s). + FragmentTimeout int + + // MaxFragmentAssemblies limits concurrent unordered fragment assemblies per connection. + // Protects against memory exhaustion from many simultaneous large messages. + // Zero uses DefaultMaxFragmentAssemblies (1000). + MaxFragmentAssemblies int } type ProxyAcceptOptions struct { @@ -389,6 +451,17 @@ type NetworkFlags struct { EnableProxyAccept bool // EnableImportantDelivery enables support 'important' flag EnableImportantDelivery bool + // EnableSimultaneousConnect enables simultaneous connect detection and resolution + EnableSimultaneousConnect bool + // EnableClockSkew enables clock skew measurement between connected nodes. + // Both nodes must have it enabled for measurements to work. + EnableClockSkew bool + // EnableTracing enables distributed tracing support. + // Both nodes must have it enabled for trace context propagation. + EnableTracing bool + // EnableSoftwareKeepAlive enables application-level keepalive with the given period in seconds. + // Zero disables keepalive. Max 255. + EnableSoftwareKeepAlive int } // we must be able to extend this structure by introducing new features. @@ -420,6 +493,22 @@ func (nf NetworkFlags) MarshalEDF(w io.Writer) error { if nf.EnableImportantDelivery == true { flags |= 64 } + if nf.EnableSimultaneousConnect == true { + flags |= 128 + } + if nf.EnableSoftwareKeepAlive > 0 { + period := nf.EnableSoftwareKeepAlive + if period > 255 { + period = 255 + } + flags |= uint64(period) << 8 + } + if nf.EnableClockSkew == true { + flags |= 1 << 16 + } + if nf.EnableTracing == true { + flags |= 1 << 17 + } binary.BigEndian.PutUint64(buf[:], flags) w.Write(buf[:]) return nil @@ -440,6 +529,10 @@ func (nf *NetworkFlags) UnmarshalEDF(buf []byte) error { nf.EnableProxyTransit = (flags & 16) > 0 nf.EnableProxyAccept = (flags & 32) > 0 nf.EnableImportantDelivery = (flags & 64) > 0 + nf.EnableSimultaneousConnect = (flags & 128) > 0 + nf.EnableSoftwareKeepAlive = int((flags >> 8) & 0xFF) + nf.EnableClockSkew = (flags & (1 << 16)) > 0 + nf.EnableTracing = (flags & (1 << 17)) > 0 return nil } @@ -504,6 +597,9 @@ type RemoteNodeInfo struct { // Reported during handshake. Messages exceeding this are rejected. MaxMessageSize int + // TLS indicates whether this connection uses TLS encryption. + TLS bool + // MessagesIn is the total number of messages received from this remote node. MessagesIn uint64 @@ -523,6 +619,43 @@ type RemoteNodeInfo struct { // TransitBytesOut is the total proxy transit bytes sent through this connection. // Only relevant if this connection is used as a proxy. TransitBytesOut uint64 + + // Reconnections is the total number of pool item reconnections. + // A non-zero value indicates connection instability. + Reconnections uint64 + + // FragmentsSent is the total number of individual fragments sent. + FragmentsSent uint64 + // FragmentMessagesSent is the total number of messages that were fragmented for sending. + FragmentMessagesSent uint64 + // FragmentsReceived is the total number of individual fragments received. + FragmentsReceived uint64 + // FragmentMessagesRecv is the total number of fragmented messages successfully reassembled. + FragmentMessagesRecv uint64 + // FragmentTimeouts is the total number of fragment assemblies that timed out. + FragmentTimeouts uint64 + + // TracedSent is the total number of messages sent with tracing wrapper. + TracedSent uint64 + // TracedReceived is the total number of messages received with tracing wrapper. + TracedReceived uint64 + + // CompressedSent is the total number of messages compressed on send. + CompressedSent uint64 + // CompressedBytesSent is the total bytes after compression (wire size). + CompressedBytesSent uint64 + // CompressedOrigBytesSent is the total bytes before compression (original size). + CompressedOrigBytesSent uint64 + // DecompressedRecv is the total number of messages decompressed on receive. + DecompressedRecv uint64 + // DecompressedBytesRecv is the total bytes before decompression (wire size). + DecompressedBytesRecv uint64 + // DecompressedOrigRecv is the total bytes after decompression (original size). + DecompressedOrigRecv uint64 + + // ClockSkew is the estimated clock offset relative to the remote node (nanoseconds). + // Positive value means remote clock is ahead. Zero if not yet measured. + ClockSkew int64 } // AcceptorOptions configures a network listener (acceptor) for incoming connections. @@ -539,13 +672,14 @@ type AcceptorOptions struct { Host string // Port is the TCP port number for incoming connections. - // Default: 15000 if not specified. + // Default: 11144 if not specified. Port uint16 - // PortRange defines the range of ports to try if Port is unavailable. - // Attempts ports from Port to (Port + PortRange). - // Example: Port=15000, PortRange=10 tries 15000-15010. - // Useful for avoiding port conflicts. + // PortRange defines how many ports to try starting from Port. + // Default (0): tries all ports from Port to 65535. + // PortRange=1: tries only the Port itself. + // PortRange=N (N>1): tries N ports starting from Port. + // Example: Port=11144, PortRange=10 tries 11144-11153. PortRange uint16 // RouteHost specifies the public/external host address to advertise in routes. @@ -603,6 +737,16 @@ type AcceptorOptions struct { // Proto overrides the default network protocol for this acceptor. // Allows mixing EDF and Erlang protocols on different ports. Proto NetworkProto + + // MaxHandshakes limits the number of simultaneous in-flight handshakes + // on this acceptor. When the limit is reached, new connections are + // rejected immediately with a "busy" reason. + // Zero (default) means unlimited. + MaxHandshakes int + + // SoftwareKeepAliveMisses sets how many consecutive keepalives from a remote node can be missed + // before the connection is considered dead. Zero inherits from NetworkOptions or uses default. + SoftwareKeepAliveMisses int } // Handshake defines handshake interface @@ -615,6 +759,9 @@ type NetworkHandshake interface { Join(NodeHandshake, net.Conn, string, HandshakeOptions) ([]byte, error) // Accept accepts handshake process initiated by another side of this connection. Accept(NodeHandshake, net.Conn, HandshakeOptions) (HandshakeResult, error) + // Reject sends a rejection message to the connecting side and is used + // when the acceptor is too busy to handle a new handshake. + Reject(net.Conn, string) error // Version Version() Version } @@ -642,6 +789,10 @@ type HandshakeOptions struct { // Communicated to peer during handshake so peer knows the limit. // Peer will reject sending messages exceeding this size. MaxMessageSize int + + // CheckPending returns true if this node has a pending outgoing + // connect to the given peer. Used for simultaneous connect detection. + CheckPending func(peer Atom) bool } type HandshakeResult struct { @@ -659,6 +810,9 @@ type HandshakeResult struct { AtomMapping map[Atom]Atom + PoolSize int + PoolDSN []string + // Tail if something is left in the buffer after the handshaking we should // pass it to the proto handler Tail []byte @@ -717,6 +871,12 @@ type NetworkInfo struct { // Indicates what features are enabled globally. Flags NetworkFlags + // ConnectionsEstablished is the cumulative number of connections established. + ConnectionsEstablished uint64 + + // ConnectionsLost is the cumulative number of connections lost. + ConnectionsLost uint64 + // EnabledSpawn lists processes that remote nodes are allowed to spawn. // Includes process name, behavior, and which nodes can spawn it. EnabledSpawn []NetworkSpawnInfo @@ -761,6 +921,10 @@ type NetworkRoute struct { AtomMapping map[Atom]Atom LogLevel LogLevel + + // SoftwareKeepAliveMisses sets how many consecutive keepalives from a remote node can be missed + // before the connection is considered dead. Zero inherits from NetworkOptions or uses default. + SoftwareKeepAliveMisses int } type NetworkProxyRoute struct { diff --git a/gen/node.go b/gen/node.go index 79dc2d728..d296c05f9 100644 --- a/gen/node.go +++ b/gen/node.go @@ -106,7 +106,14 @@ type Node interface { // More efficient than ProcessList + ProcessInfo for each. // Available in: Running state only. // Returns ErrNodeTerminated in other states. - ProcessListShortInfo(start, limit int) ([]ProcessShortInfo, error) + ProcessListShortInfo(start, limit int, filter ...func(ProcessShortInfo) bool) ([]ProcessShortInfo, error) + + // ProcessRangeShortInfo iterates over all processes calling fn for each. + // The callback receives ProcessShortInfo and returns true to continue + // or false to stop iteration. + // Available in: Running state only. + // Returns ErrNodeTerminated in other states. + ProcessRangeShortInfo(fn func(ProcessShortInfo) bool) error // ProcessName returns the registered name for the given PID. // Returns empty Atom if the process has no registered name. @@ -298,6 +305,25 @@ type Node interface { // Returns ErrNodeTerminated in other states, ErrEventUnknown if not found. UnregisterEvent(name Atom) error + // EventInfo returns information about the given event. + // Available in: Running state only. + // Returns ErrNodeTerminated in other states, ErrEventUnknown if not found. + EventInfo(event Event) (EventInfo, error) + + // EventRangeInfo iterates over all registered events calling fn for each. + // The callback receives EventInfo and returns true to continue + // or false to stop iteration. + // Available in: Running state only. + // Returns ErrNodeTerminated in other states. + EventRangeInfo(fn func(EventInfo) bool) error + + // EventListInfo returns a paginated list of events in registration order. + // timestamp: 0 = from oldest, -1 = from newest, >0 = from events created at or after this time (unix nanos). + // limit: >0 = forward (oldest first), <0 = backward (newest first), abs(limit) results max. + // filter: optional function to include only matching events. + // Available in: Running state only. + EventListInfo(timestamp int64, limit int, filter ...func(EventInfo) bool) ([]EventInfo, error) + // SendExit sends a graceful termination request to the process. // Sender is the node's core PID. // Available in: Running state only. @@ -378,25 +404,47 @@ type Node interface { // Available in all states. Log() Log - // LogLevelProcess returns the logging level for the given process. + // SetProcessLogLevel sets the logging level for the given process. // Available in: Running state only. // Returns ErrNodeTerminated in other states. - LogLevelProcess(pid PID) (LogLevel, error) + SetProcessLogLevel(pid PID, level LogLevel) error - // SetLogLevelProcess sets the logging level for the given process. + // SetProcessSendPriority sets the default message sending priority for the given process. // Available in: Running state only. - // Returns ErrNodeTerminated in other states. - SetLogLevelProcess(pid PID, level LogLevel) error + SetProcessSendPriority(pid PID, priority MessagePriority) error - // LogLevelMeta returns the logging level for the given meta process. + // SetProcessCompression enables or disables compression for the given process. // Available in: Running state only. - // Returns ErrNodeTerminated in other states. - LogLevelMeta(meta Alias) (LogLevel, error) + SetProcessCompression(pid PID, enabled bool) error + + // SetProcessCompressionType sets the compression type for the given process. + // Available in: Running state only. + SetProcessCompressionType(pid PID, ctype CompressionType) error + + // SetProcessCompressionLevel sets the compression level for the given process. + // Available in: Running state only. + SetProcessCompressionLevel(pid PID, level CompressionLevel) error + + // SetProcessCompressionThreshold sets the minimum message size that triggers compression for the given process. + // Available in: Running state only. + SetProcessCompressionThreshold(pid PID, threshold int) error - // SetLogLevelMeta sets the logging level for the given meta process. + // SetProcessKeepNetworkOrder enables or disables maintaining delivery order over the network for the given process. + // Available in: Running state only. + SetProcessKeepNetworkOrder(pid PID, order bool) error + + // SetProcessImportantDelivery enables or disables the important delivery flag for the given process. + // Available in: Running state only. + SetProcessImportantDelivery(pid PID, important bool) error + + // SetMetaLogLevel sets the logging level for the given meta process. // Available in: Running state only. // Returns ErrNodeTerminated in other states. - SetLogLevelMeta(meta Alias, level LogLevel) error + SetMetaLogLevel(meta Alias, level LogLevel) error + + // SetMetaSendPriority sets the default message sending priority for the given meta process. + // Available in: Running state only. + SetMetaSendPriority(meta Alias, priority MessagePriority) error // Loggers returns a list of registered logger names. // Available in all states. @@ -445,6 +493,53 @@ type Node interface { // Available in all states. LoggerLevels(name string) []LogLevel + // TracingExporterAddPID registers a process as a tracing exporter. + // The process will receive TracingSpan messages. + // Available in: Running state only. + // Returns ErrTaken if name already registered. + TracingExporterAddPID(pid PID, name string, flags TracingFlags) error + + // TracingExporterAdd registers a custom tracing exporter implementation. + // Available in: Running state only. + // Returns ErrTaken if name already registered. + TracingExporterAdd(name string, exporter TracingBehavior, flags TracingFlags) error + + // TracingExporterDeletePID removes a process-based tracing exporter. + // Available in all states. + TracingExporterDeletePID(pid PID) + + // TracingExporterDelete removes a tracing exporter. + // Calls exporter.Terminate() if exporter exists. + // Available in all states. + TracingExporterDelete(name string) + + // TracingExporters returns a list of registered tracing exporter names. + // Available in all states. + TracingExporters() []string + + // TracingExporterFlags returns the flags for the given tracing exporter. + // Available in all states. + TracingExporterFlags(name string) TracingFlags + + // SetTracingSampler sets the tracing sampler for node-level Send/Call. + // Use TracingSamplerDisable to turn off. + // Available in: Running state only. + SetTracingSampler(sampler TracingSampler) error + + // SetTracingAttribute sets a permanent tracing attribute on the node. + SetTracingAttribute(key, value string) + + // RemoveTracingAttribute removes a permanent tracing attribute from the node. + RemoveTracingAttribute(key string) + + // TracingSampler returns the current tracing sampler for the node. + // Available in all states. + TracingSampler() TracingSampler + + // SetProcessTracingSampler sets the tracing sampler for the given process. + // Available in: Running state only. + SetProcessTracingSampler(pid PID, sampler TracingSampler) error + // MakeRef creates a unique reference within this node. // Used for Call requests, event tokens, and correlation. // Available in: Running state only. @@ -549,6 +644,28 @@ type NodeOptions struct { // Reported in node.Version() and during network handshakes. // Includes Name, Release, License, Commit details. Version Version + + // Tracing configures tracing exporters at node startup. + Tracing TracingOptions + + // Events lists node-level events to register before starting any application. + // All events declared here are registered with the node as producer + // (Open is forced to true) and live until the node stops. Use this to + // establish node-wide event buses that application processes can subscribe + // to from Init() without the race of waiting for a producer process to + // register the event first. + Events []NodeEventSpec +} + +// NodeEventSpec declares a node-level event to be registered during node +// startup, before any application starts. +type NodeEventSpec struct { + // Name is the event name. Must be unique within the node event namespace. + Name Atom + + // Buffer is the ring buffer size for recent MessageEvent values. + // Zero means no buffer. + Buffer int } // SecurityOptions controls information exposure and security policies. @@ -714,6 +831,20 @@ type NodeInfo struct { // Loggers lists all registered loggers with their configuration. Loggers []LoggerInfo + // Tracing contains node-level tracing configuration. + Tracing TracingInfo + + // TracingExporters lists all registered tracing exporters. + TracingExporters []TracingExporterInfo + + // LogMessages contains cumulative log message counts by level. + // Indexed as: [0]=Trace, [1]=Debug, [2]=Info, [3]=Warning, [4]=Error, [5]=Panic + LogMessages [6]uint64 + + // TracingSpans contains cumulative tracing span counts by kind. + // Indexed as: [0]=Send, [1]=Request, [2]=Response, [3]=Spawn, [4]=Terminate + TracingSpans [5]uint64 + // Cron contains cron scheduler information (jobs, schedule, next run). Cron CronInfo @@ -723,9 +854,33 @@ type NodeInfo struct { // ProcessesRunning is the number of processes currently in Running state. ProcessesRunning int64 + // ProcessesWaitResponse is the number of processes blocked in a synchronous Call. + ProcessesWaitResponse int64 + // ProcessesZombee is the number of killed processes (Zombee state). ProcessesZombee int64 + // ProcessesSpawned is the cumulative number of successfully spawned processes. + ProcessesSpawned uint64 + + // ProcessesSpawnFailed is the cumulative number of failed spawn attempts. + ProcessesSpawnFailed uint64 + + // ProcessesTerminated is the cumulative number of terminated processes. + ProcessesTerminated uint64 + + // SendErrorsLocal is the cumulative number of local send delivery errors. + SendErrorsLocal uint64 + + // SendErrorsRemote is the cumulative number of remote send delivery errors. + SendErrorsRemote uint64 + + // CallErrorsLocal is the cumulative number of local call delivery errors. + CallErrorsLocal uint64 + + // CallErrorsRemote is the cumulative number of remote call delivery errors. + CallErrorsRemote uint64 + // RegisteredAliases is the total number of registered aliases. RegisteredAliases int64 @@ -735,6 +890,18 @@ type NodeInfo struct { // RegisteredEvents is the total number of registered events. RegisteredEvents int64 + // EventsPublished is the cumulative number of events published by local producers. + EventsPublished int64 + + // EventsReceived is the cumulative number of events received from remote nodes. + EventsReceived int64 + + // EventsLocalSent is the cumulative number of event messages sent to local subscribers. + EventsLocalSent int64 + + // EventsRemoteSent is the cumulative number of event messages sent to remote subscribers. + EventsRemoteSent int64 + // ApplicationsTotal is the total number of loaded applications. ApplicationsTotal int64 @@ -752,6 +919,10 @@ type NodeInfo struct { // SystemTime is the system CPU time in nanoseconds. SystemTime int64 + + // ServerTime is the current server time with timezone. + // Useful in Observer and MCP for correlating logs across nodes in different timezones. + ServerTime time.Time } // LoggerInfo describes a registered logger. @@ -768,3 +939,15 @@ type LoggerInfo struct { // Empty means logger receives all log levels. Levels []LogLevel } + +// TracingExporterInfo contains information about a registered tracing exporter. +type TracingExporterInfo struct { + // Name is the unique exporter identifier. + Name string + + // Behavior is the exporter type name. + Behavior string + + // Flags is the tracing granularity for this exporter. + Flags TracingFlags +} diff --git a/gen/process.go b/gen/process.go index a356c9fa6..41c5d6a23 100644 --- a/gen/process.go +++ b/gen/process.go @@ -340,6 +340,18 @@ type Process interface { // Available in all states. ImportantDelivery() bool + // SetTracingSampler sets the tracing sampler for this process. + // The sampler decides per outgoing message whether to start a new trace. + // Only consulted when there is no active propagating trace. + // Use TracingSamplerDisable to turn off (default). + // Available in: Init, Running states. + SetTracingSampler(sampler TracingSampler) error + + // TracingSampler returns the current tracing sampler. + // Returns TracingSamplerDisable if tracing is not enabled. + // Available in all states. + TracingSampler() TracingSampler + // CreateAlias creates a new alias associated with this process. // Other processes can send messages or make calls using this alias. // Available in: Init, Running states. @@ -760,6 +772,43 @@ type Process interface { // Available in all states. Behavior() ProcessBehavior + // BehaviorName returns the string name of the behavior type. + // Available in all states. + BehaviorName() string + + // PropagatingTrace returns the current propagating trace context. + // Zero value means no active trace. + PropagatingTrace() Tracing + + // SetPropagatingTrace sets the propagating trace context. + // Used by ProcessBehavior implementations to manage trace context + // during message handling (save/restore around handler calls). + SetPropagatingTrace(t Tracing) + + // SetTracingAttribute sets a permanent tracing attribute on this process. + // Permanent attributes are included in every span emitted by this process. + // Uses copy-on-write: safe for concurrent readers (exporters). + SetTracingAttribute(key, value string) + + // RemoveTracingAttribute removes a permanent tracing attribute. + RemoveTracingAttribute(key string) + + // SetTracingSpanAttribute sets a one-shot tracing attribute for the current handler. + // Cleared automatically after handler returns. + SetTracingSpanAttribute(key, value string) + + // TracingAttributes returns merged permanent + one-shot attributes. + // Returns a slice reference (zero alloc) when only one type exists. + TracingAttributes() []TracingAttribute + + // ClearTracingSpanAttributes clears one-shot attributes. + // Called by framework after handler returns. + ClearTracingSpanAttributes() + + // SendTracingSpan delivers a tracing span to registered exporters. + // Used by ProcessBehavior implementations to emit Processed spans. + SendTracingSpan(span TracingSpan) + // Forward forwards a mailbox message to another process with the specified priority. // This is a low-level operation for custom message routing. // Available in: Running state only. @@ -795,6 +844,8 @@ type MessageOptions struct { Compression Compression KeepNetworkOrder bool ImportantDelivery bool + Tracing Tracing + TracingAttributes []TracingAttribute // sender's attrs for Sent span, nil = none } // ProcessOptions defines configuration options for spawning a process. @@ -868,6 +919,7 @@ type ProcessOptionsExtra struct { ParentLeader PID ParentEnv map[Env]any ParentLogLevel LogLevel + Tracing Tracing Register Atom Application Atom @@ -912,6 +964,13 @@ type ProcessInfo struct { // RunningTime is the cumulative time spent in Running state (nanoseconds). RunningTime uint64 + // InitTime is the time spent in ProcessInit callback (nanoseconds). + InitTime uint64 + + // Wakeups is the cumulative number of times the process transitioned + // from Sleep to Running state to handle messages. + Wakeups uint64 + // Compression contains the compression configuration for this process. Compression Compression @@ -924,12 +983,18 @@ type ProcessInfo struct { // State is the current process state (Init, Sleep, Running, WaitResponse, Terminated, Zombee). State ProcessState + // StateTime is the elapsed time since the process entered its current state (nanoseconds). + StateTime int64 + // Parent is the PID of the parent process that spawned this process. Parent PID // Leader is the group leader PID (typically supervisor or application). Leader PID + // Tracing contains tracing configuration for this process. + Tracing TracingInfo + // Fallback contains mailbox overflow handling configuration. Fallback ProcessFallback @@ -1014,15 +1079,30 @@ type ProcessShortInfo struct { // MessagesMailbox is the total number of messages currently in mailbox queues. MessagesMailbox uint64 + // MailboxLatency is the maximum latency across all mailbox queues (nanoseconds). + // Returns -1 if built without -tags=latency (measurement disabled). + // Returns 0 if all queues are empty. + MailboxLatency int64 + // RunningTime is the cumulative time spent in Running state (nanoseconds). RunningTime uint64 + // InitTime is the time spent in ProcessInit callback (nanoseconds). + InitTime uint64 + + // Wakeups is the cumulative number of times the process transitioned + // from Sleep to Running state to handle messages. + Wakeups uint64 + // Uptime is the process uptime in seconds since creation. Uptime int64 // State is the current process state (Init, Sleep, Running, WaitResponse, Terminated, Zombee). State ProcessState + // StateTime is the elapsed time since the process entered its current state (nanoseconds). + StateTime int64 + // Parent is the PID of the parent process that spawned this process. Parent PID @@ -1083,6 +1163,26 @@ func (pm ProcessMailbox) Len() int64 { return pm.Main.Len() + pm.System.Len() + pm.Urgent.Len() + pm.Log.Len() } +// Latency returns the maximum latency across all mailbox queues (nanoseconds). +// Returns -1 if built without -tags=latency (measurement disabled). +// Returns 0 if all queues are empty. +func (pm ProcessMailbox) Latency() int64 { + lat := pm.Main.Latency() + if lat < 0 { + return -1 + } + if l := pm.System.Latency(); l > lat { + lat = l + } + if l := pm.Urgent.Latency(); l > lat { + lat = l + } + if l := pm.Log.Latency(); l > lat { + lat = l + } + return lat +} + // MailboxQueues contains message counts for each mailbox queue. // Part of ProcessInfo and ProcessShortInfo. // Represents a snapshot of mailbox load at the time of query. @@ -1099,4 +1199,17 @@ type MailboxQueues struct { // Log is the number of logging messages in the Log queue. Log int64 + + // LatencyMain is the latency of the oldest message in the Main queue (nanoseconds). + // Returns -1 if built without -tags=latency. + LatencyMain int64 + + // LatencySystem is the latency of the oldest message in the System queue (nanoseconds). + LatencySystem int64 + + // LatencyUrgent is the latency of the oldest message in the Urgent queue (nanoseconds). + LatencyUrgent int64 + + // LatencyLog is the latency of the oldest message in the Log queue (nanoseconds). + LatencyLog int64 } diff --git a/gen/registrar.go b/gen/registrar.go index c3cd47583..bc4293891 100644 --- a/gen/registrar.go +++ b/gen/registrar.go @@ -5,10 +5,10 @@ package gen // // Default Registrar Behavior (if not configured in NetworkOptions): // - Uses minimal built-in registrar (no external service required) -// - Tries to start embedded registrar server on localhost:41000 +// - Tries to start embedded registrar server on localhost:4499 // - If port taken, connects to existing registrar on localhost (TCP client) // - Same host: nodes discover each other via local registrar server -// - Different hosts: queries remote host's registrar via UDP (host:41000) +// - Different hosts: queries remote host's registrar via UDP (host:4499) // - Remote host must have registrar server running // - Remote host must be reachable and port open // - No persistent connection - query on-demand only @@ -171,6 +171,9 @@ type AcceptorInfo struct { // ProtoVersion is the network protocol version (EDF or Erlang). ProtoVersion Version + + // HandshakeErrors is the cumulative number of failed handshakes on this acceptor. + HandshakeErrors uint64 } // RegisterRoutes contains routes to publish when registering with the service registry. diff --git a/gen/target.go b/gen/target.go index 4c4a660be..ff029bc01 100644 --- a/gen/target.go +++ b/gen/target.go @@ -33,6 +33,8 @@ type TargetManager interface { UnregisterEvent(producer PID, name Atom) error PublishEvent(from PID, token Ref, options MessageOptions, message MessageEvent) error EventInfo(event Event) (EventInfo, error) + EventRangeInfo(fn func(EventInfo) bool) error + EventListInfo(timestamp int64, limit int, filter ...func(EventInfo) bool) ([]EventInfo, error) LinksFor(consumer PID) []any MonitorsFor(consumer PID) []any @@ -53,11 +55,22 @@ type TargetManager interface { // EventInfo contains event metadata and statistics type EventInfo struct { - Producer PID - BufferSize int - CurrentBuffer int - Notify bool - Subscribers int64 + CreatedAt int64 + Event Event + Producer PID + BufferSize int + CurrentBuffer int + Notify bool + Open bool + Subscribers int64 + MessagesPublished int64 + MessagesLocalSent int64 + MessagesRemoteSent int64 + // LastPublishedAt is the unix-nanos timestamp of the most recent SendEvent on this event, + // or 0 if nothing has ever been published. Useful for detecting silent producers — a + // non-zero MessagesPublished combined with a stale LastPublishedAt means the producer + // has stopped publishing. + LastPublishedAt int64 } type TargetManagerInfo struct { @@ -70,6 +83,8 @@ type TargetManagerInfo struct { ExitSignalsDelivered int64 // Total exit signals delivered by dispatchers DownMessagesProduced int64 // Total down messages generated DownMessagesDelivered int64 // Total down messages delivered - EventsPublished int64 // Total events published - EventsSent int64 // Total event messages sent to subscribers + EventsPublished int64 // Total events published by local producers + EventsReceived int64 // Total events received from remote nodes + EventsLocalSent int64 // Total event messages sent to local subscribers + EventsRemoteSent int64 // Total event messages sent to remote subscribers } diff --git a/gen/tracing.go b/gen/tracing.go new file mode 100644 index 000000000..7b966328c --- /dev/null +++ b/gen/tracing.go @@ -0,0 +1,167 @@ +package gen + +import ( + "encoding/json" + "fmt" +) + +// Tracing carries trace identity through message chains. +// Zero value (ID == [2]uint64{}) means no active tracing. +type Tracing struct { + ID [2]uint64 + SpanID uint64 + Behavior string +} + +// TracingFlags controls tracing granularity. +type TracingFlags uint32 + +const ( + TracingFlagSend TracingFlags = 1 << 0 // trace send/call/response + TracingFlagReceive TracingFlags = 1 << 1 // trace delivered to mailbox + TracingFlagProcs TracingFlags = 1 << 2 // trace spawn/terminate + TracingFlagInherit TracingFlags = 1 << 3 // children inherit tracing +) + +// TracingSpan represents a single observation point +// in the message lifecycle. +type TracingSpan struct { + TraceID [2]uint64 + SpanID uint64 + ParentSpanID uint64 // SpanID from propagating trace context (0 = root) + Point TracingPoint + Kind TracingKind + Timestamp int64 // wall clock unix nanoseconds + Node Atom + From PID + To any // PID, ProcessID, or Alias + Ref Ref + Behavior string // behavior name of the emitting process + Message string // type name of the message + Error string // empty = no error + Attributes []TracingAttribute // custom attributes, nil = none +} + +// TracingAttribute is a key-value pair attached to a tracing span. +type TracingAttribute struct { + Key string + Value string +} + +func (ts TracingSpan) MarshalJSON() ([]byte, error) { + type alias struct { + TraceID string `json:"TraceID"` + SpanID string `json:"SpanID"` + ParentSpanID string `json:"ParentSpanID,omitempty"` + Point TracingPoint `json:"Point"` + Kind TracingKind `json:"Kind"` + Timestamp int64 `json:"Timestamp"` + Node Atom `json:"Node"` + From PID `json:"From"` + To any `json:"To"` + Ref Ref `json:"Ref"` + Behavior string `json:"Behavior,omitempty"` + Message string `json:"Message"` + Error string `json:"Error,omitempty"` + Attributes []TracingAttribute `json:"Attributes,omitempty"` + } + a := alias{ + TraceID: fmt.Sprintf("%016x%016x", ts.TraceID[0], ts.TraceID[1]), + SpanID: fmt.Sprintf("%016x", ts.SpanID), + Point: ts.Point, + Kind: ts.Kind, + Timestamp: ts.Timestamp, + Node: ts.Node, + From: ts.From, + To: ts.To, + Ref: ts.Ref, + Behavior: ts.Behavior, + Message: ts.Message, + Error: ts.Error, + Attributes: ts.Attributes, + } + if ts.ParentSpanID != 0 { + a.ParentSpanID = fmt.Sprintf("%016x", ts.ParentSpanID) + } + return json.Marshal(a) +} + +// TracingPoint identifies where in the lifecycle the observation occurred. +type TracingPoint int + +const ( + TracingPointSent TracingPoint = 1 + TracingPointDelivered TracingPoint = 2 + TracingPointProcessed TracingPoint = 3 +) + +func (tp TracingPoint) String() string { + switch tp { + case TracingPointSent: + return "sent" + case TracingPointDelivered: + return "delivered" + case TracingPointProcessed: + return "processed" + } + return fmt.Sprintf("point#%d", int(tp)) +} + +func (tp TracingPoint) MarshalJSON() ([]byte, error) { + return []byte(`"` + tp.String() + `"`), nil +} + +// TracingKind identifies the type of operation being traced. +type TracingKind int + +const ( + TracingKindSend TracingKind = 1 + TracingKindRequest TracingKind = 2 + TracingKindResponse TracingKind = 3 + TracingKindSpawn TracingKind = 4 + TracingKindTerminate TracingKind = 5 +) + +func (tk TracingKind) String() string { + switch tk { + case TracingKindSend: + return "send" + case TracingKindRequest: + return "request" + case TracingKindResponse: + return "response" + case TracingKindSpawn: + return "spawn" + case TracingKindTerminate: + return "terminate" + } + return fmt.Sprintf("kind#%d", int(tk)) +} + +func (tk TracingKind) MarshalJSON() ([]byte, error) { + return []byte(`"` + tk.String() + `"`), nil +} + +// TracingInfo contains tracing configuration for a process or node. +type TracingInfo struct { + Sampler string + Attributes []TracingAttribute +} + +// TracingBehavior interface for tracing exporters. +type TracingBehavior interface { + HandleSpan(TracingSpan) + Terminate() +} + +// TracingExporter defines a named exporter with its flags. +type TracingExporter struct { + Name string + Exporter TracingBehavior + Flags TracingFlags +} + +// TracingOptions configures tracing at node startup. +type TracingOptions struct { + Exporters []TracingExporter +} diff --git a/gen/tracing_sampler.go b/gen/tracing_sampler.go new file mode 100644 index 000000000..74c32852d --- /dev/null +++ b/gen/tracing_sampler.go @@ -0,0 +1,96 @@ +package gen + +import ( + "fmt" + "sync/atomic" + "time" +) + +// TracingSampler decides whether to start a new trace for an outgoing message. +// Only consulted when there is no active propagating trace. +type TracingSampler interface { + Sample() bool + String() string +} + +// TracingSamplerAlways is a sampler that always starts a trace. +var TracingSamplerAlways TracingSampler = &samplerAlways{} + +// TracingSamplerDisable is a sampler that never starts a trace. +// Default for all processes. +var TracingSamplerDisable TracingSampler = &samplerDisable{} + +type samplerAlways struct{} + +func (s *samplerAlways) Sample() bool { return true } +func (s *samplerAlways) String() string { return "always" } + +type samplerDisable struct{} + +func (s *samplerDisable) Sample() bool { return false } +func (s *samplerDisable) String() string { return "disable" } + +// TracingSamplerRatio returns a sampler that traces the given fraction of messages. +// Rate must be between 0.0 and 1.0. +func TracingSamplerRatio(rate float64) TracingSampler { + if rate >= 1 { + return TracingSamplerAlways + } + if rate <= 0 { + return TracingSamplerDisable + } + return &samplerRatio{ + mod: uint64(1.0 / rate), + rate: rate, + } +} + +type samplerRatio struct { + counter uint64 + mod uint64 + rate float64 +} + +func (s *samplerRatio) Sample() bool { + return atomic.AddUint64(&s.counter, 1)%s.mod == 0 +} + +func (s *samplerRatio) String() string { + return fmt.Sprintf("ratio(%g)", s.rate) +} + +// TracingSamplerRateLimit returns a sampler that allows at most perSecond traces per second. +func TracingSamplerRateLimit(perSecond int) TracingSampler { + if perSecond <= 0 { + return TracingSamplerDisable + } + return &samplerRateLimit{ + tokens: int64(perSecond), + max: int64(perSecond), + lastTick: time.Now().Unix(), + perSecond: perSecond, + } +} + +type samplerRateLimit struct { + tokens int64 + max int64 + lastTick int64 + perSecond int +} + +func (s *samplerRateLimit) Sample() bool { + now := time.Now().Unix() + last := atomic.LoadInt64(&s.lastTick) + if now > last { + if atomic.CompareAndSwapInt64(&s.lastTick, last, now) { + atomic.StoreInt64(&s.tokens, s.max) + } + } + return atomic.AddInt64(&s.tokens, -1) >= 0 +} + +func (s *samplerRateLimit) String() string { + return fmt.Sprintf("rate_limit(%d/s)", s.perSecond) +} + diff --git a/gen/tracing_sampler_test.go b/gen/tracing_sampler_test.go new file mode 100644 index 000000000..c703d9a51 --- /dev/null +++ b/gen/tracing_sampler_test.go @@ -0,0 +1,120 @@ +package gen + +import ( + "testing" + "time" +) + +func TestSamplerAlways(t *testing.T) { + s := TracingSamplerAlways + for i := 0; i < 1000; i++ { + if s.Sample() == false { + t.Fatal("TracingSamplerAlways returned false") + } + } + if s.String() != "always" { + t.Fatalf("expected 'always', got %q", s.String()) + } +} + +func TestSamplerDisable(t *testing.T) { + s := TracingSamplerDisable + for i := 0; i < 1000; i++ { + if s.Sample() == true { + t.Fatal("TracingSamplerDisable returned true") + } + } + if s.String() != "disable" { + t.Fatalf("expected 'disable', got %q", s.String()) + } +} + +func TestSamplerRatioZero(t *testing.T) { + s := TracingSamplerRatio(0) + if s != TracingSamplerDisable { + t.Fatal("Ratio(0) should return TracingSamplerDisable") + } +} + +func TestSamplerRatioOne(t *testing.T) { + s := TracingSamplerRatio(1) + if s != TracingSamplerAlways { + t.Fatal("Ratio(1) should return TracingSamplerAlways") + } +} + +func TestSamplerRatio(t *testing.T) { + s := TracingSamplerRatio(0.5) + count := 0 + total := 10000 + for i := 0; i < total; i++ { + if s.Sample() { + count++ + } + } + // expect ~50% with tolerance + ratio := float64(count) / float64(total) + if ratio < 0.45 || ratio > 0.55 { + t.Fatalf("expected ~50%% samples, got %.2f%%", ratio*100) + } + if s.String() != "ratio(0.5)" { + t.Fatalf("expected 'ratio(0.5)', got %q", s.String()) + } +} + +func TestSamplerRatioSmall(t *testing.T) { + s := TracingSamplerRatio(0.01) + count := 0 + total := 100000 + for i := 0; i < total; i++ { + if s.Sample() { + count++ + } + } + ratio := float64(count) / float64(total) + if ratio < 0.005 || ratio > 0.015 { + t.Fatalf("expected ~1%% samples, got %.2f%%", ratio*100) + } +} + +func TestSamplerRateLimit(t *testing.T) { + s := TracingSamplerRateLimit(10) + count := 0 + for i := 0; i < 100; i++ { + if s.Sample() { + count++ + } + } + if count != 10 { + t.Fatalf("expected 10 samples in first burst, got %d", count) + } + if s.String() != "rate_limit(10/s)" { + t.Fatalf("expected 'rate_limit(10/s)', got %q", s.String()) + } +} + +func TestSamplerRateLimitRefill(t *testing.T) { + s := TracingSamplerRateLimit(5) + // exhaust tokens + for i := 0; i < 10; i++ { + s.Sample() + } + // wait for refill + time.Sleep(1100 * time.Millisecond) + count := 0 + for i := 0; i < 10; i++ { + if s.Sample() { + count++ + } + } + if count != 5 { + t.Fatalf("expected 5 samples after refill, got %d", count) + } +} + +func TestSamplerRateLimitZero(t *testing.T) { + s := TracingSamplerRateLimit(0) + if s != TracingSamplerDisable { + t.Fatal("RateLimit(0) should return TracingSamplerDisable") + } +} diff --git a/gen/types.go b/gen/types.go index 4fd9a81b0..6132fb9f0 100644 --- a/gen/types.go +++ b/gen/types.go @@ -31,6 +31,10 @@ func (a Atom) Host() string { return "" } +func (a Atom) CRC32Sum() uint32 { + return crc32.Checksum([]byte(a), crc32q) +} + func (a Atom) CRC32() string { if v, exist := crc32cache.Load(a); exist { return v.(string) @@ -162,8 +166,24 @@ type Event struct { } type EventOptions struct { + // Notify controls whether the producer receives MessageEventStart and + // MessageEventStop when the subscriber count crosses zero. Ignored for + // node-level events (producer is corePID), since the node core does not + // consume such messages. Notify bool + + // Buffer is the size of the ring buffer that stores the most recent + // MessageEvent values. New subscribers receive the buffer contents on + // Link or Monitor. Zero means no buffer. Buffer int + + // Open disables the token check on SendEvent. When true, any local process + // can publish to this event by name, regardless of the token value. The + // token returned by RegisterEvent is still generated but not required for + // publishing. The owner check on UnregisterEvent is unaffected: only the + // registering process (or the node, for node-level events) can unregister. + // Default: false. + Open bool } // String @@ -180,6 +200,43 @@ func (e Event) MarshalJSON() ([]byte, error) { return []byte("\"" + e.String() + "\""), nil } +// RegisteredTypeInfo describes a type registered with a wire-format proto. +type RegisteredTypeInfo struct { + // ID is the per-proto registration order. Strictly monotonic; gaps possible. + ID uint64 + // Name is the canonical wire-format name within the proto. + Name string + // Kind classifies the type for filters/badges (bool, int, struct, marshaler, ...). + Kind string + // Schema is a Go-syntax shape of the type. + Schema string + // Proto is the proto version owning this entry, set by the aggregator. + Proto string + // MinSize is the wire-format size in bytes of a zero-value of this type + // when sent as a top-level message (including the type-tag prefix). + // Real values may be larger if SizeVariable is true. + MinSize uint32 + // SizeVariable is true if the type contains string/slice/map/pointer/ + // interface fields whose encoded size depends on the actual value. + SizeVariable bool + // Stats holds per-type usage counters. Populated only when the proto + // implementation activates them (e.g. EDF when built with -tags=typestats). + // When Stats.Enabled is false, all counter fields remain zero. + Stats RegisteredTypeStats +} + +// RegisteredTypeStats holds per-type usage counters incremented on root +// encode/decode operations. Activation is controlled by the proto +// implementation. When Enabled is false, all counter fields are zero +// and unchanged across operations. +type RegisteredTypeStats struct { + Enabled bool + Encoded int64 + Decoded int64 + EncodedBytes int64 + DecodedBytes int64 +} + // Env type Env string diff --git a/go.mod b/go.mod index 1e3afc4eb..4135131d0 100644 --- a/go.mod +++ b/go.mod @@ -1,3 +1,3 @@ module ergo.services/ergo -go 1.20 +go 1.21 diff --git a/lib/buffer.go b/lib/buffer.go index 47b05f2f0..56be6decd 100644 --- a/lib/buffer.go +++ b/lib/buffer.go @@ -5,7 +5,6 @@ import ( "io" "math" "sync" - "sync/atomic" ) // Buffer @@ -25,30 +24,17 @@ var ( return b }, } - buffersTaken uint64 = 0 - buffersTakenSize uint64 = 0 - buffersReturned uint64 = 0 - buffersReturnedSize uint64 = 0 ) -func StatBuffers() { - fmt.Printf("taken: %d (size: %d)\n", buffersTaken, buffersTakenSize) - fmt.Printf("returned: %d (size: %d)\n", buffersReturned, buffersReturnedSize) -} - // TakeBuffer func TakeBuffer() *Buffer { b := buffers.Get().(*Buffer) - atomic.AddUint64(&buffersTaken, 1) - atomic.AddUint64(&buffersTakenSize, uint64(cap(b.B))) return b } // ReleaseBuffer func ReleaseBuffer(b *Buffer) { b.B = b.original[:0] - atomic.AddUint64(&buffersReturned, 1) - atomic.AddUint64(&buffersReturnedSize, uint64(cap(b.B))) buffers.Put(b) } diff --git a/lib/compress.go b/lib/compress.go index e6bd1db9a..ebd594d7f 100644 --- a/lib/compress.go +++ b/lib/compress.go @@ -15,6 +15,7 @@ import ( var ( gzipWriters [3]*sync.Pool + zlibWriters *sync.Pool gzipReaders = &sync.Pool{ New: func() any { return nil @@ -50,11 +51,14 @@ func CompressZLIB(src *Buffer, preallocate uint) (dst *Buffer, err error) { zBuffer.Allocate(int(preallocate) + 4) binary.BigEndian.PutUint32(zBuffer.B[preallocate:], uint32(src.Len())) - zWriter := zlib.NewWriter(zBuffer) - if _, err := zWriter.Write(src.B); err != nil { + zWriter := zlibWriters.Get().(*zlib.Writer) + zWriter.Reset(zBuffer) + _, err = zWriter.Write(src.B) + zWriter.Close() + zlibWriters.Put(zWriter) + if err != nil { return nil, err } - zWriter.Close() return zBuffer, nil } @@ -86,11 +90,12 @@ func CompressGZIP(src *Buffer, preallocate uint, level int) (dst *Buffer, err er } else { zWriter, _ = gzip.NewWriterLevel(zBuffer, lev) } - if _, err := zWriter.Write(src.B); err != nil { - return nil, err - } + _, err = zWriter.Write(src.B) zWriter.Close() gzipWriters[level].Put(zWriter) + if err != nil { + return nil, err + } return zBuffer, nil } @@ -183,4 +188,9 @@ func init() { }, } } + zlibWriters = &sync.Pool{ + New: func() any { + return zlib.NewWriter(io.Discard) + }, + } } diff --git a/lib/deprecation.go b/lib/deprecation.go new file mode 100644 index 000000000..183d431b9 --- /dev/null +++ b/lib/deprecation.go @@ -0,0 +1,31 @@ +package lib + +import ( + "fmt" + "io" + "os" + "sync" +) + +var ( + deprecationMu sync.Mutex + deprecationEmitted = make(map[string]struct{}) +) + +// EmitDeprecation writes a deprecation warning at most once per name. +// If w is nil, os.Stderr is used. +func EmitDeprecation(w io.Writer, name, replacement, url string) { + deprecationMu.Lock() + if _, exists := deprecationEmitted[name]; exists { + deprecationMu.Unlock() + return + } + deprecationEmitted[name] = struct{}{} + deprecationMu.Unlock() + + if w == nil { + w = os.Stderr + } + fmt.Fprintf(w, "[ergo] DEPRECATED: %s. Use %s instead. See %s\n", + name, replacement, url) +} diff --git a/lib/flusher.go b/lib/flusher.go index 1dd410ac5..c15c781c1 100644 --- a/lib/flusher.go +++ b/lib/flusher.go @@ -8,15 +8,21 @@ import ( ) const ( - latency time.Duration = 300 * time.Nanosecond + latency time.Duration = 500 * time.Nanosecond + bufioSize int = 65536 ) -func NewFlusherWithKeepAlive(w io.Writer, keepalive []byte, keepalivePeriod time.Duration) io.Writer { +type Flusher interface { + io.Writer + Reset(io.Writer) + Stop() +} + +func NewFlusherWithKeepAlive(w io.Writer, keepalive []byte, keepalivePeriod time.Duration) Flusher { f := &flusher{ - writer: bufio.NewWriter(w), + writer: bufio.NewWriterSize(w, bufioSize), } - // first time it should be longer - f.timer = time.AfterFunc(latency*10, func() { + callback := func() { f.Lock() defer f.Unlock() @@ -24,6 +30,7 @@ func NewFlusherWithKeepAlive(w io.Writer, keepalive []byte, keepalivePeriod time // nothing to write. send keepalive. f.writer.Write(keepalive) if err := f.writer.Flush(); err != nil { + f.err = err return } @@ -31,32 +38,42 @@ func NewFlusherWithKeepAlive(w io.Writer, keepalive []byte, keepalivePeriod time return } - f.writer.Flush() + if err := f.writer.Flush(); err != nil { + f.err = err + return + } f.pending = false - f.timer.Reset(latency) - }) + f.timer.Reset(keepalivePeriod) + } + f.Lock() + f.timer = time.AfterFunc(latency*10, callback) + f.Unlock() return f - } -func NewFlusher(w io.Writer) io.Writer { +func NewFlusher(w io.Writer) Flusher { f := &flusher{ - writer: bufio.NewWriter(w), + writer: bufio.NewWriterSize(w, bufioSize), } - f.timer = time.AfterFunc(latency, func() { + callback := func() { f.Lock() defer f.Unlock() if f.pending == false { - // nothing to write return } - f.writer.Flush() + if err := f.writer.Flush(); err != nil { + f.err = err + return + } f.pending = false f.timer.Reset(latency) - }) + } + f.Lock() + f.timer = time.AfterFunc(latency, callback) + f.Unlock() return f } @@ -65,21 +82,24 @@ type flusher struct { timer *time.Timer writer *bufio.Writer pending bool + err error } func (f *flusher) Write(b []byte) (n int, err error) { f.Lock() defer f.Unlock() + if f.err != nil { + return 0, f.err + } + l := len(b) - // write data to the buffer for { n, e := f.writer.Write(b) if e != nil { return n, e } - // check if something left l -= n if l > 0 { continue @@ -91,12 +111,23 @@ func (f *flusher) Write(b []byte) (n int, err error) { return len(b), nil } - // if f.writer.Size() > 65000 { - // f.writer.Flush() - // return len(b), nil - // } - f.pending = true f.timer.Reset(latency) return len(b), nil } + +func (f *flusher) Stop() { + f.Lock() + defer f.Unlock() + if f.timer != nil { + f.timer.Stop() + } +} + +func (f *flusher) Reset(w io.Writer) { + f.Lock() + defer f.Unlock() + f.writer.Reset(w) + f.pending = false + f.err = nil +} diff --git a/lib/hash.go b/lib/hash.go new file mode 100644 index 000000000..db54d5461 --- /dev/null +++ b/lib/hash.go @@ -0,0 +1,14 @@ +package lib + +// HashString64 returns a 64-bit FNV-1a hash of s. +// Used for distributing items across shards or queues. +// The algorithm is fixed so the same string always produces +// the same value across processes and across versions. +func HashString64(s string) uint64 { + h := uint64(14695981039346656037) + for i := 0; i < len(s); i++ { + h ^= uint64(s[i]) + h *= 1099511628211 + } + return h +} diff --git a/lib/mpsc.go b/lib/mpsc.go index ed69c050e..38ff6da5f 100644 --- a/lib/mpsc.go +++ b/lib/mpsc.go @@ -1,3 +1,5 @@ +//go:build !latency + // High-performance lock-free implementation of MPSC queue (Multiple Producers Single Consumer) package lib @@ -24,19 +26,6 @@ type queueLimitMPSC struct { lock uint32 } -type QueueMPSC interface { - Push(value any) bool - Pop() (any, bool) - Item() ItemMPSC - // Len returns the number of items in the queue - Len() int64 - // Size returns the limit for the queue. -1 - for unlimited - Size() int64 - - Lock() bool - Unlock() bool -} - func NewQueueMPSC() QueueMPSC { emptyItem := &itemMPSC{} return &queueMPSC{ @@ -62,12 +51,6 @@ func NewQueueLimitMPSC(limit int64, flush bool) QueueMPSC { } } -type ItemMPSC interface { - Next() ItemMPSC - Value() any - Clear() -} - type itemMPSC struct { value any next *itemMPSC @@ -137,6 +120,10 @@ func (q *queueMPSC) Size() int64 { return -1 // unlimited } +func (q *queueMPSC) Latency() int64 { + return -1 +} + func (q *queueMPSC) Lock() bool { return atomic.SwapUint32(&q.lock, 1) == 0 } @@ -154,6 +141,10 @@ func (q *queueLimitMPSC) Size() int64 { return q.limit } +func (q *queueLimitMPSC) Latency() int64 { + return -1 +} + func (q *queueLimitMPSC) Lock() bool { return atomic.SwapUint32(&q.lock, 1) == 0 } diff --git a/lib/mpsc_latency.go b/lib/mpsc_latency.go new file mode 100644 index 000000000..e8d3c739f --- /dev/null +++ b/lib/mpsc_latency.go @@ -0,0 +1,237 @@ +//go:build latency + +// High-performance lock-free MPSC queue with head-of-line latency measurement. +// Latency() returns how long the oldest item has been sitting in the queue. + +package lib + +import ( + "math" + "sync/atomic" + "unsafe" + _ "unsafe" +) + +//go:linkname nanotime runtime.nanotime +func nanotime() int64 + +type queueMPSCLatency struct { + head *itemMPSCLatency + tail *itemMPSCLatency + length int64 + oldest int64 // nanotime of the oldest item (head of queue) + lock uint32 +} + +type queueLimitMPSCLatency struct { + head *itemMPSCLatency + tail *itemMPSCLatency + length int64 + oldest int64 + limit int64 + flush bool + lock uint32 +} + +type itemMPSCLatency struct { + value any + next *itemMPSCLatency + pushed int64 +} + +func NewQueueMPSC() QueueMPSC { + emptyItem := &itemMPSCLatency{} + return &queueMPSCLatency{ + head: emptyItem, + tail: emptyItem, + } +} + +// NewQueueLimitMPSC creates MPSC queue with limited length and latency measurement. +// Enabling "flush" option makes this queue flush out the tail item if the limit has been reached. +// Warning: enabled "flush" option also makes this queue unusable +// for the concurrent environment +func NewQueueLimitMPSC(limit int64, flush bool) QueueMPSC { + if limit < 1 { + limit = math.MaxInt64 + } + emptyItem := &itemMPSCLatency{} + return &queueLimitMPSCLatency{ + limit: limit, + flush: flush, + head: emptyItem, + tail: emptyItem, + } +} + +// +// queueMPSCLatency +// + +func (q *queueMPSCLatency) Push(value any) bool { + i := &itemMPSCLatency{ + value: value, + pushed: nanotime(), + } + atomic.AddInt64(&q.length, 1) + old_head := (*itemMPSCLatency)(atomic.SwapPointer((*unsafe.Pointer)(unsafe.Pointer(&q.head)), unsafe.Pointer(i))) + atomic.StorePointer((*unsafe.Pointer)(unsafe.Pointer(&old_head.next)), unsafe.Pointer(i)) + atomic.CompareAndSwapInt64(&q.oldest, 0, i.pushed) + return true +} + +func (q *queueMPSCLatency) Pop() (any, bool) { + tail := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&q.tail)))) + tail_next := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&tail.next)))) + if tail_next == nil { + return nil, false + } + + value := tail_next.value + tail_next.value = nil + + atomic.StorePointer((*unsafe.Pointer)(unsafe.Pointer(&q.tail)), unsafe.Pointer(tail_next)) + atomic.AddInt64(&q.length, -1) + + // update oldest: check next item in queue + next := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&tail_next.next)))) + if next != nil { + atomic.StoreInt64(&q.oldest, next.pushed) + } else { + atomic.StoreInt64(&q.oldest, 0) + } + + return value, true +} + +func (q *queueMPSCLatency) Latency() int64 { + ts := atomic.LoadInt64(&q.oldest) + if ts == 0 { + return 0 + } + return nanotime() - ts +} + +func (q *queueMPSCLatency) Len() int64 { + return atomic.LoadInt64(&q.length) +} + +func (q *queueMPSCLatency) Size() int64 { + return -1 +} + +func (q *queueMPSCLatency) Lock() bool { + return atomic.SwapUint32(&q.lock, 1) == 0 +} + +func (q *queueMPSCLatency) Unlock() bool { + return atomic.SwapUint32(&q.lock, 0) == 1 +} + +func (q *queueMPSCLatency) Item() ItemMPSC { + tail := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&q.tail)))) + item := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&tail.next)))) + if item == nil { + return nil + } + return item +} + +// +// queueLimitMPSCLatency +// + +func (q *queueLimitMPSCLatency) Push(value any) bool { + if q.Len()+1 > q.limit { + if q.flush == false { + return false + } + q.Pop() + } + + i := &itemMPSCLatency{ + value: value, + pushed: nanotime(), + } + atomic.AddInt64(&q.length, 1) + old_head := (*itemMPSCLatency)(atomic.SwapPointer((*unsafe.Pointer)(unsafe.Pointer(&q.head)), unsafe.Pointer(i))) + atomic.StorePointer((*unsafe.Pointer)(unsafe.Pointer(&old_head.next)), unsafe.Pointer(i)) + atomic.CompareAndSwapInt64(&q.oldest, 0, i.pushed) + return true +} + +func (q *queueLimitMPSCLatency) Pop() (any, bool) { + tail := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&q.tail)))) + tail_next := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&tail.next)))) + if tail_next == nil { + return nil, false + } + + value := tail_next.value + tail_next.value = nil + + atomic.StorePointer((*unsafe.Pointer)(unsafe.Pointer(&q.tail)), unsafe.Pointer(tail_next)) + atomic.AddInt64(&q.length, -1) + + next := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&tail_next.next)))) + if next != nil { + atomic.StoreInt64(&q.oldest, next.pushed) + } else { + atomic.StoreInt64(&q.oldest, 0) + } + + return value, true +} + +func (q *queueLimitMPSCLatency) Latency() int64 { + ts := atomic.LoadInt64(&q.oldest) + if ts == 0 { + return 0 + } + return nanotime() - ts +} + +func (q *queueLimitMPSCLatency) Len() int64 { + return atomic.LoadInt64(&q.length) +} + +func (q *queueLimitMPSCLatency) Size() int64 { + return q.limit +} + +func (q *queueLimitMPSCLatency) Lock() bool { + return atomic.SwapUint32(&q.lock, 1) == 0 +} + +func (q *queueLimitMPSCLatency) Unlock() bool { + return atomic.SwapUint32(&q.lock, 0) == 1 +} + +func (q *queueLimitMPSCLatency) Item() ItemMPSC { + tail := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&q.tail)))) + item := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&tail.next)))) + if item == nil { + return nil + } + return item +} + +// +// itemMPSCLatency implements ItemMPSC +// + +func (i *itemMPSCLatency) Next() ItemMPSC { + next := (*itemMPSCLatency)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&i.next)))) + if next == nil { + return nil + } + return next +} + +func (i *itemMPSCLatency) Value() any { + return i.value +} + +func (i *itemMPSCLatency) Clear() { + i.value = nil +} diff --git a/lib/mpsc_types.go b/lib/mpsc_types.go new file mode 100644 index 000000000..04bc805b6 --- /dev/null +++ b/lib/mpsc_types.go @@ -0,0 +1,25 @@ +package lib + +type QueueMPSC interface { + Push(value any) bool + Pop() (any, bool) + Item() ItemMPSC + // Len returns the number of items in the queue + Len() int64 + // Size returns the limit for the queue. -1 - for unlimited + Size() int64 + // Latency returns how long the oldest item has been in the queue (nanoseconds). + // Safe to call from any goroutine. + // Returns -1 if built without -tags=latency (measurement disabled). + // Returns 0 if the queue is empty. + Latency() int64 + + Lock() bool + Unlock() bool +} + +type ItemMPSC interface { + Next() ItemMPSC + Value() any + Clear() +} diff --git a/lib/notrace.go b/lib/notrace.go deleted file mode 100644 index abe84fdc7..000000000 --- a/lib/notrace.go +++ /dev/null @@ -1,7 +0,0 @@ -//go:build !trace - -package lib - -func Trace() bool { - return false -} diff --git a/lib/noverbose.go b/lib/noverbose.go new file mode 100644 index 000000000..8cb4d73f9 --- /dev/null +++ b/lib/noverbose.go @@ -0,0 +1,7 @@ +//go:build !verbose + +package lib + +func Verbose() bool { + return false +} diff --git a/lib/trace.go b/lib/trace.go deleted file mode 100644 index 80b86c346..000000000 --- a/lib/trace.go +++ /dev/null @@ -1,7 +0,0 @@ -//go:build trace - -package lib - -func Trace() bool { - return true -} diff --git a/lib/verbose.go b/lib/verbose.go new file mode 100644 index 000000000..c4c80217f --- /dev/null +++ b/lib/verbose.go @@ -0,0 +1,7 @@ +//go:build verbose + +package lib + +func Verbose() bool { + return true +} diff --git a/net/edf/decode.go b/net/edf/decode.go index cd9740edd..7df80142b 100644 --- a/net/edf/decode.go +++ b/net/edf/decode.go @@ -37,31 +37,11 @@ func Decode(packet []byte, options Options) (_ any, _ []byte, ret error) { }() } - // Use pooled state instead of allocation state := getPooledStateDecode(options) defer putPooledStateDecode(state) state.decodeType = true - dec, packet, err := getDecoder(packet, state) - if err != nil { - return nil, nil, err - } - - if dec == nil { - return nil, packet, nil - } - state.decoder = dec - v := reflect.Indirect(reflect.New(dec.Type)) - - value, packet, err := dec.Decode(&v, packet, state) - if err != nil { - return nil, nil, fmt.Errorf("malformed EDF: %w", err) - } - - if value == nil { - return v.Interface(), packet, nil - } - return value.Interface(), packet, nil + return decodeWithStats(packet, state) } func getDecoder(packet []byte, state *stateDecode) (*decoder, []byte, error) { @@ -397,6 +377,66 @@ func decodeType(fold []byte, state *stateDecode) (*decoder, []byte, error) { case edtReg: return getRegDecoder(fold[1:], state) + + case edtPtr: + // unfold element type + decElem, f, err := decodeType(fold[1:], state) + if err != nil { + return nil, nil, fmt.Errorf("unable to unfold type (pointer): %s", err) + } + + ptrType := reflect.PointerTo(decElem.Type) + + fdec := func(value *reflect.Value, packet []byte, state *stateDecode) (*reflect.Value, []byte, error) { + if len(packet) == 0 { + return nil, nil, errDecodeEOD + } + + if packet[0] == edtNil { + // nil pointer + packet = packet[1:] + return nil, packet, nil + } + + if packet[0] != edtPtr { + return nil, nil, fmt.Errorf("incorrect pointer type %d", packet[0]) + } + packet = packet[1:] + + // use child state with decodeType = false (default) + if state.child == nil { + state.child = getPooledStateDecode(state.options) + } + state = state.child + + // decode element value + elem := reflect.Indirect(reflect.New(decElem.Type)) + _, packet, err := decElem.Decode(&elem, packet, state) + if err != nil { + return nil, nil, err + } + + // create pointer and set value + ptr := reflect.New(decElem.Type) + ptr.Elem().Set(elem) + + if value == nil { + value = &ptr + } else { + value.Set(ptr) + } + return value, packet, nil + } + + dec := decoder{ + Type: ptrType, + Decode: fdec, + } + if state.options.Cache != nil { + state.options.Cache.LoadOrStore(string(fold), &dec) + } + + return &dec, f, nil } if v, found := decoders.Load(fold[0]); found { diff --git a/net/edf/decode_ptr_test.go b/net/edf/decode_ptr_test.go new file mode 100644 index 000000000..69608d6c8 --- /dev/null +++ b/net/edf/decode_ptr_test.go @@ -0,0 +1,1877 @@ +package edf + +import ( + "fmt" + "reflect" + "testing" + "time" + + "ergo.services/ergo/gen" + "ergo.services/ergo/lib" +) + +// Basic pointer decode tests + +func TestDecodePtrInt(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := 42 + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*int) + if ok == false { + t.Fatalf("expected *int, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if *decodedPtr != 42 { + t.Fatalf("expected 42, got %d", *decodedPtr) + } +} + +func TestDecodePtrIntNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *int = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*int) + if ok == false { + t.Fatalf("expected *int, got %T", decoded) + } + if decodedPtr != nil { + t.Fatalf("expected nil, got %v", decodedPtr) + } +} + +func TestDecodePtrString(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + s := "hello" + ptr := &s + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*string) + if ok == false { + t.Fatalf("expected *string, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if *decodedPtr != "hello" { + t.Fatalf("expected 'hello', got '%s'", *decodedPtr) + } +} + +func TestDecodePtrStringNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *string = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*string) + if ok == false { + t.Fatalf("expected *string, got %T", decoded) + } + if decodedPtr != nil { + t.Fatalf("expected nil, got %v", decodedPtr) + } +} + +func TestDecodePtrFloat64(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + f := 3.14 + ptr := &f + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*float64) + if ok == false { + t.Fatalf("expected *float64, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if *decodedPtr != 3.14 { + t.Fatalf("expected 3.14, got %f", *decodedPtr) + } +} + +func TestDecodePtrFloat64Nil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *float64 = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*float64) + if ok == false { + t.Fatalf("expected *float64, got %T", decoded) + } + if decodedPtr != nil { + t.Fatalf("expected nil, got %v", decodedPtr) + } +} + +func TestDecodePtrBool(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := true + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*bool) + if ok == false { + t.Fatalf("expected *bool, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if *decodedPtr != true { + t.Fatalf("expected true, got %v", *decodedPtr) + } +} + +func TestDecodePtrBoolNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *bool = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*bool) + if ok == false { + t.Fatalf("expected *bool, got %T", decoded) + } + if decodedPtr != nil { + t.Fatalf("expected nil, got %v", decodedPtr) + } +} + +// Slice of pointers decode tests + +func TestDecodeSlicePtr(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + a, b2, c := 1, 2, 3 + slice := []*int{&a, nil, &b2, nil, &c} + + if err := Encode(slice, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedSlice, ok := decoded.([]*int) + if ok == false { + t.Fatalf("expected []*int, got %T", decoded) + } + if len(decodedSlice) != 5 { + t.Fatalf("expected len 5, got %d", len(decodedSlice)) + } + + // check values + if decodedSlice[0] == nil || *decodedSlice[0] != 1 { + t.Fatal("incorrect value at index 0") + } + if decodedSlice[1] != nil { + t.Fatal("expected nil at index 1") + } + if decodedSlice[2] == nil || *decodedSlice[2] != 2 { + t.Fatal("incorrect value at index 2") + } + if decodedSlice[3] != nil { + t.Fatal("expected nil at index 3") + } + if decodedSlice[4] == nil || *decodedSlice[4] != 3 { + t.Fatal("incorrect value at index 4") + } +} + +func TestDecodeSlicePtrEmpty(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + slice := []*int{} + + if err := Encode(slice, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedSlice, ok := decoded.([]*int) + if ok == false { + t.Fatalf("expected []*int, got %T", decoded) + } + if len(decodedSlice) != 0 { + t.Fatalf("expected len 0, got %d", len(decodedSlice)) + } +} + +func TestDecodeSlicePtrNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var slice []*int = nil + + if err := Encode(slice, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + // nil slice decodes to nil interface, not typed nil + if decoded != nil { + // if it decoded to a typed value, check it + decodedSlice, ok := decoded.([]*int) + if ok == false { + t.Fatalf("expected []*int or nil, got %T", decoded) + } + if decodedSlice != nil { + t.Fatal("expected nil slice") + } + } +} + +// Pointer to slice + +func TestDecodePtrSlice(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + slice := []int{1, 2, 3} + ptr := &slice + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*[]int) + if ok == false { + t.Fatalf("expected *[]int, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if reflect.DeepEqual(*decodedPtr, []int{1, 2, 3}) == false { + t.Fatalf("expected [1,2,3], got %v", *decodedPtr) + } +} + +func TestDecodePtrSliceNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *[]int = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*[]int) + if ok == false { + t.Fatalf("expected *[]int, got %T", decoded) + } + if decodedPtr != nil { + t.Fatalf("expected nil, got %v", decodedPtr) + } +} + +// Pointer to map + +func TestDecodePtrMap(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + m := map[string]int{"a": 1, "b": 2} + ptr := &m + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*map[string]int) + if ok == false { + t.Fatalf("expected *map[string]int, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if (*decodedPtr)["a"] != 1 || (*decodedPtr)["b"] != 2 { + t.Fatalf("incorrect map values: %v", *decodedPtr) + } +} + +func TestDecodePtrMapNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *map[string]int = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*map[string]int) + if ok == false { + t.Fatalf("expected *map[string]int, got %T", decoded) + } + if decodedPtr != nil { + t.Fatalf("expected nil, got %v", decodedPtr) + } +} + +// Map with pointer values + +func TestDecodeMapPtrValue(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v1, v2 := 10, 20 + m := map[string]*int{"a": &v1, "b": nil, "c": &v2} + + if err := Encode(m, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + dm, ok := decoded.(map[string]*int) + if ok == false { + t.Fatalf("expected map[string]*int, got %T", decoded) + } + + if dm["a"] == nil || *dm["a"] != 10 { + t.Fatal("incorrect value for key 'a'") + } + if dm["b"] != nil { + t.Fatal("expected nil for key 'b'") + } + if dm["c"] == nil || *dm["c"] != 20 { + t.Fatal("incorrect value for key 'c'") + } +} + +// Pointer in any field (struct) + +type testPtrStructWithAny struct { + Value any +} + +func init() { + RegisterTypeOf(testPtrStructWithAny{}) +} + +func TestDecodePtrInAny(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := 123 + s := testPtrStructWithAny{Value: &v} + + if err := Encode(s, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + ds, ok := decoded.(testPtrStructWithAny) + if ok == false { + t.Fatalf("expected testPtrStructWithAny, got %T", decoded) + } + + ptr, ok := ds.Value.(*int) + if ok == false { + t.Fatalf("expected *int in any, got %T", ds.Value) + } + if ptr == nil || *ptr != 123 { + t.Fatalf("expected 123, got %v", ptr) + } +} + +func TestDecodePtrInAnyNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + s := testPtrStructWithAny{Value: (*int)(nil)} + + if err := Encode(s, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + ds, ok := decoded.(testPtrStructWithAny) + if ok == false { + t.Fatalf("expected testPtrStructWithAny, got %T", decoded) + } + + // nil pointer in any becomes typed nil (*int)(nil) + ptr, ok := ds.Value.(*int) + if ok == false { + // might also be untyped nil + if ds.Value != nil { + t.Fatalf("expected *int or nil in any, got %T", ds.Value) + } + } else if ptr != nil { + t.Fatal("expected nil pointer") + } +} + +func TestDecodeSlicePtrInAny(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + a, c := 1, 3 + slice := []*int{&a, nil, &c} + s := testPtrStructWithAny{Value: slice} + + if err := Encode(s, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + ds, ok := decoded.(testPtrStructWithAny) + if ok == false { + t.Fatalf("expected testPtrStructWithAny, got %T", decoded) + } + + decodedSlice, ok := ds.Value.([]*int) + if ok == false { + t.Fatalf("expected []*int in any, got %T", ds.Value) + } + + if len(decodedSlice) != 3 { + t.Fatalf("expected len 3, got %d", len(decodedSlice)) + } + if decodedSlice[0] == nil || *decodedSlice[0] != 1 { + t.Fatal("incorrect value at index 0") + } + if decodedSlice[1] != nil { + t.Fatal("expected nil at index 1") + } + if decodedSlice[2] == nil || *decodedSlice[2] != 3 { + t.Fatal("incorrect value at index 2") + } +} + +// Pointer to nil slice (not nil pointer, pointer to nil slice value) + +func TestDecodePtrToNilSlice(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var slice []int = nil + ptr := &slice // pointer to nil slice + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*[]int) + if ok == false { + t.Fatalf("expected *[]int, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if *decodedPtr != nil { + t.Fatalf("expected nil slice, got %v", *decodedPtr) + } +} + +// Pointer to nil map + +func TestDecodePtrToNilMap(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var m map[string]int = nil + ptr := &m // pointer to nil map + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*map[string]int) + if ok == false { + t.Fatalf("expected *map[string]int, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if *decodedPtr != nil { + t.Fatalf("expected nil map, got %v", *decodedPtr) + } +} + +// Pointer to empty slice + +func TestDecodePtrToEmptySlice(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + slice := []int{} + ptr := &slice + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*[]int) + if ok == false { + t.Fatalf("expected *[]int, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if *decodedPtr == nil { + t.Fatal("expected non-nil slice") + } + if len(*decodedPtr) != 0 { + t.Fatalf("expected empty slice, got len %d", len(*decodedPtr)) + } +} + +// Corner case: pointer to pointer to slice element (nested through slice) + +func TestDecodeSlicePtrString(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + s1, s2 := "hello", "world" + slice := []*string{&s1, nil, &s2} + + if err := Encode(slice, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedSlice, ok := decoded.([]*string) + if ok == false { + t.Fatalf("expected []*string, got %T", decoded) + } + + if len(decodedSlice) != 3 { + t.Fatalf("expected len 3, got %d", len(decodedSlice)) + } + if decodedSlice[0] == nil || *decodedSlice[0] != "hello" { + t.Fatal("incorrect value at index 0") + } + if decodedSlice[1] != nil { + t.Fatal("expected nil at index 1") + } + if decodedSlice[2] == nil || *decodedSlice[2] != "world" { + t.Fatal("incorrect value at index 2") + } +} + +// Test malformed data + +func TestDecodePtrMalformedMissingValue(t *testing.T) { + // type descriptor for *int but no value bytes + packet := []byte{ + edtType, 0, 2, + edtPtr, edtInt, + // missing edtPtr or edtNil marker + } + + _, _, err := Decode(packet, Options{}) + if err == nil { + t.Fatal("expected error for malformed data") + } +} + +func TestDecodePtrMalformedInvalidMarker(t *testing.T) { + // type descriptor for *int with invalid marker + packet := []byte{ + edtType, 0, 2, + edtPtr, edtInt, + edtString, // wrong marker + } + + _, _, err := Decode(packet, Options{}) + if err == nil { + t.Fatal("expected error for invalid marker") + } +} + +// Large value through pointer + +func TestDecodePtrLargeInt(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := int64(9223372036854775807) // max int64 + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*int64) + if ok == false { + t.Fatalf("expected *int64, got %T", decoded) + } + if *decodedPtr != 9223372036854775807 { + t.Fatalf("expected max int64, got %d", *decodedPtr) + } +} + +// Slice of slices containing pointers + +func TestDecodeNestedSlicePtr(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + a, b2 := 1, 2 + c, d := 3, 4 + nested := [][]*int{ + {&a, nil, &b2}, + {&c, &d}, + } + + if err := Encode(nested, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + dn, ok := decoded.([][]*int) + if ok == false { + t.Fatalf("expected [][]*int, got %T", decoded) + } + + if len(dn) != 2 { + t.Fatalf("expected 2 outer slices, got %d", len(dn)) + } + + // first inner slice + if len(dn[0]) != 3 { + t.Fatalf("expected 3 elements in first slice, got %d", len(dn[0])) + } + if *dn[0][0] != 1 || dn[0][1] != nil || *dn[0][2] != 2 { + fmt.Printf("got: %v %v %v\n", dn[0][0], dn[0][1], dn[0][2]) + t.Fatal("incorrect values in first slice") + } + + // second inner slice + if len(dn[1]) != 2 { + t.Fatalf("expected 2 elements in second slice, got %d", len(dn[1])) + } + if *dn[1][0] != 3 || *dn[1][1] != 4 { + t.Fatal("incorrect values in second slice") + } +} + +// Pointer to binary ([]byte) + +func TestDecodePtrBinary(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + bin := []byte{1, 2, 3, 4, 5} + ptr := &bin + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*[]byte) + if ok == false { + t.Fatalf("expected *[]byte, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if reflect.DeepEqual(*decodedPtr, []byte{1, 2, 3, 4, 5}) == false { + t.Fatalf("expected [1,2,3,4,5], got %v", *decodedPtr) + } +} + +func TestDecodePtrBinaryNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *[]byte = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*[]byte) + if ok == false { + t.Fatalf("expected *[]byte, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +// Pointer to Ergo types + +func TestDecodePtrAtom(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + atom := gen.Atom("test_atom") + ptr := &atom + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.Atom) + if ok == false { + t.Fatalf("expected *gen.Atom, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if *decodedPtr != "test_atom" { + t.Fatalf("expected 'test_atom', got '%s'", *decodedPtr) + } +} + +func TestDecodePtrAtomNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *gen.Atom = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.Atom) + if ok == false { + t.Fatalf("expected *gen.Atom, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +func TestDecodePtrTime(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + tm := time.Date(2024, 1, 15, 10, 30, 0, 0, time.UTC) + ptr := &tm + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*time.Time) + if ok == false { + t.Fatalf("expected *time.Time, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if decodedPtr.Equal(tm) == false { + t.Fatalf("expected %v, got %v", tm, *decodedPtr) + } +} + +func TestDecodePtrTimeNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *time.Time = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*time.Time) + if ok == false { + t.Fatalf("expected *time.Time, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +// Array of pointers + +func TestDecodeArrayPtr(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + a, c := 1, 3 + arr := [3]*int{&a, nil, &c} + + if err := Encode(arr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedArr, ok := decoded.([3]*int) + if ok == false { + t.Fatalf("expected [3]*int, got %T", decoded) + } + + if decodedArr[0] == nil || *decodedArr[0] != 1 { + t.Fatal("incorrect value at index 0") + } + if decodedArr[1] != nil { + t.Fatal("expected nil at index 1") + } + if decodedArr[2] == nil || *decodedArr[2] != 3 { + t.Fatal("incorrect value at index 2") + } +} + +// Pointer to array + +func TestDecodePtrArray(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + arr := [3]int{1, 2, 3} + ptr := &arr + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*[3]int) + if ok == false { + t.Fatalf("expected *[3]int, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if *decodedPtr != [3]int{1, 2, 3} { + t.Fatalf("expected [1,2,3], got %v", *decodedPtr) + } +} + +func TestDecodePtrArrayNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *[3]int = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*[3]int) + if ok == false { + t.Fatalf("expected *[3]int, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +// Registered struct with pointer fields + +type testStructWithPtrField struct { + Name string + Value *int + Data *string +} + +func init() { + RegisterTypeOf(testStructWithPtrField{}) +} + +func TestDecodeStructWithPtrField(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := 42 + s := "hello" + st := testStructWithPtrField{ + Name: "test", + Value: &v, + Data: &s, + } + + if err := Encode(st, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + dst, ok := decoded.(testStructWithPtrField) + if ok == false { + t.Fatalf("expected testStructWithPtrField, got %T", decoded) + } + + if dst.Name != "test" { + t.Fatalf("expected Name='test', got '%s'", dst.Name) + } + if dst.Value == nil || *dst.Value != 42 { + t.Fatal("incorrect Value field") + } + if dst.Data == nil || *dst.Data != "hello" { + t.Fatal("incorrect Data field") + } +} + +func TestDecodeStructWithPtrFieldNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + st := testStructWithPtrField{ + Name: "test", + Value: nil, + Data: nil, + } + + if err := Encode(st, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + dst, ok := decoded.(testStructWithPtrField) + if ok == false { + t.Fatalf("expected testStructWithPtrField, got %T", decoded) + } + + if dst.Name != "test" { + t.Fatalf("expected Name='test', got '%s'", dst.Name) + } + if dst.Value != nil { + t.Fatal("expected nil Value field") + } + if dst.Data != nil { + t.Fatal("expected nil Data field") + } +} + +func TestDecodeStructWithPtrFieldMixed(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := 99 + st := testStructWithPtrField{ + Name: "mixed", + Value: &v, + Data: nil, // one nil, one non-nil + } + + if err := Encode(st, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + dst, ok := decoded.(testStructWithPtrField) + if ok == false { + t.Fatalf("expected testStructWithPtrField, got %T", decoded) + } + + if dst.Name != "mixed" { + t.Fatalf("expected Name='mixed', got '%s'", dst.Name) + } + if dst.Value == nil || *dst.Value != 99 { + t.Fatal("incorrect Value field") + } + if dst.Data != nil { + t.Fatal("expected nil Data field") + } +} + +// Additional integer pointer types + +func TestDecodePtrInt8(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := int8(127) + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*int8) + if ok == false { + t.Fatalf("expected *int8, got %T", decoded) + } + if *decodedPtr != 127 { + t.Fatalf("expected 127, got %d", *decodedPtr) + } +} + +func TestDecodePtrInt16(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := int16(32767) + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*int16) + if ok == false { + t.Fatalf("expected *int16, got %T", decoded) + } + if *decodedPtr != 32767 { + t.Fatalf("expected 32767, got %d", *decodedPtr) + } +} + +func TestDecodePtrInt32(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := int32(2147483647) + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*int32) + if ok == false { + t.Fatalf("expected *int32, got %T", decoded) + } + if *decodedPtr != 2147483647 { + t.Fatalf("expected 2147483647, got %d", *decodedPtr) + } +} + +func TestDecodePtrUint(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := uint(18446744073709551615) + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*uint) + if ok == false { + t.Fatalf("expected *uint, got %T", decoded) + } + if *decodedPtr != 18446744073709551615 { + t.Fatalf("expected max uint, got %d", *decodedPtr) + } +} + +func TestDecodePtrUint8(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := uint8(255) + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*uint8) + if ok == false { + t.Fatalf("expected *uint8, got %T", decoded) + } + if *decodedPtr != 255 { + t.Fatalf("expected 255, got %d", *decodedPtr) + } +} + +func TestDecodePtrUint16(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := uint16(65535) + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*uint16) + if ok == false { + t.Fatalf("expected *uint16, got %T", decoded) + } + if *decodedPtr != 65535 { + t.Fatalf("expected 65535, got %d", *decodedPtr) + } +} + +func TestDecodePtrUint32(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := uint32(4294967295) + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*uint32) + if ok == false { + t.Fatalf("expected *uint32, got %T", decoded) + } + if *decodedPtr != 4294967295 { + t.Fatalf("expected 4294967295, got %d", *decodedPtr) + } +} + +func TestDecodePtrUint64(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := uint64(18446744073709551615) + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*uint64) + if ok == false { + t.Fatalf("expected *uint64, got %T", decoded) + } + if *decodedPtr != 18446744073709551615 { + t.Fatalf("expected max uint64, got %d", *decodedPtr) + } +} + +func TestDecodePtrFloat32(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := float32(3.14) + ptr := &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*float32) + if ok == false { + t.Fatalf("expected *float32, got %T", decoded) + } + if *decodedPtr != float32(3.14) { + t.Fatalf("expected 3.14, got %f", *decodedPtr) + } +} + +// Ergo types pointers + +func TestDecodePtrPID(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + pid := gen.PID{Node: "test@localhost", ID: 12345, Creation: 1} + ptr := &pid + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.PID) + if ok == false { + t.Fatalf("expected *gen.PID, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if decodedPtr.Node != "test@localhost" || decodedPtr.ID != 12345 { + t.Fatalf("incorrect PID: %v", *decodedPtr) + } +} + +func TestDecodePtrPIDNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *gen.PID = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.PID) + if ok == false { + t.Fatalf("expected *gen.PID, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +func TestDecodePtrProcessID(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + pid := gen.ProcessID{Node: "test@localhost", Name: "myprocess"} + ptr := &pid + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.ProcessID) + if ok == false { + t.Fatalf("expected *gen.ProcessID, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if decodedPtr.Node != "test@localhost" || decodedPtr.Name != "myprocess" { + t.Fatalf("incorrect ProcessID: %v", *decodedPtr) + } +} + +func TestDecodePtrProcessIDNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *gen.ProcessID = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.ProcessID) + if ok == false { + t.Fatalf("expected *gen.ProcessID, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +func TestDecodePtrRef(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + ref := gen.Ref{Node: "test@localhost", Creation: 1, ID: [3]uint64{1, 2, 3}} + ptr := &ref + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.Ref) + if ok == false { + t.Fatalf("expected *gen.Ref, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if decodedPtr.Node != "test@localhost" || decodedPtr.ID != [3]uint64{1, 2, 3} { + t.Fatalf("incorrect Ref: %v", *decodedPtr) + } +} + +func TestDecodePtrRefNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *gen.Ref = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.Ref) + if ok == false { + t.Fatalf("expected *gen.Ref, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +func TestDecodePtrAlias(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + alias := gen.Alias{Node: "test@localhost", Creation: 1, ID: [3]uint64{4, 5, 6}} + ptr := &alias + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.Alias) + if ok == false { + t.Fatalf("expected *gen.Alias, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if decodedPtr.Node != "test@localhost" || decodedPtr.ID != [3]uint64{4, 5, 6} { + t.Fatalf("incorrect Alias: %v", *decodedPtr) + } +} + +func TestDecodePtrAliasNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *gen.Alias = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.Alias) + if ok == false { + t.Fatalf("expected *gen.Alias, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +func TestDecodePtrEvent(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + event := gen.Event{Node: "test@localhost", Name: "myevent"} + ptr := &event + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.Event) + if ok == false { + t.Fatalf("expected *gen.Event, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if decodedPtr.Node != "test@localhost" || decodedPtr.Name != "myevent" { + t.Fatalf("incorrect Event: %v", *decodedPtr) + } +} + +func TestDecodePtrEventNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *gen.Event = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*gen.Event) + if ok == false { + t.Fatalf("expected *gen.Event, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +// Pointer to registered struct + +type testRegisteredStruct struct { + X int + Y string +} + +func init() { + RegisterTypeOf(testRegisteredStruct{}) +} + +func TestDecodePtrToRegisteredStruct(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + st := testRegisteredStruct{X: 42, Y: "hello"} + ptr := &st + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*testRegisteredStruct) + if ok == false { + t.Fatalf("expected *testRegisteredStruct, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if decodedPtr.X != 42 || decodedPtr.Y != "hello" { + t.Fatalf("incorrect struct: %v", *decodedPtr) + } +} + +func TestDecodePtrToRegisteredStructNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *testRegisteredStruct = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*testRegisteredStruct) + if ok == false { + t.Fatalf("expected *testRegisteredStruct, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +// Slice of pointers to registered struct + +func TestDecodeSlicePtrToRegisteredStruct(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + st1 := testRegisteredStruct{X: 1, Y: "one"} + st2 := testRegisteredStruct{X: 2, Y: "two"} + slice := []*testRegisteredStruct{&st1, nil, &st2} + + if err := Encode(slice, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedSlice, ok := decoded.([]*testRegisteredStruct) + if ok == false { + t.Fatalf("expected []*testRegisteredStruct, got %T", decoded) + } + if len(decodedSlice) != 3 { + t.Fatalf("expected len 3, got %d", len(decodedSlice)) + } + if decodedSlice[0] == nil || decodedSlice[0].X != 1 { + t.Fatal("incorrect value at index 0") + } + if decodedSlice[1] != nil { + t.Fatal("expected nil at index 1") + } + if decodedSlice[2] == nil || decodedSlice[2].X != 2 { + t.Fatal("incorrect value at index 2") + } +} + +// Type alias for pointer: type myPtrType *bool + +type myPtrBool *bool + +func TestDecodeTypePtrAlias(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := true + var ptr myPtrBool = &v + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + // Note: decoded type will be *bool, not myPtrBool + // because EDF doesn't preserve type aliases + decodedPtr, ok := decoded.(*bool) + if ok == false { + t.Fatalf("expected *bool, got %T", decoded) + } + if decodedPtr == nil { + t.Fatal("expected non-nil pointer") + } + if *decodedPtr != true { + t.Fatal("expected true") + } +} + +func TestDecodeTypePtrAliasNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr myPtrBool = nil + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + decodedPtr, ok := decoded.(*bool) + if ok == false { + t.Fatalf("expected *bool, got %T", decoded) + } + if decodedPtr != nil { + t.Fatal("expected nil pointer") + } +} + +// Pointer to type alias (*myPtrBool = **bool) - should be rejected + +func TestDecodePtrToTypePtrAlias(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := true + var ptr myPtrBool = &v + ptrptr := &ptr // this is *myPtrBool which is **bool + + err := Encode(ptrptr, b, Options{}) + if err == nil { + t.Fatal("expected error for pointer to pointer type alias") + } + // Should get "nested pointer type is not supported" error +} + +// Pointer as map key + +func TestDecodeMapWithPtrKey(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + k := "key" + m := map[*string]int{&k: 42} + + if err := Encode(m, b, Options{}); err != nil { + t.Fatal(err) + } + + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + dm, ok := decoded.(map[*string]int) + if ok == false { + t.Fatalf("expected map[*string]int, got %T", decoded) + } + + if len(dm) != 1 { + t.Fatalf("expected 1 entry, got %d", len(dm)) + } + + for ptr, v := range dm { + if ptr == nil || *ptr != "key" { + t.Fatal("incorrect key") + } + if v != 42 { + t.Fatal("incorrect value") + } + } +} diff --git a/net/edf/edf.go b/net/edf/edf.go index 17441e6ae..c1f5757fc 100644 --- a/net/edf/edf.go +++ b/net/edf/edf.go @@ -12,6 +12,7 @@ type Options struct { RegCache *sync.Map // type/name => id (encoding), id => type (for decoding) ErrCache *sync.Map // error => id (for encoder), id => error (for decoder) Cache *sync.Map // common cache (caching reflect.Type => encoder, string([]byte) => decoder) + MaxDepth int // max encoding depth for pointers (default 100) } const ( @@ -40,6 +41,8 @@ const ( edtArray = byte(158) // 0x9e edtMap = byte(159) // 0x9f + edtPtr = byte(160) // 0xa0 + edtPID = byte(170) // 0xaa edtProcessID = byte(171) // 0xab edtAlias = byte(172) // 0xac diff --git a/net/edf/edf_fuzzing_test.go b/net/edf/edf_fuzzing_test.go new file mode 100644 index 000000000..3d701c7bc --- /dev/null +++ b/net/edf/edf_fuzzing_test.go @@ -0,0 +1,228 @@ +package edf + +import ( + "bytes" + "testing" + + "ergo.services/ergo/gen" + "ergo.services/ergo/lib" +) + +type fuzzInnerStruct struct { + Value int + Name string + Flag bool + FloatVal float64 +} + +type fuzzMiddleStruct struct { + Inner fuzzInnerStruct + InnerPtr *fuzzInnerStruct + Values []int + ValuesPtr []*int + Data map[string]int + DataPtr map[string]*int +} + +type fuzzComplexStruct struct { + Int8Val int8 + Int16Val int16 + Int32Val int32 + Int64Val int64 + Uint8Val uint8 + Uint16Val uint16 + Uint32Val uint32 + Uint64Val uint64 + Float32 float32 + Float64 float64 + Bool bool + String string + Binary []byte + + IntPtr *int + StringPtr *string + FloatPtr *float64 + BoolPtr *bool + + Atom gen.Atom + AtomPtr *gen.Atom + PID gen.PID + PIDPtr *gen.PID + ProcessID gen.ProcessID + Ref gen.Ref + Alias gen.Alias + Event gen.Event + + Middle fuzzMiddleStruct + MiddlePtr *fuzzMiddleStruct + + IntSlice []int + StringSlice []string + StructSlice []fuzzInnerStruct + StructPtrSlice []*fuzzInnerStruct + PtrSlice []*int + + StringIntMap map[string]int + IntStringMap map[int]string + StringStructMap map[string]fuzzInnerStruct + StringPtrMap map[string]*int +} + +func init() { + RegisterTypeOf(fuzzInnerStruct{}) + RegisterTypeOf(fuzzMiddleStruct{}) + RegisterTypeOf(fuzzComplexStruct{}) +} + +// compare by re-encoding decoded value and comparing bytes +func compareByEncode(original, decoded any) bool { + b1 := lib.TakeBuffer() + defer lib.ReleaseBuffer(b1) + b2 := lib.TakeBuffer() + defer lib.ReleaseBuffer(b2) + + if err := Encode(original, b1, Options{}); err != nil { + return false + } + if err := Encode(decoded, b2, Options{}); err != nil { + return false + } + return bytes.Equal(b1.B, b2.B) +} + +func FuzzComplexStruct(f *testing.F) { + // seed corpus - разные комбинации примитивов + f.Add( + int8(0), int16(0), int32(0), int64(0), + uint8(0), uint16(0), uint32(0), uint64(0), + float32(0), float64(0), false, "", []byte{}, + ) + f.Add( + int8(42), int16(1000), int32(100000), int64(1000000000), + uint8(255), uint16(65535), uint32(100000), uint64(1000000000), + float32(3.14), float64(3.14159), true, "hello", []byte{1, 2, 3}, + ) + f.Add( + int8(-128), int16(-32768), int32(-2147483648), int64(-9223372036854775808), + uint8(0), uint16(0), uint32(0), uint64(0), + float32(-1.5), float64(-1.5), false, "тест", []byte{0xff, 0xfe}, + ) + + f.Fuzz(func(t *testing.T, + i8 int8, i16 int16, i32 int32, i64 int64, + u8 uint8, u16 uint16, u32 uint32, u64 uint64, + f32 float32, f64 float64, b bool, s string, bin []byte, + ) { + // limit string for Atom (max 255 bytes) + if len(s) > 255 { + s = s[:255] + } + + // build pointers based on values + intVal := int(i64) + floatVal := f64 + boolVal := b + + innerVal := int(i32) + innerName := s + innerFloat := f64 + + original := fuzzComplexStruct{ + Int8Val: i8, + Int16Val: i16, + Int32Val: i32, + Int64Val: i64, + Uint8Val: u8, + Uint16Val: u16, + Uint32Val: u32, + Uint64Val: u64, + Float32: f32, + Float64: f64, + Bool: b, + String: s, + Binary: bin, + + IntPtr: &intVal, + StringPtr: &s, + FloatPtr: &floatVal, + BoolPtr: &boolVal, + + Atom: gen.Atom(s), + AtomPtr: nil, + PID: gen.PID{Node: gen.Atom(s), ID: u64, Creation: i64}, + PIDPtr: nil, + ProcessID: gen.ProcessID{Node: gen.Atom(s), Name: gen.Atom(s)}, + Ref: gen.Ref{Node: gen.Atom(s), Creation: i64, ID: [3]uint64{u64, u64, u64}}, + Alias: gen.Alias{Node: gen.Atom(s), Creation: i64, ID: [3]uint64{u64, u64, u64}}, + Event: gen.Event{Node: gen.Atom(s), Name: gen.Atom(s)}, + + Middle: fuzzMiddleStruct{ + Inner: fuzzInnerStruct{ + Value: innerVal, + Name: innerName, + Flag: b, + FloatVal: innerFloat, + }, + InnerPtr: &fuzzInnerStruct{Value: innerVal, Name: innerName, Flag: b, FloatVal: innerFloat}, + Values: []int{int(i64), int(i32), int(i16)}, + ValuesPtr: []*int{&intVal, nil, &innerVal}, + Data: map[string]int{s: int(i64)}, + DataPtr: map[string]*int{s: &intVal}, + }, + MiddlePtr: &fuzzMiddleStruct{ + Inner: fuzzInnerStruct{Value: innerVal, Name: innerName, Flag: b, FloatVal: innerFloat}, + InnerPtr: nil, + Values: []int{int(i64)}, + ValuesPtr: []*int{nil}, + Data: map[string]int{}, + DataPtr: map[string]*int{}, + }, + + IntSlice: []int{int(i64), int(i32), int(i16), int(i8)}, + StringSlice: []string{s, s}, + StructSlice: []fuzzInnerStruct{{Value: innerVal, Name: innerName, Flag: b, FloatVal: innerFloat}}, + StructPtrSlice: []*fuzzInnerStruct{nil, &fuzzInnerStruct{Value: innerVal, Name: innerName, Flag: b, FloatVal: innerFloat}}, + PtrSlice: []*int{&intVal, nil, &innerVal}, + + StringIntMap: map[string]int{s: int(i64)}, + IntStringMap: map[int]string{int(i64): s}, + StringStructMap: map[string]fuzzInnerStruct{s: {Value: innerVal, Name: innerName, Flag: b, FloatVal: innerFloat}}, + StringPtrMap: map[string]*int{s: &intVal}, + } + + buf := lib.TakeBuffer() + defer lib.ReleaseBuffer(buf) + + if err := Encode(original, buf, Options{}); err != nil { + t.Fatalf("encode error: %s", err) + } + + decoded, _, err := Decode(buf.B, Options{}) + if err != nil { + t.Fatalf("decode error: %s", err) + } + + decodedStruct, ok := decoded.(fuzzComplexStruct) + if ok == false { + t.Fatalf("expected fuzzComplexStruct, got %T", decoded) + } + + if compareByEncode(original, decodedStruct) == false { + t.Fatalf("mismatch after roundtrip") + } + }) +} + +// FuzzDecode - check decoder doesn't panic on random bytes +func FuzzDecode(f *testing.F) { + f.Add([]byte{edtNil}) + f.Add([]byte{edtBool, 1}) + f.Add([]byte{edtInt, 0, 0, 0, 0, 0, 0, 0, 42}) + f.Add([]byte{edtString, 0, 5, 'h', 'e', 'l', 'l', 'o'}) + f.Add([]byte{edtType, 0, 2, edtPtr, edtInt, edtNil}) + f.Add([]byte{edtType, 0, 2, edtPtr, edtInt, edtPtr, 0, 0, 0, 0, 0, 0, 0, 42}) + + f.Fuzz(func(t *testing.T, data []byte) { + Decode(data, Options{}) + }) +} diff --git a/net/edf/encode.go b/net/edf/encode.go index 8daf68d5a..0dcb78a32 100644 --- a/net/edf/encode.go +++ b/net/edf/encode.go @@ -11,19 +11,19 @@ import ( "ergo.services/ergo/lib" ) +const maxEncodeDepth = 100 + var ( - ErrBinaryTooLong = fmt.Errorf("binary too long - max allowed length is 2^32-1 bytes (4GB)") - ErrStringTooLong = fmt.Errorf("string too long - max allowed length is 2^16-1 (65535) bytes") - ErrAtomTooLong = fmt.Errorf("atom too long - max allowed length is 255 bytes") - ErrErrorTooLong = fmt.Errorf("error too long - max allowed length is 32767 bytes") + ErrBinaryTooLong = fmt.Errorf("binary too long - max allowed length is 2^32-1 bytes (4GB)") + ErrStringTooLong = fmt.Errorf("string too long - max allowed length is 2^16-1 (65535) bytes") + ErrAtomTooLong = fmt.Errorf("atom too long - max allowed length is 255 bytes") + ErrErrorTooLong = fmt.Errorf("error too long - max allowed length is 32767 bytes") + ErrMaxDepthExceeded = fmt.Errorf("max encoding depth exceeded (cyclic reference?)") ) type stateEncode struct { child *stateEncode - - // TODO loop detection (in slices) - //loop map[unsafe.Pointer]struct{} - //ptr unsafe.Pointer + depth int encodeType bool @@ -55,15 +55,8 @@ func Encode(x any, b *lib.Buffer, options Options) (ret error) { }() } - l := len(enc.Prefix) - if l > 1 && enc.Prefix[0] != edtReg { - buf := b.Extend(3) - buf[0] = edtType - binary.BigEndian.PutUint16(buf[1:3], uint16(l)) - } - - b.Append(enc.Prefix) - return enc.Encode(xv, b, state) + state.encodeType = true + return encodeWithStats(enc, xv, b, state) } func getEncoder(t reflect.Type, state *stateEncode) (*encoder, error) { @@ -92,6 +85,7 @@ func getEncoder(t reflect.Type, state *stateEncode) (*encoder, error) { cachedenc := &encoder{ Prefix: v.([]byte), // use cache ID (3 bytes only) instead of the full name Encode: enc.Encode, + Info: enc.Info, } if state.options.Cache == nil { return cachedenc, nil @@ -297,7 +291,63 @@ func getEncoder(t reflect.Type, state *stateEncode) (*encoder, error) { return enc, nil case reflect.Pointer: - return nil, fmt.Errorf("pointer type is not supported") + elemType := t.Elem() + // reject nested pointers + if elemType.Kind() == reflect.Pointer { + return nil, fmt.Errorf("nested pointer type is not supported") + } + + encElem, err := getEncoder(elemType, state) + if err != nil { + return nil, err + } + + elemPrefix := encElem.Prefix + if state.options.RegCache != nil { + if v, found := state.options.RegCache.Load(elemType); found { + elemPrefix = v.([]byte) + } + } + prefix := append([]byte{edtPtr}, elemPrefix...) + + fenc := func(value reflect.Value, b *lib.Buffer, state *stateEncode) error { + state.depth++ + maxDepth := state.options.MaxDepth + if maxDepth == 0 { + maxDepth = maxEncodeDepth + } + if state.depth > maxDepth { + return ErrMaxDepthExceeded + } + + if state.encodeType { + buf := b.Extend(3) + buf[0] = edtType + binary.BigEndian.PutUint16(buf[1:3], uint16(len(prefix))) + b.Append(prefix) + } + + if value.IsNil() { + state.depth-- + b.AppendByte(edtNil) + return nil + } + + b.AppendByte(edtPtr) + state.encodeType = false + err := encElem.Encode(value.Elem(), b, state) + state.depth-- + return err + } + + enc := &encoder{ + Prefix: prefix, + Encode: fenc, + } + if state.options.Cache != nil { + state.options.Cache.Store(t, enc) + } + return enc, nil } // look among the standard types diff --git a/net/edf/encode_ptr_test.go b/net/edf/encode_ptr_test.go new file mode 100644 index 000000000..a2ffbdc71 --- /dev/null +++ b/net/edf/encode_ptr_test.go @@ -0,0 +1,417 @@ +package edf + +import ( + "fmt" + "reflect" + "testing" + + "ergo.services/ergo/lib" +) + +// Basic pointer tests + +func TestEncodePtrInt(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := 42 + ptr := &v + + // type descriptor: [edtType][len=2][edtPtr][edtInt] + // value: [edtPtr][8 bytes for int64] + expect := []byte{ + edtType, 0, 2, // type header + edtPtr, edtInt, // type descriptor + edtPtr, // non-nil marker + 0, 0, 0, 0, 0, 0, 0, 42, // int value (big endian) + } + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +func TestEncodePtrIntNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *int = nil + + // type descriptor: [edtType][len=2][edtPtr][edtInt] + // value: [edtNil] + expect := []byte{ + edtType, 0, 2, // type header + edtPtr, edtInt, // type descriptor + edtNil, // nil marker + } + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +func TestEncodePtrString(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + s := "hello" + ptr := &s + + // type descriptor: [edtType][len=2][edtPtr][edtString] + // value: [edtPtr][len][string bytes] + expect := []byte{ + edtType, 0, 2, // type header + edtPtr, edtString, // type descriptor + edtPtr, // non-nil marker + 0, 5, // string len + 'h', 'e', 'l', 'l', 'o', + } + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +func TestEncodePtrStringNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *string = nil + + expect := []byte{ + edtType, 0, 2, + edtPtr, edtString, + edtNil, + } + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +func TestEncodePtrFloat64(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + f := 3.14 + ptr := &f + + expect := []byte{ + edtType, 0, 2, + edtPtr, edtFloat64, + edtPtr, + 0x40, 0x09, 0x1e, 0xb8, 0x51, 0xeb, 0x85, 0x1f, // 3.14 in IEEE 754 + } + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +func TestEncodePtrFloat64Nil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *float64 = nil + + expect := []byte{ + edtType, 0, 2, + edtPtr, edtFloat64, + edtNil, + } + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +func TestEncodePtrBool(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := true + ptr := &v + + expect := []byte{ + edtType, 0, 2, + edtPtr, edtBool, + edtPtr, + 1, // true + } + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +func TestEncodePtrBoolNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var ptr *bool = nil + + expect := []byte{ + edtType, 0, 2, + edtPtr, edtBool, + edtNil, + } + + if err := Encode(ptr, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +// Nested pointer (error case) + +func TestEncodePtrPtrInt(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := 42 + ptr := &v + ptrptr := &ptr + + err := Encode(ptrptr, b, Options{}) + if err == nil { + t.Fatal("expected error for nested pointer") + } +} + +// Slice of pointers + +func TestEncodeSlicePtr(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + a, b2, c := 1, 2, 3 + slice := []*int{&a, nil, &b2, nil, &c} + + // type: [edtType][len=3][edtSlice][edtPtr][edtInt] + // value: [edtSlice][count=5][ptr][1][nil][ptr][2][nil][ptr][3] + expect := []byte{ + edtType, 0, 3, + edtSlice, edtPtr, edtInt, + edtSlice, + 0, 0, 0, 5, // count + edtPtr, 0, 0, 0, 0, 0, 0, 0, 1, // &a + edtNil, // nil + edtPtr, 0, 0, 0, 0, 0, 0, 0, 2, // &b2 + edtNil, // nil + edtPtr, 0, 0, 0, 0, 0, 0, 0, 3, // &c + } + + if err := Encode(slice, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +func TestEncodeSlicePtrEmpty(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + slice := []*int{} + + expect := []byte{ + edtType, 0, 3, + edtSlice, edtPtr, edtInt, + edtSlice, + 0, 0, 0, 0, // count = 0 + } + + if err := Encode(slice, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +func TestEncodeSlicePtrNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + var slice []*int = nil + + expect := []byte{ + edtType, 0, 3, + edtSlice, edtPtr, edtInt, + edtNil, // nil slice + } + + if err := Encode(slice, b, Options{}); err != nil { + t.Fatal(err) + } + if reflect.DeepEqual(b.B, expect) == false { + fmt.Printf("exp %#v\n", expect) + fmt.Printf("got %#v\n", b.B) + t.Fatal("incorrect value") + } +} + +// Map with pointer values + +func TestEncodeMapPtrValue(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := 42 + m := map[string]*int{"key": &v} + + if err := Encode(m, b, Options{}); err != nil { + t.Fatal(err) + } + + // verify by decoding + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + dm, ok := decoded.(map[string]*int) + if ok == false { + t.Fatalf("expected map[string]*int, got %T", decoded) + } + if dm["key"] == nil || *dm["key"] != 42 { + t.Fatal("incorrect decoded value") + } +} + +func TestEncodeMapPtrValueNil(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + v := 42 + m := map[string]*int{"a": &v, "b": nil} + + if err := Encode(m, b, Options{}); err != nil { + t.Fatal(err) + } + + // verify by decoding + decoded, _, err := Decode(b.B, Options{}) + if err != nil { + t.Fatal(err) + } + + dm, ok := decoded.(map[string]*int) + if ok == false { + t.Fatalf("expected map[string]*int, got %T", decoded) + } + if dm["a"] == nil || *dm["a"] != 42 { + t.Fatal("incorrect decoded value for key 'a'") + } + if dm["b"] != nil { + t.Fatal("expected nil for key 'b'") + } +} + +// Max depth protection test + +// createNestedPointer creates N levels of pointer nesting without cycles +// e.g., depth=3 creates: *any -> *any -> *any -> int(42) +func createNestedPointer(depth int) any { + if depth == 0 { + return 42 + } + inner := createNestedPointer(depth - 1) + return &inner +} + +func TestEncodePtrMaxDepthError(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + // Build deeply nested pointer chain: 101 levels of *any + // This exceeds maxEncodeDepth (100) + val := createNestedPointer(101) + + err := Encode(val, b, Options{}) + if err == nil { + t.Fatal("expected error for max depth exceeded") + } + if err != ErrMaxDepthExceeded { + t.Fatalf("expected ErrMaxDepthExceeded, got: %v", err) + } +} + +func TestEncodePtrCustomMaxDepth(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + // Build 10 levels of nesting (no cycles) + val := createNestedPointer(10) + + // Should fail with MaxDepth=5 + err := Encode(val, b, Options{MaxDepth: 5}) + if err != ErrMaxDepthExceeded { + t.Fatalf("expected ErrMaxDepthExceeded with MaxDepth=5, got: %v", err) + } + + // Should succeed with MaxDepth=15 + b.Reset() + err = Encode(val, b, Options{MaxDepth: 15}) + if err != nil { + t.Fatalf("unexpected error with MaxDepth=15: %v", err) + } +} + +func TestEncodePtrCyclicReference(t *testing.T) { + b := lib.TakeBuffer() + defer lib.ReleaseBuffer(b) + + // Create cyclic reference: val points to itself + var val any = 42 + val = &val // val now contains *any pointing to val (cycle!) + + err := Encode(val, b, Options{}) + if err != ErrMaxDepthExceeded { + t.Fatalf("expected ErrMaxDepthExceeded for cyclic reference, got: %v", err) + } +} diff --git a/net/edf/init.go b/net/edf/init.go index 2836c8c40..a7c5144d8 100644 --- a/net/edf/init.go +++ b/net/edf/init.go @@ -5,7 +5,6 @@ import ( "reflect" "time" - "ergo.services/ergo/app/system/inspect" "ergo.services/ergo/gen" ) @@ -28,6 +27,12 @@ var ( gen.ApplicationDepends{}, + gen.Tracing{}, + gen.TracingFlags(0), + gen.TracingAttribute{}, + gen.TracingInfo{}, + gen.TracingExporterInfo{}, + gen.LoggerInfo{}, gen.ProcessFallback{}, gen.CronJobInfo{}, @@ -44,6 +49,8 @@ var ( gen.ApplicationOptionsExtra{}, gen.ApplicationInfo{}, gen.MetaInfo{}, + gen.EventInfo{}, + gen.LogField{}, gen.NetworkFlags{}, gen.NetworkProxyFlags{}, @@ -63,74 +70,13 @@ var ( gen.MessageEventStart{}, gen.MessageEventStop{}, - // inspector messages - - inspect.RequestInspectNode{}, - inspect.ResponseInspectNode{}, - inspect.MessageInspectNode{}, - - inspect.RequestInspectNetwork{}, - inspect.ResponseInspectNetwork{}, - inspect.MessageInspectNetwork{}, - - inspect.RequestInspectConnection{}, - inspect.ResponseInspectConnection{}, - inspect.MessageInspectConnection{}, - - inspect.RequestInspectProcessList{}, - inspect.ResponseInspectProcessList{}, - inspect.MessageInspectProcessList{}, - - inspect.RequestInspectLog{}, - inspect.ResponseInspectLog{}, - inspect.MessageInspectLogNode{}, - inspect.MessageInspectLogNetwork{}, - inspect.MessageInspectLogProcess{}, - inspect.MessageInspectLogMeta{}, - - inspect.RequestInspectProcess{}, - inspect.ResponseInspectProcess{}, - inspect.MessageInspectProcess{}, - - inspect.RequestInspectProcessState{}, - inspect.ResponseInspectProcessState{}, - inspect.MessageInspectProcessState{}, - - inspect.RequestInspectMeta{}, - inspect.ResponseInspectMeta{}, - inspect.MessageInspectMeta{}, - - inspect.RequestInspectMetaState{}, - inspect.ResponseInspectMetaState{}, - inspect.MessageInspectMetaState{}, - - inspect.RequestDoSend{}, - inspect.ResponseDoSend{}, - - inspect.RequestDoSendMeta{}, - inspect.ResponseDoSendMeta{}, - - inspect.RequestDoSendExit{}, - inspect.ResponseDoSendExit{}, - - inspect.RequestDoSendExitMeta{}, - inspect.ResponseDoSendExitMeta{}, - - inspect.RequestDoKill{}, - inspect.ResponseDoKill{}, - - inspect.RequestDoSetLogLevel{}, - inspect.RequestDoSetLogLevelProcess{}, - inspect.RequestDoSetLogLevelMeta{}, - inspect.ResponseDoSetLogLevel{}, - - inspect.RequestInspectApplicationList{}, - inspect.ResponseInspectApplicationList{}, - inspect.MessageInspectApplicationList{}, + gen.TracingPoint(0), + gen.TracingKind(0), + gen.TracingFlags(0), + gen.TracingSpan{}, - inspect.RequestInspectApplicationTree{}, - inspect.ResponseInspectApplicationTree{}, - inspect.MessageInspectApplicationTree{}, + gen.RegisteredTypeStats{}, + gen.RegisteredTypeInfo{}, } // register standard errors of the Ergo Framework @@ -154,137 +100,253 @@ var ( ) func init() { - // - // encoders - // - encoders.Store(reflect.TypeOf(gen.PID{}), &encoder{Prefix: []byte{edtPID}, Encode: encodePID}) - encoders.Store(reflect.TypeOf(gen.ProcessID{}), &encoder{Prefix: []byte{edtProcessID}, Encode: encodeProcessID}) - encoders.Store(reflect.TypeOf(gen.Ref{}), &encoder{Prefix: []byte{edtRef}, Encode: encodeRef}) - encoders.Store(reflect.TypeOf(gen.Alias{}), &encoder{Prefix: []byte{edtAlias}, Encode: encodeAlias}) - encoders.Store(reflect.TypeOf(gen.Event{}), &encoder{Prefix: []byte{edtEvent}, Encode: encodeEvent}) - encoders.Store(reflect.TypeOf(true), &encoder{Prefix: []byte{edtBool}, Encode: encodeBool}) - encoders.Store(reflect.TypeOf(gen.Atom("atom")), &encoder{Prefix: []byte{edtAtom}, Encode: encodeAtom}) - encoders.Store(reflect.TypeOf("string"), &encoder{Prefix: []byte{edtString}, Encode: encodeString}) - encoders.Store(reflect.TypeOf(int(0)), &encoder{Prefix: []byte{edtInt}, Encode: encodeInt}) - encoders.Store(reflect.TypeOf(int8(0)), &encoder{Prefix: []byte{edtInt8}, Encode: encodeInt8}) - encoders.Store(reflect.TypeOf(int16(0)), &encoder{Prefix: []byte{edtInt16}, Encode: encodeInt16}) - encoders.Store(reflect.TypeOf(int32(0)), &encoder{Prefix: []byte{edtInt32}, Encode: encodeInt32}) - encoders.Store(reflect.TypeOf(int64(0)), &encoder{Prefix: []byte{edtInt64}, Encode: encodeInt64}) - encoders.Store(reflect.TypeOf(uint(0)), &encoder{Prefix: []byte{edtUint}, Encode: encodeUint}) - encoders.Store(reflect.TypeOf(uint8(0)), &encoder{Prefix: []byte{edtUint8}, Encode: encodeUint8}) - encoders.Store(reflect.TypeOf(uint16(0)), &encoder{Prefix: []byte{edtUint16}, Encode: encodeUint16}) - encoders.Store(reflect.TypeOf(uint32(0)), &encoder{Prefix: []byte{edtUint32}, Encode: encodeUint32}) - encoders.Store(reflect.TypeOf(uint64(0)), &encoder{Prefix: []byte{edtUint64}, Encode: encodeUint64}) - encoders.Store(reflect.TypeOf([]byte(nil)), &encoder{Prefix: []byte{edtBinary}, Encode: encodeBinary}) - encoders.Store(reflect.TypeOf(float32(0.0)), &encoder{Prefix: []byte{edtFloat32}, Encode: encodeFloat32}) - encoders.Store(reflect.TypeOf(float64(0.0)), &encoder{Prefix: []byte{edtFloat64}, Encode: encodeFloat64}) - encoders.Store(reflect.TypeOf(time.Time{}), &encoder{Prefix: []byte{edtTime}, Encode: encodeTime}) - encoders.Store(anyType, &encoder{Prefix: []byte{edtAny}, Encode: encodeAny}) - - // error types - encoders.Store(errType, &encoder{Prefix: []byte{edtError}, Encode: encodeError}) - encoders.Store(reflect.TypeOf(fmt.Errorf("")), &encoder{Prefix: []byte{edtError}, Encode: encodeError}) - // wrapped error has a different type - encoders.Store(reflect.TypeOf(fmt.Errorf("%w", nil)), &encoder{Prefix: []byte{edtError}, Encode: encodeError}) - - // - // decoders - // - decPID := &decoder{reflect.TypeOf(gen.PID{}), decodePID} - decoders.Store(edtPID, decPID) - decoders.Store(decPID.Type, decPID) - - decProcessID := &decoder{reflect.TypeOf(gen.ProcessID{}), decodeProcessID} - decoders.Store(edtProcessID, decProcessID) - decoders.Store(decProcessID.Type, decProcessID) - - decRef := &decoder{reflect.TypeOf(gen.Ref{}), decodeRef} - decoders.Store(edtRef, decRef) - decoders.Store(decRef.Type, decRef) - - decAlias := &decoder{reflect.TypeOf(gen.Alias{}), decodeAlias} - decoders.Store(edtAlias, decAlias) - decoders.Store(decAlias.Type, decAlias) - - decEvent := &decoder{reflect.TypeOf(gen.Event{}), decodeEvent} - decoders.Store(edtEvent, decEvent) - decoders.Store(decEvent.Type, decEvent) - - decTime := &decoder{reflect.TypeOf(time.Time{}), decodeTime} - decoders.Store(edtTime, decTime) - decoders.Store(decTime.Type, decTime) - - decBool := &decoder{reflect.TypeOf(true), decodeBool} - decoders.Store(edtBool, decBool) - decoders.Store(decBool.Type, decBool) - - decAtom := &decoder{reflect.TypeOf(gen.Atom("atom")), decodeAtom} - decoders.Store(edtAtom, decAtom) - decoders.Store(decAtom.Type, decAtom) - - decString := &decoder{reflect.TypeOf("string"), decodeString} - decoders.Store(edtString, decString) - decoders.Store(decString.Type, decString) - - decInt := &decoder{reflect.TypeOf(int(0)), decodeInt} - decoders.Store(edtInt, decInt) - decoders.Store(decInt.Type, decInt) - - decInt8 := &decoder{reflect.TypeOf(int8(0)), decodeInt8} - decoders.Store(edtInt8, decInt8) - decoders.Store(decInt8.Type, decInt8) - - decInt16 := &decoder{reflect.TypeOf(int16(0)), decodeInt16} - decoders.Store(edtInt16, decInt16) - decoders.Store(decInt16.Type, decInt16) - - decInt32 := &decoder{reflect.TypeOf(int32(0)), decodeInt32} - decoders.Store(edtInt32, decInt32) - decoders.Store(decInt32.Type, decInt32) - - decInt64 := &decoder{reflect.TypeOf(int64(0)), decodeInt64} - decoders.Store(edtInt64, decInt64) - decoders.Store(decInt64.Type, decInt64) - - decUint := &decoder{reflect.TypeOf(uint(0)), decodeUint} - decoders.Store(edtUint, decUint) - decoders.Store(decUint.Type, decUint) - - decUint8 := &decoder{reflect.TypeOf(uint8(0)), decodeUint8} - decoders.Store(edtUint8, decUint8) - decoders.Store(decUint8.Type, decUint8) - - decUint16 := &decoder{reflect.TypeOf(uint16(0)), decodeUint16} - decoders.Store(edtUint16, decUint16) - decoders.Store(decUint16.Type, decUint16) - - decUint32 := &decoder{reflect.TypeOf(uint32(0)), decodeUint32} - decoders.Store(edtUint32, decUint32) - decoders.Store(decUint32.Type, decUint32) - - decUint64 := &decoder{reflect.TypeOf(uint64(0)), decodeUint64} - decoders.Store(edtUint64, decUint64) - decoders.Store(decUint64.Type, decUint64) - - decBinary := &decoder{reflect.TypeOf([]byte(nil)), decodeBinary} - decoders.Store(edtBinary, decBinary) - decoders.Store(decBinary.Type, decBinary) - - decFloat32 := &decoder{reflect.TypeOf(float32(0.0)), decodeFloat32} - decoders.Store(edtFloat32, decFloat32) - decoders.Store(decFloat32.Type, decFloat32) - - decFloat64 := &decoder{reflect.TypeOf(float64(0.0)), decodeFloat64} - decoders.Store(edtFloat64, decFloat64) - decoders.Store(decFloat64.Type, decFloat64) - - decAny := &decoder{anyType, decodeAny} - decoders.Store(edtAny, decAny) - decoders.Store(anyType, decAny) - - decErr := &decoder{errType, decodeError} - decoders.Store(edtError, decErr) - decoders.Store(decErr.Type, decErr) + // For each built-in type: encoder/decoder are stored first so that + // registerInfo (which calls measureZeroSize → Encode) can resolve them. + // The resulting *RegisteredTypeInfo is then attached back to the encoder + // and decoder via the Info field used by encodeWithStats/decodeWithStats. + + pidType := reflect.TypeOf(gen.PID{}) + pidEnc := &encoder{Prefix: []byte{edtPID}, Encode: encodePID} + encoders.Store(pidType, pidEnc) + pidDec := &decoder{Type: pidType, Decode: decodePID} + decoders.Store(edtPID, pidDec) + decoders.Store(pidType, pidDec) + pidInfo := registerInfo(pidType, "framework", "gen.PID") + pidEnc.Info = pidInfo + pidDec.Info = pidInfo + + processIDType := reflect.TypeOf(gen.ProcessID{}) + processIDEnc := &encoder{Prefix: []byte{edtProcessID}, Encode: encodeProcessID} + encoders.Store(processIDType, processIDEnc) + processIDDec := &decoder{Type: processIDType, Decode: decodeProcessID} + decoders.Store(edtProcessID, processIDDec) + decoders.Store(processIDType, processIDDec) + processIDInfo := registerInfo(processIDType, "framework", "gen.ProcessID") + processIDEnc.Info = processIDInfo + processIDDec.Info = processIDInfo + + refType := reflect.TypeOf(gen.Ref{}) + refEnc := &encoder{Prefix: []byte{edtRef}, Encode: encodeRef} + encoders.Store(refType, refEnc) + refDec := &decoder{Type: refType, Decode: decodeRef} + decoders.Store(edtRef, refDec) + decoders.Store(refType, refDec) + refInfo := registerInfo(refType, "framework", "gen.Ref") + refEnc.Info = refInfo + refDec.Info = refInfo + + aliasType := reflect.TypeOf(gen.Alias{}) + aliasEnc := &encoder{Prefix: []byte{edtAlias}, Encode: encodeAlias} + encoders.Store(aliasType, aliasEnc) + aliasDec := &decoder{Type: aliasType, Decode: decodeAlias} + decoders.Store(edtAlias, aliasDec) + decoders.Store(aliasType, aliasDec) + aliasInfo := registerInfo(aliasType, "framework", "gen.Alias") + aliasEnc.Info = aliasInfo + aliasDec.Info = aliasInfo + + eventType := reflect.TypeOf(gen.Event{}) + eventEnc := &encoder{Prefix: []byte{edtEvent}, Encode: encodeEvent} + encoders.Store(eventType, eventEnc) + eventDec := &decoder{Type: eventType, Decode: decodeEvent} + decoders.Store(edtEvent, eventDec) + decoders.Store(eventType, eventDec) + eventInfo := registerInfo(eventType, "framework", "gen.Event") + eventEnc.Info = eventInfo + eventDec.Info = eventInfo + + timeType := reflect.TypeOf(time.Time{}) + timeEnc := &encoder{Prefix: []byte{edtTime}, Encode: encodeTime} + encoders.Store(timeType, timeEnc) + timeDec := &decoder{Type: timeType, Decode: decodeTime} + decoders.Store(edtTime, timeDec) + decoders.Store(timeType, timeDec) + timeInfo := registerInfo(timeType, "framework", "time.Time") + timeEnc.Info = timeInfo + timeDec.Info = timeInfo + + boolType := reflect.TypeOf(true) + boolEnc := &encoder{Prefix: []byte{edtBool}, Encode: encodeBool} + encoders.Store(boolType, boolEnc) + boolDec := &decoder{Type: boolType, Decode: decodeBool} + decoders.Store(edtBool, boolDec) + decoders.Store(boolType, boolDec) + boolInfo := registerInfo(boolType, "bool", "bool") + boolEnc.Info = boolInfo + boolDec.Info = boolInfo + + atomType := reflect.TypeOf(gen.Atom("atom")) + atomEnc := &encoder{Prefix: []byte{edtAtom}, Encode: encodeAtom} + encoders.Store(atomType, atomEnc) + atomDec := &decoder{Type: atomType, Decode: decodeAtom} + decoders.Store(edtAtom, atomDec) + decoders.Store(atomType, atomDec) + atomInfo := registerInfo(atomType, "framework", "gen.Atom") + atomEnc.Info = atomInfo + atomDec.Info = atomInfo + + stringType := reflect.TypeOf("string") + stringEnc := &encoder{Prefix: []byte{edtString}, Encode: encodeString} + encoders.Store(stringType, stringEnc) + stringDec := &decoder{Type: stringType, Decode: decodeString} + decoders.Store(edtString, stringDec) + decoders.Store(stringType, stringDec) + stringInfo := registerInfo(stringType, "string", "string") + stringEnc.Info = stringInfo + stringDec.Info = stringInfo + + intType := reflect.TypeOf(int(0)) + intEnc := &encoder{Prefix: []byte{edtInt}, Encode: encodeInt} + encoders.Store(intType, intEnc) + intDec := &decoder{Type: intType, Decode: decodeInt} + decoders.Store(edtInt, intDec) + decoders.Store(intType, intDec) + intInfo := registerInfo(intType, "int", "int") + intEnc.Info = intInfo + intDec.Info = intInfo + + int8Type := reflect.TypeOf(int8(0)) + int8Enc := &encoder{Prefix: []byte{edtInt8}, Encode: encodeInt8} + encoders.Store(int8Type, int8Enc) + int8Dec := &decoder{Type: int8Type, Decode: decodeInt8} + decoders.Store(edtInt8, int8Dec) + decoders.Store(int8Type, int8Dec) + int8Info := registerInfo(int8Type, "int8", "int8") + int8Enc.Info = int8Info + int8Dec.Info = int8Info + + int16Type := reflect.TypeOf(int16(0)) + int16Enc := &encoder{Prefix: []byte{edtInt16}, Encode: encodeInt16} + encoders.Store(int16Type, int16Enc) + int16Dec := &decoder{Type: int16Type, Decode: decodeInt16} + decoders.Store(edtInt16, int16Dec) + decoders.Store(int16Type, int16Dec) + int16Info := registerInfo(int16Type, "int16", "int16") + int16Enc.Info = int16Info + int16Dec.Info = int16Info + + int32Type := reflect.TypeOf(int32(0)) + int32Enc := &encoder{Prefix: []byte{edtInt32}, Encode: encodeInt32} + encoders.Store(int32Type, int32Enc) + int32Dec := &decoder{Type: int32Type, Decode: decodeInt32} + decoders.Store(edtInt32, int32Dec) + decoders.Store(int32Type, int32Dec) + int32Info := registerInfo(int32Type, "int32", "int32") + int32Enc.Info = int32Info + int32Dec.Info = int32Info + + int64Type := reflect.TypeOf(int64(0)) + int64Enc := &encoder{Prefix: []byte{edtInt64}, Encode: encodeInt64} + encoders.Store(int64Type, int64Enc) + int64Dec := &decoder{Type: int64Type, Decode: decodeInt64} + decoders.Store(edtInt64, int64Dec) + decoders.Store(int64Type, int64Dec) + int64Info := registerInfo(int64Type, "int64", "int64") + int64Enc.Info = int64Info + int64Dec.Info = int64Info + + uintType := reflect.TypeOf(uint(0)) + uintEnc := &encoder{Prefix: []byte{edtUint}, Encode: encodeUint} + encoders.Store(uintType, uintEnc) + uintDec := &decoder{Type: uintType, Decode: decodeUint} + decoders.Store(edtUint, uintDec) + decoders.Store(uintType, uintDec) + uintInfo := registerInfo(uintType, "uint", "uint") + uintEnc.Info = uintInfo + uintDec.Info = uintInfo + + uint8Type := reflect.TypeOf(uint8(0)) + uint8Enc := &encoder{Prefix: []byte{edtUint8}, Encode: encodeUint8} + encoders.Store(uint8Type, uint8Enc) + uint8Dec := &decoder{Type: uint8Type, Decode: decodeUint8} + decoders.Store(edtUint8, uint8Dec) + decoders.Store(uint8Type, uint8Dec) + uint8Info := registerInfo(uint8Type, "uint8", "uint8") + uint8Enc.Info = uint8Info + uint8Dec.Info = uint8Info + + uint16Type := reflect.TypeOf(uint16(0)) + uint16Enc := &encoder{Prefix: []byte{edtUint16}, Encode: encodeUint16} + encoders.Store(uint16Type, uint16Enc) + uint16Dec := &decoder{Type: uint16Type, Decode: decodeUint16} + decoders.Store(edtUint16, uint16Dec) + decoders.Store(uint16Type, uint16Dec) + uint16Info := registerInfo(uint16Type, "uint16", "uint16") + uint16Enc.Info = uint16Info + uint16Dec.Info = uint16Info + + uint32Type := reflect.TypeOf(uint32(0)) + uint32Enc := &encoder{Prefix: []byte{edtUint32}, Encode: encodeUint32} + encoders.Store(uint32Type, uint32Enc) + uint32Dec := &decoder{Type: uint32Type, Decode: decodeUint32} + decoders.Store(edtUint32, uint32Dec) + decoders.Store(uint32Type, uint32Dec) + uint32Info := registerInfo(uint32Type, "uint32", "uint32") + uint32Enc.Info = uint32Info + uint32Dec.Info = uint32Info + + uint64Type := reflect.TypeOf(uint64(0)) + uint64Enc := &encoder{Prefix: []byte{edtUint64}, Encode: encodeUint64} + encoders.Store(uint64Type, uint64Enc) + uint64Dec := &decoder{Type: uint64Type, Decode: decodeUint64} + decoders.Store(edtUint64, uint64Dec) + decoders.Store(uint64Type, uint64Dec) + uint64Info := registerInfo(uint64Type, "uint64", "uint64") + uint64Enc.Info = uint64Info + uint64Dec.Info = uint64Info + + binaryType := reflect.TypeOf([]byte(nil)) + binaryEnc := &encoder{Prefix: []byte{edtBinary}, Encode: encodeBinary} + encoders.Store(binaryType, binaryEnc) + binaryDec := &decoder{Type: binaryType, Decode: decodeBinary} + decoders.Store(edtBinary, binaryDec) + decoders.Store(binaryType, binaryDec) + binaryInfo := registerInfo(binaryType, "binary", "[]byte") + binaryEnc.Info = binaryInfo + binaryDec.Info = binaryInfo + + float32Type := reflect.TypeOf(float32(0.0)) + float32Enc := &encoder{Prefix: []byte{edtFloat32}, Encode: encodeFloat32} + encoders.Store(float32Type, float32Enc) + float32Dec := &decoder{Type: float32Type, Decode: decodeFloat32} + decoders.Store(edtFloat32, float32Dec) + decoders.Store(float32Type, float32Dec) + float32Info := registerInfo(float32Type, "float32", "float32") + float32Enc.Info = float32Info + float32Dec.Info = float32Info + + float64Type := reflect.TypeOf(float64(0.0)) + float64Enc := &encoder{Prefix: []byte{edtFloat64}, Encode: encodeFloat64} + encoders.Store(float64Type, float64Enc) + float64Dec := &decoder{Type: float64Type, Decode: decodeFloat64} + decoders.Store(edtFloat64, float64Dec) + decoders.Store(float64Type, float64Dec) + float64Info := registerInfo(float64Type, "float64", "float64") + float64Enc.Info = float64Info + float64Dec.Info = float64Info + + anyEnc := &encoder{Prefix: []byte{edtAny}, Encode: encodeAny} + encoders.Store(anyType, anyEnc) + anyDec := &decoder{Type: anyType, Decode: decodeAny} + decoders.Store(edtAny, anyDec) + decoders.Store(anyType, anyDec) + anyInfo := registerInfo(anyType, "any", "any") + anyEnc.Info = anyInfo + anyDec.Info = anyInfo + + // error types: errType, *errors.errorString, *fmt.wrapError. They share + // the same encoder/decoder and the same RegisteredTypeInfo; counters + // aggregate across all error concrete types. + errEnc := &encoder{Prefix: []byte{edtError}, Encode: encodeError} + encoders.Store(errType, errEnc) + encoders.Store(reflect.TypeOf(fmt.Errorf("")), errEnc) + encoders.Store(reflect.TypeOf(fmt.Errorf("%w", nil)), errEnc) + errDec := &decoder{Type: errType, Decode: decodeError} + decoders.Store(edtError, errDec) + decoders.Store(errType, errDec) + errInfo := registerInfo(errType, "error", "error") + errEnc.Info = errInfo + errDec.Info = errInfo for _, t := range genTypes { err := RegisterTypeOf(t) diff --git a/net/edf/pools.go b/net/edf/pools.go index 711cdeaa4..c934bd48d 100644 --- a/net/edf/pools.go +++ b/net/edf/pools.go @@ -25,6 +25,7 @@ func getPooledStateEncode(options Options) *stateEncode { state := stateEncodePool.Get().(*stateEncode) // Reset state to clean condition state.child = nil + state.depth = 0 state.encodeType = false state.options = options return state diff --git a/net/edf/register.go b/net/edf/register.go index 2006bbe3d..bff0ced23 100644 --- a/net/edf/register.go +++ b/net/edf/register.go @@ -6,6 +6,9 @@ import ( "fmt" "math" "reflect" + "runtime" + "sort" + "strings" "sync" "sync/atomic" "time" @@ -14,15 +17,37 @@ import ( "ergo.services/ergo/lib" ) +const deprecationDocsURL = "https://docs.ergo.services/networking/network-transparency" + +// deprecation emits the legacy-API warning unless the call comes from +// framework-internal code (proxy in net/proto, edf init, registrar/handshake +// pre-registration, node-level atom caching). +func deprecation(name, replacement string) { + pc, _, _, _ := runtime.Caller(2) + if fn := runtime.FuncForPC(pc); fn != nil && + strings.HasPrefix(fn.Name(), "ergo.services/ergo/") { + return + } + lib.EmitDeprecation(nil, name, replacement, deprecationDocsURL) +} + type decoder struct { Type reflect.Type Decode func(*reflect.Value, []byte, *stateDecode) (*reflect.Value, []byte, error) + // Info is the per-proto type metadata. Populated for all registered + // and built-in types; nil for ad-hoc composite decoders constructed + // in getDecoder for unregistered slice/map/array types. + Info *gen.RegisteredTypeInfo } type encodeFunc func(value reflect.Value, b *lib.Buffer, state *stateEncode) error type encoder struct { Prefix []byte Encode encodeFunc + // Info is the per-proto type metadata. Populated for all registered + // and built-in types; nil for ad-hoc composite encoders constructed + // in getEncoder for unregistered slice/map/array types. + Info *gen.RegisteredTypeInfo } func regTypeName(t reflect.Type) string { @@ -30,6 +55,7 @@ func regTypeName(t reflect.Type) string { } func RegisterTypeOf(v any) error { + deprecation("edf.RegisterTypeOf", "node.Network().RegisterType") vov := reflect.ValueOf(v) tov := vov.Type() @@ -60,7 +86,12 @@ func RegisterTypeOf(v any) error { fenc := func(value reflect.Value, b *lib.Buffer, _ *stateEncode) error { v := value.Interface().(Marshaler) - buf := b.Extend(4) + // Record the offset for the length prefix instead of using the + // slice returned by Extend. If MarshalEDF triggers buffer + // reallocation, the Extend slice becomes stale but the offset + // remains valid against the new backing array. + lenPrefixOffset := b.Len() + b.Extend(4) l := b.Len() if err := v.MarshalEDF(b); err != nil { return err @@ -70,11 +101,9 @@ func RegisterTypeOf(v any) error { if int64(lenBinary) > int64(math.MaxUint32-1) { return ErrBinaryTooLong } - binary.BigEndian.PutUint32(buf, uint32(lenBinary)) + binary.BigEndian.PutUint32(b.B[lenPrefixOffset:], uint32(lenBinary)) return nil } - encoders.Store(tov, regEncoder(name, fenc)) - fdec := func(value *reflect.Value, packet []byte, state *stateDecode) (*reflect.Value, []byte, error) { if len(packet) < 4 { return nil, nil, errDecodeEOD @@ -98,10 +127,15 @@ func RegisterTypeOf(v any) error { packet = packet[l+4:] return value, packet, nil } - dec := &decoder{tov, fdec} + addRegCache(tov) + enc := regEncoder(name, fenc) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: fdec} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "marshaler", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case encoding.BinaryUnmarshaler: @@ -131,8 +165,6 @@ func RegisterTypeOf(v any) error { b.Append(bin) return nil } - encoders.Store(tov, regEncoder(name, fenc)) - fdec := func(value *reflect.Value, packet []byte, state *stateDecode) (*reflect.Value, []byte, error) { if len(packet) < 4 { return nil, nil, errDecodeEOD @@ -156,10 +188,15 @@ func RegisterTypeOf(v any) error { packet = packet[l+4:] return value, packet, nil } - dec := &decoder{tov, fdec} + addRegCache(tov) + enc := regEncoder(name, fenc) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: fdec} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "binarymarshaler", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil } @@ -181,115 +218,171 @@ func registerType(tov reflect.Type) error { switch tov.Kind() { case reflect.Bool: - encoders.Store(tov, regEncoder(name, encodeBool)) - dec := &decoder{tov, decodeBool} + addRegCache(tov) + enc := regEncoder(name, encodeBool) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeBool} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "bool", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Int: - encoders.Store(tov, regEncoder(name, encodeInt)) - dec := &decoder{tov, decodeInt} + addRegCache(tov) + enc := regEncoder(name, encodeInt) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeInt} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "int", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Int8: - encoders.Store(tov, regEncoder(name, encodeInt8)) - dec := &decoder{tov, decodeInt8} + addRegCache(tov) + enc := regEncoder(name, encodeInt8) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeInt8} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "int8", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Int16: - encoders.Store(tov, regEncoder(name, encodeInt16)) - dec := &decoder{tov, decodeInt16} + addRegCache(tov) + enc := regEncoder(name, encodeInt16) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeInt16} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "int16", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Int32: - encoders.Store(tov, regEncoder(name, encodeInt32)) - dec := &decoder{tov, decodeInt32} + addRegCache(tov) + enc := regEncoder(name, encodeInt32) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeInt32} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "int32", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Int64: - encoders.Store(tov, regEncoder(name, encodeInt64)) - dec := &decoder{tov, decodeInt64} + addRegCache(tov) + enc := regEncoder(name, encodeInt64) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeInt64} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "int64", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Uint: - encoders.Store(tov, regEncoder(name, encodeUint)) - dec := &decoder{tov, decodeUint} + addRegCache(tov) + enc := regEncoder(name, encodeUint) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeUint} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "uint", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Uint8: - encoders.Store(tov, regEncoder(name, encodeUint8)) - dec := &decoder{tov, decodeUint8} + addRegCache(tov) + enc := regEncoder(name, encodeUint8) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeUint8} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "uint8", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Uint16: - encoders.Store(tov, regEncoder(name, encodeUint16)) - dec := &decoder{tov, decodeUint16} + addRegCache(tov) + enc := regEncoder(name, encodeUint16) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeUint16} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "uint16", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Uint32: - encoders.Store(tov, regEncoder(name, encodeUint32)) - dec := &decoder{tov, decodeUint32} + addRegCache(tov) + enc := regEncoder(name, encodeUint32) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeUint32} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "uint32", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Uint64: - encoders.Store(tov, regEncoder(name, encodeUint64)) - dec := &decoder{tov, decodeUint64} + addRegCache(tov) + enc := regEncoder(name, encodeUint64) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeUint64} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "uint64", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Float32: - encoders.Store(tov, regEncoder(name, encodeFloat32)) - dec := &decoder{tov, decodeFloat32} + addRegCache(tov) + enc := regEncoder(name, encodeFloat32) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeFloat32} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "float32", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Float64: - encoders.Store(tov, regEncoder(name, encodeFloat64)) - dec := &decoder{tov, decodeFloat64} + addRegCache(tov) + enc := regEncoder(name, encodeFloat64) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeFloat64} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "float64", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.String: - encoders.Store(tov, regEncoder(name, encodeString)) - dec := &decoder{tov, decodeString} + addRegCache(tov) + enc := regEncoder(name, encodeString) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: decodeString} decoders.Store(name, dec) decoders.Store(tov, dec) - addRegCache(tov) + info := registerInfo(tov, "string", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil case reflect.Struct: @@ -331,7 +424,6 @@ func registerType(tov reflect.Type) error { } return nil } - encoders.Store(tov, regEncoder(name, fenc)) // decoder closure fdec := func(value *reflect.Value, packet []byte, state *stateDecode) (*reflect.Value, []byte, error) { @@ -355,8 +447,14 @@ func registerType(tov reflect.Type) error { } return value, packet, nil } - decoders.Store(name, &decoder{tov, fdec}) addRegCache(tov) + enc := regEncoder(name, fenc) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: fdec} + decoders.Store(name, dec) + info := registerInfo(tov, "struct", schemaFor(tov)) + enc.Info = info + dec.Info = info return nil @@ -398,7 +496,6 @@ func registerType(tov reflect.Type) error { } return nil } - encoders.Store(tov, regEncoder(name, fenc)) // decode closure fdec := func(value *reflect.Value, packet []byte, state *stateDecode) (*reflect.Value, []byte, error) { @@ -456,8 +553,14 @@ func registerType(tov reflect.Type) error { return value, packet, nil } - decoders.Store(name, &decoder{tov, fdec}) addRegCache(tov) + regEnc := regEncoder(name, fenc) + encoders.Store(tov, regEnc) + regDec := &decoder{Type: tov, Decode: fdec} + decoders.Store(name, regDec) + info := registerInfo(tov, "slice", schemaFor(tov)) + regEnc.Info = info + regDec.Info = info case reflect.Array: itemType := tov.Elem() @@ -487,7 +590,6 @@ func registerType(tov reflect.Type) error { } return nil } - encoders.Store(tov, regEncoder(name, fenc)) fdec := func(value *reflect.Value, packet []byte, state *stateDecode) (*reflect.Value, []byte, error) { if len(packet) == 0 { @@ -520,8 +622,14 @@ func registerType(tov reflect.Type) error { return value, packet, nil } - decoders.Store(name, &decoder{tov, fdec}) addRegCache(tov) + regEnc := regEncoder(name, fenc) + encoders.Store(tov, regEnc) + regDec := &decoder{Type: tov, Decode: fdec} + decoders.Store(name, regDec) + info := registerInfo(tov, "array", schemaFor(tov)) + regEnc.Info = info + regDec.Info = info case reflect.Map: typeKey := tov.Key() @@ -579,7 +687,6 @@ func registerType(tov reflect.Type) error { } return nil } - encoders.Store(tov, regEncoder(name, fenc)) fdec := func(value *reflect.Value, packet []byte, state *stateDecode) (*reflect.Value, []byte, error) { if len(packet) == 0 { @@ -647,8 +754,14 @@ func registerType(tov reflect.Type) error { return value, packet, nil } - decoders.Store(name, &decoder{tov, fdec}) addRegCache(tov) + enc := regEncoder(name, fenc) + encoders.Store(tov, enc) + dec := &decoder{Type: tov, Decode: fdec} + decoders.Store(name, dec) + info := registerInfo(tov, "map", schemaFor(tov)) + enc.Info = info + dec.Info = info default: return fmt.Errorf("type %v is not supported", tov) @@ -658,18 +771,95 @@ func registerType(tov reflect.Type) error { } func RegisterError(e error) error { + deprecation("edf.RegisterError", "node.Network().RegisterError") return addErrCache(e) } func RegisterAtom(a gen.Atom) error { + deprecation("edf.RegisterAtom", "node.Network().RegisterAtom") return addAtomCache(a) } var ( encoders sync.Map decoders sync.Map + + registeredTypes sync.Map // reflect.Type -> *gen.RegisteredTypeInfo. + registerOrder atomic.Uint64 ) +func registerInfo(t reflect.Type, kind, schema string) *gen.RegisteredTypeInfo { + custom := kind == "marshaler" || kind == "binarymarshaler" + + info := &gen.RegisteredTypeInfo{ + ID: registerOrder.Add(1), + Name: regTypeName(t), + Kind: kind, + Schema: schema, + MinSize: measureZeroSize(t, kind), + SizeVariable: custom || hasVariableSize(t, make(map[reflect.Type]bool)), + Stats: gen.RegisteredTypeStats{Enabled: statsEnabled}, + } + + if actual, loaded := registeredTypes.LoadOrStore(t, info); loaded { + return actual.(*gen.RegisteredTypeInfo) + } + return info +} + +func measureZeroSize(t reflect.Type, kind string) (size uint32) { + var fallback uint32 + switch { + case kind == "marshaler" || kind == "binarymarshaler": + // 3 bytes cached type-tag + 4 bytes length prefix + fallback = 7 + case t == anyType: + // nil interface encodes as edtNil (1 byte) + return 1 + case t == errType: + // nil error encodes as [0xff, 0xff] (2 bytes) + return 2 + } + defer func() { + if r := recover(); r != nil { + size = fallback + } + }() + v := reflect.New(t).Elem().Interface() + if v == nil { + return fallback + } + buf := lib.TakeBuffer() + defer lib.ReleaseBuffer(buf) + if err := Encode(v, buf, Options{RegCache: ®Cache}); err != nil { + return fallback + } + return uint32(buf.Len()) +} + +func hasVariableSize(t reflect.Type, visited map[reflect.Type]bool) bool { + if visited[t] { + return false + } + visited[t] = true + switch t.Kind() { + case reflect.String, reflect.Slice, reflect.Map, reflect.Pointer, reflect.Interface: + return true + case reflect.Array: + return hasVariableSize(t.Elem(), visited) + case reflect.Struct: + for i := 0; i < t.NumField(); i++ { + if hasVariableSize(t.Field(i).Type, visited) { + return true + } + } + } + return false +} + +// regEncoder creates a registered-type encoder. Info is attached separately +// by the caller after registerInfo has been called (which itself relies on +// the encoder already being stored in the encoders map). func regEncoder(name string, enc encodeFunc) *encoder { l := uint16(len(name)) if l > 4095 { @@ -727,6 +917,79 @@ func addRegCache(t reflect.Type) error { return nil } +// RegisteredTypes returns all registered EDF type metadata in registration order. +// Each entry is a value snapshot of the underlying *gen.RegisteredTypeInfo; +// counter fields reflect their values at snapshot time. +func RegisteredTypes() []gen.RegisteredTypeInfo { + var list []gen.RegisteredTypeInfo + registeredTypes.Range(func(_, v any) bool { + list = append(list, *v.(*gen.RegisteredTypeInfo)) + return true + }) + sort.Slice(list, func(i, j int) bool { + return list[i].ID < list[j].ID + }) + return list +} + +// RegisterTypesOf registers a batch with iterative resolve. Order-agnostic: +// types with unresolved dependencies are retried while progress is made. +// Returns error listing types that cannot be resolved after exhausting passes. +func RegisterTypesOf(types []any) error { + deprecation("edf.RegisterTypesOf", "node.Network().RegisterTypes") + pending := types + for len(pending) > 0 { + var next []any + progress := false + for _, t := range pending { + err := RegisterTypeOf(t) + if err == nil || err == gen.ErrTaken { + progress = true + continue + } + next = append(next, t) + } + if progress == false && len(next) > 0 { + names := make([]string, 0, len(next)) + for _, t := range next { + names = append(names, fmt.Sprintf("%T", t)) + } + return fmt.Errorf("unresolvable types: %s", strings.Join(names, ", ")) + } + pending = next + } + return nil +} + +// LookupType returns the reflect.Type for a registered type by name. +// The name can be a full EDF name ("#pkgpath/TypeName") or a short +// type name ("TypeName") which matches the first type with that suffix. +func LookupType(name string) (reflect.Type, bool) { + // Try exact match first + if v, ok := decoders.Load(name); ok { + return v.(*decoder).Type, true + } + + // Try short name match (suffix match on "/TypeName") + suffix := "/" + name + var found reflect.Type + decoders.Range(func(k, v any) bool { + s, ok := k.(string) + if ok == false { + return true + } + if len(s) > len(suffix) && s[len(s)-len(suffix):] == suffix { + found = v.(*decoder).Type + return false + } + return true + }) + if found != nil { + return found, true + } + return nil, false +} + func GetRegCache() map[uint16]string { cache := make(map[uint16]string) regCache.Range(func(k, v any) bool { diff --git a/net/edf/register_marshaler_test.go b/net/edf/register_marshaler_test.go new file mode 100644 index 000000000..d8334e116 --- /dev/null +++ b/net/edf/register_marshaler_test.go @@ -0,0 +1,77 @@ +package edf + +// Verifies that Encode/Decode roundtrip works correctly when MarshalEDF +// writes enough data to trigger lib.Buffer reallocation. A small initial +// buffer capacity (64 bytes) guarantees that MarshalEDF's Write call +// triggers append-reallocation for any non-trivial payload. + +import ( + "bytes" + "io" + "testing" + + "ergo.services/ergo/gen" + "ergo.services/ergo/lib" +) + +type largePayload struct { + Data []byte +} + +func (lp largePayload) MarshalEDF(w io.Writer) error { + _, err := w.Write(lp.Data) + return err +} + +func (lp *largePayload) UnmarshalEDF(data []byte) error { + lp.Data = make([]byte, len(data)) + copy(lp.Data, data) + return nil +} + +func TestMarshalerBufferReallocation(t *testing.T) { + if err := RegisterTypeOf(largePayload{}); err != nil && err != gen.ErrTaken { + t.Fatal(err) + } + + // Small payload that fits without reallocation, and a large payload + // that forces the buffer to grow. + sizes := []struct { + name string + n int + }{ + {"small", 100}, + {"large", lib.DefaultBufferLength * 2}, + } + + for _, tc := range sizes { + t.Run(tc.name, func(t *testing.T) { + data := make([]byte, tc.n) + for i := range data { + data[i] = byte(i % 251) + } + original := largePayload{Data: data} + + // Small capacity forces MarshalEDF to trigger buffer reallocation. + buf := &lib.Buffer{B: make([]byte, 0, 64)} + + if err := Encode(original, buf, Options{}); err != nil { + t.Fatalf("Encode: %v", err) + } + + decoded, _, err := Decode(buf.B, Options{}) + if err != nil { + t.Fatalf("Decode: %v (encoded %d bytes)", err, buf.Len()) + } + + got, ok := decoded.(largePayload) + if !ok { + t.Fatalf("expected largePayload, got %T", decoded) + } + if !bytes.Equal(got.Data, original.Data) { + t.Fatalf("roundtrip mismatch: got %d bytes, want %d bytes", + len(got.Data), len(original.Data)) + } + }) + } +} diff --git a/net/edf/schema.go b/net/edf/schema.go new file mode 100644 index 000000000..13af37999 --- /dev/null +++ b/net/edf/schema.go @@ -0,0 +1,96 @@ +package edf + +import ( + "fmt" + "reflect" + "strings" +) + +// schemaFor returns a Go-syntax representation of t's structure. +// Structs expand one level (field name + field type by name); nested +// struct fields are referred to by name only. +func schemaFor(t reflect.Type) string { + switch t.Kind() { + case reflect.Bool, reflect.String, + reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, + reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, + reflect.Float32, reflect.Float64: + return t.Kind().String() + + case reflect.Slice, reflect.Array, reflect.Map, reflect.Pointer: + return typeRefName(t) + + case reflect.Struct: + var sb strings.Builder + if t.Name() != "" { + sb.WriteString(t.Name()) + sb.WriteString(" ") + } + sb.WriteString("struct {\n") + nf := t.NumField() + for i := 0; i < nf; i++ { + f := t.Field(i) + sb.WriteString(" ") + sb.WriteString(f.Name) + sb.WriteString(" ") + sb.WriteString(typeRefName(f.Type)) + if path := typeRefPath(f.Type); path != "" { + sb.WriteString(" // ") + sb.WriteString(path) + } + sb.WriteString("\n") + } + sb.WriteString("}") + return sb.String() + + case reflect.Interface: + if t.Name() != "" { + return t.Name() + } + return "any" + } + return t.String() +} + +// typeRefName produces a short reference for a type used inside a schema. +func typeRefName(t reflect.Type) string { + if t.Name() != "" { + return t.Name() + } + switch t.Kind() { + case reflect.Slice: + return "[]" + typeRefName(t.Elem()) + case reflect.Array: + return fmt.Sprintf("[%d]%s", t.Len(), typeRefName(t.Elem())) + case reflect.Map: + return "map[" + typeRefName(t.Key()) + "]" + typeRefName(t.Elem()) + case reflect.Pointer: + return "*" + typeRefName(t.Elem()) + } + return t.String() +} + +// typeRefPath returns the fully-qualified #path/Name for a struct field +// type, used as an inline comment in the schema. Empty string for built-ins. +func typeRefPath(t reflect.Type) string { + if t.Name() != "" && t.PkgPath() != "" { + return "#" + t.PkgPath() + "/" + t.Name() + } + switch t.Kind() { + case reflect.Slice, reflect.Array, reflect.Pointer: + return typeRefPath(t.Elem()) + case reflect.Map: + keyP := typeRefPath(t.Key()) + valP := typeRefPath(t.Elem()) + if keyP != "" && valP != "" { + return "[" + keyP + "]" + valP + } + if valP != "" { + return valP + } + if keyP != "" { + return keyP + } + } + return "" +} diff --git a/net/edf/typestats_off.go b/net/edf/typestats_off.go new file mode 100644 index 000000000..b486d0c43 --- /dev/null +++ b/net/edf/typestats_off.go @@ -0,0 +1,44 @@ +//go:build !typestats + +package edf + +import ( + "fmt" + "reflect" + + "ergo.services/ergo/lib" +) + +// statsEnabled is false without -tags=typestats. Used at type +// registration to leave RegisteredTypeStats.Enabled as zero value. +const statsEnabled = false + +// encodeWithStats is a pass-through wrapper without -tags=typestats. +// The Go inliner reduces this to a direct call to enc.Encode. +func encodeWithStats(enc *encoder, xv reflect.Value, b *lib.Buffer, state *stateEncode) error { + return enc.Encode(xv, b, state) +} + +// decodeWithStats performs full root-level decoding without recording stats. +func decodeWithStats(packet []byte, state *stateDecode) (any, []byte, error) { + dec, packet, err := getDecoder(packet, state) + if err != nil { + return nil, nil, err + } + if dec == nil { + return nil, packet, nil + } + + state.decoder = dec + v := reflect.Indirect(reflect.New(dec.Type)) + + value, packet, err := dec.Decode(&v, packet, state) + if err != nil { + return nil, nil, fmt.Errorf("malformed EDF: %w", err) + } + + if value == nil { + return v.Interface(), packet, nil + } + return value.Interface(), packet, nil +} diff --git a/net/edf/typestats_on.go b/net/edf/typestats_on.go new file mode 100644 index 000000000..9b6e40bed --- /dev/null +++ b/net/edf/typestats_on.go @@ -0,0 +1,66 @@ +//go:build typestats + +package edf + +import ( + "fmt" + "reflect" + "sync/atomic" + + "ergo.services/ergo/lib" +) + +// statsEnabled is set to true under -tags=typestats. Used at type +// registration to mark RegisteredTypeStats.Enabled. +const statsEnabled = true + +// encodeWithStats wraps enc.Encode and records the operation when +// enc.Info is populated. Composite ad-hoc encoders without Info skip +// the stats path entirely. +func encodeWithStats(enc *encoder, xv reflect.Value, b *lib.Buffer, state *stateEncode) error { + if enc.Info == nil { + return enc.Encode(xv, b, state) + } + startLen := b.Len() + err := enc.Encode(xv, b, state) + if err != nil { + return err + } + atomic.AddInt64(&enc.Info.Stats.Encoded, 1) + atomic.AddInt64(&enc.Info.Stats.EncodedBytes, int64(b.Len()-startLen)) + return nil +} + +// decodeWithStats performs full root-level decoding (getDecoder + dec.Decode + +// .Interface() conversion) and records the operation. startLen is captured +// before getDecoder so that DecodedBytes includes the type-prefix bytes +// consumed by getDecoder, matching how EncodedBytes includes the prefix. +func decodeWithStats(packet []byte, state *stateDecode) (any, []byte, error) { + startLen := len(packet) + + dec, packet, err := getDecoder(packet, state) + if err != nil { + return nil, nil, err + } + if dec == nil { + return nil, packet, nil + } + + state.decoder = dec + v := reflect.Indirect(reflect.New(dec.Type)) + + value, packet, err := dec.Decode(&v, packet, state) + if err != nil { + return nil, nil, fmt.Errorf("malformed EDF: %w", err) + } + + if dec.Info != nil { + atomic.AddInt64(&dec.Info.Stats.Decoded, 1) + atomic.AddInt64(&dec.Info.Stats.DecodedBytes, int64(startLen-len(packet))) + } + + if value == nil { + return v.Interface(), packet, nil + } + return value.Interface(), packet, nil +} diff --git a/net/handshake/accept.go b/net/handshake/accept.go index 50dc95c61..c3f146a76 100644 --- a/net/handshake/accept.go +++ b/net/handshake/accept.go @@ -1,6 +1,7 @@ package handshake import ( + "crypto/hmac" "crypto/sha1" "crypto/sha256" "crypto/tls" @@ -107,8 +108,29 @@ func (h *handshake) Accept(node gen.NodeHandshake, conn net.Conn, options gen.Ha return result, fmt.Errorf("incorrect digest (accept stage 'introduce')") } + // deterministic connection ID (unconditional) + connID := generateConnectionID( + node.Name(), node.Creation(), + intro.Node, intro.Creation, + options.Cookie, + ) + + // collision detection + rejection (Erlang-style, flag-gated) + if options.Flags.EnableSimultaneousConnect == true && + intro.Flags.EnableSimultaneousConnect == true && + options.CheckPending != nil && options.CheckPending(intro.Node) { + // simultaneous connect detected + if string(node.Name()) < string(intro.Node) { + // our node name is smaller -> our outgoing wins -> reject this incoming + h.writeMessage(conn, MessageReject{Reason: "simultaneous"}) + return result, fmt.Errorf("rejected incoming from %s (simultaneous connect)", intro.Node) + } + // our node name is larger -> their outgoing wins -> accept this incoming + // our connect() will handle cleanup when it finishes + } + accept := MessageAccept{} - accept.ID = lib.RandomString(32) + accept.ID = connID accept.PoolSize = h.poolsize accept.PoolDSN = append(accept.PoolDSN, conn.LocalAddr().String()) if err := h.writeMessage(conn, accept); err != nil { @@ -149,10 +171,14 @@ func (h *handshake) Accept(node gen.NodeHandshake, conn net.Conn, options gen.Ha result.PeerMaxMessageSize = intro.MaxMessageSize result.NodeFlags = options.Flags result.NodeMaxMessageSize = options.MaxMessageSize + result.PoolSize = h.poolsize + result.PoolDSN = accept.PoolDSN result.Tail = tail + _, isTLS := conn.(*tls.Conn) custom := ConnectionOptions{ PoolSize: h.poolsize, + TLS: isTLS, EncodeAtomCache: h.makeEncodeAtomCache(intro2.AtomCache), EncodeRegCache: h.makeEncodeRegCache(intro2.RegCache), EncodeErrCache: h.makeEncodeErrCache(intro2.ErrCache), @@ -173,3 +199,16 @@ func (h *handshake) getLocalTLSFingerprint(conn net.Conn, cm gen.CertManager) [] fp := sha1.Sum(cert.Certificate[0]) return fp[:] } + +func generateConnectionID(nameA gen.Atom, creationA int64, + nameB gen.Atom, creationB int64, cookie string) string { + // canonical ordering: smaller name first + first := fmt.Sprintf("%s:%d", nameA, creationA) + second := fmt.Sprintf("%s:%d", nameB, creationB) + if string(nameA) > string(nameB) { + first, second = second, first + } + mac := hmac.New(sha256.New, []byte(cookie)) + mac.Write([]byte(first + ":" + second)) + return fmt.Sprintf("%x", mac.Sum(nil)) +} diff --git a/net/handshake/handshake.go b/net/handshake/handshake.go index 9ec78a036..e8d64ed62 100644 --- a/net/handshake/handshake.go +++ b/net/handshake/handshake.go @@ -58,6 +58,10 @@ func (h *handshake) Version() gen.Version { } } +func (h *handshake) Reject(conn net.Conn, reason string) error { + return h.writeMessage(conn, MessageReject{Reason: reason}) +} + func (h *handshake) writeMessage(conn net.Conn, message any) error { buf := lib.TakeBuffer() defer lib.ReleaseBuffer(buf) diff --git a/net/handshake/start.go b/net/handshake/start.go index d5aa15536..2d717cdb2 100644 --- a/net/handshake/start.go +++ b/net/handshake/start.go @@ -37,8 +37,13 @@ func (h *handshake) Start(node gen.NodeHandshake, conn net.Conn, options gen.Han return result, err } - hello2, ok := v.(MessageHello) - if ok == false { + var hello2 MessageHello + switch msg := v.(type) { + case MessageReject: + return result, fmt.Errorf("rejected: %s", msg.Reason) + case MessageHello: + hello2 = msg + default: return result, fmt.Errorf("malformed handshake Hello message") } hash = sha256.New() @@ -84,8 +89,13 @@ func (h *handshake) Start(node gen.NodeHandshake, conn net.Conn, options gen.Han return result, err } - accept, ok := v.(MessageAccept) - if ok == false { + var accept MessageAccept + switch msg := v.(type) { + case MessageAccept: + accept = msg + case MessageReject: + return result, fmt.Errorf("rejected: %s", msg.Reason) + default: return result, fmt.Errorf("malformed handshake Accept message") } @@ -117,11 +127,15 @@ func (h *handshake) Start(node gen.NodeHandshake, conn net.Conn, options gen.Han result.PeerMaxMessageSize = intro2.MaxMessageSize result.NodeFlags = options.Flags result.NodeMaxMessageSize = options.MaxMessageSize + result.PoolSize = accept.PoolSize + result.PoolDSN = accept.PoolDSN result.Tail = tail + _, isTLS := conn.(*tls.Conn) custom := ConnectionOptions{ PoolSize: accept.PoolSize, PoolDSN: accept.PoolDSN, + TLS: isTLS, EncodeAtomCache: h.makeEncodeAtomCache(intro.AtomCache), EncodeRegCache: h.makeEncodeRegCache(intro.RegCache), EncodeErrCache: h.makeEncodeErrCache(intro.ErrCache), diff --git a/net/handshake/types.go b/net/handshake/types.go index 1cc65fecc..f19bc05c2 100644 --- a/net/handshake/types.go +++ b/net/handshake/types.go @@ -1,9 +1,10 @@ package handshake import ( + "sync" + "ergo.services/ergo/gen" "ergo.services/ergo/net/edf" - "sync" ) const ( @@ -55,9 +56,14 @@ type MessageAccept struct { DigestCert string } +type MessageReject struct { + Reason string +} + type ConnectionOptions struct { PoolSize int PoolDSN []string + TLS bool EncodeAtomCache *sync.Map EncodeRegCache *sync.Map @@ -66,6 +72,11 @@ type ConnectionOptions struct { DecodeAtomCache *sync.Map DecodeRegCache *sync.Map DecodeErrCache *sync.Map + + SoftwareKeepAliveMisses int + FragmentSize int + FragmentTimeout int + MaxFragmentAssemblies int } func init() { @@ -74,6 +85,7 @@ func init() { MessageJoin{}, MessageIntroduce{}, MessageAccept{}, + MessageReject{}, } for _, t := range types { diff --git a/net/proto/connection.go b/net/proto/connection.go index 4e1452932..182e76220 100644 --- a/net/proto/connection.go +++ b/net/proto/connection.go @@ -23,6 +23,14 @@ const ( // lib.Buffer has 4096 of capacity // 256 messages could have at least 1Mb of allocated memory limitMemRecvQueues int64 = 1024 * 1024 * 512 + + skewRingSize = 16 + + // routeQueuesPerConn is the number of per-connection queues that drain + // protoMessageAny payloads. Matches the default TM shard count so the + // queue index equals the TM shard index for any given target. + // Must be a power of two. + routeQueuesPerConn = 16 ) type connection struct { @@ -44,6 +52,7 @@ type connection struct { pool_dsn []string pool_size int + tls bool pool_mutex sync.RWMutex pool []*pool_item @@ -51,6 +60,11 @@ type connection struct { recvQueues []lib.QueueMPSC allocatedInQueues int64 + // route queues drain protoMessageAny payloads (Link/Monitor/Spawn/...) so + // that decoding goroutines never block on synchronous core/TM/spawn work. + // Aligned with TM shard count and hashing so each queue maps to one shard. + routeQueues [routeQueuesPerConn]lib.QueueMPSC + encodeOptions edf.Options decodeOptions edf.Options @@ -64,16 +78,80 @@ type connection struct { transitIn uint64 transitOut uint64 + reconnections uint64 + + softwareKeepAlive bool + softwareKeepAliveMessage []byte + softwareKeepAlivePeriod time.Duration // how often I send + softwareKeepAliveMisses int + softwareKeepAliveTimeout time.Duration // how long I wait (peer period * misses) + + // clock skew measurement + clockSkew bool + + // tracing propagation + tracing bool + + // fragmentation + fragmentation bool + fragmentSize int + fragmentTimeout time.Duration + maxFragmentAssemblies int + nextSequenceID atomic.Uint32 + + // ordered fragment assembly (per recv queue, no mutex) + orderedFragments []map[uint32]*fragmentAssembly + + // unordered fragment assembly (order = 0 only) + sharedFragMu sync.Mutex + sharedFragments map[uint32]*fragmentAssembly + sharedFragTimer *time.Timer + + // fragment statistics + fragmentsSent atomic.Uint64 + fragmentMessagesSent atomic.Uint64 + fragmentsReceived atomic.Uint64 + fragmentMessagesRecv atomic.Uint64 + fragmentTimeouts atomic.Uint64 + + // tracing statistics + tracedSent atomic.Uint64 + tracedReceived atomic.Uint64 + + // compression statistics + compressedSent atomic.Uint64 // messages compressed on send + compressedBytesSent atomic.Uint64 // bytes after compression (wire) + compressedOrigBytesSent atomic.Uint64 // bytes before compression (original) + decompressedRecv atomic.Uint64 // messages decompressed on receive + decompressedBytesRecv atomic.Uint64 // bytes before decompression (wire) + decompressedOrigRecv atomic.Uint64 // bytes after decompression (original) + order uint32 terminated bool wg sync.WaitGroup } +type fragmentAssembly struct { + totalFragments uint16 + received uint16 + totalBytes int + payloads [][]byte + deadline time.Time +} + type pool_item struct { connection net.Conn - fl io.Writer + fl lib.Flusher timer *time.Timer handling atomic.Bool + + // skew measurement (clock offset relative to peer, nanoseconds). + // skewRing/skewIdx are only written by serve() goroutine. + // skewCount/skewValue are read by timer goroutine via atomic. + skewRing [skewRingSize]int64 + skewIdx int + skewCount atomic.Int32 + skewValue atomic.Int64 } // @@ -112,6 +190,7 @@ func (c *connection) Info() gen.RemoteNodeInfo { PoolDSN: c.pool_dsn, MaxMessageSize: c.peer_maxmessagesize, + TLS: c.tls, MessagesIn: atomic.LoadUint64(&c.messagesIn), MessagesOut: atomic.LoadUint64(&c.messagesOut), @@ -120,10 +199,136 @@ func (c *connection) Info() gen.RemoteNodeInfo { TransitBytesIn: atomic.LoadUint64(&c.transitIn), TransitBytesOut: atomic.LoadUint64(&c.transitOut), + + Reconnections: atomic.LoadUint64(&c.reconnections), + + FragmentsSent: c.fragmentsSent.Load(), + FragmentMessagesSent: c.fragmentMessagesSent.Load(), + FragmentsReceived: c.fragmentsReceived.Load(), + FragmentMessagesRecv: c.fragmentMessagesRecv.Load(), + FragmentTimeouts: c.fragmentTimeouts.Load(), + + TracedSent: c.tracedSent.Load(), + TracedReceived: c.tracedReceived.Load(), + + CompressedSent: c.compressedSent.Load(), + CompressedBytesSent: c.compressedBytesSent.Load(), + CompressedOrigBytesSent: c.compressedOrigBytesSent.Load(), + DecompressedRecv: c.decompressedRecv.Load(), + DecompressedBytesRecv: c.decompressedBytesRecv.Load(), + DecompressedOrigRecv: c.decompressedOrigRecv.Load(), + + ClockSkew: c.Skew(), } return info } +// Skew returns the aggregated clock skew across all pool connections (nanoseconds). +// Positive value means the remote node's clock is ahead of local. +func (c *connection) Skew() int64 { + c.pool_mutex.RLock() + defer c.pool_mutex.RUnlock() + if len(c.pool) == 0 { + return 0 + } + values := make([]int64, 0, len(c.pool)) + for _, pi := range c.pool { + if pi.skewCount.Load() > 0 { + values = append(values, pi.skewValue.Load()) + } + } + if len(values) == 0 { + return 0 + } + return medianInt64(values) +} + +func (c *connection) handleSkew(pi *pool_item, buf *lib.Buffer, tsRecv int64) { + ts2 := int64(binary.BigEndian.Uint64(buf.B[16:24])) + + if ts2 == 0 { + // ping - reuse buf for response, fill ts2 and ts3 + binary.BigEndian.PutUint64(buf.B[16:24], uint64(tsRecv)) + binary.BigEndian.PutUint64(buf.B[24:32], uint64(time.Now().UnixNano())) + pi.fl.Write(buf.B) + return + } + + // response - compute and store skew + ts1 := int64(binary.BigEndian.Uint64(buf.B[8:16])) + ts3 := int64(binary.BigEndian.Uint64(buf.B[24:32])) + rtt := (tsRecv - ts1) - (ts3 - ts2) + skew := ts2 - (ts1 + rtt/2) + c.pushSkew(pi, skew) +} + +func (c *connection) pushSkew(pi *pool_item, skew int64) { + pi.skewRing[pi.skewIdx%skewRingSize] = skew + pi.skewIdx++ + n := pi.skewCount.Load() + if n < skewRingSize { + n++ + pi.skewCount.Store(n) + } + + var tmp [skewRingSize]int64 + copy(tmp[:n], pi.skewRing[:n]) + pi.skewValue.Store(medianInt64(tmp[:n])) + + // schedule next ping only after receiving response + if c.terminated { + return + } + interval := 5 * time.Second + if n < skewRingSize { + interval = 100 * time.Millisecond + } + pi.timer.Reset(interval) +} + +func (c *connection) skewTick(pi *pool_item) { + if c.terminated { + return + } + c.sendSkewPing(pi) +} + +func (c *connection) sendSkewPing(pi *pool_item) { + buf := lib.TakeBuffer() + buf.Allocate(32) + buf.B[0] = protoMagic + buf.B[1] = protoVersion + binary.BigEndian.PutUint32(buf.B[2:6], 32) + buf.B[6] = 0 + buf.B[7] = protoMessageS + binary.BigEndian.PutUint64(buf.B[8:16], uint64(time.Now().UnixNano())) + binary.BigEndian.PutUint64(buf.B[16:24], 0) + binary.BigEndian.PutUint64(buf.B[24:32], 0) + pi.fl.Write(buf.B) + lib.ReleaseBuffer(buf) +} + +// medianInt64 sorts vals in place and returns the median. +func medianInt64(vals []int64) int64 { + n := len(vals) + if n == 0 { + return 0 + } + for i := 1; i < n; i++ { + v := vals[i] + j := i - 1 + for j >= 0 && vals[j] > v { + vals[j+1] = vals[j] + j-- + } + vals[j+1] = v + } + if n%2 == 0 { + return (vals[n/2-1] + vals[n/2]) / 2 + } + return vals[n/2] +} + func (c *connection) Spawn(name gen.Atom, options gen.ProcessOptions, args ...any) (gen.PID, error) { opts := gen.ProcessOptionsExtra{ ProcessOptions: options, @@ -190,7 +395,7 @@ func (c *connection) applicationStart(name gen.Atom, mode gen.ApplicationMode, o return gen.ErrNotAllowed } - ref := c.core.MakeRef() + ref := c.makeRequestRef() extra := gen.ApplicationOptionsExtra{ ApplicationOptions: options, CorePID: c.core.PID(), @@ -219,7 +424,7 @@ func (c *connection) applicationStart(name gen.Atom, mode gen.ApplicationMode, o func (c *connection) ApplicationInfo(name gen.Atom) (gen.ApplicationInfo, error) { var info gen.ApplicationInfo - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageApplicationInfo{ Name: name, Ref: ref, @@ -273,7 +478,7 @@ func (c *connection) updateCache() error { // - RegCache // - ErrCache - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageUpdateCache{ Ref: ref, // put them here @@ -327,42 +532,42 @@ func (c *connection) SendPID(from gen.PID, to gen.PID, options gen.MessageOption } buf := lib.TakeBuffer() - // 8 (header) + 8 (process id from) + 1 priority +8 (message id) + 8 (process id to) - buf.Allocate(8 + 8 + 1 + 8 + 8) + // protoWrapReserve + 8 (header) + 8 (process id from) + 1 priority + 8 (message id) + 8 (process id to) + buf.Allocate(protoWrapReserve + 8 + 8 + 1 + 8 + 8) if err := edf.Encode(message, buf, c.encodeOptions); err != nil { return err } - if c.peer_maxmessagesize > 0 && buf.Len() > c.peer_maxmessagesize { + if c.peer_maxmessagesize > 0 && buf.Len()-protoWrapReserve > c.peer_maxmessagesize { return gen.ErrTooLarge } - if buf.Len() > math.MaxUint32 { + if buf.Len()-protoWrapReserve > math.MaxUint32 { return gen.ErrTooLarge } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = orderPeer - buf.B[7] = protoMessagePID - binary.BigEndian.PutUint64(buf.B[8:16], from.ID) + h := protoWrapReserve + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = orderPeer + buf.B[h+7] = protoMessagePID + binary.BigEndian.PutUint64(buf.B[h+8:h+16], from.ID) - buf.B[16] = byte(options.Priority) // usual value 0, 1, or 2, so just cast it + buf.B[h+16] = byte(options.Priority) if options.ImportantDelivery { if c.peer_flags.EnableImportantDelivery == false { lib.ReleaseBuffer(buf) return gen.ErrUnsupported } - // set important flag - buf.B[16] |= 128 - binary.BigEndian.PutUint64(buf.B[17:25], options.Ref.ID[0]) + buf.B[h+16] |= 128 + binary.BigEndian.PutUint64(buf.B[h+17:h+25], options.Ref.ID[0]) } - binary.BigEndian.PutUint64(buf.B[25:33], to.ID) + binary.BigEndian.PutUint64(buf.B[h+25:h+33], to.ID) - return c.send(buf, order, options.Compression) + return c.send(buf, order, options.Compression, options.Tracing) } func (c *connection) SendProcessID(from gen.PID, to gen.ProcessID, options gen.MessageOptions, message any) error { @@ -392,52 +597,52 @@ func (c *connection) SendProcessID(from gen.PID, to gen.ProcessID, options gen.M buf := lib.TakeBuffer() if toNameCached > 0 { - // 8 (header) + 8 (process id from) + 1 priority + 8 (message id) + 2 (cache id) - buf.Allocate(8 + 8 + 1 + 8 + 2) + // protoWrapReserve + 8 (header) + 8 (process id from) + 1 priority + 8 (message id) + 2 (cache id) + buf.Allocate(protoWrapReserve + 8 + 8 + 1 + 8 + 2) } else { - // 8 (header) + 8 (process id from) + 1 priority + 8 (message id) + 1 (size(bname) + bname - buf.Allocate(8 + 8 + 1 + 8 + 1 + len(bname)) + // protoWrapReserve + 8 (header) + 8 (process id from) + 1 priority + 8 (message id) + 1 (size(bname) + bname + buf.Allocate(protoWrapReserve + 8 + 8 + 1 + 8 + 1 + len(bname)) } if err := edf.Encode(message, buf, c.encodeOptions); err != nil { return err } - if buf.Len() > math.MaxUint32 { + if buf.Len()-protoWrapReserve > math.MaxUint32 { return gen.ErrTooLarge } - if c.peer_maxmessagesize > 0 && buf.Len() > c.peer_maxmessagesize { + if c.peer_maxmessagesize > 0 && buf.Len()-protoWrapReserve > c.peer_maxmessagesize { return gen.ErrTooLarge } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = order // use the same order for the peer - binary.BigEndian.PutUint64(buf.B[8:16], from.ID) + h := protoWrapReserve + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = order // use the same order for the peer + binary.BigEndian.PutUint64(buf.B[h+8:h+16], from.ID) - buf.B[16] = byte(options.Priority) // usual value 0, 1, or 2, so just cast it + buf.B[h+16] = byte(options.Priority) if options.ImportantDelivery { if c.peer_flags.EnableImportantDelivery == false { lib.ReleaseBuffer(buf) return gen.ErrUnsupported } - // set important flag - buf.B[16] |= 128 - binary.BigEndian.PutUint64(buf.B[17:25], options.Ref.ID[0]) + buf.B[h+16] |= 128 + binary.BigEndian.PutUint64(buf.B[h+17:h+25], options.Ref.ID[0]) } if toNameCached > 0 { - buf.B[7] = protoMessageNameCache - binary.BigEndian.PutUint16(buf.B[25:27], toNameCached) + buf.B[h+7] = protoMessageNameCache + binary.BigEndian.PutUint16(buf.B[h+25:h+27], toNameCached) } else { - buf.B[7] = protoMessageName - buf.B[25] = byte(len(bname)) - copy(buf.B[26:], bname) + buf.B[h+7] = protoMessageName + buf.B[h+25] = byte(len(bname)) + copy(buf.B[h+26:], bname) } - return c.send(buf, order, options.Compression) + return c.send(buf, order, options.Compression, options.Tracing) } func (c *connection) SendAlias(from gen.PID, to gen.Alias, options gen.MessageOptions, message any) error { @@ -453,44 +658,44 @@ func (c *connection) SendAlias(from gen.PID, to gen.Alias, options gen.MessageOp } buf := lib.TakeBuffer() - // 8 (header) + 8 (process id from) + 1 priority + 8 (message id) + 24 (alias id [3]uint64) - buf.Allocate(8 + 8 + 1 + 8 + 24) + // protoWrapReserve + 8 (header) + 8 (process id from) + 1 priority + 8 (message id) + 24 (alias id [3]uint64) + buf.Allocate(protoWrapReserve + 8 + 8 + 1 + 8 + 24) if err := edf.Encode(message, buf, c.encodeOptions); err != nil { return err } - if buf.Len() > math.MaxUint32 { + if buf.Len()-protoWrapReserve > math.MaxUint32 { return gen.ErrTooLarge } - if c.peer_maxmessagesize > 0 && buf.Len() > c.peer_maxmessagesize { + if c.peer_maxmessagesize > 0 && buf.Len()-protoWrapReserve > c.peer_maxmessagesize { return gen.ErrTooLarge } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = orderPeer - buf.B[7] = protoMessageAlias - binary.BigEndian.PutUint64(buf.B[8:16], from.ID) + h := protoWrapReserve + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = orderPeer + buf.B[h+7] = protoMessageAlias + binary.BigEndian.PutUint64(buf.B[h+8:h+16], from.ID) - buf.B[16] = byte(options.Priority) // usual value 0, 1, or 2, so just cast it + buf.B[h+16] = byte(options.Priority) if options.ImportantDelivery { if c.peer_flags.EnableImportantDelivery == false { lib.ReleaseBuffer(buf) return gen.ErrUnsupported } - // set important flag - buf.B[16] |= 128 - binary.BigEndian.PutUint64(buf.B[17:25], options.Ref.ID[0]) + buf.B[h+16] |= 128 + binary.BigEndian.PutUint64(buf.B[h+17:h+25], options.Ref.ID[0]) } - binary.BigEndian.PutUint64(buf.B[25:33], to.ID[0]) - binary.BigEndian.PutUint64(buf.B[33:41], to.ID[1]) - binary.BigEndian.PutUint64(buf.B[41:49], to.ID[2]) + binary.BigEndian.PutUint64(buf.B[h+25:h+33], to.ID[0]) + binary.BigEndian.PutUint64(buf.B[h+33:h+41], to.ID[1]) + binary.BigEndian.PutUint64(buf.B[h+41:h+49], to.ID[2]) - return c.send(buf, order, options.Compression) + return c.send(buf, order, options.Compression, options.Tracing) } func (c *connection) SendEvent(from gen.PID, options gen.MessageOptions, message gen.MessageEvent) error { @@ -521,43 +726,44 @@ func (c *connection) SendEvent(from gen.PID, options gen.MessageOptions, message buf := lib.TakeBuffer() if eventNameCached > 0 { - // 8 (header) + 8 (process id from) + 1 priority + 8 timestamp + 2 (cache id) - buf.Allocate(8 + 8 + 1 + 8 + 2) + // protoWrapReserve + 8 (header) + 8 (process id from) + 1 priority + 8 timestamp + 2 (cache id) + buf.Allocate(protoWrapReserve + 8 + 8 + 1 + 8 + 2) } else { - // 8 (header) + 8 (process id from) + 1 priority + 8 timestamp + 1 size(bname) + bname - buf.Allocate(8 + 8 + 1 + 8 + 1 + len(bname)) + // protoWrapReserve + 8 (header) + 8 (process id from) + 1 priority + 8 timestamp + 1 size(bname) + bname + buf.Allocate(protoWrapReserve + 8 + 8 + 1 + 8 + 1 + len(bname)) } if err := edf.Encode(message.Message, buf, c.encodeOptions); err != nil { return err } - if buf.Len() > math.MaxUint32 { + if buf.Len()-protoWrapReserve > math.MaxUint32 { return gen.ErrTooLarge } - if c.peer_maxmessagesize > 0 && buf.Len() > c.peer_maxmessagesize { + if c.peer_maxmessagesize > 0 && buf.Len()-protoWrapReserve > c.peer_maxmessagesize { return gen.ErrTooLarge } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = order // use the same order for the peer - binary.BigEndian.PutUint64(buf.B[8:16], from.ID) - buf.B[16] = byte(options.Priority) // usual value 0, 1, or 2, so just cast it - binary.BigEndian.PutUint64(buf.B[17:25], uint64(message.Timestamp)) + h := protoWrapReserve + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = order + binary.BigEndian.PutUint64(buf.B[h+8:h+16], from.ID) + buf.B[h+16] = byte(options.Priority) + binary.BigEndian.PutUint64(buf.B[h+17:h+25], uint64(message.Timestamp)) if eventNameCached > 0 { - buf.B[7] = protoMessageEventCache - binary.BigEndian.PutUint16(buf.B[25:27], eventNameCached) + buf.B[h+7] = protoMessageEventCache + binary.BigEndian.PutUint16(buf.B[h+25:h+27], eventNameCached) } else { - buf.B[7] = protoMessageEvent - buf.B[25] = byte(len(bname)) - copy(buf.B[26:], bname) + buf.B[h+7] = protoMessageEvent + buf.B[h+25] = byte(len(bname)) + copy(buf.B[h+26:], bname) } - return c.send(buf, order, options.Compression) + return c.send(buf, order, options.Compression, options.Tracing) } func (c *connection) SendExit(from gen.PID, to gen.PID, reason error) error { @@ -566,8 +772,8 @@ func (c *connection) SendExit(from gen.PID, to gen.PID, reason error) error { } buf := lib.TakeBuffer() - // 8 (header) + 8 (process id from) + 1 priority + 8 (process id to) - buf.Allocate(8 + 8 + 1 + 8) + // protoWrapReserve + 8 (header) + 8 (process id from) + 1 priority + 8 (process id to) + buf.Allocate(protoWrapReserve + 8 + 8 + 1 + 8) if err := edf.Encode(reason, buf, c.encodeOptions); err != nil { return err @@ -576,16 +782,17 @@ func (c *connection) SendExit(from gen.PID, to gen.PID, reason error) error { order := uint8(from.ID % 255) orderPeer := uint8(to.ID % 255) - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = orderPeer - buf.B[7] = protoMessageExit - binary.BigEndian.PutUint64(buf.B[8:16], from.ID) - buf.B[16] = byte(gen.MessagePriorityMax) - binary.BigEndian.PutUint64(buf.B[17:25], to.ID) - - return c.send(buf, order, gen.Compression{}) + h := protoWrapReserve + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = orderPeer + buf.B[h+7] = protoMessageExit + binary.BigEndian.PutUint64(buf.B[h+8:h+16], from.ID) + buf.B[h+16] = byte(gen.MessagePriorityMax) + binary.BigEndian.PutUint64(buf.B[h+17:h+25], to.ID) + + return c.send(buf, order, gen.Compression{}, gen.Tracing{}) } func (c *connection) SendResponse(from gen.PID, to gen.PID, options gen.MessageOptions, response any) error { @@ -600,44 +807,44 @@ func (c *connection) SendResponse(from gen.PID, to gen.PID, options gen.MessageO } buf := lib.TakeBuffer() - // 8 (header) + 8 (process id from) + 1 priority + 8 (process id to) + 24 (ref [3]uint64) - buf.Allocate(8 + 8 + 1 + 8 + 24) + // protoWrapReserve + 8 (header) + 8 (process id from) + 1 priority + 8 (process id to) + 24 (ref [3]uint64) + buf.Allocate(protoWrapReserve + 8 + 8 + 1 + 8 + 24) if err := edf.Encode(response, buf, c.encodeOptions); err != nil { return err } - if buf.Len() > math.MaxUint32 { + if buf.Len()-protoWrapReserve > math.MaxUint32 { return gen.ErrTooLarge } - if c.peer_maxmessagesize > 0 && buf.Len() > c.peer_maxmessagesize { + if c.peer_maxmessagesize > 0 && buf.Len()-protoWrapReserve > c.peer_maxmessagesize { return gen.ErrTooLarge } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = orderPeer - buf.B[7] = protoMessageResponse - binary.BigEndian.PutUint64(buf.B[8:16], from.ID) + h := protoWrapReserve + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = orderPeer + buf.B[h+7] = protoMessageResponse + binary.BigEndian.PutUint64(buf.B[h+8:h+16], from.ID) - buf.B[16] = byte(options.Priority) // usual value 0, 1, or 2, so just cast it + buf.B[h+16] = byte(options.Priority) if options.ImportantDelivery { if c.peer_flags.EnableImportantDelivery == false { lib.ReleaseBuffer(buf) return gen.ErrUnsupported } - // set important flag - buf.B[16] |= 128 + buf.B[h+16] |= 128 } - binary.BigEndian.PutUint64(buf.B[17:25], to.ID) - binary.BigEndian.PutUint64(buf.B[25:33], options.Ref.ID[0]) - binary.BigEndian.PutUint64(buf.B[33:41], options.Ref.ID[1]) - binary.BigEndian.PutUint64(buf.B[41:49], options.Ref.ID[2]) + binary.BigEndian.PutUint64(buf.B[h+17:h+25], to.ID) + binary.BigEndian.PutUint64(buf.B[h+25:h+33], options.Ref.ID[0]) + binary.BigEndian.PutUint64(buf.B[h+33:h+41], options.Ref.ID[1]) + binary.BigEndian.PutUint64(buf.B[h+41:h+49], options.Ref.ID[2]) - return c.send(buf, order, options.Compression) + return c.send(buf, order, options.Compression, options.Tracing) } func (c *connection) SendResponseError(from gen.PID, to gen.PID, options gen.MessageOptions, err error) error { @@ -653,67 +860,68 @@ func (c *connection) SendResponseError(from gen.PID, to gen.PID, options gen.Mes } buf := lib.TakeBuffer() - // 8 (header) + 8 (process id from) + 1 priority + 8 (process id to) + 24 (ref [3]uint64) + 1 (err code) - buf.Allocate(8 + 8 + 1 + 8 + 24 + 1) + h := protoWrapReserve + // protoWrapReserve + 8 (header) + 8 (process id from) + 1 priority + 8 (process id to) + 24 (ref [3]uint64) + 1 (err code) + buf.Allocate(h + 8 + 8 + 1 + 8 + 24 + 1) switch err { case nil: - buf.B[49] = 0 + buf.B[h+49] = 0 case gen.ErrProcessUnknown: - buf.B[49] = 1 + buf.B[h+49] = 1 case gen.ErrProcessMailboxFull: - buf.B[49] = 2 + buf.B[h+49] = 2 case gen.ErrProcessTerminated: - buf.B[49] = 3 + buf.B[h+49] = 3 default: - buf.B[49] = 255 + buf.B[h+49] = 255 if e := edf.Encode(err, buf, c.encodeOptions); e != nil { return e } } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = orderPeer - buf.B[7] = protoMessageResponseError - binary.BigEndian.PutUint64(buf.B[8:16], from.ID) + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = orderPeer + buf.B[h+7] = protoMessageResponseError + binary.BigEndian.PutUint64(buf.B[h+8:h+16], from.ID) - buf.B[16] = byte(options.Priority) // usual value 0, 1, or 2, so just cast it + buf.B[h+16] = byte(options.Priority) if options.ImportantDelivery { if c.peer_flags.EnableImportantDelivery == false { lib.ReleaseBuffer(buf) return gen.ErrUnsupported } - // set important flag - buf.B[16] |= 128 + buf.B[h+16] |= 128 } - binary.BigEndian.PutUint64(buf.B[17:25], to.ID) - binary.BigEndian.PutUint64(buf.B[25:33], options.Ref.ID[0]) - binary.BigEndian.PutUint64(buf.B[33:41], options.Ref.ID[1]) - binary.BigEndian.PutUint64(buf.B[41:49], options.Ref.ID[2]) + binary.BigEndian.PutUint64(buf.B[h+17:h+25], to.ID) + binary.BigEndian.PutUint64(buf.B[h+25:h+33], options.Ref.ID[0]) + binary.BigEndian.PutUint64(buf.B[h+33:h+41], options.Ref.ID[1]) + binary.BigEndian.PutUint64(buf.B[h+41:h+49], options.Ref.ID[2]) - return c.send(buf, order, options.Compression) + return c.send(buf, order, options.Compression, options.Tracing) } func (c *connection) SendTerminatePID(target gen.PID, reason error) error { buf := lib.TakeBuffer() - // 8 (header) + 1 priority + 8 (target process id) - buf.Allocate(8 + 1 + 8) + // protoWrapReserve + 8 (header) + 1 priority + 8 (target process id) + buf.Allocate(protoWrapReserve + 8 + 1 + 8) if err := edf.Encode(reason, buf, c.encodeOptions); err != nil { return err } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = 0 - buf.B[7] = protoMessageTerminatePID - buf.B[8] = byte(gen.MessagePriorityHigh) - binary.BigEndian.PutUint64(buf.B[9:17], target.ID) + h := protoWrapReserve + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = 0 + buf.B[h+7] = protoMessageTerminatePID + buf.B[h+8] = byte(gen.MessagePriorityHigh) + binary.BigEndian.PutUint64(buf.B[h+9:h+17], target.ID) - return c.send(buf, 0, gen.Compression{}) + return c.send(buf, 0, gen.Compression{}, gen.Tracing{}) } func (c *connection) SendTerminateProcessID(target gen.ProcessID, reason error) error { @@ -737,55 +945,56 @@ func (c *connection) SendTerminateProcessID(target gen.ProcessID, reason error) } buf := lib.TakeBuffer() + h := protoWrapReserve if targetNameCached > 0 { - // 8 (header) + 1 priority + 2 (cache id) - buf.Allocate(8 + 1 + 2) + // protoWrapReserve + 8 (header) + 1 priority + 2 (cache id) + buf.Allocate(h + 8 + 1 + 2) } else { - // 8 (header) + 1 priority + 1 (len of bname) + bname - buf.Allocate(8 + 1 + 1 + len(bname)) + // protoWrapReserve + 8 (header) + 1 priority + 1 (len of bname) + bname + buf.Allocate(h + 8 + 1 + 1 + len(bname)) } if err := edf.Encode(reason, buf, c.encodeOptions); err != nil { return err } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = 0 // order - buf.B[8] = byte(gen.MessagePriorityHigh) + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = 0 + buf.B[h+8] = byte(gen.MessagePriorityHigh) if targetNameCached > 0 { - buf.B[7] = protoMessageTerminateNameCache - binary.BigEndian.PutUint16(buf.B[9:11], targetNameCached) + buf.B[h+7] = protoMessageTerminateNameCache + binary.BigEndian.PutUint16(buf.B[h+9:h+11], targetNameCached) } else { - buf.B[7] = protoMessageTerminateName - buf.B[9] = byte(len(bname)) - copy(buf.B[10:], bname) + buf.B[h+7] = protoMessageTerminateName + buf.B[h+9] = byte(len(bname)) + copy(buf.B[h+10:], bname) } - return c.send(buf, 0, gen.Compression{}) + return c.send(buf, 0, gen.Compression{}, gen.Tracing{}) } func (c *connection) SendTerminateAlias(target gen.Alias, reason error) error { buf := lib.TakeBuffer() - // 8 (header) + 1 priority + 24 (target alias id [3]uint64) - buf.Allocate(8 + 1 + 24) + h := protoWrapReserve + buf.Allocate(h + 8 + 1 + 24) if err := edf.Encode(reason, buf, c.encodeOptions); err != nil { return err } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = 0 - buf.B[7] = protoMessageTerminateAlias - buf.B[8] = byte(gen.MessagePriorityHigh) - binary.BigEndian.PutUint64(buf.B[9:17], target.ID[0]) - binary.BigEndian.PutUint64(buf.B[17:25], target.ID[1]) - binary.BigEndian.PutUint64(buf.B[25:33], target.ID[2]) + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = 0 + buf.B[h+7] = protoMessageTerminateAlias + buf.B[h+8] = byte(gen.MessagePriorityHigh) + binary.BigEndian.PutUint64(buf.B[h+9:h+17], target.ID[0]) + binary.BigEndian.PutUint64(buf.B[h+17:h+25], target.ID[1]) + binary.BigEndian.PutUint64(buf.B[h+25:h+33], target.ID[2]) - return c.send(buf, 0, gen.Compression{}) + return c.send(buf, 0, gen.Compression{}, gen.Tracing{}) } func (c *connection) SendTerminateEvent(target gen.Event, reason error) error { @@ -810,34 +1019,33 @@ func (c *connection) SendTerminateEvent(target gen.Event, reason error) error { } buf := lib.TakeBuffer() + h := protoWrapReserve if eventNameCached > 0 { - // 8 (header) + 1 priority + 2 (cache id) - buf.Allocate(8 + 1 + 2) + buf.Allocate(h + 8 + 1 + 2) } else { - // 8 (header) + 1 priority + 1 size(bname) + bname - buf.Allocate(8 + 1 + +1 + len(bname)) + buf.Allocate(h + 8 + 1 + 1 + len(bname)) } if err := edf.Encode(reason, buf, c.encodeOptions); err != nil { return err } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = 0 // order - buf.B[8] = byte(gen.MessagePriorityHigh) + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = 0 + buf.B[h+8] = byte(gen.MessagePriorityHigh) if eventNameCached > 0 { - buf.B[7] = protoMessageTerminateEventCache - binary.BigEndian.PutUint16(buf.B[9:11], eventNameCached) + buf.B[h+7] = protoMessageTerminateEventCache + binary.BigEndian.PutUint16(buf.B[h+9:h+11], eventNameCached) } else { - buf.B[7] = protoMessageTerminateEvent - buf.B[9] = byte(len(bname)) - copy(buf.B[10:], bname) + buf.B[h+7] = protoMessageTerminateEvent + buf.B[h+9] = byte(len(bname)) + copy(buf.B[h+10:], bname) } - return c.send(buf, 0, gen.Compression{}) + return c.send(buf, 0, gen.Compression{}, gen.Tracing{}) } func (c *connection) CallPID(from gen.PID, to gen.PID, options gen.MessageOptions, message any) error { @@ -853,39 +1061,39 @@ func (c *connection) CallPID(from gen.PID, to gen.PID, options gen.MessageOption } buf := lib.TakeBuffer() - // 8 (header) + 8 (process id from) + 1 priority + 24 (request ref) + 8 (process id to) - buf.Allocate(8 + 8 + 1 + 24 + 8) + // protoWrapReserve + 8 (header) + 8 (process id from) + 1 priority + 24 (request ref) + 8 (process id to) + buf.Allocate(protoWrapReserve + 8 + 8 + 1 + 24 + 8) if err := edf.Encode(message, buf, c.encodeOptions); err != nil { return err } - if buf.Len() > math.MaxUint32 { + if buf.Len()-protoWrapReserve > math.MaxUint32 { return gen.ErrTooLarge } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = orderPeer - buf.B[7] = protoRequestPID - binary.BigEndian.PutUint64(buf.B[8:16], from.ID) + h := protoWrapReserve + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = orderPeer + buf.B[h+7] = protoRequestPID + binary.BigEndian.PutUint64(buf.B[h+8:h+16], from.ID) - buf.B[16] = byte(options.Priority) + buf.B[h+16] = byte(options.Priority) if options.ImportantDelivery { if c.peer_flags.EnableImportantDelivery == false { lib.ReleaseBuffer(buf) return gen.ErrUnsupported } - // set important flag - buf.B[16] |= 128 + buf.B[h+16] |= 128 } - binary.BigEndian.PutUint64(buf.B[17:25], options.Ref.ID[0]) - binary.BigEndian.PutUint64(buf.B[25:33], options.Ref.ID[1]) - binary.BigEndian.PutUint64(buf.B[33:41], options.Ref.ID[2]) - binary.BigEndian.PutUint64(buf.B[41:49], to.ID) + binary.BigEndian.PutUint64(buf.B[h+17:h+25], options.Ref.ID[0]) + binary.BigEndian.PutUint64(buf.B[h+25:h+33], options.Ref.ID[1]) + binary.BigEndian.PutUint64(buf.B[h+33:h+41], options.Ref.ID[2]) + binary.BigEndian.PutUint64(buf.B[h+41:h+49], to.ID) - return c.send(buf, order, options.Compression) + return c.send(buf, order, options.Compression, options.Tracing) } func (c *connection) CallProcessID(from gen.PID, to gen.ProcessID, options gen.MessageOptions, message any) error { @@ -914,52 +1122,50 @@ func (c *connection) CallProcessID(from gen.PID, to gen.ProcessID, options gen.M } buf := lib.TakeBuffer() + h := protoWrapReserve if toNameCached > 0 { - // 8 (header) + 8 (process id from) + 1 priority + 24 (request ref) + 2 (cache id) - buf.Allocate(8 + 8 + 1 + 24 + 2) + buf.Allocate(h + 8 + 8 + 1 + 24 + 2) } else { - // 8 (header) + 8 (process id from) + 1 priority + 24 (request ref) +1 (size(bname)) + bname - buf.Allocate(8 + 8 + 1 + 24 + 1 + len(bname)) + buf.Allocate(h + 8 + 8 + 1 + 24 + 1 + len(bname)) } if err := edf.Encode(message, buf, c.encodeOptions); err != nil { return err } - if buf.Len() > math.MaxUint32 { + if buf.Len()-h > math.MaxUint32 { return gen.ErrTooLarge } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = order // use the same order for the peer - binary.BigEndian.PutUint64(buf.B[8:16], from.ID) + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = order + binary.BigEndian.PutUint64(buf.B[h+8:h+16], from.ID) - buf.B[16] = byte(options.Priority) + buf.B[h+16] = byte(options.Priority) if options.ImportantDelivery { if c.peer_flags.EnableImportantDelivery == false { lib.ReleaseBuffer(buf) return gen.ErrUnsupported } - // set important flag - buf.B[16] |= 128 + buf.B[h+16] |= 128 } - binary.BigEndian.PutUint64(buf.B[17:25], options.Ref.ID[0]) - binary.BigEndian.PutUint64(buf.B[25:33], options.Ref.ID[1]) - binary.BigEndian.PutUint64(buf.B[33:41], options.Ref.ID[2]) + binary.BigEndian.PutUint64(buf.B[h+17:h+25], options.Ref.ID[0]) + binary.BigEndian.PutUint64(buf.B[h+25:h+33], options.Ref.ID[1]) + binary.BigEndian.PutUint64(buf.B[h+33:h+41], options.Ref.ID[2]) if toNameCached > 0 { - buf.B[7] = protoRequestNameCache - binary.BigEndian.PutUint16(buf.B[41:43], toNameCached) + buf.B[h+7] = protoRequestNameCache + binary.BigEndian.PutUint16(buf.B[h+41:h+43], toNameCached) } else { - buf.B[7] = protoRequestName - buf.B[41] = byte(len(bname)) - copy(buf.B[42:], bname) + buf.B[h+7] = protoRequestName + buf.B[h+41] = byte(len(bname)) + copy(buf.B[h+42:], bname) } - return c.send(buf, order, options.Compression) + return c.send(buf, order, options.Compression, options.Tracing) } func (c *connection) CallAlias(from gen.PID, to gen.Alias, options gen.MessageOptions, message any) error { @@ -976,42 +1182,51 @@ func (c *connection) CallAlias(from gen.PID, to gen.Alias, options gen.MessageOp } buf := lib.TakeBuffer() - // 8 (header) + 8 (process id from) + 1 priority + 24 (request ref) + 24 (alias id to) - buf.Allocate(8 + 8 + 1 + 24 + 24) + h := protoWrapReserve + buf.Allocate(h + 8 + 8 + 1 + 24 + 24) if err := edf.Encode(message, buf, c.encodeOptions); err != nil { return err } - if buf.Len() > math.MaxUint32 { + if buf.Len()-h > math.MaxUint32 { return gen.ErrTooLarge } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = orderPeer - buf.B[7] = protoRequestAlias - binary.BigEndian.PutUint64(buf.B[8:16], from.ID) + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = orderPeer + buf.B[h+7] = protoRequestAlias + binary.BigEndian.PutUint64(buf.B[h+8:h+16], from.ID) - buf.B[16] = byte(options.Priority) + buf.B[h+16] = byte(options.Priority) if options.ImportantDelivery { if c.peer_flags.EnableImportantDelivery == false { lib.ReleaseBuffer(buf) return gen.ErrUnsupported } - // set important flag - buf.B[16] |= 128 + buf.B[h+16] |= 128 } - binary.BigEndian.PutUint64(buf.B[17:25], options.Ref.ID[0]) - binary.BigEndian.PutUint64(buf.B[25:33], options.Ref.ID[1]) - binary.BigEndian.PutUint64(buf.B[33:41], options.Ref.ID[2]) - binary.BigEndian.PutUint64(buf.B[41:49], to.ID[0]) - binary.BigEndian.PutUint64(buf.B[49:57], to.ID[1]) - binary.BigEndian.PutUint64(buf.B[57:65], to.ID[2]) + binary.BigEndian.PutUint64(buf.B[h+17:h+25], options.Ref.ID[0]) + binary.BigEndian.PutUint64(buf.B[h+25:h+33], options.Ref.ID[1]) + binary.BigEndian.PutUint64(buf.B[h+33:h+41], options.Ref.ID[2]) + binary.BigEndian.PutUint64(buf.B[h+41:h+49], to.ID[0]) + binary.BigEndian.PutUint64(buf.B[h+49:h+57], to.ID[1]) + binary.BigEndian.PutUint64(buf.B[h+57:h+65], to.ID[2]) + + return c.send(buf, order, options.Compression, options.Tracing) +} - return c.send(buf, order, options.Compression) +// makeRequestRef returns a reference carrying the default request timeout as +// a deadline. Receivers can use Ref.IsAlive() to drop work whose ref expired +// while sitting in queue, or to skip the response and roll back side effects +// when the operation outlived the sender's wait window. +func (c *connection) makeRequestRef() gen.Ref { + deadline := time.Now().Unix() + int64(gen.DefaultRequestTimeout) + ref, _ := c.core.MakeRefWithDeadline(deadline) + return ref } func (c *connection) LinkPID(pid gen.PID, target gen.PID) error { @@ -1020,7 +1235,7 @@ func (c *connection) LinkPID(pid gen.PID, target gen.PID) error { } order := uint8(pid.ID % 255) orderPeer := uint8(target.ID % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageLinkPID{ Source: pid, Target: target, @@ -1047,7 +1262,7 @@ func (c *connection) UnlinkPID(pid gen.PID, target gen.PID) error { } order := uint8(pid.ID % 255) orderPeer := uint8(target.ID % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageUnlinkPID{ Source: pid, Target: target, @@ -1069,7 +1284,7 @@ func (c *connection) UnlinkPID(pid gen.PID, target gen.PID) error { func (c *connection) LinkProcessID(pid gen.PID, target gen.ProcessID) error { order := uint8(pid.ID % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageLinkProcessID{ Source: pid, Target: target, @@ -1092,7 +1307,7 @@ func (c *connection) LinkProcessID(pid gen.PID, target gen.ProcessID) error { func (c *connection) UnlinkProcessID(pid gen.PID, target gen.ProcessID) error { order := uint8(pid.ID % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageUnlinkProcessID{ Source: pid, Target: target, @@ -1119,7 +1334,7 @@ func (c *connection) LinkAlias(pid gen.PID, target gen.Alias) error { } order := uint8(pid.ID % 255) orderPeer := uint8(target.ID[1] % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageLinkAlias{ Source: pid, Target: target, @@ -1146,7 +1361,7 @@ func (c *connection) UnlinkAlias(pid gen.PID, target gen.Alias) error { } order := uint8(pid.ID % 255) orderPeer := uint8(target.ID[1] % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageUnlinkAlias{ Source: pid, Target: target, @@ -1168,7 +1383,7 @@ func (c *connection) UnlinkAlias(pid gen.PID, target gen.Alias) error { func (c *connection) LinkEvent(pid gen.PID, target gen.Event) ([]gen.MessageEvent, error) { order := uint8(pid.ID % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageLinkEvent{ Source: pid, Target: target, @@ -1197,7 +1412,7 @@ func (c *connection) LinkEvent(pid gen.PID, target gen.Event) ([]gen.MessageEven func (c *connection) UnlinkEvent(pid gen.PID, target gen.Event) error { order := uint8(pid.ID % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageUnlinkEvent{ Source: pid, Target: target, @@ -1221,7 +1436,7 @@ func (c *connection) MonitorPID(pid gen.PID, target gen.PID) error { if target.Creation != c.peer_creation { return gen.ErrProcessIncarnation } - ref := c.core.MakeRef() + ref := c.makeRequestRef() order := uint8(pid.ID % 255) message := MessageMonitorPID{ Source: pid, @@ -1246,7 +1461,7 @@ func (c *connection) DemonitorPID(pid gen.PID, target gen.PID) error { if target.Creation != c.peer_creation { return gen.ErrProcessIncarnation } - ref := c.core.MakeRef() + ref := c.makeRequestRef() order := uint8(pid.ID % 255) message := MessageDemonitorPID{ Source: pid, @@ -1269,7 +1484,7 @@ func (c *connection) DemonitorPID(pid gen.PID, target gen.PID) error { func (c *connection) MonitorProcessID(pid gen.PID, target gen.ProcessID) error { order := uint8(pid.ID % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageMonitorProcessID{ Source: pid, Target: target, @@ -1291,7 +1506,7 @@ func (c *connection) MonitorProcessID(pid gen.PID, target gen.ProcessID) error { func (c *connection) DemonitorProcessID(pid gen.PID, target gen.ProcessID) error { order := uint8(pid.ID % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageDemonitorProcessID{ Source: pid, Target: target, @@ -1315,7 +1530,7 @@ func (c *connection) MonitorAlias(pid gen.PID, target gen.Alias) error { if target.Creation != c.peer_creation { return gen.ErrProcessIncarnation } - ref := c.core.MakeRef() + ref := c.makeRequestRef() order := uint8(pid.ID % 255) orderPeer := uint8(target.ID[1] % 255) message := MessageMonitorAlias{ @@ -1341,7 +1556,7 @@ func (c *connection) DemonitorAlias(pid gen.PID, target gen.Alias) error { if target.Creation != c.peer_creation { return gen.ErrProcessIncarnation } - ref := c.core.MakeRef() + ref := c.makeRequestRef() order := uint8(pid.ID % 255) orderPeer := uint8(target.ID[1] % 255) message := MessageDemonitorAlias{ @@ -1365,7 +1580,7 @@ func (c *connection) DemonitorAlias(pid gen.PID, target gen.Alias) error { func (c *connection) MonitorEvent(pid gen.PID, target gen.Event) ([]gen.MessageEvent, error) { order := uint8(pid.ID % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageMonitorEvent{ Source: pid, Target: target, @@ -1394,7 +1609,7 @@ func (c *connection) MonitorEvent(pid gen.PID, target gen.Event) ([]gen.MessageE func (c *connection) DemonitorEvent(pid gen.PID, target gen.Event) error { order := uint8(pid.ID % 255) - ref := c.core.MakeRef() + ref := c.makeRequestRef() message := MessageDemonitorEvent{ Source: pid, Target: target, @@ -1483,25 +1698,30 @@ func (c *connection) Join(conn net.Conn, id string, dial gen.NetworkDial, tail [ c.pool_mutex.Unlock() return fmt.Errorf("pool size limit") } - pi := &pool_item{ - connection: conn, - fl: lib.NewFlusher(conn), + // clear handshake write deadline + conn.SetWriteDeadline(time.Time{}) + + pi := &pool_item{connection: conn} + if c.softwareKeepAlive { + pi.fl = lib.NewFlusherWithKeepAlive(conn, c.softwareKeepAliveMessage, c.softwareKeepAlivePeriod) + } else { + pi.fl = lib.NewFlusher(conn) } c.pool = append(c.pool, pi) c.pool_mutex.Unlock() c.wg.Add(1) go func() { - if lib.Trace() { + if lib.Verbose() { defer c.log.Trace("connection %s left the pool", conn.RemoteAddr().String()) } re: // reconnected - if lib.Trace() { + if lib.Verbose() { c.log.Trace("joined new connection %s to the pool", conn.RemoteAddr().String()) } - c.serve(pi.connection, tail) + c.serve(pi, tail) if dial != nil { pool_dsn := []string{} @@ -1520,7 +1740,23 @@ func (c *connection) Join(conn net.Conn, id string, dial gen.NetworkDial, tail [ continue } pi.connection = nc + nc.SetWriteDeadline(time.Time{}) + if pi.timer != nil { + pi.timer.Stop() + } + pi.fl.Reset(nc) tail = t + atomic.AddUint64(&c.reconnections, 1) + + // reset skew state and restart measurement + if c.clockSkew { + pi.skewIdx = 0 + pi.skewCount.Store(0) + pi.skewValue.Store(0) + pi.timer = time.AfterFunc(100*time.Millisecond, func() { + c.skewTick(pi) + }) + } goto re } @@ -1558,11 +1794,27 @@ func (c *connection) Terminate(reason error) { c.pool_mutex.Lock() defer c.pool_mutex.Unlock() for _, pi := range c.pool { + if pi.timer != nil { + pi.timer.Stop() + } pi.connection.Close() } + + // cleanup fragment state + if c.sharedFragTimer != nil { + c.sharedFragTimer.Stop() + } } -func (c *connection) serve(conn net.Conn, tail []byte) { +func (c *connection) serve(pi *pool_item, tail []byte) { + conn := pi.connection + + // start skew measurement now that serve() is running + if c.clockSkew { + pi.timer = time.AfterFunc(100*time.Millisecond, func() { + c.skewTick(pi) + }) + } recvN := 0 recvNQ := len(c.recvQueues) @@ -1570,21 +1822,36 @@ func (c *connection) serve(conn net.Conn, tail []byte) { buf := lib.TakeBuffer() buf.Append(tail) - // remove the deadline - conn.SetReadDeadline(time.Time{}) + // replace handshake read deadline with keepalive deadline or infinite + if c.softwareKeepAlive { + conn.SetReadDeadline(time.Now().Add(c.softwareKeepAliveTimeout)) + } else { + conn.SetReadDeadline(time.Time{}) + } for { // read packet buftail, err := c.read(conn, buf) if err != nil || buftail == nil { if err != nil { - c.log.Trace("link with %s closed: %s", conn.RemoteAddr(), err) + if netErr, ok := err.(net.Error); ok && netErr.Timeout() { + c.log.Warning("software keepalive timeout on %s", conn.RemoteAddr()) + c.Terminate(gen.ErrTimeout) + } else { + c.log.Trace("link with %s closed: %s", conn.RemoteAddr(), err) + } } lib.ReleaseBuffer(buf) conn.Close() return } + // reset deadline after successful read + if c.softwareKeepAlive { + timeout := c.softwareKeepAliveTimeout + conn.SetReadDeadline(time.Now().Add(timeout)) + } + if buf.B[0] != protoMagic { c.log.Error("recevied malformed packet from %s (incorrect proto)", conn.RemoteAddr()) lib.ReleaseBuffer(buf) @@ -1599,10 +1866,28 @@ func (c *connection) serve(conn net.Conn, tail []byte) { return } + // handle keepalive silently, don't count, don't queue + if buf.B[7] == protoMessageK { + lib.ReleaseBuffer(buf) + buf = buftail + continue + } + + // handle skew measurement before queue dispatch for timestamp accuracy + if buf.B[7] == protoMessageS { + tsRecv := time.Now().UnixNano() + c.handleSkew(pi, buf, tsRecv) + lib.ReleaseBuffer(buf) + buf = buftail + continue + } + recvN++ - atomic.AddUint64(&c.messagesIn, 1) atomic.AddUint64(&c.bytesIn, uint64(buf.Len())) + if buf.B[7] != protoMessageF { + atomic.AddUint64(&c.messagesIn, 1) + } // TODO // c.transitIn @@ -1611,7 +1896,7 @@ func (c *connection) serve(conn net.Conn, tail []byte) { if order := int(buf.B[6]); order > 0 { qN = order % recvNQ } - if lib.Trace() { + if lib.Verbose() { c.log.Trace("received message. put it to pool[%d] of %s...", qN, conn.RemoteAddr()) } queue := c.recvQueues[qN] @@ -1619,14 +1904,14 @@ func (c *connection) serve(conn net.Conn, tail []byte) { queue.Push(buf) if queue.Lock() { - go c.handleRecvQueue(queue) + go c.handleRecvQueue(queue, qN) } buf = buftail } } -func (c *connection) handleRecvQueue(q lib.QueueMPSC) { +func (c *connection) handleRecvQueue(q lib.QueueMPSC, qIdx int) { if lib.Recover() { defer func() { if r := recover(); r != nil { @@ -1638,13 +1923,29 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { }() } - if lib.Trace() { + if lib.Verbose() { c.log.Trace("start handling the message queue") } + + // per-queue ordered fragment assembly (persists across goroutine re-entries) + var localFragments map[uint32]*fragmentAssembly + if c.fragmentation && qIdx < len(c.orderedFragments) { + localFragments = c.orderedFragments[qIdx] + } for { v, ok := q.Pop() if ok == false { - // no more items in the queue, unlock it + // queue drained, clean stale ordered assemblies (safe: we hold queue lock) + if localFragments != nil && len(localFragments) > 0 { + now := time.Now() + for seqID, asm := range localFragments { + if now.After(asm.deadline) { + delete(localFragments, seqID) + c.fragmentTimeouts.Add(1) + } + } + } + q.Unlock() // but check the queue before the exit this goroutine @@ -1670,6 +1971,8 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { releaseBuffer = false } + var tracing gen.Tracing + re: switch buf.B[7] { case protoMessagePID: // process id @@ -1680,6 +1983,7 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { idFrom := binary.BigEndian.Uint64(buf.B[8:16]) priority := gen.MessagePriority(buf.B[16] & 3) important := (buf.B[16] & 128) > 0 + refID := binary.BigEndian.Uint64(buf.B[17:25]) idTO := binary.BigEndian.Uint64(buf.B[25:33]) msg, tail, err := edf.Decode(buf.B[33:], c.decodeOptions) @@ -1692,10 +1996,6 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - from := gen.PID{ Node: c.peer, ID: idFrom, @@ -1707,7 +2007,12 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { Creation: c.core.Creation(), } + if len(tail) > 0 { + c.log.Warning("message %T from %s to %s has %d extra bytes", msg, from, to, len(tail)) + } + opts := gen.MessageOptions{ + Tracing: tracing, Priority: priority, } @@ -1719,7 +2024,7 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - opts.Ref.ID[0] = binary.BigEndian.Uint64(buf.B[17:25]) + opts.Ref.ID[0] = refID c.SendResponseError(to, from, opts, err) case protoMessageName, protoMessageNameCache: // name, chached name @@ -1767,6 +2072,7 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { idFrom := binary.BigEndian.Uint64(buf.B[8:16]) priority := gen.MessagePriority(buf.B[16] & 3) important := (buf.B[16] & 128) > 0 + refID := binary.BigEndian.Uint64(buf.B[17:25]) msg, tail, err := edf.Decode(data, c.decodeOptions) if releaseBuffer { @@ -1778,10 +2084,6 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - if c.decodeOptions.AtomMapping != nil { if v, found := c.decodeOptions.AtomMapping.Load(toName); found { toName = v.(gen.Atom) @@ -1798,7 +2100,12 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { Name: toName, } + if len(tail) > 0 { + c.log.Warning("message %T from %s to %s has %d extra bytes", msg, from, to, len(tail)) + } + opts := gen.MessageOptions{ + Tracing: tracing, Priority: priority, } @@ -1810,7 +2117,7 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - opts.Ref.ID[0] = binary.BigEndian.Uint64(buf.B[17:25]) + opts.Ref.ID[0] = refID c.SendResponseError(gen.PID{}, from, opts, err) case protoMessageAlias: @@ -1822,6 +2129,7 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { idFrom := binary.BigEndian.Uint64(buf.B[8:16]) priority := gen.MessagePriority(buf.B[16] & 3) important := (buf.B[16] & 128) > 0 + refID := binary.BigEndian.Uint64(buf.B[17:25]) idTo := [3]uint64{ binary.BigEndian.Uint64(buf.B[25:33]), binary.BigEndian.Uint64(buf.B[33:41]), @@ -1838,10 +2146,6 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - from := gen.PID{ Node: c.peer, ID: idFrom, @@ -1853,7 +2157,12 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { Creation: c.core.Creation(), } + if len(tail) > 0 { + c.log.Warning("message %T from %s to %s has %d extra bytes", msg, from, to, len(tail)) + } + opts := gen.MessageOptions{ + Tracing: tracing, Priority: priority, } err = c.core.RouteSendAlias(from, to, opts, msg) @@ -1865,7 +2174,7 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - opts.Ref.ID[0] = binary.BigEndian.Uint64(buf.B[17:25]) + opts.Ref.ID[0] = refID c.SendResponseError(gen.PID{}, from, opts, err) case protoRequestPID: @@ -1898,10 +2207,6 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - from := gen.PID{ Node: c.peer, ID: idFrom, @@ -1913,7 +2218,12 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { Creation: c.core.Creation(), } + if len(tail) > 0 { + c.log.Warning("call %T from %s to %s has %d extra bytes", msg, from, to, len(tail)) + } + opts := gen.MessageOptions{ + Tracing: tracing, Ref: ref, Priority: priority, } @@ -1998,16 +2308,18 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - if c.decodeOptions.AtomMapping != nil { if v, found := c.decodeOptions.AtomMapping.Load(to.Name); found { to.Name = v.(gen.Atom) } } + + if len(tail) > 0 { + c.log.Warning("call %T from %s to %s has %d extra bytes", msg, from, to, len(tail)) + } + opts := gen.MessageOptions{ + Tracing: tracing, Ref: ref, Priority: priority, } @@ -2062,17 +2374,18 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - from := gen.PID{ Node: c.peer, ID: idFrom, Creation: c.peer_creation, } + if len(tail) > 0 { + c.log.Warning("call %T from %s to %s has %d extra bytes", msg, from, to, len(tail)) + } + opts := gen.MessageOptions{ + Tracing: tracing, Ref: ref, Priority: priority, } @@ -2153,7 +2466,7 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { } if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) + c.log.Warning("event %s message %T from %s has %d extra bytes", message.Event.Name, msg, from, len(tail)) } message.Message = msg @@ -2179,10 +2492,6 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - reason, ok := msg.(error) if ok == false { c.log.Error("received malformed Exit message: %v", msg) @@ -2200,6 +2509,10 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { Creation: c.core.Creation(), } + if len(tail) > 0 { + c.log.Warning("exit from %s to %s (reason %T) has %d extra bytes", from, to, reason, len(tail)) + } + c.core.RouteSendExit(from, to, reason) case protoMessageResponse: @@ -2230,10 +2543,6 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - from := gen.PID{ Node: c.peer, ID: idFrom, @@ -2245,7 +2554,12 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { Creation: c.core.Creation(), } + if len(tail) > 0 { + c.log.Warning("response %T from %s to %s has %d extra bytes", msg, from, to, len(tail)) + } + opts := gen.MessageOptions{ + Tracing: tracing, Ref: ref, Priority: priority, ImportantDelivery: important, @@ -2290,6 +2604,7 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { Creation: c.core.Creation(), } opts := gen.MessageOptions{ + Tracing: tracing, Ref: ref, Priority: priority, ImportantDelivery: important, @@ -2318,16 +2633,16 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - r, ok = msg.(error) if ok == false { c.log.Error("received incorrect response error") continue } + if len(tail) > 0 { + c.log.Warning("response error %T from %s to %s has %d extra bytes", r, from, to, len(tail)) + } + default: c.log.Error("received incorrect response error id") continue @@ -2359,10 +2674,6 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - reason, ok := msg.(error) if ok == false { c.log.Error("received malformed TerminatePID message: %v", msg) @@ -2374,6 +2685,11 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { ID: idTarget, Creation: c.peer_creation, } + + if len(tail) > 0 { + c.log.Warning("terminate for %s (reason %T) has %d extra bytes", target, reason, len(tail)) + } + c.core.RouteTerminatePID(target, reason) case protoMessageTerminateName, protoMessageTerminateNameCache: @@ -2431,15 +2747,16 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - reason, ok := msg.(error) if ok == false { c.log.Error("received malformed TerminateName message: %v", msg) continue } + + if len(tail) > 0 { + c.log.Warning("terminate for %s (reason %T) has %d extra bytes", processid, reason, len(tail)) + } + c.core.RouteTerminateProcessID(processid, reason) case protoMessageTerminateEvent, protoMessageTerminateEventCache: @@ -2497,15 +2814,16 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - reason, ok := msg.(error) if ok == false { c.log.Error("received malformed TerminateEvent message: %v", msg) continue } + + if len(tail) > 0 { + c.log.Warning("terminate for %s (reason %T) has %d extra bytes", event, reason, len(tail)) + } + c.core.RouteTerminateEvent(event, reason) case protoMessageTerminateAlias: @@ -2533,16 +2851,16 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) - } - reason, ok := msg.(error) if ok == false { c.log.Error("received malformed TerminateAlias message: %v", msg) continue } + if len(tail) > 0 { + c.log.Warning("terminate for %s (reason %T) has %d extra bytes", target, reason, len(tail)) + } + c.core.RouteTerminateAlias(target, reason) case protoMessageAny: @@ -2560,10 +2878,10 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { } if len(tail) > 0 { - c.log.Warning("message has extra bytes: %#v", tail) + c.log.Warning("unaddressed message %T has %d extra bytes", msg, len(tail)) } - c.routeMessage(msg) + c.dispatchRoute(msg) case protoMessageZ: if buf.Len() < 10 { @@ -2578,6 +2896,9 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { c.log.Error("unable to decompress message (gzip), ignored: %s", err) continue } + c.decompressedRecv.Add(1) + c.decompressedBytesRecv.Add(uint64(buf.Len())) + c.decompressedOrigRecv.Add(uint64(dbuf.Len())) lib.ReleaseBuffer(buf) buf = dbuf goto re @@ -2588,6 +2909,9 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { c.log.Error("unable to decompress message (lzw), ignored: %s", err) continue } + c.decompressedRecv.Add(1) + c.decompressedBytesRecv.Add(uint64(buf.Len())) + c.decompressedOrigRecv.Add(uint64(dbuf.Len())) lib.ReleaseBuffer(buf) buf = dbuf goto re @@ -2598,6 +2922,9 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { c.log.Error("unable to decompress message (zlib), ignored: %s", err) continue } + c.decompressedRecv.Add(1) + c.decompressedBytesRecv.Add(uint64(buf.Len())) + c.decompressedOrigRecv.Add(uint64(dbuf.Len())) lib.ReleaseBuffer(buf) buf = dbuf goto re @@ -2607,9 +2934,40 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { continue } - // case protoMessageF: - // TODO fragmentation - // TODO check the message size after assembling + case protoMessageT: + if buf.Len() < 40 { + c.log.Error("malformed message (too small MessageT)") + continue + } + tracing.ID[0] = binary.BigEndian.Uint64(buf.B[8:16]) + tracing.ID[1] = binary.BigEndian.Uint64(buf.B[16:24]) + tracing.SpanID = binary.BigEndian.Uint64(buf.B[24:32]) + c.tracedReceived.Add(1) + buf.B = buf.B[32:] + goto re + + case protoMessageF: + if c.fragmentation == false { + c.log.Warning("received fragment but fragmentation disabled, ignored") + continue + } + + order := buf.B[6] + if order > 0 && localFragments != nil { + // ordered path: per-queue assembly + buf = c.handleFragmentOrdered(buf, localFragments) + if buf == nil { + continue + } + goto re + } + + // unordered path: shared assembly + buf = c.handleFragmentUnordered(buf) + if buf == nil { + continue + } + goto re // case protoMessageP: // TODO proxy @@ -2619,12 +2977,6 @@ func (c *connection) handleRecvQueue(q lib.QueueMPSC) { lib.ReleaseBuffer(buf) } - // TODO - // check if connection has been terminated - // if c.terminated { - // return - // } - } } @@ -2663,7 +3015,7 @@ func (c *connection) read(conn net.Conn, buf *lib.Buffer) (*lib.Buffer, error) { return nil, fmt.Errorf("received too long message (len: %d, limit: %d)", l, c.node_maxmessagesize) } - if lib.Trace() { + if lib.Verbose() { c.log.Trace("...recv buf.Len: %d, packet %d (expect: %d)", buf.Len(), l, expect) } @@ -2681,6 +3033,165 @@ func (c *connection) read(conn net.Conn, buf *lib.Buffer) (*lib.Buffer, error) { } } +// dispatchRoute is the entry point for protoMessageAny payloads from the +// receive goroutine. Fast-lane messages (ACKs, cache updates) are handled +// inline so their latency is bounded by a single channel send. Everything +// else (Link/Monitor/Spawn/etc) goes to a per-shard route queue and is +// processed by routeWorker, freeing the receive goroutine to keep decoding. +func (c *connection) dispatchRoute(msg any) { + const mask = uint64(routeQueuesPerConn - 1) + var idx uint64 + var ref gen.Ref + + switch m := msg.(type) { + + // Fast lane: handled inline in the receive goroutine. MessageResult + // must never queue behind heavy work — it is the ACK that wakes up + // the sender of an outbound request (including SendImportant). + case MessageResult: + c.requestsMutex.RLock() + ch, found := c.requests[m.Ref] + c.requestsMutex.RUnlock() + if found { + select { + case ch <- m: + default: + } + } + return + case MessageUpdateCache: + c.applyCacheUpdate(m) + return + + // Slow lane: extract the queue index (shard-aligned with TM) and the + // request ref (for deadline filtering) in one type switch. + case MessageLinkPID: + idx, ref = m.Target.ID&mask, m.Ref + case MessageUnlinkPID: + idx, ref = m.Target.ID&mask, m.Ref + case MessageMonitorPID: + idx, ref = m.Target.ID&mask, m.Ref + case MessageDemonitorPID: + idx, ref = m.Target.ID&mask, m.Ref + + case MessageLinkAlias: + idx, ref = m.Target.ID[1]&mask, m.Ref + case MessageUnlinkAlias: + idx, ref = m.Target.ID[1]&mask, m.Ref + case MessageMonitorAlias: + idx, ref = m.Target.ID[1]&mask, m.Ref + case MessageDemonitorAlias: + idx, ref = m.Target.ID[1]&mask, m.Ref + + case MessageLinkProcessID: + idx, ref = lib.HashString64(string(m.Target.Name))&mask, m.Ref + case MessageUnlinkProcessID: + idx, ref = lib.HashString64(string(m.Target.Name))&mask, m.Ref + case MessageMonitorProcessID: + idx, ref = lib.HashString64(string(m.Target.Name))&mask, m.Ref + case MessageDemonitorProcessID: + idx, ref = lib.HashString64(string(m.Target.Name))&mask, m.Ref + + case MessageLinkEvent: + idx, ref = lib.HashString64(string(m.Target.Name))&mask, m.Ref + case MessageUnlinkEvent: + idx, ref = lib.HashString64(string(m.Target.Name))&mask, m.Ref + case MessageMonitorEvent: + idx, ref = lib.HashString64(string(m.Target.Name))&mask, m.Ref + case MessageDemonitorEvent: + idx, ref = lib.HashString64(string(m.Target.Name))&mask, m.Ref + + case MessageSpawn: + idx, ref = lib.HashString64(string(m.Name))&mask, m.Ref + case MessageApplicationStart: + idx, ref = lib.HashString64(string(m.Name))&mask, m.Ref + case MessageApplicationInfo: + idx, ref = lib.HashString64(string(m.Name))&mask, m.Ref + + default: + c.log.Warning("unknown routed message type %T (ignored)", msg) + return + } + + // Drop requests whose deadline already passed before reaching the queue. + // Ref.IsAlive() returns true when no deadline is set, so refs from older + // peers (without a deadline) always pass through. + if ref.IsAlive() == false { + if lib.Verbose() { + c.log.Trace("dropped stale message %T at dispatch", msg) + } + return + } + + queue := c.routeQueues[idx] + queue.Push(msg) + if queue.Lock() { + go c.routeWorker(queue) + } +} + +// routeWorker drains a single route queue. Spawned on demand by dispatchRoute +// when the queue transitions from empty to non-empty. Exits when the queue +// is fully drained. Same re-entry pattern as handleRecvQueue. +func (c *connection) routeWorker(q lib.QueueMPSC) { + if lib.Recover() { + defer func() { + if r := recover(); r != nil { + pc, fn, line, _ := runtime.Caller(2) + c.log.Panic("panic in routeWorker: %#v at %s[%s:%d]", + r, runtime.FuncForPC(pc).Name(), fn, line) + c.Terminate(gen.TerminateReasonPanic) + } + }() + } + + for { + v, ok := q.Pop() + if ok == false { + q.Unlock() + if q.Item() == nil { + return + } + if locked := q.Lock(); locked == false { + return + } + continue + } + c.routeMessage(v) + } +} + +// applyCacheUpdate applies an inbound cache update and acks the sender. +// Called inline from the receive goroutine via the fast lane. +func (c *connection) applyCacheUpdate(m MessageUpdateCache) { + for k, v := range m.AtomCache { + entry, exist := c.decodeOptions.AtomCache.LoadOrStore(k, v) + if exist { + c.log.Warning("updating atom cache ignored entry (already exist): %d => %v", k, entry) + } + } + for k, v := range m.AtomMapping { + entry, exist := c.decodeOptions.AtomMapping.LoadOrStore(k, v) + if exist { + c.log.Warning("updating atom mapping ignored entry (already exist): %s => %v", k, entry) + } + } + for k, v := range m.RegCache { + entry, exist := c.decodeOptions.RegCache.LoadOrStore(k, v) + if exist { + c.log.Warning("updating reg cache ignored entry (already exist): %d => %v", k, entry) + } + } + for k, v := range m.ErrCache { + entry, exist := c.decodeOptions.ErrCache.LoadOrStore(k, v) + if exist { + c.log.Warning("updating err cache ignored entry (already exist): %d => %v", k, entry) + } + } + result := MessageResult{Ref: m.Ref} + c.sendAny(result, 0, 0, gen.Compression{}) +} + func (c *connection) routeMessage(msg any) { switch m := msg.(type) { case MessageResult: @@ -2736,163 +3247,233 @@ func (c *connection) routeMessage(msg any) { case MessageLinkPID: // TODO check the source/target node name + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteLinkPID(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + if err == nil { + c.core.RouteUnlinkPID(m.Source, m.Target) + } + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(m.Target.ID % 255) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageUnlinkPID: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteUnlinkPID(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(m.Target.ID % 255) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageLinkProcessID: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteLinkProcessID(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + if err == nil { + c.core.RouteUnlinkProcessID(m.Source, m.Target) + } + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(0) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageUnlinkProcessID: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteUnlinkProcessID(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(0) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageLinkAlias: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteLinkAlias(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + if err == nil { + c.core.RouteUnlinkAlias(m.Source, m.Target) + } + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(m.Target.ID[1] % 255) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageUnlinkAlias: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteUnlinkAlias(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(m.Target.ID[1] % 255) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageLinkEvent: + if m.Ref.IsAlive() == false { + return + } r, err := c.core.RouteLinkEvent(m.Source, m.Target) - result := MessageResult{ - Result: r, - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + if err == nil { + c.core.RouteUnlinkEvent(m.Source, m.Target) + } + return } + result := MessageResult{Result: r, Error: err, Ref: m.Ref} order := uint8(0) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageUnlinkEvent: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteUnlinkEvent(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(0) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageMonitorPID: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteMonitorPID(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + if err == nil { + c.core.RouteDemonitorPID(m.Source, m.Target) + } + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(m.Target.ID % 255) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageDemonitorPID: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteDemonitorPID(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(m.Target.ID % 255) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageMonitorProcessID: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteMonitorProcessID(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + if err == nil { + c.core.RouteDemonitorProcessID(m.Source, m.Target) + } + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(0) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageDemonitorProcessID: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteDemonitorProcessID(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(0) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageMonitorAlias: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteMonitorAlias(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + if err == nil { + c.core.RouteDemonitorAlias(m.Source, m.Target) + } + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(m.Target.ID[1] % 255) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageDemonitorAlias: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteDemonitorAlias(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(m.Target.ID[1] % 255) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageMonitorEvent: + if m.Ref.IsAlive() == false { + return + } r, err := c.core.RouteMonitorEvent(m.Source, m.Target) - result := MessageResult{ - Result: r, - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + if err == nil { + c.core.RouteDemonitorEvent(m.Source, m.Target) + } + return } + result := MessageResult{Result: r, Error: err, Ref: m.Ref} order := uint8(0) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageDemonitorEvent: + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteDemonitorEvent(m.Source, m.Target) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(0) orderPeer := uint8(m.Source.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) @@ -2902,12 +3483,19 @@ func (c *connection) routeMessage(msg any) { c.log.Warning("remote spawn is not allowed for %s", c.peer) return } + if m.Ref.IsAlive() == false { + return + } pid, err := c.core.RouteSpawn(c.core.Name(), m.Name, m.Options, c.peer) - result := MessageResult{ - Error: err, - Result: pid, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + // sender already gave up; kill the freshly-spawned process so + // the next retry doesn't see a stale instance. + if err == nil { + c.core.RouteSendExit(c.core.PID(), pid, gen.TerminateReasonKill) + } + return } + result := MessageResult{Error: err, Result: pid, Ref: m.Ref} order := uint8(0) orderPeer := uint8(m.Options.ParentPID.ID % 255) c.sendAny(result, order, orderPeer, gen.Compression{}) @@ -2917,23 +3505,30 @@ func (c *connection) routeMessage(msg any) { c.log.Warning("remote application start is not allowed for %s", c.peer) return } + if m.Ref.IsAlive() == false { + return + } err := c.core.RouteApplicationStart(m.Name, m.Mode, m.Options, c.peer) - result := MessageResult{ - Error: err, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + // app may have started; rolling it back is non-trivial so we + // just skip the reply. Sender's retry hits ErrApplicationRunning + // which is naturally idempotent. + return } + result := MessageResult{Error: err, Ref: m.Ref} order := uint8(0) orderPeer := uint8(0) c.sendAny(result, order, orderPeer, gen.Compression{}) case MessageApplicationInfo: + if m.Ref.IsAlive() == false { + return + } info, err := c.core.RouteApplicationInfo(m.Name) - - result := MessageResult{ - Error: err, - Result: info, - Ref: m.Ref, + if m.Ref.IsAlive() == false { + return } + result := MessageResult{Error: err, Result: info, Ref: m.Ref} order := uint8(0) orderPeer := uint8(0) c.sendAny(result, order, orderPeer, gen.Compression{}) @@ -2945,40 +3540,49 @@ func (c *connection) routeMessage(msg any) { func (c *connection) sendAny(msg any, order uint8, orderPeer uint8, compression gen.Compression) error { buf := lib.TakeBuffer() - buf.Allocate(8) // for the header + h := protoWrapReserve + buf.Allocate(h + 8) // reserve + header if err := edf.Encode(msg, buf, c.encodeOptions); err != nil { return err } - if buf.Len() > math.MaxUint32 { + if buf.Len()-h > math.MaxUint32 { return gen.ErrTooLarge } - buf.B[0] = protoMagic - buf.B[1] = protoVersion - binary.BigEndian.PutUint32(buf.B[2:6], uint32(buf.Len())) - buf.B[6] = orderPeer - buf.B[7] = protoMessageAny + buf.B[h+0] = protoMagic + buf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[h+2:h+6], uint32(buf.Len()-h)) + buf.B[h+6] = orderPeer + buf.B[h+7] = protoMessageAny - return c.send(buf, order, compression) + return c.send(buf, order, compression, gen.Tracing{}) } func (c *connection) wait() { c.wg.Wait() } -func (c *connection) send(buf *lib.Buffer, order uint8, compression gen.Compression) error { +func (c *connection) send(buf *lib.Buffer, order uint8, compression gen.Compression, tracing gen.Tracing) error { + // msgStart points to the beginning of the actual message within buf.B. + // Send* methods write the message at offset protoWrapReserve, + // leaving reserved space at the front for wrapping headers (tracing, proxy). + msgStart := protoWrapReserve - if compression.Enable && buf.Len() > compression.Threshold { + if c.peer_maxmessagesize > 0 && buf.Len()-msgStart > c.peer_maxmessagesize { + return gen.ErrTooLarge + } + + if compression.Enable && buf.Len()-msgStart > compression.Threshold { var zbuf *lib.Buffer var err error - // 1 - protoMagic - // 1 - protoVersion - // 4 - length - // 1 - order - // 1 - protoMessageZ - // 1 - compression type - preallocate := uint(9) + // strip reserve before compressing — only compress the actual message + orderByte := buf.B[msgStart+6] + origLen := buf.Len() - msgStart + buf.B = buf.B[msgStart:] + + // protoWrapReserve (for wrapping) + 9 (Z header: magic, version, length, order, type, compression_type) + preallocate := uint(protoWrapReserve + 9) switch compression.Type { case gen.CompressionTypeZLIB: @@ -2999,19 +3603,41 @@ func (c *connection) send(buf *lib.Buffer, order uint8, compression gen.Compress } } - zbuf.B[0] = protoMagic - zbuf.B[1] = protoVersion - binary.BigEndian.PutUint32(zbuf.B[2:6], uint32(zbuf.Len())) - zbuf.B[6] = buf.B[6] // keep order of the original message - zbuf.B[7] = protoMessageZ - zbuf.B[8] = compression.Type.ID() + h := protoWrapReserve + zbuf.B[h+0] = protoMagic + zbuf.B[h+1] = protoVersion + binary.BigEndian.PutUint32(zbuf.B[h+2:h+6], uint32(zbuf.Len()-h)) + zbuf.B[h+6] = orderByte + zbuf.B[h+7] = protoMessageZ + zbuf.B[h+8] = compression.Type.ID() + + c.compressedSent.Add(1) + c.compressedOrigBytesSent.Add(uint64(origLen)) + c.compressedBytesSent.Add(uint64(zbuf.Len() - h)) lib.ReleaseBuffer(buf) buf = zbuf + msgStart = h } - if c.peer_maxmessagesize > 0 && buf.Len() > c.peer_maxmessagesize { - return gen.ErrTooLarge + // tracing wrapper — uses reserved space, no copy + // only wrap if both nodes have tracing enabled + if c.tracing == true && tracing.ID != [2]uint64{} { + msgStart -= 32 + buf.B[msgStart+0] = protoMagic + buf.B[msgStart+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[msgStart+2:msgStart+6], uint32(buf.Len()-msgStart)) + buf.B[msgStart+6] = buf.B[msgStart+32+6] // order from inner message + buf.B[msgStart+7] = protoMessageT + binary.BigEndian.PutUint64(buf.B[msgStart+8:msgStart+16], tracing.ID[0]) + binary.BigEndian.PutUint64(buf.B[msgStart+16:msgStart+24], tracing.ID[1]) + binary.BigEndian.PutUint64(buf.B[msgStart+24:msgStart+32], tracing.SpanID) + c.tracedSent.Add(1) + } + + // fragmentation + if c.fragmentation == true && buf.Len()-msgStart > c.fragmentSize { + return c.sendFragmented(buf, msgStart, order) } var pi *pool_item @@ -3031,22 +3657,322 @@ func (c *connection) send(buf *lib.Buffer, order uint8, compression gen.Compress } c.pool_mutex.RUnlock() + dataLen := uint64(buf.Len() - msgStart) + _, err := pi.fl.Write(buf.B[msgStart:]) + lib.ReleaseBuffer(buf) + if err != nil { + pi.connection.Close() + return err + } + atomic.AddUint64(&c.messagesOut, 1) - atomic.AddUint64(&c.bytesOut, uint64(buf.Len())) + atomic.AddUint64(&c.bytesOut, dataLen) + return nil +} - // TODO - // add proxy, fragmentation support - // c.transitOut++ - // if buf.Len() < protoFragmentSize { +func (c *connection) sendFragmented(buf *lib.Buffer, msgStart int, order uint8) error { + dataLen := buf.Len() - msgStart + maxPayload := c.fragmentSize - 16 // 16 = ENP header (8) + fragment header (8) + + totalFragments := uint16((dataLen + maxPayload - 1) / maxPayload) + + sequenceID := c.nextSequenceID.Add(1) + + if lib.Verbose() { + c.log.Trace("fragmenting message: %d bytes, %d fragments (seq=%d)", + dataLen, totalFragments, sequenceID) + } + + // for ordered messages, select pool item once (all fragments must go + // through the same TCP connection to preserve order on the receiver) + var orderedPI *pool_item + if order > 0 { + c.pool_mutex.RLock() + l := len(c.pool) + if l == 0 { + c.pool_mutex.RUnlock() + lib.ReleaseBuffer(buf) + return gen.ErrNoConnection + } + orderedPI = c.pool[int(order)%l] + c.pool_mutex.RUnlock() + } + + // Zero-copy fragmentation: write fragment headers in-place. + // First fragment uses reserved space before msgStart. + // Subsequent fragments overwrite the tail of the previous (already sent) chunk. + offset := 0 + for idx := uint16(0); idx < totalFragments; idx++ { + chunkSize := dataLen - offset + if chunkSize > maxPayload { + chunkSize = maxPayload + } + + // fragment header goes 16 bytes before current chunk + hdr := msgStart + offset - 16 + buf.B[hdr+0] = protoMagic + buf.B[hdr+1] = protoVersion + binary.BigEndian.PutUint32(buf.B[hdr+2:hdr+6], uint32(16+chunkSize)) + buf.B[hdr+6] = order + buf.B[hdr+7] = protoMessageF + binary.BigEndian.PutUint32(buf.B[hdr+8:hdr+12], sequenceID) + binary.BigEndian.PutUint16(buf.B[hdr+12:hdr+14], idx) + binary.BigEndian.PutUint16(buf.B[hdr+14:hdr+16], totalFragments) + + // select pool item + pi := orderedPI + if order == 0 { + c.pool_mutex.RLock() + l := len(c.pool) + if l == 0 { + c.pool_mutex.RUnlock() + lib.ReleaseBuffer(buf) + return gen.ErrNoConnection + } + neworder := atomic.AddUint32(&c.order, 1) + pi = c.pool[int(neworder)%l] + c.pool_mutex.RUnlock() + } + + _, err := pi.fl.Write(buf.B[hdr : hdr+16+chunkSize]) + if err != nil { + pi.connection.Close() + lib.ReleaseBuffer(buf) + return err + } + + atomic.AddUint64(&c.bytesOut, uint64(16+chunkSize)) + offset += chunkSize + } + + atomic.AddUint64(&c.messagesOut, 1) - pi.fl.Write(buf.B) lib.ReleaseBuffer(buf) + + c.fragmentsSent.Add(uint64(totalFragments)) + c.fragmentMessagesSent.Add(1) + return nil +} + +const maxFragmentCount = 10000 + +func (c *connection) handleFragmentOrdered(buf *lib.Buffer, assemblies map[uint32]*fragmentAssembly) *lib.Buffer { + if buf.Len() < 16 { + c.log.Warning("fragment too short: %d bytes", buf.Len()) + return nil + } + + sequenceID := binary.BigEndian.Uint32(buf.B[8:12]) + fragIndex := binary.BigEndian.Uint16(buf.B[12:14]) + totalFragments := binary.BigEndian.Uint16(buf.B[14:16]) + payload := buf.B[16:] + + if fragIndex >= totalFragments || totalFragments == 0 { + c.log.Warning("invalid fragment index %d/%d (seq=%d)", fragIndex, totalFragments, sequenceID) + return nil + } + + asm, exists := assemblies[sequenceID] + if exists == false { + if totalFragments > maxFragmentCount { + c.log.Warning("too many fragments: %d (seq=%d)", totalFragments, sequenceID) + return nil + } + asm = &fragmentAssembly{ + totalFragments: totalFragments, + payloads: make([][]byte, 0, totalFragments), + deadline: time.Now().Add(c.fragmentTimeout), + } + assemblies[sequenceID] = asm + } + + // rejected assembly (exceeded maxmessagesize), ignore subsequent fragments + if asm.payloads == nil { + return nil + } + + if asm.totalFragments != totalFragments { + c.log.Warning("fragment total mismatch (seq=%d)", sequenceID) + delete(assemblies, sequenceID) + return nil + } + + // TCP order guarantee, append + asm.payloads = append(asm.payloads, append([]byte(nil), payload...)) + asm.received++ + asm.totalBytes += len(payload) + + // check accumulated size against receiver limit + if c.node_maxmessagesize > 0 && asm.totalBytes > c.node_maxmessagesize { + asm.payloads = nil + c.log.Warning("fragmented message exceeds max size: %d (max %d, seq=%d)", + asm.totalBytes, c.node_maxmessagesize, sequenceID) + return nil + } + + c.fragmentsReceived.Add(1) + + if asm.received < asm.totalFragments { + return nil + } + + // assembly complete + delete(assemblies, sequenceID) + + // cleanup stale assemblies from dead senders + if len(assemblies) > 0 { + now := time.Now() + for seqID, asm := range assemblies { + if now.After(asm.deadline) { + delete(assemblies, seqID) + c.fragmentTimeouts.Add(1) + } + } + } + + reassembled := lib.TakeBuffer() + reassembled.Allocate(asm.totalBytes) + + offset := 0 + for _, p := range asm.payloads { + copy(reassembled.B[offset:], p) + offset += len(p) + } + + c.fragmentMessagesRecv.Add(1) + atomic.AddUint64(&c.messagesIn, 1) + + if lib.Verbose() { + c.log.Trace("fragment assembly complete: seq=%d, %d bytes, %d fragments", + sequenceID, asm.totalBytes, asm.totalFragments) + } + + return reassembled +} + +func (c *connection) handleFragmentUnordered(buf *lib.Buffer) *lib.Buffer { + if buf.Len() < 16 { + c.log.Warning("fragment too short: %d bytes", buf.Len()) + return nil + } + + sequenceID := binary.BigEndian.Uint32(buf.B[8:12]) + fragIndex := binary.BigEndian.Uint16(buf.B[12:14]) + totalFragments := binary.BigEndian.Uint16(buf.B[14:16]) + payload := buf.B[16:] + + if fragIndex >= totalFragments || totalFragments == 0 { + c.log.Warning("invalid fragment index %d/%d (seq=%d)", fragIndex, totalFragments, sequenceID) + return nil + } + + c.sharedFragMu.Lock() + + asm, exists := c.sharedFragments[sequenceID] + if exists == false { + if totalFragments > maxFragmentCount { + c.sharedFragMu.Unlock() + c.log.Warning("too many fragments: %d (seq=%d)", totalFragments, sequenceID) + return nil + } + if len(c.sharedFragments) >= c.maxFragmentAssemblies { + c.sharedFragMu.Unlock() + c.log.Warning("too many concurrent unordered assemblies, dropping fragment") + return nil + } + + asm = &fragmentAssembly{ + totalFragments: totalFragments, + payloads: make([][]byte, totalFragments), // indexed + deadline: time.Now().Add(c.fragmentTimeout), + } + c.sharedFragments[sequenceID] = asm + c.sharedFragTimer.Reset(5 * time.Second) + } + + // rejected assembly (exceeded maxmessagesize), ignore subsequent fragments + if asm.payloads == nil { + c.sharedFragMu.Unlock() + return nil + } + + if asm.totalFragments != totalFragments { + c.sharedFragMu.Unlock() + c.log.Warning("fragment total mismatch (seq=%d)", sequenceID) + return nil + } + + // duplicate check + if asm.payloads[fragIndex] != nil { + c.sharedFragMu.Unlock() + return nil + } + + asm.payloads[fragIndex] = append([]byte(nil), payload...) + asm.received++ + asm.totalBytes += len(payload) - // } + // check accumulated size against receiver limit + if c.node_maxmessagesize > 0 && asm.totalBytes > c.node_maxmessagesize { + asm.payloads = nil + c.sharedFragMu.Unlock() + c.log.Warning("fragmented message exceeds max size: %d (max %d, seq=%d)", + asm.totalBytes, c.node_maxmessagesize, sequenceID) + return nil + } + + c.fragmentsReceived.Add(1) + + if asm.received < asm.totalFragments { + c.sharedFragMu.Unlock() + return nil + } - // message must be fragmented - // panic("TODO") + // assembly complete + delete(c.sharedFragments, sequenceID) + c.sharedFragMu.Unlock() + + // build reassembled message outside mutex + reassembled := lib.TakeBuffer() + reassembled.Allocate(asm.totalBytes) + + offset := 0 + for _, p := range asm.payloads { + copy(reassembled.B[offset:], p) + offset += len(p) + } + + c.fragmentMessagesRecv.Add(1) + atomic.AddUint64(&c.messagesIn, 1) + + if lib.Verbose() { + c.log.Trace("fragment assembly complete: seq=%d, %d bytes, %d fragments", + sequenceID, asm.totalBytes, asm.totalFragments) + } + + return reassembled +} + +func (c *connection) cleanupSharedFragments() { + c.sharedFragMu.Lock() + defer c.sharedFragMu.Unlock() + + now := time.Now() + for seqID, asm := range c.sharedFragments { + if now.After(asm.deadline) { + if lib.Verbose() { + c.log.Trace("unordered fragment timeout: seq=%d, %d/%d received", + seqID, asm.received, asm.totalFragments) + } + delete(c.sharedFragments, seqID) + c.fragmentTimeouts.Add(1) + } + } + + if len(c.sharedFragments) > 0 { + c.sharedFragTimer.Reset(5 * time.Second) + } } func (c *connection) waitResult(ref gen.Ref, ch chan MessageResult) (result MessageResult) { diff --git a/net/proto/enp.go b/net/proto/enp.go index 8b4293ee2..88b202d91 100644 --- a/net/proto/enp.go +++ b/net/proto/enp.go @@ -2,6 +2,7 @@ package proto import ( "fmt" + "reflect" "sync" "time" @@ -53,6 +54,7 @@ func (e *enp) NewConnection(core gen.Core, result gen.HandshakeResult, log gen.L pool_size: opts.PoolSize, pool_dsn: opts.PoolDSN, + tls: opts.TLS, encodeOptions: edf.Options{ AtomCache: opts.EncodeAtomCache, @@ -68,6 +70,54 @@ func (e *enp) NewConnection(core gen.Core, result gen.HandshakeResult, log gen.L Cache: new(sync.Map), }, requests: make(map[gen.Ref]chan MessageResult), + + softwareKeepAlive: result.NodeFlags.EnableSoftwareKeepAlive > 0 && + result.PeerFlags.EnableSoftwareKeepAlive > 0, + } + + if conn.softwareKeepAlive { + myPeriod := time.Duration(result.NodeFlags.EnableSoftwareKeepAlive) * time.Second + peerPeriod := time.Duration(result.PeerFlags.EnableSoftwareKeepAlive) * time.Second + misses := opts.SoftwareKeepAliveMisses + if misses == 0 { + misses = gen.DefaultSoftwareKeepAliveMisses + } + conn.softwareKeepAlivePeriod = myPeriod + conn.softwareKeepAliveMisses = misses + conn.softwareKeepAliveTimeout = peerPeriod * time.Duration(misses) + conn.softwareKeepAliveMessage = []byte{ + protoMagic, protoVersion, 0, 0, 0, 8, 0, protoMessageK, + } + } + + if result.NodeFlags.EnableClockSkew == true && + result.PeerFlags.EnableClockSkew == true { + conn.clockSkew = true + } + + if result.NodeFlags.EnableTracing == true && + result.PeerFlags.EnableTracing == true { + conn.tracing = true + } + + if result.NodeFlags.EnableFragmentation == true && + result.PeerFlags.EnableFragmentation == true { + conn.fragmentation = true + conn.fragmentSize = gen.DefaultFragmentSize + if opts.FragmentSize > 0 { + conn.fragmentSize = opts.FragmentSize + } + conn.fragmentTimeout = gen.DefaultFragmentTimeout + if opts.FragmentTimeout > 0 { + conn.fragmentTimeout = time.Duration(opts.FragmentTimeout) * time.Second + } + conn.maxFragmentAssemblies = gen.DefaultMaxFragmentAssemblies + if opts.MaxFragmentAssemblies > 0 { + conn.maxFragmentAssemblies = opts.MaxFragmentAssemblies + } + conn.sharedFragments = make(map[uint32]*fragmentAssembly) + conn.sharedFragTimer = time.AfterFunc(time.Hour, conn.cleanupSharedFragments) + conn.sharedFragTimer.Stop() } if len(result.AtomMapping) > 0 { @@ -81,10 +131,25 @@ func (e *enp) NewConnection(core gen.Core, result gen.HandshakeResult, log gen.L // init recv queues. create 4 recv queues per connection // since the decoding is more costly comparing to the encoding - for i := 0; i < opts.PoolSize*4; i++ { + numQueues := opts.PoolSize * 4 + for i := 0; i < numQueues; i++ { conn.recvQueues = append(conn.recvQueues, lib.NewQueueMPSC()) } + // init route queues for protoMessageAny dispatch. These drain Link/Monitor/ + // Spawn/etc so decoding goroutines never block on synchronous TM/core work. + for i := range conn.routeQueues { + conn.routeQueues[i] = lib.NewQueueMPSC() + } + + // init per-queue ordered fragment assembly maps + if conn.fragmentation { + conn.orderedFragments = make([]map[uint32]*fragmentAssembly, numQueues) + for i := 0; i < numQueues; i++ { + conn.orderedFragments[i] = make(map[uint32]*fragmentAssembly) + } + } + return conn, nil } @@ -115,12 +180,12 @@ func (e *enp) Serve(c gen.Connection, redial gen.NetworkDial) error { n := i % len(conn.pool_dsn) dsn := conn.pool_dsn[n] - if lib.Trace() { + if lib.Verbose() { conn.log.Trace("dialing %s (pool: %d of %d)", dsn, i+1, conn.pool_size) } nc, tail, err := redial(dsn, conn.id) if err != nil { - if lib.Trace() { + if lib.Verbose() { conn.log.Trace("dialing %s failed: %s", dsn, err) } continue @@ -143,3 +208,23 @@ func (e *enp) Version() gen.Version { License: gen.LicenseMIT, } } + +// gen.TypeRegistry implementation (wire-format type registry on top of edf). + +func (e *enp) RegisterType(v any) error { return edf.RegisterTypeOf(v) } +func (e *enp) RegisterTypes(types []any) error { return edf.RegisterTypesOf(types) } +func (e *enp) RegisterError(err error) error { return edf.RegisterError(err) } +func (e *enp) RegisterAtom(a gen.Atom) error { return edf.RegisterAtom(a) } + +func (e *enp) RegisteredTypes() []gen.RegisteredTypeInfo { + list := edf.RegisteredTypes() + ver := e.Version().Str() + for i := range list { + list[i].Proto = ver + } + return list +} + +func (e *enp) LookupType(name string) (reflect.Type, bool) { + return edf.LookupType(name) +} diff --git a/net/proto/types.go b/net/proto/types.go index 796b98c77..0dfe7eb98 100644 --- a/net/proto/types.go +++ b/net/proto/types.go @@ -40,11 +40,19 @@ const ( // any structured message (link/monitor/spawn/etc...) protoMessageAny byte = 199 - // order: compressed -> encrypted -> fragmented -> proxy + // order: compressed -> encrypted -> traced -> fragmented -> proxy protoMessageZ byte = 200 // compressed protoMessageE byte = 201 // encrypted protoMessageF byte = 202 // fragmented protoMessageP byte = 203 // proxy + protoMessageT byte = 204 // tracing wrapper + + protoMessageK byte = 208 // keepalive + protoMessageS byte = 209 // skew measurement + + // reserved space at the beginning of every message buffer + // for wrapping headers (tracing, proxy, etc.) without copying + protoWrapReserve int = 128 // TODO // protoFragmentSize int = 65000 diff --git a/net/registrar/client.go b/net/registrar/client.go index d4584da0a..498b6ed78 100644 --- a/net/registrar/client.go +++ b/net/registrar/client.go @@ -69,7 +69,7 @@ func (c *client) Resolve(name gen.Atom) ([]gen.Route, error) { return nil, gen.ErrIncorrect } dsn := net.JoinHostPort(host, strconv.Itoa(int(c.options.Port))) - if lib.Trace() { + if lib.Verbose() { c.node.Log().Trace("resolving %s using registrar %s", name, dsn) } conn, err := net.Dial("udp", dsn) diff --git a/node/acceptor.go b/node/acceptor.go index 44203a359..d212ee7ec 100644 --- a/node/acceptor.go +++ b/node/acceptor.go @@ -2,6 +2,7 @@ package node import ( "net" + "sync/atomic" "ergo.services/ergo/gen" ) @@ -24,6 +25,12 @@ type acceptor struct { proto gen.NetworkProto atom_mapping map[gen.Atom]gen.Atom + + handshaking atomic.Int32 // current number of in-flight handshakes + maxHandshakes int32 // 0 = unlimited + handshakeErrors atomic.Uint64 // cumulative handshake failures + + software_keepalive_misses int } // gen.Acceptor interface implementation @@ -76,5 +83,6 @@ func (a *acceptor) Info() gen.AcceptorInfo { info.RegistrarServer = regInfo.Server } info.RegistrarVersion = regInfo.Version + info.HandshakeErrors = a.handshakeErrors.Load() return info } diff --git a/node/application.go b/node/application.go index 40b241d58..58e6012bd 100644 --- a/node/application.go +++ b/node/application.go @@ -60,10 +60,7 @@ func (a *application) start(mode gen.ApplicationMode, options gen.ApplicationOpt deadline := time.Now().Unix() + int64(timeout) ref, err := a.node.MakeRefWithDeadline(deadline) if err != nil { - a.group.Range(func(pid gen.PID, _ bool) bool { - a.node.Kill(pid) - return true - }) + a.killMembers() atomic.StoreInt32(&a.state, int32(gen.ApplicationStateLoaded)) return err } @@ -83,10 +80,7 @@ func (a *application) start(mode gen.ApplicationMode, options gen.ApplicationOpt pid, err := a.node.spawn(item.Factory, opts) if err != nil { - a.group.Range(func(pid gen.PID, _ bool) bool { - a.node.Kill(pid) - return true - }) + a.killMembers() atomic.StoreInt32(&a.state, int32(gen.ApplicationStateLoaded)) return err } @@ -140,14 +134,14 @@ func (a *application) stop(force bool, timeout time.Duration) error { // update mode to prevent triggering 'permantent' mode a.mode = gen.ApplicationModeTemporary - a.group.Range(func(pid gen.PID, _ bool) bool { + pids := a.collectMemberPIDs() + for _, pid := range pids { if force { a.node.Kill(pid) } else { a.node.SendExit(pid, gen.TerminateReasonShutdown) } - return true - }) + } if force { a.reason = gen.TerminateReasonKill @@ -180,10 +174,7 @@ func (a *application) terminate(pid gen.PID, reason error) { a.node.Log(). Info("application %s (%s) will be stopped due to termination of %s with reason: %s", a.spec.Name, a.mode, pid, reason) a.reason = reason - a.group.Range(func(pid gen.PID, _ bool) bool { - a.node.SendExit(pid, gen.TerminateReasonShutdown) - return true - }) + a.exitMembers(gen.TerminateReasonShutdown) case gen.ApplicationModeTransient: if reason == gen.TerminateReasonNormal || reason == gen.TerminateReasonShutdown { // do nothing @@ -198,10 +189,7 @@ func (a *application) terminate(pid gen.PID, reason error) { break } a.reason = reason - a.group.Range(func(pid gen.PID, _ bool) bool { - a.node.SendExit(pid, gen.TerminateReasonShutdown) - return true - }) + a.exitMembers(gen.TerminateReasonShutdown) default: // do nothing } @@ -297,6 +285,27 @@ func (a *application) isRunning() bool { return atomic.LoadInt32(&a.state) == int32(gen.ApplicationStateRunning) } +func (a *application) collectMemberPIDs() []gen.PID { + var pids []gen.PID + a.group.Range(func(pid gen.PID, _ bool) bool { + pids = append(pids, pid) + return true + }) + return pids +} + +func (a *application) killMembers() { + for _, pid := range a.collectMemberPIDs() { + a.node.Kill(pid) + } +} + +func (a *application) exitMembers(reason error) { + for _, pid := range a.collectMemberPIDs() { + a.node.SendExit(pid, reason) + } +} + func (a *application) registerAppRoute() { appRoute := gen.ApplicationRoute{ Node: a.node.name, diff --git a/node/core.go b/node/core.go index cff0f3d1e..dcbebe42c 100644 --- a/node/core.go +++ b/node/core.go @@ -1,6 +1,7 @@ package node import ( + "reflect" "sync/atomic" "time" @@ -17,27 +18,60 @@ func (n *node) RouteSendPID(from gen.PID, to gen.PID, options gen.MessageOptions return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteSendPID from %s to %s", from, to) } + tracingActive := options.Tracing.ID != [2]uint64{} + var msgType string + var parentSpanID uint64 + var fromBehavior string + if tracingActive { + parentSpanID = options.Tracing.SpanID + fromBehavior = options.Tracing.Behavior + if message != nil { + msgType = reflect.TypeOf(message).String() + } + } + if to.Node != n.name { // remote connection, err := n.network.GetConnection(to.Node) if err != nil { + atomic.AddUint64(&n.sendErrorsRemote, 1) return err } - return connection.SendPID(from, to, options, message) + if tracingActive { + options.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } + if err := connection.SendPID(from, to, options, message); err != nil { + atomic.AddUint64(&n.sendErrorsRemote, 1) + return err + } + if tracingActive { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: options.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindSend, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + return nil } // local value, found := n.processes.Load(to) if found == false { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessUnknown } p := value.(*process) if alive := p.isAlive(); alive == false { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessTerminated } @@ -55,13 +89,19 @@ func (n *node) RouteSendPID(from gen.PID, to gen.PID, options gen.MessageOptions qm.Type = gen.MailboxMessageTypeRegular qm.Target = to qm.Message = message + qm.Tracing = options.Tracing + if tracingActive && from.Node == n.name { + qm.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } if ok := queue.Push(qm); ok == false { if p.fallback.Enable == false { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessMailboxFull } if p.fallback.Name == p.name { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessMailboxFull } @@ -74,6 +114,29 @@ func (n *node) RouteSendPID(from gen.PID, to gen.PID, options gen.MessageOptions return n.RouteSendProcessID(from, fbto, options, fbm) } atomic.AddUint64(&p.messagesIn, 1) + + if tracingActive { + nanos := time.Now().UnixNano() + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindSend, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindSend, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: p.sbehavior, Message: msgType, + Attributes: p.tracingAttrs, + }) + } + p.run() return nil } @@ -85,7 +148,7 @@ func (n *node) RouteSendProcessID(from gen.PID, to gen.ProcessID, options gen.Me return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteSendProcessID from %s to %s", from, to) } @@ -93,22 +156,55 @@ func (n *node) RouteSendProcessID(from gen.PID, to gen.ProcessID, options gen.Me to.Node = n.name } + tracingActive := options.Tracing.ID != [2]uint64{} + var msgType string + var parentSpanID uint64 + var fromBehavior string + if tracingActive { + parentSpanID = options.Tracing.SpanID + fromBehavior = options.Tracing.Behavior + if message != nil { + msgType = reflect.TypeOf(message).String() + } + } + if to.Node != n.name { // remote connection, err := n.network.GetConnection(to.Node) if err != nil { + atomic.AddUint64(&n.sendErrorsRemote, 1) return err } - return connection.SendProcessID(from, to, options, message) + if tracingActive { + options.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } + if err := connection.SendProcessID(from, to, options, message); err != nil { + atomic.AddUint64(&n.sendErrorsRemote, 1) + return err + } + if tracingActive { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: options.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindSend, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + return nil } value, found := n.names.Load(to.Name) if found == false { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessUnknown } p := value.(*process) if alive := p.isAlive(); alive == false { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessTerminated } @@ -126,13 +222,19 @@ func (n *node) RouteSendProcessID(from gen.PID, to gen.ProcessID, options gen.Me qm.Type = gen.MailboxMessageTypeRegular qm.Target = to.Name qm.Message = message + qm.Tracing = options.Tracing + if tracingActive && from.Node == n.name { + qm.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } if ok := queue.Push(qm); ok == false { if p.fallback.Enable == false { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessMailboxFull } if p.fallback.Name == p.name { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessMailboxFull } @@ -146,6 +248,29 @@ func (n *node) RouteSendProcessID(from gen.PID, to gen.ProcessID, options gen.Me } atomic.AddUint64(&p.messagesIn, 1) + + if tracingActive { + nanos := time.Now().UnixNano() + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindSend, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindSend, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: p.sbehavior, Message: msgType, + Attributes: p.tracingAttrs, + }) + } + p.run() return nil } @@ -157,26 +282,59 @@ func (n *node) RouteSendAlias(from gen.PID, to gen.Alias, options gen.MessageOpt return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteSendAlias from %s to %s", from, to) } + tracingActive := options.Tracing.ID != [2]uint64{} + var msgType string + var parentSpanID uint64 + var fromBehavior string + if tracingActive { + parentSpanID = options.Tracing.SpanID + fromBehavior = options.Tracing.Behavior + if message != nil { + msgType = reflect.TypeOf(message).String() + } + } + if to.Node != n.name { // remote connection, err := n.network.GetConnection(to.Node) if err != nil { + atomic.AddUint64(&n.sendErrorsRemote, 1) + return err + } + if tracingActive { + options.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } + if err := connection.SendAlias(from, to, options, message); err != nil { + atomic.AddUint64(&n.sendErrorsRemote, 1) return err } - return connection.SendAlias(from, to, options, message) + if tracingActive { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: options.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindSend, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + return nil } value, found := n.aliases.Load(to) if found == false { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessUnknown } p := value.(*process) if alive := p.isAlive(); alive == false { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessTerminated } @@ -185,14 +343,41 @@ func (n *node) RouteSendAlias(from gen.PID, to gen.Alias, options gen.MessageOpt qm.Type = gen.MailboxMessageTypeRegular qm.Target = to qm.Message = message + qm.Tracing = options.Tracing + if tracingActive && from.Node == n.name { + qm.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } // check if this message should be delivered to the meta process if value, found := p.metas.Load(to); found { m := value.(*meta) if ok := m.main.Push(qm); ok == false { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrMetaMailboxFull } atomic.AddUint64(&m.messagesIn, 1) + atomic.AddUint64(&p.messagesIn, 1) + if tracingActive { + nanos := time.Now().UnixNano() + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindSend, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindSend, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: p.sbehavior, Message: msgType, + Attributes: p.tracingAttrs, + }) + } m.handle() return nil } @@ -208,10 +393,12 @@ func (n *node) RouteSendAlias(from gen.PID, to gen.Alias, options gen.MessageOpt if ok := queue.Push(qm); ok == false { if p.fallback.Enable == false { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessMailboxFull } if p.fallback.Name == p.name { + atomic.AddUint64(&n.sendErrorsLocal, 1) return gen.ErrProcessMailboxFull } @@ -225,6 +412,29 @@ func (n *node) RouteSendAlias(from gen.PID, to gen.Alias, options gen.MessageOpt } atomic.AddUint64(&p.messagesIn, 1) + + if tracingActive { + nanos := time.Now().UnixNano() + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindSend, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindSend, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: p.sbehavior, Message: msgType, + Attributes: p.tracingAttrs, + }) + } + p.run() return nil } @@ -234,7 +444,7 @@ func (n *node) RouteSendEvent(from gen.PID, token gen.Ref, options gen.MessageOp return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteSendEvent from %s with token %s", from, token) } @@ -249,7 +459,7 @@ func (n *node) RouteSendExit(from gen.PID, to gen.PID, reason error) error { return gen.ErrIncorrect } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteSendExit from %s to %s with reason %q", from, to, reason) } @@ -275,17 +485,46 @@ func (n *node) RouteSendResponse(from gen.PID, to gen.PID, options gen.MessageOp return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteSendResponse from %s to %s with ref %q", from, to, options.Ref) } + tracingActive := options.Tracing.ID != [2]uint64{} + var msgType string + var parentSpanID uint64 + var fromBehavior string + if tracingActive { + parentSpanID = options.Tracing.SpanID + fromBehavior = options.Tracing.Behavior + if message != nil { + msgType = reflect.TypeOf(message).String() + } + } + if to.Node != n.name { // remote connection, err := n.network.GetConnection(to.Node) if err != nil { return err } - return connection.SendResponse(from, to, options, message) + if tracingActive { + options.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } + if err := connection.SendResponse(from, to, options, message); err != nil { + return err + } + if tracingActive { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: options.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindResponse, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + return nil } // Check if this is a node-level call response @@ -298,6 +537,31 @@ func (n *node) RouteSendResponse(from gen.PID, to gen.PID, options gen.MessageOp select { case call.done <- struct{}{}: + if tracingActive { + nanos := time.Now().UnixNano() + spanID := options.Tracing.SpanID + if from.Node == n.name { + spanID = atomic.AddUint64(&n.spanID, 1) + } + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindResponse, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindResponse, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } return nil default: return gen.ErrResponseIgnored @@ -322,9 +586,33 @@ func (n *node) RouteSendResponse(from gen.PID, to gen.PID, options gen.MessageOp select { case p.response <- resp: atomic.AddUint64(&p.messagesIn, 1) + if tracingActive { + nanos := time.Now().UnixNano() + spanID := options.Tracing.SpanID + if from.Node == n.name { + spanID = atomic.AddUint64(&n.spanID, 1) + } + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindResponse, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindResponse, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: p.sbehavior, Message: msgType, + Attributes: p.tracingAttrs, + }) + } return nil default: - // process doesn't wait for a response anymore return gen.ErrResponseIgnored } } @@ -334,17 +622,48 @@ func (n *node) RouteSendResponseError(from gen.PID, to gen.PID, options gen.Mess return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteSendResponseError from %s to %s with ref %q", from, to, options.Ref) } + tracingActive := options.Tracing.ID != [2]uint64{} + var errString string + var msgType string + var parentSpanID uint64 + var fromBehavior string + if tracingActive { + parentSpanID = options.Tracing.SpanID + fromBehavior = options.Tracing.Behavior + if err != nil { + errString = err.Error() + msgType = reflect.TypeOf(err).String() + } + } + if to.Node != n.name { // remote connection, e := n.network.GetConnection(to.Node) if e != nil { return e } - return connection.SendResponseError(from, to, options, err) + if tracingActive { + options.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } + if e := connection.SendResponseError(from, to, options, err); e != nil { + return e + } + if tracingActive { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: options.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindResponse, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, Error: errString, + Attributes: options.TracingAttributes, + }) + } + return nil } // Check if this is a node-level call response error @@ -357,6 +676,31 @@ func (n *node) RouteSendResponseError(from gen.PID, to gen.PID, options gen.Mess select { case call.done <- struct{}{}: + if tracingActive { + nanos := time.Now().UnixNano() + spanID := options.Tracing.SpanID + if from.Node == n.name { + spanID = atomic.AddUint64(&n.spanID, 1) + } + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindResponse, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, Error: errString, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindResponse, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, Error: errString, + Attributes: options.TracingAttributes, + }) + } return nil default: return gen.ErrResponseIgnored @@ -380,9 +724,33 @@ func (n *node) RouteSendResponseError(from gen.PID, to gen.PID, options gen.Mess select { case p.response <- resp: atomic.AddUint64(&p.messagesIn, 1) + if tracingActive { + nanos := time.Now().UnixNano() + spanID := options.Tracing.SpanID + if from.Node == n.name { + spanID = atomic.AddUint64(&n.spanID, 1) + } + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindResponse, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, Error: errString, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindResponse, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: p.sbehavior, Message: msgType, Error: errString, + Attributes: p.tracingAttrs, + }) + } return nil default: - // process doesn't wait for a response anymore return gen.ErrResponseIgnored } } @@ -398,27 +766,60 @@ func (n *node) RouteCallPID(from gen.PID, to gen.PID, options gen.MessageOptions return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteCallPID from %s to %s with ref %q", from, to, options.Ref) } + tracingActive := options.Tracing.ID != [2]uint64{} + var msgType string + var parentSpanID uint64 + var fromBehavior string + if tracingActive { + parentSpanID = options.Tracing.SpanID + fromBehavior = options.Tracing.Behavior + if message != nil { + msgType = reflect.TypeOf(message).String() + } + } + if to.Node != n.name { // remote connection, err := n.network.GetConnection(to.Node) if err != nil { + atomic.AddUint64(&n.callErrorsRemote, 1) return err } - return connection.CallPID(from, to, options, message) + if tracingActive { + options.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } + if err := connection.CallPID(from, to, options, message); err != nil { + atomic.AddUint64(&n.callErrorsRemote, 1) + return err + } + if tracingActive { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: options.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindRequest, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + return nil } // local value, found := n.processes.Load(to) if found == false { + atomic.AddUint64(&n.callErrorsLocal, 1) return gen.ErrProcessUnknown } p := value.(*process) if alive := p.isAlive(); alive == false { + atomic.AddUint64(&n.callErrorsLocal, 1) return gen.ErrProcessTerminated } @@ -436,11 +837,39 @@ func (n *node) RouteCallPID(from gen.PID, to gen.PID, options gen.MessageOptions qm.From = from qm.Type = gen.MailboxMessageTypeRequest qm.Message = message + qm.Tracing = options.Tracing + if tracingActive && from.Node == n.name { + qm.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } if ok := queue.Push(qm); ok == false { + atomic.AddUint64(&n.callErrorsLocal, 1) return gen.ErrProcessMailboxFull } atomic.AddUint64(&p.messagesIn, 1) + + if tracingActive { + nanos := time.Now().UnixNano() + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindRequest, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindRequest, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: p.sbehavior, Message: msgType, + Attributes: p.tracingAttrs, + }) + } + p.run() return nil } @@ -451,25 +880,58 @@ func (n *node) RouteCallProcessID(from gen.PID, to gen.ProcessID, options gen.Me if n.isRunning() == false { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteCallProcessID from %s to %s with ref %q", from, to, options.Ref) } + tracingActive := options.Tracing.ID != [2]uint64{} + var msgType string + var parentSpanID uint64 + var fromBehavior string + if tracingActive { + parentSpanID = options.Tracing.SpanID + fromBehavior = options.Tracing.Behavior + if message != nil { + msgType = reflect.TypeOf(message).String() + } + } + if to.Node != n.name { // remote connection, err := n.network.GetConnection(to.Node) if err != nil { + atomic.AddUint64(&n.callErrorsRemote, 1) + return err + } + if tracingActive { + options.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } + if err := connection.CallProcessID(from, to, options, message); err != nil { + atomic.AddUint64(&n.callErrorsRemote, 1) return err } - return connection.CallProcessID(from, to, options, message) + if tracingActive { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: options.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindRequest, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + return nil } value, found := n.names.Load(to.Name) if found == false { + atomic.AddUint64(&n.callErrorsLocal, 1) return gen.ErrProcessUnknown } p := value.(*process) if alive := p.isAlive(); alive == false { + atomic.AddUint64(&n.callErrorsLocal, 1) return gen.ErrProcessTerminated } @@ -488,11 +950,39 @@ func (n *node) RouteCallProcessID(from gen.PID, to gen.ProcessID, options gen.Me qm.Type = gen.MailboxMessageTypeRequest qm.Target = to.Name qm.Message = message + qm.Tracing = options.Tracing + if tracingActive && from.Node == n.name { + qm.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } if ok := queue.Push(qm); ok == false { + atomic.AddUint64(&n.callErrorsLocal, 1) return gen.ErrProcessMailboxFull } atomic.AddUint64(&p.messagesIn, 1) + + if tracingActive { + nanos := time.Now().UnixNano() + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindRequest, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindRequest, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: p.sbehavior, Message: msgType, + Attributes: p.tracingAttrs, + }) + } + p.run() return nil } @@ -504,25 +994,58 @@ func (n *node) RouteCallAlias(from gen.PID, to gen.Alias, options gen.MessageOpt return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteCallAlias from %s to %s with ref %q", from, to, options.Ref) } + tracingActive := options.Tracing.ID != [2]uint64{} + var msgType string + var parentSpanID uint64 + var fromBehavior string + if tracingActive { + parentSpanID = options.Tracing.SpanID + fromBehavior = options.Tracing.Behavior + if message != nil { + msgType = reflect.TypeOf(message).String() + } + } + if to.Node != n.name { // remote connection, err := n.network.GetConnection(to.Node) if err != nil { + atomic.AddUint64(&n.callErrorsRemote, 1) return err } - return connection.CallAlias(from, to, options, message) + if tracingActive { + options.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } + if err := connection.CallAlias(from, to, options, message); err != nil { + atomic.AddUint64(&n.callErrorsRemote, 1) + return err + } + if tracingActive { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: options.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindRequest, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + return nil } value, found := n.aliases.Load(to) if found == false { + atomic.AddUint64(&n.callErrorsLocal, 1) return gen.ErrProcessUnknown } p := value.(*process) if alive := p.isAlive(); alive == false { + atomic.AddUint64(&n.callErrorsLocal, 1) return gen.ErrProcessTerminated } @@ -532,14 +1055,41 @@ func (n *node) RouteCallAlias(from gen.PID, to gen.Alias, options gen.MessageOpt qm.Type = gen.MailboxMessageTypeRequest qm.Target = to qm.Message = message + qm.Tracing = options.Tracing + if tracingActive && from.Node == n.name { + qm.Tracing.SpanID = atomic.AddUint64(&n.spanID, 1) + } // check if this request should be delivered to the meta process if value, found := p.metas.Load(to); found { m := value.(*meta) if ok := m.main.Push(qm); ok == false { + atomic.AddUint64(&n.callErrorsLocal, 1) return gen.ErrMetaMailboxFull } atomic.AddUint64(&m.messagesIn, 1) + atomic.AddUint64(&p.messagesIn, 1) + if tracingActive { + nanos := time.Now().UnixNano() + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindRequest, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindRequest, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: p.sbehavior, Message: msgType, + Attributes: p.tracingAttrs, + }) + } m.handle() return nil } @@ -553,9 +1103,33 @@ func (n *node) RouteCallAlias(from gen.PID, to gen.Alias, options gen.MessageOpt queue = p.mailbox.Main } if ok := queue.Push(qm); ok == false { + atomic.AddUint64(&n.callErrorsLocal, 1) return gen.ErrProcessMailboxFull } atomic.AddUint64(&p.messagesIn, 1) + + if tracingActive { + nanos := time.Now().UnixNano() + if from.Node == n.name { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindRequest, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: fromBehavior, Message: msgType, + Attributes: options.TracingAttributes, + }) + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: qm.Tracing.SpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointDelivered, Kind: gen.TracingKindRequest, + Timestamp: nanos, Node: n.name, From: from, To: to, + Ref: options.Ref, Behavior: p.sbehavior, Message: msgType, + Attributes: p.tracingAttrs, + }) + } + p.run() return nil } @@ -565,7 +1139,7 @@ func (n *node) RouteLinkPID(pid gen.PID, target gen.PID) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteLinkPID %s with %s", pid, target) } @@ -589,7 +1163,7 @@ func (n *node) RouteUnlinkPID(pid gen.PID, target gen.PID) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteUnlinkPID %s with %s ", pid, target) } @@ -601,7 +1175,7 @@ func (n *node) RouteLinkProcessID(pid gen.PID, target gen.ProcessID) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteLinkProcessID %s with %s", pid, target) } @@ -624,7 +1198,7 @@ func (n *node) RouteUnlinkProcessID(pid gen.PID, target gen.ProcessID) error { if n.isRunning() == false { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteUnlinkProcessID %s with %s", pid, target) } @@ -636,7 +1210,7 @@ func (n *node) RouteLinkAlias(pid gen.PID, target gen.Alias) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteLinkAlias %s with %s using %s", pid, target) } @@ -654,7 +1228,7 @@ func (n *node) RouteUnlinkAlias(pid gen.PID, target gen.Alias) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteUnlinkAlias %s with %s", pid, target) } @@ -667,7 +1241,7 @@ func (n *node) RouteLinkEvent(pid gen.PID, target gen.Event) ([]gen.MessageEvent return nil, gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteLinkEvent %s with %s", pid, target) } @@ -679,7 +1253,7 @@ func (n *node) RouteUnlinkEvent(pid gen.PID, target gen.Event) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteUnlinkEvent %s with %s", pid, target) } @@ -691,7 +1265,7 @@ func (n *node) RouteMonitorPID(pid gen.PID, target gen.PID) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteMonitorPID %s with %s", pid, target) } @@ -715,7 +1289,7 @@ func (n *node) RouteDemonitorPID(pid gen.PID, target gen.PID) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteDemonitorPID %s with %s", pid, target) } @@ -727,7 +1301,7 @@ func (n *node) RouteMonitorProcessID(pid gen.PID, target gen.ProcessID) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteMonitorProcessID %s to %s", pid, target) } @@ -751,7 +1325,7 @@ func (n *node) RouteDemonitorProcessID(pid gen.PID, target gen.ProcessID) error return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteDemonitorProcessID %s to %s", pid, target) } @@ -763,7 +1337,7 @@ func (n *node) RouteMonitorAlias(pid gen.PID, target gen.Alias) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteMonitorAlias %s to %s", pid, target) } @@ -781,7 +1355,7 @@ func (n *node) RouteDemonitorAlias(pid gen.PID, target gen.Alias) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteDemonitorAlias %s to %s", pid, target) } @@ -794,7 +1368,7 @@ func (n *node) RouteMonitorEvent(pid gen.PID, target gen.Event) ([]gen.MessageEv return nil, gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteMonitorEvent %s to %s", pid, target) } @@ -807,7 +1381,7 @@ func (n *node) RouteDemonitorEvent(pid gen.PID, target gen.Event) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteDemonitorEvent %s to %s", pid, target) } @@ -819,7 +1393,7 @@ func (n *node) RouteTerminatePID(target gen.PID, reason error) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteTerminatePID %s with reason %q", target, reason) } @@ -833,7 +1407,7 @@ func (n *node) RouteTerminateProcessID(target gen.ProcessID, reason error) error return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteTerminateProcessID %s with reason %q", target, reason) } @@ -847,7 +1421,7 @@ func (n *node) RouteTerminateEvent(target gen.Event, reason error) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteTerminateEvent %s with reason %q", target, reason) } @@ -861,7 +1435,7 @@ func (n *node) RouteTerminateAlias(target gen.Alias, reason error) error { return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteTerminateAlias %s with reason %q", target, reason) } @@ -882,7 +1456,7 @@ func (n *node) RouteSpawn( return empty, gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteSpawn %s from %s to %s", name, options.ParentPID, node) } @@ -927,7 +1501,7 @@ func (n *node) RouteApplicationStart( return gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteApplicationStart %s with mode %s requested by %s", name, mode, source) } @@ -948,7 +1522,7 @@ func (n *node) RouteApplicationInfo(name gen.Atom) (gen.ApplicationInfo, error) return gen.ApplicationInfo{}, gen.ErrNodeTerminated } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteApplicationInfo %s", name) } @@ -956,12 +1530,20 @@ func (n *node) RouteApplicationInfo(name gen.Atom) (gen.ApplicationInfo, error) } func (n *node) RouteNodeDown(name gen.Atom, reason error) { - if lib.Trace() { + if lib.Verbose() { n.log.Trace("RouteNodeDown for %s ", name) } n.targets.TerminatedTargetNode(name, reason) } +func (n *node) MakeTraceID() gen.Tracing { + var t gen.Tracing + t.ID[0] = uint64(time.Now().UnixNano()) + counter := atomic.AddUint64(&n.traceID, 1) + t.ID[1] = n.nameCRC32<<32 | counter&0xFFFFFFFF + return t +} + func (n *node) MakeRef() gen.Ref { var ref gen.Ref ref.Node = n.name @@ -1006,7 +1588,7 @@ func (n *node) sendExitMessage(from gen.PID, to gen.PID, message any) error { } p := value.(*process) - if lib.Trace() { + if lib.Verbose() { n.log.Trace("...sendExitMessage from %s to %s ", from, to) } @@ -1048,7 +1630,7 @@ func (n *node) sendEventMessage( queue = p.mailbox.Main } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("...sendEventMessage from %s to %s ", from, to) } diff --git a/node/meta.go b/node/meta.go index 3f082b1ad..0285a0a0a 100644 --- a/node/meta.go +++ b/node/meta.go @@ -81,6 +81,7 @@ func (m *meta) SendResponse(to gen.PID, ref gen.Ref, message any) error { return err } atomic.AddUint64(&m.messagesOut, 1) + atomic.AddUint64(&m.p.messagesOut, 1) return nil } @@ -103,6 +104,7 @@ func (m *meta) SendResponseError(to gen.PID, ref gen.Ref, err error) error { return rerr } atomic.AddUint64(&m.messagesOut, 1) + atomic.AddUint64(&m.p.messagesOut, 1) return nil } @@ -187,6 +189,7 @@ func (m *meta) send(to any, message any) error { // so we need to increase messagesIn counter there // and run the process atomic.AddUint64(&m.p.messagesIn, 1) + atomic.AddUint64(&m.p.messagesOut, 1) m.p.run() atomic.AddUint64(&m.messagesOut, 1) @@ -213,6 +216,7 @@ func (m *meta) send(to any, message any) error { } atomic.AddUint64(&m.messagesOut, 1) + atomic.AddUint64(&m.p.messagesOut, 1) return nil } diff --git a/node/meta_start.go b/node/meta_start.go index b21521135..873d924e0 100644 --- a/node/meta_start.go +++ b/node/meta_start.go @@ -146,6 +146,7 @@ func (m *meta) handle() { } m.p.node.RouteSendResponse(m.p.pid, message.From, options, result) atomic.AddUint64(&m.messagesOut, 1) + atomic.AddUint64(&m.p.messagesOut, 1) continue case gen.MailboxMessageTypeExit: diff --git a/node/meta_start_pprof.go b/node/meta_start_pprof.go index 88f92dc5a..27d9fd679 100644 --- a/node/meta_start_pprof.go +++ b/node/meta_start_pprof.go @@ -153,6 +153,7 @@ func (m *meta) handle() { } m.p.node.RouteSendResponse(m.p.pid, message.From, options, result) atomic.AddUint64(&m.messagesOut, 1) + atomic.AddUint64(&m.p.messagesOut, 1) continue case gen.MailboxMessageTypeExit: diff --git a/node/network.go b/node/network.go index cf5bd486c..d9bb12f71 100644 --- a/node/network.go +++ b/node/network.go @@ -52,8 +52,12 @@ type network struct { handshakes sync.Map // .Version().String() -> handshake protos sync.Map // .Version().String() -> proto - cookie string - maxmessagesize int + cookie string + maxmessagesize int + softwareKeepAliveMisses int + fragmentSize int + fragmentTimeout int + maxFragmentAssemblies int staticRoutes *staticRoutes staticProxies *staticProxies @@ -62,6 +66,14 @@ type network struct { enableAppStart sync.Map connections sync.Map // gen.Atom (peer name) => gen.Connection + pending sync.Map // gen.Atom (peer name) => *pendingEntry + + connectionsEstablished atomic.Uint64 + connectionsLost atomic.Uint64 +} + +type pendingEntry struct { + ready chan struct{} // closed when connect finishes (success or failure) } func (n *network) Registrar() (gen.Registrar, error) { @@ -76,7 +88,7 @@ func (n *network) Cookie() string { } func (n *network) SetCookie(cookie string) error { n.cookie = cookie - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("updated cookie") } return nil @@ -184,7 +196,7 @@ func (n *network) AddRoute(match string, route gen.NetworkRoute, weight int) err if err := n.staticRoutes.add(match, route, weight); err != nil { return err } - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("added static route %s with weight %d", match, weight) } return nil @@ -194,7 +206,7 @@ func (n *network) RemoveRoute(match string) error { if err := n.staticRoutes.remove(match); err != nil { return err } - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("removed static route %s", match) } return nil @@ -212,7 +224,7 @@ func (n *network) AddProxyRoute(match string, route gen.NetworkProxyRoute, weigh return err } - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("added static proxy route %s with weight %d", match, weight) } return nil @@ -222,7 +234,7 @@ func (n *network) RemoveProxyRoute(match string) error { if err := n.staticProxies.remove(match); err != nil { return err } - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("removed static proxy route %s", match) } return nil @@ -432,7 +444,7 @@ func (n *network) RegisterHandshake(handshake gen.NetworkHandshake) { } _, exist := n.handshakes.LoadOrStore(handshake.Version().Str(), handshake) if exist == false { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("registered handshake %s", handshake.Version()) } } @@ -445,7 +457,7 @@ func (n *network) RegisterProto(proto gen.NetworkProto) { } _, exist := n.protos.LoadOrStore(proto.Version().Str(), proto) if exist == false { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("registered proto %s", proto.Version()) } } @@ -491,6 +503,9 @@ func (n *network) Info() (gen.NetworkInfo, error) { info.Flags = n.flags + info.ConnectionsEstablished = n.connectionsEstablished.Load() + info.ConnectionsLost = n.connectionsLost.Load() + info.EnabledSpawn = n.listEnabledSpawn() info.EnabledApplicationStart = n.listEnabledApplicationStart() @@ -501,6 +516,126 @@ func (n *network) Mode() gen.NetworkMode { return n.mode } +func (n *network) Protos() []gen.NetworkProto { + var list []gen.NetworkProto + n.protos.Range(func(_, v any) bool { + list = append(list, v.(gen.NetworkProto)) + return true + }) + return list +} + +// typeRegistryEntry pairs a proto with its TypeRegistry capability. +type typeRegistryEntry struct { + proto gen.NetworkProto + registry gen.TypeRegistry +} + +func (n *network) typeRegistries() []typeRegistryEntry { + var list []typeRegistryEntry + n.protos.Range(func(_, v any) bool { + p := v.(gen.NetworkProto) + if r, ok := p.(gen.TypeRegistry); ok { + list = append(list, typeRegistryEntry{p, r}) + } + return true + }) + return list +} + +func (n *network) RegisterType(v any) error { + regs := n.typeRegistries() + if len(regs) == 0 { + return gen.ErrUnsupported + } + var errs []string + for _, r := range regs { + err := r.registry.RegisterType(v) + if err == nil || err == gen.ErrTaken { + continue + } + errs = append(errs, fmt.Sprintf("%s: %s", r.proto.Version(), err)) + } + if len(errs) > 0 { + return fmt.Errorf("RegisterType: %s", strings.Join(errs, "; ")) + } + return nil +} + +func (n *network) RegisterTypes(types []any) error { + regs := n.typeRegistries() + if len(regs) == 0 { + return gen.ErrUnsupported + } + var errs []string + for _, r := range regs { + err := r.registry.RegisterTypes(types) + if err == nil || err == gen.ErrTaken { + continue + } + errs = append(errs, fmt.Sprintf("%s: %s", r.proto.Version(), err)) + } + if len(errs) > 0 { + return fmt.Errorf("RegisterTypes: %s", strings.Join(errs, "; ")) + } + return nil +} + +func (n *network) RegisterError(e error) error { + regs := n.typeRegistries() + if len(regs) == 0 { + return gen.ErrUnsupported + } + var errs []string + for _, r := range regs { + err := r.registry.RegisterError(e) + if err == nil || err == gen.ErrTaken { + continue + } + errs = append(errs, fmt.Sprintf("%s: %s", r.proto.Version(), err)) + } + if len(errs) > 0 { + return fmt.Errorf("RegisterError: %s", strings.Join(errs, "; ")) + } + return nil +} + +func (n *network) RegisterAtom(a gen.Atom) error { + regs := n.typeRegistries() + if len(regs) == 0 { + return gen.ErrUnsupported + } + var errs []string + for _, r := range regs { + err := r.registry.RegisterAtom(a) + if err == nil || err == gen.ErrTaken { + continue + } + errs = append(errs, fmt.Sprintf("%s: %s", r.proto.Version(), err)) + } + if len(errs) > 0 { + return fmt.Errorf("RegisterAtom: %s", strings.Join(errs, "; ")) + } + return nil +} + +func (n *network) RegisteredTypes() []gen.RegisteredTypeInfo { + var all []gen.RegisteredTypeInfo + for _, r := range n.typeRegistries() { + all = append(all, r.registry.RegisteredTypes()...) + } + return all +} + +func (n *network) LookupType(name string) (reflect.Type, bool) { + for _, r := range n.typeRegistries() { + if t, ok := r.registry.LookupType(name); ok { + return t, true + } + } + return nil, false +} + // // internals // @@ -525,36 +660,36 @@ func (n *network) GetConnection(name gen.Atom) (gen.Connection, error) { return nil, gen.ErrNoRoute } - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("trying to make connection with %s", name) } // check the static routes if sroutes, found := n.staticRoutes.lookup(string(name)); found { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("found %s static route[s] for %s", len(sroutes), name) } for i, sroute := range sroutes { sroute.InsecureSkipVerify = n.skipverify if sroute.Resolver == nil { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("use static route to %s (%d)", name, i+1) } if c, err := n.connect(name, sroute); err == nil { return c, nil } else { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("unable to connect to %s using static route: %s", name, err) } } continue } - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("use static route to %s with resolver (%d)", name, i+1) } nr, err := sroute.Resolver.Resolve(name) if err != nil { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("failed to resolve %s: %s", name, err) } continue @@ -574,7 +709,7 @@ func (n *network) GetConnection(name gen.Atom) (gen.Connection, error) { if c, err := n.connect(name, nroute); err == nil { return c, nil } else { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("unable to connect to %s using static route (with resolver): %s", name, err) } } @@ -585,12 +720,12 @@ func (n *network) GetConnection(name gen.Atom) (gen.Connection, error) { // check the static proxy routes if proutes, found := n.staticProxies.lookup(string(name)); found { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("found %d static proxy route[s] for %s", len(proutes), name) } for i, proute := range proutes { if proute.Resolver == nil { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("use static proxy route to %s (%d)", name, i+1) } if c, err := n.connectProxy(name, proute); err == nil { @@ -599,12 +734,12 @@ func (n *network) GetConnection(name gen.Atom) (gen.Connection, error) { continue } - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("use static proxy route to %s with resolver (%d)", name, i+1) } pr, err := proute.Resolver.ResolveProxy(name) if err != nil { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("failed to resolve proxy for %s: %s", name, err) } continue @@ -617,7 +752,7 @@ func (n *network) GetConnection(name gen.Atom) (gen.Connection, error) { if c, err := n.connectProxy(name, nproute); err == nil { return c, nil } else { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("unable to connect to %s using proxy route: %s", name, err) } } @@ -628,7 +763,7 @@ func (n *network) GetConnection(name gen.Atom) (gen.Connection, error) { // resolve it if nr, err := registrar.Resolver().Resolve(name); err == nil { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("resolved %d route[s] for %s", len(nr), name) } @@ -646,23 +781,23 @@ func (n *network) GetConnection(name gen.Atom) (gen.Connection, error) { if c, err := n.connect(name, nroute); err == nil { return c, nil } else { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("unable to connect to %s: %s", name, err) } } } - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("unable to connect to %s directly, looking up proxies...", name) } } else { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("attempt to resolve %s failed: %s", name, err) } } // resolve proxy if pr, err := registrar.Resolver().ResolveProxy(name); err == nil { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("resolved %d proxy routes for %s", len(pr), name) } @@ -680,7 +815,7 @@ func (n *network) GetConnection(name gen.Atom) (gen.Connection, error) { if c, err := n.connectProxy(name, nproute); err == nil { return c, nil } else { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("unable to connect to %s using resolve proxy: %s", name, err) } } @@ -690,6 +825,37 @@ func (n *network) GetConnection(name gen.Atom) (gen.Connection, error) { return nil, gen.ErrNoRoute } +// acquirePending tries to become the goroutine that connects to `name`. +// If another goroutine is already connecting, waits for it to finish and +// checks the result. Retries up to 3 times if the other goroutine fails. +// Returns: (entry, nil) if acquired; (nil, nil) if connection appeared; (nil, err) on failure. +func (n *network) acquirePending(name gen.Atom) (*pendingEntry, error) { + for attempt := 0; attempt < 3; attempt++ { + entry := &pendingEntry{ready: make(chan struct{})} + actual, loaded := n.pending.LoadOrStore(name, entry) + if loaded == false { + return entry, nil // acquired the slot + } + + // another connect in progress, wait for it + pe := actual.(*pendingEntry) + select { + case <-pe.ready: + // connect finished (success or failure) + case <-time.After(5 * time.Second): + return nil, fmt.Errorf("connection to %s: pending timeout", name) + } + + // check if connection appeared + if _, ok := n.connections.Load(name); ok { + return nil, nil // connection exists + } + + // connect failed and pending was cleared, retry LoadOrStore + } + return nil, fmt.Errorf("connection to %s: 3 attempts exhausted", name) +} + func (n *network) connect(name gen.Atom, route gen.NetworkRoute) (gen.Connection, error) { var dial func(network, addr string) (net.Conn, error) @@ -706,14 +872,32 @@ func (n *network) connect(name gen.Atom, route gen.NetworkRoute) (gen.Connection return nil, fmt.Errorf("no proto handler for %s", route.Route.ProtoVersion) } - handshake := vhandshake.(gen.NetworkHandshake) + hs := vhandshake.(gen.NetworkHandshake) proto := vproto.(gen.NetworkProto) if route.Route.Host == "" { route.Route.Host = name.Host() } - if lib.Trace() { + // acquire pending slot (waits for ongoing connect, retries on failure) + entry, err := n.acquirePending(name) + if err != nil { + return nil, err + } + if entry == nil { + // connection appeared while waiting + v, ok := n.connections.Load(name) + if ok == false { + return nil, gen.ErrNoRoute + } + return v.(gen.Connection), nil + } + defer func() { + n.pending.Delete(name) + close(entry.ready) // wake ALL waiting goroutines + }() + + if lib.Verbose() { n.node.Log().Trace("trying to connect to %s (%s:%d, tls:%v)", name, route.Route.Host, route.Route.Port, route.Route.TLS) } @@ -755,11 +939,16 @@ func (n *network) connect(name gen.Atom, route gen.NetworkRoute) (gen.Connection if err != nil { return nil, err } + conn.SetDeadline(time.Now().Add(gen.DefaultHandshakeTimeout)) hopts := gen.HandshakeOptions{ Cookie: route.Cookie, Flags: route.Flags, MaxMessageSize: n.maxmessagesize, + CheckPending: func(peer gen.Atom) bool { + _, exists := n.pending.Load(peer) + return exists + }, } if hopts.Cookie == "" { @@ -768,10 +957,15 @@ func (n *network) connect(name gen.Atom, route gen.NetworkRoute) (gen.Connection if hopts.Flags.Enable == false { hopts.Flags = n.flags } + // period for keepalive is already in hopts.Flags (from route.Flags or n.flags) - result, err := handshake.Start(n.node, conn, hopts) + result, err := hs.Start(n.node, conn, hopts) if err != nil { conn.Close() + // on simultaneous connect rejection, check if accept path established connection + if v, ok := n.connections.Load(name); ok { + return v.(gen.Connection), nil + } return nil, err } @@ -800,6 +994,15 @@ func (n *network) connect(name gen.Atom, route gen.NetworkRoute) (gen.Connection } log.setSource(logSource) + // inject options into ConnectionOptions + if opts, ok := result.Custom.(handshake.ConnectionOptions); ok { + opts.SoftwareKeepAliveMisses = n.keepAliveMisses(route.SoftwareKeepAliveMisses) + opts.FragmentSize = n.fragmentSize + opts.FragmentTimeout = n.fragmentTimeout + opts.MaxFragmentAssemblies = n.maxFragmentAssemblies + result.Custom = opts + } + pconn, err := proto.NewConnection(n.node, result, log) if err != nil { conn.Close() @@ -811,26 +1014,40 @@ func (n *network) connect(name gen.Atom, route gen.NetworkRoute) (gen.Connection if err != nil { return nil, nil, err } - tail, err := handshake.Join(n.node, c, id, hopts) + c.SetDeadline(time.Now().Add(gen.DefaultHandshakeTimeout)) + tail, err := hs.Join(n.node, c, id, hopts) if err != nil { + c.Close() return nil, nil, err } return c, tail, nil } - if c, err := n.registerConnection(result.Peer, pconn); err != nil { - if err == gen.ErrTaken { - return c, nil - } + c, err := n.registerConnection(result.Peer, pconn) + if err == nil { + pconn.Join(conn, result.ConnectionID, redial, result.Tail) + go n.serve(proto, pconn, redial) + return pconn, nil + } + + if err != gen.ErrTaken { pconn.Terminate(err) conn.Close() return nil, err } - pconn.Join(conn, result.ConnectionID, redial, result.Tail) - go n.serve(proto, pconn, redial) + // ErrTaken: another path registered first + pconn.Terminate(nil) // cleanup abandoned pconn - return pconn, nil + // with deterministic ID, join our TCP into the existing connection + if jerr := c.Join(conn, result.ConnectionID, redial, result.Tail); jerr != nil { + conn.Close() + return c, nil + } + + // expand pool (existing connection may have been registered by accept with nil redial) + go n.expandPool(c, redial, result.ConnectionID, result.PoolSize, result.PoolDSN) + return c, nil } func (n *network) serve(proto gen.NetworkProto, conn gen.Connection, redial gen.NetworkDial) { @@ -850,8 +1067,26 @@ func (n *network) serve(proto gen.NetworkProto, conn gen.Connection, redial gen. conn.Terminate(err) } +func (n *network) expandPool(c gen.Connection, redial gen.NetworkDial, + connID string, poolSize int, poolDSN []string) { + if poolSize < 2 || len(poolDSN) == 0 { + return + } + for i := 1; i < poolSize; i++ { + dsn := poolDSN[i%len(poolDSN)] + nc, tail, err := redial(dsn, connID) + if err != nil { + continue + } + if err := c.Join(nc, connID, redial, tail); err != nil { + nc.Close() + return // pool full or connection terminated + } + } +} + func (n *network) connectProxy(name gen.Atom, route gen.NetworkProxyRoute) (gen.Connection, error) { - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("trying to connect to %s (via proxy %s)", name, route.Route.Proxy) } // TODO will be implemented later @@ -860,6 +1095,16 @@ func (n *network) connectProxy(name gen.Atom, route gen.NetworkProxyRoute) (gen. return nil, gen.ErrUnsupported } +func (n *network) keepAliveMisses(v int) int { + if v > 0 { + return v + } + if n.softwareKeepAliveMisses > 0 { + return n.softwareKeepAliveMisses + } + return gen.DefaultSoftwareKeepAliveMisses +} + func (n *network) stop() error { if swapped := n.running.CompareAndSwap(true, false); swapped == false { return fmt.Errorf("network stack is already stopped") @@ -894,7 +1139,7 @@ func (n *network) start(options gen.NetworkOptions) error { return nil } - if lib.Trace() { + if lib.Verbose() { n.node.log.Trace("starting network...") } @@ -904,8 +1149,6 @@ func (n *network) start(options gen.NetworkOptions) error { n.registrar = registrar.Create(registrar.Options{}) } - n.node.validateLicenses(n.registrar.Version()) - if options.Cookie == "" { n.node.log.Warning("cookie is empty (gen.NetworkOptions), used randomized value") options.Cookie = lib.RandomString(16) @@ -917,6 +1160,10 @@ func (n *network) start(options gen.NetworkOptions) error { options.Flags = gen.DefaultNetworkFlags } n.flags = options.Flags + n.softwareKeepAliveMisses = options.SoftwareKeepAliveMisses + n.fragmentSize = options.FragmentSize + n.fragmentTimeout = options.FragmentTimeout + n.maxFragmentAssemblies = options.MaxFragmentAssemblies if options.Mode == gen.NetworkModeHidden { static, err := n.registrar.Register(n.node, gen.RegisterRoutes{}) @@ -937,7 +1184,7 @@ func (n *network) start(options gen.NetworkOptions) error { } } - if lib.Trace() { + if lib.Verbose() { n.node.log.Trace("network started (hidden) with registrar %s", n.registrar.Version()) } return nil @@ -1036,7 +1283,7 @@ func (n *network) start(options gen.NetworkOptions) error { HandshakeVersion: acceptor.handshake.Version(), ProtoVersion: acceptor.proto.Version(), } - n.node.validateLicenses(r.HandshakeVersion, r.ProtoVersion) + if a.Registrar == nil { acceptor.registrar_info = n.registrar.Info routes = append(routes, r) @@ -1091,7 +1338,7 @@ func (n *network) start(options gen.NetworkOptions) error { } } - if lib.Trace() { + if lib.Verbose() { n.node.log.Trace("network started with registrar %s", n.registrar.Version()) } return nil @@ -1115,12 +1362,14 @@ func (n *network) startAcceptor(a gen.AcceptorOptions) (*acceptor, error) { if pstart == 0 { pstart = gen.DefaultPort } - pend := a.PortRange - if pend == 0 { - pend = 50000 - } - if pend < pstart { - pend = pstart + pend := uint32(65535) + if a.PortRange > 1 { + p := uint32(pstart) + uint32(a.PortRange) - 1 + if p < 65535 { + pend = p + } + } else if a.PortRange == 1 { + pend = uint32(pstart) } acceptor := &acceptor{ @@ -1132,6 +1381,9 @@ func (n *network) startAcceptor(a gen.AcceptorOptions) (*acceptor, error) { atom_mapping: make(map[gen.Atom]gen.Atom), route_host: a.RouteHost, route_port: a.RoutePort, + maxHandshakes: int32(a.MaxHandshakes), + + software_keepalive_misses: n.keepAliveMisses(a.SoftwareKeepAliveMisses), } if a.Cookie == "" { acceptor.cookie = n.cookie @@ -1140,7 +1392,7 @@ func (n *network) startAcceptor(a gen.AcceptorOptions) (*acceptor, error) { acceptor.atom_mapping[k] = v } - for i := pstart; i < pend+1; i++ { + for i := uint32(pstart); i <= pend; i++ { hp := net.JoinHostPort(a.Host, strconv.Itoa(int(i))) lcl, err := lc.Listen(context.Background(), a.TCP, hp) if err != nil { @@ -1152,7 +1404,7 @@ func (n *network) startAcceptor(a gen.AcceptorOptions) (*acceptor, error) { continue } - acceptor.port = i + acceptor.port = uint16(i) acceptor.l = lcl break } @@ -1185,7 +1437,7 @@ func (n *network) startAcceptor(a gen.AcceptorOptions) (*acceptor, error) { go n.accept(acceptor) - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("started acceptor on %s with handshake %s and proto %s (TLS: %t)", acceptor.l.Addr(), acceptor.handshake.Version(), @@ -1200,12 +1452,22 @@ func (n *network) startAcceptor(a gen.AcceptorOptions) (*acceptor, error) { } func (n *network) accept(a *acceptor) { + cookie := a.cookie + if cookie == "" { + cookie = n.cookie + } + hopts := gen.HandshakeOptions{ - Cookie: a.cookie, + Cookie: cookie, Flags: a.flags, MaxMessageSize: a.max_message_size, CertManager: a.cert_manager, + CheckPending: func(peer gen.Atom) bool { + _, exists := n.pending.Load(peer) + return exists + }, } + // period for keepalive is already in hopts.Flags (from a.flags) for { c, err := a.l.Accept() if err != nil { @@ -1216,83 +1478,161 @@ func (n *network) accept(a *acceptor) { a.l.Addr(), a.handshake.Version(), a.proto.Version()) return } - if lib.Trace() { + if lib.Verbose() { n.node.Log().Trace("accepted new TCP-connection from %s", c.RemoteAddr().String()) } - if hopts.Cookie == "" { - hopts.Cookie = n.cookie + // check concurrency limit + if a.maxHandshakes > 0 && a.handshaking.Add(1) > a.maxHandshakes { + a.handshaking.Add(-1) + c.SetWriteDeadline(time.Now().Add(100 * time.Millisecond)) + a.handshake.Reject(c, "busy") + c.Close() + continue } - result, err := a.handshake.Accept(n.node, c, hopts) - if err != nil { - if err != io.EOF { - n.node.Log().Warning("unable to handshake with %s: %s", c.RemoteAddr().String(), err) + go func() { + if a.maxHandshakes > 0 { + defer a.handshaking.Add(-1) } - c.Close() - continue + n.handleAccepted(a, c, hopts) + }() + } +} + +func (n *network) handleAccepted(a *acceptor, c net.Conn, hopts gen.HandshakeOptions) { + c.SetDeadline(time.Now().Add(gen.DefaultHandshakeTimeout)) + result, err := a.handshake.Accept(n.node, c, hopts) + if err != nil { + if err != io.EOF { + n.node.Log().Warning("unable to handshake with %s: %s", c.RemoteAddr().String(), err) } + a.handshakeErrors.Add(1) + c.Close() + return + } + + if result.Peer == "" { + n.node.Log().Warning("%s is not introduced itself, close connection", c.RemoteAddr().String()) + a.handshakeErrors.Add(1) + c.Close() + return + } + + // update atom mapping: a.atom_mapping + result.AtomMapping + mapping := make(map[gen.Atom]gen.Atom) + for k, v := range a.atom_mapping { + mapping[k] = v + } + for k, v := range result.AtomMapping { + mapping[k] = v + } + result.AtomMapping = mapping - if result.Peer == "" { - n.node.Log().Warning("%s is not introduced itself, close connection", c.RemoteAddr().String()) + // check if we already have connection with this node + if v, exist := n.connections.Load(result.Peer); exist { + conn := v.(gen.Connection) + if err := conn.Join(c, result.ConnectionID, nil, result.Tail); err != nil { c.Close() - continue } + return + } - // update atom mapping: a.atom_mapping + result.AtomMapping - mapping := make(map[gen.Atom]gen.Atom) - for k, v := range a.atom_mapping { - mapping[k] = v + // Primary connection: announce pending BEFORE heavy work so pool + // expansion TCPs (PeerCreation == 0) can wait for registration. + if result.PeerCreation != 0 { + acceptPending := &pendingEntry{ready: make(chan struct{})} + if _, loaded := n.pending.LoadOrStore(result.Peer, acceptPending); loaded { + acceptPending = nil } - for k, v := range result.AtomMapping { - mapping[k] = v - } - result.AtomMapping = mapping + defer func() { + if acceptPending != nil { + n.pending.Delete(result.Peer) + close(acceptPending.ready) + } + }() + } - // check if we already have connection with this node - if v, exist := n.connections.Load(result.Peer); exist { - conn := v.(gen.Connection) - if err := conn.Join(c, result.ConnectionID, nil, result.Tail); err != nil { - if err == gen.ErrUnsupported { - n.node.Log().Warning("unable to accept connection with %s (join is not supported)", - result.Peer) - } else { - n.node.Log().Trace("unable to join %s to the existing connection with %s: %s", - c.RemoteAddr(), result.Peer, err) + // Pool expansion (Join handshake, PeerCreation == 0). + // + // FIXME: workaround for a handshake design flaw with TCP pool expansion. + // The initiator starts dialing pool expansion connections (Join handshake) + // immediately after completing the primary handshake. These arrive at the + // acceptor as separate goroutines running handleAccepted(). The Join + // handshake (2 messages) completes faster than the primary handshake + // (5 messages), so the pool expansion TCP can reach this point before the + // primary TCP goroutine has finished registering the connection in + // n.connections. Without the retry, the pool TCP is closed, causing + // broken pipe on the initiator side for any pool_item using that TCP. + // Once the handshake protocol is redesigned to properly synchronize pool + // formation, this workaround can be removed. + if result.PeerCreation == 0 { + for i := 0; i < 3; i++ { + if pe, exist := n.pending.Load(result.Peer); exist { + entry := pe.(*pendingEntry) + <-entry.ready + } + if v, exist := n.connections.Load(result.Peer); exist { + conn := v.(gen.Connection) + if err := conn.Join(c, result.ConnectionID, nil, result.Tail); err != nil { + c.Close() } - c.Close() + return } - continue + time.Sleep(300 * time.Millisecond) } + c.Close() + return + } - log := createLog(n.node.Log().Level(), n.node.dolog) - logSource := gen.MessageLogNetwork{ - Node: n.node.name, - Peer: result.Peer, - Creation: result.PeerCreation, - } - log.setSource(logSource) - conn, err := a.proto.NewConnection(n.node, result, log) - if err != nil { - n.node.Log().Warning("unable to create new connection: %s", err) + // inject options from acceptor into ConnectionOptions + if opts, ok := result.Custom.(handshake.ConnectionOptions); ok { + opts.SoftwareKeepAliveMisses = a.software_keepalive_misses + opts.FragmentSize = n.fragmentSize + opts.FragmentTimeout = n.fragmentTimeout + opts.MaxFragmentAssemblies = n.maxFragmentAssemblies + result.Custom = opts + } + + log := createLog(n.node.Log().Level(), n.node.dolog) + logSource := gen.MessageLogNetwork{ + Node: n.node.name, + Peer: result.Peer, + Creation: result.PeerCreation, + } + log.setSource(logSource) + conn, err := a.proto.NewConnection(n.node, result, log) + if err != nil { + n.node.Log().Warning("unable to create new connection: %s", err) + c.Close() + return + } + + if _, err := n.registerConnection(result.Peer, conn); err != nil { + // connect() registered between our Load and LoadOrStore + conn.Terminate(nil) + + // with deterministic ID, join TCP into the existing connection + existing, ok := n.connections.Load(result.Peer) + if ok == false { c.Close() - continue + return } - - if _, err := n.registerConnection(result.Peer, conn); err != nil { - n.node.Log().Warning("unable to register new connection with %s: %s", result.Peer, err) + ec := existing.(gen.Connection) + if jerr := ec.Join(c, result.ConnectionID, nil, result.Tail); jerr != nil { c.Close() - continue } - conn.Join(c, result.ConnectionID, nil, result.Tail) - go n.serve(a.proto, conn, nil) + return } + conn.Join(c, result.ConnectionID, nil, result.Tail) + go n.serve(a.proto, conn, nil) } func (n *network) registerConnection(name gen.Atom, conn gen.Connection) (gen.Connection, error) { if v, exist := n.connections.LoadOrStore(name, conn); exist { return v.(gen.Connection), gen.ErrTaken } + n.connectionsEstablished.Add(1) n.node.log.Info("new connection with %s (%s)", name, name.CRC32()) // TODO create event gen.MessageNetworkEvent return conn, nil @@ -1300,6 +1640,7 @@ func (n *network) registerConnection(name gen.Atom, conn gen.Connection) (gen.Co func (n *network) unregisterConnection(name gen.Atom, reason error) { n.connections.Delete(name) + n.connectionsLost.Add(1) if reason != nil { n.node.log.Info("connection with %s (%s) terminated with reason: %s", name, name.CRC32(), reason) } else { diff --git a/node/node.go b/node/node.go index 2483f7c0c..c1347d59a 100644 --- a/node/node.go +++ b/node/node.go @@ -70,9 +70,12 @@ type node struct { security gen.SecurityOptions certmanager gen.CertManager - corePID gen.PID - nextID uint64 - uniqID uint64 + corePID gen.PID + nextID uint64 + uniqID uint64 + traceID uint64 + spanID uint64 + nameCRC32 uint64 processes sync.Map // process pid gen.PID -> *process names sync.Map // process name gen.Atom -> *process @@ -91,6 +94,10 @@ type node struct { loggers map[gen.LogLevel]*sync.Map // level -> name -> gen.LoggerBehavior log *log + tracingExporters sync.Map // name -> tracingExporterEntry + tracing gen.Tracing + tracingSampler gen.TracingSampler + shutdownTimeout time.Duration waitprocesses sync.WaitGroup wait chan struct{} @@ -102,6 +109,25 @@ type node struct { enableCTRLC atomic.Bool ctrlc chan os.Signal + + processesSpawned uint64 + processesSpawnFailed uint64 + processesTerminated uint64 + + sendErrorsLocal uint64 + sendErrorsRemote uint64 + callErrorsLocal uint64 + callErrorsRemote uint64 + + logMessages [6]uint64 // atomic: 0=trace, 1=debug, 2=info, 3=warning, 4=error, 5=panic + tracingSpans [5]uint64 // atomic: 0=send, 1=request, 2=response, 3=spawn, 4=terminate + tracingAttrs []gen.TracingAttribute // node-level permanent, COW +} + +type tracingExporterEntry struct { + exporter gen.TracingBehavior + flags gen.TracingFlags + pid gen.PID // non-zero for process-based exporters } type eventOwner struct { @@ -142,9 +168,10 @@ func Start(name gen.Atom, options gen.NodeOptions, frameworkVersion gen.Version) framework: frameworkVersion, creation: creation, - corePID: gen.PID{Node: name, ID: 1, Creation: creation}, - nextID: startID, - uniqID: startUniqID, + corePID: gen.PID{Node: name, ID: 1, Creation: creation}, + nextID: startID, + uniqID: startUniqID, + nameCRC32: uint64(name.CRC32Sum()), certmanager: options.CertManager, security: options.Security, @@ -191,7 +218,17 @@ func Start(name gen.Atom, options gen.NodeOptions, frameworkVersion gen.Version) node.LoggerAdd(lo.Name, lo.Logger, lo.Filter...) } - node.validateLicenses(node.version) + for _, te := range options.Tracing.Exporters { + if len(te.Name) == 0 { + return nil, errors.New("tracing exporter name can not be empty") + } + if te.Exporter == nil { + return nil, errors.New("tracing exporter can not be nil") + } + if err := node.TracingExporterAdd(te.Name, te.Exporter, te.Flags); err != nil { + return nil, fmt.Errorf("tracing exporter %q: %w", te.Name, err) + } + } // create target manager (pub/sub subsystem) before network start // because registrar may call RegisterEvent during network initialization @@ -205,6 +242,19 @@ func Start(name gen.Atom, options gen.NodeOptions, frameworkVersion gen.Version) node.coreEventsToken, _ = node.RegisterEvent(gen.CoreEvent, gen.EventOptions{}) + // Pre-register user-declared node-level events before starting cron and + // applications so processes can subscribe from Init() without racing the + // producer registration. + for _, spec := range options.Events { + if _, err := node.RegisterEvent(spec.Name, gen.EventOptions{ + Buffer: spec.Buffer, + Open: true, + }); err != nil { + node.StopForce() + return nil, fmt.Errorf("unable to pre-register event %q: %w", spec.Name, err) + } + } + node.cron = createCron(node) for _, job := range options.Cron.Jobs { if err := node.cron.AddJob(job); err != nil { @@ -494,13 +544,20 @@ func (n *node) ProcessInfo(pid gen.PID) (gen.ProcessInfo, error) { info.MailboxQueues.Urgent = p.mailbox.Urgent.Len() info.MailboxQueues.System = p.mailbox.System.Len() info.MailboxQueues.Log = p.mailbox.Log.Len() + info.MailboxQueues.LatencyMain = p.mailbox.Main.Latency() + info.MailboxQueues.LatencySystem = p.mailbox.System.Latency() + info.MailboxQueues.LatencyUrgent = p.mailbox.Urgent.Latency() + info.MailboxQueues.LatencyLog = p.mailbox.Log.Latency() info.MessagesIn = atomic.LoadUint64(&p.messagesIn) info.MessagesOut = atomic.LoadUint64(&p.messagesOut) info.RunningTime = atomic.LoadUint64(&p.runningTime) + info.InitTime = p.initTime + info.Wakeups = p.wakeups info.Compression = p.compression info.MessagePriority = p.priority info.Uptime = p.Uptime() info.State = p.State() + info.StateTime = time.Now().UnixNano() - atomic.LoadInt64(&p.stateEntered) info.Parent = p.parent info.Leader = p.leader info.Fallback = p.fallback @@ -509,6 +566,10 @@ func (n *node) ProcessInfo(pid gen.PID) (gen.ProcessInfo, error) { info.LogLevel = p.log.Level() info.KeepNetworkOrder = p.keeporder info.ImportantDelivery = p.important + info.Tracing = gen.TracingInfo{ + Sampler: p.TracingSampler().String(), + Attributes: p.tracingAttrs, + } if n.security.ExposeEnvInfo { info.Env = p.EnvList() @@ -573,7 +634,7 @@ func (n *node) ProcessInfo(pid gen.PID) (gen.ProcessInfo, error) { return info, nil } -func (n *node) SetLogLevelProcess(pid gen.PID, level gen.LogLevel) error { +func (n *node) SetProcessLogLevel(pid gen.PID, level gen.LogLevel) error { if n.isRunning() == false { return gen.ErrNodeTerminated } @@ -586,22 +647,122 @@ func (n *node) SetLogLevelProcess(pid gen.PID, level gen.LogLevel) error { return p.log.SetLevel(level) } -func (n *node) LogLevelProcess(pid gen.PID) (gen.LogLevel, error) { - var level gen.LogLevel +func (n *node) SetProcessSendPriority(pid gen.PID, priority gen.MessagePriority) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + switch priority { + case gen.MessagePriorityNormal: + case gen.MessagePriorityHigh: + case gen.MessagePriorityMax: + default: + return gen.ErrIncorrect + } + value, loaded := n.processes.Load(pid) + if loaded == false { + return gen.ErrProcessUnknown + } + p := value.(*process) + p.priority = priority + return nil +} + +func (n *node) SetProcessCompression(pid gen.PID, enabled bool) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + value, loaded := n.processes.Load(pid) + if loaded == false { + return gen.ErrProcessUnknown + } + p := value.(*process) + p.compression.Enable = enabled + return nil +} + +func (n *node) SetProcessCompressionType(pid gen.PID, ctype gen.CompressionType) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + switch ctype { + case gen.CompressionTypeGZIP: + case gen.CompressionTypeLZW: + case gen.CompressionTypeZLIB: + default: + return gen.ErrIncorrect + } + value, loaded := n.processes.Load(pid) + if loaded == false { + return gen.ErrProcessUnknown + } + p := value.(*process) + p.compression.Type = ctype + return nil +} + +func (n *node) SetProcessCompressionLevel(pid gen.PID, level gen.CompressionLevel) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + switch level { + case gen.CompressionBestSize: + case gen.CompressionBestSpeed: + case gen.CompressionDefault: + default: + return gen.ErrIncorrect + } + value, loaded := n.processes.Load(pid) + if loaded == false { + return gen.ErrProcessUnknown + } + p := value.(*process) + p.compression.Level = level + return nil +} + +func (n *node) SetProcessCompressionThreshold(pid gen.PID, threshold int) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + if threshold < gen.DefaultCompressionThreshold { + return gen.ErrIncorrect + } + value, loaded := n.processes.Load(pid) + if loaded == false { + return gen.ErrProcessUnknown + } + p := value.(*process) + p.compression.Threshold = threshold + return nil +} + +func (n *node) SetProcessKeepNetworkOrder(pid gen.PID, order bool) error { if n.isRunning() == false { - return level, gen.ErrNodeTerminated + return gen.ErrNodeTerminated } value, loaded := n.processes.Load(pid) if loaded == false { - return level, gen.ErrProcessUnknown + return gen.ErrProcessUnknown } + p := value.(*process) + p.keeporder = order + return nil +} +func (n *node) SetProcessImportantDelivery(pid gen.PID, important bool) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + value, loaded := n.processes.Load(pid) + if loaded == false { + return gen.ErrProcessUnknown + } p := value.(*process) - level = p.log.Level() - return level, nil + p.important = important + return nil } -func (n *node) SetLogLevelMeta(m gen.Alias, level gen.LogLevel) error { +func (n *node) SetMetaLogLevel(m gen.Alias, level gen.LogLevel) error { if n.isRunning() == false { return gen.ErrNodeTerminated } @@ -621,26 +782,32 @@ func (n *node) SetLogLevelMeta(m gen.Alias, level gen.LogLevel) error { return mp.log.SetLevel(level) } -func (n *node) LogLevelMeta(m gen.Alias) (gen.LogLevel, error) { - var level gen.LogLevel +func (n *node) SetMetaSendPriority(m gen.Alias, priority gen.MessagePriority) error { if n.isRunning() == false { - return level, gen.ErrNodeTerminated + return gen.ErrNodeTerminated } value, loaded := n.aliases.Load(m) if loaded == false { - return level, gen.ErrProcessUnknown + return gen.ErrProcessUnknown } p := value.(*process) value, loaded = p.metas.Load(m) if loaded == false { - return level, gen.ErrMetaUnknown + return gen.ErrMetaUnknown } mp := value.(*meta) - level = mp.log.Level() - return level, nil + switch priority { + case gen.MessagePriorityNormal: + case gen.MessagePriorityHigh: + case gen.MessagePriorityMax: + default: + return gen.ErrIncorrect + } + mp.priority = priority + return nil } func (n *node) Info() (gen.NodeInfo, error) { @@ -678,6 +845,32 @@ func (n *node) Info() (gen.NodeInfo, error) { }) } + info.Tracing = gen.TracingInfo{ + Sampler: n.TracingSampler().String(), + Attributes: n.tracingAttrs, + } + + n.tracingExporters.Range(func(k, v any) bool { + entry := v.(tracingExporterEntry) + behavior := "" + if entry.exporter != nil { + behavior = strings.TrimPrefix(reflect.TypeOf(entry.exporter).String(), "*") + } + info.TracingExporters = append(info.TracingExporters, gen.TracingExporterInfo{ + Name: k.(string), + Behavior: behavior, + Flags: entry.flags, + }) + return true + }) + + for i := 0; i < 6; i++ { + info.LogMessages[i] = atomic.LoadUint64(&n.logMessages[i]) + } + for i := 0; i < 5; i++ { + info.TracingSpans[i] = atomic.LoadUint64(&n.tracingSpans[i]) + } + if n.security.ExposeEnvInfo { info.Env = n.EnvList() } else { @@ -691,13 +884,22 @@ func (n *node) Info() (gen.NodeInfo, error) { case gen.ProcessStateRunning: info.ProcessesRunning++ case gen.ProcessStateWaitResponse: - info.ProcessesRunning++ + info.ProcessesWaitResponse++ case gen.ProcessStateZombee: info.ProcessesZombee++ } return true }) + info.ProcessesSpawned = atomic.LoadUint64(&n.processesSpawned) + info.ProcessesSpawnFailed = atomic.LoadUint64(&n.processesSpawnFailed) + info.ProcessesTerminated = atomic.LoadUint64(&n.processesTerminated) + + info.SendErrorsLocal = atomic.LoadUint64(&n.sendErrorsLocal) + info.SendErrorsRemote = atomic.LoadUint64(&n.sendErrorsRemote) + info.CallErrorsLocal = atomic.LoadUint64(&n.callErrorsLocal) + info.CallErrorsRemote = atomic.LoadUint64(&n.callErrorsRemote) + n.names.Range(func(_, _ any) bool { info.RegisteredNames++ return true @@ -707,6 +909,13 @@ func (n *node) Info() (gen.NodeInfo, error) { return true }) + tmInfo := n.targets.Info() + info.RegisteredEvents = tmInfo.Events + info.EventsPublished = tmInfo.EventsPublished + info.EventsReceived = tmInfo.EventsReceived + info.EventsLocalSent = tmInfo.EventsLocalSent + info.EventsRemoteSent = tmInfo.EventsRemoteSent + info.ApplicationsTotal = int64(len(n.Applications())) info.ApplicationsRunning = int64(len(n.ApplicationsRunning())) @@ -719,6 +928,8 @@ func (n *node) Info() (gen.NodeInfo, error) { info.UserTime = utime info.SystemTime = stime + info.ServerTime = time.Now() + return info, nil } @@ -737,29 +948,28 @@ func (n *node) ProcessList() ([]gen.PID, error) { return pl, nil } -func (n *node) ProcessListShortInfo(start, limit int) ([]gen.ProcessShortInfo, error) { +func (n *node) ProcessListShortInfo(start, limit int, filter ...func(gen.ProcessShortInfo) bool) ([]gen.ProcessShortInfo, error) { if n.isRunning() == false { return nil, gen.ErrNodeTerminated } - if start < 1000 || limit < 0 { + if limit < 0 || (start >= 0 && start < 1000) { return nil, gen.ErrIncorrect } if limit == 0 { limit = 100 } - ustart := uint64(start) - psi := []gen.ProcessShortInfo{} - pid := n.corePID - for limit > 0 { + from, to, step := int64(start), int64(n.nextID)+1, int64(1) + if start < 0 { + from, to, step = int64(n.nextID), 999, -1 + } - if ustart > n.nextID { - break - } + psi := []gen.ProcessShortInfo{} + pid := n.corePID - pid.ID = ustart - ustart++ + for id := from; id != to && limit > 0; id += step { + pid.ID = uint64(id) v, found := n.processes.Load(pid) if found == false { continue @@ -778,13 +988,20 @@ func (n *node) ProcessListShortInfo(start, limit int) ([]gen.ProcessShortInfo, e MessagesIn: process.messagesIn, MessagesOut: process.messagesOut, MessagesMailbox: uint64(messagesMailbox), + MailboxLatency: process.mailbox.Latency(), RunningTime: process.runningTime, + InitTime: process.initTime, + Wakeups: process.wakeups, Uptime: process.Uptime(), State: process.State(), + StateTime: time.Now().UnixNano() - atomic.LoadInt64(&process.stateEntered), Parent: process.parent, Leader: process.leader, LogLevel: process.log.Level(), } + if len(filter) > 0 && filter[0](info) == false { + continue + } psi = append(psi, info) limit-- } @@ -793,6 +1010,43 @@ func (n *node) ProcessListShortInfo(start, limit int) ([]gen.ProcessShortInfo, e } +func (n *node) ProcessRangeShortInfo(fn func(gen.ProcessShortInfo) bool) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + + n.processes.Range(func(_, v any) bool { + p := v.(*process) + messagesMailbox := p.mailbox.Main.Len() + + p.mailbox.System.Len() + + p.mailbox.Urgent.Len() + + p.mailbox.Log.Len() + + info := gen.ProcessShortInfo{ + PID: p.pid, + Name: p.name, + Application: p.application, + Behavior: p.sbehavior, + MessagesIn: p.messagesIn, + MessagesOut: p.messagesOut, + MessagesMailbox: uint64(messagesMailbox), + MailboxLatency: p.mailbox.Latency(), + RunningTime: p.runningTime, + InitTime: p.initTime, + Wakeups: p.wakeups, + Uptime: p.Uptime(), + State: p.State(), + StateTime: time.Now().UnixNano() - atomic.LoadInt64(&p.stateEntered), + Parent: p.parent, + Leader: p.leader, + LogLevel: p.log.Level(), + } + return fn(info) + }) + + return nil +} + func (n *node) NetworkStart(options gen.NetworkOptions) error { if n.isRunning() == false { return gen.ErrNodeTerminated @@ -973,8 +1227,15 @@ func (n *node) SendWithPriority(to any, message any, priority gen.MessagePriorit return gen.ErrNodeTerminated } + var tracing gen.Tracing + if n.tracingSampler != nil && n.tracingSampler.Sample() { + tracing = n.MakeTraceID() + tracing.Behavior = "core" + } options := gen.MessageOptions{ - Priority: priority, + Priority: priority, + Tracing: tracing, + TracingAttributes: n.tracingAttrs, } switch t := to.(type) { @@ -1027,6 +1288,27 @@ func (n *node) UnregisterEvent(name gen.Atom) error { return n.unregisterEvent(name, n.corePID) } +func (n *node) EventInfo(event gen.Event) (gen.EventInfo, error) { + if n.isRunning() == false { + return gen.EventInfo{}, gen.ErrNodeTerminated + } + return n.targets.EventInfo(event) +} + +func (n *node) EventRangeInfo(fn func(gen.EventInfo) bool) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + return n.targets.EventRangeInfo(fn) +} + +func (n *node) EventListInfo(timestamp int64, limit int, filter ...func(gen.EventInfo) bool) ([]gen.EventInfo, error) { + if n.isRunning() == false { + return nil, gen.ErrNodeTerminated + } + return n.targets.EventListInfo(timestamp, limit, filter...) +} + func (n *node) SendExit(pid gen.PID, reason error) error { if n.isRunning() == false { return gen.ErrNodeTerminated @@ -1067,6 +1349,13 @@ func (n *node) callWithOptions(to any, request any, timeout int, options gen.Mes if n.isRunning() == false { return nil, gen.ErrNodeTerminated } + if n.tracingSampler != nil && n.tracingSampler.Sample() { + options.Tracing = n.MakeTraceID() + options.Tracing.Behavior = "core" + } + if options.Tracing.ID != [2]uint64{} && len(n.tracingAttrs) > 0 { + options.TracingAttributes = n.tracingAttrs + } switch t := to.(type) { case gen.Atom: @@ -1470,6 +1759,7 @@ func (n *node) Kill(pid gen.PID) error { case int32(gen.ProcessStateInit), int32(gen.ProcessStateWaitResponse), int32(gen.ProcessStateRunning): + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) // do not unregister process until its goroutine stopped return nil case int32(gen.ProcessStateTerminated): @@ -1481,6 +1771,7 @@ func (n *node) Kill(pid gen.PID) error { if old == int32(gen.ProcessStateTerminated) { return nil } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) // unregister process and stuff belonging to it n.unregisterProcess(p, gen.TerminateReasonKill) @@ -1716,9 +2007,13 @@ func (n *node) ApplicationProcessListShortInfo(name gen.Atom, limit int) ([]gen. MessagesIn: p.messagesIn, MessagesOut: p.messagesOut, MessagesMailbox: uint64(messagesMailbox), + MailboxLatency: p.mailbox.Latency(), RunningTime: p.runningTime, + InitTime: p.initTime, + Wakeups: p.wakeups, Uptime: p.Uptime(), State: p.State(), + StateTime: time.Now().UnixNano() - atomic.LoadInt64(&p.stateEntered), Parent: p.parent, Leader: p.leader, LogLevel: p.log.Level(), @@ -1751,9 +2046,13 @@ func (n *node) ApplicationProcessListShortInfo(name gen.Atom, limit int) ([]gen. MessagesIn: p.messagesIn, MessagesOut: p.messagesOut, MessagesMailbox: uint64(messagesMailbox), + MailboxLatency: p.mailbox.Latency(), RunningTime: p.runningTime, + InitTime: p.initTime, + Wakeups: p.wakeups, Uptime: p.Uptime(), State: p.State(), + StateTime: time.Now().UnixNano() - atomic.LoadInt64(&p.stateEntered), Parent: p.parent, Leader: p.leader, LogLevel: p.log.Level(), @@ -1941,7 +2240,7 @@ func (n *node) LoggerAddPID(pid gen.PID, name string, filter ...gen.LogLevel) er return err } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("node.LoggerAddPID added new process logger %s with name %q", pid, name) } return nil @@ -1971,7 +2270,7 @@ func (n *node) LoggerAdd(name string, logger gen.LoggerBehavior, filter ...gen.L } } - if lib.Trace() { + if lib.Verbose() { n.log.Trace("node.LoggerAdd added new logger with name %q", name) } return nil @@ -2022,6 +2321,194 @@ func (n *node) LoggerLevels(name string) []gen.LogLevel { return levels } +// tracing + +func (n *node) TracingExporterAddPID(pid gen.PID, name string, flags gen.TracingFlags) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + value, loaded := n.processes.Load(pid) + if loaded == false { + return gen.ErrProcessUnknown + } + p := value.(*process) + if p.tracingExporterName != "" { + return gen.ErrNotAllowed + } + entry := tracingExporterEntry{ + flags: flags, + pid: pid, + } + if _, loaded := n.tracingExporters.LoadOrStore(name, entry); loaded { + return gen.ErrTaken + } + p.tracingExporterName = name + return nil +} + +func (n *node) TracingExporterAdd(name string, exporter gen.TracingBehavior, flags gen.TracingFlags) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + if exporter == nil { + return gen.ErrIncorrect + } + entry := tracingExporterEntry{ + exporter: exporter, + flags: flags, + } + if _, loaded := n.tracingExporters.LoadOrStore(name, entry); loaded { + return gen.ErrTaken + } + return nil +} + +func (n *node) TracingExporterDeletePID(pid gen.PID) { + value, loaded := n.processes.Load(pid) + if loaded == false { + return + } + p := value.(*process) + if p.tracingExporterName != "" { + n.TracingExporterDelete(p.tracingExporterName) + p.tracingExporterName = "" + } +} + +func (n *node) TracingExporterDelete(name string) { + v, loaded := n.tracingExporters.LoadAndDelete(name) + if loaded == false { + return + } + entry := v.(tracingExporterEntry) + if entry.exporter != nil { + entry.exporter.Terminate() + } +} + +func (n *node) TracingExporters() []string { + var exporters []string + n.tracingExporters.Range(func(k, _ any) bool { + exporters = append(exporters, k.(string)) + return true + }) + return exporters +} + +func (n *node) TracingExporterFlags(name string) gen.TracingFlags { + v, ok := n.tracingExporters.Load(name) + if ok == false { + return 0 + } + return v.(tracingExporterEntry).flags +} + +func (n *node) sendTracingSpan(span gen.TracingSpan) { + if span.Kind >= 1 && span.Kind <= 5 { + atomic.AddUint64(&n.tracingSpans[span.Kind-1], 1) + } + n.tracingExporters.Range(func(k, v any) bool { + entry := v.(tracingExporterEntry) + if matchTracingFlags(entry.flags, span) == false { + return true + } + if entry.exporter != nil { + entry.exporter.HandleSpan(span) + return true + } + value, loaded := n.processes.Load(entry.pid) + if loaded == false { + return true + } + p := value.(*process) + msg := gen.TakeMailboxMessage() + msg.Type = gen.MailboxMessageTypeSpan + msg.Message = span + if p.mailbox.Main.Push(msg) == false { + gen.ReleaseMailboxMessage(msg) + return true + } + p.run() + return true + }) +} + +func matchTracingFlags(flags gen.TracingFlags, span gen.TracingSpan) bool { + switch span.Kind { + case gen.TracingKindSend, gen.TracingKindRequest, gen.TracingKindResponse: + if span.Point == gen.TracingPointSent { + return flags&gen.TracingFlagSend != 0 + } + return flags&gen.TracingFlagReceive != 0 + case gen.TracingKindSpawn, gen.TracingKindTerminate: + return flags&gen.TracingFlagProcs != 0 + } + return false +} + +func (n *node) SetTracingAttribute(key, value string) { + if len(key) > 5 && key[:5] == "ergo." { + return + } + for i, a := range n.tracingAttrs { + if a.Key == key { + attrs := make([]gen.TracingAttribute, len(n.tracingAttrs)) + copy(attrs, n.tracingAttrs) + attrs[i] = gen.TracingAttribute{Key: key, Value: value} + n.tracingAttrs = attrs + return + } + } + attrs := make([]gen.TracingAttribute, len(n.tracingAttrs)+1) + copy(attrs, n.tracingAttrs) + attrs[len(attrs)-1] = gen.TracingAttribute{Key: key, Value: value} + n.tracingAttrs = attrs +} + +func (n *node) RemoveTracingAttribute(key string) { + for i, a := range n.tracingAttrs { + if a.Key == key { + attrs := make([]gen.TracingAttribute, len(n.tracingAttrs)-1) + copy(attrs, n.tracingAttrs[:i]) + copy(attrs[i:], n.tracingAttrs[i+1:]) + n.tracingAttrs = attrs + return + } + } +} + +func (n *node) SetTracingSampler(sampler gen.TracingSampler) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + if sampler == gen.TracingSamplerDisable { + n.tracingSampler = nil + return nil + } + n.tracingSampler = sampler + return nil +} + +func (n *node) TracingSampler() gen.TracingSampler { + if n.tracingSampler == nil { + return gen.TracingSamplerDisable + } + return n.tracingSampler +} + +func (n *node) SetProcessTracingSampler(pid gen.PID, sampler gen.TracingSampler) error { + if n.isRunning() == false { + return gen.ErrNodeTerminated + } + value, loaded := n.processes.Load(pid) + if loaded == false { + return gen.ErrProcessUnknown + } + p := value.(*process) + p.tracingSampler = sampler + return nil +} + func (n *node) Loggers() []string { m := make(map[string]bool) for _, l := range n.loggers { @@ -2042,6 +2529,22 @@ func (n *node) dolog(message gen.MessageLog, loggername string) { if n.isRunning() == false { return } + + switch message.Level { + case gen.LogLevelTrace: + atomic.AddUint64(&n.logMessages[0], 1) + case gen.LogLevelDebug: + atomic.AddUint64(&n.logMessages[1], 1) + case gen.LogLevelInfo: + atomic.AddUint64(&n.logMessages[2], 1) + case gen.LogLevelWarning: + atomic.AddUint64(&n.logMessages[3], 1) + case gen.LogLevelError: + atomic.AddUint64(&n.logMessages[4], 1) + case gen.LogLevelPanic: + atomic.AddUint64(&n.logMessages[5], 1) + } + if l := n.loggers[message.Level]; l != nil { if loggername != "" { if v, found := l.Load(loggername); found { @@ -2103,27 +2606,32 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra } if factory == nil { + atomic.AddUint64(&n.processesSpawnFailed, 1) return empty, gen.ErrIncorrect } if options.ParentPID == empty || options.ParentLeader == empty { + atomic.AddUint64(&n.processesSpawnFailed, 1) return empty, gen.ErrParentUnknown } + now := time.Now() p := &process{ - node: n, - response: make(chan response, 10), - creation: time.Now().Unix(), - keeporder: true, - state: int32(gen.ProcessStateInit), - parent: options.ParentPID, - leader: options.ParentLeader, - application: options.Application, - important: options.ImportantDelivery, + node: n, + response: make(chan response, 10), + creation: now.Unix(), + keeporder: true, + state: int32(gen.ProcessStateInit), + stateEntered: now.UnixNano(), + parent: options.ParentPID, + leader: options.ParentLeader, + application: options.Application, + important: options.ImportantDelivery, } if options.Register != "" { if _, exist := n.names.LoadOrStore(options.Register, p); exist { + atomic.AddUint64(&n.processesSpawnFailed, 1) return p.pid, gen.ErrTaken } p.name = options.Register @@ -2155,7 +2663,7 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra for k, v := range options.ParentEnv { p.SetEnv(k, v) } - if lib.Trace() { + if lib.Verbose() { n.log.Trace( "...spawn new process %s (parent %s, %s) using %#v", p.pid, @@ -2197,6 +2705,7 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra behavior := factory() if behavior == nil { n.names.Delete(p.name) + atomic.AddUint64(&n.processesSpawnFailed, 1) return p.pid, errors.New("factory function must return non nil value") } p.behavior = behavior @@ -2219,6 +2728,22 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra // early registration - allows using Link/Monitor/RegisterEvent/RegisterName in Init n.processes.Store(p.pid, p) + tracingActive := options.Tracing.ID != [2]uint64{} + var spawnSpanID uint64 + var parentSpanID uint64 + if tracingActive { + parentSpanID = options.Tracing.SpanID + spawnSpanID = atomic.AddUint64(&n.spanID, 1) + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spawnSpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointSent, Kind: gen.TracingKindSpawn, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: options.ParentPID, To: pid, + Behavior: options.Tracing.Behavior, + }) + } + // Handle ProcessInit with timeout var initErr error deadline := options.Ref.ID[2] @@ -2230,6 +2755,7 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra if p.registered.Load() { n.names.Delete(p.name) } + atomic.AddUint64(&n.processesSpawnFailed, 1) return p.pid, gen.ErrTimeout } @@ -2240,7 +2766,9 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra errCh := make(chan error, 1) go func() { + initStart := time.Now() err := behavior.ProcessInit(p, options.Args...) + p.initTime = uint64(time.Since(initStart)) // try to claim "init completed" if atomic.CompareAndSwapInt32(&completed, 0, 1) { @@ -2251,6 +2779,7 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra // timeout won - main already called Kill, we do cleanup atomic.StoreInt32(&p.state, int32(gen.ProcessStateTerminated)) + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) n.cleanupProcess(p, gen.TerminateReasonKill) if lib.Recover() { defer func() { @@ -2276,6 +2805,7 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra if atomic.CompareAndSwapInt32(&completed, 0, 2) { // we won - goroutine will do cleanup when ProcessInit completes n.Kill(p.pid) + atomic.AddUint64(&n.processesSpawnFailed, 1) return p.pid, gen.ErrTimeout } // goroutine won - receive result @@ -2283,10 +2813,22 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra } } else { // no timeout - synchronous behavior + initStart := time.Now() initErr = behavior.ProcessInit(p, options.Args...) + p.initTime = uint64(time.Since(initStart)) } if initErr != nil { + if tracingActive { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spawnSpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointProcessed, Kind: gen.TracingKindSpawn, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: options.ParentPID, To: pid, + Behavior: p.sbehavior, Error: initErr.Error(), + }) + } n.cleanupProcess(p, initErr) go func() { if lib.Recover() { @@ -2300,15 +2842,28 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra } p.behavior.ProcessTerminate(initErr) }() + atomic.AddUint64(&n.processesSpawnFailed, 1) return p.pid, initErr } + if tracingActive { + n.sendTracingSpan(gen.TracingSpan{ + TraceID: options.Tracing.ID, SpanID: spawnSpanID, + ParentSpanID: parentSpanID, + Point: gen.TracingPointProcessed, Kind: gen.TracingKindSpawn, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: options.ParentPID, To: pid, + Behavior: p.sbehavior, + }) + } + if options.LinkParent { n.targets.LinkPID(p.pid, p.parent) } // switch to sleep state (process already registered above) p.state = int32(gen.ProcessStateSleep) + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) // do not count system app processes if p.application != system.Name { @@ -2319,15 +2874,33 @@ func (n *node) spawn(factory gen.ProcessFactory, options gen.ProcessOptionsExtra // so we should run this process to make sure this message is handled p.run() + atomic.AddUint64(&n.processesSpawned, 1) return p.pid, nil } // cleanupProcess performs core cleanup for a process. // Does NOT call waitprocesses.Done() - caller must handle if needed. func (n *node) cleanupProcess(p *process, reason error) { + if p.tracing.ID != [2]uint64{} { + var errString string + if reason != nil { + errString = reason.Error() + } + n.sendTracingSpan(gen.TracingSpan{ + TraceID: p.tracing.ID, SpanID: atomic.AddUint64(&n.spanID, 1), + ParentSpanID: p.tracing.SpanID, + Point: gen.TracingPointProcessed, Kind: gen.TracingKindTerminate, + Timestamp: time.Now().UnixNano(), + Node: n.name, From: p.pid, To: p.pid, + Behavior: p.sbehavior, Error: errString, + }) + } n.processes.Delete(p.pid) - n.RouteTerminatePID(p.pid, reason) // calls TerminatedTargetPID internally - n.targets.TerminatedProcess(p.pid, reason) + + if p.tracingExporterName != "" { + n.TracingExporterDelete(p.tracingExporterName) + p.tracingExporterName = "" + } n.log.Trace("...cleanupProcess %s", p.pid) @@ -2342,6 +2915,9 @@ func (n *node) cleanupProcess(p *process, reason error) { n.RouteTerminateAlias(a, reason) } + n.RouteTerminatePID(p.pid, reason) // calls TerminatedTargetPID internally + n.targets.TerminatedProcess(p.pid, reason) + p.metas.Range(func(_, v any) bool { m := v.(*meta) qm := gen.TakeMailboxMessage() @@ -2360,6 +2936,8 @@ func (n *node) cleanupProcess(p *process, reason error) { func (n *node) unregisterProcess(p *process, reason error) { n.cleanupProcess(p, reason) + atomic.AddUint64(&n.processesTerminated, 1) + if p.application != system.Name { n.waitprocesses.Done() } @@ -2434,39 +3012,3 @@ func (n *node) unregisterEvent(name gen.Atom, pid gen.PID) error { return n.targets.UnregisterEvent(pid, name) } - -func (n *node) validateLicenses(versions ...gen.Version) { - for _, version := range versions { - switch version.License { - case gen.LicenseMIT: - continue - - case "": - if lib.Trace() { - n.Log().Trace("undefined license for %s", version) - } - continue - - case gen.LicenseBSL1: - var valid bool - - if _, exist := n.licenses.LoadOrStore(version, valid); exist { - continue - } - - // TODO validate license - //if valid { - // continue - //} - - n.Log().Warning("%s is distributed under %q and can not be used "+ - "without a license for production/commercial purposes", - version, version.License) - - default: - if lib.Trace() { - n.Log().Trace("unhandled license %q for %s", version.License, version) - } - } - } -} diff --git a/node/process.go b/node/process.go index 596612144..f4ecfee9d 100644 --- a/node/process.go +++ b/node/process.go @@ -29,7 +29,8 @@ type process struct { behavior gen.ProcessBehavior sbehavior string - state int32 + state int32 + stateEntered int64 // Unix nanoseconds when current state was entered parent gen.PID leader gen.PID @@ -43,8 +44,14 @@ type process struct { messagesIn uint64 messagesOut uint64 runningTime uint64 + initTime uint64 + wakeups uint64 - compression gen.Compression + compression gen.Compression + tracing gen.Tracing + tracingSampler gen.TracingSampler + tracingAttrs []gen.TracingAttribute // permanent, COW + tracingSpanAttrs []gen.TracingAttribute // one-shot, nil after handler env sync.Map @@ -59,6 +66,9 @@ type process struct { // if act as a logger loggername string + + // if act as a tracing exporter + tracingExporterName string } type response struct { @@ -126,6 +136,8 @@ func (p *process) Spawn( Ref: ref, } + opts.Tracing = p.tracing + pid, err := p.node.spawn(factory, opts) if err != nil { return pid, err @@ -172,6 +184,9 @@ func (p *process) SpawnRegister( Args: args, Ref: ref, } + + opts.Tracing = p.tracing + pid, err := p.node.spawn(factory, opts) if err != nil { return pid, err @@ -339,7 +354,7 @@ func (p *process) RegisterName(name gen.Atom) error { return err } - p.log.setSource(gen.MessageLogProcess{Node: p.node.name, PID: p.pid, Name: p.name}) + p.log.setSource(gen.MessageLogProcess{Node: p.node.name, PID: p.pid, Name: p.name, Behavior: p.sbehavior}) return nil } @@ -498,6 +513,53 @@ func (p *process) ImportantDelivery() bool { return p.important } +func (p *process) SetTracingSampler(sampler gen.TracingSampler) error { + if p.isStateIR() == false { + return gen.ErrNotAllowed + } + if sampler == gen.TracingSamplerDisable { + p.tracingSampler = nil + return nil + } + p.tracingSampler = sampler + return nil +} + +func (p *process) TracingSampler() gen.TracingSampler { + if p.tracingSampler == nil { + return gen.TracingSamplerDisable + } + return p.tracingSampler +} + + + +func (p *process) PropagatingTrace() gen.Tracing { + return p.tracing +} + +func (p *process) SetPropagatingTrace(t gen.Tracing) { + p.tracing = t +} + +func (p *process) propagatingTrace() gen.Tracing { + if p.tracing.ID != [2]uint64{} { + t := p.tracing + t.Behavior = p.sbehavior + return t + } + // do not start new traces during Init phase + if atomic.LoadInt32(&p.state) == int32(gen.ProcessStateInit) { + return gen.Tracing{} + } + if p.tracingSampler != nil && p.tracingSampler.Sample() { + t := p.node.MakeTraceID() + t.Behavior = p.sbehavior + return t + } + return gen.Tracing{} +} + func (p *process) CreateAlias() (gen.Alias, error) { if p.isStateIR() == false { return gen.Alias{}, gen.ErrNotAllowed @@ -582,54 +644,22 @@ func (p *process) SendPID(to gen.PID, message any) error { if p.isStateIRT() == false { return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendPID to %s", to) } - // Sending to itself being in initialization stage: - // - // - we can't route this message to itself (via RouteSendPID) if this process - // is in the initialization stage since it isn't registered yet. - // - message can be routed by the process name (via RouteSendProcessID) - // because it is already registered before the invoking ProcessInit callback, - // which means we should not do this trick in SendProcessID method. - // - // So here, we should check if it is sending to itself and route this message manually - // right into the process mailbox. - - if to == p.pid { - // sending to itself - qm := gen.TakeMailboxMessage() - qm.From = p.pid - qm.Type = gen.MailboxMessageTypeRegular - qm.Target = to - qm.Message = message - - var queue lib.QueueMPSC - switch p.priority { - case gen.MessagePriorityHigh: - queue = p.mailbox.System - case gen.MessagePriorityMax: - queue = p.mailbox.Urgent - default: - queue = p.mailbox.Main - } - - if ok := queue.Push(qm); ok == false { - return gen.ErrProcessMailboxFull - } - - atomic.AddUint64(&p.messagesIn, 1) - p.run() - return nil - } - options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, ImportantDelivery: p.important, } + if options.Tracing.ID != [2]uint64{} { + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } + } if options.ImportantDelivery { ref := p.node.MakeRef() @@ -666,16 +696,22 @@ func (p *process) SendProcessID(to gen.ProcessID, message any) error { return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendProcessID to %s", to) } options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, ImportantDelivery: p.important, } + if options.Tracing.ID != [2]uint64{} { + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } + } if options.ImportantDelivery { ref := p.node.MakeRef() @@ -712,16 +748,22 @@ func (p *process) SendAlias(to gen.Alias, message any) error { return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendAlias to %s", to) } options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, ImportantDelivery: p.important, } + if options.Tracing.ID != [2]uint64{} { + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } + } if options.ImportantDelivery { ref := p.node.MakeRef() @@ -759,7 +801,7 @@ func (p *process) SendAfter(to any, message any, after time.Duration) (gen.Cance } return time.AfterFunc(after, func() { var err error - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendAfter %s to %s", after, to) } // we can't use p.Send(...) because it checks the process state @@ -798,7 +840,7 @@ func (p *process) SendWithPriorityAfter( } return time.AfterFunc(after, func() { var err error - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendWithPriorityAfter %s to %s with priority %s", after, to, priority) } // we can't use p.Send(...) because it checks the process state @@ -831,15 +873,19 @@ func (p *process) SendEvent(name gen.Atom, token gen.Ref, message any) error { return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("process SendEvent %s with token %s", name, token) } options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, } + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } em := gen.MessageEvent{ Event: gen.Event{Name: name, Node: p.node.name}, @@ -851,7 +897,6 @@ func (p *process) SendEvent(name gen.Atom, token gen.Ref, message any) error { return err } - atomic.AddUint64(&p.messagesOut, 1) return nil } @@ -870,7 +915,7 @@ func (p *process) SendExit(to gen.PID, reason error) error { return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendExit to %s", to) } err := p.node.RouteSendExit(p.pid, to, reason) @@ -897,7 +942,7 @@ func (p *process) SendExitAfter(to gen.PID, reason error, after time.Duration) ( } return time.AfterFunc(after, func() { - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendExitAfter %s to %s with reason %q", after, to, reason) } @@ -944,6 +989,7 @@ func (p *process) SendExitMeta(alias gen.Alias, reason error) error { } atomic.AddUint64(&m.messagesIn, 1) + atomic.AddUint64(&metap.messagesIn, 1) atomic.AddUint64(&p.messagesOut, 1) m.handle() return nil @@ -975,7 +1021,7 @@ func (p *process) SendExitMetaAfter(alias gen.Alias, reason error, after time.Du } return time.AfterFunc(after, func() { - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendExitMetaAfter %s to %s with reason %q", after, alias, reason) } @@ -1006,6 +1052,7 @@ func (p *process) SendExitMetaAfter(alias gen.Alias, reason error, after time.Du } atomic.AddUint64(&m.messagesIn, 1) + atomic.AddUint64(&metap.messagesIn, 1) atomic.AddUint64(&p.messagesOut, 1) m.handle() }).Stop, nil @@ -1015,16 +1062,20 @@ func (p *process) SendResponse(to gen.PID, ref gen.Ref, message any) error { if p.isStateIR() == false { return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendResponse to %s with %s", to, ref) } options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Ref: ref, Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, ImportantDelivery: p.important, } + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } atomic.AddUint64(&p.messagesOut, 1) return p.node.RouteSendResponse(p.pid, to, options, message) } @@ -1033,16 +1084,20 @@ func (p *process) SendResponseImportant(to gen.PID, ref gen.Ref, message any) er if p.isStateIR() == false { return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendResponseImportant to %s with %s", to, ref) } options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Ref: ref, Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, ImportantDelivery: true, } + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } if err := p.node.RouteSendResponse(p.pid, to, options, message); err != nil { return err } @@ -1056,16 +1111,20 @@ func (p *process) SendResponseError(to gen.PID, ref gen.Ref, err error) error { if p.isStateIR() == false { return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendResponseError to %s with %s", to, ref) } options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Ref: ref, Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, ImportantDelivery: p.important, } + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } atomic.AddUint64(&p.messagesOut, 1) return p.node.RouteSendResponseError(p.pid, to, options, err) } @@ -1074,16 +1133,20 @@ func (p *process) SendResponseErrorImportant(to gen.PID, ref gen.Ref, err error) if p.isStateIR() == false { return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("SendResponseErrorImportant to %s with %s", to, ref) } options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Ref: ref, Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, ImportantDelivery: true, } + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } if err := p.node.RouteSendResponseError(p.pid, to, options, err); err != nil { return err } @@ -1149,14 +1212,18 @@ func (p *process) CallPID(to gen.PID, message any, timeout int) (any, error) { } options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Ref: ref, Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, ImportantDelivery: p.important, } + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("CallPID to %s with %s", to, options.Ref) } @@ -1184,13 +1251,17 @@ func (p *process) CallProcessID(to gen.ProcessID, message any, timeout int) (any } options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Ref: ref, Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, ImportantDelivery: p.important, } - if lib.Trace() { + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } + if lib.Verbose() { p.log.Trace("CallProcessID %s with %s", to, options.Ref) } if err := p.node.RouteCallProcessID(p.pid, to, options, message); err != nil { @@ -1216,14 +1287,18 @@ func (p *process) CallAlias(to gen.Alias, message any, timeout int) (any, error) } options := gen.MessageOptions{ + Tracing: p.propagatingTrace(), Ref: ref, Priority: p.priority, Compression: p.compression, KeepNetworkOrder: p.keeporder, ImportantDelivery: p.important, } + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) + } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("CallAlias %s with %s", to, options.Ref) } @@ -1268,7 +1343,7 @@ func (p *process) Inspect(target gen.PID, item ...string) (map[string]string, er atomic.AddUint64(&p.messagesOut, 1) atomic.AddUint64(&targetp.messagesIn, 1) - if lib.Trace() { + if lib.Verbose() { p.log.Trace("Inspect %s with %s", target, ref) } @@ -1320,8 +1395,9 @@ func (p *process) InspectMeta(alias gen.Alias, item ...string) (map[string]strin } atomic.AddUint64(&p.messagesOut, 1) atomic.AddUint64(&m.messagesIn, 1) + atomic.AddUint64(&metap.messagesIn, 1) - if lib.Trace() { + if lib.Verbose() { m.log.Trace("Inspect meta %s with %s", alias, ref) } @@ -1340,7 +1416,7 @@ func (p *process) RegisterEvent(name gen.Atom, options gen.EventOptions) (gen.Re return empty, gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("process RegisterEvent %s", name) } @@ -1352,7 +1428,7 @@ func (p *process) UnregisterEvent(name gen.Atom) error { return gen.ErrNotAllowed } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("process UnregisterEvent %s", name) } @@ -1410,7 +1486,7 @@ func (p *process) LinkPID(target gen.PID) error { return gen.ErrTargetExist } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("LinkPID with %s", target) } @@ -1436,7 +1512,7 @@ func (p *process) UnlinkPID(target gen.PID) error { return gen.ErrTargetUnknown } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("UnlinkPID with %s", target) } @@ -1460,7 +1536,7 @@ func (p *process) LinkProcessID(target gen.ProcessID) error { return gen.ErrTargetExist } - if lib.Trace() { + if lib.Verbose() { p.log.Trace("LinkProcessID with %s", target) } @@ -1872,6 +1948,84 @@ func (p *process) Behavior() gen.ProcessBehavior { return p.behavior } +func (p *process) BehaviorName() string { + return p.sbehavior +} + +func (p *process) SetTracingAttribute(key, value string) { + if len(key) > 5 && key[:5] == "ergo." { + return + } + // COW: copy slice, overwrite existing key or append + for i, a := range p.tracingAttrs { + if a.Key == key { + attrs := make([]gen.TracingAttribute, len(p.tracingAttrs)) + copy(attrs, p.tracingAttrs) + attrs[i] = gen.TracingAttribute{Key: key, Value: value} + p.tracingAttrs = attrs + return + } + } + attrs := make([]gen.TracingAttribute, len(p.tracingAttrs)+1) + copy(attrs, p.tracingAttrs) + attrs[len(attrs)-1] = gen.TracingAttribute{Key: key, Value: value} + p.tracingAttrs = attrs +} + +func (p *process) RemoveTracingAttribute(key string) { + for i, a := range p.tracingAttrs { + if a.Key == key { + attrs := make([]gen.TracingAttribute, len(p.tracingAttrs)-1) + copy(attrs, p.tracingAttrs[:i]) + copy(attrs[i:], p.tracingAttrs[i+1:]) + p.tracingAttrs = attrs + return + } + } +} + +func (p *process) SetTracingSpanAttribute(key, value string) { + if len(key) > 5 && key[:5] == "ergo." { + return + } + for i, a := range p.tracingSpanAttrs { + if a.Key == key { + p.tracingSpanAttrs[i].Value = value + return + } + } + p.tracingSpanAttrs = append(p.tracingSpanAttrs, gen.TracingAttribute{Key: key, Value: value}) +} + +func (p *process) TracingAttributes() []gen.TracingAttribute { + if len(p.tracingAttrs) == 0 { + return p.tracingSpanAttrs + } + if len(p.tracingSpanAttrs) == 0 { + return p.tracingAttrs + } + m := make([]gen.TracingAttribute, 0, len(p.tracingAttrs)+len(p.tracingSpanAttrs)) + return append(append(m, p.tracingAttrs...), p.tracingSpanAttrs...) +} + +func (p *process) ClearTracingSpanAttributes() { + if p.tracingSpanAttrs != nil { + p.tracingSpanAttrs = nil + } +} + +func (p *process) applyTracingAttrs(options *gen.MessageOptions) { + attrs := p.TracingAttributes() + if len(attrs) == 0 { + return + } + options.TracingAttributes = attrs +} + +func (p *process) SendTracingSpan(span gen.TracingSpan) { + p.node.sendTracingSpan(span) +} + func (p *process) Forward( to gen.PID, message *gen.MailboxMessage, @@ -1906,6 +2060,7 @@ func (p *process) Forward( fp.run() return nil } + // internal func (p *process) isAlive() bool { @@ -1947,6 +2102,7 @@ func (p *process) waitResponse(ref gen.Ref, timeout int) (any, error) { atomic.StoreInt32(&p.state, prevState) return nil, gen.ErrNotAllowed } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) timer := lib.TakeTimer() defer lib.ReleaseTimer(timer) @@ -1960,7 +2116,7 @@ func (p *process) waitResponse(ref gen.Ref, timeout int) (any, error) { retry: select { case <-timer.C: - if lib.Trace() { + if lib.Verbose() { p.log.Trace("request with ref %s is timed out", ref) } err = gen.ErrTimeout @@ -1969,7 +2125,7 @@ retry: // we got a late response to the previous request that has been timed // out earlier and we made another request with the new reference - Ref. // just drop it and wait one more time - if lib.Trace() { + if lib.Verbose() { p.log.Trace("got late response on request with ref %s (exp %s). dropped", r.ref, ref) } goto retry @@ -1979,7 +2135,11 @@ retry: if r.important { // send ack for important response options := gen.MessageOptions{ - Ref: r.ref, + Tracing: p.propagatingTrace(), + Ref: r.ref, + } + if options.Tracing.ID != [2]uint64{} { + p.applyTracingAttrs(&options) } p.node.RouteSendResponseError(p.pid, r.from, options, err) } @@ -1989,5 +2149,6 @@ retry: if swapped := atomic.CompareAndSwapInt32(&p.state, int32(gen.ProcessStateWaitResponse), prevState); swapped == false { return nil, gen.ErrProcessTerminated } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) return response, err } diff --git a/node/process_run.go b/node/process_run.go index 1d41a7b36..dc2db0386 100644 --- a/node/process_run.go +++ b/node/process_run.go @@ -17,6 +17,8 @@ func (p *process) run() { int32(gen.ProcessStateRunning)) == false { // already running or terminated return } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) + atomic.AddUint64(&p.wakeups, 1) go func() { if lib.Recover() { defer func() { @@ -28,6 +30,7 @@ func (p *process) run() { if old == int32(gen.ProcessStateTerminated) { return } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) p.node.unregisterProcess(p, gen.TerminateReasonPanic) p.behavior.ProcessTerminate(gen.TerminateReasonPanic) } @@ -50,7 +53,7 @@ func (p *process) run() { if old == int32(gen.ProcessStateTerminated) { return } - + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) p.node.unregisterProcess(p, e) p.behavior.ProcessTerminate(err) return @@ -70,10 +73,12 @@ func (p *process) run() { if old == int32(gen.ProcessStateTerminated) { return } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) p.node.unregisterProcess(p, gen.TerminateReasonKill) p.behavior.ProcessTerminate(gen.TerminateReasonKill) return } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) // check if something left in the inbox and try to handle it if p.mailbox.Main.Item() == nil { if p.mailbox.System.Item() == nil { @@ -94,6 +99,7 @@ func (p *process) run() { // another goroutine is already running return } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) goto next }() } diff --git a/node/process_run_pprof.go b/node/process_run_pprof.go index 750250bfe..08caf4cff 100644 --- a/node/process_run_pprof.go +++ b/node/process_run_pprof.go @@ -19,6 +19,8 @@ func (p *process) run() { int32(gen.ProcessStateRunning)) == false { // already running or terminated return } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) + atomic.AddUint64(&p.wakeups, 1) go func() { labels := pprof.Labels("pid", p.pid.String()) pprof.Do(context.Background(), labels, func(context.Context) { @@ -32,6 +34,7 @@ func (p *process) run() { if old == int32(gen.ProcessStateTerminated) { return } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) p.node.unregisterProcess(p, gen.TerminateReasonPanic) p.behavior.ProcessTerminate(gen.TerminateReasonPanic) } @@ -54,7 +57,7 @@ func (p *process) run() { if old == int32(gen.ProcessStateTerminated) { return } - + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) p.node.unregisterProcess(p, e) p.behavior.ProcessTerminate(err) return @@ -74,10 +77,12 @@ func (p *process) run() { if old == int32(gen.ProcessStateTerminated) { return } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) p.node.unregisterProcess(p, gen.TerminateReasonKill) p.behavior.ProcessTerminate(gen.TerminateReasonKill) return } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) // check if something left in the inbox and try to handle it if p.mailbox.Main.Item() == nil { if p.mailbox.System.Item() == nil { @@ -98,6 +103,7 @@ func (p *process) run() { // another goroutine is already running return } + atomic.StoreInt64(&p.stateEntered, time.Now().UnixNano()) goto next }) }() diff --git a/node/tm/alias.go b/node/tm/alias.go index ca5a43c40..92d2b5d9b 100644 --- a/node/tm/alias.go +++ b/node/tm/alias.go @@ -3,25 +3,26 @@ package tm import "ergo.services/ergo/gen" func (tm *targetManager) LinkAlias(consumer gen.PID, target gen.Alias) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.linkRelations[key]; exists { - if consumer.Node != tm.core.Name() { - return nil - } - + _, exists := s.linkRelations[key] + if exists == true && consumer.Node != tm.core.Name() { + return nil + } + if exists == true { return gen.ErrTargetExist } - tm.linkRelations[key] = struct{}{} + s.linkRelations[key] = struct{}{} - entry := tm.targetIndex[target] + entry := s.targetIndex[target] needsRemote := false if entry == nil { @@ -29,7 +30,7 @@ func (tm *targetManager) LinkAlias(consumer gen.PID, target gen.Alias) error { allowAlwaysFirst: true, consumers: make(map[gen.PID]struct{}), } - tm.targetIndex[target] = entry + s.targetIndex[target] = entry needsRemote = true } @@ -49,49 +50,45 @@ func (tm *targetManager) LinkAlias(consumer gen.PID, target gen.Alias) error { connection, err := tm.core.GetConnection(target.Node) if err != nil { - delete(tm.linkRelations, key) + delete(s.linkRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } err = connection.LinkAlias(tm.core.PID(), target) if err != nil { - delete(tm.linkRelations, key) + delete(s.linkRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } entry.allowAlwaysFirst = false - return nil } func (tm *targetManager) UnlinkAlias(consumer gen.PID, target gen.Alias) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.linkRelations[key]; exists == false { + if _, exists := s.linkRelations[key]; exists == false { return nil } - delete(tm.linkRelations, key) + delete(s.linkRelations, key) - entry := tm.targetIndex[target] + entry := s.targetIndex[target] if entry == nil { return nil } @@ -99,9 +96,8 @@ func (tm *targetManager) UnlinkAlias(consumer gen.PID, target gen.Alias) error { delete(entry.consumers, consumer) isLast := (len(entry.consumers) == 0) - if isLast { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } if target.Node == tm.core.Name() { @@ -116,7 +112,6 @@ func (tm *targetManager) UnlinkAlias(consumer gen.PID, target gen.Alias) error { break } } - if hasLocal { return nil } @@ -128,30 +123,30 @@ func (tm *targetManager) UnlinkAlias(consumer gen.PID, target gen.Alias) error { } connection.UnlinkAlias(tm.core.PID(), target) - return nil } func (tm *targetManager) MonitorAlias(consumer gen.PID, target gen.Alias) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.monitorRelations[key]; exists { - if consumer.Node != tm.core.Name() { - return nil - } - + _, exists := s.monitorRelations[key] + if exists == true && consumer.Node != tm.core.Name() { + return nil + } + if exists == true { return gen.ErrTargetExist } - tm.monitorRelations[key] = struct{}{} + s.monitorRelations[key] = struct{}{} - entry := tm.targetIndex[target] + entry := s.targetIndex[target] needsRemote := false if entry == nil { @@ -159,7 +154,7 @@ func (tm *targetManager) MonitorAlias(consumer gen.PID, target gen.Alias) error allowAlwaysFirst: true, consumers: make(map[gen.PID]struct{}), } - tm.targetIndex[target] = entry + s.targetIndex[target] = entry needsRemote = true } @@ -179,49 +174,45 @@ func (tm *targetManager) MonitorAlias(consumer gen.PID, target gen.Alias) error connection, err := tm.core.GetConnection(target.Node) if err != nil { - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } err = connection.MonitorAlias(tm.core.PID(), target) if err != nil { - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } entry.allowAlwaysFirst = false - return nil } func (tm *targetManager) DemonitorAlias(consumer gen.PID, target gen.Alias) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.monitorRelations[key]; exists == false { + if _, exists := s.monitorRelations[key]; exists == false { return nil } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) - entry := tm.targetIndex[target] + entry := s.targetIndex[target] if entry == nil { return nil } @@ -229,9 +220,8 @@ func (tm *targetManager) DemonitorAlias(consumer gen.PID, target gen.Alias) erro delete(entry.consumers, consumer) isLast := (len(entry.consumers) == 0) - if isLast { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } if target.Node == tm.core.Name() { @@ -246,7 +236,6 @@ func (tm *targetManager) DemonitorAlias(consumer gen.PID, target gen.Alias) erro break } } - if hasLocal { return nil } @@ -258,6 +247,5 @@ func (tm *targetManager) DemonitorAlias(consumer gen.PID, target gen.Alias) erro } connection.DemonitorAlias(tm.core.PID(), target) - return nil } diff --git a/node/tm/alias_test.go b/node/tm/alias_test.go index c91f3cef8..3d57ac2d4 100644 --- a/node/tm/alias_test.go +++ b/node/tm/alias_test.go @@ -20,8 +20,7 @@ func TestLinkAlias_Local(t *testing.T) { t.Fatalf("LinkAlias failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, target); exists == false { t.Error("Link should be stored") } @@ -43,13 +42,12 @@ func TestLinkAlias_Remote_First(t *testing.T) { } // Verify link stored in linkRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, target); exists == false { t.Error("Link should be stored in linkRelations") } // Verify targetIndex created - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should be created") } @@ -92,20 +90,18 @@ func TestLinkAlias_Remote_Second(t *testing.T) { } // Verify both links stored in linkRelations - if len(tm.linkRelations) != 2 { - t.Errorf("Expected 2 link relations, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 2 { + t.Errorf("Expected 2 link relations, got %d", tm.totalLinks()) } - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.linkRelations[key1]; exists == false { + if exists := tm.hasLinkRelation(consumer1, target); exists == false { t.Error("consumer1 link should exist in linkRelations") } - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, target); exists == false { t.Error("consumer2 link should exist in linkRelations") } // Verify targetIndex has both consumers - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -145,13 +141,12 @@ func TestLinkAlias_NetworkError_Rollback(t *testing.T) { } // Verify link rolled back from linkRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("Link should be rolled back from linkRelations") } // Verify targetIndex cleaned after rollback - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned after rollback") } } @@ -174,12 +169,12 @@ func TestLinkAlias_RemoteCorePID_Duplicate_Ignored(t *testing.T) { } // Verify only ONE relation exists (duplicate was ignored) - if len(tm.linkRelations) != 1 { - t.Errorf("Expected 1 link relation, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 1 { + t.Errorf("Expected 1 link relation, got %d", tm.totalLinks()) } // Verify targetIndex has only one consumer - entry := tm.targetIndex[localTarget] + entry := tm.getTargetEntry(localTarget) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -205,13 +200,12 @@ func TestUnlinkAlias_Local(t *testing.T) { } // Verify link removed from linkRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("Link should be removed from linkRelations") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned") } } @@ -232,19 +226,17 @@ func TestUnlinkAlias_NotLast(t *testing.T) { tm.UnlinkAlias(consumer1, target) // Verify consumer1 link removed from linkRelations - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.linkRelations[key1]; exists { + if exists := tm.hasLinkRelation(consumer1, target); exists { t.Error("consumer1 link should be removed from linkRelations") } // Verify consumer2 link still exists - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, target); exists == false { t.Error("consumer2 link should still exist in linkRelations") } // Verify targetIndex still exists with only consumer2 - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should still exist") } @@ -277,13 +269,12 @@ func TestUnlinkAlias_Last(t *testing.T) { } // Verify link removed from linkRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("Link should be removed from linkRelations") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned when last consumer removed") } @@ -320,8 +311,7 @@ func TestMonitorAlias_Local(t *testing.T) { t.Fatalf("MonitorAlias failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists == false { + if exists := tm.hasMonitorRelation(consumer, target); exists == false { t.Error("Monitor should be stored") } @@ -343,13 +333,12 @@ func TestMonitorAlias_Remote_First(t *testing.T) { } // Verify monitor stored in monitorRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists == false { + if exists := tm.hasMonitorRelation(consumer, target); exists == false { t.Error("Monitor should be stored in monitorRelations") } // Verify targetIndex created - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should be created") } @@ -392,20 +381,18 @@ func TestMonitorAlias_Remote_Second(t *testing.T) { } // Verify both monitors stored in monitorRelations - if len(tm.monitorRelations) != 2 { - t.Errorf("Expected 2 monitor relations, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 2 { + t.Errorf("Expected 2 monitor relations, got %d", tm.totalMonitors()) } - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.monitorRelations[key1]; exists == false { + if exists := tm.hasMonitorRelation(consumer1, target); exists == false { t.Error("consumer1 monitor should exist in monitorRelations") } - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, target); exists == false { t.Error("consumer2 monitor should exist in monitorRelations") } // Verify targetIndex has both consumers - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -445,13 +432,12 @@ func TestMonitorAlias_NetworkError_Rollback(t *testing.T) { } // Verify monitor rolled back from monitorRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, target); exists { t.Error("Monitor should be rolled back from monitorRelations") } // Verify targetIndex cleaned after rollback - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned after rollback") } } @@ -474,12 +460,12 @@ func TestMonitorAlias_RemoteCorePID_Duplicate_Ignored(t *testing.T) { } // Verify only ONE relation exists (duplicate was ignored) - if len(tm.monitorRelations) != 1 { - t.Errorf("Expected 1 monitor relation, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 1 { + t.Errorf("Expected 1 monitor relation, got %d", tm.totalMonitors()) } // Verify targetIndex has only one consumer - entry := tm.targetIndex[localTarget] + entry := tm.getTargetEntry(localTarget) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -505,13 +491,12 @@ func TestDemonitorAlias_Local(t *testing.T) { } // Verify monitor removed from monitorRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, target); exists { t.Error("Monitor should be removed from monitorRelations") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned") } } @@ -532,19 +517,17 @@ func TestDemonitorAlias_NotLast(t *testing.T) { tm.DemonitorAlias(consumer1, target) // Verify consumer1 monitor removed from monitorRelations - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.monitorRelations[key1]; exists { + if exists := tm.hasMonitorRelation(consumer1, target); exists { t.Error("consumer1 monitor should be removed from monitorRelations") } // Verify consumer2 monitor still exists - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, target); exists == false { t.Error("consumer2 monitor should still exist in monitorRelations") } // Verify targetIndex still exists with only consumer2 - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should still exist") } @@ -577,13 +560,12 @@ func TestDemonitorAlias_Last(t *testing.T) { } // Verify monitor removed from monitorRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, target); exists { t.Error("Monitor should be removed from monitorRelations") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned when last consumer removed") } @@ -611,8 +593,7 @@ func TestDemonitorAlias_Complete(t *testing.T) { } // consumer2 still exists - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, target); exists == false { t.Error("Second monitor should still exist") } @@ -623,11 +604,11 @@ func TestDemonitorAlias_Complete(t *testing.T) { } // All cleaned - if len(tm.monitorRelations) != 0 { + if tm.totalMonitors() != 0 { t.Error("All monitors should be cleaned") } - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned") } } diff --git a/node/tm/bench_test.go b/node/tm/bench_test.go new file mode 100644 index 000000000..bd11d3400 --- /dev/null +++ b/node/tm/bench_test.go @@ -0,0 +1,380 @@ +package tm + +import ( + "fmt" + "sync" + "sync/atomic" + "testing" + "time" + + "ergo.services/ergo/gen" +) + +func benchTM(nodeName string) (*targetManager, *mockCore) { + core := newMockCore(nodeName) + tm := Create(core, Options{}).(*targetManager) + return tm, core +} + +func makePID(node string, id uint64) gen.PID { + return gen.PID{Node: gen.Atom(node), ID: id, Creation: 1} +} + +func makeEvent(node string, name string) gen.Event { + return gen.Event{Node: gen.Atom(node), Name: gen.Atom(name)} +} + +func BenchmarkLinkPID_Local(b *testing.B) { + tm, _ := benchTM("node1") + target := makePID("node1", 100) + + b.ResetTimer() + for i := 0; i < b.N; i++ { + consumer := makePID("node1", uint64(i+1000)) + tm.LinkPID(consumer, target) + } +} + +func BenchmarkMonitorPID_Local(b *testing.B) { + tm, _ := benchTM("node1") + target := makePID("node1", 100) + + b.ResetTimer() + for i := 0; i < b.N; i++ { + consumer := makePID("node1", uint64(i+1000)) + tm.MonitorPID(consumer, target) + } +} + +func BenchmarkPublishEvent_NoSubscribers(b *testing.B) { + tm, _ := benchTM("node1") + producer := makePID("node1", 10) + token, _ := tm.RegisterEvent(producer, "bench_event", gen.EventOptions{}) + event := makeEvent("node1", "bench_event") + + msg := gen.MessageEvent{Event: event} + opts := gen.MessageOptions{} + + b.ResetTimer() + for i := 0; i < b.N; i++ { + tm.PublishEvent(producer, token, opts, msg) + } +} + +func BenchmarkPublishEvent_10Subscribers(b *testing.B) { + tm, _ := benchTM("node1") + producer := makePID("node1", 10) + token, _ := tm.RegisterEvent(producer, "bench_event", gen.EventOptions{}) + event := makeEvent("node1", "bench_event") + + for i := 0; i < 10; i++ { + consumer := makePID("node1", uint64(i+100)) + tm.LinkEvent(consumer, event) + } + + msg := gen.MessageEvent{Event: event} + opts := gen.MessageOptions{} + + b.ResetTimer() + for i := 0; i < b.N; i++ { + tm.PublishEvent(producer, token, opts, msg) + } +} + +func BenchmarkPublishEvent_100Subscribers(b *testing.B) { + tm, _ := benchTM("node1") + producer := makePID("node1", 10) + token, _ := tm.RegisterEvent(producer, "bench_event", gen.EventOptions{}) + event := makeEvent("node1", "bench_event") + + for i := 0; i < 100; i++ { + consumer := makePID("node1", uint64(i+100)) + tm.LinkEvent(consumer, event) + } + + msg := gen.MessageEvent{Event: event} + opts := gen.MessageOptions{} + + b.ResetTimer() + for i := 0; i < b.N; i++ { + tm.PublishEvent(producer, token, opts, msg) + } +} + +func BenchmarkHasLink(b *testing.B) { + tm, _ := benchTM("node1") + consumer := makePID("node1", 1) + target := makePID("node1", 100) + tm.LinkPID(consumer, target) + + b.ResetTimer() + for i := 0; i < b.N; i++ { + tm.HasLink(consumer, target) + } +} + +func BenchmarkLinkPID_Parallel(b *testing.B) { + tm, _ := benchTM("node1") + + b.RunParallel(func(pb *testing.PB) { + id := uint64(0) + for pb.Next() { + id++ + consumer := makePID("node1", id+10000) + target := makePID("node1", id) + tm.LinkPID(consumer, target) + } + }) +} + +func BenchmarkPublishEvent_Parallel(b *testing.B) { + tm, _ := benchTM("node1") + producer := makePID("node1", 10) + token, _ := tm.RegisterEvent(producer, "bench_event", gen.EventOptions{}) + event := makeEvent("node1", "bench_event") + + for i := 0; i < 10; i++ { + consumer := makePID("node1", uint64(i+100)) + tm.LinkEvent(consumer, event) + } + + msg := gen.MessageEvent{Event: event} + opts := gen.MessageOptions{} + + b.ResetTimer() + b.RunParallel(func(pb *testing.PB) { + for pb.Next() { + tm.PublishEvent(producer, token, opts, msg) + } + }) +} + +func BenchmarkContention_PublishWhileMonitor(b *testing.B) { + tm, _ := benchTM("node1") + + type eventInfo struct { + producer gen.PID + token gen.Ref + event gen.Event + } + var events []eventInfo + + for e := 0; e < 10; e++ { + producer := makePID("node1", uint64(e+1)) + name := gen.Atom(fmt.Sprintf("event_%d", e)) + token, _ := tm.RegisterEvent(producer, name, gen.EventOptions{}) + event := gen.Event{Node: "node1", Name: name} + + for s := 0; s < 10; s++ { + consumer := makePID("node1", uint64(e*100+s+1000)) + tm.LinkEvent(consumer, event) + } + + events = append(events, eventInfo{producer: producer, token: token, event: event}) + } + + b.ResetTimer() + b.RunParallel(func(pb *testing.PB) { + id := uint64(0) + for pb.Next() { + id++ + if id%2 == 0 { + ei := events[id%uint64(len(events))] + msg := gen.MessageEvent{Event: ei.event} + tm.PublishEvent(ei.producer, ei.token, gen.MessageOptions{}, msg) + } else { + consumer := makePID("node1", id+50000) + target := makePID("node1", id) + tm.MonitorPID(consumer, target) + } + } + }) +} + +func BenchmarkTerminatedTargetPID(b *testing.B) { + for i := 0; i < b.N; i++ { + b.StopTimer() + tm, _ := benchTM("node1") + target := makePID("node1", 100) + + for c := 0; c < 10; c++ { + consumer := makePID("node1", uint64(c+1)) + tm.LinkPID(consumer, target) + } + + b.StartTimer() + tm.TerminatedTargetPID(target, fmt.Errorf("test")) + } +} + +func BenchmarkTerminatedProcess(b *testing.B) { + for i := 0; i < b.N; i++ { + b.StopTimer() + tm, _ := benchTM("node1") + consumer := makePID("node1", 1) + + for t := 0; t < 100; t++ { + target := makePID("node1", uint64(t+100)) + tm.LinkPID(consumer, target) + } + + b.StartTimer() + tm.TerminatedProcess(consumer, fmt.Errorf("test")) + } +} + +func BenchmarkTerminatedTargetNode(b *testing.B) { + for i := 0; i < b.N; i++ { + b.StopTimer() + tm, _ := benchTM("node1") + + for c := 0; c < 100; c++ { + consumer := makePID("node1", uint64(c+1)) + target := makePID("node2", uint64(c+1)) + s := tm.shardFor(target) + key := relationKey{consumer: consumer, target: target} + s.linkRelations[key] = struct{}{} + entry := &targetEntry{consumers: make(map[gen.PID]struct{})} + entry.consumers[consumer] = struct{}{} + s.targetIndex[target] = entry + } + + b.StartTimer() + tm.TerminatedTargetNode("node2", fmt.Errorf("test")) + } +} + +func BenchmarkContention_TerminateNodeWhileMonitor(b *testing.B) { + tm, _ := benchTM("node1") + + for c := 0; c < 100; c++ { + consumer := makePID("node1", uint64(c+1)) + target := makePID("node2", uint64(c+1)) + s := tm.shardFor(target) + key := relationKey{consumer: consumer, target: target} + s.linkRelations[key] = struct{}{} + entry := &targetEntry{consumers: make(map[gen.PID]struct{})} + entry.consumers[consumer] = struct{}{} + s.targetIndex[target] = entry + } + + b.ResetTimer() + + var wg sync.WaitGroup + wg.Add(1) + + done := make(chan struct{}) + go func() { + defer wg.Done() + id := uint64(0) + for { + select { + case <-done: + return + default: + id++ + consumer := makePID("node1", id+90000) + target := makePID("node1", id+80000) + tm.MonitorPID(consumer, target) + } + } + }() + + for i := 0; i < b.N; i++ { + tm.TerminatedTargetNode("node2", fmt.Errorf("test")) + + for c := 0; c < 100; c++ { + consumer := makePID("node1", uint64(c+1)) + target := makePID("node2", uint64(c+1)) + s := tm.shardFor(target) + s.mutex.Lock() + key := relationKey{consumer: consumer, target: target} + s.linkRelations[key] = struct{}{} + entry := &targetEntry{consumers: make(map[gen.PID]struct{})} + entry.consumers[consumer] = struct{}{} + s.targetIndex[target] = entry + s.mutex.Unlock() + } + } + + close(done) + wg.Wait() +} + +// BenchmarkMonitorLatencyUnderPublishLoad measures MonitorPID latency +// while N goroutines continuously publish events. +// This simulates the real bottleneck: sustained publish pressure starving MonitorPID. +func BenchmarkMonitorLatencyUnderPublishLoad(b *testing.B) { + for _, publishers := range []int{1, 4, 8, 14} { + b.Run(fmt.Sprintf("publishers=%d", publishers), func(b *testing.B) { + tm, _ := benchTM("node1") + + // Register events, one per publisher + type pubInfo struct { + producer gen.PID + token gen.Ref + event gen.Event + msg gen.MessageEvent + } + pubs := make([]pubInfo, publishers) + for i := 0; i < publishers; i++ { + producer := makePID("node1", uint64(i+1)) + name := gen.Atom(fmt.Sprintf("event_%d", i)) + token, _ := tm.RegisterEvent(producer, name, gen.EventOptions{}) + event := gen.Event{Node: "node1", Name: name} + + // 20 subscribers per event + for s := 0; s < 20; s++ { + consumer := makePID("node1", uint64(i*100+s+1000)) + tm.LinkEvent(consumer, event) + } + + pubs[i] = pubInfo{ + producer: producer, + token: token, + event: event, + msg: gen.MessageEvent{Event: event}, + } + } + + // Start publishers - they publish non-stop + stop := make(chan struct{}) + var publishCount atomic.Int64 + var publisherWg sync.WaitGroup + + for i := 0; i < publishers; i++ { + publisherWg.Add(1) + go func(p pubInfo) { + defer publisherWg.Done() + opts := gen.MessageOptions{} + for { + select { + case <-stop: + return + default: + tm.PublishEvent(p.producer, p.token, opts, p.msg) + publishCount.Add(1) + } + } + }(pubs[i]) + } + + // Let publishers warm up + time.Sleep(10 * time.Millisecond) + + // Benchmark: measure MonitorPID latency under load + b.ResetTimer() + for i := 0; i < b.N; i++ { + consumer := makePID("node1", uint64(i+500000)) + target := makePID("node1", uint64(i+600000)) + tm.MonitorPID(consumer, target) + } + b.StopTimer() + + close(stop) + publisherWg.Wait() + + b.ReportMetric(float64(publishCount.Load())/float64(b.N), "publishes/monitor") + }) + } +} diff --git a/node/tm/event.go b/node/tm/event.go index 5eefdb222..ed1d0b25b 100644 --- a/node/tm/event.go +++ b/node/tm/event.go @@ -7,86 +7,121 @@ import ( ) func (tm *targetManager) RegisterEvent(producer gen.PID, name gen.Atom, options gen.EventOptions) (gen.Ref, error) { - tm.mutex.Lock() - defer tm.mutex.Unlock() - event := gen.Event{Node: tm.core.Name(), Name: name} + s := tm.shardFor(event) + s.mutex.Lock() + defer s.mutex.Unlock() - // Check if already exists - if _, exists := tm.events[event]; exists { + if _, exists := s.events[event]; exists { return gen.Ref{}, gen.ErrTaken } - // Generate unique token - token := gen.Ref{ - Node: tm.core.Name(), - Creation: tm.core.PID().Creation, - ID: [3]uint64{uint64(time.Now().UnixNano()), 0, 0}, + token := tm.generateToken() + + // Node-level events (producer is corePID) do not consume MessageEventStart + // or MessageEventStop, so Notify is silently ignored for them. + notify := options.Notify + if producer == tm.core.PID() { + notify = false } - // Create event entry + id := tm.eventSeq.Add(1) entry := &eventEntry{ + id: id, + createdAt: time.Now().UnixNano(), + event: event, producer: producer, token: token, - notify: options.Notify, + notify: notify, + open: options.Open, linkSubscribersIndex: make(map[gen.PID]int), monitorSubscribersIndex: make(map[gen.PID]int), subscriberCount: 0, } - // Create buffer if configured if options.Buffer > 0 { - entry.buffer = make([]gen.MessageEvent, 0, options.Buffer) - entry.bufferSize = options.Buffer + entry.buffer = &eventRingBuffer{ + data: make([]gen.MessageEvent, options.Buffer), + size: options.Buffer, + } } - tm.events[event] = entry + s.events[event] = entry - // Add to producerEvents index - if tm.producerEvents[producer] == nil { - tm.producerEvents[producer] = make(map[gen.Event]struct{}) + if s.producerEvents[producer] == nil { + s.producerEvents[producer] = make(map[gen.Event]struct{}) } - tm.producerEvents[producer][event] = struct{}{} + s.producerEvents[producer][event] = struct{}{} + + tm.eventIndex.Store(id, entry) return token, nil } func (tm *targetManager) UnregisterEvent(producer gen.PID, name gen.Atom) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() - event := gen.Event{Node: tm.core.Name(), Name: name} - entry, exists := tm.events[event] + s := tm.shardFor(event) + s.mutex.Lock() + + entry, exists := s.events[event] if exists == false { + s.mutex.Unlock() return gen.ErrEventUnknown } if entry.producer != producer { + s.mutex.Unlock() return gen.ErrEventOwner } - // Collect all subscribers (links + monitors) remoteNodes := make(map[gen.Atom]bool) var localExitConsumers []gen.PID var localDownConsumers []gen.PID for _, consumer := range entry.linkSubscribers { - if consumer.Node == tm.core.Name() { - localExitConsumers = append(localExitConsumers, consumer) - } else { + if consumer.Node != tm.core.Name() { remoteNodes[consumer.Node] = true + continue } + localExitConsumers = append(localExitConsumers, consumer) } for _, consumer := range entry.monitorSubscribers { - if consumer.Node == tm.core.Name() { - localDownConsumers = append(localDownConsumers, consumer) - } else { + if consumer.Node != tm.core.Name() { remoteNodes[consumer.Node] = true + continue + } + localDownConsumers = append(localDownConsumers, consumer) + } + + // Cleanup relations + for key := range s.linkRelations { + if key.target == event { + delete(s.linkRelations, key) + } + } + + for key := range s.monitorRelations { + if key.target == event { + delete(s.monitorRelations, key) + } + } + + tm.eventIndex.Delete(entry.id) + delete(s.events, event) + delete(s.targetIndex, event) + + events := s.producerEvents[producer] + if events != nil { + delete(events, event) + if len(events) == 0 { + delete(s.producerEvents, producer) } } - // Send exit messages to local link subscribers + s.mutex.Unlock() + + // Dispatch without lock if len(localExitConsumers) > 0 { tm.core.RouteSendExitMessages( tm.core.PID(), @@ -95,7 +130,6 @@ func (tm *targetManager) UnregisterEvent(producer gen.PID, name gen.Atom) error ) } - // Send down messages to local monitor subscribers for _, consumer := range localDownConsumers { tm.core.RouteSendPID( tm.core.PID(), @@ -105,7 +139,6 @@ func (tm *targetManager) UnregisterEvent(producer gen.PID, name gen.Atom) error ) } - // Send to remote nodes for node := range remoteNodes { connection, err := tm.core.GetConnection(node) if err != nil { @@ -114,33 +147,6 @@ func (tm *targetManager) UnregisterEvent(producer gen.PID, name gen.Atom) error connection.SendTerminateEvent(event, gen.ErrUnregistered) } - // Cleanup event - delete(tm.events, event) - - // Cleanup producerEvents index - if events := tm.producerEvents[producer]; events != nil { - delete(events, event) - if len(events) == 0 { - delete(tm.producerEvents, producer) - } - } - - // Cleanup relations - for key := range tm.linkRelations { - if key.target == event { - delete(tm.linkRelations, key) - } - } - - for key := range tm.monitorRelations { - if key.target == event { - delete(tm.monitorRelations, key) - } - } - - // Cleanup targetIndex - delete(tm.targetIndex, event) - return nil } @@ -150,16 +156,14 @@ func (tm *targetManager) PublishEvent( options gen.MessageOptions, message gen.MessageEvent, ) error { - // Check if this is a LOCAL producer or REMOTE event if from.Node == tm.core.Name() { - // LOCAL producer - needs Lock (writes to buffer) return tm.publishEventLocalProducer(from, token, options, message) } - // REMOTE event - needs only RLock (read-only) - tm.mutex.RLock() - defer tm.mutex.RUnlock() - return tm.publishEventRemoteProducer(from, options, message) + s := tm.shardFor(message.Event) + s.mutex.RLock() + defer s.mutex.RUnlock() + return tm.publishEventRemoteProducer(s, from, options, message) } func (tm *targetManager) publishEventLocalProducer( @@ -168,49 +172,39 @@ func (tm *targetManager) publishEventLocalProducer( options gen.MessageOptions, message gen.MessageEvent, ) error { - tm.mutex.Lock() + s := tm.shardFor(message.Event) + s.mutex.RLock() - entry, exists := tm.events[message.Event] + entry, exists := s.events[message.Event] if exists == false { - tm.mutex.Unlock() + s.mutex.RUnlock() return gen.ErrEventUnknown } - // Validate token - if entry.token != token { - tm.mutex.Unlock() + if entry.open == false && entry.token != token { + s.mutex.RUnlock() return gen.ErrEventOwner } - // Store in buffer if configured (needs write lock) if entry.buffer != nil { - // Re-check after lock upgrade - entry, exists = tm.events[message.Event] - if exists == false || entry.token != token { - tm.mutex.Unlock() - return gen.ErrEventOwner - } - if len(entry.buffer) < entry.bufferSize { - entry.buffer = append(entry.buffer, message) - } else { - copy(entry.buffer, entry.buffer[1:]) - entry.buffer[entry.bufferSize-1] = message - } + entry.bufferMutex.Lock() + entry.buffer.push(message) + entry.bufferMutex.Unlock() } - // Copy slices under lock (minimize lock hold time) + entry.messagesPublished.Add(1) + entry.lastPublishedAt.Store(time.Now().UnixNano()) + + // Snapshot slices under RLock (safe: slices only modified under write lock) linkSubs := entry.linkSubscribers monitorSubs := entry.monitorSubscribers - tm.mutex.Unlock() + s.mutex.RUnlock() - // Increment published counter tm.eventsPublished.Add(1) - // Collect local consumers and remote nodes (no lock needed) var localConsumers []gen.PID remoteNodes := make(map[gen.Atom]bool) - // Fanout to link subscribers for _, consumer := range linkSubs { if consumer.Node != tm.core.Name() { remoteNodes[consumer.Node] = true @@ -219,7 +213,6 @@ func (tm *targetManager) publishEventLocalProducer( localConsumers = append(localConsumers, consumer) } - // Fanout to monitor subscribers for _, consumer := range monitorSubs { if consumer.Node != tm.core.Name() { remoteNodes[consumer.Node] = true @@ -228,72 +221,69 @@ func (tm *targetManager) publishEventLocalProducer( localConsumers = append(localConsumers, consumer) } - // Send to local consumers directly if len(localConsumers) > 0 { tm.core.RouteSendEventMessages(from, localConsumers, options, message) - tm.eventsSent.Add(int64(len(localConsumers))) + n := int64(len(localConsumers)) + entry.messagesLocalSent.Add(n) + tm.eventsLocalSent.Add(n) } - // Send to remote nodes for node := range remoteNodes { connection, err := tm.core.GetConnection(node) if err != nil { continue } connection.SendEvent(from, options, message) - tm.eventsSent.Add(1) + entry.messagesRemoteSent.Add(1) + tm.eventsRemoteSent.Add(1) } return nil } func (tm *targetManager) publishEventRemoteProducer( + s *shard, from gen.PID, options gen.MessageOptions, message gen.MessageEvent, ) error { - // Remote event arrived - deliver to local subscribers only - entry := tm.targetIndex[message.Event] + entry := s.targetIndex[message.Event] if entry == nil { return nil } - // Increment published counter - tm.eventsPublished.Add(1) + tm.eventsReceived.Add(1) - // Collect local consumers for batch delivery var localConsumers []gen.PID - for consumer := range entry.consumers { - if consumer.Node == tm.core.Name() { - localConsumers = append(localConsumers, consumer) + if consumer.Node != tm.core.Name() { + continue } + localConsumers = append(localConsumers, consumer) } if len(localConsumers) > 0 { tm.core.RouteSendEventMessages(from, localConsumers, options, message) - tm.eventsSent.Add(int64(len(localConsumers))) + tm.eventsLocalSent.Add(int64(len(localConsumers))) } return nil } func (tm *targetManager) LinkEvent(consumer gen.PID, event gen.Event) ([]gen.MessageEvent, error) { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(event) + s.mutex.Lock() + defer s.mutex.Unlock() - // Check if local or remote event if event.Node == tm.core.Name() { - // LOCAL event - return tm.linkEventLocal(consumer, event) + return tm.linkEventLocal(s, consumer, event) } - // REMOTE event - return tm.linkEventRemote(consumer, event) + return tm.linkEventRemote(s, consumer, event) } -func (tm *targetManager) linkEventLocal(consumer gen.PID, event gen.Event) ([]gen.MessageEvent, error) { - entry, exists := tm.events[event] +func (tm *targetManager) linkEventLocal(s *shard, consumer gen.PID, event gen.Event) ([]gen.MessageEvent, error) { + entry, exists := s.events[event] if exists == false { return nil, gen.ErrEventUnknown } @@ -303,26 +293,20 @@ func (tm *targetManager) linkEventLocal(consumer gen.PID, event gen.Event) ([]ge target: event, } - // Check duplicate - if _, exists := tm.linkRelations[key]; exists { - // For local event, duplicate from REMOTE CorePID is allowed - if consumer.Node != tm.core.Name() { - // Remote CorePID - return buffer anyway - return tm.getEventBuffer(entry), nil - } - + _, dup := s.linkRelations[key] + if dup == true && consumer.Node != tm.core.Name() { + return getEventBuffer(entry), nil + } + if dup == true { return nil, gen.ErrTargetExist } - // Add subscription - tm.linkRelations[key] = struct{}{} + s.linkRelations[key] = struct{}{} entry.linkSubscribersIndex[consumer] = len(entry.linkSubscribers) entry.linkSubscribers = append(entry.linkSubscribers, consumer) - // Increment counter entry.subscriberCount++ - // Check if need to send EventStart if entry.subscriberCount == 1 && entry.notify { tm.core.RouteSendPID( tm.core.PID(), @@ -332,38 +316,33 @@ func (tm *targetManager) linkEventLocal(consumer gen.PID, event gen.Event) ([]ge ) } - // Return buffer - return tm.getEventBuffer(entry), nil + return getEventBuffer(entry), nil } -func (tm *targetManager) linkEventRemote(consumer gen.PID, event gen.Event) ([]gen.MessageEvent, error) { +func (tm *targetManager) linkEventRemote(s *shard, consumer gen.PID, event gen.Event) ([]gen.MessageEvent, error) { key := relationKey{ consumer: consumer, target: event, } - if _, exists := tm.linkRelations[key]; exists { + if _, exists := s.linkRelations[key]; exists { return nil, gen.ErrTargetExist } - // Add subscription locally - tm.linkRelations[key] = struct{}{} + s.linkRelations[key] = struct{}{} - // Check targetIndex for remote request decision - entry := tm.targetIndex[event] + entry := s.targetIndex[event] needsRemote := false if entry == nil { - // First subscriber entry = &targetEntry{ - allowAlwaysFirst: true, // Start with true for buffered events + allowAlwaysFirst: true, consumers: make(map[gen.PID]struct{}), } - tm.targetIndex[event] = entry + s.targetIndex[event] = entry needsRemote = true } - // For events, allowAlwaysFirst stays true for buffered! if entry.allowAlwaysFirst == true { needsRemote = true } @@ -371,62 +350,50 @@ func (tm *targetManager) linkEventRemote(consumer gen.PID, event gen.Event) ([]g entry.consumers[consumer] = struct{}{} if needsRemote == false { - // Unbuffered event, not first - no remote request return nil, nil } - // Send remote LinkEvent connection, err := tm.core.GetConnection(event.Node) if err != nil { - // Rollback - delete(tm.linkRelations, key) + delete(s.linkRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, event) + delete(s.targetIndex, event) } - return nil, err } buffer, err := connection.LinkEvent(tm.core.PID(), event) if err != nil { - // Rollback - delete(tm.linkRelations, key) + delete(s.linkRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, event) + delete(s.targetIndex, event) } - return nil, err } - // Success! - // If buffer == nil: unbuffered, set allowAlwaysFirst=false - // If buffer != nil: buffered, keep allowAlwaysFirst=true if buffer == nil { entry.allowAlwaysFirst = false } - // else: buffered, keep allowAlwaysFirst=true for next subscribers! return buffer, nil } func (tm *targetManager) UnlinkEvent(consumer gen.PID, event gen.Event) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(event) + s.mutex.Lock() + defer s.mutex.Unlock() - // Check if local or remote if event.Node == tm.core.Name() { - return tm.unlinkEventLocal(consumer, event) + return tm.unlinkEventLocal(s, consumer, event) } - return tm.unlinkEventRemote(consumer, event) + return tm.unlinkEventRemote(s, consumer, event) } -func (tm *targetManager) unlinkEventLocal(consumer gen.PID, event gen.Event) error { - entry, exists := tm.events[event] +func (tm *targetManager) unlinkEventLocal(s *shard, consumer gen.PID, event gen.Event) error { + entry, exists := s.events[event] if exists == false { return gen.ErrEventUnknown } @@ -436,12 +403,11 @@ func (tm *targetManager) unlinkEventLocal(consumer gen.PID, event gen.Event) err target: event, } - if _, exists := tm.linkRelations[key]; exists == false { + if _, exists := s.linkRelations[key]; exists == false { return gen.ErrTargetUnknown } - // Remove subscription - delete(tm.linkRelations, key) + delete(s.linkRelations, key) // Swap-delete from slice idx := entry.linkSubscribersIndex[consumer] @@ -453,10 +419,8 @@ func (tm *targetManager) unlinkEventLocal(consumer gen.PID, event gen.Event) err entry.linkSubscribers = entry.linkSubscribers[:last] delete(entry.linkSubscribersIndex, consumer) - // Decrement counter entry.subscriberCount-- - // Send EventStop if last subscriber if entry.subscriberCount == 0 && entry.notify { tm.core.RouteSendPID( tm.core.PID(), @@ -469,19 +433,19 @@ func (tm *targetManager) unlinkEventLocal(consumer gen.PID, event gen.Event) err return nil } -func (tm *targetManager) unlinkEventRemote(consumer gen.PID, event gen.Event) error { +func (tm *targetManager) unlinkEventRemote(s *shard, consumer gen.PID, event gen.Event) error { key := relationKey{ consumer: consumer, target: event, } - if _, exists := tm.linkRelations[key]; exists == false { + if _, exists := s.linkRelations[key]; exists == false { return gen.ErrTargetUnknown } - delete(tm.linkRelations, key) + delete(s.linkRelations, key) - entry := tm.targetIndex[event] + entry := s.targetIndex[event] if entry == nil { return nil } @@ -489,12 +453,10 @@ func (tm *targetManager) unlinkEventRemote(consumer gen.PID, event gen.Event) er delete(entry.consumers, consumer) isLast := (len(entry.consumers) == 0) - if isLast { - delete(tm.targetIndex, event) + delete(s.targetIndex, event) } - // Check if need to send remote UnlinkEvent if isLast == false { hasLocal := false for pid := range entry.consumers { @@ -503,37 +465,34 @@ func (tm *targetManager) unlinkEventRemote(consumer gen.PID, event gen.Event) er break } } - if hasLocal { return nil } } - // Last local consumer - send remote UnlinkEvent connection, err := tm.core.GetConnection(event.Node) if err != nil { return nil } connection.UnlinkEvent(tm.core.PID(), event) - return nil } func (tm *targetManager) MonitorEvent(consumer gen.PID, event gen.Event) ([]gen.MessageEvent, error) { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(event) + s.mutex.Lock() + defer s.mutex.Unlock() - // Same logic as LinkEvent, but for monitors if event.Node == tm.core.Name() { - return tm.monitorEventLocal(consumer, event) + return tm.monitorEventLocal(s, consumer, event) } - return tm.monitorEventRemote(consumer, event) + return tm.monitorEventRemote(s, consumer, event) } -func (tm *targetManager) monitorEventLocal(consumer gen.PID, event gen.Event) ([]gen.MessageEvent, error) { - entry, exists := tm.events[event] +func (tm *targetManager) monitorEventLocal(s *shard, consumer gen.PID, event gen.Event) ([]gen.MessageEvent, error) { + entry, exists := s.events[event] if exists == false { return nil, gen.ErrEventUnknown } @@ -543,24 +502,20 @@ func (tm *targetManager) monitorEventLocal(consumer gen.PID, event gen.Event) ([ target: event, } - if _, exists := tm.monitorRelations[key]; exists { - if consumer.Node != tm.core.Name() { - // Remote CorePID duplicate - return buffer - return tm.getEventBuffer(entry), nil - } - + _, dup := s.monitorRelations[key] + if dup == true && consumer.Node != tm.core.Name() { + return getEventBuffer(entry), nil + } + if dup == true { return nil, gen.ErrTargetExist } - // Add subscription - tm.monitorRelations[key] = struct{}{} + s.monitorRelations[key] = struct{}{} entry.monitorSubscribersIndex[consumer] = len(entry.monitorSubscribers) entry.monitorSubscribers = append(entry.monitorSubscribers, consumer) - // Increment counter (shared with links!) entry.subscriberCount++ - // Send EventStart if first subscriber overall if entry.subscriberCount == 1 && entry.notify { tm.core.RouteSendPID( tm.core.PID(), @@ -570,23 +525,22 @@ func (tm *targetManager) monitorEventLocal(consumer gen.PID, event gen.Event) ([ ) } - // Return buffer - return tm.getEventBuffer(entry), nil + return getEventBuffer(entry), nil } -func (tm *targetManager) monitorEventRemote(consumer gen.PID, event gen.Event) ([]gen.MessageEvent, error) { +func (tm *targetManager) monitorEventRemote(s *shard, consumer gen.PID, event gen.Event) ([]gen.MessageEvent, error) { key := relationKey{ consumer: consumer, target: event, } - if _, exists := tm.monitorRelations[key]; exists { + if _, exists := s.monitorRelations[key]; exists { return nil, gen.ErrTargetExist } - tm.monitorRelations[key] = struct{}{} + s.monitorRelations[key] = struct{}{} - entry := tm.targetIndex[event] + entry := s.targetIndex[event] needsRemote := false if entry == nil { @@ -594,7 +548,7 @@ func (tm *targetManager) monitorEventRemote(consumer gen.PID, event gen.Event) ( allowAlwaysFirst: true, consumers: make(map[gen.PID]struct{}), } - tm.targetIndex[event] = entry + s.targetIndex[event] = entry needsRemote = true } @@ -610,29 +564,24 @@ func (tm *targetManager) monitorEventRemote(consumer gen.PID, event gen.Event) ( connection, err := tm.core.GetConnection(event.Node) if err != nil { - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, event) + delete(s.targetIndex, event) } - return nil, err } buffer, err := connection.MonitorEvent(tm.core.PID(), event) if err != nil { - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, event) + delete(s.targetIndex, event) } - return nil, err } - // Buffered vs unbuffered if buffer == nil { entry.allowAlwaysFirst = false } @@ -641,18 +590,19 @@ func (tm *targetManager) monitorEventRemote(consumer gen.PID, event gen.Event) ( } func (tm *targetManager) DemonitorEvent(consumer gen.PID, event gen.Event) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(event) + s.mutex.Lock() + defer s.mutex.Unlock() if event.Node == tm.core.Name() { - return tm.demonitorEventLocal(consumer, event) + return tm.demonitorEventLocal(s, consumer, event) } - return tm.demonitorEventRemote(consumer, event) + return tm.demonitorEventRemote(s, consumer, event) } -func (tm *targetManager) demonitorEventLocal(consumer gen.PID, event gen.Event) error { - entry, exists := tm.events[event] +func (tm *targetManager) demonitorEventLocal(s *shard, consumer gen.PID, event gen.Event) error { + entry, exists := s.events[event] if exists == false { return gen.ErrEventUnknown } @@ -662,11 +612,11 @@ func (tm *targetManager) demonitorEventLocal(consumer gen.PID, event gen.Event) target: event, } - if _, exists := tm.monitorRelations[key]; exists == false { + if _, exists := s.monitorRelations[key]; exists == false { return gen.ErrTargetUnknown } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) // Swap-delete from slice idx := entry.monitorSubscribersIndex[consumer] @@ -678,10 +628,8 @@ func (tm *targetManager) demonitorEventLocal(consumer gen.PID, event gen.Event) entry.monitorSubscribers = entry.monitorSubscribers[:last] delete(entry.monitorSubscribersIndex, consumer) - // Decrement counter (shared with links!) entry.subscriberCount-- - // Send EventStop if last if entry.subscriberCount == 0 && entry.notify { tm.core.RouteSendPID( tm.core.PID(), @@ -694,19 +642,19 @@ func (tm *targetManager) demonitorEventLocal(consumer gen.PID, event gen.Event) return nil } -func (tm *targetManager) demonitorEventRemote(consumer gen.PID, event gen.Event) error { +func (tm *targetManager) demonitorEventRemote(s *shard, consumer gen.PID, event gen.Event) error { key := relationKey{ consumer: consumer, target: event, } - if _, exists := tm.monitorRelations[key]; exists == false { + if _, exists := s.monitorRelations[key]; exists == false { return gen.ErrTargetUnknown } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) - entry := tm.targetIndex[event] + entry := s.targetIndex[event] if entry == nil { return nil } @@ -714,9 +662,8 @@ func (tm *targetManager) demonitorEventRemote(consumer gen.PID, event gen.Event) delete(entry.consumers, consumer) isLast := (len(entry.consumers) == 0) - if isLast { - delete(tm.targetIndex, event) + delete(s.targetIndex, event) } if isLast == false { @@ -727,7 +674,6 @@ func (tm *targetManager) demonitorEventRemote(consumer gen.PID, event gen.Event) break } } - if hasLocal { return nil } @@ -739,39 +685,163 @@ func (tm *targetManager) demonitorEventRemote(consumer gen.PID, event gen.Event) } connection.DemonitorEvent(tm.core.PID(), event) - return nil } func (tm *targetManager) EventInfo(event gen.Event) (gen.EventInfo, error) { - tm.mutex.RLock() - defer tm.mutex.RUnlock() + s := tm.shardFor(event) + s.mutex.RLock() + defer s.mutex.RUnlock() - entry, exists := tm.events[event] + entry, exists := s.events[event] if exists == false { return gen.EventInfo{}, gen.ErrEventUnknown } - // Build event info - info := gen.EventInfo{ - Producer: entry.producer, - BufferSize: entry.bufferSize, - CurrentBuffer: len(entry.buffer), - Notify: entry.notify, - Subscribers: entry.subscriberCount, + var bufSize, bufLen int + if entry.buffer != nil { + bufSize = entry.buffer.size + bufLen = entry.buffer.len + } + + return gen.EventInfo{ + CreatedAt: entry.createdAt, + Event: event, + Producer: entry.producer, + BufferSize: bufSize, + CurrentBuffer: bufLen, + Notify: entry.notify, + Open: entry.open, + Subscribers: entry.subscriberCount, + MessagesPublished: entry.messagesPublished.Load(), + MessagesLocalSent: entry.messagesLocalSent.Load(), + MessagesRemoteSent: entry.messagesRemoteSent.Load(), + LastPublishedAt: entry.lastPublishedAt.Load(), + }, nil +} + +func (tm *targetManager) EventRangeInfo(fn func(gen.EventInfo) bool) error { + var infos []gen.EventInfo + + for i := range tm.shards { + s := &tm.shards[i] + s.mutex.RLock() + for event, entry := range s.events { + var bufSize, bufLen int + if entry.buffer != nil { + bufSize = entry.buffer.size + bufLen = entry.buffer.len + } + infos = append(infos, gen.EventInfo{ + CreatedAt: entry.createdAt, + Event: event, + Producer: entry.producer, + BufferSize: bufSize, + CurrentBuffer: bufLen, + Notify: entry.notify, + Open: entry.open, + Subscribers: entry.subscriberCount, + MessagesPublished: entry.messagesPublished.Load(), + MessagesLocalSent: entry.messagesLocalSent.Load(), + MessagesRemoteSent: entry.messagesRemoteSent.Load(), + LastPublishedAt: entry.lastPublishedAt.Load(), + }) + } + s.mutex.RUnlock() + } + + for _, info := range infos { + if fn(info) == false { + break + } } - return info, nil + return nil } -// Helper: get event buffer -func (tm *targetManager) getEventBuffer(entry *eventEntry) []gen.MessageEvent { - if entry.buffer == nil { - return nil +func (tm *targetManager) EventListInfo(timestamp int64, limit int, filter ...func(gen.EventInfo) bool) ([]gen.EventInfo, error) { + maxID := tm.eventSeq.Load() + if maxID == 0 { + return nil, nil + } + + absLimit := limit + if absLimit < 0 { + absLimit = -absLimit + } + if absLimit == 0 { + return nil, nil + } + + var fn func(gen.EventInfo) bool + if len(filter) > 0 { + fn = filter[0] + } + + result := make([]gen.EventInfo, 0, absLimit) + + buildInfo := func(entry *eventEntry) gen.EventInfo { + var bufSize, bufLen int + if entry.buffer != nil { + bufSize = entry.buffer.size + bufLen = entry.buffer.len + } + return gen.EventInfo{ + CreatedAt: entry.createdAt, + Event: entry.event, + Producer: entry.producer, + BufferSize: bufSize, + CurrentBuffer: bufLen, + Notify: entry.notify, + Open: entry.open, + Subscribers: entry.subscriberCount, + MessagesPublished: entry.messagesPublished.Load(), + MessagesLocalSent: entry.messagesLocalSent.Load(), + MessagesRemoteSent: entry.messagesRemoteSent.Load(), + LastPublishedAt: entry.lastPublishedAt.Load(), + } + } + + accept := func(entry *eventEntry) bool { + if timestamp > 0 { + if limit >= 0 && entry.createdAt < timestamp { + return false + } + if limit < 0 && entry.createdAt > timestamp { + return false + } + } + if fn != nil { + return fn(buildInfo(entry)) + } + return true + } + + if timestamp == -1 || limit < 0 { + // backward: from newest + for id := maxID; id >= 1 && len(result) < absLimit; id-- { + v, ok := tm.eventIndex.Load(id) + if ok == false { + continue + } + entry := v.(*eventEntry) + if accept(entry) { + result = append(result, buildInfo(entry)) + } + } + } else { + // forward: from oldest + for id := uint64(1); id <= maxID && len(result) < absLimit; id++ { + v, ok := tm.eventIndex.Load(id) + if ok == false { + continue + } + entry := v.(*eventEntry) + if accept(entry) { + result = append(result, buildInfo(entry)) + } + } } - // Return copy of buffer - buffer := make([]gen.MessageEvent, len(entry.buffer)) - copy(buffer, entry.buffer) - return buffer + return result, nil } diff --git a/node/tm/event_test.go b/node/tm/event_test.go index 1937ae27c..b1556b640 100644 --- a/node/tm/event_test.go +++ b/node/tm/event_test.go @@ -26,7 +26,7 @@ func TestRegisterEvent_Basic(t *testing.T) { // Verify event stored event := gen.Event{Node: "node1", Name: "test"} - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry == nil { t.Fatal("Event should be stored") } @@ -35,8 +35,8 @@ func TestRegisterEvent_Basic(t *testing.T) { t.Error("Producer should match") } - if entry.bufferSize != 10 { - t.Errorf("Buffer size should be 10, got %d", entry.bufferSize) + if entry.buffer == nil || entry.buffer.size != 10 { + t.Errorf("Buffer size should be 10") } if entry.notify == false { @@ -99,7 +99,7 @@ func TestLinkEvent_Local_FirstSubscriber_EventStart(t *testing.T) { } // Verify stored - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry.subscriberCount != 1 { t.Errorf("subscriberCount should be 1, got %d", entry.subscriberCount) } @@ -133,7 +133,7 @@ func TestLinkEvent_Local_SecondSubscriber_NoEventStart(t *testing.T) { } // Counter = 2 - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry.subscriberCount != 2 { t.Errorf("subscriberCount should be 2, got %d", entry.subscriberCount) } @@ -170,7 +170,7 @@ func TestUnlinkEvent_Local_LastSubscriber_EventStop(t *testing.T) { } // Counter = 0 - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry.subscriberCount != 0 { t.Errorf("subscriberCount should be 0, got %d", entry.subscriberCount) } @@ -191,8 +191,7 @@ func TestLinkEvent_Remote_FirstSubscriber(t *testing.T) { } // Verify stored locally - key := relationKey{consumer: consumer, target: event} - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, event); exists == false { t.Error("Link should be stored locally") } @@ -202,7 +201,7 @@ func TestLinkEvent_Remote_FirstSubscriber(t *testing.T) { } // Verify allowAlwaysFirst=false (unbuffered) - entry := tm.targetIndex[event] + entry := tm.getTargetEntry(event) if entry.allowAlwaysFirst == true { t.Error("allowAlwaysFirst should be false for unbuffered event") } @@ -234,18 +233,16 @@ func TestLinkEvent_Remote_SecondSubscriber_Unbuffered_NoNetwork(t *testing.T) { } // Verify both links stored in linkRelations - key1 := relationKey{consumer: consumer1, target: event} - if _, exists := tm.linkRelations[key1]; exists == false { + if exists := tm.hasLinkRelation(consumer1, event); exists == false { t.Error("First link should be stored in linkRelations") } - key2 := relationKey{consumer: consumer2, target: event} - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, event); exists == false { t.Error("Second link should be stored in linkRelations") } // Verify targetIndex has both consumers - entry := tm.targetIndex[event] + entry := tm.getTargetEntry(event) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -295,7 +292,7 @@ func TestLinkEvent_Remote_SecondSubscriber_Buffered_SendsNetwork(t *testing.T) { } // allowAlwaysFirst still true! - entry := tm.targetIndex[event] + entry := tm.getTargetEntry(event) if entry.allowAlwaysFirst == false { t.Error("allowAlwaysFirst should stay true for buffered event") } @@ -318,17 +315,15 @@ func TestPublishEvent_Fanout(t *testing.T) { tm.LinkEvent(consumer2, event) // Verify both links stored - key1 := relationKey{consumer: consumer1, target: event} - if _, exists := tm.linkRelations[key1]; exists == false { + if exists := tm.hasLinkRelation(consumer1, event); exists == false { t.Error("consumer1 link should be stored") } - key2 := relationKey{consumer: consumer2, target: event} - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, event); exists == false { t.Error("consumer2 link should be stored") } // Verify subscriberCount - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry == nil { t.Fatal("Event entry should exist") } @@ -388,9 +383,9 @@ func TestPublishEvent_UpdatesBuffer(t *testing.T) { ) // Check buffer - entry := tm.events[event] - if len(entry.buffer) != 3 { - t.Errorf("Buffer should have 3 messages, got %d", len(entry.buffer)) + entry := tm.getEventEntry(event) + if entry.buffer.len != 3 { + t.Errorf("Buffer should have 3 messages, got %d", entry.buffer.len) } } @@ -426,17 +421,18 @@ func TestPublishEvent_BufferOverflow(t *testing.T) { ) // Buffer should have 2 messages (msg2, msg3 - msg1 flushed) - entry := tm.events[event] - if len(entry.buffer) != 2 { - t.Errorf("Buffer should have 2 messages, got %d", len(entry.buffer)) + entry := tm.getEventEntry(event) + if entry.buffer.len != 2 { + t.Errorf("Buffer should have 2 messages, got %d", entry.buffer.len) } - // msg2 and msg3 should be in buffer - if len(entry.buffer) >= 2 { - if entry.buffer[0].Message != "msg2" { + // msg2 and msg3 should be in buffer (oldest first) + snap := entry.buffer.snapshot() + if len(snap) >= 2 { + if snap[0].Message != "msg2" { t.Error("First message should be msg2 (msg1 flushed)") } - if entry.buffer[1].Message != "msg3" { + if snap[1].Message != "msg3" { t.Error("Second message should be msg3") } } @@ -459,17 +455,15 @@ func TestUnregisterEvent_SendsExit(t *testing.T) { tm.LinkEvent(consumer2, event) // Verify links stored before unregister - key1 := relationKey{consumer: consumer1, target: event} - key2 := relationKey{consumer: consumer2, target: event} - if _, exists := tm.linkRelations[key1]; exists == false { + if exists := tm.hasLinkRelation(consumer1, event); exists == false { t.Fatal("consumer1 link should exist before unregister") } - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, event); exists == false { t.Fatal("consumer2 link should exist before unregister") } // Verify producerEvents has producer - if _, exists := tm.producerEvents[producer]; exists == false { + if pe := tm.getProducerEventsMap(producer); pe == nil { t.Fatal("producerEvents should have producer before unregister") } @@ -489,20 +483,20 @@ func TestUnregisterEvent_SendsExit(t *testing.T) { } // Event removed - if _, exists := tm.events[event]; exists { + if entry := tm.getEventEntry(event); entry != nil { t.Error("Event should be removed") } // linkRelations cleaned - if _, exists := tm.linkRelations[key1]; exists { + if exists := tm.hasLinkRelation(consumer1, event); exists { t.Error("consumer1 link should be cleaned after unregister") } - if _, exists := tm.linkRelations[key2]; exists { + if exists := tm.hasLinkRelation(consumer2, event); exists { t.Error("consumer2 link should be cleaned after unregister") } // producerEvents cleaned - if _, exists := tm.producerEvents[producer]; exists { + if pe := tm.getProducerEventsMap(producer); pe != nil { t.Error("producerEvents should be cleaned after unregister") } } @@ -536,7 +530,7 @@ func TestMonitorEvent_Local_SharesCounter(t *testing.T) { } // Counter = 2 (link + monitor) - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry.subscriberCount != 2 { t.Errorf("Counter should be 2 (link+monitor), got %d", entry.subscriberCount) } @@ -559,17 +553,15 @@ func TestUnlinkDemonitor_EventStop_WhenBothGone(t *testing.T) { tm.MonitorEvent(monitorConsumer, event) // Verify both relations exist - linkKey := relationKey{consumer: linkConsumer, target: event} - monitorKey := relationKey{consumer: monitorConsumer, target: event} - if _, exists := tm.linkRelations[linkKey]; exists == false { + if exists := tm.hasLinkRelation(linkConsumer, event); exists == false { t.Fatal("Link should exist") } - if _, exists := tm.monitorRelations[monitorKey]; exists == false { + if exists := tm.hasMonitorRelation(monitorConsumer, event); exists == false { t.Fatal("Monitor should exist") } // Verify subscriberCount = 2 - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry.subscriberCount != 2 { t.Fatalf("subscriberCount should be 2, got %d", entry.subscriberCount) } @@ -586,10 +578,10 @@ func TestUnlinkDemonitor_EventStop_WhenBothGone(t *testing.T) { } // Verify link removed, monitor still exists - if _, exists := tm.linkRelations[linkKey]; exists { + if exists := tm.hasLinkRelation(linkConsumer, event); exists { t.Error("Link should be removed after unlink") } - if _, exists := tm.monitorRelations[monitorKey]; exists == false { + if exists := tm.hasMonitorRelation(monitorConsumer, event); exists == false { t.Error("Monitor should still exist") } @@ -608,10 +600,10 @@ func TestUnlinkDemonitor_EventStop_WhenBothGone(t *testing.T) { } // Verify both relations cleaned - if _, exists := tm.linkRelations[linkKey]; exists { + if exists := tm.hasLinkRelation(linkConsumer, event); exists { t.Error("Link should be removed") } - if _, exists := tm.monitorRelations[monitorKey]; exists { + if exists := tm.hasMonitorRelation(monitorConsumer, event); exists { t.Error("Monitor should be removed") } @@ -752,12 +744,10 @@ func TestUnlinkEvent_Remote_NotLast(t *testing.T) { tm.LinkEvent(consumer2, event) // Verify both stored - key1 := relationKey{consumer: consumer1, target: event} - key2 := relationKey{consumer: consumer2, target: event} - if _, exists := tm.linkRelations[key1]; exists == false { + if exists := tm.hasLinkRelation(consumer1, event); exists == false { t.Fatal("consumer1 link should exist before unlink") } - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, event); exists == false { t.Fatal("consumer2 link should exist before unlink") } @@ -768,17 +758,17 @@ func TestUnlinkEvent_Remote_NotLast(t *testing.T) { } // Verify removed from linkRelations - if _, exists := tm.linkRelations[key1]; exists { + if exists := tm.hasLinkRelation(consumer1, event); exists { t.Error("consumer1 link should be removed") } // consumer2 still exists - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, event); exists == false { t.Error("consumer2 link should still exist") } // Verify targetIndex still exists with consumer2 - entry := tm.targetIndex[event] + entry := tm.getTargetEntry(event) if entry == nil { t.Fatal("targetIndex should still exist") } @@ -807,12 +797,11 @@ func TestUnlinkEvent_Remote_LastLocal_SendsUnlink(t *testing.T) { } // Verify stored before unlink - key := relationKey{consumer: consumer, target: event} - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, event); exists == false { t.Fatal("Link should exist before unlink") } - entry := tm.targetIndex[event] + entry := tm.getTargetEntry(event) if entry == nil { t.Fatal("targetIndex should exist before unlink") } @@ -824,12 +813,12 @@ func TestUnlinkEvent_Remote_LastLocal_SendsUnlink(t *testing.T) { } // Verify removed from linkRelations - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, event); exists { t.Error("Link should be removed") } // Verify targetIndex cleaned (last consumer) - if _, exists := tm.targetIndex[event]; exists { + if entry := tm.getTargetEntry(event); entry != nil { t.Error("targetIndex should be cleaned when last consumer removed") } @@ -850,12 +839,10 @@ func TestDemonitorEvent_Remote_NotLast(t *testing.T) { tm.MonitorEvent(consumer2, event) // Verify both stored before demonitor - key1 := relationKey{consumer: consumer1, target: event} - key2 := relationKey{consumer: consumer2, target: event} - if _, exists := tm.monitorRelations[key1]; exists == false { + if exists := tm.hasMonitorRelation(consumer1, event); exists == false { t.Fatal("consumer1 monitor should exist before demonitor") } - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, event); exists == false { t.Fatal("consumer2 monitor should exist before demonitor") } @@ -866,17 +853,17 @@ func TestDemonitorEvent_Remote_NotLast(t *testing.T) { } // consumer1 removed - if _, exists := tm.monitorRelations[key1]; exists { + if exists := tm.hasMonitorRelation(consumer1, event); exists { t.Error("consumer1 monitor should be removed") } // consumer2 still exists - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, event); exists == false { t.Error("consumer2 monitor should still exist") } // Verify targetIndex still exists with consumer2 - entry := tm.targetIndex[event] + entry := tm.getTargetEntry(event) if entry == nil { t.Fatal("targetIndex should still exist") } @@ -899,12 +886,11 @@ func TestDemonitorEvent_Remote_LastLocal_SendsDemonitor(t *testing.T) { tm.MonitorEvent(consumer, event) // Verify stored before demonitor - key := relationKey{consumer: consumer, target: event} - if _, exists := tm.monitorRelations[key]; exists == false { + if exists := tm.hasMonitorRelation(consumer, event); exists == false { t.Fatal("Monitor should exist before demonitor") } - entry := tm.targetIndex[event] + entry := tm.getTargetEntry(event) if entry == nil { t.Fatal("targetIndex should exist before demonitor") } @@ -919,12 +905,12 @@ func TestDemonitorEvent_Remote_LastLocal_SendsDemonitor(t *testing.T) { } // Verify removed from monitorRelations - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, event); exists { t.Error("Monitor should be removed") } // Verify targetIndex cleaned (last consumer) - if _, exists := tm.targetIndex[event]; exists { + if entry := tm.getTargetEntry(event); entry != nil { t.Error("targetIndex should be cleaned when last consumer removed") } @@ -952,8 +938,7 @@ func TestUnlinkEvent_Remote_ConnectionError(t *testing.T) { } // Still cleaned locally - key := relationKey{consumer: consumer, target: event} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, event); exists { t.Error("Link should be cleaned even if connection fails") } } @@ -971,7 +956,7 @@ func TestMonitorEvent_Remote_Unbuffered_SecondNoNetwork(t *testing.T) { tm.MonitorEvent(consumer1, event) // allowAlwaysFirst should be false now (unbuffered) - entry := tm.targetIndex[event] + entry := tm.getTargetEntry(event) if entry == nil { t.Fatal("targetIndex should exist") } @@ -990,13 +975,11 @@ func TestMonitorEvent_Remote_Unbuffered_SecondNoNetwork(t *testing.T) { } // Verify both stored locally - key1 := relationKey{consumer: consumer1, target: event} - if _, exists := tm.monitorRelations[key1]; exists == false { + if exists := tm.hasMonitorRelation(consumer1, event); exists == false { t.Error("First monitor should exist") } - key2 := relationKey{consumer: consumer2, target: event} - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, event); exists == false { t.Error("Second monitor should exist") } @@ -1032,18 +1015,16 @@ func TestMonitorEvent_Remote_Buffered_SecondGetsBuffer(t *testing.T) { } // Verify both monitors stored in monitorRelations - key1 := relationKey{consumer: consumer1, target: event} - if _, exists := tm.monitorRelations[key1]; exists == false { + if exists := tm.hasMonitorRelation(consumer1, event); exists == false { t.Error("First monitor should be stored in monitorRelations") } - key2 := relationKey{consumer: consumer2, target: event} - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, event); exists == false { t.Error("Second monitor should be stored in monitorRelations") } // Verify targetIndex has both consumers - entry := tm.targetIndex[event] + entry := tm.getTargetEntry(event) if entry == nil { t.Fatal("targetIndex should exist") } @@ -1120,3 +1101,115 @@ func TestEventInfo_NotExist(t *testing.T) { t.Errorf("Expected ErrEventUnknown, got %v", err) } } + +// Test RegisterEvent - Open=true lets any publisher bypass token check +func TestPublishEvent_Open_BypassesToken(t *testing.T) { + core := newMockCore("node1") + tm := Create(core, Options{}).(*targetManager) + + producer := gen.PID{Node: "node1", ID: 100} + consumer := gen.PID{Node: "node1", ID: 101} + other := gen.PID{Node: "node1", ID: 102} + + _, err := tm.RegisterEvent(producer, "bus", gen.EventOptions{Open: true}) + if err != nil { + t.Fatalf("RegisterEvent failed: %v", err) + } + + event := gen.Event{Node: "node1", Name: "bus"} + if _, err := tm.LinkEvent(consumer, event); err != nil { + t.Fatalf("LinkEvent failed: %v", err) + } + + // Publish from producer with zero token: must succeed. + err = tm.PublishEvent(producer, gen.Ref{}, gen.MessageOptions{}, gen.MessageEvent{ + Event: event, + Timestamp: time.Now().UnixNano(), + Message: "from producer", + }) + if err != nil { + t.Errorf("producer publish with zero token should succeed, got %v", err) + } + + // Publish from another process with a wrong token: must also succeed. + err = tm.PublishEvent(other, gen.Ref{Node: "wrong"}, gen.MessageOptions{}, gen.MessageEvent{ + Event: event, + Timestamp: time.Now().UnixNano(), + Message: "from other", + }) + if err != nil { + t.Errorf("foreign publish with wrong token should succeed for open event, got %v", err) + } + + // Non-open event rejects wrong token. + strictProducer := gen.PID{Node: "node1", ID: 200} + _, err = tm.RegisterEvent(strictProducer, "strict", gen.EventOptions{}) + if err != nil { + t.Fatalf("RegisterEvent failed: %v", err) + } + strictEvent := gen.Event{Node: "node1", Name: "strict"} + err = tm.PublishEvent(strictProducer, gen.Ref{Node: "wrong"}, gen.MessageOptions{}, gen.MessageEvent{ + Event: strictEvent, + Timestamp: time.Now().UnixNano(), + Message: "bad token", + }) + if err != gen.ErrEventOwner { + t.Errorf("non-open event should reject wrong token: expected ErrEventOwner, got %v", err) + } +} + +// Test UnregisterEvent - Open=true does not relax the owner check +func TestUnregisterEvent_Open_StillRequiresOwner(t *testing.T) { + core := newMockCore("node1") + tm := Create(core, Options{}).(*targetManager) + + producer := gen.PID{Node: "node1", ID: 100} + other := gen.PID{Node: "node1", ID: 101} + + _, err := tm.RegisterEvent(producer, "bus", gen.EventOptions{Open: true}) + if err != nil { + t.Fatalf("RegisterEvent failed: %v", err) + } + + if err := tm.UnregisterEvent(other, "bus"); err != gen.ErrEventOwner { + t.Errorf("non-owner unregister on open event: expected ErrEventOwner, got %v", err) + } + + if err := tm.UnregisterEvent(producer, "bus"); err != nil { + t.Errorf("owner unregister on open event failed: %v", err) + } +} + +// Test RegisterEvent - Notify is ignored when the producer is the node core +func TestRegisterEvent_NotifyIgnoredForCorePID(t *testing.T) { + core := newMockCore("node1") + tm := Create(core, Options{}).(*targetManager) + + // Register as the node core with Notify=true. Notify must be silently + // suppressed since the node core does not consume MessageEventStart or + // MessageEventStop. + _, err := tm.RegisterEvent(core.PID(), "bus", gen.EventOptions{Notify: true}) + if err != nil { + t.Fatalf("RegisterEvent failed: %v", err) + } + + event := gen.Event{Node: "node1", Name: "bus"} + entry := tm.getEventEntry(event) + if entry == nil { + t.Fatal("event entry not found") + } + if entry.notify { + t.Error("entry.notify must be false for node-level events even when Notify=true was requested") + } + + // First subscriber should not produce a MessageEventStart send. + core.resetSentEventStarts() + consumer := gen.PID{Node: "node1", ID: 200} + if _, err := tm.LinkEvent(consumer, event); err != nil { + t.Fatalf("LinkEvent failed: %v", err) + } + time.Sleep(50 * time.Millisecond) + if n := core.countSentEventStarts(); n != 0 { + t.Errorf("expected 0 MessageEventStart for node-level event, got %d", n) + } +} diff --git a/node/tm/helpers_test.go b/node/tm/helpers_test.go new file mode 100644 index 000000000..3eba00673 --- /dev/null +++ b/node/tm/helpers_test.go @@ -0,0 +1,74 @@ +package tm + +import "ergo.services/ergo/gen" + +// Test helpers: aggregate views across shards + +func (tm *targetManager) totalLinks() int { + count := 0 + for i := range tm.shards { + count += len(tm.shards[i].linkRelations) + } + return count +} + +func (tm *targetManager) totalMonitors() int { + count := 0 + for i := range tm.shards { + count += len(tm.shards[i].monitorRelations) + } + return count +} + +func (tm *targetManager) totalEvents() int { + count := 0 + for i := range tm.shards { + count += len(tm.shards[i].events) + } + return count +} + +func (tm *targetManager) totalTargetIndex() int { + count := 0 + for i := range tm.shards { + count += len(tm.shards[i].targetIndex) + } + return count +} + +func (tm *targetManager) hasLinkRelation(consumer gen.PID, target any) bool { + s := tm.shardFor(target) + _, exists := s.linkRelations[relationKey{consumer: consumer, target: target}] + return exists +} + +func (tm *targetManager) hasMonitorRelation(consumer gen.PID, target any) bool { + s := tm.shardFor(target) + _, exists := s.monitorRelations[relationKey{consumer: consumer, target: target}] + return exists +} + +func (tm *targetManager) getTargetEntry(target any) *targetEntry { + s := tm.shardFor(target) + return s.targetIndex[target] +} + +func (tm *targetManager) getEventEntry(event gen.Event) *eventEntry { + s := tm.shardFor(event) + return s.events[event] +} + +func (tm *targetManager) getProducerEventsMap(producer gen.PID) map[gen.Event]struct{} { + result := make(map[gen.Event]struct{}) + for i := range tm.shards { + if pe := tm.shards[i].producerEvents[producer]; pe != nil { + for event := range pe { + result[event] = struct{}{} + } + } + } + if len(result) == 0 { + return nil + } + return result +} diff --git a/node/tm/manager.go b/node/tm/manager.go index c98d3628e..36f640ae9 100644 --- a/node/tm/manager.go +++ b/node/tm/manager.go @@ -3,24 +3,35 @@ package tm import ( "sync" "sync/atomic" + "time" "ergo.services/ergo/gen" + "ergo.services/ergo/lib" ) -// targetManager implements gen.TargetManager interface -type targetManager struct { - mutex sync.RWMutex +const defaultNumShards = 16 - core gen.CoreTargetManager +type shard struct { + mutex sync.RWMutex - // Link/Monitor relationships linkRelations map[relationKey]struct{} monitorRelations map[relationKey]struct{} targetIndex map[any]*targetEntry - // Event storage + // Events that hash to this shard events map[gen.Event]*eventEntry - producerEvents map[gen.PID]map[gen.Event]struct{} // producer -> events index + producerEvents map[gen.PID]map[gen.Event]struct{} +} + +// targetManager implements gen.TargetManager interface +type targetManager struct { + core gen.CoreTargetManager + shards []shard + numShards uint64 + + // Event ordering index: sequential ID -> *eventEntry + eventSeq atomic.Uint64 + eventIndex sync.Map // uint64 -> *eventEntry // Statistics exitSignalsProduced atomic.Int64 @@ -28,7 +39,9 @@ type targetManager struct { downMessagesProduced atomic.Int64 downMessagesDelivered atomic.Int64 eventsPublished atomic.Int64 - eventsSent atomic.Int64 + eventsReceived atomic.Int64 + eventsLocalSent atomic.Int64 + eventsRemoteSent atomic.Int64 } type relationKey struct { @@ -41,14 +54,51 @@ type targetEntry struct { consumers map[gen.PID]struct{} } +// eventRingBuffer is a fixed-size circular buffer for event messages. +// O(1) push, O(n) snapshot. No copy-shift on overflow. +type eventRingBuffer struct { + data []gen.MessageEvent + size int // capacity + head int // index of oldest element + len int // current number of elements +} + +func (rb *eventRingBuffer) push(msg gen.MessageEvent) { + if rb.len >= rb.size { + rb.data[rb.head] = msg + rb.head = (rb.head + 1) % rb.size + return + } + + idx := (rb.head + rb.len) % rb.size + rb.data[idx] = msg + rb.len++ +} + +func (rb *eventRingBuffer) snapshot() []gen.MessageEvent { + if rb.len == 0 { + return make([]gen.MessageEvent, 0) + } + result := make([]gen.MessageEvent, rb.len) + for i := 0; i < rb.len; i++ { + result[i] = rb.data[(rb.head+i)%rb.size] + } + return result +} + type eventEntry struct { + id uint64 + createdAt int64 + event gen.Event + producer gen.PID token gen.Ref notify bool + open bool - // Buffer (simple slice - protected by mutex) - buffer []gen.MessageEvent - bufferSize int + // Ring buffer (nil if unbuffered, protected by bufferMutex) + bufferMutex sync.Mutex + buffer *eventRingBuffer // Subscribers (links and monitors separately) // Slice for fast iteration, map for O(1) lookup/delete @@ -59,36 +109,97 @@ type eventEntry struct { monitorSubscribersIndex map[gen.PID]int subscriberCount int64 + + // Per-event statistics + messagesPublished atomic.Int64 + messagesLocalSent atomic.Int64 + messagesRemoteSent atomic.Int64 + lastPublishedAt atomic.Int64 // unix nanos of last SendEvent; 0 if never published } type Options struct{} func Create(core gen.CoreTargetManager, options Options) gen.TargetManager { + n := uint64(defaultNumShards) + tm := &targetManager{ - core: core, - linkRelations: make(map[relationKey]struct{}), - monitorRelations: make(map[relationKey]struct{}), - targetIndex: make(map[any]*targetEntry), - events: make(map[gen.Event]*eventEntry), - producerEvents: make(map[gen.PID]map[gen.Event]struct{}), + core: core, + shards: make([]shard, n), + numShards: n, + } + + for i := uint64(0); i < n; i++ { + tm.shards[i] = shard{ + linkRelations: make(map[relationKey]struct{}), + monitorRelations: make(map[relationKey]struct{}), + targetIndex: make(map[any]*targetEntry), + events: make(map[gen.Event]*eventEntry), + producerEvents: make(map[gen.PID]map[gen.Event]struct{}), + } } return tm } func (tm *targetManager) Info() gen.TargetManagerInfo { - tm.mutex.RLock() - defer tm.mutex.RUnlock() + var links, monitors, events int64 + + for i := range tm.shards { + s := &tm.shards[i] + s.mutex.RLock() + links += int64(len(s.linkRelations)) + monitors += int64(len(s.monitorRelations)) + events += int64(len(s.events)) + s.mutex.RUnlock() + } return gen.TargetManagerInfo{ - Links: int64(len(tm.linkRelations)), - Monitors: int64(len(tm.monitorRelations)), - Events: int64(len(tm.events)), + Links: links, + Monitors: monitors, + Events: events, ExitSignalsProduced: tm.exitSignalsProduced.Load(), ExitSignalsDelivered: tm.exitSignalsDelivered.Load(), DownMessagesProduced: tm.downMessagesProduced.Load(), DownMessagesDelivered: tm.downMessagesDelivered.Load(), EventsPublished: tm.eventsPublished.Load(), - EventsSent: tm.eventsSent.Load(), + EventsReceived: tm.eventsReceived.Load(), + EventsLocalSent: tm.eventsLocalSent.Load(), + EventsRemoteSent: tm.eventsRemoteSent.Load(), + } +} + +// shardFor returns the shard responsible for the given target. +// Uses bit masking (numShards must be power of 2). +func (tm *targetManager) shardFor(target any) *shard { + var idx uint64 + switch t := target.(type) { + case gen.PID: + idx = t.ID + case gen.Alias: + idx = t.ID[1] + case gen.ProcessID: + idx = lib.HashString64(string(t.Name)) + case gen.Event: + idx = lib.HashString64(string(t.Name)) + case gen.Atom: + idx = lib.HashString64(string(t)) + } + return &tm.shards[idx&(tm.numShards-1)] +} + +// getEventBuffer returns a snapshot of the event buffer, or nil. +func getEventBuffer(entry *eventEntry) []gen.MessageEvent { + if entry.buffer == nil { + return nil + } + return entry.buffer.snapshot() +} + +// generateToken creates a unique event token. +func (tm *targetManager) generateToken() gen.Ref { + return gen.Ref{ + Node: tm.core.Name(), + Creation: tm.core.PID().Creation, + ID: [3]uint64{uint64(time.Now().UnixNano()), 0, 0}, } } diff --git a/node/tm/node.go b/node/tm/node.go index aadadea26..808d597dd 100644 --- a/node/tm/node.go +++ b/node/tm/node.go @@ -5,28 +5,27 @@ import "ergo.services/ergo/gen" // Node operations (always local - connection monitoring) func (tm *targetManager) LinkNode(consumer gen.PID, target gen.Atom) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.linkRelations[key]; exists { + if _, exists := s.linkRelations[key]; exists { return gen.ErrTargetExist } - // Add to linkRelations - tm.linkRelations[key] = struct{}{} + s.linkRelations[key] = struct{}{} - // Add to targetIndex - entry := tm.targetIndex[target] + entry := s.targetIndex[target] if entry == nil { entry = &targetEntry{ consumers: make(map[gen.PID]struct{}), } - tm.targetIndex[target] = entry + s.targetIndex[target] = entry } entry.consumers[consumer] = struct{}{} @@ -34,55 +33,56 @@ func (tm *targetManager) LinkNode(consumer gen.PID, target gen.Atom) error { } func (tm *targetManager) UnlinkNode(consumer gen.PID, target gen.Atom) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.linkRelations[key]; exists == false { + if _, exists := s.linkRelations[key]; exists == false { return nil } - delete(tm.linkRelations, key) + delete(s.linkRelations, key) - entry := tm.targetIndex[target] - if entry != nil { - delete(entry.consumers, consumer) + entry := s.targetIndex[target] + if entry == nil { + return nil + } - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) - } + delete(entry.consumers, consumer) + if len(entry.consumers) == 0 { + delete(s.targetIndex, target) } return nil } func (tm *targetManager) MonitorNode(consumer gen.PID, target gen.Atom) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.monitorRelations[key]; exists { + if _, exists := s.monitorRelations[key]; exists { return gen.ErrTargetExist } - // Add to monitorRelations (always local - no remote for Node targets) - tm.monitorRelations[key] = struct{}{} + s.monitorRelations[key] = struct{}{} - // Add to targetIndex - entry := tm.targetIndex[target] + entry := s.targetIndex[target] if entry == nil { entry = &targetEntry{ consumers: make(map[gen.PID]struct{}), } - tm.targetIndex[target] = entry + s.targetIndex[target] = entry } entry.consumers[consumer] = struct{}{} @@ -90,27 +90,29 @@ func (tm *targetManager) MonitorNode(consumer gen.PID, target gen.Atom) error { } func (tm *targetManager) DemonitorNode(consumer gen.PID, target gen.Atom) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.monitorRelations[key]; exists == false { + if _, exists := s.monitorRelations[key]; exists == false { return nil } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) - entry := tm.targetIndex[target] - if entry != nil { - delete(entry.consumers, consumer) + entry := s.targetIndex[target] + if entry == nil { + return nil + } - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) - } + delete(entry.consumers, consumer) + if len(entry.consumers) == 0 { + delete(s.targetIndex, target) } return nil diff --git a/node/tm/node_test.go b/node/tm/node_test.go index a8da80a28..dafa845fd 100644 --- a/node/tm/node_test.go +++ b/node/tm/node_test.go @@ -21,13 +21,12 @@ func TestLinkNode_Basic(t *testing.T) { } // Stored - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, target); exists == false { t.Error("Node link should be stored") } // Verify in targetIndex - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetEntry should be created") } @@ -52,24 +51,17 @@ func TestLinkNode_Duplicate_Error(t *testing.T) { } // Verify first link still stored (not corrupted by duplicate attempt) - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, target); exists == false { t.Error("Original link should still be stored") } - // Verify only 1 link in linkRelations for this consumer - count := 0 - for k := range tm.linkRelations { - if k.consumer == consumer { - count++ - } - } - if count != 1 { - t.Errorf("Expected 1 link for consumer, got %d", count) + // Verify only 1 link total (duplicate was rejected) + if tm.totalLinks() != 1 { + t.Errorf("Expected 1 link total, got %d", tm.totalLinks()) } // Verify targetIndex has exactly 1 consumer - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex should exist") } @@ -103,7 +95,7 @@ func TestLinkNode_MultipleConsumers(t *testing.T) { } // Verify targetIndex has all consumers - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if len(entry.consumers) != 3 { t.Errorf("Expected 3 consumers, got %d", len(entry.consumers)) } @@ -126,13 +118,12 @@ func TestUnlinkNode_Basic(t *testing.T) { } // Removed - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("Node link should be removed") } // targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned") } } @@ -151,19 +142,17 @@ func TestUnlinkNode_NotLast(t *testing.T) { tm.UnlinkNode(consumer1, target) // consumer1 removed - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.linkRelations[key1]; exists { + if exists := tm.hasLinkRelation(consumer1, target); exists { t.Error("consumer1 link should be removed") } // consumer2 still exists - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, target); exists == false { t.Error("consumer2 link should still exist") } // targetIndex still exists with consumer2 - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex should still exist") } @@ -203,13 +192,12 @@ func TestMonitorNode_Basic(t *testing.T) { } // Stored in monitorRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists == false { + if exists := tm.hasMonitorRelation(consumer, target); exists == false { t.Error("Node monitor should be stored") } // Verify in targetIndex - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetEntry should be created") } @@ -238,24 +226,17 @@ func TestMonitorNode_Duplicate_Error(t *testing.T) { } // Verify first monitor still stored (not corrupted by duplicate attempt) - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists == false { + if exists := tm.hasMonitorRelation(consumer, target); exists == false { t.Error("Original monitor should still be stored") } - // Verify only 1 monitor in monitorRelations for this consumer - count := 0 - for k := range tm.monitorRelations { - if k.consumer == consumer { - count++ - } - } - if count != 1 { - t.Errorf("Expected 1 monitor for consumer, got %d", count) + // Verify only 1 monitor total (duplicate was rejected) + if tm.totalMonitors() != 1 { + t.Errorf("Expected 1 monitor total, got %d", tm.totalMonitors()) } // Verify targetIndex has exactly 1 consumer - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex should exist") } @@ -284,17 +265,15 @@ func TestMonitorNode_MultipleConsumers(t *testing.T) { } // Verify both stored in monitorRelations - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.monitorRelations[key1]; exists == false { + if exists := tm.hasMonitorRelation(consumer1, target); exists == false { t.Error("consumer1 should be in monitorRelations") } - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, target); exists == false { t.Error("consumer2 should be in monitorRelations") } // Verify targetIndex has both consumers - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex should exist") } @@ -320,13 +299,12 @@ func TestDemonitorNode_Basic(t *testing.T) { } // Removed - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, target); exists { t.Error("Node monitor should be removed") } // targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned") } } @@ -345,19 +323,17 @@ func TestDemonitorNode_NotLast(t *testing.T) { tm.DemonitorNode(consumer1, target) // consumer1 removed - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.monitorRelations[key1]; exists { + if exists := tm.hasMonitorRelation(consumer1, target); exists { t.Error("consumer1 monitor should be removed") } // consumer2 still exists - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, target); exists == false { t.Error("consumer2 monitor should still exist") } // targetIndex still exists with consumer2 - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex should still exist") } diff --git a/node/tm/pid.go b/node/tm/pid.go index 5a6f16edf..404dae880 100644 --- a/node/tm/pid.go +++ b/node/tm/pid.go @@ -3,137 +3,108 @@ package tm import "ergo.services/ergo/gen" func (tm *targetManager) LinkPID(consumer gen.PID, target gen.PID) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - // Check if already exists - if _, exists := tm.linkRelations[key]; exists { - // Special case: remote PID linking to local target (concurrent CorePID) - if consumer.Node != tm.core.Name() { - // Remote consumer - ignore duplicate (CorePID optimization) - return nil - } - + _, exists := s.linkRelations[key] + if exists == true && consumer.Node != tm.core.Name() { + return nil + } + if exists == true { return gen.ErrTargetExist } - // Add to linkRelations - tm.linkRelations[key] = struct{}{} + s.linkRelations[key] = struct{}{} - // Check targetIndex for remote request decision - entry := tm.targetIndex[target] + entry := s.targetIndex[target] needsRemote := false if entry == nil { - // First subscriber overall - create entry entry = &targetEntry{ allowAlwaysFirst: true, consumers: make(map[gen.PID]struct{}), } - tm.targetIndex[target] = entry + s.targetIndex[target] = entry needsRemote = true } - // Check if allowAlwaysFirst permits remote request if entry.allowAlwaysFirst == true { needsRemote = true } - // Add consumer to entry entry.consumers[consumer] = struct{}{} - // Check if target is local if target.Node == tm.core.Name() { - // Local target - no remote request needed return nil } - // Remote target if needsRemote == false { - // Not allowed to send remote (someone already succeeded) return nil } - // Send remote LinkPID with CorePID connection, err := tm.core.GetConnection(target.Node) if err != nil { - // Network error - rollback - delete(tm.linkRelations, key) + delete(s.linkRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } err = connection.LinkPID(tm.core.PID(), target) if err != nil { - // Remote LinkPID failed - rollback - delete(tm.linkRelations, key) + delete(s.linkRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } - // Success! Set allowAlwaysFirst=false to prevent future remote requests entry.allowAlwaysFirst = false - return nil } func (tm *targetManager) UnlinkPID(consumer gen.PID, target gen.PID) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - // Check if exists - if _, exists := tm.linkRelations[key]; exists == false { - // Idempotent - not an error if already removed + if _, exists := s.linkRelations[key]; exists == false { return nil } - // Remove from linkRelations - delete(tm.linkRelations, key) + delete(s.linkRelations, key) - // Remove from targetIndex - entry := tm.targetIndex[target] + entry := s.targetIndex[target] if entry == nil { return nil } delete(entry.consumers, consumer) - // Check if last consumer isLast := (len(entry.consumers) == 0) - if isLast { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - // Check if target is local if target.Node == tm.core.Name() { - // Local target - no remote request needed return nil } - // Remote target - check if last LOCAL consumer if isLast == false { - // Other consumers still exist - check if any are local hasLocal := false for pid := range entry.consumers { if pid.Node == tm.core.Name() && pid != tm.core.PID() { @@ -141,51 +112,41 @@ func (tm *targetManager) UnlinkPID(consumer gen.PID, target gen.PID) error { break } } - if hasLocal { - // Other local consumers exist - don't send UnlinkPID return nil } } - // Last local consumer (or isLast overall) - send remote UnlinkPID connection, err := tm.core.GetConnection(target.Node) if err != nil { - // Connection lost - not an error, remote will cleanup via RouteNodeDown return nil } - // Send UnlinkPID with CorePID (ignore errors - best effort) connection.UnlinkPID(tm.core.PID(), target) - return nil } func (tm *targetManager) MonitorPID(consumer gen.PID, target gen.PID) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - // Check if already exists - if _, exists := tm.monitorRelations[key]; exists { - // Special case: remote PID monitoring local target - if consumer.Node != tm.core.Name() { - // Remote consumer - ignore duplicate - return nil - } - + _, exists := s.monitorRelations[key] + if exists == true && consumer.Node != tm.core.Name() { + return nil + } + if exists == true { return gen.ErrTargetExist } - // Add to monitorRelations - tm.monitorRelations[key] = struct{}{} + s.monitorRelations[key] = struct{}{} - // Check targetIndex for remote request decision - entry := tm.targetIndex[target] + entry := s.targetIndex[target] needsRemote := false if entry == nil { @@ -193,7 +154,7 @@ func (tm *targetManager) MonitorPID(consumer gen.PID, target gen.PID) error { allowAlwaysFirst: true, consumers: make(map[gen.PID]struct{}), } - tm.targetIndex[target] = entry + s.targetIndex[target] = entry needsRemote = true } @@ -203,65 +164,55 @@ func (tm *targetManager) MonitorPID(consumer gen.PID, target gen.PID) error { entry.consumers[consumer] = struct{}{} - // Check if target is local if target.Node == tm.core.Name() { return nil } - // Remote target if needsRemote == false { return nil } - // Send remote MonitorPID connection, err := tm.core.GetConnection(target.Node) if err != nil { - // Rollback - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } err = connection.MonitorPID(tm.core.PID(), target) if err != nil { - // Rollback - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } - // Success entry.allowAlwaysFirst = false - return nil } func (tm *targetManager) DemonitorPID(consumer gen.PID, target gen.PID) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.monitorRelations[key]; exists == false { + if _, exists := s.monitorRelations[key]; exists == false { return nil } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) - entry := tm.targetIndex[target] + entry := s.targetIndex[target] if entry == nil { return nil } @@ -269,9 +220,8 @@ func (tm *targetManager) DemonitorPID(consumer gen.PID, target gen.PID) error { delete(entry.consumers, consumer) isLast := (len(entry.consumers) == 0) - if isLast { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } if target.Node == tm.core.Name() { @@ -286,7 +236,6 @@ func (tm *targetManager) DemonitorPID(consumer gen.PID, target gen.PID) error { break } } - if hasLocal { return nil } @@ -298,6 +247,5 @@ func (tm *targetManager) DemonitorPID(consumer gen.PID, target gen.PID) error { } connection.DemonitorPID(tm.core.PID(), target) - return nil } diff --git a/node/tm/pid_test.go b/node/tm/pid_test.go index 5873f0267..7f8a63e70 100644 --- a/node/tm/pid_test.go +++ b/node/tm/pid_test.go @@ -20,12 +20,11 @@ func TestLinkPID_Local_Basic(t *testing.T) { t.Fatalf("LinkPID failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists == false { + if tm.hasLinkRelation(consumer, target) == false { t.Error("Link should be stored in linkRelations") } - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetEntry should be created") } @@ -51,8 +50,7 @@ func TestLinkPID_Remote_FirstSubscriber(t *testing.T) { t.Fatalf("LinkPID failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists == false { + if tm.hasLinkRelation(consumer, target) == false { t.Error("Link should be stored locally") } @@ -71,7 +69,7 @@ func TestLinkPID_Remote_FirstSubscriber(t *testing.T) { } } - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry.allowAlwaysFirst == true { t.Error("allowAlwaysFirst should be false after successful remote link") } @@ -101,8 +99,7 @@ func TestLinkPID_Remote_SecondSubscriber_NoNetwork(t *testing.T) { t.Fatalf("Second LinkPID failed: %v", err) } - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.linkRelations[key2]; exists == false { + if tm.hasLinkRelation(consumer2, target) == false { t.Error("Second link should be stored locally") } @@ -110,7 +107,7 @@ func TestLinkPID_Remote_SecondSubscriber_NoNetwork(t *testing.T) { t.Errorf("Second subscriber should NOT send network request (CorePID optimization), got %d", core.countSentLinks()) } - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if len(entry.consumers) != 2 { t.Errorf("Expected 2 consumers in targetIndex, got %d", len(entry.consumers)) } @@ -133,8 +130,8 @@ func TestLinkPID_Remote_ThreeSubscribers_OneNetworkRequest(t *testing.T) { t.Errorf("Expected exactly 1 network request for 3 subscribers, got %d", core.countSentLinks()) } - if len(tm.linkRelations) != 3 { - t.Errorf("Expected 3 links stored locally, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 3 { + t.Errorf("Expected 3 links stored locally, got %d", tm.totalLinks()) } if sent, ok := core.getFirstSentLink(); ok { @@ -177,12 +174,11 @@ func TestLinkPID_NetworkError_Rollback(t *testing.T) { t.Errorf("Expected ErrNoConnection, got %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if tm.hasLinkRelation(consumer, target) { t.Error("Link should be rolled back after network error") } - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned after rollback") } } @@ -202,8 +198,7 @@ func TestLinkPID_RemoteError_Rollback(t *testing.T) { t.Errorf("Expected ErrProcessUnknown, got %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if tm.hasLinkRelation(consumer, target) { t.Error("Link should be rolled back after remote error") } } @@ -226,12 +221,12 @@ func TestLinkPID_RemoteCorePID_Duplicate_Ignored(t *testing.T) { } // Verify only ONE relation exists (duplicate was ignored, not added twice) - if len(tm.linkRelations) != 1 { - t.Errorf("Expected 1 link relation, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 1 { + t.Errorf("Expected 1 link relation, got %d", tm.totalLinks()) } // Verify targetIndex has only one consumer - entry := tm.targetIndex[localTarget] + entry := tm.getTargetEntry(localTarget) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -261,19 +256,17 @@ func TestUnlinkPID_NotLastLocal(t *testing.T) { } // Verify consumer1 link removed from linkRelations - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.linkRelations[key1]; exists { + if tm.hasLinkRelation(consumer1, target) { t.Error("consumer1 link should be removed from linkRelations") } // Verify consumer2 link still exists in linkRelations - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.linkRelations[key2]; exists == false { + if tm.hasLinkRelation(consumer2, target) == false { t.Error("consumer2 link should still exist in linkRelations") } // Verify targetIndex still exists with only consumer2 - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should still exist") } @@ -306,12 +299,11 @@ func TestUnlinkPID_LastLocal_SendsUnlink(t *testing.T) { t.Fatalf("UnlinkPID failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if tm.hasLinkRelation(consumer, target) { t.Error("Link should be removed") } - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned when last consumer removed") } @@ -344,8 +336,7 @@ func TestMonitorPID_Local_Basic(t *testing.T) { t.Fatalf("MonitorPID failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists == false { + if tm.hasMonitorRelation(consumer, target) == false { t.Error("Monitor should be stored") } @@ -411,12 +402,12 @@ func TestMonitorPID_Remote_ThreeSubscribers(t *testing.T) { } // Verify all 3 monitors stored in monitorRelations - if len(tm.monitorRelations) != 3 { - t.Errorf("Expected 3 monitor relations, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 3 { + t.Errorf("Expected 3 monitor relations, got %d", tm.totalMonitors()) } // Verify targetIndex has all 3 consumers - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -462,8 +453,7 @@ func TestMonitorPID_NetworkError_Rollback(t *testing.T) { t.Errorf("Expected ErrNoConnection, got %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if tm.hasMonitorRelation(consumer, target) { t.Error("Monitor should be rolled back") } } @@ -483,8 +473,7 @@ func TestMonitorPID_RemoteError_Rollback(t *testing.T) { t.Errorf("Expected ErrProcessUnknown, got %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if tm.hasMonitorRelation(consumer, target) { t.Error("Monitor should be rolled back") } } @@ -507,12 +496,12 @@ func TestMonitorPID_RemoteCorePID_Duplicate(t *testing.T) { } // Verify only ONE relation exists (duplicate was ignored, not added twice) - if len(tm.monitorRelations) != 1 { - t.Errorf("Expected 1 monitor relation, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 1 { + t.Errorf("Expected 1 monitor relation, got %d", tm.totalMonitors()) } // Verify targetIndex has only one consumer - entry := tm.targetIndex[localTarget] + entry := tm.getTargetEntry(localTarget) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -535,8 +524,7 @@ func TestMonitorPID_ReSubscribe(t *testing.T) { } // Verify monitor stored - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists == false { + if tm.hasMonitorRelation(consumer, target) == false { t.Error("Monitor should be stored after first MonitorPID") } @@ -545,10 +533,10 @@ func TestMonitorPID_ReSubscribe(t *testing.T) { tm.DemonitorPID(consumer, target) // Verify monitor removed after demonitor - if _, exists := tm.monitorRelations[key]; exists { + if tm.hasMonitorRelation(consumer, target) { t.Error("Monitor should be removed after DemonitorPID") } - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned after DemonitorPID") } @@ -563,12 +551,12 @@ func TestMonitorPID_ReSubscribe(t *testing.T) { } // Verify monitor stored again - if _, exists := tm.monitorRelations[key]; exists == false { + if tm.hasMonitorRelation(consumer, target) == false { t.Error("Monitor should be stored after re-subscribe") } // Verify targetIndex recreated - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should be recreated after re-subscribe") } @@ -595,19 +583,17 @@ func TestDemonitorPID_NotLast(t *testing.T) { tm.DemonitorPID(consumer1, target) // Verify consumer1 monitor removed from monitorRelations - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.monitorRelations[key1]; exists { + if tm.hasMonitorRelation(consumer1, target) { t.Error("consumer1 monitor should be removed from monitorRelations") } // Verify consumer2 monitor still exists in monitorRelations - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.monitorRelations[key2]; exists == false { + if tm.hasMonitorRelation(consumer2, target) == false { t.Error("consumer2 monitor should still exist in monitorRelations") } // Verify targetIndex still exists with only consumer2 - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should still exist") } diff --git a/node/tm/process_id.go b/node/tm/process_id.go index e766ecd86..a05537d28 100644 --- a/node/tm/process_id.go +++ b/node/tm/process_id.go @@ -3,25 +3,26 @@ package tm import "ergo.services/ergo/gen" func (tm *targetManager) LinkProcessID(consumer gen.PID, target gen.ProcessID) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.linkRelations[key]; exists { - if consumer.Node != tm.core.Name() { - return nil - } - + _, exists := s.linkRelations[key] + if exists == true && consumer.Node != tm.core.Name() { + return nil + } + if exists == true { return gen.ErrTargetExist } - tm.linkRelations[key] = struct{}{} + s.linkRelations[key] = struct{}{} - entry := tm.targetIndex[target] + entry := s.targetIndex[target] needsRemote := false if entry == nil { @@ -29,7 +30,7 @@ func (tm *targetManager) LinkProcessID(consumer gen.PID, target gen.ProcessID) e allowAlwaysFirst: true, consumers: make(map[gen.PID]struct{}), } - tm.targetIndex[target] = entry + s.targetIndex[target] = entry needsRemote = true } @@ -39,7 +40,6 @@ func (tm *targetManager) LinkProcessID(consumer gen.PID, target gen.ProcessID) e entry.consumers[consumer] = struct{}{} - // Check if target is local targetNode := target.Node if targetNode == "" { targetNode = tm.core.Name() @@ -55,49 +55,45 @@ func (tm *targetManager) LinkProcessID(consumer gen.PID, target gen.ProcessID) e connection, err := tm.core.GetConnection(targetNode) if err != nil { - delete(tm.linkRelations, key) + delete(s.linkRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } err = connection.LinkProcessID(tm.core.PID(), target) if err != nil { - delete(tm.linkRelations, key) + delete(s.linkRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } entry.allowAlwaysFirst = false - return nil } func (tm *targetManager) UnlinkProcessID(consumer gen.PID, target gen.ProcessID) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.linkRelations[key]; exists == false { + if _, exists := s.linkRelations[key]; exists == false { return nil } - delete(tm.linkRelations, key) + delete(s.linkRelations, key) - entry := tm.targetIndex[target] + entry := s.targetIndex[target] if entry == nil { return nil } @@ -105,9 +101,8 @@ func (tm *targetManager) UnlinkProcessID(consumer gen.PID, target gen.ProcessID) delete(entry.consumers, consumer) isLast := (len(entry.consumers) == 0) - if isLast { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } targetNode := target.Node @@ -127,7 +122,6 @@ func (tm *targetManager) UnlinkProcessID(consumer gen.PID, target gen.ProcessID) break } } - if hasLocal { return nil } @@ -139,30 +133,30 @@ func (tm *targetManager) UnlinkProcessID(consumer gen.PID, target gen.ProcessID) } connection.UnlinkProcessID(tm.core.PID(), target) - return nil } func (tm *targetManager) MonitorProcessID(consumer gen.PID, target gen.ProcessID) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.monitorRelations[key]; exists { - if consumer.Node != tm.core.Name() { - return nil - } - + _, exists := s.monitorRelations[key] + if exists == true && consumer.Node != tm.core.Name() { + return nil + } + if exists == true { return gen.ErrTargetExist } - tm.monitorRelations[key] = struct{}{} + s.monitorRelations[key] = struct{}{} - entry := tm.targetIndex[target] + entry := s.targetIndex[target] needsRemote := false if entry == nil { @@ -170,7 +164,7 @@ func (tm *targetManager) MonitorProcessID(consumer gen.PID, target gen.ProcessID allowAlwaysFirst: true, consumers: make(map[gen.PID]struct{}), } - tm.targetIndex[target] = entry + s.targetIndex[target] = entry needsRemote = true } @@ -195,49 +189,45 @@ func (tm *targetManager) MonitorProcessID(consumer gen.PID, target gen.ProcessID connection, err := tm.core.GetConnection(targetNode) if err != nil { - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } err = connection.MonitorProcessID(tm.core.PID(), target) if err != nil { - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) delete(entry.consumers, consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } - return err } entry.allowAlwaysFirst = false - return nil } func (tm *targetManager) DemonitorProcessID(consumer gen.PID, target gen.ProcessID) error { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.Lock() + defer s.mutex.Unlock() key := relationKey{ consumer: consumer, target: target, } - if _, exists := tm.monitorRelations[key]; exists == false { + if _, exists := s.monitorRelations[key]; exists == false { return nil } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) - entry := tm.targetIndex[target] + entry := s.targetIndex[target] if entry == nil { return nil } @@ -245,9 +235,8 @@ func (tm *targetManager) DemonitorProcessID(consumer gen.PID, target gen.Process delete(entry.consumers, consumer) isLast := (len(entry.consumers) == 0) - if isLast { - delete(tm.targetIndex, target) + delete(s.targetIndex, target) } targetNode := target.Node @@ -267,7 +256,6 @@ func (tm *targetManager) DemonitorProcessID(consumer gen.PID, target gen.Process break } } - if hasLocal { return nil } @@ -279,6 +267,5 @@ func (tm *targetManager) DemonitorProcessID(consumer gen.PID, target gen.Process } connection.DemonitorProcessID(tm.core.PID(), target) - return nil } diff --git a/node/tm/process_id_test.go b/node/tm/process_id_test.go index d49e20164..b492a1cc6 100644 --- a/node/tm/process_id_test.go +++ b/node/tm/process_id_test.go @@ -20,8 +20,7 @@ func TestLinkProcessID_Local(t *testing.T) { t.Fatalf("LinkProcessID failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, target); exists == false { t.Error("Link should be stored") } @@ -42,8 +41,7 @@ func TestLinkProcessID_Local_EmptyNode(t *testing.T) { t.Fatalf("LinkProcessID failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, target); exists == false { t.Error("Link should be stored") } @@ -65,13 +63,12 @@ func TestLinkProcessID_Remote_First(t *testing.T) { } // Verify link stored in linkRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, target); exists == false { t.Error("Link should be stored in linkRelations") } // Verify targetIndex created - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should be created") } @@ -114,20 +111,18 @@ func TestLinkProcessID_Remote_Second(t *testing.T) { } // Verify both links stored in linkRelations - if len(tm.linkRelations) != 2 { - t.Errorf("Expected 2 link relations, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 2 { + t.Errorf("Expected 2 link relations, got %d", tm.totalLinks()) } - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.linkRelations[key1]; exists == false { + if exists := tm.hasLinkRelation(consumer1, target); exists == false { t.Error("consumer1 link should exist in linkRelations") } - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, target); exists == false { t.Error("consumer2 link should exist in linkRelations") } // Verify targetIndex has both consumers - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -167,13 +162,12 @@ func TestLinkProcessID_NetworkError_Rollback(t *testing.T) { } // Verify link rolled back from linkRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("Link should be rolled back from linkRelations") } // Verify targetIndex cleaned after rollback - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned after rollback") } } @@ -196,12 +190,12 @@ func TestLinkProcessID_RemoteCorePID_Duplicate_Ignored(t *testing.T) { } // Verify only ONE relation exists (duplicate was ignored) - if len(tm.linkRelations) != 1 { - t.Errorf("Expected 1 link relation, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 1 { + t.Errorf("Expected 1 link relation, got %d", tm.totalLinks()) } // Verify targetIndex has only one consumer - entry := tm.targetIndex[localTarget] + entry := tm.getTargetEntry(localTarget) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -226,12 +220,11 @@ func TestUnlinkProcessID_Local(t *testing.T) { t.Fatalf("UnlinkProcessID failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("Link should be removed") } - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned") } } @@ -252,19 +245,17 @@ func TestUnlinkProcessID_NotLast(t *testing.T) { tm.UnlinkProcessID(consumer1, target) // Verify consumer1 link removed from linkRelations - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.linkRelations[key1]; exists { + if exists := tm.hasLinkRelation(consumer1, target); exists { t.Error("consumer1 link should be removed from linkRelations") } // Verify consumer2 link still exists - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, target); exists == false { t.Error("consumer2 link should still exist in linkRelations") } // Verify targetIndex still exists with only consumer2 - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should still exist") } @@ -297,13 +288,12 @@ func TestUnlinkProcessID_Last_SendsUnlink(t *testing.T) { } // Verify link removed from linkRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("Link should be removed from linkRelations") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned when last consumer removed") } @@ -340,8 +330,7 @@ func TestMonitorProcessID_Local(t *testing.T) { t.Fatalf("MonitorProcessID failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists == false { + if exists := tm.hasMonitorRelation(consumer, target); exists == false { t.Error("Monitor should be stored") } @@ -363,8 +352,7 @@ func TestMonitorProcessID_Local_EmptyNode(t *testing.T) { } // Verify monitor stored in monitorRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists == false { + if exists := tm.hasMonitorRelation(consumer, target); exists == false { t.Error("Monitor should be stored in monitorRelations") } @@ -387,13 +375,12 @@ func TestMonitorProcessID_Remote_First(t *testing.T) { } // Verify monitor stored in monitorRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists == false { + if exists := tm.hasMonitorRelation(consumer, target); exists == false { t.Error("Monitor should be stored in monitorRelations") } // Verify targetIndex created - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should be created") } @@ -429,20 +416,18 @@ func TestMonitorProcessID_Remote_Second(t *testing.T) { } // Verify both monitors stored in monitorRelations - if len(tm.monitorRelations) != 2 { - t.Errorf("Expected 2 monitor relations, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 2 { + t.Errorf("Expected 2 monitor relations, got %d", tm.totalMonitors()) } - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.monitorRelations[key1]; exists == false { + if exists := tm.hasMonitorRelation(consumer1, target); exists == false { t.Error("consumer1 monitor should exist in monitorRelations") } - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, target); exists == false { t.Error("consumer2 monitor should exist in monitorRelations") } // Verify targetIndex has both consumers - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -482,13 +467,12 @@ func TestMonitorProcessID_NetworkError_Rollback(t *testing.T) { } // Verify monitor rolled back from monitorRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, target); exists { t.Error("Monitor should be rolled back from monitorRelations") } // Verify targetIndex cleaned after rollback - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned after rollback") } } @@ -511,12 +495,12 @@ func TestMonitorProcessID_RemoteCorePID_Duplicate_Ignored(t *testing.T) { } // Verify only ONE relation exists (duplicate was ignored) - if len(tm.monitorRelations) != 1 { - t.Errorf("Expected 1 monitor relation, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 1 { + t.Errorf("Expected 1 monitor relation, got %d", tm.totalMonitors()) } // Verify targetIndex has only one consumer - entry := tm.targetIndex[localTarget] + entry := tm.getTargetEntry(localTarget) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -541,12 +525,11 @@ func TestDemonitorProcessID_Local(t *testing.T) { t.Fatalf("DemonitorProcessID failed: %v", err) } - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, target); exists { t.Error("Monitor should be removed") } - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned") } } @@ -567,19 +550,17 @@ func TestDemonitorProcessID_NotLast(t *testing.T) { tm.DemonitorProcessID(consumer1, target) // Verify consumer1 monitor removed from monitorRelations - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.monitorRelations[key1]; exists { + if exists := tm.hasMonitorRelation(consumer1, target); exists { t.Error("consumer1 monitor should be removed from monitorRelations") } // Verify consumer2 monitor still exists - key2 := relationKey{consumer: consumer2, target: target} - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, target); exists == false { t.Error("consumer2 monitor should still exist in monitorRelations") } // Verify targetIndex still exists with only consumer2 - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex entry should still exist") } @@ -612,13 +593,12 @@ func TestDemonitorProcessID_Last_SendsDemonitor(t *testing.T) { } // Verify monitor removed from monitorRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, target); exists { t.Error("Monitor should be removed from monitorRelations") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned when last consumer removed") } diff --git a/node/tm/query.go b/node/tm/query.go index 00dcb8a3d..d643d8648 100644 --- a/node/tm/query.go +++ b/node/tm/query.go @@ -3,75 +3,84 @@ package tm import "ergo.services/ergo/gen" func (tm *targetManager) HasLink(consumer gen.PID, target any) bool { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.RLock() + defer s.mutex.RUnlock() key := relationKey{ consumer: consumer, target: target, } - _, exists := tm.linkRelations[key] + _, exists := s.linkRelations[key] return exists } func (tm *targetManager) HasMonitor(consumer gen.PID, target any) bool { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(target) + s.mutex.RLock() + defer s.mutex.RUnlock() key := relationKey{ consumer: consumer, target: target, } - _, exists := tm.monitorRelations[key] + _, exists := s.monitorRelations[key] return exists } func (tm *targetManager) LinksFor(consumer gen.PID) []any { - tm.mutex.Lock() - defer tm.mutex.Unlock() - var targets []any - for key := range tm.linkRelations { - if key.consumer == consumer { - targets = append(targets, key.target) + for i := range tm.shards { + s := &tm.shards[i] + s.mutex.RLock() + for key := range s.linkRelations { + if key.consumer == consumer { + targets = append(targets, key.target) + } } + s.mutex.RUnlock() } return targets } func (tm *targetManager) MonitorsFor(consumer gen.PID) []any { - tm.mutex.Lock() - defer tm.mutex.Unlock() - var targets []any - for key := range tm.monitorRelations { - if key.consumer == consumer { - targets = append(targets, key.target) + for i := range tm.shards { + s := &tm.shards[i] + s.mutex.RLock() + for key := range s.monitorRelations { + if key.consumer == consumer { + targets = append(targets, key.target) + } } + s.mutex.RUnlock() } return targets } func (tm *targetManager) EventsFor(producer gen.PID) []gen.Event { - tm.mutex.Lock() - defer tm.mutex.Unlock() - - // Use producerEvents index for O(1) lookup - eventSet := tm.producerEvents[producer] - if eventSet == nil { - return nil + var events []gen.Event + + for i := range tm.shards { + s := &tm.shards[i] + s.mutex.RLock() + pe := s.producerEvents[producer] + if pe != nil { + for event := range pe { + events = append(events, event) + } + } + s.mutex.RUnlock() } - events := make([]gen.Event, 0, len(eventSet)) - for event := range eventSet { - events = append(events, event) + if len(events) == 0 { + return nil } - return events } diff --git a/node/tm/query_test.go b/node/tm/query_test.go index efe9ada11..cdab4985e 100644 --- a/node/tm/query_test.go +++ b/node/tm/query_test.go @@ -68,26 +68,21 @@ func TestHasLink_DifferentTargetTypes(t *testing.T) { } // Verify internal state: all 4 relations stored - if len(tm.linkRelations) != 4 { - t.Errorf("expected 4 linkRelations, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 4 { + t.Errorf("expected 4 linkRelations, got %d", tm.totalLinks()) } // Verify each relation in linkRelations - keyPID := relationKey{consumer: consumer, target: targetPID} - keyProcessID := relationKey{consumer: consumer, target: targetProcessID} - keyAlias := relationKey{consumer: consumer, target: targetAlias} - keyNode := relationKey{consumer: consumer, target: targetNode} - - if _, exists := tm.linkRelations[keyPID]; exists == false { + if exists := tm.hasLinkRelation(consumer, targetPID); exists == false { t.Error("linkRelations should contain PID relation") } - if _, exists := tm.linkRelations[keyProcessID]; exists == false { + if exists := tm.hasLinkRelation(consumer, targetProcessID); exists == false { t.Error("linkRelations should contain ProcessID relation") } - if _, exists := tm.linkRelations[keyAlias]; exists == false { + if exists := tm.hasLinkRelation(consumer, targetAlias); exists == false { t.Error("linkRelations should contain Alias relation") } - if _, exists := tm.linkRelations[keyNode]; exists == false { + if exists := tm.hasLinkRelation(consumer, targetNode); exists == false { t.Error("linkRelations should contain Node relation") } } @@ -173,26 +168,21 @@ func TestHasMonitor_DifferentTargetTypes(t *testing.T) { } // Verify internal state: all 4 relations stored - if len(tm.monitorRelations) != 4 { - t.Errorf("expected 4 monitorRelations, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 4 { + t.Errorf("expected 4 monitorRelations, got %d", tm.totalMonitors()) } // Verify each relation in monitorRelations - keyPID := relationKey{consumer: consumer, target: targetPID} - keyProcessID := relationKey{consumer: consumer, target: targetProcessID} - keyAlias := relationKey{consumer: consumer, target: targetAlias} - keyNode := relationKey{consumer: consumer, target: targetNode} - - if _, exists := tm.monitorRelations[keyPID]; exists == false { + if exists := tm.hasMonitorRelation(consumer, targetPID); exists == false { t.Error("monitorRelations should contain PID relation") } - if _, exists := tm.monitorRelations[keyProcessID]; exists == false { + if exists := tm.hasMonitorRelation(consumer, targetProcessID); exists == false { t.Error("monitorRelations should contain ProcessID relation") } - if _, exists := tm.monitorRelations[keyAlias]; exists == false { + if exists := tm.hasMonitorRelation(consumer, targetAlias); exists == false { t.Error("monitorRelations should contain Alias relation") } - if _, exists := tm.monitorRelations[keyNode]; exists == false { + if exists := tm.hasMonitorRelation(consumer, targetNode); exists == false { t.Error("monitorRelations should contain Node relation") } } @@ -323,19 +313,18 @@ func TestLinksFor_AfterRemovingOne(t *testing.T) { } // Verify internal state: target2 relation removed - key2 := relationKey{consumer: consumer, target: target2} - if _, exists := tm.linkRelations[key2]; exists { + if exists := tm.hasLinkRelation(consumer, target2); exists { t.Error("linkRelations should not contain removed target2") } // Verify targetIndex for target2 is cleaned - if _, exists := tm.targetIndex[target2]; exists { + if entry := tm.getTargetEntry(target2); entry != nil { t.Error("targetIndex for target2 should be cleaned") } // Verify remaining relations still exist - if len(tm.linkRelations) != 2 { - t.Errorf("expected 2 linkRelations remaining, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 2 { + t.Errorf("expected 2 linkRelations remaining, got %d", tm.totalLinks()) } } @@ -361,15 +350,15 @@ func TestLinksFor_AfterRemovingAll(t *testing.T) { } // Verify internal state: all linkRelations cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("expected 0 linkRelations, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("expected 0 linkRelations, got %d", tm.totalLinks()) } // Verify all targetIndex entries cleaned - if _, exists := tm.targetIndex[target1]; exists { + if entry := tm.getTargetEntry(target1); entry != nil { t.Error("targetIndex for target1 should be cleaned") } - if _, exists := tm.targetIndex[target2]; exists { + if entry := tm.getTargetEntry(target2); entry != nil { t.Error("targetIndex for target2 should be cleaned") } } @@ -439,12 +428,12 @@ func TestLinksFor_RemoteTargets(t *testing.T) { } // Verify internal state: both relations stored - if len(tm.linkRelations) != 2 { - t.Errorf("expected 2 linkRelations, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 2 { + t.Errorf("expected 2 linkRelations, got %d", tm.totalLinks()) } // Verify targetIndex for remote target has consumer - entry := tm.targetIndex[remoteTarget] + entry := tm.getTargetEntry(remoteTarget) if entry == nil { t.Fatal("targetIndex for remoteTarget should exist") } @@ -560,19 +549,18 @@ func TestMonitorsFor_AfterRemovingOne(t *testing.T) { } // Verify internal state: target2 relation removed - key2 := relationKey{consumer: consumer, target: target2} - if _, exists := tm.monitorRelations[key2]; exists { + if exists := tm.hasMonitorRelation(consumer, target2); exists { t.Error("monitorRelations should not contain removed target2") } // Verify targetIndex for target2 is cleaned - if _, exists := tm.targetIndex[target2]; exists { + if entry := tm.getTargetEntry(target2); entry != nil { t.Error("targetIndex for target2 should be cleaned") } // Verify remaining relations still exist - if len(tm.monitorRelations) != 2 { - t.Errorf("expected 2 monitorRelations remaining, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 2 { + t.Errorf("expected 2 monitorRelations remaining, got %d", tm.totalMonitors()) } } @@ -598,15 +586,15 @@ func TestMonitorsFor_AfterRemovingAll(t *testing.T) { } // Verify internal state: all monitorRelations cleaned - if len(tm.monitorRelations) != 0 { - t.Errorf("expected 0 monitorRelations, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 0 { + t.Errorf("expected 0 monitorRelations, got %d", tm.totalMonitors()) } // Verify all targetIndex entries cleaned - if _, exists := tm.targetIndex[target1]; exists { + if entry := tm.getTargetEntry(target1); entry != nil { t.Error("targetIndex for target1 should be cleaned") } - if _, exists := tm.targetIndex[target2]; exists { + if entry := tm.getTargetEntry(target2); entry != nil { t.Error("targetIndex for target2 should be cleaned") } } @@ -709,12 +697,12 @@ func TestMonitorsFor_RemoteTargets(t *testing.T) { } // Verify internal state: both relations stored - if len(tm.monitorRelations) != 2 { - t.Errorf("expected 2 monitorRelations, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 2 { + t.Errorf("expected 2 monitorRelations, got %d", tm.totalMonitors()) } // Verify targetIndex for remote target has consumer - entry := tm.targetIndex[remoteTarget] + entry := tm.getTargetEntry(remoteTarget) if entry == nil { t.Fatal("targetIndex for remoteTarget should exist") } @@ -745,18 +733,16 @@ func TestLinkAndMonitor_SameTarget_SeparateRelations(t *testing.T) { } // Verify stored separately - key := relationKey{consumer: consumer, target: target} - - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, target); exists == false { t.Error("Link should exist in linkRelations") } - if _, exists := tm.monitorRelations[key]; exists == false { + if exists := tm.hasMonitorRelation(consumer, target); exists == false { t.Error("Monitor should exist in monitorRelations") } // Verify targetIndex has consumer (shared) - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if _, exists := entry.consumers[consumer]; exists == false { t.Error("Consumer should be in targetIndex") } @@ -805,12 +791,12 @@ func TestEventsFor_Basic(t *testing.T) { } // Verify internal state: all events in events map - if len(tm.events) != 3 { - t.Errorf("expected 3 events in tm.events, got %d", len(tm.events)) + if tm.totalEvents() != 3 { + t.Errorf("expected 3 events in tm.events, got %d", tm.totalEvents()) } // Verify producerEvents has 3 events for this producer - producerEvts := tm.producerEvents[producer] + producerEvts := tm.getProducerEventsMap(producer) if producerEvts == nil { t.Fatal("producerEvents should exist for producer") } @@ -861,18 +847,18 @@ func TestEventsFor_AfterUnregister(t *testing.T) { // Verify internal state: event1 removed from events map event1 := gen.Event{Node: "node1", Name: "event1"} - if _, exists := tm.events[event1]; exists { + if entry := tm.getEventEntry(event1); entry != nil { t.Error("event1 should be removed from tm.events") } // Verify event2 still in events map event2 := gen.Event{Node: "node1", Name: "event2"} - if _, exists := tm.events[event2]; exists == false { + if entry := tm.getEventEntry(event2); entry == nil { t.Error("event2 should still exist in tm.events") } // Verify producerEvents has 1 event - producerEvts := tm.producerEvents[producer] + producerEvts := tm.getProducerEventsMap(producer) if len(producerEvts) != 1 { t.Errorf("expected 1 event in producerEvents, got %d", len(producerEvts)) } @@ -913,15 +899,15 @@ func TestEventsFor_AfterUnregisterAll(t *testing.T) { // Verify internal state: all events removed from events map event1 := gen.Event{Node: "node1", Name: "event1"} event2 := gen.Event{Node: "node1", Name: "event2"} - if _, exists := tm.events[event1]; exists { + if entry := tm.getEventEntry(event1); entry != nil { t.Error("event1 should be removed from tm.events") } - if _, exists := tm.events[event2]; exists { + if entry := tm.getEventEntry(event2); entry != nil { t.Error("event2 should be removed from tm.events") } // Verify producerEvents cleaned up - if _, exists := tm.producerEvents[producer]; exists { + if producerEvts := tm.getProducerEventsMap(producer); producerEvts != nil { t.Error("producerEvents for producer should be cleaned up") } } diff --git a/node/tm/scenarios_test.go b/node/tm/scenarios_test.go index c1723beb0..3b7fd1afa 100644 --- a/node/tm/scenarios_test.go +++ b/node/tm/scenarios_test.go @@ -51,13 +51,12 @@ func TestScenario_TerminationReasons_Link(t *testing.T) { } // Verify internal state: linkRelations cleaned up - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("linkRelations should be cleaned after termination") } // Verify targetIndex cleaned up - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned after termination") } }) @@ -96,13 +95,12 @@ func TestScenario_TerminationReasons_Monitor(t *testing.T) { } // Verify internal state: monitorRelations cleaned up - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, target); exists { t.Error("monitorRelations should be cleaned after termination") } // Verify targetIndex cleaned up - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned after termination") } }) @@ -167,13 +165,13 @@ func TestScenario_ProcessTermination_CleansUpAllRelations(t *testing.T) { } // Verify targetIndex cleaned up for all targets - if _, exists := tm.targetIndex[target1]; exists { + if entry := tm.getTargetEntry(target1); entry != nil { t.Error("targetIndex for target1 should be cleaned") } - if _, exists := tm.targetIndex[target2]; exists { + if entry := tm.getTargetEntry(target2); entry != nil { t.Error("targetIndex for target2 should be cleaned") } - if _, exists := tm.targetIndex[target3]; exists { + if entry := tm.getTargetEntry(target3); entry != nil { t.Error("targetIndex for target3 should be cleaned") } } @@ -227,7 +225,7 @@ func TestScenario_MultipleConsumers_SameRemoteTarget_Link(t *testing.T) { } // Verify targetIndex has all 3 consumers - entry := tm.targetIndex[remoteTarget] + entry := tm.getTargetEntry(remoteTarget) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -274,21 +272,18 @@ func TestScenario_MultipleConsumers_SameRemoteTarget_Monitor(t *testing.T) { } // Verify all relations stored - key1 := relationKey{consumer: consumer1, target: remoteTarget} - key2 := relationKey{consumer: consumer2, target: remoteTarget} - key3 := relationKey{consumer: consumer3, target: remoteTarget} - if _, exists := tm.monitorRelations[key1]; exists == false { + if exists := tm.hasMonitorRelation(consumer1, remoteTarget); exists == false { t.Error("consumer1 relation should exist in monitorRelations") } - if _, exists := tm.monitorRelations[key2]; exists == false { + if exists := tm.hasMonitorRelation(consumer2, remoteTarget); exists == false { t.Error("consumer2 relation should exist in monitorRelations") } - if _, exists := tm.monitorRelations[key3]; exists == false { + if exists := tm.hasMonitorRelation(consumer3, remoteTarget); exists == false { t.Error("consumer3 relation should exist in monitorRelations") } // Verify targetIndex has all 3 consumers - entry := tm.targetIndex[remoteTarget] + entry := tm.getTargetEntry(remoteTarget) if entry == nil { t.Fatal("targetIndex entry should exist") } @@ -332,7 +327,7 @@ func TestScenario_PartialUnlink_NoNetworkUnlinkUntilLast(t *testing.T) { t.Fatalf("expected 0 unlinks after P1, got %d", core.countSentUnlinks()) } // Verify targetIndex still has 2 consumers - entry := tm.targetIndex[remoteTarget] + entry := tm.getTargetEntry(remoteTarget) if entry == nil { t.Fatal("targetIndex should still exist after P1 unlink") } @@ -348,7 +343,7 @@ func TestScenario_PartialUnlink_NoNetworkUnlinkUntilLast(t *testing.T) { t.Fatalf("expected 0 unlinks after P2, got %d", core.countSentUnlinks()) } // Verify targetIndex still has 1 consumer - entry = tm.targetIndex[remoteTarget] + entry = tm.getTargetEntry(remoteTarget) if entry == nil { t.Fatal("targetIndex should still exist after P2 unlink") } @@ -364,7 +359,7 @@ func TestScenario_PartialUnlink_NoNetworkUnlinkUntilLast(t *testing.T) { t.Fatalf("expected 1 unlink after P3, got %d", core.countSentUnlinks()) } // Verify targetIndex cleaned up - if _, exists := tm.targetIndex[remoteTarget]; exists { + if entry := tm.getTargetEntry(remoteTarget); entry != nil { t.Error("targetIndex should be cleaned after last consumer unlinks") } } @@ -393,7 +388,7 @@ func TestScenario_PartialDemonitor_NoNetworkDemonitorUntilLast(t *testing.T) { t.Fatalf("expected 0 demonitors after P1,P2, got %d", core.countSentDemonitors()) } // Verify targetIndex still has 1 consumer (P3) - entry := tm.targetIndex[remoteTarget] + entry := tm.getTargetEntry(remoteTarget) if entry == nil { t.Fatal("targetIndex should still exist after P1,P2 demonitor") } @@ -407,7 +402,7 @@ func TestScenario_PartialDemonitor_NoNetworkDemonitorUntilLast(t *testing.T) { t.Fatalf("expected 1 demonitor after P3, got %d", core.countSentDemonitors()) } // Verify targetIndex cleaned up - if _, exists := tm.targetIndex[remoteTarget]; exists { + if entry := tm.getTargetEntry(remoteTarget); entry != nil { t.Error("targetIndex should be cleaned after last consumer demonitors") } } @@ -444,19 +439,16 @@ func TestScenario_RemoteTargetTermination_NotifiesAllLocalConsumers(t *testing.T } // Verify internal state cleaned up - key1 := relationKey{consumer: consumer1, target: remoteTarget} - key2 := relationKey{consumer: consumer2, target: remoteTarget} - key3 := relationKey{consumer: consumer3, target: remoteTarget} - if _, exists := tm.linkRelations[key1]; exists { + if exists := tm.hasLinkRelation(consumer1, remoteTarget); exists { t.Error("consumer1 linkRelation should be cleaned") } - if _, exists := tm.linkRelations[key2]; exists { + if exists := tm.hasLinkRelation(consumer2, remoteTarget); exists { t.Error("consumer2 linkRelation should be cleaned") } - if _, exists := tm.linkRelations[key3]; exists { + if exists := tm.hasLinkRelation(consumer3, remoteTarget); exists { t.Error("consumer3 linkRelation should be cleaned") } - if _, exists := tm.targetIndex[remoteTarget]; exists { + if entry := tm.getTargetEntry(remoteTarget); entry != nil { t.Error("targetIndex should be cleaned after termination") } } @@ -485,19 +477,16 @@ func TestScenario_RemoteTargetTermination_NotifiesAllLocalMonitors(t *testing.T) } // Verify internal state cleaned up - key1 := relationKey{consumer: consumer1, target: remoteTarget} - key2 := relationKey{consumer: consumer2, target: remoteTarget} - key3 := relationKey{consumer: consumer3, target: remoteTarget} - if _, exists := tm.monitorRelations[key1]; exists { + if exists := tm.hasMonitorRelation(consumer1, remoteTarget); exists { t.Error("consumer1 monitorRelation should be cleaned") } - if _, exists := tm.monitorRelations[key2]; exists { + if exists := tm.hasMonitorRelation(consumer2, remoteTarget); exists { t.Error("consumer2 monitorRelation should be cleaned") } - if _, exists := tm.monitorRelations[key3]; exists { + if exists := tm.hasMonitorRelation(consumer3, remoteTarget); exists { t.Error("consumer3 monitorRelation should be cleaned") } - if _, exists := tm.targetIndex[remoteTarget]; exists { + if entry := tm.getTargetEntry(remoteTarget); entry != nil { t.Error("targetIndex should be cleaned after termination") } } @@ -532,11 +521,10 @@ func TestScenario_NodeDown_NotifiesAllLinkedConsumers(t *testing.T) { } // Verify internal state cleaned up - key := relationKey{consumer: consumer, target: remoteTarget} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, remoteTarget); exists { t.Error("linkRelation should be cleaned after node down") } - if _, exists := tm.targetIndex[remoteTarget]; exists { + if entry := tm.getTargetEntry(remoteTarget); entry != nil { t.Error("targetIndex should be cleaned after node down") } } @@ -563,11 +551,10 @@ func TestScenario_NodeDown_NotifiesAllMonitoringConsumers(t *testing.T) { } // Verify internal state cleaned up - key := relationKey{consumer: consumer, target: remoteTarget} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, remoteTarget); exists { t.Error("monitorRelation should be cleaned after node down") } - if _, exists := tm.targetIndex[remoteTarget]; exists { + if entry := tm.getTargetEntry(remoteTarget); entry != nil { t.Error("targetIndex should be cleaned after node down") } } @@ -601,18 +588,18 @@ func TestScenario_NodeDown_MultipleTargets(t *testing.T) { } // Verify all targetIndex entries cleaned up - if _, exists := tm.targetIndex[remoteTarget1]; exists { + if entry := tm.getTargetEntry(remoteTarget1); entry != nil { t.Error("targetIndex for remoteTarget1 should be cleaned") } - if _, exists := tm.targetIndex[remoteTarget2]; exists { + if entry := tm.getTargetEntry(remoteTarget2); entry != nil { t.Error("targetIndex for remoteTarget2 should be cleaned") } - if _, exists := tm.targetIndex[remoteTarget3]; exists { + if entry := tm.getTargetEntry(remoteTarget3); entry != nil { t.Error("targetIndex for remoteTarget3 should be cleaned") } // Verify all linkRelations cleaned up - if len(tm.linkRelations) != 0 { - t.Errorf("all linkRelations should be cleaned, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("all linkRelations should be cleaned, got %d", tm.totalLinks()) } } @@ -652,11 +639,10 @@ func TestScenario_LinkNode_ReceivesExitOnDisconnect(t *testing.T) { } // Verify internal state cleaned up - key := relationKey{consumer: consumer, target: remoteNode} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, remoteNode); exists { t.Error("linkRelation should be cleaned after node disconnect") } - if _, exists := tm.targetIndex[remoteNode]; exists { + if entry := tm.getTargetEntry(remoteNode); entry != nil { t.Error("targetIndex should be cleaned after node disconnect") } } @@ -683,11 +669,10 @@ func TestScenario_MonitorNode_ReceivesDownOnDisconnect(t *testing.T) { } // Verify internal state cleaned up - key := relationKey{consumer: consumer, target: remoteNode} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, remoteNode); exists { t.Error("monitorRelation should be cleaned after node disconnect") } - if _, exists := tm.targetIndex[remoteNode]; exists { + if entry := tm.getTargetEntry(remoteNode); entry != nil { t.Error("targetIndex should be cleaned after node disconnect") } } @@ -734,7 +719,7 @@ func TestScenario_RemoteCorePIDSubscribesEvent_FirstSubscriber_EventStart(t *tes } // Verify event entry has the subscriber - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry == nil { t.Fatal("event entry should exist") } @@ -778,7 +763,7 @@ func TestScenario_RemoteCorePIDUnsubscribesEvent_LastSubscriber_EventStop(t *tes } // Verify subscriber removed from event entry - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry == nil { t.Fatal("event entry should still exist (producer owns it)") } @@ -877,14 +862,13 @@ func TestScenario_MixedLinkMonitor_SameTarget(t *testing.T) { } // Verify internal state cleaned up - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("linkRelation should be cleaned after termination") } - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, target); exists { t.Error("monitorRelation should be cleaned after termination") } - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned after termination") } } @@ -916,11 +900,10 @@ func TestScenario_ProcessID_TerminationNotifiesMonitors(t *testing.T) { } // Verify internal state cleaned up - key := relationKey{consumer: consumer, target: targetProcessID} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, targetProcessID); exists { t.Error("monitorRelation should be cleaned after termination") } - if _, exists := tm.targetIndex[targetProcessID]; exists { + if entry := tm.getTargetEntry(targetProcessID); entry != nil { t.Error("targetIndex should be cleaned after termination") } } @@ -947,11 +930,10 @@ func TestScenario_ProcessID_TerminationNotifiesLinkers(t *testing.T) { } // Verify internal state cleaned up - key := relationKey{consumer: consumer, target: targetProcessID} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, targetProcessID); exists { t.Error("linkRelation should be cleaned after termination") } - if _, exists := tm.targetIndex[targetProcessID]; exists { + if entry := tm.getTargetEntry(targetProcessID); entry != nil { t.Error("targetIndex should be cleaned after termination") } } @@ -983,11 +965,10 @@ func TestScenario_Alias_TerminationNotifiesMonitors(t *testing.T) { } // Verify internal state cleaned up - key := relationKey{consumer: consumer, target: targetAlias} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, targetAlias); exists { t.Error("monitorRelation should be cleaned after termination") } - if _, exists := tm.targetIndex[targetAlias]; exists { + if entry := tm.getTargetEntry(targetAlias); entry != nil { t.Error("targetIndex should be cleaned after termination") } } @@ -1014,11 +995,10 @@ func TestScenario_Alias_TerminationNotifiesLinkers(t *testing.T) { } // Verify internal state cleaned up - key := relationKey{consumer: consumer, target: targetAlias} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, targetAlias); exists { t.Error("linkRelation should be cleaned after termination") } - if _, exists := tm.targetIndex[targetAlias]; exists { + if entry := tm.getTargetEntry(targetAlias); entry != nil { t.Error("targetIndex should be cleaned after termination") } } @@ -1046,15 +1026,14 @@ func TestScenario_DuplicateLink_ReturnsError(t *testing.T) { } // Verify only 1 relation stored (not duplicated) - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists == false { + if exists := tm.hasLinkRelation(consumer, target); exists == false { t.Error("linkRelation should still exist") } - if len(tm.linkRelations) != 1 { - t.Errorf("expected exactly 1 linkRelation, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 1 { + t.Errorf("expected exactly 1 linkRelation, got %d", tm.totalLinks()) } // Verify targetIndex has only 1 consumer - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex should exist") } @@ -1081,15 +1060,14 @@ func TestScenario_DuplicateMonitor_ReturnsError(t *testing.T) { } // Verify only 1 relation stored (not duplicated) - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists == false { + if exists := tm.hasMonitorRelation(consumer, target); exists == false { t.Error("monitorRelation should still exist") } - if len(tm.monitorRelations) != 1 { - t.Errorf("expected exactly 1 monitorRelation, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 1 { + t.Errorf("expected exactly 1 monitorRelation, got %d", tm.totalMonitors()) } // Verify targetIndex has only 1 consumer - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetIndex should exist") } @@ -1752,11 +1730,11 @@ func TestMultipleConsumers_Link_AllNotified(t *testing.T) { } // Verify internal state cleaned up - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned after termination") } - if len(tm.linkRelations) != 0 { - t.Errorf("all linkRelations should be cleaned, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("all linkRelations should be cleaned, got %d", tm.totalLinks()) } } @@ -1788,11 +1766,11 @@ func TestMultipleConsumers_Monitor_AllNotified(t *testing.T) { } // Verify internal state cleaned up - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned after termination") } - if len(tm.monitorRelations) != 0 { - t.Errorf("all monitorRelations should be cleaned, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 0 { + t.Errorf("all monitorRelations should be cleaned, got %d", tm.totalMonitors()) } } @@ -1887,7 +1865,7 @@ func TestRemoteLink_TwoConsumers_OneNetworkRequest(t *testing.T) { } // Verify targetIndex has 2 consumers - entry := tm.targetIndex[remotePID] + entry := tm.getTargetEntry(remotePID) if entry == nil { t.Fatal("targetIndex should exist") } @@ -1923,7 +1901,7 @@ func TestRemoteUnlink_LastConsumer_SendsNetworkUnlink(t *testing.T) { } // Verify targetIndex cleaned up - if _, exists := tm.targetIndex[remotePID]; exists { + if entry := tm.getTargetEntry(remotePID); entry != nil { t.Error("targetIndex should be cleaned after last consumer unlinks") } } @@ -2042,9 +2020,10 @@ func TestCorner_LinkFromTwoNodes(t *testing.T) { // Simulate node2's CorePID also linking coreNode2 := gen.PID{Node: "node2", ID: 1} + s := tm.shardFor(target) key := relationKey{consumer: coreNode2, target: target} - tm.linkRelations[key] = struct{}{} - entry := tm.targetIndex[target] + s.linkRelations[key] = struct{}{} + entry := s.targetIndex[target] if entry != nil { entry.consumers[coreNode2] = struct{}{} } @@ -2132,7 +2111,7 @@ func TestCorner_TargetIndexCleanedAfterRollback(t *testing.T) { tm.LinkPID(consumer, target) // Cleaned - if _, exists := tm.targetIndex[target]; exists { + if entry := tm.getTargetEntry(target); entry != nil { t.Error("targetIndex should be cleaned after rollback") } } @@ -2158,7 +2137,7 @@ func TestCorner_Event_ReSubscribe(t *testing.T) { t.Error("Re-subscribe should work") } - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry.subscriberCount != 1 { t.Errorf("Counter should be 1, got %d", entry.subscriberCount) } @@ -2222,8 +2201,8 @@ func TestCorner_PublishEvent_NoSubscribers(t *testing.T) { } // Buffer updated - entry := tm.events[event] - if len(entry.buffer) != 1 { + entry := tm.getEventEntry(event) + if entry.buffer.len != 1 { t.Error("Buffer should be updated") } } @@ -2277,8 +2256,8 @@ func TestCorner_ConcurrentAdd(t *testing.T) { } // All stored - if len(tm.linkRelations) != 10 { - t.Errorf("Expected 10 links, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 10 { + t.Errorf("Expected 10 links, got %d", tm.totalLinks()) } // Only 1 network @@ -2343,8 +2322,8 @@ func TestStress_1000ConcurrentLinks(t *testing.T) { } // Verify all 1000 stored - if len(tm.linkRelations) != 1000 { - t.Errorf("Expected 1000 links, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 1000 { + t.Errorf("Expected 1000 links, got %d", tm.totalLinks()) } // Only 1 network request (CorePID optimization!) @@ -2353,7 +2332,7 @@ func TestStress_1000ConcurrentLinks(t *testing.T) { } // Verify targetIndex has all 1000 - entry := tm.targetIndex[target] + entry := tm.getTargetEntry(target) if entry == nil { t.Fatal("targetEntry should exist") } @@ -2397,13 +2376,13 @@ func TestStress_ConcurrentAddRemove(t *testing.T) { time.Sleep(100 * time.Millisecond) // All should be cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("All links should be removed, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("All links should be removed, got %d", tm.totalLinks()) } // targetIndex should be clean (or have minimal entries) - if len(tm.targetIndex) > 0 { - t.Logf("targetIndex has %d entries (may be cleanup in progress)", len(tm.targetIndex)) + if tm.totalTargetIndex() > 0 { + t.Logf("targetIndex has %d entries (may be cleanup in progress)", tm.totalTargetIndex()) } } @@ -2428,7 +2407,7 @@ func TestStress_Event_1000Subscribers(t *testing.T) { } // Verify all subscribed - entry := tm.events[event] + entry := tm.getEventEntry(event) if len(entry.linkSubscribers) != 1000 { t.Errorf("Expected 1000 subscribers, got %d", len(entry.linkSubscribers)) } @@ -2489,12 +2468,12 @@ func TestStress_RapidSubscribeUnsubscribe(t *testing.T) { } // Should be clean - if len(tm.linkRelations) != 0 { - t.Errorf("linkRelations should be empty, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("linkRelations should be empty, got %d", tm.totalLinks()) } - if len(tm.targetIndex) != 0 { - t.Errorf("targetIndex should be empty, got %d", len(tm.targetIndex)) + if tm.totalTargetIndex() != 0 { + t.Errorf("targetIndex should be empty, got %d", tm.totalTargetIndex()) } } @@ -2514,8 +2493,8 @@ func TestStress_MassTermination(t *testing.T) { } // Verify 1000 links total - if len(tm.linkRelations) != 1000 { - t.Errorf("Expected 1000 links, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 1000 { + t.Errorf("Expected 1000 links, got %d", tm.totalLinks()) } core.resetSentExits() @@ -2544,8 +2523,8 @@ func TestStress_MassTermination(t *testing.T) { } // All cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("All links should be cleaned, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("All links should be cleaned, got %d", tm.totalLinks()) } // Check statistics @@ -2605,7 +2584,7 @@ func TestStress_Event_ConcurrentOperations(t *testing.T) { time.Sleep(200 * time.Millisecond) // Verify consistency - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry == nil { t.Fatal("Event should exist") } @@ -2622,8 +2601,8 @@ func TestStress_Event_ConcurrentOperations(t *testing.T) { // eventsSent should be 100 * number_of_subscribers_at_publish_time // (varies based on timing, but should be > 0) - if info.EventsSent == 0 { - t.Error("EventsSent should be > 0") + if info.EventsLocalSent == 0 { + t.Error("EventsLocalSent should be > 0") } } @@ -2654,8 +2633,8 @@ func TestStress_TerminatedNode_1000Subscriptions(t *testing.T) { } // All cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("All links should be cleaned, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("All links should be cleaned, got %d", tm.totalLinks()) } } @@ -2675,23 +2654,23 @@ func TestStress_Memory_10KCycles(t *testing.T) { if i%1000 == 0 { // Check no memory leak every 1000 cycles - if len(tm.linkRelations) != 0 { - t.Errorf("Cycle %d: memory leak detected, %d relations", i, len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("Cycle %d: memory leak detected, %d relations", i, tm.totalLinks()) } - if len(tm.targetIndex) != 0 { - t.Errorf("Cycle %d: targetIndex leak, %d entries", i, len(tm.targetIndex)) + if tm.totalTargetIndex() != 0 { + t.Errorf("Cycle %d: targetIndex leak, %d entries", i, tm.totalTargetIndex()) } } } // Final check - if len(tm.linkRelations) != 0 { - t.Errorf("Final: linkRelations should be empty, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("Final: linkRelations should be empty, got %d", tm.totalLinks()) } - if len(tm.targetIndex) != 0 { - t.Errorf("Final: targetIndex should be empty, got %d", len(tm.targetIndex)) + if tm.totalTargetIndex() != 0 { + t.Errorf("Final: targetIndex should be empty, got %d", tm.totalTargetIndex()) } } @@ -2761,8 +2740,8 @@ func TestStress_ConcurrentTerminatedProcess(t *testing.T) { } // 1000 links total - if len(tm.linkRelations) != 1000 { - t.Errorf("Expected 1000 links, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 1000 { + t.Errorf("Expected 1000 links, got %d", tm.totalLinks()) } var wg sync.WaitGroup @@ -2784,8 +2763,8 @@ func TestStress_ConcurrentTerminatedProcess(t *testing.T) { time.Sleep(500 * time.Millisecond) // All cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("All links should be cleaned, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("All links should be cleaned, got %d", tm.totalLinks()) } // Remote Unlinks sent (500 consumers × 2 targets but CorePID optimization) @@ -2834,12 +2813,12 @@ func TestStress_MixedOperations(t *testing.T) { time.Sleep(200 * time.Millisecond) // Should be mostly clean - if len(tm.linkRelations) != 0 { - t.Logf("linkRelations: %d (some may be in flight)", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Logf("linkRelations: %d (some may be in flight)", tm.totalLinks()) } - if len(tm.monitorRelations) != 0 { - t.Logf("monitorRelations: %d (some may be in flight)", len(tm.monitorRelations)) + if tm.totalMonitors() != 0 { + t.Logf("monitorRelations: %d (some may be in flight)", tm.totalMonitors()) } } @@ -2863,8 +2842,8 @@ func TestStress_WorkerSpawnCycles(t *testing.T) { } // All 1000 links stored - if len(tm.linkRelations) != 1000 { - t.Errorf("Expected 1000 links, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 1000 { + t.Errorf("Expected 1000 links, got %d", tm.totalLinks()) } // Worker should have spawned and slept multiple times diff --git a/node/tm/terminate.go b/node/tm/terminate.go index 78bb1074f..123a1197f 100644 --- a/node/tm/terminate.go +++ b/node/tm/terminate.go @@ -3,21 +3,20 @@ package tm import "ergo.services/ergo/gen" func (tm *targetManager) TerminatedTargetPID(pid gen.PID, reason error) { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(pid) + s.mutex.Lock() remoteNodesLinks := make(map[gen.Atom]bool) remoteNodesMonitors := make(map[gen.Atom]bool) var localExitConsumers []gen.PID var localDownConsumers []gen.PID - // Process link consumers - for key := range tm.linkRelations { + for key := range s.linkRelations { if key.target != pid { continue } - delete(tm.linkRelations, key) + delete(s.linkRelations, key) if key.consumer.Node != tm.core.Name() { remoteNodesLinks[key.consumer.Node] = true @@ -27,13 +26,12 @@ func (tm *targetManager) TerminatedTargetPID(pid gen.PID, reason error) { localExitConsumers = append(localExitConsumers, key.consumer) } - // Process monitor consumers - for key := range tm.monitorRelations { + for key := range s.monitorRelations { if key.target != pid { continue } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) if key.consumer.Node != tm.core.Name() { remoteNodesMonitors[key.consumer.Node] = true @@ -43,14 +41,15 @@ func (tm *targetManager) TerminatedTargetPID(pid gen.PID, reason error) { localDownConsumers = append(localDownConsumers, key.consumer) } - // Send exit messages to local consumers + delete(s.targetIndex, pid) + s.mutex.Unlock() + if len(localExitConsumers) > 0 { tm.exitSignalsProduced.Add(1) tm.core.RouteSendExitMessages(pid, localExitConsumers, gen.MessageExitPID{PID: pid, Reason: reason}) tm.exitSignalsDelivered.Add(int64(len(localExitConsumers))) } - // Send down messages to local consumers if len(localDownConsumers) > 0 { tm.downMessagesProduced.Add(1) for _, consumer := range localDownConsumers { @@ -59,7 +58,6 @@ func (tm *targetManager) TerminatedTargetPID(pid gen.PID, reason error) { tm.downMessagesDelivered.Add(int64(len(localDownConsumers))) } - // Send to remote nodes for node := range remoteNodesLinks { connection, err := tm.core.GetConnection(node) if err != nil { @@ -75,25 +73,22 @@ func (tm *targetManager) TerminatedTargetPID(pid gen.PID, reason error) { } connection.SendTerminatePID(pid, reason) } - - delete(tm.targetIndex, pid) } func (tm *targetManager) TerminatedTargetProcessID(processID gen.ProcessID, reason error) { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(processID) + s.mutex.Lock() remoteNodes := make(map[gen.Atom]bool) var localExitConsumers []gen.PID var localDownConsumers []gen.PID - // Link consumers - for key := range tm.linkRelations { + for key := range s.linkRelations { if key.target != processID { continue } - delete(tm.linkRelations, key) + delete(s.linkRelations, key) if key.consumer.Node != tm.core.Name() { remoteNodes[key.consumer.Node] = true @@ -103,13 +98,12 @@ func (tm *targetManager) TerminatedTargetProcessID(processID gen.ProcessID, reas localExitConsumers = append(localExitConsumers, key.consumer) } - // Monitor consumers - for key := range tm.monitorRelations { + for key := range s.monitorRelations { if key.target != processID { continue } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) if key.consumer.Node != tm.core.Name() { remoteNodes[key.consumer.Node] = true @@ -119,14 +113,15 @@ func (tm *targetManager) TerminatedTargetProcessID(processID gen.ProcessID, reas localDownConsumers = append(localDownConsumers, key.consumer) } - // Send exit messages + delete(s.targetIndex, processID) + s.mutex.Unlock() + if len(localExitConsumers) > 0 { tm.exitSignalsProduced.Add(1) tm.core.RouteSendExitMessages(tm.core.PID(), localExitConsumers, gen.MessageExitProcessID{ProcessID: processID, Reason: reason}) tm.exitSignalsDelivered.Add(int64(len(localExitConsumers))) } - // Send down messages if len(localDownConsumers) > 0 { tm.downMessagesProduced.Add(1) for _, consumer := range localDownConsumers { @@ -135,7 +130,6 @@ func (tm *targetManager) TerminatedTargetProcessID(processID gen.ProcessID, reas tm.downMessagesDelivered.Add(int64(len(localDownConsumers))) } - // Send to remote nodes for node := range remoteNodes { connection, err := tm.core.GetConnection(node) if err != nil { @@ -143,25 +137,22 @@ func (tm *targetManager) TerminatedTargetProcessID(processID gen.ProcessID, reas } connection.SendTerminateProcessID(processID, reason) } - - delete(tm.targetIndex, processID) } func (tm *targetManager) TerminatedTargetAlias(alias gen.Alias, reason error) { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(alias) + s.mutex.Lock() remoteNodes := make(map[gen.Atom]bool) var localExitConsumers []gen.PID var localDownConsumers []gen.PID - // Link consumers - for key := range tm.linkRelations { + for key := range s.linkRelations { if key.target != alias { continue } - delete(tm.linkRelations, key) + delete(s.linkRelations, key) if key.consumer.Node != tm.core.Name() { remoteNodes[key.consumer.Node] = true @@ -171,13 +162,12 @@ func (tm *targetManager) TerminatedTargetAlias(alias gen.Alias, reason error) { localExitConsumers = append(localExitConsumers, key.consumer) } - // Monitor consumers - for key := range tm.monitorRelations { + for key := range s.monitorRelations { if key.target != alias { continue } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) if key.consumer.Node != tm.core.Name() { remoteNodes[key.consumer.Node] = true @@ -187,14 +177,15 @@ func (tm *targetManager) TerminatedTargetAlias(alias gen.Alias, reason error) { localDownConsumers = append(localDownConsumers, key.consumer) } - // Send exit messages + delete(s.targetIndex, alias) + s.mutex.Unlock() + if len(localExitConsumers) > 0 { tm.exitSignalsProduced.Add(1) tm.core.RouteSendExitMessages(tm.core.PID(), localExitConsumers, gen.MessageExitAlias{Alias: alias, Reason: reason}) tm.exitSignalsDelivered.Add(int64(len(localExitConsumers))) } - // Send down messages if len(localDownConsumers) > 0 { tm.downMessagesProduced.Add(1) for _, consumer := range localDownConsumers { @@ -203,7 +194,6 @@ func (tm *targetManager) TerminatedTargetAlias(alias gen.Alias, reason error) { tm.downMessagesDelivered.Add(int64(len(localDownConsumers))) } - // Send to remote nodes for node := range remoteNodes { connection, err := tm.core.GetConnection(node) if err != nil { @@ -211,25 +201,22 @@ func (tm *targetManager) TerminatedTargetAlias(alias gen.Alias, reason error) { } connection.SendTerminateAlias(alias, reason) } - - delete(tm.targetIndex, alias) } func (tm *targetManager) TerminatedTargetEvent(event gen.Event, reason error) { - tm.mutex.Lock() - defer tm.mutex.Unlock() + s := tm.shardFor(event) + s.mutex.Lock() remoteNodes := make(map[gen.Atom]bool) var localExitConsumers []gen.PID var localDownConsumers []gen.PID - // Link consumers - for key := range tm.linkRelations { + for key := range s.linkRelations { if key.target != event { continue } - delete(tm.linkRelations, key) + delete(s.linkRelations, key) if key.consumer.Node != tm.core.Name() { remoteNodes[key.consumer.Node] = true @@ -239,13 +226,12 @@ func (tm *targetManager) TerminatedTargetEvent(event gen.Event, reason error) { localExitConsumers = append(localExitConsumers, key.consumer) } - // Monitor consumers - for key := range tm.monitorRelations { + for key := range s.monitorRelations { if key.target != event { continue } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) if key.consumer.Node != tm.core.Name() { remoteNodes[key.consumer.Node] = true @@ -255,14 +241,19 @@ func (tm *targetManager) TerminatedTargetEvent(event gen.Event, reason error) { localDownConsumers = append(localDownConsumers, key.consumer) } - // Send exit messages + if entry, exists := s.events[event]; exists { + tm.eventIndex.Delete(entry.id) + } + delete(s.events, event) + delete(s.targetIndex, event) + s.mutex.Unlock() + if len(localExitConsumers) > 0 { tm.exitSignalsProduced.Add(1) tm.core.RouteSendExitMessages(tm.core.PID(), localExitConsumers, gen.MessageExitEvent{Event: event, Reason: reason}) tm.exitSignalsDelivered.Add(int64(len(localExitConsumers))) } - // Send down messages if len(localDownConsumers) > 0 { tm.downMessagesProduced.Add(1) for _, consumer := range localDownConsumers { @@ -271,7 +262,6 @@ func (tm *targetManager) TerminatedTargetEvent(event gen.Event, reason error) { tm.downMessagesDelivered.Add(int64(len(localDownConsumers))) } - // Send to remote nodes for node := range remoteNodes { connection, err := tm.core.GetConnection(node) if err != nil { @@ -279,73 +269,71 @@ func (tm *targetManager) TerminatedTargetEvent(event gen.Event, reason error) { } connection.SendTerminateEvent(event, reason) } - - // Cleanup event from events map - delete(tm.events, event) - delete(tm.targetIndex, event) } func (tm *targetManager) TerminatedTargetNode(node gen.Atom, reason error) { - tm.mutex.Lock() - defer tm.mutex.Unlock() + for i := range tm.shards { + tm.terminateNodeInShard(&tm.shards[i], node, reason) + } +} + +func (tm *targetManager) terminateNodeInShard(s *shard, node gen.Atom, reason error) { + s.mutex.Lock() - // Collect exit messages by type - exitPID := make(map[gen.PID][]gen.PID) // target -> consumers + exitPID := make(map[gen.PID][]gen.PID) exitProcessID := make(map[gen.ProcessID][]gen.PID) exitAlias := make(map[gen.Alias][]gen.PID) exitEvent := make(map[gen.Event][]gen.PID) exitNode := make(map[gen.Atom][]gen.PID) - // Cleanup linkRelations - for key := range tm.linkRelations { + for key := range s.linkRelations { shouldRemove := false - // Consumer on terminated node if key.consumer.Node == node { shouldRemove = true } - // Target on terminated node if shouldRemove == false { switch t := key.target.(type) { case gen.PID: - if t.Node == node { - shouldRemove = true - if key.consumer.Node == tm.core.Name() { - exitPID[t] = append(exitPID[t], key.consumer) - } + if t.Node != node { + break + } + shouldRemove = true + if key.consumer.Node == tm.core.Name() { + exitPID[t] = append(exitPID[t], key.consumer) } - case gen.ProcessID: - if t.Node == node { - shouldRemove = true - if key.consumer.Node == tm.core.Name() { - exitProcessID[t] = append(exitProcessID[t], key.consumer) - } + if t.Node != node { + break + } + shouldRemove = true + if key.consumer.Node == tm.core.Name() { + exitProcessID[t] = append(exitProcessID[t], key.consumer) } - case gen.Alias: - if t.Node == node { - shouldRemove = true - if key.consumer.Node == tm.core.Name() { - exitAlias[t] = append(exitAlias[t], key.consumer) - } + if t.Node != node { + break + } + shouldRemove = true + if key.consumer.Node == tm.core.Name() { + exitAlias[t] = append(exitAlias[t], key.consumer) } - case gen.Event: - if t.Node == node { - shouldRemove = true - if key.consumer.Node == tm.core.Name() { - exitEvent[t] = append(exitEvent[t], key.consumer) - } + if t.Node != node { + break + } + shouldRemove = true + if key.consumer.Node == tm.core.Name() { + exitEvent[t] = append(exitEvent[t], key.consumer) } - case gen.Atom: - if t == node { - shouldRemove = true - if key.consumer.Node == tm.core.Name() { - exitNode[t] = append(exitNode[t], key.consumer) - } + if t != node { + break + } + shouldRemove = true + if key.consumer.Node == tm.core.Name() { + exitNode[t] = append(exitNode[t], key.consumer) } } } @@ -354,29 +342,26 @@ func (tm *targetManager) TerminatedTargetNode(node gen.Atom, reason error) { continue } - delete(tm.linkRelations, key) + delete(s.linkRelations, key) - entry := tm.targetIndex[key.target] + entry := s.targetIndex[key.target] if entry == nil { continue } delete(entry.consumers, key.consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, key.target) + delete(s.targetIndex, key.target) } } - // Collect down messages by type downPID := make(map[gen.PID][]gen.PID) downProcessID := make(map[gen.ProcessID][]gen.PID) downAlias := make(map[gen.Alias][]gen.PID) downEvent := make(map[gen.Event][]gen.PID) downNode := make(map[gen.Atom][]gen.PID) - // Cleanup monitorRelations - for key := range tm.monitorRelations { + for key := range s.monitorRelations { shouldRemove := false if key.consumer.Node == node { @@ -386,43 +371,44 @@ func (tm *targetManager) TerminatedTargetNode(node gen.Atom, reason error) { if shouldRemove == false { switch t := key.target.(type) { case gen.PID: - if t.Node == node { - shouldRemove = true - if key.consumer.Node == tm.core.Name() { - downPID[t] = append(downPID[t], key.consumer) - } + if t.Node != node { + break + } + shouldRemove = true + if key.consumer.Node == tm.core.Name() { + downPID[t] = append(downPID[t], key.consumer) } - case gen.ProcessID: - if t.Node == node { - shouldRemove = true - if key.consumer.Node == tm.core.Name() { - downProcessID[t] = append(downProcessID[t], key.consumer) - } + if t.Node != node { + break + } + shouldRemove = true + if key.consumer.Node == tm.core.Name() { + downProcessID[t] = append(downProcessID[t], key.consumer) } - case gen.Alias: - if t.Node == node { - shouldRemove = true - if key.consumer.Node == tm.core.Name() { - downAlias[t] = append(downAlias[t], key.consumer) - } + if t.Node != node { + break + } + shouldRemove = true + if key.consumer.Node == tm.core.Name() { + downAlias[t] = append(downAlias[t], key.consumer) } - case gen.Event: - if t.Node == node { - shouldRemove = true - if key.consumer.Node == tm.core.Name() { - downEvent[t] = append(downEvent[t], key.consumer) - } + if t.Node != node { + break + } + shouldRemove = true + if key.consumer.Node == tm.core.Name() { + downEvent[t] = append(downEvent[t], key.consumer) } - case gen.Atom: - if t == node { - shouldRemove = true - if key.consumer.Node == tm.core.Name() { - downNode[t] = append(downNode[t], key.consumer) - } + if t != node { + break + } + shouldRemove = true + if key.consumer.Node == tm.core.Name() { + downNode[t] = append(downNode[t], key.consumer) } } } @@ -431,21 +417,30 @@ func (tm *targetManager) TerminatedTargetNode(node gen.Atom, reason error) { continue } - delete(tm.monitorRelations, key) + delete(s.monitorRelations, key) - entry := tm.targetIndex[key.target] + entry := s.targetIndex[key.target] if entry == nil { continue } delete(entry.consumers, key.consumer) - if len(entry.consumers) == 0 { - delete(tm.targetIndex, key.target) + delete(s.targetIndex, key.target) } } - // Send exit messages (batch per target) + // Cleanup events from terminated node + for event, entry := range s.events { + if event.Node == node { + tm.eventIndex.Delete(entry.id) + delete(s.events, event) + } + } + + s.mutex.Unlock() + + // Dispatch exit messages for target, consumers := range exitPID { tm.exitSignalsProduced.Add(1) tm.core.RouteSendExitMessages(tm.core.PID(), consumers, gen.MessageExitPID{PID: target, Reason: gen.ErrNoConnection}) @@ -472,7 +467,7 @@ func (tm *targetManager) TerminatedTargetNode(node gen.Atom, reason error) { tm.exitSignalsDelivered.Add(int64(len(consumers))) } - // Send down messages + // Dispatch down messages for target, consumers := range downPID { tm.downMessagesProduced.Add(1) for _, consumer := range consumers { @@ -508,317 +503,243 @@ func (tm *targetManager) TerminatedTargetNode(node gen.Atom, reason error) { } tm.downMessagesDelivered.Add(int64(len(consumers))) } - - // Cleanup events from terminated node - for event := range tm.events { - if event.Node == node { - delete(tm.events, event) - } - } } func (tm *targetManager) TerminatedProcess(pid gen.PID, reason error) { - tm.mutex.Lock() - defer tm.mutex.Unlock() + for i := range tm.shards { + tm.terminateProcessInShard(&tm.shards[i], pid, reason) + } +} - // CleanupConsumer - cleanup all subscriptions this process had +func (tm *targetManager) terminateProcessInShard(s *shard, pid gen.PID, reason error) { + s.mutex.Lock() - // Process linkRelations - for key := range tm.linkRelations { + for key := range s.linkRelations { if key.consumer != pid { continue } - delete(tm.linkRelations, key) - - // Handle events separately (need to decrement counter) - if event, ok := key.target.(gen.Event); ok { - if event.Node == tm.core.Name() { - entry := tm.events[event] - if entry != nil { - // Swap-delete from linkSubscribers - if idx, exists := entry.linkSubscribersIndex[pid]; exists { - last := len(entry.linkSubscribers) - 1 - if idx != last { - entry.linkSubscribers[idx] = entry.linkSubscribers[last] - entry.linkSubscribersIndex[entry.linkSubscribers[idx]] = idx - } - entry.linkSubscribers = entry.linkSubscribers[:last] - delete(entry.linkSubscribersIndex, pid) - } - entry.subscriberCount-- + delete(s.linkRelations, key) - if entry.subscriberCount == 0 && entry.notify { - tm.core.RouteSendPID(tm.core.PID(), entry.producer, gen.MessageOptions{Priority: gen.MessagePriorityHigh}, gen.MessageEventStop{Name: event.Name}) + event, ok := key.target.(gen.Event) + if ok == true && event.Node == tm.core.Name() { + entry := s.events[event] + if entry != nil { + idx, exists := entry.linkSubscribersIndex[pid] + if exists == true { + last := len(entry.linkSubscribers) - 1 + if idx != last { + entry.linkSubscribers[idx] = entry.linkSubscribers[last] + entry.linkSubscribersIndex[entry.linkSubscribers[idx]] = idx } + entry.linkSubscribers = entry.linkSubscribers[:last] + delete(entry.linkSubscribersIndex, pid) + } + entry.subscriberCount-- + if entry.subscriberCount == 0 && entry.notify { + tm.core.RouteSendPID(tm.core.PID(), entry.producer, gen.MessageOptions{Priority: gen.MessagePriorityHigh}, gen.MessageEventStop{Name: event.Name}) } } } - entry := tm.targetIndex[key.target] - if entry == nil { + tm.cleanupTargetEntry(s, pid, key.target, true) + } + + for key := range s.monitorRelations { + if key.consumer != pid { continue } - delete(entry.consumers, key.consumer) - - isLast := (len(entry.consumers) == 0) + delete(s.monitorRelations, key) - if isLast { - delete(tm.targetIndex, key.target) + event, ok := key.target.(gen.Event) + if ok == true && event.Node == tm.core.Name() { + entry := s.events[event] + if entry != nil { + idx, exists := entry.monitorSubscribersIndex[pid] + if exists == true { + last := len(entry.monitorSubscribers) - 1 + if idx != last { + entry.monitorSubscribers[idx] = entry.monitorSubscribers[last] + entry.monitorSubscribersIndex[entry.monitorSubscribers[idx]] = idx + } + entry.monitorSubscribers = entry.monitorSubscribers[:last] + delete(entry.monitorSubscribersIndex, pid) + } + entry.subscriberCount-- + if entry.subscriberCount == 0 && entry.notify { + tm.core.RouteSendPID(tm.core.PID(), entry.producer, gen.MessageOptions{Priority: gen.MessagePriorityHigh}, gen.MessageEventStop{Name: event.Name}) + } + } } - // Check if target is remote and need to send Unlink - isRemote := false - var targetNode gen.Atom + tm.cleanupTargetEntry(s, pid, key.target, false) + } - switch t := key.target.(type) { - case gen.PID: - targetNode = t.Node - isRemote = (t.Node != tm.core.Name()) + // Producer cleanup + tm.cleanupProducerInShard(s, pid, reason) - case gen.ProcessID: - if t.Node == "" { - targetNode = tm.core.Name() - } else { - targetNode = t.Node - } - isRemote = (targetNode != tm.core.Name()) + s.mutex.Unlock() +} - case gen.Alias: - targetNode = t.Node - isRemote = (t.Node != tm.core.Name()) +// cleanupTargetEntry handles targetIndex update and remote unlink/demonitor. +func (tm *targetManager) cleanupTargetEntry(s *shard, consumer gen.PID, target any, isLink bool) { + entry := s.targetIndex[target] + if entry == nil { + return + } - case gen.Event: - targetNode = t.Node - isRemote = (t.Node != tm.core.Name()) + delete(entry.consumers, consumer) + isLast := (len(entry.consumers) == 0) + if isLast { + delete(s.targetIndex, target) + } - case gen.Atom: - isRemote = false - } + isRemote := false + var targetNode gen.Atom - if isRemote == false { - continue + switch t := target.(type) { + case gen.PID: + targetNode = t.Node + isRemote = (t.Node != tm.core.Name()) + case gen.ProcessID: + if t.Node == "" { + targetNode = tm.core.Name() + } else { + targetNode = t.Node } + isRemote = (targetNode != tm.core.Name()) + case gen.Alias: + targetNode = t.Node + isRemote = (t.Node != tm.core.Name()) + case gen.Event: + targetNode = t.Node + isRemote = (t.Node != tm.core.Name()) + case gen.Atom: + isRemote = false + } - // Remote target - check if last local consumer - if isLast == false { - hasLocal := false - for p := range entry.consumers { - if p.Node == tm.core.Name() && p != tm.core.PID() { - hasLocal = true - break - } - } + if isRemote == false { + return + } - if hasLocal { - continue + if isLast == false { + hasLocal := false + for p := range entry.consumers { + if p.Node == tm.core.Name() && p != tm.core.PID() { + hasLocal = true + break } } - - // Last local consumer - send remote Unlink - connection, err := tm.core.GetConnection(targetNode) - if err != nil { - continue + if hasLocal { + return } + } - switch t := key.target.(type) { + connection, err := tm.core.GetConnection(targetNode) + if err != nil { + return + } + + if isLink { + switch t := target.(type) { case gen.PID: connection.UnlinkPID(tm.core.PID(), t) - case gen.ProcessID: connection.UnlinkProcessID(tm.core.PID(), t) - case gen.Alias: connection.UnlinkAlias(tm.core.PID(), t) - case gen.Event: connection.UnlinkEvent(tm.core.PID(), t) } - } - - // Process monitorRelations - for key := range tm.monitorRelations { - if key.consumer != pid { - continue - } - - delete(tm.monitorRelations, key) - - // Handle events - if event, ok := key.target.(gen.Event); ok { - if event.Node == tm.core.Name() { - entry := tm.events[event] - if entry != nil { - // Swap-delete from monitorSubscribers - if idx, exists := entry.monitorSubscribersIndex[pid]; exists { - last := len(entry.monitorSubscribers) - 1 - if idx != last { - entry.monitorSubscribers[idx] = entry.monitorSubscribers[last] - entry.monitorSubscribersIndex[entry.monitorSubscribers[idx]] = idx - } - entry.monitorSubscribers = entry.monitorSubscribers[:last] - delete(entry.monitorSubscribersIndex, pid) - } - entry.subscriberCount-- - - if entry.subscriberCount == 0 && entry.notify { - tm.core.RouteSendPID(tm.core.PID(), entry.producer, gen.MessageOptions{Priority: gen.MessagePriorityHigh}, gen.MessageEventStop{Name: event.Name}) - } - } - } - } - - entry := tm.targetIndex[key.target] - if entry == nil { - continue - } - - delete(entry.consumers, key.consumer) - - isLast := (len(entry.consumers) == 0) - - if isLast { - delete(tm.targetIndex, key.target) - } - - // Check remote and send Demonitor - isRemote := false - var targetNode gen.Atom - - switch t := key.target.(type) { - case gen.PID: - targetNode = t.Node - isRemote = (t.Node != tm.core.Name()) - - case gen.ProcessID: - if t.Node == "" { - targetNode = tm.core.Name() - } else { - targetNode = t.Node - } - isRemote = (targetNode != tm.core.Name()) - - case gen.Alias: - targetNode = t.Node - isRemote = (t.Node != tm.core.Name()) - - case gen.Event: - targetNode = t.Node - isRemote = (t.Node != tm.core.Name()) - - case gen.Atom: - isRemote = false - } - - if isRemote == false { - continue - } - - if isLast == false { - hasLocal := false - for p := range entry.consumers { - if p.Node == tm.core.Name() && p != tm.core.PID() { - hasLocal = true - break - } - } - - if hasLocal { - continue - } - } - - // Last local consumer - send remote Demonitor - connection, err := tm.core.GetConnection(targetNode) - if err != nil { - continue - } - - switch t := key.target.(type) { + } else { + switch t := target.(type) { case gen.PID: connection.DemonitorPID(tm.core.PID(), t) - case gen.ProcessID: connection.DemonitorProcessID(tm.core.PID(), t) - case gen.Alias: connection.DemonitorAlias(tm.core.PID(), t) - case gen.Event: connection.DemonitorEvent(tm.core.PID(), t) } } +} - // Cleanup events owned by terminated process (PRODUCER cleanup) - if events := tm.producerEvents[pid]; events != nil { - remoteEvents := make(map[gen.Atom][]gen.Event) +// cleanupProducerInShard handles events owned by terminated process within a shard. +func (tm *targetManager) cleanupProducerInShard(s *shard, pid gen.PID, reason error) { + events := s.producerEvents[pid] + if events == nil { + return + } - for event := range events { - entry := tm.events[event] - if entry == nil { - continue - } + remoteEvents := make(map[gen.Atom][]gen.Event) - // Send exit to link subscribers - var localExitConsumers []gen.PID - for _, consumer := range entry.linkSubscribers { - if consumer.Node != tm.core.Name() { - remoteEvents[consumer.Node] = append(remoteEvents[consumer.Node], event) - continue - } - localExitConsumers = append(localExitConsumers, consumer) - } - if len(localExitConsumers) > 0 { - tm.exitSignalsProduced.Add(1) - tm.core.RouteSendExitMessages(tm.core.PID(), localExitConsumers, gen.MessageExitEvent{Event: event, Reason: reason}) - tm.exitSignalsDelivered.Add(int64(len(localExitConsumers))) - } + for event := range events { + entry := s.events[event] + if entry == nil { + continue + } - // Send down to monitor subscribers - var localDownConsumers []gen.PID - for _, consumer := range entry.monitorSubscribers { - if consumer.Node != tm.core.Name() { - remoteEvents[consumer.Node] = append(remoteEvents[consumer.Node], event) - continue - } - localDownConsumers = append(localDownConsumers, consumer) - } - if len(localDownConsumers) > 0 { - tm.downMessagesProduced.Add(1) - for _, consumer := range localDownConsumers { - tm.core.RouteSendPID(tm.core.PID(), consumer, gen.MessageOptions{Priority: gen.MessagePriorityHigh}, gen.MessageDownEvent{Event: event, Reason: reason}) - } - tm.downMessagesDelivered.Add(int64(len(localDownConsumers))) + // Send exit to link subscribers + var localExitConsumers []gen.PID + for _, consumer := range entry.linkSubscribers { + if consumer.Node != tm.core.Name() { + remoteEvents[consumer.Node] = append(remoteEvents[consumer.Node], event) + continue } + localExitConsumers = append(localExitConsumers, consumer) + } + if len(localExitConsumers) > 0 { + tm.exitSignalsProduced.Add(1) + tm.core.RouteSendExitMessages(tm.core.PID(), localExitConsumers, gen.MessageExitEvent{Event: event, Reason: reason}) + tm.exitSignalsDelivered.Add(int64(len(localExitConsumers))) + } - // Cleanup relations for this event - for key := range tm.linkRelations { - if key.target == event { - delete(tm.linkRelations, key) - } + // Send down to monitor subscribers + var localDownConsumers []gen.PID + for _, consumer := range entry.monitorSubscribers { + if consumer.Node != tm.core.Name() { + remoteEvents[consumer.Node] = append(remoteEvents[consumer.Node], event) + continue } - for key := range tm.monitorRelations { - if key.target == event { - delete(tm.monitorRelations, key) - } + localDownConsumers = append(localDownConsumers, consumer) + } + if len(localDownConsumers) > 0 { + tm.downMessagesProduced.Add(1) + for _, consumer := range localDownConsumers { + tm.core.RouteSendPID(tm.core.PID(), consumer, gen.MessageOptions{Priority: gen.MessagePriorityHigh}, gen.MessageDownEvent{Event: event, Reason: reason}) } - - delete(tm.targetIndex, event) - delete(tm.events, event) + tm.downMessagesDelivered.Add(int64(len(localDownConsumers))) } - // Send to remote nodes - for remoteNode, nodeEvents := range remoteEvents { - connection, err := tm.core.GetConnection(remoteNode) - if err != nil { - continue + // Cleanup relations for this event + for key := range s.linkRelations { + if key.target == event { + delete(s.linkRelations, key) } - for _, event := range nodeEvents { - connection.SendTerminateEvent(event, reason) + } + for key := range s.monitorRelations { + if key.target == event { + delete(s.monitorRelations, key) } } - delete(tm.producerEvents, pid) + tm.eventIndex.Delete(entry.id) + delete(s.targetIndex, event) + delete(s.events, event) + } + + // Send to remote nodes + for remoteNode, nodeEvents := range remoteEvents { + connection, err := tm.core.GetConnection(remoteNode) + if err != nil { + continue + } + for _, event := range nodeEvents { + connection.SendTerminateEvent(event, reason) + } } + + delete(s.producerEvents, pid) } diff --git a/node/tm/terminate_test.go b/node/tm/terminate_test.go index d329e83b2..11028e286 100644 --- a/node/tm/terminate_test.go +++ b/node/tm/terminate_test.go @@ -34,13 +34,12 @@ func TestTerminatedTargetPID_LinkSubscribers(t *testing.T) { } // Verify subscriptions cleaned - key1 := relationKey{consumer: consumer1, target: target} - if _, exists := tm.linkRelations[key1]; exists { + if exists := tm.hasLinkRelation(consumer1, target); exists { t.Error("Link should be removed after target terminated") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned") } } @@ -72,8 +71,8 @@ func TestTerminatedTargetPID_MonitorSubscribers(t *testing.T) { } // Verify cleaned - if len(tm.monitorRelations) != 0 { - t.Errorf("monitorRelations should be empty, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 0 { + t.Errorf("monitorRelations should be empty, got %d", tm.totalMonitors()) } } @@ -108,17 +107,17 @@ func TestTerminatedTargetPID_Mixed(t *testing.T) { } // Verify linkRelations cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("linkRelations should be empty, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("linkRelations should be empty, got %d", tm.totalLinks()) } // Verify monitorRelations cleaned - if len(tm.monitorRelations) != 0 { - t.Errorf("monitorRelations should be empty, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 0 { + t.Errorf("monitorRelations should be empty, got %d", tm.totalMonitors()) } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned") } } @@ -145,13 +144,12 @@ func TestTerminatedTargetProcessID(t *testing.T) { } // Cleaned from linkRelations - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("Link should be removed") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned") } } @@ -177,13 +175,12 @@ func TestTerminatedTargetAlias(t *testing.T) { } // Verify linkRelations cleaned - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(consumer, target); exists { t.Error("Link should be removed") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned") } } @@ -198,11 +195,12 @@ func TestTerminatedTargetNode_RemoteConsumer(t *testing.T) { // Remote consumer links to local target // (Simulates: node2's CorePID linked to node1's process) - tm.linkRelations[relationKey{consumer: remoteConsumer, target: localTarget}] = struct{}{} + s := tm.shardFor(localTarget) + s.linkRelations[relationKey{consumer: remoteConsumer, target: localTarget}] = struct{}{} entry := &targetEntry{consumers: make(map[gen.PID]struct{})} entry.consumers[remoteConsumer] = struct{}{} - tm.targetIndex[localTarget] = entry + s.targetIndex[localTarget] = entry core.resetSentExits() @@ -217,8 +215,7 @@ func TestTerminatedTargetNode_RemoteConsumer(t *testing.T) { } // But subscription cleaned! - key := relationKey{consumer: remoteConsumer, target: localTarget} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(remoteConsumer, localTarget); exists { t.Error("Link from remote consumer should be removed") } } @@ -247,13 +244,12 @@ func TestTerminatedTargetNode_RemoteTarget(t *testing.T) { } // Cleaned from linkRelations - key := relationKey{consumer: localConsumer, target: remoteTarget} - if _, exists := tm.linkRelations[key]; exists { + if exists := tm.hasLinkRelation(localConsumer, remoteTarget); exists { t.Error("Link should be removed") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[remoteTarget]; exists { + if tm.getTargetEntry(remoteTarget) != nil { t.Error("targetIndex should be cleaned") } } @@ -273,7 +269,8 @@ func TestTerminatedTargetNode_Mixed(t *testing.T) { tm.LinkPID(localConsumer, remoteTarget) // Remote → Local (manual setup) - tm.linkRelations[relationKey{consumer: remoteConsumer, target: localTarget}] = struct{}{} + s := tm.shardFor(localTarget) + s.linkRelations[relationKey{consumer: remoteConsumer, target: localTarget}] = struct{}{} core.resetSentExits() @@ -288,12 +285,12 @@ func TestTerminatedTargetNode_Mixed(t *testing.T) { } // Both links cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("All links should be cleaned, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("All links should be cleaned, got %d", tm.totalLinks()) } // Verify targetIndex cleaned for remoteTarget - if _, exists := tm.targetIndex[remoteTarget]; exists { + if tm.getTargetEntry(remoteTarget) != nil { t.Error("targetIndex for remoteTarget should be cleaned") } } @@ -305,7 +302,8 @@ func TestTerminatedTargetNode_EventsCleaned(t *testing.T) { // Manually add event from node2 event := gen.Event{Node: "node2", Name: "test"} - tm.events[event] = &eventEntry{ + s := tm.shardFor(event) + s.events[event] = &eventEntry{ producer: gen.PID{Node: "node2", ID: 100}, } @@ -315,7 +313,7 @@ func TestTerminatedTargetNode_EventsCleaned(t *testing.T) { time.Sleep(50 * time.Millisecond) // Event cleaned - if _, exists := tm.events[event]; exists { + if tm.getEventEntry(event) != nil { t.Error("Event from terminated node should be removed") } } @@ -342,13 +340,12 @@ func TestTerminatedTargetProcessID_Monitors(t *testing.T) { } // Verify monitorRelations cleaned - key := relationKey{consumer: consumer, target: target} - if _, exists := tm.monitorRelations[key]; exists { + if exists := tm.hasMonitorRelation(consumer, target); exists { t.Error("Monitor should be removed") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned") } } @@ -374,7 +371,7 @@ func TestTerminatedTargetAlias_Monitors(t *testing.T) { } // Cleaned - if len(tm.monitorRelations) != 0 { + if tm.totalMonitors() != 0 { t.Error("monitorRelations should be empty") } } @@ -438,18 +435,18 @@ func TestTerminatedTargetNode_MultipleTypes(t *testing.T) { } // All links cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("All links should be cleaned, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("All links should be cleaned, got %d", tm.totalLinks()) } // Verify all targetIndex entries cleaned - if _, exists := tm.targetIndex[pidTarget]; exists { + if tm.getTargetEntry(pidTarget) != nil { t.Error("targetIndex for pidTarget should be cleaned") } - if _, exists := tm.targetIndex[processIDTarget]; exists { + if tm.getTargetEntry(processIDTarget) != nil { t.Error("targetIndex for processIDTarget should be cleaned") } - if _, exists := tm.targetIndex[aliasTarget]; exists { + if tm.getTargetEntry(aliasTarget) != nil { t.Error("targetIndex for aliasTarget should be cleaned") } } @@ -493,8 +490,8 @@ func TestTerminatedProcess_CleanupLinks(t *testing.T) { time.Sleep(50 * time.Millisecond) // Links cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("All links should be cleaned, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("All links should be cleaned, got %d", tm.totalLinks()) } // Remote Unlink sent @@ -503,10 +500,10 @@ func TestTerminatedProcess_CleanupLinks(t *testing.T) { } // Verify targetIndex cleaned for both targets - if _, exists := tm.targetIndex[target1]; exists { + if tm.getTargetEntry(target1) != nil { t.Error("targetIndex for target1 should be cleaned") } - if _, exists := tm.targetIndex[target2]; exists { + if tm.getTargetEntry(target2) != nil { t.Error("targetIndex for target2 should be cleaned") } } @@ -540,7 +537,7 @@ func TestTerminatedProcess_LastSubscriber_EventStop(t *testing.T) { } // Counter = 0 - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry.subscriberCount != 0 { t.Errorf("subscriberCount should be 0, got %d", entry.subscriberCount) } @@ -576,20 +573,18 @@ func TestTerminatedProcess_NotLast_NoEventStop(t *testing.T) { } // Counter = 1 - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry.subscriberCount != 1 { t.Errorf("subscriberCount should be 1, got %d", entry.subscriberCount) } // Verify consumer1 removed from linkRelations - key1 := relationKey{consumer: consumer1, target: event} - if _, exists := tm.linkRelations[key1]; exists { + if exists := tm.hasLinkRelation(consumer1, event); exists { t.Error("consumer1 link should be removed") } // Verify consumer2 still in linkRelations - key2 := relationKey{consumer: consumer2, target: event} - if _, exists := tm.linkRelations[key2]; exists == false { + if exists := tm.hasLinkRelation(consumer2, event); exists == false { t.Error("consumer2 link should still exist") } } @@ -613,12 +608,12 @@ func TestTerminatedProcess_RemoteEvent_LastLocal_SendsUnlink(t *testing.T) { time.Sleep(50 * time.Millisecond) // Verify linkRelations cleaned - if len(tm.linkRelations) != 0 { + if tm.totalLinks() != 0 { t.Error("Link should be cleaned") } // Verify targetIndex cleaned (last local subscriber) - if _, exists := tm.targetIndex[event]; exists { + if tm.getTargetEntry(event) != nil { t.Error("targetIndex should be cleaned") } } @@ -641,7 +636,7 @@ func TestTerminatedProcess_Event_LinkAndMonitor(t *testing.T) { time.Sleep(20 * time.Millisecond) // Counter = 2 (link + monitor) - entry := tm.events[event] + entry := tm.getEventEntry(event) if entry.subscriberCount != 2 { t.Errorf("subscriberCount should be 2, got %d", entry.subscriberCount) } @@ -664,14 +659,12 @@ func TestTerminatedProcess_Event_LinkAndMonitor(t *testing.T) { } // Verify linkRelations cleaned for consumer - linkKey := relationKey{consumer: consumer, target: event} - if _, exists := tm.linkRelations[linkKey]; exists { + if exists := tm.hasLinkRelation(consumer, event); exists { t.Error("Link should be removed") } // Verify monitorRelations cleaned for consumer - monitorKey := relationKey{consumer: consumer, target: event} - if _, exists := tm.monitorRelations[monitorKey]; exists { + if exists := tm.hasMonitorRelation(consumer, event); exists { t.Error("Monitor should be removed") } } @@ -705,12 +698,12 @@ func TestTerminatedEvent_LinkSubscribers(t *testing.T) { } // Event removed from tm.events - if _, exists := tm.events[event]; exists { + if tm.getEventEntry(event) != nil { t.Error("Event should be removed") } // Subscriptions cleaned - if len(tm.linkRelations) != 0 { + if tm.totalLinks() != 0 { t.Error("linkRelations should be empty") } } @@ -744,7 +737,7 @@ func TestTerminatedEvent_MonitorSubscribers(t *testing.T) { } // Cleaned - if len(tm.monitorRelations) != 0 { + if tm.totalMonitors() != 0 { t.Error("monitorRelations should be empty") } } @@ -783,18 +776,18 @@ func TestTerminatedEvent_Mixed(t *testing.T) { } // Event removed - if _, exists := tm.events[event]; exists { + if tm.getEventEntry(event) != nil { t.Error("Event should be removed") } // Verify linkRelations cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("linkRelations should be empty, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("linkRelations should be empty, got %d", tm.totalLinks()) } // Verify monitorRelations cleaned - if len(tm.monitorRelations) != 0 { - t.Errorf("monitorRelations should be empty, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 0 { + t.Errorf("monitorRelations should be empty, got %d", tm.totalMonitors()) } } @@ -822,7 +815,7 @@ func TestTerminatedEvent_NoSubscribers(t *testing.T) { } // Event still removed - if _, exists := tm.events[event]; exists { + if tm.getEventEntry(event) != nil { t.Error("Event should be removed") } } @@ -840,9 +833,9 @@ func TestTerminatedTargetPID_RemoteCorePIDSubscriber(t *testing.T) { tm.LinkPID(localConsumer, target) // Simulate remote CorePID also linked (from node2) - key := relationKey{consumer: remoteCorePID, target: target} - tm.linkRelations[key] = struct{}{} - entry := tm.targetIndex[target] + s := tm.shardFor(target) + s.linkRelations[relationKey{consumer: remoteCorePID, target: target}] = struct{}{} + entry := tm.getTargetEntry(target) if entry != nil { entry.consumers[remoteCorePID] = struct{}{} } @@ -860,12 +853,12 @@ func TestTerminatedTargetPID_RemoteCorePIDSubscriber(t *testing.T) { } // Both links cleaned - if len(tm.linkRelations) != 0 { + if tm.totalLinks() != 0 { t.Error("All links should be cleaned") } // Verify targetIndex cleaned - if _, exists := tm.targetIndex[target]; exists { + if tm.getTargetEntry(target) != nil { t.Error("targetIndex should be cleaned") } } @@ -901,12 +894,12 @@ func TestTerminatedProcess_ProducerCleanup_LinkSubscribers(t *testing.T) { } // Event removed - if _, exists := tm.events[event]; exists { + if tm.getEventEntry(event) != nil { t.Error("Event should be removed after producer terminates") } // producerEvents index cleaned - if tm.producerEvents[producer] != nil { + if tm.getProducerEventsMap(producer) != nil { t.Error("producerEvents index should be cleaned") } } @@ -942,17 +935,17 @@ func TestTerminatedProcess_ProducerCleanup_MonitorSubscribers(t *testing.T) { } // Event removed - if _, exists := tm.events[event]; exists { + if tm.getEventEntry(event) != nil { t.Error("Event should be removed after producer terminates") } // Verify monitorRelations cleaned - if len(tm.monitorRelations) != 0 { - t.Errorf("monitorRelations should be empty, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 0 { + t.Errorf("monitorRelations should be empty, got %d", tm.totalMonitors()) } // Verify producerEvents cleaned - if tm.producerEvents[producer] != nil { + if tm.getProducerEventsMap(producer) != nil { t.Error("producerEvents index should be cleaned") } } @@ -992,18 +985,18 @@ func TestTerminatedProcess_ProducerCleanup_MultipleEvents(t *testing.T) { } // All events removed - if len(tm.events) != 0 { - t.Errorf("All events should be removed, got %d", len(tm.events)) + if tm.totalEvents() != 0 { + t.Errorf("All events should be removed, got %d", tm.totalEvents()) } // producerEvents index cleaned - if tm.producerEvents[producer] != nil { + if tm.getProducerEventsMap(producer) != nil { t.Error("producerEvents index should be cleaned") } // Verify linkRelations cleaned - if len(tm.linkRelations) != 0 { - t.Errorf("linkRelations should be empty, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 0 { + t.Errorf("linkRelations should be empty, got %d", tm.totalLinks()) } } @@ -1024,11 +1017,11 @@ func TestTerminatedProcess_ProducerCleanup_RelationsCleaned(t *testing.T) { tm.MonitorEvent(consumer, event) // Verify relations exist - if len(tm.linkRelations) != 1 { - t.Fatalf("Expected 1 link relation, got %d", len(tm.linkRelations)) + if tm.totalLinks() != 1 { + t.Fatalf("Expected 1 link relation, got %d", tm.totalLinks()) } - if len(tm.monitorRelations) != 1 { - t.Fatalf("Expected 1 monitor relation, got %d", len(tm.monitorRelations)) + if tm.totalMonitors() != 1 { + t.Fatalf("Expected 1 monitor relation, got %d", tm.totalMonitors()) } // Producer terminates @@ -1037,15 +1030,11 @@ func TestTerminatedProcess_ProducerCleanup_RelationsCleaned(t *testing.T) { time.Sleep(50 * time.Millisecond) // Relations for the event should be cleaned - for key := range tm.linkRelations { - if key.target == event { - t.Error("Link relation for event should be cleaned") - } + if exists := tm.hasLinkRelation(consumer, event); exists { + t.Error("Link relation for event should be cleaned") } - for key := range tm.monitorRelations { - if key.target == event { - t.Error("Monitor relation for event should be cleaned") - } + if exists := tm.hasMonitorRelation(consumer, event); exists { + t.Error("Monitor relation for event should be cleaned") } } @@ -1086,7 +1075,7 @@ func TestTerminatedProcess_ProducerCleanup_RemoteSubscribers(t *testing.T) { tm.LinkEvent(localConsumer, event) // Simulate remote subscriber (as if from node2 via network) - entry := tm.events[event] + entry := tm.getEventEntry(event) entry.linkSubscribersIndex[remoteConsumer] = len(entry.linkSubscribers) entry.linkSubscribers = append(entry.linkSubscribers, remoteConsumer) @@ -1103,7 +1092,7 @@ func TestTerminatedProcess_ProducerCleanup_RemoteSubscribers(t *testing.T) { } // Event removed - if _, exists := tm.events[event]; exists { + if tm.getEventEntry(event) != nil { t.Error("Event should be removed") } } diff --git a/testing/tests/002_distributed/t009_simultaneous_connect_test.go b/testing/tests/002_distributed/t009_simultaneous_connect_test.go new file mode 100644 index 000000000..d2c8eb405 --- /dev/null +++ b/testing/tests/002_distributed/t009_simultaneous_connect_test.go @@ -0,0 +1,238 @@ +package distributed + +import ( + "fmt" + "sync" + "testing" + "time" + + "ergo.services/ergo" + "ergo.services/ergo/gen" +) + +func TestT9SimultaneousConnect(t *testing.T) { + // Test: two nodes connect to each other at the same time. + // One side wins via connect(), the other gets its connection via accept(). + // The rejected connect() returns error, but the accept path registers the connection. + // Verify: after both goroutines complete, both nodes have exactly one connection. + options1 := gen.NodeOptions{} + options1.Network.Cookie = "simconnect" + options1.Log.DefaultLogger.Disable = true + + options2 := gen.NodeOptions{} + options2.Network.Cookie = "simconnect" + options2.Log.DefaultLogger.Disable = true + + node1, err := ergo.StartNode("distT9node1simcon@localhost", options1) + if err != nil { + t.Fatal(err) + } + defer node1.Stop() + + node2, err := ergo.StartNode("distT9node2simcon@localhost", options2) + if err != nil { + t.Fatal(err) + } + defer node2.Stop() + + // simultaneously connect from both sides + var wg sync.WaitGroup + wg.Add(2) + go func() { + defer wg.Done() + node1.Network().GetNode(node2.Name()) + }() + go func() { + defer wg.Done() + node2.Network().GetNode(node1.Name()) + }() + wg.Wait() + + // the accept path on the rejected side needs a moment to finish + // its handshake and register the connection + time.Sleep(300 * time.Millisecond) + + // both nodes must have exactly one connection to each other + r1, err := node1.Network().Node(node2.Name()) + if err != nil { + t.Fatalf("node1 should have connection to node2: %s", err) + } + r2, err := node2.Network().Node(node1.Name()) + if err != nil { + t.Fatalf("node2 should have connection to node1: %s", err) + } + + // verify exactly one connection per node (no duplicates) + nodes1 := node1.Network().Nodes() + nodes2 := node2.Network().Nodes() + if len(nodes1) != 1 { + t.Fatalf("node1 should have exactly 1 connection, got %d", len(nodes1)) + } + if len(nodes2) != 1 { + t.Fatalf("node2 should have exactly 1 connection, got %d", len(nodes2)) + } + + // verify connection info is consistent + r1info := r1.Info() + r2info := r2.Info() + if r1info.Node != node2.Name() { + t.Fatal("connection info mismatch on node1") + } + if r2info.Node != node1.Name() { + t.Fatal("connection info mismatch on node2") + } +} + +func TestT9SimultaneousConnectNoFlag(t *testing.T) { + // Test: one node has EnableSimultaneousConnect, the other does not. + // Should fall back to current behavior (no collision detection). + options1 := gen.NodeOptions{} + options1.Network.Cookie = "simconnect2" + options1.Log.DefaultLogger.Disable = true + + options2 := gen.NodeOptions{} + options2.Network.Cookie = "simconnect2" + options2.Log.DefaultLogger.Disable = true + options2.Network.Flags = gen.NetworkFlags{ + Enable: true, + EnableRemoteSpawn: true, + EnableRemoteApplicationStart: true, + EnableProxyAccept: true, + EnableImportantDelivery: true, + EnableSimultaneousConnect: false, // explicitly disabled + } + + node1, err := ergo.StartNode("distT9node1noflag@localhost", options1) + if err != nil { + t.Fatal(err) + } + defer node1.Stop() + + node2, err := ergo.StartNode("distT9node2noflag@localhost", options2) + if err != nil { + t.Fatal(err) + } + defer node2.Stop() + + remote1, err := node1.Network().GetNode(node2.Name()) + if err != nil { + t.Fatalf("node1 -> node2 connection failed: %s", err) + } + + time.Sleep(100 * time.Millisecond) + + remote2, err := node2.Network().Node(node1.Name()) + if err != nil { + t.Fatalf("node2 should see node1: %s", err) + } + + if remote1.Name() != node2.Name() { + t.Fatal("incorrect remote1 node name") + } + if remote2.Name() != node1.Name() { + t.Fatal("incorrect remote2 node name") + } + + r1info := remote1.Info() + if r1info.NetworkFlags.EnableSimultaneousConnect == true { + t.Fatal("node2 should not have EnableSimultaneousConnect flag") + } + + r2info := remote2.Info() + if r2info.NetworkFlags.EnableSimultaneousConnect == false { + t.Fatal("node1 should have EnableSimultaneousConnect flag") + } +} + +func TestT9SimultaneousConnectCluster(t *testing.T) { + // Test: N nodes all connect to each other simultaneously. + // Reproduces the real-world scenario where a cluster of nodes + // starts up and every node tries to reach every other node at once. + // Verify: after the storm, every pair has exactly one connection, + // no dead loops, no leaked connections. + const N = 50 // 50 nodes = 1225 pairs + + cookie := "simcluster" + nodes := make([]gen.Node, N) + + for i := 0; i < N; i++ { + opts := gen.NodeOptions{} + opts.Network.Cookie = cookie + opts.Log.DefaultLogger.Disable = true + name := gen.Atom(fmt.Sprintf("distT9cluster%03d@localhost", i)) + nd, err := ergo.StartNode(name, opts) + if err != nil { + // stop already started nodes + for j := 0; j < i; j++ { + nodes[j].Stop() + } + t.Fatalf("failed to start node %d: %s", i, err) + } + nodes[i] = nd + } + defer func() { + for _, nd := range nodes { + nd.Stop() + } + }() + + // every node connects to every other node simultaneously + var wg sync.WaitGroup + for i := 0; i < N; i++ { + for j := 0; j < N; j++ { + if i == j { + continue + } + wg.Add(1) + go func(src, dst int) { + defer wg.Done() + nodes[src].Network().GetNode(nodes[dst].Name()) + }(i, j) + } + } + wg.Wait() + + // let accept paths finish + time.Sleep(3 * time.Second) + + // retry missing connections (TCP backlog overflow under 9900 concurrent dials + // can drop some connections -- same as in a real cluster, retry resolves it) + for retry := 0; retry < 3; retry++ { + missing := 0 + for i := 0; i < N; i++ { + for j := i + 1; j < N; j++ { + if _, err := nodes[i].Network().Node(nodes[j].Name()); err != nil { + missing++ + nodes[i].Network().GetNode(nodes[j].Name()) + } + } + } + if missing == 0 { + break + } + t.Logf("retry %d: %d missing connections, retrying...", retry+1, missing) + time.Sleep(time.Second) + } + + // verify: every node sees exactly N-1 peers + for i := 0; i < N; i++ { + peers := nodes[i].Network().Nodes() + if len(peers) != N-1 { + t.Fatalf("node %d has %d connections, expected %d", i, len(peers), N-1) + } + } + + // verify: every pair has a bidirectional connection + for i := 0; i < N; i++ { + for j := i + 1; j < N; j++ { + _, err := nodes[i].Network().Node(nodes[j].Name()) + if err != nil { + t.Fatalf("node %d -> node %d: no connection: %s", i, j, err) + } + _, err = nodes[j].Network().Node(nodes[i].Name()) + if err != nil { + t.Fatalf("node %d -> node %d: no connection: %s", j, i, err) + } + } + } +} diff --git a/testing/tests/002_distributed/t010_fragmentation_test.go b/testing/tests/002_distributed/t010_fragmentation_test.go new file mode 100644 index 000000000..e6c385233 --- /dev/null +++ b/testing/tests/002_distributed/t010_fragmentation_test.go @@ -0,0 +1,297 @@ +package distributed + +import ( + "fmt" + "reflect" + "testing" + "time" + + "ergo.services/ergo" + "ergo.services/ergo/act" + "ergo.services/ergo/gen" + "ergo.services/ergo/lib" +) + +var ( + t10pongCh chan any +) + +func factory_t10pong() gen.ProcessBehavior { + return &t10pong{} +} + +type t10pong struct { + act.Actor +} + +func (t *t10pong) HandleMessage(from gen.PID, message any) error { + select { + case t10pongCh <- message: + default: + } + return nil +} + +func factory_t10() gen.ProcessBehavior { + return &t10{} +} + +type t10 struct { + act.Actor + + remote gen.Atom + testcase *testcase +} + +func (t *t10) Init(args ...any) error { + t.remote = args[0].(gen.Atom) + return nil +} + +func (t *t10) HandleMessage(from gen.PID, message any) error { + if t.testcase == nil { + t.testcase = message.(*testcase) + message = initcase{} + } + + method := reflect.ValueOf(t).MethodByName(t.testcase.name) + if method.IsValid() == false { + t.testcase.err <- fmt.Errorf("unknown method %q", t.testcase.name) + t.testcase = nil + return nil + } + method.Call([]reflect.Value{reflect.ValueOf(message)}) + return nil +} + +// TestFragmentSendOrdered sends a large message with KeepNetworkOrder (order > 0). +// All fragments go to the same pool item -> same TCP -> same recv queue. +func (t *t10) TestFragmentSendOrdered(input any) { + defer func() { t.testcase = nil }() + + pid, err := t.RemoteSpawn(t.remote, "t10pong", gen.ProcessOptions{}) + if err != nil { + t.testcase.err <- err + return + } + + t10pongCh = make(chan any, 1) + + // 5000 bytes string, FragmentSize=1000 -> ~5 fragments + pingvalue := lib.RandomString(5000) + + if err := t.Send(pid, pingvalue); err != nil { + t.testcase.err <- err + return + } + + select { + case pong := <-t10pongCh: + if reflect.DeepEqual(pingvalue, pong) == false { + t.testcase.err <- fmt.Errorf("pong value mismatch (ordered)") + return + } + case <-time.NewTimer(5 * time.Second).C: + t.testcase.err <- gen.ErrTimeout + return + } + + t.testcase.err <- nil +} + +// TestFragmentSendUnordered sends a large message without KeepNetworkOrder (order = 0). +// Fragments round-robin across pool items -> different recv queues. +func (t *t10) TestFragmentSendUnordered(input any) { + defer func() { t.testcase = nil }() + + pid, err := t.RemoteSpawn(t.remote, "t10pong", gen.ProcessOptions{}) + if err != nil { + t.testcase.err <- err + return + } + + t10pongCh = make(chan any, 1) + + pingvalue := lib.RandomString(5000) + + t.SetKeepNetworkOrder(false) + defer t.SetKeepNetworkOrder(true) + + if err := t.Send(pid, pingvalue); err != nil { + t.testcase.err <- err + return + } + + select { + case pong := <-t10pongCh: + if reflect.DeepEqual(pingvalue, pong) == false { + t.testcase.err <- fmt.Errorf("pong value mismatch (unordered)") + return + } + case <-time.NewTimer(5 * time.Second).C: + t.testcase.err <- gen.ErrTimeout + return + } + + t.testcase.err <- nil +} + +// TestFragmentSendCompressed sends a large compressed message that still exceeds FragmentSize. +func (t *t10) TestFragmentSendCompressed(input any) { + defer func() { t.testcase = nil }() + + pid, err := t.RemoteSpawn(t.remote, "t10pong", gen.ProcessOptions{}) + if err != nil { + t.testcase.err <- err + return + } + + t10pongCh = make(chan any, 1) + + // large enough that even compressed it exceeds FragmentSize=1000 + pingvalue := lib.RandomString(10000) + + t.SetCompression(true) + defer t.SetCompression(false) + + if err := t.Send(pid, pingvalue); err != nil { + t.testcase.err <- err + return + } + + select { + case pong := <-t10pongCh: + if reflect.DeepEqual(pingvalue, pong) == false { + t.testcase.err <- fmt.Errorf("pong value mismatch (compressed+fragmented)") + return + } + case <-time.NewTimer(5 * time.Second).C: + t.testcase.err <- gen.ErrTimeout + return + } + + t.testcase.err <- nil +} + +// TestFragmentSendImportant sends a large important message. +// ACK must arrive after full reassembly. +func (t *t10) TestFragmentSendImportant(input any) { + defer func() { t.testcase = nil }() + + pid, err := t.RemoteSpawn(t.remote, "t10pong", gen.ProcessOptions{}) + if err != nil { + t.testcase.err <- err + return + } + + t10pongCh = make(chan any, 1) + + pingvalue := lib.RandomString(5000) + + if err := t.SendImportant(pid, pingvalue); err != nil { + t.testcase.err <- err + return + } + + select { + case pong := <-t10pongCh: + if reflect.DeepEqual(pingvalue, pong) == false { + t.testcase.err <- fmt.Errorf("pong value mismatch (important)") + return + } + case <-time.NewTimer(5 * time.Second).C: + t.testcase.err <- gen.ErrTimeout + return + } + + t.testcase.err <- nil +} + +// TestFragmentSendSmall sends a message smaller than FragmentSize. +// Should NOT be fragmented, just sent normally. +func (t *t10) TestFragmentSendSmall(input any) { + defer func() { t.testcase = nil }() + + pid, err := t.RemoteSpawn(t.remote, "t10pong", gen.ProcessOptions{}) + if err != nil { + t.testcase.err <- err + return + } + + t10pongCh = make(chan any, 1) + + // small message, no fragmentation + pingvalue := "hello" + + if err := t.Send(pid, pingvalue); err != nil { + t.testcase.err <- err + return + } + + select { + case pong := <-t10pongCh: + if reflect.DeepEqual(pingvalue, pong) == false { + t.testcase.err <- fmt.Errorf("pong value mismatch (small)") + return + } + case <-time.NewTimer(5 * time.Second).C: + t.testcase.err <- gen.ErrTimeout + return + } + + t.testcase.err <- nil +} + +func TestT10Fragmentation(t *testing.T) { + options1 := gen.NodeOptions{} + options1.Network.Cookie = "fragtest" + options1.Network.FragmentSize = 1000 + options1.Log.DefaultLogger.Disable = false + options1.Log.Level = gen.LogLevelTrace + node1, err := ergo.StartNode("distT10node1Frag@localhost", options1) + if err != nil { + t.Fatal(err) + } + defer node1.Stop() + + options2 := gen.NodeOptions{} + options2.Network.Cookie = "fragtest" + options2.Network.FragmentSize = 1000 + options2.Log.DefaultLogger.Disable = false + options2.Log.Level = gen.LogLevelTrace + node2, err := ergo.StartNode("distT10node2Frag@localhost", options2) + if err != nil { + t.Fatal(err) + } + defer node2.Stop() + + if err := node2.Network().EnableSpawn("t10pong", factory_t10pong); err != nil { + t.Fatal(err) + } + + // establish connection + if _, err := node1.Network().GetNode(node2.Name()); err != nil { + t.Fatal(err) + } + + pid, err := node1.Spawn(factory_t10, gen.ProcessOptions{}, node2.Name()) + if err != nil { + panic(err) + } + + t10cases := []*testcase{ + {"TestFragmentSendSmall", nil, nil, make(chan error)}, + {"TestFragmentSendOrdered", nil, nil, make(chan error)}, + {"TestFragmentSendUnordered", nil, nil, make(chan error)}, + {"TestFragmentSendCompressed", nil, nil, make(chan error)}, + {"TestFragmentSendImportant", nil, nil, make(chan error)}, + } + for _, tc := range t10cases { + t.Run(tc.name, func(t *testing.T) { + node1.Send(pid, tc) + if err := tc.wait(10); err != nil { + t.Fatal(err) + } + }) + } +} diff --git a/testing/tests/002_distributed/t011_fragmentation_load_test.go b/testing/tests/002_distributed/t011_fragmentation_load_test.go new file mode 100644 index 000000000..b9b387bfd --- /dev/null +++ b/testing/tests/002_distributed/t011_fragmentation_load_test.go @@ -0,0 +1,325 @@ +package distributed + +import ( + "fmt" + "sync" + "sync/atomic" + "testing" + "time" + + "ergo.services/ergo" + "ergo.services/ergo/act" + "ergo.services/ergo/gen" + "ergo.services/ergo/lib" +) + +// t11receiver counts received messages and signals when target is reached +type t11receiver struct { + act.Actor + + target int64 + count atomic.Int64 + done chan struct{} + mismatch atomic.Int64 +} + +var ( + t11done chan struct{} + t11target int64 + t11count atomic.Int64 + t11mismatch atomic.Int64 +) + +func factory_t11receiver() gen.ProcessBehavior { + return &t11receiver{} +} + +func (t *t11receiver) Init(args ...any) error { + return nil +} + +func (t *t11receiver) HandleMessage(from gen.PID, message any) error { + s, ok := message.(string) + if ok == false { + t11mismatch.Add(1) + return nil + } + // verify payload integrity: first 10 chars are the expected length + expected := fmt.Sprintf("%010d", len(s)) + if len(s) < 10 || s[:10] != expected { + t11mismatch.Add(1) + return nil + } + + n := t11count.Add(1) + if n >= t11target { + select { + case t11done <- struct{}{}: + default: + } + } + return nil +} + +// t11sender sends N messages of given size to remote pid +type t11sender struct { + act.Actor +} + +func factory_t11sender() gen.ProcessBehavior { + return &t11sender{} +} + +type t11sendJob struct { + target gen.PID + size int + count int + noOrder bool + done chan error +} + +func (t *t11sender) HandleMessage(from gen.PID, message any) error { + job, ok := message.(*t11sendJob) + if ok == false { + return nil + } + + if job.noOrder { + t.SetKeepNetworkOrder(false) + } + + for i := 0; i < job.count; i++ { + // create payload with size prefix for integrity check + s := lib.RandomString(job.size) + prefix := fmt.Sprintf("%010d", len(s)) + s = prefix + s[10:] + + if err := t.Send(job.target, s); err != nil { + job.done <- fmt.Errorf("send error at msg %d: %w", i, err) + return nil + } + } + job.done <- nil + return nil +} + +func TestT11FragmentationLoad(t *testing.T) { + t.Run("OrderedConcurrent", testFragLoadOrderedConcurrent) + t.Run("UnorderedConcurrent", testFragLoadUnorderedConcurrent) + t.Run("MixedOrderConcurrent", testFragLoadMixedConcurrent) +} + +// 10 senders, each sends 100 ordered large messages +func testFragLoadOrderedConcurrent(t *testing.T) { + node1, node2, receiverPID := setupFragLoadNodes(t, "Ordered") + defer node1.Stop() + defer node2.Stop() + + numSenders := 10 + msgsPerSender := 100 + msgSize := 5000 // ~6 fragments each at FragmentSize=1000 + totalMsgs := numSenders * msgsPerSender + + t11done = make(chan struct{}, 1) + t11target = int64(totalMsgs) + t11count.Store(0) + t11mismatch.Store(0) + + // spawn senders and fire jobs + var wg sync.WaitGroup + for i := 0; i < numSenders; i++ { + wg.Add(1) + go func() { + defer wg.Done() + pid, err := node1.Spawn(factory_t11sender, gen.ProcessOptions{}) + if err != nil { + t.Errorf("spawn sender: %v", err) + return + } + job := &t11sendJob{ + target: receiverPID, + size: msgSize, + count: msgsPerSender, + noOrder: false, + done: make(chan error, 1), + } + node1.Send(pid, job) + if err := <-job.done; err != nil { + t.Errorf("sender error: %v", err) + } + }() + } + + // wait for all sends to complete + wg.Wait() + + // wait for all messages to be received + select { + case <-t11done: + case <-time.After(30 * time.Second): + t.Fatalf("timeout: received %d/%d messages", t11count.Load(), totalMsgs) + } + + if m := t11mismatch.Load(); m > 0 { + t.Fatalf("payload integrity errors: %d", m) + } + t.Logf("OK: %d messages delivered, %d fragments each (~%d total fragments)", + totalMsgs, msgSize/984+1, totalMsgs*(msgSize/984+1)) +} + +// 10 senders, each sends 100 unordered large messages +func testFragLoadUnorderedConcurrent(t *testing.T) { + node1, node2, receiverPID := setupFragLoadNodes(t, "Unordered") + defer node1.Stop() + defer node2.Stop() + + numSenders := 10 + msgsPerSender := 100 + msgSize := 5000 + totalMsgs := numSenders * msgsPerSender + + t11done = make(chan struct{}, 1) + t11target = int64(totalMsgs) + t11count.Store(0) + t11mismatch.Store(0) + + var wg sync.WaitGroup + for i := 0; i < numSenders; i++ { + wg.Add(1) + go func() { + defer wg.Done() + pid, err := node1.Spawn(factory_t11sender, gen.ProcessOptions{}) + if err != nil { + t.Errorf("spawn sender: %v", err) + return + } + job := &t11sendJob{ + target: receiverPID, + size: msgSize, + count: msgsPerSender, + noOrder: true, + done: make(chan error, 1), + } + node1.Send(pid, job) + if err := <-job.done; err != nil { + t.Errorf("sender error: %v", err) + } + }() + } + + wg.Wait() + + select { + case <-t11done: + case <-time.After(30 * time.Second): + t.Fatalf("timeout: received %d/%d messages", t11count.Load(), totalMsgs) + } + + if m := t11mismatch.Load(); m > 0 { + t.Fatalf("payload integrity errors: %d", m) + } + t.Logf("OK: %d messages delivered (unordered, shared assembly)", totalMsgs) +} + +// 10 senders: 5 ordered + 5 unordered, mixed sizes +func testFragLoadMixedConcurrent(t *testing.T) { + node1, node2, receiverPID := setupFragLoadNodes(t, "Mixed") + defer node1.Stop() + defer node2.Stop() + + numSenders := 10 + msgsPerSender := 100 + totalMsgs := numSenders * msgsPerSender + + t11done = make(chan struct{}, 1) + t11target = int64(totalMsgs) + t11count.Store(0) + t11mismatch.Store(0) + + var wg sync.WaitGroup + for i := 0; i < numSenders; i++ { + wg.Add(1) + idx := i + go func() { + defer wg.Done() + pid, err := node1.Spawn(factory_t11sender, gen.ProcessOptions{}) + if err != nil { + t.Errorf("spawn sender: %v", err) + return + } + // odd senders: unordered, larger messages + // even senders: ordered, smaller messages + noOrder := idx%2 == 1 + size := 3000 + if noOrder { + size = 8000 + } + job := &t11sendJob{ + target: receiverPID, + size: size, + count: msgsPerSender, + noOrder: noOrder, + done: make(chan error, 1), + } + node1.Send(pid, job) + if err := <-job.done; err != nil { + t.Errorf("sender error: %v", err) + } + }() + } + + wg.Wait() + + select { + case <-t11done: + case <-time.After(30 * time.Second): + t.Fatalf("timeout: received %d/%d messages", t11count.Load(), totalMsgs) + } + + if m := t11mismatch.Load(); m > 0 { + t.Fatalf("payload integrity errors: %d", m) + } + t.Logf("OK: %d messages delivered (mixed ordered/unordered)", totalMsgs) +} + +func setupFragLoadNodes(t *testing.T, suffix string) (gen.Node, gen.Node, gen.PID) { + t.Helper() + + uid := lib.RandomString(6) + + options1 := gen.NodeOptions{} + options1.Network.Cookie = "fragload" + options1.Network.FragmentSize = 1000 + options1.Log.DefaultLogger.Disable = true + node1, err := ergo.StartNode(gen.Atom(fmt.Sprintf("distT11n1%s%s@localhost", suffix, uid)), options1) + if err != nil { + t.Fatal(err) + } + + options2 := gen.NodeOptions{} + options2.Network.Cookie = "fragload" + options2.Network.FragmentSize = 1000 + options2.Log.DefaultLogger.Disable = true + node2, err := ergo.StartNode(gen.Atom(fmt.Sprintf("distT11n2%s%s@localhost", suffix, uid)), options2) + if err != nil { + node1.Stop() + t.Fatal(err) + } + + // spawn receiver on node2 + receiverPID, err := node2.Spawn(factory_t11receiver, gen.ProcessOptions{}) + if err != nil { + node1.Stop() + node2.Stop() + t.Fatal(err) + } + + // establish connection + if _, err := node1.Network().GetNode(node2.Name()); err != nil { + node1.Stop() + node2.Stop() + t.Fatal(err) + } + + return node1, node2, receiverPID +} diff --git a/testing/unit/network.go b/testing/unit/network.go index cc7656cc6..ee2a9c4f5 100644 --- a/testing/unit/network.go +++ b/testing/unit/network.go @@ -2,6 +2,7 @@ package unit import ( "fmt" + "reflect" "regexp" "sort" @@ -358,6 +359,14 @@ func (n *TestNetwork) Mode() gen.NetworkMode { return n.mode } +func (n *TestNetwork) Protos() []gen.NetworkProto { return nil } +func (n *TestNetwork) RegisterType(v any) error { return nil } +func (n *TestNetwork) RegisterTypes(types []any) error { return nil } +func (n *TestNetwork) RegisterError(e error) error { return nil } +func (n *TestNetwork) RegisterAtom(a gen.Atom) error { return nil } +func (n *TestNetwork) RegisteredTypes() []gen.RegisteredTypeInfo { return nil } +func (n *TestNetwork) LookupType(name string) (reflect.Type, bool) { return nil, false } + // Helper method to set network mode (for testing) func (n *TestNetwork) SetMode(mode gen.NetworkMode) { n.mode = mode diff --git a/testing/unit/node.go b/testing/unit/node.go index ef78d2007..9db58f414 100644 --- a/testing/unit/node.go +++ b/testing/unit/node.go @@ -230,7 +230,7 @@ func (tn *TestNode) ProcessList() ([]gen.PID, error) { return pids, nil } -func (tn *TestNode) ProcessListShortInfo(start, limit int) ([]gen.ProcessShortInfo, error) { +func (tn *TestNode) ProcessListShortInfo(start, limit int, filter ...func(gen.ProcessShortInfo) bool) ([]gen.ProcessShortInfo, error) { var infos []gen.ProcessShortInfo for pid := range tn.processes { infos = append(infos, gen.ProcessShortInfo{ @@ -241,6 +241,19 @@ func (tn *TestNode) ProcessListShortInfo(start, limit int) ([]gen.ProcessShortIn return infos, nil } +func (tn *TestNode) ProcessRangeShortInfo(fn func(gen.ProcessShortInfo) bool) error { + for pid := range tn.processes { + info := gen.ProcessShortInfo{ + PID: pid, + State: gen.ProcessStateRunning, + } + if fn(info) == false { + break + } + } + return nil +} + func (tn *TestNode) ProcessName(pid gen.PID) (gen.Atom, error) { // Simple stub - returns empty name return "", nil @@ -596,6 +609,18 @@ func (tn *TestNode) UnregisterEvent(name gen.Atom) error { return nil } +func (tn *TestNode) EventInfo(event gen.Event) (gen.EventInfo, error) { + return gen.EventInfo{}, gen.ErrEventUnknown +} + +func (tn *TestNode) EventRangeInfo(fn func(gen.EventInfo) bool) error { + return nil +} + +func (tn *TestNode) EventListInfo(timestamp int64, limit int, filter ...func(gen.EventInfo) bool) ([]gen.EventInfo, error) { + return nil, nil +} + func (tn *TestNode) SendExit(pid gen.PID, reason error) error { // Check for failure injection if err := tn.CheckMethodFailure("SendExit", pid, reason); err != nil { @@ -613,19 +638,43 @@ func (tn *TestNode) Log() gen.Log { return tn.log } -func (tn *TestNode) LogLevelProcess(pid gen.PID) (gen.LogLevel, error) { - return gen.LogLevelInfo, nil +func (tn *TestNode) SetProcessLogLevel(pid gen.PID, level gen.LogLevel) error { + return nil +} + +func (tn *TestNode) SetProcessSendPriority(pid gen.PID, priority gen.MessagePriority) error { + return nil +} + +func (tn *TestNode) SetProcessCompression(pid gen.PID, enabled bool) error { + return nil } -func (tn *TestNode) SetLogLevelProcess(pid gen.PID, level gen.LogLevel) error { +func (tn *TestNode) SetProcessCompressionType(pid gen.PID, ctype gen.CompressionType) error { return nil } -func (tn *TestNode) LogLevelMeta(meta gen.Alias) (gen.LogLevel, error) { - return gen.LogLevelInfo, nil +func (tn *TestNode) SetProcessCompressionLevel(pid gen.PID, level gen.CompressionLevel) error { + return nil +} + +func (tn *TestNode) SetProcessCompressionThreshold(pid gen.PID, threshold int) error { + return nil +} + +func (tn *TestNode) SetProcessKeepNetworkOrder(pid gen.PID, order bool) error { + return nil } -func (tn *TestNode) SetLogLevelMeta(meta gen.Alias, level gen.LogLevel) error { +func (tn *TestNode) SetProcessImportantDelivery(pid gen.PID, important bool) error { + return nil +} + +func (tn *TestNode) SetMetaLogLevel(meta gen.Alias, level gen.LogLevel) error { + return nil +} + +func (tn *TestNode) SetMetaSendPriority(meta gen.Alias, priority gen.MessagePriority) error { return nil } @@ -658,6 +707,43 @@ func (tn *TestNode) LoggerLevels(name string) []gen.LogLevel { return []gen.LogLevel{gen.LogLevelInfo} } +func (tn *TestNode) TracingExporterAddPID(pid gen.PID, name string, flags gen.TracingFlags) error { + return nil +} + +func (tn *TestNode) TracingExporterAdd(name string, exporter gen.TracingBehavior, flags gen.TracingFlags) error { + return nil +} + +func (tn *TestNode) TracingExporterDeletePID(pid gen.PID) { +} + +func (tn *TestNode) TracingExporterDelete(name string) { +} + +func (tn *TestNode) TracingExporters() []string { + return nil +} + +func (tn *TestNode) TracingExporterFlags(name string) gen.TracingFlags { + return 0 +} + +func (tn *TestNode) SetTracingSampler(sampler gen.TracingSampler) error { + return nil +} + +func (tn *TestNode) SetTracingAttribute(key, value string) {} +func (tn *TestNode) RemoveTracingAttribute(key string) {} + +func (tn *TestNode) TracingSampler() gen.TracingSampler { + return gen.TracingSamplerDisable +} + +func (tn *TestNode) SetProcessTracingSampler(pid gen.PID, sampler gen.TracingSampler) error { + return nil +} + func (tn *TestNode) MakeRef() gen.Ref { return makeTestRefWithCreation(tn.options.NodeName, tn.options.NodeCreation) } diff --git a/testing/unit/process.go b/testing/unit/process.go index 8aec2bd66..8bd361573 100644 --- a/testing/unit/process.go +++ b/testing/unit/process.go @@ -102,6 +102,16 @@ func (tp *TestProcess) Behavior() gen.ProcessBehavior { return tp.behavior } +func (tp *TestProcess) BehaviorName() string { + return "" +} + +func (tp *TestProcess) SetTracingAttribute(key, value string) {} +func (tp *TestProcess) RemoveTracingAttribute(key string) {} +func (tp *TestProcess) SetTracingSpanAttribute(key, value string) {} +func (tp *TestProcess) TracingAttributes() []gen.TracingAttribute { return nil } +func (tp *TestProcess) ClearTracingSpanAttributes() {} + func (tp *TestProcess) State() gen.ProcessState { return tp.state } @@ -585,6 +595,23 @@ func (tp *TestProcess) ImportantDelivery() bool { return tp.options.ImportantDelivery } +func (tp *TestProcess) SetTracingSampler(sampler gen.TracingSampler) error { + return nil +} + +func (tp *TestProcess) TracingSampler() gen.TracingSampler { + return gen.TracingSamplerDisable +} + +func (tp *TestProcess) PropagatingTrace() gen.Tracing { + return gen.Tracing{} +} + +func (tp *TestProcess) SetPropagatingTrace(t gen.Tracing) { +} + +func (tp *TestProcess) SendTracingSpan(span gen.TracingSpan) {} + // Compression methods func (tp *TestProcess) Compression() bool { return false // Default to false for testing diff --git a/version.go b/version.go index c68b71b24..8eccdce3d 100644 --- a/version.go +++ b/version.go @@ -5,7 +5,7 @@ import "ergo.services/ergo/gen" var ( FrameworkVersion = gen.Version{ Name: "Ergo Framework", - Release: "3.2.0", + Release: "3.3.0-development", License: gen.LicenseMIT, } )