Skip to content

als-computing/splash_timepix

BranchesTags

Repository files navigation

splash_timepix

Streaming and pre-processing time-resolved TimePix3 detector data

Overview

This package provides a pipeline for streaming, pre-processing, and visualizing data from Amsterdam Scientific Instruments (ASI) TimePix3 detectors. It implements time-resolved spectroscopy binning with TDC (Time-to-Digital Converter) triggering and supports both real-time visualization and ZMQ publishing for downstream analysis.

It also ships a PySide6 acquisition UI (tpx-ui) that drives the full pipeline — beam alignment, acquisition control, live heatmaps/spectra, clustering parameter sweeps, unified logs, and a session timeline (see Acquisition UI).

Platform Requirements

  • Python 3.9+
  • Linux/Ubuntu recommended (input handling uses select.select() which is Unix-specific)
  • TimePix3 detector with ASI live-cli software

Installation

Clone the repository and install dependencies:

git clone https://github.com/als-computing/splash_timepix.git
cd splash_timepix
python3 -m venv .venv
source .venv/bin/activate
pip install -e .[dev]

To use the PySide6 acquisition UI, also install the optional ui extra:

pip install -e ".[ui]"

Linux: UDP receive buffers (Serval / streaming host)

On any Linux machine that will receive high-rate detector UDP traffic (for example the Serval or acquisition PC), configure kernel UDP socket buffer limits before your first live acquisition. Serval expects a large receive buffer; the default kernel ceiling can silently cap it and increase the risk of packet loss under burst traffic.

Follow the step-by-step sysctl / sysctl.d instructions in UDP receive buffers (first-time Linux setup).

Optional: register the UI with GNOME

On a fresh Linux/GNOME machine the UI launches fine via tpx-ui, but the window will appear in the dock with a generic gear icon and a main.py tooltip until the desktop integration is installed. The repository ships a script that creates the icon, an installed .desktop launcher, and a clickable shortcut on the Desktop, all derived from the current clone path:

./scripts/install_desktop_integration.sh

Re-running is safe (every output is overwritten). To remove:

./scripts/install_desktop_integration.sh --uninstall

What it sets up:

Path Purpose
~/.local/share/icons/hicolor/256x256/apps/splash_timepix.png Themed icon for the dock and Activities.
~/.local/share/applications/splash_timepix.desktop Launcher GNOME indexes; StartupWMClass=splash_timepix pairs the running window to this entry.
~/Desktop/splash_timepix.desktop Clickable shortcut on the user's Desktop.

After running it once, log out and back in (or restart the dock extension) so GNOME picks up the new launcher. On Wayland-GNOME the dock will then show the bundled icon instead of the gear.

Dependencies

  • numpy - Array operations
  • h5py - HDF5 I/O (centroider outputs)
  • opencv-python - Real-time visualization
  • pyzmq - ZMQ publishing
  • msgpack - Serialization for ZMQ
  • typer - CLI interface
  • psutil - System monitoring
  • requests - HTTP client (Serval status)
  • pydantic - Message schema validation

UI extra (pip install -e ".[ui]"): PySide6, pyqtgraph.

Configuration

  • All tool and code style configurations are in pyproject.toml
  • Pre-commit hooks are configured in .pre-commit-config.yaml
  • To enable pre-commit hooks (recommended):
pre-commit install

Documentation (info/)

Additional markdown guides in the repository:

Architecture

The system consists of five main components:

1. Socket Server (socket_server.py)

Multi-threaded TCP server that receives 12-byte TimePix3 packets and parses them with NumPy (parser):

  • Thread 1: Socket listener (reads TCP chunks, batches complete packets into raw byte buffers)
  • Thread 2: Vectorized parser (parse_batch) → callback receives a BatchParseResult (arrays per packet type)

See info/SOCKET_SERVER_README.md for details.

2. Main Application (app.py)

Main application implementing time-resolved binning:

  • Receives packets via callback from socket server
  • Bins pixel events into 3D arrays (x, y, time) based on TDC triggers
  • Flushes accumulated 3D arrays to processing queue
  • Provides statistics display and user commands
  • Publishes a heartbeat with queue depths and server state on a separate ZMQ port (default 5658) so the UI can monitor pipeline health
  • Supports an --alignment mode: TDCs are ignored and wall-clock-gated 2D X/Y histograms are emitted at 1–30 Hz (used by the UI's Alignment tab)

3. Worker Threads

Two alternative workers for consuming 3D arrays:

Plotting Worker (--plot flag):

  • Real-time OpenCV visualization
  • Displays 2D heatmap (x vs time, summed over y)
  • Shows statistics overlay
  • Interactive (press 'q' to close)

ZMQ Worker (default):

  • Publishes arrays via ZMQ PUB socket
  • Uses msgpack serialization
  • Multi-part messages (metadata + array bytes)
  • Supports multiple subscribers
  • Publishes start/stop control messages for acquisition lifecycle tracking

4. Simulated Source (simulator_cli.py)

Simulated TimePix3 data source for testing:

  • Generates realistic packet streams
  • Configurable pixel count rate and TDC frequency
  • Interactive CLI for control

5. Acquisition UI (tpx-ui)

PySide6 desktop application (src/splash_timepix/ui/) that orchestrates Serval, the streaming server, and live-cli, and visualizes the ZMQ stream. Tabs:

  • Alignment (default) — live 2D X/Y heatmap at 1–30 Hz for beam alignment, with grayscale LUT, binarize/log/integrate display modes, target overlay, and a rolling cps strip chart. Includes a local simulator mode that needs no hardware.
  • Operator — acquisition control (Start / Preview / Simulator / Replay), live current-flush and running-average heatmaps, ROI cursors with energy-calibrated spectra, pipeline queue monitoring, and saving of averaged data (PNG, CSV, energy/time axes, JSON metadata) under a shared per-scan slug.
  • Logs — unified system/server/Serval log viewer; all logs are also written to <repo>/logs/ for post-mortem debugging.
  • Timeline — chronological overview of the session's runs and log events in one place.
  • Centroider — clustering parameter sweeps (eps-s × eps-t) on .tpx3 files via the bundled tpx3dump, with side-by-side x-histogram comparison. See the Centroider tutorial.

Launch with:

tpx-ui

Widget preferences (acquisition settings, alignment display options) persist across sessions. Only one UI instance can run at a time (a lock with a takeover prompt guards against double-starts).

Quick Start - Basic Usage Examples

The package installs three console entry points (see pyproject.toml):

Command Equivalent Purpose
tpx-stream python -m splash_timepix.app Streaming server
tpx-ui python -m splash_timepix.ui.main Acquisition UI
tpx-sim python -m splash_timepix.simulator_cli Simulated data source

Production Mode (ZMQ Publishing)

Terminal 1 - Start server:

tpx-stream
# OR via
python -m splash_timepix.app

Terminal 2 - Subscribe to published data:

# Use (or modify) the example
python -m splash_timepix.example_zmq_sub
# OR connect with your application

Terminal 3 - Start detector:

./ASI/live-cli

Display Real-time Data (Live Plotting)

Terminal 1 - Start server with visualization:

tpx-stream --plot

Terminal 2 - Start data source:

# Using real detector
./ASI/live-cli
# OR replaying from file
./ASI/live-cli --source-files path/to/recording.tpx3
# OR using the simulator
python -m splash_timepix.simulator_cli

Development Mode (Debugging)

Terminal 1 - Start server with verbose output

tpx-stream --verbose

Terminal 2

# (I) Low Count Rates (Simulator)
python -m splash_timepix.simulator_cli
cps 3
tdc 1
start 60
# OR (II) High Count Rates [<100kcps] (Simulator)
python -m splash_timepix.simulator_cli
cps 100000
tdc 0.1
start 60
# OR (III) Using Replay From File (live-cli)
./ASI/live-cli --source-files path/to/recording.tpx3

Command-Line Options

app.py Options

python -m splash_timepix.app [OPTIONS]

Flags:

  • --plot: Enable real-time plotting (default: ZMQ publishing)
  • --verbose: Show detailed logs and packet samples (default: warnings only)

Server Options:

  • --host STR: Host for server to bind to (default: "localhost")
  • --port INT: Socket server port matching live-cli client port (default: 9090)
  • --buffer-size INT: Internal message queue size (default: 1000)
  • --callback-batch-size INT: Number of parsed packets to batch per callback (default: 10000)
  • --zmq-port INT: ZMQ publishing port (default: 5657)
  • --heartbeat-port INT: ZMQ heartbeat/status port (default: 5658)
  • --exit-on-disconnect: Stop the server when the client disconnects (default: keep running)

Time-Resolved Binning Options:

  • --tdc-ch INT: TDC channel to use - 0=both, 1=ch1, 2=ch2 (default: 0)
  • --tdc-edge STR: TDC edge to trigger on - "rising" or "falling" (default: rising)
  • --tdc-frequency FLOAT: Expected TDC trigger frequency in Hz (default: 100.0)
  • --t-delta-ns FLOAT: Time bin width in nanoseconds (default: -1 = auto; bin width is derived from --n-bins)
  • --n-bins INT: Number of time bins per TDC cycle, used when --t-delta-ns is not given (default: 350)
  • --flush-interval FLOAT: Time between 3D x, y, t array flushes in seconds (default: 1.0)
  • --collapse-y: Sum over the Y axis and emit 2D (x, t) arrays instead of 3D

Alignment Mode:

  • --alignment: Ignore TDCs and emit wall-clock-gated 2D X/Y histograms (used by the UI's Alignment tab)
  • --alignment-rate-hz FLOAT: Flush rate for alignment mode, 1–30 Hz (default: 30.0)

Display Options:

  • --stats-update-time INT: Stats refresh interval in seconds (default: 1)

simulator_cli.py Interactive Commands

After starting the test source, use these commands:

  • cps <value> - Set pixel count rate (events/second)
  • tdc <value> - Set TDC frequency (Hz)
  • count <y/n> - Enable/disable packet counting statistics
  • batch <seconds> - Wire-level burst batching interval (0 disables; sends per-packet)
  • start <duration> - Start streaming for duration (seconds)
  • stop - Stop streaming data
  • quit - Exit

ZMQ Message Format

The system publishes three types of messages via ZMQ. For a concrete start payload example, see info/SAMPLE_START_MESSAGE.md.

1. Start Message (Single-part)

Published when data acquisition begins (first data arrives):

{
    'msg_type': 'start',
    'scan_name': 'a1b2c3d4-e5f6-4a7b-8c9d-0e1f23456789',  # full UUID4
    'tdc_frequency_hz': 10.0,
    'detector_size_x': 256,
    'detector_size_y': 256,
    'n_bins': 350,
    't_delta_ns': 285714.29,
    'mode': 'timing',  # 'timing' (default) or 'alignment'
    # ... other configuration parameters
}

2. Event Message (Multi-part)

Published for each data flush:

Part 1: Metadata (msgpack)

{
    'msg_type': 'event',
    'shape': (256, 256, 350),     # Array dimensions
    'dtype': 'uint32',             # Numpy dtype
    'timestamp': 1699999999.123,   # Unix timestamp
    'flush_number': 1,             # Sequential flush number
    'cycles_in_flush': 10,         # TDC cycles in this flush
    'total_cycles': 10,            # Cumulative cycle count
    # ... other metadata
}

Part 2: Array Data (raw bytes)

  • Raw numpy array bytes
  • Reconstruct with np.frombuffer(bytes, dtype).reshape(shape)

3. Stop Message (Single-part)

Published when data acquisition ends (client disconnects or server shuts down):

{
    'msg_type': 'stop',
    'scan_name': 'a1b2c3d4-e5f6-4a7b-8c9d-0e1f23456789',  # same UUID as start
    'total_flushes': 9,
    'total_cycles': 99,
    'total_packets': 50000,
    'acquisition_duration_s': 28.91,
    # ... statistics
}

Example Subscriber

See also example_zmq_sub.py for a complete example that handles all message types.

import zmq
import msgpack
import numpy as np

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:5657")
socket.setsockopt(zmq.SUBSCRIBE, b"")

while True:
    # Receive first part (metadata)
    metadata_bytes = socket.recv()
    metadata = msgpack.unpackb(metadata_bytes)
    msg_type = metadata.get('msg_type')

    if msg_type == 'start':
        print(f"Acquisition started: {metadata['scan_name']}")
    elif msg_type == 'stop':
        print(f"Acquisition stopped: {metadata['scan_name']}")
    elif msg_type == 'event' or msg_type is None:
        # Event message - receive array data
        array_bytes = socket.recv()
        array = np.frombuffer(array_bytes, dtype=metadata['dtype']).reshape(metadata['shape'])
        print(f"Received flush #{metadata.get('flush_number')}: {array.shape}")

Using the Listener Pattern

For a more structured approach, use SplashTimePixZMQListener (similar to ArroyoXPS):

See also example_listener.py

from splash_timepix.listener import SplashTimePixZMQListener
from splash_timepix.schemas import TimePixStart, TimePixEvent, TimePixStop

def my_operator(message):
    if isinstance(message, TimePixStart):
        # Initialize processing
        print(f"Start: {message.scan_name}")
    elif isinstance(message, TimePixEvent):
        # Process data array
        print(f"Event: flush #{message.flush_number}, shape={message.array.shape}")
    elif isinstance(message, TimePixStop):
        # Finalize processing
        print(f"Stop: {message.scan_name}, {message.total_flushes} flushes")

listener = SplashTimePixZMQListener(
    zmq_address="tcp://localhost:5657",
    operator=my_operator
)
listener.start()  # Blocks until stopped

Statistics Display

The application displays real-time statistics:

Overall Stats

  • Server uptime
  • Physical memory usage (RSS)
  • Total packet count and rate
  • Unknown packet count

Queue Statistics

  • Message queue: Raw byte batches (socket reader → vectorized parser)
  • Callback: One BatchParseResult per batch (pixel/TDC/control arrays), not a list of packet objects
  • x, y, t array queue: 3D arrays (callback → worker)
  • Single array size: Memory per 3D array

Session Stats

  • Duration since last reset
  • Packet count and rate for current session
  • Reset with r key

Verbose Mode (--verbose)

  • Last 10 valid packets
  • Recent error messages

Time-Resolved Binning Explained

The application bins pixel events into 3D arrays based on TDC triggers:

  1. TDC trigger arrives → sets time zero (t_zero)
  2. Pixel events are binned relative to t_zero:
    • Calculate time bin: bin = (pixel_time - t_zero) / t_delta
    • Increment: array[x, y, bin] += 1
  3. After flush_interval time, the array is flushed to the worker
  4. Worker either plots or publishes the array

Key parameters:

  • tdc_frequency: Inverse of total time window to capture per TDC trigger (t_cycle = 1 / tdc_frequency)
  • t_delta: Width of each time bin (temporal resolution)
  • n_bins: Number of time bins per cycle

You specify either the bin width or the bin count; the other is derived:

  • With --t-delta-ns set: n_bins = ceil(t_cycle / t_delta)
  • Without it (default): t_delta = t_cycle / n_bins (with --n-bins, default 350)

Pixels outside the time window or before the first TDC are discarded and counted for diagnostics.

Data Sources

Simulator/ Test Source (simulator_cli.py)

  • Generates Poisson-distributed pixel events
  • Realistic TDC pulses with configurable frequency
  • Useful for testing and development

Live Detector (live-cli)

Real-time streaming from TimePix3:

./ASI/live-cli

Recorded Data (live-cli --source-files)

Replay recorded .tpx3 files:

./ASI/live-cli --source-files path/to/file.tpx3

Development

For step-by-step testing (server, simulator, ZMQ subscribers, optional ArroyoXPS), see info/TESTING_GUIDE.md.

Run Tests

# Run all tests
pytest

# Run start/stop message tests
pytest tests/test_start_stop_messages.py -v

# Run quick manual test (requires server running)
python tests/test_start_stop_quick.py

Run Pre-commit Checks

pre-commit run --all-files

Architecture Overview

Data Source → Socket Server → Callback (Binning) → Processing Queue → Worker
                ↓                    ↓                                    ↓
      Parser (NumPy batches)   3D Array (x,y,t)               Plot or Publish
                                                                    ↓
                                                          ZMQ PUB (start/event/stop)
                                                                    ↓
                                                          SplashTimePixZMQListener
                                                                    ↓
                                                          Operator (your processing)

Threading:

  1. Socket Listener Thread (I/O bound)
  2. Data Processor Thread (CPU bound)
  3. Plotting/ZMQ Worker Thread (output bound)
  4. Input Listener Thread (user commands)

See info/SOCKET_SERVER_README.md for server details. The start/stop wire format is documented in info/SAMPLE_START_MESSAGE.md, and the schema definitions live in src/splash_timepix/schemas.py.

Troubleshooting

No data appearing in plots/ZMQ

  • Check TDC channel matches your trigger source (--tdc-ch)
  • Check TDC edge setting (--tdc-edge)
  • Verify you're receiving TDC packets (use --verbose)

High memory usage

  • Increase t_delta (fewer time bins, smaller 3D array size)
  • Reduce flush_interval (flush more frequently)
  • Increase xyt_queue consumption rate

Queue overflow warnings

  • Increase --buffer-size
  • Reduce callback processing time
  • Increase --callback-batch-size
  • Check if worker thread is keeping up

Qt warnings on shutdown

  • Press ENTER after Ctrl+C to clear buffered warnings
  • These are harmless and occur during daemon thread cleanup

Warnings about pixels outside time window

When using the test simulator, you may see warnings like:

WARNING - Pixel outside time window: t_relative=... ps

This is expected behavior. The simulator uses real-time scheduling while packets use detector timestamps, which can drift slightly. These warnings indicate the application is correctly identifying and discarding events that fall outside the configured time window. In production with real detector hardware, similar edge cases may occur due to timing jitter, and the application handles them correctly.

Copyright Notice

Detector Interface for Streaming, Control, and Open-source integration (DISCO) Copyright (c) 2026, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.

If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Intellectual Property Office at IPO@lbl.gov.

NOTICE. This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit others to do so.

This software is distributed under a BSD-style license (with an added Enhancements paragraph). See LICENSE.txt for the full terms.

Note: The third-party binaries under ASI/ (e.g. Serval, live-cli, tpx3dump) are provided by Amsterdam Scientific Instruments and are covered by their own respective licenses, not the DISCO license above.

About

Streaming analysis of timepix data

Topics

beamline als asi lbl timepix timepix3

Resources

Readme

License

View license
Activity
Custom properties

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages

Generated from pypa/sampleproject