Skip to content

Add optional embedded terminal and IPython console tabs to the default PyDM GUI - #1287

Open
ruck314 wants to merge 3 commits into
pre-releasefrom
pydm-gui-embedded-linux-terminal
Open

Add optional embedded terminal and IPython console tabs to the default PyDM GUI#1287
ruck314 wants to merge 3 commits into
pre-releasefrom
pydm-gui-embedded-linux-terminal

Conversation

@ruck314

@ruck314 ruck314 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Description

Adds two optional tabs to the default PyDM GUI, both off by default and
independent of each other. Both are appended after the existing tabs, so no tab
position changes and Debug Tree is still shown on startup.

  • Terminal, an interactive local shell, enabled with
    runPyDM(enableTerminal=True) or python -m pyrogue gui --terminal. Job
    control works, so Ctrl-C, Ctrl-Z, fg and jobs behave normally, as do
    full-screen programs such as vim and htop. Color output, 2000 lines of
    scrollback that retains color, and a dark theme.
  • IPython, an IPython session with a Rogue client already connected,
    enabled with runPyDM(enableIPython=True) or python -m pyrogue gui --ipython.
    client, root and pr are in scope against the same server the rest of the
    GUI is using, which is the session the Rogue server suggests at startup opened
    rather than typed out. Tab completion on the tree, highlighting, history search
    and %magics all work. Rich output does not, so no inline plotting, because the
    view is a text screen.

Both start their process the first time the tab is displayed, detach into their
own window and reattach without disturbing the running session, and start a fresh
session if the old one exits.

Both run on the machine displaying the GUI, as the user who launched it, not
on the Rogue server. Anyone who can reach the window gets a shell, or arbitrary
Python, with that user's privileges, and PyDM read-only mode restricts neither.
That is why they are opt-in, and the documentation covers the considerations.

Also fixes a VirtualClient bug this work exposed: its link monitor was not a
daemon thread, so a process that connected a client and never called stop()
could not exit. That hangs the one-line client the server banner suggests when it
is used from python -i, and any script that connects and falls off the end.

Adds pyte and ipython as dependencies, declared with the rest of the GUI
stack.

Details

  • Emulation. Qt has no terminal widget, so pyte does the VT emulation and
    the result is painted into a QPlainTextEdit. TerminalScreen overrides
    write_process_input, a pyte no-op that cursor position reports reply
    through, and keeps the scrollback rows pyte would otherwise drop.
  • Spawn. openpty plus subprocess, since os.forkpty warns in a
    multi-threaded process from Python 3.12 on. start_new_session cannot grant a
    controlling terminal, so the child goes through /bin/sh reopening the pty
    slave inside the new session; without that there is no foreground process group
    and Ctrl-C stops working. A negative-control test pins it.
  • Console. IPython on the same pty rather than an embedded qtconsole: no
    new dependency, and it is a TerminalPanel subclass supplying an argv. Started
    as sys.executable -m IPython -i -c so it imports the same pyrogue, with the
    address interpolated only through repr() and its own client in its own
    process. Neither widget is registered for Qt Designer, which instantiates
    eagerly.
  • Tree completion. jedi cannot follow the tree's __getattr__ and crashes on
    its nodes, after which IPython offers only the crash text, so Tab looked
    ignored while ordinary objects completed fine. --IPCompleter.use_jedi=False
    falls back to IPython's own dir() based completer; the cost is jedi's type
    inference on ordinary Python.
  • Client shutdown. The monitor thread is joined before atexit runs, so only
    a daemon thread or an explicit stop() lets the process exit. That join is why
    exiting the console did not restart it: the child stayed alive, so the pty never
    reported end of file.
  • Docker. Quotes the pydm specifier in docker/rogue/Dockerfile. Unquoted,
    >=1.18.0 was a shell redirection and the image installed pydm unpinned.
  • Docs. New terminal_tab.rst and ipython_tab.rst, with
    starting_gui.rst, rogue_widgets.rst and new API pages updated.

Testing. 8 new test files collecting 205 tests, 4 extended: controlling
terminal and job control, descriptor and zombie leaks over repeated spawn and
close cycles, key translation, screen and color rendering, detach and reattach,
option plumbing for both tabs, and for the console the Qt-free startup code, a
real session on a pty, tree completion at a real prompt, exiting with a client
attached, and reads and writes against a real server. Full suite: 863 pass, 10
skip, 3 fail. scripts/run_linters.sh clean.

Two failures are pre-existing and unrelated, confirmed with this branch's changes
stashed: test_pydm_rogue_plugin.py calls object.__new__ on a class that
overrides it, which Python 3.14 rejects, and test_epicsV7.py needs softioc,
absent from conda.yml. The third,
test_terminal_takes_focus_when_its_tab_is_selected, is added here; it passes
alone and fails when the rest of its file runs in one process against a forwarded
X display, which points at Qt tests competing for focus rather than at the
widget. It skips in CI, which installs pydm but no Qt binding.

@ruck314
ruck314 force-pushed the pydm-gui-embedded-linux-terminal branch from d4b1b40 to 8e5a931 Compare August 25, 2026 23:02
@ruck314 ruck314 changed the title Add optional embedded Linux terminal tab to the default PyDM GUI Add optional embedded terminal and IPython console tabs to the default PyDM GUI Aug 26, 2026
Adds a top-level Terminal tab beside System and Debug Tree, off by default and
enabled with `runPyDM(enableTerminal=True)` or `python -m pyrogue gui
--terminal`. The shell is fully interactive, so job control, tab completion and
full-screen programs work, and it renders 16 color, 256 color and 24 bit output
with 2000 lines of scrollback that retains color. A button detaches the terminal
into its own window and reattaches it without disturbing the running shell. The
shell runs on the machine displaying the GUI as the user who launched it, not on
the Rogue server, and is not started until the tab is displayed for the first
time.

Qt provides no terminal emulator widget and the real ones are separate C++
libraries needing system packages, so pyte does the VT screen emulation and the
result is painted into a QPlainTextEdit. The pseudo-terminal and screen half
lives in terminal_core.py and imports no Qt, which keeps the controlling
terminal setup, child reaping and descriptor handling testable in a CI
environment with no Qt binding.

Notes on the parts that are not obvious:

- The shell is started with openpty plus subprocess rather than pty.fork,
  because os.forkpty raises a DeprecationWarning in a multi-threaded process
  from Python 3.12 onward and this GUI always has Rogue, ZeroMQ and Qt threads
  running. start_new_session alone cannot give the child a controlling terminal,
  since one is only acquired when a session leader opens a tty itself and here
  the parent opened the slave. The child is therefore reached through /bin/sh
  reopening the same pty slave from inside the new session. Without that step
  there is no foreground process group, so terminal generated signals go nowhere
  and Ctrl-C silently stops working. A negative control test pins this.
- pyte.Screen.write_process_input is a no-op by default, and it is what cursor
  position reports and device attribute requests reply through. Left alone,
  readline, less and many shell prompts block waiting for an answer.
  TerminalScreen overrides it.
- pyte.HistoryScreen is deliberately not used. It hooks __getattribute__ and
  rebuilds a wrapper closure on every access to a wrapped event, including draw,
  which the stream calls once per character. Scrollback is captured in index()
  instead, and resize() keeps the rows pyte would otherwise delete off the top.

pyte is declared across the conda environment, the conda recipe, both pip
requirement sets and the generated package metadata, and mocked for the docs
build. The pydm specifier in docker/rogue/Dockerfile was unquoted in Docker's
shell form, so `>=1.18.0` was parsed as an output redirection and the image
silently installed pydm unpinned; quoting it is also what allows a second
specifier on that line.

Tests cover the controlling terminal and job control end to end, absence of
descriptor leaks and zombies over repeated spawn and close cycles, key
translation, screen and color rendering, scroll position handling, detach and
reattach, and the option plumbing. New terminal_tab.rst documents how to enable
it, where the shell runs, appearance, color, detaching, the security
considerations and the limitations; starting_gui.rst and rogue_widgets.rst are
updated, with a new API page for the widgets.
Adds a top-level IPython tab, off by default and enabled with
`runPyDM(enableIPython=True)` or `python -m pyrogue gui --ipython`. It is
independent of the terminal tab: either can be enabled without the other, and
both are appended, so neither moves the tabs the other adds.

The session starts with a connected VirtualClient bound as `client` and the tree
root as `root`. That is the session the Rogue server already suggests when it
starts, opened rather than typed out. The client targets the server the rest of
the GUI is using, resolved through the same parseAddress call the PyDM tools
make, so a GUI launched against a remote server does not get a console quietly
pointed at localhost.

It runs IPython on the pseudo-terminal the Terminal tab already uses rather than
embedding qtconsole, which keeps the feature to no new third-party packages and
reuses machinery that already has tests behind it. The cost is rich output:
there is no inline plotting, because the view is a text screen. LinuxTerminal
and TerminalPanel now take the program and the presentation strings as keyword
arguments defaulting to today's values, so the console is a subclass supplying
an argv rather than a second copy of the widget, and restart on exit comes for
free through the same _spawn.

Notes on the parts that are not obvious:

- Started as `sys.executable -m IPython` rather than through an ipython found on
  PATH. A console on another interpreter would import a different pyrogue, or
  none, and would not see the PYTHONPATH of a local build.
- `-i` is what keeps the session alive after the `-c` startup code has run, so
  the connected client stays in the namespace the user types into.
- The startup code imports pyrogue.interfaces explicitly. Importing the package
  alone does not bind the submodule, and the AttributeError that follows would
  be reported as a connection failure on every server.
- Every interpolation of the address into the startup code goes through repr(),
  so an address can neither terminate the literal it sits in nor smuggle in a
  statement.
- The console holds its own client in its own process, not the one the PyDM
  channels share, so a session stopped or wedged from the prompt cannot take the
  GUI's displays down with it.
- It starts with --IPCompleter.use_jedi=False, because completing on the tree is
  the reason the tab exists and jedi cannot do it. The tree resolves children
  through __getattr__, which jedi's static analysis cannot follow and in fact
  crashes on with "'TreeInstance' object has no attribute 'with_generics'";
  IPython then offers that crash text as the only candidate, so Tab does nothing
  after `root.` while ordinary objects and modules complete normally. IPython's
  own completer works from dir(), which the tree answers correctly. What is given
  up is jedi's type inference on ordinary Python.
- Kept out of the Qt Designer plugin group for the same reason as the terminal:
  Designer instantiates registered widgets eagerly, which would start an IPython
  session inside the design tool.

IPython becomes a runtime requirement of the GUI rather than only a convenience
of the development environment. It was already in conda.yml, the conda recipe
and the Docker image, so conda and Docker users see no change; this adds it to
the pip requirement sets and to extras_require['gui'].

The command line and startup code live in a module that imports no Qt, so they
stay testable in a CI environment with no Qt binding, and one case there spawns
a real console on a pty to pin the `-i -c` behaviour the design rests on. The
integration tests drive a real server end to end: a value read back through the
tree, one written that the server side then sees, and Tab completing on the
tree. They point IPYTHONDIR at a temporary directory, because prompt_toolkit
renders a suggestion drawn from the developer's own history as ghost text that
reads exactly like a completion. New ipython_tab.rst documents what is already
connected, where it runs, when the session starts, detaching, the security
considerations and the limitations it inherits from the terminal's rendering.
A process that connects a VirtualClient and never calls stop() cannot exit. The
monitor thread is joined during interpreter shutdown, and that join does not
return: the loop only ends when stop() clears its flag, and atexit callbacks run
after the join, too late to clear anything. So the client the server banner
suggests hangs a `python -i` session on exit, and any script that connects and
falls off the end hangs on the way out.

The IPython tab was the visible symptom. Exiting the console left the child alive
in shutdown, the pty therefore never reported end of file, and the widget never
learned the session had ended, so the restart it does for a shell never happened.
The restart path itself was correct all along.

The thread polls the link once a second and holds nothing worth flushing, so
running it as a daemon costs nothing and stop() still joins it when it is called.

The new test spawns a child that connects and returns, and requires it to exit,
which is what nothing covered before. The widget-level test exits a session with
a live client attached and requires a new one to come up connected. The FakeThread
stub in the unit tests has to accept the keyword.
@ruck314
ruck314 force-pushed the pydm-gui-embedded-linux-terminal branch from 0796610 to 604b334 Compare August 26, 2026 04:30
@ruck314
ruck314 marked this pull request as ready for review August 26, 2026 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant