Add optional embedded terminal and IPython console tabs to the default PyDM GUI - #1287
Open
ruck314 wants to merge 3 commits into
Open
Add optional embedded terminal and IPython console tabs to the default PyDM GUI#1287ruck314 wants to merge 3 commits into
ruck314 wants to merge 3 commits into
Conversation
ruck314
force-pushed
the
pydm-gui-embedded-linux-terminal
branch
from
August 25, 2026 23:02
d4b1b40 to
8e5a931
Compare
Adds a top-level Terminal tab beside System and Debug Tree, off by default and enabled with `runPyDM(enableTerminal=True)` or `python -m pyrogue gui --terminal`. The shell is fully interactive, so job control, tab completion and full-screen programs work, and it renders 16 color, 256 color and 24 bit output with 2000 lines of scrollback that retains color. A button detaches the terminal into its own window and reattaches it without disturbing the running shell. The shell runs on the machine displaying the GUI as the user who launched it, not on the Rogue server, and is not started until the tab is displayed for the first time. Qt provides no terminal emulator widget and the real ones are separate C++ libraries needing system packages, so pyte does the VT screen emulation and the result is painted into a QPlainTextEdit. The pseudo-terminal and screen half lives in terminal_core.py and imports no Qt, which keeps the controlling terminal setup, child reaping and descriptor handling testable in a CI environment with no Qt binding. Notes on the parts that are not obvious: - The shell is started with openpty plus subprocess rather than pty.fork, because os.forkpty raises a DeprecationWarning in a multi-threaded process from Python 3.12 onward and this GUI always has Rogue, ZeroMQ and Qt threads running. start_new_session alone cannot give the child a controlling terminal, since one is only acquired when a session leader opens a tty itself and here the parent opened the slave. The child is therefore reached through /bin/sh reopening the same pty slave from inside the new session. Without that step there is no foreground process group, so terminal generated signals go nowhere and Ctrl-C silently stops working. A negative control test pins this. - pyte.Screen.write_process_input is a no-op by default, and it is what cursor position reports and device attribute requests reply through. Left alone, readline, less and many shell prompts block waiting for an answer. TerminalScreen overrides it. - pyte.HistoryScreen is deliberately not used. It hooks __getattribute__ and rebuilds a wrapper closure on every access to a wrapped event, including draw, which the stream calls once per character. Scrollback is captured in index() instead, and resize() keeps the rows pyte would otherwise delete off the top. pyte is declared across the conda environment, the conda recipe, both pip requirement sets and the generated package metadata, and mocked for the docs build. The pydm specifier in docker/rogue/Dockerfile was unquoted in Docker's shell form, so `>=1.18.0` was parsed as an output redirection and the image silently installed pydm unpinned; quoting it is also what allows a second specifier on that line. Tests cover the controlling terminal and job control end to end, absence of descriptor leaks and zombies over repeated spawn and close cycles, key translation, screen and color rendering, scroll position handling, detach and reattach, and the option plumbing. New terminal_tab.rst documents how to enable it, where the shell runs, appearance, color, detaching, the security considerations and the limitations; starting_gui.rst and rogue_widgets.rst are updated, with a new API page for the widgets.
Adds a top-level IPython tab, off by default and enabled with `runPyDM(enableIPython=True)` or `python -m pyrogue gui --ipython`. It is independent of the terminal tab: either can be enabled without the other, and both are appended, so neither moves the tabs the other adds. The session starts with a connected VirtualClient bound as `client` and the tree root as `root`. That is the session the Rogue server already suggests when it starts, opened rather than typed out. The client targets the server the rest of the GUI is using, resolved through the same parseAddress call the PyDM tools make, so a GUI launched against a remote server does not get a console quietly pointed at localhost. It runs IPython on the pseudo-terminal the Terminal tab already uses rather than embedding qtconsole, which keeps the feature to no new third-party packages and reuses machinery that already has tests behind it. The cost is rich output: there is no inline plotting, because the view is a text screen. LinuxTerminal and TerminalPanel now take the program and the presentation strings as keyword arguments defaulting to today's values, so the console is a subclass supplying an argv rather than a second copy of the widget, and restart on exit comes for free through the same _spawn. Notes on the parts that are not obvious: - Started as `sys.executable -m IPython` rather than through an ipython found on PATH. A console on another interpreter would import a different pyrogue, or none, and would not see the PYTHONPATH of a local build. - `-i` is what keeps the session alive after the `-c` startup code has run, so the connected client stays in the namespace the user types into. - The startup code imports pyrogue.interfaces explicitly. Importing the package alone does not bind the submodule, and the AttributeError that follows would be reported as a connection failure on every server. - Every interpolation of the address into the startup code goes through repr(), so an address can neither terminate the literal it sits in nor smuggle in a statement. - The console holds its own client in its own process, not the one the PyDM channels share, so a session stopped or wedged from the prompt cannot take the GUI's displays down with it. - It starts with --IPCompleter.use_jedi=False, because completing on the tree is the reason the tab exists and jedi cannot do it. The tree resolves children through __getattr__, which jedi's static analysis cannot follow and in fact crashes on with "'TreeInstance' object has no attribute 'with_generics'"; IPython then offers that crash text as the only candidate, so Tab does nothing after `root.` while ordinary objects and modules complete normally. IPython's own completer works from dir(), which the tree answers correctly. What is given up is jedi's type inference on ordinary Python. - Kept out of the Qt Designer plugin group for the same reason as the terminal: Designer instantiates registered widgets eagerly, which would start an IPython session inside the design tool. IPython becomes a runtime requirement of the GUI rather than only a convenience of the development environment. It was already in conda.yml, the conda recipe and the Docker image, so conda and Docker users see no change; this adds it to the pip requirement sets and to extras_require['gui']. The command line and startup code live in a module that imports no Qt, so they stay testable in a CI environment with no Qt binding, and one case there spawns a real console on a pty to pin the `-i -c` behaviour the design rests on. The integration tests drive a real server end to end: a value read back through the tree, one written that the server side then sees, and Tab completing on the tree. They point IPYTHONDIR at a temporary directory, because prompt_toolkit renders a suggestion drawn from the developer's own history as ghost text that reads exactly like a completion. New ipython_tab.rst documents what is already connected, where it runs, when the session starts, detaching, the security considerations and the limitations it inherits from the terminal's rendering.
A process that connects a VirtualClient and never calls stop() cannot exit. The monitor thread is joined during interpreter shutdown, and that join does not return: the loop only ends when stop() clears its flag, and atexit callbacks run after the join, too late to clear anything. So the client the server banner suggests hangs a `python -i` session on exit, and any script that connects and falls off the end hangs on the way out. The IPython tab was the visible symptom. Exiting the console left the child alive in shutdown, the pty therefore never reported end of file, and the widget never learned the session had ended, so the restart it does for a shell never happened. The restart path itself was correct all along. The thread polls the link once a second and holds nothing worth flushing, so running it as a daemon costs nothing and stop() still joins it when it is called. The new test spawns a child that connects and returns, and requires it to exit, which is what nothing covered before. The widget-level test exits a session with a live client attached and requires a new one to come up connected. The FakeThread stub in the unit tests has to accept the keyword.
ruck314
force-pushed
the
pydm-gui-embedded-linux-terminal
branch
from
August 26, 2026 04:30
0796610 to
604b334
Compare
ruck314
marked this pull request as ready for review
August 26, 2026 04:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds two optional tabs to the default PyDM GUI, both off by default and
independent of each other. Both are appended after the existing tabs, so no tab
position changes and
Debug Treeis still shown on startup.Terminal, an interactive local shell, enabled withrunPyDM(enableTerminal=True)orpython -m pyrogue gui --terminal. Jobcontrol works, so
Ctrl-C,Ctrl-Z,fgandjobsbehave normally, as dofull-screen programs such as
vimandhtop. Color output, 2000 lines ofscrollback that retains color, and a dark theme.
IPython, an IPython session with a Rogue client already connected,enabled with
runPyDM(enableIPython=True)orpython -m pyrogue gui --ipython.client,rootandprare in scope against the same server the rest of theGUI is using, which is the session the Rogue server suggests at startup opened
rather than typed out. Tab completion on the tree, highlighting, history search
and
%magicsall work. Rich output does not, so no inline plotting, because theview is a text screen.
Both start their process the first time the tab is displayed, detach into their
own window and reattach without disturbing the running session, and start a fresh
session if the old one exits.
Both run on the machine displaying the GUI, as the user who launched it, not
on the Rogue server. Anyone who can reach the window gets a shell, or arbitrary
Python, with that user's privileges, and PyDM read-only mode restricts neither.
That is why they are opt-in, and the documentation covers the considerations.
Also fixes a
VirtualClientbug this work exposed: its link monitor was not adaemon thread, so a process that connected a client and never called
stop()could not exit. That hangs the one-line client the server banner suggests when it
is used from
python -i, and any script that connects and falls off the end.Adds
pyteandipythonas dependencies, declared with the rest of the GUIstack.
Details
pytedoes the VT emulation andthe result is painted into a
QPlainTextEdit.TerminalScreenoverrideswrite_process_input, apyteno-op that cursor position reports replythrough, and keeps the scrollback rows
pytewould otherwise drop.openptyplussubprocess, sinceos.forkptywarns in amulti-threaded process from Python 3.12 on.
start_new_sessioncannot grant acontrolling terminal, so the child goes through
/bin/shreopening the ptyslave inside the new session; without that there is no foreground process group
and
Ctrl-Cstops working. A negative-control test pins it.qtconsole: nonew dependency, and it is a
TerminalPanelsubclass supplying an argv. Startedas
sys.executable -m IPython -i -cso it imports the samepyrogue, with theaddress interpolated only through
repr()and its own client in its ownprocess. Neither widget is registered for Qt Designer, which instantiates
eagerly.
__getattr__and crashes onits nodes, after which IPython offers only the crash text, so Tab looked
ignored while ordinary objects completed fine.
--IPCompleter.use_jedi=Falsefalls back to IPython's own
dir()based completer; the cost is jedi's typeinference on ordinary Python.
atexitruns, so onlya daemon thread or an explicit
stop()lets the process exit. That join is whyexiting the console did not restart it: the child stayed alive, so the pty never
reported end of file.
pydmspecifier indocker/rogue/Dockerfile. Unquoted,>=1.18.0was a shell redirection and the image installedpydmunpinned.terminal_tab.rstandipython_tab.rst, withstarting_gui.rst,rogue_widgets.rstand new API pages updated.Testing. 8 new test files collecting 205 tests, 4 extended: controlling
terminal and job control, descriptor and zombie leaks over repeated spawn and
close cycles, key translation, screen and color rendering, detach and reattach,
option plumbing for both tabs, and for the console the Qt-free startup code, a
real session on a pty, tree completion at a real prompt, exiting with a client
attached, and reads and writes against a real server. Full suite: 863 pass, 10
skip, 3 fail.
scripts/run_linters.shclean.Two failures are pre-existing and unrelated, confirmed with this branch's changes
stashed:
test_pydm_rogue_plugin.pycallsobject.__new__on a class thatoverrides it, which Python 3.14 rejects, and
test_epicsV7.pyneedssoftioc,absent from
conda.yml. The third,test_terminal_takes_focus_when_its_tab_is_selected, is added here; it passesalone and fails when the rest of its file runs in one process against a forwarded
X display, which points at Qt tests competing for focus rather than at the
widget. It skips in CI, which installs
pydmbut no Qt binding.