Added design document for remote and multi-machine launching #297
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,185 @@ | ||||||
| # Multi-Machine and Remote Launching | ||||||
| ## Goals | ||||||
| - Connect to a remote host and run nodes on it | ||||||
| - Ensure this capability does not compromise security | ||||||
| - Support arbitrary remote execution methods | ||||||
| - Monitor the status of and manage lifecycles of nodes across systems | ||||||
| - Shutdown nodes remotely | ||||||
| - Ability to detach the machine used to launch a system and have the system continue running | ||||||
| - Be able to reconnect to the system to monitor and manage nodes running remotely | ||||||
|
|
||||||
|
|
||||||
| To achieve our goals, this update will consist of two primary components: an ExecuteRemote action, and a LaunchServiceNode. | ||||||
|
Contributor
There was a problem hiding this comment. Based on the re-scope of this design - it seems like this "Goals" section shouldn't be talking about the |
||||||
|
|
||||||
| The ExecuteRemote action will build off of the refactor he is working on described here: https://github.com/ros2/design/pull/272 | ||||||
| Since we don't want to limit remote execution to a single protocol, ExecuteRemote will be a base class and we will create an implementation using SSH which we will call ExecuteRemoteSSH. | ||||||
| Remote machines will be described by a Machine class which will also be a base class for protocol-specific implementations. | ||||||
| Together these components will provide the means to execute processes remotely and should cover most basic use cases. | ||||||
|
|
||||||
| The LaunchServiceNode(s) will expose ROS2 services and topics for handling startup and shutdown of processes on remote machines. | ||||||
|
Contributor
There was a problem hiding this comment. This LaunchServiceNode could be designed as an independent component - that is possible to use purely locally to detach a launch command from an ongoing launch context - it could then be used by the remote launching functionality. |
||||||
| This component won't be necessary for basic execution of remote nodes, but would allow remote system to persist beyond the life of the initial LaunchService. | ||||||
|
Collaborator
There was a problem hiding this comment. I am not clear on this use case, could you elaborate a little bit? especially on the difference from There was a problem hiding this comment. The design is intended to allow you to run a node pretty much like they currently would run, that is, bound to the lifespan of the launch process itself. When the launch process dies, the SSH connection dies, and anything still running on that SSH terminal dies with it. The This is certainly somewhat similar to some of the functionality envisioned by (Obviously, these same arguments also apply to the
Contributor
There was a problem hiding this comment. Question to explicitly state in the doc: Who starts the LaunchServiceNode? How is its lifecycle expected to be managed? |
||||||
| The topics and services will allow remote tools to view and manage the processes running remotely. | ||||||
| To prevent execution of completely arbitrary processes, the node will be given a list of commands it is allowed to run at startup, each with a unique ID assigned by the user. | ||||||
| Requests to start or stop processes would then use the unique ID to indicate to the node which process to start or stop. | ||||||
|
|
||||||
|
|
||||||
| ## New Classes | ||||||
|
|
||||||
| #### launch.descriptions.Machine | ||||||
| ##### Members | ||||||
| Members would be defined by child classes, since different protocols will require different types of information | ||||||
|
|
||||||
| #### launch.descriptions.SSHMachine | ||||||
| Contains information needed to run a process on another machine | ||||||
| ##### Members | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | hostname | String | Hostname of the machine. | | ||||||
| | port | int | Port on to connect to (default 22) | | ||||||
| | ssh\_keys | String | Path to ssh key file | | ||||||
|
Contributor
There was a problem hiding this comment. should this be singular?
Suggested change
|
||||||
| | passphrase | String | Optional passphrase if the key uses one. | | ||||||
|
|
||||||
| #### launch.actions.ExecuteRemote | ||||||
| Base class for actions that use connection informationn contained in `machine` to start a process on a remote machine. | ||||||
|
|
||||||
| ##### Members | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | process\_description |`launch.descriptions.Executable` | An object describing the process to be executed.| | ||||||
| | shell | boolean | If True, a shell is used to execute the process.| | ||||||
| | sigterm\_timeout | List[Substitution] | Time until shutdown should escalate to `SIGTERM`, as a string or a list of strings and `Substitution`s to be resolved at runtime, defaults to the `LaunchConfiguration` called `sigterm_timeout` | | ||||||
| | sigkill\_timeout | List[Substitution] | Time until escalating to `SIGKILL` after `SIGTERM`, as a string or a list of strings and `Substitution`s to be resolved at runtime, defaults to the `LaunchConfiguration` called `sigkill_timeout` | | ||||||
| | emulate\_tty | boolean | Emulate a tty (terminal), defaults to `False`, but can be overridden with the `LaunchConfiguration` called `emulate_tty`, the value of which is evaluated as true or false according to `evaluate_condition_expression`. | | ||||||
| | prefix | List[Substitution] | A set of commands/arguments to preceed the `cmd`, used for things like `gdb`/`valgrind` and defaults to the `LaunchConfiguration` called `launch-prefix` | | ||||||
| | output | ?? | Configuration for process output logging. Defaults to `log` i.e. log both `stdout` and `stderr` to launch main log file and stderr to the screen. | | ||||||
| | output\_format | ?? | For logging each output line, supporting `str.format()` substitutions with the following keys in scope: `line` to reference the raw output line and `this` to reference this action instance. | | ||||||
| | log\_cmd | boolean | If `True`, prints the final cmd before executing the process, which is useful for debugging when substitutions are involved. | | ||||||
|
Contributor
There was a problem hiding this comment. maybe should have a note that this will avoid printing any key passphrases or other sensitive information
Author
There was a problem hiding this comment. Oh, good point, I will include a note about that. |
||||||
| | on\_exit | List[LaunchDescriptionEntity] | List of actions to execute upon process exit.| | ||||||
| | persistent\_connection | boolean | Whether the LaunchService should maintain a persistent connection to the ssh instance | | ||||||
| | machine | `launch.descriptions.SSHMachine` | The machine on which to execute the process | | ||||||
|
Contributor
There was a problem hiding this comment. Want this to take the abstract interface, right?
Suggested change
|
||||||
| ##### Methods | ||||||
| | Name | Type | Description | | ||||||
| | ------------------ | --- | ------------------------------------------------------------ | | ||||||
| | get\_sub\_entities | List[LaunchDescriptionEntity] | Override. If on\_exit was provided in the constructor, returns that; otherwise returns an empty list. | | ||||||
| | execute | Optional[List[LaunchDescriptionEntity]] | Override. Establishes event handlers for process execution, passes the execution context to the process definition for substitution expansion, then uses `osrf_pycommon.process_utils.async_execute_process` to launch the defined process. | | ||||||
| | get\_asyncio\_future | Optional[asyncio.Future] | Override. Return an asyncio Future, used to let the launch system know when we're done. | | ||||||
|
|
||||||
|
|
||||||
| #### launch.actions.ExecuteRemoteSSH | ||||||
| Inherits from `launch.actions.ExecuteRemote` and manages the ssh connection | ||||||
| ##### Members | ||||||
| Inherited | ||||||
| ##### Methods | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | open\_ssh\_connection | SSHConnection | Object for interacting with the SSH connection | | ||||||
| | exec\_remote\_command | void | Executes a command on the remote machine | ||||||
|
|
||||||
| #### launch.substitutions.RemoteSubstitution | ||||||
| Uses a connection to a machine to run a command remotely and retuns the result as a string | ||||||
| ##### Members | ||||||
| ##### Constructor | ||||||
| | Argument | Type | Description | | ||||||
| |---|---|---| | ||||||
| | command | Text | The command to run on the remote machine | | ||||||
| | machine | `launch.descriptions.Machine` | An object decribing the machine that the command should be run on. | | ||||||
| ##### Methods | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | describe | Text | Override method. Returns a description of this substitution as a string. | | ||||||
| | perform | Text | Override method. Perform the substitution, given the launch context, and return it as a string. | | ||||||
|
|
||||||
|
|
||||||
| ## ROS | ||||||
|
|
||||||
| ### Nodes | ||||||
| #### launch\_ros.LaunchServiceNode | ||||||
| This node runs on each machine in a system is is repsonsible for launching and managing remote processes. | ||||||
|
Collaborator
There was a problem hiding this comment. nitpick:
Suggested change
There was a problem hiding this comment. nit: This node runs on each machine in a system |
||||||
| #### Params | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | allowable\_processes | List[Executable] | List of executables the node is allowed to launch | | ||||||
|
Contributor
There was a problem hiding this comment. Explicit flag to allow remote launching any process/node? Early in development this could be super useful for testing out various nodes via remote context from developer machine to robot.
Contributor
There was a problem hiding this comment. Or - is it not worth having this allow-list in here at all. Maybe SROS2 takes care of permissions well enough that we don't need to reimplement it. "If you use this you are exposing a vulnerability if you don't use SROS"
Contributor
There was a problem hiding this comment. Perhaps, only allow the LaunchServiceNode to only start known ros nodes on the remote system, rather than arbitrary commands. Restricting to ROS-specific in the remote context, since the local launch can execute arbitrary SSH commands anyways |
||||||
| ##### Topics | ||||||
| ###### Sent | ||||||
| - heartbeat | ||||||
| ###### Received | ||||||
| ##### Services | ||||||
| - query\_status | ||||||
| - start\_process | ||||||
| - stop\_process | ||||||
| - shutdown | ||||||
|
|
||||||
| ### Messages | ||||||
| #### Heartbeat | ||||||
|
Contributor
There was a problem hiding this comment. Do all LaunchServiceNodes use a unique topic, or do all publish on the same topic (in which case would need to put machine/service ID in the messages)
Contributor
There was a problem hiding this comment. Maybe this should be configurable, "heartbeat_topic", "process_status_topic" |
||||||
| Contains basic status information and enough information for launch related tools to find the node. | ||||||
| DDS might already provide enough info making this superfluous. | ||||||
| ##### Fields | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | timestamp | `builtin_interface.Timestamp` | Time at which heartbeat message was sent | | ||||||
| | hostname | String | Hostname of machine sending the status | | ||||||
| | num\_procs | int | Number of processes currently being tracked by the LaunchServiceNode. | | ||||||
|
|
||||||
| #### ProcessStatus | ||||||
| ##### Fields | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | timestamp | `builtin_interface.Timestamp` | Time at which message was sent | | ||||||
| | process\_name | String | Name of the process | | ||||||
| | process\_id | int | A unique numerical identifier for the process. This value is generated by the launch system and should not be expected to be the same as the proc id on the machine. | | ||||||
| | process\_status | int | An enumerated status value. STOPPED=0; RUNNING=1; SUSPENDED=2; | | ||||||
| | mem\_usage | double | Percent of memory used by the process | | ||||||
| | processor\_load | double | Percent of processor load coming from this process | | ||||||
| | err\_msg | String | Error string | | ||||||
|
|
||||||
| ### Services | ||||||
|
Contributor
There was a problem hiding this comment. Noting that each LaunchServiceNode will need to namespace itself to make service calls uniquely-fulfillable |
||||||
|
|
||||||
| #### QueryStatus | ||||||
| Service to be called by launch tools to get detailed information about the processes being managed by the LaunchServiceNode | ||||||
| ##### Request | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| Empty since no info is needed to answer the request. | ||||||
| ##### Response | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | timestamp | `builtin_interface.Timestamp` | Time at which message was sent | | ||||||
| | hostname | String | Hostname of machine sending the status | | ||||||
| | total\_mem\_usage | double | Percent of memory used | | ||||||
| | total\_processor\_load | double | processor load as a percent | | ||||||
| | num\_procs | int | Number of processes being managed by the node | | ||||||
| | processes | List[ProcessStatus] | List of ProcessStatus objects | | ||||||
|
|
||||||
| #### StartProcess | ||||||
| ##### Request | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | cmd | String | The command to be run when starting the process. | | ||||||
| | name | String | The name to use for the process. | | ||||||
| | uid | int | A unique numerical id for use within the launch system. | | ||||||
| | cwd | String | The directory from which to run the process on the remote machine. | | ||||||
| | env | List[common\_interfaces.KeyValue] | Dictionary of environment variables to be used from a clean environment. | | ||||||
| ##### Response | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | success | boolean | Whether the process was sucessfully started or not | | ||||||
| | status | ProcessStatus | Status of the process | | ||||||
|
|
||||||
| #### StopProcess | ||||||
| ##### Request | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | proc\_id | int | The launch id of the process that should be stopped | | ||||||
| ##### Response | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | success | boolean | Whether the process was sucessfully stopped or not | | ||||||
| | status | ProcessStatus | Status of the process | | ||||||
|
|
||||||
| #### Shutdown | ||||||
| ##### Request | ||||||
| Empty message | ||||||
| ##### Response | ||||||
| | Name | Type | Description | | ||||||
| |---|---|---| | ||||||
| | success | boolean | Whether the process was sucessfully stopped or not | | ||||||
| | status | ProcessStatus | Status of the process | | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're currently working on launch integration for security. In order to make sure it scales to remote launching, we might want to consider ways to send security files over to the remote machine.