Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion e2e-tests/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,16 @@ exist to close that gap:
observable over SSH or in the journal, assert it there — it's deterministic
and fast. The suite already does this where it can: see `Check Home Directory`
and `Check If Owner Was Registered` in `resources/broker.resource`, which use
`SSH.Execute` instead of reading the screen.
`SSH.Execute` and `getent` instead of reading the screen.
2. **OCR (`Match Text` / `Find Text` / `Read Text`).** Use this only when the
behavior is genuinely only observable on screen (GDM, the PAM CLI/TUI inside
a `machinectl` session, terminal output you can't reach over SSH). Follow the
OCR hygiene rules below.

Note that SSH is often *not* possible — e.g. anything happening inside the GDM
greeter or a `machinectl login` session before the user's shell exists. Don't
force an SSH assertion where it doesn't fit; use OCR and make it robust.

## Don't sleep — poll instead

Avoid `Builtin.Sleep` to wait for asynchronous work to finish (e.g. authd's
Expand Down Expand Up @@ -145,6 +149,27 @@ file. The console history is saved to `rfdebug_history.log` in the output dir.
For a region that OCR keeps misreading, the console also exposes `Grab
Templates` and a region-of-interest selector to crop and scope matches.

## OCR hygiene rules

These come straight from patterns the existing suite relies on — reproduce them:

- **Strip spaces and normalize case** when matching machine-generated codes. The
device user code handling in `Continue Log In With Remote User: Authenticate
In External Browser` removes OCR-inserted spaces and upper-cases the result,
because OCR adds phantom spaces and mis-cases characters.
- **Prefer `regex:` matches** for anything variable. `Find Text` / `Match Text`
accept `regex:<pattern>`; use it instead of brittle exact strings when the
text contains values that vary or that OCR renders inconsistently.
- **Scope with a `region`** when a short string risks matching elsewhere on
screen, or when OCR is unreliable over the full frame.
- **Give realistic timeouts.** `Match Text` waits up to its timeout for text to
appear; remote/network-dependent screens use generous timeouts (e.g. `120`)
in the existing tests — match that, don't use the short default for slow
screens.
- **Don't match the command echo.** When checking terminal output, use the
`base64 cmd-finished` trick (above) so you don't match the typed command
before it has run.

## Running and reading results

Run a single test (or omit the file to run all) using the existing script. It automatically loads the
Expand Down
3 changes: 2 additions & 1 deletion e2e-tests/resources/SSH.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@ async def execute_as_user(self, user:str, command: str, timeout: int|None = 30)
command = (f"sudo -u {user} "
f"XDG_RUNTIME_DIR=/run/user/$(id -u {user}) "
f"DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u {user})/bus "
f"{command}")
"-- "
f"sh -c \"{command}\"")
return await self.execute(command, timeout)

@keyword
Expand Down
20 changes: 20 additions & 0 deletions e2e-tests/resources/authd.resource
Original file line number Diff line number Diff line change
Expand Up @@ -72,3 +72,23 @@ Check That Remote User Can Run Sudo Commands
SSH.Execute echo ${local_password} | sudo -S touch sudo-test.txt
${output} = SSH.Execute ls -l sudo-test.txt
Should Contain ${output} sudo-test.txt


Check that GNOME keyring is unlocked
SSH.Execute As Current User echo '' | secret-tool store --label=test test key


Store Secret In GNOME Keyring
[Arguments] ${secret}
# Storing succeeds only if the keyring is unlocked, so this both seeds a
# known secret and asserts the keyring is currently unlocked.
SSH.Execute As Current User printf '%s' '${secret}' | secret-tool store --label=e2e-test e2e-test keyring-persistence


Check Secret In GNOME Keyring
[Arguments] ${expected_secret}
# Looking up the secret succeeds only if the *same* keyring that stored it is
# unlocked. If the password change re-created the keyring instead of re-keying
# it, the secret is gone and this fails.
${secret} = SSH.Execute As Current User secret-tool lookup e2e-test keyring-persistence
Should Be Equal ${secret} ${expected_secret}
2 changes: 1 addition & 1 deletion e2e-tests/resources/utils.resource
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ Submit Run Command Dialog

Launch App
[Arguments] ${app-id}
SSH.Execute As Current User gtk-launch ${app-id}
SSH.Execute As Current User gtk-launch ${app-id} timeout=60


Cancel Operation
Expand Down
2 changes: 2 additions & 0 deletions e2e-tests/tests/login_gdm.robot
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Test login with GDM

# Log in with remote user with device code flow via GDM
Log In With Remote User Through GDM: QR Code ${username} ${local_password}
Check that GNOME keyring is unlocked

# Check remote user is properly added to the system
Check If User Was Added Properly ${username}
Expand All @@ -30,3 +31,4 @@ Test login with GDM

# Log in with remote user with local password via GDM
Log In With Remote User Through GDM: Local Password ${username} ${local_password}
Check that GNOME keyring is unlocked
48 changes: 48 additions & 0 deletions e2e-tests/tests/passwd_changes_keyring_password.robot
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
*** Settings ***
Resource resources/utils.resource
Resource resources/authd.resource

Resource resources/broker.resource

# Test Tags robot:exit-on-failure

Test Setup utils.Test Setup snapshot=%{BROKER}-installed
Test Teardown utils.Test Teardown


*** Variables ***
${snapshot} %{BROKER}-installed
${username} %{E2E_USER}
${local_password} qwer1234
${new_password} passwd1234
${keyring_secret} s3cr3t-survives-passwd


*** Test Cases ***
Changing the local password also changes the keyring password
[Documentation] Changing a remote user's local password with `passwd` must re-key the
... GNOME login keyring so it keeps unlocking with the new password.
...
... Regression guard: the keyring is unlocked from PAM_AUTHTOK at login. If the
... password change does not propagate the old/new password to the keyring's PAM
... module, the next login creates a fresh keyring and any previously stored secret
... is lost. We seed a secret before the change and require it to still be there
... after logging in again with the new password.

# Log in with device authentication. This creates the login keyring and
# unlocks it with ${local_password}.
Log In With Remote User Through GDM: QR Code ${username} ${local_password}
Check that GNOME keyring is unlocked
Store Secret In GNOME Keyring ${keyring_secret}

# Change the local password with `passwd`.
Open Terminal
Change Password ${local_password} ${new_password}
Close Focused Window
Log Out

# Log in again with the new local password. The keyring must unlock with it
# and still contain the secret stored before the change.
Log In With Remote User Through GDM: Local Password ${username} ${new_password}
Check that GNOME keyring is unlocked
Check Secret In GNOME Keyring ${keyring_secret}
2 changes: 2 additions & 0 deletions e2e-tests/vm/cloud-init-template-noble.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ packages:
- gnome-terminal
- gsettings-ubuntu-schemas
- libpam-pwquality
# Required to test that the keyring is unlocked (ships secret-tool)
- libsecret-tools
- openssh-server
- policykit-desktop-privileges
# Required to get the VM's IP address
Expand Down
2 changes: 2 additions & 0 deletions e2e-tests/vm/cloud-init-template-resolute.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ packages:
- gnome-shell-ubuntu-extensions
- gsettings-ubuntu-schemas
- libpam-pwquality
# Required to test that the keyring is unlocked (ships secret-tool)
- libsecret-tools
- openssh-server
- policykit-desktop-privileges
# Required to get the VM's IP address
Expand Down
2 changes: 2 additions & 0 deletions e2e-tests/vm/cloud-init-template-stonking.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ packages:
- gnome-shell-ubuntu-extensions
- gsettings-ubuntu-schemas
- libpam-pwquality
# Required to test that the keyring is unlocked (ships secret-tool)
- libsecret-tools
- openssh-server
- policykit-desktop-privileges
# Required to get the VM's IP address
Expand Down
2 changes: 1 addition & 1 deletion e2e-tests/vm/provision-authd.sh
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ assert_env_vars RELEASE BROKER
ARTIFACTS_DIR="${ARTIFACTS_DIR:-${DATA_DIR}/${RELEASE}}"

if [ -z "${VM_NAME:-}" ]; then
VM_NAME="${VM_NAME_BASE}-${RELEASE}"
export VM_NAME="${VM_NAME_BASE}-${RELEASE}"
fi
export VM_NAME

Expand Down
2 changes: 1 addition & 1 deletion e2e-tests/vm/provision-ubuntu.sh
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ ARTIFACTS_DIR="${ARTIFACTS_DIR:-${DATA_DIR}/${RELEASE}}"
CLOUD_INIT_TEMPLATE="${SCRIPT_DIR}/cloud-init-template-${RELEASE_NAME}.yaml"

if [ -z "${VM_NAME:-}" ]; then
VM_NAME="${VM_NAME_BASE}-${RELEASE}"
export VM_NAME="${VM_NAME_BASE}-${RELEASE}"
fi

function cloud_init_finished() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-local-groups-integration-auth-cli@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ New password:

Press escape key to go back to choose the provider
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-can-reset@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ Confirm password:

Press escape key to go back to choose the provider
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "authd2404"
PAM Authenticate()
User: "user-needs-reset-integration-mandatory@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ Confirm password:

Press escape key to go back to choose the provider
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "authd2404"
PAM Authenticate()
User: "user-needs-reset-integration-case-insensitive-authenticate-user-and-reset-password-with-case-insensitive-user-selection@example.com"
Result: success
Expand All @@ -78,6 +79,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "authd2404"
PAM Authenticate()
User: "user-needs-reset-integration-case-insensitive-authenticate-user-and-reset-password-with-case-insensitive-user-selection@example.com"
Result: success
Expand All @@ -98,6 +100,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "authd2404"
PAM Authenticate()
User: "user-needs-reset-integration-case-insensitive-authenticate-user-and-reset-password-with-case-insensitive-user-selection@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-integration-simple-testcliauthenticate@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-integration-was-empty@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-integration-invalid-timeout-testcliauthenticate@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Gimme your password:

Press escape key to go back to choose the provider
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-auth-modes-password-integration-cli@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-integration-preset-testcliauthenticate@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-integration-upper-case-testcliauthenticate@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-integration-preset-upper-case-testcliauthenticate@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-integration-switch-mode@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-integration-username-switched@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ Enter your one time credential:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "temporary pass00"
PAM Authenticate()
User: "user-integration-form-w-button@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ Unlock your phone +33... or accept request on web interface

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-mfa@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ Confirm password:

Press escape key to go back to choose the provider
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "authd2404"
PAM Authenticate()
User: "user-mfa-with-reset@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user2@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ Confirm password:

Press escape key to go back to choose the provider
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "authd2404"
PAM Authenticate()
User: "user-needs-reset@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Gimme your password:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "goodpass"
PAM Authenticate()
User: "user-integration-pam-preset@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ Enter your one time credential:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "temporary pass0"
PAM Authenticate()
User: "user-integration-remember-mode@example.com"
Result: success
Expand All @@ -65,6 +66,7 @@ Enter your one time credential:

Press escape key to go back to select the authentication method
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "temporary pass0"
PAM Authenticate()
User: "user-integration-remember-mode@example.com"
Result: success
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,8 @@ Confirm password:

Press escape key to go back to choose the provider
────────────────────────────────────────────────────────────────────────────────
PAM_AUTHTOK: "authd2404"
PAM_OLDAUTHTOK: "goodpass"
PAM ChangeAuthTok()
User: "user-mfa-integration-cli-passwd@example.com"
Result: success
Expand Down
Loading
Loading