` is joined only by the proxy. The proxy accepts
+ only exact `www.southkesteven.gov.uk` and
+ `selfservice.southkesteven.gov.uk` destinations on ports 80 and 443.
+
+The allowlist proxy never logs a request path, query, header, payload, postcode,
+house number, or denied hostname. Its complete log vocabulary is an allowlisted
+host (or the constant `[DENIED]`) and a proxy status code. Redirects to any other
+origin fail at the next proxy request. The proxy has a read-only filesystem,
+drops every capability, has no mounts, and is the only process with an egress
+route.
+
+Chrome must receive both of these Selenium image environment values:
+
+```text
+SE_BROWSER_ARGS_UKBCD_PROXY=--proxy-server=http://ukbcd-live-proxy:3128
+SE_BROWSER_ARGS_UKBCD_NO_BYPASS=--proxy-bypass-list=<-loopback>
+```
+
+Chromium normally bypasses proxies for loopback. The special `<-loopback>` token
+removes that implicit bypass and adds no replacement bypass host. Background
+networking, sync, and default apps are disabled as additional noise controls.
+
+## Reviewed orchestration shape
+
+The following is a reference sequence, not an executable script. Substitute the
+already-recorded digest-pinned candidate and Selenium image names. Build/pull
+images only during the plan's preparation window. Never substitute mutable tags
+in the evidence run.
+
+```bash
+set -eu
+RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"
+INTERNAL="ukbcd-canary-internal-${RUN_ID}"
+EGRESS="ukbcd-canary-egress-${RUN_ID}"
+PROXY="ukbcd-live-proxy-${RUN_ID}"
+SELENIUM="ukbcd-live-selenium-${RUN_ID}"
+RUNNER="ukbcd-live-runner-${RUN_ID}"
+PROXY_IMAGE_REF="LOCAL_LIVE_PROXY_IMAGE@sha256:RECORD_BEFORE_RUN"
+SELENIUM_IMAGE_REF="PINNED_SELENIUM_IMAGE@sha256:RECORD_BEFORE_RUN"
+RUNNER_IMAGE_REF="LOCAL_LIVE_CANARY_IMAGE@sha256:RECORD_BEFORE_RUN"
+
+# Record the full local immutable image IDs which container inspect will expose.
+PROXY_IMAGE_ID="$(podman image inspect --format '{{.Id}}' "$PROXY_IMAGE_REF")"
+SELENIUM_IMAGE_ID="$(podman image inspect --format '{{.Id}}' "$SELENIUM_IMAGE_REF")"
+RUNNER_IMAGE_ID="$(podman image inspect --format '{{.Id}}' "$RUNNER_IMAGE_REF")"
+
+podman network create --internal "$INTERNAL"
+podman network create "$EGRESS"
+
+podman create --name "$PROXY" \
+ --network "$INTERNAL:alias=ukbcd-live-proxy" \
+ --read-only --cap-drop=ALL --security-opt=no-new-privileges \
+ --pids-limit=64 --cpus=0.25 --memory=192m \
+ --tmpfs /tmp:rw,noexec,nosuid,nodev,size=16m \
+ "$PROXY_IMAGE_ID"
+podman network connect "$EGRESS" "$PROXY"
+
+podman create --name "$SELENIUM" \
+ --network "$INTERNAL:alias=selenium" \
+ --read-only --cap-drop=ALL --security-opt=no-new-privileges \
+ --pids-limit=512 --cpus=3 --memory=3500m --shm-size=2g \
+ --tmpfs /tmp:rw,nosuid,nodev,size=512m \
+ --tmpfs /home/seluser:rw,nosuid,nodev,size=512m \
+ -e SE_BROWSER_ARGS_UKBCD_PROXY=--proxy-server=http://ukbcd-live-proxy:3128 \
+ -e 'SE_BROWSER_ARGS_UKBCD_NO_BYPASS=--proxy-bypass-list=<-loopback>' \
+ -e SE_BROWSER_ARGS_UKBCD_BACKGROUND=--disable-background-networking \
+ -e SE_BROWSER_ARGS_UKBCD_SYNC=--disable-sync \
+ -e SE_BROWSER_ARGS_UKBCD_DEFAULT_APPS=--disable-default-apps \
+ -e SE_NODE_MAX_SESSIONS=1 -e SE_NODE_OVERRIDE_MAX_SESSIONS=false \
+ -e SE_OFFLINE=true -e SE_ENABLE_TRACING=false \
+ "$SELENIUM_IMAGE_ID"
+
+podman create --name "$RUNNER" \
+ --network "$INTERNAL" \
+ --read-only --cap-drop=ALL --security-opt=no-new-privileges \
+ --pids-limit=96 --cpus=0.5 --memory=512m \
+ --tmpfs /tmp:rw,noexec,nosuid,nodev,size=32m \
+ -e UKBCD_LIVE_CANARY_APPROVED=one-public-fixture-lookup \
+ "$RUNNER_IMAGE_ID"
+
+python tests/disposable_ha/live_canary_safety_check.py \
+ --internal-network "$INTERNAL" --egress-network "$EGRESS" \
+ --proxy "$PROXY" --selenium "$SELENIUM" --runner "$RUNNER" \
+ --expected-proxy-image "$PROXY_IMAGE_ID" \
+ --expected-selenium-image "$SELENIUM_IMAGE_ID" \
+ --expected-runner-image "$RUNNER_IMAGE_ID" \
+ --output "live-canary-safety-${RUN_ID}.json"
+```
+
+The safety check must pass while all three application containers are still in
+`created` state. Podman 4.9 does not expose unstarted endpoints in the container
+map returned by `network inspect`, so the validator also inventories every local
+container and resolves its planned networks from `container inspect`. That
+complete planned membership must be exact; if `network inspect` reports any
+members, its runtime view must also agree exactly. This detects an extra
+unstarted container without starting any reviewed payload. The check fails if
+Podman is not rootless; either application container has egress; the proxy lacks
+either network; any host port, mount, device, socket, added capability, writable
+root, restart policy, or missing resource limit is found; Chrome has a bypass;
+or the total limits exceed 6 CPUs/7 GiB. Capability removal
+must be present as `--cap-drop all` in each inspected create command and as the
+complete expected drop set in container state; a merely non-empty drop list does
+not pass. The full local image IDs must match the three separately recorded
+immutable values; mutable image names and short IDs do not pass. Host PID, IPC,
+user, UTS, or cgroup namespace modes do not pass where the runtime exposes them.
+For a `Cmd`-only image, Podman 4.9 reports the command's first value as `Path`
+and repeats the complete command in `Args`; the validator derives and binds that
+exact runtime representation from the immutable reviewed image metadata.
+
+After that check passes, start the proxy and Selenium, wait only for Selenium's
+internal `/status` endpoint, and attach to the runner once. Do not use `podman
+run`, because that would bypass the inspected `created` containers. Export only:
+
+- the pre-start safety JSON;
+- the runner's one-line JSON summary;
+- the proxy's safe host/status log;
+- image digests and the candidate wheel/local commit hashes.
+
+Do not export Selenium logs, browser profiles, page source, screenshots, `/tmp`,
+or any container filesystem. The runner suppresses collector stdout/stderr and
+prints only a bin-row count or a short redacted error. Review exported text with
+the same postcode/number/URL redactor before moving it to Windows.
+
+Remove only the exact three container and two network names after evidence is
+safe. Never use a global prune. A failed safety check is terminal for that run;
+do not add host networking, a host port, a bind mount, privilege, or a socket to
+make the canary pass.
diff --git a/tests/disposable_ha/LiveCanary.Containerfile b/tests/disposable_ha/LiveCanary.Containerfile
new file mode 100644
index 0000000000..3d8797151c
--- /dev/null
+++ b/tests/disposable_ha/LiveCanary.Containerfile
@@ -0,0 +1,11 @@
+ARG CANDIDATE_IMAGE
+FROM ${CANDIDATE_IMAGE}
+
+COPY tests/disposable_ha/live_canary_runner.py /opt/ukbcd/live_canary_runner.py
+
+ENV PYTHONDONTWRITEBYTECODE=1
+USER 1000:1000
+ENTRYPOINT ["python", "/opt/ukbcd/live_canary_runner.py", "--confirm-one-public-fixture-lookup"]
+
+LABEL org.opencontainers.image.title="UKBCD one-shot South Kesteven canary"
+LABEL org.opencontainers.image.source="local-only"
diff --git a/tests/disposable_ha/LiveProxy.Containerfile b/tests/disposable_ha/LiveProxy.Containerfile
new file mode 100644
index 0000000000..d82c1add11
--- /dev/null
+++ b/tests/disposable_ha/LiveProxy.Containerfile
@@ -0,0 +1,12 @@
+ARG PYTHON_IMAGE
+FROM ${PYTHON_IMAGE}
+
+COPY tests/disposable_ha/live_allowlist_proxy.py /opt/ukbcd/live_allowlist_proxy.py
+
+ENV PYTHONDONTWRITEBYTECODE=1
+USER 65532:65532
+ENTRYPOINT ["python3", "/opt/ukbcd/live_allowlist_proxy.py"]
+CMD ["--max-requests", "256", "--minimum-interval-ms", "25"]
+
+LABEL org.opencontainers.image.title="UKBCD South Kesteven allowlist proxy"
+LABEL org.opencontainers.image.source="local-only"
diff --git a/tests/disposable_ha/SouthKestevenFixtureCouncil.py b/tests/disposable_ha/SouthKestevenFixtureCouncil.py
new file mode 100644
index 0000000000..a4e54dbd62
--- /dev/null
+++ b/tests/disposable_ha/SouthKestevenFixtureCouncil.py
@@ -0,0 +1,26 @@
+"""Disposable loopback adapter around the production South Kesteven flow."""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+from uk_bin_collection.uk_bin_collection.councils.SouthKestevenDistrictCouncil import (
+ CouncilClass as ProductionCouncilClass,
+)
+
+
+class CouncilClass(ProductionCouncilClass):
+ """Use the production flow against the pod's loopback-only fixture service."""
+
+ BIN_DAY_URL = "http://127.0.0.1:8081/binday"
+ CHECKER_SCHEME = "http"
+ CHECKER_HOST = "127.0.0.1"
+
+ def parse_data(self, page: str, **kwargs) -> dict:
+ evidence_dir = os.environ.get("UKBCD_TEST_EVIDENCE_DIR")
+ if evidence_dir:
+ evidence_path = Path(evidence_dir) / "south_kesteven_scrape_calls"
+ with evidence_path.open("a", encoding="utf-8") as handle:
+ handle.write("1\n")
+ return super().parse_data(page, **kwargs)
diff --git a/tests/disposable_ha/WheelBuilder.Containerfile b/tests/disposable_ha/WheelBuilder.Containerfile
new file mode 100644
index 0000000000..a251262a9b
--- /dev/null
+++ b/tests/disposable_ha/WheelBuilder.Containerfile
@@ -0,0 +1,16 @@
+ARG PYTHON_IMAGE
+FROM ${PYTHON_IMAGE}
+
+COPY tests/disposable_ha/build-1.3.0-py3-none-any.whl /tmp/wheelhouse/
+COPY tests/disposable_ha/packaging-26.2-py3-none-any.whl /tmp/wheelhouse/
+COPY tests/disposable_ha/poetry_core-1.9.1-py3-none-any.whl /tmp/wheelhouse/
+COPY tests/disposable_ha/pyproject_hooks-1.2.0-py3-none-any.whl /tmp/wheelhouse/
+RUN PIP_NO_INDEX=1 python -m pip install --no-deps /tmp/wheelhouse/*.whl \
+ && rm -rf /tmp/wheelhouse
+
+COPY . /workspace
+WORKDIR /workspace
+RUN python -m build --wheel --no-isolation --outdir /dist
+
+LABEL org.opencontainers.image.title="UKBCD disposable offline wheel builder"
+LABEL org.opencontainers.image.source="local-only"
diff --git a/tests/disposable_ha/__init__.py b/tests/disposable_ha/__init__.py
new file mode 100644
index 0000000000..0fd535c39e
--- /dev/null
+++ b/tests/disposable_ha/__init__.py
@@ -0,0 +1 @@
+"""Disposable Home Assistant and canary validation helpers."""
diff --git a/tests/disposable_ha/collision_probe.py b/tests/disposable_ha/collision_probe.py
new file mode 100644
index 0000000000..95b355955a
--- /dev/null
+++ b/tests/disposable_ha/collision_probe.py
@@ -0,0 +1,88 @@
+"""Reproduce the raw collision, then prove the candidate fails before import."""
+
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+from uk_bin_collection.uk_bin_collection.dependency_validation import (
+ validate_websocket_client,
+)
+from uk_bin_collection.uk_bin_collection.exceptions import DependencyShadowingError
+
+
+def main() -> None:
+ evidence = Path(os.environ.get("UKBCD_TEST_EVIDENCE_DIR", "/evidence"))
+ evidence.mkdir(parents=True, exist_ok=True)
+ with tempfile.TemporaryDirectory() as temp_dir:
+ poison_root = Path(temp_dir)
+ poison = poison_root / "websocket"
+ poison.mkdir()
+ execution_marker = poison_root / "poison_executed"
+ (poison / "__init__.py").write_text(
+ "from pathlib import Path\n"
+ f"Path({str(execution_marker)!r}).write_text('executed')\n"
+ "from ..const import DOMAIN\n",
+ encoding="utf-8",
+ )
+
+ env = dict(os.environ)
+ env["PYTHONPATH"] = os.pathsep.join(
+ [str(poison_root), env.get("PYTHONPATH", "")]
+ )
+ raw = subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ "from selenium.webdriver.remote.websocket_connection import "
+ "WebSocketConnection",
+ ],
+ env=env,
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ raw_reproduced = (
+ raw.returncode != 0
+ and "attempted relative import beyond top-level package" in raw.stderr
+ and execution_marker.exists()
+ )
+
+ execution_marker.unlink(missing_ok=True)
+ sys.path.insert(0, str(poison_root))
+ sys.modules.pop("websocket", None)
+ try:
+ validate_websocket_client()
+ except DependencyShadowingError as exc:
+ typed_error = type(exc).__name__
+ typed_message = str(exc)
+ else:
+ typed_error = None
+ typed_message = None
+ finally:
+ sys.path.remove(str(poison_root))
+
+ candidate_blocked_before_execution = (
+ typed_error == "DependencyShadowingError" and not execution_marker.exists()
+ )
+
+ report = {
+ "raw_import_reproduced": raw_reproduced,
+ "raw_return_code": raw.returncode,
+ "candidate_error": typed_error,
+ "candidate_message": typed_message,
+ "candidate_blocked_before_poison_execution": candidate_blocked_before_execution,
+ }
+ (evidence / "collision_probe.json").write_text(
+ json.dumps(report, indent=2, sort_keys=True), encoding="utf-8"
+ )
+ if not raw_reproduced or not candidate_blocked_before_execution:
+ raise SystemExit(json.dumps(report, sort_keys=True))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/disposable_ha/containerignore b/tests/disposable_ha/containerignore
new file mode 100644
index 0000000000..b298ccccfb
--- /dev/null
+++ b/tests/disposable_ha/containerignore
@@ -0,0 +1,9 @@
+.git
+**/__pycache__
+**/*.pyc
+.pytest_cache
+.coverage
+build
+dist
+test_results_map.png
+SwaggerUI.png
diff --git a/tests/disposable_ha/fixture_server.py b/tests/disposable_ha/fixture_server.py
new file mode 100644
index 0000000000..0d081395c5
--- /dev/null
+++ b/tests/disposable_ha/fixture_server.py
@@ -0,0 +1,129 @@
+"""Loopback-only deterministic HTML service for the South Kesteven browser flow."""
+
+from __future__ import annotations
+
+import json
+import os
+from datetime import date, timedelta
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+
+HOST = "127.0.0.1"
+PORT = 8081
+EVIDENCE = Path(os.environ.get("UKBCD_TEST_EVIDENCE_DIR", "/evidence"))
+
+
+def _future_date(days: int) -> str:
+ return (date.today() + timedelta(days=days)).strftime("%A %d %B, %Y")
+
+
+def _checker_html() -> str:
+ first_date = _future_date(7)
+ second_date = _future_date(14)
+ return f"""
+Offline bin checker
+
+
+
+
+
+
+
+
+
+
+
+
+
+"""
+
+
+def _is_fixture_browser_user_agent(value: str) -> bool:
+ """Require the disposable browser marker, not merely a generic browser UA."""
+ normalized = value.casefold()
+ return "mozilla" in normalized and "ukbcd-disposable-fixture" in normalized
+
+
+class Handler(BaseHTTPRequestHandler):
+ """Serve only the three fixture routes needed by the validation."""
+
+ def do_GET(self) -> None: # noqa: N802 - BaseHTTPRequestHandler API
+ EVIDENCE.mkdir(parents=True, exist_ok=True)
+ with (EVIDENCE / "fixture_requests.jsonl").open(
+ "a", encoding="utf-8"
+ ) as handle:
+ user_agent = self.headers.get("User-Agent", "").lower()
+ handle.write(
+ json.dumps(
+ {
+ "path": self.path.split("?", 1)[0],
+ "browser": "mozilla" in user_agent,
+ "custom_user_agent": "ukbcd-disposable-fixture" in user_agent,
+ },
+ sort_keys=True,
+ )
+ + "\n"
+ )
+
+ if self.path == "/health":
+ self._send(200, "text/plain", "ok")
+ return
+ if self.path == "/binday":
+ user_agent = self.headers.get("User-Agent", "")
+ if not _is_fixture_browser_user_agent(user_agent):
+ self._send(403, "text/html", "403 Forbidden
")
+ return
+ self._send(
+ 200,
+ "text/html",
+ ''
+ "Postcode bin day checker",
+ )
+ return
+ if self.path == "/checker":
+ self._send(200, "text/html", _checker_html())
+ return
+ if self.path == "/static":
+ self._send(200, "application/json", '{"fixture": true}')
+ return
+ self._send(404, "text/plain", "not found")
+
+ def _send(self, status: int, content_type: str, body: str) -> None:
+ payload = body.encode("utf-8")
+ self.send_response(status)
+ self.send_header("Content-Type", f"{content_type}; charset=utf-8")
+ self.send_header("Content-Length", str(len(payload)))
+ self.end_headers()
+ self.wfile.write(payload)
+
+ def log_message(self, format: str, *args) -> None:
+ del format, args
+
+
+if __name__ == "__main__":
+ ThreadingHTTPServer((HOST, PORT), Handler).serve_forever()
diff --git a/tests/disposable_ha/import_all_councils.py b/tests/disposable_ha/import_all_councils.py
new file mode 100644
index 0000000000..2f0eb01cb0
--- /dev/null
+++ b/tests/disposable_ha/import_all_councils.py
@@ -0,0 +1,39 @@
+"""Import every shipped council module without performing network activity."""
+
+from __future__ import annotations
+
+import json
+
+from uk_bin_collection.uk_bin_collection.collect_data import (
+ import_council_module,
+ registered_council_modules,
+)
+
+
+def main() -> None:
+ failures: dict[str, str] = {}
+ modules = sorted(registered_council_modules())
+ for module_name in modules:
+ try:
+ import_council_module(module_name)
+ except Exception as exc: # Report the complete deterministic import inventory.
+ missing_name = getattr(exc, "name", None)
+ failures[module_name] = (
+ f"{type(exc).__name__}:{missing_name}"
+ if missing_name
+ else type(exc).__name__
+ )
+
+ report = {
+ "registered_council_count": len(modules),
+ "imported_council_count": len(modules) - len(failures),
+ "failure_count": len(failures),
+ "failures": failures,
+ }
+ print(json.dumps(report, indent=2, sort_keys=True))
+ if failures:
+ raise SystemExit(1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/disposable_ha/live_allowlist_proxy.py b/tests/disposable_ha/live_allowlist_proxy.py
new file mode 100644
index 0000000000..978d8ad34a
--- /dev/null
+++ b/tests/disposable_ha/live_allowlist_proxy.py
@@ -0,0 +1,347 @@
+"""Small fail-closed proxy for the separately approved live SKDC canary.
+
+The proxy deliberately has no general forwarding mode. It accepts only the two
+published South Kesteven origins, never records a request target or payload, and
+applies a small global connection delay/budget. It is intended to be the only
+container attached to both the canary's internal network and its egress network.
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import selectors
+import socket
+import socketserver
+import threading
+import time
+from dataclasses import dataclass
+from urllib.parse import urlsplit
+
+ALLOWED_HOSTS = frozenset(
+ {
+ "www.southkesteven.gov.uk",
+ "selfservice.southkesteven.gov.uk",
+ }
+)
+ALLOWED_PORTS = frozenset({80, 443})
+ALLOWED_HTTP_METHODS = frozenset({"GET", "HEAD", "OPTIONS", "POST"})
+MAX_HEADER_BYTES = 64 * 1024
+SOCKET_TIMEOUT_SECONDS = 15
+TUNNEL_IDLE_TIMEOUT_SECONDS = 45
+
+_LOGGER = logging.getLogger("ukbcd.live_allowlist_proxy")
+
+
+class ProxyRequestDenied(ValueError):
+ """The request is outside the deliberately tiny egress policy."""
+
+
+class ProxyRateLimitExceeded(RuntimeError):
+ """The one-shot canary exhausted its bounded proxy request budget."""
+
+
+@dataclass(frozen=True)
+class ProxyRequest:
+ """Validated upstream request metadata.
+
+ ``forward_header`` may contain a URL path and must never be logged.
+ """
+
+ method: str
+ host: str
+ port: int
+ is_tunnel: bool
+ forward_header: bytes | None
+
+
+class RequestLimiter:
+ """Serialize upstream connection starts and enforce a hard request budget."""
+
+ def __init__(
+ self,
+ *,
+ max_requests: int,
+ minimum_interval_seconds: float,
+ clock=time.monotonic,
+ sleeper=time.sleep,
+ ) -> None:
+ if max_requests < 1:
+ raise ValueError("max_requests must be positive")
+ if minimum_interval_seconds < 0:
+ raise ValueError("minimum_interval_seconds cannot be negative")
+ self._max_requests = max_requests
+ self._minimum_interval = minimum_interval_seconds
+ self._clock = clock
+ self._sleeper = sleeper
+ self._count = 0
+ self._last_started: float | None = None
+ self._lock = threading.Lock()
+
+ @property
+ def count(self) -> int:
+ return self._count
+
+ def acquire(self) -> None:
+ with self._lock:
+ if self._count >= self._max_requests:
+ raise ProxyRateLimitExceeded("proxy request budget exhausted")
+
+ now = self._clock()
+ if self._last_started is not None:
+ delay = self._minimum_interval - (now - self._last_started)
+ if delay > 0:
+ self._sleeper(delay)
+ now = self._clock()
+
+ self._count += 1
+ self._last_started = now
+
+
+def _parse_authority(authority: str) -> tuple[str, int]:
+ """Return a validated exact allowlisted host and explicit/default port."""
+ if not authority or any(character in authority for character in "/?#"):
+ raise ProxyRequestDenied("invalid proxy authority")
+
+ parsed = urlsplit(f"//{authority}")
+ if parsed.username is not None or parsed.password is not None:
+ raise ProxyRequestDenied("credentials are forbidden in proxy authorities")
+ if not parsed.hostname:
+ raise ProxyRequestDenied("proxy authority has no hostname")
+
+ host = parsed.hostname.casefold()
+ if host not in ALLOWED_HOSTS:
+ raise ProxyRequestDenied("hostname is not allowlisted")
+ try:
+ port = parsed.port
+ except ValueError as exc:
+ raise ProxyRequestDenied("invalid proxy port") from exc
+ port = 443 if port is None else port
+ if port not in ALLOWED_PORTS:
+ raise ProxyRequestDenied("port is not allowlisted")
+ return host, port
+
+
+def parse_proxy_request(header: bytes) -> ProxyRequest:
+ """Validate a single HTTP proxy header without contacting the network."""
+ if not header.endswith(b"\r\n\r\n") or len(header) > MAX_HEADER_BYTES:
+ raise ProxyRequestDenied("invalid proxy header framing")
+ try:
+ text = header.decode("iso-8859-1")
+ except UnicodeDecodeError as exc: # pragma: no cover - ISO-8859-1 is total
+ raise ProxyRequestDenied("invalid proxy header encoding") from exc
+
+ lines = text[:-4].split("\r\n")
+ if not lines:
+ raise ProxyRequestDenied("missing request line")
+ request_parts = lines[0].split(" ")
+ if len(request_parts) != 3:
+ raise ProxyRequestDenied("invalid request line")
+ method, target, version = request_parts
+ method = method.upper()
+ if version not in {"HTTP/1.0", "HTTP/1.1"}:
+ raise ProxyRequestDenied("unsupported HTTP version")
+ if any(line.startswith((" ", "\t")) for line in lines[1:]):
+ raise ProxyRequestDenied("folded proxy headers are forbidden")
+
+ if method == "CONNECT":
+ host, port = _parse_authority(target)
+ return ProxyRequest(method, host, port, True, None)
+
+ if method not in ALLOWED_HTTP_METHODS:
+ raise ProxyRequestDenied("HTTP method is not allowlisted")
+
+ parsed = urlsplit(target)
+ if (
+ parsed.scheme.casefold() != "http"
+ or not parsed.hostname
+ or parsed.username is not None
+ or parsed.password is not None
+ or parsed.fragment
+ ):
+ raise ProxyRequestDenied("only absolute allowlisted HTTP targets are accepted")
+ host = parsed.hostname.casefold()
+ if host not in ALLOWED_HOSTS:
+ raise ProxyRequestDenied("hostname is not allowlisted")
+ try:
+ port = parsed.port or 80
+ except ValueError as exc:
+ raise ProxyRequestDenied("invalid proxy port") from exc
+ if port not in ALLOWED_PORTS:
+ raise ProxyRequestDenied("port is not allowlisted")
+
+ origin_target = parsed.path or "/"
+ if parsed.query:
+ origin_target = f"{origin_target}?{parsed.query}"
+
+ forwarded_headers: list[str] = []
+ for line in lines[1:]:
+ if not line or ":" not in line:
+ if line:
+ raise ProxyRequestDenied("malformed proxy header")
+ continue
+ name, value = line.split(":", 1)
+ normalized_name = name.strip().casefold()
+ if normalized_name in {
+ "connection",
+ "host",
+ "proxy-authorization",
+ "proxy-connection",
+ }:
+ continue
+ forwarded_headers.append(f"{name.strip()}:{value}")
+
+ host_header = host if port == 80 else f"{host}:{port}"
+ forward_lines = [
+ f"{method} {origin_target} {version}",
+ f"Host: {host_header}",
+ *forwarded_headers,
+ "Connection: close",
+ "",
+ "",
+ ]
+ return ProxyRequest(
+ method,
+ host,
+ port,
+ False,
+ "\r\n".join(forward_lines).encode("iso-8859-1"),
+ )
+
+
+def safe_log_line(host: str | None, status: int) -> str:
+ """Return a log line containing no URL, payload, or denied hostname."""
+ safe_host = host if host in ALLOWED_HOSTS else "[DENIED]"
+ return f"host={safe_host} status={int(status)}"
+
+
+def _read_header(connection: socket.socket) -> tuple[bytes, bytes]:
+ buffer = bytearray()
+ while b"\r\n\r\n" not in buffer:
+ chunk = connection.recv(4096)
+ if not chunk:
+ raise ProxyRequestDenied("connection ended before proxy header")
+ buffer.extend(chunk)
+ if len(buffer) > MAX_HEADER_BYTES:
+ raise ProxyRequestDenied("proxy header is too large")
+ marker = buffer.index(b"\r\n\r\n") + 4
+ return bytes(buffer[:marker]), bytes(buffer[marker:])
+
+
+def _relay(left: socket.socket, right: socket.socket) -> None:
+ selector = selectors.DefaultSelector()
+ selector.register(left, selectors.EVENT_READ, right)
+ selector.register(right, selectors.EVENT_READ, left)
+ try:
+ while True:
+ events = selector.select(TUNNEL_IDLE_TIMEOUT_SECONDS)
+ if not events:
+ return
+ for key, _ in events:
+ source = key.fileobj
+ destination = key.data
+ chunk = source.recv(64 * 1024)
+ if not chunk:
+ return
+ destination.sendall(chunk)
+ finally:
+ selector.close()
+
+
+class AllowlistProxyHandler(socketserver.BaseRequestHandler):
+ """Handle one validated tunnel or one close-delimited HTTP exchange."""
+
+ server: "AllowlistProxyServer"
+
+ def handle(self) -> None:
+ self.request.settimeout(SOCKET_TIMEOUT_SECONDS)
+ allowed_host: str | None = None
+ try:
+ header, buffered_body = _read_header(self.request)
+ decision = parse_proxy_request(header)
+ allowed_host = decision.host
+ self.server.limiter.acquire()
+
+ with socket.create_connection(
+ (decision.host, decision.port), timeout=SOCKET_TIMEOUT_SECONDS
+ ) as upstream:
+ upstream.settimeout(TUNNEL_IDLE_TIMEOUT_SECONDS)
+ self.request.settimeout(TUNNEL_IDLE_TIMEOUT_SECONDS)
+ if decision.is_tunnel:
+ self.request.sendall(b"HTTP/1.1 200 Connection Established\r\n\r\n")
+ else:
+ assert decision.forward_header is not None
+ upstream.sendall(decision.forward_header)
+ if buffered_body:
+ upstream.sendall(buffered_body)
+ _LOGGER.info(safe_log_line(decision.host, 200))
+ _relay(self.request, upstream)
+ except ProxyRateLimitExceeded:
+ self._send_error(429)
+ _LOGGER.warning(safe_log_line(allowed_host, 429))
+ except ProxyRequestDenied:
+ self._send_error(403)
+ _LOGGER.warning(safe_log_line(None, 403))
+ except (OSError, TimeoutError):
+ self._send_error(502)
+ _LOGGER.warning(safe_log_line(allowed_host, 502))
+
+ def _send_error(self, status: int) -> None:
+ reason = {403: "Forbidden", 429: "Too Many Requests", 502: "Bad Gateway"}[
+ status
+ ]
+ response = (
+ f"HTTP/1.1 {status} {reason}\r\n"
+ "Content-Length: 0\r\n"
+ "Connection: close\r\n\r\n"
+ ).encode("ascii")
+ try:
+ self.request.sendall(response)
+ except OSError:
+ pass
+
+
+class AllowlistProxyServer(socketserver.ThreadingTCPServer):
+ """Threaded proxy server with a process-wide limiter."""
+
+ allow_reuse_address = False
+ daemon_threads = True
+
+ def __init__(self, address, handler, *, limiter: RequestLimiter):
+ self.limiter = limiter
+ super().__init__(address, handler)
+
+ def handle_error(self, request, client_address) -> None:
+ """Suppress default tracebacks so unexpected failures cannot leak data."""
+ del request, client_address
+ _LOGGER.error(safe_log_line(None, 500))
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--listen", default="0.0.0.0")
+ parser.add_argument("--port", type=int, default=3128)
+ parser.add_argument("--max-requests", type=int, default=256)
+ parser.add_argument("--minimum-interval-ms", type=int, default=25)
+ args = parser.parse_args()
+
+ if not 1 <= args.port <= 65535:
+ parser.error("port must be between 1 and 65535")
+ if not 1 <= args.max_requests <= 256:
+ parser.error("max-requests must be between 1 and 256")
+ if not 25 <= args.minimum_interval_ms <= 10_000:
+ parser.error("minimum-interval-ms must be between 25 and 10000")
+
+ logging.basicConfig(level=logging.INFO, format="%(message)s")
+ limiter = RequestLimiter(
+ max_requests=args.max_requests,
+ minimum_interval_seconds=args.minimum_interval_ms / 1000,
+ )
+ with AllowlistProxyServer(
+ (args.listen, args.port), AllowlistProxyHandler, limiter=limiter
+ ) as server:
+ server.serve_forever(poll_interval=0.25)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/disposable_ha/live_canary_runner.py b/tests/disposable_ha/live_canary_runner.py
new file mode 100644
index 0000000000..22bde24b60
--- /dev/null
+++ b/tests/disposable_ha/live_canary_runner.py
@@ -0,0 +1,230 @@
+"""One-shot, redaction-safe South Kesteven public-fixture canary runner.
+
+This module is intentionally not part of the normal test suite's network path.
+Running ``main`` requires both a deliberate command-line acknowledgement and a
+fixed environment acknowledgement. Those controls record intent for the single
+plan-approved run; they are not an additional approval gate. Address values can
+only come from the repository's already-public
+``uk_bin_collection/tests/input.json`` entry.
+"""
+
+from __future__ import annotations
+
+import argparse
+import contextlib
+import io
+import json
+import os
+import re
+import sys
+import time
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Callable
+from urllib.parse import urlsplit
+
+COUNCIL_NAME = "SouthKestevenDistrictCouncil"
+PUBLIC_FIXTURE_PATH = Path("/workspace/uk_bin_collection/tests/input.json")
+PUBLIC_BINDAY_URL = "https://www.southkesteven.gov.uk/binday"
+SELENIUM_INTERNAL_URL = "http://selenium:4444"
+APPROVAL_ENVIRONMENT_VARIABLE = "UKBCD_LIVE_CANARY_APPROVED"
+APPROVAL_VALUE = "one-public-fixture-lookup"
+LOCK_PATH = Path("/tmp/ukbcd-live-canary-one-shot.lock")
+MINIMUM_START_DELAY_SECONDS = 5.0
+
+_URL_PATTERN = re.compile(r"(?i)\bhttps?://[^\s\"'<>]+")
+_UK_POSTCODE_PATTERN = re.compile(
+ r"(?i)\b(?:GIR\s?0AA|(?:[A-Z]{1,2}[0-9][0-9A-Z]?)\s?[0-9][A-Z]{2})\b"
+)
+_HOUSEHOLD_IDENTIFIER_PATTERN = re.compile(
+ r"(?i)\b((?:uprn|usrn)\s*[:=]?\s*)[0-9]{6,15}\b"
+)
+
+
+@dataclass(frozen=True)
+class PublicFixture:
+ """The public South Kesteven canary fields committed to input.json."""
+
+ postcode: str
+ house_number: str
+ url: str
+ web_driver: str
+
+
+def load_public_fixture(path: Path = PUBLIC_FIXTURE_PATH) -> PublicFixture:
+ """Load and fail-closed validate the repository's public fixture."""
+ raw = json.loads(path.read_text(encoding="utf-8"))
+ entry = raw.get(COUNCIL_NAME)
+ if not isinstance(entry, dict):
+ raise ValueError("public South Kesteven fixture is missing")
+
+ postcode = entry.get("postcode")
+ house_number = entry.get("house_number")
+ url = entry.get("url")
+ web_driver = entry.get("web_driver")
+ if not all(
+ isinstance(value, str) and value.strip()
+ for value in (postcode, house_number, url, web_driver)
+ ):
+ raise ValueError("public South Kesteven fixture is incomplete")
+ if entry.get("skip_get_url") is not True:
+ raise ValueError(
+ "public South Kesteven fixture must skip the generic HTTP preflight"
+ )
+ if url != PUBLIC_BINDAY_URL:
+ raise ValueError("public South Kesteven URL changed from the reviewed origin")
+
+ driver = urlsplit(web_driver)
+ if (
+ driver.scheme != "http"
+ or driver.hostname != "selenium"
+ or driver.port != 4444
+ or driver.path not in {"", "/"}
+ or driver.query
+ or driver.fragment
+ or driver.username is not None
+ or driver.password is not None
+ ):
+ raise ValueError("public WebDriver fixture is not the isolated Selenium alias")
+
+ return PublicFixture(
+ postcode=postcode.strip(),
+ house_number=house_number.strip(),
+ url=url,
+ web_driver=SELENIUM_INTERNAL_URL,
+ )
+
+
+def redact_output(value: object, fixture: PublicFixture | None = None) -> str:
+ """Remove household values and all URLs from text intended for output."""
+ text = str(value)
+ text = _URL_PATTERN.sub("[REDACTED_URL]", text)
+ text = _UK_POSTCODE_PATTERN.sub("[REDACTED_POSTCODE]", text)
+ text = _HOUSEHOLD_IDENTIFIER_PATTERN.sub(r"\1[REDACTED]", text)
+
+ if fixture is not None:
+ explicit_values = {
+ fixture.postcode,
+ fixture.postcode.replace(" ", ""),
+ fixture.house_number,
+ fixture.url,
+ fixture.web_driver,
+ }
+ for sensitive in sorted(explicit_values, key=len, reverse=True):
+ if not sensitive:
+ continue
+ text = re.sub(re.escape(sensitive), "[REDACTED]", text, flags=re.I)
+ return text
+
+
+def parse_collection_result(result: object) -> dict[str, object]:
+ """Return a deliberately non-identifying summary of collector JSON."""
+ payload = json.loads(result) if isinstance(result, str) else result
+ if not isinstance(payload, dict) or not isinstance(payload.get("bins"), list):
+ raise ValueError("collector did not return a bins list")
+ bins = payload["bins"]
+ for row in bins:
+ if not isinstance(row, dict):
+ raise ValueError("collector returned a malformed bin row")
+ if not isinstance(row.get("type"), str) or not isinstance(
+ row.get("collectionDate"), str
+ ):
+ raise ValueError("collector returned an incomplete bin row")
+ return {
+ "status": "passed",
+ "council": COUNCIL_NAME,
+ "logical_lookups": 1,
+ "bin_count": len(bins),
+ }
+
+
+def _invoke_collector(fixture: PublicFixture) -> object:
+ """Invoke the production CLI application once using only public values."""
+ from uk_bin_collection.uk_bin_collection.collect_data import UKBinCollectionApp
+
+ app = UKBinCollectionApp()
+ app.set_args(
+ [
+ COUNCIL_NAME,
+ fixture.url,
+ "--postcode",
+ fixture.postcode,
+ "--number",
+ fixture.house_number,
+ "--skip_get_url",
+ "--web_driver",
+ fixture.web_driver,
+ ]
+ )
+ return app.run()
+
+
+def reserve_one_shot(path: Path = LOCK_PATH) -> None:
+ """Fail if this immutable canary container already attempted its lookup."""
+ descriptor = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
+ with os.fdopen(descriptor, "w", encoding="ascii") as lock_file:
+ lock_file.write("reserved\n")
+
+
+def run_canary(
+ fixture: PublicFixture,
+ *,
+ invoke: Callable[[PublicFixture], object] = _invoke_collector,
+ sleeper: Callable[[float], None] = time.sleep,
+ minimum_delay_seconds: float = MINIMUM_START_DELAY_SECONDS,
+) -> dict[str, object]:
+ """Perform exactly one deliberately delayed logical lookup."""
+ if minimum_delay_seconds < MINIMUM_START_DELAY_SECONDS:
+ raise ValueError("live canary delay cannot be reduced below five seconds")
+ sleeper(minimum_delay_seconds)
+
+ # Do not allow third-party or legacy print/log output to escape. Only the
+ # fixed summary below is emitted by main().
+ captured = io.StringIO()
+ with contextlib.redirect_stdout(captured), contextlib.redirect_stderr(captured):
+ result = invoke(fixture)
+ return parse_collection_result(result)
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--confirm-one-public-fixture-lookup",
+ action="store_true",
+ help="Required acknowledgement for the one plan-approved live canary.",
+ )
+ args = parser.parse_args()
+
+ if (
+ not args.confirm_one_public_fixture_lookup
+ or os.environ.get(APPROVAL_ENVIRONMENT_VARIABLE) != APPROVAL_VALUE
+ ):
+ parser.error(
+ "live canary requires the command acknowledgement and fixed approval environment value"
+ )
+
+ fixture: PublicFixture | None = None
+ try:
+ reserve_one_shot()
+ fixture = load_public_fixture()
+ summary = run_canary(fixture)
+ print(json.dumps(summary, sort_keys=True))
+ return 0
+ except Exception as exc:
+ # Never emit a traceback or arbitrary response content. The class and a
+ # short redacted message are enough to correlate with separately-redacted
+ # scraper diagnostics.
+ message = redact_output(exc, fixture)[:240]
+ failure = {
+ "status": "failed",
+ "council": COUNCIL_NAME,
+ "logical_lookups": 0 if fixture is None else 1,
+ "error_type": type(exc).__name__,
+ "message": message,
+ }
+ print(json.dumps(failure, sort_keys=True))
+ return 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tests/disposable_ha/live_canary_safety_check.py b/tests/disposable_ha/live_canary_safety_check.py
new file mode 100644
index 0000000000..8a382cc61b
--- /dev/null
+++ b/tests/disposable_ha/live_canary_safety_check.py
@@ -0,0 +1,685 @@
+"""Validate the live-canary topology before any container is started.
+
+Unlike the fully-offline HA pod, this canary uses two rootless Podman networks:
+runner/Selenium are attached only to an internal network, while the minimal
+allowlist proxy is the sole dual-homed container. The validator is intentionally
+strict and writes only topology metadata, never fixture values or container logs.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+import subprocess
+from pathlib import Path
+
+PROXY_SERVER_ARGUMENT = "--proxy-server=http://ukbcd-live-proxy:3128"
+# Chromium has an implicit loopback bypass. The special <-loopback> token
+# subtracts it, leaving no hosts outside the proxy policy.
+PROXY_NO_BYPASS_ARGUMENT = "--proxy-bypass-list=<-loopback>"
+APPROVAL_ENVIRONMENT_VARIABLE = "UKBCD_LIVE_CANARY_APPROVED"
+APPROVAL_VALUE = "one-public-fixture-lookup"
+HOST_NAMESPACE_FIELDS = (
+ "PidMode",
+ "IpcMode",
+ "UsernsMode",
+ "UTSMode",
+ "CgroupnsMode",
+ "CgroupMode",
+)
+RUNNER_PROCESS = (
+ "python",
+ (
+ "/opt/ukbcd/live_canary_runner.py",
+ "--confirm-one-public-fixture-lookup",
+ ),
+)
+PROXY_PROCESS = (
+ "python3",
+ (
+ "/opt/ukbcd/live_allowlist_proxy.py",
+ "--max-requests",
+ "256",
+ "--minimum-interval-ms",
+ "25",
+ ),
+)
+_SHA256_PATTERN = re.compile(r"^(?:sha256:)?([0-9a-fA-F]{64})$")
+_CAPABILITY_PATTERN = re.compile(r"^(?:CAP_)?[A-Z][A-Z0-9_]*$")
+SENSITIVE_ENVIRONMENT_NAME_MARKERS = frozenset(
+ {
+ "token",
+ "password",
+ "passwd",
+ "secret",
+ "credential",
+ "postcode",
+ "house_number",
+ "housenumber",
+ "house_name",
+ "housename",
+ "paon",
+ "uprn",
+ "usrn",
+ "address",
+ }
+)
+
+
+def _podman_json(*arguments: str):
+ result = subprocess.run(
+ ["podman", *arguments],
+ check=True,
+ capture_output=True,
+ text=True,
+ )
+ return json.loads(result.stdout)
+
+
+def _inspect_container(name: str) -> dict:
+ return _podman_json("inspect", name)[0]
+
+
+def _inspect_image(image_id: str) -> dict:
+ return _podman_json("image", "inspect", image_id)[0]
+
+
+def _inspect_network(name: str) -> dict:
+ return _podman_json("network", "inspect", name)[0]
+
+
+def _require(condition: bool, message: str) -> None:
+ if not condition:
+ raise RuntimeError(message)
+
+
+def _is_internal(network: dict) -> bool:
+ return bool(network.get("internal", network.get("Internal", False)))
+
+
+def _is_rootless(info: dict) -> bool:
+ reported_values: list[object] = []
+ if "rootless" in info.get("host", {}).get("security", {}):
+ reported_values.append(info["host"]["security"]["rootless"])
+ if "Rootless" in info.get("Host", {}).get("Security", {}):
+ reported_values.append(info["Host"]["Security"]["Rootless"])
+ return bool(reported_values) and all(value is True for value in reported_values)
+
+
+def _attached_networks(container: dict) -> set[str]:
+ return set(container.get("NetworkSettings", {}).get("Networks", {}))
+
+
+def _environment(container: dict) -> set[str]:
+ return set(container.get("Config", {}).get("Env") or [])
+
+
+def _environment_mapping(subject: dict) -> dict[str, str]:
+ environment: dict[str, str] = {}
+ for raw_value in subject.get("Config", {}).get("Env") or []:
+ variable_name, separator, value = str(raw_value).partition("=")
+ if variable_name:
+ environment[variable_name] = value if separator else ""
+ return environment
+
+
+def _is_sensitive_environment_name(variable_name: str) -> bool:
+ normalized_name = variable_name.casefold()
+ return any(
+ marker in normalized_name for marker in SENSITIVE_ENVIRONMENT_NAME_MARKERS
+ )
+
+
+def _require_no_sensitive_environment_overrides(
+ name: str, container: dict, image: dict
+) -> None:
+ """Reject sensitive runtime env not identically baked into the reviewed image."""
+ container_environment = _environment_mapping(container)
+ reviewed_environment = _environment_mapping(image)
+ sensitive_overrides = sorted(
+ variable_name
+ for variable_name, value in container_environment.items()
+ if _is_sensitive_environment_name(variable_name)
+ and reviewed_environment.get(variable_name) != value
+ )
+ _require(
+ not sensitive_overrides,
+ f"{name} receives a sensitive environment override: {sensitive_overrides}",
+ )
+
+
+def _network_members(network: dict) -> set[str]:
+ """Extract actual container names from a Podman network-inspect result."""
+ raw_members = network.get("containers", network.get("Containers"))
+ if isinstance(raw_members, dict):
+ rows = raw_members.values()
+ elif isinstance(raw_members, list):
+ rows = raw_members
+ else:
+ return set()
+
+ members: set[str] = set()
+ for row in rows:
+ if not isinstance(row, dict):
+ continue
+ name = row.get("name", row.get("Name"))
+ if isinstance(name, str) and name:
+ members.add(name)
+ return members
+
+
+def _all_container_names() -> set[str]:
+ """Return every local container name from Podman's JSON inventory."""
+ rows = _podman_json("ps", "--all", "--format", "json")
+ _require(
+ isinstance(rows, list), "Podman returned a malformed container inventory"
+ )
+
+ names: set[str] = set()
+ for row in rows:
+ _require(isinstance(row, dict), "Podman returned a malformed container row")
+ raw_names = row.get("Names", row.get("names"))
+ if isinstance(raw_names, str):
+ row_names = [raw_names]
+ elif isinstance(raw_names, list) and all(
+ isinstance(name, str) for name in raw_names
+ ):
+ row_names = raw_names
+ else:
+ raise RuntimeError("Podman returned malformed container names")
+ names.update(name for name in row_names if name)
+ return names
+
+
+def _planned_network_members(network_name: str) -> set[str]:
+ """Resolve unstarted network intent from every container inspect record.
+
+ Podman 4.9 does not populate ``network inspect``'s container map until a
+ container starts. Scanning the complete local inventory retains exact
+ extra-container detection while the reviewed application containers remain
+ in the required ``created`` state.
+ """
+ return {
+ name
+ for name in _all_container_names()
+ if network_name in _attached_networks(_inspect_container(name))
+ }
+
+
+def _validated_network_members(
+ network_name: str, network: dict, expected: set[str]
+) -> tuple[set[str], set[str], str]:
+ """Validate exact planned membership and reconcile runtime membership."""
+ planned = _planned_network_members(network_name)
+ reported = _network_members(network)
+ _require_exact_members(network_name, planned, expected)
+ if reported:
+ _require_exact_members(f"{network_name} runtime", reported, planned)
+ source = "network-inspect-and-container-inspect"
+ else:
+ source = "all-container-inspect-unstarted-intent"
+ return planned, reported, source
+
+
+def _require_exact_members(boundary: str, actual: set[str], expected: set[str]) -> None:
+ missing = expected - actual
+ extra = actual - expected
+ _require(
+ not missing and not extra,
+ f"{boundary} membership mismatch; missing={sorted(missing)} extra={sorted(extra)}",
+ )
+
+
+def _normalise_capability(value: object) -> str:
+ capability = str(value).strip().upper()
+ return capability.removeprefix("CAP_")
+
+
+def _parse_capabilities(value: object) -> frozenset[str]:
+ """Parse one concrete capability universe reported by Podman."""
+ if isinstance(value, str):
+ raw_capabilities = [] if not value.strip() else value.split(",")
+ elif isinstance(value, list) and all(isinstance(item, str) for item in value):
+ raw_capabilities = value
+ else:
+ raise RuntimeError("Podman reports malformed rootless capabilities")
+
+ capabilities: set[str] = set()
+ for raw_capability in raw_capabilities:
+ normalized_input = raw_capability.strip().upper()
+ _require(
+ bool(_CAPABILITY_PATTERN.fullmatch(normalized_input)),
+ "Podman reports malformed rootless capabilities",
+ )
+ capability = _normalise_capability(normalized_input)
+ _require(
+ capability != "ALL",
+ "Podman rootless capability universe must contain concrete names",
+ )
+ capabilities.add(capability)
+ return frozenset(capabilities)
+
+
+def _rootless_capabilities(info: dict) -> frozenset[str]:
+ """Return the exact capability universe advertised by the rootless engine."""
+ reported: list[frozenset[str]] = []
+ lower_security = info.get("host", {}).get("security", {})
+ if "capabilities" in lower_security:
+ reported.append(_parse_capabilities(lower_security["capabilities"]))
+ upper_security = info.get("Host", {}).get("Security", {})
+ if "Capabilities" in upper_security:
+ reported.append(_parse_capabilities(upper_security["Capabilities"]))
+
+ _require(bool(reported), "Podman does not report rootless capabilities")
+ _require(
+ all(value == reported[0] for value in reported[1:]),
+ "Podman reports conflicting rootless capability universes",
+ )
+ return reported[0]
+
+
+def _create_command_requests_cap_drop_all(container: dict) -> bool:
+ command = container.get("Config", {}).get("CreateCommand") or []
+ if not isinstance(command, list):
+ return False
+ tokens = [str(token).casefold() for token in command]
+ if "--cap-drop=all" in tokens:
+ return True
+ return any(
+ token == "--cap-drop" and index + 1 < len(tokens) and tokens[index + 1] == "all"
+ for index, token in enumerate(tokens)
+ )
+
+
+def _cap_drop_covers_available_set(
+ container: dict, available_capabilities: frozenset[str]
+) -> bool:
+ host = container.get("HostConfig", {})
+ raw_drops = host.get("CapDrop")
+ _require(
+ isinstance(raw_drops, list)
+ and all(isinstance(value, str) for value in raw_drops),
+ "Podman reports malformed dropped capabilities",
+ )
+ drops: set[str] = set()
+ for value in raw_drops:
+ normalized_input = value.strip().upper()
+ _require(
+ bool(_CAPABILITY_PATTERN.fullmatch(normalized_input)),
+ "Podman reports malformed dropped capabilities",
+ )
+ drops.add(_normalise_capability(normalized_input))
+ return "ALL" in drops or available_capabilities <= drops
+
+
+def _require_cap_drop_all(
+ name: str, container: dict, available_capabilities: frozenset[str]
+) -> None:
+ cap_add = container.get("HostConfig", {}).get("CapAdd")
+ _require(
+ isinstance(cap_add, list),
+ f"{name} reports malformed added capabilities",
+ )
+ _require(
+ _create_command_requests_cap_drop_all(container),
+ f"{name} was not created with --cap-drop all",
+ )
+ _require(
+ not cap_add,
+ f"{name} adds Linux capabilities",
+ )
+ _require(
+ _cap_drop_covers_available_set(container, available_capabilities),
+ f"{name} does not report the complete rootless capability drop set",
+ )
+
+
+def _host_namespace_modes(container: dict) -> dict[str, str]:
+ host = container.get("HostConfig", {})
+ return {
+ field: str(host[field])
+ for field in HOST_NAMESPACE_FIELDS
+ if field in host
+ and str(host[field]).strip().casefold().split(":", 1)[0] == "host"
+ }
+
+
+def _require_no_host_namespaces(name: str, container: dict) -> None:
+ host_modes = _host_namespace_modes(container)
+ _require(
+ not host_modes,
+ f"{name} joins host namespaces: {sorted(host_modes)}",
+ )
+
+
+def _normalise_image_id(value: object) -> str | None:
+ text = str(value).strip()
+ match = _SHA256_PATTERN.fullmatch(text)
+ if not match:
+ return None
+ return f"sha256:{match.group(1).lower()}"
+
+
+def _inspected_image_ids(container: dict) -> set[str]:
+ identifiers: set[str] = set()
+ for field in ("Image", "ImageID", "ImageDigest", "Digest"):
+ if field not in container:
+ continue
+ normalized = _normalise_image_id(container[field])
+ if normalized is not None:
+ identifiers.add(normalized)
+ return identifiers
+
+
+def _require_expected_image(name: str, container: dict, expected: str) -> str:
+ normalized_expected = _normalise_image_id(expected)
+ _require(
+ normalized_expected is not None,
+ f"{name} expected image must be a full immutable SHA-256 ID or digest",
+ )
+ actual_ids = _inspected_image_ids(container)
+ _require(bool(actual_ids), f"{name} inspect data has no immutable image ID")
+ _require(
+ normalized_expected in actual_ids,
+ f"{name} immutable image ID does not match the reviewed value",
+ )
+ return normalized_expected
+
+
+def _require_exact_process(
+ name: str,
+ container: dict,
+ expected: tuple[str, tuple[str, ...]],
+) -> None:
+ actual = (str(container.get("Path", "")), tuple(container.get("Args") or []))
+ _require(actual == expected, f"{name} entrypoint or arguments differ from policy")
+
+
+def _process_sha256(container: dict) -> str:
+ payload = {
+ "path": str(container.get("Path", "")),
+ "args": [str(value) for value in container.get("Args") or []],
+ }
+ serialized = json.dumps(
+ payload, ensure_ascii=False, separators=(",", ":"), sort_keys=True
+ ).encode("utf-8")
+ return f"sha256:{hashlib.sha256(serialized).hexdigest()}"
+
+
+def _require_expected_process(name: str, container: dict, expected: str) -> str:
+ normalized_expected = _normalise_image_id(expected)
+ _require(
+ normalized_expected is not None,
+ f"{name} expected process must be a full SHA-256 digest",
+ )
+ actual = _process_sha256(container)
+ _require(
+ actual == normalized_expected,
+ f"{name} entrypoint or arguments differ from the reviewed payload",
+ )
+ return actual
+
+
+def _reviewed_image_process_sha256(image: dict) -> str:
+ config = image.get("Config", {})
+ entrypoint = config.get("Entrypoint") or []
+ command = config.get("Cmd") or []
+ _require(
+ isinstance(entrypoint, list)
+ and all(isinstance(value, str) for value in entrypoint)
+ and isinstance(command, list)
+ and all(isinstance(value, str) for value in command),
+ "reviewed Selenium image process metadata is malformed",
+ )
+ if entrypoint:
+ process = [*entrypoint, *command]
+ else:
+ # Podman 4.9 represents a Cmd-only image with the command's first value
+ # as Path and the complete command (including argv[0]) as Args. Bind to
+ # that exact inspected representation instead of assuming Docker's
+ # Path/Args normalization.
+ process = [command[0], *command] if command else []
+ _require(
+ bool(process) and bool(process[0].strip()),
+ "reviewed Selenium image has no default process",
+ )
+ return _process_sha256({"Path": process[0], "Args": process[1:]})
+
+
+def _validate_container(
+ name: str,
+ container: dict,
+ *,
+ expected_networks: set[str],
+ available_capabilities: frozenset[str],
+) -> dict:
+ host = container["HostConfig"]
+ _require(container["State"]["Status"] == "created", f"{name} already started")
+ _require(not container.get("Pod"), f"{name} unexpectedly belongs to a pod")
+ _require(not host["Privileged"], f"{name} is privileged")
+ _require(host["ReadonlyRootfs"], f"{name} root filesystem is writable")
+ _require(not host.get("PortBindings"), f"{name} publishes a host port")
+ _require(not host.get("PublishAllPorts"), f"{name} publishes all ports")
+ _require(not host.get("Devices"), f"{name} receives a host device")
+ _require_cap_drop_all(name, container, available_capabilities)
+ _require_no_host_namespaces(name, container)
+ _require(
+ "no-new-privileges" in host.get("SecurityOpt", []),
+ f"{name} permits privilege escalation",
+ )
+ _require(
+ host.get("RestartPolicy", {}).get("Name") in ("", "no"),
+ f"{name} has a restart policy",
+ )
+ _require(host.get("PidsLimit", 0) > 0, f"{name} has no PID limit")
+ _require(host.get("NanoCpus", 0) > 0, f"{name} has no CPU limit")
+ _require(host.get("Memory", 0) > 0, f"{name} has no memory limit")
+ _require(not container.get("Mounts"), f"{name} has a mount")
+ _require(
+ host.get("NetworkMode") not in {"host", "none"}
+ and not str(host.get("NetworkMode", "")).startswith("container:"),
+ f"{name} uses an unexpected network namespace",
+ )
+
+ networks = _attached_networks(container)
+ _require(
+ networks == expected_networks,
+ f"{name} networks differ from policy: {sorted(networks)}",
+ )
+
+ serialized = json.dumps(container).casefold()
+ _require("podman.sock" not in serialized, f"{name} receives the Podman socket")
+ _require("docker.sock" not in serialized, f"{name} receives the Docker socket")
+ _require("/mnt/" not in serialized, f"{name} references a Windows-drive mount")
+
+ return {
+ "state": "created",
+ "read_only_rootfs": True,
+ "privileged": False,
+ "networks": sorted(networks),
+ "published_ports": 0,
+ "mounts": 0,
+ "devices": 0,
+ "capabilities_added": 0,
+ "capabilities_dropped": "all",
+ "rootless_capabilities_dropped": sorted(available_capabilities),
+ "cap_drop_all": True,
+ "host_namespaces": [],
+ "no_new_privileges": True,
+ "pid_limit": host["PidsLimit"],
+ "cpu_limit": host["NanoCpus"] / 1_000_000_000,
+ "memory_limit_bytes": host["Memory"],
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--internal-network", required=True)
+ parser.add_argument("--egress-network", required=True)
+ parser.add_argument("--runner", required=True)
+ parser.add_argument("--selenium", required=True)
+ parser.add_argument("--proxy", required=True)
+ parser.add_argument("--expected-proxy-image", required=True)
+ parser.add_argument("--expected-selenium-image", required=True)
+ parser.add_argument("--expected-runner-image", required=True)
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args()
+
+ info = _podman_json("info", "--format", "json")
+ _require(_is_rootless(info), "Podman is not running rootless")
+ available_capabilities = _rootless_capabilities(info)
+
+ internal = _inspect_network(args.internal_network)
+ egress = _inspect_network(args.egress_network)
+ _require(_is_internal(internal), "runner/Selenium network is not internal")
+ _require(not _is_internal(egress), "proxy egress network is marked internal")
+ _require(
+ internal.get("driver", internal.get("Driver")) == "bridge",
+ "runner/Selenium network is not a rootless bridge",
+ )
+ _require(
+ egress.get("driver", egress.get("Driver")) == "bridge",
+ "proxy egress network is not a rootless bridge",
+ )
+ internal_members, internal_reported_members, internal_membership_source = (
+ _validated_network_members(
+ args.internal_network,
+ internal,
+ {args.runner, args.selenium, args.proxy},
+ )
+ )
+ egress_members, egress_reported_members, egress_membership_source = (
+ _validated_network_members(
+ args.egress_network,
+ egress,
+ {args.proxy},
+ )
+ )
+
+ runner = _inspect_container(args.runner)
+ selenium = _inspect_container(args.selenium)
+ proxy = _inspect_container(args.proxy)
+ image_ids = {
+ args.runner: _require_expected_image(
+ args.runner, runner, args.expected_runner_image
+ ),
+ args.selenium: _require_expected_image(
+ args.selenium, selenium, args.expected_selenium_image
+ ),
+ args.proxy: _require_expected_image(
+ args.proxy, proxy, args.expected_proxy_image
+ ),
+ }
+ containers = {
+ args.runner: runner,
+ args.selenium: selenium,
+ args.proxy: proxy,
+ }
+ images: dict[str, dict] = {}
+ for name, container in containers.items():
+ images[name] = _inspect_image(image_ids[name])
+ _require_no_sensitive_environment_overrides(name, container, images[name])
+ checks = {
+ args.runner: _validate_container(
+ args.runner,
+ runner,
+ expected_networks={args.internal_network},
+ available_capabilities=available_capabilities,
+ ),
+ args.selenium: _validate_container(
+ args.selenium,
+ selenium,
+ expected_networks={args.internal_network},
+ available_capabilities=available_capabilities,
+ ),
+ args.proxy: _validate_container(
+ args.proxy,
+ proxy,
+ expected_networks={args.internal_network, args.egress_network},
+ available_capabilities=available_capabilities,
+ ),
+ }
+ selenium_environment = _environment(selenium)
+ expected_proxy_environment = {
+ f"SE_BROWSER_ARGS_UKBCD_PROXY={PROXY_SERVER_ARGUMENT}",
+ f"SE_BROWSER_ARGS_UKBCD_NO_BYPASS={PROXY_NO_BYPASS_ARGUMENT}",
+ "SE_BROWSER_ARGS_UKBCD_BACKGROUND=--disable-background-networking",
+ "SE_BROWSER_ARGS_UKBCD_SYNC=--disable-sync",
+ "SE_BROWSER_ARGS_UKBCD_DEFAULT_APPS=--disable-default-apps",
+ }
+ _require(
+ expected_proxy_environment <= selenium_environment,
+ "Selenium does not contain the reviewed Chrome proxy arguments",
+ )
+ bypass_values = {
+ value
+ for value in selenium_environment
+ if "proxy-bypass-list" in value.casefold()
+ }
+ _require(
+ bypass_values
+ == {f"SE_BROWSER_ARGS_UKBCD_NO_BYPASS={PROXY_NO_BYPASS_ARGUMENT}"},
+ "Selenium contains an additional proxy bypass rule",
+ )
+
+ runner_environment = _environment(runner)
+ _require(
+ f"{APPROVAL_ENVIRONMENT_VARIABLE}={APPROVAL_VALUE}" in runner_environment,
+ "runner lacks the exact one-shot approval value",
+ )
+
+ _require_exact_process(args.runner, runner, RUNNER_PROCESS)
+ _require_exact_process(args.proxy, proxy, PROXY_PROCESS)
+ selenium_process_digest = _require_expected_process(
+ args.selenium,
+ selenium,
+ _reviewed_image_process_sha256(images[args.selenium]),
+ )
+ checks[args.selenium]["process_sha256"] = selenium_process_digest
+
+ total_cpu = sum(check["cpu_limit"] for check in checks.values())
+ total_memory = sum(check["memory_limit_bytes"] for check in checks.values())
+ _require(total_cpu <= 6, f"aggregate CPU limit exceeds 6: {total_cpu}")
+ _require(total_memory <= 7 * 1024**3, "aggregate memory exceeds 7 GiB")
+
+ report = {
+ "status": "passed",
+ "rootless": True,
+ "rootless_capability_universe": sorted(available_capabilities),
+ "internal_network": {
+ "name": args.internal_network,
+ "internal": True,
+ "members": sorted(internal_members),
+ "runtime_reported_members": sorted(internal_reported_members),
+ "membership_source": internal_membership_source,
+ },
+ "egress_network": {
+ "name": args.egress_network,
+ "internal": False,
+ "members": sorted(egress_members),
+ "runtime_reported_members": sorted(egress_reported_members),
+ "membership_source": egress_membership_source,
+ },
+ "allowlisted_origins": [
+ "selfservice.southkesteven.gov.uk",
+ "www.southkesteven.gov.uk",
+ ],
+ "application_containers": checks,
+ "immutable_image_ids": image_ids,
+ "aggregate_cpu_limit": total_cpu,
+ "aggregate_memory_limit_bytes": total_memory,
+ "host_ports": 0,
+ "host_mounts": 0,
+ "container_sockets": 0,
+ "chrome_proxy_bypass_hosts": 0,
+ }
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ args.output.write_text(
+ json.dumps(report, indent=2, sort_keys=True), encoding="utf-8"
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/disposable_ha/migration_runner.py b/tests/disposable_ha/migration_runner.py
new file mode 100644
index 0000000000..087fcc121e
--- /dev/null
+++ b/tests/disposable_ha/migration_runner.py
@@ -0,0 +1,376 @@
+"""Verify actual HA config-entry migration and restart persistence offline."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.metadata
+import json
+import os
+import re
+import sys
+import time
+from pathlib import Path
+from urllib.parse import quote, quote_plus
+
+CONFIG = Path("/config")
+EVIDENCE = Path(os.environ.get("UKBCD_TEST_EVIDENCE_DIR", "/evidence"))
+HA_VERSION = "2026.7.2"
+ENTRY_IDS = (
+ "ddddddddddddddddddddddddddddddd1",
+ "ddddddddddddddddddddddddddddddd2",
+ "ddddddddddddddddddddddddddddddd3",
+)
+EXPECTED_ENTITY_SUFFIXES = (
+ "Fixture Bin",
+ "Fixture Bin_bin_type",
+ "Fixture Bin_calendar",
+ "Fixture Bin_colour",
+ "Fixture Bin_days_until_collection",
+ "Fixture Bin_next_collection_date",
+ "Fixture Bin_next_collection_human_readable",
+ "raw_json",
+)
+SNAPSHOT_PATH = EVIDENCE / "migration_v4_snapshot.json"
+SENSITIVE_SENTINELS = (
+ "Synthetic Address V1",
+ "Synthetic Address V2",
+ "Synthetic Address V3",
+ "Ignored legacy house value",
+ "Ignored legacy PAON value",
+)
+
+COMMON_DATA = {
+ "council": "FixtureCouncil",
+ "url": "http://127.0.0.1:8081/static",
+ "timeout": 75,
+ "icon_color_mapping": "{}",
+}
+EXPECTED_DATA = {
+ ENTRY_IDS[0]: {
+ **COMMON_DATA,
+ "name": "Offline migration v1",
+ "update_interval": 12,
+ "manual_refresh_only": True,
+ "number": "Synthetic Address V1",
+ "web_driver": "http://127.0.0.1:4444",
+ "headless": True,
+ "local_browser": False,
+ "skip_get_url": True,
+ },
+ ENTRY_IDS[1]: {
+ **COMMON_DATA,
+ "name": "Offline migration v2",
+ "update_interval": 6,
+ "manual_refresh_only": True,
+ "number": "Synthetic Address V2",
+ "web_driver": "http://127.0.0.1:4444/wd/hub",
+ "headless": False,
+ "local_browser": True,
+ "skip_get_url": True,
+ },
+ ENTRY_IDS[2]: {
+ **COMMON_DATA,
+ "name": "Offline migration v3",
+ "update_interval": 12,
+ "manual_refresh_only": True,
+ "number": "Synthetic Address V3",
+ "web_driver": "http://127.0.0.1:4444",
+ "headless": True,
+ "local_browser": False,
+ "skip_get_url": True,
+ },
+}
+
+
+def _wait_for_file(path: Path, timeout: float) -> Path:
+ deadline = time.monotonic() + timeout
+ while time.monotonic() < deadline:
+ if path.is_file() and path.stat().st_size:
+ return path
+ time.sleep(0.5)
+ raise RuntimeError(f"Timed out waiting for disposable file: {path.name}")
+
+
+def _network_assertions() -> tuple[list[str], int]:
+ interfaces = sorted(path.name for path in Path("/sys/class/net").iterdir())
+ routes = Path("/proc/net/route").read_text(encoding="utf-8")
+ rows = [line.split() for line in routes.splitlines()[1:] if line.split()]
+ default_routes = [row for row in rows if len(row) > 1 and row[1] == "00000000"]
+ if interfaces != ["lo"] or default_routes:
+ raise RuntimeError(
+ f"Offline network assertion failed: interfaces={interfaces}, "
+ f"default_routes={default_routes}"
+ )
+ return interfaces, len(default_routes)
+
+
+def _load_entry_projection() -> dict[str, dict]:
+ storage = json.loads(
+ _wait_for_file(CONFIG / ".storage" / "core.config_entries", 15).read_text(
+ encoding="utf-8"
+ )
+ )
+ return {
+ entry["entry_id"]: {"version": entry["version"], "data": entry["data"]}
+ for entry in storage["data"]["entries"]
+ if entry.get("entry_id") in ENTRY_IDS
+ }
+
+
+def _expected_projection() -> dict[str, dict]:
+ return {
+ entry_id: {"version": 4, "data": data}
+ for entry_id, data in EXPECTED_DATA.items()
+ }
+
+
+def _wait_for_projection(timeout: float) -> dict[str, dict]:
+ expected = _expected_projection()
+ deadline = time.monotonic() + timeout
+ projection: dict[str, dict] = {}
+ while time.monotonic() < deadline:
+ try:
+ projection = _load_entry_projection()
+ except (FileNotFoundError, json.JSONDecodeError, KeyError):
+ pass
+ if projection == expected:
+ return projection
+ time.sleep(0.5)
+ raise RuntimeError(
+ "Persisted config entries did not match the exact v4 migration contract: "
+ f"observed_entry_ids={sorted(projection)}"
+ )
+
+
+def _scrape_count() -> int:
+ path = EVIDENCE / "non_selenium_scrape_calls"
+ if not path.exists():
+ return 0
+ return len(path.read_text(encoding="utf-8").splitlines())
+
+
+def _wait_for_scrape_count(expected: int, timeout: float) -> None:
+ deadline = time.monotonic() + timeout
+ while time.monotonic() < deadline:
+ if _scrape_count() >= expected:
+ time.sleep(2)
+ observed = _scrape_count()
+ if observed != expected:
+ raise RuntimeError(
+ f"Expected {expected} fixture scrapes, observed {observed}"
+ )
+ return
+ time.sleep(0.5)
+ raise RuntimeError(
+ f"Timed out waiting for {expected} fixture scrapes; observed {_scrape_count()}"
+ )
+
+
+def _entry_setup_markers() -> tuple[str, ...]:
+ return tuple(
+ f"async_setup_entry finished for entry_id={entry_id}" for entry_id in ENTRY_IDS
+ )
+
+
+def _wait_for_first_log(timeout: float) -> bytes:
+ path = CONFIG / "home-assistant.log"
+ deadline = time.monotonic() + timeout
+ while time.monotonic() < deadline:
+ if path.exists():
+ content = path.read_bytes()
+ text = content.decode("utf-8", errors="replace")
+ if all(marker in text for marker in _entry_setup_markers()):
+ return content
+ time.sleep(0.5)
+ raise RuntimeError("Timed out waiting for first-boot HA setup markers")
+
+
+def _wait_for_restart_log(snapshot: dict, timeout: float) -> bytes:
+ """Return only log bytes written after the first validated boot when possible."""
+ path = CONFIG / "home-assistant.log"
+ prefix_size = snapshot["first_log_size"]
+ prefix_sha256 = snapshot["first_log_sha256"]
+ deadline = time.monotonic() + timeout
+ while time.monotonic() < deadline:
+ if path.exists():
+ content = path.read_bytes()
+ if (
+ len(content) >= prefix_size
+ and hashlib.sha256(content[:prefix_size]).hexdigest() == prefix_sha256
+ ):
+ current_boot = content[prefix_size:]
+ else:
+ # HA may rotate or recreate its log on restart. In that case the
+ # current file is the second boot's log and is safe to inspect whole.
+ current_boot = content
+ text = current_boot.decode("utf-8", errors="replace")
+ if all(marker in text for marker in _entry_setup_markers()):
+ return current_boot
+ time.sleep(0.5)
+ raise RuntimeError("Timed out waiting for second-boot HA setup markers")
+
+
+def _entity_ids() -> set[str | None]:
+ storage = json.loads(
+ _wait_for_file(CONFIG / ".storage" / "core.entity_registry", 30).read_text(
+ encoding="utf-8"
+ )
+ )
+ return {
+ entity.get("unique_id")
+ for entity in storage["data"].get("entities", [])
+ if entity.get("platform") == "uk_bin_collection"
+ and any(
+ str(entity.get("unique_id", "")).startswith(entry_id)
+ for entry_id in ENTRY_IDS
+ )
+ }
+
+
+def _assert_entity_contract(timeout: float) -> int:
+ expected = {
+ f"{entry_id}_{suffix}"
+ for entry_id in ENTRY_IDS
+ for suffix in EXPECTED_ENTITY_SUFFIXES
+ }
+ deadline = time.monotonic() + timeout
+ actual: set[str | None] = set()
+ while time.monotonic() < deadline:
+ try:
+ actual = _entity_ids()
+ except (FileNotFoundError, json.JSONDecodeError, KeyError):
+ pass
+ if actual == expected:
+ return len(actual)
+ time.sleep(0.5)
+ raise RuntimeError(
+ "Migrated entries did not retain the exact sensor/calendar identity contract"
+ )
+
+
+def _redaction_variants() -> set[str]:
+ variants: set[str] = set()
+ for value in SENSITIVE_SENTINELS:
+ variants.update(
+ (value, "".join(value.split()), quote(value), quote_plus(value))
+ )
+ return {value for value in variants if value}
+
+
+def _write_redacted_log(log_bytes: bytes, phase: str) -> None:
+ text = log_bytes.decode("utf-8", errors="replace")
+ redacted = text
+ leaked: list[str] = []
+ for value in sorted(_redaction_variants(), key=len, reverse=True):
+ if value.casefold() in text.casefold():
+ leaked.append(value)
+ redacted = re.sub(re.escape(value), "[REDACTED]", redacted, flags=re.IGNORECASE)
+ (EVIDENCE / f"migration_{phase}.redacted.log").write_text(
+ redacted, encoding="utf-8"
+ )
+ if leaked:
+ raise RuntimeError("HA migration log contains a synthetic address sentinel")
+
+
+def _runtime_contract() -> tuple[str, str, str]:
+ ha_version = importlib.metadata.version("homeassistant")
+ core_version = importlib.metadata.version("uk-bin-collection")
+ if ha_version != HA_VERSION or sys.version_info[:2] != (3, 14):
+ raise RuntimeError("The disposable runtime is not HA 2026.7.2/Python 3.14")
+ return ha_version, sys.version, core_version
+
+
+def _run(phase: str, timeout: float) -> dict:
+ interfaces, default_route_count = _network_assertions()
+ expected_scrapes = 3 if phase == "first" else 6
+ _wait_for_scrape_count(expected_scrapes, timeout)
+ projection = _wait_for_projection(timeout)
+ projection_sha256 = hashlib.sha256(
+ json.dumps(projection, sort_keys=True, separators=(",", ":")).encode()
+ ).hexdigest()
+ entity_count = _assert_entity_contract(timeout)
+
+ if phase == "first":
+ log_bytes = _wait_for_first_log(timeout)
+ log_text = log_bytes.decode("utf-8", errors="replace")
+ for entry_id in ENTRY_IDS:
+ marker = f"Migrated config entry {entry_id} to version 4"
+ if log_text.count(marker) != 1:
+ raise RuntimeError(
+ f"Expected one first-boot migration marker for {entry_id}"
+ )
+ snapshot = {
+ "projection": projection,
+ "first_log_size": len(log_bytes),
+ "first_log_sha256": hashlib.sha256(log_bytes).hexdigest(),
+ }
+ SNAPSHOT_PATH.write_text(
+ json.dumps(snapshot, indent=2, sort_keys=True), encoding="utf-8"
+ )
+ migration_reran = False
+ else:
+ snapshot = json.loads(
+ _wait_for_file(SNAPSHOT_PATH, 10).read_text(encoding="utf-8")
+ )
+ if projection != snapshot["projection"]:
+ raise RuntimeError("The v4 config-entry projection changed across restart")
+ log_bytes = _wait_for_restart_log(snapshot, timeout)
+ log_text = log_bytes.decode("utf-8", errors="replace")
+ migration_reran = "Migrated config entry" in log_text
+ if migration_reran:
+ raise RuntimeError("A persisted v4 entry was migrated again after restart")
+
+ integration_errors = [
+ line
+ for line in log_text.splitlines()
+ if "ERROR" in line and "[custom_components.uk_bin_collection]" in line
+ ]
+ if integration_errors or "Unexpected coordinator error" in log_text:
+ raise RuntimeError("Home Assistant logged an unexpected migration/setup error")
+ _write_redacted_log(log_bytes, phase)
+ ha_version, python_version, core_version = _runtime_contract()
+ return {
+ "network_interfaces": interfaces,
+ "default_route_count": default_route_count,
+ "home_assistant_version": ha_version,
+ "python_version": python_version,
+ "core_package_version": core_version,
+ "checks": {
+ "source_versions": [1, 2, 3],
+ "persisted_target_version": 4,
+ "persisted_entry_count": len(projection),
+ "registered_entity_count": entity_count,
+ "fixture_scrape_calls": expected_scrapes,
+ "legacy_aliases_absent": True,
+ "boolean_fields_are_boolean": True,
+ "persisted_projection_sha256": projection_sha256,
+ "migration_reran_after_restart": migration_reran,
+ "projection_preserved_across_restart": phase == "restart",
+ },
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--phase", choices=("first", "restart"), required=True)
+ parser.add_argument("--timeout", type=float, default=180)
+ args = parser.parse_args()
+ EVIDENCE.mkdir(parents=True, exist_ok=True)
+
+ report = {"phase": args.phase, "status": "failed"}
+ try:
+ report.update(_run(args.phase, args.timeout))
+ report["status"] = "passed"
+ except Exception as exc:
+ report["failure_type"] = type(exc).__name__
+ report["failure_message"] = str(exc)
+ raise
+ finally:
+ (EVIDENCE / f"migration_{args.phase}_report.json").write_text(
+ json.dumps(report, indent=2, sort_keys=True), encoding="utf-8"
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/disposable_ha/offline_runner.py b/tests/disposable_ha/offline_runner.py
new file mode 100644
index 0000000000..19b40892b1
--- /dev/null
+++ b/tests/disposable_ha/offline_runner.py
@@ -0,0 +1,353 @@
+"""Validate an offline HA run from a sibling container in the same pod."""
+
+from __future__ import annotations
+
+import argparse
+import importlib.metadata
+import json
+import os
+import re
+import sys
+import time
+from pathlib import Path
+from urllib.error import HTTPError
+from urllib.parse import quote, quote_plus
+from urllib.request import urlopen
+
+CONFIG = Path("/config")
+EVIDENCE = Path(os.environ.get("UKBCD_TEST_EVIDENCE_DIR", "/evidence"))
+HA_VERSION = "2026.7.2"
+REPAIR_PERSIST_TIMEOUT = 210
+SUCCESS_ENTRY_ID = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+CONTROL_ENTRY_ID = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
+COLLISION_ENTRY_ID = "cccccccccccccccccccccccccccccccc"
+SENSITIVE_VALUES = ("ZZ99 9ZZ", "Codex Test House")
+
+
+def _wait_for_url(url: str, timeout: float) -> dict | str:
+ deadline = time.monotonic() + timeout
+ last_error: Exception | None = None
+ while time.monotonic() < deadline:
+ try:
+ with urlopen(url, timeout=2) as response: # noqa: S310 - loopback only
+ body = response.read().decode("utf-8")
+ if response.status == 200:
+ try:
+ return json.loads(body)
+ except json.JSONDecodeError:
+ return body
+ except Exception as exc: # service may still be starting
+ last_error = exc
+ time.sleep(0.5)
+ raise RuntimeError(
+ f"Timed out waiting for loopback service: {type(last_error).__name__}"
+ )
+
+
+def _wait_for_log(markers: tuple[str, ...], timeout: float) -> str:
+ log_path = CONFIG / "home-assistant.log"
+ deadline = time.monotonic() + timeout
+ text = ""
+ while time.monotonic() < deadline:
+ if log_path.exists():
+ text = log_path.read_text(encoding="utf-8", errors="replace")
+ if all(marker in text for marker in markers):
+ return text
+ time.sleep(1)
+ raise RuntimeError(f"Missing HA log markers: {markers!r}")
+
+
+def _wait_for_file(path: Path, timeout: float) -> Path:
+ deadline = time.monotonic() + timeout
+ while time.monotonic() < deadline:
+ if path.is_file() and path.stat().st_size:
+ return path
+ time.sleep(0.5)
+ raise RuntimeError(f"Timed out waiting for disposable storage file: {path.name}")
+
+
+def _line_count(path: Path) -> int:
+ if not path.exists():
+ return 0
+ return len(path.read_text(encoding="utf-8").splitlines())
+
+
+def _read_requests() -> list[dict]:
+ path = _wait_for_file(EVIDENCE / "fixture_requests.jsonl", 15)
+ return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()]
+
+
+def _storage_data(name: str, timeout: float = 30) -> dict:
+ path = _wait_for_file(CONFIG / ".storage" / name, timeout)
+ return json.loads(path.read_text(encoding="utf-8"))["data"]
+
+
+def _assert_entity_ids(expected: set[str]) -> int:
+ deadline = time.monotonic() + 30
+ actual: set[str | None] = set()
+ while time.monotonic() < deadline:
+ entities = _storage_data("core.entity_registry").get("entities", [])
+ actual = {
+ entity.get("unique_id")
+ for entity in entities
+ if entity.get("platform") == "uk_bin_collection"
+ }
+ if expected <= actual:
+ return len(actual)
+ time.sleep(0.5)
+ missing = expected - actual
+ raise RuntimeError(
+ f"Expected sensor/calendar entities were not registered: {sorted(missing)}"
+ )
+
+
+def _assert_one_dependency_repair(timeout: float) -> None:
+ expected_id = f"dependency_{COLLISION_ENTRY_ID}"
+ # HA intentionally delays registry writes created during startup by 180
+ # seconds. Read the persisted registry only after that bounded delay; the
+ # in-memory issue is already visible to the Repairs UI immediately.
+ issues = _storage_data("repairs.issue_registry", timeout).get("issues", [])
+ # Transient HA Repairs deliberately persist only their stable domain and
+ # issue id; their translation metadata is regenerated in memory at startup.
+ matches = [
+ issue
+ for issue in issues
+ if issue.get("domain") == "uk_bin_collection"
+ and issue.get("issue_id") == expected_id
+ ]
+ if len(matches) != 1:
+ raise RuntimeError(
+ "Expected exactly one structured dependency Repair, "
+ f"observed {len(matches)}"
+ )
+
+
+def _redaction_variants() -> set[str]:
+ variants: set[str] = set()
+ for value in SENSITIVE_VALUES:
+ variants.update(
+ {
+ value,
+ "".join(value.split()),
+ quote(value),
+ quote_plus(value),
+ }
+ )
+ return {value for value in variants if value}
+
+
+def _write_redacted_ha_log(log_text: str) -> None:
+ redacted = log_text
+ leaked: list[str] = []
+ for value in sorted(_redaction_variants(), key=len, reverse=True):
+ if value.casefold() in log_text.casefold():
+ leaked.append(value)
+ redacted = re.sub(re.escape(value), "[REDACTED]", redacted, flags=re.IGNORECASE)
+ (EVIDENCE / "home-assistant.redacted.log").write_text(redacted, encoding="utf-8")
+ if leaked:
+ raise RuntimeError("Home Assistant log contains a synthetic household sentinel")
+
+
+def _assert_fixture_request_contract(mode: str) -> dict[str, int | bool]:
+ requests = _read_requests()
+ binday = [request for request in requests if request.get("path") == "/binday"]
+ browser = [request for request in binday if request.get("browser")]
+ direct = [request for request in binday if not request.get("browser")]
+ if len(direct) != 1:
+ raise RuntimeError(
+ f"Expected one deliberate direct-HTTP /binday request, observed {len(direct)}"
+ )
+ if mode == "success":
+ if len(browser) != 1:
+ raise RuntimeError(
+ "Expected exactly one browser /binday request; skip_get_url may be broken"
+ )
+ if not browser[0].get("custom_user_agent"):
+ raise RuntimeError("The configured browser user agent was not propagated")
+ elif browser:
+ raise RuntimeError("The collision run created a browser session unexpectedly")
+ return {
+ "binday_browser_requests": len(browser),
+ "binday_direct_requests": len(direct),
+ "custom_user_agent_observed": bool(
+ browser and browser[0].get("custom_user_agent")
+ ),
+ }
+
+
+def _assert_no_selenium_sessions() -> None:
+ # Selenium 4.45 removed the legacy GET /sessions route. The supported
+ # status payload exposes every node slot and its current session, which is
+ # also a stronger cleanup assertion for a bounded one-node test grid.
+ status = _wait_for_url("http://127.0.0.1:4444/status", 15)
+ value = status.get("value") if isinstance(status, dict) else None
+ nodes = value.get("nodes") if isinstance(value, dict) else None
+ if not isinstance(nodes, list) or not value.get("ready"):
+ raise RuntimeError("Selenium returned an invalid status payload")
+
+ active_sessions = 0
+ for node in nodes:
+ slots = node.get("slots") if isinstance(node, dict) else None
+ if not isinstance(slots, list):
+ raise RuntimeError("Selenium returned an invalid node-slot payload")
+ active_sessions += sum(
+ 1
+ for slot in slots
+ if not isinstance(slot, dict) or slot.get("session") is not None
+ )
+
+ if active_sessions:
+ raise RuntimeError("A Selenium session remained active after the HA lookup")
+
+
+def _network_assertions() -> tuple[list[str], list[list[str]]]:
+ interfaces = sorted(path.name for path in Path("/sys/class/net").iterdir())
+ routes = Path("/proc/net/route").read_text(encoding="utf-8")
+ route_rows = [line.split() for line in routes.splitlines()[1:] if line.split()]
+ default_routes = [
+ row for row in route_rows if len(row) > 1 and row[1] == "00000000"
+ ]
+ if interfaces != ["lo"] or default_routes:
+ raise RuntimeError(
+ f"Offline network assertion failed: interfaces={interfaces}, "
+ f"default_routes={default_routes}"
+ )
+ return interfaces, default_routes
+
+
+def _run(mode: str, timeout: float) -> dict:
+ interfaces, default_routes = _network_assertions()
+ fixture_health = _wait_for_url("http://127.0.0.1:8081/health", 30)
+ selenium_status = _wait_for_url("http://127.0.0.1:4444/status", 90)
+ if not (
+ isinstance(selenium_status, dict)
+ and selenium_status.get("value", {}).get("ready")
+ ):
+ raise RuntimeError("Selenium status did not report ready")
+
+ try:
+ urlopen("http://127.0.0.1:8081/binday", timeout=3) # noqa: S310
+ except HTTPError as exc:
+ direct_preflight_status = exc.code
+ else:
+ direct_preflight_status = 200
+ if direct_preflight_status != 403:
+ raise RuntimeError("The deterministic direct-HTTP denial was not reproduced")
+
+ if mode == "success":
+ markers = (
+ "Initial data fetched successfully",
+ "Setting up UK Bin Collection Data platform",
+ "Setting up UK Bin Collection Calendar platform",
+ "async_setup_entry finished",
+ )
+ _wait_for_log(markers, timeout)
+ time.sleep(12)
+ log_text = (CONFIG / "home-assistant.log").read_text(
+ encoding="utf-8", errors="replace"
+ )
+ scrape_calls = _line_count(EVIDENCE / "south_kesteven_scrape_calls")
+ if scrape_calls != 1:
+ raise RuntimeError(f"Expected one initial scrape, observed {scrape_calls}")
+ integration_errors = [
+ line
+ for line in log_text.splitlines()
+ if "ERROR" in line and "[custom_components.uk_bin_collection]" in line
+ ]
+ if integration_errors or "Unexpected coordinator error" in log_text:
+ raise RuntimeError("Home Assistant logged an unexpected integration error")
+ entity_count = _assert_entity_ids(
+ {
+ f"{SUCCESS_ENTRY_ID}_Black Bin",
+ f"{SUCCESS_ENTRY_ID}_Grey Bin",
+ f"{SUCCESS_ENTRY_ID}_Black Bin_calendar",
+ f"{SUCCESS_ENTRY_ID}_Grey Bin_calendar",
+ }
+ )
+ checks = {
+ "south_kesteven_scrape_calls": scrape_calls,
+ "registered_entity_count": entity_count,
+ "sensor_platform_marker": markers[1] in log_text,
+ "calendar_platform_marker": markers[2] in log_text,
+ }
+ else:
+ _wait_for_log(("A Python dependency is missing or shadowed",), timeout)
+ time.sleep(12)
+ log_text = (CONFIG / "home-assistant.log").read_text(
+ encoding="utf-8", errors="replace"
+ )
+ non_selenium_calls = _line_count(EVIDENCE / "non_selenium_scrape_calls")
+ collision_calls = _line_count(EVIDENCE / "south_kesteven_scrape_calls")
+ if non_selenium_calls != 1:
+ raise RuntimeError("The non-Selenium control entry did not remain isolated")
+ if collision_calls != 1:
+ raise RuntimeError("The collision entry retried unexpectedly")
+ if (EVIDENCE / "ha_poison_executed").exists():
+ raise RuntimeError("The top-level shadow package executed inside HA")
+ _assert_one_dependency_repair(max(timeout, REPAIR_PERSIST_TIMEOUT))
+ if "attempted relative import beyond top-level package" in log_text:
+ raise RuntimeError("The raw relative-import failure escaped into HA")
+ if "Unexpected coordinator error" in log_text:
+ raise RuntimeError("The typed dependency error reached the generic handler")
+ entity_count = _assert_entity_ids(
+ {
+ f"{CONTROL_ENTRY_ID}_Fixture Bin",
+ f"{CONTROL_ENTRY_ID}_Fixture Bin_calendar",
+ }
+ )
+ checks = {
+ "non_selenium_scrape_calls": non_selenium_calls,
+ "collision_entry_calls": collision_calls,
+ "dependency_repair_persisted": True,
+ "poison_module_not_executed": True,
+ "registered_control_entity_count": entity_count,
+ "raw_import_error_absent": True,
+ }
+
+ _assert_no_selenium_sessions()
+ checks.update(_assert_fixture_request_contract(mode))
+ _write_redacted_ha_log(log_text)
+
+ home_assistant_version = importlib.metadata.version("homeassistant")
+ if home_assistant_version != HA_VERSION or sys.version_info[:2] != (3, 14):
+ raise RuntimeError(
+ "The disposable runtime is not the accepted HA/Python version"
+ )
+
+ return {
+ "network_interfaces": interfaces,
+ "default_route_count": len(default_routes),
+ "fixture_health": fixture_health,
+ "selenium_ready": True,
+ "selenium_sessions_after_lookup": 0,
+ "direct_http_preflight_status": direct_preflight_status,
+ "home_assistant_version": home_assistant_version,
+ "python_version": sys.version,
+ "core_package_version": importlib.metadata.version("uk-bin-collection"),
+ "checks": checks,
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--mode", choices=("success", "collision"), required=True)
+ parser.add_argument("--timeout", type=float, default=180)
+ args = parser.parse_args()
+ EVIDENCE.mkdir(parents=True, exist_ok=True)
+
+ report = {"mode": args.mode, "status": "failed"}
+ try:
+ report.update(_run(args.mode, args.timeout))
+ report["status"] = "passed"
+ except Exception as exc:
+ report["failure_type"] = type(exc).__name__
+ report["failure_message"] = str(exc)
+ raise
+ finally:
+ (EVIDENCE / f"offline_{args.mode}_report.json").write_text(
+ json.dumps(report, indent=2, sort_keys=True), encoding="utf-8"
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/disposable_ha/pod_safety_check.py b/tests/disposable_ha/pod_safety_check.py
new file mode 100644
index 0000000000..cdf7d1e44d
--- /dev/null
+++ b/tests/disposable_ha/pod_safety_check.py
@@ -0,0 +1,501 @@
+"""Fail closed unless an unstarted disposable Podman pod is tightly contained."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+import subprocess
+from pathlib import Path
+
+HOST_NAMESPACE_FIELDS = (
+ "PidMode",
+ "IpcMode",
+ "UsernsMode",
+ "UTSMode",
+ "CgroupnsMode",
+ "CgroupMode",
+)
+_SHA256_PATTERN = re.compile(r"^(?:sha256:)?([0-9a-fA-F]{64})$")
+_CAPABILITY_PATTERN = re.compile(r"^(?:CAP_)?[A-Z][A-Z0-9_]*$")
+SENSITIVE_ENVIRONMENT_NAME_MARKERS = frozenset(
+ {
+ "token",
+ "password",
+ "passwd",
+ "secret",
+ "credential",
+ "postcode",
+ "house_number",
+ "housenumber",
+ "house_name",
+ "housename",
+ "paon",
+ "uprn",
+ "usrn",
+ "address",
+ }
+)
+
+
+def _podman_json(*arguments: str):
+ result = subprocess.run(
+ ["podman", *arguments],
+ check=True,
+ capture_output=True,
+ text=True,
+ )
+ return json.loads(result.stdout)
+
+
+def _inspect(name: str) -> dict:
+ return _podman_json("inspect", name)[0]
+
+
+def _inspect_image(image_id: str) -> dict:
+ return _podman_json("image", "inspect", image_id)[0]
+
+
+def _require(condition: bool, message: str) -> None:
+ if not condition:
+ raise RuntimeError(message)
+
+
+def _is_rootless(info: dict) -> bool:
+ reported_values: list[object] = []
+ if "rootless" in info.get("host", {}).get("security", {}):
+ reported_values.append(info["host"]["security"]["rootless"])
+ if "Rootless" in info.get("Host", {}).get("Security", {}):
+ reported_values.append(info["Host"]["Security"]["Rootless"])
+ return bool(reported_values) and all(value is True for value in reported_values)
+
+
+def _named_members(raw_members) -> set[str]:
+ """Extract names from Podman pod/network inspect member structures."""
+ if isinstance(raw_members, dict):
+ rows = raw_members.values()
+ elif isinstance(raw_members, list):
+ rows = raw_members
+ else:
+ return set()
+
+ members: set[str] = set()
+ for row in rows:
+ if not isinstance(row, dict):
+ continue
+ name = row.get("Name", row.get("name"))
+ if isinstance(name, str) and name:
+ members.add(name)
+ return members
+
+
+def _pod_members(pod: dict) -> set[str]:
+ return _named_members(pod.get("Containers", pod.get("containers")))
+
+
+def _require_exact_members(boundary: str, actual: set[str], expected: set[str]) -> None:
+ missing = expected - actual
+ extra = actual - expected
+ _require(
+ not missing and not extra,
+ f"{boundary} membership mismatch; missing={sorted(missing)} extra={sorted(extra)}",
+ )
+
+
+def _normalise_capability(value: object) -> str:
+ capability = str(value).strip().upper()
+ return capability.removeprefix("CAP_")
+
+
+def _parse_capabilities(value: object) -> frozenset[str]:
+ """Parse one concrete capability universe reported by Podman."""
+ if isinstance(value, str):
+ raw_capabilities = [] if not value.strip() else value.split(",")
+ elif isinstance(value, list) and all(isinstance(item, str) for item in value):
+ raw_capabilities = value
+ else:
+ raise RuntimeError("Podman reports malformed rootless capabilities")
+
+ capabilities: set[str] = set()
+ for raw_capability in raw_capabilities:
+ normalized_input = raw_capability.strip().upper()
+ _require(
+ bool(_CAPABILITY_PATTERN.fullmatch(normalized_input)),
+ "Podman reports malformed rootless capabilities",
+ )
+ capability = _normalise_capability(normalized_input)
+ _require(
+ capability != "ALL",
+ "Podman rootless capability universe must contain concrete names",
+ )
+ capabilities.add(capability)
+ return frozenset(capabilities)
+
+
+def _rootless_capabilities(info: dict) -> frozenset[str]:
+ """Return the exact capability universe advertised by the rootless engine."""
+ reported: list[frozenset[str]] = []
+ lower_security = info.get("host", {}).get("security", {})
+ if "capabilities" in lower_security:
+ reported.append(_parse_capabilities(lower_security["capabilities"]))
+ upper_security = info.get("Host", {}).get("Security", {})
+ if "Capabilities" in upper_security:
+ reported.append(_parse_capabilities(upper_security["Capabilities"]))
+
+ _require(bool(reported), "Podman does not report rootless capabilities")
+ _require(
+ all(value == reported[0] for value in reported[1:]),
+ "Podman reports conflicting rootless capability universes",
+ )
+ return reported[0]
+
+
+def _create_command_requests_cap_drop_all(container: dict) -> bool:
+ command = container.get("Config", {}).get("CreateCommand") or []
+ if not isinstance(command, list):
+ return False
+ tokens = [str(token).casefold() for token in command]
+ if "--cap-drop=all" in tokens:
+ return True
+ return any(
+ token == "--cap-drop" and index + 1 < len(tokens) and tokens[index + 1] == "all"
+ for index, token in enumerate(tokens)
+ )
+
+
+def _cap_drop_covers_available_set(
+ container: dict, available_capabilities: frozenset[str]
+) -> bool:
+ host = container.get("HostConfig", {})
+ raw_drops = host.get("CapDrop")
+ _require(
+ isinstance(raw_drops, list)
+ and all(isinstance(value, str) for value in raw_drops),
+ "Podman reports malformed dropped capabilities",
+ )
+ drops: set[str] = set()
+ for value in raw_drops:
+ normalized_input = value.strip().upper()
+ _require(
+ bool(_CAPABILITY_PATTERN.fullmatch(normalized_input)),
+ "Podman reports malformed dropped capabilities",
+ )
+ drops.add(_normalise_capability(normalized_input))
+ return "ALL" in drops or available_capabilities <= drops
+
+
+def _require_cap_drop_all(
+ name: str, container: dict, available_capabilities: frozenset[str]
+) -> None:
+ cap_add = container.get("HostConfig", {}).get("CapAdd")
+ _require(
+ isinstance(cap_add, list),
+ f"{name} reports malformed added capabilities",
+ )
+ _require(
+ _create_command_requests_cap_drop_all(container),
+ f"{name} was not created with --cap-drop all",
+ )
+ _require(
+ not cap_add,
+ f"{name} adds Linux capabilities",
+ )
+ _require(
+ _cap_drop_covers_available_set(container, available_capabilities),
+ f"{name} does not report the complete rootless capability drop set",
+ )
+
+
+def _host_namespace_modes(container: dict) -> dict[str, str]:
+ host = container.get("HostConfig", {})
+ return {
+ field: str(host[field])
+ for field in HOST_NAMESPACE_FIELDS
+ if field in host
+ and str(host[field]).strip().casefold().split(":", 1)[0] == "host"
+ }
+
+
+def _require_no_host_namespaces(name: str, container: dict) -> None:
+ host_modes = _host_namespace_modes(container)
+ _require(
+ not host_modes,
+ f"{name} joins host namespaces: {sorted(host_modes)}",
+ )
+
+
+def _environment_mapping(subject: dict) -> dict[str, str]:
+ environment: dict[str, str] = {}
+ for raw_value in subject.get("Config", {}).get("Env") or []:
+ variable_name, separator, value = str(raw_value).partition("=")
+ if variable_name:
+ environment[variable_name] = value if separator else ""
+ return environment
+
+
+def _is_sensitive_environment_name(variable_name: str) -> bool:
+ normalized_name = variable_name.casefold()
+ return any(
+ marker in normalized_name for marker in SENSITIVE_ENVIRONMENT_NAME_MARKERS
+ )
+
+
+def _require_no_sensitive_environment_overrides(
+ name: str, container: dict, image: dict
+) -> None:
+ """Reject sensitive runtime env not identically baked into the reviewed image."""
+ container_environment = _environment_mapping(container)
+ reviewed_environment = _environment_mapping(image)
+ sensitive_overrides = sorted(
+ variable_name
+ for variable_name, value in container_environment.items()
+ if _is_sensitive_environment_name(variable_name)
+ and reviewed_environment.get(variable_name) != value
+ )
+ _require(
+ not sensitive_overrides,
+ f"{name} receives a sensitive environment override: {sensitive_overrides}",
+ )
+
+
+def _normalise_sha256(value: object) -> str | None:
+ match = _SHA256_PATTERN.fullmatch(str(value).strip())
+ if not match:
+ return None
+ return f"sha256:{match.group(1).lower()}"
+
+
+def _inspected_image_ids(container: dict) -> set[str]:
+ identifiers: set[str] = set()
+ for field in ("Image", "ImageID", "ImageDigest", "Digest"):
+ if field not in container:
+ continue
+ normalized = _normalise_sha256(container[field])
+ if normalized is not None:
+ identifiers.add(normalized)
+ return identifiers
+
+
+def _require_expected_image(name: str, container: dict, expected: str) -> str:
+ normalized_expected = _normalise_sha256(expected)
+ _require(
+ normalized_expected is not None,
+ f"{name} expected image must be a full immutable SHA-256 ID or digest",
+ )
+ actual_ids = _inspected_image_ids(container)
+ _require(bool(actual_ids), f"{name} inspect data has no immutable image ID")
+ _require(
+ normalized_expected in actual_ids,
+ f"{name} immutable image ID does not match the reviewed value",
+ )
+ return normalized_expected
+
+
+def _process_sha256(container: dict) -> str:
+ payload = {
+ "path": str(container.get("Path", "")),
+ "args": [str(value) for value in container.get("Args") or []],
+ }
+ serialized = json.dumps(
+ payload, ensure_ascii=False, separators=(",", ":"), sort_keys=True
+ ).encode("utf-8")
+ return f"sha256:{hashlib.sha256(serialized).hexdigest()}"
+
+
+def _require_expected_process(name: str, container: dict, expected: str) -> str:
+ normalized_expected = _normalise_sha256(expected)
+ _require(
+ normalized_expected is not None,
+ f"{name} expected process must be a full SHA-256 digest",
+ )
+ actual = _process_sha256(container)
+ _require(
+ actual == normalized_expected,
+ f"{name} entrypoint or arguments differ from the reviewed payload",
+ )
+ return actual
+
+
+def _parse_name_mapping(
+ values: list[str], expected_names: set[str], label: str
+) -> dict[str, str]:
+ mapping: dict[str, str] = {}
+ for value in values:
+ name, separator, expected = value.partition("=")
+ _require(
+ bool(separator and name and expected),
+ f"{label} values must use NAME=SHA256",
+ )
+ _require(name not in mapping, f"duplicate {label} for {name}")
+ mapping[name] = expected
+ _require_exact_members(label, set(mapping), expected_names)
+ return mapping
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--pod", required=True)
+ parser.add_argument("--infra", required=True)
+ parser.add_argument("--container", action="append", default=[], required=True)
+ parser.add_argument("--expected-image", action="append", default=[], required=True)
+ parser.add_argument(
+ "--expected-process-sha256", action="append", default=[], required=True
+ )
+ parser.add_argument("--allowed-volume", action="append", default=[])
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args()
+
+ info = _podman_json("info", "--format", "json")
+ _require(_is_rootless(info), "Podman is not running rootless")
+ available_capabilities = _rootless_capabilities(info)
+
+ pod = _inspect(args.pod)
+ infra = _inspect(args.infra)
+ infra_id = infra["Id"]
+ infra_host = infra["HostConfig"]
+
+ pod_members = _pod_members(pod)
+ expected_pod_members = {args.infra, *args.container}
+ _require_exact_members(args.pod, pod_members, expected_pod_members)
+ container_names = set(args.container)
+ expected_images = _parse_name_mapping(
+ args.expected_image, container_names, "expected images"
+ )
+ expected_processes = _parse_name_mapping(
+ args.expected_process_sha256,
+ container_names,
+ "expected process digests",
+ )
+
+ _require(pod["State"] == "Created", "pod was started before safety validation")
+ _require(infra["State"]["Status"] == "created", "infra was already started")
+ _require(
+ infra["Config"]["Image"].startswith("localhost/podman-pause:"),
+ "infra image is not Podman's built-in pause image",
+ )
+ _require(infra["Path"] == "/catatonit", "infra runs an unexpected process")
+ _require_no_host_namespaces(args.infra, infra)
+ _require(infra_host["NetworkMode"] == "none", "pod has a network route")
+ _require(not infra_host["Privileged"], "infra is privileged")
+ _require(not infra_host.get("PortBindings"), "infra publishes a host port")
+ _require(not infra_host.get("Devices"), "infra has a host device")
+ _require(not infra.get("Mounts"), "infra has a host or volume mount")
+ _require(
+ "no-new-privileges" in infra_host.get("SecurityOpt", []),
+ "infra permits privilege escalation",
+ )
+ _require(
+ infra_host.get("RestartPolicy", {}).get("Name") in ("", "no"),
+ "infra has a restart policy",
+ )
+
+ allowed_volumes = set(args.allowed_volume)
+ total_cpus = 0.0
+ total_memory = 0
+ checks: dict[str, dict] = {}
+ for name in args.container:
+ container = _inspect(name)
+ host = container["HostConfig"]
+ image_id = _require_expected_image(name, container, expected_images[name])
+ _require_no_sensitive_environment_overrides(
+ name, container, _inspect_image(image_id)
+ )
+ process_digest = _require_expected_process(
+ name, container, expected_processes[name]
+ )
+ _require(container["State"]["Status"] == "created", f"{name} already started")
+ _require(not host["Privileged"], f"{name} is privileged")
+ _require(host["ReadonlyRootfs"], f"{name} root filesystem is writable")
+ _require(
+ host["NetworkMode"] == f"container:{infra_id}",
+ f"{name} does not share the offline pod namespace",
+ )
+ _require(not host.get("PortBindings"), f"{name} publishes a host port")
+ _require(not host.get("PublishAllPorts"), f"{name} publishes all ports")
+ _require(not host.get("Devices"), f"{name} has a host device")
+ _require_cap_drop_all(name, container, available_capabilities)
+ _require_no_host_namespaces(name, container)
+ _require(
+ "no-new-privileges" in host.get("SecurityOpt", []),
+ f"{name} permits privilege escalation",
+ )
+ _require(
+ host.get("RestartPolicy", {}).get("Name") in ("", "no"),
+ f"{name} has a restart policy",
+ )
+ _require(host.get("PidsLimit", 0) > 0, f"{name} has no PID limit")
+ _require(host.get("NanoCpus", 0) > 0, f"{name} has no CPU limit")
+ _require(host.get("Memory", 0) > 0, f"{name} has no memory limit")
+
+ mounts = container.get("Mounts", [])
+ _require(
+ all(mount.get("Type") == "volume" for mount in mounts),
+ f"{name} has a bind or non-volume mount",
+ )
+ mounted_names = {mount.get("Name") for mount in mounts}
+ _require(
+ mounted_names <= allowed_volumes,
+ f"{name} has an unapproved named volume: {sorted(mounted_names)}",
+ )
+ serialized = json.dumps(container).casefold()
+ _require("podman.sock" not in serialized, f"{name} receives the Podman socket")
+ _require("docker.sock" not in serialized, f"{name} receives the Docker socket")
+
+ total_cpus += host["NanoCpus"] / 1_000_000_000
+ total_memory += host["Memory"]
+ checks[name] = {
+ "image_id": image_id,
+ "process_sha256": process_digest,
+ "read_only_rootfs": True,
+ "privileged": False,
+ "network_namespace": "offline-pod-loopback",
+ "published_ports": 0,
+ "devices": 0,
+ "capabilities_added": 0,
+ "capabilities_dropped": "all",
+ "rootless_capabilities_dropped": sorted(available_capabilities),
+ "cap_drop_all": True,
+ "host_namespaces": [],
+ "no_new_privileges": True,
+ "pid_limit": host["PidsLimit"],
+ "cpu_limit": host["NanoCpus"] / 1_000_000_000,
+ "memory_limit_bytes": host["Memory"],
+ "named_volumes": sorted(mounted_names),
+ "tmpfs": sorted(host.get("Tmpfs", {})),
+ }
+
+ _require(total_cpus <= 6, f"aggregate CPU limit exceeds 6: {total_cpus}")
+ _require(total_memory <= 7 * 1024**3, "aggregate memory limit exceeds 7 GiB")
+
+ report = {
+ "status": "passed",
+ "rootless": True,
+ "rootless_capability_universe": sorted(available_capabilities),
+ "pod": args.pod,
+ "pod_members": sorted(pod_members),
+ "pod_state_before_test": "Created",
+ "infra": {
+ "image": infra["Config"]["Image"],
+ "process": [infra["Path"], *infra["Args"]],
+ "network_mode": "none",
+ "published_ports": 0,
+ "devices": 0,
+ "mounts": 0,
+ "privileged": False,
+ "no_new_privileges": True,
+ "host_namespaces": [],
+ "note": "Podman's static namespace-holding pause process has no application payload.",
+ },
+ "application_containers": checks,
+ "aggregate_cpu_limit": total_cpus,
+ "aggregate_memory_limit_bytes": total_memory,
+ }
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ args.output.write_text(
+ json.dumps(report, indent=2, sort_keys=True), encoding="utf-8"
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/disposable_ha/prepare_config.py b/tests/disposable_ha/prepare_config.py
new file mode 100644
index 0000000000..4ee17af39f
--- /dev/null
+++ b/tests/disposable_ha/prepare_config.py
@@ -0,0 +1,198 @@
+"""Create a fresh, synthetic Home Assistant config for one offline test mode."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import shutil
+from pathlib import Path
+
+CONFIG = Path("/config")
+COMPONENT_SOURCE = Path("/workspace/custom_components/uk_bin_collection")
+
+
+def _entry(entry_id: str, title: str, data: dict, *, version: int = 4) -> dict:
+ return {
+ "entry_id": entry_id,
+ "version": version,
+ "minor_version": 1,
+ "domain": "uk_bin_collection",
+ "title": title,
+ "data": data,
+ "options": {},
+ "pref_disable_new_entities": False,
+ "pref_disable_polling": False,
+ "source": "user",
+ "unique_id": None,
+ "disabled_by": None,
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--mode", choices=("success", "collision", "migration"), required=True
+ )
+ args = parser.parse_args()
+
+ if any(CONFIG.iterdir()):
+ raise RuntimeError("Refusing to populate a non-empty disposable config volume")
+
+ component_target = CONFIG / "custom_components" / "uk_bin_collection"
+ component_target.parent.mkdir(parents=True)
+ shutil.copytree(COMPONENT_SOURCE, component_target)
+ storage = CONFIG / ".storage"
+ storage.mkdir(mode=0o700)
+
+ common = {
+ "manual_refresh_only": True,
+ "update_interval": 12,
+ "timeout": 75,
+ "icon_color_mapping": "{}",
+ }
+ if args.mode == "success":
+ entries = [
+ _entry(
+ "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+ "Offline South Kesteven",
+ {
+ **common,
+ "name": "Offline South Kesteven",
+ "council": "SouthKestevenFixtureCouncil",
+ "url": "http://127.0.0.1:8081/binday",
+ "postcode": "ZZ99 9ZZ",
+ "number": "Codex Test House",
+ "web_driver": "http://127.0.0.1:4444",
+ "headless": True,
+ "skip_get_url": True,
+ "user_agent": "Mozilla/5.0 UKBCD-Disposable-Fixture",
+ },
+ )
+ ]
+ elif args.mode == "collision":
+ entries = [
+ _entry(
+ "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+ "Offline non-Selenium control",
+ {
+ **common,
+ "name": "Offline non-Selenium control",
+ "council": "FixtureCouncil",
+ "url": "http://127.0.0.1:8081/static",
+ "skip_get_url": True,
+ },
+ ),
+ _entry(
+ "cccccccccccccccccccccccccccccccc",
+ "Offline dependency collision",
+ {
+ **common,
+ "name": "Offline dependency collision",
+ "council": "SouthKestevenFixtureCouncil",
+ "url": "http://127.0.0.1:8081/binday",
+ "postcode": "ZZ99 9ZZ",
+ "number": "Codex Test House",
+ "web_driver": "http://127.0.0.1:4444",
+ "headless": True,
+ "skip_get_url": True,
+ },
+ ),
+ ]
+ poison = CONFIG / "websocket"
+ poison.mkdir()
+ (poison / "__init__.py").write_text(
+ "import os\n"
+ "from pathlib import Path\n"
+ "Path(os.environ.get('UKBCD_TEST_EVIDENCE_DIR', '/evidence'))"
+ ".joinpath('ha_poison_executed').write_text('executed', encoding='utf-8')\n"
+ "from ..const import DOMAIN\n",
+ encoding="utf-8",
+ )
+ else:
+ # These entries deliberately exercise every historical starting version in
+ # an actual HA storage file. FixtureCouncil never performs network I/O,
+ # and every value is synthetic or loopback-only.
+ migration_common = {
+ "council": "FixtureCouncil",
+ "url": "http://127.0.0.1:8081/static",
+ "timeout": 75,
+ "icon_color_mapping": "{}",
+ }
+ entries = [
+ _entry(
+ "ddddddddddddddddddddddddddddddd1",
+ "Offline migration v1",
+ {
+ **migration_common,
+ "name": "Offline migration v1",
+ "house_number": "Synthetic Address V1",
+ "selenium_url": " http://127.0.0.1:4444/ ",
+ "skip_get_url": "true",
+ },
+ version=1,
+ ),
+ _entry(
+ "ddddddddddddddddddddddddddddddd2",
+ "Offline migration v2",
+ {
+ **migration_common,
+ "name": "Offline migration v2",
+ "update_interval": 6,
+ "paon": "Synthetic Address V2",
+ "webdriver": " http://127.0.0.1:4444/wd/hub/ ",
+ "headless": "false",
+ "local_browser": "yes",
+ "skip_get_url": "on",
+ },
+ version=2,
+ ),
+ _entry(
+ "ddddddddddddddddddddddddddddddd3",
+ "Offline migration v3",
+ {
+ **migration_common,
+ "name": "Offline migration v3",
+ "update_interval": None,
+ "manual_refresh_only": "true",
+ "number": "Synthetic Address V3",
+ "house_number": "Ignored legacy house value",
+ "paon": "Ignored legacy PAON value",
+ "web_driver": " http://127.0.0.1:4444/ ",
+ "selenium_url": "http://127.0.0.1:5555/",
+ "webdriver": "http://127.0.0.1:6666/",
+ "headless": True,
+ "local_browser": "0",
+ "skip_get_url": "1",
+ },
+ version=3,
+ ),
+ ]
+
+ store = {
+ "version": 1,
+ "minor_version": 1,
+ "key": "core.config_entries",
+ "data": {"entries": entries},
+ }
+ (storage / "core.config_entries").write_text(
+ json.dumps(store, indent=2, sort_keys=True), encoding="utf-8"
+ )
+ (CONFIG / "configuration.yaml").write_text(
+ """homeassistant:
+ name: UKBCD Offline Test
+ latitude: 0
+ longitude: 0
+ elevation: 0
+ unit_system: metric
+ time_zone: Europe/London
+logger:
+ default: info
+ logs:
+ custom_components.uk_bin_collection: debug
+""",
+ encoding="utf-8",
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/disposable_ha/test_live_canary_harness.py b/tests/disposable_ha/test_live_canary_harness.py
new file mode 100644
index 0000000000..9e9ce68e37
--- /dev/null
+++ b/tests/disposable_ha/test_live_canary_harness.py
@@ -0,0 +1,261 @@
+"""Offline-only tests for the live-canary policy and redaction helpers."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from tests.disposable_ha.fixture_server import _is_fixture_browser_user_agent
+from tests.disposable_ha.live_allowlist_proxy import (
+ ALLOWED_HOSTS,
+ ProxyRateLimitExceeded,
+ ProxyRequestDenied,
+ RequestLimiter,
+ parse_proxy_request,
+ safe_log_line,
+)
+
+
+def test_fixture_rejects_legacy_generic_get_user_agent() -> None:
+ legacy = (
+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+ "AppleWebKit/537.36 Chrome/108.0.0.0 Safari/537.36"
+ )
+ configured = "Mozilla/5.0 UKBCD-Disposable-Fixture"
+
+ assert _is_fixture_browser_user_agent(legacy) is False
+ assert _is_fixture_browser_user_agent(configured) is True
+
+
+from tests.disposable_ha.live_canary_runner import (
+ COUNCIL_NAME,
+ MINIMUM_START_DELAY_SECONDS,
+ PUBLIC_BINDAY_URL,
+ PublicFixture,
+ load_public_fixture,
+ parse_collection_result,
+ redact_output,
+ run_canary,
+)
+
+
+@pytest.mark.parametrize(
+ "authority",
+ [
+ "www.southkesteven.gov.uk:443",
+ "selfservice.southkesteven.gov.uk:443",
+ "www.southkesteven.gov.uk:80",
+ "selfservice.southkesteven.gov.uk:80",
+ ],
+)
+def test_proxy_allows_only_reviewed_connect_authorities(authority: str) -> None:
+ request = parse_proxy_request(
+ f"CONNECT {authority} HTTP/1.1\r\nHost: redacted.invalid\r\n\r\n".encode()
+ )
+
+ assert request.is_tunnel is True
+ assert request.host in ALLOWED_HOSTS
+ assert request.port in {80, 443}
+ assert request.forward_header is None
+
+
+@pytest.mark.parametrize(
+ "target",
+ [
+ "evil.example:443",
+ "www.southkesteven.gov.uk.evil.example:443",
+ "127.0.0.1:443",
+ "www.southkesteven.gov.uk:22",
+ "user@www.southkesteven.gov.uk:443",
+ "[::1]:443",
+ ],
+)
+def test_proxy_denies_non_allowlisted_connect_targets(target: str) -> None:
+ with pytest.raises(ProxyRequestDenied):
+ parse_proxy_request(f"CONNECT {target} HTTP/1.1\r\n\r\n".encode())
+
+
+def test_proxy_rewrites_allowed_http_but_safe_log_has_no_request_data() -> None:
+ sensitive_path = "/binday?postcode=NG31%208XG&number=43"
+ request = parse_proxy_request(
+ (
+ "POST http://www.southkesteven.gov.uk"
+ f"{sensitive_path} HTTP/1.1\r\n"
+ "Host: attacker.invalid\r\n"
+ "Proxy-Authorization: secret\r\n"
+ "Content-Length: 0\r\n\r\n"
+ ).encode()
+ )
+
+ assert request.is_tunnel is False
+ assert request.host == "www.southkesteven.gov.uk"
+ assert sensitive_path.encode() in request.forward_header
+ assert b"attacker.invalid" not in request.forward_header
+ assert b"Proxy-Authorization" not in request.forward_header
+
+ log_line = safe_log_line(request.host, 200)
+ assert log_line == "host=www.southkesteven.gov.uk status=200"
+ assert "/binday" not in log_line
+ assert "postcode" not in log_line
+ assert "NG31" not in log_line
+ assert "43" not in log_line
+
+
+@pytest.mark.parametrize(
+ "request_line",
+ [
+ "GET http://evil.example/path HTTP/1.1",
+ "GET https://www.southkesteven.gov.uk/path HTTP/1.1",
+ "DELETE http://www.southkesteven.gov.uk/path HTTP/1.1",
+ "GET /relative HTTP/1.1",
+ ],
+)
+def test_proxy_denies_unsafe_http_forms(request_line: str) -> None:
+ with pytest.raises(ProxyRequestDenied):
+ parse_proxy_request(f"{request_line}\r\nHost: ignored\r\n\r\n".encode())
+
+
+def test_denied_proxy_log_never_discloses_denied_hostname() -> None:
+ assert safe_log_line("metadata.google.internal", 403) == (
+ "host=[DENIED] status=403"
+ )
+
+
+def test_proxy_limiter_paces_and_bounds_requests() -> None:
+ now = [10.0]
+ delays: list[float] = []
+
+ def clock() -> float:
+ return now[0]
+
+ def sleeper(delay: float) -> None:
+ delays.append(delay)
+ now[0] += delay
+
+ limiter = RequestLimiter(
+ max_requests=2,
+ minimum_interval_seconds=0.25,
+ clock=clock,
+ sleeper=sleeper,
+ )
+ limiter.acquire()
+ now[0] += 0.1
+ limiter.acquire()
+
+ assert limiter.count == 2
+ assert delays == pytest.approx([0.15])
+ with pytest.raises(ProxyRateLimitExceeded):
+ limiter.acquire()
+
+
+def test_loads_only_repository_public_fixture() -> None:
+ fixture_path = (
+ Path(__file__).parents[2] / "uk_bin_collection" / "tests" / "input.json"
+ )
+ fixture = load_public_fixture(fixture_path)
+
+ source = json.loads(fixture_path.read_text(encoding="utf-8"))[COUNCIL_NAME]
+ assert fixture.postcode == source["postcode"]
+ assert fixture.house_number == source["house_number"]
+ assert fixture.url == source["url"] == PUBLIC_BINDAY_URL
+ assert fixture.web_driver == source["web_driver"] == "http://selenium:4444"
+
+
+def test_fixture_loader_fails_closed_on_webdriver_override(tmp_path: Path) -> None:
+ fixture_path = tmp_path / "input.json"
+ fixture_path.write_text(
+ json.dumps(
+ {
+ COUNCIL_NAME: {
+ "postcode": "NG31 8XG",
+ "house_number": "43",
+ "url": PUBLIC_BINDAY_URL,
+ "web_driver": "http://host.containers.internal:4444",
+ "skip_get_url": True,
+ }
+ }
+ ),
+ encoding="utf-8",
+ )
+
+ with pytest.raises(ValueError, match="isolated Selenium alias"):
+ load_public_fixture(fixture_path)
+
+
+def test_runner_redacts_postcode_number_identifiers_and_urls() -> None:
+ fixture = PublicFixture(
+ postcode="NG31 8XG",
+ house_number="43",
+ url=PUBLIC_BINDAY_URL,
+ web_driver="http://selenium:4444",
+ )
+ raw = (
+ "postcode NG31 8XG / NG318XG number 43 "
+ "UPRN=123456789012 and https://example.test/path?address=43"
+ )
+
+ redacted = redact_output(raw, fixture)
+
+ assert "NG31" not in redacted
+ assert "NG318XG" not in redacted
+ assert "123456789012" not in redacted
+ assert "https://" not in redacted
+ assert "example.test" not in redacted
+ assert " 43" not in redacted
+
+
+def test_result_parser_returns_only_non_identifying_counts() -> None:
+ summary = parse_collection_result(
+ json.dumps(
+ {
+ "bins": [
+ {"type": "Grey Bin", "collectionDate": "20/07/2026"},
+ {"type": "Food Bin", "collectionDate": "20/07/2026"},
+ ]
+ }
+ )
+ )
+
+ assert summary == {
+ "status": "passed",
+ "council": COUNCIL_NAME,
+ "logical_lookups": 1,
+ "bin_count": 2,
+ }
+
+
+def test_runner_invokes_exactly_once_after_minimum_delay() -> None:
+ fixture = PublicFixture(
+ postcode="NG31 8XG",
+ house_number="43",
+ url=PUBLIC_BINDAY_URL,
+ web_driver="http://selenium:4444",
+ )
+ invocations: list[PublicFixture] = []
+ delays: list[float] = []
+
+ def invoke(value: PublicFixture) -> str:
+ invocations.append(value)
+ return json.dumps(
+ {"bins": [{"type": "Grey Bin", "collectionDate": "20/07/2026"}]}
+ )
+
+ summary = run_canary(fixture, invoke=invoke, sleeper=delays.append)
+
+ assert invocations == [fixture]
+ assert delays == [MINIMUM_START_DELAY_SECONDS]
+ assert summary["logical_lookups"] == 1
+
+
+def test_runner_refuses_reduced_delay() -> None:
+ fixture = PublicFixture("NG31 8XG", "43", PUBLIC_BINDAY_URL, "http://selenium:4444")
+
+ with pytest.raises(ValueError, match="cannot be reduced"):
+ run_canary(
+ fixture,
+ invoke=lambda _: None,
+ sleeper=lambda _: None,
+ minimum_delay_seconds=0,
+ )
diff --git a/tests/disposable_ha/test_migration_runner.py b/tests/disposable_ha/test_migration_runner.py
new file mode 100644
index 0000000000..a664cc4e13
--- /dev/null
+++ b/tests/disposable_ha/test_migration_runner.py
@@ -0,0 +1,54 @@
+"""Unit tests for disposable Home Assistant migration validation."""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+import pytest
+
+
+_RUNNER_PATH = Path(__file__).with_name("migration_runner.py")
+_RUNNER_SPEC = importlib.util.spec_from_file_location(
+ "ukbcd_disposable_migration_runner", _RUNNER_PATH
+)
+assert _RUNNER_SPEC is not None and _RUNNER_SPEC.loader is not None
+migration_runner = importlib.util.module_from_spec(_RUNNER_SPEC)
+_RUNNER_SPEC.loader.exec_module(migration_runner)
+
+
+def _expected_entity_ids() -> set[str]:
+ return {
+ f"{entry_id}_{suffix}"
+ for entry_id in migration_runner.ENTRY_IDS
+ for suffix in migration_runner.EXPECTED_ENTITY_SUFFIXES
+ }
+
+
+def test_entity_contract_accepts_all_existing_sensor_and_calendar_ids(
+ monkeypatch,
+) -> None:
+ expected = _expected_entity_ids()
+ monkeypatch.setattr(migration_runner, "_entity_ids", lambda: expected)
+
+ assert migration_runner._assert_entity_contract(0.1) == 24
+
+
+@pytest.mark.parametrize(
+ "missing_suffix",
+ (
+ "Fixture Bin",
+ "Fixture Bin_calendar",
+ "Fixture Bin_next_collection_date",
+ "raw_json",
+ ),
+)
+def test_entity_contract_rejects_a_missing_existing_identity(
+ monkeypatch, missing_suffix: str
+) -> None:
+ expected = _expected_entity_ids()
+ expected.remove(f"{migration_runner.ENTRY_IDS[0]}_{missing_suffix}")
+ monkeypatch.setattr(migration_runner, "_entity_ids", lambda: expected)
+
+ with pytest.raises(RuntimeError, match="identity contract"):
+ migration_runner._assert_entity_contract(0)
diff --git a/tests/disposable_ha/test_offline_runner.py b/tests/disposable_ha/test_offline_runner.py
new file mode 100644
index 0000000000..94aff9783b
--- /dev/null
+++ b/tests/disposable_ha/test_offline_runner.py
@@ -0,0 +1,120 @@
+"""Unit tests for disposable Home Assistant result validation."""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+import pytest
+
+
+# Some Home Assistant dependencies install an unrelated top-level ``tests``
+# package. Load the sibling runner directly so this isolated harness test does
+# not depend on site-packages ordering.
+_RUNNER_PATH = Path(__file__).with_name("offline_runner.py")
+_RUNNER_SPEC = importlib.util.spec_from_file_location(
+ "ukbcd_disposable_offline_runner", _RUNNER_PATH
+)
+assert _RUNNER_SPEC is not None and _RUNNER_SPEC.loader is not None
+offline_runner = importlib.util.module_from_spec(_RUNNER_SPEC)
+_RUNNER_SPEC.loader.exec_module(offline_runner)
+
+
+def test_session_cleanup_accepts_empty_selenium_status_slots(monkeypatch) -> None:
+ monkeypatch.setattr(
+ offline_runner,
+ "_wait_for_url",
+ lambda url, timeout: {
+ "value": {
+ "ready": True,
+ "nodes": [
+ {
+ "slots": [
+ {"session": None},
+ {"session": None},
+ ]
+ }
+ ],
+ }
+ },
+ )
+
+ offline_runner._assert_no_selenium_sessions()
+
+
+def test_session_cleanup_rejects_an_active_selenium_slot(monkeypatch) -> None:
+ monkeypatch.setattr(
+ offline_runner,
+ "_wait_for_url",
+ lambda url, timeout: {
+ "value": {
+ "ready": True,
+ "nodes": [
+ {"slots": [{"session": {"sessionId": "synthetic"}}]}
+ ],
+ }
+ },
+ )
+
+ with pytest.raises(RuntimeError, match="remained active"):
+ offline_runner._assert_no_selenium_sessions()
+
+
+@pytest.mark.parametrize(
+ "status",
+ (
+ None,
+ {},
+ {"value": {"ready": False, "nodes": []}},
+ {"value": {"ready": True, "nodes": [{"slots": None}]}},
+ ),
+)
+def test_session_cleanup_rejects_malformed_status(monkeypatch, status) -> None:
+ monkeypatch.setattr(
+ offline_runner,
+ "_wait_for_url",
+ lambda url, timeout: status,
+ )
+
+ with pytest.raises(RuntimeError, match="invalid"):
+ offline_runner._assert_no_selenium_sessions()
+
+
+def test_dependency_repair_waits_for_persisted_startup_registry(monkeypatch) -> None:
+ observed: dict[str, object] = {}
+
+ def storage_data(name: str, timeout: float) -> dict:
+ observed.update(name=name, timeout=timeout)
+ return {
+ "issues": [
+ {
+ "domain": "uk_bin_collection",
+ "issue_id": (
+ f"dependency_{offline_runner.COLLISION_ENTRY_ID}"
+ ),
+ "is_persistent": False,
+ }
+ ]
+ }
+
+ monkeypatch.setattr(offline_runner, "_storage_data", storage_data)
+
+ offline_runner._assert_one_dependency_repair(210)
+
+ assert observed == {"name": "repairs.issue_registry", "timeout": 210}
+
+
+def test_dependency_repair_rejects_duplicate_matches(monkeypatch) -> None:
+ issue = {
+ "domain": "uk_bin_collection",
+ "issue_id": f"dependency_{offline_runner.COLLISION_ENTRY_ID}",
+ "is_persistent": False,
+ }
+ monkeypatch.setattr(
+ offline_runner,
+ "_storage_data",
+ lambda name, timeout: {"issues": [issue, issue.copy()]},
+ )
+
+ with pytest.raises(RuntimeError, match="observed 2"):
+ offline_runner._assert_one_dependency_repair(210)
diff --git a/tests/disposable_ha/test_runtime_requirements.py b/tests/disposable_ha/test_runtime_requirements.py
new file mode 100644
index 0000000000..dec0cdc7db
--- /dev/null
+++ b/tests/disposable_ha/test_runtime_requirements.py
@@ -0,0 +1,9 @@
+"""Tests for the candidate image's offline dependency-closure verifier."""
+
+from tests.disposable_ha.verify_runtime_requirements import dependency_errors
+
+
+def test_runtime_verifier_reports_missing_root_distribution() -> None:
+ assert dependency_errors("ukbcd-definitely-not-installed-7f930b9a") == [
+ "missing distribution: ukbcd-definitely-not-installed-7f930b9a"
+ ]
diff --git a/tests/disposable_ha/test_safety_validators.py b/tests/disposable_ha/test_safety_validators.py
new file mode 100644
index 0000000000..5ee6fa702b
--- /dev/null
+++ b/tests/disposable_ha/test_safety_validators.py
@@ -0,0 +1,637 @@
+"""Offline tests for fail-closed Podman safety validator decisions."""
+
+from __future__ import annotations
+
+import pytest
+
+from tests.disposable_ha import live_canary_safety_check as live
+from tests.disposable_ha import pod_safety_check as offline
+
+VALIDATORS = (offline, live)
+ROOTLESS_CAPABILITIES = frozenset(
+ {
+ "CHOWN",
+ "DAC_OVERRIDE",
+ "FOWNER",
+ "FSETID",
+ "KILL",
+ "NET_BIND_SERVICE",
+ "SETFCAP",
+ "SETGID",
+ "SETPCAP",
+ "SETUID",
+ "SYS_CHROOT",
+ }
+)
+
+SENSITIVE_ENVIRONMENT_NAMES = (
+ "HA_TOKEN",
+ "ADMIN_PASSWORD",
+ "API_SECRET",
+ "CLIENT_CREDENTIAL",
+ "POSTCODE",
+ "HOUSE_NUMBER",
+ "HOUSE_NAME",
+ "PAON",
+ "UPRN",
+ "USRN",
+ "PROPERTY_ADDRESS",
+)
+
+
+def _container_with_drops(drops: list[str], command: list[str] | None = None) -> dict:
+ return {
+ "Config": {
+ "CreateCommand": (
+ command
+ if command is not None
+ else ["podman", "create", "--cap-drop", "all", "image"]
+ ),
+ },
+ "HostConfig": {"CapAdd": [], "CapDrop": drops},
+ }
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_rejects_rootful_podman(validator) -> None:
+ assert validator._is_rootless({"host": {"security": {"rootless": False}}}) is False
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_accepts_both_rootless_info_schemas(validator) -> None:
+ assert validator._is_rootless({"host": {"security": {"rootless": True}}}) is True
+ assert validator._is_rootless({"Host": {"Security": {"Rootless": True}}}) is True
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_rejects_missing_or_conflicting_rootless_status(validator) -> None:
+ assert validator._is_rootless({}) is False
+ assert (
+ validator._is_rootless(
+ {
+ "host": {"security": {"rootless": True}},
+ "Host": {"Security": {"Rootless": False}},
+ }
+ )
+ is False
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_rejects_partial_capability_drop(validator) -> None:
+ partial = _container_with_drops(["CAP_CHOWN", "CAP_NET_RAW"])
+
+ with pytest.raises(RuntimeError, match="complete rootless capability drop set"):
+ validator._require_cap_drop_all("partial", partial, ROOTLESS_CAPABILITIES)
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_requires_create_command_cap_drop_all(validator) -> None:
+ full_report = _container_with_drops(
+ ["ALL"],
+ command=["podman", "create", "--cap-drop", "NET_RAW", "image"],
+ )
+
+ with pytest.raises(RuntimeError, match="not created with --cap-drop all"):
+ validator._require_cap_drop_all(
+ "wrong-command", full_report, ROOTLESS_CAPABILITIES
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_accepts_all_sentinel_and_expanded_drop_set(validator) -> None:
+ validator._require_cap_drop_all(
+ "sentinel", _container_with_drops(["ALL"]), ROOTLESS_CAPABILITIES
+ )
+
+ expanded_runtime = ROOTLESS_CAPABILITIES | {
+ "AUDIT_WRITE",
+ "MKNOD",
+ "NET_RAW",
+ "FUTURE_CONCRETE_CAPABILITY",
+ }
+ validator._require_cap_drop_all(
+ "expanded-runtime",
+ _container_with_drops([f"CAP_{name}" for name in expanded_runtime]),
+ expanded_runtime,
+ )
+ expanded = [f"CAP_{name}" for name in ROOTLESS_CAPABILITIES]
+ validator._require_cap_drop_all(
+ "expanded",
+ _container_with_drops(
+ expanded,
+ command=["podman", "create", "--cap-drop=ALL", "image"],
+ ),
+ ROOTLESS_CAPABILITIES,
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_uses_rootless_engine_capability_universe(validator) -> None:
+ actual = ",".join(f"CAP_{name}" for name in sorted(ROOTLESS_CAPABILITIES))
+ assert (
+ validator._rootless_capabilities(
+ {"host": {"security": {"capabilities": actual}}}
+ )
+ == ROOTLESS_CAPABILITIES
+ )
+ assert (
+ validator._rootless_capabilities(
+ {
+ "Host": {
+ "Security": {
+ "Capabilities": [
+ name.casefold() for name in ROOTLESS_CAPABILITIES
+ ]
+ }
+ }
+ }
+ )
+ == ROOTLESS_CAPABILITIES
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+@pytest.mark.parametrize(
+ "info",
+ [
+ {},
+ {"host": {"security": {"capabilities": 7}}},
+ {"host": {"security": {"capabilities": ["CAP_CHOWN", 7]}}},
+ {"host": {"security": {"capabilities": "CAP_CHOWN,"}}},
+ {"host": {"security": {"capabilities": "ALL"}}},
+ {"host": {"security": {"capabilities": "CAP_ALL"}}},
+ {"host": {"security": {"capabilities": "CAP_CHOWN;CAP_SETUID"}}},
+ ],
+)
+def test_validator_rejects_missing_or_malformed_capability_universe(
+ validator, info
+) -> None:
+ with pytest.raises(RuntimeError):
+ validator._rootless_capabilities(info)
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_rejects_conflicting_capability_schemas(validator) -> None:
+ with pytest.raises(RuntimeError, match="conflicting"):
+ validator._rootless_capabilities(
+ {
+ "host": {"security": {"capabilities": "CAP_CHOWN"}},
+ "Host": {"Security": {"Capabilities": "CAP_SETUID"}},
+ }
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_rejects_missing_advertised_capability(validator) -> None:
+ missing_one = ROOTLESS_CAPABILITIES - {"SETUID"}
+ drops = [f"CAP_{name}" for name in missing_one | {"NET_RAW"}]
+ with pytest.raises(RuntimeError, match="complete rootless capability drop set"):
+ validator._require_cap_drop_all(
+ "missing-setuid", _container_with_drops(drops), ROOTLESS_CAPABILITIES
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_rejects_empty_drop_for_nonempty_universe(validator) -> None:
+ with pytest.raises(RuntimeError, match="complete rootless capability drop set"):
+ validator._require_cap_drop_all(
+ "empty-drop", _container_with_drops([]), ROOTLESS_CAPABILITIES
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_accepts_explicitly_empty_capability_universe(validator) -> None:
+ info = {"host": {"security": {"capabilities": ""}}}
+ available = validator._rootless_capabilities(info)
+ assert available == frozenset()
+ validator._require_cap_drop_all(
+ "already-empty", _container_with_drops([]), available
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_rejects_capability_addition(validator) -> None:
+ container = _container_with_drops(["ALL"])
+ container["HostConfig"]["CapAdd"] = ["CAP_NET_ADMIN"]
+ with pytest.raises(RuntimeError, match="adds Linux capabilities"):
+ validator._require_cap_drop_all(
+ "added-capability", container, ROOTLESS_CAPABILITIES
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+@pytest.mark.parametrize("cap_add", [None, "", "CAP_NET_ADMIN"])
+def test_validator_rejects_malformed_capability_additions(validator, cap_add) -> None:
+ container = _container_with_drops(["ALL"])
+ container["HostConfig"]["CapAdd"] = cap_add
+ with pytest.raises(RuntimeError, match="malformed added capabilities"):
+ validator._require_cap_drop_all(
+ "malformed-cap-add", container, ROOTLESS_CAPABILITIES
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+@pytest.mark.parametrize("cap_drop", [None, "ALL", ["CAP_CHOWN", 7], ["CAP_CHOWN,"]])
+def test_validator_rejects_malformed_capability_drops(validator, cap_drop) -> None:
+ container = _container_with_drops([])
+ container["HostConfig"]["CapDrop"] = cap_drop
+ with pytest.raises(RuntimeError, match="malformed dropped capabilities"):
+ validator._require_cap_drop_all(
+ "malformed-cap-drop", container, ROOTLESS_CAPABILITIES
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+@pytest.mark.parametrize(
+ "field",
+ [
+ "PidMode",
+ "IpcMode",
+ "UsernsMode",
+ "UTSMode",
+ "CgroupnsMode",
+ "CgroupMode",
+ ],
+)
+def test_validator_rejects_host_namespace(validator, field: str) -> None:
+ with pytest.raises(RuntimeError, match="joins host namespaces"):
+ validator._require_no_host_namespaces("unsafe", {"HostConfig": {field: "host"}})
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_allows_private_or_container_namespace(validator) -> None:
+ validator._require_no_host_namespaces(
+ "contained",
+ {
+ "HostConfig": {
+ "PidMode": "private",
+ "IpcMode": "container:infra-id",
+ "UsernsMode": "keep-id",
+ "UTSMode": "container:infra-id",
+ "CgroupnsMode": "private",
+ }
+ },
+ )
+
+
+def test_live_validator_rejects_wrong_or_mutable_image_identity() -> None:
+ actual = "a" * 64
+ container = {"Image": f"sha256:{actual}"}
+
+ assert live._require_expected_image("runner", container, actual) == (
+ f"sha256:{actual}"
+ )
+ with pytest.raises(RuntimeError, match="does not match"):
+ live._require_expected_image("runner", container, "b" * 64)
+ with pytest.raises(RuntimeError, match="full immutable SHA-256"):
+ live._require_expected_image("runner", container, "candidate:latest")
+
+
+def test_offline_validator_rejects_wrong_or_mutable_image_identity() -> None:
+ actual = "a" * 64
+ container = {"Image": f"sha256:{actual}"}
+
+ assert offline._require_expected_image("ha", container, actual) == (
+ f"sha256:{actual}"
+ )
+ with pytest.raises(RuntimeError, match="does not match"):
+ offline._require_expected_image("ha", container, "b" * 64)
+ with pytest.raises(RuntimeError, match="full immutable SHA-256"):
+ offline._require_expected_image("ha", container, "candidate:latest")
+
+
+def test_offline_validator_binds_exact_entrypoint_and_arguments() -> None:
+ reviewed = {
+ "Path": "python3",
+ "Args": ["/workspace/tests/disposable_ha/offline_runner.py", "success"],
+ }
+ expected = offline._process_sha256(reviewed)
+
+ assert offline._require_expected_process("runner", reviewed, expected) == expected
+ with pytest.raises(RuntimeError, match="entrypoint or arguments"):
+ offline._require_expected_process(
+ "runner",
+ {**reviewed, "Args": [*reviewed["Args"], "unexpected"]},
+ expected,
+ )
+ with pytest.raises(RuntimeError, match="full SHA-256"):
+ offline._require_expected_process("runner", reviewed, "reviewed-command")
+
+
+def test_live_validator_binds_selenium_entrypoint_and_arguments_by_digest() -> None:
+ image = {
+ "Config": {
+ "Entrypoint": ["/opt/bin/entry_point.sh"],
+ "Cmd": ["--selenium-manager", "false"],
+ }
+ }
+ reviewed = {
+ "Path": "/opt/bin/entry_point.sh",
+ "Args": ["--selenium-manager", "false"],
+ }
+ expected = live._reviewed_image_process_sha256(image)
+
+ assert live._require_expected_process("selenium", reviewed, expected) == expected
+ with pytest.raises(RuntimeError, match="entrypoint or arguments"):
+ live._require_expected_process(
+ "selenium",
+ {**reviewed, "Args": [*reviewed["Args"], "unexpected"]},
+ expected,
+ )
+ with pytest.raises(RuntimeError, match="full SHA-256"):
+ live._require_expected_process("selenium", reviewed, "reviewed-command")
+
+
+def test_live_validator_normalizes_podman_cmd_only_process_metadata() -> None:
+ command = "/opt/bin/entry_point.sh"
+ image = {"Config": {"Entrypoint": None, "Cmd": [command]}}
+ podman_inspect = {"Path": command, "Args": [command]}
+
+ expected = live._process_sha256(podman_inspect)
+
+ assert live._reviewed_image_process_sha256(image) == expected
+ assert (
+ live._require_expected_process("selenium", podman_inspect, expected)
+ == expected
+ )
+
+
+@pytest.mark.parametrize(
+ "image",
+ [
+ {"Config": {}},
+ {"Config": {"Entrypoint": [""], "Cmd": []}},
+ {"Config": {"Entrypoint": "entry-point.sh", "Cmd": []}},
+ ],
+)
+def test_live_validator_rejects_missing_or_malformed_image_process(image) -> None:
+ with pytest.raises(RuntimeError, match="reviewed Selenium image"):
+ live._reviewed_image_process_sha256(image)
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+@pytest.mark.parametrize("environment_name", SENSITIVE_ENVIRONMENT_NAMES)
+def test_validator_rejects_added_sensitive_environment_variable(
+ validator, environment_name: str
+) -> None:
+ container = {
+ "Config": {
+ "Env": ["PATH=/usr/bin", f"{environment_name}=must-not-enter-container"]
+ }
+ }
+ image = {"Config": {"Env": ["PATH=/usr/bin"]}}
+
+ with pytest.raises(RuntimeError, match="sensitive environment override"):
+ validator._require_no_sensitive_environment_overrides(
+ "application", container, image
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_rejects_changed_sensitive_base_image_environment(validator) -> None:
+ container = {
+ "Config": {"Env": ["PATH=/usr/bin", "SE_VNC_PASSWORD=host-supplied-secret"]}
+ }
+ image = {"Config": {"Env": ["PATH=/usr/bin", "SE_VNC_PASSWORD=reviewed-default"]}}
+
+ with pytest.raises(RuntimeError, match="SE_VNC_PASSWORD"):
+ validator._require_no_sensitive_environment_overrides(
+ "application", container, image
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_rejects_malformed_sensitive_environment_entry(validator) -> None:
+ container = {"Config": {"Env": ["PATH=/usr/bin", "HA_TOKEN"]}}
+ image = {"Config": {"Env": ["PATH=/usr/bin"]}}
+
+ with pytest.raises(RuntimeError, match="HA_TOKEN"):
+ validator._require_no_sensitive_environment_overrides(
+ "application", container, image
+ )
+
+
+@pytest.mark.parametrize("validator", VALIDATORS)
+def test_validator_allows_unchanged_reviewed_base_image_environment(validator) -> None:
+ container = {
+ "Config": {
+ "Env": [
+ "PATH=/usr/bin",
+ "SE_VNC_PASSWORD=reviewed-default",
+ "NORMAL_RUNTIME_SETTING=enabled",
+ ]
+ }
+ }
+ image = {"Config": {"Env": ["PATH=/usr/bin", "SE_VNC_PASSWORD=reviewed-default"]}}
+
+ validator._require_no_sensitive_environment_overrides(
+ "application", container, image
+ )
+
+
+def test_offline_expected_mappings_require_every_container_exactly_once() -> None:
+ assert offline._parse_name_mapping(
+ [f"ha={'a' * 64}", f"runner={'b' * 64}"],
+ {"ha", "runner"},
+ "expected images",
+ ) == {"ha": "a" * 64, "runner": "b" * 64}
+
+ with pytest.raises(RuntimeError, match="membership mismatch"):
+ offline._parse_name_mapping(
+ [f"ha={'a' * 64}"], {"ha", "runner"}, "expected images"
+ )
+ with pytest.raises(RuntimeError, match="duplicate"):
+ offline._parse_name_mapping(
+ [f"ha={'a' * 64}", f"ha={'b' * 64}"],
+ {"ha"},
+ "expected images",
+ )
+
+
+@pytest.mark.parametrize(
+ ("name", "container", "expected"),
+ [
+ (
+ "runner",
+ {
+ "Path": "python3",
+ "Args": [
+ "/opt/ukbcd/live_canary_runner.py",
+ "--confirm-one-public-fixture-lookup",
+ ],
+ },
+ live.RUNNER_PROCESS,
+ ),
+ (
+ "proxy",
+ {
+ "Path": "python3",
+ "Args": [
+ "/opt/ukbcd/live_allowlist_proxy.py",
+ "--max-requests",
+ "256",
+ "--minimum-interval-ms",
+ "0",
+ ],
+ },
+ live.PROXY_PROCESS,
+ ),
+ ],
+)
+def test_live_validator_rejects_wrong_payload_or_arguments(
+ name: str,
+ container: dict,
+ expected: tuple[str, tuple[str, ...]],
+) -> None:
+ with pytest.raises(RuntimeError, match="entrypoint or arguments"):
+ live._require_exact_process(name, container, expected)
+
+
+def test_live_validator_accepts_exact_runner_and_proxy_processes() -> None:
+ live._require_exact_process(
+ "runner",
+ {"Path": live.RUNNER_PROCESS[0], "Args": list(live.RUNNER_PROCESS[1])},
+ live.RUNNER_PROCESS,
+ )
+ live._require_exact_process(
+ "proxy",
+ {"Path": live.PROXY_PROCESS[0], "Args": list(live.PROXY_PROCESS[1])},
+ live.PROXY_PROCESS,
+ )
+
+
+def test_offline_pod_members_come_from_inspect() -> None:
+ pod = {
+ "Containers": [
+ {"Id": "infra-id", "Name": "infra"},
+ {"Id": "runner-id", "Name": "runner"},
+ ]
+ }
+
+ assert offline._pod_members(pod) == {"infra", "runner"}
+ offline._require_exact_members(
+ "offline-pod", offline._pod_members(pod), {"infra", "runner"}
+ )
+
+
+@pytest.mark.parametrize(
+ ("actual", "expected"),
+ [
+ ({"infra", "runner", "unexpected"}, {"infra", "runner"}),
+ ({"infra"}, {"infra", "runner"}),
+ ],
+)
+def test_offline_pod_rejects_extra_or_missing_member(
+ actual: set[str], expected: set[str]
+) -> None:
+ with pytest.raises(RuntimeError, match="membership mismatch"):
+ offline._require_exact_members("offline-pod", actual, expected)
+
+
+def test_live_network_members_come_from_network_inspect() -> None:
+ network = {
+ "containers": {
+ "runner-id": {"name": "runner"},
+ "selenium-id": {"name": "selenium"},
+ "proxy-id": {"name": "proxy"},
+ }
+ }
+
+ actual = live._network_members(network)
+ assert actual == {"runner", "selenium", "proxy"}
+ live._require_exact_members(
+ "internal-network", actual, {"runner", "selenium", "proxy"}
+ )
+
+
+def test_live_planned_members_scan_every_unstarted_container(monkeypatch) -> None:
+ containers = {
+ "runner": {"NetworkSettings": {"Networks": {"internal": {}}}},
+ "selenium": {"NetworkSettings": {"Networks": {"internal": {}}}},
+ "proxy": {
+ "NetworkSettings": {"Networks": {"internal": {}, "egress": {}}}
+ },
+ "unrelated": {"NetworkSettings": {"Networks": {"other": {}}}},
+ }
+ monkeypatch.setattr(live, "_all_container_names", lambda: set(containers))
+ monkeypatch.setattr(live, "_inspect_container", containers.__getitem__)
+
+ assert live._planned_network_members("internal") == {
+ "runner",
+ "selenium",
+ "proxy",
+ }
+ assert live._planned_network_members("egress") == {"proxy"}
+
+
+def test_live_accepts_empty_runtime_members_for_exact_unstarted_plan(
+ monkeypatch,
+) -> None:
+ expected = {"runner", "selenium", "proxy"}
+ monkeypatch.setattr(live, "_planned_network_members", lambda name: expected)
+
+ planned, reported, source = live._validated_network_members(
+ "internal", {"containers": {}}, expected
+ )
+
+ assert planned == expected
+ assert reported == set()
+ assert source == "all-container-inspect-unstarted-intent"
+
+
+def test_live_rejects_extra_unstarted_member_when_runtime_map_is_empty(
+ monkeypatch,
+) -> None:
+ monkeypatch.setattr(
+ live,
+ "_planned_network_members",
+ lambda name: {"runner", "selenium", "proxy", "unexpected"},
+ )
+
+ with pytest.raises(RuntimeError, match="membership mismatch"):
+ live._validated_network_members(
+ "internal",
+ {"containers": {}},
+ {"runner", "selenium", "proxy"},
+ )
+
+
+def test_live_rejects_runtime_membership_that_disagrees_with_the_plan(
+ monkeypatch,
+) -> None:
+ expected = {"runner", "selenium", "proxy"}
+ monkeypatch.setattr(live, "_planned_network_members", lambda name: expected)
+ network = {
+ "containers": {
+ "runner-id": {"name": "runner"},
+ "selenium-id": {"name": "selenium"},
+ }
+ }
+
+ with pytest.raises(RuntimeError, match="membership mismatch"):
+ live._validated_network_members("internal", network, expected)
+
+
+@pytest.mark.parametrize(
+ ("network", "expected"),
+ [
+ (
+ {
+ "containers": {
+ "proxy-id": {"name": "proxy"},
+ "unexpected-id": {"name": "unexpected"},
+ }
+ },
+ {"proxy"},
+ ),
+ ({"containers": {}}, {"proxy"}),
+ ],
+)
+def test_live_network_rejects_extra_or_missing_member(
+ network: dict, expected: set[str]
+) -> None:
+ with pytest.raises(RuntimeError, match="membership mismatch"):
+ live._require_exact_members(
+ "egress-network", live._network_members(network), expected
+ )
diff --git a/tests/disposable_ha/verify_runtime_requirements.py b/tests/disposable_ha/verify_runtime_requirements.py
new file mode 100644
index 0000000000..d00c215c19
--- /dev/null
+++ b/tests/disposable_ha/verify_runtime_requirements.py
@@ -0,0 +1,69 @@
+"""Fail an offline candidate build if its installed dependency graph is incomplete."""
+
+from __future__ import annotations
+
+import argparse
+from collections import deque
+from importlib import metadata
+
+from packaging.requirements import Requirement
+from packaging.utils import canonicalize_name
+
+
+def dependency_errors(distribution_name: str) -> list[str]:
+ """Return missing or incompatible requirements reachable from a distribution."""
+ pending = deque([distribution_name])
+ visited: set[str] = set()
+ errors: list[str] = []
+
+ while pending:
+ requested_name = pending.popleft()
+ canonical_name = canonicalize_name(requested_name)
+ if canonical_name in visited:
+ continue
+ visited.add(canonical_name)
+
+ try:
+ distribution = metadata.distribution(requested_name)
+ except metadata.PackageNotFoundError:
+ errors.append(f"missing distribution: {canonical_name}")
+ continue
+
+ for raw_requirement in distribution.requires or ():
+ requirement = Requirement(raw_requirement)
+ if requirement.marker and not requirement.marker.evaluate({"extra": ""}):
+ continue
+
+ required_name = canonicalize_name(requirement.name)
+ try:
+ installed_version = metadata.version(requirement.name)
+ except metadata.PackageNotFoundError:
+ errors.append(f"missing dependency: {required_name}")
+ continue
+
+ if requirement.specifier and not requirement.specifier.contains(
+ installed_version, prereleases=True
+ ):
+ errors.append(
+ f"incompatible dependency: {required_name}=={installed_version}"
+ )
+ continue
+ pending.append(requirement.name)
+
+ return sorted(set(errors))
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--distribution", required=True)
+ args = parser.parse_args()
+
+ errors = dependency_errors(args.distribution)
+ if errors:
+ raise SystemExit("\n".join(errors))
+ print(f"Installed dependency closure is complete for {args.distribution}.")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/tests/test_pin_release_manifest.py b/tests/test_pin_release_manifest.py
new file mode 100644
index 0000000000..429b3ccc29
--- /dev/null
+++ b/tests/test_pin_release_manifest.py
@@ -0,0 +1,192 @@
+"""Tests for fail-closed release-manifest pinning."""
+
+from __future__ import annotations
+
+import json
+import runpy
+import sys
+from pathlib import Path
+
+import pytest
+
+from scripts import pin_release_manifest
+from scripts.pin_release_manifest import pin_exact_requirement
+
+
+def _write_release_files(tmp_path: Path, requirement: str) -> tuple[Path, Path]:
+ pyproject = tmp_path / "pyproject.toml"
+ pyproject.write_text(
+ '[tool.poetry]\nname = "uk-bin-collection"\nversion = "1.2.3"\n',
+ encoding="utf-8",
+ )
+ manifest = tmp_path / "manifest.json"
+ manifest.write_text(
+ json.dumps(
+ {
+ "domain": "uk_bin_collection",
+ "version": "1.2.3",
+ "requirements": [requirement],
+ }
+ ),
+ encoding="utf-8",
+ )
+ return pyproject, manifest
+
+
+def test_release_manifest_is_pinned_only_to_matching_core(tmp_path: Path) -> None:
+ pyproject, manifest = _write_release_files(tmp_path, "uk-bin-collection>=1.2.3")
+
+ assert (
+ pin_exact_requirement("1.2.3", pyproject_path=pyproject, manifest_path=manifest)
+ == []
+ )
+ assert json.loads(manifest.read_text(encoding="utf-8"))["requirements"] == [
+ "uk-bin-collection==1.2.3"
+ ]
+
+
+def test_check_mode_does_not_mutate_manifest(tmp_path: Path) -> None:
+ pyproject, manifest = _write_release_files(tmp_path, "uk-bin-collection>=1.2.3")
+ before = manifest.read_bytes()
+
+ assert (
+ pin_exact_requirement(
+ "1.2.3",
+ pyproject_path=pyproject,
+ manifest_path=manifest,
+ check_only=True,
+ )
+ == []
+ )
+ assert manifest.read_bytes() == before
+
+
+def test_pin_refuses_version_mismatch_or_unrelated_requirements(
+ tmp_path: Path,
+) -> None:
+ pyproject, manifest = _write_release_files(tmp_path, "other-package==1.2.3")
+ before = manifest.read_bytes()
+
+ errors = pin_exact_requirement(
+ "9.9.9", pyproject_path=pyproject, manifest_path=manifest
+ )
+
+ assert "pyproject version does not match the requested release" in errors
+ assert "manifest version does not match the requested release" in errors
+ assert any("only uk-bin-collection" in error for error in errors)
+ assert manifest.read_bytes() == before
+
+
+@pytest.mark.parametrize(
+ "requirements",
+ [
+ None,
+ "uk-bin-collection>=1.2.3",
+ [],
+ ["uk-bin-collection>=1.2.3", "other-package==1.0"],
+ [123],
+ ],
+ ids=[
+ "missing",
+ "not-a-list",
+ "empty",
+ "multiple-packages",
+ "non-string",
+ ],
+)
+def test_pin_refuses_malformed_requirement_shapes(
+ tmp_path: Path,
+ requirements: object,
+) -> None:
+ pyproject, manifest = _write_release_files(tmp_path, "uk-bin-collection>=1.2.3")
+ payload = json.loads(manifest.read_text(encoding="utf-8"))
+ payload["requirements"] = requirements
+ manifest.write_text(json.dumps(payload), encoding="utf-8")
+ before = manifest.read_bytes()
+
+ errors = pin_exact_requirement(
+ "1.2.3", pyproject_path=pyproject, manifest_path=manifest
+ )
+
+ assert errors == [
+ "manifest requirements must contain only uk-bin-collection before pinning"
+ ]
+ assert manifest.read_bytes() == before
+
+
+@pytest.mark.parametrize(
+ ("argv", "expected_check_only", "expected_action"),
+ [
+ (["--version", "1.2.3"], False, "pinned"),
+ (["--version", "1.2.3", "--check"], True, "can be pinned"),
+ ],
+)
+def test_main_reports_success_without_bypassing_requested_mode(
+ monkeypatch: pytest.MonkeyPatch,
+ capsys: pytest.CaptureFixture[str],
+ argv: list[str],
+ expected_check_only: bool,
+ expected_action: str,
+) -> None:
+ calls: list[tuple[str, bool]] = []
+
+ def successful_pin(version: str, *, check_only: bool = False) -> list[str]:
+ calls.append((version, check_only))
+ return []
+
+ monkeypatch.setattr(pin_release_manifest, "pin_exact_requirement", successful_pin)
+
+ assert pin_release_manifest.main(argv) == 0
+ captured = capsys.readouterr()
+ assert calls == [("1.2.3", expected_check_only)]
+ assert (
+ captured.out == f"HA manifest {expected_action} to uk-bin-collection==1.2.3.\n"
+ )
+ assert captured.err == ""
+
+
+def test_main_reports_every_validation_error(
+ monkeypatch: pytest.MonkeyPatch,
+ capsys: pytest.CaptureFixture[str],
+) -> None:
+ errors = ["version mismatch", "unexpected requirement"]
+
+ monkeypatch.setattr(
+ pin_release_manifest,
+ "pin_exact_requirement",
+ lambda version, *, check_only=False: errors,
+ )
+
+ assert pin_release_manifest.main(["--version", "1.2.3", "--check"]) == 1
+ captured = capsys.readouterr()
+ assert captured.out == ""
+ assert captured.err == ("ERROR: version mismatch\nERROR: unexpected requirement\n")
+
+
+def test_module_entrypoint_runs_repository_check_without_mutating_manifest(
+ monkeypatch: pytest.MonkeyPatch,
+ capsys: pytest.CaptureFixture[str],
+) -> None:
+ manifest_before = pin_release_manifest.MANIFEST.read_bytes()
+ version = json.loads(manifest_before)["version"]
+ monkeypatch.setattr(
+ sys,
+ "argv",
+ [
+ str(pin_release_manifest.__file__),
+ "--version",
+ version,
+ "--check",
+ ],
+ )
+
+ with pytest.raises(SystemExit) as exc_info:
+ runpy.run_path(str(pin_release_manifest.__file__), run_name="__main__")
+
+ assert exc_info.value.code == 0
+ assert pin_release_manifest.MANIFEST.read_bytes() == manifest_before
+ captured = capsys.readouterr()
+ assert (
+ captured.out == f"HA manifest can be pinned to uk-bin-collection=={version}.\n"
+ )
+ assert captured.err == ""
diff --git a/tests/test_release_workflow_order.py b/tests/test_release_workflow_order.py
new file mode 100644
index 0000000000..f8afd3da57
--- /dev/null
+++ b/tests/test_release_workflow_order.py
@@ -0,0 +1,44 @@
+"""Regression checks for safe core/component publication ordering."""
+
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def test_core_is_verified_before_manifest_bearing_branch_is_pushed() -> None:
+ workflow = (ROOT / ".github" / "workflows" / "bump.yml").read_text(encoding="utf-8")
+
+ publish = workflow.index("name: Publish core to PyPI before repository writes")
+ verify = workflow.index(
+ "name: Verify published core bytes before repository writes"
+ )
+ repository_push = workflow.index("git push --atomic origin master")
+
+ assert publish < verify < repository_push
+ assert "--pypi-json dist/pypi-release.json" in workflow[verify:repository_push]
+ push_step = workflow[
+ workflow.index("name: Push installable release commit and tag") :
+ ]
+ assert push_step.count("git push") == 1
+ assert "refs/tags/${{ steps.bump.outputs.version }}" in push_step
+
+
+def test_tag_release_cannot_publish_or_load_pypi_credentials() -> None:
+ workflow = (ROOT / ".github" / "workflows" / "release.yml").read_text(
+ encoding="utf-8"
+ )
+
+ assert "poetry publish" not in workflow
+ assert "PYPI_API_KEY" not in workflow
+ assert "Verify pre-published exact core artifact from PyPI" in workflow
+
+
+def test_prepublication_and_tag_rebuild_use_same_reproducible_epoch() -> None:
+ bump = (ROOT / ".github" / "workflows" / "bump.yml").read_text(encoding="utf-8")
+ release = (ROOT / ".github" / "workflows" / "release.yml").read_text(
+ encoding="utf-8"
+ )
+
+ marker = "SOURCE_DATE_EPOCH=315532800"
+ assert bump.count(marker) == 1
+ assert release.count(marker) == 1
diff --git a/tests/test_validate_release_contract.py b/tests/test_validate_release_contract.py
new file mode 100644
index 0000000000..7e8a181daa
--- /dev/null
+++ b/tests/test_validate_release_contract.py
@@ -0,0 +1,308 @@
+"""Tests for the fail-closed release contract."""
+
+from __future__ import annotations
+
+import hashlib
+import importlib.util
+import json
+import runpy
+import sys
+import zipfile
+from pathlib import Path
+
+import pytest
+
+SCRIPT = (
+ Path(__file__).resolve().parents[1] / "scripts" / "validate_release_contract.py"
+)
+SPEC = importlib.util.spec_from_file_location("validate_release_contract", SCRIPT)
+assert SPEC and SPEC.loader
+release_contract = importlib.util.module_from_spec(SPEC)
+SPEC.loader.exec_module(release_contract)
+
+
+def _write_project(
+ tmp_path: Path,
+ *,
+ requirement: str,
+ project_version: str = "1.2.3",
+ manifest_version: str = "1.2.3",
+) -> tuple[Path, Path]:
+ pyproject = tmp_path / "pyproject.toml"
+ pyproject.write_text(
+ f'[tool.poetry]\nversion = "{project_version}"\n', encoding="utf-8"
+ )
+ manifest = tmp_path / "manifest.json"
+ manifest.write_text(
+ json.dumps(
+ {
+ "version": manifest_version,
+ "requirements": [requirement],
+ }
+ ),
+ encoding="utf-8",
+ )
+ return pyproject, manifest
+
+
+def _write_wheel(tmp_path: Path, *, version: str = "1.2.3") -> Path:
+ wheel = tmp_path / f"uk_bin_collection-{version}-py3-none-any.whl"
+ with zipfile.ZipFile(wheel, "w") as archive:
+ archive.writestr(
+ f"uk_bin_collection-{version}.dist-info/METADATA",
+ "Metadata-Version: 2.4\n"
+ "Name: uk-bin-collection\n"
+ f"Version: {version}\n"
+ "Requires-Python: >=3.12,<3.15\n",
+ )
+ return wheel
+
+
+def test_broad_requirement_is_allowed_before_release_but_not_at_release(
+ tmp_path: Path,
+) -> None:
+ pyproject, manifest = _write_project(
+ tmp_path, requirement="uk-bin-collection>=1.2.3"
+ )
+ assert (
+ release_contract.validate_release_metadata(
+ "1.2.3",
+ require_exact=False,
+ pyproject_path=pyproject,
+ manifest_path=manifest,
+ )
+ == []
+ )
+ assert (
+ "manifest must contain only"
+ in release_contract.validate_release_metadata(
+ "1.2.3",
+ require_exact=True,
+ pyproject_path=pyproject,
+ manifest_path=manifest,
+ )[0]
+ )
+
+
+def test_release_metadata_reports_core_and_component_version_mismatches(
+ tmp_path: Path,
+) -> None:
+ pyproject, manifest = _write_project(
+ tmp_path,
+ requirement="uk-bin-collection==1.2.3",
+ project_version="1.2.2",
+ manifest_version="1.2.4",
+ )
+
+ assert release_contract.validate_release_metadata(
+ "1.2.3",
+ require_exact=True,
+ pyproject_path=pyproject,
+ manifest_path=manifest,
+ ) == [
+ "pyproject version '1.2.2' does not match '1.2.3'",
+ "manifest version '1.2.4' does not match '1.2.3'",
+ ]
+
+
+def test_wheel_metadata_must_match_release_version(tmp_path: Path) -> None:
+ wheel = _write_wheel(tmp_path, version="1.2.4")
+ errors = release_contract.validate_wheel(wheel, "1.2.3")
+ assert any("wheel version" in error for error in errors)
+
+
+def test_wheel_must_contain_exactly_one_metadata_file(tmp_path: Path) -> None:
+ wheel = tmp_path / "missing-metadata.whl"
+ with zipfile.ZipFile(wheel, "w") as archive:
+ archive.writestr("uk_bin_collection/__init__.py", "")
+ assert release_contract.validate_wheel(wheel, "1.2.3") == [
+ "wheel must contain exactly one dist-info/METADATA; found 0"
+ ]
+
+ wheel = tmp_path / "duplicate-metadata.whl"
+ with zipfile.ZipFile(wheel, "w") as archive:
+ archive.writestr(
+ "one.dist-info/METADATA",
+ "Name: uk-bin-collection\nVersion: 1.2.3\nRequires-Python: >=3.12\n",
+ )
+ archive.writestr(
+ "two.dist-info/METADATA",
+ "Name: uk-bin-collection\nVersion: 1.2.3\nRequires-Python: >=3.12\n",
+ )
+ assert release_contract.validate_wheel(wheel, "1.2.3") == [
+ "wheel must contain exactly one dist-info/METADATA; found 2"
+ ]
+
+
+def test_wheel_must_be_a_readable_zip_archive(tmp_path: Path) -> None:
+ wheel = tmp_path / "invalid.whl"
+ wheel.write_text("not a zip archive", encoding="utf-8")
+
+ errors = release_contract.validate_wheel(wheel, "1.2.3")
+
+ assert len(errors) == 1
+ assert errors[0].startswith(f"cannot read wheel {wheel}:")
+
+
+def test_wheel_rejects_wrong_distribution_and_missing_python_metadata(
+ tmp_path: Path,
+) -> None:
+ wheel = tmp_path / "wrong-package.whl"
+ with zipfile.ZipFile(wheel, "w") as archive:
+ archive.writestr(
+ "wrong_package-1.2.3.dist-info/METADATA",
+ "Metadata-Version: 2.4\n" "Name: wrong_package\n" "Version: 1.2.3\n",
+ )
+
+ assert release_contract.validate_wheel(wheel, "1.2.3") == [
+ "wheel distribution 'wrong_package' is not 'uk-bin-collection'",
+ "wheel is missing Requires-Python metadata",
+ ]
+
+
+def test_pypi_digest_must_match_local_wheel(tmp_path: Path) -> None:
+ wheel = _write_wheel(tmp_path)
+ digest = hashlib.sha256(wheel.read_bytes()).hexdigest()
+ release_json = tmp_path / "release.json"
+ release_json.write_text(
+ json.dumps(
+ {
+ "info": {"version": "1.2.3"},
+ "urls": [
+ {
+ "filename": wheel.name,
+ "digests": {"sha256": digest},
+ }
+ ],
+ }
+ ),
+ encoding="utf-8",
+ )
+ assert release_contract.validate_pypi_digest(wheel, release_json, "1.2.3") == []
+
+ payload = json.loads(release_json.read_text(encoding="utf-8"))
+ payload["urls"][0]["digests"]["sha256"] = "0" * 64
+ release_json.write_text(json.dumps(payload), encoding="utf-8")
+ assert (
+ "does not match"
+ in release_contract.validate_pypi_digest(wheel, release_json, "1.2.3")[0]
+ )
+
+
+def test_pypi_release_json_must_be_readable(tmp_path: Path) -> None:
+ wheel = _write_wheel(tmp_path)
+ release_json = tmp_path / "release.json"
+ release_json.write_text("{", encoding="utf-8")
+
+ errors = release_contract.validate_pypi_digest(wheel, release_json, "1.2.3")
+
+ assert len(errors) == 1
+ assert errors[0].startswith(f"cannot read PyPI release JSON {release_json}:")
+
+
+def test_pypi_release_must_match_version_and_contain_the_wheel(
+ tmp_path: Path,
+) -> None:
+ wheel = _write_wheel(tmp_path)
+ release_json = tmp_path / "release.json"
+ release_json.write_text(
+ json.dumps(
+ {
+ "info": {"version": "1.2.4"},
+ "urls": [{"filename": "another-file.whl"}],
+ }
+ ),
+ encoding="utf-8",
+ )
+
+ assert release_contract.validate_pypi_digest(wheel, release_json, "1.2.3") == [
+ "PyPI JSON version '1.2.4' does not match '1.2.3'",
+ f"PyPI JSON must contain exactly one {wheel.name!r}; found 0",
+ ]
+
+
+def test_main_runs_all_requested_validators_and_reports_success(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+ wheel = tmp_path / "candidate.whl"
+ release_json = tmp_path / "release.json"
+ calls: list[tuple[object, ...]] = []
+
+ def validate_metadata(version: str, *, require_exact: bool) -> list[str]:
+ calls.append(("metadata", version, require_exact))
+ return []
+
+ def validate_wheel(wheel_path: Path, version: str) -> list[str]:
+ calls.append(("wheel", wheel_path, version))
+ return []
+
+ def validate_digest(
+ wheel_path: Path, pypi_json_path: Path, version: str
+ ) -> list[str]:
+ calls.append(("pypi", wheel_path, pypi_json_path, version))
+ return []
+
+ monkeypatch.setattr(
+ release_contract, "validate_release_metadata", validate_metadata
+ )
+ monkeypatch.setattr(release_contract, "validate_wheel", validate_wheel)
+ monkeypatch.setattr(release_contract, "validate_pypi_digest", validate_digest)
+
+ result = release_contract.main(
+ [
+ "--version",
+ "1.2.3",
+ "--require-exact",
+ "--wheel",
+ str(wheel),
+ "--pypi-json",
+ str(release_json),
+ ]
+ )
+
+ assert result == 0
+ assert calls == [
+ ("metadata", "1.2.3", True),
+ ("wheel", wheel, "1.2.3"),
+ ("pypi", wheel, release_json, "1.2.3"),
+ ]
+ assert capsys.readouterr().out == (
+ "Release contract valid for 1.2.3: core/component versions, "
+ "exact HA core requirement, wheel metadata, published wheel SHA-256.\n"
+ )
+
+
+def test_main_reports_all_errors_and_requires_a_wheel_for_pypi_json(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+ release_json = tmp_path / "release.json"
+ monkeypatch.setattr(
+ release_contract,
+ "validate_release_metadata",
+ lambda version, *, require_exact: ["metadata mismatch"],
+ )
+
+ result = release_contract.main(
+ ["--version", "1.2.3", "--pypi-json", str(release_json)]
+ )
+
+ assert result == 1
+ assert capsys.readouterr().err == (
+ "ERROR: metadata mismatch\nERROR: --pypi-json requires --wheel\n"
+ )
+
+
+def test_script_entrypoint_validates_the_repository_release_contract(
+ monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
+) -> None:
+ project = release_contract.tomllib.loads(
+ (SCRIPT.parents[1] / "pyproject.toml").read_text(encoding="utf-8")
+ )
+ version = project["tool"]["poetry"]["version"]
+ monkeypatch.setattr(sys, "argv", [str(SCRIPT), "--version", version])
+
+ with pytest.raises(SystemExit) as exc_info:
+ runpy.run_path(str(SCRIPT), run_name="__main__")
+
+ assert exc_info.value.code == 0
+ assert capsys.readouterr().out.startswith(f"Release contract valid for {version}:")
diff --git a/tests/test_workflow_actions_pinned.py b/tests/test_workflow_actions_pinned.py
new file mode 100644
index 0000000000..3ead81ad80
--- /dev/null
+++ b/tests/test_workflow_actions_pinned.py
@@ -0,0 +1,34 @@
+"""Supply-chain checks for GitHub Actions workflow dependencies."""
+
+from pathlib import Path
+import re
+
+WORKFLOWS = Path(__file__).parents[1] / ".github" / "workflows"
+USES_PATTERN = re.compile(r"\buses:\s*[\"']?([^\"'\s#]+)")
+FULL_COMMIT_PATTERN = re.compile(r"^[0-9a-f]{40}$")
+
+
+def test_external_actions_are_pinned_to_full_commit_shas() -> None:
+ """Reject mutable tags, branches, and abbreviated commit references."""
+
+ violations: list[str] = []
+
+ for workflow in sorted(WORKFLOWS.glob("*.yml")):
+ for line_number, line in enumerate(
+ workflow.read_text(encoding="utf-8").splitlines(), start=1
+ ):
+ match = USES_PATTERN.search(line)
+ if not match:
+ continue
+
+ action = match.group(1)
+ if action.startswith("./"):
+ continue
+
+ _, separator, revision = action.rpartition("@")
+ if not separator or not FULL_COMMIT_PATTERN.fullmatch(revision):
+ violations.append(f"{workflow.name}:{line_number}: {action}")
+
+ assert not violations, "External actions must use full commit SHAs:\n" + "\n".join(
+ violations
+ )
diff --git a/uk_bin_collection/tests/test_birmingham_city_council.py b/uk_bin_collection/tests/test_birmingham_city_council.py
new file mode 100644
index 0000000000..5fe090870d
--- /dev/null
+++ b/uk_bin_collection/tests/test_birmingham_city_council.py
@@ -0,0 +1,64 @@
+"""Regression tests for Birmingham's standalone runtime dependency boundary."""
+
+from __future__ import annotations
+
+import builtins
+import importlib
+import sys
+from types import SimpleNamespace
+from unittest.mock import MagicMock, patch
+
+
+MODULE_NAME = (
+ "uk_bin_collection.uk_bin_collection.councils.BirminghamCityCouncil"
+)
+COLLECTION_URL = (
+ "https://www.birmingham.gov.uk/info/50388/check_your_collection_day"
+)
+
+
+def test_birmingham_import_does_not_require_yarl(monkeypatch) -> None:
+ """The packaged council module must use only declared runtime dependencies."""
+ sys.modules.pop(MODULE_NAME, None)
+ real_import = builtins.__import__
+
+ def guarded_import(name, *args, **kwargs):
+ if name == "yarl" or name.startswith("yarl."):
+ raise ModuleNotFoundError("No module named 'yarl'", name="yarl")
+ return real_import(name, *args, **kwargs)
+
+ monkeypatch.setattr(builtins, "__import__", guarded_import)
+
+ module = importlib.import_module(MODULE_NAME)
+
+ assert module.CouncilClass.__name__ == "CouncilClass"
+
+
+def test_birmingham_passes_address_as_requests_query_parameters() -> None:
+ module = importlib.import_module(MODULE_NAME)
+ response = SimpleNamespace(
+ text='',
+ raise_for_status=MagicMock(),
+ )
+ postcode = "B1 1AA"
+ uprn = "100000000001"
+
+ with (
+ patch.object(module, "check_uprn"),
+ patch.object(module, "check_postcode"),
+ patch.object(module.requests, "get", return_value=response) as get,
+ ):
+ result = module.CouncilClass().parse_data(
+ "",
+ postcode=postcode,
+ uprn=uprn,
+ )
+
+ assert result == {"bins": []}
+ response.raise_for_status.assert_called_once_with()
+ get.assert_called_once_with(
+ COLLECTION_URL,
+ params={"postcode": postcode, "uprn": uprn},
+ headers=module.HEADERS,
+ timeout=30,
+ )
diff --git a/uk_bin_collection/tests/test_collect_data.py b/uk_bin_collection/tests/test_collect_data.py
index 086b9ff12f..171894f2fa 100644
--- a/uk_bin_collection/tests/test_collect_data.py
+++ b/uk_bin_collection/tests/test_collect_data.py
@@ -1,7 +1,37 @@
-from unittest.mock import MagicMock, patch
import argparse
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import MagicMock, patch
+
import pytest
-from uk_bin_collection.collect_data import UKBinCollectionApp, import_council_module
+from uk_bin_collection.uk_bin_collection import collect_data
+from uk_bin_collection.uk_bin_collection.collect_data import (
+ UKBinCollectionApp,
+)
+from uk_bin_collection.uk_bin_collection.exceptions import (
+ InvalidCouncilModuleError,
+ MissingDependencyError,
+)
+
+
+def _configure_registered_council(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> Path:
+ """Configure a trusted synthetic council for loader branch tests."""
+ council_file = tmp_path / "ExampleCouncil.py"
+ council_file.write_text("# synthetic council\n", encoding="utf-8")
+ monkeypatch.setattr(
+ collect_data,
+ "registered_council_modules",
+ lambda: frozenset({"ExampleCouncil"}),
+ )
+ monkeypatch.setattr(
+ collect_data, "council_requires_selenium", lambda _module_name: False
+ )
+ monkeypatch.setattr(
+ collect_data, "_council_file", lambda _module_name: council_file
+ )
+ return council_file
# Test UKBinCollectionApp setup_arg_parser
@@ -21,7 +51,10 @@ def test_setup_arg_parser():
"number",
"skip_get_url",
"uprn",
+ "usrn",
"web_driver",
+ "artifact_dir",
+ "user_agent",
"headless",
"local_browser",
"dev_mode",
@@ -41,6 +74,7 @@ def test_set_args():
assert app.parsed_args.module == "council_module"
assert app.parsed_args.URL == "http://example.com"
assert app.parsed_args.postcode == "AB1 2CD"
+ assert app.parsed_args.headless is True
# Test UKBinCollectionApp client_code method
@@ -56,11 +90,11 @@ def test_client_code():
# Test the run() function with logging setup
-@patch("uk_bin_collection.collect_data.setup_logging") # Correct patch path
-@patch("uk_bin_collection.collect_data.UKBinCollectionApp.run") # Correct patch path
+@patch("uk_bin_collection.uk_bin_collection.collect_data.setup_logging")
+@patch("uk_bin_collection.uk_bin_collection.collect_data.UKBinCollectionApp.run")
@patch("sys.argv", ["uk_bin_collection.py", "council_module", "http://example.com"])
def test_run_function(mock_app_run, mock_setup_logging):
- from uk_bin_collection.collect_data import run
+ from uk_bin_collection.uk_bin_collection.collect_data import run
mock_setup_logging.return_value = MagicMock()
mock_app_run.return_value = None
@@ -70,3 +104,168 @@ def test_run_function(mock_app_run, mock_setup_logging):
# Ensure logging was set up and the app run method was called
mock_setup_logging.assert_called_once()
mock_app_run.assert_called_once()
+
+
+def test_run_propagates_all_supported_arguments():
+ app = UKBinCollectionApp()
+ app.set_args(
+ [
+ "ExampleCouncil",
+ "https://example.invalid/collections",
+ "--postcode=AB1 2CD",
+ "--number=42A",
+ "--uprn=100012345",
+ "--usrn=200012345",
+ "--skip_get_url",
+ "--web_driver=http://selenium.invalid:4444/wd/hub",
+ "--artifact_dir=/tmp/ukbcd-artifacts",
+ "--user_agent=UKBCD contract test",
+ "--not-headless",
+ "--local_browser",
+ "--dev_mode",
+ ]
+ )
+ council = MagicMock()
+ module = MagicMock()
+ module.CouncilClass.return_value = council
+
+ with patch(
+ "uk_bin_collection.uk_bin_collection.collect_data.import_council_module",
+ return_value=module,
+ ):
+ app.run()
+
+ council.template_method.assert_called_once_with(
+ "https://example.invalid/collections",
+ postcode="AB1 2CD",
+ paon="42A",
+ uprn="100012345",
+ usrn="200012345",
+ skip_get_url=True,
+ web_driver="http://selenium.invalid:4444/wd/hub",
+ artifact_dir="/tmp/ukbcd-artifacts",
+ user_agent="UKBCD contract test",
+ headless=False,
+ local_browser=True,
+ dev_mode=True,
+ council_module_str="ExampleCouncil",
+ )
+
+
+def test_council_requires_selenium_detects_direct_import(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+ council_file = tmp_path / "DirectImportCouncil.py"
+ council_file.write_text("import selenium.webdriver\n", encoding="utf-8")
+ monkeypatch.setattr(
+ collect_data, "_council_file", lambda _module_name: council_file
+ )
+
+ collect_data.council_requires_selenium.cache_clear()
+ try:
+ assert collect_data.council_requires_selenium("DirectImportCouncil") is True
+ finally:
+ collect_data.council_requires_selenium.cache_clear()
+
+
+def test_council_requires_selenium_rejects_uninspectable_source(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+ council_file = tmp_path / "BrokenCouncil.py"
+ council_file.write_text("from selenium import\n", encoding="utf-8")
+ monkeypatch.setattr(
+ collect_data, "_council_file", lambda _module_name: council_file
+ )
+
+ collect_data.council_requires_selenium.cache_clear()
+ try:
+ with pytest.raises(
+ InvalidCouncilModuleError, match="cannot be inspected safely"
+ ) as error:
+ collect_data.council_requires_selenium("BrokenCouncil")
+ finally:
+ collect_data.council_requires_selenium.cache_clear()
+
+ assert isinstance(error.value.__cause__, SyntaxError)
+
+
+def test_council_loader_wraps_selenium_resolution_errors(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+ _configure_registered_council(monkeypatch, tmp_path)
+ monkeypatch.setattr(
+ collect_data, "council_requires_selenium", lambda _module_name: True
+ )
+ resolution_error = ValueError("selenium.__spec__ is invalid")
+ find_spec = MagicMock(side_effect=resolution_error)
+ import_module = MagicMock()
+ monkeypatch.setattr(collect_data.import_util, "find_spec", find_spec)
+ monkeypatch.setattr(collect_data.importlib, "import_module", import_module)
+
+ with pytest.raises(MissingDependencyError, match="cannot safely resolve") as error:
+ collect_data.import_council_module("ExampleCouncil")
+
+ assert error.value.__cause__ is resolution_error
+ find_spec.assert_called_once_with("selenium")
+ import_module.assert_not_called()
+
+
+def test_council_loader_wraps_qualified_module_resolution_errors(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+ _configure_registered_council(monkeypatch, tmp_path)
+ resolution_error = ImportError("qualified module cannot be resolved")
+ find_spec = MagicMock(side_effect=resolution_error)
+ import_module = MagicMock()
+ monkeypatch.setattr(collect_data.import_util, "find_spec", find_spec)
+ monkeypatch.setattr(collect_data.importlib, "import_module", import_module)
+
+ with pytest.raises(
+ InvalidCouncilModuleError, match="cannot be resolved safely"
+ ) as error:
+ collect_data.import_council_module("ExampleCouncil")
+
+ assert error.value.__cause__ is resolution_error
+ find_spec.assert_called_once_with(
+ "uk_bin_collection.uk_bin_collection.councils.ExampleCouncil"
+ )
+ import_module.assert_not_called()
+
+
+def test_council_loader_rejects_module_loaded_from_unexpected_location(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+ council_file = _configure_registered_council(monkeypatch, tmp_path)
+ specification = SimpleNamespace(origin=str(council_file))
+ loaded_module = SimpleNamespace(
+ __file__=str(tmp_path / "shadow" / "ExampleCouncil.py"),
+ CouncilClass=object,
+ )
+ monkeypatch.setattr(
+ collect_data.import_util, "find_spec", lambda _qualified_name: specification
+ )
+ monkeypatch.setattr(
+ collect_data.importlib, "import_module", lambda _qualified_name: loaded_module
+ )
+
+ with pytest.raises(
+ InvalidCouncilModuleError, match="loaded from an unexpected location"
+ ):
+ collect_data.import_council_module("ExampleCouncil")
+
+
+def test_council_loader_requires_council_class(
+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+ council_file = _configure_registered_council(monkeypatch, tmp_path)
+ specification = SimpleNamespace(origin=str(council_file))
+ loaded_module = SimpleNamespace(__file__=str(council_file))
+ monkeypatch.setattr(
+ collect_data.import_util, "find_spec", lambda _qualified_name: specification
+ )
+ monkeypatch.setattr(
+ collect_data.importlib, "import_module", lambda _qualified_name: loaded_module
+ )
+
+ with pytest.raises(InvalidCouncilModuleError, match="does not expose CouncilClass"):
+ collect_data.import_council_module("ExampleCouncil")
diff --git a/uk_bin_collection/tests/test_common_functions.py b/uk_bin_collection/tests/test_common_functions.py
index ba8f279d1d..467d2d00c9 100644
--- a/uk_bin_collection/tests/test_common_functions.py
+++ b/uk_bin_collection/tests/test_common_functions.py
@@ -5,24 +5,59 @@
import pytest
from selenium.common.exceptions import WebDriverException
-from uk_bin_collection.common import *
-from urllib3.exceptions import MaxRetryError
+from uk_bin_collection.uk_bin_collection import common as common_module
+from uk_bin_collection.uk_bin_collection.common import *
def test_check_postcode_valid():
valid_postcode = "SW1A 1AA"
- result = check_postcode(valid_postcode)
+ response = MagicMock(status_code=200)
+ with patch(
+ "uk_bin_collection.uk_bin_collection.common.requests.get", return_value=response
+ ) as get:
+ result = check_postcode(valid_postcode)
assert result is True
+ get.assert_called_once_with(
+ "https://api.postcodes.io/postcodes/SW1A 1AA", timeout=(3.05, 10)
+ )
def test_check_postcode_invalid():
invalid_postcode = "BADPOSTCODE"
- with pytest.raises(ValueError) as exc_info:
- result = check_postcode(invalid_postcode)
+ response = MagicMock(status_code=404)
+ response.json.return_value = {"error": "Invalid postcode", "status": 404}
+ with (
+ patch(
+ "uk_bin_collection.uk_bin_collection.common.requests.get",
+ return_value=response,
+ ),
+ pytest.raises(ValueError) as exc_info,
+ ):
+ check_postcode(invalid_postcode)
assert exc_info._excinfo[1].args[0] == "Exception: Invalid postcode Status: 404"
assert exc_info.type == ValueError
+def test_check_postcode_handles_non_json_rejection():
+ response = MagicMock(status_code=503)
+ response.json.side_effect = ValueError("not JSON")
+
+ with (
+ patch(
+ "uk_bin_collection.uk_bin_collection.common.requests.get",
+ return_value=response,
+ ),
+ pytest.raises(
+ ValueError,
+ match=(
+ "Exception: Postcode validation service rejected the request "
+ "Status: 503"
+ ),
+ ),
+ ):
+ check_postcode("SW1A 1AA")
+
+
def test_check_paon():
valid_house_num = "1"
result = check_paon(valid_house_num)
@@ -34,7 +69,7 @@ def test_check_paon_invalid(capfd):
with pytest.raises(SystemExit) as exc_info:
result = check_paon(invalid_house_num)
out, err = capfd.readouterr()
- assert out.startswith("Exception encountered: Invalid house number")
+ assert out.startswith("Exception encountered: ValueError")
assert exc_info.type == SystemExit
assert exc_info.value.code == 1
@@ -222,8 +257,8 @@ def test_get_next_occurrence_from_day_month_true():
assert result == pd.Timestamp("2024-01-01 00:00:00")
-@patch("uk_bin_collection.common.load_data", return_value={})
-@patch("uk_bin_collection.common.save_data")
+@patch("uk_bin_collection.uk_bin_collection.common.load_data", return_value={})
+@patch("uk_bin_collection.uk_bin_collection.common.save_data")
def test_update_input_json(mock_save_data, mock_load_data):
update_input_json(
"test_council",
@@ -248,20 +283,18 @@ def test_update_input_json(mock_save_data, mock_load_data):
mock_save_data.assert_called_once_with("path/to/input.json", expected_data)
-@patch("uk_bin_collection.common.load_data")
-@patch("uk_bin_collection.common.save_data")
+@patch("uk_bin_collection.uk_bin_collection.common.load_data")
+@patch("uk_bin_collection.uk_bin_collection.common.save_data")
def test_update_input_json_ioerror(mock_save_data, mock_load_data):
mock_load_data.side_effect = IOError("Unable to access file")
with patch("builtins.print") as mock_print:
update_input_json("test_council", "TEST_URL", "path/to/input.json")
- mock_print.assert_called_once_with(
- "Error updating the JSON file: Unable to access file"
- )
+ mock_print.assert_called_once_with("Error updating the JSON file: OSError")
-@patch("uk_bin_collection.common.load_data")
-@patch("uk_bin_collection.common.save_data")
+@patch("uk_bin_collection.uk_bin_collection.common.load_data")
+@patch("uk_bin_collection.uk_bin_collection.common.save_data")
def test_update_input_json_jsondecodeerror(mock_save_data, mock_load_data):
mock_load_data.side_effect = json.JSONDecodeError("Expecting value", "doc", 0)
@@ -295,8 +328,12 @@ def test_load_data_non_existing_file():
def test_load_data_invalid_json():
# Create a mock file with invalid JSON content
mock_file_data = '{"key": "value"'
- with patch("builtins.open", mock_open(read_data=mock_file_data)), patch(
- "json.load", side_effect=json.JSONDecodeError("Expecting ',' delimiter", "", 0)
+ with (
+ patch("builtins.open", mock_open(read_data=mock_file_data)),
+ patch(
+ "json.load",
+ side_effect=json.JSONDecodeError("Expecting ',' delimiter", "", 0),
+ ),
):
data = load_data("path/to/invalid.json")
assert data == {} # Modify based on your desired behavior
@@ -359,18 +396,69 @@ def test_contains_date_with_mixed_content():
def test_create_webdriver_local():
- result = create_webdriver(
- None, headless=True, user_agent="FireFox", session_name="test-session"
- )
+ mock_webdriver = mock.MagicMock()
+ mock_options = mock.MagicMock()
+ mock_webdriver.ChromeOptions.return_value = mock_options
+ mock_driver = mock.MagicMock(name="local-driver")
+ mock_driver.name = "chrome"
+ mock_webdriver.Chrome.return_value = mock_driver
+ mock_service = mock.MagicMock()
+ mock_manager = mock.MagicMock()
+ mock_manager.return_value.install.return_value = "/drivers/chromedriver"
+
+ with (
+ mock.patch(
+ "uk_bin_collection.uk_bin_collection.common._load_selenium_dependencies",
+ return_value=(mock_webdriver, WebDriverException),
+ ),
+ mock.patch(
+ "uk_bin_collection.uk_bin_collection.common._load_local_chrome_dependencies",
+ return_value=(mock_service, mock_manager),
+ ),
+ ):
+ result = create_webdriver(
+ None, headless=True, user_agent="FireFox", session_name="test-session"
+ )
+
assert result.name in ["chrome", "chrome-headless-shell"]
+ mock_webdriver.Chrome.assert_called_once()
def test_create_webdriver_remote_failure():
# Test the scenario where the remote server is not available
- with pytest.raises(MaxRetryError) as exc_info:
+ mock_webdriver = mock.MagicMock()
+ mock_webdriver.ChromeOptions.return_value = mock.MagicMock()
+ mock_webdriver.Remote.side_effect = WebDriverException("server unavailable")
+ with (
+ mock.patch(
+ "uk_bin_collection.uk_bin_collection.common._load_selenium_dependencies",
+ return_value=(mock_webdriver, WebDriverException),
+ ),
+ pytest.raises(BrowserUnavailableError),
+ ):
create_webdriver("http://invalid-url:4444", False)
+@pytest.mark.parametrize(
+ "web_driver",
+ ["/path/to/webdriver", "selenium:4444", "ftp://selenium:4444", "http://:4444"],
+)
+def test_create_webdriver_rejects_malformed_remote_url(web_driver):
+ mock_webdriver = mock.MagicMock()
+ mock_webdriver.ChromeOptions.return_value = mock.MagicMock()
+
+ with (
+ mock.patch(
+ "uk_bin_collection.uk_bin_collection.common._load_selenium_dependencies",
+ return_value=(mock_webdriver, WebDriverException),
+ ),
+ pytest.raises(BrowserUnavailableError, match="WebDriver URL"),
+ ):
+ create_webdriver(web_driver)
+
+ mock_webdriver.Remote.assert_not_called()
+
+
def test_create_webdriver_remote_with_session_name():
# Test creating a remote WebDriver with a session name
session_name = "test-session"
@@ -378,22 +466,142 @@ def test_create_webdriver_remote_with_session_name():
"http://localhost:4444/wd/hub" # Use a valid remote WebDriver URL for testing
)
- # Mock the Remote WebDriver
- with mock.patch("uk_bin_collection.common.webdriver.Remote") as mock_remote:
+ # Mock the lazily loaded Remote WebDriver
+ mock_webdriver = mock.MagicMock()
+ mock_options = mock.MagicMock()
+ mock_options._caps = {}
+ mock_options.set_capability.side_effect = mock_options._caps.__setitem__
+ mock_webdriver.ChromeOptions.return_value = mock_options
+ mock_service = mock.MagicMock()
+ mock_manager = mock.MagicMock()
+ with mock.patch(
+ "uk_bin_collection.uk_bin_collection.common._load_selenium_dependencies",
+ return_value=(mock_webdriver, WebDriverException),
+ ):
mock_instance = mock.MagicMock()
mock_instance.name = "chrome"
- mock_remote.return_value = mock_instance
+ mock_webdriver.Remote.return_value = mock_instance
# Call the function with the test parameters
result = create_webdriver(web_driver=web_driver_url, session_name=session_name)
# Check if the session name was set in capabilities
- args, kwargs = mock_remote.call_args
+ args, kwargs = mock_webdriver.Remote.call_args
options = kwargs["options"]
assert options._caps.get("se:name") == session_name
assert result.name == "chrome"
+def test_create_webdriver_remote_uses_bounded_client_config():
+ mock_webdriver = mock.MagicMock()
+ mock_webdriver.ChromeOptions.return_value = mock.MagicMock()
+ mock_driver = mock.MagicMock(name="remote-driver")
+ mock_webdriver.Remote.return_value = mock_driver
+ client_config = object()
+
+ with (
+ mock.patch(
+ "uk_bin_collection.uk_bin_collection.common._load_selenium_dependencies",
+ return_value=(mock_webdriver, WebDriverException),
+ ),
+ mock.patch(
+ "uk_bin_collection.uk_bin_collection.common._build_remote_client_config",
+ return_value=client_config,
+ ) as build_config,
+ mock.patch(
+ "uk_bin_collection.uk_bin_collection.common._load_local_chrome_dependencies",
+ side_effect=AssertionError(
+ "remote WebDriver must not require local Chrome dependencies"
+ ),
+ ) as local_dependencies,
+ ):
+ result = create_webdriver(web_driver="http://selenium:4444", command_timeout=17)
+
+ assert result is mock_driver
+ local_dependencies.assert_not_called()
+ build_config.assert_called_once_with("http://selenium:4444", 17)
+ assert mock_webdriver.Remote.call_args.kwargs["client_config"] is client_config
+
+
+def test_remote_webdriver_url_rejects_invalid_port():
+ with pytest.raises(BrowserUnavailableError, match="URL is invalid"):
+ common_module._validate_remote_webdriver_url("http://selenium:not-a-port")
+
+
+def test_remote_client_config_validates_dependency_and_bounds_timeout():
+ client_config_class = MagicMock()
+ client_config = object()
+ client_config_class.return_value = client_config
+
+ with (
+ patch.object(common_module, "validate_websocket_client") as validate,
+ patch(
+ "selenium.webdriver.remote.client_config.ClientConfig",
+ client_config_class,
+ ),
+ ):
+ result = common_module._build_remote_client_config("http://selenium:4444", 0.25)
+
+ assert result is client_config
+ validate.assert_called_once_with()
+ client_config_class.assert_called_once_with(
+ remote_server_addr="http://selenium:4444",
+ timeout=1,
+ websocket_timeout=1.0,
+ )
+
+
+def test_selenium_dependency_loader_wraps_resolution_errors():
+ resolution_error = ValueError("invalid selenium module metadata")
+
+ with (
+ patch.object(common_module, "find_spec", side_effect=resolution_error),
+ pytest.raises(MissingDependencyError, match="cannot safely resolve") as raised,
+ ):
+ common_module._load_selenium_dependencies()
+
+ assert raised.value.__cause__ is resolution_error
+
+
+def test_selenium_dependency_preflight_delegates_to_lazy_loader():
+ with patch.object(common_module, "_load_selenium_dependencies") as loader:
+ common_module.ensure_selenium_dependencies()
+
+ loader.assert_called_once_with()
+
+
+def test_local_chrome_dependency_loader_returns_runtime_classes():
+ chrome_service, chrome_driver_manager = (
+ common_module._load_local_chrome_dependencies()
+ )
+
+ assert chrome_service.__name__ == "Service"
+ assert chrome_driver_manager.__name__ == "ChromeDriverManager"
+
+
+def test_webdriver_cleanup_failure_preserves_browser_creation_error():
+ mock_webdriver = MagicMock()
+ mock_webdriver.ChromeOptions.return_value = MagicMock()
+ driver = MagicMock()
+ creation_error = WebDriverException("window setup failed")
+ driver.set_window_position.side_effect = creation_error
+ driver.quit.side_effect = RuntimeError("cleanup failed")
+ mock_webdriver.Remote.return_value = driver
+
+ with (
+ patch.object(
+ common_module,
+ "_load_selenium_dependencies",
+ return_value=(mock_webdriver, WebDriverException),
+ ),
+ pytest.raises(BrowserUnavailableError) as raised,
+ ):
+ create_webdriver(web_driver="http://selenium:4444")
+
+ assert raised.value.__cause__ is creation_error
+ driver.quit.assert_called_once_with()
+
+
def test_string_with_numbers():
assert has_numbers("abc123") is True
assert has_numbers("1a2b3c") is True
@@ -457,7 +665,7 @@ def test_string_with_whitespace_and_numbers():
def test_get_next_day_of_week(today_str, day_name, expected):
mock_today = datetime.strptime(today_str, "%Y-%m-%d")
with patch(
- "uk_bin_collection.common.datetime"
+ "uk_bin_collection.uk_bin_collection.common.datetime"
) as mock_datetime: # replace 'your_module' with the actual module name
mock_datetime.now.return_value = mock_today
mock_datetime.side_effect = lambda *args, **kw: datetime(*args, **kw)
@@ -485,7 +693,10 @@ def test_build_retry_session_custom_headers_and_methods():
def test_get_scraper_user_agent_uses_installed_version():
- with patch("uk_bin_collection.common._pkg_version", return_value="1.2.3"):
+ with patch(
+ "uk_bin_collection.uk_bin_collection.common._pkg_version",
+ return_value="1.2.3",
+ ):
result = get_scraper_user_agent()
assert result == (
"uk-bin-collection/1.2.3 (+https://github.com/robbrad/UKBinCollectionData)"
@@ -494,7 +705,8 @@ def test_get_scraper_user_agent_uses_installed_version():
def test_get_scraper_user_agent_falls_back_when_not_installed():
with patch(
- "uk_bin_collection.common._pkg_version", side_effect=PackageNotFoundError
+ "uk_bin_collection.uk_bin_collection.common._pkg_version",
+ side_effect=PackageNotFoundError,
):
result = get_scraper_user_agent()
assert result == (
@@ -504,7 +716,8 @@ def test_get_scraper_user_agent_falls_back_when_not_installed():
def test_get_scraper_user_agent_custom_fallback():
with patch(
- "uk_bin_collection.common._pkg_version", side_effect=PackageNotFoundError
+ "uk_bin_collection.uk_bin_collection.common._pkg_version",
+ side_effect=PackageNotFoundError,
):
result = get_scraper_user_agent(fallback_version="dev")
assert result == (
diff --git a/uk_bin_collection/tests/test_core_import_hardening.py b/uk_bin_collection/tests/test_core_import_hardening.py
new file mode 100644
index 0000000000..6b87f2c2e4
--- /dev/null
+++ b/uk_bin_collection/tests/test_core_import_hardening.py
@@ -0,0 +1,447 @@
+"""Focused regression tests for council and Selenium import boundaries."""
+
+from __future__ import annotations
+
+import ast
+import builtins
+import importlib
+import sys
+from pathlib import Path, PurePosixPath
+from types import ModuleType, SimpleNamespace
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from uk_bin_collection.uk_bin_collection import common, dependency_validation
+from uk_bin_collection.uk_bin_collection import collect_data
+from uk_bin_collection.uk_bin_collection.exceptions import (
+ DependencyShadowingError,
+ InvalidCouncilModuleError,
+ MissingDependencyError,
+)
+
+
+class _FakeDistribution:
+ def __init__(self, root: Path, *, include_websocket: bool = True) -> None:
+ self.root = root
+ self.files = (
+ [PurePosixPath("websocket/__init__.py")] if include_websocket else []
+ )
+
+ def locate_file(self, relative_path: PurePosixPath) -> Path:
+ return self.root.joinpath(*relative_path.parts)
+
+
+def test_websocket_client_accepts_file_owned_by_distribution(tmp_path: Path) -> None:
+ expected = tmp_path / "site-packages" / "websocket" / "__init__.py"
+ expected.parent.mkdir(parents=True)
+ expected.touch()
+ distribution = _FakeDistribution(tmp_path / "site-packages")
+
+ with (
+ patch.object(
+ dependency_validation.metadata,
+ "distribution",
+ return_value=distribution,
+ ),
+ patch.object(
+ dependency_validation.util,
+ "find_spec",
+ return_value=SimpleNamespace(origin=str(expected)),
+ ),
+ ):
+ result = dependency_validation.validate_websocket_client()
+
+ assert result == expected.resolve()
+
+
+def test_websocket_client_reports_stale_distribution_metadata(
+ tmp_path: Path,
+) -> None:
+ expected = tmp_path / "site-packages" / "websocket" / "__init__.py"
+ distribution = _FakeDistribution(tmp_path / "site-packages")
+
+ with (
+ patch.object(
+ dependency_validation.metadata,
+ "distribution",
+ return_value=distribution,
+ ),
+ patch.object(
+ dependency_validation.util,
+ "find_spec",
+ return_value=SimpleNamespace(origin=str(expected)),
+ ),
+ pytest.raises(MissingDependencyError, match="missing"),
+ ):
+ dependency_validation.validate_websocket_client()
+
+
+def test_websocket_client_rejects_shadow_package(tmp_path: Path) -> None:
+ distribution = _FakeDistribution(tmp_path / "site-packages")
+ shadow = tmp_path / "config" / "websocket" / "__init__.py"
+
+ with (
+ patch.object(
+ dependency_validation.metadata,
+ "distribution",
+ return_value=distribution,
+ ),
+ patch.object(
+ dependency_validation.util,
+ "find_spec",
+ return_value=SimpleNamespace(origin=str(shadow)),
+ ),
+ pytest.raises(DependencyShadowingError, match="shadowed"),
+ ):
+ dependency_validation.validate_websocket_client()
+
+
+def test_websocket_client_reports_missing_distribution() -> None:
+ with (
+ patch.object(
+ dependency_validation.metadata,
+ "distribution",
+ side_effect=dependency_validation.metadata.PackageNotFoundError,
+ ),
+ pytest.raises(MissingDependencyError, match="websocket-client"),
+ ):
+ dependency_validation.validate_websocket_client()
+
+
+def test_websocket_client_rejects_broken_loaded_module_metadata(
+ tmp_path: Path,
+) -> None:
+ distribution = _FakeDistribution(tmp_path / "site-packages")
+ with (
+ patch.object(
+ dependency_validation.metadata,
+ "distribution",
+ return_value=distribution,
+ ),
+ patch.object(
+ dependency_validation.util,
+ "find_spec",
+ side_effect=ValueError("websocket.__spec__ is None"),
+ ),
+ pytest.raises(DependencyShadowingError, match="cannot safely resolve"),
+ ):
+ dependency_validation.validate_websocket_client()
+
+
+def test_selenium_is_loaded_only_when_browser_is_requested() -> None:
+ with (
+ patch.object(common, "find_spec", return_value=None),
+ pytest.raises(MissingDependencyError, match="selenium"),
+ ):
+ common._load_selenium_dependencies()
+
+
+def test_browser_session_is_closed_when_initialisation_fails() -> None:
+ webdriver = MagicMock()
+ options = MagicMock()
+ webdriver.ChromeOptions.return_value = options
+ driver = MagicMock()
+ webdriver.Remote.return_value = driver
+
+ class FakeWebDriverError(Exception):
+ pass
+
+ driver.set_window_position.side_effect = FakeWebDriverError("window unavailable")
+ with (
+ patch.object(
+ common,
+ "_load_selenium_dependencies",
+ return_value=(webdriver, FakeWebDriverError),
+ ),
+ pytest.raises(common.BrowserUnavailableError),
+ ):
+ common.create_webdriver("http://selenium:4444")
+
+ driver.quit.assert_called_once_with()
+
+
+@pytest.mark.parametrize(
+ "module_name",
+ ["", "../SouthKestevenDistrictCouncil", "os.path", "name-with-dash", 123],
+)
+def test_council_loader_rejects_non_identifier_names(module_name: object) -> None:
+ with pytest.raises(InvalidCouncilModuleError):
+ collect_data.import_council_module(module_name) # type: ignore[arg-type]
+
+
+def test_council_loader_rejects_custom_source_path() -> None:
+ with pytest.raises(InvalidCouncilModuleError, match="Custom"):
+ collect_data.import_council_module("SouthKestevenDistrictCouncil", "../tmp")
+
+
+def test_council_loader_imports_only_from_councils_package() -> None:
+ expected_file = collect_data._council_file("ExampleCouncil")
+ council_module = SimpleNamespace(
+ CouncilClass=MagicMock(), __file__=str(expected_file)
+ )
+ with (
+ patch.object(
+ collect_data,
+ "registered_council_modules",
+ return_value=frozenset({"ExampleCouncil"}),
+ ),
+ patch.object(collect_data, "council_requires_selenium", return_value=False),
+ patch.object(
+ collect_data.import_util,
+ "find_spec",
+ return_value=SimpleNamespace(origin=str(expected_file)),
+ ),
+ patch.object(
+ collect_data.importlib,
+ "import_module",
+ return_value=council_module,
+ ) as import_module,
+ ):
+ result = collect_data.import_council_module("ExampleCouncil")
+
+ assert result is council_module
+ import_module.assert_called_once_with(
+ "uk_bin_collection.uk_bin_collection.councils.ExampleCouncil"
+ )
+
+
+def test_council_loader_rejects_shadowed_qualified_module(tmp_path: Path) -> None:
+ shadow = tmp_path / "config" / "ExampleCouncil.py"
+ with (
+ patch.object(
+ collect_data,
+ "registered_council_modules",
+ return_value=frozenset({"ExampleCouncil"}),
+ ),
+ patch.object(collect_data, "council_requires_selenium", return_value=False),
+ patch.object(
+ collect_data.import_util,
+ "find_spec",
+ return_value=SimpleNamespace(origin=str(shadow)),
+ ),
+ patch.object(collect_data.importlib, "import_module") as import_module,
+ pytest.raises(InvalidCouncilModuleError, match="outside"),
+ ):
+ collect_data.import_council_module("ExampleCouncil")
+
+ import_module.assert_not_called()
+
+
+def test_selenium_council_validates_websocket_before_module_import() -> None:
+ with (
+ patch.object(
+ collect_data,
+ "registered_council_modules",
+ return_value=frozenset({"ExampleCouncil"}),
+ ),
+ patch.object(collect_data, "council_requires_selenium", return_value=True),
+ patch.object(
+ collect_data,
+ "validate_websocket_client",
+ side_effect=DependencyShadowingError("shadowed"),
+ ) as validate,
+ patch.object(collect_data.import_util, "find_spec") as find_spec,
+ patch.object(collect_data.importlib, "import_module") as import_module,
+ pytest.raises(DependencyShadowingError, match="shadowed"),
+ ):
+ collect_data.import_council_module("ExampleCouncil")
+
+ validate.assert_called_once_with()
+ find_spec.assert_called_once_with("selenium")
+ import_module.assert_not_called()
+
+
+def test_selenium_council_reports_missing_optional_dependency_before_import() -> None:
+ with (
+ patch.object(
+ collect_data,
+ "registered_council_modules",
+ return_value=frozenset({"ExampleCouncil"}),
+ ),
+ patch.object(collect_data, "council_requires_selenium", return_value=True),
+ patch.object(collect_data.import_util, "find_spec", return_value=None),
+ patch.object(collect_data, "validate_websocket_client") as validate,
+ patch.object(collect_data.importlib, "import_module") as import_module,
+ pytest.raises(MissingDependencyError, match="selenium"),
+ ):
+ collect_data.import_council_module("ExampleCouncil")
+
+ validate.assert_not_called()
+ import_module.assert_not_called()
+
+
+def test_selenium_preflight_detects_eager_and_lazy_imports(
+ tmp_path: Path,
+) -> None:
+ eager_source = tmp_path / "EagerCouncil.py"
+ eager_source.write_text("from selenium import webdriver\n", encoding="utf-8")
+ lazy_source = tmp_path / "LazyCouncil.py"
+ lazy_source.write_text(
+ "def load_browser():\n from selenium import webdriver\n",
+ encoding="utf-8",
+ )
+
+ collect_data.council_requires_selenium.cache_clear()
+ with patch.object(collect_data, "_council_file", return_value=eager_source):
+ assert collect_data.council_requires_selenium("EagerCouncil") is True
+
+ collect_data.council_requires_selenium.cache_clear()
+ with patch.object(collect_data, "_council_file", return_value=lazy_source):
+ assert collect_data.council_requires_selenium("LazyCouncil") is True
+
+ collect_data.council_requires_selenium.cache_clear()
+
+
+def test_isle_of_wight_lazy_selenium_import_is_preflighted() -> None:
+ """A real lazy-import council cannot bypass dependency ownership checks."""
+ collect_data.council_requires_selenium.cache_clear()
+ assert collect_data.council_requires_selenium("IsleOfWightCouncil") is True
+ collect_data.council_requires_selenium.cache_clear()
+
+
+def test_council_loader_rejects_identifier_not_in_registry() -> None:
+ with (
+ patch.object(
+ collect_data,
+ "registered_council_modules",
+ return_value=frozenset({"KnownCouncil"}),
+ ),
+ pytest.raises(InvalidCouncilModuleError, match="installed registry"),
+ ):
+ collect_data.import_council_module("UnknownCouncil")
+
+
+def test_non_selenium_council_imports_with_shadow_websocket(monkeypatch) -> None:
+ """An unrelated websocket package must not break an HTTP-only council import."""
+ fake_websocket = ModuleType("websocket")
+ fake_websocket.__file__ = "/config/websocket/__init__.py"
+ monkeypatch.setitem(sys.modules, "websocket", fake_websocket)
+
+ real_import = builtins.__import__
+
+ def guarded_import(name, *args, **kwargs):
+ if name == "selenium" or name.startswith("selenium."):
+ raise AssertionError("HTTP-only council unexpectedly imported Selenium")
+ return real_import(name, *args, **kwargs)
+
+ monkeypatch.setattr(builtins, "__import__", guarded_import)
+ importlib.reload(common)
+ sys.modules.pop(
+ "uk_bin_collection.uk_bin_collection.councils.AberdeenCityCouncil", None
+ )
+ original_import_path = list(sys.path)
+
+ module = collect_data.import_council_module("AberdeenCityCouncil")
+
+ assert module.CouncilClass.__name__ == "CouncilClass"
+ assert sys.path == original_import_path
+
+
+def test_all_council_selenium_imports_are_lazy_and_prevalidated() -> None:
+ council_root = (
+ Path(__file__).resolve().parents[1] / "uk_bin_collection" / "councils"
+ )
+
+ for path in sorted(council_root.glob("*.py")):
+ tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
+ selenium_imports = [
+ node
+ for node in ast.walk(tree)
+ if (
+ isinstance(node, ast.Import)
+ and any(
+ alias.name == "selenium" or alias.name.startswith("selenium.")
+ for alias in node.names
+ )
+ )
+ or (
+ isinstance(node, ast.ImportFrom)
+ and node.module
+ and (node.module == "selenium" or node.module.startswith("selenium."))
+ )
+ ]
+ if not selenium_imports:
+ continue
+
+ assert not any(node in tree.body for node in selenium_imports), path.name
+ council_class = next(
+ node
+ for node in tree.body
+ if isinstance(node, ast.ClassDef) and node.name == "CouncilClass"
+ )
+ parse_data = next(
+ node
+ for node in council_class.body
+ if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
+ and node.name == "parse_data"
+ )
+ parse_imports = {
+ node for node in ast.walk(parse_data) if node in selenium_imports
+ }
+ if parse_imports != set(selenium_imports):
+ support_loader = next(
+ (
+ node
+ for node in council_class.body
+ if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
+ and node.name == "_load_selenium_support"
+ ),
+ None,
+ )
+ assert support_loader is not None, path.name
+ assert {
+ node for node in ast.walk(support_loader) if node in selenium_imports
+ } == set(selenium_imports), path.name
+ validation_calls = [
+ node
+ for node in ast.walk(support_loader)
+ if isinstance(node, ast.Call)
+ and isinstance(node.func, ast.Name)
+ and node.func.id == "validate_websocket_client"
+ ]
+ assert len(validation_calls) == 1, path.name
+ assert validation_calls[0].lineno < min(
+ node.lineno for node in selenium_imports
+ ), path.name
+ loader_calls = [
+ node
+ for node in ast.walk(parse_data)
+ if isinstance(node, ast.Call)
+ and isinstance(node.func, ast.Attribute)
+ and node.func.attr == "_load_selenium_support"
+ ]
+ assert len(loader_calls) == 1, path.name
+ continue
+
+ validation_calls = [
+ node
+ for node in ast.walk(parse_data)
+ if isinstance(node, ast.Call)
+ and isinstance(node.func, ast.Name)
+ and node.func.id == "ensure_selenium_dependencies"
+ ]
+ assert len(validation_calls) == 1, path.name
+ assert validation_calls[0].lineno < min(
+ node.lineno for node in selenium_imports
+ ), path.name
+
+
+def test_browser_only_optional_dependencies_are_not_imported_at_module_scope() -> None:
+ council_root = (
+ Path(__file__).resolve().parents[1] / "uk_bin_collection" / "councils"
+ )
+ browser_packages = {"selenium", "undetected_chromedriver", "webdriver_manager"}
+
+ for path in sorted(council_root.glob("*.py")):
+ tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
+ for node in tree.body:
+ if isinstance(node, ast.Import):
+ imported = {
+ alias.name.split(".", maxsplit=1)[0] for alias in node.names
+ }
+ elif isinstance(node, ast.ImportFrom) and node.module:
+ imported = {node.module.split(".", maxsplit=1)[0]}
+ else:
+ continue
+ assert imported.isdisjoint(browser_packages), path.name
diff --git a/uk_bin_collection/tests/test_council_log_redaction.py b/uk_bin_collection/tests/test_council_log_redaction.py
new file mode 100644
index 0000000000..8aff87aa12
--- /dev/null
+++ b/uk_bin_collection/tests/test_council_log_redaction.py
@@ -0,0 +1,332 @@
+"""Package-wide regression checks for sensitive diagnostic output."""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+CORE_DIRECTORY = Path(__file__).resolve().parents[1] / "uk_bin_collection"
+COUNCILS_DIRECTORY = CORE_DIRECTORY / "councils"
+HOME_ASSISTANT_DIRECTORY = (
+ Path(__file__).resolve().parents[2] / "custom_components" / "uk_bin_collection"
+)
+AUDITED_FILES = (
+ tuple(sorted(COUNCILS_DIRECTORY.glob("*.py")))
+ + tuple(sorted(CORE_DIRECTORY.glob("*.py")))
+ + tuple(sorted(HOME_ASSISTANT_DIRECTORY.glob("*.py")))
+)
+LOG_METHODS = {"critical", "debug", "error", "exception", "info", "warning"}
+SAFE_STATE_SUFFIXES = (
+ "_count",
+ "_found",
+ "_length",
+ "_matched",
+ "_present",
+ "_status",
+ "_status_code",
+ "_total",
+)
+SAFE_STATE_NAMES = {
+ "council_key",
+ "http_status",
+ "manual_refresh",
+ "response_status",
+ "status",
+ "status_code",
+ "timeout",
+ "update_interval",
+ "update_interval_hours",
+}
+SENSITIVE_NAME_PARTS = (
+ "address",
+ "bin",
+ "collection",
+ "data",
+ "date",
+ "html",
+ "markup",
+ "mapping",
+ "month",
+ "option",
+ "output",
+ "page_source",
+ "paon",
+ "postcode",
+ "result",
+ "raw",
+ "response",
+ "service",
+ "tag",
+ "text",
+ "uprn",
+ "usrn",
+ "waste",
+ "web_driver",
+)
+SENSITIVE_EXACT_NAMES = {
+ "body",
+ "item",
+ "key",
+ "page",
+ "payload",
+ "row",
+ "soup",
+ "url",
+ "value",
+}
+SENSITIVE_ATTRIBUTES = {
+ "_attribute_type",
+ "_bin_type",
+ "content",
+ "data",
+ "page_source",
+ "url",
+}
+
+
+def _diagnostic_sinks(tree: ast.AST):
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Call):
+ continue
+ if isinstance(node.func, ast.Name) and node.func.id == "print":
+ yield node
+ elif isinstance(node.func, ast.Attribute) and node.func.attr in LOG_METHODS:
+ yield node
+
+
+def _exception_names(tree: ast.AST) -> set[str]:
+ return {
+ node.name
+ for node in ast.walk(tree)
+ if isinstance(node, ast.ExceptHandler) and isinstance(node.name, str)
+ }
+
+
+def _root_name(node: ast.AST) -> str | None:
+ while isinstance(node, (ast.Attribute, ast.Call, ast.Subscript)):
+ if isinstance(node, ast.Attribute):
+ node = node.value
+ elif isinstance(node, ast.Call):
+ node = node.func
+ else:
+ node = node.value
+ return node.id if isinstance(node, ast.Name) else None
+
+
+def _is_sensitive_name(name: str) -> bool:
+ lowered = name.lower()
+ if lowered in SAFE_STATE_NAMES or lowered.endswith(SAFE_STATE_SUFFIXES):
+ return False
+ if lowered in SENSITIVE_EXACT_NAMES:
+ return True
+ if lowered.startswith("url_") or lowered.endswith("_url"):
+ return True
+ return any(part in lowered for part in SENSITIVE_NAME_PARTS)
+
+
+def _is_safe_summary(node: ast.AST, exception_names: set[str]) -> bool:
+ if (
+ isinstance(node, ast.Attribute)
+ and node.attr == "__name__"
+ and isinstance(node.value, ast.Call)
+ and isinstance(node.value.func, ast.Name)
+ and node.value.func.id == "type"
+ and len(node.value.args) == 1
+ and isinstance(node.value.args[0], ast.Name)
+ and node.value.args[0].id in exception_names
+ ):
+ return True
+ if (
+ isinstance(node, ast.Call)
+ and isinstance(node.func, ast.Name)
+ and node.func.id == "len"
+ and len(node.args) == 1
+ ):
+ return True
+ return isinstance(node, ast.Attribute) and node.attr == "status_code"
+
+
+def _expression_findings(
+ node: ast.AST,
+ exception_names: set[str],
+ tainted_names: set[str],
+) -> set[str]:
+ if _is_safe_summary(node, exception_names):
+ return set()
+
+ findings = set()
+ if isinstance(node, ast.Name):
+ if node.id.lower() in SAFE_STATE_NAMES or node.id.lower().endswith(
+ SAFE_STATE_SUFFIXES
+ ):
+ return findings
+ if node.id in exception_names:
+ findings.add(f"raw exception '{node.id}'")
+ if node.id in tainted_names or _is_sensitive_name(node.id):
+ findings.add(f"sensitive value '{node.id}'")
+ return findings
+
+ if isinstance(node, ast.Call):
+ if isinstance(node.func, ast.Attribute):
+ findings.update(
+ _expression_findings(node.func.value, exception_names, tainted_names)
+ )
+ for argument in node.args:
+ findings.update(
+ _expression_findings(argument, exception_names, tainted_names)
+ )
+ for keyword in node.keywords:
+ findings.update(
+ _expression_findings(keyword.value, exception_names, tainted_names)
+ )
+ return findings
+
+ if isinstance(node, ast.Constant) and isinstance(node.value, str):
+ if "http://" in node.value.lower() or "https://" in node.value.lower():
+ findings.add("literal URL")
+ return findings
+
+ if isinstance(node, ast.Attribute):
+ root = (_root_name(node) or "").lower()
+ if node.attr in SENSITIVE_ATTRIBUTES:
+ findings.add(f"sensitive attribute '.{node.attr}'")
+ elif node.attr in {"name", "summary"} and any(
+ marker in root for marker in ("calendar", "entity", "event")
+ ):
+ findings.add(f"sensitive attribute '.{node.attr}'")
+ elif node.attr == "text" and any(
+ marker in root
+ for marker in ("address", "dropdown", "html", "option", "page", "response")
+ ):
+ findings.add("sensitive attribute '.text'")
+
+ for child in ast.iter_child_nodes(node):
+ findings.update(_expression_findings(child, exception_names, tainted_names))
+ return findings
+
+
+def _assignment_value(node: ast.AST) -> ast.AST | None:
+ if isinstance(node, (ast.Assign, ast.AnnAssign, ast.NamedExpr)):
+ return node.value
+ return None
+
+
+def _assignment_targets(node: ast.AST):
+ targets = []
+ if isinstance(node, ast.Assign):
+ targets = node.targets
+ elif isinstance(node, (ast.AnnAssign, ast.NamedExpr)):
+ targets = [node.target]
+ pending = list(targets)
+ while pending:
+ target = pending.pop()
+ if isinstance(target, ast.Name):
+ yield target.id
+ elif isinstance(target, (ast.List, ast.Tuple)):
+ pending.extend(target.elts)
+ elif isinstance(target, ast.Starred):
+ pending.append(target.value)
+
+
+def _tainted_names(tree: ast.AST, exception_names: set[str]) -> set[str]:
+ tainted = {
+ node.id
+ for node in ast.walk(tree)
+ if isinstance(node, ast.Name) and _is_sensitive_name(node.id)
+ }
+ assignments = [
+ node for node in ast.walk(tree) if _assignment_value(node) is not None
+ ]
+ changed = True
+ while changed:
+ changed = False
+ for assignment in assignments:
+ value = _assignment_value(assignment)
+ if value is None or not _expression_findings(
+ value, exception_names, tainted
+ ):
+ continue
+ for target in _assignment_targets(assignment):
+ if (
+ target.lower() not in SAFE_STATE_NAMES
+ and not target.lower().endswith(SAFE_STATE_SUFFIXES)
+ and target not in tainted
+ ):
+ tainted.add(target)
+ changed = True
+ return tainted
+
+
+def _tree_violations(tree: ast.AST, source_name: str):
+ exception_names = _exception_names(tree)
+ tainted_names = _tainted_names(tree, exception_names)
+ violations = []
+ for sink in _diagnostic_sinks(tree):
+ findings = _expression_findings(sink, exception_names, tainted_names)
+ if any(
+ keyword.arg in {"exc_info", "stack_info"}
+ and not (
+ isinstance(keyword.value, ast.Constant)
+ and keyword.value.value in {False, None}
+ )
+ for keyword in sink.keywords
+ ) or (
+ isinstance(sink.func, ast.Attribute)
+ and sink.func.attr == "exception"
+ and (_root_name(sink.func.value) or "").lower()
+ in {"logger", "logging", "_logger"}
+ ):
+ findings.add("raw traceback output")
+ if findings:
+ violations.append((source_name, sink.lineno, sorted(findings)))
+ return violations
+
+
+def test_sensitive_values_do_not_flow_to_diagnostics():
+ """Normal print and logger diagnostics must contain summaries only."""
+ violations = []
+ for source_path in AUDITED_FILES:
+ tree = ast.parse(
+ source_path.read_text(encoding="utf-8"), filename=str(source_path)
+ )
+ violations.extend(_tree_violations(tree, source_path.name))
+
+ assert violations == [], violations
+
+
+def test_gate_detects_sensitive_and_allows_summary_diagnostics():
+ """Exercise each protected source class and each supported safe summary."""
+ unsafe_samples = (
+ "try:\n pass\nexcept Exception as exc:\n print(f'{exc}')\n",
+ "print(driver.page_source)\n",
+ "print(html)\n",
+ "logger.error('URL: %s', response.url)\n",
+ "print('https://council.invalid/private')\n",
+ "print(postcode, paon, uprn, usrn)\n",
+ "selected = address_options[0].text\nlogger.info('%s', selected)\n",
+ "print(dict_data, bin_data, collection, raw_text)\n",
+ "logger.debug('artifact: %s', json.dumps(data))\n",
+ "print(bin_type, collection_date, waste_service)\n",
+ "logger.warning('%s', icon_color_mapping)\n",
+ "logger.debug('%s', config_entry.data)\n",
+ "logger.info('%s', event.summary)\n",
+ "print(raw_name, month_txt, start_tag, key, value, payload, row)\n",
+ "logger.info('%s', entity._bin_type)\n",
+ "logger.exception('collection failed')\n",
+ "try:\n pass\nexcept Exception:\n logger.error('failed', exc_info=True)\n",
+ )
+ safe_samples = (
+ "try:\n pass\nexcept Exception as exc:\n print(type(exc).__name__)\n",
+ "print(f'Found {len(address_options)} address options')\n",
+ "logger.info('HTTP %s', response.status_code)\n",
+ "print('Address selected successfully')\n",
+ "logger.error('failed', exc_info=False)\n",
+ )
+
+ for index, source in enumerate(unsafe_samples):
+ tree = ast.parse(source, filename=f"unsafe_{index}.py")
+ assert _tree_violations(tree, f"unsafe_{index}.py"), source
+
+ for index, source in enumerate(safe_samples):
+ tree = ast.parse(source, filename=f"safe_{index}.py")
+ assert _tree_violations(tree, f"safe_{index}.py") == [], source
diff --git a/uk_bin_collection/uk_bin_collection/collect_data.py b/uk_bin_collection/uk_bin_collection/collect_data.py
index 29c3649691..27f9a4e5a1 100755
--- a/uk_bin_collection/uk_bin_collection/collect_data.py
+++ b/uk_bin_collection/uk_bin_collection/collect_data.py
@@ -1,22 +1,148 @@
import argparse
+import ast
import importlib
+import logging
import os
+import re
import sys
-import logging
+from functools import lru_cache
+from importlib import util as import_util
+from pathlib import Path
+
from uk_bin_collection.uk_bin_collection.get_bin_data import (
setup_logging,
LOGGING_CONFIG,
)
+from uk_bin_collection.uk_bin_collection.exceptions import (
+ InvalidCouncilModuleError,
+ MissingDependencyError,
+)
+from uk_bin_collection.uk_bin_collection.dependency_validation import (
+ validate_websocket_client,
+)
_LOGGER = logging.getLogger(__name__)
+_COUNCIL_NAME_PATTERN = re.compile(r"^[A-Za-z][A-Za-z0-9_]*$")
+_COUNCILS_PACKAGE = "uk_bin_collection.uk_bin_collection.councils"
+_COUNCILS_DIRECTORY = Path(__file__).resolve().parent / "councils"
+
+
+def _normalised_path(path: os.PathLike[str] | str) -> str:
+ """Return a real, case-normalised path for import-origin comparisons."""
+ return os.path.normcase(os.path.realpath(os.fspath(path)))
+
+
+def _council_file(module_name: str) -> Path:
+ """Return the trusted source location for a registered council."""
+ return _COUNCILS_DIRECTORY / f"{module_name}.py"
+
+
+@lru_cache(maxsize=None)
+def council_requires_selenium(module_name: str) -> bool:
+ """Return whether a trusted council imports Selenium at any runtime scope.
+
+ Older adapters use both eager and function-local imports. Inspecting the full
+ trusted source before import lets the loader validate Selenium's
+ collision-prone ``websocket`` dependency before either form can execute.
+ """
+ council_file = _council_file(module_name)
+ try:
+ syntax_tree = ast.parse(
+ council_file.read_text(encoding="utf-8"), filename=str(council_file)
+ )
+ except (OSError, SyntaxError, UnicodeError) as exc:
+ raise InvalidCouncilModuleError(
+ f"Council module {module_name!r} cannot be inspected safely."
+ ) from exc
+
+ for node in ast.walk(syntax_tree):
+ if isinstance(node, ast.Import) and any(
+ imported.name == "selenium" or imported.name.startswith("selenium.")
+ for imported in node.names
+ ):
+ return True
+ if (
+ isinstance(node, ast.ImportFrom)
+ and node.module
+ and (node.module == "selenium" or node.module.startswith("selenium."))
+ ):
+ return True
+ return False
+
+
+@lru_cache(maxsize=1)
+def registered_council_modules() -> frozenset[str]:
+ """Return council module names shipped in the installed core package."""
+ return frozenset(
+ council_file.stem
+ for council_file in _COUNCILS_DIRECTORY.iterdir()
+ if council_file.is_file()
+ and council_file.suffix == ".py"
+ and _COUNCIL_NAME_PATTERN.fullmatch(council_file.stem)
+ )
+
-def import_council_module(module_name, src_path="councils"):
- """Dynamically import the council processor module."""
- module_path = os.path.realpath(os.path.join(os.path.dirname(__file__), src_path))
- if module_path not in sys.path:
- sys.path.append(module_path)
- return importlib.import_module(module_name)
+def import_council_module(module_name: str, src_path: str = "councils"):
+ """Import an allowlisted council by its fully qualified package name."""
+ if src_path != "councils":
+ raise InvalidCouncilModuleError(
+ "Custom council import paths are not supported."
+ )
+ if not isinstance(module_name, str) or not _COUNCIL_NAME_PATTERN.fullmatch(
+ module_name
+ ):
+ raise InvalidCouncilModuleError(
+ "Council names must be simple Python identifiers."
+ )
+
+ if module_name not in registered_council_modules():
+ raise InvalidCouncilModuleError(
+ f"Council module {module_name!r} is not present in the installed registry."
+ )
+
+ if council_requires_selenium(module_name):
+ try:
+ selenium_spec = import_util.find_spec("selenium")
+ except (AttributeError, ImportError, ValueError) as exc:
+ raise MissingDependencyError(
+ "Python cannot safely resolve the optional 'selenium' dependency."
+ ) from exc
+ if selenium_spec is None:
+ raise MissingDependencyError(
+ "The optional dependency 'selenium' is required for this council."
+ )
+ validate_websocket_client()
+
+ qualified_name = f"{_COUNCILS_PACKAGE}.{module_name}"
+ expected_file = _normalised_path(_council_file(module_name))
+ try:
+ specification = import_util.find_spec(qualified_name)
+ except (AttributeError, ImportError, ValueError) as exc:
+ raise InvalidCouncilModuleError(
+ f"Council module {module_name!r} cannot be resolved safely."
+ ) from exc
+
+ if (
+ specification is None
+ or specification.origin is None
+ or _normalised_path(specification.origin) != expected_file
+ ):
+ raise InvalidCouncilModuleError(
+ f"Council module {module_name!r} resolves outside the installed registry."
+ )
+
+ council_module = importlib.import_module(qualified_name)
+ module_file = getattr(council_module, "__file__", None)
+ if module_file is None or _normalised_path(module_file) != expected_file:
+ raise InvalidCouncilModuleError(
+ f"Council module {module_name!r} loaded from an unexpected location."
+ )
+ if not hasattr(council_module, "CouncilClass"):
+ raise InvalidCouncilModuleError(
+ f"Council module {module_name!r} does not expose CouncilClass."
+ )
+ return council_module
class UKBinCollectionApp:
@@ -55,6 +181,9 @@ def setup_arg_parser(self):
self.parser.add_argument(
"-u", "--uprn", type=str, help="UPRN to parse", required=False
)
+ self.parser.add_argument(
+ "-us", "--usrn", type=str, help="USRN to parse", required=False
+ )
self.parser.add_argument(
"-w",
"--web_driver",
@@ -70,6 +199,14 @@ def setup_arg_parser(self):
help="Directory for council-specific debug artifacts when a live scrape fails",
required=False,
)
+ self.parser.add_argument(
+ "--user-agent",
+ "--user_agent",
+ dest="user_agent",
+ type=str,
+ help="Optional HTTP/browser user agent for council requests",
+ required=False,
+ )
self.parser.add_argument(
"--headless",
dest="headless",
@@ -112,9 +249,11 @@ def run(self):
postcode=self.parsed_args.postcode,
paon=self.parsed_args.number,
uprn=self.parsed_args.uprn,
+ usrn=self.parsed_args.usrn,
skip_get_url=self.parsed_args.skip_get_url,
web_driver=self.parsed_args.web_driver,
artifact_dir=self.parsed_args.artifact_dir,
+ user_agent=self.parsed_args.user_agent,
headless=self.parsed_args.headless,
local_browser=self.parsed_args.local_browser,
dev_mode=self.parsed_args.dev_mode,
diff --git a/uk_bin_collection/uk_bin_collection/common.py b/uk_bin_collection/uk_bin_collection/common.py
index c6eb844596..ccbcca244b 100644
--- a/uk_bin_collection/uk_bin_collection/common.py
+++ b/uk_bin_collection/uk_bin_collection/common.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
import calendar
import json
import os
@@ -5,17 +7,28 @@
from datetime import datetime, timedelta
from enum import Enum
from importlib.metadata import PackageNotFoundError, version as _pkg_version
+from importlib.util import find_spec
+from typing import TYPE_CHECKING, Any
+from urllib.parse import urlsplit
import holidays
import pandas as pd
import requests
from dateutil.parser import parse
from requests.adapters import HTTPAdapter
-from selenium import webdriver
-from selenium.webdriver.chrome.service import Service as ChromeService
from urllib3.exceptions import MaxRetryError
from urllib3.util.retry import Retry
-from webdriver_manager.chrome import ChromeDriverManager
+
+from uk_bin_collection.uk_bin_collection.dependency_validation import (
+ validate_websocket_client,
+)
+from uk_bin_collection.uk_bin_collection.exceptions import (
+ BrowserUnavailableError,
+ MissingDependencyError,
+)
+
+if TYPE_CHECKING:
+ from selenium.webdriver.remote.webdriver import WebDriver
date_format = "%d/%m/%Y"
days_of_week = {
@@ -42,13 +55,20 @@ def check_postcode(postcode: str):
:param postcode: Postcode to parse
"""
postcode_api_url = "https://api.postcodes.io/postcodes/"
- postcode_api_response = requests.get(f"{postcode_api_url}{postcode}")
+ postcode_api_response = requests.get(
+ f"{postcode_api_url}{postcode}", timeout=(3.05, 10)
+ )
if postcode_api_response.status_code != 200:
- val_error = json.loads(postcode_api_response.text)
- raise ValueError(
- f"Exception: {val_error['error']} Status: {val_error['status']}"
+ try:
+ val_error = postcode_api_response.json()
+ except (ValueError, TypeError):
+ val_error = {}
+ error = val_error.get(
+ "error", "Postcode validation service rejected the request"
)
+ status = val_error.get("status", postcode_api_response.status_code)
+ raise ValueError(f"Exception: {error} Status: {status}")
return True
@@ -62,7 +82,7 @@ def check_paon(paon: str):
raise ValueError("Invalid house number")
return True
except Exception as ex:
- print(f"Exception encountered: {ex}")
+ print(f"Exception encountered: {type(ex).__name__}")
print("Please check the provided house number.")
exit(1)
@@ -77,7 +97,7 @@ def check_uprn(uprn: str):
raise ValueError("Invalid UPRN")
return True
except Exception as ex:
- print(f"Exception encountered: {ex}")
+ print(f"Exception encountered: {type(ex).__name__}")
print("Please check the provided UPRN.")
@@ -91,7 +111,7 @@ def check_usrn(usrn: str):
raise ValueError("Invalid USRN")
return True
except Exception as ex:
- print(f"Exception encountered: {ex}")
+ print(f"Exception encountered: {type(ex).__name__}")
print("Please check the provided USRN.")
@@ -271,7 +291,7 @@ def update_input_json(council: str, url: str, input_file_path: str, **kwargs):
save_data(input_file_path, data)
except IOError as e:
- print(f"Error updating the JSON file: {e}")
+ print(f"Error updating the JSON file: {type(e).__name__}")
except json.JSONDecodeError:
print("Failed to decode JSON, check the integrity of the input file.")
@@ -366,7 +386,8 @@ def create_webdriver(
headless: bool = True,
user_agent: str = None,
session_name: str = None,
-) -> webdriver.Chrome:
+ command_timeout: float | None = None,
+) -> WebDriver:
"""
Create and return a Chrome WebDriver configured for optional headless operation.
@@ -374,9 +395,14 @@ def create_webdriver(
:param headless: Whether to run the browser in headless mode.
:param user_agent: Optional custom user agent string.
:param session_name: Optional custom session name string.
+ :param command_timeout: Optional bound for remote WebDriver HTTP/WebSocket calls.
:return: An instance of a Chrome WebDriver.
- :raises WebDriverException: If the WebDriver cannot be created.
+ :raises DependencyError: If Selenium's dependencies cannot be imported safely.
+ :raises BrowserUnavailableError: If the WebDriver cannot be created.
"""
+ webdriver, WebDriverException = _load_selenium_dependencies()
+ if web_driver:
+ web_driver = _validate_remote_webdriver_url(web_driver)
options = webdriver.ChromeOptions()
if headless:
options.add_argument("--headless")
@@ -391,10 +417,20 @@ def create_webdriver(
if session_name and web_driver:
options.set_capability("se:name", session_name)
+ driver = None
try:
if web_driver:
- driver = webdriver.Remote(command_executor=web_driver, options=options)
+ remote_kwargs = {
+ "command_executor": web_driver,
+ "options": options,
+ }
+ if command_timeout is not None:
+ remote_kwargs["client_config"] = _build_remote_client_config(
+ web_driver, command_timeout
+ )
+ driver = webdriver.Remote(**remote_kwargs)
else:
+ ChromeService, ChromeDriverManager = _load_local_chrome_dependencies()
driver = webdriver.Chrome(
service=ChromeService(ChromeDriverManager().install()), options=options
)
@@ -403,6 +439,110 @@ def create_webdriver(
driver.set_window_position(0, 0)
return driver
- except MaxRetryError as e:
- print(f"Failed to create WebDriver: {e}")
- raise
+ except (
+ MaxRetryError,
+ WebDriverException,
+ requests.exceptions.RequestException,
+ OSError,
+ ) as exc:
+ if driver is not None:
+ try:
+ driver.quit()
+ except Exception:
+ # Preserve the browser-creation failure as the actionable cause.
+ pass
+ raise BrowserUnavailableError(
+ "Unable to create or reach the configured Selenium WebDriver."
+ ) from exc
+
+
+def _validate_remote_webdriver_url(web_driver: str) -> str:
+ """Return a normalized HTTP(S) WebDriver URL or fail with a typed error."""
+ normalized = str(web_driver).strip().rstrip("/")
+ try:
+ parsed = urlsplit(normalized)
+ # Accessing hostname/port performs urllib's bracket and port validation.
+ hostname = parsed.hostname
+ parsed.port
+ except (TypeError, ValueError) as exc:
+ raise BrowserUnavailableError(
+ "The configured Selenium WebDriver URL is invalid."
+ ) from exc
+
+ if (
+ parsed.scheme.casefold() not in {"http", "https"}
+ or not hostname
+ or any(character.isspace() for character in normalized)
+ ):
+ raise BrowserUnavailableError(
+ "The configured Selenium WebDriver URL must use HTTP or HTTPS and "
+ "include a host."
+ )
+ return normalized
+
+
+def _build_remote_client_config(web_driver: str, timeout: float):
+ """Build Selenium's bounded remote connection config lazily."""
+ validate_websocket_client()
+ try:
+ from selenium.webdriver.remote.client_config import ClientConfig
+ except ImportError as exc:
+ raise MissingDependencyError(
+ "Selenium's remote client configuration support is unavailable."
+ ) from exc
+
+ timeout_seconds = max(1, int(timeout))
+ return ClientConfig(
+ remote_server_addr=web_driver,
+ timeout=timeout_seconds,
+ websocket_timeout=float(timeout_seconds),
+ )
+
+
+def _load_selenium_dependencies() -> tuple[Any, type[Exception]]:
+ """Import Selenium only when a caller actually needs a browser."""
+ try:
+ selenium_spec = find_spec("selenium")
+ except (AttributeError, ImportError, ValueError) as exc:
+ raise MissingDependencyError(
+ "Python cannot safely resolve the optional 'selenium' dependency."
+ ) from exc
+
+ if selenium_spec is None:
+ raise MissingDependencyError(
+ "The optional dependency 'selenium' is required for this council."
+ )
+
+ # Selenium's remote driver imports ``websocket``. Validate its ownership before
+ # allowing that import to execute potentially conflicting top-level code.
+ validate_websocket_client()
+
+ try:
+ from selenium import webdriver
+ from selenium.common.exceptions import WebDriverException
+ except ImportError as exc:
+ # Recheck the most security-sensitive transitive dependency in case import
+ # state changed between validation and Selenium's own import.
+ validate_websocket_client()
+ raise MissingDependencyError("Selenium is installed incompletely.") from exc
+
+ return webdriver, WebDriverException
+
+
+def ensure_selenium_dependencies() -> None:
+ """Validate and import Selenium before a council performs lazy imports."""
+ _load_selenium_dependencies()
+
+
+def _load_local_chrome_dependencies() -> tuple[Any, Any]:
+ """Import dependencies needed only when launching a local Chrome driver."""
+ try:
+ from selenium.webdriver.chrome.service import Service as ChromeService
+ from webdriver_manager.chrome import ChromeDriverManager
+ except ImportError as exc:
+ raise MissingDependencyError(
+ "The optional dependency 'webdriver-manager' is required for a local "
+ "browser, but is not required for a remote WebDriver."
+ ) from exc
+
+ return ChromeService, ChromeDriverManager
diff --git a/uk_bin_collection/uk_bin_collection/councils/AdurAndWorthingCouncils.py b/uk_bin_collection/uk_bin_collection/councils/AdurAndWorthingCouncils.py
index 37616efe45..ed752cb27e 100644
--- a/uk_bin_collection/uk_bin_collection/councils/AdurAndWorthingCouncils.py
+++ b/uk_bin_collection/uk_bin_collection/councils/AdurAndWorthingCouncils.py
@@ -78,13 +78,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
bin_date = bin_date.replace(year=current_year + 1)
collections.append((bin_type, bin_date))
- print(
- f"Successfully parsed date for {bin_type}: {bin_date}"
- )
+ print("Collection date parsed successfully")
except ValueError as e:
print(
- f"Failed to parse date '{date_str}' for {bin_type}: {e}"
+ "Failed to parse a collection date:",
+ type(e).__name__,
)
continue
@@ -92,7 +91,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
break
except Exception as e:
- print(f"Error processing bin row: {e}")
+ print(f"Error processing bin row: {type(e).__name__}")
continue
if not collections:
diff --git a/uk_bin_collection/uk_bin_collection/councils/AngusCouncil.py b/uk_bin_collection/uk_bin_collection/councils/AngusCouncil.py
index 95d1135523..dd49ee5e94 100644
--- a/uk_bin_collection/uk_bin_collection/councils/AngusCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/AngusCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import re
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
-from selenium.common.exceptions import TimeoutException, NoSuchElementException
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -13,6 +11,17 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, NoSuchElementException, Select, TimeoutException, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+ from selenium.common.exceptions import TimeoutException, NoSuchElementException
+
driver = None
try:
user_postcode = kwargs.get("postcode")
@@ -59,7 +68,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except NoSuchElementException:
# Print page source for debugging
- print("Page source:", driver.page_source[:1000])
+ print("Page source omitted from diagnostics.")
raise ValueError("Could not find postcode input field")
postcode_input.clear()
postcode_input.send_keys(user_postcode)
@@ -170,7 +179,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
return bin_data
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/ArgyllandButeCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ArgyllandButeCouncil.py
index 0ae0d16bd8..f5b7d14935 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ArgyllandButeCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ArgyllandButeCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,7 +19,7 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
"""
Fetch upcoming bin collection types and dates for an Argyll and Bute address.
-
+
Parameters:
page (str): Unused; the function always targets the Argyll and Bute bin collection page.
**kwargs:
@@ -28,12 +27,22 @@ def parse_data(self, page: str, **kwargs) -> dict:
postcode (str): The postcode to search.
web_driver: Optional webdriver configuration or path passed to create_webdriver.
headless (bool): Whether to run the browser in headless mode.
-
+
Returns:
dict: A dictionary with a "bins" key containing a list of collections. Each collection is a dict with:
- "type" (str): Human-readable bin type (e.g., "General waste").
- "collectionDate" (str): Collection date formatted according to the module's `date_format`.
"""
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
page = "https://www.argyll-bute.gov.uk/rubbish-and-recycling/household-waste/bin-collection"
@@ -146,11 +155,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
# This block ensures that the driver is closed regardless of an exception
if driver:
driver.quit()
- return bin_data
\ No newline at end of file
+ return bin_data
diff --git a/uk_bin_collection/uk_bin_collection/councils/ArunCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ArunCouncil.py
index eabc5fd06f..7e81acf892 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ArunCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ArunCouncil.py
@@ -1,10 +1,8 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
-from selenium import webdriver
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +17,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait, webdriver
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium import webdriver
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
try:
# Make a BS4 object
data = {"bins": []}
@@ -88,7 +97,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/AshfieldDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/AshfieldDistrictCouncil.py
index 3e668ca263..7c725d4297 100644
--- a/uk_bin_collection/uk_bin_collection/councils/AshfieldDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/AshfieldDistrictCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium import webdriver
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +17,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait, webdriver
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium import webdriver
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
# Get and check UPRN
@@ -105,7 +114,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
bindata["bins"].append(dict_data)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/AshfordBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/AshfordBoroughCouncil.py
index 6af1164536..b2afd3cdb9 100644
--- a/uk_bin_collection/uk_bin_collection/councils/AshfordBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/AshfordBoroughCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
from datetime import datetime
import requests
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +17,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
# Get and check UPRN
@@ -133,7 +142,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/BaberghDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BaberghDistrictCouncil.py
index 7f79634f06..412dd17591 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BaberghDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BaberghDistrictCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import re
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.common.exceptions import TimeoutException
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -39,6 +37,17 @@ def parse_data(self, page: str, **kwargs) -> dict:
Returns:
A dict with a ``bins`` list of collection type/date entries.
"""
+ global By, EC, Select, TimeoutException, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.common.exceptions import TimeoutException
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
web_driver = kwargs.get("web_driver")
@@ -195,7 +204,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/BarkingDagenham.py b/uk_bin_collection/uk_bin_collection/councils/BarkingDagenham.py
index 0a6caeb990..ab2c3e04f7 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BarkingDagenham.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BarkingDagenham.py
@@ -1,15 +1,11 @@
+from __future__ import annotations
+
# This script pulls bin collection data from Barking and Dagenham Council
# Example URL: https://www.lbbd.gov.uk/rubbish-recycling/household-bin-collection/check-your-bin-collection-days
import time
from bs4 import BeautifulSoup
from dateutil.parser import parse
-from selenium.common.exceptions import NoSuchElementException, TimeoutException
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -18,6 +14,19 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, NoSuchElementException, Select, TimeoutException, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.common.exceptions import NoSuchElementException, TimeoutException
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -28,16 +37,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
headless = kwargs.get("headless")
url = kwargs.get("url")
- print(
- f"Starting parse_data with parameters: postcode={postcode}, paon={user_paon}"
- )
- print(
- f"Creating webdriver with: web_driver={web_driver}, headless={headless}"
- )
+ print("Starting parse_data with configured address parameters")
+ print(f"Creating configured webdriver (headless={headless})")
user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
driver = create_webdriver(web_driver, headless, user_agent, __name__)
- print(f"Navigating to URL: {url}")
+ print("Navigating to the configured council page")
driver.get(url)
print("Successfully loaded the page")
@@ -82,7 +87,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
post_code_input.clear()
post_code_input.send_keys(postcode)
- print(f"Entered postcode: {postcode}")
+ print("Postcode entered")
driver.switch_to.active_element.send_keys(Keys.TAB + Keys.ENTER)
print("Pressed ENTER on Search button")
@@ -152,13 +157,13 @@ def parse_data(self, page: str, **kwargs) -> dict:
"collectionDate": bin_date,
}
data["bins"].append(dict_data)
- print(f"Successfully added collection: {dict_data}")
+ print("Collection added successfully")
except Exception as e:
- print(f"Error processing item: {e}")
+ print(f"Error processing item ({type(e).__name__})")
continue
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred ({type(e).__name__})")
raise
finally:
print("Cleaning up webdriver...")
diff --git a/uk_bin_collection/uk_bin_collection/councils/BarnetCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BarnetCouncil.py
index 311acc726c..2058f549a5 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BarnetCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BarnetCouncil.py
@@ -1,11 +1,10 @@
+from __future__ import annotations
+
import re
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +18,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
user_postcode = kwargs.get("postcode")
@@ -132,9 +141,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
date_text = date_el.get_text(strip=True).strip()
# Remove ordinal suffixes (1st, 2nd, 3rd, 4th, etc.)
- date_text = re.sub(
- r"(\d+)(st|nd|rd|th)", r"\1", date_text
- )
+ date_text = re.sub(r"(\d+)(st|nd|rd|th)", r"\1", date_text)
# Parse "Monday, 6 April"
try:
@@ -172,7 +179,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
return bin_data
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/BasildonCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BasildonCouncil.py
index fb78cafd00..999564cd5f 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BasildonCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BasildonCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import requests
import json
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium import webdriver
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support.ui import WebDriverWait
-from selenium.webdriver.support import expected_conditions as EC
from uk_bin_collection.uk_bin_collection.common import (
check_uprn,
date_format as DATE_FORMAT,
@@ -19,30 +17,43 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait, webdriver
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium import webdriver
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support.ui import WebDriverWait
+ from selenium.webdriver.support import expected_conditions as EC
+
uprn = kwargs.get("uprn")
check_uprn(uprn)
-
+
# Try API first
try:
return self._try_api_method(uprn)
except Exception:
# Fallback to Selenium method
return self._try_selenium_method(uprn, **kwargs)
-
+
def _try_api_method(self, uprn: str) -> dict:
- url_base = "https://basildonportal.azurewebsites.net/api/getPropertyRefuseInformation"
+ url_base = (
+ "https://basildonportal.azurewebsites.net/api/getPropertyRefuseInformation"
+ )
payload = {"uprn": uprn}
headers = {"Content-Type": "application/json"}
-
+
response = requests.post(url_base, data=json.dumps(payload), headers=headers)
-
+
if response.status_code != 200:
raise Exception(f"API failed with status {response.status_code}")
-
+
data = response.json()
bins = []
available_services = data.get("refuse", {}).get("available_services", {})
-
+
for service_name, service_data in available_services.items():
match service_data["container"]:
case "Green Wheelie Bin":
@@ -56,55 +67,69 @@ def _try_api_method(self, uprn: str) -> dict:
type_descr = service_data.get("name", "Unknown Service")
case _:
type_descr = service_data.get("container", "Unknown Container")
-
+
date_str = service_data.get("current_collection_date")
if date_str:
try:
date_obj = datetime.strptime(date_str, "%Y-%m-%d")
formatted_date = date_obj.strftime(DATE_FORMAT)
- bins.append({
- "type": type_descr,
- "collectionDate": formatted_date,
- })
+ bins.append(
+ {
+ "type": type_descr,
+ "collectionDate": formatted_date,
+ }
+ )
except ValueError:
pass # Skip bins with invalid dates
-
+
return {"bins": bins}
-
+
def _try_selenium_method(self, uprn: str, **kwargs) -> dict:
driver = kwargs.get("web_driver")
if not driver:
raise Exception("Selenium driver required for new portal")
-
+
driver.get("https://mybasildon.powerappsportals.com/check/where_i_live/")
-
+
# Wait for and find postcode input
wait = WebDriverWait(driver, 10)
postcode_input = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "input[type='text']"))
)
-
+
# Get postcode from UPRN lookup (simplified - would need actual lookup)
postcode_input.send_keys("SS14 1EY") # Default postcode for testing
-
+
# Submit form
- submit_btn = driver.find_element(By.CSS_SELECTOR, "button[type='submit'], input[type='submit']")
+ submit_btn = driver.find_element(
+ By.CSS_SELECTOR, "button[type='submit'], input[type='submit']"
+ )
submit_btn.click()
-
+
# Wait for results and parse
- wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".collection-info, .bin-info")))
-
+ wait.until(
+ EC.presence_of_element_located(
+ (By.CSS_SELECTOR, ".collection-info, .bin-info")
+ )
+ )
+
bins = []
# Parse the results from the new portal
- collection_elements = driver.find_elements(By.CSS_SELECTOR, ".collection-info, .bin-info")
-
+ collection_elements = driver.find_elements(
+ By.CSS_SELECTOR, ".collection-info, .bin-info"
+ )
+
for element in collection_elements:
bin_type = element.find_element(By.CSS_SELECTOR, ".bin-type").text
- collection_date = element.find_element(By.CSS_SELECTOR, ".collection-date").text
-
- bins.append({
- "type": bin_type,
- "collectionDate": collection_date,
- })
-
+ collection_date = element.find_element(
+ By.CSS_SELECTOR, ".collection-date"
+ ).text
+
+ bins.append(
+ {
+ "type": bin_type,
+ "collectionDate": collection_date,
+ }
+ )
+
return {"bins": bins}
diff --git a/uk_bin_collection/uk_bin_collection/councils/BexleyCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BexleyCouncil.py
index f7355296c0..3e8fffa1ee 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BexleyCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BexleyCouncil.py
@@ -71,10 +71,15 @@ def parse_data(self, page: str, **kwargs) -> dict:
next_collection = dd.get_text(strip=True)
try:
- if "collected today" in next_collection.lower() or "could not be collected today" in next_collection.lower():
+ if (
+ "collected today" in next_collection.lower()
+ or "could not be collected today" in next_collection.lower()
+ ):
parsed_date = datetime.now()
else:
- cleaned = remove_ordinal_indicator_from_date_string(next_collection)
+ cleaned = remove_ordinal_indicator_from_date_string(
+ next_collection
+ )
try:
parsed_date = datetime.strptime(cleaned, "%A, %d %B %Y")
except ValueError:
@@ -87,7 +92,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
)
except ValueError as e:
- print(f"Error parsing date for {bin_type}: {e}")
+ print(f"Collection date parsing failed ({type(e).__name__})")
break
return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/BirminghamCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BirminghamCityCouncil.py
index a535a992ff..a07782dac1 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BirminghamCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BirminghamCityCouncil.py
@@ -4,7 +4,6 @@
from datetime import datetime
from dateutil.parser import parse as dateutil_parse
from dateutil.parser import ParserError
-from yarl import URL
from uk_bin_collection.uk_bin_collection.common import (
check_uprn,
@@ -89,10 +88,13 @@ def parse_data(self, page: str, **kwargs: Any) -> Dict[str, List[Dict[str, str]]
"postcode": postcode,
"uprn": uprn,
}
- url = URL(
- "https://www.birmingham.gov.uk/info/50388/check_your_collection_day"
- ).with_query(query_string)
- response = requests.get(url, headers=HEADERS, timeout=30)
+ url = "https://www.birmingham.gov.uk/info/50388/check_your_collection_day"
+ response = requests.get(
+ url,
+ params=query_string,
+ headers=HEADERS,
+ timeout=30,
+ )
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
diff --git a/uk_bin_collection/uk_bin_collection/councils/BlaenauGwentCountyBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BlaenauGwentCountyBoroughCouncil.py
index 119fef279f..a62e4a6c0a 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BlaenauGwentCountyBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BlaenauGwentCountyBoroughCouncil.py
@@ -1,8 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
import re
import time
@@ -12,6 +10,17 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -25,10 +34,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Create Selenium webdriver
user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
driver = create_webdriver(web_driver, headless, user_agent, __name__)
-
+
# Navigate to the main page first
driver.get("https://www.blaenau-gwent.gov.uk/en/resident/waste-recycling/")
-
+
# Handle cookie overlay if present
try:
# Wait a moment for any overlays to appear
@@ -40,7 +49,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
"//button[contains(text(), 'Accept')]",
"//button[contains(text(), 'OK')]",
"//button[@id='ccc-recommended-settings']",
- "//button[contains(@class, 'cookie')]"
+ "//button[contains(@class, 'cookie')]",
]
for button_xpath in cookie_buttons:
try:
@@ -52,13 +61,15 @@ def parse_data(self, page: str, **kwargs) -> dict:
continue
except:
pass # No cookie overlay found
-
+
# Find and extract the collection day URL
find_collection_link = WebDriverWait(driver, 10).until(
- EC.presence_of_element_located((By.XPATH, "//a[contains(text(), 'Find Your Collection Day')]"))
+ EC.presence_of_element_located(
+ (By.XPATH, "//a[contains(text(), 'Find Your Collection Day')]")
+ )
)
collection_url = find_collection_link.get_attribute("href")
-
+
# Navigate to the collection portal
driver.get(collection_url)
@@ -70,7 +81,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Click Find button
find_button = WebDriverWait(driver, 10).until(
- EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Find')]"))
+ EC.element_to_be_clickable(
+ (By.XPATH, "//button[contains(text(), 'Find')]")
+ )
)
find_button.click()
@@ -83,42 +96,47 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Wait for collection data to load
time.sleep(3) # Give JavaScript time to process the selection
-
+
# Wait for the actual collection data to appear
WebDriverWait(driver, 20).until(
- lambda d: "Your next collections" in d.page_source and ("Recycling" in d.page_source or "Refuse" in d.page_source)
+ lambda d: "Your next collections" in d.page_source
+ and ("Recycling" in d.page_source or "Refuse" in d.page_source)
)
soup = BeautifulSoup(driver.page_source, features="html.parser")
page_text = soup.get_text()
-
+
# Find the collections section in the text
if "Your next collections" in page_text:
# Extract the section after "Your next collections"
collections_section = page_text.split("Your next collections")[1]
- collections_section = collections_section.split("Related content")[0] # Stop at Related content
-
+ collections_section = collections_section.split("Related content")[
+ 0
+ ] # Stop at Related content
+
# Use regex to find collection patterns
# Pattern to match: "Collection Type" followed by "Day Date Month" (stopping before 'followed')
- pattern = r'(Recycling collection|Refuse Bin)([A-Za-z]+ \d+ [A-Za-z]+)(?=followed|$|[A-Z])'
+ pattern = r"(Recycling collection|Refuse Bin)([A-Za-z]+ \d+ [A-Za-z]+)(?=followed|$|[A-Z])"
matches = re.findall(pattern, collections_section)
-
+
for bin_type, date_text in matches:
try:
# Clean up the date text
date_text = date_text.strip()
if "followed by" in date_text:
date_text = date_text.split("followed by")[0].strip()
-
+
# Parse the date
collection_date = datetime.strptime(date_text, "%A %d %B")
-
+
# Set the correct year
current_year = datetime.now().year
current_month = datetime.now().month
-
+
if (current_month > 10) and (collection_date.month < 3):
- collection_date = collection_date.replace(year=(current_year + 1))
+ collection_date = collection_date.replace(
+ year=(current_year + 1)
+ )
else:
collection_date = collection_date.replace(year=current_year)
@@ -131,9 +149,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
pass # Skip if date parsing fails
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
driver.quit()
- return data
\ No newline at end of file
+ return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/BoltonCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BoltonCouncil.py
index 6f617dcde5..0c8a8ade7d 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BoltonCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BoltonCouncil.py
@@ -114,7 +114,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
diff --git a/uk_bin_collection/uk_bin_collection/councils/BostonBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BostonBoroughCouncil.py
index fdc09dffe0..ffed100924 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BostonBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BostonBoroughCouncil.py
@@ -1,14 +1,8 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
-from selenium.common.exceptions import (
- ElementClickInterceptedException,
- NoSuchElementException,
- TimeoutException,
-)
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -24,19 +18,34 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
"""
Retrieve bin collection types and upcoming collection dates for the given address.
-
+
Parameters:
page (str): Unused by this implementation (kept for interface compatibility).
paon (str, in kwargs): Property/PAON text used to select the correct address option.
postcode (str, in kwargs): Postcode to search for addresses.
web_driver (optional, in kwargs): Selenium WebDriver instance or web driver identifier to use when creating the driver.
headless (bool, optional, in kwargs): Whether to run the browser in headless mode.
-
+
Returns:
data (dict): Dictionary with a single key "bins" whose value is a list of dictionaries. Each entry contains:
- "type" (str): The bin/collection type name.
- "collectionDate" (str): The next collection date formatted according to the module's date_format.
"""
+ global By, EC, ElementClickInterceptedException, NoSuchElementException, TimeoutException, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.common.exceptions import (
+ ElementClickInterceptedException,
+ NoSuchElementException,
+ TimeoutException,
+ )
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -98,9 +107,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
# pattern as a fallback in case the council re-renames it again.
WebDriverWait(driver, 15).until(
EC.presence_of_element_located(
- (By.XPATH,
- "//select[contains(@id, 'ADDRESSUPRN') or contains(@id, 'ADDRESSSELECTION')]"
- " | //div[contains(@id, 'ADDRESSUPRN_chosen') or contains(@id, 'ADDRESSSELECTION_chosen') or contains(@class, 'chosen-container')]")
+ (
+ By.XPATH,
+ "//select[contains(@id, 'ADDRESSUPRN') or contains(@id, 'ADDRESSSELECTION')]"
+ " | //div[contains(@id, 'ADDRESSUPRN_chosen') or contains(@id, 'ADDRESSSELECTION_chosen') or contains(@class, 'chosen-container')]",
+ )
)
)
@@ -110,11 +121,13 @@ def parse_data(self, page: str, **kwargs) -> dict:
dropdown_containers = driver.find_elements(
By.XPATH,
"//div[contains(@id, 'ADDRESSUPRN_chosen') or contains(@id, 'ADDRESSSELECTION_chosen')]"
- " | //div[contains(@class, 'chosen-container')]"
+ " | //div[contains(@class, 'chosen-container')]",
)
if dropdown_containers:
dropdown = dropdown_containers[0]
- driver.execute_script("arguments[0].scrollIntoView({block:'center'});", dropdown)
+ driver.execute_script(
+ "arguments[0].scrollIntoView({block:'center'});", dropdown
+ )
try:
dropdown.click()
except ElementClickInterceptedException:
@@ -125,7 +138,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
try:
search_input = driver.find_element(
By.XPATH,
- "//div[contains(@class, 'chosen-container')]//input[contains(@class, 'chosen-search-input') or @type='text']"
+ "//div[contains(@class, 'chosen-container')]//input[contains(@class, 'chosen-search-input') or @type='text']",
)
search_input.clear()
search_input.send_keys(user_paon)
@@ -138,10 +151,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
desired_option = WebDriverWait(driver, 10).until(
- EC.element_to_be_clickable((
- By.XPATH,
- f"//li[contains(@class, 'active-result') and contains(., '{user_paon}')]"
- ))
+ EC.element_to_be_clickable(
+ (
+ By.XPATH,
+ f"//li[contains(@class, 'active-result') and contains(., '{user_paon}')]",
+ )
+ )
)
desired_option.click()
else:
@@ -164,7 +179,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Click the next button to proceed
next_button = WebDriverWait(driver, 10).until(
EC.presence_of_element_located(
- (By.XPATH, "//button[contains(@id, 'NEXT') and contains(@id, 'BBCWASTECOLLECTIONSV2')]")
+ (
+ By.XPATH,
+ "//button[contains(@id, 'NEXT') and contains(@id, 'BBCWASTECOLLECTIONSV2')]",
+ )
)
)
next_button.click()
@@ -172,7 +190,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Wait for the collections information to appear
WebDriverWait(driver, 10).until(
EC.presence_of_element_located(
- (By.XPATH, "//div[contains(@class, 'item__title') or contains(@class, 'grid__cell--listitem')]")
+ (
+ By.XPATH,
+ "//div[contains(@class, 'item__title') or contains(@class, 'grid__cell--listitem')]",
+ )
)
)
@@ -192,18 +213,18 @@ def parse_data(self, page: str, **kwargs) -> dict:
bin_type_elem = bin_div.find("h2", class_="item__title")
if not bin_type_elem:
continue
-
+
bin_type = bin_type_elem.text.strip()
# Find the next collection date
content_div = bin_div.find("div", class_="item__content")
if not content_div:
continue
-
+
date_div = content_div.find("div")
if not date_div:
continue
-
+
next_collection = date_div.text.strip().replace("Next: ", "")
next_collection = datetime.strptime(
@@ -224,7 +245,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/BrentCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BrentCouncil.py
index 50fe942a57..8ceed738ee 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BrentCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BrentCouncil.py
@@ -27,9 +27,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
payload = {"postcode": user_postcode}
s = requests.Session()
- s.headers.update({
- "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
- })
+ s.headers.update(
+ {
+ "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
+ }
+ )
# Make the POST request
response = s.post(URI, data=payload)
@@ -125,8 +127,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
),
}
data["bins"].append(dict_data)
- print(dict_data)
+ print("Collection added successfully")
except ValueError as e:
- print(f"Error parsing date {next_collection}: {e}")
+ print(
+ "Collection date parsing failed "
+ f"({type(e).__name__})"
+ )
return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/BrightonandHoveCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BrightonandHoveCityCouncil.py
index df4e588ec5..434e262ee4 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BrightonandHoveCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BrightonandHoveCityCouncil.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
# This script pulls (in one hit) the data from Bromley Council Bins Data
import datetime
import re
@@ -6,11 +8,6 @@
import requests
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -27,9 +24,9 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
"""
Fetch and parse bin collection data for a given address from Brighton & Hove's collections page.
-
+
This function drives a Selenium browser to the fixed Brighton & Hove collections URL, submits the provided postcode, selects the matching PAON (primary addressable object name) from the resulting address dropdown, submits the selection, and parses the resulting list view into structured bin collection entries.
-
+
Parameters:
page (str): Unused; included for compatibility with caller signature.
uprn (str, optional): Unique Property Reference Number for the address (passed via kwargs).
@@ -37,15 +34,27 @@ def parse_data(self, page: str, **kwargs) -> dict:
postcode (str, optional): Postcode to search on the council site (passed via kwargs).
web_driver (str or WebDriver, optional): Specification or instance used by create_webdriver to start the browser (passed via kwargs).
headless (bool, optional): Whether to run the browser in headless mode (passed via kwargs).
-
+
Returns:
dict: A dictionary with a single key "bins" whose value is a list of objects each containing:
- "type": bin type string
- "collectionDate": collection date string formatted according to the module's date_format
-
+
Raises:
Exception: If no dropdown option matching `paon` is found or any other error occurs during navigation or parsing.
"""
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -137,11 +146,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
data["bins"].append(dict_data)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
# This block ensures that the driver is closed regardless of an exception
if driver:
driver.quit()
- return data
\ No newline at end of file
+ return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/BroadlandDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BroadlandDistrictCouncil.py
index bd3fe772d6..d20c2f2935 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BroadlandDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BroadlandDistrictCouncil.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
# This script pulls (in one hit) the data from Broadland District Council Bins Data
# Working command line:
# python collect_data.py BroadlandDistrictCouncil "https://area.southnorfolkandbroadland.gov.uk/FindAddress" -p "NR10 3FD" -n "1 Park View, Horsford, Norfolk, NR10 3FD"
@@ -7,11 +9,6 @@
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +17,18 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -32,16 +41,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
headless = kwargs.get("headless")
url = kwargs.get("url")
- print(
- f"Starting parse_data with parameters: postcode={postcode}, paon={user_paon}"
- )
- print(
- f"Creating webdriver with: web_driver={web_driver}, headless={headless}"
- )
+ print("Starting parse_data with configured address parameters")
+ print(f"Creating configured webdriver (headless={headless})")
user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
driver = create_webdriver(web_driver, headless, user_agent, __name__)
- print(f"Navigating to URL: {url}")
+ print("Navigating to the configured council page")
driver.get(url)
print("Successfully loaded the page")
@@ -86,7 +91,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
dropdown_select = Select(address_dropdown)
- print(f"Looking for address containing: {user_paon}")
+ print("Looking for the configured address")
found = False
user_paon_clean = user_paon.lower().strip()
@@ -95,12 +100,15 @@ def parse_data(self, page: str, **kwargs) -> dict:
option_text_clean = option.text.lower().strip()
if (
- option_text_clean == user_paon_clean # Exact match if full address given
- or option_text_clean.startswith(f"{user_paon_clean} ") # Startswith match if just a number
+ option_text_clean
+ == user_paon_clean # Exact match if full address given
+ or option_text_clean.startswith(
+ f"{user_paon_clean} "
+ ) # Startswith match if just a number
):
option.click()
found = True
- print(f"Selected address: {option.text.strip()}")
+ print("Address selected successfully")
break
if not found:
@@ -169,7 +177,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Extract the full text and remove the bin type to get the date part
full_text = text_container.get_text(strip=True)
date_text = full_text.replace(bin_type, "").strip()
- print(f"Unparsed collection date: {date_text}")
+ print("Collection date value found")
# Parse the date
# First, remove any ordinal indicators (1st, 2nd, 3rd, etc.)
@@ -189,9 +197,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
"collectionDate": bin_date,
}
data["bins"].append(dict_data)
- print(f"Added bin data: {dict_data}")
+ print("Collection added successfully")
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred ({type(e).__name__})")
raise
finally:
print("Cleaning up webdriver...")
diff --git a/uk_bin_collection/uk_bin_collection/councils/BromleyBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BromleyBoroughCouncil.py
index 11b054efae..93569c34f0 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BromleyBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BromleyBoroughCouncil.py
@@ -1,12 +1,11 @@
+from __future__ import annotations
+
# This script pulls (in one hit) the data from Bromley Council Bins Data
import datetime
from datetime import datetime
from bs4 import BeautifulSoup
from dateutil.relativedelta import relativedelta
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -21,6 +20,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import WebDriverWait
+
driver = None
try:
bin_data_dict = {"bins": []}
@@ -97,7 +106,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
data["bins"].append(dict_data)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.py
index 3b9c717ec7..8117d431c7 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BroxbourneCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
from datetime import datetime
import time
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -13,49 +11,66 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
user_uprn = kwargs.get("uprn")
user_postcode = kwargs.get("postcode")
web_driver = kwargs.get("web_driver")
headless = kwargs.get("headless")
-
+
check_uprn(user_uprn)
check_postcode(user_postcode)
-
+
bindata = {"bins": []}
# Use a realistic user agent to help bypass Cloudflare
user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
driver = create_webdriver(web_driver, headless, user_agent, __name__)
-
+
try:
driver.get("https://www.broxbourne.gov.uk/bin-collection-date")
-
+
# Wait for Cloudflare challenge to complete
print("Waiting for page to load (Cloudflare check)...")
try:
WebDriverWait(driver, 45).until(
- lambda d: "Just a moment" not in d.title and d.title != "" and len(d.find_elements(By.TAG_NAME, "input")) > 0
+ lambda d: "Just a moment" not in d.title
+ and d.title != ""
+ and len(d.find_elements(By.TAG_NAME, "input")) > 0
)
- print(f"Page loaded: {driver.title}")
+ print("Page loaded successfully.")
except:
- print(f"Timeout waiting for page load. Current title: {driver.title}")
+ print("Timeout waiting for page load.")
# Try to continue anyway
pass
-
+
time.sleep(8)
-
+
# Handle cookie banner with multiple attempts
try:
cookie_btn = WebDriverWait(driver, 15).until(
- EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Allow all')]"))
+ EC.element_to_be_clickable(
+ (By.XPATH, "//button[contains(text(), 'Allow all')]")
+ )
)
cookie_btn.click()
except:
pass
-
+
# Find postcode input
postcode_input = WebDriverWait(driver, 20).until(
- EC.element_to_be_clickable((By.XPATH, "//input[@autocomplete='postal-code']"))
+ EC.element_to_be_clickable(
+ (By.XPATH, "//input[@autocomplete='postal-code']")
+ )
)
postcode_input.clear()
@@ -74,52 +89,76 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
)
Select(address_select).select_by_value(user_uprn)
-
+
# Click Next button
next_btn = WebDriverWait(driver, 15).until(
- EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Next')]"))
+ EC.element_to_be_clickable(
+ (By.XPATH, "//button[contains(text(), 'Next')]")
+ )
)
next_btn.click()
-
+
# Get results
WebDriverWait(driver, 15).until(
- EC.presence_of_element_located((By.XPATH, "//h1[contains(text(), 'When is my bin collection date?')]"))
+ EC.presence_of_element_located(
+ (
+ By.XPATH,
+ "//h1[contains(text(), 'When is my bin collection date?')]",
+ )
+ )
)
-
+
table = WebDriverWait(driver, 15).until(
- EC.presence_of_element_located((By.XPATH, "//h1[contains(text(), 'When is my bin collection date?')]/following::table[1]"))
+ EC.presence_of_element_located(
+ (
+ By.XPATH,
+ "//h1[contains(text(), 'When is my bin collection date?')]/following::table[1]",
+ )
+ )
)
-
- soup = BeautifulSoup(table.get_attribute('outerHTML'), 'html.parser')
- rows = soup.find_all('tr')
-
+
+ soup = BeautifulSoup(table.get_attribute("outerHTML"), "html.parser")
+ rows = soup.find_all("tr")
+
current_year = datetime.now().year
current_month = datetime.now().month
-
+
for row in rows[1:]:
- columns = row.find_all('td')
+ columns = row.find_all("td")
if len(columns) >= 2:
collection_date_text = columns[0].get_text().strip()
service = columns[1].get_text().strip()
-
+
if collection_date_text:
try:
- collection_date = datetime.strptime(collection_date_text, "%a %d %b")
+ collection_date = datetime.strptime(
+ collection_date_text, "%a %d %b"
+ )
if collection_date.month == 1 and current_month != 1:
- collection_date = collection_date.replace(year=current_year + 1)
+ collection_date = collection_date.replace(
+ year=current_year + 1
+ )
else:
- collection_date = collection_date.replace(year=current_year)
-
- bindata["bins"].append({
- "type": service,
- "collectionDate": collection_date.strftime("%d/%m/%Y")
- })
+ collection_date = collection_date.replace(
+ year=current_year
+ )
+
+ bindata["bins"].append(
+ {
+ "type": service,
+ "collectionDate": collection_date.strftime(
+ "%d/%m/%Y"
+ ),
+ }
+ )
except ValueError:
continue
-
- bindata["bins"].sort(key=lambda x: datetime.strptime(x["collectionDate"], "%d/%m/%Y"))
-
+
+ bindata["bins"].sort(
+ key=lambda x: datetime.strptime(x["collectionDate"], "%d/%m/%Y")
+ )
+
finally:
driver.quit()
-
- return bindata
\ No newline at end of file
+
+ return bindata
diff --git a/uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py
index c5f98c39d7..26a48929a2 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BroxtoweBoroughCouncil.py
@@ -1,8 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -17,6 +15,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
page = "https://selfservice.broxtowe.gov.uk/renderform.aspx?t=217&k=9D2EF214E144EE796430597FB475C3892C43C528"
@@ -70,7 +79,14 @@ def parse_data(self, page: str, **kwargs) -> dict:
paon_lower = user_paon.strip().lower()
for option in dropdownSelect.options:
text = option.text.strip().lower()
- if text and paon_lower and (text.startswith(paon_lower + " ") or text.startswith(paon_lower + ",")):
+ if (
+ text
+ and paon_lower
+ and (
+ text.startswith(paon_lower + " ")
+ or text.startswith(paon_lower + ",")
+ )
+ ):
option.click()
matched = True
break
@@ -126,7 +142,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/BuckinghamshireCouncil.py b/uk_bin_collection/uk_bin_collection/councils/BuckinghamshireCouncil.py
index 2bb9908113..d7bae960ee 100644
--- a/uk_bin_collection/uk_bin_collection/councils/BuckinghamshireCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/BuckinghamshireCouncil.py
@@ -111,7 +111,7 @@ def parse_data(self, _: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/CanterburyCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/CanterburyCityCouncil.py
index 00f34a02fd..49f911d5f2 100644
--- a/uk_bin_collection/uk_bin_collection/councils/CanterburyCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/CanterburyCityCouncil.py
@@ -45,7 +45,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
# Loop through each collection in bin_collection
for collection in collections:
- print(collection)
+ print("Processing collection entry")
if len(collections[collection]) <= 0:
continue
diff --git a/uk_bin_collection/uk_bin_collection/councils/CardiffCouncil.py b/uk_bin_collection/uk_bin_collection/councils/CardiffCouncil.py
index cca28a3a12..b12e5f0c22 100644
--- a/uk_bin_collection/uk_bin_collection/councils/CardiffCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/CardiffCouncil.py
@@ -79,7 +79,7 @@ def get_jwt() -> str:
raise ValueError("Invalid server response code getting JWT!")
except Exception as ex:
- print(f"Exception encountered: {ex}")
+ print(f"Exception encountered: {type(ex).__name__}")
exit(1)
token = parse_token(response.text)
options.close()
@@ -144,7 +144,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
raise ValueError("Invalid server response code finding UPRN!")
except Exception as ex:
- print(f"Exception encountered: {ex}")
+ print(f"Exception encountered: {type(ex).__name__}")
exit(1)
result = json.loads(response.text)
diff --git a/uk_bin_collection/uk_bin_collection/councils/CastlepointDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/CastlepointDistrictCouncil.py
index 4766d304ac..a8d8f455f9 100644
--- a/uk_bin_collection/uk_bin_collection/councils/CastlepointDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/CastlepointDistrictCouncil.py
@@ -74,7 +74,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
continue
month_txt = container.find("tr", class_="calendar").get_text(strip=True)
month = datetime.strptime(month_txt, "%B").strftime("%m")
- print(month_txt)
+ print("Collection month parsed")
pink_days = [
td.get_text(strip=True)
diff --git a/uk_bin_collection/uk_bin_collection/councils/CeredigionCountyCouncil.py b/uk_bin_collection/uk_bin_collection/councils/CeredigionCountyCouncil.py
index 7ca1fab762..414cbf6485 100644
--- a/uk_bin_collection/uk_bin_collection/councils/CeredigionCountyCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/CeredigionCountyCouncil.py
@@ -1,10 +1,8 @@
+from __future__ import annotations
+
from time import sleep
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.common.exceptions import NoSuchElementException
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +17,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, NoSuchElementException, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.common.exceptions import NoSuchElementException
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
house_number = kwargs.get("paon")
@@ -178,14 +187,14 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
data["bins"].append(dict_data)
except Exception as e:
- print(f"Skipping one panel due to: {e}")
+ print(f"Skipping one panel due to: {type(e).__name__}")
data["bins"].sort(
key=lambda x: datetime.strptime(x.get("collectionDate"), date_format)
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/ChelmsfordCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ChelmsfordCityCouncil.py
index 77e79a2caa..a427b2a8b5 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ChelmsfordCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ChelmsfordCityCouncil.py
@@ -37,9 +37,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
postcode = kwargs.get("postcode")
user_paon = kwargs.get("paon")
- headers = {
- "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
- }
+ headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
session = requests.Session()
base_url = "https://www.chelmsford.gov.uk/bins-and-recycling/check-your-collection-day/"
@@ -142,7 +140,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
)
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/ChesterfieldBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ChesterfieldBoroughCouncil.py
index 97277c8369..ba3e8de27e 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ChesterfieldBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ChesterfieldBoroughCouncil.py
@@ -9,7 +9,6 @@
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
import urllib3
-
# Suppress only the single warning from urllib3 needed.
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
@@ -159,7 +158,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
dt_local = dt_utc.astimezone(None)
collection_date = dt_local.date()
except (IndexError, KeyError, ValueError) as e:
- _LOGGER.warning(f"Failed to parse date for {waste_type}: {e}")
+ _LOGGER.warning(
+ "Failed to parse a collection date (%s).",
+ type(e).__name__,
+ )
continue
# Append to bin_schedule
@@ -179,10 +181,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except requests.RequestException as e:
- _LOGGER.error(f"Network error occurred: {e}")
+ _LOGGER.error(f"Network error occurred: {type(e).__name__}")
except json.JSONDecodeError as e:
- _LOGGER.error(f"JSON decoding failed: {e}")
+ _LOGGER.error(f"JSON decoding failed: {type(e).__name__}")
except Exception as e:
- _LOGGER.error(f"An unexpected error occurred: {e}")
+ _LOGGER.error(f"An unexpected error occurred: {type(e).__name__}")
return bindata
diff --git a/uk_bin_collection/uk_bin_collection/councils/ChichesterDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ChichesterDistrictCouncil.py
index c133e12b90..83ab20a21c 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ChichesterDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ChichesterDistrictCouncil.py
@@ -1,16 +1,9 @@
+from __future__ import annotations
+
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support.ui import WebDriverWait, Select
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.common.exceptions import (
- StaleElementReferenceException,
- TimeoutException,
- NoSuchElementException,
-)
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -36,6 +29,22 @@ class CouncilClass(AbstractGetBinDataClass):
DROPDOWN_ID = "WASTECOLLECTIONCALENDARV7_CALENDAR_ADDRESSLOOKUPADDRESS"
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, NoSuchElementException, Select, StaleElementReferenceException, TimeoutException, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support.ui import WebDriverWait, Select
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.common.exceptions import (
+ StaleElementReferenceException,
+ TimeoutException,
+ NoSuchElementException,
+ )
+
driver = None
try:
start_url = "https://www.chichester.gov.uk/checkyourbinday"
@@ -171,9 +180,7 @@ def dropdown_has_addresses(d):
sel_element = driver.find_element(By.ID, self.DROPDOWN_ID)
select = Select(sel_element)
- raise Exception(
- f"Failed to select '{target_text}' after retries"
- )
+ raise Exception(f"Failed to select '{target_text}' after retries")
@staticmethod
def _pick_option(options, house_number: str):
@@ -182,8 +189,10 @@ def _pick_option(options, house_number: str):
for opt in options:
text = opt.text.strip()
low = text.lower()
- if low == target or low.startswith(f"{target},") or low.startswith(
- f"{target} "
+ if (
+ low == target
+ or low.startswith(f"{target},")
+ or low.startswith(f"{target} ")
):
return text
# Fuzzy: substring match
diff --git a/uk_bin_collection/uk_bin_collection/councils/ColchesterCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ColchesterCityCouncil.py
index 346b74946f..b04cb814b5 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ColchesterCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ColchesterCityCouncil.py
@@ -1,7 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -16,6 +15,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -94,7 +103,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/CotswoldDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/CotswoldDistrictCouncil.py
index ed8c7ff70f..c62809cad0 100644
--- a/uk_bin_collection/uk_bin_collection/councils/CotswoldDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/CotswoldDistrictCouncil.py
@@ -1,24 +1,40 @@
+from __future__ import annotations
+
import time
import re
from datetime import datetime, timedelta
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
_BIN_TYPES = {
- "180 litre refuse", "black recycling box", "blue bag", "white bag",
- "outdoor food caddy", "indoor food caddy", "garden waste",
- "240 litre refuse", "recycling box", "food caddy",
+ "180 litre refuse",
+ "black recycling box",
+ "blue bag",
+ "white bag",
+ "outdoor food caddy",
+ "indoor food caddy",
+ "garden waste",
+ "240 litre refuse",
+ "recycling box",
+ "food caddy",
}
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import WebDriverWait
+
driver = None
try:
page = "https://community.cotswold.gov.uk/s/waste-collection-enquiry"
@@ -26,7 +42,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
house_number = kwargs.get("paon")
postcode = kwargs.get("postcode")
- if house_number and postcode and postcode.upper() not in house_number.upper():
+ if (
+ house_number
+ and postcode
+ and postcode.upper() not in house_number.upper()
+ ):
full_address = f"{house_number}, {postcode}"
else:
full_address = house_number or postcode or ""
@@ -41,9 +61,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
time.sleep(8)
address_entry_field = wait.until(
- EC.presence_of_element_located(
- (By.XPATH, "//input[@role='combobox']")
- )
+ EC.presence_of_element_located((By.XPATH, "//input[@role='combobox']"))
)
address_entry_field.click()
@@ -52,9 +70,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
time.sleep(4)
wait.until(
- EC.element_to_be_clickable(
- (By.XPATH, "//li[@role='presentation']")
- )
+ EC.element_to_be_clickable((By.XPATH, "//li[@role='presentation']"))
)
all_opts = driver.find_elements(By.XPATH, "//li[@role='presentation']")
if not all_opts:
@@ -78,7 +94,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
for _ in range(8):
time.sleep(5)
- if driver.find_elements(By.XPATH, "//*[contains(text(), 'Collection day')]"):
+ if driver.find_elements(
+ By.XPATH, "//*[contains(text(), 'Collection day')]"
+ ):
break
soup = BeautifulSoup(driver.page_source, features="html.parser")
@@ -92,23 +110,23 @@ def parse_data(self, page: str, **kwargs) -> dict:
td = row.find("td")
if not th or not td:
continue
- container_type = (
- th.get("data-cell-value", "").strip()
- or th.get_text(strip=True)
- )
- raw_date = (
- td.get("data-cell-value", "").strip()
- or td.get_text(strip=True)
+ container_type = th.get(
+ "data-cell-value", ""
+ ).strip() or th.get_text(strip=True)
+ raw_date = td.get("data-cell-value", "").strip() or td.get_text(
+ strip=True
)
if container_type and raw_date:
try:
parsed_date = self._parse_date(raw_date, current_year)
except (ValueError, AttributeError):
continue
- data["bins"].append({
- "type": container_type,
- "collectionDate": parsed_date,
- })
+ data["bins"].append(
+ {
+ "type": container_type,
+ "collectionDate": parsed_date,
+ }
+ )
except (ValueError, AttributeError):
continue
else:
@@ -122,13 +140,15 @@ def parse_data(self, page: str, **kwargs) -> dict:
parsed_date = self._parse_date(raw_date, current_year)
except (ValueError, AttributeError):
continue
- data["bins"].append({
- "type": line,
- "collectionDate": parsed_date,
- })
+ data["bins"].append(
+ {
+ "type": line,
+ "collectionDate": parsed_date,
+ }
+ )
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
@@ -138,7 +158,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
@staticmethod
def _looks_like_date(text):
t = text.lower().strip()
- return t in ("today", "tomorrow") or bool(re.match(r"^(mon|tue|wed|thu|fri|sat|sun)", t))
+ return t in ("today", "tomorrow") or bool(
+ re.match(r"^(mon|tue|wed|thu|fri|sat|sun)", t)
+ )
@staticmethod
def _parse_date(raw_date, current_year):
diff --git a/uk_bin_collection/uk_bin_collection/councils/CroydonCouncil.py b/uk_bin_collection/uk_bin_collection/councils/CroydonCouncil.py
index df642a45fe..ffa6cde199 100644
--- a/uk_bin_collection/uk_bin_collection/councils/CroydonCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/CroydonCouncil.py
@@ -1,11 +1,8 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
-from selenium import webdriver
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +16,18 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait, webdriver
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium import webdriver
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
user_postcode = kwargs.get("postcode")
@@ -120,14 +129,17 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
bin_data["bins"].append(bin_info)
except ValueError as e:
- print(f"Error parsing date '{collection_date_string}': {e}")
+ print(
+ "Collection date parsing failed "
+ f"({type(e).__name__})"
+ )
if not bin_data["bins"]:
raise ValueError("No bin collection data found")
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/DacorumBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/DacorumBoroughCouncil.py
index 089c7462a5..c683cfac66 100644
--- a/uk_bin_collection/uk_bin_collection/councils/DacorumBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/DacorumBoroughCouncil.py
@@ -1,7 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -16,6 +15,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -81,17 +90,30 @@ def parse_data(self, page: str, **kwargs) -> dict:
if strong_element:
BinType = strong_element.text.strip()
# Skip if this is not a bin type (e.g., informational text)
- if BinType and not any(skip_text in BinType.lower() for skip_text in
- ["please note", "we may collect", "bank holiday", "different day"]):
- date_cells = Collection.find_all("div", {"style": "display:table-cell;"})
+ if BinType and not any(
+ skip_text in BinType.lower()
+ for skip_text in [
+ "please note",
+ "we may collect",
+ "bank holiday",
+ "different day",
+ ]
+ ):
+ date_cells = Collection.find_all(
+ "div", {"style": "display:table-cell;"}
+ )
if len(date_cells) > 1:
date_text = date_cells[1].get_text().strip()
if date_text:
try:
- CollectionDate = datetime.strptime(date_text, "%a, %d %b %Y")
+ CollectionDate = datetime.strptime(
+ date_text, "%a, %d %b %Y"
+ )
dict_data = {
"type": BinType,
- "collectionDate": CollectionDate.strftime("%d/%m/%Y"),
+ "collectionDate": CollectionDate.strftime(
+ "%d/%m/%Y"
+ ),
}
# Check for duplicates before adding
if dict_data not in data["bins"]:
@@ -102,7 +124,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/DerbyshireDalesDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/DerbyshireDalesDistrictCouncil.py
index 8cbf19b5ad..2795160bd6 100644
--- a/uk_bin_collection/uk_bin_collection/councils/DerbyshireDalesDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/DerbyshireDalesDistrictCouncil.py
@@ -94,7 +94,7 @@ def get_hidden_value(soup, name):
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/DurhamCouncil.py b/uk_bin_collection/uk_bin_collection/councils/DurhamCouncil.py
index 810b4c7f10..64e060463d 100644
--- a/uk_bin_collection/uk_bin_collection/councils/DurhamCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/DurhamCouncil.py
@@ -78,9 +78,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
# volume suffix. Whitespace is flexible throughout. fullmatch() is
# used so unexpected leading/trailing content surfaces as a warning
# rather than being silently absorbed.
- name_pattern = re.compile(
- r"(?:Empty\s+Bin\s+)?(.+?)(?:\s+\d+\s*[Ll])?"
- )
+ name_pattern = re.compile(r"(?:Empty\s+Bin\s+)?(.+?)(?:\s+\d+\s*[Ll])?")
next_dates = {}
for job in soup.find_all("job"):
@@ -92,7 +90,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
raw_name = name_tag.get_text(strip=True)
match = name_pattern.fullmatch(raw_name)
if not match:
- _LOGGER.warning("Could not parse bin name %r", raw_name)
+ _LOGGER.warning("Could not parse a collection type")
continue
label = match.group(1).strip()
@@ -101,11 +99,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
start_tag.get_text(strip=True)[:10], "%Y-%m-%d"
)
except ValueError:
- _LOGGER.warning(
- "Could not parse scheduled date %r for bin %r",
- start_tag.get_text(strip=True),
- raw_name,
- )
+ _LOGGER.warning("Could not parse a scheduled collection date")
continue
if scheduled < today:
@@ -115,9 +109,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
data = {"bins": []}
for bin_type, collection_date in next_dates.items():
- data["bins"].append({
- "type": bin_type,
- "collectionDate": collection_date.strftime(date_format),
- })
+ data["bins"].append(
+ {
+ "type": bin_type,
+ "collectionDate": collection_date.strftime(date_format),
+ }
+ )
return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/EastLindseyDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/EastLindseyDistrictCouncil.py
index 82a57e3b7a..ce0577754c 100644
--- a/uk_bin_collection/uk_bin_collection/councils/EastLindseyDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/EastLindseyDistrictCouncil.py
@@ -1,7 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -16,6 +15,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -101,7 +110,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/EastRenfrewshireCouncil.py b/uk_bin_collection/uk_bin_collection/councils/EastRenfrewshireCouncil.py
index 6dcfc2ce03..fd789fdf4d 100644
--- a/uk_bin_collection/uk_bin_collection/councils/EastRenfrewshireCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/EastRenfrewshireCouncil.py
@@ -1,8 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
-from selenium.webdriver.support.ui import Select
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -16,6 +14,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+ from selenium.webdriver.support.ui import Select
+
driver = None
try:
data = {"bins": []}
@@ -41,9 +50,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Click search button
search_button = WebDriverWait(driver, 10).until(
- EC.element_to_be_clickable(
- (By.XPATH, "//button[text()='Search']")
- )
+ EC.element_to_be_clickable((By.XPATH, "//button[text()='Search']"))
)
search_button.click()
@@ -53,7 +60,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
(By.XPATH, "//label[text()='Addresses']/following-sibling::select")
)
)
-
+
# Select the appropriate address based on UPRN or house number
select = Select(addresses_select)
if user_uprn:
@@ -79,31 +86,31 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Wait for the results table to appear
WebDriverWait(driver, 10).until(
- EC.presence_of_element_located(
- (By.XPATH, "//th[text()='Bin Type']")
- )
+ EC.presence_of_element_located((By.XPATH, "//th[text()='Bin Type']"))
)
soup = BeautifulSoup(driver.page_source, features="html.parser")
-
+
# Find the table with bin collection data
table = soup.find("th", string="Bin Type").find_parent("table")
rows = table.find_all("tr")[1:] # Skip header row
-
+
for row in rows:
cells = row.find_all("td")
if len(cells) >= 3:
date_cell = cells[0].get_text().strip()
bin_type_cell = cells[2]
-
+
# Only process rows that have a date
if date_cell:
# Get all text content including line breaks
- bin_type_text = bin_type_cell.get_text(separator='\n').strip()
-
+ bin_type_text = bin_type_cell.get_text(separator="\n").strip()
+
# Split multiple bin types that appear on separate lines
- bin_types = [bt.strip() for bt in bin_type_text.split('\n') if bt.strip()]
-
+ bin_types = [
+ bt.strip() for bt in bin_type_text.split("\n") if bt.strip()
+ ]
+
for bin_type in bin_types:
dict_data = {
"type": bin_type,
@@ -115,7 +122,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
key=lambda x: datetime.strptime(x.get("collectionDate"), "%d/%m/%Y")
)
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/EastStaffordshireBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/EastStaffordshireBoroughCouncil.py
index 8926bbbca2..4412c4def3 100644
--- a/uk_bin_collection/uk_bin_collection/councils/EastStaffordshireBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/EastStaffordshireBoroughCouncil.py
@@ -1,9 +1,7 @@
+from __future__ import annotations
+
import requests
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +17,17 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
bindata = {"bins": []}
soup = BeautifulSoup(page.text, features="html.parser")
diff --git a/uk_bin_collection/uk_bin_collection/councils/EastSuffolkCouncil.py b/uk_bin_collection/uk_bin_collection/councils/EastSuffolkCouncil.py
index 082cceac31..d93465c653 100644
--- a/uk_bin_collection/uk_bin_collection/councils/EastSuffolkCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/EastSuffolkCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
from time import sleep
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +19,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
user_uprn = kwargs.get("uprn")
@@ -121,7 +130,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/EastleighBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/EastleighBoroughCouncil.py
index 9b6c46b29d..8f95881f63 100644
--- a/uk_bin_collection/uk_bin_collection/councils/EastleighBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/EastleighBoroughCouncil.py
@@ -1,7 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -16,6 +15,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
try:
uprn = kwargs.get("uprn")
# Check the UPRN is valid
@@ -81,7 +90,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
import traceback
error_message = f"Error fetching/parsing data for Eastleigh: {str(e)}\n{traceback.format_exc()}"
- print(error_message)
+ print(f"Eastleigh data retrieval failed ({type(e).__name__}).")
# Use the correct date format for the error fallback
today = datetime.now().strftime("%d/%m/%Y")
return {"bins": [{"type": "Error", "collectionDate": today}]}
diff --git a/uk_bin_collection/uk_bin_collection/councils/EnfieldCouncil.py b/uk_bin_collection/uk_bin_collection/councils/EnfieldCouncil.py
index f91d7707a2..2b073a96f0 100644
--- a/uk_bin_collection/uk_bin_collection/councils/EnfieldCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/EnfieldCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
import pdb
from uk_bin_collection.uk_bin_collection.common import *
@@ -19,6 +18,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
user_postcode = kwargs.get("postcode")
@@ -42,18 +51,20 @@ def parse_data(self, page: str, **kwargs) -> dict:
for attempt in range(max_attempts):
try:
WebDriverWait(driver, 60).until(
- lambda d: "Just a moment" not in d.title and d.title != "" and len(d.find_elements(By.TAG_NAME, "input")) > 1
+ lambda d: "Just a moment" not in d.title
+ and d.title != ""
+ and len(d.find_elements(By.TAG_NAME, "input")) > 1
)
- print(f"Page loaded: {driver.title}")
+ print("Page loaded successfully.")
break
except:
- print(f"Attempt {attempt + 1}: Timeout waiting for page load. Current title: {driver.title}")
+ print(f"Attempt {attempt + 1}: timeout waiting for page load.")
if attempt < max_attempts - 1:
time.sleep(10)
driver.refresh()
else:
print("Failed to bypass Cloudflare after multiple attempts")
-
+
time.sleep(8)
try:
@@ -70,27 +81,30 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Check for multiple iframes and find the correct one
try:
iframes = driver.find_elements(By.TAG_NAME, "iframe")
-
+
# Try each iframe to find the one with the bin collection form
for i, iframe in enumerate(iframes):
try:
driver.switch_to.frame(iframe)
-
+
# Check if this iframe has the postcode input
time.sleep(2)
inputs = driver.find_elements(By.TAG_NAME, "input")
-
+
# Look for address-related inputs
for inp in inputs:
- aria_label = inp.get_attribute('aria-label') or ''
- placeholder = inp.get_attribute('placeholder') or ''
- if 'address' in aria_label.lower() or 'postcode' in placeholder.lower():
+ aria_label = inp.get_attribute("aria-label") or ""
+ placeholder = inp.get_attribute("placeholder") or ""
+ if (
+ "address" in aria_label.lower()
+ or "postcode" in placeholder.lower()
+ ):
break
else:
# This iframe doesn't have the form, try the next one
driver.switch_to.default_content()
continue
-
+
# Found the right iframe, break out of the loop
break
except Exception as e:
@@ -108,9 +122,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
'[aria-label="Enter your address"]',
'input[placeholder*="postcode"]',
'input[placeholder*="address"]',
- 'input[type="text"]'
+ 'input[type="text"]',
]
-
+
for selector in selectors:
try:
postcode_input = WebDriverWait(driver, 5).until(
@@ -119,7 +133,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
break
except:
continue
-
+
if not postcode_input:
raise ValueError("Could not find postcode input field")
@@ -214,7 +228,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/EpsomandEwellBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/EpsomandEwellBoroughCouncil.py
index 4de73ac35b..8dbe4e5e4c 100644
--- a/uk_bin_collection/uk_bin_collection/councils/EpsomandEwellBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/EpsomandEwellBoroughCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import re
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +17,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.ui import WebDriverWait
+
user_uprn = kwargs.get("uprn")
user_postcode = kwargs.get("postcode")
check_uprn(user_uprn)
@@ -31,18 +40,18 @@ def parse_data(self, page: str, **kwargs) -> dict:
kwargs.get("web_driver"),
kwargs.get("headless", True),
user_agent,
- __name__
+ __name__,
)
-
+
# Navigate to the iTouchVision portal
portal_url = "https://iportal.itouchvision.com/icollectionday/collection-day/?uuid=8E7DCC4BD90D8405D154BE053147018A8C0B5F09"
driver.get(portal_url)
-
+
# Wait for postcode input to be present
postcode_input = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "postcodeSearch"))
)
-
+
# Enter postcode using JavaScript to trigger React events
if user_postcode:
postcode = user_postcode
@@ -50,7 +59,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
# If no postcode provided, we need to derive it from UPRN
# For now, raise an error
raise ValueError("Postcode is required for EpsomandEwellBoroughCouncil")
-
+
driver.execute_script(f"""
const input = document.getElementById('postcodeSearch');
const nativeInputValueSetter = Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype, 'value').set;
@@ -58,94 +67,101 @@ def parse_data(self, page: str, **kwargs) -> dict:
input.dispatchEvent(new Event('input', {{ bubbles: true }}));
input.dispatchEvent(new Event('change', {{ bubbles: true }}));
""")
-
+
# Click the Find button
find_button = driver.find_element(By.CSS_SELECTOR, ".govuk-button")
find_button.click()
-
+
# Wait for address dropdown to appear and be populated
address_select = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "addressSelect"))
)
-
+
# Wait a bit for options to populate
import time
+
time.sleep(2)
-
+
# Select the address by UPRN value using JavaScript
driver.execute_script(f"""
const select = document.getElementById('addressSelect');
select.value = '{user_uprn}';
select.dispatchEvent(new Event('change', {{ bubbles: true }}));
""")
-
+
# Wait for collection data to load (look for h3 elements with bin types)
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.TAG_NAME, "h3"))
)
-
+
# Wait a bit more for all data to render
time.sleep(3)
-
+
# Get the page source and parse with BeautifulSoup
soup = BeautifulSoup(driver.page_source, "html.parser")
-
+
# Find all h3 elements (these contain bin types)
h3_elements = soup.find_all("h3")
-
+
for h3 in h3_elements:
bin_type = h3.text.strip()
-
+
# Skip if empty
if not bin_type:
continue
-
+
# Get the next sibling element which should contain the date
next_elem = h3.find_next_sibling()
if not next_elem:
continue
-
+
date_text = next_elem.text.strip()
-
+
# Parse date in format "Thursday 11 December"
# Need to add current year
try:
# Extract day and month from "Thursday 11 December" format
- match = re.search(r'(\w+)\s+(\d{1,2})\s+(\w+)', date_text)
+ match = re.search(r"(\w+)\s+(\d{1,2})\s+(\w+)", date_text)
if match:
day = match.group(2)
month = match.group(3)
-
+
# Determine the year (if month is in the past, use next year)
current_date = datetime.now()
current_year = current_date.year
-
+
# Try parsing with current year
try:
- date_obj = datetime.strptime(f"{day} {month} {current_year}", "%d %B %Y")
+ date_obj = datetime.strptime(
+ f"{day} {month} {current_year}", "%d %B %Y"
+ )
# If the date is more than 30 days in the past, assume it's next year
if (current_date - date_obj).days > 30:
- date_obj = datetime.strptime(f"{day} {month} {current_year + 1}", "%d %B %Y")
+ date_obj = datetime.strptime(
+ f"{day} {month} {current_year + 1}", "%d %B %Y"
+ )
except ValueError:
# Try with next year
- date_obj = datetime.strptime(f"{day} {month} {current_year + 1}", "%d %B %Y")
-
+ date_obj = datetime.strptime(
+ f"{day} {month} {current_year + 1}", "%d %B %Y"
+ )
+
collection_date = date_obj.strftime(date_format)
-
+
dict_data = {
"type": bin_type,
"collectionDate": collection_date,
}
bindata["bins"].append(dict_data)
except Exception as e:
- print(f"Error parsing date '{date_text}': {e}")
+ print("Collection date parsing failed " f"({type(e).__name__})")
continue
-
+
# Sort by collection date
bindata["bins"].sort(
key=lambda x: datetime.strptime(x.get("collectionDate"), "%d/%m/%Y")
)
-
+
finally:
if driver:
driver.quit()
diff --git a/uk_bin_collection/uk_bin_collection/councils/FalkirkCouncil.py b/uk_bin_collection/uk_bin_collection/councils/FalkirkCouncil.py
index 977d7ae2b9..5e2f263529 100644
--- a/uk_bin_collection/uk_bin_collection/councils/FalkirkCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/FalkirkCouncil.py
@@ -35,8 +35,8 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Loop through the dates for each collection type
for date in collection_dates:
- print(f"Bin Type: {bin_type}")
- print(f"Collection Date: {date}")
+ print("Collection type parsed")
+ print("Collection date parsed")
dict_data = {
"type": bin_type,
diff --git a/uk_bin_collection/uk_bin_collection/councils/FifeCouncil.py b/uk_bin_collection/uk_bin_collection/councils/FifeCouncil.py
index 6fdd028f1e..38d7d35ad5 100644
--- a/uk_bin_collection/uk_bin_collection/councils/FifeCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/FifeCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -21,22 +19,33 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
"""
Parse bin collection data for a given postcode and house identifier by driving the council bin-calendar web UI and returning structured collection entries.
-
+
Parameters:
page (str): Unused; kept for API compatibility.
postcode (str, in kwargs): The postcode to search for.
paon (str, in kwargs): The property identifier (house number or name) used to pick the best address option from the dropdown.
web_driver (str, optional, in kwargs): WebDriver backend identifier passed to the webdriver factory.
headless (bool, optional, in kwargs): Whether to run the browser in headless mode.
-
+
Returns:
dict: A dictionary with a "bins" key containing a list of entries. Each entry is a dict with:
- "type": the collection colour/name extracted from the image alt text (or None if missing).
- "collectionDate": the collection date string formatted according to the module's date_format.
-
+
Raises:
ValueError: If the provided paon cannot be matched to any dropdown address or if the collections table cannot be found.
"""
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
# Get and check UPRN
@@ -92,9 +101,9 @@ def _best_option():
# Prefer exact contains on visible text; fallback to casefold contains
"""
Selects the first dropdown option whose visible text contains the normalized PAON.
-
+
Performs a case-insensitive containment check using the precomputed `paon_norm` against each option's visible text and returns the first match.
-
+
Returns:
`WebElement` of the first matching option if found, `None` otherwise.
"""
@@ -159,11 +168,11 @@ def _best_option():
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
# This block ensures that the driver is closed regardless of an exception
if driver:
driver.quit()
- return bindata
\ No newline at end of file
+ return bindata
diff --git a/uk_bin_collection/uk_bin_collection/councils/ForestOfDeanDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ForestOfDeanDistrictCouncil.py
index bec7e856b2..ba37e7dae3 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ForestOfDeanDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ForestOfDeanDistrictCouncil.py
@@ -1,24 +1,40 @@
+from __future__ import annotations
+
import time
import re
from datetime import datetime, timedelta
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
_BIN_TYPES = {
- "180 litre refuse", "black recycling box", "blue bag", "white bag",
- "outdoor food caddy", "indoor food caddy", "garden waste",
- "240 litre refuse", "recycling box", "food caddy",
+ "180 litre refuse",
+ "black recycling box",
+ "blue bag",
+ "white bag",
+ "outdoor food caddy",
+ "indoor food caddy",
+ "garden waste",
+ "240 litre refuse",
+ "recycling box",
+ "food caddy",
}
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import WebDriverWait
+
driver = None
try:
page = "https://community.fdean.gov.uk/s/waste-collection-enquiry"
@@ -26,7 +42,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
house_number = kwargs.get("paon")
postcode = kwargs.get("postcode")
- if house_number and postcode and postcode.upper() not in house_number.upper():
+ if (
+ house_number
+ and postcode
+ and postcode.upper() not in house_number.upper()
+ ):
full_address = f"{house_number}, {postcode}"
else:
full_address = house_number or postcode or ""
@@ -41,9 +61,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
time.sleep(8)
address_entry_field = wait.until(
- EC.presence_of_element_located(
- (By.XPATH, "//input[@role='combobox']")
- )
+ EC.presence_of_element_located((By.XPATH, "//input[@role='combobox']"))
)
address_entry_field.click()
@@ -52,9 +70,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
time.sleep(4)
wait.until(
- EC.element_to_be_clickable(
- (By.XPATH, "//li[@role='presentation']")
- )
+ EC.element_to_be_clickable((By.XPATH, "//li[@role='presentation']"))
)
all_opts = driver.find_elements(By.XPATH, "//li[@role='presentation']")
if len(all_opts) > 1:
@@ -72,7 +88,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
for _ in range(8):
time.sleep(5)
- if driver.find_elements(By.XPATH, "//*[contains(text(), 'Collection day')]"):
+ if driver.find_elements(
+ By.XPATH, "//*[contains(text(), 'Collection day')]"
+ ):
break
soup = BeautifulSoup(driver.page_source, features="html.parser")
@@ -87,19 +105,21 @@ def parse_data(self, page: str, **kwargs) -> dict:
td = row.find("td")
if not th or not td:
continue
- container_type = (
- th.get("data-cell-value", "").strip()
- or th.get_text(strip=True)
- )
- raw_date = (
- td.get("data-cell-value", "").strip()
- or td.get_text(strip=True)
+ container_type = th.get(
+ "data-cell-value", ""
+ ).strip() or th.get_text(strip=True)
+ raw_date = td.get("data-cell-value", "").strip() or td.get_text(
+ strip=True
)
if container_type and raw_date:
- data["bins"].append({
- "type": container_type,
- "collectionDate": self._parse_date(raw_date, current_year),
- })
+ data["bins"].append(
+ {
+ "type": container_type,
+ "collectionDate": self._parse_date(
+ raw_date, current_year
+ ),
+ }
+ )
except (ValueError, AttributeError):
continue
else:
@@ -109,13 +129,17 @@ def parse_data(self, page: str, **kwargs) -> dict:
if line.lower() in _BIN_TYPES and i + 1 < len(lines):
raw_date = lines[i + 1]
if self._looks_like_date(raw_date):
- data["bins"].append({
- "type": line,
- "collectionDate": self._parse_date(raw_date, current_year),
- })
+ data["bins"].append(
+ {
+ "type": line,
+ "collectionDate": self._parse_date(
+ raw_date, current_year
+ ),
+ }
+ )
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
@@ -125,7 +149,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
@staticmethod
def _looks_like_date(text):
t = text.lower().strip()
- return t in ("today", "tomorrow") or bool(re.match(r"^(mon|tue|wed|thu|fri|sat|sun)", t))
+ return t in ("today", "tomorrow") or bool(
+ re.match(r"^(mon|tue|wed|thu|fri|sat|sun)", t)
+ )
@staticmethod
def _parse_date(raw_date, current_year):
diff --git a/uk_bin_collection/uk_bin_collection/councils/GatesheadCouncil.py b/uk_bin_collection/uk_bin_collection/councils/GatesheadCouncil.py
index 2b20e38e89..702fd268a7 100644
--- a/uk_bin_collection/uk_bin_collection/councils/GatesheadCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/GatesheadCouncil.py
@@ -1,8 +1,7 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -17,6 +16,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -177,7 +186,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
data["bins"].append(dict_data)
except Exception as e:
- print(f"Error parsing date for row: {e}")
+ print(f"Error parsing date for row: {type(e).__name__}")
continue
data["bins"].sort(
@@ -185,7 +194,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/GlasgowCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/GlasgowCityCouncil.py
index 1737719dba..eaad30c1eb 100644
--- a/uk_bin_collection/uk_bin_collection/councils/GlasgowCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/GlasgowCityCouncil.py
@@ -55,7 +55,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
collection_date = (datetime.today() + timedelta(days=1)).strftime(
date_format
)
- print(collection_date)
+ print("Collection date parsed")
else:
collection_date = datetime.strptime(
collection_date,
diff --git a/uk_bin_collection/uk_bin_collection/councils/GloucesterCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/GloucesterCityCouncil.py
index 2825e3e928..0363038de9 100644
--- a/uk_bin_collection/uk_bin_collection/councils/GloucesterCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/GloucesterCityCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -21,6 +19,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
page = "https://gloucester-self.achieveservice.com/service/Bins___Check_your_bin_day"
@@ -112,7 +121,7 @@ def is_a_collection_date(t):
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/GreatYarmouthBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/GreatYarmouthBoroughCouncil.py
index ed4f655d88..8829e691e5 100644
--- a/uk_bin_collection/uk_bin_collection/councils/GreatYarmouthBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/GreatYarmouthBoroughCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +18,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
user_uprn = kwargs.get("uprn")
@@ -106,7 +115,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
try:
# Parse the date text
- date_obj = datetime.strptime(date_text + " " + str(datetime.today().year), "%A %d %B %Y")
+ date_obj = datetime.strptime(
+ date_text + " " + str(datetime.today().year), "%A %d %B %Y"
+ )
if date_obj.date() < datetime.today().date():
continue # Skip past dates
except ValueError:
@@ -131,21 +142,19 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
):
next_collections[name] = bin_entry
- print(
- f"Found next collection for {name}: {formatted_date}"
- ) # Debug output
+ print("Next collection parsed successfully")
break
# Add the next collections to the bin_data
bin_data["bins"] = list(next_collections.values())
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
driver.quit()
print("\nFinal bin data:")
- print(bin_data) # Debug output
+ print(f"Parsed {len(bin_data.get('bins', []))} collections")
return bin_data
diff --git a/uk_bin_collection/uk_bin_collection/councils/GuildfordCouncil.py b/uk_bin_collection/uk_bin_collection/councils/GuildfordCouncil.py
index a2ada06efc..1d906d3bdc 100644
--- a/uk_bin_collection/uk_bin_collection/councils/GuildfordCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/GuildfordCouncil.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
# This script pulls (in one hit) the data from Bromley Council Bins Data
import datetime
import re
@@ -6,11 +8,6 @@
import requests
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -25,6 +22,18 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
postcode = kwargs.get("postcode")
@@ -140,7 +149,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
data["bins"].append(dict_data)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/HaltonBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/HaltonBoroughCouncil.py
index 792ba23fa2..e19ff48dfa 100644
--- a/uk_bin_collection/uk_bin_collection/councils/HaltonBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/HaltonBoroughCouncil.py
@@ -1,12 +1,9 @@
+from __future__ import annotations
+
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -24,20 +21,32 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
"""
Retrieve bin collection dates for a property from Halton Council's waste service.
-
+
This method loads the council's waste service page, submits the provided property identifier and postcode, parses the resulting collection schedule, and returns structured bin collection entries.
-
+
Parameters:
paon (str, via kwargs): Property identifier â house number or property name.
postcode (str, via kwargs): Property postcode.
web_driver (str or selenium.webdriver, via kwargs): Optional webdriver backend identifier or instance passed to create_webdriver.
headless (bool, via kwargs): If True, the browser is created in headless mode.
-
+
Returns:
dict: A dictionary with a single key "bins" containing a list of collection entries. Each entry is a dict with:
- "type" (str): Waste type name (capitalized).
- "collectionDate" (str): Collection date formatted as "DD/MM/YYYY".
"""
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -155,11 +164,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
# This block ensures that the driver is closed regardless of an exception
if driver:
driver.quit()
- return data
\ No newline at end of file
+ return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/HerefordshireCouncil.py b/uk_bin_collection/uk_bin_collection/councils/HerefordshireCouncil.py
index da561c9815..4b1e7c9021 100644
--- a/uk_bin_collection/uk_bin_collection/councils/HerefordshireCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/HerefordshireCouncil.py
@@ -48,7 +48,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
soup = BeautifulSoup(page.text, "html.parser")
soup.prettify
- checkValid = any("Your next collection days" in h2.get_text() for h2 in soup.find_all("h2"))
+ checkValid = any(
+ "Your next collection days" in h2.get_text() for h2 in soup.find_all("h2")
+ )
if not checkValid:
raise ValueError("Address/UPRN not found")
@@ -74,12 +76,14 @@ def parse_data(self, page: str, **kwargs) -> dict:
next_date = li.get_text(strip=True).replace(" (next collection)", "")
- logging.info(f"Bin type: {bin_type} - Collection date: {next_date}")
+ logging.info("Collection entry parsed successfully")
data["bins"].append(
{
"type": bin_type,
- "collectionDate": datetime.strptime(next_date, "%A %d %B %Y").strftime(date_format),
+ "collectionDate": datetime.strptime(
+ next_date, "%A %d %B %Y"
+ ).strftime(date_format),
}
)
diff --git a/uk_bin_collection/uk_bin_collection/councils/HertsmereBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/HertsmereBoroughCouncil.py
index 2e1b55d796..35cb19a8c8 100644
--- a/uk_bin_collection/uk_bin_collection/councils/HertsmereBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/HertsmereBoroughCouncil.py
@@ -1,12 +1,10 @@
+from __future__ import annotations
+
import re
import time
import requests
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -21,6 +19,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
user_paon = kwargs.get("paon")
@@ -51,11 +60,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Wait for results to appear
WebDriverWait(driver, 10).until(
- EC.presence_of_element_located(
- (By.CSS_SELECTOR, "ul.result_list li")
- )
+ EC.presence_of_element_located((By.CSS_SELECTOR, "ul.result_list li"))
)
-
+
# Use JavaScript to click the correct address
# Add space after house number to match exactly (e.g., "1 " not "10", "11", etc.)
driver.execute_script(f"""
@@ -96,16 +103,14 @@ def parse_data(self, page: str, **kwargs) -> dict:
table_data = []
for row in table.find("tbody").find_all("tr"):
# Extract cell data from each tag
- row_data = [
- cell.get_text(strip=True) for cell in row.find_all("td")
- ]
+ row_data = [cell.get_text(strip=True) for cell in row.find_all("td")]
table_data.append(row_data)
# The table structure is: [Bin Type, Collection Day, Round Code]
# All bins are collected on the same day (e.g., "Thursday")
if not table_data or len(table_data[0]) < 2:
raise Exception("Unable to parse collection schedule from table.")
-
+
collection_day = table_data[0][1] # e.g., "Thursday"
# Extract all bin types
@@ -116,7 +121,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Calculate next collection dates based on the collection day
from datetime import datetime, timedelta
-
+
days_of_week = [
"Monday",
"Tuesday",
@@ -138,7 +143,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
next_day = today + timedelta(days=days_until_target)
# Generate collection dates for the next 12 weeks (all bins collected weekly)
- all_dates = get_dates_every_x_days(next_day, 7, 12) # 12 collections, every 7 days
+ all_dates = get_dates_every_x_days(
+ next_day, 7, 12
+ ) # 12 collections, every 7 days
# Assign all bin types to each collection date
for date in all_dates:
@@ -155,7 +162,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/HighPeakCouncil.py b/uk_bin_collection/uk_bin_collection/councils/HighPeakCouncil.py
index 32e356751d..dcf8d0b666 100644
--- a/uk_bin_collection/uk_bin_collection/councils/HighPeakCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/HighPeakCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
import re
from datetime import datetime
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +19,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import WebDriverWait
+
driver = None
try:
user_postcode = kwargs.get("postcode")
@@ -98,7 +107,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
key=lambda x: datetime.strptime(x["collectionDate"], date_format)
)
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/Hillingdon.py b/uk_bin_collection/uk_bin_collection/councils/Hillingdon.py
index 09b8d3d410..6240bbf1d3 100644
--- a/uk_bin_collection/uk_bin_collection/councils/Hillingdon.py
+++ b/uk_bin_collection/uk_bin_collection/councils/Hillingdon.py
@@ -1,19 +1,11 @@
+from __future__ import annotations
+
import json
from datetime import datetime, timedelta
from typing import Any, Dict
from bs4 import BeautifulSoup
from dateutil.parser import parse
-from selenium.common.exceptions import (
- NoSuchElementException,
- StaleElementReferenceException,
- TimeoutException,
-)
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.remote.webdriver import WebDriver
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -86,17 +78,37 @@ def get_bank_holiday_changes(driver: WebDriver) -> Dict[str, str]:
)
changes[normal_date] = revised_date
except Exception as e:
- print(f"Error parsing dates: {e}")
+ print(f"Error parsing dates: {type(e).__name__}")
continue
except Exception as e:
- print(f"An error occurred while fetching bank holiday changes: {e}")
+ print(
+ "An error occurred while fetching bank holiday changes "
+ f"({type(e).__name__})."
+ )
return changes
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs: Any) -> Dict[str, Any]:
+ global By, EC, Keys, NoSuchElementException, StaleElementReferenceException, TimeoutException, WebDriver, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.common.exceptions import (
+ NoSuchElementException,
+ StaleElementReferenceException,
+ TimeoutException,
+ )
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.remote.webdriver import WebDriver
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data: Dict[str, Any] = {"bins": []}
@@ -264,7 +276,7 @@ def parse_data(self, page: str, **kwargs: Any) -> Dict[str, Any]:
}
)
except Exception as e:
- print(f"Error processing item: {e}")
+ print(f"Error processing item: {type(e).__name__}")
continue
# Get bank holiday changes
@@ -276,13 +288,11 @@ def parse_data(self, page: str, **kwargs: Any) -> Dict[str, Any]:
original_date = bin_data["collectionDate"]
if original_date in bank_holiday_changes:
new_date = bank_holiday_changes[original_date]
- print(
- f"Bank holiday change: {bin_data['type']} collection moved from {original_date} to {new_date}"
- )
+ print("Applied a bank-holiday collection-date adjustment.")
bin_data["collectionDate"] = new_date
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
@@ -290,6 +300,6 @@ def parse_data(self, page: str, **kwargs: Any) -> Dict[str, Any]:
# Print the final data dictionary for debugging
print("\nFinal data dictionary:")
- print(json.dumps(data, indent=2))
+ print(f"Parsed {len(data.get('bins', []))} collections")
return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/HyndburnBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/HyndburnBoroughCouncil.py
index 4fbed58d2f..901d0c984c 100644
--- a/uk_bin_collection/uk_bin_collection/councils/HyndburnBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/HyndburnBoroughCouncil.py
@@ -1,10 +1,8 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +17,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
user_postcode = kwargs.get("postcode")
@@ -123,19 +132,19 @@ def parse_data(self, page: str, **kwargs) -> dict:
bin_info = {"type": bin_type, "collectionDate": formatted_date}
bin_data["bins"].append(bin_info)
except ValueError as e:
- print(f"Error parsing date {collection_date_string}: {e}")
+ print("Collection date parsing failed " f"({type(e).__name__})")
continue
if not bin_data["bins"]:
raise ValueError("No collection data found")
- print(bin_data)
+ print(f"Parsed {len(bin_data.get('bins', []))} collections")
return bin_data
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/IsleOfAngleseyCouncil.py b/uk_bin_collection/uk_bin_collection/councils/IsleOfAngleseyCouncil.py
index 9a97c9ec62..cf54cb59a0 100644
--- a/uk_bin_collection/uk_bin_collection/councils/IsleOfAngleseyCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/IsleOfAngleseyCouncil.py
@@ -27,7 +27,7 @@ class CouncilClass(AbstractGetBinDataClass):
def __init__(self):
"""
Initialize the CouncilClass instance.
-
+
Calls the superclass initializer, creates a requests.Session assigned to self._session, and sets self._have_session to False to indicate that an authenticated session has not yet been established.
"""
super().__init__()
@@ -37,9 +37,9 @@ def __init__(self):
def _initialise_session(self) -> None:
"""
Establish an authenticated session with the remote service and mark the instance as having a valid session.
-
+
Performs an HTTP GET to the configured session endpoint and verifies the JSON response contains an "auth-session" indicator. On success sets the instance flag that a session is available.
-
+
Raises:
requests.HTTPError: If the session request returned an HTTP error status.
ValueError: If the response JSON cannot be decoded or does not contain an "auth-session" key.
@@ -58,14 +58,14 @@ def _initialise_session(self) -> None:
def _run_lookup(self, lookup_id: str, payload: dict) -> dict:
"""
Run a lookup request and return the lookup's transformed rows data.
-
+
Parameters:
lookup_id (str): Lookup identifier appended as the `id` query parameter to the lookup endpoint.
payload (dict): JSON body sent with the POST request.
-
+
Returns:
The `rows_data` value extracted from the response's `integration.transformed` object.
-
+
Raises:
ValueError: If the response is not valid JSON or does not contain the expected `integration.transformed.rows_data` structure.
"""
@@ -83,20 +83,20 @@ def _run_lookup(self, lookup_id: str, payload: dict) -> dict:
except requests.exceptions.JSONDecodeError as e:
raise ValueError("Failed to decode lookup response as JSON") from e
except KeyError as e:
- logger.debug(f"Lookup response content: {response.text}")
+ logger.debug("Lookup response content omitted after schema mismatch.")
raise ValueError("Unexpected response structure from lookup") from e
def _get_uprn_from_postcode_and_paon(self, postcode: str, paon: str) -> str:
"""
Return the UPRN for the address at the given postcode matching the provided PAON.
-
+
Parameters:
postcode (str): Postcode to search.
paon (str): Primary Addressable Object Name â house number or name to match.
-
+
Returns:
str: The matching UPRN.
-
+
Raises:
ValueError: If no addresses are found for the postcode or no address matches the PAON.
"""
@@ -135,7 +135,7 @@ def _get_uprn_from_postcode_and_paon(self, postcode: str, paon: str) -> str:
def parse_data(self, page: str, **kwargs) -> dict:
"""
Obtain the bin collection schedule for a property identified by UPRN or by postcode and house number/name.
-
+
Parameters:
page (str): Unused but required by the interface.
**kwargs: Identification parameters â provide either:
@@ -143,10 +143,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
OR
postcode (str): The property's postcode.
paon (str) or number (str): The property's primary addressable object name/number (required with postcode).
-
+
Returns:
dict: A dictionary with a "bins" key mapping to a list of collection entries, each containing `type` and `collectionDate`.
-
+
Raises:
ValueError: If required identification parameters are missing, input validation fails, or the remote lookup cannot resolve the property.
"""
@@ -191,16 +191,16 @@ def parse_data(self, page: str, **kwargs) -> dict:
def _extract_bin_data(schedule: dict) -> dict:
"""
Convert a schedule response into the standardized bins list.
-
+
Parameters:
schedule (dict): Mapping of schedule rows returned by the API where each value is a dict containing at least the keys "Service" and "Date".
-
+
Returns:
dict: {"bins": [ {"type": , "collectionDate": } , ... ]}
-
+
Raises:
ValueError: If `schedule` is empty.
-
+
Notes:
- Rows missing required fields or containing unparsable dates are skipped and a warning is logged.
"""
@@ -232,6 +232,6 @@ def _extract_bin_data(schedule: dict) -> dict:
}
)
except (KeyError, ValueError) as e:
- logger.warning(f"Skipping invalid row: {e}", exc_info=True)
+ logger.warning("Skipping invalid row (%s).", type(e).__name__)
- return {"bins": bins}
\ No newline at end of file
+ return {"bins": bins}
diff --git a/uk_bin_collection/uk_bin_collection/councils/IsleOfWightCouncil.py b/uk_bin_collection/uk_bin_collection/councils/IsleOfWightCouncil.py
index e45e131177..6ccc8e95e5 100644
--- a/uk_bin_collection/uk_bin_collection/councils/IsleOfWightCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/IsleOfWightCouncil.py
@@ -13,22 +13,32 @@
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
RECYCLING_COLORS = [
- (0.5, 0.0, 1.0, 0.0), # CMYK purple (2024-25 PDF)
- (0.584, 0.757, 0.12), # RGB green (2025-26 PDF)
+ (0.5, 0.0, 1.0, 0.0), # CMYK purple (2024-25 PDF)
+ (0.584, 0.757, 0.12), # RGB green (2025-26 PDF)
]
NON_RECYCLABLE_COLORS = [
- (0.0, 0.0, 0.0, 0.383), # CMYK grey (2024-25 PDF)
- (0.713, 0.713, 0.712), # RGB grey (2025-26 PDF)
+ (0.0, 0.0, 0.0, 0.383), # CMYK grey (2024-25 PDF)
+ (0.713, 0.713, 0.712), # RGB grey (2025-26 PDF)
]
HEADER_BG_COLORS = [
(0.527, 0.323, 0.0, 0.0), # CMYK brown (2024-25 PDF)
- (0.525, 0.628, 0.828), # RGB blue (2025-26 PDF)
+ (0.525, 0.628, 0.828), # RGB blue (2025-26 PDF)
]
COLOR_TOLERANCE = 0.08
MONTHS = [
- "JANUARY", "FEBRUARY", "MARCH", "APRIL", "MAY", "JUNE",
- "JULY", "AUGUST", "SEPTEMBER", "OCTOBER", "NOVEMBER", "DECEMBER",
+ "JANUARY",
+ "FEBRUARY",
+ "MARCH",
+ "APRIL",
+ "MAY",
+ "JUNE",
+ "JULY",
+ "AUGUST",
+ "SEPTEMBER",
+ "OCTOBER",
+ "NOVEMBER",
+ "DECEMBER",
]
PDF_CACHE_DIR = "/tmp/iow-pdf-cache"
@@ -76,12 +86,14 @@ def _parse_calendar_pdf(pdf_path, collection_day):
for w in words:
upper = w["text"].upper()
if upper in MONTHS:
- month_headers.append({
- "month": MONTHS.index(upper) + 1,
- "x0": w["x0"],
- "top": w["top"],
- "bottom": w.get("bottom", w["top"] + 12),
- })
+ month_headers.append(
+ {
+ "month": MONTHS.index(upper) + 1,
+ "x0": w["x0"],
+ "top": w["top"],
+ "bottom": w.get("bottom", w["top"] + 12),
+ }
+ )
if not month_headers:
raise ValueError("No month headers found in PDF calendar")
@@ -91,8 +103,13 @@ def _parse_calendar_pdf(pdf_path, collection_day):
mh["year"] = start_year if mh["month"] >= first_month else start_year + 1
day_map = {
- "MONDAY": 0, "TUESDAY": 1, "WEDNESDAY": 2,
- "THURSDAY": 3, "FRIDAY": 4, "SATURDAY": 5, "SUNDAY": 6,
+ "MONDAY": 0,
+ "TUESDAY": 1,
+ "WEDNESDAY": 2,
+ "THURSDAY": 3,
+ "FRIDAY": 4,
+ "SATURDAY": 5,
+ "SUNDAY": 6,
}
target_weekday = day_map.get(collection_day.upper())
if target_weekday is None:
@@ -226,9 +243,9 @@ def _select_address(options_text, user_paon):
if user_paon:
paon_lower = user_paon.lower().strip()
for label in options_text:
- if label.lower().startswith(
- paon_lower + ","
- ) or label.lower().startswith(paon_lower + " "):
+ if label.lower().startswith(paon_lower + ",") or label.lower().startswith(
+ paon_lower + " "
+ ):
return label
for label in options_text:
@@ -245,9 +262,7 @@ def _extract_collection_info(html):
collection_day_el = soup.find("strong", string=re.compile(r"Collection Day:"))
if collection_day_el:
collection_day = (
- collection_day_el.parent.get_text()
- .replace("Collection Day:", "")
- .strip()
+ collection_day_el.parent.get_text().replace("Collection Day:", "").strip()
)
else:
raise ValueError("Could not find collection day in page")
@@ -270,10 +285,12 @@ def _build_bins(collection_dates):
for dt, bin_type in collection_dates:
if dt >= today:
- data["bins"].append({
- "type": bin_type,
- "collectionDate": dt.strftime(date_format),
- })
+ data["bins"].append(
+ {
+ "type": bin_type,
+ "collectionDate": dt.strftime(date_format),
+ }
+ )
return data
@@ -286,6 +303,7 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ ensure_selenium_dependencies()
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select, WebDriverWait
@@ -355,14 +373,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
"Referer": "https://digitalservices.iow.gov.uk/wasteday",
}
- pdf_path = _download_pdf_cached(
- pdf_url, cookies=cookies, headers=headers
- )
+ pdf_path = _download_pdf_cached(pdf_url, cookies=cookies, headers=headers)
collection_dates = _parse_calendar_pdf(pdf_path, collection_day)
return _build_bins(collection_dates)
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/KingstonUponThamesCouncil.py b/uk_bin_collection/uk_bin_collection/councils/KingstonUponThamesCouncil.py
index 2807caabae..26c01a430e 100644
--- a/uk_bin_collection/uk_bin_collection/councils/KingstonUponThamesCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/KingstonUponThamesCouncil.py
@@ -1,6 +1,8 @@
+from __future__ import annotations
+
# alternative implementation for retrieving bin data from Kingston Upon Thames Council
# principal URL is https://waste-services.kingston.gov.uk/waste/[uprn]
-# https://www.kingston.gov.uk/bins-and-recycling/collections/check-your-bin-collection-day
+# https://www.kingston.gov.uk/bins-and-recycling/collections/check-your-bin-collection-day
# switched to using Selenium as the htmx elements are not rendered reliably with requests
# updated Jan 2026 due to small website formatting changes
@@ -8,10 +10,6 @@
import re
from bs4 import BeautifulSoup
-from selenium import webdriver
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -27,6 +25,17 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait, webdriver
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium import webdriver
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import WebDriverWait
+
driver = None
try:
@@ -65,7 +74,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
f"Kingston parser: missing dl.govuk-summary-list for {service_name}"
)
- rows = summary_list.find_all("div", {"class": "govuk-summary-list__row"})
+ rows = summary_list.find_all(
+ "div", {"class": "govuk-summary-list__row"}
+ )
for row in rows:
dt = row.find("dt")
if dt and dt.get_text().strip().lower() == "next collection":
@@ -74,9 +85,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
raise ValueError(
f"Kingston parser: missing dd element for 'next collection' in {service_name}"
)
- collection_date = remove_ordinal_indicator_from_date_string(
- dd.get_text()
- ).strip().replace(" (In progress)", "")
+ collection_date = (
+ remove_ordinal_indicator_from_date_string(dd.get_text())
+ .strip()
+ .replace(" (In progress)", "")
+ )
# strip out any text inside of the date string
collection_date = re.sub(
r"\n\s*\(this.*?\)", "", collection_date
@@ -100,7 +113,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/KnowsleyMBCouncil.py b/uk_bin_collection/uk_bin_collection/councils/KnowsleyMBCouncil.py
index 1473d2fe30..c9b96c5754 100644
--- a/uk_bin_collection/uk_bin_collection/councils/KnowsleyMBCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/KnowsleyMBCouncil.py
@@ -1,10 +1,8 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
from datetime import datetime
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -12,6 +10,17 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import WebDriverWait
+
driver = None
try:
bindata = {"bins": []}
@@ -135,7 +144,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/LeedsCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/LeedsCityCouncil.py
index 8237b18bee..fd217e138a 100644
--- a/uk_bin_collection/uk_bin_collection/councils/LeedsCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/LeedsCityCouncil.py
@@ -40,7 +40,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Send GET request
response = requests.get(URI, params=params, headers=headers)
- print(response.content)
+ print(f"Council response received (HTTP {response.status_code}).")
collections = json.loads(response.content)
@@ -59,7 +59,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/LondonBoroughHarrow.py b/uk_bin_collection/uk_bin_collection/councils/LondonBoroughHarrow.py
index 66c8268145..6ae270270c 100644
--- a/uk_bin_collection/uk_bin_collection/councils/LondonBoroughHarrow.py
+++ b/uk_bin_collection/uk_bin_collection/councils/LondonBoroughHarrow.py
@@ -15,13 +15,12 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
-
"""
Fetch bin collection data for a property and return normalized bin types with formatted dates.
-
+
Parameters:
uprn (str): Unique Property Reference Number used to query the council's bin collections endpoint (passed via kwargs).
-
+
Returns:
dict: A dictionary with a "bins" key mapping to a list of collection records. Each record is a dict with:
- "type" (str): Bin type.
@@ -45,7 +44,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
for collection in bin_collection["results"]["collections"]["all"]:
CollectTime = (collection["eventTime"]).split("T")[0]
- print(CollectTime)
+ print("Collection schedule parsed")
dict_data = {
"type": collection["binType"],
@@ -55,4 +54,4 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
bindata["bins"].append(dict_data)
- return bindata
\ No newline at end of file
+ return bindata
diff --git a/uk_bin_collection/uk_bin_collection/councils/LondonBoroughRedbridge.py b/uk_bin_collection/uk_bin_collection/councils/LondonBoroughRedbridge.py
index 1534115b35..2541268cfb 100644
--- a/uk_bin_collection/uk_bin_collection/councils/LondonBoroughRedbridge.py
+++ b/uk_bin_collection/uk_bin_collection/councils/LondonBoroughRedbridge.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
# This script pulls (in one hit) the data from Bromley Council Bins Data
import datetime
import re
@@ -6,11 +8,6 @@
import requests
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -25,6 +22,18 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -96,7 +105,9 @@ def extract_collection_data(collection_div, collection_type, class_prefix):
) # e.g., "February 2026 "
# Parse the date string (format: "04 February 2026 ")
- date_string = f"{collection_date} {collection_month.strip()}"
+ date_string = (
+ f"{collection_date} {collection_month.strip()}"
+ )
try:
# Convert the date string to a datetime object
@@ -121,13 +132,12 @@ def extract_collection_data(collection_div, collection_type, class_prefix):
except ValueError as e:
# Handle the case where the date format is invalid
print(
- f"Error parsing date '{date_string}' for {collection_type}: {e}"
+ "Collection date parsing failed "
+ f"({type(e).__name__})"
)
# Extract Refuse collection data
- refuse_div = soup.find(
- "div", class_="container-fluid RegularCollectionDay"
- )
+ refuse_div = soup.find("div", class_="container-fluid RegularCollectionDay")
if refuse_div and refuse_div.find(class_="refuse-container"):
extract_collection_data(refuse_div, "Refuse", "refuse")
@@ -150,7 +160,7 @@ def extract_collection_data(collection_div, collection_type, class_prefix):
# Print the extracted data
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/MaidstoneBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/MaidstoneBoroughCouncil.py
index ede74bde7a..af8dae7292 100644
--- a/uk_bin_collection/uk_bin_collection/councils/MaidstoneBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/MaidstoneBoroughCouncil.py
@@ -1,12 +1,9 @@
+from __future__ import annotations
+
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +17,18 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
page = "https://my.maidstone.gov.uk/service/Find-your-bin-day"
@@ -97,11 +106,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
)
except Exception as inner_e:
- print(f"Skipping one panel due to error: {inner_e}")
+ print(f"Skipping one panel due to error: {type(inner_e).__name__}")
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/ManchesterCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ManchesterCityCouncil.py
index dd5fa7104b..082cbde233 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ManchesterCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ManchesterCityCouncil.py
@@ -64,12 +64,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
r.raise_for_status()
result = r.json()
- print(result["data"])
+ print("Council response decoded successfully.")
for key, value in result["data"].items():
if key.startswith("ahtm_dates_"):
- print(key)
- print(value)
+ print("Collection field parsed")
+ print("Collection field value parsed")
dates_list = [
datetime.strptime(date.strip(), "%d/%m/%Y %H:%M:%S").date()
diff --git a/uk_bin_collection/uk_bin_collection/councils/MeltonBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/MeltonBoroughCouncil.py
index 9aed04a3a8..a6a060ce9d 100644
--- a/uk_bin_collection/uk_bin_collection/councils/MeltonBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/MeltonBoroughCouncil.py
@@ -77,6 +77,6 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
data["bins"].append(dict_data)
- print(json.dumps(data, indent=2))
+ print(f"Parsed {len(data.get('bins', []))} collections")
return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/MidSuffolkDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/MidSuffolkDistrictCouncil.py
index 2e357b2759..501c4dd1f9 100644
--- a/uk_bin_collection/uk_bin_collection/councils/MidSuffolkDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/MidSuffolkDistrictCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +19,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
web_driver = kwargs.get("web_driver")
@@ -152,7 +161,7 @@ def _populated_select(d):
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py
index ccfa4cfbb7..14f4938727 100644
--- a/uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/MidSussexDistrictCouncil.py
@@ -49,8 +49,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
alink = soup.find(
"a",
- string=lambda s: s
- and "View my collections" in s,
+ string=lambda s: s and "View my collections" in s,
)
if alink is None:
@@ -114,5 +113,5 @@ def parse_data(self, page: str, **kwargs) -> dict:
return bindata
except Exception as e:
- logging.error(f"An error occurred: {e}")
+ logging.error(f"An error occurred: {type(e).__name__}")
raise
diff --git a/uk_bin_collection/uk_bin_collection/councils/MidUlsterDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/MidUlsterDistrictCouncil.py
index 88f9237f4e..69debd91ac 100644
--- a/uk_bin_collection/uk_bin_collection/councils/MidUlsterDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/MidUlsterDistrictCouncil.py
@@ -1,12 +1,10 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
# import selenium keys
-from selenium.webdriver.common.keys import Keys
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -21,6 +19,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+ from selenium.webdriver.common.keys import Keys
+
driver = None
try:
user_postcode = kwargs.get("postcode")
@@ -106,7 +115,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
else:
collection_date = None
except Exception as e:
- print(f"Failed to parse date: {e}")
+ print(f"Failed to parse date: {type(e).__name__}")
collection_date = None
# 2. Extract bin types
@@ -130,7 +139,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/MiddlesbroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/MiddlesbroughCouncil.py
index bd4bc6e02b..c471aad043 100644
--- a/uk_bin_collection/uk_bin_collection/councils/MiddlesbroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/MiddlesbroughCouncil.py
@@ -12,14 +12,14 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
"""
Retrieve upcoming bin collection types and the next collection date for the provided address (paon) within the Middlesbrough service.
-
+
Queries the Recollect API to resolve the provided address to a place_id, fetches the place calendar payload, extracts the "Next Collection" date and associated bin types, and returns them in a normalized structure. If the address cannot be resolved to a place_id, the function returns None.
-
+
Parameters:
page (str): Page or URL context (not used by this implementation).
kwargs:
paon (str): The primary addressable object name/number (e.g., house number or name). Required to look up the address.
-
+
Returns:
dict: A dictionary with the key "bins" mapping to a list of collection entries:
{
@@ -137,5 +137,5 @@ def extract_next_collection(payload: dict):
return data
except Exception as e:
- print(f"An error occurred: {e}")
- raise
\ No newline at end of file
+ print(f"An error occurred: {type(e).__name__}")
+ raise
diff --git a/uk_bin_collection/uk_bin_collection/councils/MidlothianCouncil.py b/uk_bin_collection/uk_bin_collection/councils/MidlothianCouncil.py
index 837ad1ad3e..e81577fb05 100644
--- a/uk_bin_collection/uk_bin_collection/councils/MidlothianCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/MidlothianCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
import time
from datetime import datetime
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support.ui import Select, WebDriverWait
-from selenium.webdriver.support import expected_conditions as EC
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -34,6 +33,16 @@ class CouncilClass(AbstractGetBinDataClass):
}
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+ from selenium.webdriver.support import expected_conditions as EC
+
house_identifier = (kwargs.get("paon") or kwargs.get("number") or "").strip()
user_postcode = kwargs.get("postcode")
web_driver = kwargs.get("web_driver")
@@ -51,11 +60,13 @@ def parse_data(self, page: str, **kwargs) -> dict:
# undetected_chromedriver in non-headless mode via the Xvfb display
# available on the VPS.
import os
+
driver = None
try:
if os.environ.get("DISPLAY") and web_driver is None:
try:
import undetected_chromedriver as uc
+
uc_opts = uc.ChromeOptions()
uc_opts.add_argument("--no-sandbox")
uc_opts.add_argument("--disable-dev-shm-usage")
@@ -126,9 +137,7 @@ def dropdown_populated(d):
}
)
- bins.sort(
- key=lambda x: datetime.strptime(x["collectionDate"], date_format)
- )
+ bins.sort(key=lambda x: datetime.strptime(x["collectionDate"], date_format))
return {"bins": bins}
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/NeathPortTalbotCouncil.py b/uk_bin_collection/uk_bin_collection/councils/NeathPortTalbotCouncil.py
index c358b1aa9d..1d72cb19c5 100644
--- a/uk_bin_collection/uk_bin_collection/councils/NeathPortTalbotCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/NeathPortTalbotCouncil.py
@@ -1,10 +1,8 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +17,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -112,8 +121,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
if date_text != "Bank Holidays":
try:
bin_date = datetime.strptime(
- date_text
- .removesuffix("(Today)")
+ date_text.removesuffix("(Today)")
.removesuffix("(Tomorrow)")
.replace(" ", " ")
+ " "
@@ -128,7 +136,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
},
):
if bin_date and bin_type_wrapper:
- bin_type = bin_type_wrapper.find("a").get_text(strip=True)
+ bin_type = bin_type_wrapper.find("a").get_text(
+ strip=True
+ )
bin_type += (
" ("
+ bin_type_wrapper.find("span").get_text(strip=True)
@@ -148,7 +158,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/NewForestCouncil.py b/uk_bin_collection/uk_bin_collection/councils/NewForestCouncil.py
index b88ba8033f..35e40244b0 100644
--- a/uk_bin_collection/uk_bin_collection/councils/NewForestCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/NewForestCouncil.py
@@ -1,14 +1,9 @@
+from __future__ import annotations
+
import logging
import time
from bs4 import BeautifulSoup
-from selenium import webdriver
-from selenium.common.exceptions import NoSuchElementException
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -42,7 +37,7 @@ def get_legacy_bins(self, page: str) -> []:
"collectionDate": next_collection,
}
)
- logging.info(f"Rubbish and Recycling: {str(next_collection)}")
+ logging.info("Rubbish and recycling date parsed.")
# Glass collection
glass_collection = soup.find("span", class_="CTID-78-_ eb-78-textControl")
@@ -54,7 +49,7 @@ def get_legacy_bins(self, page: str) -> []:
legacy_bins.append(
{"type": "Glass collection", "collectionDate": match.group(1)}
)
- logging.info(f"Glass: {str(match.group(1))}")
+ logging.info("Glass collection date parsed.")
# Garden waste
garden_waste = soup.find("div", class_="eb-2HIpCnWC-Override-EditorInput")
@@ -64,12 +59,26 @@ def get_legacy_bins(self, page: str) -> []:
legacy_bins.append(
{"type": "Garden waste", "collectionDate": match.group(1)}
)
- logging.info(f"Garden: {str(match.group(1))}")
+ logging.info("Garden-waste collection date parsed.")
# return bins
return legacy_bins
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, NoSuchElementException, Select, WebDriverWait, webdriver
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium import webdriver
+ from selenium.common.exceptions import NoSuchElementException
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
bins = []
@@ -106,7 +115,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
"arguments[0].scrollIntoView();", input_element_postcode
)
- logging.info(f"Entering postcode '{str(user_postcode)}'")
+ logging.info("Entering the configured postcode.")
# Force the value through the DOM cos send_keys just don't work for some reason :(
driver.execute_script(
f"arguments[0].value='{str(user_postcode)}'", input_element_postcode
@@ -193,7 +202,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
return {"bins": bins}
except Exception as e:
- logging.error(f"An error occurred: {e}")
+ logging.error(f"An error occurred: {type(e).__name__}")
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/NewportCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/NewportCityCouncil.py
index d90a66bd97..1c8469bdab 100644
--- a/uk_bin_collection/uk_bin_collection/councils/NewportCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/NewportCityCouncil.py
@@ -32,10 +32,10 @@ class CouncilClass(AbstractGetBinDataClass):
def encode_body(self, newport_input: NewportInput):
"""
Encrypt a NewportInput dataclass using AES-CBC and encode the resulting ciphertext as a hex string.
-
+
Parameters:
newport_input (NewportInput): Dataclass instance to serialize to JSON and encrypt. The instance is converted to a dict via `asdict()` before serialization.
-
+
Returns:
str: Hex-encoded AES-CBC ciphertext of the JSON-serialized input. Encryption uses the module-level `key_hex` and `iv_hex` values and applies PKCS#7 padding.
"""
@@ -56,13 +56,12 @@ def encode_body(self, newport_input: NewportInput):
return ciphertext.hex()
def decode_response(self, hex_input: str):
-
"""
Decrypts a hex-encoded AES-CBC ciphertext and returns the parsed JSON payload.
-
+
Parameters:
hex_input (str): Hex-encoded AES-CBC ciphertext to decrypt.
-
+
Returns:
The Python object produced by JSON decoding the decrypted UTF-8 plaintext (typically a dict).
"""
@@ -84,13 +83,13 @@ def decode_response(self, hex_input: str):
def parse_data(self, _: str, **kwargs) -> dict:
"""
Fetch collection-day information for a given UPRN and return it as a normalized bins dictionary.
-
+
Parameters:
_: str
Unused placeholder parameter kept for signature compatibility.
kwargs:
uprn (str): Unique Property Reference Number to query; this value is validated before use.
-
+
Returns:
dict: A dictionary with a "bins" key containing a list of mappings:
- "type": the bin type string from the service response.
@@ -116,10 +115,12 @@ def parse_data(self, _: str, **kwargs) -> dict:
)
output = response.text
-
+
# Check if API returned HTML error page instead of encrypted data
- if output.strip().startswith('<'):
- raise ValueError(f"API returned HTML error page instead of encrypted data. Status: {response.status_code}")
+ if output.strip().startswith("<"):
+ raise ValueError(
+ f"API returned HTML error page instead of encrypted data. Status: {response.status_code}"
+ )
decoded_bins = self.decode_response(output)
data: dict[str, list[dict[str, str]]] = {}
@@ -135,7 +136,7 @@ def parse_data(self, _: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
- return data
\ No newline at end of file
+ return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/NorthDevonCountyCouncil.py b/uk_bin_collection/uk_bin_collection/councils/NorthDevonCountyCouncil.py
index 992353c682..952692ae41 100644
--- a/uk_bin_collection/uk_bin_collection/councils/NorthDevonCountyCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/NorthDevonCountyCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
from time import sleep
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -18,6 +17,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
user_uprn = kwargs.get("uprn")
@@ -143,14 +152,17 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
data["bins"].append(dict_data)
except Exception as e:
- print(f"Skipping invalid date '{full_date}': {e}")
+ print(
+ "Skipping an invalid collection date "
+ f"({type(e).__name__})"
+ )
data["bins"].sort(
key=lambda x: datetime.strptime(x.get("collectionDate"), date_format)
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/NorthEastDerbyshireDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/NorthEastDerbyshireDistrictCouncil.py
index 4980010bad..bde83ad7cd 100644
--- a/uk_bin_collection/uk_bin_collection/councils/NorthEastDerbyshireDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/NorthEastDerbyshireDistrictCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
from datetime import datetime
from time import sleep
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +17,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
page = (
@@ -97,7 +106,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
bin_types = bin_label.get_text().strip() if bin_label else None
if collection_date and bin_types:
- print(f"Found collection: {collection_date} - {bin_types}")
+ print("Collection entry parsed")
# Parse date once
formatted_date = datetime.strptime(
@@ -115,10 +124,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
print(f"Found {len(data['bins'])} collections")
- print(f"Final data: {data}")
+ print("Collection data parsing complete")
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/NorthHertfordshireDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/NorthHertfordshireDistrictCouncil.py
index 35cbd52344..4255602c06 100644
--- a/uk_bin_collection/uk_bin_collection/councils/NorthHertfordshireDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/NorthHertfordshireDistrictCouncil.py
@@ -1,17 +1,14 @@
+from __future__ import annotations
+
import re
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
-
_ORDINAL_RE = re.compile(r"(\d+)(?:st|nd|rd|th)", re.IGNORECASE)
@@ -35,6 +32,17 @@ def parse_data(self, page: str, **kwargs) -> dict:
against the typeahead list item text.
web_driver, headless: passed through to create_webdriver.
"""
+ global By, EC, Keys, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
user_postcode = kwargs.get("postcode")
@@ -57,10 +65,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Typeahead input â Liberty's visible field uses this class
search_input = wait.until(
- EC.element_to_be_clickable((
- By.CSS_SELECTOR,
- "input.relation_path_type_ahead_search",
- ))
+ EC.element_to_be_clickable(
+ (
+ By.CSS_SELECTOR,
+ "input.relation_path_type_ahead_search",
+ )
+ )
)
search_input.click()
search_input.clear()
@@ -101,23 +111,25 @@ def parse_data(self, page: str, **kwargs) -> dict:
time.sleep(0.5)
submit_btn = wait.until(
- EC.element_to_be_clickable((
- By.CSS_SELECTOR,
- 'input[type="submit"][aria-label="Select address and continue"]',
- ))
+ EC.element_to_be_clickable(
+ (
+ By.CSS_SELECTOR,
+ 'input[type="submit"][aria-label="Select address and continue"]',
+ )
+ )
)
submit_btn.click()
# Wait for the result page â URL pattern ends in "show-details"
- WebDriverWait(driver, 30).until(
- lambda d: "show-details" in d.current_url
- )
+ WebDriverWait(driver, 30).until(lambda d: "show-details" in d.current_url)
# Wait for the bin content to render
WebDriverWait(driver, 30).until(
- EC.presence_of_element_located((
- By.XPATH,
- "//strong[normalize-space()='Next collection']",
- ))
+ EC.presence_of_element_located(
+ (
+ By.XPATH,
+ "//strong[normalize-space()='Next collection']",
+ )
+ )
)
soup = BeautifulSoup(driver.page_source, "html.parser")
@@ -173,10 +185,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
parsed = datetime.strptime(raw, "%A %d %B %Y")
except ValueError:
continue
- data["bins"].append({
- "type": current_type,
- "collectionDate": parsed.strftime(date_format),
- })
+ data["bins"].append(
+ {
+ "type": current_type,
+ "collectionDate": parsed.strftime(date_format),
+ }
+ )
# Dedupe while preserving order
seen = set()
diff --git a/uk_bin_collection/uk_bin_collection/councils/NorthNorfolkDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/NorthNorfolkDistrictCouncil.py
index cc0b9e2f8b..0cef48e28b 100644
--- a/uk_bin_collection/uk_bin_collection/councils/NorthNorfolkDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/NorthNorfolkDistrictCouncil.py
@@ -1,12 +1,10 @@
+from __future__ import annotations
+
import re
import time
from datetime import timedelta
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +18,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -79,9 +88,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Select matching address
address_select = Select(address_dropdown)
matching_options = [
- o
- for o in address_select.options
- if user_paon.lower() in o.text.lower()
+ o for o in address_select.options if user_paon.lower() in o.text.lower()
]
if not matching_options:
raise ValueError(
@@ -96,9 +103,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
time.sleep(2)
# Click Next to get bin results
- next_btn2 = wait.until(
- EC.element_to_be_clickable((By.ID, "NextButton"))
- )
+ next_btn2 = wait.until(EC.element_to_be_clickable((By.ID, "NextButton")))
next_btn2.click()
# Wait for results page
@@ -146,7 +151,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
raise ValueError("No bin collection data found on the results page.")
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/NorthTynesideCouncil.py b/uk_bin_collection/uk_bin_collection/councils/NorthTynesideCouncil.py
index 3b05938a04..008217df54 100644
--- a/uk_bin_collection/uk_bin_collection/councils/NorthTynesideCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/NorthTynesideCouncil.py
@@ -55,14 +55,14 @@ def parse_data(self, page: str, **kwargs) -> dict:
if getattr(response, "raise_for_status", None):
response.raise_for_status()
-
# Parse form page and get the day of week and week offsets
soup = BeautifulSoup(response.text, features="html.parser")
schedule = soup.find("div", {"class": "waste-collection__schedule"})
if schedule is None:
- raise ValueError("No waste-collection schedule found. The page structure may have changed.")
-
+ raise ValueError(
+ "No waste-collection schedule found. The page structure may have changed."
+ )
# Find days of form:
#
@@ -94,13 +94,17 @@ def parse_data(self, page: str, **kwargs) -> dict:
type_span = day.find("span", {"class": "waste-collection__day--type"})
# Direct text only (exclude nested spans, e.g., bank-holiday note)
- bin_type_text = type_span.find(text=True, recursive=False) if type_span else None
+ bin_type_text = (
+ type_span.find(text=True, recursive=False) if type_span else None
+ )
if not bin_type_text:
logger.warning("Skipping day: missing type")
continue
bin_type = bin_type_text.strip()
- colour_span = day.find("span", {"class": "waste-collection__day--colour"})
+ colour_span = day.find(
+ "span", {"class": "waste-collection__day--colour"}
+ )
if not colour_span:
logger.warning("Skipping day: missing colour")
continue
@@ -108,7 +112,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
collections.append((f"{bin_type} ({bin_colour})", collection_date))
except (AttributeError, KeyError, TypeError, ValueError) as e:
- logger.warning(f"Skipping unparsable day node: {e}")
+ logger.warning(f"Skipping unparsable day node: {type(e).__name__}")
continue
return {
diff --git a/uk_bin_collection/uk_bin_collection/councils/NorthumberlandCouncil.py b/uk_bin_collection/uk_bin_collection/councils/NorthumberlandCouncil.py
index fd0a22a59a..0822b12856 100644
--- a/uk_bin_collection/uk_bin_collection/councils/NorthumberlandCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/NorthumberlandCouncil.py
@@ -1,13 +1,10 @@
+from __future__ import annotations
+
import datetime
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.common.exceptions import TimeoutException
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -23,10 +20,10 @@ class CouncilClass(AbstractGetBinDataClass):
def extract_styles(self, style_str: str) -> dict:
"""
Parse an inline CSS style string into a dictionary of property-value pairs.
-
+
Parameters:
style_str (str): Inline CSS style text with semicolon-separated declarations (e.g. "color: red; margin: 0;").
-
+
Returns:
dict: Mapping of CSS property names to their values, with surrounding whitespace removed from both keys and values.
"""
@@ -40,7 +37,7 @@ def extract_styles(self, style_str: str) -> dict:
def parse_data(self, page: str, **kwargs) -> dict:
"""
Fetches bin collection dates from the Northumberland council postcode lookup and returns them as structured entries.
-
+
Parameters:
page (str): Ignored; the method uses the council postcode lookup URL.
**kwargs:
@@ -48,12 +45,24 @@ def parse_data(self, page: str, **kwargs) -> dict:
uprn (str|int): Property UPRN; will be padded to 12 digits before use.
web_driver: Optional Selenium WebDriver factory or identifier passed to create_webdriver.
headless (bool): Optional flag controlling headless browser creation.
-
+
Returns:
dict: A dictionary with a "bins" key mapping to a list of entries. Each entry is a dict with:
- "type" (str): The bin type (e.g., "General waste", "Recycling", "Garden waste").
- "collectionDate" (str): The collection date formatted according to the module's date_format.
"""
+ global By, EC, Keys, Select, TimeoutException, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.common.exceptions import TimeoutException
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
page = "https://bincollection.northumberland.gov.uk/postcode"
@@ -166,9 +175,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
)
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
driver.quit()
- return data
\ No newline at end of file
+ return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/NuneatonBedworthBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/NuneatonBedworthBoroughCouncil.py
index a579b72666..4ecf0c2787 100644
--- a/uk_bin_collection/uk_bin_collection/councils/NuneatonBedworthBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/NuneatonBedworthBoroughCouncil.py
@@ -937,7 +937,7 @@ def get_bin_data(self, url) -> dict:
output = bin_data[filename]
else:
- print(bin_day_response.content)
+ print("Bin data download link was not found.")
Exception("Bin data Download link not found.")
else:
diff --git a/uk_bin_collection/uk_bin_collection/councils/PembrokeshireCountyCouncil.py b/uk_bin_collection/uk_bin_collection/councils/PembrokeshireCountyCouncil.py
index fa2718a709..8c0004845f 100644
--- a/uk_bin_collection/uk_bin_collection/councils/PembrokeshireCountyCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/PembrokeshireCountyCouncil.py
@@ -91,6 +91,6 @@ def parse_data(self, page: str, **kwargs) -> dict:
key=lambda x: datetime.strptime(x["collectionDate"], "%d/%m/%Y")
)
- print(data)
+ print(f"Parsed {len(data.get('bins', []))} collections")
return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/PeterboroughCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/PeterboroughCityCouncil.py
index 2f3c2d3c6d..f225bb4c85 100644
--- a/uk_bin_collection/uk_bin_collection/councils/PeterboroughCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/PeterboroughCityCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -18,6 +17,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
user_poan = kwargs.get("paon")
@@ -135,7 +144,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
if not next_collection:
continue
- print(f"Found next collection for {bin_type}: '{next_collection}'")
+ print("Next collection parsed successfully")
parsed_date = datetime.strptime(next_collection, input_date_format)
formatted_date = parsed_date.strftime(output_date_format)
@@ -148,9 +157,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
- print(
- f"Error processing panel for bin '{bin_type if 'bin_type' in locals() else 'unknown'}': {e}"
- )
+ print(f"Error processing a collection panel ({type(e).__name__}).")
# Sort the data
data["bins"].sort(
@@ -158,7 +165,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/PortsmouthCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/PortsmouthCityCouncil.py
index c88f591093..25b922306e 100644
--- a/uk_bin_collection/uk_bin_collection/councils/PortsmouthCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/PortsmouthCityCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -21,6 +19,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
page = "https://my.portsmouth.gov.uk/en/AchieveForms/?form_uri=sandbox-publish://AF-Process-26e27e70-f771-47b1-a34d-af276075cede/AF-Stage-cd7cc291-2e59-42cc-8c3f-1f93e132a2c9/definition.json&redirectlink=%2F&cancelRedirectLink=%2F"
@@ -122,7 +131,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/PrestonCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/PrestonCityCouncil.py
index 36a8693beb..12ed1ab695 100644
--- a/uk_bin_collection/uk_bin_collection/councils/PrestonCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/PrestonCityCouncil.py
@@ -1,11 +1,8 @@
+from __future__ import annotations
+
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -21,6 +18,18 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
page = "https://selfservice.preston.gov.uk/service/Forms/FindMyNearest.aspx?Service=bins"
@@ -88,7 +97,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/ReigateAndBansteadBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ReigateAndBansteadBoroughCouncil.py
index 73c90934d7..7ce854ccd8 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ReigateAndBansteadBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ReigateAndBansteadBoroughCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -17,6 +16,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import WebDriverWait
+
driver = None
try:
user_uprn = kwargs.get("uprn")
@@ -100,7 +109,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
key=lambda x: datetime.strptime(x.get("collectionDate"), date_format)
)
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/RotherhamCouncil.py b/uk_bin_collection/uk_bin_collection/councils/RotherhamCouncil.py
index 18ab217120..71bdead0a9 100644
--- a/uk_bin_collection/uk_bin_collection/councils/RotherhamCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/RotherhamCouncil.py
@@ -46,9 +46,7 @@ def _resolve_premise(self, postcode: str, paon: str) -> str:
target = str(paon).strip().lower()
if not target:
if not rows:
- raise ValueError(
- f"No addresses found for postcode {postcode}"
- )
+ raise ValueError(f"No addresses found for postcode {postcode}")
return str(rows[0].get("PremiseID"))
# Match against Address2 (house number/name) first, then Street.
@@ -64,15 +62,12 @@ def _resolve_premise(self, postcode: str, paon: str) -> str:
# against a paon of "22A".
for row in rows:
blob = " ".join(
- str(row.get(k, "")).strip()
- for k in ("Address1", "Address2", "Street")
+ str(row.get(k, "")).strip() for k in ("Address1", "Address2", "Street")
).lower()
if target and target in blob:
return str(row.get("PremiseID"))
- raise ValueError(
- f"No address matching '{paon}' for postcode {postcode}"
- )
+ raise ValueError(f"No address matching '{paon}' for postcode {postcode}")
def parse_data(self, page: str, **kwargs) -> dict:
premises = kwargs.get("premisesid")
@@ -104,14 +99,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
timeout=15,
)
except Exception as exc:
- print(f"Error contacting Rotherham API: {exc}")
+ print(f"Error contacting Rotherham API ({type(exc).__name__}).")
return {"bins": []}
if resp.status_code != 200:
- print(
- f"Rotherham API request failed ({resp.status_code}). "
- f"URL: {resp.url}"
- )
+ print(f"Rotherham API request failed ({resp.status_code}).")
return {"bins": []}
try:
@@ -123,12 +115,8 @@ def parse_data(self, page: str, **kwargs) -> dict:
data = {"bins": []}
seen = set()
for item in collections:
- bin_type = (
- item.get("BinType") or item.get("bintype") or "Unknown"
- )
- date_str = (
- item.get("CollectionDate") or item.get("collectionDate")
- )
+ bin_type = item.get("BinType") or item.get("bintype") or "Unknown"
+ date_str = item.get("CollectionDate") or item.get("collectionDate")
if not date_str:
continue
try:
@@ -141,13 +129,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
if key in seen:
continue
seen.add(key)
- data["bins"].append(
- {"type": bin_type, "collectionDate": formatted}
- )
+ data["bins"].append({"type": bin_type, "collectionDate": formatted})
data["bins"].sort(
- key=lambda x: datetime.strptime(
- x["collectionDate"], date_format
- )
+ key=lambda x: datetime.strptime(x["collectionDate"], date_format)
)
return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/RoyalBoroughofGreenwich.py b/uk_bin_collection/uk_bin_collection/councils/RoyalBoroughofGreenwich.py
index ff5b224164..fb471ada92 100644
--- a/uk_bin_collection/uk_bin_collection/councils/RoyalBoroughofGreenwich.py
+++ b/uk_bin_collection/uk_bin_collection/councils/RoyalBoroughofGreenwich.py
@@ -112,7 +112,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
original_collection_date, "%A %d %B"
)
- original_collection_date = self._set_year_from_month(original_collection_date)
+ original_collection_date = self._set_year_from_month(
+ original_collection_date
+ )
new_collection_date = row.find_all("td")[1].get_text(strip=True)
new_collection_date = new_collection_date.replace(
@@ -131,7 +133,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
IndexError,
ValueError,
) as e:
- logger.warning(f"Failed to scrape bank holiday dates: {e}")
+ logger.warning(f"Failed to scrape bank holiday dates: {type(e).__name__}")
greenstartDate = datetime(2025, 12, 29)
bluestartDate = datetime(2025, 12, 29)
diff --git a/uk_bin_collection/uk_bin_collection/councils/RushcliffeBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/RushcliffeBoroughCouncil.py
index 9770167487..07a13f5922 100644
--- a/uk_bin_collection/uk_bin_collection/councils/RushcliffeBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/RushcliffeBoroughCouncil.py
@@ -1,8 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -17,6 +15,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
page = "https://selfservice.rushcliffe.gov.uk/renderform.aspx?t=1242&k=86BDCD8DE8D868B9E23D10842A7A4FE0F1023CCA"
@@ -110,7 +119,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/SandwellBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/SandwellBoroughCouncil.py
index 714788c5bd..050793c433 100644
--- a/uk_bin_collection/uk_bin_collection/councils/SandwellBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/SandwellBoroughCouncil.py
@@ -94,14 +94,14 @@ def parse_data(self, page: str, **kwargs) -> dict:
rows_data = result["integration"]["transformed"]["rows_data"]
if not isinstance(rows_data, dict):
- logger.warning("Unexpected rows_data format: %s", rows_data)
+ logger.warning("Unexpected rows_data format.")
continue
for row in rows_data.values():
date = row.get(date_key)
if not date:
logger.warning(
- "Date key '%s' missing in row: %s", date_key, row
+ "Required collection date field missing from parsed row"
)
continue
@@ -111,11 +111,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except requests.RequestException as e:
- logger.error("API request failed: %s", e)
+ logger.error("API request failed: %s", type(e).__name__)
continue
except (KeyError, ValueError, TypeError) as e:
- logger.warning("Unexpected structure in response: %s", e)
+ logger.warning("Unexpected structure in response: %s", type(e).__name__)
continue
- logger.info("Parsed bins: %s", bindata["bins"])
+ logger.info("Parsed %s bins", len(bindata["bins"]))
return bindata
diff --git a/uk_bin_collection/uk_bin_collection/councils/SevenoaksDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/SevenoaksDistrictCouncil.py
index 55cc96e11f..0e4e92203b 100644
--- a/uk_bin_collection/uk_bin_collection/councils/SevenoaksDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/SevenoaksDistrictCouncil.py
@@ -1,12 +1,9 @@
+from __future__ import annotations
+
import time
from typing import Any
from dateutil.parser import parse
-from selenium.common.exceptions import NoSuchElementException, TimeoutException
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import create_webdriver, date_format
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -21,6 +18,18 @@ def wait_for_element_conditions(self, driver, conditions, timeout: int = 5):
raise
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, NoSuchElementException, Select, TimeoutException, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.common.exceptions import NoSuchElementException, TimeoutException
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
web_driver = kwargs.get("web_driver")
@@ -83,7 +92,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
)[1].text
# Skip if the message indicates service suspension
- if "suspended" in raw_next_collection_date.lower() or "restarting" in raw_next_collection_date.lower():
+ if (
+ "suspended" in raw_next_collection_date.lower()
+ or "restarting" in raw_next_collection_date.lower()
+ ):
continue
parsed_bin_date = parse(
@@ -101,7 +113,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
print("Error finding element for bin")
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/SomersetCouncil.py b/uk_bin_collection/uk_bin_collection/councils/SomersetCouncil.py
index 7fd94e8867..14a10cc819 100644
--- a/uk_bin_collection/uk_bin_collection/councils/SomersetCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/SomersetCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -18,6 +17,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -127,7 +136,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/SouthGloucestershireCouncil.py b/uk_bin_collection/uk_bin_collection/councils/SouthGloucestershireCouncil.py
index 06825b1de0..90ea3788d8 100644
--- a/uk_bin_collection/uk_bin_collection/councils/SouthGloucestershireCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/SouthGloucestershireCouncil.py
@@ -7,9 +7,9 @@
def format_bin_data(key: str, date: datetime):
formatted_date = date.strftime(date_format)
servicename = key.get("hso_servicename")
- print(servicename)
+ print("Processing collection service")
if re.match(r"^Recycl", servicename) is not None:
- return [ ("Recycling", formatted_date) ]
+ return [("Recycling", formatted_date)]
elif re.match(r"^Refuse", servicename) is not None:
return [("General Waste (Black Bin)", formatted_date)]
elif re.match(r"^Garden", servicename) is not None:
@@ -24,16 +24,16 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
"""
Parse waste collection data for the given UPRN and return upcoming bin collections within the next eight weeks.
-
+
Parameters:
page (str): Raw page content (unused by this implementation; included for signature compatibility).
uprn (str, keyword): Unique Property Reference Number used to query the South Gloucestershire collection API.
-
+
Returns:
dict: A mapping with a "bins" key containing a list of collection entries. Each entry is a dict with:
- "type" (str): Human-friendly bin type (e.g., "Recycling", "General Waste (Black Bin)").
- "collectionDate" (str): Formatted collection date string.
-
+
Raises:
ValueError: If the API returns no collection data for the provided UPRN.
"""
@@ -53,15 +53,15 @@ def parse_data(self, page: str, **kwargs) -> dict:
if not json_response:
raise ValueError("No collection data found for provided UPRN.")
- collection_data = json_response.get('value')
+ collection_data = json_response.get("value")
today = datetime.today()
eight_weeks = datetime.today() + timedelta(days=8 * 7)
data = {"bins": []}
collection_tuple = []
for collection in collection_data:
- print(collection)
- item = collection.get('hso_nextcollection')
+ print("Processing collection entry")
+ item = collection.get("hso_nextcollection")
if not item:
continue
@@ -82,4 +82,4 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
data["bins"].append(dict_data)
- return data
\ No newline at end of file
+ return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/SouthKestevenDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/SouthKestevenDistrictCouncil.py
index 78dca587b2..3bacffde75 100644
--- a/uk_bin_collection/uk_bin_collection/councils/SouthKestevenDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/SouthKestevenDistrictCouncil.py
@@ -1,24 +1,58 @@
+"""South Kesteven District Council collection data via its live browser flow."""
+
+from __future__ import annotations
+
import json
+import logging
+import re
+import unicodedata
+import uuid
from datetime import datetime
from pathlib import Path
-from urllib.parse import urljoin
+from time import monotonic
+from types import SimpleNamespace
+from urllib.parse import quote, quote_plus, urljoin, urlsplit, urlunsplit
from bs4 import BeautifulSoup
-import requests
-from requests import RequestException
-from selenium.common.exceptions import NoSuchElementException, TimeoutException
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
+
from uk_bin_collection.uk_bin_collection.common import create_webdriver, date_format
+from uk_bin_collection.uk_bin_collection.dependency_validation import (
+ validate_websocket_client,
+)
+from uk_bin_collection.uk_bin_collection.exceptions import (
+ AddressMismatchError,
+ BrowserUnavailableError,
+ ConfigurationError,
+ MissingDependencyError,
+ SiteChanged,
+ UKBinCollectionError,
+ UpstreamAccessDenied,
+)
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
+_LOGGER = logging.getLogger(__name__)
+
class CouncilClass(AbstractGetBinDataClass):
- """South Kesteven District Council bin collections via the live binday flow."""
+ """Collect South Kesteven bin dates from the supported postcode checker."""
+ # This adapter owns the complete browser navigation flow. The opt-out is
+ # intrinsic so callers cannot accidentally reinstate the direct HTTP
+ # preflight that South Kesteven rejects in some environments.
+ skip_generic_get = True
BIN_DAY_URL = "https://www.southkesteven.gov.uk/binday"
CHECKER_LINK_TEXT = "Postcode bin day checker"
+ CHECKER_SCHEME = "https"
+ CHECKER_HOST = "selfservice.southkesteven.gov.uk"
+ PAGE_LOAD_TIMEOUT_SECONDS = 30
+ ELEMENT_TIMEOUT_SECONDS = 30
+ TOTAL_RUN_TIMEOUT_SECONDS = 90
+ ACCESS_DENIED_MARKERS = (
+ "403 forbidden",
+ "error 403",
+ "access denied",
+ "request blocked",
+ )
BIN_TYPE_MAP = {
"240 Litre Recycling": "Grey Bin",
"23lt Food Caddy": "Food Bin",
@@ -34,179 +68,339 @@ class CouncilClass(AbstractGetBinDataClass):
SUBMIT_BUTTON_ID = "submit-button"
BODY_CONTENT_ID = "body-content"
CURRENT_SECTION_ID = "current-section-id"
- DEFAULT_ARTIFACT_ROOT = "artifacts/SouthKestevenDistrictCouncil"
+ CHECKER_ARTIFACT_PATHS = frozenset(
+ {
+ "/renderform",
+ "/renderform.aspx",
+ "/renderform/Form",
+ }
+ )
+ SENSITIVE_OPAQUE_FIELD_PREFIXES = ("ff5265",)
def parse_data(self, page: str, **kwargs) -> dict:
- binday_url = kwargs.get("url") or self.BIN_DAY_URL
+ """Run the complete lookup in one bounded Selenium browser session."""
+ del page # The generic HTTP response is intentionally never used.
+
+ # This adapter has one reviewed entry point. Do not let standalone/API
+ # callers redirect its privileged browser session to another origin.
+ binday_url = self.BIN_DAY_URL
postcode = kwargs.get("postcode")
paon = kwargs.get("paon")
web_driver = kwargs.get("web_driver")
user_agent = kwargs.get("user_agent")
headless = kwargs.get("headless", True)
artifact_root = self._get_artifact_root(kwargs.get("artifact_dir"))
+ deadline = monotonic() + self.TOTAL_RUN_TIMEOUT_SECONDS
if headless is None:
headless = True
-
if not postcode:
- raise ValueError("Postcode is required for South Kesteven.")
+ raise ConfigurationError("Postcode is required for South Kesteven.")
if not paon:
- raise ValueError(
- "Property number or name (paon) is required for South Kesteven."
+ raise ConfigurationError(
+ "Property number or name is required for South Kesteven."
)
+ if not str(web_driver or "").strip():
+ raise ConfigurationError(
+ "A remote Selenium WebDriver URL is required for South Kesteven."
+ )
+ web_driver = str(web_driver).strip()
- checker_url = self._resolve_checker_url(
- binday_url, page=page, user_agent=user_agent
- )
-
- driver = create_webdriver(web_driver, headless, user_agent, __name__)
+ driver = None
+ support = None
+ stage = "create_browser"
try:
- wait = WebDriverWait(driver, 30)
+ # create_webdriver validates websocket-client ownership before importing
+ # Selenium, so a /config/websocket collision is reported safely.
+ driver = create_webdriver(
+ web_driver,
+ headless,
+ user_agent,
+ __name__,
+ command_timeout=min(
+ self.PAGE_LOAD_TIMEOUT_SECONDS,
+ self._remaining_seconds(deadline),
+ ),
+ )
+ support = self._load_selenium_support()
+
+ stage = "open_binday"
+ self._set_page_load_timeout(driver, deadline)
+ driver.get(binday_url)
+ self._raise_if_access_denied(driver.page_source)
+
+ stage = "resolve_checker"
+ checker_url = self._wait_for_checker_url(
+ self._new_wait(driver, support, deadline), support, binday_url
+ )
+
+ stage = "open_checker"
+ self._set_page_load_timeout(driver, deadline)
driver.get(checker_url)
+ self._raise_if_access_denied(driver.page_source)
+ stage = "enter_postcode"
postcode_input = self._wait_for_clickable(
- wait,
- (By.ID, self.POSTCODE_INPUT_ID),
- "Unable to find the postcode input on the South Kesteven checker.",
+ self._new_wait(driver, support, deadline),
+ support,
+ (support.By.ID, self.POSTCODE_INPUT_ID),
+ "The postcode input is missing from the South Kesteven checker.",
)
postcode_input.clear()
postcode_input.send_keys(postcode)
- initial_section_id = self._get_current_section_id(driver)
- initial_body_markup = self._get_body_markup(driver)
+ initial_section_id = self._get_current_section_id(driver, support)
+ initial_body_markup = self._get_body_markup(driver, support)
search_button = self._wait_for_clickable(
- wait,
- (By.ID, self.SEARCH_BUTTON_ID),
- "Unable to find the search button after entering the postcode.",
+ self._new_wait(driver, support, deadline),
+ support,
+ (support.By.ID, self.SEARCH_BUTTON_ID),
+ "The postcode search button is missing from the checker.",
)
search_button.click()
+ self._raise_if_driver_access_denied(driver)
- address_select = self._wait_for_address_options(wait)
- self._select_address(address_select, paon)
- self._wait_for_address_confirmation(wait)
+ stage = "select_address"
+ address_select = self._wait_for_address_options(
+ self._new_wait(driver, support, deadline), support
+ )
+ self._raise_if_driver_access_denied(driver)
+ self._select_address(address_select, paon, support)
+ self._wait_for_address_confirmation(
+ self._new_wait(driver, support, deadline), support
+ )
+ self._raise_if_driver_access_denied(driver)
+ stage = "submit_address"
submit_button = self._wait_for_clickable(
- wait,
- (By.ID, self.SUBMIT_BUTTON_ID),
- "Unable to find the submit button after selecting the address.",
+ self._new_wait(driver, support, deadline),
+ support,
+ (support.By.ID, self.SUBMIT_BUTTON_ID),
+ "The address submit button is missing from the checker.",
)
submit_button.click()
+ self._raise_if_driver_access_denied(driver)
self._wait_for_results_container(
- wait,
+ self._new_wait(driver, support, deadline),
+ support,
initial_section_id,
initial_body_markup,
)
+ self._raise_if_driver_access_denied(driver)
+ stage = "parse_results"
+ self._remaining_seconds(deadline)
return {"bins": self._parse_collection_rows(driver.page_source)}
+ except UKBinCollectionError as exc:
+ artifact_path = self._capture_debug_artifacts(
+ driver,
+ artifact_root,
+ stage=stage,
+ redactions=(str(postcode), str(paon)),
+ )
+ if artifact_path is not None and hasattr(exc, "add_note"):
+ exc.add_note(f"Redacted debug artifacts: {artifact_path}")
+ raise
except Exception as exc:
artifact_path = self._capture_debug_artifacts(
driver,
artifact_root,
- {"postcode": postcode, "paon": paon, "binday_url": binday_url},
+ stage=stage,
+ redactions=(str(postcode), str(paon)),
)
- raise RuntimeError(
- self._with_artifact_hint(str(exc), artifact_path)
- ) from exc
+ try:
+ self._raise_if_driver_access_denied(driver)
+ except UpstreamAccessDenied as translated:
+ if artifact_path is not None and hasattr(translated, "add_note"):
+ translated.add_note(f"Redacted debug artifacts: {artifact_path}")
+ raise translated from exc
+ if support is not None and isinstance(exc, support.WebDriverException):
+ translated: UKBinCollectionError = BrowserUnavailableError(
+ "The configured Selenium browser stopped responding."
+ )
+ else:
+ translated = SiteChanged(
+ "The South Kesteven checker did not complete the expected flow."
+ )
+ if artifact_path is not None and hasattr(translated, "add_note"):
+ translated.add_note(f"Redacted debug artifacts: {artifact_path}")
+ raise translated from exc
finally:
- if driver:
- driver.quit()
-
- def _wait_for_clickable(self, wait, locator, error_message):
+ if driver is not None:
+ try:
+ driver.quit()
+ except Exception:
+ _LOGGER.warning("South Kesteven WebDriver cleanup failed.")
+
+ @staticmethod
+ def _load_selenium_support() -> SimpleNamespace:
+ """Import Selenium support classes only after dependency validation."""
+ validate_websocket_client()
try:
- return wait.until(EC.element_to_be_clickable(locator))
- except TimeoutException as exc:
- raise RuntimeError(error_message) from exc
+ from selenium.common.exceptions import (
+ NoSuchElementException,
+ TimeoutException,
+ UnexpectedTagNameException,
+ WebDriverException,
+ )
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+ except ImportError as exc:
+ raise MissingDependencyError(
+ "Selenium support modules are not installed completely."
+ ) from exc
+
+ return SimpleNamespace(
+ By=By,
+ EC=EC,
+ Select=Select,
+ WebDriverWait=WebDriverWait,
+ NoSuchElementException=NoSuchElementException,
+ TimeoutException=TimeoutException,
+ UnexpectedTagNameException=UnexpectedTagNameException,
+ WebDriverException=WebDriverException,
+ )
+
+ def _remaining_seconds(self, deadline: float) -> float:
+ remaining = deadline - monotonic()
+ if remaining <= 0:
+ raise SiteChanged(
+ "The South Kesteven browser flow exceeded its total timeout."
+ )
+ return remaining
- def _wait_for_presence(self, wait, locator, error_message):
+ def _new_wait(self, driver, support, deadline: float):
+ timeout = min(
+ self.ELEMENT_TIMEOUT_SECONDS,
+ self._remaining_seconds(deadline),
+ )
+ return support.WebDriverWait(driver, timeout)
+
+ def _set_page_load_timeout(self, driver, deadline: float) -> None:
+ if not hasattr(driver, "set_page_load_timeout"):
+ return
+ timeout = min(
+ self.PAGE_LOAD_TIMEOUT_SECONDS,
+ self._remaining_seconds(deadline),
+ )
+ driver.set_page_load_timeout(timeout)
+
+ def _wait_for_checker_url(self, wait, support, binday_url: str) -> str:
try:
- return wait.until(EC.presence_of_element_located(locator))
- except TimeoutException as exc:
- raise RuntimeError(error_message) from exc
+ return wait.until(
+ lambda driver: self._find_checker_url(
+ driver.page_source,
+ binday_url,
+ )
+ )
+ except support.TimeoutException as exc:
+ driver = getattr(wait, "_driver", None)
+ self._raise_if_driver_access_denied(driver)
+ raise SiteChanged(
+ "The supported postcode checker link is missing from the binday page."
+ ) from exc
- def _resolve_checker_url(
- self,
- binday_url: str,
- page: str | object | None = None,
- user_agent: str | None = None,
- ) -> str:
- html = self._get_binday_html(binday_url, page=page, user_agent=user_agent)
- soup = BeautifulSoup(html, "html.parser")
+ def _find_checker_url(self, html: str, binday_url: str) -> str | bool:
+ if not html or not html.strip():
+ return False
+ soup = BeautifulSoup(html, "html.parser")
for link in soup.find_all("a", href=True):
- link_text = link.get_text(" ", strip=True).lower()
- if self.CHECKER_LINK_TEXT.lower() in link_text:
- return urljoin(binday_url, link["href"])
+ link_text = link.get_text(" ", strip=True).casefold()
+ if self.CHECKER_LINK_TEXT.casefold() not in link_text:
+ continue
+ checker_url = urljoin(binday_url, link["href"])
+ parsed = urlsplit(checker_url)
+ if (
+ parsed.scheme != self.CHECKER_SCHEME
+ or parsed.hostname != self.CHECKER_HOST
+ ):
+ raise SiteChanged(
+ "The postcode checker link points outside the supported service."
+ )
+ return checker_url
+ return False
+
+ def _raise_if_access_denied(self, html: str) -> None:
+ text = BeautifulSoup(html or "", "html.parser").get_text(" ", strip=True)
+ normalized = self._normalize_text(text)
+ if any(marker in normalized for marker in self.ACCESS_DENIED_MARKERS):
+ raise UpstreamAccessDenied(
+ "South Kesteven denied browser access to the bin checker."
+ )
- raise RuntimeError(
- "Unable to find the postcode bin day checker link on the binday page."
- )
+ def _raise_if_driver_access_denied(self, driver) -> None:
+ """Classify a browser denial without trusting arbitrary driver values."""
+ if driver is None:
+ return
+ try:
+ page_source = getattr(driver, "page_source", "")
+ except Exception:
+ return
+ if not isinstance(page_source, str):
+ return
+ self._raise_if_access_denied(page_source)
- def _get_binday_html(
- self,
- binday_url: str,
- page: str | object | None = None,
- user_agent: str | None = None,
- ) -> str:
- if hasattr(page, "text"):
- html = str(page.text)
- elif isinstance(page, str) and page.strip():
- html = page
- else:
- headers = {}
- if user_agent:
- headers["User-Agent"] = user_agent
- try:
- response = requests.get(binday_url, headers=headers, timeout=30)
- response.raise_for_status()
- except RequestException as exc:
- raise RuntimeError(
- f"Unable to load the South Kesteven binday page at {binday_url}."
- ) from exc
- html = response.text
-
- if not html.strip():
- raise RuntimeError(
- f"Unable to load the South Kesteven binday page at {binday_url}."
- )
+ def _raise_if_wait_access_denied(self, wait) -> None:
+ self._raise_if_driver_access_denied(getattr(wait, "_driver", None))
- return html
+ def _wait_for_clickable(self, wait, support, locator, error_message):
+ try:
+ return wait.until(support.EC.element_to_be_clickable(locator))
+ except support.TimeoutException as exc:
+ self._raise_if_wait_access_denied(wait)
+ raise SiteChanged(error_message) from exc
- def _wait_for_address_options(self, wait):
+ def _wait_for_address_options(self, wait, support):
try:
- return wait.until(self._address_options_ready)
- except TimeoutException as exc:
- raise RuntimeError(
- "Unable to find the address dropdown after searching for the postcode."
+ return wait.until(
+ lambda driver: self._address_options_ready(driver, support)
+ )
+ except support.TimeoutException as exc:
+ self._raise_if_wait_access_denied(wait)
+ raise SiteChanged(
+ "The address dropdown did not appear after the postcode search."
) from exc
- def _address_options_ready(self, driver):
+ def _address_options_ready(self, driver, support):
try:
- address_select = driver.find_element(By.ID, self.ADDRESS_SELECT_ID)
- except NoSuchElementException:
+ address_select = driver.find_element(support.By.ID, self.ADDRESS_SELECT_ID)
+ except support.NoSuchElementException:
return False
if not address_select.is_displayed():
return False
- options = [
- option for option in Select(address_select).options if option.text.strip()
- ]
+ try:
+ options = [
+ option
+ for option in support.Select(address_select).options
+ if option.text.strip()
+ ]
+ except support.UnexpectedTagNameException as exc:
+ raise SiteChanged(
+ "The South Kesteven address control is no longer a selection list."
+ ) from exc
return address_select if options else False
- def _wait_for_address_confirmation(self, wait):
+ def _wait_for_address_confirmation(self, wait, support):
try:
- return wait.until(self._address_confirmation_ready)
- except TimeoutException as exc:
- raise RuntimeError(
- "Unable to confirm the selected address before submitting the lookup."
+ return wait.until(
+ lambda driver: self._address_confirmation_ready(driver, support)
+ )
+ except support.TimeoutException as exc:
+ self._raise_if_wait_access_denied(wait)
+ raise SiteChanged(
+ "The selected address was not confirmed by the checker."
) from exc
- def _address_confirmation_ready(self, driver):
+ def _address_confirmation_ready(self, driver, support):
try:
- address_value = driver.find_element(By.ID, self.ADDRESS_VALUE_ID)
- change_button = driver.find_element(By.ID, self.CHANGE_BUTTON_ID)
- except NoSuchElementException:
+ address_value = driver.find_element(support.By.ID, self.ADDRESS_VALUE_ID)
+ change_button = driver.find_element(support.By.ID, self.CHANGE_BUTTON_ID)
+ except support.NoSuchElementException:
return False
selected_value = (address_value.get_attribute("value") or "").strip()
@@ -214,128 +408,312 @@ def _address_confirmation_ready(self, driver):
return True
try:
- display_name = driver.find_element(By.ID, self.ADDRESS_DISPLAY_ID)
- except NoSuchElementException:
+ display_name = driver.find_element(support.By.ID, self.ADDRESS_DISPLAY_ID)
+ except support.NoSuchElementException:
return False
return bool((display_name.text or "").strip()) and change_button.is_displayed()
def _wait_for_results_container(
- self, wait, initial_section_id: str | None, initial_body_markup: str
+ self,
+ wait,
+ support,
+ initial_section_id: str | None,
+ initial_body_markup: str,
):
try:
return wait.until(
lambda driver: self._results_container_ready(
driver,
+ support,
initial_section_id,
initial_body_markup,
)
)
- except TimeoutException as exc:
- raise RuntimeError(
- "Unable to load the collection results after submitting the address."
+ except support.TimeoutException as exc:
+ self._raise_if_wait_access_denied(wait)
+ raise SiteChanged(
+ "Collection results did not load after the address was submitted."
) from exc
def _results_container_ready(
- self, driver, initial_section_id: str | None, initial_body_markup: str
+ self,
+ driver,
+ support,
+ initial_section_id: str | None,
+ initial_body_markup: str,
):
try:
- body_markup = self._get_body_markup(driver)
- except RuntimeError:
+ body_markup = self._get_body_markup(driver, support)
+ except SiteChanged:
return False
if not body_markup or body_markup == initial_body_markup:
return False
- current_section_id = self._get_current_section_id(driver)
+ current_section_id = self._get_current_section_id(driver, support)
section_changed = bool(
initial_section_id
and current_section_id
and current_section_id != initial_section_id
)
address_form_gone = self.POSTCODE_INPUT_ID not in body_markup
+ normalized_markup = body_markup.casefold()
has_collection_markup = (
- "alloy-table" in body_markup.lower()
- or "your collections" in body_markup.lower()
+ "alloy-table" in normalized_markup
+ or "your collections" in normalized_markup
)
-
return address_form_gone and (section_changed or has_collection_markup)
- def _select_address(self, address_select, paon: str) -> None:
- target = str(paon).strip().lower()
- select = Select(address_select)
+ def _select_address(self, address_select, paon: str, support) -> None:
+ target = self._normalize_text(str(paon))
+ try:
+ select = support.Select(address_select)
+ except support.UnexpectedTagNameException as exc:
+ raise SiteChanged(
+ "The South Kesteven address control is no longer a selection list."
+ ) from exc
+ matches = [
+ option
+ for option in select.options
+ if self._address_component(option.text, target) == target
+ ]
+
+ if not matches:
+ raise AddressMismatchError(
+ "The configured property was not found in the address results."
+ )
+ if len(matches) > 1:
+ raise AddressMismatchError(
+ "The configured property matched more than one address."
+ )
+ select.select_by_visible_text(matches[0].text)
- for option in select.options:
- option_text = option.text.strip().lower()
- if target in option_text:
- select.select_by_visible_text(option.text)
- return
+ @classmethod
+ def _address_component(cls, address: str, target: str) -> str:
+ normalized_address = cls._normalize_text(address)
+ if re.fullmatch(r"[0-9]+[a-z]?", target):
+ return re.split(r"[\s,]", normalized_address, maxsplit=1)[0]
+ return normalized_address.split(",", maxsplit=1)[0].strip()
- raise RuntimeError(
- f"Unable to find the property '{paon}' in the address dropdown."
- )
+ @staticmethod
+ def _normalize_text(value: str) -> str:
+ normalized = unicodedata.normalize("NFKC", value or "")
+ return " ".join(normalized.split()).casefold()
- def _get_current_section_id(self, driver) -> str | None:
+ def _get_current_section_id(self, driver, support) -> str | None:
try:
- return driver.find_element(By.ID, self.CURRENT_SECTION_ID).get_attribute(
- "value"
- )
- except NoSuchElementException:
+ return driver.find_element(
+ support.By.ID, self.CURRENT_SECTION_ID
+ ).get_attribute("value")
+ except support.NoSuchElementException:
return None
- def _get_body_markup(self, driver) -> str:
+ def _get_body_markup(self, driver, support) -> str:
try:
- body_content = driver.find_element(By.ID, self.BODY_CONTENT_ID)
- except NoSuchElementException as exc:
- raise RuntimeError(
- "Unable to find the body-content container on the checker page."
- ) from exc
-
+ body_content = driver.find_element(support.By.ID, self.BODY_CONTENT_ID)
+ except support.NoSuchElementException as exc:
+ raise SiteChanged("The checker body container is missing.") from exc
return body_content.get_attribute("innerHTML") or ""
- def _get_artifact_root(self, artifact_dir: str | None) -> Path:
- if artifact_dir:
- return Path(artifact_dir)
- return Path.cwd() / self.DEFAULT_ARTIFACT_ROOT
+ @staticmethod
+ def _get_artifact_root(artifact_dir: str | None) -> Path | None:
+ if not artifact_dir or not str(artifact_dir).strip():
+ return None
+ return Path(artifact_dir)
def _capture_debug_artifacts(
- self, driver, artifact_root: Path, context: dict[str, str]
+ self,
+ driver,
+ artifact_root: Path | None,
+ *,
+ stage: str,
+ redactions: tuple[str, ...],
) -> Path | None:
- if not driver:
+ """Write opt-in HTML/metadata without screenshots or household values."""
+ if driver is None or artifact_root is None:
return None
- timestamp = datetime.now().strftime("%Y%m%d-%H%M%S")
- artifact_path = artifact_root / timestamp
- artifact_path.mkdir(parents=True, exist_ok=True)
+ try:
+ timestamp = datetime.now().strftime("%Y%m%d-%H%M%S-%f")
+ artifact_path = artifact_root / f"{timestamp}-{uuid.uuid4().hex[:8]}"
+ artifact_path.mkdir(parents=True, exist_ok=False)
- metadata = {
- "current_url": str(getattr(driver, "current_url", "")) or None,
- **context,
+ html = self._redact_html(
+ str(getattr(driver, "page_source", "") or ""), redactions
+ )
+
+ metadata = {
+ "stage": stage,
+ "current_url": self._sanitize_url(
+ str(getattr(driver, "current_url", "") or "")
+ ),
+ }
+ (artifact_path / "page.html").write_text(html, encoding="utf-8")
+ (artifact_path / "metadata.json").write_text(
+ json.dumps(metadata, indent=2, sort_keys=True),
+ encoding="utf-8",
+ )
+ return artifact_path.resolve()
+ except Exception:
+ _LOGGER.warning("Unable to write redacted South Kesteven debug artifacts.")
+ return None
+
+ @classmethod
+ def _redact_html(cls, html: str, redactions: tuple[str, ...]) -> str:
+ """Remove configured and form-derived household values from HTML.
+
+ Form state is default-deny: values and option/display text are removed
+ even when the upstream service uses opaque field names. Selector names
+ and structural markup remain available for diagnosing site drift.
+ """
+ redacted = html
+ for value in redactions:
+ if not value or value == "None":
+ continue
+ variants = {
+ value,
+ re.sub(r"\s+", "", value),
+ quote(value),
+ quote_plus(value),
+ }
+ for variant in sorted(variants, key=len, reverse=True):
+ if variant:
+ redacted = re.sub(
+ re.escape(variant),
+ "[REDACTED]",
+ redacted,
+ flags=re.IGNORECASE,
+ )
+
+ soup = BeautifulSoup(redacted, "html.parser")
+ for script in soup.find_all("script"):
+ script.clear()
+ script.append("[REDACTED SCRIPT]")
+
+ sensitive_markers = (
+ "address",
+ "displayname",
+ "postcode",
+ "property",
+ "paon",
+ "uprn",
+ "usrn",
+ )
+ form_control_tags = {
+ "button",
+ "input",
+ "option",
+ "output",
+ "select",
+ "textarea",
+ }
+ value_attributes = {
+ "aria-label",
+ "aria-valuetext",
+ "placeholder",
+ "title",
+ "value",
}
- screenshot_path = artifact_path / "page.png"
- html_path = artifact_path / "page.html"
- metadata_path = artifact_path / "metadata.json"
+ for field in soup.find_all(True):
+ descriptor = " ".join(
+ str(field.get(attribute, ""))
+ for attribute in (
+ "id",
+ "name",
+ "for",
+ "aria-label",
+ "autocomplete",
+ )
+ ).casefold()
+ field_id = str(field.get("id", "")).casefold()
+ opaque_sensitive = any(
+ field_id.startswith(prefix)
+ for prefix in cls.SENSITIVE_OPAQUE_FIELD_PREFIXES
+ )
+ semantic_sensitive = any(
+ marker in descriptor for marker in sensitive_markers
+ )
+ style = str(field.get("style", "")).casefold().replace(" ", "")
+ hidden = (
+ field.has_attr("hidden")
+ or str(field.get("aria-hidden", "")).casefold() == "true"
+ or (
+ field.name == "input"
+ and str(field.get("type", "")).casefold() == "hidden"
+ )
+ or "display:none" in style
+ or "visibility:hidden" in style
+ )
+ form_control = field.name in form_control_tags
+
+ if form_control or opaque_sensitive or semantic_sensitive or hidden:
+ for attribute in list(field.attrs):
+ normalized_attribute = attribute.casefold()
+ if (
+ normalized_attribute in value_attributes
+ or normalized_attribute.startswith("data-")
+ ):
+ field[attribute] = "[REDACTED]"
+
+ if field.name == "option":
+ field.attrs.pop("selected", None)
+
+ redact_text = (
+ field.name in {"option", "output", "textarea"}
+ or opaque_sensitive
+ or semantic_sensitive
+ or hidden
+ )
+ if redact_text:
+ for text_node in list(field.find_all(string=True)):
+ if text_node.strip():
+ text_node.replace_with("[REDACTED]")
+
+ redacted = str(soup)
+ return re.sub(
+ r"(?i)((?:uprn|usrn)[^0-9]{0,20})[0-9]{6,15}",
+ r"\1[REDACTED]",
+ redacted,
+ )
+ @classmethod
+ def _sanitize_url(cls, value: str) -> str | None:
+ """Retain only an allowlisted origin and a known non-identifier route."""
+ if not value:
+ return None
+ parsed = urlsplit(value)
+ if parsed.scheme not in {"http", "https"} or not parsed.hostname:
+ return None
try:
- html_path.write_text(driver.page_source, encoding="utf-8")
- except Exception as exc:
- metadata["page_html_error"] = str(exc)
+ parsed_port = parsed.port
+ except ValueError:
+ return None
- try:
- metadata["screenshot_saved"] = bool(
- driver.save_screenshot(str(screenshot_path))
- )
- except Exception as exc:
- metadata["screenshot_error"] = str(exc)
+ default_port = 443 if parsed.scheme == "https" else 80
+ if parsed_port not in {None, default_port}:
+ return None
- metadata_path.write_text(json.dumps(metadata, indent=4), encoding="utf-8")
- return artifact_path.resolve()
+ binday = urlsplit(cls.BIN_DAY_URL)
+ origin = (parsed.scheme.casefold(), parsed.hostname.casefold())
+ binday_origin = (binday.scheme.casefold(), (binday.hostname or "").casefold())
+ checker_origin = (cls.CHECKER_SCHEME.casefold(), cls.CHECKER_HOST.casefold())
+
+ safe_path = ""
+ if origin == binday_origin:
+ if parsed.path == binday.path:
+ safe_path = binday.path
+ elif origin == checker_origin:
+ if parsed.path in cls.CHECKER_ARTIFACT_PATHS:
+ safe_path = parsed.path
+ else:
+ return None
- def _with_artifact_hint(self, message: str, artifact_path: Path | None) -> str:
- if artifact_path is None:
- return message
- return f"{message} Debug artifacts saved to: {artifact_path}"
+ host = parsed.hostname
+ if ":" in host and not host.startswith("["):
+ host = f"[{host}]"
+ return urlunsplit((parsed.scheme.casefold(), host, safe_path, "", ""))
def _normalize_bin_type(self, raw_bin_type: str) -> str:
return self.BIN_TYPE_MAP.get(raw_bin_type, raw_bin_type)
@@ -353,7 +731,6 @@ def _parse_collection_rows(self, page_source: str) -> list[dict]:
raw_date = cols[0].get_text(" ", strip=True).replace(",", "")
raw_bin_type = cols[1].get_text(" ", strip=True)
-
try:
collection_date = datetime.strptime(
raw_date, "%A %d %B %Y"
@@ -369,8 +746,7 @@ def _parse_collection_rows(self, page_source: str) -> list[dict]:
)
if not bins:
- raise RuntimeError(
- "Unable to find any collection rows on the South Kesteven results page."
+ raise SiteChanged(
+ "No collection rows were found on the South Kesteven results page."
)
-
return bins
diff --git a/uk_bin_collection/uk_bin_collection/councils/SouthLanarkshireCouncil.py b/uk_bin_collection/uk_bin_collection/councils/SouthLanarkshireCouncil.py
index a53a4811a9..c9e3f31926 100644
--- a/uk_bin_collection/uk_bin_collection/councils/SouthLanarkshireCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/SouthLanarkshireCouncil.py
@@ -9,6 +9,7 @@
logger = logging.getLogger(__name__)
+
# import the wonderful Beautiful Soup and the URL grabber
class CouncilClass(AbstractGetBinDataClass):
"""
@@ -76,7 +77,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
td_cell = row.find("td")
if th_cell is None or td_cell is None:
- logger.warning("Skipping schedule row with missing th or td cell")
+ logger.warning(
+ "Skipping schedule row with missing th or td cell"
+ )
continue
schedule_type = th_cell.get_text().strip()
@@ -91,8 +94,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
if " " not in td_text:
logger.warning(
- "Skipping schedule cadence parsing for unexpected schedule text: %s",
- td_text,
+ "Skipping schedule cadence parsing for unexpected schedule text"
)
schedule_cadence = ""
else:
diff --git a/uk_bin_collection/uk_bin_collection/councils/SouthTynesideCouncil.py b/uk_bin_collection/uk_bin_collection/councils/SouthTynesideCouncil.py
index e5a9e090ba..10f6fdd5b8 100644
--- a/uk_bin_collection/uk_bin_collection/councils/SouthTynesideCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/SouthTynesideCouncil.py
@@ -59,7 +59,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
if user_paon is None:
raise ValueError("Invalid house number")
except Exception as ex:
- print(f"Exception encountered: {ex}")
+ print(f"Exception encountered: {type(ex).__name__}")
print(
"Please check the provided house number. If this error continues, please first trying setting the "
"house number manually on line 25 before raising an issue."
diff --git a/uk_bin_collection/uk_bin_collection/councils/StHelensBC.py b/uk_bin_collection/uk_bin_collection/councils/StHelensBC.py
index f88ea53135..d618e4eaa9 100644
--- a/uk_bin_collection/uk_bin_collection/councils/StHelensBC.py
+++ b/uk_bin_collection/uk_bin_collection/councils/StHelensBC.py
@@ -1,8 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -17,6 +15,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -113,7 +122,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/StaffordshireMoorlandsDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/StaffordshireMoorlandsDistrictCouncil.py
index 79c8281fcb..3fbaf48a79 100644
--- a/uk_bin_collection/uk_bin_collection/councils/StaffordshireMoorlandsDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/StaffordshireMoorlandsDistrictCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
import re
from datetime import datetime
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -22,6 +21,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import WebDriverWait
+
driver = None
try:
user_postcode = kwargs.get("postcode")
@@ -99,7 +108,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
key=lambda x: datetime.strptime(x["collectionDate"], date_format)
)
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/StocktonOnTeesCouncil.py b/uk_bin_collection/uk_bin_collection/councils/StocktonOnTeesCouncil.py
index 2e828c9560..23980b20e3 100644
--- a/uk_bin_collection/uk_bin_collection/councils/StocktonOnTeesCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/StocktonOnTeesCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
from dateutil.relativedelta import relativedelta
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +18,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -163,7 +172,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
print()
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/StroudDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/StroudDistrictCouncil.py
index 6d580311a2..e334798362 100644
--- a/uk_bin_collection/uk_bin_collection/councils/StroudDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/StroudDistrictCouncil.py
@@ -67,7 +67,7 @@ def parse_data(self, page: Any, **kwargs: Any) -> Dict[str, Any]:
date = item.find_all("strong")[1].text.strip()
extracted_data[key] = date
- print("Extracted data:", extracted_data)
+ print(f"Extracted {len(extracted_data)} collection entries")
# Transform the data to the required schema
bin_data = {"bins": []}
diff --git a/uk_bin_collection/uk_bin_collection/councils/SwaleBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/SwaleBoroughCouncil.py
index 62c3ad0c82..93aec1152a 100644
--- a/uk_bin_collection/uk_bin_collection/councils/SwaleBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/SwaleBoroughCouncil.py
@@ -1,7 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -34,6 +33,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
# Get postcode and UPRN from kwargs
@@ -128,7 +137,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/TeignbridgeCouncil.py b/uk_bin_collection/uk_bin_collection/councils/TeignbridgeCouncil.py
index c4b7c87024..a3d6604417 100644
--- a/uk_bin_collection/uk_bin_collection/councils/TeignbridgeCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/TeignbridgeCouncil.py
@@ -1,7 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -16,6 +15,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
user_uprn = kwargs.get("uprn")
@@ -59,7 +68,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/TendringDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/TendringDistrictCouncil.py
index aab8e1e7b4..1367f7aeeb 100644
--- a/uk_bin_collection/uk_bin_collection/councils/TendringDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/TendringDistrictCouncil.py
@@ -1,12 +1,10 @@
+from __future__ import annotations
+
import re
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.common.exceptions import TimeoutException
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import (
check_postcode,
@@ -28,19 +26,30 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
"""
Scrape Tendring District Council's rubbish and recycling collection days for a given address and return upcoming collections.
-
+
This navigates the council's canonical service page, enters the supplied postcode, selects the address by UPRN, parses the resulting waste collection table, and returns a list of future collection entries. Entries with collection dates on or before today are excluded.
-
+
Parameters:
page (str): Ignored; the method always uses the canonical Tendring service URL.
uprn (int | str, via kwargs["uprn"]): Unique Property Reference Number used to select the address.
postcode (str, via kwargs["postcode"]): Postcode to populate the address search field.
web_driver (optional, via kwargs["web_driver"]): Selenium driver configuration or remote endpoint; if omitted a local driver is created.
headless (bool, via kwargs["headless"]): Whether to run the browser headlessly; defaults to True when not provided.
-
+
Returns:
dict: {"bins": [{"type": , "collectionDate": }, ...]} where each entry describes a waste type and its upcoming collection date. `collectionDate` is formatted using the module's configured `date_format`.
"""
+ global By, EC, Select, TimeoutException, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.common.exceptions import TimeoutException
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
bin_data: dict[str, list[dict]] = {"bins": []}
@@ -101,16 +110,16 @@ def parse_data(self, page: str, **kwargs) -> dict:
input_postcode.clear()
input_postcode.send_keys(user_postcode)
# Wait for dropdown to appear and be populated
- dropdown = wait.until(EC.element_to_be_clickable((By.NAME, "selectAddress")))
+ dropdown = wait.until(
+ EC.element_to_be_clickable((By.NAME, "selectAddress"))
+ )
wait.until(lambda _: len(Select(dropdown).options) > 1)
# Select address by UPRN
Select(dropdown).select_by_value(str(user_uprn))
# Wait for results table
- wait.until(
- EC.presence_of_element_located((By.CLASS_NAME, "wasteTable"))
- )
+ wait.until(EC.presence_of_element_located((By.CLASS_NAME, "wasteTable")))
# Parse HTML
soup = BeautifulSoup(driver.page_source, "html.parser")
@@ -145,9 +154,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
continue
# Normalise bin type (strip parentheses)
- bin_type = re.sub(
- r"\([^)]*\)", "", cols[type_idx].get_text(strip=True)
- )
+ bin_type = re.sub(r"\([^)]*\)", "", cols[type_idx].get_text(strip=True))
# Extract a dd/mm/YYYY from the 'Next collection' cell
cell_txt = cols[next_idx].get_text(" ", strip=True)
diff --git a/uk_bin_collection/uk_bin_collection/councils/TestValleyBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/TestValleyBoroughCouncil.py
index 2563c2e061..b6b766c0ee 100644
--- a/uk_bin_collection/uk_bin_collection/councils/TestValleyBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/TestValleyBoroughCouncil.py
@@ -1,9 +1,8 @@
+from __future__ import annotations
+
import datetime as dt
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -11,6 +10,16 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -49,6 +58,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
).click()
import time
+
time.sleep(8)
soup = BeautifulSoup(driver.page_source, features="html.parser")
@@ -68,10 +78,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
next_date = self._parse_date(next_collection_text)
if next_date:
- data["bins"].append({
- "type": bin_type,
- "collectionDate": next_date.strftime(date_format),
- })
+ data["bins"].append(
+ {
+ "type": bin_type,
+ "collectionDate": next_date.strftime(date_format),
+ }
+ )
followed_div = collection.find(
lambda t: (
@@ -82,16 +94,20 @@ def parse_data(self, page: str, **kwargs) -> dict:
if followed_div:
following_text = followed_div.get_text(strip=True)
following_date = self._parse_date(
- following_text.replace("followed by ", "").replace("Followed by ", "")
+ following_text.replace("followed by ", "").replace(
+ "Followed by ", ""
+ )
)
if following_date:
- data["bins"].append({
- "type": bin_type,
- "collectionDate": following_date.strftime(date_format),
- })
+ data["bins"].append(
+ {
+ "type": bin_type,
+ "collectionDate": following_date.strftime(date_format),
+ }
+ )
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
@@ -101,6 +117,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
@staticmethod
def _parse_date(text: str):
import re
+
text = re.sub(r"(st|nd|rd|th)", "", text).strip()
try:
parsed = dt.datetime.strptime(text, "%A %d %B").date()
diff --git a/uk_bin_collection/uk_bin_collection/councils/TewkesburyBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/TewkesburyBoroughCouncil.py
index ac06158f82..c133ca893c 100644
--- a/uk_bin_collection/uk_bin_collection/councils/TewkesburyBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/TewkesburyBoroughCouncil.py
@@ -10,7 +10,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
user_uprn = kwargs.get("uprn")
check_uprn(user_uprn)
- url = f"https://api-2.tewkesbury.gov.uk/incab/rounds/{user_uprn}/next-collection"
+ url = (
+ f"https://api-2.tewkesbury.gov.uk/incab/rounds/{user_uprn}/next-collection"
+ )
response = requests.get(url, timeout=30)
response.raise_for_status()
@@ -19,17 +21,23 @@ def parse_data(self, page: str, **kwargs) -> dict:
data = {"bins": []}
# Legacy format: {"status":"OK","body":[{"collectionType":"...","NextCollection":"YYYY-MM-DD"}]}
- if isinstance(json_data, dict) and json_data.get("status") == "OK" and "body" in json_data:
+ if (
+ isinstance(json_data, dict)
+ and json_data.get("status") == "OK"
+ and "body" in json_data
+ ):
for entry in json_data["body"]:
bin_type = entry.get("collectionType")
date_str = entry.get("NextCollection")
if bin_type and date_str:
try:
collection_date = datetime.strptime(date_str, "%Y-%m-%d")
- data["bins"].append({
- "type": bin_type,
- "collectionDate": collection_date.strftime(date_format)
- })
+ data["bins"].append(
+ {
+ "type": bin_type,
+ "collectionDate": collection_date.strftime(date_format),
+ }
+ )
except ValueError:
continue
# Current format: {"food":{"nextCollectionDate":"...Z"},"garden":{...},"recycling":{...},"refuse":{...}}
@@ -50,10 +58,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
try:
# "2026-04-23T01:00:00.000Z"
collection_date = datetime.strptime(date_str[:10], "%Y-%m-%d")
- data["bins"].append({
- "type": display_name,
- "collectionDate": collection_date.strftime(date_format)
- })
+ data["bins"].append(
+ {
+ "type": display_name,
+ "collectionDate": collection_date.strftime(date_format),
+ }
+ )
except ValueError:
continue
@@ -61,5 +71,5 @@ def parse_data(self, page: str, **kwargs) -> dict:
key=lambda x: datetime.strptime(x["collectionDate"], date_format)
)
- print(json.dumps(data, indent=2))
+ print(f"Parsed {len(data.get('bins', []))} collections")
return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/ThanetDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ThanetDistrictCouncil.py
index 5f4e7afe61..990776f6ce 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ThanetDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ThanetDistrictCouncil.py
@@ -1,11 +1,10 @@
+from __future__ import annotations
+
import json
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -19,6 +18,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import WebDriverWait
+
user_uprn = kwargs.get("uprn")
check_uprn(user_uprn)
bindata = {"bins": []}
@@ -32,7 +41,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
driver = create_webdriver(web_driver, headless, user_agent, __name__)
try:
- print(f"Navigating to URL: {url}")
+ print("Navigating to the configured council page")
driver.get(url)
# Wait for Cloudflare to complete its check
@@ -67,10 +76,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
bindata["bins"].sort(
key=lambda x: datetime.strptime(x.get("collectionDate"), "%d/%m/%Y")
)
- print(bindata)
+ print(f"Parsed {len(bindata.get('bins', []))} collections")
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
print("Cleaning up WebDriver...")
diff --git a/uk_bin_collection/uk_bin_collection/councils/ThreeRiversDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/ThreeRiversDistrictCouncil.py
index eccde4bfae..9e2cfbe4da 100644
--- a/uk_bin_collection/uk_bin_collection/councils/ThreeRiversDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/ThreeRiversDistrictCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import logging
import time
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -18,6 +16,17 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -133,7 +142,7 @@ def click_element(by, value):
return bin_data
except Exception as e:
- logging.error(f"An error occurred: {e}")
+ logging.error(f"An error occurred: {type(e).__name__}")
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/TorbayCouncil.py b/uk_bin_collection/uk_bin_collection/councils/TorbayCouncil.py
index dd27724d1d..ac668f8927 100644
--- a/uk_bin_collection/uk_bin_collection/councils/TorbayCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/TorbayCouncil.py
@@ -1,13 +1,9 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
from dateutil.parser import parse
-from selenium.common.exceptions import NoSuchElementException, TimeoutException
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -16,6 +12,19 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, NoSuchElementException, Select, TimeoutException, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.common.exceptions import NoSuchElementException, TimeoutException
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -28,16 +37,12 @@ def parse_data(self, page: str, **kwargs) -> dict:
check_postcode(user_postcode)
- print(
- f"Starting parse_data with parameters: postcode={user_postcode}, uprn={user_uprn}"
- )
- print(
- f"Creating webdriver with: web_driver={web_driver}, headless={headless}"
- )
+ print("Starting parse_data with configured address parameters")
+ print(f"Creating configured webdriver (headless={headless})")
user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
driver = create_webdriver(web_driver, headless, user_agent, __name__)
- print(f"Navigating to URL: {url}")
+ print("Navigating to the configured council page")
driver.get("https://www.torbay.gov.uk/recycling/bin-collections/")
print("Successfully loaded the page")
@@ -96,7 +101,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
post_code_input.clear()
post_code_input.send_keys(user_postcode)
- print(f"Entered postcode: {user_postcode}")
+ print("Postcode entered")
post_code_input.send_keys(Keys.TAB + Keys.ENTER)
# driver.switch_to.active_element.send_keys(Keys.TAB + Keys.ENTER)
@@ -116,28 +121,28 @@ def parse_data(self, page: str, **kwargs) -> dict:
options = address_select.find_elements(By.TAG_NAME, "option")
print(f"Found {len(options)} options in dropdown")
- # Print all options first for debugging
- print("\nAvailable options:")
+ # Inspect all options without writing their household data to diagnostics.
+ print("Inspecting available address options")
for opt in options:
value = opt.get_attribute("value")
text = opt.text
- print(f"Value: '{value}', Text: '{text}'")
+ print("Address option inspected")
# Try to find our specific UPRN
target_uprn = f"U{user_uprn}|"
- print(f"\nLooking for UPRN pattern: {target_uprn}")
+ print("Looking for the configured property identifier")
found = False
for option in options:
value = option.get_attribute("value")
if value and target_uprn in value:
- print(f"Found matching address with value: {value}")
+ print("Found a matching address")
option.click()
found = True
break
if not found:
- print(f"No matching address found for UPRN: {user_uprn}")
+ print("No matching address found")
return data
print("Address selected successfully")
@@ -209,18 +214,18 @@ def parse_data(self, page: str, **kwargs) -> dict:
"collectionDate": bin_date,
}
data["bins"].append(dict_data)
- print(f"Successfully added collection: {dict_data}")
+ print("Collection added successfully")
except Exception as e:
- print(f"Error processing collection row: {e}")
+ print(f"Error processing collection row ({type(e).__name__})")
continue
# Debug: Print the complete dict_data
print("\nFinal bin collection data:")
- print(data)
+ print(f"Parsed {len(data.get('bins', []))} collections")
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred ({type(e).__name__})")
raise
finally:
print("Cleaning up webdriver...")
diff --git a/uk_bin_collection/uk_bin_collection/councils/TorfaenCouncil.py b/uk_bin_collection/uk_bin_collection/councils/TorfaenCouncil.py
index 1f8cb7075b..df1dad727d 100644
--- a/uk_bin_collection/uk_bin_collection/councils/TorfaenCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/TorfaenCouncil.py
@@ -1,12 +1,10 @@
+from __future__ import annotations
+
import re
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -28,13 +26,22 @@ def _parse_date(text):
return parsed
except ValueError:
continue
- raise ValueError(
- f"Could not parse date '{text}' with any known format"
- )
+ raise ValueError(f"Could not parse date '{text}' with any known format")
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -89,7 +96,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
soup = BeautifulSoup(driver.page_source, "html.parser")
- collections_div = soup.find("h2", string=re.compile(r"Your next collections", re.I))
+ collections_div = soup.find(
+ "h2", string=re.compile(r"Your next collections", re.I)
+ )
if not collections_div:
raise ValueError("Collection results not found on page")
@@ -100,7 +109,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
cards = parent.find_all("h3")
for card_heading in cards:
bin_type = card_heading.get_text(strip=True)
- card = card_heading.find_parent("div", class_=re.compile(r"ant-col|col"))
+ card = card_heading.find_parent(
+ "div", class_=re.compile(r"ant-col|col")
+ )
if not card:
card = card_heading.find_parent("div")
@@ -108,7 +119,8 @@ def parse_data(self, page: str, **kwargs) -> dict:
date_matches = re.findall(
r"(?:Monday|Tuesday|Wednesday|Thursday|Friday|Saturday|Sunday)\s+\d{1,2}\s+\w+",
- card_text, re.I
+ card_text,
+ re.I,
)
seen = set()
@@ -121,15 +133,15 @@ def parse_data(self, page: str, **kwargs) -> dict:
key = (bin_type, cd)
if key not in seen:
seen.add(key)
- data["bins"].append({
- "type": bin_type,
- "collectionDate": cd,
- })
+ data["bins"].append(
+ {
+ "type": bin_type,
+ "collectionDate": cd,
+ }
+ )
if not data["bins"]:
- raise ValueError(
- "No bin collection data found for this address"
- )
+ raise ValueError("No bin collection data found for this address")
data["bins"].sort(
key=lambda x: datetime.strptime(x.get("collectionDate"), date_format)
@@ -138,7 +150,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
return data
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/TowerHamletsCouncil.py b/uk_bin_collection/uk_bin_collection/councils/TowerHamletsCouncil.py
index 155e3139e6..13733aae81 100644
--- a/uk_bin_collection/uk_bin_collection/councils/TowerHamletsCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/TowerHamletsCouncil.py
@@ -1,12 +1,10 @@
+from __future__ import annotations
+
import re
import time
from datetime import datetime, timedelta
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +18,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
page = (
@@ -126,9 +135,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
if not selected:
- raise ValueError(
- f"No addresses found for postcode {user_postcode}"
- )
+ raise ValueError(f"No addresses found for postcode {user_postcode}")
time.sleep(8)
@@ -190,7 +197,7 @@ def _field(name):
return data
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/UttlesfordDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/UttlesfordDistrictCouncil.py
index c293eac3cc..4d4b6f7217 100644
--- a/uk_bin_collection/uk_bin_collection/councils/UttlesfordDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/UttlesfordDistrictCouncil.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
import logging
import pickle
import time
@@ -5,12 +7,6 @@
import requests
from bs4 import BeautifulSoup
-from selenium import webdriver
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -24,6 +20,19 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait, webdriver
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium import webdriver
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
user_paon = kwargs.get("paon")
@@ -72,15 +81,21 @@ def parse_data(self, page: str, **kwargs) -> dict:
paon_lower = str(user_paon).strip().lower()
for option in drop_down_values.options:
text = option.text.strip().lower()
- if text and paon_lower and (text.startswith(paon_lower + " ") or text.startswith(paon_lower + ",") or text == paon_lower):
+ if (
+ text
+ and paon_lower
+ and (
+ text.startswith(paon_lower + " ")
+ or text.startswith(paon_lower + ",")
+ or text == paon_lower
+ )
+ ):
option.click()
matched = True
break
if not matched:
- raise ValueError(
- f"Address '{user_paon}' not found in dropdown"
- )
+ raise ValueError(f"Address '{user_paon}' not found in dropdown")
input_element_address_btn = wait.until(
EC.element_to_be_clickable(
@@ -135,7 +150,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
return bin_data
except Exception as e:
- logging.error(f"An error occurred: {e}")
+ logging.error(f"An error occurred: {type(e).__name__}")
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/WakefieldCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/WakefieldCityCouncil.py
index 7ee5d2f52a..43d90cdec6 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WakefieldCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WakefieldCityCouncil.py
@@ -68,7 +68,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
data["bins"].append(dict_data)
except ValueError:
# Skip if the date isn't a valid date
- print(f"Skipping invalid date: {date_text}")
+ print("Skipping an invalid collection date")
# Get future collections
future_collections_section = row.find("ul", class_="u-mt-4")
@@ -98,7 +98,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except ValueError:
# Skip if the future collection date isn't valid
print(
- f"Skipping invalid future date: {future_date_text}"
+ "Skipping an invalid future collection date"
)
# Sort the collections by date
@@ -106,7 +106,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
key=lambda x: datetime.strptime(x.get("collectionDate"), date_format)
)
except Exception as e:
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
raise
finally:
if driver:
diff --git a/uk_bin_collection/uk_bin_collection/councils/WalsallCouncil.py b/uk_bin_collection/uk_bin_collection/councils/WalsallCouncil.py
index 9a7cf8e4ea..bd68a0253f 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WalsallCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WalsallCouncil.py
@@ -40,9 +40,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
bin_url = "https://cag.walsall.gov.uk" + item.contents[1]["href"]
r = requests.get(bin_url, headers=headers)
if r.status_code != 200:
- print(
- f"Collection details for {bin_colour.lower()} bin could not be retrieved."
- )
+ print("Collection details could not be retrieved")
break
soup = BeautifulSoup(r.text, "html.parser")
table = soup.findAll("tr")
diff --git a/uk_bin_collection/uk_bin_collection/councils/WalthamForest.py b/uk_bin_collection/uk_bin_collection/councils/WalthamForest.py
index 1582eef726..d8789fe6ba 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WalthamForest.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WalthamForest.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +18,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
page = "https://portal.walthamforest.gov.uk/AchieveForms/?mode=fill&consentMessage=yes&form_uri=sandbox-publish://AF-Process-d62ccdd2-3de9-48eb-a229-8e20cbdd6393/AF-Stage-8bf39bf9-5391-4c24-857f-0dc2025c67f4/definition.json&process=1&process_uri=sandbox-processes://AF-Process-d62ccdd2-3de9-48eb-a229-8e20cbdd6393&process_id=AF-Process-d62ccdd2-3de9-48eb-a229-8e20cbdd6393"
@@ -117,7 +126,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/WestBerkshireCouncil.py b/uk_bin_collection/uk_bin_collection/councils/WestBerkshireCouncil.py
index 08c77e0558..ca62d70d52 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WestBerkshireCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WestBerkshireCouncil.py
@@ -1,11 +1,9 @@
+from __future__ import annotations
+
import time
from bs4 import BeautifulSoup
from dateutil.relativedelta import relativedelta
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +18,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -136,7 +145,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
print()
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/WestDevonBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/WestDevonBoroughCouncil.py
index eac75e5230..ddd7902d43 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WestDevonBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WestDevonBoroughCouncil.py
@@ -1,12 +1,10 @@
+from __future__ import annotations
+
import json
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium import webdriver
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -20,6 +18,17 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait, webdriver
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium import webdriver
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import WebDriverWait
+
user_uprn = kwargs.get("uprn")
user_postcode = kwargs.get("postcode")
user_paon = kwargs.get("paon")
@@ -34,9 +43,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
driver.get(base_url)
WebDriverWait(driver, 15).until(
- EC.presence_of_element_located(
- (By.CSS_SELECTOR, "input[type='text']")
- )
+ EC.presence_of_element_located((By.CSS_SELECTOR, "input[type='text']"))
)
fcc_token = None
@@ -65,9 +72,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
addresses = result.get("addresses", {})
if not addresses:
- raise ValueError(
- f"No addresses found for postcode {user_postcode}"
- )
+ raise ValueError(f"No addresses found for postcode {user_postcode}")
paon_lower = user_paon.lower().strip()
matched_uprn = None
@@ -123,9 +128,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
for coll in collections:
service_name = coll.find("h3").text.strip()
- det_wrap = coll.find(
- "div", class_="wdshDetWrap"
- ) or coll.find("div", class_="detWrap")
+ det_wrap = coll.find("div", class_="wdshDetWrap") or coll.find(
+ "div", class_="detWrap"
+ )
if not det_wrap:
continue
@@ -141,9 +146,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
if next_collection.startswith("today"):
next_collection = next_collection.split("today, ")[1]
elif next_collection.startswith("tomorrow"):
- next_collection = next_collection.split(
- "tomorrow, "
- )[1]
+ next_collection = next_collection.split("tomorrow, ")[1]
collection_date = datetime.strptime(
next_collection, "%A, %d %B %Y"
@@ -161,9 +164,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
bindata["bins"].append(dict_data)
bindata["bins"].sort(
- key=lambda x: datetime.strptime(
- x.get("collectionDate"), "%d/%m/%Y"
- )
+ key=lambda x: datetime.strptime(x.get("collectionDate"), "%d/%m/%Y")
)
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/WestLothianCouncil.py b/uk_bin_collection/uk_bin_collection/councils/WestLothianCouncil.py
index cad85b83c1..09d2dfe576 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WestLothianCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WestLothianCouncil.py
@@ -1,7 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -16,6 +15,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -94,7 +103,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/WestOxfordshireDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/WestOxfordshireDistrictCouncil.py
index 7138ed69ee..bed4c3c93c 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WestOxfordshireDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WestOxfordshireDistrictCouncil.py
@@ -1,12 +1,9 @@
+from __future__ import annotations
+
import time
from datetime import datetime
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -24,19 +21,31 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
"""
Query West Oxfordshire's waste collection site for a property's bin types and upcoming collection dates.
-
+
Parameters:
page (str): Ignored; the function always queries the West Oxfordshire waste collection enquiry page.
paon (str, in kwargs): Property house number or name.
postcode (str, in kwargs): Property postcode.
web_driver (str or WebDriver, in kwargs): WebDriver identifier or instance used to create the Selenium driver.
headless (bool, in kwargs): Whether to run the browser in headless mode.
-
+
Returns:
dict: A dictionary with a "bins" key mapping to a list of objects, each containing:
- "type" (str): Bin/container type (e.g., "General waste", "Recycling").
- "collectionDate" (str): Next collection date formatted as "DD/MM/YYYY".
"""
+ global By, EC, Keys, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
page = "https://community.westoxon.gov.uk/s/waste-collection-enquiry"
@@ -53,7 +62,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
driver = create_webdriver(web_driver, headless, user_agent, __name__)
driver.get(page)
-
+
# Scroll to top-left to ensure components are visible
driver.execute_script("window.scrollTo(0, 0);")
@@ -67,7 +76,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
address_entry_field.click()
address_entry_field.send_keys(str(full_address))
-
+
# Trigger dropdown by sending backspace and re-typing last character
time.sleep(1) # Give the search a moment to process
address_entry_field.send_keys(Keys.BACKSPACE)
@@ -77,7 +86,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
# The second item contains the actual address (first is "Show more results")
first_found_address = wait.until(
EC.element_to_be_clickable(
- (By.XPATH, '(//lightning-base-combobox-item)[2]')
+ (By.XPATH, "(//lightning-base-combobox-item)[2]")
)
)
first_found_address.click()
@@ -129,11 +138,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
)
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
# This block ensures that the driver is closed regardless of an exception
if driver:
driver.quit()
- return data
\ No newline at end of file
+ return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/WinchesterCityCouncil.py b/uk_bin_collection/uk_bin_collection/councils/WinchesterCityCouncil.py
index 41fd2a3f65..4873dd8582 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WinchesterCityCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WinchesterCityCouncil.py
@@ -1,7 +1,6 @@
+from __future__ import annotations
+
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
@@ -18,7 +17,7 @@ class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
"""
Parse Winchester council bin calendar and extract upcoming collection types and dates.
-
+
Parameters:
page (str): Unused by this implementation; kept for interface compatibility.
**kwargs:
@@ -26,15 +25,25 @@ def parse_data(self, page: str, **kwargs) -> dict:
postcode (str): Postcode to search for addresses.
web_driver: Optional identifier or configuration for the Selenium webdriver.
headless (bool): Whether to run the webdriver in headless mode.
-
+
Returns:
dict: A dictionary with a single key "bins" whose value is a list of dictionaries, each containing:
- "type" (str): The bin type/name.
- "collectionDate" (str): Collection date formatted as "dd/mm/YYYY".
-
+
Raises:
ValueError: If the page does not contain the expected collections container.
"""
+ global By, EC, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -92,7 +101,9 @@ def parse_data(self, page: str, **kwargs) -> dict:
# Find the main container and then each card. Use class contains so small CSS changes don't break parsing.
recyclingcalendar = soup.find(
"div",
- class_=lambda c: c and "ant-row" in c and "justify-content-between" in c,
+ class_=lambda c: c
+ and "ant-row" in c
+ and "justify-content-between" in c,
)
if not recyclingcalendar:
@@ -135,11 +146,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
# This block ensures that the driver is closed regardless of an exception
if driver:
driver.quit()
- return data
\ No newline at end of file
+ return data
diff --git a/uk_bin_collection/uk_bin_collection/councils/WirralCouncil.py b/uk_bin_collection/uk_bin_collection/councils/WirralCouncil.py
index 485bb2d1cf..02d27dfb79 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WirralCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WirralCouncil.py
@@ -154,7 +154,11 @@ def parse_data(self, page: str, **kwargs) -> dict:
continue
try:
parsed_month = datetime.strptime(month_name, "%B").month
- year = current_year + 1 if parsed_month < current_month else current_year
+ year = (
+ current_year + 1
+ if parsed_month < current_month
+ else current_year
+ )
collection_date = datetime(
year,
parsed_month,
@@ -169,7 +173,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
)
except ValueError as e:
- logging.warning(f"Failed to parse date {day} {month_name}: {e}")
+ logging.warning(
+ "Failed to parse a primary date (%s).",
+ type(e).__name__,
+ )
j += 1
else:
i += 1
@@ -182,15 +189,17 @@ def parse_data(self, page: str, **kwargs) -> dict:
if parent:
text = parent.get_text(separator="\n", strip=True)
# Parse "Friday 15 May\nNon-recyclable waste" pattern
- date_match = re.search(
- r"(\w+day)\s+(\d{1,2})\s+(\w+)", text
- )
+ date_match = re.search(r"(\w+day)\s+(\d{1,2})\s+(\w+)", text)
if date_match:
day = int(date_match.group(2))
month = date_match.group(3)
try:
parsed_month = datetime.strptime(month, "%B").month
- year = current_year + 1 if parsed_month < current_month else current_year
+ year = (
+ current_year + 1
+ if parsed_month < current_month
+ else current_year
+ )
collection_date = datetime.strptime(
f"{day} {month} {year}", "%d %B %Y"
)
@@ -210,7 +219,10 @@ def parse_data(self, page: str, **kwargs) -> dict:
}
)
except ValueError as e:
- logging.warning(f"Failed to parse fallback date {day} {month}: {e}")
+ logging.warning(
+ "Failed to parse a fallback date (%s).",
+ type(e).__name__,
+ )
if not data["bins"]:
raise ValueError("No collection data found on page")
diff --git a/uk_bin_collection/uk_bin_collection/councils/WrexhamCountyBoroughCouncil.py b/uk_bin_collection/uk_bin_collection/councils/WrexhamCountyBoroughCouncil.py
index fb707e5459..69da97d092 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WrexhamCountyBoroughCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WrexhamCountyBoroughCouncil.py
@@ -1,14 +1,12 @@
+from __future__ import annotations
+
from time import sleep
from bs4 import BeautifulSoup
-from selenium.webdriver.common.by import By
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select, WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.get_bin_data import AbstractGetBinDataClass
-
# import the wonderful Beautiful Soup and the URL grabber
@@ -20,6 +18,16 @@ class CouncilClass(AbstractGetBinDataClass):
"""
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Select, WebDriverWait
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select, WebDriverWait
+
driver = None
try:
page = "https://www.wrexham.gov.uk/service/when-are-my-bins-collected"
@@ -114,7 +122,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
except Exception as e:
# Here you can log the exception if needed
- print(f"An error occurred: {e}")
+ print(f"An error occurred: {type(e).__name__}")
# Optionally, re-raise the exception if you want it to propagate
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/WychavonDistrictCouncil.py b/uk_bin_collection/uk_bin_collection/councils/WychavonDistrictCouncil.py
index 15bf8b7eb8..70ec2588d2 100644
--- a/uk_bin_collection/uk_bin_collection/councils/WychavonDistrictCouncil.py
+++ b/uk_bin_collection/uk_bin_collection/councils/WychavonDistrictCouncil.py
@@ -1,15 +1,11 @@
+from __future__ import annotations
+
import logging
import pickle
import time
import requests
from bs4 import BeautifulSoup
-from selenium import webdriver
-from selenium.webdriver.common.by import By
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.support import expected_conditions as EC
-from selenium.webdriver.support.ui import Select
-from selenium.webdriver.support.wait import WebDriverWait
from uk_bin_collection.uk_bin_collection.common import *
from uk_bin_collection.uk_bin_collection.common import *
@@ -24,6 +20,19 @@
class CouncilClass(AbstractGetBinDataClass):
def parse_data(self, page: str, **kwargs) -> dict:
+ global By, EC, Keys, Select, WebDriverWait, webdriver
+ from uk_bin_collection.uk_bin_collection.common import (
+ ensure_selenium_dependencies,
+ )
+
+ ensure_selenium_dependencies()
+ from selenium import webdriver
+ from selenium.webdriver.common.by import By
+ from selenium.webdriver.common.keys import Keys
+ from selenium.webdriver.support import expected_conditions as EC
+ from selenium.webdriver.support.ui import Select
+ from selenium.webdriver.support.wait import WebDriverWait
+
driver = None
try:
data = {"bins": []}
@@ -148,7 +157,7 @@ def parse_data(self, page: str, **kwargs) -> dict:
return bin_data
except Exception as e:
- logging.error(f"An error occurred: {e}")
+ logging.error(f"An error occurred: {type(e).__name__}")
raise
finally:
diff --git a/uk_bin_collection/uk_bin_collection/councils/tests/test_south_kesteven_district_council.py b/uk_bin_collection/uk_bin_collection/councils/tests/test_south_kesteven_district_council.py
index 221a2993ed..2224e36d15 100644
--- a/uk_bin_collection/uk_bin_collection/councils/tests/test_south_kesteven_district_council.py
+++ b/uk_bin_collection/uk_bin_collection/councils/tests/test_south_kesteven_district_council.py
@@ -1,74 +1,81 @@
-"""Unit tests for the South Kesteven District Council live checker flow."""
+"""Deterministic tests for the South Kesteven browser-only checker flow."""
+
+from __future__ import annotations
import json
-from unittest.mock import ANY, MagicMock, patch
+from types import SimpleNamespace
+from unittest.mock import MagicMock, call, patch
import pytest
-from selenium.webdriver.common.by import By
+from uk_bin_collection.uk_bin_collection.collect_data import UKBinCollectionApp
from uk_bin_collection.uk_bin_collection.councils.SouthKestevenDistrictCouncil import (
CouncilClass,
)
+from uk_bin_collection.uk_bin_collection.exceptions import (
+ AddressMismatchError,
+ BrowserUnavailableError,
+ ConfigurationError,
+ SiteChanged,
+ UpstreamAccessDenied,
+)
MODULE_PATH = (
"uk_bin_collection.uk_bin_collection.councils.SouthKestevenDistrictCouncil"
)
CHECKER_URL = (
- "https://selfservice.southkesteven.gov.uk/renderform?"
- "t=213&k=2074C945A63DDC0D18F1EB74DA230AC3122958B1"
+ "https://selfservice.southkesteven.gov.uk/renderform?" "t=213&k=public-test-token"
)
BINDAY_HTML = f"""
-
-
-
- Postcode bin day checker
-
-
-
+
+ Postcode bin day checker
+
"""
RESULTS_HTML = """
-
-
- Your Collections
-
-
- | Thursday 16 April, 2026 |
- 240 Litre Refuse |
-
-
- | Thursday 23 April, 2026 |
- 240 Litre Recycling |
-
-
- | Thursday 30 April, 2026 |
- 23lt Food Caddy |
-
-
- | Thursday 07 May, 2026 |
- 240 Litre Paper and Card |
-
-
- Your Collections
+
+ | Thursday 16 April, 2026 | 240 Litre Refuse |
+ | Thursday 23 April, 2026 | 240 Litre Recycling |
+ | Thursday 30 April, 2026 | 23lt Food Caddy |
+ | Thursday 07 May, 2026 | 240 Litre Paper and Card |
+
+
- Your Collections
-
-
- | Thursday 30 April, 2026 |
- Glass Box Collection |
-
-
- Your Collections
+
+ | Thursday 30 April, 2026 | Glass Box Collection |
+
+ |