diff --git a/adr/20260310-seqera-dataset-filesystem.md b/adr/20260310-seqera-dataset-filesystem.md new file mode 100644 index 0000000000..3e03f84c7b --- /dev/null +++ b/adr/20260310-seqera-dataset-filesystem.md @@ -0,0 +1,136 @@ +# NIO Filesystem for Seqera Platform Datasets + +- Authors: Jorge Ejarque +- Status: draft +- Date: 2026-03-10 +- Tags: nio, filesystem, seqera, datasets, nf-tower + +Technical Story: Enable Nextflow pipelines to read Seqera Platform datasets as ordinary file paths using `seqera://` URIs. + +## Summary + +Add a Java NIO `FileSystemProvider` to the `nf-tower` plugin that registers the `seqera://` scheme, allowing pipelines to reference Seqera Platform datasets (CSV/TSV) as standard file paths without manual download steps. The implementation reuses the existing `TowerClient` for all HTTP communication, inheriting authentication and retry behaviour. + +## Problem Statement + +Nextflow users managing datasets on the Seqera Platform must currently download dataset files manually or through custom scripts before referencing them in pipelines. There is no native integration between Nextflow's file abstraction and the Seqera Platform dataset API. This creates friction in workflows where datasets are the primary input and forces users to handle authentication, versioning, and file staging outside the pipeline definition. + +## Goals or Decision Drivers + +- Transparent access to Seqera Platform datasets using standard Nextflow file path syntax +- Reuse of existing nf-tower plugin infrastructure (authentication, HTTP client, retry/backoff) +- Hierarchical path browsing matching the platform's org/workspace/dataset structure +- Extensible architecture that can support future Seqera-managed resource types (e.g. data-links) +- No new plugin or module — feature lives within nf-tower + +## Non-goals + +- Streaming large datasets — the Platform API does not support streaming; content is fully buffered on download +- Implementing resource types beyond `datasets` — only the extensible architecture is required +- Local caching across pipeline runs — Nextflow's standard task staging handles caching +- Dataset management operations (delete, rename) — the filesystem is read-only in the initial implementation + +## Considered Options + +### Option 1: Standalone plugin with dedicated HTTP client + +A new `nf-seqera-fs` plugin with its own HTTP client configuration and authentication setup. + +- Good, because it isolates the filesystem code from the nf-tower plugin +- Bad, because it duplicates authentication configuration and HTTP client setup +- Bad, because two separate HTTP clients sharing a refresh token would corrupt each other's auth state + +### Option 2: NIO filesystem within nf-tower using TowerClient delegation + +Add the filesystem to nf-tower, delegating all HTTP through the existing `TowerClient` singleton via a typed `SeqeraDatasetClient` wrapper. + +- Good, because it shares authentication and token refresh with TowerClient +- Good, because it reuses existing retry/backoff configuration +- Good, because no new dependencies are needed + +### Option 3: Direct HxClient usage within nf-tower + +Add the filesystem to nf-tower but use `HxClient` directly rather than going through TowerClient. + +- Good, because it gives full control over request construction +- Bad, because exposing HxClient internals couples the filesystem to implementation details +- Bad, because token refresh coordination with TowerClient becomes manual + +## Solution or decision outcome + +Option 2 — NIO filesystem within nf-tower using TowerClient delegation. All HTTP calls go through `TowerClient.sendApiRequest()`, ensuring a single point of authentication and retry logic. + +## Rationale & discussion + +### Path Hierarchy + +The `seqera://` path encodes the Platform's organizational structure directly: + +``` +seqera:// → ROOT (directory, depth 0) + └── / → ORGANIZATION (directory, depth 1) + └── / → WORKSPACE (directory, depth 2) + └── datasets/ → RESOURCE TYPE (directory, depth 3) + └── [@] → DATASET (file, depth 4) +``` + +Each level is a directory except the leaf dataset, which is a file. Version pinning uses an `@version` suffix on the dataset name segment (e.g. `seqera://acme/research/datasets/samples@2`). Without it, the latest non-disabled version is resolved. + +### Name-to-ID Resolution + +The path uses human-readable names but the Platform API requires numeric IDs. Resolution is built from two API calls at filesystem initialization: + +1. `GET /user-info` → obtain `userId` +2. `GET /user/{userId}/workspaces` → returns all accessible org/workspace pairs + +This single source provides both directory listing content and name→ID mapping. Results are cached in `SeqeraFileSystem` with invalidation on write operations. `GET /orgs` is intentionally not used as it returns all platform orgs, not scoped to user membership. + +### Component Structure + +``` +plugins/nf-tower/src/main/io/seqera/tower/plugin/ +├── fs/ ← NIO layer +│ ├── SeqeraFileSystemProvider ← FileSystemProvider (scheme: "seqera") +│ ├── SeqeraFileSystem ← FileSystem with org/workspace/dataset caches +│ ├── SeqeraPath ← Path implementation (depth 0–4) +│ ├── SeqeraFileAttributes ← BasicFileAttributes +│ ├── SeqeraPathFactory ← PF4J FileSystemPathFactory extension +│ └── DatasetInputStream ← SeekableByteChannel over InputStream +├── dataset/ ← API client layer +│ ├── SeqeraDatasetClient ← Typed HTTP client wrapping TowerClient +│ ├── DatasetDto ← Dataset API response model +│ ├── DatasetVersionDto ← Version API response model +│ ├── OrgAndWorkspaceDto ← Org/workspace list model +│ └── WorkspaceOrgDto ← Workspace/org mapping model +└── resources/META-INF/services/ + └── java.nio.file.spi.FileSystemProvider +``` + +### Key Design Decisions + +1. **TowerClient delegation**: `SeqeraDatasetClient` delegates all HTTP through `TowerFactory.client()` → `TowerClient.sendApiRequest()`. This ensures shared authentication state and avoids the token refresh corruption that would occur with separate HTTP client instances. + +2. **One filesystem per JVM**: `SeqeraFileSystemProvider` maintains a single `SeqeraFileSystem` keyed by scheme. This matches the `TowerClient` singleton-per-session pattern. + +3. **Read-only initial scope**: The filesystem reports `isReadOnly()=true`. Write support (dataset upload via multipart POST) is deferred to a future iteration. + +4. **Download filename constraint**: The Platform API's download endpoint (`GET /datasets/{id}/v/{version}/n/{fileName}`) requires the exact filename from upload time. The implementation always resolves `DatasetVersionDto.fileName` from `GET /datasets/{id}/versions` before constructing the download URL. + +5. **Extensible resource types**: The path hierarchy reserves depth 3 for a resource type segment (currently only `datasets`). Adding support for data-links or other resource types requires only a new handler at the directory listing and I/O layers, with no changes to path resolution or authentication. + +6. **Thread safety**: `SeqeraFileSystem` cache methods and `SeqeraFileSystemProvider` lifecycle methods are `synchronized`. The filesystem map uses `LinkedHashMap` with external synchronization rather than `ConcurrentHashMap`, matching the low-contention access pattern. + +### Limitations + +- **No size metadata**: `SeqeraFileAttributes.size()` returns 0 for all paths because the Platform API does not expose content length in dataset metadata. +- **Single endpoint per JVM**: The filesystem key is scheme-only; concurrent access to different Platform endpoints in the same JVM is not supported. + +### Streaming Downloads + +Dataset downloads use `TowerClient.sendStreamingRequest()` which calls `HxClient.sendAsStream()` — the response body is returned as an `InputStream` streamed directly from the HTTP connection. This avoids the triple-buffering problem (`String` → `getBytes()` → `ByteArrayInputStream`) that would otherwise consume ~40 MB heap per 10 MB dataset. The `HxClient.sendAsStream()` method goes through the same `sendWithRetry()` path as `sendAsString()`, so retry logic and token refresh are preserved. + +## Links + +- [Spec](../specs/260310-seqera-dataset-fs/spec.md) +- [Implementation plan](../specs/260310-seqera-dataset-fs/plan.md) +- [Data model](../specs/260310-seqera-dataset-fs/data-model.md) diff --git a/modules/nf-commons/src/main/nextflow/file/FileHelper.groovy b/modules/nf-commons/src/main/nextflow/file/FileHelper.groovy index 035f4d6cfa..2ded21331e 100644 --- a/modules/nf-commons/src/main/nextflow/file/FileHelper.groovy +++ b/modules/nf-commons/src/main/nextflow/file/FileHelper.groovy @@ -362,7 +362,7 @@ class FileHelper { return asPath(toPathURI(str)) } - static final private Map PLUGINS_MAP = [s3:'nf-amazon', gs:'nf-google', az:'nf-azure'] + static final private Map PLUGINS_MAP = [s3:'nf-amazon', gs:'nf-google', az:'nf-azure', seqera:'nf-tower'] static final private Map SCHEME_CHECKED = new HashMap<>() @@ -373,6 +373,7 @@ class FileHelper { // find out the default plugin for the given scheme and try to load it final pluginId = PLUGINS_MAP.get(scheme) if( pluginId ) try { + log.debug "Detected required plugin '$pluginId'" if( Plugins.startIfMissing(pluginId) ) { log.debug "Started plugin '$pluginId' required to handle file: $str" // return true to signal a new plugin was loaded diff --git a/plugins/nf-tower/build.gradle b/plugins/nf-tower/build.gradle index e47fe5ad37..4d77f48afc 100644 --- a/plugins/nf-tower/build.gradle +++ b/plugins/nf-tower/build.gradle @@ -32,7 +32,8 @@ nextflowPlugin { 'io.seqera.tower.plugin.TowerFactory', 'io.seqera.tower.plugin.TowerFusionToken', 'io.seqera.tower.plugin.auth.AuthCommandImpl', - 'io.seqera.tower.plugin.launch.LaunchCommandImpl' + 'io.seqera.tower.plugin.launch.LaunchCommandImpl', + 'io.seqera.tower.plugin.fs.SeqeraPathFactory' ] } @@ -57,6 +58,9 @@ dependencies { compileOnly 'io.seqera:lib-httpx:2.1.0' api 'io.seqera:lib-platform-oidc:0.1.0' + api('io.seqera:tower-api:1.121.0') { + exclude group: 'io.micronaut.servlet' + } api "com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.21.1" api "com.fasterxml.jackson.core:jackson-databind:2.21.1" diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/BaseCommandImpl.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/BaseCommandImpl.groovy index f415efa852..b4a2a7e0f2 100644 --- a/plugins/nf-tower/src/main/io/seqera/tower/plugin/BaseCommandImpl.groovy +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/BaseCommandImpl.groovy @@ -19,44 +19,26 @@ package io.seqera.tower.plugin import groovy.json.JsonSlurper import groovy.transform.Memoized import groovy.util.logging.Slf4j -import io.seqera.http.HxClient import nextflow.Const +import nextflow.SysEnv import nextflow.config.ConfigBuilder - -import java.net.http.HttpRequest -import java.net.http.HttpResponse -import java.time.Duration +import nextflow.util.Duration @Slf4j class BaseCommandImpl { - private static final int API_TIMEOUT_MS = 10_000 + protected static final int API_TIMEOUT_MS = 10_000 /** - * Provides common API operations for Seqera Platform - */ - protected TowerCommonApi commonApi - - BaseCommandImpl(){ - this.commonApi = new TowerCommonApi() - } - - BaseCommandImpl( TowerCommonApi commonApi ) { - this.commonApi = commonApi - } - - /** - * Creates an HxClient instance with optional authentication token. + * Creates a TowerClient instance with optional authentication token. * + * @param apiUrl Seqera Platform API url * @param accessToken Optional personal access token for authentication (PAT) - * @return Configured HxClient instance with timeout settings + * @return Configured TowerClient instance with timeout settings */ @Memoized - protected HxClient createHttpClient(String accessToken = null) { - return HxClient.newBuilder() - .connectTimeout(Duration.ofMillis(API_TIMEOUT_MS)) - .bearerToken(accessToken) - .build() + protected TowerClient createTowerClient(String apiUrl, String accessToken) { + return new TowerClient( new TowerConfig( [accessToken: accessToken, endpoint: apiUrl, httpConnectTimeout: Duration.of(API_TIMEOUT_MS)], SysEnv.get())) } /** @@ -73,69 +55,26 @@ class BaseCommandImpl { return builder.buildConfigObject().flatten() } - protected List listUserWorkspaces(HxClient client, String endpoint, String userId) { - final url = "${endpoint}/user/${userId}/workspaces" - log.debug "Platform list workspaces - GET ${url}" - final request = HttpRequest.newBuilder() - .uri(URI.create(url)) - .GET() - .build() - - final response = client.send(request, HttpResponse.BodyHandlers.ofString()) - - if( response.statusCode() != 200 ) { - final error = response.body() ?: "HTTP ${response.statusCode()}" - throw new RuntimeException("Failed to get workspaces: ${error}") - } - - final json = new JsonSlurper().parseText(response.body()) as Map - final orgsAndWorkspaces = json.orgsAndWorkspaces as List - - return orgsAndWorkspaces.findAll { ((Map) it).workspaceId != null } + protected List listUserWorkspaces(TowerClient client, String userId) { + return client.listUserWorkspacesAndOrgs(userId).findAll { ((Map) it).workspaceId != null } } - protected List listComputeEnvironments(HxClient client, String endpoint, String workspaceId) { - final uri = workspaceId - ? "${endpoint}/compute-envs?workspaceId=${workspaceId}" - : "${endpoint}/compute-envs" - log.debug "Platform list compute env - GET ${uri}" - - final request = HttpRequest.newBuilder() - .uri(URI.create(uri)) - .GET() - .build() - - final response = client.send(request, HttpResponse.BodyHandlers.ofString()) - - if( response.statusCode() != 200 ) { - final error = response.body() ?: "HTTP ${response.statusCode()}" - throw new RuntimeException("Failed to get compute environments: ${error}") + protected List listComputeEnvironments(TowerClient client, String workspaceId) { + try { + final json = client.apiGet("/compute-envs", workspaceId ? [workspaceId: workspaceId] : [:]) + return json.computeEnvs as List ?: [] + } catch ( Exception e ) { + throw new RuntimeException("Failed to get compute environments: ${e.message}", e) } - - final json = new JsonSlurper().parseText(response.body()) as Map - return json.computeEnvs as List ?: [] } - protected Map getComputeEnvironment(HxClient client, String endpoint, String computeEnvId, String workspaceId) { - final uri = workspaceId ? - "${endpoint}/compute-envs/${computeEnvId}?workspaceId=${workspaceId}" : - "${endpoint}/compute-envs" - log.debug "Platform get compute env - GET ${uri}" - - final request = HttpRequest.newBuilder() - .uri(URI.create(uri)) - .GET() - .build() - - final response = client.send(request, HttpResponse.BodyHandlers.ofString()) - - if( response.statusCode() != 200 ) { - final error = response.body() ?: "HTTP ${response.statusCode()}" - throw new RuntimeException("Failed to get compute environment: ${error}") + protected Map getComputeEnvironment(TowerClient client, String computeEnvId, String workspaceId) { + try { + final json = client.apiGet(workspaceId ? "/compute-envs/${computeEnvId}" : "/compute-envs", workspaceId ? [workspaceId: workspaceId] : [:]) + return unifyComputeEnvDescription(json.computeEnv as Map ?: [:]) + } catch ( Exception e ) { + throw new RuntimeException("Failed to get compute environments: ${e.message}", e) } - - final json = new JsonSlurper().parseText(response.body()) as Map - return unifyComputeEnvDescription(json.computeEnv as Map ?: [:]) } private Map unifyComputeEnvDescription(Map computeEnv) { diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy index 73974530c6..fc97b5d50f 100644 --- a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerClient.groovy @@ -18,58 +18,34 @@ package io.seqera.tower.plugin import java.net.http.HttpClient import java.net.http.HttpRequest -import java.time.Instant -import java.time.OffsetDateTime -import java.time.ZoneId -import java.util.concurrent.ConcurrentHashMap -import java.util.concurrent.LinkedBlockingQueue -import java.util.concurrent.TimeUnit import groovy.json.JsonGenerator import groovy.json.JsonOutput import groovy.json.JsonSlurper import groovy.transform.CompileStatic -import groovy.transform.ToString import groovy.transform.TupleConstructor import groovy.util.logging.Slf4j import io.seqera.http.HxClient +import io.seqera.tower.plugin.exception.ForbiddenException +import io.seqera.tower.plugin.exception.NotFoundException +import io.seqera.tower.plugin.exception.UnauthorizedException import io.seqera.util.trace.TraceUtils import nextflow.BuildInfo -import nextflow.Session -import nextflow.container.resolver.ContainerMeta +import nextflow.SysEnv import nextflow.exception.AbortOperationException -import nextflow.processor.TaskHandler -import nextflow.processor.TaskId -import nextflow.processor.TaskProcessor -import nextflow.script.PlatformMetadata -import nextflow.trace.ResourcesAggregator -import nextflow.trace.TraceObserverV2 -import nextflow.trace.TraceRecord -import nextflow.trace.event.FilePublishEvent -import nextflow.trace.event.TaskEvent import nextflow.util.Duration -import nextflow.util.LoggerHelper -import nextflow.util.ProcessHelper import nextflow.util.TestOnly -import nextflow.util.Threads /** - * Send out messages via HTTP to a configured URL on different workflow - * execution events. + * Perform HTTP call to Seqera platform. * * @author Paolo Di Tommaso */ @Slf4j @CompileStatic -class TowerClient implements TraceObserverV2 { +class TowerClient { static final public String DEF_ENDPOINT_URL = 'https://api.cloud.seqera.io' - static private final int TASKS_PER_REQUEST = 100 - - static private final Duration REQUEST_INTERVAL = Duration.of('1 sec') - - static private final Duration ALIVE_INTERVAL = Duration.of('1 min') - static private final String TOKEN_PREFIX = '@token:' @TupleConstructor @@ -80,139 +56,40 @@ class TowerClient implements TraceObserverV2 { boolean isError() { code < 200 || code >= 300 } } - @ToString(includeNames = true) - static class ProcessEvent { - TraceRecord trace - boolean completed - } - - private Session session - - /** - * Workflow identifier, will be taken from the Session() object later - */ - private String runName - - /** - * Store the sessions unique ID for downstream reference purposes - */ - private String runId - private HxClient httpClient private JsonGenerator generator - private String workflowId - - private String watchUrl - private String endpoint - private ResourcesAggregator aggregator - - protected Map env = System.getenv() - - private LinkedBlockingQueue events = new LinkedBlockingQueue() - - private Thread sender - - private Duration requestInterval = REQUEST_INTERVAL - - private Duration aliveInterval = ALIVE_INTERVAL - - private LinkedHashSet processNames = new LinkedHashSet<>(20) - private Map schema = Collections.emptyMap() - private int maxRetries = 5 - - private int backOffDelay - - private int backOffBase - - private boolean towerLaunch - private String accessToken - private String workspaceId - - private TowerReports reports - private TowerRetryPolicy retryPolicy - private Map allContainers = new ConcurrentHashMap<>() - - protected TowerCommonApi commonApi - private Duration readTimeout = TowerConfig.DEFAULT_READ_TIMEOUT private Duration connectTimeout = TowerConfig.DEFAULT_CONNECT_TIMEOUT - TowerClient(Session session, TowerConfig config) { - this.session = session + TowerClient(TowerConfig config) { this.endpoint = checkUrl(config.endpoint) this.accessToken = config.accessToken - this.workspaceId = config.workspaceId this.retryPolicy = config.retryPolicy this.readTimeout = config.httpReadTimeout this.connectTimeout = config.httpConnectTimeout this.schema = loadSchema() this.generator = TowerJsonGenerator.create(schema) - this.reports = new TowerReports(session) - this.commonApi = new TowerCommonApi() - } - - TowerClient withEnvironment(Map env) { - this.env = env - return this - } - - TowerClient withRequestTimeout(Duration duration) { - this.readTimeout = duration - return this + initHttpClient() } @TestOnly protected TowerClient() { this.generator = TowerJsonGenerator.create(Collections.EMPTY_MAP) - this.commonApi = new TowerCommonApi() } - @Override - boolean enableMetrics() { true } - String getEndpoint() { endpoint } - String getWorkflowId() { workflowId } - - boolean getTowerLaunch() { towerLaunch } - - String getRunName() { runName } - - String getRunId() { runId } - - void setAliveInterval(Duration d) { - this.aliveInterval = d - } - - void setRequestInterval(Duration d) { - this.requestInterval = d - } - - void setMaxRetries( int value ) { - this.maxRetries = value - } - - void setBackOffBase( int value ) { - this.backOffBase = value - } - - void setBackOffDelay( int value ) { - this.backOffDelay = value - } - - String getWorkspaceId() { workspaceId } - /** * Check the URL and create an HttpPost() object. If a invalid i.e. protocol is used, * the constructor will raise an exception. @@ -240,139 +117,87 @@ class TowerClient implements TraceObserverV2 { return "${url.protocol}://${url.authority}" } - protected String getUrlTraceCreate() { + Map traceCreate(Map req, String workspaceId){ + return sendAndProcessRequest( getUrlTraceCreate(workspaceId), req, 'POST') + } + + Map traceBegin(Map req, String workspaceId, String workflowId){ + return sendAndProcessRequest( getUrlTraceBegin(workspaceId, workflowId), req, 'POST') + } + + void traceComplete(Map req, String workspaceId, String workflowId) { + final url = getUrlTraceComplete(workspaceId, workflowId) + final resp = sendHttpMessage(url, req, 'PUT') + logHttpResponse(url, resp) + } + + void traceHeartbeat(Map req, String workspaceId, String workflowId) { + final url = getUrlTraceHeartbeat( workspaceId, workflowId) + final resp = sendHttpMessage(url, req, 'PUT') + logHttpResponse(url, resp) + } + + void traceProgress(Map req, String workspaceId, String workflowId) { + final url = getUrlTraceProgress( workspaceId, workflowId ) + final resp = sendHttpMessage(url, req, 'PUT') + logHttpResponse(url, resp) + } + + protected Map sendAndProcessRequest(String url, Map req, String method){ + final resp = sendHttpMessage(url, req, method) + if( resp.error ) { + log.debug """\ + Unexpected HTTP response + - endpoint : $url + - status code : $resp.code + - response msg: $resp.cause + """.stripIndent(true) + throw new AbortOperationException(resp.message) + } + return parseTowerResponse(resp) + } + + protected String getUrlTraceCreate(String workspaceId) { def result = this.endpoint + '/trace/create' if( workspaceId ) result += "?workspaceId=$workspaceId" return result } - protected String getUrlTraceBegin() { + protected String getUrlTraceBegin(String workspaceId, String workflowId) { def result = "$endpoint/trace/$workflowId/begin" if( workspaceId ) result += "?workspaceId=$workspaceId" return result } - protected String getUrlTraceComplete() { + protected String getUrlTraceComplete(String workspaceId, String workflowId) { def result = "$endpoint/trace/$workflowId/complete" if( workspaceId ) result += "?workspaceId=$workspaceId" return result } - protected String getUrlTraceHeartbeat() { + protected String getUrlTraceHeartbeat(String workspaceId, String workflowId) { def result = "$endpoint/trace/$workflowId/heartbeat" if( workspaceId ) result += "?workspaceId=$workspaceId" return result } - protected String getUrlTraceProgress() { + protected String getUrlTraceProgress(String workspaceId, String workflowId) { def result = "$endpoint/trace/$workflowId/progress" if( workspaceId ) result += "?workspaceId=$workspaceId" return result } - /** - * On workflow start, submit a message with some basic - * information, like Id, activity and an ISO 8601 formatted - * timestamp. - * @param session The current Nextflow session object - */ - @Override - void onFlowCreate(Session session) { - log.debug "Creating Seqera Platform observer -- endpoint=$endpoint; requestInterval=$requestInterval; aliveInterval=$aliveInterval; maxRetries=$maxRetries; backOffBase=$backOffBase; backOffDelay=$backOffDelay" - - this.session = session - this.aggregator = new ResourcesAggregator() - this.runName = session.getRunName() - this.runId = session.getUniqueId() - this.httpClient = newHttpClient() - - // send hello to verify auth - final req = makeCreateReq(session) - final resp = sendHttpMessage(urlTraceCreate, req, 'POST') - if( resp.error ) { - log.debug """\ - Unexpected HTTP response - - endpoint : $urlTraceCreate - - status code : $resp.code - - response msg: $resp.cause - """.stripIndent(true) - throw new AbortOperationException(resp.message) - } - final ret = parseTowerResponse(resp) - this.workflowId = ret.workflowId - if( !workflowId ) - throw new AbortOperationException("Invalid Seqera Platform API response - Missing workflow Id") - log.debug "Platform workflow id: $workflowId; workflow url: ${ret.watchUrl}" - session.workflowMetadata.platform.workflowId = workflowId - // note: `watchUrl` in the create response requires Platform 26.01 or later - this.watchUrl = ret.watchUrl as String - session.workflowMetadata.platform.workflowUrl = watchUrl - if( ret.message ) - log.warn(ret.message.toString()) - // populate platform metadata from the create response - if( ret.metadata ) - applyPlatformMetadata(ret.metadata as Map) - - // Prepare to collect report paths if tower configuration has a 'reports' section - reports.flowCreate(workflowId) - } - - - /** - * Apply platform metadata received inline from the trace create response. - * This avoids extra API calls to fetch user, workspace, and launch details. - */ - protected void applyPlatformMetadata(Map metadata) { - try { - final platform = session.workflowMetadata.platform - // user info - if( metadata.userId ) - platform.user = new PlatformMetadata.User( - id: metadata.userId as String, - userName: metadata.userName as String, - organization: metadata.userOrganization as String - ) - // workspace info - if( metadata.workspaceId ) - platform.workspace = new PlatformMetadata.Workspace( - workspaceId: metadata.workspaceId as String, - workspaceName: metadata.workspaceName as String, - workspaceFullName: metadata.workspaceFullName as String, - orgName: metadata.orgName as String - ) - // launch details (only present for Platform-submitted runs) - if( metadata.computeEnvId ) - platform.computeEnv = new PlatformMetadata.ComputeEnv( - id: metadata.computeEnvId as String, - name: metadata.computeEnvName as String, - platform: metadata.computeEnvPlatform as String - ) - if( metadata.pipelineName ) - platform.pipeline = new PlatformMetadata.Pipeline( - id: metadata.pipelineId as String, - name: metadata.pipelineName as String, - revision: metadata.revision as String, - commitId: metadata.commitId as String - ) - if( metadata.labels ) - platform.labels = metadata.labels as List - } - catch( Exception e ) { - log.debug("Failed to apply platform metadata from create response", e) - } - } - - protected HxClient newHttpClient() { + protected void initHttpClient() { final builder = HxClient.newBuilder() // auth settings setupClientAuth(builder, getAccessToken()) // retry settings - builder + this.httpClient = builder .retryConfig(this.retryPolicy) .followRedirects(HttpClient.Redirect.NORMAL) .version(HttpClient.Version.HTTP_1_1) @@ -382,7 +207,7 @@ class TowerClient implements TraceObserverV2 { protected void setupClientAuth(HxClient.Builder config, String token) { // check for plain jwt token - final refreshToken = env.get('TOWER_REFRESH_TOKEN') + final refreshToken = SysEnv.get('TOWER_REFRESH_TOKEN') final refreshUrl = refreshToken ? "$endpoint/oauth/access_token" : null if( token.count('.')==2 ) { config.bearerToken(token) @@ -414,157 +239,12 @@ class TowerClient implements TraceObserverV2 { config.basicAuth(TOKEN_PREFIX + token) } - protected Map makeCreateReq(Session session) { - def result = new HashMap(5) - result.sessionId = session.uniqueId.toString() - result.runName = session.runName - result.projectName = session.workflowMetadata.projectName - result.repository = session.workflowMetadata.repository - result.workflowId = env.get('TOWER_WORKFLOW_ID') - result.instant = Instant.now().toEpochMilli() - this.towerLaunch = result.workflowId != null - return result - } - - @Override - void onProcessCreate(TaskProcessor process) { - log.trace "Creating process ${process.name}" - if( !processNames.add(process.name) ) - throw new IllegalStateException("Process name `${process.name}` already used") - } - - @Override - void onFlowBegin() { - // configure error retry - - final req = makeBeginReq(session) - final resp = sendHttpMessage(urlTraceBegin, req, 'PUT') - if( resp.error ) { - log.debug """\ - Unexpected HTTP response - - endpoint : $urlTraceBegin - - status code : $resp.code - - response msg: $resp.cause - """.stripIndent(true) - throw new AbortOperationException(resp.message) - } - - final payload = parseTowerResponse(resp) - this.watchUrl ?= payload.watchUrl - session.workflowMetadata.platform.workflowUrl ?= watchUrl - this.sender = Threads.start('Tower-thread', this.&sendTasks0) - final msg = "Monitor the execution with Seqera Platform using this URL: ${watchUrl}" - log.info(LoggerHelper.STICKY, msg) - } - String getAccessToken() { if( !accessToken ) throw new AbortOperationException("Missing Seqera Platform access token -- Make sure there's a variable TOWER_ACCESS_TOKEN in your environment") return accessToken } - /** - * Send an HTTP message when the workflow is completed. - */ - @Override - void onFlowComplete() { - // publish runtime reports - reports.publishRuntimeReports() - // submit the completion record - if( sender ) { - events << new ProcessEvent(completed: true) - // wait the submission of pending events - sender.join() - } - // wait and flush reports content - reports.flowComplete() - // notify the workflow completion - // note: only send complete if onFlowBegin was invoked (sender is set there) - if( workflowId && sender ) { - final req = makeCompleteReq(session) - final resp = sendHttpMessage(urlTraceComplete, req, 'PUT') - logHttpResponse(urlTraceComplete, resp) - } - } - - @Override - void onTaskPending(TaskEvent event) { - events << new ProcessEvent(trace: event.trace) - } - - /** - * Send an HTTP message when a process has been submitted - * - * @param handler A {@link TaskHandler} object representing the task submitted - * @param trace A {@link TraceRecord} object holding the task metadata and runtime info - */ - @Override - void onTaskSubmit(TaskEvent event) { - events << new ProcessEvent(trace: event.trace) - } - - /** - * Send an HTTP message, when a process has started - * - * @param handler A {@link TaskHandler} object representing the task started - * @param trace A {@link TraceRecord} object holding the task metadata and runtime info - */ - @Override - void onTaskStart(TaskEvent event) { - events << new ProcessEvent(trace: event.trace) - } - - /** - * Send an HTTP message, when a process completed - * - * @param handler A {@link TaskHandler} object representing the task completed - * @param trace A {@link TraceRecord} object holding the task metadata and runtime info - */ - @Override - void onTaskComplete(TaskEvent event) { - events << new ProcessEvent(trace: event.trace) - - synchronized (this) { - aggregator.aggregate(event.trace) - } - } - - @Override - void onTaskCached(TaskEvent event) { - // event was triggered by a stored task, ignore it - if( !event.trace ) - return - - // add the cached task event - events << new ProcessEvent(trace: event.trace) - - // remove the record from the current records - synchronized (this) { - aggregator.aggregate(event.trace) - } - } - - /** - * Send an HTTP message, when a workflow has failed - * - * @param handler A {@link TaskHandler} object representing the task that caused the workflow execution to fail (it may be null) - * @param trace A {@link TraceRecord} object holding the task metadata and runtime info (it may be null) - */ - @Override - void onFlowError(TaskEvent event) { - events << new ProcessEvent(trace: event.trace) - } - - /** - * Update reports file when a file is published - * - * @param destination File path at `publishDir` of the published file. - */ - @Override - void onFilePublish(FilePublishEvent event) { - reports.filePublish(event.target) - } - /** * Little helper method that sends a HTTP POST message as JSON with * the current run status, ISO 8601 UTC timestamp, run name and the TraceRecord @@ -600,15 +280,58 @@ class TowerClient implements TraceObserverV2 { } } - protected HttpRequest makeRequest(String url, String payload, String verb) { - assert payload, "Tower request cannot be empty" + Response sendApiRequest(String url, Map payload=null, String method='GET') { + sendHttpMessage(url, payload, method) + } + /** + * Send a GET request and return the response body as a streaming {@link InputStream} + * instead of buffering the entire response into a {@link String}. + * Uses {@code HxClient.sendAsStream()} which goes through the same retry and + * auth chain as {@code sendAsString()}. + * + * Status codes are checked before returning — on error the stream is closed and + * the same exceptions as {@link #checkResponse} are thrown. + * + * @param url the full API URL to GET + * @return an InputStream over the response body + * @throws UnauthorizedException on 401 + * @throws ForbiddenException on 403 + * @throws NotFoundException on 404 + */ + InputStream sendStreamingRequest(String url) { + log.trace "HTTP streaming GET url=$url" + final req = makeRequest(url, null, 'GET') + final resp = httpClient.sendAsStream(req) + final status = resp.statusCode() + if( status >= 200 && status < 300 ) + return resp.body() + // Error — close the stream and throw + resp.body()?.close() + if( status == 401 ) + throw new UnauthorizedException("Seqera authentication failed — check tower.accessToken or TOWER_ACCESS_TOKEN") + if( status == 403 ) + throw new ForbiddenException("Forbidden — check permissions") + if( status == 404 ) + throw new NotFoundException("Resource $url not found") + throw new IOException("Seqera API error: HTTP ${status} for ${url}") + } + + protected HttpRequest makeRequest(String url, String payload, String verb) { final builder = HttpRequest.newBuilder(URI.create(url)) - .header('Content-Type', 'application/json; charset=utf-8') .header('User-Agent', "Nextflow/$BuildInfo.version") .header('Traceparent', TraceUtils.rndTrace()) .timeout(java.time.Duration.ofMillis(readTimeout.millis)) + if( verb == 'GET' ) + return builder.GET().build() + + if( verb == 'DELETE' ) + return builder.DELETE().build() + + assert payload, "Tower request cannot be empty" + builder.header('Content-Type', 'application/json; charset=utf-8') + if( verb == 'PUT' ) return builder.PUT(HttpRequest.BodyPublishers.ofString(payload)).build() @@ -618,175 +341,6 @@ class TowerClient implements TraceObserverV2 { throw new IllegalArgumentException("Unsupported HTTP verb: $verb") } - protected boolean isCliLogsEnabled() { - return env.get('TOWER_ALLOW_NEXTFLOW_LOGS') == 'true' - } - - protected String getOperationId() { - if( !isCliLogsEnabled() ) - return null - try { - if( env.get('AWS_BATCH_JOB_ID') ) - return "aws-batch::${env.get('AWS_BATCH_JOB_ID')}" - else - return "local-platform::${ProcessHelper.selfPid()}" - } - catch (Exception e) { - log.warn "Unable to retrieve native environment operation id", e - return null - } - } - - protected String getLogFile() { - return isCliLogsEnabled() ? env.get('NXF_LOG_FILE') : null - } - - protected String getOutFile() { - return isCliLogsEnabled() ? env.get('NXF_OUT_FILE') : null - } - - protected Map makeBeginReq(Session session) { - def workflow = session.getWorkflowMetadata().toMap() - workflow.params = session.getParams() - workflow.id = getWorkflowId() - workflow.remove('stats') - - // render as a string - workflow.container = mapToString(workflow.container) - workflow.configText = session.resolvedConfig - // extra metadata - workflow.operationId = getOperationId() - workflow.logFile = getLogFile() - workflow.outFile = getOutFile() - - def result = new LinkedHashMap(5) - result.workflow = workflow - result.processNames = new ArrayList(processNames) - result.towerLaunch = towerLaunch - result.instant = Instant.now().toEpochMilli() - return result - } - - protected Map makeCompleteReq(Session session) { - def workflow = session.getWorkflowMetadata().toMap() - //Remove retrieved platform info - if( workflow.platform ) - workflow.remove('platform') - - workflow.params = session.getParams() - workflow.id = getWorkflowId() - // render as a string - workflow.container = mapToString(workflow.container) - workflow.configText = session.resolvedConfig - // extra metadata - workflow.operationId = getOperationId() - workflow.logFile = getLogFile() - workflow.outFile = getOutFile() - - def result = new LinkedHashMap(5) - result.workflow = workflow - result.metrics = getMetricsList() - result.progress = getWorkflowProgress(false) - result.instant = Instant.now().toEpochMilli() - return result - } - - protected Map makeHeartbeatReq() { - def result = new HashMap(1) - result.progress = getWorkflowProgress(true) - result.instant = Instant.now().toEpochMilli() - return result - } - - protected String mapToString(def obj) { - if( obj == null ) - return null - if( obj instanceof CharSequence ) - return obj.toString() - if( obj instanceof Map ) { - // turn this off for multiple containers because the string representation is broken - return null - } - throw new IllegalArgumentException("Illegal container attribute type: ${obj.getClass().getName()} = ${obj}" ) - } - - protected Map makeTaskMap0(TraceRecord trace) { - Map record = new LinkedHashMap<>(trace.store.size()) - for( Map.Entry entry : trace.store.entrySet() ) { - def name = entry.key - // remove '%' char from field prefix - if( name.startsWith('%') ) - name = 'p' + name.substring(1) - // normalise to camelCase - name = underscoreToCamelCase(name) - // put the value - record.put(name, fixTaskField(name,entry.value)) - } - - // prevent invalid tag data - if( record.tag!=null && !(record.tag instanceof CharSequence)) { - final msg = "Invalid tag value for process: ${record.process} -- A string is expected instead of type: ${record.tag.getClass().getName()}; offending value=${record.tag}" - log.warn1(msg, cacheKey: record.process) - record.tag = null - } - - // add transient fields - record.executor = trace.getExecutorName() - record.cloudZone = trace.getMachineInfo()?.zone - record.machineType = trace.getMachineInfo()?.type - record.priceModel = trace.getMachineInfo()?.priceModel?.toString() - record.numSpotInterruptions = trace.getNumSpotInterruptions() - record.logStreamId = trace.getLogStreamId() - record.resourceAllocation = trace.getResourceAllocation() - record.gpuMetrics = trace.getGpuMetrics() - - return record - } - - - static protected Object fixTaskField(String name, value) { - if( TraceRecord.FIELDS[name] == 'date' ) - return value ? OffsetDateTime.ofInstant(Instant.ofEpochMilli(value as long), ZoneId.systemDefault()) : null - else - return value - } - - protected Map makeTasksReq(Collection tasks) { - - def payload = new ArrayList(tasks.size()) - for( TraceRecord rec : tasks ) { - payload << makeTaskMap0(rec) - } - - final result = new LinkedHashMap(5) - result.put('tasks', payload) - result.put('progress', getWorkflowProgress(true)) - result.put('containers', getNewContainers(tasks)) - result.instant = Instant.now().toEpochMilli() - return result - } - - protected List getNewContainers(Collection tasks) { - final result = new ArrayList() - for( TraceRecord it : tasks ) { - final meta = it.getContainerMeta() - if( meta && !allContainers.get(meta.targetImage) ) { - allContainers.put(meta.targetImage, Boolean.TRUE) - result.add(meta) - } - } - return result - } - - protected List getMetricsList() { - return aggregator.computeSummaryList() - } - - protected WorkflowProgress getWorkflowProgress(boolean quick) { - def stats = quick ? session.getStatsObserver().getQuickStats() : session.getStatsObserver().getStats() - new WorkflowProgress(stats) - } - /** * Little helper function that can be called for logging upon an incoming HTTP response */ @@ -839,18 +393,6 @@ class TowerClient implements TraceObserverV2 { } } - protected String underscoreToCamelCase(String str) { - if( !str.contains('_') ) - return str - - final words = str.tokenize('_') - def result = words[0] - for( int i=1; i loadSchema() { final props = new Properties() @@ -863,49 +405,108 @@ class TowerClient implements TraceObserverV2 { return result } - protected void sendTasks0(dummy) { - final tasks = new HashMap(TASKS_PER_REQUEST) - boolean complete = false - long previous = System.currentTimeMillis() - final long period = requestInterval.millis - final long delay = period / 10 as long - - while( !complete ) { - final ProcessEvent ev = events.poll(delay, TimeUnit.MILLISECONDS) - // reconcile task events ie. send out only the last event - if( ev ) { - log.trace "Tower event=$ev" - if( ev.trace ) - tasks[ev.trace.taskId] = ev.trace - if( ev.completed ) - complete = true - } + String buildUrl( String path, Map queryParams) { + def url = new StringBuilder(endpoint) + if( !path.startsWith('/') ) { + url.append('/') + } + url.append(path) - // check if there's something to send - final now = System.currentTimeMillis() - final delta = now -previous - - if( !tasks ) { - if( delta > aliveInterval.millis ) { - final req = makeHeartbeatReq() - final resp = sendHttpMessage(urlTraceHeartbeat, req, 'PUT') - logHttpResponse(urlTraceHeartbeat, resp) - previous = now - } - continue - } + if( queryParams && !queryParams.isEmpty() ) { + url.append('?') + url.append(queryParams.collect { k, v -> "${URLEncoder.encode(k.toString(), 'UTF-8')}=${URLEncoder.encode(v.toString(), 'UTF-8')}" }.join('&')) + } + + return url.toString() + } + + Map apiGet(String path, Map queryParams = [:]) { + final url = buildUrl( path, queryParams) + + final response = sendApiRequest(url) + checkResponse(response, url) + return new JsonSlurper().parseText(response.message) as Map + } + + Map apiPost(String path, Map queryParams, Map payload) { + final url = buildUrl( path, queryParams) + final response= sendApiRequest(url, payload, 'POST') + checkResponse(response, url) + return response.message ? new JsonSlurper().parseText(response.message) as Map : [:] + } - if( delta > period || tasks.size() >= TASKS_PER_REQUEST || complete ) { - // send - final req = makeTasksReq(tasks.values()) - final resp = sendHttpMessage(urlTraceProgress, req, 'PUT') - logHttpResponse(urlTraceProgress, resp) + /** + * @return current user info (id, userName, etc.) from GET /user-info + */ + Map getUserInfo() { + final json = apiGet("/user-info") + return json.user as Map + } + + /** + * Calls the Seqera Platform to retrieve the workflow information. + * + * @param workflowId Id of the workflow + * @return Map containing workflow information + * @throws RuntimeException if the API call fails + */ + Map getWorkflowDetails( String workflowId, Map queryParams = [:]) { + final json = apiGet("/workflow/${workflowId}", queryParams) + return json.workflow as Map + } - // clean up for next iteration - previous = now - tasks.clear() + + List listUserWorkspacesAndOrgs(String userId) { + final json = apiGet("/user/${userId}/workspaces") + return json.orgsAndWorkspaces as List + } + + private static void checkResponse(Response resp, String url) { + if (!resp.error) return + final code = resp.code + if (code == 401) + throw new UnauthorizedException("Seqera authentication failed — check tower.accessToken or TOWER_ACCESS_TOKEN") + if (code == 403) + throw new ForbiddenException("Forbidden — check permissions") + if (code == 404) + throw new NotFoundException("Resource $url not found") + throw new Exception("Seqera API error: HTTP ${code} for ${url}${resp.message ? ' - ' + resp.message :''}") + } + + /** + * Calls the Seqera Platform to retrieve the user's workspaces information + * and select the one matching with the workspace Id. + * + * @param userId Id of the workspace user + * @param workspaceId Id of the workspace + * @return Map containing workspace information + * @throws RuntimeException if the API call fails + */ + Map getUserWorkspaceDetails( String userId, String workspaceId) { + if( !userId || !workspaceId ) { + return null + } + try { + final orgsAndWorkspaces = listUserWorkspacesAndOrgs(userId) + + final workspace = orgsAndWorkspaces.find { ((Map) it).workspaceId?.toString() == workspaceId } + if( workspace ) { + final ws = workspace as Map + return [ + orgName : ws.orgName, + workspaceId : ws.workspaceId, + workspaceName : ws.workspaceName, + workspaceFullName: ws.workspaceFullName, + roles : ws.roles + ] } + + return null + } catch( Exception e ) { + log.debug("Failed to get workspace details for workspace ${workspaceId}: ${e.message}", e) + return null } } + } diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerCommonApi.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerCommonApi.groovy deleted file mode 100644 index 1847b7a656..0000000000 --- a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerCommonApi.groovy +++ /dev/null @@ -1,134 +0,0 @@ -/* - * Copyright 2013-2025, Seqera Labs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -package io.seqera.tower.plugin - -import groovy.json.JsonSlurper -import groovy.transform.CompileStatic -import groovy.util.logging.Slf4j -import io.seqera.http.HxClient - -import java.net.http.HttpRequest -import java.net.http.HttpResponse - -/** - * Class with common API calls used in different classes - * - * @author Jorge Ejarque - */ -@Slf4j -@CompileStatic -class TowerCommonApi { - - /** - * Calls the Seqera Platform user-info API to retrieve user information. - * - * @param client HTTP client to perform the API calls - * @param endpoint Seqera Platform API endpoint - * @return Map containing user information (id, userName, email, etc.) - * @throws RuntimeException if the API call fails - */ - Map getUserInfo(HxClient client, String endpoint) { - final json = apiGet(client, endpoint, "/user-info") - return json.user as Map - } - - /** - * Calls the Seqera Platform to retrieve the user's workspaces information - * and select the one matching with the workspace Id. - * - * @param client HTTP client to perform the API calls - * @param userId Id of the workspace user - * @param endpoint Seqera Platform API endpoint - * @param workspaceId Id of the workspace - * @return Map containing workspace information - * @throws RuntimeException if the API call fails - */ - Map getUserWorkspaceDetails(HxClient client, String userId, String endpoint, String workspaceId) { - if( !userId || !workspaceId ) { - return null - } - try { - final json = apiGet(client, endpoint, "/user/${userId}/workspaces") - - final orgsAndWorkspaces = json.orgsAndWorkspaces as List - - final workspace = orgsAndWorkspaces.find { ((Map) it).workspaceId?.toString() == workspaceId } - if( workspace ) { - final ws = workspace as Map - return [ - orgName : ws.orgName, - workspaceId : ws.workspaceId, - workspaceName : ws.workspaceName, - workspaceFullName: ws.workspaceFullName, - roles : ws.roles - ] - } - - return null - } catch( Exception e ) { - log.debug("Failed to get workspace details for workspace ${workspaceId}: ${e.message}", e) - return null - } - } - - /** - * Calls the Seqera Platform to retrieve the workflow information. - * - * @param client HTTP client to perform the API calls - * @param endpoint Seqera Platform API endpoint - * @param workflowId Id of the workflow - * @return Map containing workflow information - * @throws RuntimeException if the API call fails - */ - Map getWorkflowDetails(HxClient client, String endpoint, String workflowId, Map queryParams = [:]) { - final json = apiGet(client, endpoint, "/workflow/${workflowId}", queryParams) - return json.workflow as Map - } - - Map apiGet(HxClient client, String apiEndpoint, String path, Map queryParams = [:]) { - final url = buildUrl(apiEndpoint, path, queryParams) - log.debug "Platform API - GET ${url}" - final request = HttpRequest.newBuilder() - .uri(URI.create(url)) - .GET() - .build() - - final response = client.send(request, HttpResponse.BodyHandlers.ofString()) - - if( response.statusCode() != 200 ) { - final error = response.body() ?: "HTTP ${response.statusCode()}" - throw new RuntimeException("API GET request ${url} failed: ${error}") - } - - return new JsonSlurper().parseText(response.body()) as Map - } - - String buildUrl(String endpoint, String path, Map queryParams) { - def url = new StringBuilder(endpoint) - if( !path.startsWith('/') ) { - url.append('/') - } - url.append(path) - - if( queryParams && !queryParams.isEmpty() ) { - url.append('?') - url.append(queryParams.collect { k, v -> "${URLEncoder.encode(k.toString(), 'UTF-8')}=${URLEncoder.encode(v.toString(), 'UTF-8')}" }.join('&')) - } - - return url.toString() - } -} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerFactory.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerFactory.groovy index 42aabece2d..24e6f3d22e 100644 --- a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerFactory.groovy +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerFactory.groovy @@ -44,31 +44,23 @@ class TowerFactory implements TraceObserverFactoryV2 { @Override Collection create(Session session) { - final client = client(session, env) - if( !client ) + final config = new TowerConfig(session.config.tower as Map ?: Collections.emptyMap(), env) + if( !isEnabled(session, config, env) ) return Collections.emptyList() - final result = new ArrayList(1) - // create the tower client - result.add(client) + // create the tower observer + result.add( new TowerObserver(session, client(session, env), config.workspaceId, env)) // create the logs checkpoint if( session.cloudCachePath ) result.add( new LogsCheckpoint() ) return result } - static protected TowerClient createTowerClient0(Session session, TowerConfig config, Map env) { + @Memoized + static TowerClient client(Session session, Map env) { final opts = session.config.tower as Map ?: Collections.emptyMap() - - Duration requestInterval = opts.requestInterval as Duration - Duration aliveInterval = opts.aliveInterval as Duration - - final tower = new TowerClient(session, config).withEnvironment(env) - if( aliveInterval ) - tower.aliveInterval = aliveInterval - if( requestInterval ) - tower.requestInterval = requestInterval - + final config = new TowerConfig(opts, env) + final tower = new TowerClient(config) // register auth provider // note: this is needed to authorize access to resources via XFileSystemProvider used by NF // it's not needed by the tower client logic @@ -83,17 +75,11 @@ class TowerFactory implements TraceObserverFactoryV2 { return new TowerXAuth(endpoint, accessToken, refreshToken) } - @Memoized - static TowerClient client(Session session, Map env) { - final opts = session.config.tower as Map ?: Collections.emptyMap() - final config = new TowerConfig(opts, env) - Boolean isEnabled = config.enabled || env.get('TOWER_WORKFLOW_ID') || session.config.navigate('fusion.enabled') as Boolean - return isEnabled - ? createTowerClient0(session, config, env) - : null + private static boolean isEnabled(Session session, TowerConfig config, Map env) { + return config.enabled || env.get('TOWER_WORKFLOW_ID') || session.config.navigate('fusion.enabled') as Boolean } static TowerClient client() { - client(Global.session as Session, SysEnv.get()) + return client(Global.session as Session, SysEnv.get()) } } diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerObserver.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerObserver.groovy new file mode 100644 index 0000000000..93d9f40cfd --- /dev/null +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerObserver.groovy @@ -0,0 +1,575 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin + +import groovy.transform.CompileStatic +import groovy.transform.ToString +import groovy.util.logging.Slf4j +import nextflow.Session +import nextflow.container.resolver.ContainerMeta +import nextflow.exception.AbortOperationException +import nextflow.processor.TaskHandler +import nextflow.processor.TaskId +import nextflow.processor.TaskProcessor +import nextflow.script.PlatformMetadata +import nextflow.trace.ResourcesAggregator +import nextflow.trace.TraceObserverV2 +import nextflow.trace.TraceRecord +import nextflow.trace.event.FilePublishEvent +import nextflow.trace.event.TaskEvent +import nextflow.util.Duration +import nextflow.util.LoggerHelper +import nextflow.util.ProcessHelper +import nextflow.util.Threads + +import java.time.Instant +import java.time.OffsetDateTime +import java.time.ZoneId +import java.util.concurrent.ConcurrentHashMap +import java.util.concurrent.LinkedBlockingQueue +import java.util.concurrent.TimeUnit + +/** + * Send out messages via HTTP to a configured URL on different workflow + * execution events. + * + * @author Paolo Di Tommaso + */ +@Slf4j +@CompileStatic +class TowerObserver implements TraceObserverV2 { + + static private final int TASKS_PER_REQUEST = 100 + + static private final Duration REQUEST_INTERVAL = Duration.of('1 sec') + + static private final Duration ALIVE_INTERVAL = Duration.of('1 min') + + @ToString(includeNames = true) + static class ProcessEvent { + TraceRecord trace + boolean completed + } + + private Session session + + /** + * Workflow identifier, will be taken from the Session() object later + */ + private String runName + + /** + * Store the sessions unique ID for downstream reference purposes + */ + private String runId + + private String workflowId + + private String watchUrl + + private ResourcesAggregator aggregator + + protected Map env = System.getenv() + + private LinkedBlockingQueue events = new LinkedBlockingQueue() + + private Thread sender + + private Duration requestInterval = REQUEST_INTERVAL + + private Duration aliveInterval = ALIVE_INTERVAL + + private LinkedHashSet processNames = new LinkedHashSet<>(20) + + private boolean towerLaunch + + private String workspaceId + + private TowerReports reports + + private TowerClient client + + private Map allContainers = new ConcurrentHashMap<>() + + TowerObserver(Session session, TowerClient client, String workspaceId, Map env) { + this.session = session + this.workspaceId = workspaceId + this.reports = new TowerReports(session) + this.client = client + if( env ) + this.env = env + } + + + @Override + boolean enableMetrics() { true } + + String getWorkflowId() { workflowId } + + boolean getTowerLaunch() { towerLaunch } + + String getRunName() { runName } + + String getRunId() { runId } + + void setAliveInterval(Duration d) { + this.aliveInterval = d + } + + void setRequestInterval(Duration d) { + this.requestInterval = d + } + + String getWorkspaceId() { workspaceId } + + + /** + * On workflow start, submit a message with some basic + * information, like Id, activity and an ISO 8601 formatted + * timestamp. + * @param session The current Nextflow session object + */ + @Override + void onFlowCreate(Session session) { + log.debug "Creating Seqera Platform observer -- endpoint=$client.endpoint; requestInterval=$requestInterval; aliveInterval=$aliveInterval;" + + this.session = session + this.aggregator = new ResourcesAggregator() + this.runName = session.getRunName() + this.runId = session.getUniqueId() + + // send hello to verify auth + final ret = client.traceCreate(makeCreateReq(session), workspaceId) + this.workflowId = ret.workflowId + if( !workflowId ) + throw new AbortOperationException("Invalid Seqera Platform API response - Missing workflow Id") + log.debug "Platform workflow id: $workflowId; workflow url: ${ret.watchUrl}" + session.workflowMetadata.platform.workflowId = workflowId + // note: `watchUrl` in the create response requires Platform 26.01 or later + this.watchUrl = ret.watchUrl as String + session.workflowMetadata.platform.workflowUrl = watchUrl + if( ret.message ) + log.warn(ret.message.toString()) + // populate platform metadata from the create response + if( ret.metadata ) + applyPlatformMetadata(ret.metadata as Map) + + // Prepare to collect report paths if tower configuration has a 'reports' section + reports.flowCreate(workflowId) + } + + + /** + * Apply platform metadata received inline from the trace create response. + * This avoids extra API calls to fetch user, workspace, and launch details. + */ + protected void applyPlatformMetadata(Map metadata) { + try { + final platform = session.workflowMetadata.platform + // user info + if( metadata.userId ) + platform.user = new PlatformMetadata.User( + id: metadata.userId as String, + userName: metadata.userName as String, + organization: metadata.userOrganization as String + ) + // workspace info + if( metadata.workspaceId ) + platform.workspace = new PlatformMetadata.Workspace( + workspaceId: metadata.workspaceId as String, + workspaceName: metadata.workspaceName as String, + workspaceFullName: metadata.workspaceFullName as String, + orgName: metadata.orgName as String + ) + // launch details (only present for Platform-submitted runs) + if( metadata.computeEnvId ) + platform.computeEnv = new PlatformMetadata.ComputeEnv( + id: metadata.computeEnvId as String, + name: metadata.computeEnvName as String, + platform: metadata.computeEnvPlatform as String + ) + if( metadata.pipelineName ) + platform.pipeline = new PlatformMetadata.Pipeline( + id: metadata.pipelineId as String, + name: metadata.pipelineName as String, + revision: metadata.revision as String, + commitId: metadata.commitId as String + ) + if( metadata.labels ) + platform.labels = metadata.labels as List + } + catch( Exception e ) { + log.debug("Failed to apply platform metadata from create response", e) + } + } + + protected Map makeCreateReq(Session session) { + def result = new HashMap(5) + result.sessionId = session.uniqueId.toString() + result.runName = session.runName + result.projectName = session.workflowMetadata.projectName + result.repository = session.workflowMetadata.repository + result.workflowId = env.get('TOWER_WORKFLOW_ID') + result.instant = Instant.now().toEpochMilli() + this.towerLaunch = result.workflowId != null + return result + } + + @Override + void onProcessCreate(TaskProcessor process) { + log.trace "Creating process ${process.name}" + if( !processNames.add(process.name) ) + throw new IllegalStateException("Process name `${process.name}` already used") + } + + @Override + void onFlowBegin() { + // configure error retry + + final payload = client.traceBegin(makeBeginReq(session), workspaceId, workflowId) + this.watchUrl ?= payload.watchUrl + session.workflowMetadata.platform.workflowUrl ?= watchUrl + this.sender = Threads.start('Tower-thread', this.&sendTasks0) + final msg = "Monitor the execution with Seqera Platform using this URL: ${watchUrl}" + log.info(LoggerHelper.STICKY, msg) + } + + /** + * Send an HTTP message when the workflow is completed. + */ + @Override + void onFlowComplete() { + // publish runtime reports + reports.publishRuntimeReports() + // submit the completion record + if( sender ) { + events << new ProcessEvent(completed: true) + // wait the submission of pending events + sender.join() + } + // wait and flush reports content + reports.flowComplete() + // notify the workflow completion + // note: only send complete if onFlowBegin was invoked (sender is set there) + if( workflowId && sender ) { + client.traceComplete(makeCompleteReq(session), workspaceId, workflowId) + } + } + + @Override + void onTaskPending(TaskEvent event) { + events << new ProcessEvent(trace: event.trace) + } + + /** + * Send an HTTP message when a process has been submitted + * + * @param handler A {@link TaskHandler} object representing the task submitted + * @param trace A {@link TraceRecord} object holding the task metadata and runtime info + */ + @Override + void onTaskSubmit(TaskEvent event) { + events << new ProcessEvent(trace: event.trace) + } + + /** + * Send an HTTP message, when a process has started + * + * @param handler A {@link TaskHandler} object representing the task started + * @param trace A {@link TraceRecord} object holding the task metadata and runtime info + */ + @Override + void onTaskStart(TaskEvent event) { + events << new ProcessEvent(trace: event.trace) + } + + /** + * Send an HTTP message, when a process completed + * + * @param handler A {@link TaskHandler} object representing the task completed + * @param trace A {@link TraceRecord} object holding the task metadata and runtime info + */ + @Override + void onTaskComplete(TaskEvent event) { + events << new ProcessEvent(trace: event.trace) + + synchronized (this) { + aggregator.aggregate(event.trace) + } + } + + @Override + void onTaskCached(TaskEvent event) { + // event was triggered by a stored task, ignore it + if( !event.trace ) + return + + // add the cached task event + events << new ProcessEvent(trace: event.trace) + + // remove the record from the current records + synchronized (this) { + aggregator.aggregate(event.trace) + } + } + + /** + * Send an HTTP message, when a workflow has failed + * + * @param handler A {@link TaskHandler} object representing the task that caused the workflow execution to fail (it may be null) + * @param trace A {@link TraceRecord} object holding the task metadata and runtime info (it may be null) + */ + @Override + void onFlowError(TaskEvent event) { + events << new ProcessEvent(trace: event.trace) + } + + /** + * Update reports file when a file is published + * + * @param destination File path at `publishDir` of the published file. + */ + @Override + void onFilePublish(FilePublishEvent event) { + reports.filePublish(event.target) + } + + protected boolean isCliLogsEnabled() { + return env.get('TOWER_ALLOW_NEXTFLOW_LOGS') == 'true' + } + + protected String getOperationId() { + if( !isCliLogsEnabled() ) + return null + try { + if( env.get('AWS_BATCH_JOB_ID') ) + return "aws-batch::${env.get('AWS_BATCH_JOB_ID')}" + else + return "local-platform::${ProcessHelper.selfPid()}" + } + catch (Exception e) { + log.warn "Unable to retrieve native environment operation id", e + return null + } + } + + protected String getLogFile() { + return isCliLogsEnabled() ? env.get('NXF_LOG_FILE') : null + } + + protected String getOutFile() { + return isCliLogsEnabled() ? env.get('NXF_OUT_FILE') : null + } + + protected Map makeBeginReq(Session session) { + def workflow = session.getWorkflowMetadata().toMap() + workflow.params = session.getParams() + workflow.id = getWorkflowId() + workflow.remove('stats') + + // render as a string + workflow.container = mapToString(workflow.container) + workflow.configText = session.resolvedConfig + // extra metadata + workflow.operationId = getOperationId() + workflow.logFile = getLogFile() + workflow.outFile = getOutFile() + + def result = new LinkedHashMap(5) + result.workflow = workflow + result.processNames = new ArrayList(processNames) + result.towerLaunch = towerLaunch + result.instant = Instant.now().toEpochMilli() + return result + } + + protected Map makeCompleteReq(Session session) { + def workflow = session.getWorkflowMetadata().toMap() + //Remove retrieved platform info + if( workflow.platform ) + workflow.remove('platform') + + workflow.params = session.getParams() + workflow.id = getWorkflowId() + // render as a string + workflow.container = mapToString(workflow.container) + workflow.configText = session.resolvedConfig + // extra metadata + workflow.operationId = getOperationId() + workflow.logFile = getLogFile() + workflow.outFile = getOutFile() + + def result = new LinkedHashMap(5) + result.workflow = workflow + result.metrics = getMetricsList() + result.progress = getWorkflowProgress(false) + result.instant = Instant.now().toEpochMilli() + return result + } + + protected Map makeHeartbeatReq() { + def result = new HashMap(1) + result.progress = getWorkflowProgress(true) + result.instant = Instant.now().toEpochMilli() + return result + } + + protected String mapToString(def obj) { + if( obj == null ) + return null + if( obj instanceof CharSequence ) + return obj.toString() + if( obj instanceof Map ) { + // turn this off for multiple containers because the string representation is broken + return null + } + throw new IllegalArgumentException("Illegal container attribute type: ${obj.getClass().getName()} = ${obj}" ) + } + + protected Map makeTaskMap0(TraceRecord trace) { + Map record = new LinkedHashMap<>(trace.store.size()) + for( Map.Entry entry : trace.store.entrySet() ) { + def name = entry.key + // remove '%' char from field prefix + if( name.startsWith('%') ) + name = 'p' + name.substring(1) + // normalise to camelCase + name = underscoreToCamelCase(name) + // put the value + record.put(name, fixTaskField(name,entry.value)) + } + + // prevent invalid tag data + if( record.tag!=null && !(record.tag instanceof CharSequence)) { + final msg = "Invalid tag value for process: ${record.process} -- A string is expected instead of type: ${record.tag.getClass().getName()}; offending value=${record.tag}" + log.warn1(msg, cacheKey: record.process) + record.tag = null + } + + // add transient fields + record.executor = trace.getExecutorName() + record.cloudZone = trace.getMachineInfo()?.zone + record.machineType = trace.getMachineInfo()?.type + record.priceModel = trace.getMachineInfo()?.priceModel?.toString() + record.numSpotInterruptions = trace.getNumSpotInterruptions() + record.logStreamId = trace.getLogStreamId() + record.resourceAllocation = trace.getResourceAllocation() + record.gpuMetrics = trace.getGpuMetrics() + return record + } + + + static protected Object fixTaskField(String name, value) { + if( TraceRecord.FIELDS[name] == 'date' ) + return value ? OffsetDateTime.ofInstant(Instant.ofEpochMilli(value as long), ZoneId.systemDefault()) : null + else + return value + } + + protected Map makeTasksReq(Collection tasks) { + + def payload = new ArrayList(tasks.size()) + for( TraceRecord rec : tasks ) { + payload << makeTaskMap0(rec) + } + + final result = new LinkedHashMap(5) + result.put('tasks', payload) + result.put('progress', getWorkflowProgress(true)) + result.put('containers', getNewContainers(tasks)) + result.instant = Instant.now().toEpochMilli() + return result + } + + protected List getNewContainers(Collection tasks) { + final result = new ArrayList() + for( TraceRecord it : tasks ) { + final meta = it.getContainerMeta() + if( meta && !allContainers.get(meta.targetImage) ) { + allContainers.put(meta.targetImage, Boolean.TRUE) + result.add(meta) + } + } + return result + } + + protected List getMetricsList() { + return aggregator.computeSummaryList() + } + + protected WorkflowProgress getWorkflowProgress(boolean quick) { + def stats = quick ? session.getStatsObserver().getQuickStats() : session.getStatsObserver().getStats() + new WorkflowProgress(stats) + } + + protected String underscoreToCamelCase(String str) { + if( !str.contains('_') ) + return str + + final words = str.tokenize('_') + def result = words[0] + for( int i=1; i(TASKS_PER_REQUEST) + boolean complete = false + long previous = System.currentTimeMillis() + final long period = requestInterval.millis + final long delay = period / 10 as long + + while( !complete ) { + final ProcessEvent ev = events.poll(delay, TimeUnit.MILLISECONDS) + // reconcile task events ie. send out only the last event + if( ev ) { + log.trace "Tower event=$ev" + if( ev.trace ) + tasks[ev.trace.taskId] = ev.trace + if( ev.completed ) + complete = true + } + + // check if there's something to send + final now = System.currentTimeMillis() + final delta = now -previous + + if( !tasks ) { + if( delta > aliveInterval.millis ) { + final req = makeHeartbeatReq() + client.traceHeartbeat(req, workspaceId, workflowId) + previous = now + } + continue + } + + if( delta > period || tasks.size() >= TASKS_PER_REQUEST || complete ) { + // send + final req = makeTasksReq(tasks.values()) + client.traceProgress(req, workspaceId, workflowId) + + // clean up for next iteration + previous = now + tasks.clear() + } + } + } + +} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerPlugin.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerPlugin.groovy index 89f419bdee..eee8da159b 100644 --- a/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerPlugin.groovy +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/TowerPlugin.groovy @@ -18,6 +18,8 @@ package io.seqera.tower.plugin import groovy.transform.CompileStatic import groovy.util.logging.Slf4j +import io.seqera.tower.plugin.fs.SeqeraFileSystemProvider +import nextflow.file.FileHelper import nextflow.plugin.BasePlugin import nextflow.cli.PluginExecAware import org.pf4j.PluginWrapper @@ -37,4 +39,10 @@ class TowerPlugin extends BasePlugin implements PluginExecAware { this.delegate = new CacheCommand() } + @Override + void start() { + super.start() + FileHelper.getOrInstallProvider(SeqeraFileSystemProvider) + } + } diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/auth/AuthCommandImpl.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/auth/AuthCommandImpl.groovy index dc8a48d152..cc28ff1cd2 100644 --- a/plugins/nf-tower/src/main/io/seqera/tower/plugin/auth/AuthCommandImpl.groovy +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/auth/AuthCommandImpl.groovy @@ -19,6 +19,7 @@ package io.seqera.tower.plugin.auth import io.seqera.http.HxClient import io.seqera.platform.auth.oidc.OidcLoginFlow import io.seqera.tower.plugin.BaseCommandImpl +import io.seqera.tower.plugin.TowerClient import java.awt.* import java.net.http.HttpRequest @@ -26,9 +27,9 @@ import java.net.http.HttpResponse import java.nio.file.Files import java.nio.file.Path import java.nio.file.StandardOpenOption +import java.time.Duration import java.util.List -import groovy.json.JsonBuilder import groovy.json.JsonSlurper import groovy.transform.Canonical import groovy.transform.CompileStatic @@ -142,8 +143,8 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { }) // Verify login by calling /user-info - final userInfo = commonApi.getUserInfo(createHttpClient(accessToken), apiUrl) - println "\n\n${colorize('✔', 'green', true)} Authentication successful" + final userInfo = createTowerClient(apiUrl, accessToken).getUserInfo() + println "\n\n${colorize('✔', 'green', true)} Authentication successful ($userInfo.userName)" // Generate PAT final pat = generatePAT(accessToken, apiUrl) @@ -218,24 +219,17 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { final timestamp = new Date().format("yyyy-MM-dd-HH-mm") final tokenName = "nextflow-auth-${username}-${timestamp}" - final requestBody = new JsonBuilder([name: tokenName]).toString() - - final client = createHttpClient(accessToken) + final client = createTowerClient(apiUrl, accessToken) log.debug "Platform auth API - POST ${tokensUrl}" - final request = HttpRequest.newBuilder() - .uri(URI.create(tokensUrl)) - .header('Content-Type', 'application/json') - .POST(HttpRequest.BodyPublishers.ofString(requestBody)) - .build() - final response = client.send(request, HttpResponse.BodyHandlers.ofString()) + final response = client.sendApiRequest(tokensUrl, [name: tokenName], 'POST' ) - if( response.statusCode() != 200 ) { - final error = response.body() ?: "HTTP ${response.statusCode()}" + if( response.code != 200 ) { + final error = response.message ?: "HTTP ${response.code}" throw new RuntimeException("Failed to generate PAT: ${error}") } - final json = new JsonSlurper().parseText(response.body()) as Map + final json = new JsonSlurper().parseText(response.message) as Map return json.accessKey as String } @@ -313,7 +307,7 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { // Validate token by calling /user-info API try { - final userInfo = commonApi.getUserInfo( createHttpClient(existingToken as String), apiUrl) + final userInfo = createTowerClient(apiUrl, existingToken as String).getUserInfo() printColored(" - Token is valid for user: $userInfo.userName", "dim") } catch( Exception e ) { printColored("Failed to validate token: ${e.message}", "red") @@ -361,18 +355,14 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { } private void deleteTokenViaApi(String token, String apiUrl, String tokenId) { - final client = createHttpClient(token) + final client = createTowerClient(apiUrl, token) final url = "${apiUrl}/tokens/${tokenId}" log.debug "Platform auth API - DELETE ${url}" - final request = HttpRequest.newBuilder() - .uri(URI.create(url)) - .DELETE() - .build() - final response = client.send(request, HttpResponse.BodyHandlers.ofString()) + final response = client.sendApiRequest(url, null, 'DELETE') - if( response.statusCode() != 200 && response.statusCode() != 204 ) { - final error = response.body() ?: "HTTP ${response.statusCode()}" + if( response.code != 200 && response.code != 204 ) { + final error = response.message ?: "HTTP ${response.code}" throw new RuntimeException("Failed to delete token: ${error}") } @@ -440,8 +430,8 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { try { // Get user info to validate token and get user ID - final httpClient = createHttpClient(existingToken as String) - final userInfo = commonApi.getUserInfo(httpClient, endpoint as String) + final httpClient = createTowerClient(endpoint as String, existingToken as String) + final userInfo = httpClient.getUserInfo() printColored(" - Authenticated as: $userInfo.userName", "dim") println "" @@ -452,7 +442,7 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { configChanged |= configureEnabled(config) // Configure workspace - final workspaceResult = configureWorkspace(httpClient, config, endpoint as String, userInfo.id as String) + final workspaceResult = configureWorkspace(httpClient, config, userInfo.id as String) configChanged = configChanged || (workspaceResult.changed as boolean) // Configure compute environment for the workspace (always run after workspace selection) @@ -460,7 +450,7 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { def workspaceMetadata = workspaceResult.metadata as Map if( !workspaceMetadata && currentWorkspaceId ) { // Get workspace metadata if not already available (e.g., when user kept existing workspace) - workspaceMetadata = commonApi.getUserWorkspaceDetails(httpClient, userInfo.id as String, endpoint as String, currentWorkspaceId) + workspaceMetadata = httpClient.getUserWorkspaceDetails(userInfo.id as String, currentWorkspaceId) } final computeEnvResult = configureComputeEnvironment(config as Map, existingToken as String, endpoint as String, currentWorkspaceId, workspaceMetadata) configChanged = configChanged || (computeEnvResult.changed as boolean) @@ -508,11 +498,10 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { * * @param client * @param config Configuration map to update - * @param endpoint Seqera Platform API endpoint * @param userId User ID for fetching workspaces * @return Map containing 'changed' (boolean) and 'metadata' (workspace info) */ - private Map configureWorkspace(HxClient client, Map config, String endpoint, String userId) { + private Map configureWorkspace(TowerClient client, Map config, String userId) { // Check if TOWER_WORKFLOW_ID environment variable is set final envWorkspaceId = SysEnv.get('TOWER_WORKFLOW_ID') if( envWorkspaceId ) { @@ -522,7 +511,7 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { } // Get all workspaces for the user - final workspaces = listUserWorkspaces(client, endpoint, userId) + final workspaces = listUserWorkspaces(client, userId) if( !workspaces ) { println "\nNo workspaces found for your account." @@ -548,23 +537,21 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { } } - private String getCurrentWorkspaceName(List workspaces, currentWorkspaceId) { + private String getCurrentWorkspaceName(List workspaces, currentWorkspaceId) { final currentWorkspace = workspaces.find { ((Map) it).workspaceId.toString() == currentWorkspaceId?.toString() } as Map return currentWorkspace ? "${currentWorkspace.orgName} / ${currentWorkspace.workspaceName}" : "None (Personal workspace)" } - private Map selectWorkspaceFromAll(Map config, List workspaces, final currentWorkspaceId) { + private Map selectWorkspaceFromAll(Map config, List workspaces, final currentWorkspaceId) { println "\nAvailable workspaces:" final isPersonalWorkspace = !currentWorkspaceId final currentIndicator = isPersonalWorkspace ? colorize(' (current)', 'bold') : '' println " 0. ${colorize('None (Personal workspace)', 'cyan', true)} ${colorize('[no organization]', 'dim', true)}${currentIndicator}" // Sort workspaces by org name, then workspace name - final sortedWorkspaces = workspaces.sort { a, b -> - final aMap = a as Map - final bMap = b as Map - final orgCompare = (aMap.orgName as String ?: '').compareToIgnoreCase(bMap.orgName as String ?: '') - orgCompare != 0 ? orgCompare : (aMap.workspaceName as String ?: '').compareToIgnoreCase(bMap.workspaceName as String ?: '') + final sortedWorkspaces = workspaces.sort { Map a, Map b -> + final orgCompare = (a.orgName as String ?: '').compareToIgnoreCase(b.orgName as String ?: '') + orgCompare != 0 ? orgCompare : (a.workspaceName as String ?: '').compareToIgnoreCase(b.workspaceName as String ?: '') } sortedWorkspaces.eachWithIndex { workspace, index -> @@ -605,9 +592,9 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { private Map selectWorkspaceByOrg(Map config, Map orgWorkspaces, final currentWorkspaceId) { // Get current workspace info for prompts - final allWorkspaces = [] + final allWorkspaces = [] as List orgWorkspaces.values().each { workspaceList -> - allWorkspaces.addAll(workspaceList as List) + allWorkspaces.addAll(workspaceList as List) } final currentWorkspaceDisplay = getCurrentWorkspaceName(allWorkspaces, currentWorkspaceId) @@ -683,7 +670,7 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { private Map configureComputeEnvironment(Map config, String accessToken, String endpoint, String workspaceId, Map workspaceMetadata) { try { // Get compute environments for the workspace - final computeEnvs = listComputeEnvironments(createHttpClient(accessToken), endpoint, workspaceId) + final computeEnvs = listComputeEnvironments(createTowerClient(endpoint, accessToken), workspaceId) // If there are zero compute environments, log a warning and provide a link if( computeEnvs.isEmpty() ) { @@ -836,7 +823,7 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { if( accessToken ) { try { - final userInfo = commonApi.getUserInfo(createHttpClient(accessToken), endpoint) + final userInfo = createTowerClient(endpoint, accessToken).getUserInfo() final currentUser = userInfo.userName as String status.table.add(['Authentication', "${colorize('✔ OK', 'green')} (user: $currentUser)".toString(), tokenSource]) } catch( Exception e ) { @@ -858,9 +845,9 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { // Try to get workspace name and roles from API if we have a token def workspaceDetails = null if( accessToken ) { - final httpClient = createHttpClient(accessToken) - final userInfo = commonApi.getUserInfo(httpClient, endpoint) - workspaceDetails = commonApi.getUserWorkspaceDetails(httpClient, userInfo.id as String, endpoint, workspaceId) + final httpClient = createTowerClient(endpoint, accessToken) + final userInfo = httpClient.getUserInfo() + workspaceDetails = httpClient.getUserWorkspaceDetails(userInfo.id as String, workspaceId) } if( workspaceDetails ) { @@ -883,12 +870,12 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { // Compute environment and work directory def computeEnv = null if( accessToken ) { - final httpClient = createHttpClient(accessToken) + final httpClient = createTowerClient(endpoint, accessToken) try { if( config['tower.computeEnvId'] ) { - computeEnv = getComputeEnvironment(httpClient, endpoint, config['tower.computeEnvId'] as String, workspaceId) + computeEnv = getComputeEnvironment(httpClient, config['tower.computeEnvId'] as String, workspaceId) } else { - final computeEnvs = listComputeEnvironments(httpClient, endpoint, workspaceId) + final computeEnvs = listComputeEnvironments(httpClient, workspaceId) computeEnv = computeEnvs.find { ((Map) it).primary == true } as Map } } catch( Exception e ) { @@ -1007,7 +994,7 @@ class AuthCommandImpl extends BaseCommandImpl implements CmdAuth.AuthCommand { protected boolean checkApiConnection(String endpoint) { try { - final client = createHttpClient() + final client = HxClient.newBuilder().connectTimeout(Duration.ofMillis(API_TIMEOUT_MS)).build() final url = "${endpoint}/service-info" log.debug "Platform auth API - GET ${url}" final request = HttpRequest.newBuilder() diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/dataset/SeqeraDatasetClient.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/dataset/SeqeraDatasetClient.groovy new file mode 100644 index 0000000000..143b193c23 --- /dev/null +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/dataset/SeqeraDatasetClient.groovy @@ -0,0 +1,202 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.dataset + +import io.seqera.tower.plugin.exception.ForbiddenException +import io.seqera.tower.plugin.exception.NotFoundException +import io.seqera.tower.plugin.exception.UnauthorizedException + +import java.nio.file.AccessDeniedException +import java.nio.file.NoSuchFileException +import java.time.OffsetDateTime + +import groovy.json.JsonSlurper +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.tower.model.DatasetDto +import io.seqera.tower.model.DatasetVersionDto +import io.seqera.tower.model.OrgAndWorkspaceDto +import io.seqera.tower.plugin.TowerClient +import nextflow.exception.AbortOperationException + +/** + * Typed client for Seqera Platform dataset API endpoints. + * Delegates HTTP execution to {@link TowerClient#sendApiRequest}, inheriting its + * authentication token management and retry policy without exposing the underlying + * HTTP client. + * + * @author Seqera Labs + */ +@Slf4j +@CompileStatic +class SeqeraDatasetClient { + + private final TowerClient towerClient + + SeqeraDatasetClient(TowerClient towerClient) { + this.towerClient = towerClient + } + + private String getEndpoint() { towerClient.endpoint } + + /** + * @return current user info (id, userName, etc.) from GET /user-info + */ + Long getUserId() { + try { + final info = towerClient.getUserInfo() + if( info?.id == null ) + throw new AbortOperationException("Unable to retrieve user ID from Seqera Platform — check your access token") + return info.id as long + }catch( UnauthorizedException e ){ + throw new AbortOperationException(e.getMessage()) + }catch( ForbiddenException e){ + throw new AccessDeniedException("${endpoint}/user-info", null, e.message) + }catch(NotFoundException e){ + throw new NoSuchFileException("${endpoint}/user-info") + } + } + + /** + * @return all orgs and workspaces accessible to the given user from GET /user/{userId}/workspaces + */ + List listUserWorkspacesAndOrgs(long userId) { + try { + final list = towerClient.listUserWorkspacesAndOrgs(userId as String) + return list.collect { m -> mapOrgAndWorkspace(m) } + } catch( UnauthorizedException e ){ + throw new AbortOperationException(e.getMessage()) + } catch( ForbiddenException e){ + throw new AccessDeniedException("${endpoint}/user/$userId/workspaces", null, e.message) + } catch(NotFoundException e){ + throw new NoSuchFileException("${endpoint}/user/$userId/workspaces") + } + } + + + + /** + * @return all datasets in the given workspace from GET /datasets?workspaceId={workspaceId} + */ + List listDatasets(long workspaceId) { + final url = "${endpoint}/datasets?workspaceId=${workspaceId}" + log.debug "SeqeraDatasetClient GET $url" + final resp = towerClient.sendApiRequest(url) + checkFsResponse(resp, url) + final json = new JsonSlurper().parseText(resp.message) as Map + final list = json.datasets as List + return list ? list.collect { m -> mapDataset(m) } : Collections.emptyList() + } + + /** + * Create a new dataset in the given workspace via POST /datasets?workspaceId={workspaceId}. + * @return the created dataset DTO + */ + DatasetDto createDataset(long workspaceId, String name) { + final url = "${endpoint}/datasets?workspaceId=${workspaceId}" + log.debug "SeqeraDatasetClient POST $url name=$name" + final resp = towerClient.sendApiRequest(url, [name: name], 'POST') + checkFsResponse(resp, url) + final json = new JsonSlurper().parseText(resp.message) as Map + return mapDataset(json.dataset as Map) + } + + /** + * @return all versions for the given dataset from GET /datasets/{datasetId}/versions + */ + List listVersions(String datasetId, long workspaceId) { + final url = "${endpoint}/datasets/${datasetId}/versions?workspaceId=${workspaceId}" + log.debug "SeqeraDatasetClient GET $url" + final resp = towerClient.sendApiRequest(url) + checkFsResponse(resp, url) + final json = new JsonSlurper().parseText(resp.message) as Map + final list = json.versions as List + return list ? list.collect { m -> mapVersion(m) } : Collections.emptyList() + } + + /** + * Download a dataset version as an InputStream. + * GET /datasets/{datasetId}/v/{version}/n/{fileName} + * The fileName must exactly match DatasetVersionDto.fileName from upload time. + */ + InputStream downloadDataset(String datasetId, String version, String fileName, long workspaceId) { + final encodedName = new URI(null, null, fileName, null).rawPath + final url = "${endpoint}/datasets/${datasetId}/v/${version}/n/${encodedName}?workspaceId=$workspaceId" + log.debug "SeqeraDatasetClient GET $url (streaming)" + try { + return towerClient.sendStreamingRequest(url) + } + catch (UnauthorizedException e) { + throw new AbortOperationException("Seqera authentication failed — check tower.accessToken or TOWER_ACCESS_TOKEN") + } + catch (ForbiddenException e) { + throw new AccessDeniedException(url, null, "Forbidden — check workspace permissions") + } + catch (NotFoundException e) { + throw new NoSuchFileException(url) + } + } + + // ---- private helpers ---- + + private static void checkFsResponse(TowerClient.Response resp, String url) { + if (!resp.error) return + final code = resp.code + if (code == 401) + throw new AbortOperationException("Seqera authentication failed — check tower.accessToken or TOWER_ACCESS_TOKEN") + if (code == 403) + throw new AccessDeniedException(url, null, "Forbidden — check workspace permissions") + if (code == 404) + throw new NoSuchFileException(url) + throw new IOException("Seqera API error: HTTP ${code} for ${url}") + } + + private static OrgAndWorkspaceDto mapOrgAndWorkspace(Map m) { + final dto = new OrgAndWorkspaceDto() + dto.orgId = (m.orgId as Long) ?: 0L + dto.orgName = m.orgName as String + dto.workspaceId = (m.workspaceId as Long) ?: 0L + dto.workspaceName = m.workspaceName as String + dto.workspaceFullName = m.workspaceFullName as String + return dto + } + + private static DatasetDto mapDataset(Map m) { + final dto = new DatasetDto() + dto.id = m.id as String + dto.name = m.name as String + dto.description = m.description as String + dto.version = (m.version as Long) ?: 0L + dto.mediaType = m.mediaType as String + dto.workspaceId = (m.workspaceId as Long) ?: 0L + dto.dateCreated = m.dateCreated ? OffsetDateTime.parse(m.dateCreated as String) : null + dto.lastUpdated = m.lastUpdated ? OffsetDateTime.parse(m.lastUpdated as String) : null + return dto + } + + private static DatasetVersionDto mapVersion(Map m) { + final dto = new DatasetVersionDto() + dto.datasetId = m.datasetId as String + dto.version = (m.version as Long) ?: 0L + dto.fileName = m.fileName as String + dto.mediaType = m.mediaType as String + dto.hasHeader = (m.hasHeader as Boolean) ?: false + dto.dateCreated = m.dateCreated ? OffsetDateTime.parse(m.dateCreated as String) : null + dto.disabled = (m.disabled as Boolean) ?: false + return dto + } +} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/exception/ForbiddenException.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/exception/ForbiddenException.groovy new file mode 100644 index 0000000000..fdfb756bed --- /dev/null +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/exception/ForbiddenException.groovy @@ -0,0 +1,22 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.seqera.tower.plugin.exception + +import groovy.transform.InheritConstructors + +@InheritConstructors +class ForbiddenException extends RuntimeException { +} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/exception/NotFoundException.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/exception/NotFoundException.groovy new file mode 100644 index 0000000000..f7d8d0da7d --- /dev/null +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/exception/NotFoundException.groovy @@ -0,0 +1,22 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package io.seqera.tower.plugin.exception + +import groovy.transform.InheritConstructors + +@InheritConstructors +class NotFoundException extends RuntimeException { +} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/DatasetInputStream.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/DatasetInputStream.groovy new file mode 100644 index 0000000000..7231e806c6 --- /dev/null +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/DatasetInputStream.groovy @@ -0,0 +1,80 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.fs + +import java.nio.ByteBuffer +import java.nio.channels.ClosedChannelException +import java.nio.channels.SeekableByteChannel + +import groovy.transform.CompileStatic + +/** + * Minimal {@link SeekableByteChannel} backed by an {@link InputStream}. + * Supports sequential reads only (no seek/position). + * + * @author Seqera Labs + */ +@CompileStatic +class DatasetInputStream implements SeekableByteChannel { + private final InputStream inputStream + private long position0 = 0L + private boolean open = true + private byte[] buf = new byte[0] + + DatasetInputStream(InputStream inputStream) { + this.inputStream = inputStream + } + + @Override + int read(ByteBuffer dst) throws IOException { + if( !open ) + throw new ClosedChannelException() + final len = dst.remaining() + if (buf.length < len) + buf = new byte[len] + final n = inputStream.read(buf, 0, len) + if (n > 0) { + dst.put(buf, 0, n) + position0 += n + } + return n + } + + @Override + int write(ByteBuffer src) { throw new UnsupportedOperationException() } + + @Override + long position() { position0 } + + @Override + SeekableByteChannel position(long newPosition) { throw new UnsupportedOperationException("seek not supported") } + + @Override + long size() { throw new UnsupportedOperationException("size not available for streaming dataset channel") } + + @Override + SeekableByteChannel truncate(long size) { throw new UnsupportedOperationException() } + + @Override + boolean isOpen() { open } + + @Override + void close() throws IOException { + open = false + inputStream.close() + } +} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraFileAttributes.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraFileAttributes.groovy new file mode 100644 index 0000000000..3246bd35e2 --- /dev/null +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraFileAttributes.groovy @@ -0,0 +1,87 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.fs + +import java.nio.file.attribute.BasicFileAttributes +import java.nio.file.attribute.FileTime +import java.time.Instant + +import groovy.transform.CompileStatic +import io.seqera.tower.model.DatasetDto + +/** + * {@link BasicFileAttributes} for {@code seqera://} paths. + * For depth < 4 (directory paths): {@code isDirectory=true}, {@code size=0}. + * For depth 4 (dataset file paths): {@code isRegularFile=true}, timestamps from {@link DatasetDto}. + * + * @author Seqera Labs + */ +@CompileStatic +class SeqeraFileAttributes implements BasicFileAttributes { + + private final boolean directory + private final DatasetDto dataset + + /** Construct attributes for a virtual directory (depth 0–3). */ + SeqeraFileAttributes(boolean isDir) { + this.directory = isDir + this.dataset = null + } + + /** Construct attributes for a dataset file (depth 4). */ + SeqeraFileAttributes(DatasetDto dataset) { + this.directory = false + this.dataset = dataset + } + + @Override + FileTime lastModifiedTime() { + if (dataset?.lastUpdated) { + return FileTime.from(dataset.lastUpdated.toInstant()) + } + return FileTime.from(Instant.EPOCH) + } + + @Override + FileTime lastAccessTime() { lastModifiedTime() } + + @Override + FileTime creationTime() { + if (dataset?.dateCreated) { + return FileTime.from(dataset.dateCreated.toInstant()) + } + return FileTime.from(Instant.EPOCH) + } + + @Override + boolean isRegularFile() { !directory } + + @Override + boolean isDirectory() { directory } + + @Override + boolean isSymbolicLink() { false } + + @Override + boolean isOther() { false } + + @Override + long size() { 0L } + + @Override + Object fileKey() { dataset?.id } +} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraFileSystem.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraFileSystem.groovy new file mode 100644 index 0000000000..4f639facd6 --- /dev/null +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraFileSystem.groovy @@ -0,0 +1,214 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.fs + +import java.nio.file.FileStore +import java.nio.file.FileSystem +import java.nio.file.NoSuchFileException +import java.nio.file.Path +import java.nio.file.PathMatcher +import java.nio.file.WatchService +import java.nio.file.attribute.UserPrincipalLookupService +import java.nio.file.spi.FileSystemProvider + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.tower.model.DatasetDto +import io.seqera.tower.model.DatasetVersionDto +import io.seqera.tower.model.OrgAndWorkspaceDto +import io.seqera.tower.plugin.dataset.SeqeraDatasetClient + +/** + * FileSystem instance for the {@code seqera://} scheme. + * One instance per (endpoint + credentials) pair, cached by {@link SeqeraFileSystemProvider}. + * + * Lazily populates org/workspace/dataset caches on first access. + * Cache is invalidated on dataset write operations. + * + * @author Seqera Labs + */ +@Slf4j +@CompileStatic +class SeqeraFileSystem extends FileSystem { + + private final SeqeraFileSystemProvider provider0 + final SeqeraDatasetClient client + + /** orgName → orgId */ + private final Map orgCache = new LinkedHashMap<>() + /** "orgName/workspaceName" → workspaceId */ + private final Map workspaceCache = new LinkedHashMap<>() + /** workspaceId → list of DatasetDto */ + private final Map> datasetCache = new LinkedHashMap<>() + /** datasetId → list of DatasetVersionDto */ + private final Map> versionCache = new LinkedHashMap<>() + + private volatile boolean orgWorkspaceCacheLoaded = false + + SeqeraFileSystem(SeqeraFileSystemProvider provider, SeqeraDatasetClient client) { + this.provider0 = provider + this.client = client + } + + @Override + FileSystemProvider provider() { provider0 } + + @Override + void close() { /* no-op: platform API connection is stateless */ } + + @Override + boolean isOpen() { true } + + @Override + boolean isReadOnly() { true } + + @Override + String getSeparator() { '/' } + + @Override + Iterable getRootDirectories() { + return [getPath('seqera://')] as Iterable + } + + @Override + Iterable getFileStores() { Collections.emptyList() } + + @Override + Set supportedFileAttributeViews() { Collections.singleton('basic') } + + @Override + Path getPath(String first, String... more) { + final full = more ? ([first] + more.toList()).join(getSeparator()) : first + return new SeqeraPath(this, full) + } + + @Override + PathMatcher getPathMatcher(String syntaxAndPattern) { + throw new UnsupportedOperationException("PathMatcher not supported by seqera:// filesystem") + } + + @Override + UserPrincipalLookupService getUserPrincipalLookupService() { + throw new UnsupportedOperationException("UserPrincipalLookupService not supported by seqera:// filesystem") + } + + @Override + WatchService newWatchService() { + throw new UnsupportedOperationException("WatchService not supported by seqera:// filesystem") + } + + // ---- cache management ---- + + /** + * Ensure the org/workspace cache is populated. Thread-safe: loads at most once. + * Calls GET /user-info then GET /user/{userId}/workspaces. + */ + synchronized void loadOrgWorkspaceCache() { + if (orgWorkspaceCacheLoaded) return + log.debug "Loading Seqera org/workspace cache" + final entries = client.listUserWorkspacesAndOrgs(client.getUserId()) + for (OrgAndWorkspaceDto entry : entries) { + if (entry.orgName) + orgCache.put(entry.orgName, entry.orgId) + if (entry.orgName && entry.workspaceName && entry.workspaceId) + workspaceCache.put("${entry.orgName}/${entry.workspaceName}" as String, entry.workspaceId) + } + orgWorkspaceCacheLoaded = true + } + + /** + * @return distinct org names visible to the authenticated user + */ + synchronized Set listOrgNames() { + loadOrgWorkspaceCache() + return Collections.unmodifiableSet(orgCache.keySet()) + } + + /** + * @return workspace names for the given org + */ + synchronized List listWorkspaceNames(String org) { + loadOrgWorkspaceCache() + return workspaceCache.keySet() + .findAll { String k -> k.startsWith("${org}/") } + .collect { String k -> k.substring(org.length() + 1) } + } + + /** + * Resolve a workspace ID by org and workspace name. + * @throws NoSuchFileException if the org or workspace is not in the cache + */ + synchronized long resolveWorkspaceId(String org, String workspace) throws NoSuchFileException { + loadOrgWorkspaceCache() + final key = "${org}/${workspace}" as String + final id = workspaceCache.get(key) + if (id == null) + throw new NoSuchFileException("seqera://${key}", null, "Org or workspace not found or not accessible") + return id + } + + /** + * Return datasets for the given workspace, populating the cache on first access. + */ + synchronized List resolveDatasets(long workspaceId) { + List cached = datasetCache.get(workspaceId) + if (cached == null) { + cached = client.listDatasets(workspaceId) + datasetCache.put(workspaceId, cached) + } + return cached + } + + /** + * Invalidate the dataset and version caches for a workspace (call after a write operation). + */ + synchronized void invalidateDatasetCache(long workspaceId) { + // Remove version caches for all datasets in this workspace + final datasets = datasetCache.get(workspaceId) + if (datasets) { + for (DatasetDto ds : datasets) { + versionCache.remove(ds.id) + } + } + datasetCache.remove(workspaceId) + } + + /** + * Resolve a DatasetDto by name within a workspace. + * @throws NoSuchFileException if no dataset with the given name exists + */ + synchronized DatasetDto resolveDataset(long workspaceId, String name) throws NoSuchFileException { + final datasets = resolveDatasets(workspaceId) + return datasets.find { DatasetDto d -> d.name == name } + } + + /** + * Return versions for the given dataset, populating the cache on first access. + * Note: the version cache is only invalidated when the workspace dataset cache is invalidated + * (e.g. after a write operation). Versions published externally during a pipeline run will not + * be visible until the cache is cleared. + */ + synchronized List resolveVersions(String datasetId, long workspaceId) { + List cached = versionCache.get(datasetId) + if (cached == null) { + cached = client.listVersions(datasetId, workspaceId) + versionCache.put(datasetId, cached) + } + return cached + } + +} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraFileSystemProvider.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraFileSystemProvider.groovy new file mode 100644 index 0000000000..bed6667dd3 --- /dev/null +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraFileSystemProvider.groovy @@ -0,0 +1,325 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.fs + +import java.nio.channels.SeekableByteChannel +import java.nio.file.AccessDeniedException +import java.nio.file.AccessMode +import java.nio.file.CopyOption +import java.nio.file.DirectoryStream +import java.nio.file.FileStore +import java.nio.file.FileSystem +import java.nio.file.FileSystemAlreadyExistsException +import java.nio.file.FileSystemNotFoundException +import java.nio.file.Files +import java.nio.file.LinkOption +import java.nio.file.DirectoryIteratorException +import java.nio.file.NoSuchFileException +import java.nio.file.NotDirectoryException +import java.nio.file.OpenOption +import java.nio.file.Path +import java.nio.file.ProviderMismatchException +import java.nio.file.StandardOpenOption +import java.nio.file.attribute.BasicFileAttributes +import java.nio.file.attribute.FileAttribute +import java.nio.file.attribute.FileAttributeView +import java.nio.file.spi.FileSystemProvider + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import io.seqera.tower.model.DatasetDto +import io.seqera.tower.model.DatasetVersionDto +import io.seqera.tower.plugin.TowerClient +import io.seqera.tower.plugin.TowerFactory +import io.seqera.tower.plugin.dataset.SeqeraDatasetClient + +/** + * NIO {@link FileSystemProvider} for the {@code seqera://} scheme. + * Registered via {@code META-INF/services/java.nio.file.spi.FileSystemProvider}. + * + * Enables Nextflow pipelines to read Seqera Platform datasets as ordinary file paths: + * {@code seqera:////datasets/} + * + * Follows the {@code LinFileSystemProvider} pattern for structure. + * Write support follows the {@code AzFileSystemProvider} buffered-upload pattern. + * + * @author Seqera Labs + */ +@Slf4j +@CompileStatic +class SeqeraFileSystemProvider extends FileSystemProvider { + + public static final String SCHEME = 'seqera' + + /** Single filesystem instance — TowerClient is a singleton per session */ + private volatile SeqeraFileSystem fileSystem + + @Override + String getScheme() { SCHEME } + + // ---- FileSystem lifecycle ---- + + @Override + synchronized FileSystem newFileSystem(URI uri, Map env) throws IOException { + checkScheme(uri) + if (fileSystem) + throw new FileSystemAlreadyExistsException("File system `seqera://` already exists") + final TowerClient towerClient = TowerFactory.client() + if (!towerClient) + throw new IllegalStateException("File system `seqera://` requires the Seqera Platform access token to be provided - use `tower.accessToken` config option or TOWER_ACCESS_TOKEN env variable") + final client = new SeqeraDatasetClient(towerClient) + fileSystem = new SeqeraFileSystem(this, client) + return fileSystem + } + + @Override + synchronized FileSystem getFileSystem(URI uri) { + checkScheme(uri) + if (!fileSystem) throw new FileSystemNotFoundException("No seqera:// filesystem has been created yet") + return fileSystem + } + + synchronized SeqeraFileSystem getOrCreateFileSystem(URI uri, Map env) { + checkScheme(uri) + if (!fileSystem) { + final envMap = env ?: Collections.emptyMap() + newFileSystem(uri, envMap as Map) + } + return fileSystem + } + + @Override + SeqeraPath getPath(URI uri) { + final fs = getOrCreateFileSystem(uri, Collections.emptyMap()) + return new SeqeraPath(fs, uri.toString()) + } + + // ---- Read operations ---- + + @Override + InputStream newInputStream(Path path, OpenOption... options) throws IOException { + final sp = toSeqeraPath(path) + if (sp.depth() != 4) + throw new IllegalArgumentException("Operation `newInputStream` requires a dataset path (depth 4): $path") + final fs = sp.getFileSystem() as SeqeraFileSystem + final workspaceId = fs.resolveWorkspaceId(sp.org, sp.workspace) + final dataset = fs.resolveDataset(workspaceId, sp.datasetName) + if (!dataset) + throw new NoSuchFileException(sp.toString(), null, "Dataset '${sp.datasetName}' not found in workspace $sp.workspace") + final version = resolveVersion(fs, dataset, sp) + log.debug "Downloading dataset '${sp.datasetName}' version ${version.version} (${version.fileName}) from workspace $workspaceId" + return fs.client.downloadDataset(dataset.id, String.valueOf(version.version), version.fileName, dataset.workspaceId) + } + + @Override + SeekableByteChannel newByteChannel(Path path, Set options, FileAttribute... attrs) throws IOException { + if (options?.contains(StandardOpenOption.WRITE) || options?.contains(StandardOpenOption.APPEND)) + throw new UnsupportedOperationException("File system `seqera://` is read-only") + final inputStream = newInputStream(path) + return new DatasetInputStream(inputStream) + } + + // ---- Metadata ---- + + @Override + A readAttributes(Path path, Class type, LinkOption... options) throws IOException { + if (!BasicFileAttributes.isAssignableFrom(type)) + throw new UnsupportedOperationException("Attribute type not supported: $type") + final sp = toSeqeraPath(path) + final fs = sp.getFileSystem() as SeqeraFileSystem + final d = sp.depth() + if (d < 4) { + // Virtual directory — validate the path exists (throws NoSuchFileException if not) + validateDirectoryExists(fs, sp) + return (A) new SeqeraFileAttributes(true) + } + // Dataset file + final workspaceId = fs.resolveWorkspaceId(sp.org, sp.workspace) + final dataset = fs.resolveDataset(workspaceId, sp.datasetName) + if (!dataset) + throw new NoSuchFileException(sp.toString(), null, "Dataset '${sp.datasetName}' not found in workspace $sp.workspace") + return (A) new SeqeraFileAttributes(dataset) + } + + @Override + Map readAttributes(Path path, String attributes, LinkOption... options) throws IOException { + throw new UnsupportedOperationException("Operation `readAttributes(String)` not supported by `seqera://` file system") + } + + // ---- Access check ---- + + @Override + void checkAccess(Path path, AccessMode... modes) throws IOException { + final sp = toSeqeraPath(path) + for (AccessMode m : modes) { + if (m == AccessMode.WRITE || m == AccessMode.EXECUTE) + throw new AccessDeniedException(path.toString(), null, "seqera:// filesystem is read-only") + } + // For READ, verify the path resolves without throwing NoSuchFileException + if (sp.depth() >= 1) { + final fs = sp.getFileSystem() as SeqeraFileSystem + if (sp.depth() == 1) { + fs.loadOrgWorkspaceCache() + if (!fs.listOrgNames().contains(sp.org)) + throw new NoSuchFileException(path.toString(), null, "Organisation not found") + } else { + fs.resolveWorkspaceId(sp.org, sp.workspace) + } + } + } + + // ---- Directory stream ---- + + @Override + DirectoryStream newDirectoryStream(Path dir, DirectoryStream.Filter filter) throws IOException { + final sp = toSeqeraPath(dir) + final fs = sp.getFileSystem() as SeqeraFileSystem + final d = sp.depth() + List entries + if (d == 0) { + // Root: list distinct org names + fs.loadOrgWorkspaceCache() + entries = fs.listOrgNames().collect { String org -> sp.resolve(org) as Path } + } else if (d == 1) { + // Org: list workspace names + fs.loadOrgWorkspaceCache() + entries = fs.listWorkspaceNames(sp.org).collect { String ws -> sp.resolve(ws) as Path } + } else if (d == 2) { + // Workspace: static resource types + entries = ['datasets'].collect { String rt -> sp.resolve(rt) as Path } + } else if (d == 3) { + // Resource type directory: list dataset names + final workspaceId = fs.resolveWorkspaceId(sp.org, sp.workspace) + entries = fs.resolveDatasets(workspaceId).collect { DatasetDto ds -> + sp.resolve(ds.name) as Path + } + } else { + throw new NotDirectoryException(dir.toString()) + } + + final filtered = filter ? entries.findAll { Path p -> + try { filter.accept(p) } + catch (IOException e) { throw new DirectoryIteratorException(e) } + } : entries + + return new DirectoryStream() { + @Override Iterator iterator() { filtered.iterator() } + @Override void close() {} + } + } + + // ---- Copy ---- + + @Override + void copy(Path source, Path target, CopyOption... options) throws IOException { + toSeqeraPath(source) + if (target instanceof SeqeraPath) + throw new UnsupportedOperationException("seqera:// filesystem is read-only") + // cross-provider (seqera → local): stream to target + try (final InputStream is = newInputStream(source)) { + Files.copy(is, target, options) + } + } + + // ---- Unsupported mutations ---- + + @Override + void move(Path source, Path target, CopyOption... options) { + throw new UnsupportedOperationException("move() not supported by seqera:// filesystem") + } + + @Override + void delete(Path path) { + throw new UnsupportedOperationException("delete() not supported by seqera:// filesystem") + } + + @Override + void createDirectory(Path dir, FileAttribute... attrs) { + throw new UnsupportedOperationException("createDirectory() not supported by seqera:// filesystem") + } + + // ---- Misc ---- + + @Override + boolean isSameFile(Path path, Path path2) throws IOException { + return path == path2 + } + + @Override + boolean isHidden(Path path) { false } + + @Override + FileStore getFileStore(Path path) { + throw new UnsupportedOperationException("getFileStore() not supported by seqera:// filesystem") + } + + @Override + V getFileAttributeView(Path path, Class type, LinkOption... options) { + return null + } + + @Override + void setAttribute(Path path, String attribute, Object value, LinkOption... options) { + throw new UnsupportedOperationException("setAttribute() not supported by seqera:// filesystem") + } + + // ---- private helpers ---- + + private static SeqeraPath toSeqeraPath(Path path) { + if (path !instanceof SeqeraPath) + throw new ProviderMismatchException() + return (SeqeraPath) path + } + + private static void checkScheme(URI uri) { + if (uri.scheme?.toLowerCase() != SCHEME) + throw new IllegalArgumentException("Not a seqera:// URI: $uri") + } + + private static void validateDirectoryExists(SeqeraFileSystem fs, SeqeraPath sp) throws NoSuchFileException { + final d = sp.depth() + if (d == 0) return + // Depth 1+: ensure org/workspace cache is loaded + fs.loadOrgWorkspaceCache() + if (d >= 1 && !fs.listOrgNames().contains(sp.org)) + throw new NoSuchFileException("seqera://${sp.org}", null, "Organisation not found") + if (d >= 2) + fs.resolveWorkspaceId(sp.org, sp.workspace) + if (d >= 3 && sp.resourceType != 'datasets') + throw new NoSuchFileException("seqera://${sp.org}/${sp.workspace}/${sp.resourceType}", null, "Unsupported resource type") + } + + private static DatasetVersionDto resolveVersion(SeqeraFileSystem fs, DatasetDto dataset, SeqeraPath sp) throws IOException { + final pinnedVersion = sp.version + final versions = fs.resolveVersions(dataset.id, dataset.workspaceId) + if (versions.isEmpty()) + throw new NoSuchFileException(sp.toString(), null, "No versions available for dataset '${dataset.name}'") + if (pinnedVersion) { + final found = versions.find { DatasetVersionDto v -> String.valueOf(v.version) == pinnedVersion } + if (!found) + throw new NoSuchFileException(sp.toString(), null, "Version '${pinnedVersion}' not found for dataset '${dataset.name}'") + return found + } + // Latest non-disabled version + final latest = versions.findAll { DatasetVersionDto v -> !v.disabled } + .max { DatasetVersionDto v -> v.version } + if (!latest) + throw new NoSuchFileException(sp.toString(), null, "No enabled versions for dataset '${dataset.name}'") + return latest + } + +} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraPath.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraPath.groovy new file mode 100644 index 0000000000..af2b39165c --- /dev/null +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraPath.groovy @@ -0,0 +1,490 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.fs + +import java.nio.file.FileSystem +import java.nio.file.InvalidPathException +import java.nio.file.LinkOption +import java.nio.file.Path +import java.nio.file.ProviderMismatchException +import java.nio.file.WatchEvent +import java.nio.file.WatchKey +import java.nio.file.WatchService + +import groovy.transform.CompileStatic + +/** + * {@link Path} implementation for the {@code seqera://} scheme. + * + * Path hierarchy: + *
+ *   depth 0  seqera://                                   (root — directory)
+ *   depth 1  seqera://<org>                              (org — directory)
+ *   depth 2  seqera://<org>/<workspace>                  (workspace — directory)
+ *   depth 3  seqera://<org>/<workspace>/datasets          (resource type — directory)
+ *   depth 4  seqera://<org>/<workspace>/datasets/<name>  (dataset file)
+ *            seqera://<org>/<workspace>/datasets/<name@ver>  (pinned version)
+ * 
+ * + * @author Seqera Labs + */ +@CompileStatic +class SeqeraPath implements Path { + + /** URI scheme */ + public static final String SCHEME = 'seqera' + public static final String PROTOCOL = "${SCHEME}://" + public static final String SEPARATOR = '/' + + private final SeqeraFileSystem fs + /** path segments in order: [org, workspace, resourceType, datasetName] — null for missing levels */ + private final String org + private final String workspace + private final String resourceType + private final String datasetName + /** version string extracted from {@code @version} suffix; null when not pinned */ + private final String version + /** + * Raw relative path string — non-null only for relative {@code SeqeraPath} instances + * created by {@link #relativize(Path)}. When non-null, {@link #fs} is {@code null} + * and all segment fields are {@code null}. + */ + private final String relPath + + /** + * Parse a {@code seqera://} URI string into a SeqeraPath. + * The URI authority is the org; path segments are workspace, resourceType, datasetName. + * The last segment may contain a {@code @version} suffix. + */ + SeqeraPath(SeqeraFileSystem fs, String uriString) { + this.fs = fs + this.relPath = null + if (!uriString.startsWith("${SCHEME}://")) + throw new InvalidPathException(uriString, "Not a seqera:// URI") + // strip scheme: seqera://rest + final withoutScheme = uriString.substring("${SCHEME}://".length()) + // split on '/' + final parts = withoutScheme.split('/', -1).toList().findAll { it != null } as List + // parts[0]=org, parts[1]=workspace, parts[2]=resourceType, parts[3]=datasetName[@version] + this.org = parts.size() > 0 && parts[0] ? parts[0] : null + this.workspace = parts.size() > 1 && parts[1] ? parts[1] : null + this.resourceType = parts.size() > 2 && parts[2] ? parts[2] : null + if (parts.size() > 3 && parts[3]) { + final last = parts[3] + final atIdx = last.lastIndexOf('@') + if (atIdx > 0) { + this.datasetName = last.substring(0, atIdx) + this.version = last.substring(atIdx + 1) + } else { + this.datasetName = last + this.version = null + } + } else { + this.datasetName = null + this.version = null + } + validatePath(uriString) + } + + /** Internal constructor for programmatic absolute path creation */ + SeqeraPath(SeqeraFileSystem fs, String org, String workspace, String resourceType, String datasetName, String version) { + this.fs = fs + this.relPath = null + this.org = org + this.workspace = workspace + this.resourceType = resourceType + this.datasetName = datasetName + this.version = version + validatePath(null) + } + + /** + * Constructor for relative paths produced by {@link #relativize(Path)}. + * The {@code relPath} is a slash-separated string of the differing path segments. + * All segment fields are {@code null}; {@link #isAbsolute()} returns {@code false}. + */ + SeqeraPath(String relPath) { + this.fs = null + this.relPath = relPath ?: '' + this.org = null + this.workspace = null + this.resourceType = null + this.datasetName = null + this.version = null + } + + /** + * Validate structural integrity: deeper segments require all shallower ones, + * and no segment may contain {@code /}. + * + * @param original original URI string used in error messages (null → derive from fields) + * @throws InvalidPathException if the path is malformed + */ + private void validatePath(String original) { + final label = original ?: rawPath() + if (datasetName && !workspace) + throw new InvalidPathException(label, "Dataset path requires a workspace segment") + if (resourceType && !workspace) + throw new InvalidPathException(label, "Resource type requires a workspace segment") + if (workspace && !org) + throw new InvalidPathException(label, "Workspace requires an org segment") + // Segments from URI parsing never contain '/', but guard the internal constructor too + if (org?.contains('/')) + throw new InvalidPathException(label, "Org name cannot contain '/'") + if (workspace?.contains('/')) + throw new InvalidPathException(label, "Workspace name cannot contain '/'") + if (resourceType?.contains('/')) + throw new InvalidPathException(label, "Resource type cannot contain '/'") + if (datasetName?.contains('/')) + throw new InvalidPathException(label, "Dataset name cannot contain '/'") + } + + /** Return a list of name component strings (works for both absolute and relative paths). */ + private List nameComponents() { + if (isAbsolute()) { + final d = depth() + final result = new ArrayList(d) + for (int i = 0; i < d; i++) + result.add(getName(i).toString()) + return result + } + if (!relPath) return Collections.emptyList() + return relPath.split('/').toList().findAll { String s -> s } as List + } + + /** Build a raw path string from the current fields, for use in exception messages. */ + private String rawPath() { + final sb = new StringBuilder("${SCHEME}://") + if (org) sb.append(org) + if (workspace) sb.append('/').append(workspace) + if (resourceType) sb.append('/').append(resourceType) + if (datasetName) { + sb.append('/').append(datasetName) + if (version) sb.append('@').append(version) + } + return sb.toString() + } + + // ---- path component accessors ---- + + String getOrg() { org } + String getWorkspace() { workspace } + String getResourceType() { resourceType } + String getDatasetName() { datasetName } + String getVersion() { version } + + /** + * Path depth: 0=root, 1=org, 2=workspace, 3=resourceType, 4=dataset file. + */ + int depth() { + if (datasetName) return 4 + if (resourceType) return 3 + if (workspace) return 2 + if (org) return 1 + return 0 + } + + boolean isDirectory() { depth() < 4 } + boolean isRegularFile() { depth() == 4 } + + // ---- Path API ---- + + @Override + FileSystem getFileSystem() { fs } + + @Override + boolean isAbsolute() { fs != null } + + @Override + Path getRoot() { new SeqeraPath(fs, null, null, null, null, null) } + + @Override + Path getFileName() { + final d = depth() + if (d == 0) return null + final name = d == 4 ? (version ? "${datasetName}@${version}" : datasetName) + : d == 3 ? resourceType + : d == 2 ? workspace + : org + return new SeqeraPath( name as String) + } + + @Override + Path getParent() { + final d = depth() + if (d == 0) return null + if (d == 1) return new SeqeraPath(fs, null, null, null, null, null) + if (d == 2) return new SeqeraPath(fs, org, null, null, null, null) + if (d == 3) return new SeqeraPath(fs, org, workspace, null, null, null) + return new SeqeraPath(fs, org, workspace, resourceType, null, null) + } + + @Override + int getNameCount() { depth() } + + @Override + Path getName(int index) { + final d = depth() + if (index < 0 || index >= d) + throw new IllegalArgumentException("Index out of range: $index") + if (index == 0) return new SeqeraPath(org) + if (index == 1) return new SeqeraPath(workspace) + if (index == 2) return new SeqeraPath(resourceType) + return new SeqeraPath((version ? "${datasetName}@${version}" : datasetName) as String) + } + + @Override + Path subpath(int beginIndex, int endIndex) { + throw new UnsupportedOperationException("subpath not supported by seqera:// paths") + } + + @Override + boolean startsWith(Path other) { + if (other !instanceof SeqeraPath) + return false + final that = (SeqeraPath) other + if (this.isAbsolute() != that.isAbsolute()) + return false + final thisNames = this.nameComponents() + final thatNames = that.nameComponents() + if (thatNames.size() > thisNames.size()) + return false + for (int i = 0; i < thatNames.size(); i++) { + if (thisNames[i] != thatNames[i]) + return false + } + return true + } + + @Override + boolean startsWith(String other) { + if (!other) return false + try { + final Path p = SeqeraPath.isSeqeraUri(other) ? new SeqeraPath(fs, other) : new SeqeraPath(other) + return startsWith(p) + } catch (Exception ignored) { + return false + } + } + + @Override + boolean endsWith(Path other) { + if (other !instanceof SeqeraPath) + return false + final that = (SeqeraPath) other + if (that.isAbsolute()) + return this.equals(that) + final thisNames = this.nameComponents() + final thatNames = that.nameComponents() + if (thatNames.isEmpty() || thatNames.size() > thisNames.size()) + return false + final offset = thisNames.size() - thatNames.size() + for (int i = 0; i < thatNames.size(); i++) { + if (thisNames[offset + i] != thatNames[i]) + return false + } + return true + } + + @Override + boolean endsWith(String other) { + if (!other) return false + try { + final Path p = SeqeraPath.isSeqeraUri(other) ? new SeqeraPath(fs, other) : new SeqeraPath(other) + return endsWith(p) + } catch (Exception ignored) { + return false + } + } + + @Override + Path normalize() { this } + + @Override + Path resolve(Path other) { + if (other instanceof SeqeraPath) { + final that = (SeqeraPath) other + if (that.isAbsolute()) return that + // Relative SeqeraPath: resolve each segment of relPath against this + return resolve(that.relPath) + } + return resolve(other.toString()) + } + + @Override + Path resolve(String segment) { + if (!segment) return this + // Absolute seqera:// URI — parse and return directly + if (segment.startsWith(PROTOCOL)) + return new SeqeraPath(fs, segment) + // Strip a single leading slash + final stripped = segment.startsWith(SEPARATOR) ? segment.substring(1) : segment + if (!stripped) return this + // Multi-segment: split and resolve one segment at a time + final segs = stripped.split(SEPARATOR, -1).findAll { String s -> s } as List + SeqeraPath result = this + for (String seg : segs) { + result = result.resolveOne(seg) + } + return result + } + + /** Resolve a single (non-empty, slash-free) segment against this path. */ + private SeqeraPath resolveOne(String seg) { + final d = depth() + if (d == 0) return new SeqeraPath(fs, seg, null, null, null, null) + if (d == 1) return new SeqeraPath(fs, org, seg, null, null, null) + if (d == 2) return new SeqeraPath(fs, org, workspace, seg, null, null) + if (d == 3) { + final atIdx = seg.lastIndexOf('@') + if (atIdx > 0) + return new SeqeraPath(fs, org, workspace, resourceType, seg.substring(0, atIdx), seg.substring(atIdx + 1)) + return new SeqeraPath(fs, org, workspace, resourceType, seg, null) + } + throw new IllegalStateException("Cannot resolve a path segment on a depth-4 path: $this") + } + + @Override + Path resolveSibling(Path other) { + final parent = getParent() + return parent != null ? parent.resolve(other) : other + } + + @Override + Path resolveSibling(String other) { + final parent = getParent() + return parent != null ? parent.resolve(other) : new SeqeraPath(fs, other) + } + + @Override + Path relativize(Path other) { + if (other !instanceof SeqeraPath) + throw new ProviderMismatchException() + final that = (SeqeraPath) other + if (!this.isAbsolute() || !that.isAbsolute()) + throw new IllegalArgumentException("Both paths must be absolute to relativize: ${this} vs ${other}") + final thisNames = this.nameComponents() + final thatNames = that.nameComponents() + // Find common prefix length + int common = 0 + while (common < thisNames.size() && common < thatNames.size() + && thisNames[common] == thatNames[common]) + common++ + // Build ".." for each remaining segment in this, then append remaining segments of other + final parts = new ArrayList() + for (int i = common; i < thisNames.size(); i++) + parts.add('..') + for (int i = common; i < thatNames.size(); i++) + parts.add(thatNames[i]) + return new SeqeraPath(parts.join(SEPARATOR)) + } + + @Override + URI toUri() { + // Build path component for depth >= 2 + String uriPath = null + if (workspace) { + final segments = [workspace] + if (resourceType) segments.add(resourceType) + if (datasetName) segments.add(version ? "${datasetName}@${version}" as String : datasetName) + uriPath = '/' + segments.join('/') + } + // new URI(scheme, authority, path, query, fragment) avoids URI.create() pitfalls for edge cases + return new URI(SCHEME, org ?: '', uriPath, null, null) + } + + @Override + String toString() { + if (!isAbsolute()) return relPath + // Return the canonical human-readable representation + final d = depth() + if (d == 0) return "${SCHEME}://" + return toUri().toString() + } + + @Override + Path toAbsolutePath() { + if (!isAbsolute()) + throw new IllegalStateException("Cannot convert relative SeqeraPath to absolute — no default directory context") + return this + } + + @Override + Path toRealPath(LinkOption... options) { this } + + @Override + File toFile() { + throw new UnsupportedOperationException("toFile() not supported for seqera:// paths") + } + + @Override + WatchKey register(WatchService watcher, WatchEvent.Kind[] events, WatchEvent.Modifier... modifiers) { + throw new UnsupportedOperationException("WatchService not supported by seqera:// paths") + } + + @Override + WatchKey register(WatchService watcher, WatchEvent.Kind... events) { + throw new UnsupportedOperationException("WatchService not supported by seqera:// paths") + } + + @Override + Iterator iterator() { + final d = depth() + final List parts = new ArrayList<>(d) + for (int i = 0; i < d; i++) { + parts.add(getName(i)) + } + return parts.iterator() + } + + @Override + int compareTo(Path other) { + return toString().compareTo(other.toString()) + } + + @Override + boolean equals(Object obj) { + if (obj == this) return true + if (obj !instanceof SeqeraPath) return false + return toString() == obj.toString() + } + + @Override + int hashCode() { toString().hashCode() } + + static URI asUri(String path) { + if( !path ) + throw new IllegalArgumentException("Missing 'path' argument") + if( !path.startsWith(PROTOCOL) ) + throw new IllegalArgumentException("Invalid Seqera file system path URI - it must start with '${PROTOCOL}' prefix - offending value: $path") + if( path.startsWith(PROTOCOL + SEPARATOR) && path.length() > PROTOCOL.length() + 1 ) + throw new IllegalArgumentException("Invalid Seqera file system path URI - make sure the scheme prefix does not contain more than two slash characters or a query in the root '/' - offending value: $path") + + //URI strings like seqera://./something are converted to seqera://something + if( path.startsWith(PROTOCOL + './') ) { + path = PROTOCOL + path.substring(PROTOCOL.length() + 2) + } + + if( path == PROTOCOL || path == PROTOCOL + '.') //Empty path case + return new URI(PROTOCOL + '/') + return new URI(path) + } + + static boolean isSeqeraUri(String path) { + return path && path.startsWith(PROTOCOL) + } +} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraPathFactory.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraPathFactory.groovy new file mode 100644 index 0000000000..487a2996bd --- /dev/null +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/fs/SeqeraPathFactory.groovy @@ -0,0 +1,69 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.fs + +import nextflow.file.FileHelper + +import java.nio.file.Path + +import groovy.transform.CompileStatic +import groovy.util.logging.Slf4j +import nextflow.file.FileSystemPathFactory + +/** + * PF4J extension that registers the {@code seqera://} URI scheme with Nextflow's file helper, + * allowing pipeline scripts to use {@code file('seqera://org/ws/datasets/name')} transparently. + * + * Registered as a PF4J extension point via {@code extensionPoints} in {@code build.gradle}. + * + * @author Seqera Labs + */ +@Slf4j +@CompileStatic +class SeqeraPathFactory extends FileSystemPathFactory { + + + @Override + Path parseUri(String str) { + return str?.startsWith(SeqeraPath.PROTOCOL) ? create(str) : null + } + + @Override + String toUriString(Path path) { + if (path instanceof SeqeraPath) + return path.toUri().toString() + return null + } + + @Override + String getBashLib(Path target) { + // No bash-level staging for seqera:// — handled via NIO newInputStream/copy + return null + } + + @Override + String getUploadCmd(String source, Path target) { + // No bash upload command — seqera:// filesystem is read-only + return null + } + + static SeqeraPath create(String path) { + final uri = SeqeraPath.asUri(path) + return (SeqeraPath) FileHelper.getOrCreateFileSystemFor(uri).provider().getPath(uri) + } + +} diff --git a/plugins/nf-tower/src/main/io/seqera/tower/plugin/launch/LaunchCommandImpl.groovy b/plugins/nf-tower/src/main/io/seqera/tower/plugin/launch/LaunchCommandImpl.groovy index 8921353a3d..09f9932ffc 100644 --- a/plugins/nf-tower/src/main/io/seqera/tower/plugin/launch/LaunchCommandImpl.groovy +++ b/plugins/nf-tower/src/main/io/seqera/tower/plugin/launch/LaunchCommandImpl.groovy @@ -25,6 +25,7 @@ import groovy.util.logging.Slf4j import io.seqera.http.HxClient import io.seqera.tower.plugin.BaseCommandImpl import io.seqera.tower.plugin.TowerClient +import io.seqera.tower.plugin.exception.ForbiddenException import nextflow.BuildInfo import nextflow.cli.CmdLaunch import nextflow.util.ColorUtil @@ -154,14 +155,14 @@ class LaunchCommandImpl extends BaseCommandImpl implements CmdLaunch.LaunchComma // Resolve workspace final workspaceId = resolveWorkspaceId(config, options.workspace, accessToken, apiEndpoint) - final httpClient = createHttpClient(accessToken) - final userInfo = commonApi.getUserInfo(httpClient, apiEndpoint) + final httpClient = createTowerClient(apiEndpoint, accessToken) + final userInfo = httpClient.getUserInfo() final userName = userInfo.name as String final userId = userInfo.id as String String orgName = null String workspaceName = null if (workspaceId) { - final wsDetails = commonApi.getUserWorkspaceDetails(httpClient, userId, apiEndpoint, workspaceId.toString()) + final wsDetails = httpClient.getUserWorkspaceDetails(userId, workspaceId.toString()) orgName = wsDetails?.orgName as String workspaceName = wsDetails?.workspaceName as String log.debug "Using workspace '${workspaceName}' (ID: ${workspaceId})" @@ -199,7 +200,7 @@ class LaunchCommandImpl extends BaseCommandImpl implements CmdLaunch.LaunchComma // Submit to API final queryParams = context.workspaceId ? [workspaceId: context.workspaceId.toString()] : [:] - final response = apiPost('/workflow/launch', launchRequest, queryParams, context.accessToken, context.apiEndpoint) + final response = postLaunch( launchRequest, queryParams, context.accessToken, context.apiEndpoint) // Fetch workflow details for accurate launch info final workflowDetails = fetchWorkflowDetails(response.workflowId as String, context.workspaceId, @@ -242,7 +243,7 @@ class LaunchCommandImpl extends BaseCommandImpl implements CmdLaunch.LaunchComma log.debug "Fetching workflow details for ID: ${workflowId}" final queryParams = workspaceId ? [workspaceId: workspaceId.toString()] : [:] - return commonApi.apiGet( createHttpClient(accessToken), apiEndpoint, "/workflow/${workflowId}", queryParams) + return createTowerClient(apiEndpoint, accessToken).apiGet("/workflow/${workflowId}", queryParams) } /** @@ -290,13 +291,13 @@ class LaunchCommandImpl extends BaseCommandImpl implements CmdLaunch.LaunchComma * Resolve compute environment by flag name config computeEnvId or get primary */ protected Map resolveComputeEnvironment(Map config, String computeEnvName, Long workspaceId, String accessToken, String apiEndpoint) { - final client = createHttpClient(accessToken) + final client = createTowerClient(apiEndpoint, accessToken) Map computeEnvInfo = null if (!computeEnvName && config?.get('tower.computeEnvId')) { - computeEnvInfo = getComputeEnvironment(client, apiEndpoint, config['tower.computeEnvId'] as String, workspaceId?.toString()) + computeEnvInfo = getComputeEnvironment(client, config['tower.computeEnvId'] as String, workspaceId?.toString()) } else { log.debug "Looking up compute environment: ${computeEnvName ?: '(primary)'}" - computeEnvInfo = findComputeEnv(client, computeEnvName, workspaceId, apiEndpoint) + computeEnvInfo = findComputeEnv(client, computeEnvName, workspaceId) } if (!computeEnvInfo) { if (computeEnvName) { @@ -508,12 +509,12 @@ class LaunchCommandImpl extends BaseCommandImpl implements CmdLaunch.LaunchComma try { spinner.start() - final client = createHttpClient(accessToken) + final client = createTowerClient(apiEndpoint, accessToken) while (!shouldExit.get() && !Thread.currentThread().isInterrupted()) { try { // Fetch workflow status and logs - final status = fetchWorkflowStatus(client, workflowId, queryParams, apiEndpoint) - final logEntries = fetchWorkflowLogs(client, workflowId, queryParams, apiEndpoint) + final status = fetchWorkflowStatus(client, workflowId, queryParams) + final logEntries = fetchWorkflowLogs(client, workflowId, queryParams) // Update spinner with status if it changed if (status && status != lastStatus) { @@ -644,8 +645,8 @@ class LaunchCommandImpl extends BaseCommandImpl implements CmdLaunch.LaunchComma /** * Fetch workflow status from API */ - private String fetchWorkflowStatus(HxClient client, String workflowId, Map queryParams, String apiEndpoint) { - final workflow = commonApi.getWorkflowDetails(client, apiEndpoint, workflowId, queryParams) + private String fetchWorkflowStatus(TowerClient client, String workflowId, Map queryParams) { + final workflow = client.getWorkflowDetails(workflowId, queryParams) final status = workflow?.status as String log.debug "Workflow status: ${status}" return status @@ -654,8 +655,8 @@ class LaunchCommandImpl extends BaseCommandImpl implements CmdLaunch.LaunchComma /** * Fetch workflow logs from API */ - private List fetchWorkflowLogs(HxClient client, String workflowId, Map queryParams, String apiEndpoint) { - final logResponse = commonApi.apiGet(client, apiEndpoint,"/workflow/${workflowId}/log", queryParams) + private List fetchWorkflowLogs(TowerClient client, String workflowId, Map queryParams) { + final logResponse = client.apiGet( "/workflow/${workflowId}/log", queryParams) final logData = logResponse.log as Map return logData?.entries as List ?: [] } @@ -950,30 +951,15 @@ class LaunchCommandImpl extends BaseCommandImpl implements CmdLaunch.LaunchComma } } - // ===== API Helper Methods ===== - - protected Map apiPost(String path, Map body, Map queryParams = [:], String accessToken, String apiEndpoint) { - final url = commonApi.buildUrl(apiEndpoint, path, queryParams) - log.debug "Platform API - POST ${url}" - final requestBody = new JsonBuilder(body).toString() - final client = createHttpClient(accessToken) - final HttpRequest request = HttpRequest.newBuilder() - .uri(URI.create(url)) - .header('Content-Type', 'application/json') - .POST(HttpRequest.BodyPublishers.ofString(requestBody)) - .build() - - final HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()) - - if (response.statusCode() != 200) { - if (response.statusCode() == 403) { - throw new AbortOperationException("ERROR: Unable to launch workflow.\nCheck your credentials with 'nextflow auth status' and your user role in the workspace (required: 'maintain' or higher).") - } - final error = response.body() ?: "HTTP ${response.statusCode()}" - throw new RuntimeException("Failed to launch workflow: ${error}") + protected Map postLaunch( Map body, Map queryParams = [:], String accessToken, String apiEndpoint) { + try { + final client = createTowerClient(apiEndpoint, accessToken) + return client.apiPost('/workflow/launch', queryParams, body) + }catch (ForbiddenException e) { + throw new AbortOperationException("ERROR: Unable to launch workflow.\nCheck your credentials with 'nextflow auth status' and your user role in the workspace (required: 'maintain' or higher).") + }catch (Exception e) { + throw new RuntimeException("Failed to launch workflow: ${e.message}", e) } - - return new JsonSlurper().parseText(response.body()) as Map } // ===== Workspace & User Helper Methods ===== @@ -987,10 +973,10 @@ class LaunchCommandImpl extends BaseCommandImpl implements CmdLaunch.LaunchComma // If workspace name provided, look it up if (workspaceName) { - final httpClient = createHttpClient(accessToken) - final userInfo = commonApi.getUserInfo(httpClient, apiEndpoint) as Map + final httpClient = createTowerClient(apiEndpoint, accessToken) + final userInfo = httpClient.getUserInfo() as Map final userId = userInfo.id as String - final workspaces = listUserWorkspaces(httpClient, apiEndpoint, userId) + final workspaces = listUserWorkspaces(httpClient, userId) final matchingWorkspace = workspaces.find { workspace -> final ws = workspace as Map @@ -1008,8 +994,8 @@ class LaunchCommandImpl extends BaseCommandImpl implements CmdLaunch.LaunchComma return null } - protected Map findComputeEnv(HxClient client, String computeEnvName, Long workspaceId, String apiEndpoint) { - final computeEnvs = listComputeEnvironments(client, apiEndpoint, workspaceId ? workspaceId.toString() : null) + protected Map findComputeEnv(TowerClient client, String computeEnvName, Long workspaceId) { + final computeEnvs = listComputeEnvironments(client, workspaceId ? workspaceId.toString() : null) log.debug "Looking for ${computeEnvName ? "compute environment with name: ${computeEnvName}" : "primary compute environment"} ${workspaceId ? "in workspace ID ${workspaceId}" : "in personal workspace"}" diff --git a/plugins/nf-tower/src/resources/META-INF/services/java.nio.file.spi.FileSystemProvider b/plugins/nf-tower/src/resources/META-INF/services/java.nio.file.spi.FileSystemProvider new file mode 100644 index 0000000000..d17b471f4b --- /dev/null +++ b/plugins/nf-tower/src/resources/META-INF/services/java.nio.file.spi.FileSystemProvider @@ -0,0 +1 @@ +io.seqera.tower.plugin.fs.SeqeraFileSystemProvider diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerClientTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerClientTest.groovy index 6fb91f7b42..8c3d13f84e 100644 --- a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerClientTest.groovy +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerClientTest.groovy @@ -17,29 +17,13 @@ package io.seqera.tower.plugin import java.net.http.HttpResponse -import java.nio.file.Files import java.time.Instant -import java.time.OffsetDateTime -import java.time.ZoneId import com.github.tomakehurst.wiremock.WireMockServer import com.github.tomakehurst.wiremock.client.WireMock import io.seqera.http.HxClient -import nextflow.Session -import nextflow.SysEnv -import nextflow.cloud.types.CloudMachineInfo -import nextflow.cloud.types.PriceModel -import nextflow.container.DockerConfig -import nextflow.container.resolver.ContainerMeta import nextflow.exception.AbortOperationException -import nextflow.script.PlatformMetadata -import nextflow.script.ScriptBinding -import nextflow.script.WorkflowMetadata -import nextflow.trace.TraceRecord -import nextflow.trace.WorkflowStats -import nextflow.trace.WorkflowStatsObserver import nextflow.util.Duration -import nextflow.util.ProcessHelper import spock.lang.Specification /** * @@ -70,57 +54,6 @@ class TowerClientTest extends Specification { thrown(Exception) } - def 'should create message map' () { - given: - def session = Mock(Session) - def params = new ScriptBinding.ParamsMap(x: "hello") - def meta = Mock(WorkflowMetadata) - - def tower = Spy(TowerClient) - tower.@runName = session.runName - tower.@workflowId = '12ef' - - when: - def map = tower.makeCompleteReq(session) - then: - 1 * session.getWorkflowMetadata() >> meta - 1 * session.getParams() >> params - 1 * meta.toMap() >> [foo:1, bar:2, container: [p1: 'c1', p2: 'c2']] - 1 * tower.getMetricsList() >> [[process:'foo', cpu: [min: 1, max:5], time: [min: 6, max: 9]]] - 1 * tower.getWorkflowProgress(false) >> new WorkflowProgress() - 1 * tower.getOutFile() >> 'bar.out' - 1 * tower.getLogFile() >> 'foo.out' - 1 * tower.getOperationId() >> 'op-12345' - then: - map.workflow.foo == 1 - map.workflow.bar == 2 - map.workflow.id == '12ef' - map.workflow.params == [x: 'hello'] - map.workflow.container == null - map.metrics == [[process:'foo', cpu: [min: 1, max:5], time: [min: 6, max: 9]]] - map.progress == new WorkflowProgress() - and: - aroundNow(map.instant) - and: - map.workflow.outFile == 'bar.out' - map.workflow.logFile == 'foo.out' - map.workflow.operationId == 'op-12345' - } - - def 'should capitalise underscores' () { - given: - def tower = new TowerClient() - - expect: - tower.underscoreToCamelCase(STR) == EXPECTED - where: - STR | EXPECTED - 'abc' | 'abc' - 'a_b_c' | 'aBC' - 'foo__bar' | 'fooBar' - } - - def 'should validate URL' () { given: def observer = new TowerClient() @@ -156,324 +89,37 @@ class TowerClientTest extends Specification { } def 'should get access token' () { - given: - def session = Mock(Session) - when: def config = new TowerConfig([accessToken: 'abc'], [TOWER_ACCESS_TOKEN: 'xyz']) - def observer = new TowerClient(session, config) + def client = new TowerClient(config) then: // the token in the config overrides the one in the env - observer.getAccessToken() == 'abc' + client.getAccessToken() == 'abc' when: config = new TowerConfig([accessToken: 'abc'], [TOWER_ACCESS_TOKEN: 'xyz', TOWER_WORKFLOW_ID: '111222333']) - observer = new TowerClient(session, config) + client = new TowerClient(config) then: // the token from the env is taken because is a tower launch aka TOWER_WORKFLOW_ID is set - observer.getAccessToken() == 'xyz' + client.getAccessToken() == 'xyz' when: config = new TowerConfig([:], [TOWER_ACCESS_TOKEN: 'xyz']) - observer = new TowerClient(session, config) + client = new TowerClient(config) then: - observer.getAccessToken() == 'xyz' + client.getAccessToken() == 'xyz' when: - config = new TowerConfig([:], [:]) - observer = new TowerClient(session, config) - observer.getAccessToken() + def c = new TowerClient() + c.getAccessToken() then: thrown(AbortOperationException) } - def 'should post task records' () { - given: - def URL = 'http://foo.com' - def PROGRESS = Mock(WorkflowProgress) { getRunning()>>1; getSucceeded()>>2; getFailed()>>3 } - def client = Mock(HxClient) - def observer = Spy(TowerClient) - observer.@httpClient = client - observer.@workflowId = 'xyz-123' - - def nowTs = System.currentTimeMillis() - def submitTs = nowTs-2000 - def startTs = nowTs-1000 - - def trace = new TraceRecord([ - taskId: 10, - process: 'foo', - workdir: "/work/dir", - cpus: 1, - submit: submitTs, - start: startTs, - complete: nowTs ]) - trace.executorName= 'batch' - trace.machineInfo = new CloudMachineInfo('m4.large', 'eu-west-1b', PriceModel.spot) - trace.containerMeta = new ContainerMeta(requestId: '12345', sourceImage: 'ubuntu:latest', targetImage: 'wave.io/12345/ubuntu:latest') - when: - def req = observer.makeTasksReq([trace]) - then: - observer.getWorkflowProgress(true) >> PROGRESS - and: - req.tasks[0].taskId == 10 - req.tasks[0].process == 'foo' - req.tasks[0].workdir == "/work/dir" - req.tasks[0].cpus == 1 - req.tasks[0].submit == OffsetDateTime.ofInstant(Instant.ofEpochMilli(submitTs), ZoneId.systemDefault()) - req.tasks[0].start == OffsetDateTime.ofInstant(Instant.ofEpochMilli(startTs), ZoneId.systemDefault()) - req.tasks[0].executor == 'batch' - req.tasks[0].machineType == 'm4.large' - req.tasks[0].cloudZone == 'eu-west-1b' - req.tasks[0].priceModel == 'spot' - and: - req.progress.running == 1 - req.progress.succeeded == 2 - req.progress.failed == 3 - and: - req.containers[0].requestId == '12345' - req.containers[0].sourceImage == 'ubuntu:latest' - req.containers[0].targetImage == 'wave.io/12345/ubuntu:latest' - and: - aroundNow(req.instant) - - when: - observer.sendHttpMessage(URL, req) - then: - 1 * client.sendAsString(_) >> Mock(HttpResponse) - - } - - - static now_millis = System.currentTimeMillis() - static now_instant = OffsetDateTime.ofInstant(Instant.ofEpochMilli(now_millis), ZoneId.systemDefault()) - - def 'should fix field types' () { - - expect: - TowerClient.fixTaskField(FIELD,VALUE) == EXPECTED - - where: - FIELD | VALUE | EXPECTED - 'foo' | 'hola' | 'hola' - 'submit' | now_millis | now_instant - 'start' | now_millis | now_instant - 'complete' | now_millis | now_instant - 'complete' | 0 | null - } - - - def 'should create workflow json' () { - - given: - def sessionId = UUID.randomUUID() - def dir = Files.createTempDirectory('test') - def http = Mock(HxClient) - TowerClient client = Spy(new TowerClient([httpClient: http, env: ENV])) - and: - client.getOperationId() >> 'op-112233' - client.getLogFile() >> 'log.file' - client.getOutFile() >> 'out.file' - - and: - def session = Mock(Session) - session.getUniqueId() >> sessionId - session.getRunName() >> 'foo' - session.config >> [:] - session.containerConfig >> new DockerConfig([:]) - session.getParams() >> new ScriptBinding.ParamsMap([foo:'Hello', bar:'World']) - - def meta = new WorkflowMetadata( - session: session, - projectName: 'the-project-name', - repository: 'git://repo.com/foo' ) - session.getWorkflowMetadata() >> meta - session.getStatsObserver() >> Mock(WorkflowStatsObserver) { getStats() >> new WorkflowStats() } - - when: - def req1 = client.makeCreateReq(session) - then: - req1.sessionId == sessionId.toString() - req1.runName == 'foo' - req1.projectName == 'the-project-name' - req1.repository == 'git://repo.com/foo' - req1.workflowId == WORKFLOW_ID - and: - aroundNow(req1.instant) - - when: - def req = client.makeBeginReq(session) - then: - client.getWorkflowId() >> '12345' - and: - req.workflow.id == '12345' - req.workflow.params == [foo:'Hello', bar:'World'] - req.workflow.outFile == 'out.file' - req.workflow.logFile == 'log.file' - req.workflow.operationId == 'op-112233' - and: - req.towerLaunch == TOWER_LAUNCH - and: - aroundNow(req.instant) - - cleanup: - dir?.deleteDir() - - where: - ENV | WORKFLOW_ID | TOWER_LAUNCH - [:] | null | false - [TOWER_WORKFLOW_ID: '1234'] | '1234' | true - - } - - def 'should convert map' () { - given: - def tower = new TowerClient() - - expect: - tower.mapToString(null) == null - tower.mapToString('ciao') == 'ciao' - tower.mapToString([:]) == null - tower.mapToString([p:'foo', q:'bar']) == null - } - - - def 'should load schema col len' () { - given: - def tower = new TowerClient() - - when: - def schema = tower.loadSchema() - then: - schema.get('workflow.start') == null - schema.get('workflow.profile') == 100 - schema.get('workflow.projectDir') == 255 - } - - def 'should create init request' () { - given: - def uuid = UUID.randomUUID() - def client = new TowerClient(env: [TOWER_WORKFLOW_ID: 'x123']) - def meta = Mock(WorkflowMetadata) { - getProjectName() >> 'the-project-name' - getRepository() >> 'git://repo.com/foo' - } - def session = Mock(Session) { - getUniqueId() >> uuid - getRunName() >> 'foo_bar' - getWorkflowMetadata() >> meta - } - - when: - def req = client.makeCreateReq(session) - then: - req.sessionId == uuid.toString() - req.runName == 'foo_bar' - req.projectName == 'the-project-name' - req.repository == 'git://repo.com/foo' - req.workflowId == 'x123' - and: - aroundNow(req.instant) - - and: - client.towerLaunch - } - - def 'should post create request' () { - given: - def uuid = UUID.randomUUID() - def platform = new PlatformMetadata() - def meta = Mock(WorkflowMetadata) { - getProjectName() >> 'the-project-name' - getRepository() >> 'git://repo.com/foo' - getPlatform() >> platform - } - def session = Mock(Session) { - getUniqueId() >> uuid - getRunName() >> 'foo_bar' - getWorkflowMetadata() >> meta - } - def config = new TowerConfig([:], [:]) - - def client = Spy(new TowerClient(session, config)) - - when: - client.onFlowCreate(session) - then: - 0 * client.applyPlatformMetadata(_) - 1 * client.getAccessToken() >> 'secret' - 1 * client.makeCreateReq(session) >> [runName: 'foo'] - 1 * client.sendHttpMessage('https://api.cloud.seqera.io/trace/create', [runName: 'foo'], 'POST') >> new TowerClient.Response(200, '{"workflowId":"xyz123","watchUrl":"https://cloud.seqera.io/watch/xyz123"}') - and: - client.runName == 'foo_bar' - client.runId == uuid.toString() - and: - client.workflowId == 'xyz123' - client.@watchUrl == 'https://cloud.seqera.io/watch/xyz123' - !client.towerLaunch - and: - platform.workflowId == 'xyz123' - platform.workflowUrl == 'https://cloud.seqera.io/watch/xyz123' - - } - - def 'should set workflowUrl on platform metadata during onFlowBegin' () { - given: - def platform = new PlatformMetadata() - def meta = Mock(WorkflowMetadata) { - getPlatform() >> platform - } - def session = Mock(Session) { - getWorkflowMetadata() >> meta - } - def config = new TowerConfig([:], [:]) - def client = Spy(new TowerClient(session, config)) - client.@workflowId = 'abc123' - - when: - client.onFlowBegin() - then: - 1 * client.makeBeginReq(session) >> [foo: 'bar'] - 1 * client.sendHttpMessage(_, [foo: 'bar'], 'PUT') >> new TowerClient.Response(200, '{"watchUrl":"https://cloud.seqera.io/watch/abc123"}') - and: - client.@watchUrl == 'https://cloud.seqera.io/watch/abc123' - platform.workflowUrl == 'https://cloud.seqera.io/watch/abc123' - } - - def 'should get trace endpoint' () { - given: - def config = new TowerConfig([:], [:]) - def tower = new TowerClient(Mock(Session), config) - tower.workflowId = '12345' - - expect: - tower.getUrlTraceCreate() == 'https://api.cloud.seqera.io/trace/create' - tower.getUrlTraceBegin() == 'https://api.cloud.seqera.io/trace/12345/begin' - tower.getUrlTraceProgress() == 'https://api.cloud.seqera.io/trace/12345/progress' - tower.getUrlTraceHeartbeat() == 'https://api.cloud.seqera.io/trace/12345/heartbeat' - tower.getUrlTraceComplete() == 'https://api.cloud.seqera.io/trace/12345/complete' - } - - def 'should get trace endpoint with workspace' () { - given: - def config = new TowerConfig([workspaceId: '300'], [:]) - def tower = new TowerClient(Mock(Session), config) - tower.workflowId = '12345' - - expect: - tower.getUrlTraceCreate() == 'https://api.cloud.seqera.io/trace/create?workspaceId=300' - tower.getUrlTraceBegin() == 'https://api.cloud.seqera.io/trace/12345/begin?workspaceId=300' - tower.getUrlTraceProgress() == 'https://api.cloud.seqera.io/trace/12345/progress?workspaceId=300' - tower.getUrlTraceHeartbeat() == 'https://api.cloud.seqera.io/trace/12345/heartbeat?workspaceId=300' - tower.getUrlTraceComplete() == 'https://api.cloud.seqera.io/trace/12345/complete?workspaceId=300' - } - def 'should set the auth token' () { given: def http = Mock(HxClient.Builder) - def session = Mock(Session) - def config = new TowerConfig([:], [:]) - def client = Spy(new TowerClient(session, config)) + def client = new TowerClient() and: def SIMPLE = '4ffbf1009ebabea77db3d72efefa836dfbb71271' def BEARER = 'eyJ0aWQiOiA1fS5jZmM1YjVhOThjZjM2MTk1NjBjZWU1YmMwODUxYzA1ZjkzMDdmN2Iz' @@ -496,81 +142,42 @@ class TowerClientTest extends Specification { 1 * http.refreshTokenUrl(_) >> http } - def 'should fetch workflow meta' () { + def 'should get trace endpoint' () { given: - def client = Spy(new TowerClient(env: ENV)) + def client = new TowerClient() + client.@endpoint = TowerClient.DEF_ENDPOINT_URL expect: - client.getOperationId() == OP_ID - client.getLogFile() == LOG_FILE - client.getOutFile() == OUT_FILE - - where: - OP_ID | OUT_FILE | LOG_FILE | ENV - null | null | null | [:] - "local-platform::${ProcessHelper.selfPid()}" | null | null | [TOWER_ALLOW_NEXTFLOW_LOGS:'true'] - 'aws-batch::1234z' | 'xyz.out' | 'hola.log' | [TOWER_ALLOW_NEXTFLOW_LOGS:'true', AWS_BATCH_JOB_ID: '1234z', NXF_OUT_FILE: 'xyz.out', NXF_LOG_FILE: 'hola.log'] + client.getUrlTraceCreate(null) == 'https://api.cloud.seqera.io/trace/create' + client.getUrlTraceBegin(null, '12345') == 'https://api.cloud.seqera.io/trace/12345/begin' + client.getUrlTraceProgress(null, '12345') == 'https://api.cloud.seqera.io/trace/12345/progress' + client.getUrlTraceHeartbeat(null, '12345') == 'https://api.cloud.seqera.io/trace/12345/heartbeat' + client.getUrlTraceComplete(null, '12345') == 'https://api.cloud.seqera.io/trace/12345/complete' } - def 'should deduplicate containers' () { + def 'should get trace endpoint with workspace' () { given: - def client = Spy(new TowerClient()) - and: - def c1 = new ContainerMeta(requestId: '12345', sourceImage: 'ubuntu:latest', targetImage: 'wave.io/12345/ubuntu:latest') - def c2 = new ContainerMeta(requestId: '54321', sourceImage: 'ubuntu:latest', targetImage: 'wave.io/54321/ubuntu:latest') - and: - def trace1 = new TraceRecord( - taskId: 1, - process: 'foo', - workdir: "/work/dir", - cpus: 1, - submit: System.currentTimeMillis(), - start: System.currentTimeMillis(), - complete: System.currentTimeMillis()) - trace1.containerMeta = c1 - and: - def trace2 = new TraceRecord( - taskId: 2, - process: 'foo', - workdir: "/work/dir", - cpus: 1, - submit: System.currentTimeMillis(), - start: System.currentTimeMillis(), - complete: System.currentTimeMillis()) - trace2.containerMeta = c2 - and: - def trace3 = new TraceRecord( - taskId: 3, - process: 'foo', - workdir: "/work/dir", - cpus: 1, - submit: System.currentTimeMillis(), - start: System.currentTimeMillis(), - complete: System.currentTimeMillis()) - trace3.containerMeta = c2 + def client = new TowerClient() + client.@endpoint = TowerClient.DEF_ENDPOINT_URL expect: - client.getNewContainers([trace1]) == [c1] - and: - client.getNewContainers([trace1]) == [] - and: - client.getNewContainers([trace1, trace2, trace3]) == [c2] + client.getUrlTraceCreate('300') == 'https://api.cloud.seqera.io/trace/create?workspaceId=300' + client.getUrlTraceBegin('300', '12345') == 'https://api.cloud.seqera.io/trace/12345/begin?workspaceId=300' + client.getUrlTraceProgress('300', '12345') == 'https://api.cloud.seqera.io/trace/12345/progress?workspaceId=300' + client.getUrlTraceHeartbeat('300', '12345') == 'https://api.cloud.seqera.io/trace/12345/heartbeat?workspaceId=300' + client.getUrlTraceComplete('300', '12345') == 'https://api.cloud.seqera.io/trace/12345/complete?workspaceId=300' } - def 'should not send complete request when onFlowBegin was not invoked' () { + def 'should load schema col len' () { given: - def client = Spy(new TowerClient()) - client.@workflowId = 'xyz-123' - client.@sender = null - client.@reports = Mock(TowerReports) + def tower = new TowerClient() when: - client.onFlowComplete() - + def schema = tower.loadSchema() then: - 1 * client.@reports.publishRuntimeReports() - 1 * client.@reports.flowComplete() - 0 * client.sendHttpMessage(_, _, _) + schema.get('workflow.start') == null + schema.get('workflow.profile') == 100 + schema.get('workflow.projectDir') == 255 } def 'should handle HTTP request with content'() { @@ -585,157 +192,19 @@ class TowerClientTest extends Specification { request.uri().toString() == 'http://example.com/test' } - def 'should apply platform metadata from trace create response'() { - given: - def metadata = new WorkflowMetadata() - def session = Mock(Session) { - getWorkflowMetadata() >> metadata - } - def config = new TowerConfig([accessToken: 'token-1234', workspaceId: '1234'], SysEnv.get()) - def towerClient = new TowerClient(session, config) - - def responseMetadata = [ - userId: 39, - userName: 'user', - userOrganization: 'ACME Inc.', - workspaceId: 1234, - workspaceName: 'Workspace-Name', - workspaceFullName: 'Full Workspace Name', - orgName: 'ACME Inc.', - computeEnvId: 'ce1234', - computeEnvName: 'ce-test', - computeEnvPlatform: 'aws-batch', - pipelineName: 'test-pipeline', - pipelineId: 'pipe1234', - revision: 'v1.1', - commitId: 'abcd12345' - ] - - when: - towerClient.applyPlatformMetadata(responseMetadata) - - then: - metadata.platform.user.id == '39' - metadata.platform.user.userName == 'user' - metadata.platform.user.organization == 'ACME Inc.' - metadata.platform.workspace.id == '1234' - metadata.platform.workspace.name == 'Workspace-Name' - metadata.platform.workspace.fullName == 'Full Workspace Name' - metadata.platform.workspace.organization == 'ACME Inc.' - metadata.platform.computeEnv.id == 'ce1234' - metadata.platform.computeEnv.name == 'ce-test' - metadata.platform.computeEnv.platform == 'aws-batch' - metadata.platform.pipeline.id == 'pipe1234' - metadata.platform.pipeline.name == 'test-pipeline' - metadata.platform.pipeline.revision == 'v1.1' - metadata.platform.pipeline.commitId == 'abcd12345' - } - - def 'should include numSpotInterruptions in task map'() { + def 'should send http message' () { given: - def client = Spy(new TowerClient()) - client.getWorkflowProgress(true) >> new WorkflowProgress() - - def now = System.currentTimeMillis() - def trace = new TraceRecord([ - taskId: 42, - process: 'foo', - workdir: "/work/dir", - cpus: 1, - submit: now-2000, - start: now-1000, - complete: now - ]) - trace.setNumSpotInterruptions(3) - - when: - def req = client.makeTasksReq([trace]) - - then: - req.tasks.size() == 1 - req.tasks[0].numSpotInterruptions == 3 - } - - def 'should include logStreamId in task map'() { - given: - def client = Spy(new TowerClient()) - client.getWorkflowProgress(true) >> new WorkflowProgress() - - def now = System.currentTimeMillis() - def trace = new TraceRecord([ - taskId: 42, - process: 'foo', - workdir: "/work/dir", - cpus: 1, - submit: now-2000, - start: now-1000, - complete: now - ]) - trace.setLogStreamId('arn:aws:logs:us-east-1:123456789:log-group:/ecs/task:log-stream:abc123') - - when: - def req = client.makeTasksReq([trace]) - - then: - req.tasks.size() == 1 - req.tasks[0].logStreamId == 'arn:aws:logs:us-east-1:123456789:log-group:/ecs/task:log-stream:abc123' - } - - def 'should include resourceAllocation in task map'() { - given: - def client = Spy(new TowerClient()) - client.getWorkflowProgress(true) >> new WorkflowProgress() - - def now = System.currentTimeMillis() - def trace = new TraceRecord([ - taskId: 42, - process: 'foo', - workdir: "/work/dir", - cpus: 1, - submit: now-2000, - start: now-1000, - complete: now, - accelerator: 2, - acceleratorType: 'v100' - ]) - trace.setResourceAllocation([cpuShares: 2048, memoryMiB: 4096, time: '1h']) - - when: - def req = client.makeTasksReq([trace]) - - then: - req.tasks.size() == 1 - req.tasks[0].accelerator == 2 - req.tasks[0].acceleratorType == 'v100' - req.tasks[0].resourceAllocation == [cpuShares: 2048, memoryMiB: 4096, time: '1h'] - } - - def 'should include gpuMetrics in task map'() { - given: - def client = Spy(new TowerClient()) - client.getWorkflowProgress(true) >> new WorkflowProgress() - - def now = System.currentTimeMillis() - def trace = new TraceRecord([ - taskId: 42, - process: 'foo', - workdir: "/work/dir", - cpus: 1, - submit: now-2000, - start: now-1000, - complete: now - ]) - trace.setGpuMetrics([name: 'Tesla T4', mem: 15360, driver: '580.126.09', active_time: 651030, pct: 75, peak: 100]) + def client = Mock(HxClient) + def tower = new TowerClient() + tower.@httpClient = client when: - def req = client.makeTasksReq([trace]) - + def resp = tower.sendHttpMessage('http://foo.com', [foo: 'bar'], 'POST') then: - req.tasks.size() == 1 - req.tasks[0].gpuMetrics.name == 'Tesla T4' - req.tasks[0].gpuMetrics.mem == 15360 - req.tasks[0].gpuMetrics.pct == 75 - req.tasks[0].gpuMetrics.peak == 100 + 1 * client.sendAsString(_) >> Mock(HttpResponse) { statusCode() >> 200; body() >> '{}' } + and: + !resp.error + resp.code == 200 } def 'should return error response on http request timeout' () { @@ -751,15 +220,13 @@ class TowerClientTest extends Specification { ) and: 'a TowerClient whose requests carry a 200ms timeout' - TowerClient client = Spy(new TowerClient().withRequestTimeout(Duration.of('200 ms'))) { - // inject a short per-request timeout so the test doesn't wait 5 seconds - - newHttpClient() >> HxClient.newBuilder() - .connectTimeout(java.time.Duration.ofSeconds(5)) - .build() + TowerConfig config = Mock(TowerConfig) { + getHttpReadTimeout() >> Duration.of('200 ms') + getHttpConnectTimeout() >> Duration.of('5 s') + getEndpoint() >> wireMock.baseUrl() + getAccessToken() >> 'token' } - client.@httpClient = client.newHttpClient() - client.@endpoint = wireMock.baseUrl() + TowerClient client = new TowerClient(config) when: def response = client.sendHttpMessage("${wireMock.baseUrl()}/trace/create", [runName: 'test'], 'POST') @@ -772,4 +239,41 @@ class TowerClientTest extends Specification { wireMock.stop() } + def 'should build URL without query params'() { + given: + def client = new TowerClient() + client.@endpoint = 'https://api.cloud.seqera.io' + + when: + def url = client.buildUrl( '/workflow/launch', [:]) + + then: + url == 'https://api.cloud.seqera.io/workflow/launch' + } + + def 'should build URL with query params'() { + given: + def client = new TowerClient() + client.@endpoint = 'https://api.cloud.seqera.io' + + when: + def url = client.buildUrl( '/workflow/launch', [workspaceId: '12345']) + + then: + url.contains('https://api.cloud.seqera.io/workflow/launch?') + url.contains('workspaceId=12345') + } + + def 'should URL encode query params'() { + given: + def client = new TowerClient() + client.@endpoint = 'https://api.cloud.seqera.io' + + when: + def url = client.buildUrl( '/workflow', [name: 'test workflow']) + + then: + url.contains('name=test+workflow') + } + } diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerCommonApiTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerCommonApiTest.groovy deleted file mode 100644 index eefc4ac972..0000000000 --- a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerCommonApiTest.groovy +++ /dev/null @@ -1,57 +0,0 @@ -/* - * Copyright 2013-2026, Seqera Labs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package io.seqera.tower.plugin - -import spock.lang.Specification - -class TowerCommonApiTest extends Specification{ - - - def 'should build URL without query params'() { - given: - def api = new TowerCommonApi() - - when: - def url = api.buildUrl('https://api.cloud.seqera.io', '/workflow/launch', [:]) - - then: - url == 'https://api.cloud.seqera.io/workflow/launch' - } - - def 'should build URL with query params'() { - given: - def api = new TowerCommonApi() - - when: - def url = api.buildUrl('https://api.cloud.seqera.io', '/workflow/launch', [workspaceId: '12345']) - - then: - url.contains('https://api.cloud.seqera.io/workflow/launch?') - url.contains('workspaceId=12345') - } - - def 'should URL encode query params'() { - given: - def api = new TowerCommonApi() - - when: - def url = api.buildUrl('https://api.cloud.seqera.io', '/workflow', [name: 'test workflow']) - - then: - url.contains('name=test+workflow') - } -} diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerFactoryTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerFactoryTest.groovy index 8aa4e7d53f..7eb0e7f3f1 100644 --- a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerFactoryTest.groovy +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerFactoryTest.groovy @@ -26,21 +26,21 @@ import spock.lang.Unroll */ class TowerFactoryTest extends Specification { - def 'should create an tower observer' () { + def 'should create a tower observer' () { given: def factory = new TowerFactory(env: [TOWER_ACCESS_TOKEN: '123']) when: def session = Mock(Session) { getConfig() >> [tower: [enabled: true]] } - def client = factory.create(session)[0] as TowerClient + def observer = factory.create(session)[0] as TowerObserver then: - client.endpoint == TowerClient.DEF_ENDPOINT_URL + observer.@client.endpoint == TowerClient.DEF_ENDPOINT_URL when: session = Mock(Session) { getConfig() >> [tower: [enabled: true, endpoint:'http://foo.com/api', accessToken: 'xyz']] } - client = factory.create(session)[0] as TowerClient + observer = factory.create(session)[0] as TowerObserver then: - client.endpoint == 'http://foo.com/api' + observer.@client.endpoint == 'http://foo.com/api' } def 'should not create a tower observer' () { @@ -56,16 +56,16 @@ class TowerFactoryTest extends Specification { result == [] } - def 'should create with with workspace id'() { + def 'should create with workspace id'() { // // the workspace id is taken from the env // when: def session = Mock(Session) { getConfig() >> [tower: [enabled: true, accessToken: 'xyz']] } def factory = new TowerFactory(env: [TOWER_WORKSPACE_ID: '100']) - def client = (TowerClient) factory.create(session)[0] + def observer = (TowerObserver) factory.create(session)[0] then: - client.getWorkspaceId() == '100' + observer.getWorkspaceId() == '100' // // the workspace id is taken from the config @@ -73,9 +73,9 @@ class TowerFactoryTest extends Specification { when: session = Mock(Session) { getConfig() >> [tower: [enabled: true, workspaceId: '200', accessToken: 'xyz']] } factory = new TowerFactory(env: [:]) - client = (TowerClient) factory.create(session)[0] + observer = (TowerObserver) factory.create(session)[0] then: - client.getWorkspaceId() == '200' + observer.getWorkspaceId() == '200' // // the workspace id is set both in the config and the env @@ -84,9 +84,9 @@ class TowerFactoryTest extends Specification { when: session = Mock(Session) { getConfig() >> [tower: [enabled: true, workspaceId: '200', accessToken: 'xyz']] } factory = new TowerFactory(env: [TOWER_WORKSPACE_ID: '100']) - client = (TowerClient) factory.create(session)[0] + observer = (TowerObserver) factory.create(session)[0] then: - client.getWorkspaceId() == '200' + observer.getWorkspaceId() == '200' // // when TOWER_WORKFLOW_ID is set is a tower launch @@ -95,20 +95,20 @@ class TowerFactoryTest extends Specification { when: session = Mock(Session) { getConfig() >> [tower: [enabled: true, workspaceId: '200', accessToken: 'xyz']] } factory = new TowerFactory(env: [TOWER_WORKSPACE_ID: '100', TOWER_WORKFLOW_ID: '111222333', TOWER_ACCESS_TOKEN: 'xyz']) - client = (TowerClient) factory.create(session)[0] + observer = (TowerObserver) factory.create(session)[0] then: - client.getWorkspaceId() == '100' + observer.getWorkspaceId() == '100' // // when enabled is false but `TOWER_WORKFLOW_ID` is provided - // then the client should be created + // then the observer should be created // when: session = Mock(Session) { getConfig() >> [tower: [enabled: false]]} factory = new TowerFactory(env: [TOWER_WORKSPACE_ID: '100', TOWER_WORKFLOW_ID: '111222333', TOWER_ACCESS_TOKEN: 'xyz']) - client = (TowerClient) factory.create(session)[0] + observer = (TowerObserver) factory.create(session)[0] then: - client.getWorkspaceId() == '100' + observer.getWorkspaceId() == '100' } @Unroll diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerFusionEnvTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerFusionEnvTest.groovy index 22fa09f72c..9887cb2553 100644 --- a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerFusionEnvTest.groovy +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerFusionEnvTest.groovy @@ -16,6 +16,8 @@ package io.seqera.tower.plugin +import nextflow.exception.AbortOperationException + import static com.github.tomakehurst.wiremock.client.WireMock.* import java.time.Instant @@ -70,6 +72,7 @@ class TowerFusionEnvTest extends Specification { def 'should return the endpoint from the config'() { given: 'a session' + SysEnv.push(['TOWER_API_ENDPOINT': 'https://tower.nf', 'TOWER_ACCESS_TOKEN': 'abc123']) Global.session = Mock(Session) { config >> [ tower: [ @@ -83,11 +86,13 @@ class TowerFusionEnvTest extends Specification { then: 'the endpoint has the expected value' provider.endpoint == 'https://tower.nf' + cleanup: + SysEnv.pop() } def 'should return the endpoint from the environment'() { setup: - SysEnv.push(['TOWER_API_ENDPOINT': 'https://tower.nf']) + SysEnv.push(['TOWER_API_ENDPOINT': 'https://tower.nf', 'TOWER_ACCESS_TOKEN': 'abc123']) Global.session = Mock(Session) { config >> [:] } @@ -103,6 +108,7 @@ class TowerFusionEnvTest extends Specification { } def 'should return the default endpoint'() { + SysEnv.push(['TOWER_ACCESS_TOKEN': 'abc123']) when: 'session config is empty' Global.session = Mock(Session) { config >> [ @@ -177,6 +183,9 @@ class TowerFusionEnvTest extends Specification { then: 'the endpoint has the expected value' provider.endpoint == TowerClient.DEF_ENDPOINT_URL + + cleanup: + SysEnv.pop() } def 'should return the access token from the config'() { @@ -249,7 +258,8 @@ class TowerFusionEnvTest extends Specification { def provider = new TowerFusionToken() then: 'the access token has the expected value' - provider.accessToken == null + def e = thrown(AbortOperationException) + e.message.contains("Missing Seqera Platform access token") cleanup: SysEnv.pop() @@ -307,8 +317,8 @@ class TowerFusionEnvTest extends Specification { ) ) and: - def client = TowerFactory.client(session, SysEnv.get()) - client.onFlowCreate(session) + def observer = new TowerFactory().create(session)[0] + observer.onFlowCreate(session) and: 'a mock endpoint returning a valid token' final now = Instant.now() @@ -371,7 +381,7 @@ class TowerFusionEnvTest extends Specification { ) ) and: - def client = TowerFactory.client(session, SysEnv.get()) + def client = new TowerFactory().create(session)[0] client.onFlowCreate(session) and: 'a mock endpoint returning a valid token' @@ -439,7 +449,7 @@ class TowerFusionEnvTest extends Specification { ) ) and: - def client = TowerFactory.client(session, SysEnv.get()) + def client = new TowerFactory().create(session)[0] client.onFlowCreate(session) and: 'prepare stubs' @@ -531,7 +541,7 @@ class TowerFusionEnvTest extends Specification { ) ) and: - def client = TowerFactory.client(session, SysEnv.get()) + def client = new TowerFactory().create(session)[0] client.onFlowCreate(session) and: 'a mock endpoint returning an error' wireMockServer.stubFor( diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerObserverTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerObserverTest.groovy new file mode 100644 index 0000000000..87f5c0f4af --- /dev/null +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/TowerObserverTest.groovy @@ -0,0 +1,581 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin + +import nextflow.script.PlatformMetadata + +import java.net.http.HttpResponse +import java.nio.file.Files +import java.time.Instant +import java.time.OffsetDateTime +import java.time.ZoneId + +import com.github.tomakehurst.wiremock.WireMockServer +import com.github.tomakehurst.wiremock.client.WireMock +import io.seqera.http.HxClient +import nextflow.Session +import nextflow.SysEnv +import nextflow.cloud.types.CloudMachineInfo +import nextflow.cloud.types.PriceModel +import nextflow.container.DockerConfig +import nextflow.container.resolver.ContainerMeta +import nextflow.exception.AbortOperationException +import nextflow.script.PlatformMetadata +import nextflow.script.ScriptBinding +import nextflow.script.WorkflowMetadata +import nextflow.trace.TraceRecord +import nextflow.trace.WorkflowStats +import nextflow.trace.WorkflowStatsObserver +import nextflow.util.Duration +import nextflow.util.ProcessHelper +import spock.lang.Specification +/** + * + * @author Paolo Di Tommaso + */ +class TowerObserverTest extends Specification { + + protected boolean aroundNow(value) { + def now = Instant.now().toEpochMilli() + value > now-1_000 && value <= now + } + + private TowerObserver newObserver(Session session, Map env = [:]) { + def client = Mock(TowerClient) + def observer = new TowerObserver(session, client, null, env) + observer.@reports = Mock(TowerReports) + return observer + } + + def 'should create message map' () { + given: + def session = Mock(Session) + def params = new ScriptBinding.ParamsMap(x: "hello") + def meta = Mock(WorkflowMetadata) + def observer = Spy(newObserver(session)) + observer.@workflowId = '12ef' + + when: + def map = observer.makeCompleteReq(session) + then: + 1 * session.getWorkflowMetadata() >> meta + 1 * session.getParams() >> params + 1 * meta.toMap() >> [foo:1, bar:2, container: [p1: 'c1', p2: 'c2']] + 1 * observer.getMetricsList() >> [[process:'foo', cpu: [min: 1, max:5], time: [min: 6, max: 9]]] + 1 * observer.getWorkflowProgress(false) >> new WorkflowProgress() + 1 * observer.getOutFile() >> 'bar.out' + 1 * observer.getLogFile() >> 'foo.out' + 1 * observer.getOperationId() >> 'op-12345' + then: + map.workflow.foo == 1 + map.workflow.bar == 2 + map.workflow.id == '12ef' + map.workflow.params == [x: 'hello'] + map.workflow.container == null + map.metrics == [[process:'foo', cpu: [min: 1, max:5], time: [min: 6, max: 9]]] + map.progress == new WorkflowProgress() + and: + aroundNow(map.instant) + and: + map.workflow.outFile == 'bar.out' + map.workflow.logFile == 'foo.out' + map.workflow.operationId == 'op-12345' + } + + def 'should capitalise underscores' () { + given: + def tower = new TowerObserver(Mock(Session), Mock(TowerClient), "ws1234", [:] ) + + expect: + tower.underscoreToCamelCase(STR) == EXPECTED + where: + STR | EXPECTED + 'abc' | 'abc' + 'a_b_c' | 'aBC' + 'foo__bar' | 'fooBar' + } + + def 'should post task records' () { + given: + def session = Mock(Session) + def PROGRESS = Mock(WorkflowProgress) { getRunning()>>1; getSucceeded()>>2; getFailed()>>3 } + def observer = Spy(newObserver(session)) + observer.@workflowId = 'xyz-123' + + def nowTs = System.currentTimeMillis() + def submitTs = nowTs-2000 + def startTs = nowTs-1000 + + def trace = new TraceRecord([ + taskId: 10, + process: 'foo', + workdir: "/work/dir", + cpus: 1, + submit: submitTs, + start: startTs, + complete: nowTs ]) + trace.executorName= 'batch' + trace.machineInfo = new CloudMachineInfo('m4.large', 'eu-west-1b', PriceModel.spot) + trace.containerMeta = new ContainerMeta(requestId: '12345', sourceImage: 'ubuntu:latest', targetImage: 'wave.io/12345/ubuntu:latest') + + when: + def req = observer.makeTasksReq([trace]) + then: + observer.getWorkflowProgress(true) >> PROGRESS + and: + req.tasks[0].taskId == 10 + req.tasks[0].process == 'foo' + req.tasks[0].workdir == "/work/dir" + req.tasks[0].cpus == 1 + req.tasks[0].submit == OffsetDateTime.ofInstant(Instant.ofEpochMilli(submitTs), ZoneId.systemDefault()) + req.tasks[0].start == OffsetDateTime.ofInstant(Instant.ofEpochMilli(startTs), ZoneId.systemDefault()) + req.tasks[0].executor == 'batch' + req.tasks[0].machineType == 'm4.large' + req.tasks[0].cloudZone == 'eu-west-1b' + req.tasks[0].priceModel == 'spot' + and: + req.progress.running == 1 + req.progress.succeeded == 2 + req.progress.failed == 3 + and: + req.containers[0].requestId == '12345' + req.containers[0].sourceImage == 'ubuntu:latest' + req.containers[0].targetImage == 'wave.io/12345/ubuntu:latest' + and: + aroundNow(req.instant) + } + + static now_millis = System.currentTimeMillis() + static now_instant = OffsetDateTime.ofInstant(Instant.ofEpochMilli(now_millis), ZoneId.systemDefault()) + + def 'should fix field types' () { + + expect: + TowerObserver.fixTaskField(FIELD, VALUE) == EXPECTED + + where: + FIELD | VALUE | EXPECTED + 'foo' | 'hola' | 'hola' + 'submit' | now_millis | now_instant + 'start' | now_millis | now_instant + 'complete' | now_millis | now_instant + 'complete' | 0 | null + } + + def 'should create workflow json' () { + + given: + def sessionId = UUID.randomUUID() + def dir = Files.createTempDirectory('test') + def session = Mock(Session) + session.getUniqueId() >> sessionId + session.getRunName() >> 'foo' + session.config >> [:] + session.containerConfig >> new DockerConfig([:]) + session.getParams() >> new ScriptBinding.ParamsMap([foo:'Hello', bar:'World']) + + def meta = new WorkflowMetadata( + session: session, + projectName: 'the-project-name', + repository: 'git://repo.com/foo' ) + session.getWorkflowMetadata() >> meta + session.getStatsObserver() >> Mock(WorkflowStatsObserver) { getStats() >> new WorkflowStats() } + + def observer = Spy(newObserver(session, ENV)) + observer.getOperationId() >> 'op-112233' + observer.getLogFile() >> 'log.file' + observer.getOutFile() >> 'out.file' + + when: + def req1 = observer.makeCreateReq(session) + then: + req1.sessionId == sessionId.toString() + req1.runName == 'foo' + req1.projectName == 'the-project-name' + req1.repository == 'git://repo.com/foo' + req1.workflowId == WORKFLOW_ID + and: + aroundNow(req1.instant) + + when: + def req = observer.makeBeginReq(session) + then: + observer.getWorkflowId() >> '12345' + and: + req.workflow.id == '12345' + req.workflow.params == [foo:'Hello', bar:'World'] + req.workflow.outFile == 'out.file' + req.workflow.logFile == 'log.file' + req.workflow.operationId == 'op-112233' + and: + req.towerLaunch == TOWER_LAUNCH + and: + aroundNow(req.instant) + + cleanup: + dir?.deleteDir() + + where: + ENV | WORKFLOW_ID | TOWER_LAUNCH + [:] | null | false + [TOWER_WORKFLOW_ID: '1234'] | '1234' | true + + } + + def 'should convert map' () { + given: + def tower = new TowerObserver(Mock(Session), Mock(TowerClient), "ws1234", [:] ) + + expect: + tower.mapToString(null) == null + tower.mapToString('ciao') == 'ciao' + tower.mapToString([:]) == null + tower.mapToString([p:'foo', q:'bar']) == null + } + + def 'should create init request' () { + given: + def uuid = UUID.randomUUID() + def meta = Mock(WorkflowMetadata) { + getProjectName() >> 'the-project-name' + getRepository() >> 'git://repo.com/foo' + } + def session = Mock(Session) { + getUniqueId() >> uuid + getRunName() >> 'foo_bar' + getWorkflowMetadata() >> meta + } + def observer = newObserver(session, [TOWER_WORKFLOW_ID: 'x123']) + + when: + def req = observer.makeCreateReq(session) + then: + req.sessionId == uuid.toString() + req.runName == 'foo_bar' + req.projectName == 'the-project-name' + req.repository == 'git://repo.com/foo' + req.workflowId == 'x123' + and: + aroundNow(req.instant) + + and: + observer.towerLaunch + } + + def 'should post create request' () { + given: + def uuid = UUID.randomUUID() + def platform = new PlatformMetadata() + def meta = Mock(WorkflowMetadata) { + getProjectName() >> 'the-project-name' + getRepository() >> 'git://repo.com/foo' + getPlatform() >> platform + } + def session = Mock(Session) { + getUniqueId() >> uuid + getRunName() >> 'foo_bar' + getWorkflowMetadata() >> meta + } + def towerClient = Mock(TowerClient) + def observer = Spy(new TowerObserver(session, towerClient, null, [:])) + observer.@reports = Mock(TowerReports) + + when: + observer.onFlowCreate(session) + then: + 1 * observer.makeCreateReq(session) >> [runName: 'foo'] + 1 * towerClient.traceCreate([runName: 'foo'], null) >> [workflowId: 'xyz123', watchUrl: 'https://cloud.seqera.io/watch/xyz123'] + and: + observer.runName == 'foo_bar' + observer.runId == uuid.toString() + and: + observer.workflowId == 'xyz123' + observer.@watchUrl == 'https://cloud.seqera.io/watch/xyz123' + !observer.towerLaunch + and: + platform.workflowId == 'xyz123' + platform.workflowUrl == 'https://cloud.seqera.io/watch/xyz123' + + } + + def 'should set workflowUrl on platform metadata during onFlowBegin' () { + given: + def platform = new PlatformMetadata() + def meta = Mock(WorkflowMetadata) { + getPlatform() >> platform + } + def session = Mock(Session) { + getWorkflowMetadata() >> meta + } + def towerClient = Mock(TowerClient) + def observer = Spy(new TowerObserver(session, towerClient, null, [:])) + observer.@reports = Mock(TowerReports) + observer.@workflowId = 'abc123' + + when: + observer.onFlowBegin() + then: + 1 * observer.makeBeginReq(session) >> [foo: 'bar'] + 1 * towerClient.traceBegin([foo: 'bar'], null, 'abc123') >> [watchUrl: 'https://cloud.seqera.io/watch/abc123'] + and: + observer.@watchUrl == 'https://cloud.seqera.io/watch/abc123' + platform.workflowUrl == 'https://cloud.seqera.io/watch/abc123' + + cleanup: + observer.@sender?.interrupt() + } + + def 'should fetch workflow meta' () { + given: + def session = Mock(Session) + def observer = newObserver(session, ENV) + + expect: + observer.getOperationId() == OP_ID + observer.getLogFile() == LOG_FILE + observer.getOutFile() == OUT_FILE + + where: + OP_ID | OUT_FILE | LOG_FILE | ENV + null | null | null | [:] + "local-platform::${ProcessHelper.selfPid()}" | null | null | [TOWER_ALLOW_NEXTFLOW_LOGS:'true'] + 'aws-batch::1234z' | 'xyz.out' | 'hola.log' | [TOWER_ALLOW_NEXTFLOW_LOGS:'true', AWS_BATCH_JOB_ID: '1234z', NXF_OUT_FILE: 'xyz.out', NXF_LOG_FILE: 'hola.log'] + } + + def 'should deduplicate containers' () { + given: + def session = Mock(Session) + def observer = newObserver(session) + and: + def c1 = new ContainerMeta(requestId: '12345', sourceImage: 'ubuntu:latest', targetImage: 'wave.io/12345/ubuntu:latest') + def c2 = new ContainerMeta(requestId: '54321', sourceImage: 'ubuntu:latest', targetImage: 'wave.io/54321/ubuntu:latest') + and: + def trace1 = new TraceRecord( + taskId: 1, + process: 'foo', + workdir: "/work/dir", + cpus: 1, + submit: System.currentTimeMillis(), + start: System.currentTimeMillis(), + complete: System.currentTimeMillis()) + trace1.containerMeta = c1 + and: + def trace2 = new TraceRecord( + taskId: 2, + process: 'foo', + workdir: "/work/dir", + cpus: 1, + submit: System.currentTimeMillis(), + start: System.currentTimeMillis(), + complete: System.currentTimeMillis()) + trace2.containerMeta = c2 + and: + def trace3 = new TraceRecord( + taskId: 3, + process: 'foo', + workdir: "/work/dir", + cpus: 1, + submit: System.currentTimeMillis(), + start: System.currentTimeMillis(), + complete: System.currentTimeMillis()) + trace3.containerMeta = c2 + + expect: + observer.getNewContainers([trace1]) == [c1] + and: + observer.getNewContainers([trace1]) == [] + and: + observer.getNewContainers([trace1, trace2, trace3]) == [c2] + } + + def 'should not send complete request when onFlowBegin was not invoked' () { + given: + def session = Mock(Session) + def towerClient = Mock(TowerClient) + def observer = Spy(new TowerObserver(session, towerClient, null, [:])) + def reports = Mock(TowerReports) + observer.@reports = reports + observer.@workflowId = 'xyz-123' + observer.@sender = null + + when: + observer.onFlowComplete() + + then: + 1 * reports.publishRuntimeReports() + 1 * reports.flowComplete() + 0 * towerClient.traceComplete(_, _, _) + } + + def 'should apply platform metadata from trace create response'() { + given: + def metadata = new WorkflowMetadata() + def session = Mock(Session) { + getWorkflowMetadata() >> metadata + } + def observer = new TowerObserver(session, Mock(TowerClient), '1234', SysEnv.get()) + + def responseMetadata = [ + userId: 39, + userName: 'user', + userOrganization: 'ACME Inc.', + workspaceId: 1234, + workspaceName: 'Workspace-Name', + workspaceFullName: 'Full Workspace Name', + orgName: 'ACME Inc.', + computeEnvId: 'ce1234', + computeEnvName: 'ce-test', + computeEnvPlatform: 'aws-batch', + pipelineName: 'test-pipeline', + pipelineId: 'pipe1234', + revision: 'v1.1', + commitId: 'abcd12345' + ] + + when: + observer.applyPlatformMetadata(responseMetadata) + + then: + metadata.platform.user.id == '39' + metadata.platform.user.userName == 'user' + metadata.platform.user.organization == 'ACME Inc.' + metadata.platform.workspace.id == '1234' + metadata.platform.workspace.name == 'Workspace-Name' + metadata.platform.workspace.fullName == 'Full Workspace Name' + metadata.platform.workspace.organization == 'ACME Inc.' + metadata.platform.computeEnv.id == 'ce1234' + metadata.platform.computeEnv.name == 'ce-test' + metadata.platform.computeEnv.platform == 'aws-batch' + metadata.platform.pipeline.id == 'pipe1234' + metadata.platform.pipeline.name == 'test-pipeline' + metadata.platform.pipeline.revision == 'v1.1' + metadata.platform.pipeline.commitId == 'abcd12345' + } + + def 'should include numSpotInterruptions in task map'() { + given: + def session = Mock(Session) + def observer = Spy(newObserver(session)) + observer.getWorkflowProgress(true) >> new WorkflowProgress() + + def now = System.currentTimeMillis() + def trace = new TraceRecord([ + taskId: 42, + process: 'foo', + workdir: "/work/dir", + cpus: 1, + submit: now-2000, + start: now-1000, + complete: now + ]) + trace.setNumSpotInterruptions(3) + + when: + def req = observer.makeTasksReq([trace]) + + then: + req.tasks.size() == 1 + req.tasks[0].numSpotInterruptions == 3 + } + + def 'should include logStreamId in task map'() { + given: + def session = Mock(Session) + def observer = Spy(newObserver(session)) + observer.getWorkflowProgress(true) >> new WorkflowProgress() + + def now = System.currentTimeMillis() + def trace = new TraceRecord([ + taskId: 42, + process: 'foo', + workdir: "/work/dir", + cpus: 1, + submit: now-2000, + start: now-1000, + complete: now + ]) + trace.setLogStreamId('arn:aws:logs:us-east-1:123456789:log-group:/ecs/task:log-stream:abc123') + + when: + def req = observer.makeTasksReq([trace]) + + then: + req.tasks.size() == 1 + req.tasks[0].logStreamId == 'arn:aws:logs:us-east-1:123456789:log-group:/ecs/task:log-stream:abc123' + } + + def 'should include resourceAllocation in task map'() { + given: + def session = Mock(Session) + def observer = Spy(newObserver(session)) + observer.getWorkflowProgress(true) >> new WorkflowProgress() + + def now = System.currentTimeMillis() + def trace = new TraceRecord([ + taskId: 42, + process: 'foo', + workdir: "/work/dir", + cpus: 1, + submit: now-2000, + start: now-1000, + complete: now, + accelerator: 2, + acceleratorType: 'v100' + ]) + trace.setResourceAllocation([cpuShares: 2048, memoryMiB: 4096, time: '1h']) + + when: + def req = observer.makeTasksReq([trace]) + + then: + req.tasks.size() == 1 + req.tasks[0].accelerator == 2 + req.tasks[0].acceleratorType == 'v100' + req.tasks[0].resourceAllocation == [cpuShares: 2048, memoryMiB: 4096, time: '1h'] + } + + def 'should include gpuMetrics in task map'() { + given: + def session = Mock(Session) + def observer = Spy(newObserver(session)) + observer.getWorkflowProgress(true) >> new WorkflowProgress() + + def now = System.currentTimeMillis() + def trace = new TraceRecord([ + taskId: 42, + process: 'foo', + workdir: "/work/dir", + cpus: 1, + submit: now-2000, + start: now-1000, + complete: now + ]) + trace.setGpuMetrics([name: 'Tesla T4', mem: 15360, driver: '580.126.09', active_time: 651030, pct: 75, peak: 100]) + + when: + def req = observer.makeTasksReq([trace]) + + then: + req.tasks.size() == 1 + req.tasks[0].gpuMetrics.name == 'Tesla T4' + req.tasks[0].gpuMetrics.mem == 15360 + req.tasks[0].gpuMetrics.pct == 75 + req.tasks[0].gpuMetrics.peak == 100 + } + + +} diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/auth/AuthCommandImplTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/auth/AuthCommandImplTest.groovy index 56b515b0ee..ff6e7f1fda 100644 --- a/plugins/nf-tower/src/test/io/seqera/tower/plugin/auth/AuthCommandImplTest.groovy +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/auth/AuthCommandImplTest.groovy @@ -16,8 +16,7 @@ package io.seqera.tower.plugin.auth -import io.seqera.http.HxClient -import io.seqera.tower.plugin.TowerCommonApi +import io.seqera.tower.plugin.TowerClient import nextflow.Const import nextflow.SysEnv import nextflow.util.ColorUtil @@ -26,7 +25,6 @@ import spock.lang.Specification import spock.lang.TempDir import test.OutputCapture -import java.net.http.HttpResponse import java.nio.file.Files import java.nio.file.Path @@ -652,13 +650,14 @@ param2 = 'value2'""" ] // Mock API calls - def commonApi = Mock(TowerCommonApi){ - getUserInfo(_, _) >> [userName: 'testuser', id: '123'] - getWorkflowDetails(_, _, _) >> null + def client = Mock(TowerClient){ + getUserInfo() >> [userName: 'testuser', id: '123'] + getWorkflowDetails(_, _) >> null } - def cmd = Spy(new AuthCommandImpl(commonApi)) + def cmd = Spy(new AuthCommandImpl()) + cmd.createTowerClient(_,_) >> client cmd.checkApiConnection(_) >> true - cmd.listComputeEnvironments(_, _, _) >> [[name: 'ce_test', platform: 'aws', workDir: 's3://test', primary: true]] + cmd.listComputeEnvironments(_, _) >> [[name: 'ce_test', platform: 'aws', workDir: 's3://test', primary: true]] when: def status = cmd.collectStatus(config) @@ -734,10 +733,11 @@ param2 = 'value2'""" def config = ['tower.accessToken': 'invalid-token'] SysEnv.push([:]) // Isolate from actual environment variables - def commonApi = Mock(TowerCommonApi){ - getUserInfo(_, _) >> { throw new RuntimeException('Invalid token') } + def client = Mock(TowerClient){ + getUserInfo() >> { throw new RuntimeException('Invalid token') } } - def cmd = Spy(new AuthCommandImpl(commonApi)) + def cmd = Spy(new AuthCommandImpl()) + cmd.createTowerClient(_,_) >> client cmd.checkApiConnection(_) >> true when: def status = cmd.collectStatus(config) @@ -791,15 +791,16 @@ param2 = 'value2'""" 'tower.accessToken': 'test-token', 'tower.workspaceId': '12345' ] - def commonApi = Mock(TowerCommonApi){ - getUserInfo(_, _) >> [userName: 'testuser', id: '123'] - getUserWorkspaceDetails(_, _, _, _) >> [ + def client = Mock(TowerClient){ + getUserInfo() >> [userName: 'testuser', id: '123'] + getUserWorkspaceDetails(_, _) >> [ orgName: 'TestOrg', workspaceName: 'TestWorkspace', workspaceFullName: 'test-org/test-workspace' ] } - def cmd = Spy(new AuthCommandImpl(commonApi)) + def cmd = Spy(new AuthCommandImpl()) + cmd.createTowerClient(_,_) >> client cmd.checkApiConnection(_) >> true when: @@ -817,15 +818,16 @@ param2 = 'value2'""" def 'should collect status with workspace ID but no details'() { given: - def commonApi = Mock(TowerCommonApi){ - getUserInfo(_, _) >> [userName: 'testuser', id: '123'] - getUserWorkspaceDetails(_, _, _, _) >> null + def client = Mock(TowerClient){ + getUserInfo() >> [userName: 'testuser', id: '123'] + getUserWorkspaceDetails(_, _) >> null } def config = [ 'tower.accessToken': 'test-token', 'tower.workspaceId': '12345' ] - def cmd = Spy(new AuthCommandImpl(commonApi)) + def cmd = Spy(new AuthCommandImpl()) + cmd.createTowerClient(_,_) >> client cmd.checkApiConnection(_) >> true when: @@ -840,14 +842,15 @@ param2 = 'value2'""" def 'should collect status from environment variables'() { given: - def commonApi = Mock(TowerCommonApi){ - getUserInfo(_, _) >> [userName: 'envuser', id: '456'] - getUserWorkspaceDetails(_, _, _, _) >> [:] + def client = Mock(TowerClient){ + getUserInfo() >> [userName: 'envuser', id: '456'] + getUserWorkspaceDetails(_, _) >> [:] } def config = [:] - def cmd = Spy(new AuthCommandImpl(commonApi)) + def cmd = Spy(new AuthCommandImpl()) + cmd.createTowerClient(_,_) >> client cmd.checkApiConnection(_) >> true - cmd.listComputeEnvironments(_,_,_) >> [] + cmd.listComputeEnvironments(_,_) >> [] SysEnv.push(['TOWER_ACCESS_TOKEN': 'env-token', 'TOWER_API_ENDPOINT': 'https://env.example.com', @@ -892,13 +895,13 @@ param2 = 'value2'""" def 'should collect status with mixed sources'() { given: - def commonApi = Mock(TowerCommonApi) { - getUserInfo(_, _) >> [userName: 'mixeduser', id: '789'] + def client = Mock(TowerClient) { + getUserInfo() >> [userName: 'mixeduser', id: '789'] } - def cmd = Spy(new AuthCommandImpl(commonApi)) + def cmd = Spy(new AuthCommandImpl()) + cmd.createTowerClient(_,_) >> client def authFile = tempDir.resolve('seqera-auth.config') def configFile = tempDir.resolve('config') - cmd.getAuthFile() >> authFile cmd.getConfigFile() >> configFile @@ -1071,16 +1074,16 @@ param2 = 'value2'""" given: def cmd = Spy(AuthCommandImpl) def username = System.getProperty('user.name') - - cmd.createHttpClient(_) >> { - def mockClient = Mock(HxClient) - def mockResponse = Mock(HttpResponse) - mockResponse.statusCode() >> 200 - mockResponse.body() >> '{"accessKey":"generated-pat-123","id":"token-id-456"}' - mockClient.send(_, _) >> mockResponse - return mockClient + def mockApiResponse = Mock(TowerClient.Response){ + getCode() >> 200 + getMessage() >> '{"accessKey":"generated-pat-123","id":"token-id-456"}' + } + def mockClient = Mock(TowerClient){ + sendApiRequest(_,_,_) >> mockApiResponse } + cmd.createTowerClient(_,_) >> mockClient + when: def pat = cmd.generatePAT('auth-token', 'https://api.cloud.seqera.io') @@ -1092,15 +1095,16 @@ param2 = 'value2'""" given: def cmd = Spy(AuthCommandImpl) - cmd.createHttpClient(_) >> { - def mockClient = Mock(HxClient) - def mockResponse = Mock(HttpResponse) - mockResponse.statusCode() >> 401 - mockResponse.body() >> 'Unauthorized' - mockClient.send(_, _) >> mockResponse - return mockClient + def mockApiResponse = Mock(TowerClient.Response){ + getCode() >> 401 + getMessage() >> 'Unauthorized' + } + def mockClient = Mock(TowerClient){ + sendApiRequest(_,_,_) >> mockApiResponse } + cmd.createTowerClient(_,_) >> mockClient + when: cmd.generatePAT('invalid-token', 'https://api.cloud.seqera.io') diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/dataset/SeqeraDatasetClientTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/dataset/SeqeraDatasetClientTest.groovy new file mode 100644 index 0000000000..25884bf3fa --- /dev/null +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/dataset/SeqeraDatasetClientTest.groovy @@ -0,0 +1,217 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.dataset + +import java.nio.file.AccessDeniedException +import java.nio.file.NoSuchFileException + +import groovy.json.JsonOutput +import io.seqera.tower.plugin.TowerClient +import io.seqera.tower.plugin.exception.ForbiddenException +import io.seqera.tower.plugin.exception.NotFoundException +import io.seqera.tower.plugin.exception.UnauthorizedException +import nextflow.exception.AbortOperationException +import spock.lang.Specification + +/** + * Tests for {@link SeqeraDatasetClient} using a mock {@link TowerClient}. + */ +class SeqeraDatasetClientTest extends Specification { + + private TowerClient mockTower(String endpoint = 'https://api.example.com') { + def tc = Mock(TowerClient) + tc.endpoint >> endpoint + return tc + } + private TowerClient spyTower(String endpoint = 'https://api.example.com') { + def tc = Spy(TowerClient) + tc.@endpoint = endpoint + return tc + } + + private static TowerClient.Response ok(String body) { + new TowerClient.Response(200, body) + } + + private static TowerClient.Response error(int code) { + new TowerClient.Response(code, "error $code") + } + + // ---- listUserWorkspacesAndOrgs ---- + + def "listUserWorkspacesAndOrgs returns parsed DTOs"() { + given: + def body = JsonOutput.toJson([orgsAndWorkspaces: [ + [orgId: 1, orgName: 'acme', workspaceId: 10, workspaceName: 'research', workspaceFullName: 'acme/research'] + ]]) + def tc = spyTower() + tc.sendApiRequest('https://api.example.com/user/42/workspaces') >> ok(body) + def client = new SeqeraDatasetClient(tc) + + when: + def list = client.listUserWorkspacesAndOrgs(42L) + + then: + list.size() == 1 + list[0].orgName == 'acme' + list[0].workspaceId == 10L + list[0].workspaceName == 'research' + } + + // ---- listDatasets ---- + + def "listDatasets returns parsed DatasetDto list"() { + given: + def body = JsonOutput.toJson([datasets: [ + [id: 'ds-1', name: 'samples', version: 2, mediaType: 'text/csv', + dateCreated: '2024-01-01T00:00:00Z', lastUpdated: '2024-01-02T00:00:00Z'] + ], totalSize: 1]) + def tc = mockTower() + tc.sendApiRequest('https://api.example.com/datasets?workspaceId=99') >> ok(body) + def client = new SeqeraDatasetClient(tc) + + when: + def list = client.listDatasets(99L) + + then: + list.size() == 1 + list[0].id == 'ds-1' + list[0].name == 'samples' + list[0].version == 2L + } + + def "listDatasets returns empty list when no datasets"() { + given: + def tc = mockTower() + tc.sendApiRequest('https://api.example.com/datasets?workspaceId=99') >> + ok(JsonOutput.toJson([datasets: [], totalSize: 0])) + def client = new SeqeraDatasetClient(tc) + + when: + def list = client.listDatasets(99L) + + then: + list.isEmpty() + } + + // ---- listVersions ---- + + def "listVersions returns parsed DatasetVersionDto list"() { + given: + def body = JsonOutput.toJson([versions: [ + [datasetId: 'ds-1', version: 1, fileName: 'samples.csv', + mediaType: 'text/csv', hasHeader: true, dateCreated: '2024-01-01T00:00:00Z', disabled: false] + ]]) + def tc = mockTower() + tc.sendApiRequest('https://api.example.com/datasets/ds-1/versions?workspaceId=1234') >> ok(body) + def client = new SeqeraDatasetClient(tc) + + when: + def list = client.listVersions('ds-1', 1234) + + then: + list.size() == 1 + list[0].version == 1L + list[0].fileName == 'samples.csv' + list[0].hasHeader + !list[0].disabled + } + + // ---- downloadDataset ---- + + def "downloadDataset returns InputStream with correct content"() { + given: + def content = 'col1,col2\n1,2\n' + def tc = mockTower() + tc.sendStreamingRequest('https://api.example.com/datasets/ds-1/v/1/n/samples.csv?workspaceId=1234') >> new ByteArrayInputStream(content.getBytes('UTF-8')) + def client = new SeqeraDatasetClient(tc) + + when: + def stream = client.downloadDataset('ds-1', '1', 'samples.csv', 1234) + + then: + stream.text == content + } + + def "downloadDataset URL-encodes the filename"() { + given: + def tc = mockTower() + def client = new SeqeraDatasetClient(tc) + + when: + client.downloadDataset('ds-1', '1', 'my file.csv',1234) + + then: + 1 * tc.sendStreamingRequest('https://api.example.com/datasets/ds-1/v/1/n/my%20file.csv?workspaceId=1234') >> new ByteArrayInputStream('data'.getBytes('UTF-8')) + } + + def "downloadDataset throws NoSuchFileException on 404"() { + given: + def tc = mockTower() + tc.sendStreamingRequest(_) >> { throw new NotFoundException("not found") } + def client = new SeqeraDatasetClient(tc) + + when: + client.downloadDataset('ds-missing', '1', 'file.csv', 1234) + + then: + thrown(NoSuchFileException) + } + + def "downloadDataset throws AccessDeniedException on 403"() { + given: + def tc = mockTower() + tc.sendStreamingRequest(_) >> { throw new ForbiddenException("forbidden") } + def client = new SeqeraDatasetClient(tc) + + when: + client.downloadDataset('ds-1', '1', 'file.csv', 1234) + + then: + thrown(AccessDeniedException) + } + + def "downloadDataset throws AbortOperationException on 401"() { + given: + def tc = mockTower() + tc.sendStreamingRequest(_) >> { throw new UnauthorizedException("unauthorized") } + def client = new SeqeraDatasetClient(tc) + + when: + client.downloadDataset('ds-1', '1', 'file.csv', 1234) + + then: + thrown(AbortOperationException) + } + + // ---- createDataset ---- + + def "createDataset posts and returns created dataset"() { + given: + def responseBody = JsonOutput.toJson([dataset: [id: 'ds-new', name: 'results']]) + def tc = mockTower() + tc.sendApiRequest('https://api.example.com/datasets?workspaceId=10', [name: 'results'], 'POST') >> ok(responseBody) + def client = new SeqeraDatasetClient(tc) + + when: + def dto = client.createDataset(10L, 'results') + + then: + dto.id == 'ds-new' + dto.name == 'results' + } +} diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/DatasetInputStreamTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/DatasetInputStreamTest.groovy new file mode 100644 index 0000000000..6f9fde6847 --- /dev/null +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/DatasetInputStreamTest.groovy @@ -0,0 +1,141 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.fs + +import java.nio.ByteBuffer + +import spock.lang.Specification + +class DatasetInputStreamTest extends Specification { + + def 'should read bytes into buffer'() { + given: + def data = 'hello world'.bytes + def channel = new DatasetInputStream(new ByteArrayInputStream(data)) + def buf = ByteBuffer.allocate(data.length) + + when: + def n = channel.read(buf) + + then: + n == data.length + buf.array() == data + } + + def 'should advance position after read'() { + given: + def data = 'abcdef'.bytes + def channel = new DatasetInputStream(new ByteArrayInputStream(data)) + + when: + channel.read(ByteBuffer.allocate(3)) + + then: + channel.position() == 3 + + when: + channel.read(ByteBuffer.allocate(3)) + + then: + channel.position() == 6 + } + + def 'should return -1 at end of stream'() { + given: + def channel = new DatasetInputStream(new ByteArrayInputStream(new byte[0])) + + when: + def n = channel.read(ByteBuffer.allocate(8)) + + then: + n == -1 + channel.position() == 0 + } + + def 'should read partial buffer when stream has fewer bytes'() { + given: + def data = 'hi'.bytes + def channel = new DatasetInputStream(new ByteArrayInputStream(data)) + def buf = ByteBuffer.allocate(100) + + when: + def n = channel.read(buf) + + then: + n == 2 + channel.position() == 2 + } + + def 'should be open initially and closed after close()'() { + given: + def channel = new DatasetInputStream(new ByteArrayInputStream(new byte[0])) + + expect: + channel.isOpen() + + when: + channel.close() + + then: + !channel.isOpen() + } + + def 'should close underlying stream on close()'() { + given: + def stream = Mock(InputStream) + def channel = new DatasetInputStream(stream) + + when: + channel.close() + + then: + 1 * stream.close() + !channel.isOpen() + } + + def 'should throw on size'() { + when: + new DatasetInputStream(new ByteArrayInputStream(new byte[0])).size() + + then: + thrown(UnsupportedOperationException) + } + + def 'should throw on write'() { + when: + new DatasetInputStream(new ByteArrayInputStream(new byte[0])).write(ByteBuffer.allocate(1)) + + then: + thrown(UnsupportedOperationException) + } + + def 'should throw on seek'() { + when: + new DatasetInputStream(new ByteArrayInputStream(new byte[0])).position(0L) + + then: + thrown(UnsupportedOperationException) + } + + def 'should throw on truncate'() { + when: + new DatasetInputStream(new ByteArrayInputStream(new byte[0])).truncate(0L) + + then: + thrown(UnsupportedOperationException) + } +} diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/SeqeraFileSystemProviderTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/SeqeraFileSystemProviderTest.groovy new file mode 100644 index 0000000000..7a698c1d5a --- /dev/null +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/SeqeraFileSystemProviderTest.groovy @@ -0,0 +1,411 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.fs + +import java.nio.file.AccessDeniedException +import java.nio.file.DirectoryStream +import java.nio.file.FileSystemAlreadyExistsException +import java.nio.file.InvalidPathException +import java.nio.file.NoSuchFileException +import java.nio.file.attribute.BasicFileAttributes + +import groovy.json.JsonOutput +import io.seqera.tower.plugin.TowerClient +import io.seqera.tower.plugin.dataset.SeqeraDatasetClient +import nextflow.exception.AbortOperationException +import spock.lang.Specification + +/** + * Tests for {@link SeqeraFileSystemProvider} using a mock {@link TowerClient}. + */ +class SeqeraFileSystemProviderTest extends Specification { + + private static final String ENDPOINT = 'https://api.example.com' + + private TowerClient spyTower() { + def tc = Spy(TowerClient) + tc.@endpoint = ENDPOINT + return tc + } + + private static TowerClient.Response ok(String body) { + new TowerClient.Response(200, body) + } + + private static TowerClient.Response error(int code) { + new TowerClient.Response(code, "error $code") + } + + private SeqeraFileSystem buildFs(TowerClient tc) { + final client = new SeqeraDatasetClient(tc) + final provider = new SeqeraFileSystemProvider() + return new SeqeraFileSystem(provider, client) + } + + private static String userInfoJson() { + JsonOutput.toJson([user: [id: 42L, userName: 'testuser']]) + } + + private static String workspacesJson() { + JsonOutput.toJson([orgsAndWorkspaces: [ + [orgId: 1L, orgName: 'acme', workspaceId: 10L, workspaceName: 'research', workspaceFullName: 'acme/research'] + ]]) + } + + private static String datasetsJson() { + JsonOutput.toJson([datasets: [ + [id: 'ds-1', name: 'samples', version: 2L, mediaType: 'text/csv', + workspaceId: 10L, + dateCreated: '2024-01-01T00:00:00Z', lastUpdated: '2024-01-02T00:00:00Z'] + ], totalSize: 1]) + } + + private static String versionsJson() { + JsonOutput.toJson([versions: [ + [datasetId: 'ds-1', version: 1L, fileName: 'samples.csv', + mediaType: 'text/csv', hasHeader: true, dateCreated: '2024-01-01T00:00:00Z', disabled: false], + [datasetId: 'ds-1', version: 2L, fileName: 'samples_v2.csv', + mediaType: 'text/csv', hasHeader: true, dateCreated: '2024-01-02T00:00:00Z', disabled: false] + ]]) + } + + // ---- newInputStream - latest version ---- + + def "newInputStream resolves latest version and downloads correct content"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> ok(datasetsJson()) + tc.sendApiRequest("${ENDPOINT}/datasets/ds-1/versions?workspaceId=10") >> ok(versionsJson()) + final csvContent = 'col1,col2\n1,2\n3,4\n' + tc.sendStreamingRequest("${ENDPOINT}/datasets/ds-1/v/2/n/samples_v2.csv?workspaceId=10") >> new ByteArrayInputStream(csvContent.getBytes('UTF-8')) + + final fs = buildFs(tc) + final path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + when: + final text = fs.provider().newInputStream(path).text + + then: + text == csvContent + } + + // ---- newInputStream - pinned version ---- + + def "newInputStream uses pinned version when @ver suffix given"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> ok(datasetsJson()) + tc.sendApiRequest("${ENDPOINT}/datasets/ds-1/versions?workspaceId=10") >> ok(versionsJson()) + final csvContent = 'col1,col2\n1,2\n' + tc.sendStreamingRequest("${ENDPOINT}/datasets/ds-1/v/1/n/samples.csv?workspaceId=10") >> new ByteArrayInputStream(csvContent.getBytes('UTF-8')) + + final fs = buildFs(tc) + final path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples@1') + + when: + final text = fs.provider().newInputStream(path).text + + then: + text == csvContent + } + + // ---- newInputStream - missing dataset ---- + + def "newInputStream throws NoSuchFileException for unknown dataset"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> + ok(JsonOutput.toJson([datasets: [], totalSize: 0])) + + final fs = buildFs(tc) + final path = new SeqeraPath(fs, 'seqera://acme/research/datasets/missing-dataset') + + when: + fs.provider().newInputStream(path) + + then: + thrown(NoSuchFileException) + } + + // ---- newInputStream - pinned version not found ---- + + def "newInputStream throws NoSuchFileException for unknown pinned version"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> ok(datasetsJson()) + tc.sendApiRequest("${ENDPOINT}/datasets/ds-1/versions?workspaceId=10") >> ok(versionsJson()) + + final fs = buildFs(tc) + final path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples@99') + + when: + fs.provider().newInputStream(path) + + then: + thrown(NoSuchFileException) + } + + // ---- readAttributes ---- + + def "readAttributes returns directory attributes for depth < 4"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + + final fs = buildFs(tc) + final path = new SeqeraPath(fs, 'seqera://acme/research') + + when: + final attrs = fs.provider().readAttributes(path, java.nio.file.attribute.BasicFileAttributes) + + then: + attrs.isDirectory() + !attrs.isRegularFile() + } + + def "readAttributes returns file attributes for dataset path"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> ok(datasetsJson()) + + final fs = buildFs(tc) + final path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + when: + final attrs = fs.provider().readAttributes(path, BasicFileAttributes) + + then: + !attrs.isDirectory() + attrs.isRegularFile() + } + + // ---- newDirectoryStream (T023) ---- + + def "newDirectoryStream on root returns org names"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + + final fs = buildFs(tc) + final root = new SeqeraPath(fs, 'seqera://') + + when: + def entries = fs.provider().newDirectoryStream(root, null).toList() + + then: + entries.size() == 1 + entries[0].toString() == 'seqera://acme' + } + + def "newDirectoryStream on org returns workspace names"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + + final fs = buildFs(tc) + final orgPath = new SeqeraPath(fs, 'seqera://acme') + + when: + def entries = fs.provider().newDirectoryStream(orgPath, null).toList() + + then: + entries.size() == 1 + entries[0].toString() == 'seqera://acme/research' + } + + def "newDirectoryStream on workspace returns datasets resource type"() { + given: + def tc = spyTower() + final fs = buildFs(tc) + final wsPath = new SeqeraPath(fs, 'seqera://acme/research') + + when: + def entries = fs.provider().newDirectoryStream(wsPath, null).toList() + + then: + entries.size() == 1 + entries[0].toString() == 'seqera://acme/research/datasets' + } + + def "newDirectoryStream on datasets dir returns dataset names"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> ok(datasetsJson()) + + final fs = buildFs(tc) + final dsDir = new SeqeraPath(fs, 'seqera://acme/research/datasets') + + when: + def entries = fs.provider().newDirectoryStream(dsDir, null).toList() + + then: + entries.size() == 1 + entries[0].toString() == 'seqera://acme/research/datasets/samples' + } + + def "newDirectoryStream on datasets dir with empty workspace returns empty stream"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> + ok(JsonOutput.toJson([datasets: [], totalSize: 0])) + + final fs = buildFs(tc) + final dsDir = new SeqeraPath(fs, 'seqera://acme/research/datasets') + + when: + def entries = fs.provider().newDirectoryStream(dsDir, null).toList() + + then: + entries.isEmpty() + } + + def "newDirectoryStream filter is applied to entries"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> ok(JsonOutput.toJson([datasets: [ + [id: 'ds-1', name: 'samples', version: 1L, mediaType: 'text/csv', workspaceId: 10L, + dateCreated: '2024-01-01T00:00:00Z', lastUpdated: '2024-01-02T00:00:00Z'], + [id: 'ds-2', name: 'results', version: 1L, mediaType: 'text/csv', workspaceId: 10L, + dateCreated: '2024-01-01T00:00:00Z', lastUpdated: '2024-01-02T00:00:00Z'] + ], totalSize: 2])) + + final fs = buildFs(tc) + final dsDir = new SeqeraPath(fs, 'seqera://acme/research/datasets') + final filter = { java.nio.file.Path p -> p.toString().contains('results') } as DirectoryStream.Filter + + when: + def entries = fs.provider().newDirectoryStream(dsDir, filter).toList() + + then: + entries.size() == 1 + entries[0].toString() == 'seqera://acme/research/datasets/results' + } + + // ---- error scenarios (T028) ---- + + def "readAttributes throws NoSuchFileException for unknown org"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + + final fs = buildFs(tc) + final path = new SeqeraPath(fs, 'seqera://unknown-org/research') + + when: + fs.provider().readAttributes(path, BasicFileAttributes) + + then: + thrown(NoSuchFileException) + } + + def "newInputStream throws NoSuchFileException containing dataset name for missing dataset"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> + ok(JsonOutput.toJson([datasets: [], totalSize: 0])) + + final fs = buildFs(tc) + final path = new SeqeraPath(fs, 'seqera://acme/research/datasets/missing-dataset') + + when: + fs.provider().newInputStream(path) + + then: + def e = thrown(NoSuchFileException) + e.file?.contains('missing-dataset') + } + + def "getUserInfo 401 propagates as AbortOperationException"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> new TowerClient.Response(401, 'Unauthorized') + + final fs = buildFs(tc) + final path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + when: + fs.provider().newInputStream(path) + + then: + thrown(AbortOperationException) + } + + def "getUserInfo 403 propagates as AccessDeniedException"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> new TowerClient.Response(403, 'Forbidden') + + final fs = buildFs(tc) + final path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + when: + fs.provider().newInputStream(path) + + then: + thrown(AccessDeniedException) + } + + def "SeqeraPath constructor throws InvalidPathException for path with empty workspace segment"() { + given: + def tc = spyTower() + final fs = buildFs(tc) + + when: + new SeqeraPath(fs, 'seqera://acme//datasets/samples') + + then: + thrown(InvalidPathException) + } + + // ---- newFileSystem contract ---- + + def "newFileSystem throws FileSystemAlreadyExistsException when filesystem exists"() { + given: 'a provider with an existing filesystem' + def tc = spyTower() + def provider = new SeqeraFileSystemProvider() + def fs = new SeqeraFileSystem(provider, new SeqeraDatasetClient(tc)) + provider.@fileSystem = fs + + when: + provider.newFileSystem(new URI('seqera://test'), [:]) + + then: + thrown(FileSystemAlreadyExistsException) + } +} diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/SeqeraFileSystemTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/SeqeraFileSystemTest.groovy new file mode 100644 index 0000000000..c7db27a534 --- /dev/null +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/SeqeraFileSystemTest.groovy @@ -0,0 +1,204 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.fs + +import java.nio.file.NoSuchFileException + +import groovy.json.JsonOutput +import io.seqera.tower.plugin.TowerClient +import io.seqera.tower.plugin.dataset.SeqeraDatasetClient +import spock.lang.Specification + +/** + * Tests for {@link SeqeraFileSystem} caching and workspace resolution using a mock {@link TowerClient}. + */ +class SeqeraFileSystemTest extends Specification { + + private static final String ENDPOINT = 'https://api.example.com' + + private TowerClient spyTower() { + def tc = Spy(TowerClient) + tc.@endpoint = ENDPOINT + return tc + } + + private static TowerClient.Response ok(String body) { + new TowerClient.Response(200, body) + } + + private static String userInfoJson() { + JsonOutput.toJson([user: [id: 42L, userName: 'testuser']]) + } + + private static String workspacesJson() { + JsonOutput.toJson([orgsAndWorkspaces: [ + [orgId: 1L, orgName: 'acme', workspaceId: 10L, workspaceName: 'research', workspaceFullName: 'acme/research'], + [orgId: 1L, orgName: 'acme', workspaceId: 20L, workspaceName: 'dev', workspaceFullName: 'acme/dev'], + [orgId: 2L, orgName: 'other', workspaceId: 30L, workspaceName: 'ws', workspaceFullName: 'other/ws'] + ]]) + } + + private SeqeraFileSystem buildFs(TowerClient tc) { + new SeqeraFileSystem(new SeqeraFileSystemProvider(), new SeqeraDatasetClient(tc)) + } + + // ---- cache loading ---- + + def "loadOrgWorkspaceCache is called only once across multiple invocations"() { + given: + def tc = spyTower() + final fs = buildFs(tc) + + when: + fs.loadOrgWorkspaceCache() + fs.loadOrgWorkspaceCache() + + then: + 1 * tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + 1 * tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + } + + def "listOrgNames returns distinct org names from cache"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + final fs = buildFs(tc) + + when: + def orgs = fs.listOrgNames() + + then: + orgs.size() == 2 + orgs.contains('acme') + orgs.contains('other') + } + + def "listWorkspaceNames returns workspace names for the given org"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + final fs = buildFs(tc) + + when: + def names = fs.listWorkspaceNames('acme') + + then: + names.size() == 2 + names.containsAll(['research', 'dev']) + } + + // ---- resolveWorkspaceId ---- + + def "resolveWorkspaceId returns correct ID for known org and workspace"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + final fs = buildFs(tc) + + when: + def id = fs.resolveWorkspaceId('acme', 'research') + + then: + id == 10L + } + + def "resolveWorkspaceId throws NoSuchFileException for unknown org"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + final fs = buildFs(tc) + + when: + fs.resolveWorkspaceId('unknown-org', 'research') + + then: + thrown(NoSuchFileException) + } + + def "resolveWorkspaceId throws NoSuchFileException for unknown workspace within known org"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/user-info") >> ok(userInfoJson()) + tc.sendApiRequest("${ENDPOINT}/user/42/workspaces") >> ok(workspacesJson()) + final fs = buildFs(tc) + + when: + fs.resolveWorkspaceId('acme', 'no-such-ws') + + then: + thrown(NoSuchFileException) + } + + // ---- dataset cache ---- + + def "resolveDatasets populates cache and returns datasets"() { + given: + def tc = spyTower() + tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> + ok(JsonOutput.toJson([datasets: [ + [id: 'ds-1', name: 'samples', version: 1L, mediaType: 'text/csv', + dateCreated: '2024-01-01T00:00:00Z', lastUpdated: '2024-01-02T00:00:00Z'] + ], totalSize: 1])) + final fs = buildFs(tc) + + when: + def datasets = fs.resolveDatasets(10L) + + then: + datasets.size() == 1 + datasets[0].name == 'samples' + } + + def "resolveDatasets returns cached result on second call without extra API request"() { + given: + def tc = spyTower() + final datasetsJson = JsonOutput.toJson([datasets: [ + [id: 'ds-1', name: 'samples', version: 1L, mediaType: 'text/csv', + dateCreated: '2024-01-01T00:00:00Z', lastUpdated: '2024-01-02T00:00:00Z'] + ], totalSize: 1]) + final fs = buildFs(tc) + + when: + fs.resolveDatasets(10L) + fs.resolveDatasets(10L) + + then: + 1 * tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> ok(datasetsJson) + } + + def "invalidateDatasetCache forces re-fetch on next resolveDatasets call"() { + given: + def tc = spyTower() + final datasetsJson = JsonOutput.toJson([datasets: [ + [id: 'ds-1', name: 'samples', version: 1L, mediaType: 'text/csv', + dateCreated: '2024-01-01T00:00:00Z', lastUpdated: '2024-01-02T00:00:00Z'] + ], totalSize: 1]) + final fs = buildFs(tc) + + when: + fs.resolveDatasets(10L) + fs.invalidateDatasetCache(10L) + fs.resolveDatasets(10L) + + then: + 2 * tc.sendApiRequest("${ENDPOINT}/datasets?workspaceId=10") >> ok(datasetsJson) + } +} diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/SeqeraPathTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/SeqeraPathTest.groovy new file mode 100644 index 0000000000..69a5e26915 --- /dev/null +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/fs/SeqeraPathTest.groovy @@ -0,0 +1,521 @@ +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.seqera.tower.plugin.fs + +import io.seqera.tower.plugin.dataset.SeqeraDatasetClient +import spock.lang.Specification + +/** + * Unit tests for {@link SeqeraPath}. + */ +class SeqeraPathTest extends Specification { + + private SeqeraFileSystem mockFs() { + def provider = new SeqeraFileSystemProvider() + def client = Mock(SeqeraDatasetClient) + return new SeqeraFileSystem(provider, client) + } + + def "depth 0 - root path"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://') + + expect: + path.depth() == 0 + path.isDirectory() + !path.isRegularFile() + path.org == null + path.workspace == null + } + + def "depth 1 - org path"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme') + + expect: + path.depth() == 1 + path.isDirectory() + !path.isRegularFile() + path.org == 'acme' + path.workspace == null + } + + def "depth 2 - workspace path"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research') + + expect: + path.depth() == 2 + path.isDirectory() + path.org == 'acme' + path.workspace == 'research' + path.resourceType == null + } + + def "depth 3 - resource type path"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets') + + expect: + path.depth() == 3 + path.isDirectory() + path.org == 'acme' + path.workspace == 'research' + path.resourceType == 'datasets' + path.datasetName == null + } + + def "depth 4 - dataset file path"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + expect: + path.depth() == 4 + !path.isDirectory() + path.isRegularFile() + path.org == 'acme' + path.workspace == 'research' + path.resourceType == 'datasets' + path.datasetName == 'samples' + path.version == null + } + + def "depth 4 - dataset with pinned version"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples@2') + + expect: + path.depth() == 4 + path.datasetName == 'samples' + path.version == '2' + } + + def "toUri round-trip - no version"() { + given: + def fs = mockFs() + def uri = 'seqera://acme/research/datasets/samples' + def path = new SeqeraPath(fs, uri) + + expect: + path.toUri().toString() == uri + path.toString() == uri + } + + def "toUri round-trip - with version"() { + given: + def fs = mockFs() + def uri = 'seqera://acme/research/datasets/samples@2' + def path = new SeqeraPath(fs, uri) + + expect: + path.toUri().toString() == uri + } + + def "getParent - depth 4 returns depth 3"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + when: + def parent = path.getParent() + + then: + parent.toString() == 'seqera://acme/research/datasets' + (parent as SeqeraPath).depth() == 3 + } + + def "getParent - depth 3 returns depth 2"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets') + + expect: + path.getParent().toString() == 'seqera://acme/research' + } + + def "getParent - depth 1 returns depth 0 root"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme') + + expect: + path.getParent().toString() == 'seqera://' + } + + def "getParent - root returns null"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://') + + expect: + path.getParent() == null + } + + def "resolve - appends segment to workspace"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research') + + when: + def resolved = path.resolve('datasets') + + then: + resolved.toString() == 'seqera://acme/research/datasets' + (resolved as SeqeraPath).depth() == 3 + } + + def "resolve - appends dataset name to resource type"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets') + + when: + def resolved = path.resolve('my-dataset') + + then: + resolved.toString() == 'seqera://acme/research/datasets/my-dataset' + } + + def "resolve - dataset name with version"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets') + + when: + def resolved = path.resolve('samples@3') + + then: + (resolved as SeqeraPath).datasetName == 'samples' + (resolved as SeqeraPath).version == '3' + } + + def "equality and hashCode"() { + given: + def fs = mockFs() + def p1 = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + def p2 = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + def p3 = new SeqeraPath(fs, 'seqera://acme/research/datasets/other') + + expect: + p1 == p2 + p1.hashCode() == p2.hashCode() + p1 != p3 + } + + def "isAbsolute always true"() { + given: + def fs = mockFs() + + expect: + new SeqeraPath(fs, 'seqera://acme').isAbsolute() + new SeqeraPath(fs, 'seqera://').isAbsolute() + } + + def "getNameCount equals depth"() { + given: + def fs = mockFs() + + expect: + new SeqeraPath(fs, 'seqera://').nameCount == 0 + new SeqeraPath(fs, 'seqera://acme').nameCount == 1 + new SeqeraPath(fs, 'seqera://acme/research/datasets/samples').nameCount == 4 + } + + // ---- relativize ---- + + def "relativize returns correct relative path string"() { + given: + def fs = mockFs() + + expect: + new SeqeraPath(fs, base).relativize(new SeqeraPath(fs, other)).toString() == expected + + where: + base | other | expected + 'seqera://acme' | 'seqera://acme/research' | 'research' + 'seqera://acme/research' | 'seqera://acme/research/datasets' | 'datasets' + 'seqera://acme/research' | 'seqera://acme/research/datasets/samples' | 'datasets/samples' + 'seqera://acme/research/datasets' | 'seqera://acme/research/datasets/samples' | 'samples' + 'seqera://acme/research/datasets/samples' | 'seqera://acme/research/datasets/samples' | '' + } + + def "relativize result round-trips through resolve"() { + given: + def fs = mockFs() + def base = new SeqeraPath(fs, 'seqera://acme/research') + def target = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + when: + def rel = base.relativize(target) + def restored = base.resolve(rel) + + then: + rel.toString() == 'datasets/samples' + !rel.isAbsolute() + restored == target + } + + def "relativize produces '..' segments for upward traversal"() { + given: + def fs = mockFs() + + expect: + new SeqeraPath(fs, base).relativize(new SeqeraPath(fs, other)).toString() == expected + + where: + base | other | expected + 'seqera://acme/research' | 'seqera://acme/dev' | '../dev' + 'seqera://acme/research/datasets' | 'seqera://acme/dev' | '../../dev' + 'seqera://acme' | 'seqera://other' | '../other' + 'seqera://acme/ws1' | 'seqera://acme/ws2' | '../ws2' + 'seqera://acme/research/datasets/samples' | 'seqera://acme/research/datasets/other' | '../other' + } + + // ---- multi-segment resolve ---- + + def "resolve with multi-segment string builds correct path"() { + given: + def fs = mockFs() + def base = new SeqeraPath(fs, 'seqera://acme/research') + + expect: + base.resolve('datasets/samples').toString() == 'seqera://acme/research/datasets/samples' + base.resolve('datasets').toString() == 'seqera://acme/research/datasets' + } + + def "resolve with absolute seqera URI returns that URI"() { + given: + def fs = mockFs() + def base = new SeqeraPath(fs, 'seqera://acme/research') + def absolute = 'seqera://other/ws/datasets/report' + + expect: + base.resolve(absolute).toString() == absolute + } + + def "isAbsolute is false for relative paths produced by relativize"() { + given: + def fs = mockFs() + def rel = new SeqeraPath(fs, 'seqera://acme').relativize(new SeqeraPath(fs, 'seqera://acme/research')) + + expect: + !rel.isAbsolute() + rel.toString() == 'research' + } + + // ---- getFileName ---- + + def "getFileName returns relative path for each depth"() { + given: + def fs = mockFs() + + expect: + new SeqeraPath(fs, 'seqera://').getFileName() == null + new SeqeraPath(fs, 'seqera://acme').getFileName().toString() == 'acme' + !new SeqeraPath(fs, 'seqera://acme').getFileName().isAbsolute() + new SeqeraPath(fs, 'seqera://acme/research').getFileName().toString() == 'research' + new SeqeraPath(fs, 'seqera://acme/research/datasets').getFileName().toString() == 'datasets' + new SeqeraPath(fs, 'seqera://acme/research/datasets/samples').getFileName().toString() == 'samples' + new SeqeraPath(fs, 'seqera://acme/research/datasets/samples@2').getFileName().toString() == 'samples@2' + } + + def "getFileName is not absolute (uses relative constructor)"() { + given: + def fs = mockFs() + def name = new SeqeraPath(fs, 'seqera://acme/research').getFileName() + + expect: + !name.isAbsolute() + name.toString() == 'research' + name.getFileSystem() == null + } + + // ---- asUri ---- + + def "asUri - valid full path round-trips"() { + expect: + SeqeraPath.asUri('seqera://acme/research/datasets/samples').toString() == 'seqera://acme/research/datasets/samples' + SeqeraPath.asUri('seqera://acme/research').toString() == 'seqera://acme/research' + } + + def "asUri - empty path returns root URI"() { + expect: + SeqeraPath.asUri('seqera://').toString() == 'seqera:///' + } + + def "asUri - path starting with dot has dot stripped"() { + expect: + // seqera://. → strips dot → seqera:// → hits empty-path case → seqera:/// + SeqeraPath.asUri('seqera://.').toString() == 'seqera:///' + // seqera://./foo/bar → strips dot only (substring from index 10) → seqera:///foo/bar + SeqeraPath.asUri('seqera://./foo/bar').toString() == 'seqera://foo/bar' + } + + def "asUri - triple slash path throws IllegalArgumentException"() { + when: + SeqeraPath.asUri('seqera:///something') + + then: + thrown(IllegalArgumentException) + } + + def "asUri - missing protocol prefix throws IllegalArgumentException"() { + when: + SeqeraPath.asUri('s3://bucket/key') + + then: + thrown(IllegalArgumentException) + } + + def "asUri - null or empty throws IllegalArgumentException"() { + when: + SeqeraPath.asUri(null) + + then: + thrown(IllegalArgumentException) + + when: + SeqeraPath.asUri('') + + then: + thrown(IllegalArgumentException) + } + + // ---- startsWith ---- + + def "startsWith - same path returns true"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + expect: + path.startsWith(new SeqeraPath(fs, 'seqera://acme/research/datasets/samples')) + } + + def "startsWith - prefix path returns true"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + expect: + path.startsWith(new SeqeraPath(fs, 'seqera://acme')) + path.startsWith(new SeqeraPath(fs, 'seqera://acme/research')) + path.startsWith(new SeqeraPath(fs, 'seqera://')) + } + + def "startsWith - component-wise not substring"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme-corp/research/datasets/samples') + + expect: 'acme is a substring prefix of acme-corp but not a component prefix' + !path.startsWith(new SeqeraPath(fs, 'seqera://acme')) + } + + def "startsWith - longer path returns false"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme') + + expect: + !path.startsWith(new SeqeraPath(fs, 'seqera://acme/research')) + } + + def "startsWith with string"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + expect: + path.startsWith('seqera://acme') + !path.startsWith('seqera://acm') + } + + // ---- endsWith ---- + + def "endsWith - absolute path requires exact match"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + expect: + path.endsWith(new SeqeraPath(fs, 'seqera://acme/research/datasets/samples')) + !path.endsWith(new SeqeraPath(fs, 'seqera://acme/research')) + } + + def "endsWith - relative path matches trailing components"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + expect: + path.endsWith(new SeqeraPath('samples')) + path.endsWith(new SeqeraPath('datasets/samples')) + !path.endsWith(new SeqeraPath('other')) + } + + def "endsWith - component-wise not substring"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets/my-samples') + + expect: 'samples is a substring suffix of my-samples but not a component match' + !path.endsWith(new SeqeraPath('samples')) + } + + // ---- iterator ---- + + def "iterator returns relative name components"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme/research/datasets/samples') + + when: + def parts = path.iterator().collect { it.toString() } + + then: + parts == ['acme', 'research', 'datasets', 'samples'] + path.iterator().every { !it.isAbsolute() } + } + + def "iterator on root returns empty"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://') + + expect: + !path.iterator().hasNext() + } + + def "iterator on org returns single element"() { + given: + def fs = mockFs() + def path = new SeqeraPath(fs, 'seqera://acme') + + when: + def parts = path.iterator().collect { it.toString() } + + then: + parts == ['acme'] + } +} diff --git a/plugins/nf-tower/src/test/io/seqera/tower/plugin/launch/LaunchCommandImplTest.groovy b/plugins/nf-tower/src/test/io/seqera/tower/plugin/launch/LaunchCommandImplTest.groovy index 044928c4d1..e58053986b 100644 --- a/plugins/nf-tower/src/test/io/seqera/tower/plugin/launch/LaunchCommandImplTest.groovy +++ b/plugins/nf-tower/src/test/io/seqera/tower/plugin/launch/LaunchCommandImplTest.groovy @@ -17,7 +17,7 @@ package io.seqera.tower.plugin.launch import io.seqera.http.HxClient -import io.seqera.tower.plugin.TowerCommonApi +import io.seqera.tower.plugin.TowerClient import nextflow.cli.CmdLaunch import nextflow.exception.AbortOperationException import org.junit.Rule @@ -379,11 +379,12 @@ class LaunchCommandImplTest extends Specification { 'tower.accessToken': 'test-token', 'tower.endpoint': 'https://api.cloud.seqera.io' ] - def commonApi = Mock(TowerCommonApi){ - getUserInfo(_, _) >> [name: 'testuser', id: '123'] - getUserWorkspaceDetails(_,_, _, _) >> null + def client = Mock(TowerClient){ + getUserInfo() >> [name: 'testuser', id: '123'] + getUserWorkspaceDetails(_,_) >> null } - def cmd = Spy(new LaunchCommandImpl(commonApi)) + def cmd = Spy(new LaunchCommandImpl()) + cmd.createTowerClient(_,_) >> client cmd.readConfig() >> config cmd.resolveWorkspaceId(_, _, _, _) >> null cmd.resolveComputeEnvironment(_,_, _, _, _) >> [id: 'ce-123', name: 'test-ce', workDir: 's3://bucket/work'] @@ -405,11 +406,12 @@ class LaunchCommandImplTest extends Specification { def 'should use default endpoint when not configured'() { given: def config = ['tower.accessToken': 'test-token'] - def commonApi = Mock(TowerCommonApi){ - getUserInfo(_, _) >> [name: 'testuser', id: '123'] - getUserWorkspaceDetails(_, _, _, _) >> null + def client = Mock(TowerClient){ + getUserInfo() >> [name: 'testuser', id: '123'] + getUserWorkspaceDetails(_, _) >> null } - def cmd = Spy(new LaunchCommandImpl(commonApi)) + def cmd = Spy(new LaunchCommandImpl()) + cmd.createTowerClient(_,_) >> client cmd.readConfig() >> config cmd.resolveWorkspaceId(_, _, _, _) >> null cmd.resolveComputeEnvironment(_,_, _, _, _) >> [id: 'ce-123', name: 'test-ce', workDir: 's3://bucket/work'] @@ -427,11 +429,12 @@ class LaunchCommandImplTest extends Specification { given: def config = ['tower.accessToken': 'test-token', 'tower.workspaceId': 12345] - def commonApi = Mock(TowerCommonApi){ - getUserInfo(_, _) >> [name: 'testuser', id: '123'] - getUserWorkspaceDetails(_, _, _, _) >> [orgName: 'TestOrg', workspaceName: 'TestWS'] - } - def cmd = Spy(new LaunchCommandImpl(commonApi)) + def client = Mock(TowerClient){ + getUserInfo() >> [name: 'testuser', id: '123'] + getUserWorkspaceDetails(_, _) >> [orgName: 'TestOrg', workspaceName: 'TestWS'] + } + def cmd = Spy(new LaunchCommandImpl()) + cmd.createTowerClient(_,_) >> client cmd.readConfig() >> config cmd.resolveWorkspaceId(_, _, _, _) >> 12345L cmd.resolveComputeEnvironment(_, _, _, _, _) >> [id: 'ce-123', name: 'test-ce', workDir: 's3://bucket/work'] @@ -456,10 +459,10 @@ class LaunchCommandImplTest extends Specification { [id: 'ce-1', name: 'primary-ce', primary: true], [id: 'ce-2', name: 'secondary-ce', primary: false] ] - cmd.listComputeEnvironments(_, _, _) >> computeEnvs + cmd.listComputeEnvironments(_, _) >> computeEnvs when: - def result = cmd.findComputeEnv(Mock(HxClient), 'secondary-ce', null, 'endpoint') + def result = cmd.findComputeEnv(Mock(TowerClient), 'secondary-ce', null) then: result.id == 'ce-2' @@ -473,10 +476,10 @@ class LaunchCommandImplTest extends Specification { [id: 'ce-1', name: 'primary-ce', primary: true], [id: 'ce-2', name: 'secondary-ce', primary: false] ] - cmd.listComputeEnvironments(_, _, _) >> computeEnvs + cmd.listComputeEnvironments(_, _) >> computeEnvs when: - def result = cmd.findComputeEnv( Mock(HxClient) ,null, null, 'endpoint') + def result = cmd.findComputeEnv( Mock(TowerClient) ,null, null) then: result.id == 'ce-1' @@ -487,10 +490,10 @@ class LaunchCommandImplTest extends Specification { def 'should return null when compute environment not found'() { given: def cmd = Spy(LaunchCommandImpl) - cmd.listComputeEnvironments(_, _, _) >> [] + cmd.listComputeEnvironments(_, _) >> [] when: - def result = cmd.findComputeEnv(Mock(HxClient), 'nonexistent', null, 'endpoint') + def result = cmd.findComputeEnv(Mock(TowerClient), 'nonexistent', null) then: result == null @@ -500,7 +503,7 @@ class LaunchCommandImplTest extends Specification { given: def cmd = Spy(LaunchCommandImpl) // Mock findComputeEnv to return null (not found) - cmd.findComputeEnv(_,'nonexistent', null, 'https://api.cloud.seqera.io') >> null + cmd.findComputeEnv(_,'nonexistent', null) >> null when: cmd.resolveComputeEnvironment(null, 'nonexistent', null, 'token', 'https://api.cloud.seqera.io') @@ -514,7 +517,8 @@ class LaunchCommandImplTest extends Specification { given: def cmd = Spy(LaunchCommandImpl) // Mock findComputeEnv to return null (no primary found) - cmd.findComputeEnv(_ , null, null, 'https://api.cloud.seqera.io') >> null + cmd.createTowerClient(_,_) >> Mock(TowerClient) + cmd.findComputeEnv(_ , null, null) >> null when: cmd.resolveComputeEnvironment(null, null, null, 'token', 'https://api.cloud.seqera.io') @@ -687,11 +691,12 @@ class LaunchCommandImplTest extends Specification { [workspaceId: 111, workspaceName: 'ws1'], [workspaceId: 222, workspaceName: 'ws2'] ] - def commonApi = Mock(TowerCommonApi) { - getUserInfo(_, _) >> [id: 'user-123'] + def client = Mock(TowerClient) { + getUserInfo() >> [id: 'user-123'] } - def cmd = Spy(new LaunchCommandImpl(commonApi)) - cmd.listUserWorkspaces(_, _, _) >> workspaces + def cmd = Spy(new LaunchCommandImpl()) + cmd.createTowerClient(_,_) >> client + cmd.listUserWorkspaces(_, _) >> workspaces when: def workspaceId = cmd.resolveWorkspaceId(config, 'ws2', 'token', 'endpoint') @@ -703,10 +708,11 @@ class LaunchCommandImplTest extends Specification { def 'should throw error when workspace not found by name'() { given: def config = [:] - def commonApi = Mock(TowerCommonApi) { - getUserInfo(_, _) >> [id: 'user-123'] + def client = Mock(TowerClient) { + getUserInfo() >> [id: 'user-123'] } - def cmd = Spy(new LaunchCommandImpl(commonApi)) + def cmd = Spy(new LaunchCommandImpl()) + cmd.createTowerClient(_,_) >> client cmd.listUserWorkspaces(_, _, _) >> [] when: diff --git a/tests/checks/seqera-dataset.nf/.checks b/tests/checks/seqera-dataset.nf/.checks new file mode 100644 index 0000000000..19f82b6c63 --- /dev/null +++ b/tests/checks/seqera-dataset.nf/.checks @@ -0,0 +1,17 @@ +set -e +export NXF_PLUGINS_DEFAULT=nf-tower + +# Skip test if Seqera Platform token is missing +if [[ ! $TOWER_ACCESS_TOKEN ]]; then + echo "Skip seqera-dataset test since TOWER_ACCESS_TOKEN is not available" + exit 0 +fi + +# +# run normal mode +# +$NXF_RUN | tee stdout + +[[ `grep INFO .nextflow.log | grep -c 'Submitted process > TEST'` == 1 ]] || false +[[ `grep -c 'patient,sex,status,sample,lane,fastq_1,fastq_2' stdout` == 1 ]] || false + diff --git a/tests/seqera-dataset.nf b/tests/seqera-dataset.nf new file mode 100644 index 0000000000..fa122afbb3 --- /dev/null +++ b/tests/seqera-dataset.nf @@ -0,0 +1,33 @@ +#!/usr/bin/env nextflow +/* + * Copyright 2013-2026, Seqera Labs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +params.dataset = 'seqera://seqeralabs/showcase/datasets/sarek_samples' + +process TEST { + input: + path(file) + output: + stdout + script: + """ + cat $file + """ +} + +workflow { + TEST(file(params.dataset)).view() +}