Adding gfaidx backend support for indexed genome graph queries - #136
Adding gfaidx backend support for indexed genome graph queries#136fawaz-dabbaghieh wants to merge 7 commits into
Conversation
… ticket ID, HTTP endpoints in backend/gfaidxapi.go, Tests covering validation, command construction, submission, polling, and result retrieval, adding POST request for gfaidx subgraph and region extraction
… and coordinates for a graph, parse the region paths tsv and return a JSON with this information, add discovery, parsing, and command-construction tests
… source code in the gfaidx repository, because gfaidx doesn't really use much dependencies, it should be easy and straightforward to compile, seems to work locally on my mack when I run the docker image I created and gfaidx is compiling without problems and was able to make API requests
| r.HandleFunc("/ticket/folddisco", ticketFolddiscoHandlerFunc).Methods("POST") | ||
| r.HandleFunc("/ticket/riboseek", ticketRiboseekHandlerFunc).Methods("POST") | ||
| } | ||
| // Register the optional gfaidx routes alongside, but independently from, existing app routes. |
There was a problem hiding this comment.
Gate behind:
if config.App == AppFoldseek && config.Gfaidx != nil {
...
So you cant misconfigure a CF server
|
|
||
| FROM $downloader AS downloader | ||
|
|
||
| # Build gfaidx from a pinned source revision on Docker's target platform. This |
There was a problem hiding this comment.
Can we download built binaries? None of the other projects build their binary on the fly
There was a problem hiding this comment.
Yeah, I have to make some changes to the gfaidx repo and build there, I'll look into this.
| @@ -0,0 +1,300 @@ | |||
| package main | |||
There was a problem hiding this comment.
can we move all files to backend/gfaidx/...?
…the problem with Foldseek Interface, I just change the docker config to point to foldseek binary, I wasn't sure how to remove it cleanly, as there are mentions of config.Paths.FoldseekInterface in several places in the backend
|
Made a new commit, the main changes were gating gfaidx behind |
… so far, but I haven't been testing Foldseek related commands, only gfaidx
|
Made another commit removing the FoldseekInterface part as it's now integrated in Foldseek, it still works for me, but I'm only testing the gfaidx parts locally |
|
Can you delete the code fully? I would like to not keep dead comments |
…hecked the parameters against foldseek binary and things seems to work
|
OK, I removed dead comments and references to interface search, now it's just foldseek |
2 participants
The main idea here is that my graph indexer allowes basically random access through the indexed graphs, and I implemented a simple API on the frontend app that I've been working on for graph visualization. Martin then suggested that I have
gfaidxintegrated in the MMseqs2-App API, so we can offer my frontend graph visualizer as a service as well for user to explore the indexed graphs without having to download the big files.What is Implemented
These functionalities were added:
I reuse the existing ticket, queue, worker, status, and result-storage infrastructure. So my frontend will basically receive the job ID, and checks for the status every second, once it's done, it will retrieve the graph and visualize it.
Only the get_region and get_subgraph gfaidx commands are accepted. Command arguments are constructed by the backend from validated JSON fields; clients cannot submit arbitrary command-line arguments or filesystem paths.
Graphs Databases
Similar to how it works now with a .params file, I will add a .params for each indexed graph, and the front end will show the names of the available indexed graph. Example:
{ "name": "HPRC v2 Minigraph-Cactus", "description": "HPRC release 2 pangenome graph", "version": "v2", "path": "hprc_v2_mc.indexed.gfa.gz" }Docker
Dockerfile.backend now builds and installs gfaidx into the existing backend image. The source is pinned to a checksummed revision reporting gfaidx 1.9.7, which contains the query options used by the new backend. The build works for both ARM64 and AMD64 targets.
docker-compose.gfaidx.ymlenables the integration as it configures:This is optional, so deployment can be done with only MMseqs2 and Foldseek do not need to enable or configure gfaidx.
Compatiblitiy
gfaidxintegration uses its own routes, job types, worker dispatch and other configurations, I tried to keep this as separate as possible while using as much as possible from the already existing backend structure. But nothing should change for MMseqs2 and Foldseeks jobs. If gfaidx is not configured, its routes are not registered and the application continues to operate as before.Testing
I tested this on my mac, I tested the backend separately (compiled the go backend and ran the binary), also tested docker ARM64 and simulated AMD64 builds, then tested the graph discovery and some simple graph requests, all seems to work.
What is still needed
This needs to be tested on the servers, but I don't forsee much problems in compiling gfaidx, as I don't use much dependencies other than zlib, then we need to offer the frontend, right now the frontend app is here (https://github.com/fawaz-dabbaghieh/graphviz_wasm), I made some changes for the frontend to be able to make the correct requests that this backend accepts, but probably still needs a bit of work. This frontend has the layout core as a webassembly object, an old and very simple version of the app lives here if you're interested in checking it out (https://hulk.mmseqs.com/fawaz/bandagejs/)