The raw-reads-processing service currently invokes ENA ReadTools in quick mode, which validates only the first 100,000 reads per file. Malformed records occurring later may therefore go undetected.
Deacon subsequently reads the complete files and detects some structural failures, but it does not necessarily enforce every ReadTools check.
ReadTools provides --full. But it also sizes internal Bloom filters according to that larger limit, substantially increasing memory usage.
In a local benchmark using the deployed ReadTools jar, a 43 MB paired input used:
| Mode |
Runtime |
Peak RSS |
| Quick |
1.37s |
0.55 GB |
| Full |
2.34s |
2.2 GB |
We should investigate a resource-efficient way to validate entire files.
The raw-reads-processing service currently invokes ENA ReadTools in quick mode, which validates only the first 100,000 reads per file. Malformed records occurring later may therefore go undetected.
Deacon subsequently reads the complete files and detects some structural failures, but it does not necessarily enforce every ReadTools check.
ReadTools provides
--full. But it also sizes internal Bloom filters according to that larger limit, substantially increasing memory usage.In a local benchmark using the deployed ReadTools jar, a 43 MB paired input used:
We should investigate a resource-efficient way to validate entire files.