Reduce gunzip() peak memory to ~1x the output size - #19
Open
peximus-yo wants to merge 1 commit into
Open
Conversation
Decompressing a large input previously peaked at close to 3x the decompressed size: the StringInfo output buffer grows by doubling (each enlargement holding old + new copies transiently), and the finished buffer was then copied once more into a freshly allocated bytea. On memory-constrained systems (containers with cgroup limits, small instances) that transient spike can draw the OOM killer, which SIGKILLs the backend and forces the whole cluster through crash recovery -- even for compressed inputs of only a few MB, since highly repetitive data (text, CSV) commonly inflates 20-100x. Two changes bound the peak to roughly the output size itself: * Use the gzip ISIZE trailer (uncompressed size mod 2^32) as a pre-allocation hint, clamped to gzip.max_size and MaxAllocSize, so the buffer is sized once up front instead of via repeated enlarge-and-copy cycles. The value is attacker-controlled, so it is only ever a hint; the gzip.max_size enforcement in the inflate loop is unchanged. * Reserve the varlena header at the head of the StringInfo and return the buffer directly as the result bytea, removing the final full-size copy. Measured on a 300MB decompression (11KB compressed): peak backend RSS drops from ~600MB to ~304MB. Also add CHECK_FOR_INTERRUPTS() to the deflate/inflate loops so large (de)compressions respond to cancellation, make the zlib alloc shim overflow-safe, and document gzip.max_size in the README.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
During an Oracle→PostgreSQL migration we stored gzipped XML documents in
byteacolumns and decompressed them withgunzip()in queries. Inputs of only 5–10MB compressed would intermittently get the backend killed withsignal 9and force the whole cluster through crash recovery. The root cause: text/XML routinely inflates 10–100x, andpg_gunzip()peaks at close to 3x the decompressed size — the StringInfo output buffer grows by doubling (each enlargement transiently holding old + new copies), and the finished buffer is then copied once more into a freshly palloc'dbytea. On memory-constrained deployments (cgroup limits, containers, small instances) that transient spike invokes the kernel OOM killer, which SIGKILLs the backend. Thegzip.max_sizeguard added in v1.1.0 bounds the output, but the ~3x amplification still applies to whatever limit is configured.Changes
Two changes bound the peak to roughly the output size itself:
gzip.max_sizeandMaxAllocSize— and the existinggzip.max_sizeenforcement inside the inflate loop is unchanged. This replaces the O(log n) enlarge-and-copy cycles with a single up-front allocation.byteain place. ReserveVARHDRSZbytes at the head of the StringInfo, stamp the varlena header at the end, and return the buffer directly — removing the final full-size copy.Also:
CHECK_FOR_INTERRUPTS()in the deflate/inflate loops, so large (de)compressions respond to query cancellation.(Size)items * itemsize).gzip.max_size(currently undocumented) and the zip-bomb/OOM hazard.Measurements
Peak backend RSS (
log_executor_stats, fresh connection per query), PostgreSQL 17, singlegunzip()call:Reproduced end-to-end in a Docker container (
postgres:17.4,--memory=1g,gzip.max_size=-1): master gets OOM-killed (server process was terminated by signal 9) from 500 MB of output onward; this branch completes up to 700 MB, md5-verified against the source files. Withgzip.max_sizeset the failure mode is a clean cancellableERRORin both cases — the two mechanisms are complementary.Behavior is unchanged for multi-member streams, truncated input, and corrupted trailers (a forged ISIZE only affects the initial allocation size, never the enforced limit), and the regression suite passes as-is.