`ValueError: I/O operation on closed file` when parsing feeds with leading newline in multiprocessing

[Alvaro-Kothe]

When parsing an RSS feed string that starts with a newline before the XML declaration inside a ProcessPoolExecutor, feedparser.parse returns a FeedParserDict object that cannot be pickled without raising:

ValueError: I/O operation on closed file

This error happens during the result pickling step after the worker finishes, not directly inside feedparser.parse.

Minimal Reproducible Example

import feedparser
from concurrent.futures import ProcessPoolExecutor

FEED_STR = """
<?xml version='1.0' encoding='UTF-8'?>
<rss version="2.0">
<channel>
<title>Foo</title>
<link>https://foo.com/</link>
<item><title>Title 1</title><link>https://foo.com/1</link><pubDate>Thu, 05 Jun 2025 18:27:58 -0000</pubDate></item>
</channel>
</rss>
"""

def parse_and_return_full(raw_feed: str):
    return feedparser.parse(raw_feed)

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        future = pool.submit(parse_and_return_full, FEED_STR)
        result = future.result()

Removing the initial newline from FEED_STR prevents the error.

[daaquan]

I reproduced this on 6.0.14 and traced it. There are two independent defects; the leading newline only triggers the first one.

1. convert_to_utf8 emits two XML declarations when the document starts with whitespace

RE_XML_DECLARATION is anchored at the start of the string with no allowance for leading whitespace — feedparser/encodings.py:68 (main @ a22c552):

RE_XML_DECLARATION = re.compile(r"^<\?xml[^>]*?>")

and encodings.py:306-310:

new_declaration = """<?xml version='1.0' encoding='utf-8'?>"""
if RE_XML_DECLARATION.search(text):
    text = RE_XML_DECLARATION.sub(new_declaration, text)
else:
    text = new_declaration + "\n" + text

With a leading newline the pattern does not match, so the else branch prepends a declaration instead of replacing the existing one:

b"<?xml version='1.0' encoding='utf-8'?>\n\n<?xml version='1.0' encoding='UTF-8'?>"

expat then fails with <unknown>:3:0: XML or text declaration not at start of entity, bozo is set, and LooseFeedParser takes over — which is why entries is still populated and nothing looks wrong until you try to serialize the result. I believe this is also the cause of #508.

2. The stored bozo_exception is not picklable

feedparser/api.py:330 stores the SAX exception as-is:

result["bozo_exception"] = feed_parser.exc or e

SAXParseException keeps _locator, which is the ExpatParser, which keeps its InputSource, which keeps the BytesIO. expat closes that stream on the parse error, so pickle walks

exc -> ExpatParser -> InputSource -> BytesIO(closed=True)

and raises ValueError: I/O operation on closed file. ProcessPoolExecutor only exposes it; FeedParserDict itself is fine.

This is not specific to leading newlines. Any feed that fails strict parsing produces an unpicklable result (6.0.14):

input	bozo	pickle.dumps(result)
leading newline	1	ValueError: I/O operation on closed file
unclosed tag	1	ValueError: I/O operation on closed file
mismatched tag	1	ValueError: I/O operation on closed file
raw & in text	1	ValueError: I/O operation on closed file
junk before root element	1	ValueError: I/O operation on closed file
well-formed	False	OK

Since malformed feeds are the reason bozo exists, this affects any multiprocessing / ProcessPoolExecutor / Celery consumer handling real-world feeds.

Worth noting that detaching the locator is not sufficient — SAXParseException has no __reduce__ and SAXException.__init__ never populates args, so reconstruction fails too:

as-is         -> ValueError: I/O operation on closed file
locator=None  -> TypeError: SAXParseException.__init__() missing 2 required
                 positional arguments: 'exception' and 'locator'

The line, column and system id are already cached on the exception at __init__ time (CPython xml/sax/_exceptions.py), so a small picklable stand-in keeps all the diagnostics.

Fix for defect 1, verified

Allowing leading whitespace in the pattern makes sub() replace the declaration and drop the leading whitespace with it:

RE_XML_DECLARATION = re.compile(r"^\s*<\?xml[^>]*?>")

Against main @ a22c552, Python 3.12.14:

	encodings.py:68	test suite	reproducer from this issue
upstream	r"^<\?xml[^>]*?>"	4297 passed, 8 skipped	ValueError: I/O operation on closed file
patched	r"^\s*<\?xml[^>]*?>"	4297 passed, 8 skipped	passes

I also checked that the wider pattern does not change behaviour for documents with no declaration, or with leading spaces/tabs before one.

Workarounds today

Both confirmed on 6.0.14:

feedparser.parse(text.strip())          # avoids the double declaration
result.pop("bozo_exception", None)      # makes any bozo result picklable

Happy to send a PR for either half — the declaration fix on its own, or that plus a picklable bozo_exception.

Reproduction script (no network needed after install)

python -m venv .venv
.venv/bin/pip install feedparser==6.0.14
.venv/bin/python check.py

"""Minimal end-to-end check for feedparser issue #511 / #508."""
import pickle, xml.sax, feedparser

NL = ("\n<?xml version='1.0' encoding='UTF-8'?>\n"
      '<rss version="2.0"><channel><title>Foo</title>'
      "<item><title>T1</title></item></channel></rss>\n")
BAD = "<?xml version='1.0'?>\n<rss version=\"2.0\"><channel><title>A</channel></rss>"

def pick(d):
    try:
        pickle.dumps(d); return "OK"
    except Exception as e:
        return f"FAIL({type(e).__name__}: {e})"

print("1) what convert_to_utf8 hands to the SAX parser")
from feedparser import encodings
from feedparser.util import FeedParserDict
print("   leading newline ->", repr(encodings.convert_to_utf8({}, NL.encode(), FeedParserDict())[:78]))
print("   stripped        ->", repr(encodings.convert_to_utf8({}, NL.strip().encode(), FeedParserDict())[:78]))

print("2) parse result")
d = feedparser.parse(NL)
print("   bozo:", d.bozo, "| exc:", type(d.get('bozo_exception')).__name__,
      "|", str(d.get('bozo_exception')), "| entries:", len(d.entries))
print("   pickle:", pick(d))

print("3) the object graph pickle walks")
e = d["bozo_exception"]; loc = e._locator; src = loc._source
bs = src.getByteStream()
print(f"   exc -> {type(loc).__name__} -> {type(src).__name__} -> "
      f"{type(bs).__name__}(closed={bs.closed})")

print("4) not limited to leading newlines: any strict-parse failure")
d2 = feedparser.parse(BAD)
print("   mismatched tag -> bozo:", d2.bozo, "| pickle:", pick(d2))

print("5) SAXParseException cannot be made picklable in place")
for label, fn in (("as-is", lambda x: x), ("locator=None", lambda x: (setattr(x, '_locator', None), x)[1])):
    ex = feedparser.parse(BAD)["bozo_exception"]
    try:
        pickle.loads(pickle.dumps(fn(ex))); print(f"   {label:<13} -> OK")
    except Exception as err:
        print(f"   {label:<13} -> FAIL {type(err).__name__}: {err}")

print("6) workarounds that do work today")
print("   parse(text.strip())      -> pickle:", pick(feedparser.parse(NL.strip())))
d3 = feedparser.parse(NL); d3.pop("bozo_exception", None)
print("   del d['bozo_exception']  -> pickle:", pick(d3))
class FeedXMLParseError(Exception):
    def __init__(self, message, line=None, column=None):
        super().__init__(message); self.line, self.column = line, column
d4 = feedparser.parse(BAD); ex = d4["bozo_exception"]
d4["bozo_exception"] = FeedXMLParseError(str(ex), ex.getLineNumber(), ex.getColumnNumber())
print("   picklable surrogate      -> pickle:", pick(d4),
      "| roundtrip:", str(pickle.loads(pickle.dumps(d4))["bozo_exception"]))

Output:

1) what convert_to_utf8 hands to the SAX parser
   leading newline -> b"<?xml version='1.0' encoding='utf-8'?>\n\n<?xml version='1.0' encoding='UTF-8'?>"
   stripped        -> b'<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<rss version="2.0"><channel><title>Foo<'
2) parse result
   bozo: 1 | exc: SAXParseException | <unknown>:3:0: XML or text declaration not at start of entity | entries: 1
   pickle: FAIL(ValueError: I/O operation on closed file.)
3) the object graph pickle walks
   exc -> ExpatParser -> InputSource -> BytesIO(closed=True)
4) not limited to leading newlines: any strict-parse failure
   mismatched tag -> bozo: 1 | pickle: FAIL(ValueError: I/O operation on closed file.)
5) SAXParseException cannot be made picklable in place
   as-is         -> FAIL ValueError: I/O operation on closed file.
   locator=None  -> FAIL TypeError: SAXParseException.__init__() missing 2 required positional arguments: 'exception' and 'locator'
6) workarounds that do work today
   parse(text.strip())      -> pickle: OK
   del d['bozo_exception']  -> pickle: OK
   picklable surrogate      -> pickle: OK | roundtrip: <unknown>:2:38: mismatched tag