Skip to content

Parse list_object_versions responses with xmerl SAX instead of a DOM - #336

Closed
velimir wants to merge 1 commit into
ex-aws:mainfrom
velimir:faster-object-versions-parser
Closed

velimir wants to merge 1 commit into
ex-aws:mainfrom
velimir:faster-object-versions-parser

Conversation

@velimir

@velimir velimir commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem

parse_object_versions/1 is expensive. On OTP 28, one ListVersionsResult page with 1000 entries takes approximately 220 ms and 9.5M reductions to parse. When code walks the version history of a large prefix with ExAws.S3.list_object_versions/2 |> ExAws.stream!(), this adds up: 1000 pages take approximately 3.7 minutes of CPU time only to parse.

The cost is mostly garbage collection, not the XML scan or the xpath queries. SweetXml.xpath/3 builds the full xmerl DOM for the page, with every text node as a charlist. That keeps 10 to 15 MB live while the xpath queries allocate more data. A GC trace of one page shows 40 major and 82 minor collections, and each major collection copies the DOM again. Each step alone is fast: the scan takes 36 ms and xpath on a DOM that is already built takes 24 ms. Together they take 222 ms.

Change

parse_object_versions/1 now uses ExAws.S3.Parsers.ObjectVersions, which builds the result maps directly from :xmerl_sax_parser events. It builds no DOM. xmerl_sax_parser is part of OTP, so there is no new dependency.

The new module does not call SweetXml, but it keeps the if Code.ensure_loaded?(SweetXml) check that wraps ExAws.S3.Parsers. ex_aws_s3 does not list xmerl in its applications. sweet_xml is optional, and it is the dependency that brings xmerl into a release. With the check, list_object_versions keeps the same requirement as the other operations that parse XML: without sweet_xml, it still raises "Missing XML parser".

The result has the same shape and the same string values as before. I compared the old and the new parser on generated pages and on edge cases: entities, character references, CDATA, comments, whitespace-only values, UTF-8 and UTF-16, nested and repeated elements, unknown elements and namespace prefixes. Apart from the changes below, the output is identical.

Performance

One page of 1000 entries, 950 Version and 50 DeleteMarker, 498 KB, with keys of approximately 140 bytes. Each value is the median of 21 runs, and each run is in a new process. OTP 28.5, Elixir 1.19.5, xmerl 2.1.9.

Parser Time Reductions Memory collected
SweetXml / xmerl DOM 222 ms 9.5M 221 MB
xmerl SAX 34 ms 2.0M 36 MB

Almost all of the remaining time is in xmerl_sax_parser_utf8, which reads each character. The event function uses approximately 1% of the time.

Behaviour changes

  • An entry without <Owner> now gives owner: nil. Before, the parse crashed with Protocol.UndefinedError ([Bug] While trying to fetch object versions #291). nil is the same value that parse_list_objects/1 gives for a missing Owner.
  • &#13; now decodes to "\r". Before, it decoded to "\n" (\r by itself is mapped to \n which causes things like AWS S3 Key names to have information loss when parsed by SweetXxml kbrw/sweet_xml#92), so a key with a carriage return did not match the real object key. This agrees with the Smithy restXml protocol test SimpleScalarPropertiesComplexEscapes, which expects "\r\n" for &#xD;&#10;. The old parser gives "\n\n".
  • Entity declarations, external DTDs and undeclared entity references are rejected with exit({:fatal, _}) on every supported OTP release. The event function does the check, so it does not need the disallow_entities option, which arrived in OTP 24.3.4.12 and 25.3.2.1. On OTP 24.3.4.2, the old parser expanded entities and read local files through <!ENTITY x SYSTEM "file:///...">. On OTP 24.3.4.2 and 28.5, the old parser opened the file of an external DTD subset.
  • 2 rare text cases now follow the XML rules. A literal CR inside CDATA becomes LF, and whitespace after a comment stays in the value. S3 does not send either of them.
  • Malformed XML still exits with {:fatal, reason}. A root element other than an unprefixed ListVersionsResult now exits with {:fatal, {:expected_root, ~c"ListVersionsResult"}}. Before, it raised Protocol.UndefinedError.

Tests

The new tests use explicit XML and the expected maps. They include the string cases from the Smithy restXml protocol tests: SimpleScalarPropertiesComplexEscapes, SimpleScalarPropertiesWithEscapedCharacter, SimpleScalarPropertiesWithWhiteSpace, SimpleScalarPropertiesPureWhiteSpace, XmlEmptySelfClosedStrings and SimpleScalarPropertiesWithXMLPreamble. The existing ListVersionsResult test now compares full entry maps.

The parser tests pass on OTP 24.3.4.2 with Elixir 1.13, on OTP 25.3.2, and on OTP 28.5. On OTP 27.3 with Elixir 1.18, the full suite, mix format --check-formatted and mix dialyzer pass. I did not change mix.lock. On OTP 28, the locked cowlib 2.9.1 does not compile, so I ran the parser tests there outside mix test.

On OTP 28 one 1000-entry ListVersionsResult page (~500 KB) took ~220 ms
and 9.5M reductions to parse. Most of that is garbage collection: the
xmerl DOM, with every text node as a charlist, keeps 10-15 MB live
while the xpath queries allocate, so the process runs ~40 major
collections per page. Code that walks the version history of a large
prefix spends whole cores on this.

The parser now builds the result maps from xmerl_sax_parser events.
The same page takes ~34 ms and 2.0M reductions, with no new dependency.

Behaviour changes:
- an entry without <Owner> returns owner: nil instead of crashing (ex-aws#291)
- &ex-aws#13; decodes to "\r" instead of "\n", so keys with a carriage
  return survive the round trip
- entity declarations and external DTDs are rejected on every OTP
  release; before, OTP releases older than 24.3.4.12 / 25.3.2.1
  expanded entities, and external DTD files were opened on all releases
@velimir
velimir marked this pull request as draft October 1, 2026 16:49
@velimir velimir closed this Oct 1, 2026
@velimir
velimir deleted the faster-object-versions-parser branch October 1, 2026 17:47
@velimir
velimir restored the faster-object-versions-parser branch October 2, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant