Repository navigation
Conversation
On OTP 28 one 1000-entry ListVersionsResult page (~500 KB) took ~220 ms and 9.5M reductions to parse. Most of that is garbage collection: the xmerl DOM, with every text node as a charlist, keeps 10-15 MB live while the xpath queries allocate, so the process runs ~40 major collections per page. Code that walks the version history of a large prefix spends whole cores on this. The parser now builds the result maps from xmerl_sax_parser events. The same page takes ~34 ms and 2.0M reductions, with no new dependency. Behaviour changes: - an entry without <Owner> returns owner: nil instead of crashing (ex-aws#291) - &ex-aws#13; decodes to "\r" instead of "\n", so keys with a carriage return survive the round trip - entity declarations and external DTDs are rejected on every OTP release; before, OTP releases older than 24.3.4.12 / 25.3.2.1 expanded entities, and external DTD files were opened on all releases
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
parse_object_versions/1is expensive. On OTP 28, oneListVersionsResultpage with 1000 entries takes approximately 220 ms and 9.5M reductions to parse. When code walks the version history of a large prefix withExAws.S3.list_object_versions/2 |> ExAws.stream!(), this adds up: 1000 pages take approximately 3.7 minutes of CPU time only to parse.The cost is mostly garbage collection, not the XML scan or the xpath queries.
SweetXml.xpath/3builds the full xmerl DOM for the page, with every text node as a charlist. That keeps 10 to 15 MB live while the xpath queries allocate more data. A GC trace of one page shows 40 major and 82 minor collections, and each major collection copies the DOM again. Each step alone is fast: the scan takes 36 ms and xpath on a DOM that is already built takes 24 ms. Together they take 222 ms.Change
parse_object_versions/1now usesExAws.S3.Parsers.ObjectVersions, which builds the result maps directly from:xmerl_sax_parserevents. It builds no DOM.xmerl_sax_parseris part of OTP, so there is no new dependency.The new module does not call SweetXml, but it keeps the
if Code.ensure_loaded?(SweetXml)check that wrapsExAws.S3.Parsers. ex_aws_s3 does not listxmerlin its applications.sweet_xmlis optional, and it is the dependency that bringsxmerlinto a release. With the check,list_object_versionskeeps the same requirement as the other operations that parse XML: withoutsweet_xml, it still raises "Missing XML parser".The result has the same shape and the same string values as before. I compared the old and the new parser on generated pages and on edge cases: entities, character references, CDATA, comments, whitespace-only values, UTF-8 and UTF-16, nested and repeated elements, unknown elements and namespace prefixes. Apart from the changes below, the output is identical.
Performance
One page of 1000 entries, 950
Versionand 50DeleteMarker, 498 KB, with keys of approximately 140 bytes. Each value is the median of 21 runs, and each run is in a new process. OTP 28.5, Elixir 1.19.5, xmerl 2.1.9.Almost all of the remaining time is in
xmerl_sax_parser_utf8, which reads each character. The event function uses approximately 1% of the time.Behaviour changes
<Owner>now givesowner: nil. Before, the parse crashed withProtocol.UndefinedError([Bug] While trying to fetch object versions #291).nilis the same value thatparse_list_objects/1gives for a missingOwner. now decodes to"\r". Before, it decoded to"\n"(\r by itself is mapped to \n which causes things like AWS S3 Key names to have information loss when parsed by SweetXxml kbrw/sweet_xml#92), so a key with a carriage return did not match the real object key. This agrees with the Smithy restXml protocol testSimpleScalarPropertiesComplexEscapes, which expects"\r\n"for
 . The old parser gives"\n\n".exit({:fatal, _})on every supported OTP release. The event function does the check, so it does not need thedisallow_entitiesoption, which arrived in OTP 24.3.4.12 and 25.3.2.1. On OTP 24.3.4.2, the old parser expanded entities and read local files through<!ENTITY x SYSTEM "file:///...">. On OTP 24.3.4.2 and 28.5, the old parser opened the file of an external DTD subset.{:fatal, reason}. A root element other than an unprefixedListVersionsResultnow exits with{:fatal, {:expected_root, ~c"ListVersionsResult"}}. Before, it raisedProtocol.UndefinedError.Tests
The new tests use explicit XML and the expected maps. They include the string cases from the Smithy restXml protocol tests:
SimpleScalarPropertiesComplexEscapes,SimpleScalarPropertiesWithEscapedCharacter,SimpleScalarPropertiesWithWhiteSpace,SimpleScalarPropertiesPureWhiteSpace,XmlEmptySelfClosedStringsandSimpleScalarPropertiesWithXMLPreamble. The existingListVersionsResulttest now compares full entry maps.The parser tests pass on OTP 24.3.4.2 with Elixir 1.13, on OTP 25.3.2, and on OTP 28.5. On OTP 27.3 with Elixir 1.18, the full suite,
mix format --check-formattedandmix dialyzerpass. I did not changemix.lock. On OTP 28, the lockedcowlib2.9.1 does not compile, so I ran the parser tests there outsidemix test.