[jeremy]

Fixes #48.

What changes

Tika expresses detection of 38 XML vocabularies as <root-XML localName=… namespaceURI=…> rules that script/generate_tables.rb ignored, so RSS/Atom bodies detected as application/xml. Two live gaps on main:

1. Content-only detection: <?xml …?><rss version="2.0"> → application/xml. Feeds are the visible tip; kml, plist, xliff, DITA, ONIX, XSLT, iWork flat-XML, Word/Excel 2003 XML are all dropped on the floor.
2. Name + content together: for(io, name: "feed.rss") → application/xml, because application/rss+xml had no parent link to application/xml, so most_specific_type couldn't let the filename refine the magic verdict.

This PR mirrors the merged ZIP central-directory refiner (#169): generated data, a bounded fail-closed procedural refinement layered over the declarative magic result, no dependencies.

Generator

• Reads root-XML rules into a new ROOT_XML table keyed [namespace_or_nil, local_name], with MimeData.local_name / namespace_uri validators so the data is validated before emission and every string goes through RubySource.string (the inert-literal test now covers a code-shaped namespace).
• Tika semantics reproduced exactly (verified from MimeType.RootXML#matches and MimeTypes#init): matching is strict two-way equality, so the one namespace-only rule (RSS 1.0) can never match a real root element — it's routed through UnsupportedRules and the pin moves from 58 to 59. Cross-type precedence is lexicographic type name, first match wins; since each document matches at most one key, a per-key min at generation time is equivalent. The ONIX collision resolves to application/onix-message+xml, matching Tika's own tests.
• 72 rules → 69 keys. TYPE_EXTS is byte-identical; the only other table change is six new TYPE_PARENTS entries.

Runtime — Marcel::Magic::Xml.refine(io, base_type)

• Gate: base_type == "application/xml" only (Tika's text/html leg is moot: marcel's XML magic never yields it).
• One bounded read of at most 64KB (Tika's getMinLength).
• Fail-closed at the token level, matching what a namespace-aware SAX parser rejects before startElement (XmlRootExtractor takes no root on a parse failure):
  • UTF-16 (BOM guaranteed by the magic gate) is transcoded strictly; the only tolerated incompleteness is a code unit or surrogate pair split by the 64KB cut, which lies beyond the scan by construction.
  • Other input is validated against the declared encoding (<?xml … encoding="…"?>, UTF-8 by default; unknown, dummy, binary and Ruby pseudo-names like locale refine nothing) over exactly the bytes consumed up to the end of the root start-tag — so an invalid byte in a comment or an unrelated attribute before the root fails closed, while one after the root, or a multibyte character cut at 64KB past the root, is harmless.
  • The XML declaration must be first and is parsed in full — version, then optional encoding, then optional standalone="yes|no", in that order, with nothing else before ?> — so trailing or duplicated tokens cannot hide behind the PI skipper (<?xml version="1.0" garbage?>, duplicate encoding, standalone="maybe", out-of-order pseudo-attributes all fail closed). The declaration is limited to SAX-supported versions 1.0 and 1.1. No other PI may use the reserved xml target (<?XML …?>, a second declaration, a declaration after a comment all fail closed; xml-stylesheet and other targets are fine), and PI data must be separated from its target by whitespace (<?pi?x?> fails closed); comments may not contain --.
  • The consumed prefix must be free of characters the XML version forbids as literals, wherever they fall: XML 1.0 forbids the C0 controls outright but admits literal C1; XML 1.1 restricts both C0 and C1 (NEL excepted) to character references. A raw < inside a comment stays legal. Each side is pinned by positive controls so strictness cannot overshoot the spec.
  • Character references are parsed and validated against the version's Char production —  fails closed under 1.0 but passes under 1.1, and surrogates, noncharacters and � fail closed under both — and entity references must be predefined (amp/lt/gt/quot/apos) unless a DTD that could declare them was seen. References are validated, never resolved.
  • Names — the root and attribute QName parts, namespace prefixes, PI targets and entity names — are validated as Unicode NCNames after decoding (·a= fails closed since U+00B7 is NameChar but not NameStartChar; café= is a valid name), not as arbitrary high bytes.
  • The root start-tag is validated under the namespace rules: attribute QNames must be well-formed (a:b:c= fails closed), prefixes bound (p:x= with no xmlns:p fails closed), expanded [namespace, local] attribute names unique (xmlns:a="urn:x" xmlns:b="urn:x" a:x b:x fails closed; the default namespace does not apply to attributes), the xml/xmlns names and namespace URIs reserved (xmlns:xml only to its own URI, xmlns:xmlns never; xmlns:p="" is a Namespaces 1.1 undeclaration honoured under version 1.1 and an error under 1.0, and xml can never be undeclared), and attribute values free of raw < and malformed references (bare & fails closed; &/A are matched but never resolved).
• Prolog skipper handles whitespace, PIs, comments, and <!DOCTYPE> with quoted literals and an internal subset — <!ENTITY x "<foo>"> and <!-- don't --> inside the subset don't derail it. Linear in the prefix; every step advances by an index jump, no backtracking regexes.
• Root start-tag parsed as qname + quoted attribute triples so a > in a value can't truncate the tag; duplicate attributes are a well-formedness error (Unique Att Spec) and refine nothing — last-one-wins could otherwise flip the namespace; namespace resolved from the root's own xmlns / xmlns:<prefix>; xmlns="" un-declares; an undeclared prefix is a namespace error and refines nothing.
• Result is canonicalized: Tika's application/x-xliff+xml is an alias key in marcel, so an xliff document yields application/xliff+xml, preserving Respect MIME type aliases #171's canonical-output invariant.
• custom.xml adds <sub-class-of type="application/xml"/> for atom+xml, rss+xml, xhtml+xml, xslfo+xml, xslt+xml, xspf+xml, repeating each type's Tika globs and comment because the generator records parents only for glob-bearing types and overwrites the type wholesale (same pattern as the existing OOXML overrides).

Release manifest (script/verify_gem.rb)

release.yml's inline gem manifest was already missing lib/marcel/magic/zip.rb (since #169) and would have aborted the build at its exact-manifest assertion; CI's "Release toolchain" job never executes that block. The manifest and identity checks now live in script/verify_gem.rb: the release build calls it, and test/package_test.rb both uses its manifest constant and runs the script's CLI against a freshly built gem on every push (plus a negative case), so CI and release hold the gem to one list. Verified locally by replaying the release build step (gem build → ruby script/verify_gem.rb pkg/marcel-1.2.1.gem 1.2.1 → Verified marcel-1.2.1: 13 files).

Behavior changes beyond feeds, all Tika-faithful

• Un-namespaced <?xml?><body>, <p>, <script>, <frameset>, <iframe>, <link> roots (either case) → text/html (TIKA-327). <?xml?><html> was already text/html via the runtime HTML matcher.
• A comment-prefixed HTML document whose <html> lies past the 4KB HTML-magic window but within 64KB is now text/html rather than application/xml (html_magic_test updated; the 4KB bound itself is re-pinned with a >64KB variant).
• Prefixed roots now resolve: <k:kml xmlns:k=…>, <h:html xmlns:h="http://www.w3.org/1999/xhtml"> → kml / xhtml.
• Marcel::Magic.new("application/rss+xml").text? is now true (via application/xml → text/plain), as for every other XML subtype.

Documented posture

• refine can return the 10 glob-less Tika types absent from TYPE_EXTS (e.g. application/vnd.ms-word2006ml) and ignores runtime Magic.remove — the posture Zip::DOCUMENT_FAMILIES established in Sniff ZIP central directories to identify OOXML documents #169.
• No existing fixture changes type: the comment-prefixed fixtures are claimed by earlier matchers and magic/application/xml/xml.xml's <_/> root matches no key. A test pins these.
• Deliberate divergences from Tika: no TIKA-2419 downgrade to text/plain (marcel stays on application/xml); a bounded token-level scanner instead of a full SAX parse, so it enforces encoding validity and prolog token rules but does not resolve character references in xmlns, apply <!ATTLIST> #FIXED defaults, normalize attribute values, or validate DOCTYPE grammar beyond bracket/quote/comment/PI structure — each divergence is pinned as application/xml in the tests. Normalization can never change a verdict anyway: no key contains whitespace.
• Unseekable-but-rewindable IOs are refined (the scan needs only a read from the start), unlike the ZIP refiner which needs seek/size.

Tests

test/xml_sniffing_test.rb (49 tests): happy paths for every vocabulary family above, canonicalization, Tika-fidelity cases, a prolog gauntlet (comment-first, internal DTD subset with quoted > and commented ', PI before root, multi-line start-tag with a non-ASCII attribute name, xmlns="", all three BOMs), encoding strictness (invalid UTF-8 in a comment / in an unrelated attribute / after the root, lone UTF-16 surrogate in a comment, split surrogate pair at the cut, ISO-8859-1 and Shift_JIS validated as declared, unknown/pseudo/dummy encodings), prolog well-formedness (<?XML, missing version, duplicate and misplaced declarations, full-declaration grammar with positive standalone/spacing controls, -- in comments incl. inside the internal subset), fail-closed cases (start-tag straddling the 64KB limit, root beyond it, garbage after the prolog, unterminated comment/DOCTYPE/tag, odd-length UTF-16 truncation inside vs. after the root), bounded-time adversarial prologs, a 10MB document asserting exactly MAX_SCAN bytes read, the gap-2 regression (name: "feed.rss" on generic XML → application/rss+xml), sibling-hint non-override, fixture regressions, and io.pos == 0 after refine. Generator tests cover collision resolution, the [nil, name] key shape, parameterized values, and rejection of code-shaped names/namespaces. Three new authored fixtures (rss, atom, kml) ride the magic sweep.

bundle exec rake (test + tables:check): 659 runs, 1660 assertions, green; the branch is rebased onto #174's merge (90855e0), so the reviewed tree is the merge tree. Zero runtime dependencies unchanged.