Precompute ancestor hierarchy to speed up term indexing

[alexskr]

Summary

Closes #278

• Replace per-class SPARQL ancestor traversal in index_doc with a single bulk query + in-memory transitive closure
• Before the indexing loop, fetch all parent-child edges in one paginated SPARQL query and compute ancestors via memoized BFS — O(V + E) total
• Store the precomputed ancestor map as a class-level cache on LinkedData::Models::Class, cleared in an ensure block after indexing
• For a 100K-class ontology, this eliminates ~800K SPARQL round-trips
• Observed 2-5x overall indexing performance improvement on large ontologies

Changes

• submission_indexer.rb — compute_ancestors_map, fetch_all_parent_edges, compute_ancestors_for precompute the full ancestor map; validate_class_ancestors provides temporary per-class comparison against the old traversal, gated behind OP_VALIDATE_ANCESTORS env var, with per-class timing logs
• class.rb — ancestors_cache class accessor; index_doc reads from cache instead of calling retrieve_hierarchy_ids
• test_ancestors_precompute.rb — 9 unit tests covering linear chain, diamond inheritance, multiple roots, cycles, complex DAG, memoization, and edge cases

Validation

Run with OP_VALIDATE_ANCESTORS=1 to enable per-class comparison of precomputed ancestors against the old SPARQL traversal. Every class is checked and logs include timing for old vs new approach plus match/mismatch status. Once validated against production data, the validation code can be removed.

Test plan

• Unit tests pass: bundle exec rake test:docker:fs TEST='test/models/test_ancestors_precompute.rb' (9/9)
• Full indexing pipeline: bundle exec rake test:docker:fs TEST='test/models/test_search.rb'
• Validation run on a large ontology with OP_VALIDATE_ANCESTORS=1 — confirm zero mismatches

[mdorf]

This is a SOLID performance optimization, and we should definitely implement it in a short working order. However, it introduces significant changes to a core model and should NOT be bundled together with the upcoming major release. I propose that we revisit this feature SHORTLY after the main release.

[mdorf]

Hard dependency on cache — no fallback (High)

class.rb:270 now unconditionally reads from ancestors_cache:

path_ids = (self.class.ancestors_cache[class_id] || Set.new).dup

If index_doc is ever called outside of bulk indexing (e.g., individual class re-indexing, tests, or other code paths), ancestors_cache will be nil, and calling [class_id] on nil will raise NoMethodError. The commented-out fallback to retrieve_hierarchy_ids should be uncommented:

if self.class.ancestors_cache
  path_ids = (self.class.ancestors_cache[class_id] || Set.new).dup
else
  path_ids = retrieve_hierarchy_ids(:ancestors)
end

[alexskr]

I was thinking about it but is there a case where index_doc is called outside of bulk indexing?

[mdorf]

Thread safety of class-level mutable state (High)

ancestors_cache is a class-level attr_accessor on LinkedData::Models::Class. If two ontologies are indexed concurrently (separate threads/workers), one worker could overwrite or nil out the cache while another is still reading it. Consider:

• Making the cache per-submission (pass it through the indexing context rather than a global class variable)
• Or at minimum, documenting/asserting that concurrent indexing is not supported

[alexskr]

Thread safety of class-level mutable state (High)

The ancestors_cache is computed once before worker threads are spawned. During indexing, threads only read from it (hash lookups + .dup on the Sets) — no writes occur until all threads complete and the cache is cleared.

The class-level state concern would apply if different ontologies were indexed concurrently in the same process. Currently ncbo_cron processes ontologies sequentially, and if we ever move to concurrent indexing, we would likely use separate worker processes subscribing to a queue — each with their own isolated class-level state. In the unlikely event we needed concurrency within a shared process, we could scope the cache via Thread.current (same pattern already used for RequestStore.store[:requested_lang] in this file).

[mdorf]

Instance variables leaked from validate_class_ancestors (Medium)

In validate_class_ancestors, @old_ancestors_result and @new_ancestors_result are instance variables on the indexer but are only used within Benchmark.realtime blocks. These should be local variables instead. The current approach leaks state across calls:

old_time = Benchmark.realtime do
  old_ancestors = cls.retrieve_hierarchy_ids(:ancestors)
  old_ancestors.select! { |x| !x["owl#Thing"] }
end