Delete search index in an indexing task

[labkey-jeckels]

Rationale

A recent server deadlock involved three concurrent attempts to modify exp.DataIndexed. Two were admin-initiated requests to delete the index (the second was presumably a retry as it was kicked off 5.5 minutes later), and the third was background indexing of data objects.

Additionally, we can capture more context in thread dumps for search indexing tasks.

Changes

• Move index deletion and clearing of LastModified to a search task, meaning it will queue and avoid parallel self-execution
• Capture indexing work as thread extra context
• Show extra context in Admin Console thread dumps too
• Test coverage for index deletion and crawler pausing

Tasks 📍

• Manual Testing @labkey-danield
  • Delete index, ensure it goes away
  • Admin Console->Running Threads while indexing work is happening. Ensure SearchService thread shows extra context for the current runnable, including its time spent executing so far
• Needs Automation