Performance: datahub export filter is O(n*m) over a 3x-duplicated subject id array

[gdevenyi]

Current Implementation and Limitations

handleExportSelection in apps/web/src/routes/_app/datahub/index.tsx:227 downloads the whole group's export from the server and then filters it in the browser down to the subjects currently listed in the table:

const listedSubjects = table
  .getPrePaginationRowModel()
  .rows.flatMap((row) => row.getVisibleCells().map((cell) => removeSubjectIdScope(cell.row.original.id)));

const filteredData = data.filter((dataEntry) => listedSubjects.includes(dataEntry.subjectId));

Three separate problems in these four lines:

1. listedSubjects contains one entry per visible cell, not per row. The flatMap iterates row.getVisibleCells() but ignores cell entirely and reads cell.row.original.id — the same value for every cell in the row. The master table has three data columns, so the array holds at least three identical copies of every subject id, and removeSubjectIdScope() is called three times per row to produce them.

2. listedSubjects.includes(...) inside data.filter(...) is O(n × m). includes on an array is a linear scan. With n export rows and m entries in listedSubjects, this is n × m string comparisons — and because of (1), m is already 3× larger than it needs to be.

3. The filtering happens on the client at all. The groupId is the only thing sent to the server (params: { groupId: currentGroup?.id }); every other active filter — the sex filter, the date-of-birth range, the search string, the "with records only" toggle — is applied after the full group export has crossed the network. A user who filters down to 10 subjects still downloads the export for all of them.

The same anti-pattern appears at logs.tsx:141 in the audit log download, which builds its payload from the client's in-memory row model.

Measurements

Combining with the measurements from the export endpoint issue: a group with 25,000 instrument records produces 750,000 export rows and a 111 MiB response body. The master table for that group lists 5,000 subjects, so listedSubjects holds 15,000 strings (3 × 5,000 duplicates).

That makes the client-side filter 750,000 × 15,000 = 1.1 × 10¹⁰ string comparisons in the worst case (a row whose subject is not listed scans the entire array). This runs synchronously on the main thread, after the browser has already parsed a 111 MiB msgpack payload. In practice the tab will be unresponsive for a long time or crash.

Deduplicating and using a Set reduces the same work to 750,000 hash lookups — a reduction of roughly four orders of magnitude — and is a two-line change.

Associated Application Components

Client, Server

Proposed Solution

1. Immediate, low-risk fix — dedupe and use a Set:

const listedSubjects = new Set(
  table.getPrePaginationRowModel().rows.map((row) => removeSubjectIdScope(row.original.id))
);

const filteredData = data.filter((dataEntry) => listedSubjects.has(dataEntry.subjectId));

This drops the redundant getVisibleCells() traversal, calls removeSubjectIdScope once per row instead of three times, and turns the O(n × m) filter into O(n). It changes no behaviour.

2. Real fix — filter server-side. Send the listed subject ids (or, better, the filter criteria themselves) to /v1/instrument-records/export so the server only produces the rows the user asked for. This composes directly with the streaming rework proposed in the export endpoint issue: with a subjectIds parameter the export becomes proportional to what is being exported rather than to the size of the group.

3. Apply the same fix to the audit log download (logs.tsx:140), which has the same "build the export from the client's row model" shape and additionally captures a stale table via useCallback(..., []).

Estimated Difficulty

Low

Priority

High

[joshunrau]

@gdevenyi Claude can do this

[gdevenyi]

Correction to the original report: the duplication factor is 4×, not 3×. I measured it against the real table rather than counting columns by eye — the master table has three data columns plus the row-actions column libui injects when rowActions is set:

cellsPerRow=4  rows=5000
oldArrayLength=20000  newSetSize=5000  factor=4.0

#1425 fixes item 1, measured at 3,070 ms → 17 ms for 200,000 export rows against 5,000 listed subjects (identical results, 167,000 matched either way).

Item 2 is still open — the filtering remains client-side, so every active filter is applied only after the whole group's export has crossed the network. That is better done alongside the streaming rework in #1407 than by adding a subjectIds parameter to an endpoint that is about to be restructured.

Item 3 (the same pattern in the audit log download) is handled in #1419.