Memory leak?

[saschabrockel]

Bug Type

Performance problem

Affected Area

Other

DOCSight Version

Latest release

Exact Version or Commit

No response

What happened?

I realized today that my DOCSight Docker Container (running on my Unraid server) was using 9 GB of RAM. I updated the image and restarted the container and it started with using 300 MB approximately.

After 30 minutes it are already 600 MB used. I will check this again but even that it used 9 GB og RAM confused me a lot like what is the tool doing?

Steps to Reproduce

1. Run DOCSight
2. See the RAM usage getting more and more

What did you expect to happen?

No increasing RAM usage.

What actually happened?

Memory usage seems to increase over time

Deployment Method

Docker Compose

Modem or Router Model

FRITZ!Box 6690 Cable

Relevant Logs or Console Errors

Screenshots, Exports, or Extra Context

No response

[itsDNNS]

Thanks for reporting this. I compared it with a comparable Docker deployment running DOCSight v2026-07-16.1 against a FRITZ!Box with a 300-second polling interval.

That installation has more than 44,000 snapshots and a roughly 540 MiB SQLite history database. During a monitored interval that included a complete modem poll:

• Docker-adjusted memory stayed between 122.5 and 123.1 MiB
• the Python process RSS stayed around 136 MiB
• there were no restarts or OOM events
• raw cgroup memory was about 618 MiB, but roughly 495 MiB of that was reclaimable inactive file cache

This distinction is important for DOCSight because Linux will cache pages from the SQLite history database. Some dashboards may include that file cache in the displayed container memory, while docker stats subtracts inactive file cache. The approximately 600 MiB seen shortly after startup may therefore be cache accounting rather than evidence of a growing Python heap.

The earlier 9 GiB value still needs investigation, so I do not want to dismiss this as only a display issue yet. Could you please provide the following after the container has been running for a while, replacing docsight if your container has a different name?

docker stats --no-stream docsight
docker inspect --format 'Started={{.State.StartedAt}} Restarts={{.RestartCount}} MemoryLimit={{.HostConfig.Memory}}' docsight
docker exec docsight python -c 'print(next(line.strip() for line in open("/proc/1/status") if line.startswith("VmRSS:")))'
docker exec docsight python -c 'import os; p="/data/docsis_history.db"; print(f"{os.path.getsize(p)/(1024*1024):.1f} MiB")'

It would also help to know:

• the exact DOCSight version
• the configured polling interval and history retention
• container uptime when the value reaches several GiB
• whether docker stats and the Unraid dashboard show substantially different values
• a screenshot of the Unraid memory metric, if possible

Please do not attach config.json, as it can contain credentials. With the values above we should be able to distinguish file cache from actual anonymous/process-memory growth.

[saschabrockel]

First of all, thanks for the response. Now it seems to be normal. Between 400 and 600 MB. But I don't know as well why I saw 9 GB today.

My version now is v2026-07-16.1. So the version where it happened was v2026-07-05.1.

docker inspect --format 'Started={{.State.StartedAt}} Restarts={{.RestartCount}} MemoryLimit={{.HostConfig.Memory}}' docsight
Started=2026-07-16T12:49:21.566186684Z Restarts=0 MemoryLimit=0

docker exec docsight python -c 'print(next(line.strip() for line in open("/proc/1/status") if line.startswith("VmRSS:")))'
VmRSS:    508616 kB

docker exec docsight python -c 'import os; p="/data/docsis_history.db"; print(f"{os.path.getsize(p)/(1024*1024):.1f} MiB")'
13276.0 MiB

The polling interval is 900 seconds.
History retention is 90 days.

As it is normal and okay now, I will check if it needs a few days to reach that level again.

[itsDNNS]

Thanks, this is useful. At the time of your measurements the container had been running for about eight hours without a restart. The Python process RSS was about 497 MiB, while the SQLite history database was about 12.96 GiB.

I rechecked the comparison installation mentioned above in more detail. It currently contains 44,733 snapshots spanning more than five months, uses a 300-second polling interval and unlimited retention, yet its complete database is only 541.6 MiB and contains no reusable free pages. Against that baseline, a 12.96 GiB database is disproportionate for a 900-second interval and 90-day retention. It could still have contributed to the 9 GiB Unraid reading if database pages were cached, but we should not treat that size as a normal cache footprint or assume cache is the whole explanation. First we need to determine whether the file mainly contains reusable pages or unusually large live data.

I also checked the two releases: the SQLite storage and retention code is unchanged between v2026-07-05.1 and v2026-07-16.1. The restart during the update reset the observed memory, but there is no known storage-memory fix in that version change, so I will keep this open while we gather more evidence.

Please let the container continue running as planned. If the Unraid value starts climbing again, please capture these values at roughly the same time and before restarting it:

docker stats --no-stream docsight

docker exec docsight sh -c 'grep -E "^(Rss|Pss|Anonymous|Private_Clean|Private_Dirty|Shared_Clean|Shared_Dirty):" /proc/1/smaps_rollup'

docker exec docsight sh -c 'printf "memory.current="; cat /sys/fs/cgroup/memory.current; grep -E "^(anon|file|inactive_file|active_file) " /sys/fs/cgroup/memory.stat'

The following one-time database check would also help determine whether much of the 13 GiB file consists of reusable free pages:

docker exec -i docsight python - <<'PY'
import sqlite3

path = "/data/docsis_history.db"
with sqlite3.connect(path) as db:
    page_size = db.execute("PRAGMA page_size").fetchone()[0]
    page_count = db.execute("PRAGMA page_count").fetchone()[0]
    free_pages = db.execute("PRAGMA freelist_count").fetchone()[0]
    count, oldest, newest = db.execute(
        "SELECT COUNT(*), MIN(timestamp), MAX(timestamp) FROM snapshots"
    ).fetchone()

print(f"allocated={page_size * page_count / 1024**3:.2f} GiB")
print(f"reusable_free={page_size * free_pages / 1024**3:.2f} GiB")
print(f"snapshots={count} oldest={oldest} newest={newest}")
PY

These readings should distinguish actual Python/anonymous-memory growth from reclaimable SQLite file cache and explain why this database is much larger than the comparison installation.

[saschabrockel]

I want to share the following AI conversation with you (it's in German, unfortunately) where it seems to be a Python thing with glibc:

https://claude.ai/share/19fef64d-ee4b-4825-9f82-2d78a584e807

[itsDNNS]

Thanks for sharing this. glibc arena fragmentation is technically possible because the DOCSight image uses Debian/glibc and the process uses multiple threads. The fact that another Python application on the same host shows similar behavior makes a shared host or allocator-related factor worth considering, but it does not establish glibc fragmentation as the cause.

At present there is no evidence of a general DOCSight memory leak. A comparable long-running installation with more frequent polling, a longer history, and substantially more snapshots remains stable, and there is no broader pattern of similar reports. That does not rule out a code path triggered by a particular database, configuration, workload, or host environment.

The 12.96 GiB SQLite database remains the most significant instance-specific difference, and allocator fragmentation would not explain the database file itself. VmRSS and docker stats alone also cannot fully distinguish anonymous process memory from file-backed pages. Docker subtracts inactive file cache, but active file pages can still remain in the reported usage.

Please avoid setting MALLOC_ARENA_MAX, changing the trim threshold, adding scheduled restarts, or otherwise changing the baseline for now. The smaps_rollup, cgroup memory.stat, and SQLite page-count results requested in my previous comment remain the next useful step:

• growing anon/Anonymous with stable file memory would make an application leak or allocator fragmentation more plausible
• growing file memory with stable anonymous memory would point toward SQLite/file cache
• the SQLite freelist and snapshot range will show whether the 12.96 GiB file mainly contains reusable pages or unusually large live data

Once those readings are available, we can decide whether a controlled MALLOC_ARENA_MAX=2 comparison is warranted without masking the original behavior.

[itsDNNS]

Hi @saschabrockel, has the memory usage remained stable since your last update, or did the issue occur again?

If it has remained stable and no further diagnostic data is available, we’ll close this as not reproducible for now. We can reopen it if the behavior returns and the requested measurements can be captured.

[saschabrockel]

@itsDNNS good that you reminded me. The memory usage is now again between 5-6 GB and somehow now that I watched it increasing to 9GB again. That does not seem normal.

docker stats --no-stream docsight
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
42e650cd37da   docsight   0.34%     28.15GiB / 94GiB    29.94%    599MB / 13.6MB   11.6TB / 27.1MB   31

This is really a memory leak this time....

docker exec docsight sh -c 'grep -E "^(Rss|Pss|Anonymous|Private_Clean|Private_Dirty|Shared_Clean|Shared_Dirty):" /proc/1/smaps_rollup'
grep: /proc/1/smaps_rollup: Permission denied

docker exec docsight sh -c 'printf "memory.current="; cat /sys/fs/cgroup/memory.current; grep -E "^(anon|file|inactive_file|active_file) " /sys/fs/cgroup/memory.stat'
memory.current=30611267584
anon 30147522560
file 399261696
inactive_file 394506240
active_file 4444160

Version is v2026-07-25.1

Edit: it increased until it reached 28.2GiB of memory usage and somehow dropped again to normal, but starts increasing again:

root@UNRAID-Server:~# docker stats --no-stream docsight
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
42e650cd37da   docsight   0.20%     28.2GiB / 94GiB     30.00%    600MB / 16.1MB   11.6TB / 27.1MB   31
root@UNRAID-Server:~# docker stats --no-stream docsight
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
42e650cd37da   docsight   98.27%    795.7MiB / 94GiB    0.83%     600MB / 16.1MB   11.6TB / 27.1MB   31
root@UNRAID-Server:~# docker stats --no-stream docsight
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
42e650cd37da   docsight   100.00%   3.518GiB / 94GiB    3.74%     600MB / 16.1MB   11.6TB / 27.1MB   31

[itsDNNS]

Thanks, the updated measurements narrow this down considerably.

At the 28.2 GiB peak, about 28.1 GiB was anonymous container memory and only about 0.37 GiB was file-backed. This rules out the Linux file cache as the explanation for that peak. The drop to about 796 MiB followed by another rise at 100% CPU also suggests a repeatable long-running operation rather than simple monotonic retention.

I found two relevant defects in the backup path:

• backups are currently built completely in memory, and the file-write path creates a second full copy of the compressed archive;
• a collector that runs longer than the orchestrator's 120-second wait can finish after its future is no longer observed. Its successful completion is then not recorded, so an overdue scheduled job can start again immediately.

Both are valid defects and will be addressed independently, but I do not want to call them the cause of this report until the runtime correlation is confirmed. Could you provide the following?

1. Is automatic backup enabled, what interval is configured, and was a manual backup started during the observed period? Please also provide the exact sizes and timestamps of the most recent generated backup files.
2. Please provide the matching backup log lines:

docker logs --since 48h --timestamps docsight 2>&1 \
  | grep -Ei 'backup|timed out after 120s' \
  | tail -100

3. Please check whether the drop was a restart or OOM event:

docker inspect --format '{{.RestartCount}} {{.State.StartedAt}} {{.State.OOMKilled}} {{.State.Pid}}' docsight

docker exec -u 0 docsight sh -c '
  grep -E "^(VmRSS|VmHWM|Threads):" /proc/1/status
  cat /sys/fs/cgroup/memory.events
'

4. Please retry the missing SQLite page information as root inside the container:

docker exec -u 0 docsight python -c '
import sqlite3
p = "/data/docsis_history.db"
db = sqlite3.connect(f"file:{p}?mode=ro", uri=True)
for name in ("page_size", "page_count", "freelist_count"):
    print(name, db.execute(f"PRAGMA {name}").fetchone()[0])
print("snapshots", db.execute("SELECT COUNT(*), MIN(timestamp), MAX(timestamp) FROM snapshots").fetchone())
'

Please redact WAN IPs, MAC addresses, serial numbers, credentials, or tokens if any unrelated log lines contain them.

[saschabrockel]

root@UNRAID-Server:~# docker logs --since 48h --timestamps docsight 2>&1 \
  | grep -Ei 'backup|timed out after 120s' \
  | tail -100
2026-07-28T05:11:34.999213271Z 2026-07-28 07:11:34,999 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_070333.tar.gz
2026-07-28T05:11:36.686211953Z 2026-07-28 07:11:36,685 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_060310.tar.gz
2026-07-28T05:11:36.686246943Z 2026-07-28 07:11:36,686 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_070333.tar.gz
2026-07-28T05:13:36.781406699Z 2026-07-28 07:13:36,781 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T05:19:32.401797537Z 2026-07-28 07:19:32,401 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_071136.tar.gz
2026-07-28T05:19:34.517213427Z 2026-07-28 07:19:34,516 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_061126.tar.gz
2026-07-28T05:19:34.517324119Z 2026-07-28 07:19:34,516 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_071136.tar.gz
2026-07-28T05:21:35.187052966Z 2026-07-28 07:21:35,186 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T05:27:28.664598441Z 2026-07-28 07:27:28,664 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_071935.tar.gz
2026-07-28T05:27:30.982370145Z 2026-07-28 07:27:30,982 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_061949.tar.gz
2026-07-28T05:27:30.982399448Z 2026-07-28 07:27:30,982 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_071935.tar.gz
2026-07-28T05:29:31.447941284Z 2026-07-28 07:29:31,447 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T05:36:24.280949825Z 2026-07-28 07:36:24,280 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_072731.tar.gz
2026-07-28T05:36:27.532134680Z 2026-07-28 07:36:27,531 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_062800.tar.gz
2026-07-28T05:36:27.532277166Z 2026-07-28 07:36:27,532 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_072731.tar.gz
2026-07-28T05:38:28.074676032Z 2026-07-28 07:38:28,074 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T05:44:16.570116011Z 2026-07-28 07:44:16,569 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_073628.tar.gz
2026-07-28T05:44:18.404081352Z 2026-07-28 07:44:18,403 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_063830.tar.gz
2026-07-28T05:44:18.404171032Z 2026-07-28 07:44:18,404 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_073628.tar.gz
2026-07-28T05:46:19.402384522Z 2026-07-28 07:46:19,402 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T05:52:36.757072789Z 2026-07-28 07:52:36,756 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_074419.tar.gz
2026-07-28T05:52:38.307388916Z 2026-07-28 07:52:38,307 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_064659.tar.gz
2026-07-28T05:52:38.307435222Z 2026-07-28 07:52:38,307 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_074419.tar.gz
2026-07-28T05:54:38.556534537Z 2026-07-28 07:54:38,556 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T06:00:16.994920722Z 2026-07-28 08:00:16,994 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_075238.tar.gz
2026-07-28T06:00:19.337244600Z 2026-07-28 08:00:19,337 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_065522.tar.gz
2026-07-28T06:00:19.337281266Z 2026-07-28 08:00:19,337 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_075238.tar.gz
2026-07-28T06:02:19.949002300Z 2026-07-28 08:02:19,948 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T06:08:07.708837085Z 2026-07-28 08:08:07,708 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_080019.tar.gz
2026-07-28T06:08:09.975110561Z 2026-07-28 08:08:09,974 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_070333.tar.gz
2026-07-28T06:08:09.975209980Z 2026-07-28 08:08:09,974 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_080019.tar.gz
2026-07-28T06:10:10.594647992Z 2026-07-28 08:10:10,594 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T06:16:08.228904059Z 2026-07-28 08:16:08,228 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_080810.tar.gz
2026-07-28T06:16:10.201544457Z 2026-07-28 08:16:10,201 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_071136.tar.gz
2026-07-28T06:16:10.201582162Z 2026-07-28 08:16:10,201 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_080810.tar.gz
2026-07-28T06:18:11.126167469Z 2026-07-28 08:18:11,125 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T06:25:20.540680938Z 2026-07-28 08:25:20,540 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_081611.tar.gz
2026-07-28T06:25:22.992392604Z 2026-07-28 08:25:22,992 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_071935.tar.gz
2026-07-28T06:25:22.992445132Z 2026-07-28 08:25:22,992 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_081611.tar.gz
2026-07-28T06:27:23.318674206Z 2026-07-28 08:27:23,318 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T06:33:44.444540114Z 2026-07-28 08:33:44,444 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_082523.tar.gz
2026-07-28T06:33:46.504751768Z 2026-07-28 08:33:46,504 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_072731.tar.gz
2026-07-28T06:33:46.509770822Z 2026-07-28 08:33:46,509 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_082523.tar.gz
2026-07-28T06:35:55.881918468Z 2026-07-28 08:35:55,881 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T06:44:00.863820506Z 2026-07-28 08:44:00,863 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_083355.tar.gz
2026-07-28T06:44:03.028857843Z 2026-07-28 08:44:03,028 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_073628.tar.gz
2026-07-28T06:44:03.028907672Z 2026-07-28 08:44:03,028 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_083355.tar.gz
2026-07-28T06:46:03.723089649Z 2026-07-28 08:46:03,722 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T06:52:09.309186919Z 2026-07-28 08:52:09,309 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_084403.tar.gz
2026-07-28T06:52:11.243436460Z 2026-07-28 08:52:11,243 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_074419.tar.gz
2026-07-28T06:52:11.243502955Z 2026-07-28 08:52:11,243 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_084403.tar.gz
2026-07-28T06:54:12.107248587Z 2026-07-28 08:54:12,107 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T07:00:44.409053983Z 2026-07-28 09:00:44,408 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_085212.tar.gz
2026-07-28T07:00:46.317378826Z 2026-07-28 09:00:46,317 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_075238.tar.gz
2026-07-28T07:00:46.317411944Z 2026-07-28 09:00:46,317 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_085212.tar.gz
2026-07-28T07:02:47.239380529Z 2026-07-28 09:02:47,239 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T07:08:56.177712132Z 2026-07-28 09:08:56,177 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_090047.tar.gz
2026-07-28T07:08:58.324187913Z 2026-07-28 09:08:58,324 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_080019.tar.gz
2026-07-28T07:08:58.324235628Z 2026-07-28 09:08:58,324 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_090047.tar.gz
2026-07-28T07:10:59.033968054Z 2026-07-28 09:10:59,033 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T07:17:02.572130961Z 2026-07-28 09:17:02,571 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_090859.tar.gz
2026-07-28T07:17:05.180616719Z 2026-07-28 09:17:05,180 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_080810.tar.gz
2026-07-28T07:17:05.180661388Z 2026-07-28 09:17:05,180 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_090859.tar.gz
2026-07-28T07:19:05.444543363Z 2026-07-28 09:19:05,444 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T07:25:11.637637917Z 2026-07-28 09:25:11,637 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_091705.tar.gz
2026-07-28T07:25:13.549150187Z 2026-07-28 09:25:13,548 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_081611.tar.gz
2026-07-28T07:25:13.549194990Z 2026-07-28 09:25:13,549 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_091705.tar.gz
2026-07-28T07:27:14.409747963Z 2026-07-28 09:27:14,409 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T07:32:53.225332925Z 2026-07-28 09:32:53,225 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_092514.tar.gz
2026-07-28T07:32:55.070595818Z 2026-07-28 09:32:55,070 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_082523.tar.gz
2026-07-28T07:32:55.070765874Z 2026-07-28 09:32:55,070 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_092514.tar.gz
2026-07-28T07:34:56.035813381Z 2026-07-28 09:34:56,035 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T07:41:20.785860244Z 2026-07-28 09:41:20,784 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_093256.tar.gz
2026-07-28T07:41:23.027260402Z 2026-07-28 09:41:23,026 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_083355.tar.gz
2026-07-28T07:41:23.027343717Z 2026-07-28 09:41:23,027 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_093256.tar.gz
2026-07-28T07:43:23.591263536Z 2026-07-28 09:43:23,591 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T07:49:39.357982929Z 2026-07-28 09:49:39,357 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_094123.tar.gz
2026-07-28T07:49:40.744740106Z 2026-07-28 09:49:40,744 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_084403.tar.gz
2026-07-28T07:49:40.744983795Z 2026-07-28 09:49:40,744 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_094123.tar.gz
2026-07-28T07:51:41.151375265Z 2026-07-28 09:51:41,151 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T07:57:48.778434037Z 2026-07-28 09:57:48,778 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_094941.tar.gz
2026-07-28T07:57:50.589329650Z 2026-07-28 09:57:50,589 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_085212.tar.gz
2026-07-28T07:57:50.589357272Z 2026-07-28 09:57:50,589 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_094941.tar.gz
2026-07-28T07:59:50.628799915Z 2026-07-28 09:59:50,628 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T08:06:17.743889110Z 2026-07-28 10:06:17,743 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_095750.tar.gz
2026-07-28T08:06:19.622686565Z 2026-07-28 10:06:19,622 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_090047.tar.gz
2026-07-28T08:06:19.622733592Z 2026-07-28 10:06:19,622 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_095750.tar.gz
2026-07-28T08:08:19.729443691Z 2026-07-28 10:08:19,729 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T08:14:59.649985430Z 2026-07-28 10:14:59,649 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_100619.tar.gz
2026-07-28T08:15:02.331153011Z 2026-07-28 10:15:02,330 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_090859.tar.gz
2026-07-28T08:15:02.331194204Z 2026-07-28 10:15:02,331 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_100619.tar.gz
2026-07-28T08:17:02.548052109Z 2026-07-28 10:17:02,547 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T08:23:26.624337214Z 2026-07-28 10:23:26,624 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_101502.tar.gz
2026-07-28T08:23:28.730975197Z 2026-07-28 10:23:28,730 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_091705.tar.gz
2026-07-28T08:23:28.731008474Z 2026-07-28 10:23:28,730 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_101502.tar.gz
2026-07-28T08:25:29.772249700Z 2026-07-28 10:25:29,772 [ERROR] docsis.main: backup: timed out after 120s
2026-07-28T08:31:28.154341479Z 2026-07-28 10:31:28,154 [INFO] docsis.backup: Backup saved to /backup/docsight_backup_2026-07-28_102329.tar.gz
2026-07-28T08:31:30.527474822Z 2026-07-28 10:31:30,527 [INFO] docsis.backup: Deleted old backup: docsight_backup_2026-07-28_092514.tar.gz
2026-07-28T08:31:30.527506481Z 2026-07-28 10:31:30,527 [INFO] docsis.collector.backup: Scheduled backup created: docsight_backup_2026-07-28_102329.tar.gz
2026-07-28T08:33:31.058521710Z 2026-07-28 10:33:31,058 [ERROR] docsis.main: backup: timed out after 120s

docker inspect --format '{{.RestartCount}} {{.State.StartedAt}} {{.State.OOMKilled}} {{.State.Pid}}' docsight
0 2026-07-26T02:03:31.217789465Z false 3818768

docker exec -u 0 docsight sh -c '
  grep -E "^(VmRSS|VmHWM|Threads):" /proc/1/status
  cat /sys/fs/cgroup/memory.events
'
VmHWM:  29619220 kB
VmRSS:  29547792 kB
Threads:        17
low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0

docker exec -u 0 docsight python -c '
import sqlite3
p = "/data/docsis_history.db"
db = sqlite3.connect(f"file:{p}?mode=ro", uri=True)
for name in ("page_size", "page_count", "freelist_count"):
    print(name, db.execute(f"PRAGMA {name}").fetchone()[0])
print("snapshots", db.execute("SELECT COUNT(*), MIN(timestamp), MAX(timestamp) FROM snapshots").fetchone())
'
page_size 4096
page_count 3840667
freelist_count 2
snapshots (8114, '2026-04-29T08:51:07Z', '2026-07-28T08:38:31Z')

[itsDNNS]

Thank you — this confirms the diagnosis.

• There was no restart or OOM event: RestartCount=0, OOMKilled=false, and all cgroup OOM counters are zero.
• The memory is owned by the DOCSight process itself: VmRSS is about 28.18 GiB and VmHWM about 28.25 GiB.
• The logs show the scheduled backup repeatedly running for roughly 8 minutes. Every run crosses the 120-second orchestrator limit, its later successful completion is not recorded, and another overdue backup starts immediately. The repeated timed out after 120s → Backup saved → next timeout sequence is the scheduler defect we identified.
• The database contains about 14.65 GiB of live pages. With only two pages on the freelist, VACUUM INTO cannot materially shrink it. This also makes the observed archive-sized memory amplification quantitatively consistent with the in-memory backup implementation.

I am therefore treating the repeating scheduled backup plus the full in-memory archive copy as the root cause of the reported memory pattern. Both code paths are being corrected: archives will be written to disk without full RAM copies, and long-running collectors will remain tracked until their result is recorded instead of being resubmitted.

Until a fixed release is available, please temporarily disable automatic backups and restart the container once to release the retained memory. Avoid manually creating a backup from the UI during that period, because the current manual path also builds the archive in memory. Existing database contents are not affected by disabling the schedule.

[itsDNNS]

Fixed and merged in #755.

The reporter data confirmed two interacting defects:

• backup archives were fully materialized and copied in process memory for a 14.65 GiB live database;
• backups running longer than 120 seconds were no longer tracked after the timeout, so their successful completion was not recorded and the scheduler started another overdue backup.

The merged change writes archives through disk-backed temporary files, publishes scheduled backups atomically, cleans manual-download files when the response closes, and retains long-running collector futures until their actual result is processed. Logical collector names are also kept single-flight during configuration hot-swaps.

Validation completed successfully:

• 3002 non-browser tests passed, 1 skipped;
• Linux, Windows, mobile E2E, full browser E2E, audit, build, i18n, and CodeQL checks passed in CI;
• a 64 MiB incompressible archive smoke test reduced peak RSS from 156.0 MiB to 27.7 MiB while producing the same archive size.

The fix will be available with the next release. Until the installation has been updated, keep automatic backups disabled and restart the current container once to release the retained memory. After updating, automatic backups can be enabled again.

[itsDNNS]

This fix is now available in DOCSight v2026-07-28.1 and in the matching Docker image tags:

• ghcr.io/itsdnns/docsight:v2026-07-28.1
• ghcr.io/itsdnns/docsight:latest

Thanks again for the report and the detailed diagnostics that helped confirm the cause.