Skip to content

Sharp memory increase and OOM after filter update on OpenWrt #8491

Closed
#8498
Closed
Sharp memory increase and OOM after filter update on OpenWrt#8491
#8498
Assignees
Chinaski1
Labels
Priority: P4waiting for dataWaiting for users to provide more data.

Description

@Daedalus86
Daedalus86
opened on Jul 25, 2026

Prerequisites

Platform (OS and CPU architecture)

Linux, ARM64

Installation

Custom package (OpenWrt, HomeAssistant, etc; please mention in the description)

Setup

On a router, DHCP is handled by the router

AdGuard Home version

v0.107.78

Action

AdGuard Home is running on an OpenWrt router with approximately 1 GB of RAM and no swap.

The problem occurs intermittently while AdGuard Home is operating normally.

During the most recently monitored occurrence, the following sequence was observed:

  1. At 23:00 local time, the AdGuard Home process with PID 12975 used 436528 kB RSS.
  2. At 23:23 local time, AdGuard Home started an automatic filter update.
  3. One of the updated filters contained 2103241 rules and had a downloaded size of 43460590 bytes.
  4. At 00:00 local time, the same AdGuard Home process with PID 12975 used 888148 kB RSS.
  5. AdGuard Home subsequently stopped responding.
  6. After AdGuard Home restarted, DNS resolution and the web interface worked again.

The AdGuard Home debug log uses UTC. The filter update therefore appears in the log at approximately 21:23 UTC.

Relevant AdGuard Home log entries:

2026/07/25 21:23:31.510459 12975#88 [debug] filtering: starting update

2026/07/25 21:23:33.545031 12975#88 [debug] filtering: downloading update for filter id=1758964205 url=https://adguardteam.github.io/HostlistsRegistry/assets/filter_44.txt

2026/07/25 21:23:46.202887 12975#88 [info] filtering: filter updated id=1758964205 bytes_written=43460590 rules_count=2103241

2026/07/25 21:23:46.382012 12975#88 [info] filtering: updated filter id=1758964205 rules_count=2103241 prev_rules_count=2080045

The automatic filter update strongly correlates with the sudden memory increase. I do not yet have a heap profile proving that the filter update is the direct root cause.

Expected result

AdGuard Home should update and reload filter lists without causing a permanent increase of several hundred megabytes in resident memory.

After the filter update has completed, memory usage should either return to approximately its previous level or remain low enough for AdGuard Home to continue operating normally.

DNS resolution and the AdGuard Home web interface should remain available after an automatic filter update.

Actual result

The resident memory usage of the same AdGuard Home process increased from 436528 kB to 888148 kB within the monitored one-hour interval.

Observed values:

2026-07-25 23:00 local time

PID: 12975
VmSize: 2250940 kB
VmRSS: 436528 kB
Threads: 14

2026-07-26 00:00 local time

PID: 12975
VmSize: 2250940 kB
VmRSS: 888148 kB
Threads: 14

The PID, virtual memory size and thread count remained the same, while resident memory increased by approximately 451620 kB.

AdGuard Home then stopped responding. DNS resolution and the AdGuard Home web interface were unavailable until AdGuard Home restarted.

The Linux kernel log also confirms that AdGuard Home has been killed by the OOM-killer on multiple occasions at a similar memory level:

Out of memory: Killed process 2464 (AdGuardHome) total-vm:2182124kB, anon-rss:880568kB, file-rss:1536kB, shmem-rss:0kB, UID:0 pgtables:1904kB oom_score_adj:0

Out of memory: Killed process 6193 (AdGuardHome) total-vm:2181940kB, anon-rss:878876kB, file-rss:4992kB, shmem-rss:0kB, UID:0 pgtables:1900kB oom_score_adj:0

These kernel log entries demonstrate that the problem has occurred more than once. They are not being presented as the kernel log entry for PID 12975 specifically.

Additional information and/or screenshots

Device: GL.iNet Flint 2 / GL-MT6000

Operating system: OpenWrt 24.10.7

CPU architecture: aarch64 / ARM64

AdGuard Home version: v0.107.78

Installation: standalone manual installation on OpenWrt

AdGuard Home binary: /opt/AdGuardHome/AdGuardHome

Configuration file: /opt/AdGuardHome/AdGuardHome.yaml

Available memory:

Approximately 1 GB RAM

The query log was checked during the relevant period and was not being rotated:

2026/07/25 21:03:05.978536 12975#90 [debug] querylog: oldest log entry entry_time=2026-07-22T20:46:44.143393617Z

2026/07/25 21:03:05.978680 12975#90 [debug] querylog: not rotating now=2026-07-25T21:03:05Z rotate_time=2026-08-21T20:46:44Z

A statistics flush also occurred at 21:00 UTC:

2026/07/25 21:00:00.340962 12975#89 [debug] stats: flushing unit id=495836 req_num=1342

I have enabled the AdGuard Home pprof endpoint and configured additional memory monitoring. The monitoring checks memory usage every 15 minutes and attempts to capture heap and goroutine profiles when configured RSS thresholds are exceeded.

At this moment, the automatic filter update is a strong correlation, but I do not yet have a heap profile proving that it directly causes the memory increase.

Activity

  1. Daedalus86
    changed the title [-]AdGuard Home is OOM-killed after automatic update of a large filter list[/-] [+]Intermittent sharp memory increase and OOM on OpenWrt[/+] on Jul 25, 2026

  2. Daedalus86 commented on Jul 25, 2026

    @Daedalus86
    Daedalus86
    on Jul 25, 2026
    Author

    I was able to reproduce the memory increase by manually checking the filter lists for updates.

    Before the manual check:

    VmSize: 2250940 kB
    VmRSS: 457456 kB
    Threads: 14

    During and after the check, RSS increased progressively:

    535000 kB
    675524 kB
    716432 kB
    767784 kB
    809700 kB
    833764 kB
    835640 kB
    870836 kB
    879084 kB

    The latest measurement was still 874696 kB RSS.

    VmSize remained 2250940 kB and the thread count remained 14 throughout the test. The same AdGuard Home process was monitored during the complete increase.

    The RSS therefore increased by approximately 421628 kB, or about 412 MiB, after manually checking the filter lists. It did not return to the previous level after the update activity appeared to finish.

    This makes the filter update/check operation reproducible as the trigger for the sharp memory increase, although I do not yet know which internal stage of the filter update causes the retained memory.

    After restarting AdGuard Home, both the thread count and memory usage returned to a lower baseline:

    Before restart, after the manual filter update:

    VmSize: 2250940 kB
    VmRSS: approximately 875000 kB
    Threads: 14

    After restarting AdGuard Home:

    VmSize: 1711620 kB
    VmRSS: 399216 kB
    Threads: 10

    The restart therefore reduced VmSize by approximately 539 MB and RSS by approximately 476 MB. The thread count also returned from 14 to 10.

    The four additional threads do not appear sufficient to explain the full RSS increase, but they may indicate that the filter update leaves additional worker threads or runtime resources active until the process is restarted.

  3. Daedalus86
    changed the title [-]Intermittent sharp memory increase and OOM on OpenWrt[/-] [+]Sharp memory increase and OOM after filter update on OpenWrt[/+] on Jul 25, 2026

  4. Master-of-Mouse commented on Jul 26, 2026

    @Master-of-Mouse
    Master-of-Mouse
    on Jul 26, 2026

    This looks like a regression introduced in one of the recent releases.

    I started seeing this issue roughly 4 AdGuard Home updates ago, and I am currently on v0.107.78.

    Any configuration change can make AdGuard Home hang completely. In my case, even a simple block/unblock action from the Query Log is enough to trigger the problem.

    Once it happens:

    • DNS resolution stops working for all clients on the LAN
    • Devices on the network can no longer browse the Internet because DNS queries are not answered
    • The AdGuard Home web interface becomes unreachable or unusable
    • I cannot revert the change from the UI
    • The only reliable recovery method is rebooting the entire LXC container from Proxmox

    Environment:

    • AdGuard Home v0.107.78
    • Ubuntu LXC container
    • Proxmox VE 9.2.4
    • Unprivileged container
    • Dedicated container for AdGuard Home

    This does not appear to be caused by lack of resources. The Proxmox host has 16 CPU threads and 62 GB RAM, with plenty of available capacity, and the AdGuard Home container does not show significant CPU or memory pressure before the hang.

    The impact is quite severe because AdGuard Home is the main DNS resolver for my network. When it hangs, the whole LAN loses DNS resolution until the container is rebooted.

    Please let me know which logs, debug settings, or diagnostic commands would be useful to collect the next time this happens, before rebooting the container.

  5. Daedalus86 commented on Jul 26, 2026

    @Daedalus86
    Daedalus86
    on Jul 26, 2026
    Author

    The issue started for me after updating to v0.107.78. I did not observe this behaviour in previous versions, although I cannot completely rule out that it may have occurred unnoticed during automatic nightly filter updates.

    This may be related to #8297, which reports periodic high memory usage and OOM kills after upgrading to v0.107.73. In my case, I was able to reproduce the sharp RSS increase by manually checking the DNS blocklists for updates.

  6. tjharman commented on Jul 26, 2026

    @tjharman
    tjharman
    on Jul 26, 2026

    Can those of you with this problem please indicate which Blocklist(s) you're using? Just the standard ones, or list any 3rd party ones you're using?
    Thank you.

  7. tjharman
    added
    waiting for dataWaiting for users to provide more data.
    on Jul 26, 2026

  8. Daedalus86 commented on Jul 26, 2026

    @Daedalus86
    Daedalus86
    on Jul 26, 2026
    Author

    These are the filters I am using:

    filters:

    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt
    name: AdGuard DNS filter

    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_50.txt
    name: uBlock₀ filters – Badware risks

    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_34.txt
    name: HaGeZi's Normal Blocklist

    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_44.txt
    name: HaGeZi's Threat Intelligence Feeds

    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_11.txt
    name: Malicious URL Blocklist (URLHaus)

    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_71.txt
    name: HaGeZi's DNS Rebind Protection

  9. Mahtabctg commented on Jul 27, 2026

    @Mahtabctg
    Mahtabctg
    on Jul 27, 2026

    AdGuard Home v0.107.78
    Device: Keenetic KN-1810
    It's happening for me also for MIPS build. After filter update or spike in DNS request memory increase got very high and AGH crashes. Rolled back to AdGuard Home v0.107.77. Old version also has memory leak but not as much as 78.

  10. tjharman commented on Jul 27, 2026

    @tjharman
    tjharman
    on Jul 27, 2026

    Thanks for the details. I have flagged this to the AGH team to have a look at - I expect they'll have more questions.
    Just to confirm though, rolling back to .77 makes the issue go away again?

  11. Mahtabctg commented on Jul 27, 2026

    @Mahtabctg
    Mahtabctg
    on Jul 27, 2026

    @tjharman Crash went away, but still minor memory leak is there as I can see memory usage increase over time. Probably previous versions also had it, bust didn't notice as application didn't crash very often.

  12. Daedalus86 commented on Jul 27, 2026

    @Daedalus86
    Daedalus86
    on Jul 27, 2026
    Author

    I created two valid heap profiles from the same AdGuard Home version and the same filter configuration:

    Baseline after restarting AdGuard Home:
    PID 26498
    VmRSS: 348464 kB
    VmSize: 1781452 kB
    Threads: 10

    Post-update state after the large memory spike had partially subsided:
    PID 28911
    VmRSS: approximately 431988 kB
    VmSize: 2250740 kB
    Threads: 12

    Both heap profiles were downloaded successfully from the pprof endpoint and passed gzip validation.

    The post-update process remained approximately 84 MB higher in RSS than the clean baseline, even after the earlier peak of approximately 890 MB had mostly subsided.

    Attached files:

    heap-baseline-after-restart.pb.gz
    heap-post-update-430mb.pb.gz

    The profiles were created on AdGuard Home v0.107.78 with the same enabled blocklists.

    heap-baseline-after-restart.pb.gz
    heap-post-update-430mb.pb.gz

  13. kurt-k commented on Jul 28, 2026

    @kurt-k
    kurt-k
    on Jul 28, 2026

    I have the same problem after the update

    Jul 28 21:42:11 adguard systemd[1]: AdGuardHome.service: A process of this unit has been killed by the OOM killer.
    Jul 28 21:42:11 adguard systemd[1]: AdGuardHome.service: Main process exited, code=killed, status=9/KILL
    Jul 28 21:42:11 adguard systemd[1]: AdGuardHome.service: Failed with result 'oom-kill'.
    Jul 28 21:42:11 adguard systemd[1]: AdGuardHome.service: Consumed 19min 8.983s CPU time, 506.4M memory peak, 500.5M memory swap peak
    Jul 28 21:42:21 adguard systemd[1]: AdGuardHome.service: Scheduled restart job, restart counter is at 8.
    Jul 28 21:42:21 adguard systemd[1]: Started AdGuardHome.service - AdGuard Home: Network-level blocker.

  14. netsecninja commented on Jul 29, 2026

    @netsecninja
    netsecninja
    on Jul 29, 2026

    I've downgraded to v0.107.77, and the OOM still happens daily. I've downgraded again to v0.107.76 and will see if it still locks up.

  15. alelaba commented on Jul 29, 2026

    @alelaba
    alelaba
    on Jul 29, 2026

    I don’t know which blocklists everyone in this thread is using in AGH, but HaGeZi's Threat Intelligence Feeds has now doubled in volume size in the past weeks.
    I suggest to use Mini version on these devices with less than 1 GB RAM
    Check HaGeZi’s page for details https://github.com/hagezi/dns-blocklists#tif

  16. 35 remaining items

  17. Daedalus86 commented on Aug 14, 2026

    @Daedalus86
    Daedalus86
    on Aug 14, 2026
    Author

    Could be worth checking how you pull HaGeZi's TIF (or his other lists) as for the last few days his github has been down. He has mirrors ofc (and internal adguard copies), but if you were pointing at the github and it couldnt grab the list, then that could mask when it would have increased memory usage.

    That explains some things, I use the URL's provided by AGH.
    But to get to the core if things I think the patch linked to this problem does improve "Memory management" in AGH. so I hope it is in released version soon.

  18. adguard-bot
    assigned
    Chinaski1
    on Aug 14, 2026
  19. adguard-bot
    added
    Priority: P4
    on Aug 14, 2026

  20. alelaba commented on Aug 14, 2026

    @alelaba
    alelaba
    on Aug 14, 2026

    Could be worth checking how you pull HaGeZi's TIF (or his other lists) as for the last few days his github has been down. He has mirrors ofc (and internal adguard copies), but if you were pointing at the github and it couldnt grab the list, then that could mask when it would have increased memory usage.

    That explains some things, I use the URL's provided by AGH. But to get to the core if things I think the patch linked to this problem does improve "Memory management" in AGH. so I hope it is in released version soon.

    I don’t think both situations have a link each other.
    When HaGeZi’s GitHub was down, his lists were just not updated in AGH GitHub repo. I verified that with the date written in the file header: the date was stuck on 8/9/2026.
    It should not use more memory usage on our AGH endpoints.

  21. vortexilation commented on Aug 15, 2026

    @vortexilation
    vortexilation
    on Aug 15, 2026

    @Daedalus86

    It seems that adguard teams has change the status of this ticket, and looks like they have no interest on accepting my PR,so i guess i leave it with them then, i hope that it will be great result.

  22. Daedalus86 commented on Aug 15, 2026

    @Daedalus86
    Daedalus86
    on Aug 15, 2026
    Author

    @vortexilation I hope they will use your solution. In any way thanks for your effort regarding this issue.

  23. vortexilation
    added a commit that references this issue on Aug 17, 2026

    filtering: collect before the engine rebuild too

    5030799

  24. mrdannywarren commented on Aug 22, 2026

    @mrdannywarren
    mrdannywarren
    on Aug 22, 2026

    I'm also seeing this on AdGuard Home v0.107.79.

    I have a large filtering setup using several HaGeZi lists, including the Threat Intelligence Feed with ~2.14 million rules. The instance starts around 400–500 MB RAM, but memory jumps significantly when the filtering engine is rebuilt, eventually reaching ~1 GB, heavy swap usage, and becoming unresponsive.

    I can reproduce the problem by simply adding a custom allowlist/whitelist rule. The filtering change triggers the same memory spike and AdGuard eventually becomes unresponsive.

    I also run a separate AdGuard Media instance with tiny filter lists and a much smaller workload. It remains stable at around 40 MB RAM and does not show the same issue.

    Once it becomes unresponsive, the only way I can recover it is to stop and restart the LXC container from Proxmox.

  25. EugeneOne1 commented on Sep 10, 2026

    @EugeneOne1
    EugeneOne1
    on Sep 10, 2026
    Member

    Hello everyone, thank you for the detailed reports, heap profiles and testing. They were very helpful.

    Having benchmarked the versions mentioned and analysed the profiles, we conclude that memory consumption scales proportionally with the total size of the enabled filtering lists. Therefore, this is not strictly a regression in AdGuard Home. With very large lists, such as "HaGeZi's Threat Intelligence Feeds" with around 2.1M rules, and limited RAM, the machine simply does not have enough memory for the filtering engines.

    We have also took a look at #8498, which contains some meaningful optimisations, and we appreciate the work that has gone into it. However, we currently don't have much time to properly finish it before merging, so we can't commit to a timeline. We will, however, look into it in the future and see what can be taken from it.

    In the meantime, we suggest that users of devices with less than ≈1 GiB of RAM use smaller filter lists (e.g. the "mini" variants of HaGeZi's lists). The memory footprint, including the peak during updates, is directly tied to the total list size.

    Thanks again for your patience, and for the shared data.

  26. EugeneOne1
    closed this as completedon Sep 10, 2026
  27. vortexilation
    added 2 commits that reference this issue on Sep 10, 2026

    filtering: fix engine rebuild memory usage

    8aa284d

    filtering: collect before the engine rebuild too

    6625f6b

  28. ZolKush commented on Sep 24, 2026

    @ZolKush
    ZolKush
    on Sep 24, 2026

    I encountered a similar failure on OpenWrt 25.12.5 / mediatek MT7622 with only ~182k filtering rules, rather than multi-million-rule lists.

    The router has ~243 MiB RAM and ~121 MiB zram. Shortly after an automatic filter update, AdGuard Home reached:

    VmRSS: 85916 kB
    VmSwap: 59588 kB

    System memory at the time:

    MemAvailable: 0
    Swap: 120708 / 120828 kB used
    Load average: 10.76 10.65 6.02
    CPU: 65% sys, 30% iowait, 0% idle

    Stopping AdGuard Home immediately recovered the router.

    After restarting with:

    GOGC=40
    GOMAXPROCS=1
    GOMEMLIMIT=104857600

    normal DNS traffic remained stable at roughly 60–67 MiB RSS with 0 swap.

    pprof under normal traffic did not show an obvious leak. The failure appears correlated with filter engine rebuild/update rather than ordinary DNS traffic.

    The largest active filter contains about 182,617 rules (~4.37 MB).

    I can collect a heap profile during the failure if it happens again.

  29. vortexilation
    added 2 commits that reference this issue on Oct 6, 2026

    filtering: fix engine rebuild memory usage

    e91cdb1

    filtering: collect before the engine rebuild too

    c7371cf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Priority: P4waiting for dataWaiting for users to provide more data.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions