Skip to content

refactor: remove NApps directory observer (watchdog) feature - #638

Merged
italovalcy merged 4 commits into
masterfrom
fix/issue-635
Sep 21, 2026
Merged

italovalcy merged 4 commits into
masterfrom
fix/issue-635

Conversation

@italovalcy

@italovalcy italovalcy commented Sep 16, 2026 •

Copy link
Copy Markdown

Closes #635

Summary

See updated changelog file.

The NApps directory observer (NAppDirListener) relied on watchdog/inotify to auto load/unload NApps when their enabled-directory symlink appeared or disappeared. Its default inotify instance limit (usually 128) is quickly exhausted when running heavy applications or many kytos instances in parallel on the same host, and when the limit is hit the observer fails to start and kytosd crashes on startup (see the traceback below).

This PR initially made the observer opt-in and fail-safe, but following review (thanks @quadflow) the approach was changed to fully remove the feature instead, since it was half-baked (its path regex assumed kytos/napps is always part of the path and did not handle custom NApp paths), an opt-in toggle would be confusing to document, and removing it lets us drop the watchdog dependency.

The observer's only job was to load/unload a NApp when its enabled-dir symlink changed. Disabling a NApp already calls unload_napp directly, so this PR makes enabling symmetric — NAppsManager.enable now calls load_napp directly after creating the symlink (the meta-NApp dependency loop was fixed to no longer shadow the enabled NApp's name, so a meta-NApp loads itself correctly) — and then removes the observer entirely:

  • Removed NAppDirListener and its wiring in Controller (start/stop).
  • Removed the enable_napps_observer config option / -N/--enable_napps_observer CLI flag / conf template entry that were added earlier in this PR.
  • Dropped the watchdog dependency from requirements/.

The original problem

In a docker container with a single Kytos instance, the crash can be reproduced by lowering the inotify limit:

root@73badf2a0bef:/src/kytos# sysctl -w fs.inotify.max_user_watches=16
fs.inotify.max_user_watches = 16
root@73badf2a0bef:/src/kytos# kytosd -f
...
Kytos couldn't start because of [Errno 28] inotify watch limit reached Traceback (most recent call last):
  File ".../kytos/core/controller.py", line 414, in start_controller
    self.napp_dir_listener.start()
  File ".../kytos/core/napps/napp_dir_listener.py", line 39, in start
    self.observer.start()
  ...
  File ".../watchdog/observers/inotify_c.py", line 413, in _raise_error
    raise OSError(errno.ENOSPC, "inotify watch limit reached")
OSError: [Errno 28] inotify watch limit reached

Shutting down Kytos...
Terminated

With this PR the observer (and the watchdog dependency) no longer exists, so this failure path is gone entirely, and enabling/disabling a NApp loads/unloads it directly.

Local Tests

Ran locally.

  • Deleted tests/unit/test_core/test_napp_dir_listener.py (feature removed) and the observer-specific controller tests.
  • tests/unit/test_core/test_napps_manager.py::test_enable now asserts the NApp is loaded via controller.load_napp after being enabled.
  • Full tests/unit suite passing.

End-to-End Tests

Not needed for this case; there are no e2e tests covering the NApps observer. Enabling/disabling NApps through the existing REST endpoints continues to load/unload them.

The NApps directory observer uses inotify, whose default instance limit
(usually 128) is easily exhausted when running many kytos instances on the
same host, causing kytosd to crash on startup.

- Wrap the observer start in a try/except so a failure is logged (with
  traceback) and kytosd keeps running without it.
- Add an ``enable_napps_observer`` setting (kytos.conf) and a
  ``-N``/``--enable_napps_observer`` CLI flag, disabled by default, to make
  the observer opt-in.
@italovalcy
italovalcy marked this pull request as ready for review September 16, 2026 17:00
@italovalcy
italovalcy requested a review from a team as a code owner September 16, 2026 17:00
@italovalcy italovalcy self-assigned this Sep 16, 2026
@italovalcy
italovalcy requested a review from quadflow September 16, 2026 17:01
Comment thread kytos/core/controller.py Outdated
Following review on PR #638, fully remove the watchdog-based NApps
directory observer (NAppDirListener) instead of keeping it as an opt-in
feature. The observer was half-baked (its path regex did not handle
custom NApp paths) and could exhaust the system inotify instance limit
when running many Kytos instances on the same host.

The observer's only job was to load/unload a NApp when its enabled-dir
symlink appeared/disappeared. Disabling already unloads directly, so make
enabling load directly too (symmetric), and drop the observer entirely:

- NAppsManager.enable now calls controller.load_napp after creating the
  symlink; the meta-NApp dependency loop no longer shadows the enabled
  NApp's name so a meta-NApp loads itself correctly.
- Delete NAppDirListener and its wiring in Controller.
- Remove the enable_napps_observer option (config, CLI flag, conf template)
  added earlier in this PR.
- Drop the watchdog dependency from requirements.
@italovalcy italovalcy changed the title fix: make NApps observer opt-in and fail-safe on startup refactor: remove NApps directory observer (watchdog) feature Sep 18, 2026
@italovalcy
italovalcy requested a review from quadflow September 18, 2026 17:44

@quadflow quadflow left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome how this turned out, thanks @italovalcy.

I also ran it locally with a custom napps path, I completely reinstalled the env from scratch, disabled/enabled napps, no unexpected side effects observed.

@italovalcy
italovalcy merged commit 42c8e60 into master Sep 21, 2026
2 checks passed
@italovalcy
italovalcy deleted the fix/issue-635 branch September 21, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Napps observer should have a fallback alternative to not crash Kytos

2 participants