Skip to content

feat(agent): Add debounce for watch events #17048

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

neelayu
Copy link
Contributor

@neelayu neelayu commented May 21, 2025

Summary

This PR implements a debounce mechanism for configuration file watching that batches multiple file changes into a single reload operation.

Changes Made

  • Added --watch-debounce-interval CLI flag to configure the debounce duration (default: 0s for backward compatibility)
  • Implemented timer-based debouncing in watchLocalConfig() function that:
    • Starts a timer when the first file change is detected
    • Resets the timer if additional changes occur during the wait period
    • Only triggers reload (SIGHUP) after the timer expires without interruption
  • Enhanced state tracking with reloadPending flag to prevent unnecessary operations
  • Applied debouncing to all file events: Modified, Deleted, Truncated, and Created
  • Proper resource cleanup with timer management in all exit paths

Behavior Change

  • Before: 500 config file changes → 500 separate reloads
  • After: 500 config file changes → 1 batched reload (when debounce interval > 0)
  • Backward Compatible: Default behavior unchanged (0s debounce = immediate reload)

Usage Example

# Enable debouncing with 5-second wait period
telegraf --config-directory /configs --watch-config=notify --watch-debounce-interval=5s

Checklist

  • No AI generated code was used in this PR

Related issues

resolves #17047

telegraf git:(debounce-reload) ✗ go run . --config-directory debounce --watch-config=notify --config agent.conf --watch-debounce-interval=10s
2025-05-21T16:52:12Z I! Loading config: agent.conf
2025-05-21T16:52:12Z I! Loading config: debounce/file0.conf
2025-05-21T16:52:12Z I! Loading config: debounce/file1.conf
2025-05-21T16:52:12Z I! Loading config: debounce/file2.conf
2025-05-21T16:52:12Z I! Starting Telegraf unknown brought to you by InfluxData the makers of InfluxDB
2025-05-21T16:52:12Z I! Available plugins: 241 inputs, 9 aggregators, 34 processors, 26 parsers, 64 outputs, 5 secret-stores
2025-05-21T16:52:12Z I! Loaded inputs: mem (3x)
2025-05-21T16:52:12Z I! Loaded aggregators:
2025-05-21T16:52:12Z I! Loaded processors:
2025-05-21T16:52:12Z I! Loaded secretstores:
2025-05-21T16:52:12Z I! Loaded outputs: file (3x)
2025-05-21T16:52:12Z I! Tags enabled: host=hostname
2025-05-21T16:52:12Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"hostname", Flush Interval:10s
2025-05-21T16:52:12Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
2025-05-21T16:52:12Z I! Config watcher started for debounce/file0.conf
2025-05-21T16:52:12Z I! Config watcher started for debounce/file2.conf
2025-05-21T16:52:12Z I! Config watcher started for debounce/file1.conf
2025-05-21T16:52:12Z I! Config watcher started for agent.conf
2025-05-21T16:52:12Z I! Config watcher started for debounce
2025-05-21T16:52:28Z I! Config file/directory "debounce" modified
2025-05-21T16:52:28Z I! Config file/directory "debounce" modified
2025-05-21T16:52:28Z I! Config file/directory "debounce" modified
2025-05-21T16:52:38Z I! Debounce period elapsed, triggering config reload for "debounce"
2025-05-21T16:52:38Z I! Reloading Telegraf config
2025-05-21T16:52:38Z I! [agent] Hang on, flushing any cached metrics before shutdown
2025-05-21T16:52:38Z I! [agent] Stopping running outputs
2025-05-21T16:52:38Z I! Config watcher started for agent.conf
2025-05-21T16:52:38Z I! Config watcher started for debounce/file0.conf
2025-05-21T16:52:38Z I! Loading config: agent.conf
2025-05-21T16:52:38Z I! Config watcher started for debounce/file2.conf
2025-05-21T16:52:38Z I! Config watcher started for debounce/file1.conf
2025-05-21T16:52:38Z I! Config watcher started for debounce
2025-05-21T16:52:38Z I! Loading config: debounce/file0.conf
2025-05-21T16:52:38Z I! Loading config: debounce/file1.conf
2025-05-21T16:52:38Z I! Loading config: debounce/file2.conf
2025-05-21T16:52:38Z I! Starting Telegraf unknown brought to you by InfluxData the makers of InfluxDB
2025-05-21T16:52:38Z I! Available plugins: 241 inputs, 9 aggregators, 34 processors, 26 parsers, 64 outputs, 5 secret-stores
2025-05-21T16:52:38Z I! Loaded inputs: mem (3x)
2025-05-21T16:52:38Z I! Loaded aggregators:
2025-05-21T16:52:38Z I! Loaded processors:
2025-05-21T16:52:38Z I! Loaded secretstores:
2025-05-21T16:52:38Z I! Loaded outputs: file (3x)
2025-05-21T16:52:38Z I! Tags enabled: host=hostname
2025-05-21T16:52:38Z I! [agent] Config: Interval:120s, Quiet:false, Hostname:"hostname", Flush Interval:10s
2025-05-21T16:52:38Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!

Essentially, the config watcher will wait for 10 seconds after the last change to the config directory before reloading the config. This is useful when you have multiple files in a directory and you want to avoid reloading the config multiple times if they are changed in quick succession.

@telegraf-tiger telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label May 21, 2025
@telegraf-tiger
Copy link
Contributor

@skartikey
Copy link
Contributor

@neelayu Could you please rebase your branch on top of master? The changes to tls.go are already present in the master branch.

@skartikey skartikey self-assigned this May 28, 2025
@neelayu neelayu force-pushed the debounce-reload branch from 687c44b to aa9f773 Compare May 28, 2025 15:34
@neelayu
Copy link
Contributor Author

neelayu commented May 28, 2025

Yes. I think I added it to check if tests were passing or not. Rebased now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support debounce while watching configuration files and directories
2 participants