Skip to content

feat: incident.io Notifier #4372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

rorymalcolm
Copy link

@rorymalcolm rorymalcolm commented Apr 27, 2025

Closes #4367

  • Adds the technical implementation, and tests, for the incident.io notifier

  • Configured through the following config:

receivers:
  - name: 'incidentio-notifications'
    incidentio_configs:
      - url: '$alert_source_url'
        alert_source_token: '$alert_source_token'

@rorymalcolm rorymalcolm changed the title - Adds the technical implementation, and tests, for the incident.io n… incident.io Notifier Apr 27, 2025
@rorymalcolm rorymalcolm force-pushed the rorymalcolm/incidentio-notifier branch 2 times, most recently from 98ed08f to 416cac1 Compare April 27, 2025 22:09
@rorymalcolm rorymalcolm changed the title incident.io Notifier feat: incident.io Notifier Apr 27, 2025
@rorymalcolm rorymalcolm force-pushed the rorymalcolm/incidentio-notifier branch 2 times, most recently from 2de7c7f to 118a310 Compare April 27, 2025 22:28
- Adds the technical implementation, and tests, for the incident.io notifier

- Configured through the following config:

```yaml
receivers:
  - name: 'incidentio-notifications'
    incidentio_configs:
      - url: '$alert_source_url'
        alert_source_token: '$alert_source_token'
```

Signed-off-by: Rory Malcolm <[email protected]>
@rorymalcolm rorymalcolm force-pushed the rorymalcolm/incidentio-notifier branch from 118a310 to 4bc00a7 Compare April 28, 2025 09:49
@@ -521,6 +528,49 @@ func (c *SlackConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
return nil
}

// IncidentioConfig configures notifications via incident.io.
type IncidentioConfig struct {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about adding a field Metadata to this struct? Something similar to the Details field in the OpsgenieConfig struct.
This would enable users to define additional data (see incident.io API definition for reference).

@grobinson-grafana
Copy link
Collaborator

grobinson-grafana commented May 3, 2025

Looks good, a couple comments. It also needs docs here and here 👍

@rorymalcolm rorymalcolm force-pushed the rorymalcolm/incidentio-notifier branch 3 times, most recently from 17d6a6f to 255befe Compare May 7, 2025 07:06
@rorymalcolm
Copy link
Author

Looks good, a couple comments. It also needs docs here and here 👍

All done - I think? 🙏

@grobinson-grafana
Copy link
Collaborator

You have some lint failures in notify/incidentio/incidentio_test.go

@rorymalcolm rorymalcolm force-pushed the rorymalcolm/incidentio-notifier branch 2 times, most recently from 8f16fcd to bbc39ce Compare May 9, 2025 16:14
@grobinson-grafana
Copy link
Collaborator

You still have some failing tests I'm afraid https://github.com/prometheus/alertmanager/actions/runs/14933150149/job/41982648209?pr=4372

--- FAIL: TestIncidentIORetry (0.01s)
    incidentio_test.go:48: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:48
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIORetry
--- FAIL: TestIncidentIORedactedURL (0.01s)
    incidentio_test.go:69: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:69
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIORedactedURL
--- FAIL: TestIncidentIOURLFromFile (0.01s)
    incidentio_test.go:91: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:91
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIOURLFromFile
--- FAIL: TestIncidentIONotify (0.01s)
    incidentio_test.go:[143](https://github.com/prometheus/alertmanager/actions/runs/14933150149/job/41982648209?pr=4372#step:6:144): 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:143
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIONotify
--- FAIL: TestIncidentIORetryScenarios (0.03s)
    --- FAIL: TestIncidentIORetryScenarios/success_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/success_response
    --- FAIL: TestIncidentIORetryScenarios/rate_limit_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/rate_limit_response
    --- FAIL: TestIncidentIORetryScenarios/server_error_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/server_error_response
    --- FAIL: TestIncidentIORetryScenarios/client_error_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/client_error_response

- Made a change to demarcate `AlertSourceToken` as required

- Now error if an authorization header is set, and a alert_source_token is set

- Ensure the alert source endpoint is no longer secret

- Add documentation for the incidentio_config

Signed-off-by: Rory Malcolm <[email protected]>
@rorymalcolm rorymalcolm force-pushed the rorymalcolm/incidentio-notifier branch from bbc39ce to 07604e3 Compare May 11, 2025 21:03
@rorymalcolm
Copy link
Author

You still have some failing tests I'm afraid https://github.com/prometheus/alertmanager/actions/runs/14933150149/job/41982648209?pr=4372

--- FAIL: TestIncidentIORetry (0.01s)
    incidentio_test.go:48: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:48
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIORetry
--- FAIL: TestIncidentIORedactedURL (0.01s)
    incidentio_test.go:69: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:69
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIORedactedURL
--- FAIL: TestIncidentIOURLFromFile (0.01s)
    incidentio_test.go:91: 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:91
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIOURLFromFile
--- FAIL: TestIncidentIONotify (0.01s)
    incidentio_test.go:[143](https://github.com/prometheus/alertmanager/actions/runs/14933150149/job/41982648209?pr=4372#step:6:144): 
        	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:143
        	Error:      	Received unexpected error:
        	            	one of alert_source_token or alert_source_token_file must be configured
        	Test:       	TestIncidentIONotify
--- FAIL: TestIncidentIORetryScenarios (0.03s)
    --- FAIL: TestIncidentIORetryScenarios/success_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/success_response
    --- FAIL: TestIncidentIORetryScenarios/rate_limit_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/rate_limit_response
    --- FAIL: TestIncidentIORetryScenarios/server_error_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/server_error_response
    --- FAIL: TestIncidentIORetryScenarios/client_error_response (0.01s)
        incidentio_test.go:223: 
            	Error Trace:	/__w/alertmanager/alertmanager/notify/incidentio/incidentio_test.go:223
            	Error:      	Received unexpected error:
            	            	one of alert_source_token or alert_source_token_file must be configured
            	Test:       	TestIncidentIORetryScenarios/client_error_response

Ah - apols; fixed!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support an incident.io notifier
3 participants