Datadog Destination missing on Kinesis Firehose

[boyer-victor]

Description

Kinesis firehose supports Datadog as a destination. This requires configuring the following (according to the Console UI, includes default configurations and non-datadog specifics):
HTTP Endpoint Name : Datadog
Authentication: Use API Key / Secrets Manager
API Key: Sensitive string
Content Encoding: Not enabled/ Gzip
Retry Duration: 0-7200s, default 60s
Parameters: key-value pairs
Buffer size: 1Mib-4Mib, reccomended 4 Mib
Buffer internal: 0s-900s, reccomended 60s

Affected Resource(s) or Data Source(s)

• aws_kinesis_firehose_delivery_stream

Potential Terraform Configuration

resource "aws_kinesis_firehose_delivery_stream" "extended_s3_stream" {
  name        = "terraform-kinesis-firehose-extended-s3-test-stream"
  destination = "datadog"

  datadog_configuration {
    buffering_interval = 4 # default 4Mib
    buffering_size = 60 # default 60s

    authentication {
      api_key = "key" # or
      secret_name = "arn:aws:secretsmanager:us-east-1:111222333444:secret:datadog-api-ID0o2R"
    }

    content_encoding = "GZIP" # default "", disabled

    s3_configuration {
      role_arn           = aws_iam_role.firehose_role.arn
      bucket_arn         = aws_s3_bucket.bucket.arn
      buffering_size     = 10
      buffering_interval = 400
      compression_format = "GZIP"
    }

    s3_backup_configuration {
      role_arn           = aws_iam_role.firehose_role.arn
      bucket_arn         = aws_s3_bucket.bucket.arn
      buffering_size     = 15
      buffering_interval = 300
      compression_format = "GZIP"
    }
    
    advanced_settings {
      enable_sse = true
      send_logs_to_cloudwatch = true
      service_access_iam = aws_iam_role.firehose_role.arn
      
    }
  }
  
  tags = {}
}

References

Not yet supported in the go sdk. I understand this will need to wait until then.

Would you like to implement the enhancement?

No

[github-actions]

Community Guidelines

This comment is added to every new Issue to provide quick reference to how the Terraform AWS Provider is maintained. Please review the information below, and thank you for contributing to the community that keeps the provider thriving! 🚀

Voting for Prioritization

• Please vote on this Issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize it.
• Please see our prioritization guide for additional information on how the maintainers handle prioritization.
• Please do not leave +1 or other comments that do not add relevant new information or questions; they generate extra noise for others following the Issue and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.

[tabito-hara]

@boyer-victor

Thank you for your report.

The Datadog endpoint is one kind of HTTP endpoints.
You can configure it as follows:

resource "aws_kinesis_firehose_delivery_stream" "test_stream" {
  name        = "terraform-kinesis-firehose-test-stream"
  destination = "http_endpoint"
  http_endpoint_configuration {
    url                = "datadog_endpoint"
    name               = "Datadog"
    access_key         = "your_api_key"

 ....
  }
 ....
}

Reference
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream#http-endpoint-eg-new-relic-destination

[boyer-victor]

@tabito-hara Got it, apoligies I should have dug further into if an existing solution (HTTP in this case) would actually cover the use case. I was creating this through the api hence my confusion when terraform did not directly document the datadog usecase. Would it be beneficial to open a pr with an example for datadog configuration on the terraform provider documentation?

[tabito-hara]

@boyer-victor

Thank you for the clarification — no worries at all!

If you encounter any issues while trying to output to Datadog, please feel free to report them here.

I think opening a PR to clarify that the existing HTTP endpoint example also applies to Datadog would be very helpful. It would definitely improve the documentation for others who are using the API and might encounter the same confusion.

[AdamJCavanaugh]

I don't see how this example/documentation would be substantially different from the linked NewRelic example. Perhaps a link in the http_endpoint_configuration block to this documentation or that documentation instead?

If we want to move forward with a Datadog destination example, I'm happy to do it - let me know if we'd prefer a minimal example as @tabito-hara provided or a complete one.