[DOCS-11051] Add Azure Event Hub using Kafka source #30211
Conversation
📝 Documentation Team Review RequiredThis pull request requires approval from the @DataDog/documentation team before it can be merged. Please ensure your changes follow our documentation guidelines and wait for a team member to review and approve your changes. |
Preview links (active after the
|
| 1. Toggle the switch to enable SASL authentication. | ||
| 1. In the **Mechanism** dropdown menu, select **PLAIN**. | ||
| 1. Enable TLS. | ||
| 1. Download the certificate from [https://curl.se/docs/caextract.html](https://curl.se/docs/caextract.html) and save it to `/var/lib/observability-pipelines-worker/config/cert.pem`. |
There was a problem hiding this comment.
Maybe we should host this cert in docs? I don't think this third party site is something we can rely on. But I think certs expire, so I am not really sure what to do here. I found this out via: https://datadoghq.atlassian.net/wiki/spaces/PRODUCTSA/pages/5118492913/One+Oncology+2025#:~:text=Third%20TLS%20needs%20to%20be%20enabled.%20But%20I%20didn%27t%20know%20what%20cert%20to%20use.%20I%20found%20this%20really%20old%20github%20issue%20post
There was a problem hiding this comment.
We might also want to be specific saying "Save this cert to this path on your OP worker host/container"? If it is containerized they will likely have to mount a volume to load it.
This is a bit chicken and egg problem because this guide assumes they haven't deployed OPW yet, so that directory won't yet exist until OPW has been installed.
There was a problem hiding this comment.
Does this work?
| 1. Download the certificate from [https://curl.se/docs/caextract.html](https://curl.se/docs/caextract.html) and save it to `/var/lib/observability-pipelines-worker/config/cert.pem`. | |
| 1. Download the certificate from [https://curl.se/docs/caextract.html](https://curl.se/docs/caextract.html) and save it to this path on your Worker host or container:`/var/lib/observability-pipelines-worker/config/cert.pem`. If you are using a container, you might have to mount a volume to load it. |
There was a problem hiding this comment.
Maybe we should host this cert in docs? I don't think this third party site is something we can rely on. But I think certs expire, so I am not really sure what to do here. I found this out via:
Hm I'm also not sure what's best here either..if it expires, how does someone find another one to use in this case?
There was a problem hiding this comment.
Suggested change looks good to me.
Regarding cert, I don't really know 😓 I could ask the broader PSA team and see if someone smarter than me on these things has any thoughts?
There was a problem hiding this comment.
I asked PSA team for suggestions, I am out of my depth here
There was a problem hiding this comment.
Someone Smarter than me helped me out :) @krlv (Thank you so much!)
His verbatim response below:
okay, I think I got it. Azure Events Hub supports SASL/PLAIN auth only over SSL encrypted connection (kinda make sense -- you're sending the password in a plain text over the wire). So in your case, OPW has to use TLS encryption for the authentication request.
Moreover, based on this comment, i have a hunch that any trusted root cert will work -- even the one from the ca-certificates/OpenSSL. if you're running OPW on a linux instance, you should probably use
/etc/ssl/certs/ca-certificates.crtor/usr/lib/ssl/certs/ca-certificates.crtinstead of downloading it from the cURL website.Back to your original question in GH -- no, we shouldn't host the CA certs on our docs site. We don't host root certs; even when we changed the one used to sign Datadog certs, we just pointed users to download it directly from the DigiCert website: https://docs.datadoghq.com/data_security/guide/tls_cert_chain_of_trust/?tab=g1rootcertificateold#action-needed
My 2 cents on the doc: I'd highly recommend testing the ca-certificates package (using
/etc/ssl/certs/ca-certificates.crtas the "Certificate path" in the OP UI) before suggesting customers download and mount root CA cert from the internet. Hopefully OPW can handle .crt files in addition to .pem. If this works, we can simply reference ca-certificates in the docs and provide/etc/ssl/certs/ca-certificates.crtas an example.
@emarsha94 and/or I will need to test this, I think we must do this before we proceed with publishing. I'll also check container land for these certs to see if they can be used. Hopefully our Azure Event Hubs environment is still around so we can test this without having to set everything up again 🤞
There was a problem hiding this comment.
Slacked this as well, but putting here for visibility:
If it's going to take some time to test, do you think we could just say something like:
Download a trusted root certificate and save it to this path on your Worker host or container:
/var/lib/observability-pipelines-worker/config/cert.pem. If you're running the Worker on a Linux instance, you should use the certifiate:/etc/ssl/certs/ca-certificates.crtor/usr/lib/ssl/certs/ca-certificates.crt. If you are using a container, you might have to mount a volume to load the certificate.
There was a problem hiding this comment.
I just don't know if those certs already exist on a standard distribution or in our container image and if they'll work, thus why I want to test again.
| 1. For **Kafka Bootstrap Servers**, enter `\<NAMESPACE\>.servicebus.windows.net:9093` (for example, `myeventhubns.servicebus.windows.net:9093`). | ||
| 1. For **Kafka SASL Username**, enter `$ConnectionString`. | ||
| 1. For **Kafka SASL Password**, enter the full connection string (for example, `Endpoint=sb://<NAMESPACE>.servicebus.windows.net/;SharedAccessKeyName=<PolicyName>;SharedAccessKey=<Key>`). | ||
| 1. Enter your Kafka TLS passphrase. |
There was a problem hiding this comment.
I think we left this blank in our test, but @emarsha94 will have to confirm -- I don't see in either of our notes where we denoted needing this.
There was a problem hiding this comment.
No, this is the art we didnt give to the customer before in the notes, and we had to update. $$ConnectionString is definitely needed for instance
There was a problem hiding this comment.
Since I am going to have to set this up to test it again for the certificate, I'll make note if we leave this blank or not when we get it to a working state.
layouts/shortcodes/observability_pipelines/destination_settings/microsoft_sentinel.md
Outdated
Show resolved
Hide resolved
Co-authored-by: cecilia saixue watt <[email protected]>
|
Thanks @cswatt! I've commit your suggestion, could you re-review? |
What does this PR do? What is the motivation?
Merge instructions
Merge readiness:
For Datadog employees:
Your branch name MUST follow the
<name>/<description>convention and include the forward slash (/). Without this format, your pull request will not pass CI, the GitLab pipeline will not run, and you won't get a branch preview. Getting a branch preview makes it easier for us to check any issues with your PR, such as broken links.If your branch doesn't follow this format, rename it or create a new branch and PR.
[6/5/2025] Merge queue has been disabled on the documentation repo. If you have write access to the repo, the PR has been reviewed by a Documentation team member, and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #documentation channel in Slack.
Additional notes