Skip to content

Commit 01b77bd

Browse files
guilloadestherk15
andauthored
Add minor fixes (#29401)
* Add minor fixes * Relative reference link --------- Co-authored-by: Esther Kim <[email protected]>
1 parent 3033158 commit 01b77bd

File tree

7 files changed

+54
-41
lines changed

7 files changed

+54
-41
lines changed

content/en/cloudprem/_index.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
title: CloudPrem
33
description: Learn how to deploy and manage Datadog CloudPrem, a self-hosted log management solution for cost-effective log ingestion, processing, indexing, and search capabilities
4-
private: true
4+
private: true
55
---
66

77
<div class="alert alert-warning">CloudPrem is in Preview.</div>
@@ -25,7 +25,7 @@ Datadog CloudPrem is a self-hosted log management solution that enables cost-eff
2525
{{< /whatsnext >}}
2626

2727

28-
[1]: https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/
28+
[1]: https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/
2929
[2]: /cloudprem/installation/
3030
[3]: /cloudprem/processing/
31-
[4]: /cloudprem/architecture/
31+
[4]: /cloudprem/architecture/

content/en/cloudprem/architecture.md

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -40,14 +40,17 @@ The CloudPrem cluster, typically deployed on Kubernetes (EKS), consists of sever
4040
: Responsible for receiving logs from Datadog Agents. Indexers process, index, and store logs in index files called _splits_ to the object storage (for example, Amazon S3).
4141

4242
**Searchers**
43-
: Handle search queries from the Datadog UI, reading metadata from Metastore and index data from the object storage.
43+
: Handle search queries from the Datadog UI, reading metadata from Metastore and fetching data from the object storage.
4444

4545
**Metastore**
46-
: Stores metadata about the indexes, including splits locations on the object storage. CloudPrem uses PostgreSQL for this purpose.
46+
: Stores metadata about the indexes, including split locations on the object storage. CloudPrem uses PostgreSQL for this purpose.
4747

48-
**Janitor/Control Plane**
49-
: Responsible for tasks like indexing tasks scheduling and delete tasks.
48+
**Control Plane**
49+
: Schedules indexing jobs called _indexing pipelines_ on indexers.
50+
51+
**Janitor**
52+
: Performs maintenance tasks, applying retention policies, garbage collecting expired splits, and running delete query jobs.
5053

5154
## Further reading
5255

53-
{{< partial name="whats-next/whats-next.html" >}}
56+
{{< partial name="whats-next/whats-next.html" >}}

content/en/cloudprem/aws_config.md

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,9 @@ To deploy CloudPrem on AWS, you need to configure:
3535

3636
When starting a node, CloudPrem attempts to find AWS credentials using the credential provider chain implemented by [rusoto\_core::ChainProvider][2] and looks for credentials in this order:
3737

38-
1. Environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, or `AWS_SESSION_TOKEN` (optional).
39-
2. Credential profiles file, typically located at `~/.aws/credentials` or otherwise specified by the `AWS_SHARED_CREDENTIALS_FILE` and `AWS_PROFILE` environment variables if set and not empty.
40-
3. Amazon ECS container credentials, loaded from the Amazon ECS container if the environment variable `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` is set.
38+
1. Environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, or `AWS_SESSION_TOKEN` (optional).
39+
2. Credential profiles file, typically located at `~/.aws/credentials` or otherwise specified by the `AWS_SHARED_CREDENTIALS_FILE` and `AWS_PROFILE` environment variables if set and not empty.
40+
3. Amazon ECS container credentials, loaded from the Amazon ECS container if the environment variable `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` is set.
4141
4. Instance profile credentials, used on Amazon EC2 instances, and delivered through the Amazon EC2 metadata service.
4242

4343
An error is returned if no credentials are found in the chain.
@@ -46,20 +46,20 @@ An error is returned if no credentials are found in the chain.
4646

4747
CloudPrem attempts to find the AWS region from multiple sources, using the following order of precedence:
4848

49-
1. **Environment variables**: Checks `AWS_REGION`, then `AWS_DEFAULT_REGION`.
50-
2. **AWS config file**: Typically located at `~/.aws/config`, or at the path specified by the `AWS_CONFIG_FILE` environment variable (if set and not empty).
51-
3. **EC2 instance metadata**: Uses the region of the currently running Amazon EC2 instance.
49+
1. **Environment variables**: Checks `AWS_REGION`, then `AWS_DEFAULT_REGION`.
50+
2. **AWS config file**: Typically located at `~/.aws/config`, or at the path specified by the `AWS_CONFIG_FILE` environment variable (if set and not empty).
51+
3. **EC2 instance metadata**: Uses the region of the currently running Amazon EC2 instance.
5252
4. **Default**: Falls back to `us-east-1` if no other source provides a region.
5353

5454
## IAM permissions for S3
5555

5656
Required authorized actions:
5757

58-
* `ListBucket` (on the bucket directly)
59-
* `GetObject`
60-
* `PutObject`
61-
* `DeleteObject`
62-
* `ListMultipartUploadParts`
58+
* `ListBucket` (on the bucket directly)
59+
* `GetObject`
60+
* `PutObject`
61+
* `DeleteObject`
62+
* `ListMultipartUploadParts`
6363
* `AbortMultipartUpload`
6464

6565
Here is an example of a bucket policy:
@@ -97,7 +97,7 @@ Here is an example of a bucket policy:
9797

9898
## Further reading
9999

100-
{{< partial name="whats-next/whats-next.html" >}}
100+
{{< partial name="whats-next/whats-next.html" >}}
101101

102102
[1]: /cloudprem/ingress/
103-
[2]: https://docs.rs/rusoto_credential/latest/rusoto_credential/struct.ChainProvider.html
103+
[2]: https://docs.rs/rusoto_credential/latest/rusoto_credential/struct.ChainProvider.html

content/en/cloudprem/cluster.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,4 +71,4 @@ The following components are typically lightweight:
7171

7272
## Further reading
7373

74-
{{< partial name="whats-next/whats-next.html" >}}
74+
{{< partial name="whats-next/whats-next.html" >}}

content/en/cloudprem/ingress.md

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -47,15 +47,15 @@ This setup ensures that only authenticated Datadog services can access the Cloud
4747

4848
{{< img src="/cloudprem/ingress/public_ingress.png" alt="Diagram showing CloudPrem public ingress architecture with Datadog services connecting through an internet-facing AWS ALB using mTLS authentication to access the CloudPrem gRPC API" style="width:100%;" >}}
4949

50-
### IP ranges API
51-
The Datadog control plane and query services connect to CloudPrem clusters using a set of fixed IP ranges, which can be retrieved for each Datadog site from the Datadog IP Ranges API, specifically under the "webhooks" section. For example, to fetch the IP ranges for the datadoghq.eu site, you can run:
50+
### IP Allowlisting
51+
52+
The Datadog control plane and query services connect to CloudPrem clusters using a set of fixed IP ranges, which can be retrieved for each Datadog site from the Datadog [IP Ranges API][1], specifically under the "webhooks" section. For example, to fetch the IP ranges for the datadoghq.eu site, you can run:
5253
```
5354
curl -X GET "https://ip-ranges.datadoghq.eu/" \
5455
-H "Accept: application/json" |
5556
jq '.webhooks'
5657
```
5758

58-
5959
## Internal ingress
6060

6161
The internal ingress enables log ingestion from Datadog Agents and other log collectors within your environment through HTTP.
@@ -89,7 +89,7 @@ rules:
8989
service:
9090
name: <RELEASE_NAME>-indexer
9191
port:
92-
name: rest
92+
name: rest
9393
- path: /api/v2/logs
9494
pathType: Prefix
9595
backend:
@@ -120,4 +120,6 @@ rules:
120120

121121
## Further reading
122122

123-
{{< partial name="whats-next/whats-next.html" >}}
123+
{{< partial name="whats-next/whats-next.html" >}}
124+
125+
[1]: /api/latest/ip-ranges/

content/en/cloudprem/installation.md

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ Before getting started with CloudPrem, ensure you have:
5050
3. [Configure the Datadog Agent to send Kubernetes logs](#send-kubernetes-logs-to-cloudprem-with-the-datadog-agent).
5151
4. [Configure your Datadog account](#configure-your-datadog-account).
5252

53-
## Install the CloudPrem Helm chart
53+
## Install the CloudPrem Helm chart
5454

5555
1. Add and update the Datadog Helm repository:
5656

@@ -70,7 +70,7 @@ Before getting started with CloudPrem, ensure you have:
7070
```bash
7171
kubectl create secret generic <SECRET_NAME> \
7272
-n <NAMESPACE_NAME> \
73-
--from-literal QW_METASTORE_URI=postgres://<USERNAME>:<PASSWORD>@<ENDPOINT>:<PORT>/<DATABASE>
73+
--from-literal QW_METASTORE_URI=postgres://<USERNAME>:<PASSWORD>@<ENDPOINT>:<PORT>/<DATABASE>
7474
```
7575

7676
4. Customize the Helm chart
@@ -200,6 +200,14 @@ Before getting started with CloudPrem, ensure you have:
200200
memory: "16Gi"
201201
```
202202
203+
5. Install or upgrade the Helm chart
204+
205+
```bash
206+
helm upgrade --install <RELEASE_NAME> datadog/cloudprem \
207+
-n <NAMESPACE_NAME> \
208+
-f datadog-values.yaml
209+
```
210+
203211
To check if the deployment went well:
204212
- Look at the logs from the metastore or indexer pod.
205213
- Look at the status of deployment resources, such as the ingress status.
@@ -224,13 +232,13 @@ spec:
224232
keyName: api-key
225233
env:
226234
- name: DD_LOGS_CONFIG_LOGS_DD_URL
227-
value: http://<RELEASE_NAME>-indexer.<NAMESPACE>.svc.cluster.local:7280
235+
value: http://<RELEASE_NAME>-indexer.<NAMESPACE_NAME>.svc.cluster.local:7280
228236

229237
features:
230238
logCollection:
231239
enabled: true
232240
containerCollectAll: true
233-
241+
234242
otlp:
235243
receiver:
236244
protocols:
@@ -244,7 +252,7 @@ spec:
244252

245253
```
246254

247-
- Within the cluster, use the indexer service for the logs endpoint URL: `DD_LOGS_CONFIG_LOGS_DD_URL:http://<RELEASE_NAME>-indexer.<NAMESPACE>.svc.cluster.local:7280`.
255+
- Within the cluster, use the indexer service for the logs endpoint URL: `DD_LOGS_CONFIG_LOGS_DD_URL:http://<RELEASE_NAME>-indexer.<NAMESPACE_NAME>.svc.cluster.local:7280`.
248256
- Outside the cluster, use the host of the internal ingress.
249257
- To send cluster metrics to Datadog, enable `prometheusScrape`.
250258
- To send cluster logs to Datadog, enable `OTLP/gRPC`.
@@ -266,7 +274,7 @@ After your Datadog account is configured, you are ready to search into the `clou
266274
To uninstall CloudPrem:
267275

268276
```bash
269-
helm uninstall <DEPLOYMENT_NAME>
277+
helm uninstall <RELEASE_NAME>
270278
```
271279

272280
## Further reading

content/en/cloudprem/processing.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -32,13 +32,13 @@ further_reading:
3232

3333
## Overview
3434

35-
CloudPrem includes a processing feature that allows you to parse and enrich logs. It automatically parses logs formatted in JSON. You can define pipelines and processors to extract meaningful information or attributes from semi-structured text, which can then be used for aggregations.
35+
CloudPrem includes a processing feature that allows you to parse and enrich logs. It automatically parses logs formatted in JSON. You can define pipelines and processors to extract meaningful information or attributes from semi-structured text, which can then be used for aggregations.
3636

3737
<div class="alert alert-info">CloudPrem log pipelines and processors are designed to be match the capabilities of Datadog's <a href="/logs/log_configuration/pipelines/?tab=source">cloud-based log pipelines and processors</a>. </div>
3838

3939
For a list of supported and unsupported processors, see [Compatibility with cloud-based pipelines](#compatibility-with-cloud-based-pipelines).
4040

41-
You configure log processing pipelines using a JSON file that adheres to the same format as Datadog pipeline configurations.
41+
You can configure log processing pipelines using a JSON file that adheres to the same format as Datadog pipeline configurations.
4242

4343
## Setting up processing
4444

@@ -48,24 +48,24 @@ You configure log processing pipelines using a JSON file that adheres to the sam
4848
curl -X GET "https://api.datadoghq.com/api/v1/logs/config/pipelines" \
4949
-H "Accept: application/json" \
5050
-H "DD-API-KEY: ${DD_API_KEY}" \
51-
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" > my-pipelines_config.json
51+
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" > pipelines-config.json
5252
```
5353

54-
This JSON file can be used directly with CloudPrem.
54+
This JSON file can be used directly with CloudPrem.
5555

5656
2. To set the configuration in the Helm Chart, provide the path to your JSON configuration file using the `pipelinesConfig` parameter in the CloudPrem Helm chart:
5757

5858
```bash
5959
helm repo update
60-
helm upgrade <RELEASE_NAME> -n <NAMESPACE> --set-file pipelinesConfig=./pipelines_config.json -f your_custom_values_file.yaml
60+
helm upgrade <RELEASE_NAME> -n <NAMESPACE_NAME> --set-file pipelinesConfig=./pipelines-config.json -f datadog-values.yaml
6161
```
6262

6363
CloudPrem logs an informational message (`Successfully read pipeline config file`) when it successfully reads the configuration file. Any processors defined in the file that are not supported by CloudPrem are ignored during startup.
6464
**Note**: Helm imposes a 1 MB size limit on the configuration file due to its underlying etcd storage.
6565

6666
## Configuration file format
6767

68-
The configuration is a JSON array, where each element represents a processor or a nested pipeline.
68+
The configuration is a JSON array, where each element represents a processor or a nested pipeline.
6969

7070
The order of elements in the array defines the sequential execution order of the processors. The structure mirrors the output of the Datadog API endpoint `api/v1/logs/config/pipelines`.
7171

@@ -150,8 +150,8 @@ Ignored processors appear as a warning in the indexer logs.
150150

151151
## Further reading
152152

153-
{{< partial name="whats-next/whats-next.html" >}}
153+
{{< partial name="whats-next/whats-next.html" >}}
154154

155155
[1]: /logs/log_configuration/pipelines/?tab=source
156156
[2]: /api/latest/logs-pipelines/#get-all-pipelines
157-
[3]: /logs/log_configuration/processors/
157+
[3]: /logs/log_configuration/processors/

0 commit comments

Comments
 (0)