Skip to content

Update new package README template #2707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docs/howto/add_package_readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,18 @@ Each package should have a README to explain all the details related to a given

In order to help developers, this file is intended to be auto-generated by the `elastic-package build` command.
The template used to generate the final README must be located at `_dev/build/docs/README.md`.
This is the file that developers should be writting/updating.
This is the file that developers should be writing/updating.

As a note, there could be more than one readme file present in `_dev/build/docs/*.md`.
`elastic-package build` command will render and write all those files into `docs/` folder.

When generating README files, `elastic-package` automatically prepends a header comment to indicate that the file
is auto-generated and should not be edited manually:
```
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
```

# Markdown templates

Files in `_dev/build/docs/*.md` follow Markdown syntax and they are rendered using [text/template](https://pkg.go.dev/text/template) package.
Expand Down
6 changes: 6 additions & 0 deletions internal/docs/readme.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,11 @@ type ReadmeFile struct {
Error error
}

const (
doNotModifyStr string = `<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->`
)

// AreReadmesUpToDate function checks if all the .md readme files are up-to-date.
func AreReadmesUpToDate() ([]ReadmeFile, error) {
packageRoot, err := packages.MustFindPackageRoot()
Expand Down Expand Up @@ -210,6 +215,7 @@ func renderReadme(fileName, packageRoot, templatePath string, linksMap linkMap)
}

var rendered bytes.Buffer
fmt.Fprintln(&rendered, doNotModifyStr)
err = t.Execute(&rendered, nil)
if err != nil {
return nil, fmt.Errorf("executing template failed: %w", err)
Expand Down
24 changes: 18 additions & 6 deletions internal/docs/readme_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,9 @@ func TestGenerateReadme(t *testing.T) {
readmeTemplateContents: `
# README
Introduction to the package`,
expected: `
expected: `<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->

# README
Introduction to the package`,
},
Expand Down Expand Up @@ -83,7 +85,9 @@ func TestRenderReadmeWithLinks(t *testing.T) {
Introduction to the package
{{ url "foo" }}
{{ url "foo" "Example" }}`,
expected: `
expected: `<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->

# README
Introduction to the package
http://www.example.com/bar
Expand Down Expand Up @@ -127,7 +131,9 @@ func TestRenderReadmeWithSampleEvent(t *testing.T) {
# README
Introduction to the package
{{ event "example" }}`,
expected: `
expected: `<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->

# README
Introduction to the package
An example event for ` + "`example`" + ` looks as following:
Expand Down Expand Up @@ -184,7 +190,9 @@ func TesRenderReadmeWithFields(t *testing.T) {
# README
Introduction to the package
{{ fields }}`,
expected: `
expected: `<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->

# README
Introduction to the package
**Exported fields**
Expand All @@ -207,7 +215,9 @@ Introduction to the package
# README
Introduction to the package
{{ fields "example" }}`,
expected: `
expected: `<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->

# README
Introduction to the package
**Exported fields**
Expand All @@ -230,7 +240,9 @@ Introduction to the package
# README
Introduction to the package
{{ fields "notexist" }}`,
expected: `
expected: `<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->

# README
Introduction to the package
**Exported fields**
Expand Down
134 changes: 84 additions & 50 deletions internal/packages/archetype/_static/package-docs-readme.md.tmpl
Original file line number Diff line number Diff line change
@@ -1,84 +1,118 @@
<!-- Use this template language as a starting point, replacing {placeholder text} with details about the integration. -->
<!-- Find more detailed documentation guidelines in https://github.com/elastic/integrations/blob/main/docs/documentation_guidelines.md -->
<!-- This template can be used as a starting point for writing documentation for your new integration. For each section, fill in the details
described in the comments.

# {{.Manifest.Title}}
Find more detailed documentation guidelines in https://www.elastic.co/docs/extend/integrations/documentation-guidelines
-->

<!-- The {{.Manifest.Title}} integration allows you to monitor {name of service}. {name of service} is {describe service}.
# {{.Manifest.Title}} Integration for Elastic

Use the {{.Manifest.Title}} integration to {purpose}. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference {data stream type} when troubleshooting an issue.
## Overview

For example, if you wanted to {sample use case} you could {action}. Then you can {visualize|alert|troubleshoot} by {action}. -->
<!-- Complete this section with a short summary of what data this integration collects and what use cases it enables -->
The {{.Manifest.Title}} integration for Elastic enables collection of ...
This integration facilitates ...

## Data streams
### Compatibility

<!-- The {{.Manifest.Title}} integration collects {one|two} type{s} of data streams: {logs and/or metrics}. -->
<!-- Complete this section with information on what 3rd party software or hardware versions this integration is compatible with -->
This integration is compatible with ...

<!-- If applicable -->
<!-- **Logs** help you keep a record of events happening in {service}.
Log data streams collected by the {name} integration include {sample data stream(s)} and more. See more details in the [Logs](#logs-reference). -->
### How it works

<!-- If applicable -->
<!-- **Metrics** give you insight into the state of {service}.
Metric data streams collected by the {name} integration include {sample data stream(s)} and more. See more details in the [Metrics](#metrics-reference). -->
<!-- Add a high level overview on how this integration works. For example, does it collect data from API calls or recieving data from a network or file.-->

<!-- Optional: Any additional notes on data streams -->
## What data does this integration collect?

## Requirements
<!-- Complete this section with information on what types of data the integration collects, and link to reference documentation if available -->
The {{.Manifest.Title}} integration collects log messages of the following types:
* ...

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
### Supported use cases

<!--
Optional: Other requirements including:
* System compatibility
* Supported versions of third-party products
* Permissions needed
* Anything else that could block a user from successfully using the integration
-->
<!-- Add details on the use cases that can be enabled by using this integration. Explain why a user would want to install and use this integration. -->

## Setup
## What do I need to use this integration?

<!-- Any prerequisite instructions -->
<!-- List any vendor-specific prerequisites needed before starting to install the integration. -->

For step-by-step instructions on how to set up an integration, see the
[Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide.
## How do I deploy this integration?

<!-- Additional set up instructions -->
### Agent-based deployment

<!-- If applicable -->
<!-- ## Logs reference -->
Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host.

<!-- Repeat for each data stream of the current type -->
<!-- ### {Data stream name}
Elastic Agent is required to stream data from the syslog or log file receiver and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.

<!-- If agentless is available for this integration, we'll want to include that here as well. -->
<!-- ### Agentless deployment

Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. Agentless deployments provide a means to ingest data while avoiding the orchestration, management, and maintenance needs associated with standard ingest infrastructure. Using an agentless deployment makes manual agent deployment unnecessary, allowing you to focus on your data instead of the agent that collects it.

For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html)
-->

### Onboard / configure

<!-- List the steps that will need to be followed in order to completely set up a working inte completely set up a working integration.
For integrations that support multiple input types, be sure to add steps for all inputs.
-->

### Validation

The `{data stream name}` data stream provides events from {source} of the following types: {list types}. -->
<!-- How can the user test whether the integration is working? Including example commands or test files if applicable -->

<!-- Optional -->
<!-- #### Example
## Troubleshooting

An example event for `{data stream name}` looks as following:
For help with Elastic ingest tools, check [Common problems](https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems).

{code block with example} -->
<!-- Add any vendor specific troubleshooting here.

<!-- #### Exported fields
Are there common issues or “gotchas” for deploying this integration? If so, how can they be resolved?
If applicable, links to the third-party software’s troubleshooting documentation.
-->

## Scaling

{insert table} -->
For more information on architectures that can be used for scaling this integration, check the [Ingest Architectures](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures) documentation.

<!-- If applicable -->
<!-- ## Metrics reference -->
<!-- Add any vendor specific scaling information here -->

## Reference

<!-- Repeat for each data stream of the current type -->
<!-- ### {Data stream name}

The `{data stream name}` data stream provides events from {source} of the following types: {list types}. -->
The `{data stream name}` data stream provides events from {source} of the following types: {list types}.

For each data_stream_name, include an optional summary of the datastream, the exported fields reference table and the sample event. -->

<!-- The fields template function will be replaced by a generated list of all fields from the `fields/` directory of the data stream when building the integration. -->
<!--
#### {data stream name} fields

To include a generated list of fields from the `fields/` directory, uncomment and use:
(Remove the spaces between curly braces when using)
{ { fields "data_stream_name" } }
Comment on lines +94 to +95
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can directly generate the expected curly braces.

Suggested change
(Remove the spaces between curly braces when using)
{ { fields "data_stream_name" } }
{{ "{{ fields \"data_stream_name\" }}" }}

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano if I try to change it to that (and the other one), I get test failures

=== Failed
=== FAIL: internal/packages/archetype TestDataStream/valid-logs (0.01s)
New package has been created: /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/_build-test-1661138179/go_unit_test_package
New data stream has been created: go_unit_test_data_stream
    package_test.go:116: 
                Error Trace:    /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/package_test.go:116
                                                        /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/data_stream_test.go:71
                                                        /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/data_stream_test.go:22
                Error:          Received unexpected error:
                                updating readme file README.md failed: rendering Readme failed: executing template failed: template: README.md:95:3: executing "README.md" at <fields "data_stream_name">: error calling fields: can't create fields validator instance (path: /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/_build-test-1661138179/go_unit_test_package/data_stream/data_stream_name): package root not found and dependency management is enabled
                Test:           TestDataStream/valid-logs

=== FAIL: internal/packages/archetype TestDataStream/valid-metrics (0.01s)
New package has been created: /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/_build-test-668793173/go_unit_test_package
New data stream has been created: go_unit_test_data_stream
    package_test.go:116: 
                Error Trace:    /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/package_test.go:116
                                                        /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/data_stream_test.go:71
                                                        /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/data_stream_test.go:29
                Error:          Received unexpected error:
                                updating readme file README.md failed: rendering Readme failed: executing template failed: template: README.md:95:3: executing "README.md" at <fields "data_stream_name">: error calling fields: can't create fields validator instance (path: /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/_build-test-668793173/go_unit_test_package/data_stream/data_stream_name): package root not found and dependency management is enabled
                Test:           TestDataStream/valid-metrics

=== FAIL: internal/packages/archetype TestDataStream (0.03s)

=== FAIL: internal/packages/archetype TestPackage/valid (0.01s)
New package has been created: /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/_build-test-1228717663/go_unit_test_package
    package_test.go:116: 
                Error Trace:    /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/package_test.go:116
                                                        /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/package_test.go:46
                                                        /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/package_test.go:24
                Error:          Received unexpected error:
                                updating readme file README.md failed: rendering Readme failed: executing template failed: template: README.md:95:3: executing "README.md" at <fields "data_stream_name">: error calling fields: can't create fields validator instance (path: /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/_build-test-1228717663/go_unit_test_package/data_stream/data_stream_name): package root not found and dependency management is enabled
                Test:           TestPackage/valid

=== FAIL: internal/packages/archetype TestPackage/input-package (0.01s)
New package has been created: /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/_build-test-3570254830/go_unit_test_package
    package_test.go:116: 
                Error Trace:    /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/package_test.go:116
                                                        /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/package_test.go:46
                                                        /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/package_test.go:33
                Error:          Received unexpected error:
                                updating readme file README.md failed: rendering Readme failed: executing template failed: template: README.md:95:3: executing "README.md" at <fields "data_stream_name">: error calling fields: can't create fields validator instance (path: /Users/kymeli/Documents/git/elastic-package/internal/packages/archetype/_build-test-3570254830/go_unit_test_package/data_stream/data_stream_name): package root not found and dependency management is enabled
                Test:           TestPackage/input-package

=== FAIL: internal/packages/archetype TestPackage (0.04s)

DONE 820 tests, 6 failures in 11.524s


<!-- Optional -->
<!-- #### Example
-->

<!-- The event template function will be replace by a sample event, taken from `sample_event.json`, when building this integration. -->
<!--

To include a sample event from `sample_event.json`, uncomment and use:
(Remove the spaces between curly braces when using)
{ { event "data_stream_name" } }
Comment on lines +103 to +104
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(Remove the spaces between curly braces when using)
{ { event "data_stream_name" } }
{{ "{{ event \"data_stream_name\" }}" }}


-->

An example event for `{data stream name}` looks as following:
### Inputs used

{code block with example} -->
<!-- List inputs used in this integration, and link to the documentation -->
These inputs can be used with this integration:
* ...

<!-- #### Exported fields
### API usage

{insert table} -->
<!-- For integrations that use APIs to collect data, document all the APIs that are used, and link to relevent information -->
These APIs are used with this integration:
* ...
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"description": "This is an example sample-event for {{.Manifest.Title}}. Replace it with a real sample event. Hint: If system tests exist, running `elastic-package test system --generate` will generate this file."
}
16 changes: 16 additions & 0 deletions internal/packages/archetype/package.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,14 @@ func createPackageInDir(packageDescriptor PackageDescriptor, cwd string) error {
return fmt.Errorf("can't render package README: %w", err)
}

if packageDescriptor.Manifest.Type != "content" {
logger.Debugf("Write docs readme to _dev")
err = renderResourceFile(packageDocsReadme, &packageDescriptor, filepath.Join(baseDir, "_dev", "build", "docs", "README.md"))
if err != nil {
return fmt.Errorf("can't render package README in _dev: %w", err)
}
}

if license := packageDescriptor.Manifest.Source.License; license != "" {
logger.Debugf("Write license file")
err = licenses.WriteTextToFile(license, filepath.Join(baseDir, "LICENSE.txt"))
Expand All @@ -78,6 +86,14 @@ func createPackageInDir(packageDescriptor PackageDescriptor, cwd string) error {
return fmt.Errorf("can't render sample screenshot: %w", err)
}

if packageDescriptor.Manifest.Type == "integration" {
logger.Debugf("Write sample sample_event")
err = renderResourceFile(packageSampleEvent, &packageDescriptor, filepath.Join(baseDir, "sample_event.json"))
if err != nil {
return fmt.Errorf("can't render sample sample_event: %w", err)
}
}

if packageDescriptor.Manifest.Type == "input" {
logger.Debugf("Write base fields")
err = renderResourceFile(fieldsBaseTemplate, &packageDescriptor, filepath.Join(baseDir, "fields", "base-fields.yml"))
Expand Down
3 changes: 3 additions & 0 deletions internal/packages/archetype/resources.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ var packageImgSampleIcon []byte
//go:embed _static/sampleScreenshot.png.b64
var packageImgSampleScreenshot string

//go:embed _static/package-sample-event.json.tmpl
var packageSampleEvent string

// Input Package templates

//go:embed _static/input-package-agent-config.yml.tmpl
Expand Down
2 changes: 2 additions & 0 deletions test/packages/false_positives/cisco_asa/docs/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Cisco ASA Integration

This integration is for Cisco ASA network device's logs. It includes the following
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Test integration
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Custom API input integration

The custom API input integration is used to ingest data from custom RESTful API's that do not currently have an existing integration.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Long Integer Tests

An example event for `test` looks as following:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Nginx Integration

This integration periodically fetches metrics from [Nginx](https://nginx.org/) servers. It can parse access and error
Expand Down
2 changes: 2 additions & 0 deletions test/packages/false_positives/prometheus/docs/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Prometheus Integration

This integration can collect metrics from:
Expand Down
2 changes: 2 additions & 0 deletions test/packages/other/ecs_mappings/docs/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# ECS Mappings Test

Test package to verify support for ECS mappings available to integrations running on stack version 8.13.0 and later.
Expand Down
2 changes: 2 additions & 0 deletions test/packages/other/fields_tests/docs/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Fields Tests

An example event for `first` looks as following:
Expand Down
2 changes: 2 additions & 0 deletions test/packages/other/imported_mappings_tests/docs/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Imported Mappings Tests

An example event for `first` looks as following:
Expand Down
2 changes: 2 additions & 0 deletions test/packages/other/long_integers/docs/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Long Integer Tests

An example event for `test` looks as following:
Expand Down
2 changes: 2 additions & 0 deletions test/packages/other/readme_links/docs/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Readme Links

Examples of links:
Expand Down
2 changes: 2 additions & 0 deletions test/packages/other/with_links/docs/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
<!-- NOTICE: Do not edit this file manually.-->
<!-- This file is automatically generated by Elastic Package -->
# Imported Mappings Tests

An example event for `first` looks as following:
Expand Down
Loading