OPCT-226: cmd/report UX enhancements for CLI and Web report

This change introduce several improvements in the UX while reviewing the
report by:
- creationg an intuitive HTML report allowing users to quickly see
  issues and navigate to the logs for each test failure
- introduce several gates/SLO/checks to be used as post-processor and
  get better visibility in the results, based in existing knowledge
base/CI data or external ssytems

See the PR with details of improvements:
#76

----

fix sippy query in the flake filter

create a rank of tags, adding percentage of counters

create html report saving data to json

improve report html

add log links and rank by error count

update camgi tab

add waiting interval support

status: improve message field with plugin phase

report: generating filter.html file with failures table

renameing vfs path for html templates

report html - update rank and output as json

using report as data source of CLI report; apply docs

add timers to the runtime report

update report collecting more env information

report: update html templates to improve filters

adding suite errors menu

Update suite error menu using native js

create a rank of error counters

adding an option to extract errors from must-gather

extracting must-gather event filters adding a tab to the report

adding the structure of report checks

introducing dynamic checks moving report to summary

Adding checks with acceptance values from baseline

adding alert pop to Checks menu

parallelism processing must-gather, loweing 3 times proc time

adding support to all plugins in report

http file server to serve report

review: documenting and linking check rules

reorg mkdocs/todo

report: embeding etcd error parser to json report

report/must-gather: consolidating etd parser

report: supporting etcd, network check and embed data

remove codegen script / not used

distributing report to packages

parsing meta/run.log extracting runtime information

introducing new parsers: opct and meta config

renaming sippy to its package

renaming packages and cleanup

review report html and threasholds

review timers for plugins

add etcd checks ; cleanup

review thresholds

bumping plugins image

review documentation and doc strings

update check platform type

fix tests in tags

fix dev docs

dev/report: adding initial flow of report

Fix Containerfile for CI when releasing

Fix windows build

update makefile to fix windows build

isolating status watch interval from #87

reverting unrelated docs changes

improving unit tests

adding metaconfig unit tests and test data

adding tests for meta-run.log parser

adding parsers and tests for config maps

fixes in the opct metrics report

gen metrics report

adding sample document to generate batch reports

Review checks to support attributes

review report to improve checks

increasing the error pattern information for etcd

taking notes for report dev doc

adding plugin log extraction and link to the plugin name

collecting node info

extracting install-config

review report html and metrics with adm parse cmd

adding charts with ploty

remove comments when plotting

fix log save

opct adm parse-etcd-logs to quickly access parsed logs (#4)

increasing documentation coverage

bump to use new quay.io org

create target directory before extractors

rename title of checks on report cli

fixes in the report check prevent empty data

fix/report/cli: show other plugin than k8s

report/check: 012 - check plugin failure

fixes after rebase

rename namespace

Supporting PDB to opct server

add new gen openshift-tests plugin based in go

allow custom openshift-tests image to devel w/ kind

working version with entrypoint for plugin

plugin manifests fixes to support remote entrypoint to tests image

update plugin manifests to new plugin version

refact CLI UI tables to enhance results

intro yamllint and fixing yaml assets/manifests

plugins working version the default flow

enhance pre-run checks for missing config

create local log file for all levels

updatem plugins according to the latest goplugin version

add collector image to the manifest template

review packages / moving to report package using ETL-like

cleanup must-gather/extractor packages

refact/must-gather: isolate and tune leaky bucket processor

cmd/run: fix flag to use full image path to allow CI presubmits

review the result filter pipeline

OPCT-292: cmd/publish - add experimental publish used in CI

Add a experimental command to allow Prow CI to publish results in the
opct-storage (S3) without needing to install dependencies in the test
step, preventing failures.

OPCT-292: cmd/publish - add metadata option when publishing artifact

doc: add review/support documentations

feat: upload baseline artifact on s3

report/feat: opct adm baseline (get|list|indexer|publish)

review baseline reitrieving summary/baseline from service/API

feat: add replay to filter pipeline

feat: introduce adm setup-ndoe to helper in tests

review plugin filter order

fix checks to good tresholds

Author: mtulio

.github/workflows/go.yaml

@@ -1,3 +1,4 @@
+---
 name: "OPCT"
 
 on:
@@ -49,8 +50,8 @@ jobs:
     name: go-test
     runs-on: ubuntu-latest
     needs:
-    - go-lint
-    - go-static
+      - go-lint
+      - go-static
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-go@v5
@@ -63,8 +64,8 @@ jobs:
     name: "go-vet"
     runs-on: ubuntu-latest
     needs:
-    - go-lint
-    - go-static
+      - go-lint
+      - go-static
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-go@v5
@@ -77,8 +78,8 @@ jobs:
     name: "build-artifact"
     runs-on: ubuntu-latest
     needs:
-    - go-test
-    - go-vet
+      - go-test
+      - go-vet
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-go@v5
@@ -121,8 +122,7 @@ jobs:
   e2e-cmd_report:
     name: "e2e-cmd_report"
     runs-on: ubuntu-latest
-    needs:
-    - build
+    needs: [build]
     steps:
       - name: Download artifacts
         uses: actions/download-artifact@v4
@@ -132,24 +132,30 @@ jobs:
 
       - name: Running report
         env:
-          RESULT_ARTIFACT_URL: "https://openshift-provider-certification.s3.us-west-2.amazonaws.com"
-          RESULT_ARTIFACT_VERSION: "v0.4.0/default/4.15.0-20240228-HighlyAvailable-vsphere-None.tar.gz"
-          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
+          BUCKET: openshift-provider-certification
+          REGION: us-west-2
+          OPCT_MODE: v0.4.0/default
+          EXEC_MODE: default
+          ARTIFACT: 4.15.0-20240228-HighlyAvailable-vsphere-None.tar.gz
+          OPCT: /tmp/build/opct-linux-amd64
         run: |
-          echo "> Downloading sample artifact: ${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
-          wget -qO /tmp/result.tar.gz "${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
+          URI=${OPCT_MODE}/${ARTIFACT}
+          URL=https://${BUCKET}.s3.${REGION}.amazonaws.com/${URI}
+
+          echo "> Downloading sample artifact: ${URL}"
+          wget -qO /tmp/result.tar.gz "${URL}"
 
           echo "> Setting run permissions to OPCT:"
-          chmod u+x ${CUSTOM_BUILD_PATH}
+          chmod u+x ${OPCT}
 
           echo "> Running OPCT report:"
-          ${CUSTOM_BUILD_PATH} report /tmp/result.tar.gz
+          ${OPCT} report /tmp/result.tar.gz
 
   e2e-cmd_adm-parse-etcd-logs:
     name: "e2e-cmd_adm-parse-etcd-logs"
     runs-on: ubuntu-latest
     needs:
-    - build
+      - build
     steps:
       - name: Download artifacts
         uses: actions/download-artifact@v4
@@ -159,14 +165,16 @@ jobs:
 
       - name: Preparing testdata
         env:
-          RESULT_ARTIFACT_URL: "https://openshift-provider-certification.s3.us-west-2.amazonaws.com"
-          RESULT_ARTIFACT_VERSION: "testdata/must-gather-etcd-logs.tar.gz"
+          BUCKET: openshift-provider-certification
+          REGION: us-west-2
+          VERSION: "testdata/must-gather-etcd-logs.tar.gz"
           CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
           LOCAL_TEST_DATA: /tmp/must-gather.tar.gz
           LOCAL_TEST_DATA_DIR: /tmp/must-gather
         run: |
-          echo "> Downloading sample artifact: ${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
-          wget -qO ${LOCAL_TEST_DATA} "${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
+          URL=https://${BUCKET}.s3.${REGION}.amazonaws.com
+          echo "> Downloading sample artifact: ${URL}/${VERSION}"
+          wget -qO ${LOCAL_TEST_DATA} "${URL}/${VERSION}"
 
           echo "> Setting run permissions to OPCT:"
           chmod u+x ${CUSTOM_BUILD_PATH}
@@ -178,37 +186,43 @@ jobs:
       - name: "e2e stdin reader: cat <etcd pod logs> | opct adm parse-etcd-logs"
         env:
           CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
-          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+          TEST_DATA_DIR: /tmp/must-gather
+          NS_PATH_ETCD: namespaces/openshift-etcd/pods
+          LOG_PATH: etcd/etcd/logs
         run: |
-          cat ${LOCAL_TEST_DATA_DIR}/*/*/namespaces/openshift-etcd/pods/*/etcd/etcd/logs/*.log | \
+          cat ${TEST_DATA_DIR}/*/*/${NS_PATH_ETCD}/*/${LOG_PATH}/*.log | \
             ${CUSTOM_BUILD_PATH} adm parse-etcd-logs
 
       - name: "e2e must-gather dir: opct adm parse-etcd-logs <must-gather-dir>"
         env:
           CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
-          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+          TEST_DATA_DIR: /tmp/must-gather
         run: |
-          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs ${LOCAL_TEST_DATA_DIR}
+          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs ${TEST_DATA_DIR}
 
-      - name: "e2e aggregate by hour: opct adm parse-etcd-logs --aggregator hour <must-gather-dir>"
+      - name: |
+          e2e aggregate by hour: opct adm parse-etcd-logs
+            --aggregator hour <must-gather-dir>
         env:
-          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
-          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+          OPCT: /tmp/build/opct-linux-amd64
+          TEST_DATA_DIR: /tmp/must-gather
         run: |
-          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs --aggregator hour ${LOCAL_TEST_DATA_DIR}
+          ${OPCT} adm parse-etcd-logs --aggregator hour ${TEST_DATA_DIR}
 
-      - name: "e2e ignore error counters: opct adm parse-etcd-logs --skip-error-counter=true <must-gather-dir>"
+      - name: |
+          e2e ignore error counters: opct adm parse-etcd-logs
+            --skip-error-counter=true <must-gather-dir>
         env:
-          CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
-          LOCAL_TEST_DATA_DIR: /tmp/must-gather
+          OPCT: /tmp/build/opct-linux-amd64
+          TEST_DATA_DIR: /tmp/must-gather
         run: |
-          ${CUSTOM_BUILD_PATH} adm parse-etcd-logs --skip-error-counter=true ${LOCAL_TEST_DATA_DIR}
+          ${OPCT} adm parse-etcd-logs \
+            --skip-error-counter=true ${TEST_DATA_DIR}
 
   e2e-cmd_adm-parse-metrics:
     name: "e2e-cmd_adm-parse-metrics"
     runs-on: ubuntu-latest
-    needs:
-    - build
+    needs: [build]
     steps:
       - name: Download artifacts
         uses: actions/download-artifact@v4
@@ -218,21 +232,60 @@ jobs:
 
       - name: Preparing testdata
         env:
-          RESULT_ARTIFACT_URL: "https://openshift-provider-certification.s3.us-west-2.amazonaws.com"
-          RESULT_ARTIFACT_VERSION: "testdata/ci-external-aws-ccm_1757495441294888960-artifacts_must-gather-metrics.tar.xz"
+          BUCKET: openshift-provider-certification
+          REGION: us-west-2
+          PREFIX: testdata/ci-external-aws-ccm_
+          VERSION: 1757495441294888960-artifacts_must-gather-metrics.tar.xz
           CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
           LOCAL_TEST_DATA: /tmp/opct-metrics.tar.xz
         run: |
-          echo "> Downloading sample artifact: ${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
-          wget -qO ${LOCAL_TEST_DATA} "${RESULT_ARTIFACT_URL}/${RESULT_ARTIFACT_VERSION}"
+          DATA_VERSION=${PREFIX}${VERSION}
+          URL=https://${BUCKET}.s3.${REGION}.amazonaws.com
+          echo "> Downloading sample artifact: ${URL}/${DATA_VERSION}"
+          wget -qO ${LOCAL_TEST_DATA} "${URL}/${DATA_VERSION}"
 
-          echo "> Setting run permissions to OPCT:"
+          echo "> Setting exec permissions to OPCT:"
           chmod u+x ${CUSTOM_BUILD_PATH}
 
       - name: "e2e parse metrics: opct adm parse-etcd-logs <must-gather-dir>"
         env:
           CUSTOM_BUILD_PATH: /tmp/build/opct-linux-amd64
           LOCAL_TEST_DATA: /tmp/opct-metrics.tar.xz
         run: |
-          ${CUSTOM_BUILD_PATH} adm parse-metrics --input ${LOCAL_TEST_DATA} --output /tmp/metrics
+          ${CUSTOM_BUILD_PATH} adm parse-metrics \
+            --input ${LOCAL_TEST_DATA} --output /tmp/metrics
           tree /tmp/metrics
+
+  e2e-cmd_adm-baseline:
+    name: "e2e-cmd_adm-baseline"
+    runs-on: ubuntu-latest
+    needs: [build]
+    steps:
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          name: opct-linux-amd64
+          path: /tmp/build/
+
+      - name: Preparing testdata
+        env:
+          OPCT: /tmp/build/opct-linux-amd64
+        run: |
+          echo "> Setting exec permissions to OPCT:"
+          chmod u+x ${OPCT}
+
+      - name: "e2e adm baseline: opct adm baseline (list|get)"
+        env:
+          OPCT: /tmp/build/opct-linux-amd64
+        run: |
+          echo -e "\n\t#>> List latest baseline results"
+          ${OPCT} adm baseline list
+
+          echo -e "\n\t#>> List all baseline results"
+          ${OPCT} adm baseline list --all
+
+          echo -e "\n\t#>> Retrieve a baseline result by name"
+          ${OPCT} adm baseline get --name 4.16_None_latest --dump
+
+          echo -e "\n\t#>> Retrieve a baseline result by release and platform"
+          ${OPCT} adm baseline get --release 4.15 --platform None

.github/workflows/lint-yaml.yaml

@@ -0,0 +1,28 @@
+---
+name: Lint YAML
+
+on:
+  push:
+    branches:
+      - main
+      - release-*
+  pull_request:
+    branches:
+      - main
+      - release-*
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Install yamllint
+        run: pip install yamllint
+
+      - name: Lint YAML files
+        run: |
+          echo ">> Running YAML lint on data/templates/plugins/"
+          yamllint data/templates/plugins/

.github/workflows/release.yaml

@@ -1,3 +1,4 @@
+---
 name: OPCT Release
 
 on:
@@ -48,7 +49,9 @@ jobs:
           podman login -u="${QUAY_USER}" -p="${QUAY_PASS}" quay.io
 
           echo "> Build container image:"
-          podman build -t ${CONTAINER_REPO}:${RELEASE_TAG} -f hack/Containerfile.ci .
+          podman build \
+            -t ${CONTAINER_REPO}:${RELEASE_TAG} \
+            -f hack/Containerfile.ci .
 
           echo "> Publish container image:"
           podman push ${CONTAINER_REPO}:${RELEASE_TAG}
@@ -83,6 +86,6 @@ jobs:
           body: |
             ## Changelog
             ${{steps.github_release.outputs.changelog}}
-            
+
             ## Container Images
             - [${{ env.CONTAINER_REPO }}:${{ env.RELEASE_TAG }}](https://quay.io/repository/ocp-cert/opct?tab=tags)

.gitignore

@@ -5,6 +5,7 @@ kubeconfig
 
 # build files
 dist/
+build/
 
 # changelog is generated automaticaly by hack/generate-changelog.sh
 # available only in the rendered webpage (built by mkdocs).

Makefile

@@ -57,15 +57,30 @@ build-darwin-arm64: build
 linux-amd64-container: build-linux-amd64
 	podman build -t $(IMG):latest -f hack/Containerfile --build-arg=RELEASE_TAG=$(RELEASE_TAG) .
 
-.PHONY: image-mirror-sonobuoy
-image-mirror-sonobuoy:
-	./hack/image-mirror-sonobuoy/mirror.sh
-
-# Utils dev
-.PHONY: update-go
-update-go:
-	go get -u
-	go mod tidy
+# Publish devel binaries (non-official). Must be used only for troubleshooting in development/support.
+.PHONY: publish-amd64-devel
+publish-amd64-devel: build-linux-amd64
+	aws s3 cp $(BUILD_DIR)/opct-linux-amd64 s3://openshift-provider-certification/bin/opct-linux-amd64-devel
+
+.PHONY: publish-darwin-arm64-devel
+publish-darwin-arm64-devel: build-darwin-arm64
+	aws s3 cp $(BUILD_DIR)/opct-darwin-arm64 s3://openshift-provider-certification/bin/opct-darwin-arm64-devel
+
+.PHONY: publish-devel
+publish-devel: publish-amd64-devel
+publish-devel: publish-darwin-arm64-devel
+
+#
+# Test
+#
+
+.PHONY: test-lint
+test-lint:
+	@echo "Running linting tools"
+	# Download https://github.com/golangci/golangci-lint/releases/tag/v1.59.1
+	golangci-lint run --timeout=10m
+	# yamllint: pip install yamllint
+	yamllint .github/workflows/*.yaml
 
 .PHONY: test
 test:
@@ -90,3 +105,13 @@ build-changelog:
 .PHONY: build-docs
 build-docs: build-changelog
 	mkdocs build --site-dir ./site
+
+.PHONY: image-mirror-sonobuoy
+image-mirror-sonobuoy:
+	./hack/image-mirror-sonobuoy/mirror.sh
+
+# Utils dev
+.PHONY: update-go
+update-go:
+	go get -u
+	go mod tidy