Description
Summary
After enabling JWT authorization on Temporal cluster, system workers (like history scanner) fail to authenticate and cannot connect to the frontend service, resulting in "Request unauthorized" errors.
Environment
Temporal Version: 1.24.3
Deployment Method: Kubernetes with Temporal Operator
Authentication Method: JWT with Keycloak/OIDC
Cluster Configuration: Single frontend service with JWT authorization enabled
Problem Description
When JWT authorization is enabled on a Temporal cluster, internal system workers cannot authenticate with the frontend service and fail with authorization errors. This appears to be a common issue where system workers need authentication credentials but there's no clear documentation or configuration method to provide them.
Error Details
{
"level": "error",
"ts": "2025-06-03T01:04:50.770Z",
"msg": "error starting temporal-sys-history-scanner-workflow workflow",
"service": "worker",
"error": "Request unauthorized.",
"logging-call-at": "scanner.go:289",
"stacktrace": "go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/service/worker/scanner.(*Scanner).startWorkflow\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/scanner.go:289\ngo.temporal.io/server/service/worker/scanner.(*Scanner).startWorkflowWithRetry.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/scanner.go:259\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:143\ngo.temporal.io/server/service/worker/scanner.(*Scanner).startWorkflowWithRetry\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/worker/scanner/scanner.go:258"
}
Current Configuration
Authorization Configuration
authorization:
jwtKeyProvider:
keySourceURIs:
- https://myapp.domain.com/auth/realms/default/protocol/openid-connect/certs
refreshInterval: "1m"
permissionsClaimName: "permissions"
authorizer: default
claimMapper: default
Services Configuration
services:
frontend:
httpPort: 7243
membershipPort: 6933
port: 7233
replicas: 2
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
history:
httpPort: 0
membershipPort: 6934
port: 7234
replicas: 2
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
matching:
httpPort: 0
membershipPort: 6935
port: 7235
replicas: 2
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
worker:
httpPort: 0
membershipPort: 6939
port: 7239
replicas: 2
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
Expected Behavior
System workers should be able to operate normally when JWT authorization is enabled, either by:
Having a built-in bypass mechanism for system operations
Supporting configuration of JWT credentials for system workers
Clear documentation on how to configure system worker authentication
Actual Behavior
System workers fail with "Request unauthorized" errors and cannot perform essential operations like history scanning, workflow cleanup, etc.