Skip to content

Commit 32fe99a

Browse files
authored
Docs: Handle concurrent pipeline execution in separate containers (#3799)
* Handle concurrent pipeline execution in separate containers * Update pipeline execution for production environments
1 parent 830db48 commit 32fe99a

File tree

1 file changed

+25
-0
lines changed

1 file changed

+25
-0
lines changed

docs/book/user-guide/tutorial/trigger-pipelines-from-external-systems.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -746,6 +746,31 @@ You can extend this API to support additional features:
746746
4. **Advanced Authentication**: Implement JWT or OAuth2 for better security
747747
5. **Pipeline Scheduling**: Add endpoints to schedule pipeline runs
748748

749+
### Handling Concurrent Pipeline Execution
750+
751+
{% hint style="warning" %}
752+
**Important Limitation: ZenML Prevents Concurrent Pipeline Execution**
753+
754+
ZenML's current implementation uses shared global state (like active stack and active pipeline), which prevents running multiple pipelines concurrently in the same process. If you attempt to trigger multiple pipelines simultaneously, subsequent calls will be blocked with the error:
755+
756+
```
757+
Preventing execution of pipeline '<pipeline_name>'. If this is not intended behavior, make sure to unset the environment variable 'ZENML_PREVENT_PIPELINE_EXECUTION'.
758+
```
759+
{% endhint %}
760+
761+
The FastAPI example above uses threading, but due to ZenML's architecture, concurrent pipeline execution will fail. For production environments that need to handle concurrent pipeline requests, consider deploying your pipeline triggers through container orchestration platforms.
762+
763+
#### Recommended Solutions for Concurrent Execution
764+
765+
For production deployments, consider using:
766+
767+
1. **Kubernetes Jobs**: Deploy each pipeline execution as a separate Kubernetes Job for resource management and scaling
768+
2. **Docker Containers**: Use a container orchestration platform like Docker Swarm or ECS to run separate container instances
769+
3. **Cloud Container Services**: Leverage services like AWS ECS, Google Cloud Run, or Azure Container Instances
770+
4. **Serverless Functions**: Deploy pipeline triggers as serverless functions (AWS Lambda, Azure Functions, etc.)
771+
772+
These approaches ensure each pipeline runs in its own isolated environment, avoiding the concurrency limitations of ZenML's shared state architecture.
773+
749774
### Security Considerations
750775

751776
When deploying this API in production:

0 commit comments

Comments
 (0)