Skip to content

Improve python sdk #197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jun 22, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sdks/fs-python/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ build-image:
docker build -t functionstream/fs-python-base .

test:
pytest -v
PYTHONPATH=. python -m pytest
41 changes: 22 additions & 19 deletions sdks/fs-python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,31 +16,31 @@

# FunctionStream Python SDK

FunctionStream SDK is a powerful Python library for building and deploying serverless functions that process messages from Apache Pulsar. It provides a simple yet flexible framework for creating event-driven applications with robust error handling, metrics collection, and resource management.
FunctionStream SDK is a powerful Python library for building and deploying serverless functions that process messages
from Apache Pulsar. It provides a simple yet flexible framework for creating event-driven applications with robust error
handling, metrics collection, and resource management.

## Features

- **Easy Function Development**: Simple API for creating serverless functions
- **Message Processing**: Built-in support for Apache Pulsar message processing
- **Metrics Collection**: Automatic collection of performance metrics
- **Resource Management**: Efficient handling of connections and resources
- **Graceful Shutdown**: Proper cleanup of resources during shutdown
- **Configuration Management**: Flexible configuration through YAML files
- **Schema Validation**: Input and output schema validation
- **Error Handling**: Comprehensive error handling and logging

## Installation

```bash
pip install fs-sdk
pip install function-stream
```

## Quick Start

1. Create a function that processes messages:

```python
from fs_sdk import FSFunction
from function_stream import FSFunction

async def my_process_function(request_data: dict) -> dict:
# Process the request data
Expand Down Expand Up @@ -105,6 +105,7 @@ modules:
### FSFunction

The main class for creating serverless functions. It handles:

- Message consumption and processing
- Response generation
- Resource management
Expand All @@ -114,6 +115,7 @@ The main class for creating serverless functions. It handles:
### Configuration

The SDK uses YAML configuration files to define:

- Pulsar connection settings
- Module selection
- Topic subscriptions
Expand All @@ -123,6 +125,7 @@ The SDK uses YAML configuration files to define:
### Metrics

Built-in metrics collection for:

- Request processing time
- Success/failure rates
- Message throughput
Expand All @@ -140,24 +143,24 @@ Check out the `examples` directory for complete examples:
## Best Practices

1. **Error Handling**
- Always handle exceptions in your process functions
- Use proper logging for debugging
- Implement graceful shutdown
- Always handle exceptions in your process functions
- Use proper logging for debugging
- Implement graceful shutdown

2. **Resource Management**
- Close resources properly
- Use context managers when possible
- Monitor resource usage
- Close resources properly
- Use context managers when possible
- Monitor resource usage

3. **Configuration**
- Use environment variables for sensitive data
- Validate configuration values
- Document configuration options
- Use environment variables for sensitive data
- Validate configuration values
- Document configuration options

4. **Testing**
- Write unit tests for your functions
- Test error scenarios
- Validate input/output schemas
- Write unit tests for your functions
- Test error scenarios
- Validate input/output schemas

## Development

Expand All @@ -180,13 +183,13 @@ source venv/bin/activate # Linux/Mac
pip install -r requirements.txt

# Install the package in development mode
pip install -e .
python -m pip install -e .
```

### Running Tests

```bash
pytest
make test
```

## Support
Expand Down
8 changes: 4 additions & 4 deletions sdks/fs-python/examples/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ WORKDIR /function
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY string_function.py .
COPY config.yaml .
COPY package.yaml .
COPY main.py .

CMD ["python", "string_function.py"]
RUN chmod +x main.py

CMD ["python", "main.py"]
2 changes: 1 addition & 1 deletion sdks/fs-python/examples/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
# This configuration file defines the settings for the string processing function example.

pulsar:
serviceUrl: "pulsar://192.168.31.80:6650" # Required: URL of the Pulsar broker
serviceUrl: "pulsar://127.0.0.1:6650" # Required: URL of the Pulsar broker
authPlugin: "" # Optional: Authentication plugin class name
authParams: "" # Optional: Authentication parameters

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@
import asyncio
from typing import Dict, Any

from fs_sdk import FSFunction
from fs_sdk.context import FSContext
from function_stream import FSFunction, FSContext

async def string_process_function(context: FSContext, data: Dict[str, Any]) -> Dict[str, Any]:
"""
Expand Down
1 change: 0 additions & 1 deletion sdks/fs-python/examples/test_string_function.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
import pulsar
import json
import uuid
import time

async def test_string_function():
"""
Expand Down
4 changes: 0 additions & 4 deletions sdks/fs-python/fs_sdk/__init__.py

This file was deleted.

63 changes: 0 additions & 63 deletions sdks/fs-python/fs_sdk/config.py

This file was deleted.

27 changes: 27 additions & 0 deletions sdks/fs-python/function_stream/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
from .config import Config, PulsarConfig, PulsarSourceConfig, SourceSpec, SinkSpec, Metric
from .context import FSContext
from .function import FSFunction
from .metrics import Metrics, MetricsServer
from .module import FSModule

__version__ = "0.6.0rc1"
__all__ = [
# Core classes
"FSFunction",
"FSModule",

# Configuration classes
"Config",
"PulsarConfig",
"PulsarSourceConfig",
"SourceSpec",
"SinkSpec",
"Metric",

# Context and utilities
"FSContext",

# Metrics and monitoring
"Metrics",
"MetricsServer"
]
Loading
Loading