This is a Bedrock (and other generative AI tools from AWS) API forwarding tool that can issue virtual keys, log chats, and manage costs.
It is compatible with any OPENAI client that can define Host and API Key.
Docker Image: DockerHub, Public ECR
Docker Image for Lambda: DockerHub, Public ECR
- Supports all current and future large language models from Bedrock (supported through bedrock-converse).
- Supports models deployed through Sagemaker LMI (partial models).
- Supports other forms of custom models, including Ollama, etc.
- More AI workflow applications, such as internet search, AWS command executors, etc.
- Create API keys. Keys can be created for regular users and administrators. Regular users can chat, while administrators can manage API keys and costs.
- Record the cost of each call and use it as a basis for cost control.
- Cost control. You can set monthly quotas and account balances for each API key. When the monthly quota or account balance is insufficient, it cannot be used.
- Calculate the overall cost.
Important
The cost calculation of this project cannot be used as the basis for AWS billing. Please refer to the AWS bill for actual expenses. Please refer to the official website for the Bedrock pricing.
Models and their parameters can be defined from the backend.
Once defined, models can be bound to groups or API Keys.
-
Enhancement: Streaming Mode reasoning_content and tool_calls - In streaming mode, the system now outputs
reasoning_contentandtool_callscontent. This addresses the issue mentioned in issue #74 where reasoning_content was missing in non-streaming mode. -
Standardization: OpenAI Format for Function Calling - Function calling in both streaming and non-streaming modes has been adapted to follow OpenAI's standard format, improving compatibility with existing tools and libraries. Document
-
Improvement: Cost Statistics Total - The total in cost statistics is now directly modified using SQL update, preventing monthly consumption from exceeding total consumption.
-
New: ECS Deployment Script - Added ECS deployment script.
-
Fix: Feishu Message Duplication - Fixed duplicate messages in Feishu when using streaming mode.
- New Provider: sagemaker-deepseek - Added a new provider for DeepSeek R1 models deployed on SageMaker. This provider supports streaming output with Deepseek-style
reasoning_content. Currently supports three deployment methods on SageMaker: LMI, JumpStart, and Bedrock.
This connector is an open-source software aimed at providing proxy services for using Bedrock services. We make our best efforts to ensure the security and legality of the software, but we are not responsible for the users' behavior.
The connector is intended solely for personal learning and research purposes. Users shall not use it for any illegal activities, including but not limited to hacking, spreading illegal information, etc. Otherwise, users shall bear the corresponding legal responsibilities themselves. Users are responsible for complying with the laws and regulations in their respective jurisdictions and shall not use the connector for any illegal or non-compliant purposes. The developers and maintainers of this software shall not be liable for any disputes, losses, or legal liabilities arising from the use of this connector.
We reserve the right to modify or terminate the connector's code at any time without further notice. Users are expected to understand and comply with the relevant local laws and regulations.
If you have any questions regarding this disclaimer, please feel free to contact us through the open-source channels.
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.
