APIs stack and repositories structure

APIs repository structure

Glossary

• API: An "Application programming interface". In this context is the definition on how to interact with a certain service. Implementation-wise is a set of proto files defining a gRPC interface. This is programming language agnostic.

• Binding: Programming language-specific code that implements the API messages as structures and RPC calls as functions. Usually automatically generated by protoc (a command-line tool to generate language-specific code from proto files). Bindings can be used to implement an API service and/or client. We only generate public bindings for Python for now.

• Service: A process (daemon, server) that implements an API. This is implemented using using bindings for the language used to build the service, for example Rust or Go.

• Client: A library that can be used to connect and interact with a service. These are programming language-specific and use the bindings for this language underneath. It provides a more convenient an idiomatic interface than the (automatically-generated) bindings. For Python it also convert gRPC streams to use frequenz-channels-python.

• Actor: An actor in the context of APIs usually refers to an actor that uses the client to implement the client-side business logic of the API. It usually have some background tasks talking to the service and keeping some state. It is also a library.

• High-level interface: Another library layer that provides a more convenient and idiomatic way to interact with the actor by using simple function calls to send information to it, or channels to receive information from it. It is usually in charge of starting the actor and managing its life-cycle too.

• Application: A process (daemon, server) that usually mix several API's high-level interfaces (although it can use only one too) to implement business logic or "use cases", like peak shaving or FCR.

Stack Diagram

                                                        Application
                                                         "(Python)"
                                   .-------------------------------------------------.
                                   |                    Business logic               |
                                   +------------------------+------------------------+
    Service 1                      |         API 1          |         API 2          |                      Service 2
     "(Rust)"                      | "High-level interface" | "High-level interface" |                        "(Go)"
.----------------.                 +------------------------+------------------------+                 .----------------.
| Business logic |                 |         Actor          |         Actor          |                 | Business logic |
+----------------+                 +------------------------+------------------------+                 +----------------+
|     Server     |       API 1     |         Client         |         Client         |       API 2     |     Server     |
+----------------+  "(proto/gRPC)" +------------------------+------------------------+  "(proto/gRPC)" +----------------+
|    Bindings    | <┄┄┄┄┄┄┄┄┄┄┄┄┄> |        Bindings        |        Bindings        | <┄┄┄┄┄┄┄┄┄┄┄┄┄> |    Bindings    |
'----------------'     network     '------------------------'------------------------'     network     '----------------'

Repository structure

Current state of affairs

Currently we have the following repository structure to represent the above stack:

• frequenz-client-base-python: Python library to write async gRPC clients using channels.

• frequenz-api-xxx: The API (protobuf definitions) and the Python Bindings. Sometimes it also contains the Python Client.

  Internal dependencies: frequenz-api-common. If it contains the Client too, then it also depends on frequenz-client-base-python.

• frequenz-service-xxx: The Service implementation.

  Internal dependencies: frequenz-api-xxx.

• frequenz-client-xxx-python: The Python Client implementation.

  Internal dependencies: frequenz-api-xxx, frequenz-client-base-python.

• frequenz-actor-xxx: The Actor implementation.

  Internal dependencies: frequenz-api-xxx or frequenz-client-xxx if there is one, frequenz-channels-python, frequenz-sdk-python.

• frequenz-sdk-python: The High-level interface implementation.

  Internal dependencies: frequenz-api-xxx or frequenz-client-xxx or frequenz-actor-xxx if there is one, frequenz-channels-python.

  So far we only have one High-level interface implementation, for the frequenz-api-microgrid. And the Microgrid API doesn't have a client or actor repository yet, all of them live in the SDK.

Future direction

We want to have a more consistent approach for mapping the stack layers to repositories. This is the bare minimum of splitting and homogenization we want to achieve:

• frequenz-client-base-python: Python library to write async gRPC clients using channels.

• frequenz-api-xxx: The API (protobuf definitions).

  Internal dependencies: frequenz-api-common.

• frequenz-service-xxx: The Service implementation.

  Internal dependencies: frequenz-api-xxx.

• frequenz-client-xxx-python: The Python Client implementation and the Python Bindings. It is still in discussion if we want to have one repository for the Bindings separated from the Client.

  Internal dependencies: frequenz-api-xxx, frequenz-client-base-python.

• frequenz-actor-xxx: The Actor implementation.

  Internal dependencies: frequenz-api-xxx or frequenz-client-xxx if there is one, frequenz-channels-python, frequenz-sdk-python.

• frequenz-sdk-python: The High-level interface implementation.

Still in discussion

We probably will want to eventually separate the Bindings and Client into independent repositories, as the versions and release cycles could differ.

We might also want to put the High-level interface into its own repository, so we can make the dependency on different APIs in the SDK optional, otherwise if one wants to write an Application that only uses the Electricity Trading API (for example), it is not necessary to pull all the dependencies for all other APIs. This would mean splitting the SDK in smaller reusable packages, as we did with frequenz-channels-python, like frequenz-actor-python, frequenz-quantity-python, etc.

See https://github.com/frequenz-floss/frequenz-sdk-python/discussions/854 for more details.

• frequenz-bindings-xxx-python: The Python Bindings (only automatically generated files).

  Internal dependencies: frequenz-api-xxx.

• frequenz-client-xxx-python: The Python Client implementation (only manually written code).

  Internal dependencies: frequenz-bindings-xxx-python, frequenz-client-base-python.

• frequenz-actor-xxx: The Actor implementation.

  Internal dependencies: frequenz-client-xxx, frequenz-channels-python (frequenz-actor-python, frequenz-quantity-python, etc.).

• frequenz-xxx-python: The High-level interface implementation.

  Internal dependencies: frequenz-actor-xxx, frequenz-channels-python (frequenz-actor-python, frequenz-quantity-python, etc.).

• frequenz-sdk-python: Glue code to bring different APIs and common infrastructure (like configuration and logging management) together.

  Internal dependencies: frequenz-xxx-python, frequenz-yyy-python, ..., frequenz-channels-python, frequenz-actor-python, frequenz-quantity-python, etc.

API (protobuf) dependencies

All APIs protobuf definitions depend on the frequenz-api-common repository, which host some common definitions shared by many APIs.

The frequenz-api-common repository depends on the googleapis/api-common-protos which host also some core extensions to the base protobuf types. So all APIs depend on this repository too, at least indirectly.

Some APIs will also depend on googleapis/api-common-protos directly.

flowchart BT
    google[googleapis/api-common-protos]
    common[frequenz-api-common]
    api1[frequenz-api-xxx]
    api2[frequenz-api-yyy]
    apiN[frequenz-api-...]


    api1 ---> google
    common --> google

    style s opacity:0
    subgraph s[" "]
        api1 --> common
        api2 --> common
        apiN --> common
    end

    apiN -.->|sometimes| google

Loading