MRG: Merge pull request #587 from octue/feature/add-get-latest-service-function

Add ability to get latest revision of a service

Author: cortadocodes

.pre-commit-config.yaml

@@ -77,7 +77,7 @@ repos:
           - "^dependencies/([a-z][a-z0-9]*)(-[a-z0-9]+)*$"
 
   - repo: https://github.com/octue/conventional-commits
-    rev: 0.8.1
+    rev: 0.9.0
     hooks:
       - id: check-commit-message-is-conventional
         stages: [commit-msg]

docs/source/asking_questions.rst

@@ -4,43 +4,11 @@
 Asking services questions
 =========================
 
-Octue services
-==============
-
-There's a growing range of live :ref:`services <service_definition>` in the Octue ecosystem that you can ask questions
-to and get answers from. Currently, all of them are related to wind energy. Here's a quick glossary of terms before we
-tell you more:
-
-.. admonition:: Definitions
-
-    Child
-        An Octue service that can be asked a question. This name reflects the tree structure of services (specifically,
-        `a DAG <https://en.wikipedia.org/wiki/Directed_acyclic_graph>`_) formed by the service asking the question (the
-        parent), the child it asks the question to, any children that the child asks questions to as part of forming
-        its answer, and so on.
-
-    Parent
-        An Octue service that asks a question to another Octue service (a child).
-
-    Asking a question
-        Sending data (input values and/or an input manifest) to a child for processing/analysis.
-
-    Receiving an answer
-       Receiving data (output values and/or an output manifest) from a child you asked a question to.
-
-    Octue ecosystem
-       The set of services running the ``octue`` SDK as their backend. These services guarantee:
-
-       - Defined JSON schemas and validation for input and output data
-       - An easy interface for asking them questions and receiving their answers
-       - Logs and exceptions (and potentially monitor messages) forwarded to you
-       - High availability if deployed in the cloud
-
-
 How to ask a question
 =====================
-You can ask any service a question if you have its service ID, project name, and permissions. The question is formed of
-input values and/or an input manifest.
+Questions are always asked to a *revision* of a service. You can ask a service a question if you have its
+:ref:`SRUID <sruid_definition>`, project name, and the necessary permissions. The question is formed of input values
+and/or an input manifest.
 
 .. code-block:: python
 
@@ -63,6 +31,16 @@ input values and/or an input manifest.
     >>> <FilterSet({<Datafile('my_file.csv')>, <Datafile('another_file.csv')>})>
 
 
+.. _using_latest_revision_tag:
+
+.. note::
+
+    Using the ``latest`` service revision tag, or not including one at all, will cause your question to be sent to the
+    latest deployed revision of the service. This is determined by making a request to a `service registry
+    <https://django-twined.readthedocs.io/en/latest/>`_ if one or more :ref:`registries are defined
+    <using_service_registries>`. If none of the service registries contain an entry for this service, a specific service
+    revision tag must be used.
+
 You can also set the following options when you call :mod:`Child.ask <octue.resources.child.Child.ask>`:
 
 - ``children`` - If the child has children of its own (i.e. grandchildren of the parent), this optional argument can be used to override the child's "default" children. This allows you to specify particular versions of grandchildren to use (see :ref:`this subsection below <overriding_children>`).
@@ -230,3 +208,45 @@ Overriding beyond the first generation
 It's an intentional choice to only go one generation deep with overriding children. If you need to be able to specify a
 whole tree of children, grandchildren, and so on, please `upvote this issue.
 <https://github.com/octue/octue-sdk-python/issues/528>`_
+
+
+.. _using_service_registries:
+
+Using a service registry
+========================
+When asking a question, you can optionally specify one or more `service registries
+<https://django-twined.readthedocs.io/en/latest/>`_ to resolve SRUIDs against. This is analogous to specifying a
+different ``pip`` index for resolving package names when using ``pip install``. If you don't specify any registries, the
+default Octue service registry is used.
+
+Specifying service registries can be useful if:
+
+- You have your own private services that aren't on the default Octue service registry
+- You want services from one service registry with the same name as in another service registry to be prioritised
+
+Specifying service registries
+-----------------------------
+You can specify service registries in two ways:
+
+1. Globally for all questions asked inside a service. In the service configuration (``octue.yaml`` file):
+
+    .. code-block:: yaml
+
+        services:
+          - namespace: my-organisation
+            name: my-app
+            service_registries:
+              - name: my-registry
+                endpoint: blah.com/services
+
+2. For questions to a specific child, inside or outside a service:
+
+    .. code-block:: python
+
+        child = Child(
+            id="my-organisation/my-service:latest",
+            backend={"name": "GCPPubSubBackend", "project_name": "my-project"},
+            service_registries=[
+                {"name": "my-registry", "endpoint": "blah.com/services"},
+            ]
+        )

docs/source/creating_services.rst

@@ -113,49 +113,10 @@ Dockerfile (optional)
 
         As always, if you need help with this, feel free to drop us a message or raise an issue!
 
-Naming services
-===============
-
-.. admonition:: Definitions
-
-    Service revision
-        A specific instance of an Octue service that can be individually addressed. The revision could correspond to a
-        version of the service, a dynamic development branch for it, or a deliberate duplication or variation of it.
-
-    Service revision unique identifier (SRUID)
-        The combination of a service revisions's namespace, name, and revision tag that uniquely identifies it. For
-        example, ``octue/my-service:1.3.0`` where the namespace is ``octue``, the name is ``my-service``, and the
-        revision tag is ``1.3.0``.
-
-    Service namespace
-        The group to which the service belongs e.g. your name or your organisation's name. If in doubt, use the GitHub
-        handle of the user or organisation publishing the services.
-
-        Namespaces must be lower kebab case (i.e. they may contain the letters [a-z], numbers [0-9], and hyphens [-]).
-        They may not begin or end with hyphens.
-
-    Service name
-        A name to uniquely identify the service within its namespace. This usually corresponds to the name of the GitHub
-        repository for the service. Names must be lower kebab case (i.e. they may contain the letters [a-z],
-        numbers [0-9] and hyphens [-]). They may not begin or end with hyphens.
-
-    Service revision tag
-        A tag that uniquely identifies a particular revision of a service. The revision tag could correspond to a commit
-        hash like ``a3eb45``, a release number like ``0.12.4``, a branch name (e.g. ``development``), a particular
-        environment the service is deployed in (e.g. ``production``), or a combination like ``0.12.4-production``. Tags
-        may contain lowercase and uppercase letters, numbers, underscores, periods, and hyphens, but can't start with a
-        period or a dash. They can contain a maximum of 128 characters. These requirements are the same as the `Docker
-        tag format <https://docs.docker.com/engine/reference/commandline/tag/>`_.
-
-    Service ID
-        The SRUID is a special case of the service ID. A service ID can be an SRUID or just the service namespace and
-        name. It can be used to ask a question to a service without specifying a specific revision of it. This enables
-        asking questions to, for example, the service ``octue/my-service`` and automatically having them routed to its
-        latest revision. Note that this will be a future feature; currently, you will still be required to provide a
-        revision tag (i.e. a full SRUID).
 
 Where to specify the namespace, name, and revision tag
 ------------------------------------------------------
+See :ref:`here <service_naming>` for service naming requirements.
 
 **Namespace**
 

docs/source/index.rst

@@ -10,8 +10,8 @@ groundwork so you have more time for the science!
 .. admonition:: Definition
 
     Octue service
-        An Octue data service, digital twin, or application that can be asked questions, process them, and send answers.
-        Octue services can communicate with each other with no extra setup.
+        An Octue data service, digital twin, or application that can be asked questions, process them, and return
+        answers. Octue services can communicate with each other with minimal extra setup.
 
 
 Key features
@@ -82,6 +82,7 @@ We use `GitHub Issues <https://github.com/octue/octue-sdk-python/issues>`_ [#]_
    datafile
    dataset
    manifest
+   services
    asking_questions
    creating_services
    running_services_locally

docs/source/services.rst

@@ -0,0 +1,90 @@
+.. _services:
+
+==============
+Octue services
+==============
+
+There's a growing range of live :ref:`services <service_definition>` in the Octue ecosystem that you can ask questions
+to and get answers from. Currently, all of them are related to wind energy. Here's a quick glossary of terms before we
+tell you more:
+
+.. admonition:: Definitions
+
+    Octue service
+        See :ref:`here <service_definition>`.
+
+    Child
+        An Octue service that can be asked a question. This name reflects the tree structure of services (specifically,
+        `a DAG <https://en.wikipedia.org/wiki/Directed_acyclic_graph>`_) formed by the service asking the question (the
+        parent), the child it asks the question to, any children that the child asks questions to as part of forming
+        its answer, and so on.
+
+    Parent
+        An Octue service that asks a question to another Octue service (a child).
+
+    Asking a question
+        Sending data (input values and/or an input manifest) to a child for processing/analysis.
+
+    Receiving an answer
+       Receiving data (output values and/or an output manifest) from a child you asked a question to.
+
+    Octue ecosystem
+       The set of services running the Octue SDK as their backend. These services guarantee:
+
+       - Defined input/output JSON schemas and validation
+       - An easy and consistent interface for asking them questions and receiving their answers
+       - Logs, exceptions, and monitor messages forwarded to you
+       - High availability (if deployed in the cloud)
+
+
+.. _service_naming:
+
+Service names
+=============
+
+Questions are always asked to a *revision* of a service. Services revisions are named in a similar way to docker images.
+They look like ``namespace/name:tag`` where the tag is often a semantic version (but doesn't have to be).
+
+.. admonition:: Definitions
+
+    Service revision
+        A specific instance of an Octue service that can be individually addressed. The revision could correspond to a
+        version of the service, a dynamic development branch for it, or a deliberate duplication or variation of it.
+
+    .. _sruid_definition:
+
+    Service revision unique identifier (SRUID)
+        The combination of a service revision's namespace, name, and revision tag that uniquely identifies it. For
+        example, ``octue/my-service:1.3.0`` where the namespace is ``octue``, the name is ``my-service``, and the
+        revision tag is ``1.3.0``.
+
+    Service namespace
+        The group to which the service belongs e.g. your name or your organisation's name. If in doubt, use the GitHub
+        handle of the user or organisation publishing the services.
+
+        Namespaces must be lower kebab case (i.e. they may contain the letters [a-z], numbers [0-9], and hyphens [-]).
+        They may not begin or end with hyphens.
+
+    Service name
+        A name to uniquely identify the service within its namespace. This usually corresponds to the name of the GitHub
+        repository for the service. Names must be lower kebab case (i.e. they may contain the letters [a-z], numbers
+        [0-9] and hyphens [-]). They may not begin or end with hyphens.
+
+    Service revision tag
+        A tag that uniquely identifies a particular revision of a service. The revision tag could be a:
+
+        - Commit hash (e.g. ``a3eb45``)
+        - Semantic version (e.g. ``0.12.4``)
+        - Branch name (e.g. ``development``)
+        - Particular environment the service is deployed in (e.g. ``production``)
+        - Combination of these (e.g. ``0.12.4-production``)
+
+        Tags may contain lowercase and uppercase letters, numbers, underscores, periods, and hyphens, but can't start
+        with a period or a dash. They can contain a maximum of 128 characters. These requirements are the same as the
+        `Docker tag format <https://docs.docker.com/engine/reference/commandline/tag/>`_.
+
+    Service ID
+        The SRUID is a special case of a service ID. A service ID can be an SRUID or just the service namespace and
+        name. It can be used to ask a question to a service without specifying a specific revision of it. This enables
+        asking questions to, for example, the service ``octue/my-service`` and automatically having them routed to its
+        latest revision. :ref:`See here for more info<using_latest_revision_tag>`.

octue/cli.py

@@ -1,19 +1,19 @@
 import copy
 import functools
+import importlib.metadata
 import importlib.util
 import json
 import logging
 import os
 import sys
 
 import click
-import pkg_resources
 from google import auth
 
 from octue.cloud import storage
 from octue.cloud.pub_sub import Subscription, Topic
 from octue.cloud.pub_sub.service import Service
-from octue.cloud.service_id import convert_service_id_to_pub_sub_form, create_service_sruid, get_service_sruid_parts
+from octue.cloud.service_id import convert_service_id_to_pub_sub_form, create_sruid, get_sruid_parts
 from octue.cloud.storage import GoogleCloudStorageClient
 from octue.configuration import load_service_and_app_configuration
 from octue.definitions import MANIFEST_FILENAME, VALUES_FILENAME
@@ -53,7 +53,7 @@
     show_default=True,
     help="Forces a reset of analysis cache and outputs [For future use, currently not implemented]",
 )
-@click.version_option(version=pkg_resources.get_distribution("octue").version)
+@click.version_option(version=importlib.metadata.version("octue"))
 def octue_cli(id, logger_uri, log_level, force_reset):
     """The CLI for the Octue SDK. Use it to start an Octue data service or digital twin locally or run an analysis on
     one locally.
@@ -137,6 +137,7 @@ def run(service_config, input_dir, output_file, output_manifest_file, monitor_me
         children=app_configuration.children,
         output_location=app_configuration.output_location,
         crash_diagnostics_cloud_path=service_configuration.crash_diagnostics_cloud_path,
+        service_registries=service_configuration.service_registries,
     )
 
     if monitor_messages_file:
@@ -212,7 +213,7 @@ def start(service_config, revision_tag, timeout, no_rm):
     """
     service_revision_tag_override = revision_tag
     service_configuration, app_configuration = load_service_and_app_configuration(service_config)
-    service_namespace, service_name, service_revision_tag = get_service_sruid_parts(service_configuration)
+    service_namespace, service_name, service_revision_tag = get_sruid_parts(service_configuration)
 
     if service_revision_tag_override and service_revision_tag:
         logger.warning(
@@ -222,7 +223,7 @@ def start(service_config, revision_tag, timeout, no_rm):
             service_revision_tag_override,
         )
 
-    service_sruid = create_service_sruid(
+    service_sruid = create_sruid(
         namespace=service_namespace,
         name=service_name,
         revision_tag=service_revision_tag_override or service_revision_tag,
@@ -262,7 +263,7 @@ def start(service_config, revision_tag, timeout, no_rm):
 
     except ServiceAlreadyExists:
         # Generate and use a new revision tag if the service already exists.
-        service_sruid = create_service_sruid(namespace=service_namespace, name=service_name)
+        service_sruid = create_sruid(namespace=service_namespace, name=service_name)
 
         while True:
             user_confirmation = input(
@@ -438,7 +439,7 @@ def create_push_subscription(
     PUSH_ENDPOINT is the HTTP/HTTPS endpoint of the service to push to. It should be fully formed and include the
     'https://' prefix
     """
-    service_sruid = create_service_sruid(namespace=service_namespace, name=service_name, revision_tag=revision_tag)
+    service_sruid = create_sruid(namespace=service_namespace, name=service_name, revision_tag=revision_tag)
     pub_sub_sruid = convert_service_id_to_pub_sub_form(service_sruid)
 
     topic = Topic(name=pub_sub_sruid, project_name=project_name)

octue/cloud/deployment/google/answer_pub_sub_question.py

@@ -1,7 +1,7 @@
 import logging
 
 from octue.cloud.pub_sub.service import Service
-from octue.cloud.service_id import create_service_sruid, get_service_sruid_parts
+from octue.cloud.service_id import create_sruid, get_sruid_parts
 from octue.configuration import load_service_and_app_configuration
 from octue.resources.service_backends import GCPPubSubBackend
 from octue.runner import Runner
@@ -22,9 +22,9 @@ def answer_question(question, project_name):
     :return None:
     """
     service_configuration, app_configuration = load_service_and_app_configuration(DEFAULT_SERVICE_CONFIGURATION_PATH)
-    service_namespace, service_name, service_revision_tag = get_service_sruid_parts(service_configuration)
+    service_namespace, service_name, service_revision_tag = get_sruid_parts(service_configuration)
 
-    service_sruid = create_service_sruid(
+    service_sruid = create_sruid(
         namespace=service_namespace,
         name=service_name,
         revision_tag=service_revision_tag,
@@ -46,6 +46,7 @@ def answer_question(question, project_name):
             crash_diagnostics_cloud_path=service_configuration.crash_diagnostics_cloud_path,
             project_name=project_name,
             service_id=service_sruid,
+            service_registries=service_configuration.service_registries,
         )
 
         service.run_function = runner.run