Fix and improve crash diagnostics

Summary

This release fixes and adds new features to service crash diagnostics, making it much easier to locally reconstruct errors in a service. This especially true if it has children - crash diagnostics now include a child emulator for every question asked to a child by the service before it failed.

Contents (#541)

IMPORTANT: There are 3 breaking changes.

New features

• Add ability to load crash diagnostics data as a test fixture
• Record messages from each of a service's children when it crashes

Enhancements

• 💥 BREAKING CHANGE: Avoid downloading datasets by default in get-crash-diagnostics
• 💥 BREAKING CHANGE: Save recorded messages to attribute instead of file
• Log a warning if trying to download files from empty directory
• Add Serialisable.from_file method
• Add Manifest.update_dataset_paths method
• Add representation for Runner
• Avoid uploading null strings when values or manifests not present
• Add option to include .octue files when instantiating datasets
• Add ability to filter the files downloaded in GoogleCloudStorageClient.download_all_files

Fixes

• Save original unmutated input and configuration data for crash diagnostics
• Update manifests with local paths if datasets downloaded locally
• Ensure crash diagnostics upload works without manifests
• Ensure empty values or manifests are uploaded for crash diagnostics
• Fix splitting bucket name from signed cloud storage URLs

Refactoring

• 💥 BREAKING CHANGE: Rename filename parameter to path in Serialisable.to_file
• Factor out child instantiation in Runner into a method
• Move AnalysisLogHandlerSwitcher into log_handlers module
• Move AppFrom into new app_loading module
• Remove unused Service._sent_messages attribute
• Rename recorded_messages attributes to received_messages properties
• Factor out manifest/dataset and values uploading for crash diagnostics
• Reduce nesting in Dataset._instantiate_from_cloud
• Factor out app loading and running in Runner

Dependencies

• Update poetry.lock to avoid usage of insecure py package by pytest

Testing

• Make testing downloading crash diagnostics more efficient
• Test crash diagnostics with service that has its own children
• Use test fixture to speed up octue get-crash-diagnostics tests
• Test that dataset/file metadata is uploaded with crash diagnostics
• Test metadata is preserved with octue get-crash-diagnostics

Other

• Clarify service troubleshooting doc

---

Upgrade instructions

💥 Avoid downloading datasets by default in `get-crash-diagnostics`

To keep the previous behaviour, add the --download-datasets flag when using the octue get-crash-diagnostics CLI command

💥 Save recorded messages to attribute instead of file

Instead of using the record_messages_to parameter, set the record_messages parameter to True when using Child.ask or Service.wait_for_answer. Retrieve messages from the received_messages property of either class instance.

💥 Rename filename parameter to path in Serialisable.to_file

If using the positional argument filename in the to_file method of an instance of a Serialisable mixed-in class as a keyword argument, replace it with path.

Fix manifest upload and analysis error logging

Contents (#536)

Fixes

• Always use OctueJSONEncoder when encoding JSON
• Log analysis errors before attempting to save crash diagnostics in Runner

Operations

• Update author email address

Refactoring

• Use Manifest.serialise instead of Manifest.to_primitive in Manifest.to_cloud

Remove `mkver.conf` file and update troubleshooting docs

Contents (#533)

Operations

• Use new version of octue/check-semantic-version action and remove mkver.conf file
• Use latest octue/generate-pull-request-description action
• Add prettier to pre-commit checks

Other

• Fix octue.yaml code block in docs
• Improve troubleshooting services doc
• Update service ID usage in docs

Fully support service namespacing and revision tagging

Summary

This release add support and requirement for service namespaces and service revision tags. This allows services to be versioned and provides a more familiar format for service IDs that's, for example, similar to Docker image names (e.g. octue/example-service:latest). Full validation of service IDs comes as part of this.

The new octue deploy create-push-subscription has also been added as requested by users and will be accompanied by a GitHub action in another repository.

Contents (#529)

IMPORTANT: There are 4 breaking changes.

New features

• 💥 BREAKING CHANGE: Add support for and require revision tags for services
• 💥 BREAKING CHANGE: Validate service IDs before creating or questioning services
• Add ability to set service namespace, name, and revision tag by environment variable
• Add octue deploy create-push-subscription CLI command

Enhancements

• 💥 BREAKING CHANGE: Require services to have a namespace and replace use of organisation with namespace in service configurations
• 💥 BREAKING CHANGE: Remove namespace parameter from Topic and Subscription and apply octue.services Pub/Sub namespace to all topic/subscription paths
• Remove application of octue.services Pub/Sub namespace from Service
• Use a coolname revision tag to allow a service revision with a non-unique SRUID to start as a unique service revision when using the octue start CLI command
• Set the Service.id attribute to the user-friendly SRUID instead of the Pub/Sub-friendly SRUID
• Return SRUID from DataflowDeployer.deploy

Fixes

• Handle inability to find/load version compatibility data file
• Use latest Apache Beam base image in Dataflow Dockerfile
• Fix some log and error messages

Operations

• Speed up and simplify release workflow
• Use new Octue GitHub actions in workflows
• Improve installation process in main Dockerfile
• Use latest commit message checker

Dependencies

• Update Dataflow setup.py file

Refactoring

• Rename OCTUE_NAMESPACE to OCTUE_SERVICES_NAMESPACE
• Remove cool-naming of services when instantiated without id and simplify the name argument

---

Upgrade instructions

💥 Add support for and require revision tags for services

The ServiceConfiguration.service_id attribute and the SERVICE_ID environment variable have been removed.

• ServiceConfiguration: Use octue.cloud.service_id.create_service_id to create IDs from ServiceConfiguration.namespace and ServiceConfiguration.name

• SERVICE_ID environment variable: Use the new OCTUE_SERVICE_NAMESPACE, OCTUE_SERVICE_NAME, and OCTUE_SERVICE_REVISION_TAG environment variables.

💥 Validate service IDs before creating or questioning services

Update your services' namespaces, names, and revision tags to follow the requirements set out in the Creating services doc.

💥 Require services to have a namespace and replace use of organisation with namespace in service configurations

Provide the namespace key in the service configuration (the octue.yaml file). If you were using the organisation key before, the namespace key now replaces it. Examples of a namespace are your organisation's name or your GitHub username.

💥 Remove namespace parameter from Topic and Subscription and apply octue.services Pub/Sub namespace to all topic/subscription paths

The octue.services namespace is now mandatory and applied automatically in topic and subscription paths. Please stop providing the namespace argument if you were providing it before.

Add dynamic children override

Summary

Allow parents to override the default children its child uses (i.e. the parent's grandchildren). This allows services that produce the same type of results but use, for example, different versions of a model to be swapped in and out for each other without modifying the child.

Note that the child has to be running version 0.39.0 to be able to accept the override (otherwise it will just use its default children).

Contents (#524)

New features

• Add ability to dynamically override a child's children when asking it a question

Enhancements

• Use raw service IDs in logs and errors

Fixes

• Use full service ID in MockService when answering question

Refactoring

• Refactor OrderedMessageHandler methods into smaller methods

Testing

• Remove middle-man ask_question_and_wait_for_answer method in TestService

Fix serialisation of datasets instantiated from files

Summary

Ensure that datasets constructed using the files argument are successfully serialised/deserialised without losing file information.

Contents (#525)

Fixes

• Fix serialisation/deserialisation of datasets in Manifest when the datasets have been constructed using the files argument
• Use datafile cloud path in files field if available in Dataset.to_primitive
• Use Manifest.serialise instead of Manifest.to_primitive in Analysis.finalise

Dependencies

• Use twined=^0.5.1

Make Topic and Subscription classes standalone

Summary

Make the Topic and Subscription classes standalone instead of relying on a Service instance and a related subscriber.

Contents (#523)

IMPORTANT: There are 2 breaking changes.

Enhancements

• 💥 BREAKING CHANGE: Remove service argument from Topic constructor
• 💥 BREAKING CHANGE: Remove subscriber argument from Subscription constructor
• Implicitly get Service credentials from environment and remove credentials attribute

Fixes

• Move finally clause to correct nesting level in OrderedMessageHandler.handle_messages

---

Upgrade instructions

💥 Remove `service` argument from `Topic` constructor

Provide the project_name argument instead of the old service argument.

💥 Remove `subscriber` argument from `Subscription` constructor

Remove the old subscriber argument as it's no longer needed.

Enforce unique names for services

Summary

Only allow services with unique names to be started via the octue start CLI command. This ensures that, when using the command:

• A new service can't intercept/receive questions meant for an existing service of the same name
• When running a service locally that's already deployed, a modified service name is generated and used, preventing the deletion of the deployed service's topic/subscription on exit

Contents (#515)

IMPORTANT: There is 1 breaking change.

Enhancements

• 💥 BREAKING CHANGE: Remove the --rm option and instead delete the service topic and subscription by default on exit of the octue start CLI command
• Add --no-rm flag to octue start to avoid topic and subscription deletion
• Allow octue start users to create a service with a variant of the ID in octue.yaml if the service already exists
• Add allow_existing parameter to Service.serve and default it to False (it was always True within the method before)
• Add creation_triggered_locally property to Topic and Subscription to facilitate avoiding deletion of subscriptions and topics that weren't created by the local service

Fixes

• Don't allow answer subscriptions to already exist
• Don't allow a service to be started by octue start if its ID is the same as an existing service's ID
• Ensure "what next" log message for starting a service is displayed at the right time

Testing

• Speed up Topic.exists test
• Simplify Topic tests

Operations

• Improve table generated by version compatibility table printing script

---

Upgrade instructions

💥 Delete topic/subscription at end of octue start command by default

Stop providing the --rm / --delete-topic-and-subscription-on-exit option to the octue start CLI command. Use the --no-rm flag or deploy your service if you want the service topic and subscription to persist after exiting this command.

Add inter-service communication compatibility checking

Summary

This PR adds the use of empirical data (instead of intuition) on compatible versions of octue for inter-service communication to warn about version compatibilities when errors are raised during communication.

Contents (#498)

New features

• Add octue version compatibility checker that uses empirical compatibility data

Enhancements

• Warn if incompatible SDK version detected if an error is raised while handling messages in parent or answering questions in child
• Add recorded questions and empirical version compatibility data for versions since 0.16.0 inclusively

Operations

• Add workflow that checks version compatibility testing has been carried out for the current version
• Add script that prints a table of version compatibilities

Automatically upload output datasets to analysis output location if provided

Summary

When finalising analyses that produce output datasets, automatically upload the datasets to the output_location if it's specified in the app configuration. Additionally, the datasets are put in unique subdirectories within the output_location to avoid unintended overwriting or combining of sets of datasets.

These changes remove the need to explicitly call Analysis.finalise or construct a unique path to upload datasets to within an app when an output_location is provided in the app configuration.

Contents (#519)

Enhancements

• Automatically upload output datasets to output location specified in app configuration in Analysis.finalise
• Automatically use a unique subdirectory within the output location in Analysis.finalise to avoid data loss/pollution