Add Google Cloud Storage support; deprecate python < 3.8

Contents

New Features

• Add GoogleCloudStorageClient
• Write manifest, its datasets, and its datafiles to cloud in Analysis.finalise (#96)
• Closes #84 - add auto tag and release workflow
• Allow Google Cloud storage blobs to be represented by Pathable
• Add Datafile, Dataset, and Manifest to_cloud and from_cloud methods
• Allow regular GCP files to be represented as Datafiles

Minor fixes and improvements

• Add cloud storage emulator once for all tests
• Add disk usage and file age utilities
• Allow Datasets to have custom names
• Add storage.path module akin to os.path but for Google Cloud Storage paths
• Allow Hashables' hash values to be set
• Pass GCP project and bucket names to tests from environment (#93)
• Add ability to delete topic and subscription when a Service has finished serving
• Facilitate graceful exit for serving Services on KeyboardInterrupt
• Use latest versions of flake8, isort, and black in pre-commit and across all files (#87)
• Fix CI test skipping flag
• Fix documentation links (#92)

Breaking changes

• Remove testing and explicit support for python3.6 and python3.7
• Remove base_from from Pathable and replace with more transparent method
• Rename Datafile.posix_timestamp to Datafile.timestamp and remove default value
• Make Datafile.last_modified private
• Rename persistence subpackage to storage

Testing

• Test that children can question their own children as part of answering a question
• Close #94 - delete topics and subscriptions at the end of each test
• Remove timeouts from tests and replace with thread executor shutdown upon test pass, meaning that tests that connect to Google Pub/Sub won't fail because the connection is slower than expected

Child services, documentation, easier logging, and CI

Contents

New Features

• Enable use of child services - solving #46.
  • To solve #57 we need to be able to define and run local children as well as remote ones.
  • This means we must allow multiple services to run locally and independently...
  • Which probably means we can also solve octue/twined-server#2 at the same time
• Enable Documentation Build and Serve, Update README #70
  • Ultimately we wish to unify documentation between twined and octue-sdk-python, but this is best done at the time of refactoring large chunks of octue-sdk-python into twined ( See #69 ) but at the moment we wish to just serve what we've got so we can at least link to it.
• Add option to handle developer logs separately from Scientist logs (#78)
• Allow skipping of CI tests if #skip_ci_tests is in the commit body - the use case is to reduce unnecessary computation when knowing the tests will fail for a commit but still wanting to commit.

Minor fixes and improvements

• Implement a proper issue template, either derived from .github repo or applied directly (c.f. octue/twined#60 )
• Close #32 - stop CLI tests leaving output files in working area.

Patch release: Add filtering capability and data input hashing

New Features

• Add Hashable mixin to create data hashes of Datafiles and their containers, including metadata
• Add Filterable mixin to make Datafiles in a Dataset and Tags in a TagSet filterable
• Add FilterSet class for storing and facilitating filtering of Datafiles in Dataset and Tags in TagSet
• Add FilterList for the output of ordering FilterSets
• Add a plethora of type- and interface-based filters for use on FilterSets and FilterLists

Breaking changes

• Any sha256 properties have been replaced with blake3_hash properties
• Replace TagSet.has_tag with native `contains method
• Rename TagSet.starts_with to TagSet.any_tag_starts_with
• Rename TagSet.ends_with to TagSet.any_tag_ends_with

Minor fixes and improvements

• Replace SHA256 hashing with BLAKE3 hashing - this is reportedly around 10 times as fast!
• Neaten up #40/#42 by combining the two workflow files while still keeping separate checks on GitHub. This also makes the version check display less verbosely on GitHub
• Add Tag class, neatening up the retrieval and filtering of subtags
• Rename TagGroup to TagSet to reflect that it's set-based

Version consistency checks and Output Manifest fixes

Contents

New Features

• Closes #40 - Add Github setup.py/branch version consistency check
• Codecov - README badge points to correct base branch

Minor fixes and improvements

• Closes #45 - Analysis attributes are None when they shouldn't be

Improve basic CLI functionality

New Features

• Connect pre-existing CLI options to Runner (#34)
• Allow remote logging to a websocket (#35)

Other

• Remove travis builds (#39)
• Add GitHub actions
• Add issue and PR templates

CLI Improvements

New Features

• Updated CLI to a central CLI rather than a generative process that gives a CLI for each application #9
• Updated dependencies of the fractal example for test with twined server (a tide-us-over stopgap)
• Addition of a Pathable Class, which resolves discussions in #16 for managing paths
• Adds a Context Manager for Datafiles which elegantly handles construction of data paths and opening of files in write mode #3
• Adds a complete example for use and creation of datasets via manifests.

Minor Changes

• Various performance tuning and refactoring for simpler code by @cordatocodes
• Negates need for #15 because Pathable uses the os.path builtins properly, to clean paths and resolve between absolute and relative
• Serialisables can now exclude fields as well as explicitly including them
• Style reapplied throughout (pre-commit didn't get installed)
• Resolve issue #27 problem where the two tests of the run command of the CLI pass by themselves, but not when run in serial with test_fractal_configuration
• Decision: Extra unused strand data should be passed to Runner to enable simplified CLI (see test_exception_raised_when_extra_strand_data_present)
• [ x] Removes unused tmp_dir #30
• Fix #20 CLI raises TypeError when running from command [NB this should be fixed, but have yet to test the caller]

Major API improvement for Twined==0.0.12

Merge pull request #18 from octue/release/0.1.3

Release/0.1.3

Release to provide basic SDK functionality

v0.1.2

Version bump for release

Base release to verify PyPI works correctly

v0.1.0

General updates figuring out how PyPI works