Exploit a hierarchical multi-table schema for sklearn API

[marcboulle]

Description

La spécification actuelle (10.3.1) des datasets pour l'API sklearn est basée sur un schéma entité-relation, proche de celui utilisé par l'outil FeatureTools.

X = {
    "main_table": "Accidents",
    "tables": {
        "Accidents": (accidents_df.drop("Gravity", axis=1), "AccidentId"),
        "Vehicles": (vehicles_df, ["AccidentId", "VehicleId"]),
        "Users": (users_df, ["AccidentId", "VehicleId"]),
        "Places": (places_df, "AccidentId"),
    },
    "relations": [
        ("Accidents", "Vehicles"),
        ("Vehicles", "Users"),
        ("Accidents", "Places", True),
    ],
}

Il s'agit de passer à un schéma hiérarchique, qui est celui des dictionnaires multi-table de Khiops.

      X = {
          "main_table": (accidents_df,  ["AccidentId"]),
          "additional_data_tables": {
              "Vehicles": (vehicles_df, ["AccidentId", "VehicleId"]),
              "Vehicles/Users": (users_df, ["AccidentId", "VehicleId"]),
              "Place": (places_df, ["AccidentId"], True),
          }
      }

Cela présente les avantages suivants:

• conformité au modèle de donnée des dictionnaires Khiops
• simplicité accrue
• transition facilitée de l'API sklearn vers l'API core

Questions/Ideas

Prévoir un chemin de déprécation, en acceptant l'ancienne spécification avec un warning

[popescu-v]

We need to set a suppression target for the older syntax. For a start, we can put 11.1.0.

[popescu-v]

We lose the main table name with the new schema, which would result in less readable reports. To get it back, there are several possibilities:

1. Adding a new, optional, "main_table_name" key to the main dictionary ("main_table" is used as main table name if this field is not specified):

      X = {
          "main_table": (accidents_df,  ["AccidentId"]),
          "main_table_name": "Accidents", # Optional
          "additional_data_tables": {
              "Vehicles": (vehicles_df, ["AccidentId", "VehicleId"]),
              "Vehicles/Users": (users_df, ["AccidentId", "VehicleId"]),
              "Place": (places_df, ["AccidentId"], True),
          }
      }

2. Adding the main table name to the tuple which represents the value of the "main_table" key:

      X = {
          "main_table": ("Accidents", accidents_df,  ["AccidentId"]),
          "additional_data_tables": {
              "Vehicles": (vehicles_df, ["AccidentId", "VehicleId"]),
              "Vehicles/Users": (users_df, ["AccidentId", "VehicleId"]),
              "Place": (places_df, ["AccidentId"], True),
          }
      }

3. Adding the main table name as a dictionary key:

      X = {
          "main_table": {"Accidents": (accidents_df,  ["AccidentId"])},
          "additional_data_tables": {
              "Vehicles": (vehicles_df, ["AccidentId", "VehicleId"]),
              "Vehicles/Users": (users_df, ["AccidentId", "VehicleId"]),
              "Place": (places_df, ["AccidentId"], True),
          }
      }

Solution 2 has the following disadvantages:

• it obliges the users to specify the main table name (or to specify None should they want to omit the main table name, thus to use "main_table" as the main table name;
• it breaks the symmetry with the 2-tuples which represents the values of the secondary table data paths.

Solution 3 has the disadvantage of imposing the usage of a single-key dictionary for the main table specification, which seems unnecessarily complex / costly.

Hence, solution 1 seems preferable among the 3.

[popescu-v]

After discussion, it results that the initially-proposed specification is more consistent with the mono-table behavior, where the single table is always named "main_table".

Moreover, now that we use data paths for specifying the paths from the main table to the secondary tables, the names of the secondary tables aren't explicitly specified either, and this is consistent with the main table situation.

In conclusion, the initial specification would be kept as stated in the issue.

However, we need to provide non-colliding internal names for the secondary tables, as these will be used for:

• naming the secondary tables' files (for perusal by Khiops Core via the Core API);
• naming the dictionaries (according to the tables' file names).

For implementation purposes only: one such way of providing them is to concatenate "main_table" with the data paths, replacing / with _ and making sure the total name length is < 128 characters. If the total name length is >= 128 characters, then an indexing scheme like _i can be used.