Feature/40 Added training implementation for AzureML tut (#55)

Author: MarleneKress79789

tutorials/machine-learning/python/AzureML/ConnectAzureMLtoExasol.ipynb

@@ -14,6 +14,7 @@
    "source": [
     "In this Tutorial we will:\n",
     " - Connect to Exasol SaaS from AzureML\n",
+    " - Preprocess data\n",
     " - Export Exasol tables to an Azure Blobstore Container\n",
     " - Create a Datastore"
    ],
@@ -125,6 +126,73 @@
     }
    }
   },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "## Data Preprocessing\n",
+    "explanation\n",
+    "    - why here not in azure\n",
+    "    - what gets done\n",
+    "\n",
+    "\"There are two things we need to do:\n",
+    "\n",
+    "    Split into train and validation data\n",
+    "    Replace CLASS column by a column with boolean values\n",
+    "\n",
+    "For the split we add a column SPLIT that has a random value between 0 and 1, so we can partition the data by a condition on that column.\n",
+    "\n",
+    "In addition, we replace the CLASS with the text values pos and neg by a new column CLASS_POS with boolean values.\"\n",
+    "     - mention test table to"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "all_columns = exasol.export_to_pandas(\"SELECT * FROM IDA.TRAIN LIMIT 1;\")\n",
+    "column_names = list(all_columns)\n",
+    "column_names.remove(\"CLASS\")\n",
+    "exasol.execute(\"\"\"CREATE OR REPLACE TABLE IDA.TRAIN_PREPARED AS (\n",
+    "               SELECT RANDOM() AS SPLIT,\n",
+    "               (CLASS = 'pos') as CLASS_POS, {all_columns_except_class!q} FROM IDA.TRAIN)\"\"\",\n",
+    "               {\"all_columns_except_class\": column_names})\n",
+    "\n",
+    "\n",
+    "\n",
+    "exasol.export_to_pandas(\"SELECT * FROM IDA.TRAIN_PREPARED LIMIT 4\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "exasol.execute(\"\"\"CREATE OR REPLACE TABLE IDA.TEST_PREPARED AS (\n",
+    "               SELECT\n",
+    "               (CLASS = 'pos') as CLASS_POS, {all_columns_except_class!q} FROM IDA.TEST)\"\"\",\n",
+    "               {\"all_columns_except_class\": column_names})\n",
+    "\n",
+    "\n",
+    "\n",
+    "exasol.export_to_pandas(\"SELECT * FROM IDA.TEST_PREPARED LIMIT 4\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
   {
    "cell_type": "markdown",
    "source": [
@@ -170,19 +238,78 @@
     "collapsed": false
    }
   },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "## todo\n",
+    "- change and add explanation to preprocessing\n",
+    "- update image of loaded tables (reload without split column beforehand)\n",
+    "- add notze about selecing columns\n",
+    "- add note about importing more than once -> appends not make new file!"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "table = \"TEST_PREPARED\"\n",
+    "column_names = ['CLASS_POS', 'AA_000', 'AG_005', 'AH_000', 'AL_000', 'AM_0', 'AN_000', 'AO_000', 'AP_000', 'AQ_000',\n",
+    "                    'AZ_004', 'BA_002', 'BB_000', 'BC_000', 'BD_000', 'BE_000',\n",
+    "                    'BF_000', 'BG_000', 'BH_000', 'BI_000', 'BJ_000', 'BS_000', 'BT_000', 'BU_000', 'BV_000',\n",
+    "                    'BX_000', 'BY_000', 'BZ_000', 'CA_000', 'CB_000', 'CC_000', 'CI_000', 'CN_004', 'CQ_000',\n",
+    "                    'CS_001', 'DD_000', 'DE_000', 'DN_000', 'DS_000', 'DU_000', 'DV_000', 'EB_000', 'EE_005']\n",
+    "\n",
+    "blobstorage_name = \"azureml-tutorial\"   # change, remember to you might need to remove the \"_datastore\" suffix\n",
+    "save_path = f'{blobstorage_name}/ida/{table}'\n",
+    "sql_export = \"EXPORT (SELECT {column_names!q}\" + f\" FROM IDA.{table}) INTO CSV AT CLOUD AZURE BLOBSTORAGE 'DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net'\"\\\n",
+    "                f\"USER '{my_storage_account_name}' IDENTIFIED BY '{credentials.account_key}' FILE '{save_path}' WITH COLUMN NAMES\"\n",
+    "exasol.execute(sql_export, {\"column_names\": column_names})\n",
+    "print(f\"saved {table} in file {save_path}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "\n",
+    "table = \"TRAIN_PREPARED\"\n",
+    "save_path = f'{blobstorage_name}/ida/{table}'\n",
+    "sql_export = \"EXPORT (SELECT {column_names!q}\" + f\" FROM IDA.{table} WHERE SPLIT <= 0.8) INTO CSV AT CLOUD AZURE BLOBSTORAGE 'DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net'\"\\\n",
+    "                f\"USER '{my_storage_account_name}' IDENTIFIED BY '{credentials.account_key}' FILE '{save_path}' WITH COLUMN NAMES\"\n",
+    "exasol.execute(sql_export, {\"column_names\": column_names})\n",
+    "print(f\"saved {table} in file {save_path}\")\n",
+    "\n",
+    "save_path = f'{blobstorage_name}/ida/VALIDATE_PREPARED'\n",
+    "sql_export = \"EXPORT (SELECT {column_names!q}\" + f\" FROM IDA.{table} WHERE SPLIT > 0.8) INTO CSV AT CLOUD AZURE BLOBSTORAGE 'DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net'\"\\\n",
+    "                f\"USER '{my_storage_account_name}' IDENTIFIED BY '{credentials.account_key}' FILE '{save_path}' WITH COLUMN NAMES\"\n",
+    "exasol.execute(sql_export, {\"column_names\": column_names})\n",
+    "print(f\"saved {table} in file {save_path}\")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "outputs": [],
    "source": [
-    "azure_storage_container_name = \"your-container-name\"   # change, remember to you might need to remove the \"_datastore\" suffix\n",
-    "\n",
-    "for table in [\"TEST\", \"TRAIN\"]:\n",
-    "    save_path = f'{azure_storage_container_name}/ida/{table}'\n",
-    "    sql_export = f\"EXPORT TABLE IDA.{table} INTO CSV AT CLOUD AZURE BLOBSTORAGE 'DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net'\"\\\n",
-    "                f\"USER '{my_storage_account_name}' IDENTIFIED BY '{credentials.account_key}' FILE '{save_path}'\"\n",
-    "    exasol.execute(sql_export)\n",
-    "    print(f\"Saved {table} in file {save_path}\")\n"
+    "for table in [\"TRAIN_PREPARED\", \"TEST_PREPARED\"]:\n",
+    "    exasol.execute(f\"DROP TABLE IDA.{table};\")"
    ],
    "metadata": {
     "collapsed": false,

tutorials/machine-learning/python/AzureML/InvokeModelFromExasolDBwithUDF.ipynb

@@ -9,9 +9,14 @@
     }
    },
    "source": [
+    "- In this Tutorial You will ..\n",
+    "- Prerequisites\n",
     "- use model to predict in azureml from exasol? return result single/table as input\n",
     "- \"set up and deploy a web service.. You can then use the API URL and Key of the BES Post URL found on the Batch Exaction Help page. (https://github.com/toddkitta/azure-content/blob/master/articles/machine-learning/machine-learning-walkthrough-5-publish-web-service.md)\" necessary?\n",
-    "- \"Click on Submit Request to expand the section displaying the JSON strings for Execution request and ExecutionResults.\" in azureML? add these to exasol to create data structures?"
+    "- \"Click on Submit Request to expand the section displaying the JSON strings for Execution request and ExecutionResults.\" in azureML? add these to exasol to create data structures?\n",
+    "\n",
+    "- load model (with web service input and output modules?) and invoke from exasol\n",
+    " - or load model into exasol udf"
    ]
   }
  ],