48: Added documentation to azure ml tut p2 (#56)

* switched tensoflow for scikit-learn

Author: MarleneKress79789

tutorials/machine-learning/python/AzureML/ConnectAzureMLtoExasol.ipynb

@@ -44,7 +44,7 @@
    "source": [
     "## Why using Azure blobstorage is necessary\n",
     "\n",
-    "In this tutorial we copy the data from an Exasol Saas database into an Azure Blobstorage Container. This is necessary because while AzureML has functionality to import directly from SQL databases, the Exasol SQL dialect is not supported by AzureML at the moment of writing.\n"
+    "In this tutorial we copy the data from an Exasol Saas Database into an Azure Blob Storage Container. This is necessary because while AzureML has functionality to import directly from SQL databases, the Exasol SQL dialect is not supported by AzureML at the moment of writing.\n"
    ],
    "metadata": {
     "collapsed": false
@@ -129,20 +129,9 @@
   {
    "cell_type": "markdown",
    "source": [
-    "## Data Preprocessing\n",
-    "explanation\n",
-    "    - why here not in azure\n",
-    "    - what gets done\n",
+    "We will also need to get access to the Azure Storage Account, which we will use later to transfer the data. For that, you need to insert your Azure Storage Account Name and Access Key. To find your Access Key, in the Azure portal navigate to your Storage Account, and click on \"Access Keys\" under \"Security + networking\" and copy one of your Access Keys.\n",
     "\n",
-    "\"There are two things we need to do:\n",
-    "\n",
-    "    Split into train and validation data\n",
-    "    Replace CLASS column by a column with boolean values\n",
-    "\n",
-    "For the split we add a column SPLIT that has a random value between 0 and 1, so we can partition the data by a condition on that column.\n",
-    "\n",
-    "In addition, we replace the CLASS with the text values pos and neg by a new column CLASS_POS with boolean values.\"\n",
-    "     - mention test table to"
+    "![](img_src/access_key_azure.png)\n"
    ],
    "metadata": {
     "collapsed": false
@@ -153,17 +142,12 @@
    "execution_count": null,
    "outputs": [],
    "source": [
-    "all_columns = exasol.export_to_pandas(\"SELECT * FROM IDA.TRAIN LIMIT 1;\")\n",
-    "column_names = list(all_columns)\n",
-    "column_names.remove(\"CLASS\")\n",
-    "exasol.execute(\"\"\"CREATE OR REPLACE TABLE IDA.TRAIN_PREPARED AS (\n",
-    "               SELECT RANDOM() AS SPLIT,\n",
-    "               (CLASS = 'pos') as CLASS_POS, {all_columns_except_class!q} FROM IDA.TRAIN)\"\"\",\n",
-    "               {\"all_columns_except_class\": column_names})\n",
-    "\n",
+    "from azure.ai.ml.entities import AccountKeyConfiguration\n",
     "\n",
+    "my_storage_account_name = \"your_storage_account_name\"   # change\n",
+    "account_key=\"your_storage_account_key\"                  # change\n",
     "\n",
-    "exasol.export_to_pandas(\"SELECT * FROM IDA.TRAIN_PREPARED LIMIT 4\")"
+    "credentials= AccountKeyConfiguration(account_key)"
    ],
    "metadata": {
     "collapsed": false,
@@ -172,19 +156,37 @@
     }
    }
   },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "### Data Preprocessing\n",
+    "\n",
+    "Now that we are set up for the data transfer, we are first going to preprocess the data in the Exasol Database before pulling the data into Azure. We want to replace the text based \"CLASS\" column all data tables with a boolean column called \"CLASS_POS\" which will make classifying easier.\n",
+    "\n",
+    "For your own project, you need to evaluate which preprocessing steps to run in the efficient Exasol Database and which might be easier to accomplish later on the CSV files in Azure Blob Storage.\n",
+    "\n",
+    "First, we create a new table \"TRAIN_PREPARED\" which is a copy of the \"TRAIN\" table, with the replaced \"CLASS_POS\" column."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "outputs": [],
    "source": [
-    "exasol.execute(\"\"\"CREATE OR REPLACE TABLE IDA.TEST_PREPARED AS (\n",
-    "               SELECT\n",
-    "               (CLASS = 'pos') as CLASS_POS, {all_columns_except_class!q} FROM IDA.TEST)\"\"\",\n",
+    "all_columns = exasol.export_to_pandas(\"SELECT * FROM IDA.TRAIN LIMIT 1;\")\n",
+    "column_names = list(all_columns)\n",
+    "column_names.remove(\"CLASS\")\n",
+    "exasol.execute(\"\"\"CREATE OR REPLACE TABLE IDA.TRAIN_PREPARED AS (\n",
+    "                SELECT\n",
+    "               (CLASS = 'pos') as CLASS_POS, {all_columns_except_class!q} FROM IDA.TRAIN)\"\"\",\n",
     "               {\"all_columns_except_class\": column_names})\n",
     "\n",
     "\n",
     "\n",
-    "exasol.export_to_pandas(\"SELECT * FROM IDA.TEST_PREPARED LIMIT 4\")"
+    "exasol.export_to_pandas(\"SELECT * FROM IDA.TRAIN_PREPARED LIMIT 4\")"
    ],
    "metadata": {
     "collapsed": false,
@@ -196,13 +198,7 @@
   {
    "cell_type": "markdown",
    "source": [
-    "\n",
-    "### Load data into AzureML Blobstore\n",
-    "\n",
-    "\n",
-    "For this step, we need to access the Azure Storage Account. For that you need to insert your Azure storage account name and access key. To find your access key, in the Azure portal navigate to your storage account, and click on \"Access Keys\" under \"Security + networking\" and copy one of your access Keys.\n",
-    "\n",
-    "![](img_src/access_key_azure.png)\n"
+    "Then we create a new \"TEST_PREPARED\" table as a copy of the \"TEST\" table with replaced \"CLASS_POS\" column."
    ],
    "metadata": {
     "collapsed": false
@@ -213,12 +209,14 @@
    "execution_count": null,
    "outputs": [],
    "source": [
-    "from azure.ai.ml.entities import AccountKeyConfiguration\n",
+    "exasol.execute(\"\"\"CREATE OR REPLACE TABLE IDA.TEST_PREPARED AS (\n",
+    "               SELECT\n",
+    "               (CLASS = 'pos') as CLASS_POS, {all_columns_except_class!q} FROM IDA.TEST)\"\"\",\n",
+    "               {\"all_columns_except_class\": column_names})\n",
     "\n",
-    "my_storage_account_name = \"your_storage_account_name\"   # change\n",
-    "account_key=\"your_storage_account_key\"                  # change\n",
     "\n",
-    "credentials= AccountKeyConfiguration(account_key)"
+    "\n",
+    "exasol.export_to_pandas(\"SELECT * FROM IDA.TEST_PREPARED LIMIT 4\")"
    ],
    "metadata": {
     "collapsed": false,
@@ -230,9 +228,11 @@
   {
    "cell_type": "markdown",
    "source": [
-    "Lastly, we use an \"EXPORT TABLE\" command for each of our data tables to export them  into a CSV file in our Blobstorage using \"INTO CSV AT CLOUD AZURE BLOBSTORAGE\". You can find [the domumentation for this export command](https://docs.exasol.com/db/latest/sql/export.htm) in the Exasol documentation.\n",
-    "If you choose an existing \"azure_storage_container_name\", this command will save your files in this container. Otherwise, an azure storage container with that name will be created automatically.\n",
-    "When you created your AzureML workspace, an Azure blob container was [created automatically](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data) and added as a Datastore named \"workspaceblobstore\" to your workspace. You can use it here and then scip the \"Create a Datastore\" step below if you want. For this you would need to find its name (\"azureml-blobstore-some-ID\") in the datastore info and insert it here."
+    "### Load Data into AzureML Blob Storage\n",
+    "\n",
+    "Now that our data is prepared and we have access to our Azure Storage Account and our Exasol Saas Cluster, we use an \"EXPORT TABLE\" command for each of our data tables to export them into a CSV file in our Blob Storage using \"INTO CSV AT CLOUD AZURE BLOBSTORAGE\". You can find [the domumentation for this export command](https://docs.exasol.com/db/latest/sql/export.htm) in the Exasol documentation.\n",
+    "If you choose an existing Azure Blob Storage container, this command will save your files in this container. Otherwise, a new container with the given name will be created automatically.\n",
+    "When you created your AzureML Workspace, an Azure Blob Container was [created automatically](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data) and added as a Datastore named \"workspaceblobstore\" to your workspace. You can use it here and then skip the \"Create a Datastore\" step below if you want. For this you would need to find its name (\"azureml-blobstore-some-ID\") in the Datastore Information and insert it here."
    ],
    "metadata": {
     "collapsed": false
@@ -241,11 +241,9 @@
   {
    "cell_type": "markdown",
    "source": [
-    "## todo\n",
-    "- change and add explanation to preprocessing\n",
-    "- update image of loaded tables (reload without split column beforehand)\n",
-    "- add notze about selecing columns\n",
-    "- add note about importing more than once -> appends not make new file!"
+    "Some of the 170 features of the Scania Trucks dataset do not have a notable influence on the classification or contain a big amount of empty values. Because of this we select only some columns to actually use for the training. Since we only want to use them, we import only these features to Azure.\n",
+    "\n",
+    "Once we have selected the column names we want to use, we transfer the \"TEST_PREPARED\" table using the exasol EXPORT command."
    ],
    "metadata": {
     "collapsed": false
@@ -264,10 +262,19 @@
     "                    'CS_001', 'DD_000', 'DE_000', 'DN_000', 'DS_000', 'DU_000', 'DV_000', 'EB_000', 'EE_005']\n",
     "\n",
     "blobstorage_name = \"azureml-tutorial\"   # change, remember to you might need to remove the \"_datastore\" suffix\n",
+    "\n",
     "save_path = f'{blobstorage_name}/ida/{table}'\n",
-    "sql_export = \"EXPORT (SELECT {column_names!q}\" + f\" FROM IDA.{table}) INTO CSV AT CLOUD AZURE BLOBSTORAGE 'DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net'\"\\\n",
-    "                f\"USER '{my_storage_account_name}' IDENTIFIED BY '{credentials.account_key}' FILE '{save_path}' WITH COLUMN NAMES\"\n",
-    "exasol.execute(sql_export, {\"column_names\": column_names})\n",
+    "sql_export = \"\"\"EXPORT (SELECT {column_names!q} FROM IDA.{table!q})\n",
+    "                INTO CSV AT CLOUD AZURE BLOBSTORAGE 'DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net'\n",
+    "                USER '{my_storage_account_name!q}' IDENTIFIED BY '{account_key!q}'\n",
+    "                FILE '{save_path!q}' WITH COLUMN NAMES REPLACE\"\"\"\n",
+    "\n",
+    "\n",
+    "exasol.execute(sql_export, {\"column_names\": column_names,\n",
+    "                            \"table\": table,\n",
+    "                            \"my_storage_account_name\": my_storage_account_name,\n",
+    "                            \"account_key\": credentials.account_key,\n",
+    "                            \"save_path\": save_path})\n",
     "print(f\"saved {table} in file {save_path}\")"
    ],
    "metadata": {
@@ -277,23 +284,28 @@
     }
    }
   },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Then we do the same with the TRAIN_PREPARED table:"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "outputs": [],
    "source": [
-    "\n",
     "table = \"TRAIN_PREPARED\"\n",
     "save_path = f'{blobstorage_name}/ida/{table}'\n",
-    "sql_export = \"EXPORT (SELECT {column_names!q}\" + f\" FROM IDA.{table} WHERE SPLIT <= 0.8) INTO CSV AT CLOUD AZURE BLOBSTORAGE 'DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net'\"\\\n",
-    "                f\"USER '{my_storage_account_name}' IDENTIFIED BY '{credentials.account_key}' FILE '{save_path}' WITH COLUMN NAMES\"\n",
-    "exasol.execute(sql_export, {\"column_names\": column_names})\n",
-    "print(f\"saved {table} in file {save_path}\")\n",
-    "\n",
-    "save_path = f'{blobstorage_name}/ida/VALIDATE_PREPARED'\n",
-    "sql_export = \"EXPORT (SELECT {column_names!q}\" + f\" FROM IDA.{table} WHERE SPLIT > 0.8) INTO CSV AT CLOUD AZURE BLOBSTORAGE 'DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net'\"\\\n",
-    "                f\"USER '{my_storage_account_name}' IDENTIFIED BY '{credentials.account_key}' FILE '{save_path}' WITH COLUMN NAMES\"\n",
-    "exasol.execute(sql_export, {\"column_names\": column_names})\n",
+    "\n",
+    "exasol.execute(sql_export, {\"column_names\": column_names,\n",
+    "                            \"table\": table,\n",
+    "                            \"my_storage_account_name\": my_storage_account_name,\n",
+    "                            \"account_key\": credentials.account_key,\n",
+    "                            \"save_path\": save_path})\n",
     "print(f\"saved {table} in file {save_path}\")"
    ],
    "metadata": {
@@ -303,6 +315,15 @@
     }
    }
   },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Delete the temporary tables from the Exasol Saas Database in order to not pollute the database."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -351,7 +372,7 @@
     "You can now see your data directly in AzureML by navigating to \"Datastores\" and clicking on <your_datastore_name> . If you then change into the \"Browse\" view you can open your files and have a look at them if you want.\n",
     "\n",
     "\n",
-    "Great, we successfully connected to our Exasol Saas instance and loaded data from there into our Azure Blobstorage!\n",
+    "Great, we successfully connected to our Exasol Saas instance and loaded data from there into our Azure Blob Storage!\n",
     "\n",
     "Now we move on to [working with the data in AzureML and training a model on it](TrainModelInAzureML.ipynb)."
    ],

tutorials/machine-learning/python/AzureML/InvokeModelFromExasolDBwithUDF.ipynb

@@ -16,7 +16,10 @@
     "- \"Click on Submit Request to expand the section displaying the JSON strings for Execution request and ExecutionResults.\" in azureML? add these to exasol to create data structures?\n",
     "\n",
     "- load model (with web service input and output modules?) and invoke from exasol\n",
-    " - or load model into exasol udf"
+    "- or load model into exasol udf\n",
+    "\n",
+    "+ todo either implement normalization in udf or exasol, or append normalization to model with tensoflow\n",
+    "dont forget to stop compute!"
    ]
   }
  ],