#47: Added documentation for AzureML tutorial p1 (#52)

Author: MarleneKress79789

tutorials/machine-learning/python/AzureML/ConnectAzureMLtoExasol.ipynb

@@ -13,10 +13,9 @@
    "cell_type": "markdown",
    "source": [
     "In this Tutorial we will:\n",
-    " - create blob storage\n",
-    " - connect to Exasol SaaS\n",
-    " - load tables in AzureML\n",
-    " - save tables to blob storage as csv files\n"
+    " - Connect to Exasol SaaS from AzureML\n",
+    " - Export Exasol tables to an Azure Blobstore Container\n",
+    " - Create a Datastore"
    ],
    "metadata": {
     "collapsed": false
@@ -26,7 +25,14 @@
    "cell_type": "markdown",
    "source": [
     "## Prerequisites\n",
-    " (..)"
+    "\n",
+    "You will need:\n",
+    " - Your running Exasol Saas Cluster with your data loaded into it\n",
+    " - Authentication information for your Exasol Saas Cluster\n",
+    " - An AzureML account and Azure Storage account\n",
+    " - AzureML set up with a:\n",
+    "    - Workspace\n",
+    "    - Compute instance"
    ],
    "metadata": {
     "collapsed": false
@@ -35,9 +41,9 @@
   {
    "cell_type": "markdown",
    "source": [
-    "## Why blob storage is necessary\n",
-    "(explanation)\n",
-    "(word on data duplication?)"
+    "## Why using Azure blobstorage is necessary\n",
+    "\n",
+    "In this tutorial we copy the data from an Exasol Saas database into an Azure Blobstorage Container. This is necessary because while AzureML has functionality to import directly from SQL databases, the Exasol SQL dialect is not supported by AzureML at the moment of writing.\n"
    ],
    "metadata": {
     "collapsed": false
@@ -48,40 +54,8 @@
    "source": [
     "## AzureML setup\n",
     "\n",
-    "\n",
-    "    - create\n",
-    "        - resource group,\n",
-    "        - workspace,\n",
-    "        - launch  azML studio\n",
-    "        - compute instance\n",
-    "        - notebook (this notebook) link your compute\n",
-    "    - in compute find public ip (if load data was run from inside azurML skip this)\n",
-    "    add to exasol saas like local above\n",
-    "\n",
-    "\"When you create a workspace, an Azure blob container and an Azure file share are automatically registered as datastores to the workspace. They're named workspaceblobstore and workspacefilestore, respectively. \"\"https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data\"\n",
-    "can find in data in azureml and in azure portal storage account in container as \"azureml-blobstore-someID\"\n",
-    "\n",
-    "if you do not want to use default blobstore\n",
-    "   - create blob storage:\n",
-    "    - in azure portal of storage account:\n",
-    "        -> data storage -> containers -> + container -> in menu on left enter name -> create\n",
-    "\n",
-    "        while here:     -> navigate security & networking -> access keys : remember access key and storage account name\n",
-    "\n",
-    "\n",
-    "\n",
-    "create datastore in azureML :(this step might be moved to following tutorial or removed)\n",
-    "    - go to assets -> data  on th left -> datastores -> create\n",
-    "    menu opens on right. ender:\n",
-    "        - datastore name\n",
-    "        - datastore type (we use blob storage because the offered SQL databases are not compatible with exasol atm.)\n",
-    "        - subscription if needed\n",
-    "        - your storage account from dropdown ( from step before) pic\n",
-    "        - newly created blob storage/blob container from step above\n",
-    "        - authentication info (key from step before)\n",
-    "            pic\n",
-    "\n",
-    "\n"
+    "If you do not know how to set up your AzureML studio, please refer to the [AzureML documentation](https://learn.microsoft.com/en-us/azure/machine-learning/quickstart-create-resources).\n",
+    "Once you are set up with a workspace and compute instance, you can copy this notebook into your notebook files. Open it and select your compute instance in the drop-down menu at the top of your notebook. Now we can get started with connecting to the Exasol Saas cluster.\n"
    ],
    "metadata": {
     "collapsed": false
@@ -90,7 +64,10 @@
   {
    "cell_type": "markdown",
    "source": [
-    "explanation (link to pyexasol)"
+    "### Connect to Exasol Saas\n",
+    "\n",
+    "\n",
+    "We are going to use the [PyExasol](https://docs.exasol.com/db/latest/connect_exasol/drivers/python/pyexasol.htm) package in order to connect to the Exasol database and read the data. First we need to install PyExasol using pip in your AzureML Compute."
    ],
    "metadata": {
     "collapsed": false
@@ -113,7 +90,8 @@
   {
    "cell_type": "markdown",
    "source": [
-    "explanation"
+    "Then we need to connect with PyExasol to our Exasol Saas Cluster with the data. Change these values to reflect your Cluster.\n",
+    "We ask for 10 lines of our \"IDA.TEST\" table from the [Scania Trucks](https://archive.ics.uci.edu/ml/datasets/IDA2016Challenge) to check if our connection is working."
    ],
    "metadata": {
     "collapsed": false
@@ -126,17 +104,19 @@
    "source": [
     "import pyexasol\n",
     "import pandas\n",
-    "EXASOL_HOST = \"your.clusters.exasol.com\" # change\n",
-    "EXASOL_PORT = \"8563\" # change if needed\n",
-    "EXASOL_USER = \"integration-team\" # change if needed\n",
-    "EXASOL_PASSWORD = \"exa_pat_your_password\" #change\n",
-    "EXASOL_SCHEMA = \"IDA\" # change if needed\n",
+    "\n",
+    "EXASOL_HOST = \"<your>.clusters.exasol.com\"      # change\n",
+    "EXASOL_PORT = \"8563\"                            # change if needed\n",
+    "EXASOL_USER = \"<your-exasol-user>\"              # change\n",
+    "EXASOL_PASSWORD = \"exa_pat_<your_password>\"     # change\n",
+    "EXASOL_SCHEMA = \"IDA\"                           # change if needed\n",
+    "\n",
+    "# get the connection\n",
     "EXASOL_CONNECTION = \"{host}:{port}\".format(host=EXASOL_HOST, port=EXASOL_PORT)\n",
     "exasol = pyexasol.connect(dsn=EXASOL_CONNECTION, user=EXASOL_USER, password=EXASOL_PASSWORD, compression=True)\n",
     "\n",
-    "# check if working\n",
-    "data = exasol.export_to_pandas(\"SELECT * FROM TABLE IDA.TEST LIMIT 10\")\n",
-    "print(data)"
+    "# check if the connection is working\n",
+    "exasol.export_to_pandas(\"SELECT * FROM TABLE IDA.TEST LIMIT 10\")"
    ],
    "metadata": {
     "collapsed": false,
@@ -148,7 +128,13 @@
   {
    "cell_type": "markdown",
    "source": [
-    "explanation"
+    "\n",
+    "### Load data into AzureML Blobstore\n",
+    "\n",
+    "\n",
+    "For this step, we need to access the Azure Storage Account. For that you need to insert your Azure storage account name and access key. To find your access key, in the Azure portal navigate to your storage account, and click on \"Access Keys\" under \"Security + networking\" and copy one of your access Keys.\n",
+    "\n",
+    "![](img_src/access_key_azure.png)\n"
    ],
    "metadata": {
     "collapsed": false
@@ -161,10 +147,10 @@
    "source": [
     "from azure.ai.ml.entities import AccountKeyConfiguration\n",
     "\n",
-    "my_storage_account_name = \"your_storage_account_name\"\n",
-    "credentials= AccountKeyConfiguration(\n",
-    "        account_key=\"your_storage_account_key\"\n",
-    "    )"
+    "my_storage_account_name = \"your_storage_account_name\"   # change\n",
+    "account_key=\"your_storage_account_key\"                  # change\n",
+    "\n",
+    "credentials= AccountKeyConfiguration(account_key)"
    ],
    "metadata": {
     "collapsed": false,
@@ -176,8 +162,9 @@
   {
    "cell_type": "markdown",
    "source": [
-    "explanation\n",
-    "mention small table size in example"
+    "Lastly, we use an \"EXPORT TABLE\" command for each of our data tables to export them  into a CSV file in our Blobstorage using \"INTO CSV AT CLOUD AZURE BLOBSTORAGE\". You can find [the domumentation for this export command](https://docs.exasol.com/db/latest/sql/export.htm) in the Exasol documentation.\n",
+    "If you choose an existing \"azure_storage_container_name\", this command will save your files in this container. Otherwise, an azure storage container with that name will be created automatically.\n",
+    "When you created your AzureML workspace, an Azure blob container was [created automatically](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data) and added as a Datastore named \"workspaceblobstore\" to your workspace. You can use it here and then scip the \"Create a Datastore\" step below if you want. For this you would need to find its name (\"azureml-blobstore-some-ID\") in the datastore info and insert it here."
    ],
    "metadata": {
     "collapsed": false
@@ -188,10 +175,14 @@
    "execution_count": null,
    "outputs": [],
    "source": [
+    "azure_storage_container_name = \"your-container-name\"   # change, remember to you might need to remove the \"_datastore\" suffix\n",
+    "\n",
     "for table in [\"TEST\", \"TRAIN\"]:\n",
+    "    save_path = f'{azure_storage_container_name}/ida/{table}'\n",
     "    sql_export = f\"EXPORT TABLE IDA.{table} INTO CSV AT CLOUD AZURE BLOBSTORAGE 'DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net'\"\\\n",
-    "                f\"USER '{my_storage_account_name}' IDENTIFIED BY '{credentials.account_key}' FILE 'azureml-tutorial/ida/{table}'\"\n",
-    "    exasol.execute(sql_export)\n"
+    "                f\"USER '{my_storage_account_name}' IDENTIFIED BY '{credentials.account_key}' FILE '{save_path}'\"\n",
+    "    exasol.execute(sql_export)\n",
+    "    print(f\"Saved {table} in file {save_path}\")\n"
    ],
    "metadata": {
     "collapsed": false,
@@ -203,8 +194,39 @@
   {
    "cell_type": "markdown",
    "source": [
-    "- Success!\n",
-    "- pic of show tables in AzureML storage"
+    "You can check the success of the command by navigating to your Container in the Azure portal using your Azure storage account.\n",
+    "In the menu on the left, you can find \"Containers\" under \"Data Storage\". Find the container named \"your-container-name\" and click on it. Your files should be there.\n"
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "### Create a Datastore\n",
+    "\n",
+    "We recommend that you create a connection between your Azure Storage Container and your AzureML Workspace. For this, enter your workspace in AzureML Studio and select \"Data\" under \"Assets\" in the menu on the left. Now select \"Datastores\" and click on \"+Create\".\n",
+    "\n",
+    "![](img_src/create_datastore.png)\n",
+    "\n",
+    "In the view that opens you need to enter the info for your datastore. Enter a name and select the type as \"Azure Blob Storage\". Then select your Azure subscription and the blob container we loaded the data into from the drop-down menu. Use Authentication type Account key and enter your Azure storage account access key. Click create."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "![](img_src/data_blobstore.png)\n",
+    "\n",
+    "You can now see your data directly in AzureML by navigating to \"Datastores\" and clicking on <your_datastore_name> . If you then change into the \"Browse\" view you can open your files and have a look at them if you want.\n",
+    "\n",
+    "\n",
+    "Great, we successfully connected to our Exasol Saas instance and loaded data from there into our Azure Blobstorage!\n",
+    "\n",
+    "Now we move on to [working with the data in AzureML and training a model on it](TrainModelInAzureML.ipynb)."
    ],
    "metadata": {
     "collapsed": false