{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2d7656f8-f188-4b96-8d47-c4225ae0e96a",
   "metadata": {},
   "source": [
    "# Dask clusters\n",
    "\n",
    "This page describes how to create, scale, use and shut down Dask clusters.\n",
    "\n",
    "For an example of using Dask clusters to analyse data from Ag3, see the video below.\n",
    "\n",
    "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/aeRoyZdDojU\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f32d775d-fba3-46ff-bb98-af76d2782e83",
   "metadata": {},
   "source": [
    "## Accessing the gateway\n",
    "\n",
    "@@TODO explain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d4473b8d-0ccf-4355-a443-7f6d120832a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask_gateway import Gateway\n",
    "gateway = Gateway()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95413fef-d79d-49be-9498-b9080ad69453",
   "metadata": {},
   "source": [
    "## Creating a cluster\n",
    "\n",
    "@@TODO explain, selecting environment, selecting profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4ac8a099-2c6f-4798-9363-fe030d5001cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'binder-v3.0.0'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "conda_environment = os.environ['CONDA_PREFIX'].split('/')[-1]\n",
    "conda_environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "81c22b02-89d9-4802-8cc4-274b76a75334",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ad3fc077b1b4a99a52bfc472272fbbf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HTML(value='<h2>Cluster Options</h2>'), GridBox(children=(HTML(value=\"<p style='font-weight: bo…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cluster_options = gateway.cluster_options()\n",
    "cluster_options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c0c12aa3-45bc-4837-bc61-c0767a311683",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Standard Worker',\n",
       " 'High Memory Worker',\n",
       " 'Very High Memory Worker',\n",
       " 'Standard Worker Not Preemptible')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster_options._fields['profile'].options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "dbeab462-2885-492e-a854-d6bb55295c3e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f1068953d611453983b1e9af874f2851",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HTML(value='<h2>GatewayCluster</h2>'), HBox(children=(HTML(value='\\n<div>\\n<style scoped>\\n    …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cluster = gateway.new_cluster(environment=conda_environment, profile=\"Standard Worker\")\n",
    "cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2509fdf-d319-4cfa-b09c-84d77993dddb",
   "metadata": {},
   "source": [
    "## Scaling a cluster\n",
    "\n",
    "@@TODO explain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ccb4f596-6b28-4680-a12b-7c4f4e22b6fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.scale(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b4d9f207-4f02-49d5-932f-e8e41a2ca4eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.scale(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e8655d71-499f-44e5-9bbc-b8cfd99353b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.adapt(minimum=0, maximum=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "170d748f-31a4-4dad-95f2-157deaf5ec58",
   "metadata": {},
   "source": [
    "## Connecting to a cluster\n",
    "\n",
    "@@TODO explain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "268d41e4-2677-4b59-b0fc-98cd3553ea08",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = cluster.get_client()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4137c4d9-8d0a-47d4-8d78-085cc933a9f9",
   "metadata": {},
   "source": [
    "## Running a computation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "bc814f9f-5e03-4775-ae21-1a7405f2fa42",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr>\n",
       "<td>\n",
       "<table>\n",
       "  <thead>\n",
       "    <tr><td> </td><th> Array </th><th> Chunk </th></tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr><th> Bytes </th><td> 4.00 MB </td> <td> 8.00 kB </td></tr>\n",
       "    <tr><th> Shape </th><td> (500000,) </td> <td> (1000,) </td></tr>\n",
       "    <tr><th> Count </th><td> 59000 Tasks </td><td> 500 Chunks </td></tr>\n",
       "    <tr><th> Type </th><td> float64 </td><td> numpy.ndarray </td></tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</td>\n",
       "<td>\n",
       "<svg width=\"170\" height=\"75\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"120\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"0\" y1=\"25\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"0\" y2=\"25\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"6\" y1=\"0\" x2=\"6\" y2=\"25\" />\n",
       "  <line x1=\"12\" y1=\"0\" x2=\"12\" y2=\"25\" />\n",
       "  <line x1=\"18\" y1=\"0\" x2=\"18\" y2=\"25\" />\n",
       "  <line x1=\"25\" y1=\"0\" x2=\"25\" y2=\"25\" />\n",
       "  <line x1=\"31\" y1=\"0\" x2=\"31\" y2=\"25\" />\n",
       "  <line x1=\"37\" y1=\"0\" x2=\"37\" y2=\"25\" />\n",
       "  <line x1=\"44\" y1=\"0\" x2=\"44\" y2=\"25\" />\n",
       "  <line x1=\"50\" y1=\"0\" x2=\"50\" y2=\"25\" />\n",
       "  <line x1=\"56\" y1=\"0\" x2=\"56\" y2=\"25\" />\n",
       "  <line x1=\"63\" y1=\"0\" x2=\"63\" y2=\"25\" />\n",
       "  <line x1=\"69\" y1=\"0\" x2=\"69\" y2=\"25\" />\n",
       "  <line x1=\"75\" y1=\"0\" x2=\"75\" y2=\"25\" />\n",
       "  <line x1=\"82\" y1=\"0\" x2=\"82\" y2=\"25\" />\n",
       "  <line x1=\"88\" y1=\"0\" x2=\"88\" y2=\"25\" />\n",
       "  <line x1=\"94\" y1=\"0\" x2=\"94\" y2=\"25\" />\n",
       "  <line x1=\"101\" y1=\"0\" x2=\"101\" y2=\"25\" />\n",
       "  <line x1=\"107\" y1=\"0\" x2=\"107\" y2=\"25\" />\n",
       "  <line x1=\"113\" y1=\"0\" x2=\"113\" y2=\"25\" />\n",
       "  <line x1=\"120\" y1=\"0\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"0.0,0.0 120.0,0.0 120.0,25.412616514582485 0.0,25.412616514582485\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"60.000000\" y=\"45.412617\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >500000</text>\n",
       "  <text x=\"140.000000\" y=\"12.706308\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(0,140.000000,12.706308)\">1</text>\n",
       "</svg>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "dask.array<mean_agg-aggregate, shape=(500000,), dtype=float64, chunksize=(1000,), chunktype=numpy.ndarray>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import dask.array as da\n",
    "x = da.random.random((500_000, 50000), chunks=(1000, 1000))\n",
    "y = x.mean(axis=1)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "dc52bf35-0bbb-4185-bfdf-eb01390f032d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.50043175, 0.50108079, 0.50329407, ..., 0.50033652, 0.49935371,\n",
       "       0.49858524])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# watch the Dask dashboard\n",
    "y.compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b8b69e9-86f1-456e-99b5-ed6612fb44c1",
   "metadata": {},
   "source": [
    "## Shutting down a cluster\n",
    "\n",
    "@@TODO explain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "e6c76197-4a22-44af-a6f8-1e8b6a176eb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.shutdown()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccabc7a5-a6d1-4c58-8ea6-1c1426890a8c",
   "metadata": {},
   "source": [
    "## Reconnecting to an existing cluster\n",
    "\n",
    "@@TODO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "78f4e159-0254-4db1-b111-a4da54f7b5cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ClusterReport<name=dev.600f7e441656459ca1327e7287db951b, status=RUNNING>]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gateway.list_clusters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d016613-eeca-47b4-9fae-f3a0cb355de3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:binder-v3.0.0]",
   "language": "python",
   "name": "conda-env-binder-v3.0.0-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}