{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Model Evaluation\n", "\n", "Applying machine learning in an applied science context is often method work. We build a prototype model and expect want to show that this method can be applied to our specific problem. This means that we have to guarantee that the insights we glean from this application generalize to new data from the same problem set.\n", "\n", "This is why we usually import `train_test_split()` from scikit-learn to get a validation set and a test set. But in my experience, in real-world applications, this isn’t always enough. In science, we usually deal with data that has some kind of correlation in some kind of dimension. Sometimes we have geospatial data and have to account for Tobler’s Law, i.e. things that are closer to each other matter more to each other than those data points at a larger distance. Sometimes we have temporal correlations, dealing with time series, where data points closer in time may influence each other.\n", "\n", "Not taking care of proper validation, will often lead to additional review cycles in a paper submission. It might lead to a rejection of the manuscript which is bad enough. In the worst case scenario, our research might report incorrect conclusions and have to be retracted. No one wants rejections or even retractions.\n", "\n", "So we’ll go into some methods to properly evaluate machine learning models even when our data is not “independent and identically distributed”." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:10.608399Z", "iopub.status.busy": "2022-12-13T01:42:10.607899Z", "iopub.status.idle": "2022-12-13T01:42:10.619565Z", "shell.execute_reply": "2022-12-13T01:42:10.619064Z" }, "tags": [] }, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "DATA_FOLDER = Path(\"..\", \"..\") / \"data\"\n", "DATA_FILEPATH = DATA_FOLDER / \"penguins_clean.csv\"" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:10.622066Z", "iopub.status.busy": "2022-12-13T01:42:10.621565Z", "iopub.status.idle": "2022-12-13T01:42:11.022636Z", "shell.execute_reply": "2022-12-13T01:42:11.022136Z" }, "lines_to_next_cell": 2, "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Culmen Length (mm)Culmen Depth (mm)Flipper Length (mm)SexSpecies
039.118.7181.0MALEAdelie Penguin (Pygoscelis adeliae)
139.517.4186.0FEMALEAdelie Penguin (Pygoscelis adeliae)
240.318.0195.0FEMALEAdelie Penguin (Pygoscelis adeliae)
336.719.3193.0FEMALEAdelie Penguin (Pygoscelis adeliae)
439.320.6190.0MALEAdelie Penguin (Pygoscelis adeliae)
\n", "
" ], "text/plain": [ " Culmen Length (mm) Culmen Depth (mm) Flipper Length (mm) Sex \\\n", "0 39.1 18.7 181.0 MALE \n", "1 39.5 17.4 186.0 FEMALE \n", "2 40.3 18.0 195.0 FEMALE \n", "3 36.7 19.3 193.0 FEMALE \n", "4 39.3 20.6 190.0 MALE \n", "\n", " Species \n", "0 Adelie Penguin (Pygoscelis adeliae) \n", "1 Adelie Penguin (Pygoscelis adeliae) \n", "2 Adelie Penguin (Pygoscelis adeliae) \n", "3 Adelie Penguin (Pygoscelis adeliae) \n", "4 Adelie Penguin (Pygoscelis adeliae) " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "penguins = pd.read_csv(DATA_FILEPATH)\n", "penguins.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Splitting \n", "The simplest method of splitting data into a training and test data set is `train_test_split()`, which randomly selects samples from our dataframe. \n", "\n", "This method essentially makes a very big assumption. That assumption being that our data is \"independent and identically distributed\" or i.i.d..\n", "\n", "That simply means that each measurement for a penguin does not depend on another measurement. Luckily for penguins that is mostly true. For other data? Not so much.\n", "And it means that we expect that we have a similar distribution of measurements of our penguins to the unseen data or future measurements.\n", "\n", "
\n", "Tip: The i.i.d. assumption lies at the core of most machine learning and is an important concept to dive into and understand.
" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:11.025637Z", "iopub.status.busy": "2022-12-13T01:42:11.025136Z", "iopub.status.idle": "2022-12-13T01:42:11.193166Z", "shell.execute_reply": "2022-12-13T01:42:11.192666Z" }, "tags": [] }, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'sklearn'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [3], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmodel_selection\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m train_test_split\n\u001b[0;32m 2\u001b[0m num_features \u001b[38;5;241m=\u001b[39m [\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCulmen Length (mm)\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCulmen Depth (mm)\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFlipper Length (mm)\u001b[39m\u001b[38;5;124m\"\u001b[39m]\n\u001b[0;32m 3\u001b[0m cat_features \u001b[38;5;241m=\u001b[39m [\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSex\u001b[39m\u001b[38;5;124m\"\u001b[39m]\n", "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sklearn'" ] } ], "source": [ "from sklearn.model_selection import train_test_split\n", "num_features = [\"Culmen Length (mm)\", \"Culmen Depth (mm)\", \"Flipper Length (mm)\"]\n", "cat_features = [\"Sex\"]\n", "features = num_features + cat_features\n", "target = [\"Species\"]\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(penguins[features], penguins[target], train_size=.7, random_state=42)\n", "X_train" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stratification \n", "Usually, our target class or another feature we use isn't distributed equally." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:11.195666Z", "iopub.status.busy": "2022-12-13T01:42:11.195666Z", "iopub.status.idle": "2022-12-13T01:42:11.596221Z", "shell.execute_reply": "2022-12-13T01:42:11.595720Z" } }, "outputs": [], "source": [ "from matplotlib import pyplot as plt" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:11.599721Z", "iopub.status.busy": "2022-12-13T01:42:11.599221Z", "iopub.status.idle": "2022-12-13T01:42:11.735745Z", "shell.execute_reply": "2022-12-13T01:42:11.735245Z" }, "tags": [] }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAALECAYAAAAii8t8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABjwklEQVR4nO3deXRN5+L/8c9JQgYyCE1IGyQoooagNHTQGmKooXRCa6ZVU2lVtYaixm8HlFI1u1pUaWlVaagxhiSEmocgRUJpEkmQ8feH1fPraXS6N8ne7PdrrbOWPHvn+Og9l0+e/exn23Jzc3MFAABgIk5GBwAAAPgjCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdF6MD/DdycnJ04cIFeXp6ymazGR0HAAD8A7m5ubp27ZoCAgLk5PTXcyR3ZEG5cOGCAgMDjY4BAAD+C/Hx8brvvvv+8pw7sqB4enpKuvUH9PLyMjgNAAD4J1JSUhQYGGj/d/yv3JEF5bfLOl5eXhQUAADuMP9keQaLZAEAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOlQUAAAgOm4GB3gblb+zW+NjnDXODOpldERAACFiBkUAABgOhQUAABgOhQUAABgOhQUAABgOhQUAABgOhQUAABgOhQUAABgOhQUAABgOhQUAABgOhQUAABgOv+6oGzdulWtW7dWQECAbDabvvrqqz899+WXX5bNZtPUqVMdxq9evarOnTvLy8tLPj4+6tmzp1JTU/9tFAAAcJf61wUlLS1NNWvW1MyZM//yvNWrV2vXrl0KCAjIc6xz5846dOiQNm7cqG+++UZbt25Vnz59/m0UAABwl/rXDwts0aKFWrRo8ZfnnD9/XgMGDND333+vVq0cH/J25MgRrV+/Xnv37lXdunUlSR999JFatmyp995777aF5ubNm7p586b965SUlH8bGwAA3EHyfQ1KTk6OXnzxRQ0dOlTVqlXLczwyMlI+Pj72ciJJTZo0kZOTk3bv3n3b95w4caK8vb3tr8DAwPyODQAATCTfC8rkyZPl4uKigQMH3vZ4QkKC/Pz8HMZcXFzk6+urhISE237P8OHDlZycbH/Fx8fnd2wAAGAi//oSz1+Jjo7WtGnTFBMTI5vNlm/v6+rqKldX13x7PwAAYG75OoOybds2Xbp0SWXLlpWLi4tcXFx09uxZvfbaaypfvrwkqXTp0rp06ZLD92VlZenq1asqXbp0fsYBAAB3qHydQXnxxRfVpEkTh7Hw8HC9+OKL6t69uyQpLCxMSUlJio6OVp06dSRJmzZtUk5OjurXr5+fcQAAwB3qXxeU1NRUnTx50v51XFyc9u/fL19fX5UtW1YlS5Z0OL9IkSIqXbq0KleuLEmqWrWqmjdvrt69e2v27NnKzMxU//799fzzz9/2Dh4AAGA9//oST1RUlEJDQxUaGipJGjJkiEJDQzVq1Kh//B5Lly5VlSpV1LhxY7Vs2VIPP/yw5syZ82+jAACAu9S/nkFp1KiRcnNz//H5Z86cyTPm6+urzz777N/+1gAAwCJ4Fg8AADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADCdf11Qtm7dqtatWysgIEA2m01fffWV/VhmZqaGDRum6tWrq1ixYgoICFCXLl104cIFh/e4evWqOnfuLC8vL/n4+Khnz55KTU39n/8wAADg7vCvC0paWppq1qypmTNn5jmWnp6umJgYjRw5UjExMVq1apWOHTumNm3aOJzXuXNnHTp0SBs3btQ333yjrVu3qk+fPv/9nwIAANxVbLm5ubn/9TfbbFq9erXatWv3p+fs3btX9erV09mzZ1W2bFkdOXJEISEh2rt3r+rWrStJWr9+vVq2bKmff/5ZAQEBf/v7pqSkyNvbW8nJyfLy8vpv4xe48m9+a3SEu8aZSa2MjgAA+B/9m3+/C3wNSnJysmw2m3x8fCRJkZGR8vHxsZcTSWrSpImcnJy0e/fu277HzZs3lZKS4vACAAB3L5eCfPMbN25o2LBh6tixo70pJSQkyM/PzzGEi4t8fX2VkJBw2/eZOHGixowZU5BRAQAGYbY5/9xNs80FNoOSmZmpZ599Vrm5uZo1a9b/9F7Dhw9XcnKy/RUfH59PKQEAgBkVyAzKb+Xk7Nmz2rRpk8N1ptKlS+vSpUsO52dlZenq1asqXbr0bd/P1dVVrq6uBREVAACYUL7PoPxWTk6cOKEffvhBJUuWdDgeFhampKQkRUdH28c2bdqknJwc1a9fP7/jAACAO9C/nkFJTU3VyZMn7V/HxcVp//798vX1VZkyZfT0008rJiZG33zzjbKzs+3rSnx9fVW0aFFVrVpVzZs3V+/evTV79mxlZmaqf//+ev755//RHTwAAODu968LSlRUlB5//HH710OGDJEkde3aVe+8847WrFkjSapVq5bD923evFmNGjWSJC1dulT9+/dX48aN5eTkpA4dOmj69On/5R8BAADcbf51QWnUqJH+auuUf7Ktiq+vrz777LN/+1sDAACL4Fk8AADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdP51Qdm6datat26tgIAA2Ww2ffXVVw7Hc3NzNWrUKJUpU0bu7u5q0qSJTpw44XDO1atX1blzZ3l5ecnHx0c9e/ZUamrq//QHAQAAd49/XVDS0tJUs2ZNzZw587bHp0yZounTp2v27NnavXu3ihUrpvDwcN24ccN+TufOnXXo0CFt3LhR33zzjbZu3ao+ffr8938KAABwV3H5t9/QokULtWjR4rbHcnNzNXXqVI0YMUJt27aVJC1evFj+/v766quv9Pzzz+vIkSNav3699u7dq7p160qSPvroI7Vs2VLvvfeeAgIC/oc/DgAAuBvk6xqUuLg4JSQkqEmTJvYxb29v1a9fX5GRkZKkyMhI+fj42MuJJDVp0kROTk7avXv3bd/35s2bSklJcXgBAIC7V74WlISEBEmSv7+/w7i/v7/9WEJCgvz8/ByOu7i4yNfX137OH02cOFHe3t72V2BgYH7GBgAAJnNH3MUzfPhwJScn21/x8fFGRwIAAAUoXwtK6dKlJUmJiYkO44mJifZjpUuX1qVLlxyOZ2Vl6erVq/Zz/sjV1VVeXl4OLwAAcPfK14ISFBSk0qVLKyIiwj6WkpKi3bt3KywsTJIUFhampKQkRUdH28/ZtGmTcnJyVL9+/fyMAwAA7lD/+i6e1NRUnTx50v51XFyc9u/fL19fX5UtW1avvvqq3n33XVWqVElBQUEaOXKkAgIC1K5dO0lS1apV1bx5c/Xu3VuzZ89WZmam+vfvr+eff547eAAAgKT/oqBERUXp8ccft389ZMgQSVLXrl21cOFCvfHGG0pLS1OfPn2UlJSkhx9+WOvXr5ebm5v9e5YuXar+/furcePGcnJyUocOHTR9+vR8+OMAAIC7gS03NzfX6BD/VkpKiry9vZWcnGzq9Sjl3/zW6Ah3jTOTWhkdAUAB4e/K/GP2vyv/zb/fd8RdPAAAwFooKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHQoKAAAwHTyvaBkZ2dr5MiRCgoKkru7uypUqKBx48YpNzfXfk5ubq5GjRqlMmXKyN3dXU2aNNGJEyfyOwoAALhD5XtBmTx5smbNmqUZM2boyJEjmjx5sqZMmaKPPvrIfs6UKVM0ffp0zZ49W7t371axYsUUHh6uGzdu5HccAABwB3LJ7zfcuXOn2rZtq1atWkmSypcvr88//1x79uyRdGv2ZOrUqRoxYoTatm0rSVq8eLH8/f311Vdf6fnnn8/vSAAA4A6T7zMoDRo0UEREhI4fPy5Jio2N1fbt29WiRQtJUlxcnBISEtSkSRP793h7e6t+/fqKjIy87XvevHlTKSkpDi8AAHD3yvcZlDfffFMpKSmqUqWKnJ2dlZ2drfHjx6tz586SpISEBEmSv7+/w/f5+/vbj/3RxIkTNWbMmPyOCgAATCrfZ1BWrFihpUuX6rPPPlNMTIwWLVqk9957T4sWLfqv33P48OFKTk62v+Lj4/MxMQAAMJt8n0EZOnSo3nzzTftakurVq+vs2bOaOHGiunbtqtKlS0uSEhMTVaZMGfv3JSYmqlatWrd9T1dXV7m6uuZ3VAAAYFL5PoOSnp4uJyfHt3V2dlZOTo4kKSgoSKVLl1ZERIT9eEpKinbv3q2wsLD8jgMAAO5A+T6D0rp1a40fP15ly5ZVtWrVtG/fPn3wwQfq0aOHJMlms+nVV1/Vu+++q0qVKikoKEgjR45UQECA2rVrl99xAADAHSjfC8pHH32kkSNH6pVXXtGlS5cUEBCgl156SaNGjbKf88YbbygtLU19+vRRUlKSHn74Ya1fv15ubm75HQcAANyBbLm/3+L1DpGSkiJvb28lJyfLy8vL6Dh/qvyb3xod4a5xZlIroyMAKCD8XZl/zP535b/595tn8QAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANOhoAAAANNxMToAgMJV/s1vjY5wVzgzqZXREYC7GjMoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdCgoAADAdAqkoJw/f14vvPCCSpYsKXd3d1WvXl1RUVH247m5uRo1apTKlCkjd3d3NWnSRCdOnCiIKAAA4A6U7wXl119/VcOGDVWkSBF99913Onz4sN5//32VKFHCfs6UKVM0ffp0zZ49W7t371axYsUUHh6uGzdu5HccAABwB3LJ7zecPHmyAgMDtWDBAvtYUFCQ/de5ubmaOnWqRowYobZt20qSFi9eLH9/f3311Vd6/vnn8zsSAAC4w+T7DMqaNWtUt25dPfPMM/Lz81NoaKg+/fRT+/G4uDglJCSoSZMm9jFvb2/Vr19fkZGRt33PmzdvKiUlxeEFAADuXvleUE6fPq1Zs2apUqVK+v7779W3b18NHDhQixYtkiQlJCRIkvz9/R2+z9/f337sjyZOnChvb2/7KzAwML9jAwAAE8n3gpKTk6PatWtrwoQJCg0NVZ8+fdS7d2/Nnj37v37P4cOHKzk52f6Kj4/Px8QAAMBs8r2glClTRiEhIQ5jVatW1blz5yRJpUuXliQlJiY6nJOYmGg/9keurq7y8vJyeAEAgLtXvheUhg0b6tixYw5jx48fV7ly5STdWjBbunRpRURE2I+npKRo9+7dCgsLy+84AADgDpTvd/EMHjxYDRo00IQJE/Tss89qz549mjNnjubMmSNJstlsevXVV/Xuu++qUqVKCgoK0siRIxUQEKB27drldxwAAHAHyveC8uCDD2r16tUaPny4xo4dq6CgIE2dOlWdO3e2n/PGG28oLS1Nffr0UVJSkh5++GGtX79ebm5u+R0HAADcgfK9oEjSk08+qSeffPJPj9tsNo0dO1Zjx44tiN8eAADc4XgWDwAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMJ0CLyiTJk2SzWbTq6++ah+7ceOG+vXrp5IlS6p48eLq0KGDEhMTCzoKAAC4QxRoQdm7d68++eQT1ahRw2F88ODBWrt2rb744gtt2bJFFy5cUPv27QsyCgAAuIMUWEFJTU1V586d9emnn6pEiRL28eTkZM2bN08ffPCBnnjiCdWpU0cLFizQzp07tWvXroKKAwAA7iAFVlD69eunVq1aqUmTJg7j0dHRyszMdBivUqWKypYtq8jIyNu+182bN5WSkuLwAgAAdy+XgnjTZcuWKSYmRnv37s1zLCEhQUWLFpWPj4/DuL+/vxISEm77fhMnTtSYMWMKIioAADChfJ9BiY+P16BBg7R06VK5ubnly3sOHz5cycnJ9ld8fHy+vC8AADCnfC8o0dHRunTpkmrXri0XFxe5uLhoy5Ytmj59ulxcXOTv76+MjAwlJSU5fF9iYqJKly592/d0dXWVl5eXwwsAANy98v0ST+PGjXXw4EGHse7du6tKlSoaNmyYAgMDVaRIEUVERKhDhw6SpGPHjuncuXMKCwvL7zgAAOAOlO8FxdPTUw888IDDWLFixVSyZEn7eM+ePTVkyBD5+vrKy8tLAwYMUFhYmB566KH8jgMAAO5ABbJI9u98+OGHcnJyUocOHXTz5k2Fh4fr448/NiIKAAAwoUIpKD/++KPD125ubpo5c6ZmzpxZGL89AAC4w/AsHgAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDoUFAAAYDr5XlAmTpyoBx98UJ6envLz81O7du107Ngxh3Nu3Lihfv36qWTJkipevLg6dOigxMTE/I4CAADuUPleULZs2aJ+/fpp165d2rhxozIzM9WsWTOlpaXZzxk8eLDWrl2rL774Qlu2bNGFCxfUvn37/I4CAADuUC75/Ybr1693+HrhwoXy8/NTdHS0Hn30USUnJ2vevHn67LPP9MQTT0iSFixYoKpVq2rXrl166KGH8jsSAAC4wxT4GpTk5GRJkq+vryQpOjpamZmZatKkif2cKlWqqGzZsoqMjLzte9y8eVMpKSkOLwAAcPcq0IKSk5OjV199VQ0bNtQDDzwgSUpISFDRokXl4+PjcK6/v78SEhJu+z4TJ06Ut7e3/RUYGFiQsQEAgMEKtKD069dPP/30k5YtW/Y/vc/w4cOVnJxsf8XHx+dTQgAAYEb5vgblN/3799c333yjrVu36r777rOPly5dWhkZGUpKSnKYRUlMTFTp0qVv+16urq5ydXUtqKgAAMBk8n0GJTc3V/3799fq1au1adMmBQUFORyvU6eOihQpooiICPvYsWPHdO7cOYWFheV3HAAAcAfK9xmUfv366bPPPtPXX38tT09P+7oSb29vubu7y9vbWz179tSQIUPk6+srLy8vDRgwQGFhYdzBAwAAJBVAQZk1a5YkqVGjRg7jCxYsULdu3SRJH374oZycnNShQwfdvHlT4eHh+vjjj/M7CgAAuEPle0HJzc3923Pc3Nw0c+ZMzZw5M79/ewAAcBfgWTwAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0KCgAAMB0DC0oM2fOVPny5eXm5qb69etrz549RsYBAAAmYVhBWb58uYYMGaLRo0crJiZGNWvWVHh4uC5dumRUJAAAYBKGFZQPPvhAvXv3Vvfu3RUSEqLZs2fLw8ND8+fPNyoSAAAwCRcjftOMjAxFR0dr+PDh9jEnJyc1adJEkZGRec6/efOmbt68af86OTlZkpSSklLwYf8HOTfTjY5w1zD7/9Z3Ej6X+YPPZP7hM5l/zP65/C1fbm7u355rSEH55ZdflJ2dLX9/f4dxf39/HT16NM/5EydO1JgxY/KMBwYGFlhGmIv3VKMTAI74TMKM7pTP5bVr1+Tt7f2X5xhSUP6t4cOHa8iQIfavc3JydPXqVZUsWVI2m83AZHe+lJQUBQYGKj4+Xl5eXkbHAfhMwnT4TOaf3NxcXbt2TQEBAX97riEFpVSpUnJ2dlZiYqLDeGJiokqXLp3nfFdXV7m6ujqM+fj4FGREy/Hy8uL/eDAVPpMwGz6T+ePvZk5+Y8gi2aJFi6pOnTqKiIiwj+Xk5CgiIkJhYWFGRAIAACZi2CWeIUOGqGvXrqpbt67q1aunqVOnKi0tTd27dzcqEgAAMAnDCspzzz2ny5cva9SoUUpISFCtWrW0fv36PAtnUbBcXV01evToPJfQAKPwmYTZ8Jk0hi33n9zrAwAAUIh4Fg8AADAdCgoAADAdCgoAADAdCgoAADAdCgoAADCdO2KrexSMmzdvctscDBcXF6dt27bp7NmzSk9P1z333KPQ0FCFhYXJzc3N6HiwID6T5kBBsZDvvvtOy5Yt07Zt2xQfH6+cnBwVK1ZMoaGhatasmbp37/6Pno8A5IelS5dq2rRpioqKkr+/vwICAuTu7q6rV6/q1KlTcnNzU+fOnTVs2DCVK1fO6LiwAD6T5sI+KBawevVqDRs2TNeuXVPLli1Vr149h//j/fTTT9q2bZsiIyPVrVs3jRs3Tvfcc4/RsXEXCw0NVdGiRdW1a1e1bt06z5PJb968qcjISC1btkxffvmlPv74Yz3zzDMGpYUV8Jk0HwqKBYSFhWnEiBFq0aKFnJz+fNnR+fPn9dFHH8nf31+DBw8uxISwmu+//17h4eH/6NwrV67ozJkzqlOnTgGngpXxmTQfCgoAADAd1qBYVEZGhuLi4lShQgW5uPAxgDncuHFDGRkZDmM83h5G4jNpHG4ztpj09HT17NlTHh4eqlatms6dOydJGjBggCZNmmRwOlhRenq6+vfvLz8/PxUrVkwlSpRweAGFjc+kOVBQLGb48OGKjY3Vjz/+6HC7XJMmTbR8+XIDk8Gqhg4dqk2bNmnWrFlydXXV3LlzNWbMGAUEBGjx4sVGx4MF8Zk0B9agWEy5cuW0fPlyPfTQQ/L09FRsbKyCg4N18uRJ1a5dWykpKUZHhMWULVtWixcvVqNGjeTl5aWYmBhVrFhRS5Ys0eeff65169YZHREWw2fSHJhBsZjLly/Lz88vz3haWppsNpsBiWB1V69eVXBwsKRb1/avXr0qSXr44Ye1detWI6PBovhMmgMFxWLq1q2rb7/91v71b6Vk7ty5CgsLMyoWLCw4OFhxcXGSpCpVqmjFihWSpLVr18rHx8fAZLAqPpPmwO0bFjNhwgS1aNFChw8fVlZWlqZNm6bDhw9r586d2rJli9HxYEHdu3dXbGysHnvsMb355ptq3bq1ZsyYoczMTH3wwQdGx4MF8Zk0B9agWNCpU6c0adIkxcbGKjU1VbVr19awYcNUvXp1o6MBOnv2rKKjo1WxYkXVqFHD6DgAn0mDUFAAAIDpcInHwtiACGYwcOBAVaxYUQMHDnQYnzFjhk6ePKmpU6caEwyWNXbs2L88PmrUqEJKYm3MoFhMenq63njjDa1YsUJXrlzJczw7O9uAVLCye++9V2vWrMnzXJOYmBi1adNGP//8s0HJYFWhoaEOX2dmZiouLk4uLi6qUKGCYmJiDEpmLcygWMzQoUO1efNmzZo1Sy+++KJmzpyp8+fP65NPPmEnWRjiypUr8vb2zjPu5eWlX375xYBEsLp9+/blGUtJSVG3bt301FNPGZDImrjN2GLWrl2rjz/+WB06dJCLi4seeeQRjRgxQhMmTNDSpUuNjgcLqlixotavX59n/LvvvrPvRQEYzcvLS2PGjNHIkSONjmIZzKBYzF9tQNS3b18jo8GihgwZov79++vy5ct64oknJEkRERF6//33WX8CU0lOTlZycrLRMSyDgmIxv21AVLZsWfsGRPXq1WMDIhimR48eunnzpsaPH69x48ZJksqXL69Zs2apS5cuBqeDFU2fPt3h69zcXF28eFFLlixRixYtDEplPSyStZgPP/xQzs7OGjhwoH744Qe1bt1aubm59g2IBg0aZHREWNjly5fl7u6u4sWLGx0FFhYUFOTwtZOTk+655x498cQTGj58uDw9PQ1KZi0UFItjAyIAgBlRUAAUutq1aysiIkIlSpRQaGjoXz6okls6YaT4+HhJUmBgoMFJrIc1KBYwffp09enTR25ubnmurf7RHzfLAgpC27Zt5erqav81T9KGmWRlZWnMmDGaPn26UlNTJUnFixfXgAEDNHr0aBUpUsTghNbADIoFBAUFKSoqSiVLlsxzbfX3bDabTp8+XYjJAMB8+vbtq1WrVmns2LH2p7xHRkbqnXfeUbt27TRr1iyDE1oDBQWAoYKDg7V3716VLFnSYTwpKUm1a9emNKPQeXt7a9myZXnu2Fm3bp06duzIrcaFhI3aABjqzJkzt33Ews2bN9nmHoZwdXVV+fLl84wHBQWpaNGihR/IoliDYgFDhgz5x+d+8MEHBZgE+P/WrFlj//X333/vsN19dna2IiIi/vKSJFBQ+vfvr3HjxmnBggX2tVK/7dXTv39/g9NZB5d4LODxxx//R+fZbDZt2rSpgNMAtzg53ZrAtdls+uNfQ0WKFFH58uX1/vvv68knnzQiHizsqaeeUkREhFxdXVWzZk1JUmxsrDIyMtS4cWOHc1etWmVEREtgBsUCNm/ebHQEII+cnBxJt6bN9+7dq1KlShmcCLjFx8dHHTp0cBjjNuPCxwyKRZ08eVKnTp3So48+Knd3d+Xm5nKrJwDANFgkazFXrlxR48aNdf/996tly5a6ePGiJKlnz5567bXXDE4HKxo4cOBt9+eZMWOGXn311cIPBMAUKCgWM3jwYBUpUkTnzp2Th4eHffy555677SPvgYL25ZdfqmHDhnnGGzRooJUrVxqQCJBWrlypZ599Vg899JBq167t8ELhoKBYzIYNGzR58mTdd999DuOVKlXS2bNnDUoFK7ty5YrDHTy/8fLy0i+//GJAIljd9OnT1b17d/n7+2vfvn2qV6+eSpYsqdOnT/M040JEQbGYtLQ0h5mT31y9etV+Ox1QmCpWrHjb2bvvvvtOwcHBBiSC1X388ceaM2eOPvroIxUtWlRvvPGGNm7cqIEDB7JJWyHiLh6LeeSRR7R48WKNGzdO0q1bPHNycjRlypR/fDsykJ+GDBmi/v376/Lly3riiSckSREREXr//fc1depUY8PBks6dO6cGDRpIktzd3XXt2jVJ0osvvqiHHnpIM2bMMDKeZVBQLGbKlClq3LixoqKilJGRoTfeeEOHDh3S1atXtWPHDqPjwYJ69Ohh3wTrt+Jcvnx5zZo1S126dDE4HayodOnSunr1qsqVK6eyZctq165dqlmzpuLi4vLs2YOCwyUei3nggQd0/PhxPfzww2rbtq3S0tLUvn177du3TxUqVDA6HiwmKytLixcvVvv27fXzzz8rMTFRKSkpOn36NOUEhnniiSfsOx13795dgwcPVtOmTfXcc8/pqaeeMjiddbAPCgBDeXh46MiRIypXrpzRUQBJtzYRzMnJkYvLrYsMy5Yt086dO1WpUiW99NJLPI+nkFBQLODAgQP/+NwaNWoUYBIgr0aNGunVV19Vu3btjI4CwERYg2IBtWrVsj/v5Pe7xf7WTX8/drunygIF6ZVXXtFrr72mn3/+WXXq1FGxYsUcjlOaYYRff/1V8+bN05EjRyRJISEh6t69u3x9fQ1OZh3MoFjA7/c32bdvn15//XUNHTpUYWFhkqTIyEi9//77mjJlCj/FotD99tDA3/t9oaY0o7Bt3bpVbdq0kZeXl+rWrStJio6OVlJSktauXatHH33U4ITWQEGxmHr16umdd95Ry5YtHcbXrVunkSNHKjo62qBksKq/2yCQtSkobNWrV1dYWJhmzZolZ2dnSbdml1955RXt3LlTBw8eNDihNVBQLMbd3V0xMTGqWrWqw/iRI0dUu3ZtXb9+3aBkAGAO7u7u2r9/vypXruwwfuzYMdWqVYu/JwsJa1AspmrVqpo4caLmzp1rX4mekZGhiRMn5iktQGE6fPiwzp07p4yMDIfxNm3aGJQIVlW7dm0dOXIkT0E5cuSIatasaVAq66GgWMzs2bPVunVr3XffffbFhwcOHJDNZtPatWsNTgcrOn36tJ566ikdPHjQvvZE+v+Lt1mDgsI2cOBADRo0SCdPntRDDz0kSdq1a5dmzpypSZMmOdwZySLugsMlHgtKS0vT0qVLdfToUUm3ZlU6deqU5+4JoDC0bt1azs7Omjt3roKCgrRnzx5duXJFr732mt577z098sgjRkeExdxu4fbvsYi7cFBQABiqVKlS2rRpk2rUqCFvb2/t2bNHlStX1qZNm/Taa69p3759RkeExfybJ7uziLvgcInHorjeD7PIzs6Wp6enpFtl5cKFC6pcubLKlSunY8eOGZwOVkTpMAcKisVwvR9m88ADDyg2NlZBQUGqX7++pkyZoqJFi2rOnDkKDg42Oh4Ag/CwQIsZNGiQgoKCdOnSJXl4eOjQoUPaunWr6tatqx9//NHoeLCgESNGKCcnR5I0duxYxcXF6ZFHHtG6des0bdo0g9MBMAprUCyG6/24E1y9elUlSpRweAwDAGthBsVibne9XxLX+2GYHj166Nq1aw5jvr6+Sk9PV48ePQxKBcBoFBSL+e16vyT79f4dO3Zo7NixXO+HIRYtWnTbnTmvX7+uxYsXG5AIVhcfH6+ff/7Z/vWePXv06quvas6cOQamsh4KisX81fX+6dOnG5wOVpKSkqLk5GTl5ubq2rVrSklJsb9+/fVXrVu3Tn5+fkbHhAV16tRJmzdvliQlJCSoadOm2rNnj95++22NHTvW4HTWwRoUcL0fhnBycvrLz5zNZtOYMWP09ttvF2IqQCpRooR27dqlypUra/r06Vq+fLl27NihDRs26OWXX9bp06eNjmgJ3GYM+fr6Gh0BFrR582bl5ubqiSee0JdffunwOSxatKjKlSungIAAAxPCqjIzM+Xq6ipJ+uGHH+z7Q1WpUkUXL140MpqlUFAAGOKxxx6TJMXFxSkwMPBvtxcHCku1atU0e/ZstWrVShs3btS4ceMkSRcuXFDJkiUNTmcdXOIBYLikpCTt2bNHly5dsq+R+k2XLl0MSgWr+vHHH/XUU08pJSVFXbt21fz58yVJb731lo4ePapVq1YZnNAaKCgADLV27Vp17txZqamp8vLycliXYrPZdPXqVQPTwaqys7OVkpKiEiVK2MfOnDkjDw8PFm8XEgoKAEPdf//9atmypSZMmCAPDw+j4wAwCQqKxSxatEilSpVSq1atJElvvPGG5syZo5CQEH3++ec8JAuFrlixYjp48CD78MBQtWvXVkREhEqUKKHQ0NC/vMMsJiamEJNZF4tkLWbChAmaNWuWJCkyMlIzZ87Uhx9+qG+++UaDBw/m2ioKXXh4uKKioigoMFTbtm3td+60a9fO2DCQxAyK5Xh4eOjo0aMqW7ashg0bposXL2rx4sU6dOiQGjVqpMuXLxsdERYzb948jR07Vt27d1f16tVVpEgRh+O/3eIJwFqYQbGY4sWL68qVKypbtqw2bNigIUOGSJLc3Nxuu904UNB69+4tSbfdodNmsyk7O7uwIwEwAQqKxTRt2lS9evVSaGiojh8/rpYtW0qSDh06pPLlyxsbDpb0x9uKASP8m920ubOscFBQLGbmzJkaMWKE4uPj9eWXX9o3HYqOjlbHjh0NTgcAxpg6darREfAHrEEBYLi0tDRt2bJF586dU0ZGhsOxgQMHGpQKgJEoKBZw4MABPfDAA3JyctKBAwf+8twaNWoUUirgln379qlly5ZKT09XWlqafH199csvv9g3xOLBbDDCqVOntGDBAp06dUrTpk2Tn5+fvvvuO5UtW1bVqlUzOp4lUFAswMnJSQkJCfLz87M/Qfb3/7P/9jULEmGERo0a6f7779fs2bPl7e2t2NhYFSlSRC+88IIGDRqk9u3bGx0RFrNlyxa1aNFCDRs21NatW3XkyBEFBwdr0qRJioqK0sqVK42OaAkUFAs4e/asypYtK5vNprNnz/7luWzUhsLm4+Oj3bt3q3LlyvLx8VFkZKSqVq2q3bt3q2vXrjp69KjREWExYWFheuaZZzRkyBB5enoqNjZWwcHB2rNnj9q3b6+ff/7Z6IiWwCJZC/h96aCAwGyKFClif5Kxn5+fzp07p6pVq8rb21vx8fEGp4MVHTx4UJ999lmecT8/P/3yyy8GJLImCooFrFmz5h+fy6ZYKGyhoaHau3evKlWqpMcee0yjRo3SL7/8oiVLluiBBx4wOh4syMfHRxcvXlRQUJDD+L59+3TvvfcalMp6uMRjAb/9dPp3WIMCI0RFRenatWt6/PHHdenSJXXp0kU7d+5UpUqVNH/+fNWsWdPoiLCY119/Xbt379YXX3yh+++/XzExMUpMTFSXLl3UpUsXjR492uiIlkBBAQDgdzIyMtSvXz8tXLhQ2dnZcnFxUXZ2tjp16qSFCxfK2dnZ6IiWQEGxsBs3bsjNzc3oGABgSvHx8Tp48KBSU1MVGhqqSpUqGR3JUigoFpOdna0JEyZo9uzZSkxM1PHjxxUcHKyRI0eqfPny6tmzp9ERAQDQP1ucgLvG+PHjtXDhQk2ZMkVFixa1jz/wwAOaO3eugckAwBw6dOigyZMn5xmfMmWKnnnmGQMSWRMFxWIWL16sOXPmqHPnzg7XUWvWrMl+EwAgaevWrfYHqf5eixYttHXrVgMSWRMFxWLOnz+vihUr5hnPyclRZmamAYmAvJKSkoyOAAtLTU11mGH+TZEiRZSSkmJAImuioFhMSEiItm3blmd85cqVCg0NNSARrG7y5Mlavny5/etnn31WJUuW1L333qvY2FgDk8Gqqlev7vCZ/M2yZcsUEhJiQCJrYqM2ixk1apS6du2q8+fPKycnR6tWrdKxY8e0ePFiffPNN0bHgwXNnj1bS5culSRt3LhRGzdu1HfffacVK1Zo6NCh2rBhg8EJYTUjR45U+/btderUKT3xxBOSpIiICH3++ef64osvDE5nHdzFY0Hbtm3T2LFjFRsbq9TUVNWuXVujRo1Ss2bNjI4GC3J3d9fx48cVGBioQYMG6caNG/rkk090/Phx1a9fX7/++qvREWFB3377rSZMmKD9+/fL3d1dNWrU0OjRo/XYY48ZHc0yKCgADBUQEKCVK1eqQYMGqly5st59910988wzOnbsmB588EGu+QMWxSUei9m7d69ycnJUv359h/Hdu3fL2dlZdevWNSgZrKp9+/bq1KmTKlWqpCtXrqhFixaSbj335HYLuoGCFh8fL5vNpvvuu0+StGfPHn322WcKCQlRnz59DE5nHSyStZh+/frd9gmx58+fV79+/QxIBKv78MMP1b9/f4WEhGjjxo0qXry4JOnixYt65ZVXDE4HK+rUqZM2b94sSUpISFCTJk20Z88evf322xo7dqzB6ayDSzwWU7x4cR04cEDBwcEO43FxcapRo4auXbtmUDIAMIcSJUpo165dqly5sqZPn67ly5drx44d2rBhg15++WWdPn3a6IiWwCUei3F1dVViYmKegnLx4kW5uPBxQOFYs2aNWrRooSJFimjNmjV/eW6bNm0KKRVwS2ZmplxdXSVJP/zwg/0zWKVKFV28eNHIaJbCDIrFdOzYURcvXtTXX38tb29vSbc2xWrXrp38/Py0YsUKgxPCCpycnJSQkCA/Pz85Of35lWabzabs7OxCTAZI9evX1+OPP65WrVqpWbNm2rVrl2rWrKldu3bp6aef1s8//2x0REugoFjM+fPn9eijj+rKlSv2jdn2798vf39/bdy4UYGBgQYnBABj/fjjj3rqqaeUkpKirl27av78+ZKkt956S0ePHtWqVasMTmgNFBQLSktL09KlSxUbG2u/v79jx44qUqSI0dEAwBSys7OVkpKiEiVK2MfOnDkjDw8P+fn5GZjMOigoAArd9OnT//G5AwcOLMAkwJ+7fPmyjh07JkmqXLmy7rnnHoMTWQsFxWIWLVqkUqVKqVWrVpKkN954Q3PmzFFISIg+//xzlStXzuCEsIKgoKB/dJ7NZuOOCRS6tLQ0DRgwQIsXL1ZOTo4kydnZWV26dNFHH30kDw8PgxNaAwXFYipXrqxZs2bpiSeeUGRkpBo3bqypU6fqm2++kYuLC9dWAVjeSy+9pB9++EEzZsxQw4YNJUnbt2/XwIED1bRpU82aNcvghNZAQbEYDw8PHT16VGXLltWwYcN08eJFLV68WIcOHVKjRo10+fJloyPCojIyMhQXF6cKFSpwyzsMVapUKa1cuVKNGjVyGN+8ebOeffZZ/p4sJOwkazHFixfXlStXJEkbNmxQ06ZNJUlubm66fv26kdFgUenp6erZs6c8PDxUrVo1nTt3TpI0YMAATZo0yeB0sKL09HT5+/vnGffz81N6eroBiayJgmIxTZs2Va9evdSrVy8dP35cLVu2lCQdOnRI5cuXNzYcLGn48OGKjY3Vjz/+KDc3N/t4kyZNtHz5cgOTwarCwsI0evRo3bhxwz52/fp1jRkzRmFhYQYmsxbmUS1m5syZGjFihOLj4/Xll1+qZMmSkqTo6Gh17NjR4HSwoq+++krLly/XQw89JJvNZh+vVq2aTp06ZWAyWNW0adMUHh6u++67TzVr1pQkxcbGys3NTd9//73B6ayDNSgADOXh4aGffvpJwcHB8vT0VGxsrIKDgxUbG6tHH31UycnJRkeEBaWnp2vp0qU6evSoJKlq1arq3Lmz3N3dDU5mHcygWFBSUpLmzZunI0eOSLr1k2qPHj3sW98Dhalu3br69ttvNWDAAEmyz6LMnTuX6XQYxsPDQ7179zY6hqUxg2IxUVFRCg8Pl7u7u+rVqydJ2rt3r65fv64NGzaodu3aBieE1Wzfvl0tWrTQCy+8oIULF+qll17S4cOHtXPnTm3ZskV16tQxOiIs5s8eYGmz2eTm5qaKFSv+47188N+joFjMI488oooVK+rTTz+138qZlZWlXr166fTp09q6davBCWFFp06d0qRJkxQbG6vU1FTVrl1bw4YNU/Xq1Y2OBgtycnKSzWbTH/95/G3MZrPp4Ycf1ldffeWwFT7yFwXFYtzd3bVv3z5VqVLFYfzw4cOqW7cut9ABsLyIiAi9/fbbGj9+vH2mec+ePRo5cqRGjBghb29vvfTSS6pfv77mzZtncNq7F2tQLMbLy0vnzp3LU1Di4+Pl6elpUCpY2bp16+Ts7Kzw8HCH8e+//145OTlq0aKFQclgVYMGDdKcOXPUoEED+1jjxo3l5uamPn366NChQ5o6dap69OhhYMq7H/ugWMxzzz2nnj17avny5YqPj1d8fLyWLVumXr16cZsxDPHmm28qOzs7z3hubq7efPNNAxLB6k6dOiUvL688415eXvZnQ1WqVEm//PJLYUezFGZQLOa9996TzWZTly5dlJWVJUkqUqSI+vbty66dMMSJEycUEhKSZ7xKlSo6efKkAYlgdXXq1NHQoUO1ePFi+xOML1++rDfeeEMPPvigpFuf28DAQCNj3vUoKBZTtGhRTZs2TRMnTrRvglWhQgWezgnDeHt76/Tp03l2Mj558qSKFStmTChY2rx589S2bVvdd9999hISHx+v4OBgff3115Kk1NRUjRgxwsiYdz0WyQIw1EsvvaTIyEitXr1aFSpUkHSrnHTo0EEPPvig5s6da3BCWFFOTo42bNig48ePS7r1JPimTZvKyYmVEYWFgmIxTz31lMN24r/5/f39nTp1UuXKlQ1IBytKTk5W8+bNFRUVpfvuu0+S9PPPP+uRRx7RqlWr5OPjY2xAWM7p06cVHBxsdAzLo6BYTLdu3fTVV1/Jx8fHvgFWTEyMkpKS1KxZM8XGxurMmTOKiIhQw4YNDU4Lq8jNzdXGjRsVGxsrd3d31ahRQ48++qjRsWBRTk5Oeuyxx9SzZ089/fTTDg+xROGhoFjMm2++qZSUFM2YMcM+VZmTk6NBgwbJ09NT48eP18svv6xDhw5p+/btBqeFVSUlJTFzAsPs379fCxYs0Oeff66MjAw999xz6tGjh+rXr290NEuhoFjMPffcox07duj+++93GD9+/LgaNGigX375RQcPHtQjjzyipKQkY0LCUiZPnqzy5cvrueeekyQ9++yz+vLLL1W6dGmtW7fO/jRZoLBlZWVpzZo1WrhwodavX6/7779fPXr00Isvvmi/uwcFh9U+FpOVlWV/OufvHT161L4XhZub223XqQAFYfbs2fY7JTZu3KiNGzfqu+++U4sWLTR06FCD08HKXFxc1L59e33xxReaPHmyTp48qddff12BgYHq0qWLLl68aHTEuxq3GVvMiy++qJ49e+qtt96y38+/d+9eTZgwQV26dJEkbdmyRdWqVTMyJiwkISHBXlC++eYbPfvss2rWrJnKly/PlDoMFRUVpfnz52vZsmUqVqyYXn/9dfXs2VM///yzxowZo7Zt22rPnj1Gx7xrUVAs5sMPP5S/v7+mTJmixMRESZK/v78GDx6sYcOGSZKaNWum5s2bGxkTFlKiRAnFx8crMDBQ69ev17vvvivp1sLZ2+0wCxS0Dz74QAsWLNCxY8fUsmVLLV68WC1btrSv2wsKCtLChQvz7N2D/MUaFAtLSUmRpNtu6QwUlv79++ubb75RpUqVtG/fPp05c0bFixfXsmXLNGXKFMXExBgdERZTqVIl9ejRQ926dVOZMmVue05GRoY+//xzde3atZDTWQcFxWJGjx6tHj16qFy5ckZHASRJmZmZmjZtmuLj49WtWzeFhoZKujXb5+npqV69ehmcEIARKCgWU6tWLf3000/2e/w7dOggV1dXo2MBgOHS0tL0+uuva82aNcrIyFDjxo310UcfcceOQSgoFrRv3z77Pf5ZWVl6/vnn1aNHD/uiWaCwnTp1SlOnTtWRI0ckSSEhIXr11VfZzROFasiQIZozZ446d+4sNzc3ff7552rYsKFWr15tdDRLoqBYWGZmptauXasFCxbo+++/V5UqVdSzZ09169ZN3t7eRseDRXz//fdq06aNatWqZd+9eMeOHYqNjdXatWvVtGlTgxPCKoKCgjRlyhQ988wzkqTo6Gg99NBDun79ulxcuKeksFFQLCwjI0OrV6/W/PnztWnTJjVo0EAXLlxQYmKiPv30U/vGWUBBCg0NVXh4uCZNmuQw/uabb2rDhg0skkWhKVKkiM6ePauAgAD7mIeHh44ePaqyZcsamMya2KjNgqKjo9W/f3+VKVNGgwcPVmhoqI4cOaItW7boxIkTGj9+vAYOHGh0TFjEkSNH1LNnzzzjPXr00OHDhw1IBKvKyclRkSJFHMZcXFy43d0gzFlZTPXq1XX06FE1a9ZM8+bNU+vWreXs7OxwTseOHTVo0CCDEsJq7rnnHu3fv1+VKlVyGN+/f7/8/PwMSgUrys3NVePGjR0u56Snp6t169YqWrSofYxZvcJBQbGYZ599Vj169NC99977p+eUKlVKOTk5hZgKVta7d2/16dNHp0+fVoMGDSTdWoMyefJkDRkyxOB0sJLRo0fnGWvbtq0BSSCxBsVSUlJStHv3bmVkZKhevXrcOgdTyM3N1dSpU/X+++/rwoULkqSAgAANHTpUAwcO5LlQgEVRUCxi//79atmypRITE5WbmytPT0+tWLFC4eHhRkcD7K5duyZJ8vT0NDgJAKOxSNYihg0bpqCgIG3fvl3R0dFq3Lix+vfvb3QswIGnpyflBIZo3ry5du3a9bfnXbt2TZMnT9bMmTMLIZW1MYNiEaVKldKGDRtUu3ZtSVJSUpJ8fX2VlJTEs3hgqNDQ0NtexrHZbHJzc1PFihXVrVs3Pf744wakg1XMmzdPo0aNkre3t1q3bq26desqICBAbm5u+vXXX3X48GFt375d69atU6tWrfR///d/3HpcwCgoFuHk5KSEhASHuyI8PT114MABBQUFGZgMVjd8+HDNmjVL1atXV7169SRJe/fu1YEDB9StWzcdPnxYERERWrVqFQsWUaBu3rypL774QsuXL9f27duVnJws6VZZDgkJUXh4uHr27KmqVasanNQaKCgW4eTkpE2bNsnX19c+1qBBA61YsUL33XeffaxGjRpGxIOF9e7dW2XLltXIkSMdxt99912dPXtWn376qUaPHq1vv/1WUVFRBqWEFSUnJ+v69esqWbJknv1RUPAoKBbh5OQkm82m2/3P/du4zWZjQyIUOm9vb0VHR6tixYoO4ydPnlSdOnWUnJyso0eP6sEHH7QvogVw92MfFIuIi4szOgJwW25ubtq5c2eegrJz5065ublJurXD52+/BmANFBSLKFeunNERgNsaMGCAXn75ZUVHR9ufqL13717NnTtXb731lqRbDxSsVauWgSkBFDYu8VjAuXPn/tVq8/Pnz//lTrNAflu6dKlmzJihY8eOSZIqV66sAQMGqFOnTpKk69ev2+/qAWANFBQL8Pf3V7t27dSrVy/7T6h/lJycrBUrVmjatGnq06cPDwsEABiKSzwWcPjwYY0fP15NmzaVm5ub6tSpk+f+/kOHDql27dqaMmWKWrZsaXRkWEjXrl3Vs2dPPfroo0ZHARxkZGTo0qVLeZ5Nxv4nhYMZFAu5fv26vv32W23fvl1nz57V9evXVapUKYWGhio8PFwPPPCA0RFhQe3atdO6detUrlw5de/eXV27duUSIwx14sQJ9ejRQzt37nQY527HwkVBAWC4y5cva8mSJVq0aJEOHz6sJk2aqGfPnmrbti37T6DQNWzYUC4uLnrzzTdVpkyZPDsd16xZ06Bk1kJBAWAqMTExWrBggebOnavixYvrhRde0CuvvKJKlSoZHQ0WUaxYMUVHR6tKlSpGR7E0HhYIwDQuXryojRs3auPGjXJ2dlbLli118OBBhYSE6MMPPzQ6HiwiJCREv/zyi9ExLI8ZFACGyszM1Jo1a7RgwQJt2LBBNWrUUK9evdSpUyf7gyxXr16tHj166NdffzU4Laxg06ZNGjFihCZMmKDq1avnuczIA1YLBwUFgKFKlSqlnJwcdezYUb17977thmxJSUkKDQ1lR2QUCienWxcX/rj2hEWyhYuCAsBQS5Ys0TPPPMMmbDCNLVu2/OXxxx57rJCSWBsFxYJOnDihzZs33/b+/lGjRhmUClZ05swZbdy4UZmZmXrsscdUrVo1oyMBMAkKisV8+umn6tu3r0qVKqXSpUs7TGHabDbFxMQYmA5WsnnzZj355JO6fv26JMnFxUXz58/XCy+8YHAyWNGBAwf0wAMPyMnJSQcOHPjLc2vUqFFIqayNgmIx5cqV0yuvvKJhw4YZHQUW9/DDD6tUqVKaNWuW3NzcNGLECK1evVoXLlwwOhosyMnJSQkJCfLz85OTk5NsNptu988ja1AKDwXFYry8vLR//34FBwcbHQUW5+Pjo507dyokJESSlJ6eLi8vLyUmJqpkyZIGp4PVnD17VmXLlpXNZtPZs2f/8lyeDl84KCgW07NnTz344IN6+eWXjY4Ci/v9T6y/8fT0VGxsLAUaAA8LtJqKFStq5MiR2rVr123v7+cpxihM33//vby9ve1f5+TkKCIiQj/99JN9rE2bNkZEg4UtXrz4L4936dKlkJJYGzMoFhMUFPSnx2w2m06fPl2IaWBlv+018Ve43g8jlChRwuHrzMxMpaenq2jRovLw8NDVq1cNSmYtzKBYDBtdwSz+eIs7YBa327H4xIkT6tu3r4YOHWpAImtiBgUAgH8gKipKL7zwgo4ePWp0FEtgBsUChgwZonHjxqlYsWIaMmTIX577wQcfFFIqWNmuXbv00EMP/aNz09PTFRcXxyZuMJyLiwu3wRciCooF7Nu3T5mZmfZf/5k/PncCKCgvvviigoOD1atXL7Vs2VLFihXLc87hw4f1n//8RwsWLNDkyZMpKCg0a9ascfg6NzdXFy9e1IwZM9SwYUODUlkPl3gAFLrMzEzNmjVLM2fO1OnTp3X//fcrICBAbm5u+vXXX3X06FGlpqbqqaee0ltvvaXq1asbHRkW8scF3DabTffcc4+eeOIJvf/++ypTpoxByayFggLAUFFRUdq+fbvOnj2r69evq1SpUgoNDdXjjz8uX19fo+MBMAgFxWIef/zxv7yUs2nTpkJMAwDA7bEGxWJq1arl8HVmZqb279+vn376SV27djUmFACYyJ/dTGCz2eTm5qaKFSuqbdu2zPAVMGZQIEl65513lJqaqvfee8/oKABgqMcff1wxMTHKzs5W5cqVJUnHjx+Xs7OzqlSpomPHjslms2n79u32Z0kh/1FQIEk6efKk6tWrxw6JACxv6tSp2rZtmxYsWCAvLy9JUnJysnr16qWHH35YvXv3VqdOnXT9+nV9//33Bqe9e1FQIElasmSJhg0bxj3+ACzv3nvv1caNG/PMjhw6dEjNmjXT+fPnFRMTo2bNmumXX34xKOXdjzUoFtO+fXuHr3+7vz8qKkojR440KBUAmEdycrIuXbqUp6BcvnxZKSkpkiQfHx9lZGQYEc8yKCgW8/snx0q37vevXLmyxo4dq2bNmhmUClYXERGhiIgIXbp0Kc8zeubPn29QKlhV27Zt1aNHD73//vt68MEHJUl79+7V66+/rnbt2kmS9uzZo/vvv9/AlHc/LvEAMNSYMWM0duxY1a1bV2XKlMlzG/zq1asNSgarSk1N1eDBg7V48WJlZWVJurXNfdeuXfXhhx+qWLFi2r9/v6S8d0Yi/1BQABiqTJkymjJlil588UWjowAOUlNTdfr0aUlScHCwihcvbnAia6GgWEyJEiVuu1Hb7+/v79atm7p3725AOlhRyZIltWfPHlWoUMHoKABMhDUoFjNq1CiNHz9eLVq0UL169STdupa6fv169evXT3Fxcerbt6+ysrLUu3dvg9PCCnr16qXPPvuMRdowjbS0NE2aNOlP10X9NquCgkVBsZjt27fr3Xff1csvv+ww/sknn2jDhg368ssvVaNGDU2fPp2CgkJx48YNzZkzRz/88INq1KihIkWKOBz/4IMPDEoGq+rVq5e2bNmiF1988bbrolA4uMRjMcWLF9f+/ftVsWJFh/GTJ0+qVq1aSk1N1alTp1SjRg2lpaUZlBJW8vjjj//pMZvNxvOhUOh8fHz07bffqmHDhkZHsTRmUCzG19dXa9eu1eDBgx3G165da3+uRFpamjw9PY2IBwvavHmz0REAByVKlOA5OyZAQbGYkSNHqm/fvtq8ebN9DcrevXu1bt06zZ49W5K0ceNGPfbYY0bGBADDjBs3TqNGjdKiRYvk4eFhdBzL4hKPBe3YsUMzZszQsWPHJEmVK1fWgAED1KBBA4OTwSrat2+vhQsXysvLK8/uxn+0atWqQkoF3BIaGqpTp04pNzdX5cuXz7MuKiYmxqBk1sIMigU1bNiQa6swlLe3t33h4R93NwaM9ttusTAWMygWlJOTo5MnT9729rlHH33UoFQAAPx/zKBYzK5du9SpUyedPXtWf+ymNptN2dnZBiUDAPNISkrSypUrderUKQ0dOlS+vr6KiYmRv7+/7r33XqPjWQIzKBZTq1Yt3X///RozZsxt7+9nuh2FLSgo6C/3mWBTLBS2AwcOqEmTJvL29taZM2d07NgxBQcHa8SIETp37pwWL15sdERLYAbFYk6cOKGVK1fm2QcFMMqrr77q8HVmZqb27dun9evXa+jQocaEgqUNGTJE3bp105QpUxy2XGjZsqU6depkYDJroaBYTP369XXy5EkKCkxj0KBBtx2fOXOmoqKiCjkNcGvrhU8++STP+L333quEhAQDElkTBcViBgwYoNdee00JCQmqXr16ntvnatSoYVAywFGLFi00fPhwLViwwOgosBhXV1elpKTkGT9+/LjuueceAxJZE2tQLMbJySnPmM1mU25uLotkYSpTpkzRxx9/rDNnzhgdBRbTq1cvXblyRStWrJCvr68OHDggZ2dntWvXTo8++qimTp1qdERLoKBYzNmzZ//yeLly5QopCXBLaGiowyLZ3NxcJSQk6PLly/r444/Vp08fA9PBipKTk/X0008rKipK165dU0BAgBISEhQWFqZ169apWLFiRke0BAoKAEONGTPG4WsnJyfdc889atSokapUqWJQKuDW098PHDig1NRU1a5dW02aNDE6kqVQUCxoyZIlmj17tuLi4hQZGaly5cpp6tSpCgoKUtu2bY2OBwCA8i5IwF1t1qxZGjJkiFq2bKmkpCT7mhMfHx+uq8IQKSkpt31du3ZNGRkZRseDRUVEROjJJ59UhQoVVKFCBT355JP64YcfjI5lKRQUi/noo4/06aef6u2335azs7N9vG7dujp48KCByWBVPj4+KlGiRJ6Xj4+P3N3dVa5cOY0ePTrPYxmAgvLxxx+refPm8vT01KBBgzRo0CB5eXmpZcuWmjlzptHxLIPbjC0mLi5OoaGhecZdXV2VlpZmQCJY3cKFC/X222+rW7duqlevniRpz549WrRokUaMGKHLly/rvffek6urq9566y2D08IKJkyYoA8//FD9+/e3jw0cOFANGzbUhAkT1K9fPwPTWQcFxWKCgoK0f//+PHfrrF+/XlWrVjUoFaxs0aJFev/99/Xss8/ax1q3bq3q1avrk08+UUREhMqWLavx48dTUFAokpKS1Lx58zzjzZo107BhwwxIZE1c4rGYIUOGqF+/flq+fLlyc3O1Z88ejR8/XsOHD9cbb7xhdDxY0M6dO287qxcaGqrIyEhJ0sMPP6xz584VdjRYVJs2bbR69eo8419//bWefPJJAxJZEzMoFtOrVy+5u7trxIgRSk9PV6dOnRQQEKBp06bp+eefNzoeLCgwMFDz5s3TpEmTHMbnzZunwMBASdKVK1dUokQJI+LBgkJCQjR+/Hj9+OOPCgsLk3TrSfA7duzQa6+9punTp9vPHThwoFEx73rcZmwxN2/eVFZWlooVK6b09HSlpqbKz8/P6FiwsDVr1uiZZ55RlSpV9OCDD0qSoqKidPToUa1cuVJPPvmkZs2apRMnTuiDDz4wOC2sICgo6B+dZ7PZeNp2AaKgWMTly5fVpUsX/fDDD8rJydGDDz6opUuXqkKFCkZHAxQXF6dPPvlEx48flyRVrlxZL730ksqXL29sMACGoaBYRI8ePfTdd99p4MCBcnNz0yeffKIyZcpo8+bNRkcDACAPCopFBAYGau7cuQoPD5cknThxQlWrVlVaWppcXV0NTgerS0pK0p49e3Tp0qU8+5106dLFoFQAjERBsQhnZ2edP39epUuXto8VK1ZMhw4dYhodhlq7dq06d+6s1NRUeXl5OTw40Gaz6erVqwamA2AUbjO2kN/vHPvb1/RTGO21115Tjx49lJqaqqSkJP3666/2F+UEsC5mUCzCyclJ3t7eDj+dJiUlycvLS05O/7+n8g8CCluxYsV08OBBBQcHGx0FgImwD4pFLFiwwOgIwG2Fh4crKiqKggJTSUpK0rx583TkyBFJUrVq1dSjRw95e3sbnMw6mEEBYKh58+Zp7Nix6t69u6pXr64iRYo4HG/Tpo1ByWBVUVFRCg8Pl7u7u/35UHv37tX169e1YcMG1a5d2+CE1kBBAWCo319i/CObzabs7OxCTANIjzzyiCpWrKhPP/1ULi63LjRkZWWpV69eOn36tLZu3WpwQmugoAAA8Dvu7u7at2+fqlSp4jB++PBh1a1bV+np6QYlsxbu4gFgGjdu3DA6AiAvL6/bPpwyPj5enp6eBiSyJgoKAENlZ2dr3Lhxuvfee1W8eHH7s01GjhypefPmGZwOVvTcc8+pZ8+eWr58ueLj4xUfH69ly5apV69e6tixo9HxLIOCYlEZGRk6duyYsrKyjI4Cixs/frwWLlyoKVOmqGjRovbxBx54QHPnzjUwGazqvffeU/v27dWlSxeVL19e5cuXV7du3fT0009r8uTJRsezDNagWEx6eroGDBigRYsWSZKOHz+u4OBgDRgwQPfee6/efPNNgxPCaipWrKhPPvlEjRs3lqenp2JjYxUcHKyjR48qLCxMv/76q9ERYVHp6ek6deqUJKlChQry8PAwOJG1MINiMcOHD1dsbKx+/PFHubm52cebNGmi5cuXG5gMVnX+/HlVrFgxz3hOTo4yMzMNSATc4uHhoRIlSqhEiRKUEwNQUCzmq6++0owZM/Twww877CpbrVo1+08KQGEKCQnRtm3b8oyvXLlSoaGhBiSC1eXk5Gjs2LHy9vZWuXLlVK5cOfn4+GjcuHF5HmaJgsNOshZz+fJl+fn55RlPS0tzKCxAYRk1apS6du2q8+fPKycnR6tWrdKxY8e0ePFiffPNN0bHgwW9/fbbmjdvniZNmqSGDRtKkrZv36533nlHN27c0Pjx4w1OaA2sQbGYRx99VM8884wGDBggT09PHThwQEFBQRowYIBOnDih9evXGx0RFrRt2zaNHTtWsbGxSk1NVe3atTVq1Cg1a9bM6GiwoICAAM2ePTvPLsZff/21XnnlFZ0/f96gZNbCDIrFTJgwQS1atNDhw4eVlZWladOm6fDhw9q5c6e2bNlidDxY1COPPKKNGzcaHQOQdOuhqX/cpE2SqlSpwgNVCxFrUCzm4Ycf1v79+5WVlaXq1atrw4YN8vPzU2RkpOrUqWN0PFhYVFSUlixZoiVLlig6OtroOLCwmjVrasaMGXnGZ8yYoZo1axqQyJq4xAPAUD///LM6duyoHTt2yMfHR9KtJ8k2aNBAy5Yt03333WdsQFjOli1b1KpVK5UtW1ZhYWGSpMjISMXHx2vdunV65JFHDE5oDcygWEBKSorDr//qBRS2Xr16KTMzU0eOHNHVq1d19epVHTlyRDk5OerVq5fR8WBBjz32mI4fP66nnnpKSUlJSkpKUvv27XXs2DHKSSFiBsUCnJ2ddfHiRfn5+cnJyem2d+vk5uby5FgYwt3dXTt37sxzS3F0dLQeeeQRHsyGQnfu3DkFBgbe9u/Kc+fOqWzZsgaksh4WyVrApk2b5OvrK0navHmzwWkAR4GBgbfdkC07O1sBAQEGJILVBQUF2X+o+70rV64oKCiIH+QKCQXFAh577LHb/howg//7v//TgAEDNHPmTNWtW1fSrQWzgwYN0nvvvWdwOljRbzPKf5SamuqwAzcKFpd4LODAgQP/+NwaNWoUYBIgrxIlSig9PV1ZWVlycbn1M9Nvvy5WrJjDudziiYI0ZMgQSdK0adPUu3dvh+3ts7OztXv3bjk7O2vHjh1GRbQUZlAsoFatWrLZbPq7LsoaFBhh6tSpRkcAJEn79u2TdGsG5eDBgw5P1y5atKhq1qyp119/3ah4lsMMigWcPXv2H59brly5AkwCAObXvXt3TZs2TV5eXkZHsTQKCgAAMB32QbGgJUuWqGHDhgoICLDPrkydOlVff/21wckAwHhpaWkaOXKkGjRooIoVKyo4ONjhhcLBGhSLmTVrlkaNGqVXX31V48ePt6858fHx0dSpU9W2bVuDEwKAsXr16qUtW7boxRdfVJkyZXjSu0G4xGMxISEhmjBhgtq1aydPT0/FxsYqODhYP/30kxo1aqRffvnF6IgAYCgfHx99++23atiwodFRLI1LPBYTFxeXZ8dOSXJ1dVVaWpoBiYD/Lz4+XvHx8UbHgMWVKFHCvrkljENBsZigoCDt378/z/j69etVtWrVwg8Ey8vKytLIkSPl7e2t8uXLq3z58vL29taIESNuu8MsUNDGjRunUaNG8ZgFg7EGxWKGDBmifv366caNG8rNzdWePXv0+eefa+LEiZo7d67R8WBBAwYM0KpVqzRlyhSHJ8e+8847unLlimbNmmVwQljN+++/r1OnTsnf31/ly5dXkSJFHI7HxMQYlMxaWINiQUuXLtU777yjU6dOSZICAgI0ZswY9ezZ0+BksCJvb28tW7ZMLVq0cBhft26dOnbsqOTkZIOSwarGjBnzl8dHjx5dSEmsjYJiYenp6UpNTc3zQCygMPn5+WnLli15LjEeOXJEjz76qC5fvmxQMgBGYg2KhXl4eFBOYLj+/ftr3Lhxunnzpn3s5s2bGj9+vPr3729gMlhZUlKS5s6dq+HDh9ufARUTE6Pz588bnMw6mEGxgNDQ0H98Hz/XVlHYnnrqKUVERMjV1VU1a9aUJMXGxiojI0ONGzd2OHfVqlVGRITFHDhwQE2aNJG3t7fOnDmjY8eOKTg4WCNGjNC5c+e0ePFioyNaAotkLaBdu3b2X9+4cUMff/yxQkJC7AsSd+3apUOHDumVV14xKCGszMfHRx06dHAYCwwMNCgNcOtmgm7dumnKlCny9PS0j7ds2VKdOnUyMJm1MINiMb169VKZMmU0btw4h/HRo0crPj5e8+fPNygZAJiDt7e3YmJiVKFCBYcNLc+ePavKlSvrxo0bRke0BNagWMwXX3yhLl265Bl/4YUX9OWXXxqQCADMxdXVVSkpKXnGjx8/rnvuuceARNbEJR6LcXd3144dO1SpUiWH8R07dsjNzc2gVLC6lStXasWKFTp37pwyMjIcjrEuCoWtTZs2Gjt2rFasWCFJstlsOnfunIYNG5bnciQKDjMoFvPqq6+qb9++GjhwoP7zn//oP//5jwYMGKB+/fpp8ODBRseDBU2fPl3du3eXv7+/9u3bp3r16qlkyZI6ffp0nr1RgMLw/vvv27dguH79uh577DFVrFhRnp6eGj9+vNHxLIM1KBa0YsUKTZs2TUeOHJEkVa1aVYMGDdKzzz5rcDJYUZUqVTR69Gh17NjR4Xr/qFGjdPXqVc2YMcPoiLCo7du368CBA0pNTVXt2rXVpEkToyNZCgUFdj/99JMeeOABo2PAYjw8PHTkyBGVK1dOfn5+2rhxo2rWrKkTJ07ooYce0pUrV4yOCMAArEGxuGvXrunzzz/X3LlzFR0drezsbKMjwWJKly6tq1evqly5cipbtqx27dqlmjVrKi4uTvz8hMJ0/fp1RURE6Mknn5QkDR8+3GEDQWdnZ40bN471eoWEgmJRW7du1dy5c7Vq1SoFBASoffv2mjlzptGxYEFPPPGE1qxZo9DQUHXv3l2DBw/WypUrFRUVpfbt2xsdDxayaNEiffvtt/aCMmPGDFWrVk3u7u6SpKNHjyogIID1eoWESzwWkpCQoIULF2revHlKSUnRs88+q9mzZys2NlYhISFGx4NF5eTkKCcnRy4ut35eWrZsmXbu3KlKlSrppZdeUtGiRQ1OCKt45JFH9MYbb6h169aS5LAmSpL+85//aObMmYqMjDQypmVQUCyidevW2rp1q1q1aqXOnTurefPmcnZ2VpEiRSgoMExWVpYmTJigHj166L777jM6DiyuTJkyioyMVPny5SVJ99xzj/bu3Wv/+vjx43rwwQd5wnYh4TZji/juu+/Us2dPjRkzRq1atZKzs7PRkQC5uLhoypQpysrKMjoKoKSkJIc1J5cvX7aXE+nWbN/vj6NgUVAsYvv27bp27Zrq1Kmj+vXra8aMGfrll1+MjgWocePG2rJli9ExAN1333366aef/vT4gQMHmOkrRFzisZi0tDQtX75c8+fP1549e5Sdna0PPvhAPXr0cHgoFlBYZs+erTFjxqhz586qU6eOihUr5nC8TZs2BiWD1QwaNEg//PCDoqOj89ypc/36ddWtW1dNmjTRtGnTDEpoLRQUCzt27JjmzZunJUuWKCkpSU2bNtWaNWuMjgWLcXL684lcm83Gre8oNImJiapVq5aKFi2q/v376/7775d06+/KGTNmKCsrS/v27ZO/v7/BSa2BggJlZ2dr7dq1mj9/PgUFgKXFxcWpb9++2rhxo30fHpvNpqZNm+rjjz+239GDgkdBAWCoxYsX67nnnpOrq6vDeEZGhpYtW3bbp28DBe3q1as6efKkJKlixYry9fU1OJH1UFAAGMrZ2VkXL16Un5+fw/iVK1fk5+fHJR7AoriLB4ChcnNzZbPZ8oz//PPP8vb2NiARADNgq3sAhggNDZXNZpPNZlPjxo3tO8lKt9ZFxcXFqXnz5gYmBGAkCgoAQ7Rr106StH//foWHh6t48eL2Y0WLFlX58uXVoUMHg9IBMBprUAAYatGiRXr++efzLJIFYG0UFACGio+Pl81ms+/QuWfPHn322WcKCQlRnz59DE4HwCgskgVgqE6dOmnz5s2Sbj1xu0mTJtqzZ4/efvttjR071uB0AIxCQQFgqJ9++kn16tWTJK1YsULVq1fXzp07tXTpUi1cuNDYcAAMQ0EBYKjMzEz7+pMffvjB/uydKlWq6OLFi0ZGA2AgCgoAQ1WrVk2zZ8/Wtm3btHHjRvutxRcuXFDJkiUNTgfAKBQUAIaaPHmyPvnkEzVq1EgdO3ZUzZo1JUlr1qyxX/oBYD3cxQPAcNnZ2UpJSVGJEiXsY2fOnJGHh0eeLfABWAMFBQAAmA6XeAAYKjExUS+++KICAgLk4uIiZ2dnhxcAa2KrewCG6tatm86dO6eRI0eqTJkyt31wIADr4RIPAEN5enpq27ZtqlWrltFRAJgIl3gAGCowMFD8nATgjygoAAw1depUvfnmmzpz5ozRUQCYCJd4ABiqRIkSSk9PV1ZWljw8PFSkSBGH41evXjUoGQAjsUgWgKGmTp1qdAQAJsQMCgAAMB1mUAAUupSUFHl5edl//Vd+Ow+AtTCDAqDQOTs76+LFi/Lz85OTk9Nt9z7Jzc2VzWZTdna2AQkBGI0ZFACFbtOmTfL19ZUkbd682eA0AMyIGRQAAGA6zKAAMFxSUpL27NmjS5cuKScnx+FYly5dDEoFwEjMoAAw1Nq1a9W5c2elpqbKy8vLYT2KzWZjHxTAoigoAAx1//33q2XLlpowYYI8PDyMjgPAJCgoAAxVrFgxHTx4UMHBwUZHAWAiPIsHgKHCw8MVFRVldAwAJsMiWQCFbs2aNfZft2rVSkOHDtXhw4dVvXr1PM/iadOmTWHHA2ACXOIBUOicnP7Z5C0btQHWRUEBAACmwxoUAABgOhQUAIbYtGmTQkJCbvuwwOTkZFWrVk1bt241IBkAM6CgADDE1KlT1bt379s+rdjb21svvfSSPvzwQwOSATADCgoAQ8TGxqp58+Z/erxZs2aKjo4uxEQAzISCAsAQiYmJeW4p/j0XFxddvny5EBMBMBMKCgBD3Hvvvfrpp5/+9PiBAwdUpkyZQkwEwEwoKAAM0bJlS40cOVI3btzIc+z69esaPXq0nnzySQOSATAD9kEBYIjExETVrl1bzs7O6t+/vypXrixJOnr0qGbOnKns7GzFxMTI39/f4KQAjEBBAWCYs2fPqm/fvvr+++/1219FNptN4eHhmjlzpoKCggxOCMAoFBQAhvv111918uRJ5ebmqlKlSipRooTRkQAYjIICAABMh0WyAADAdCgoAADAdCgoAADAdCgoAADAdCgoAO44NptNX331ldExABQgCgqAf+Xy5cvq27evypYtK1dXV5UuXVrh4eHasWNHoWW4ePGiWrRoUWi/H4DC52J0AAB3lg4dOigjI0OLFi1ScHCwEhMTFRERoStXrhRahtKlSxfa7wXAGMygAPjHkpKStG3bNk2ePFmPP/64ypUrp3r16mn48OFq06aNpFuXX2bNmqUWLVrI3d1dwcHBWrlypcP7xMfH69lnn5WPj498fX3Vtm1bnTlzxuGc+fPnq1q1anJ1dVWZMmXUv39/+7E/XuL5u/f78ccfVa9ePRUrVkw+Pj5q2LChzp49m+//fQDkHwoKgH+sePHiKl68uL766ivdvHnzT88bOXKkOnTooNjYWHXu3FnPP/+8jhw5IknKzMxUeHi4PD09tW3bNu3YsUPFixdX8+bNlZGRIUmaNWuW+vXrpz59+ujgwYNas2aNKlaseNvf6+/eLysrS+3atdNjjz2mAwcOKDIyUn369JHNZsv//0AA8k8uAPwLK1euzC1RokSum5tbboMGDXKHDx+eGxsbaz8uKffll192+J769evn9u3bNzc3Nzd3yZIluZUrV87NycmxH79582auu7t77vfff5+bm5ubGxAQkPv222//aQZJuatXr/5H73flypVcSbk//vjj//xnB1B4mEEB8K906NBBFy5c0Jo1a9S8eXP9+OOPql27thYuXGg/JywszOF7wsLC7DMosbGxOnnypDw9Pe0zMr6+vrpx44ZOnTqlS5cu6cKFC2rcuPE/yvN37+fr66tu3bopPDxcrVu31rRp03Tx4sV8++8BoGCwSBbAv+bm5qamTZuqadOmGjlypHr16qXRo0erW7duf/u9qampqlOnjpYuXZrn2D333CMnp3/3c9PfvZ8kLViwQAMHDtT69eu1fPlyjRgxQhs3btRDDz30r34vAIWHGRQA/7OQkBClpaXZv961a5fD8V27dqlq1aqSpNq1a+vEiRPy8/NTxYoVHV7e3t7y9PRU+fLlFRER8Y9+7797v9+EhoZq+PDh2rlzpx544AF99tln+fAnB1BQKCgA/rErV67oiSee0H/+8x8dOHBAcXFx+uKLLzRlyhS1bdvWft4XX3yh+fPn6/jx4xo9erT27Nljvwunc+fOKlWqlNq2batt27YpLi5OP/74owYOHKiff/5ZkvTOO+/o/fff1/Tp03XixAnFxMToo48+um2mv3u/uLg4DR8+XJGRkTp79qw2bNigEydO2AsTAHPiEg+Af6x48eKqX7++PvzwQ506dUqZmZkKDAxU79699dZbb9nPGzNmjJYtW6ZXXnlFZcqU0eeff66QkBBJkoeHh7Zu3aphw4apffv2unbtmu699141btxYXl5ekqSuXbvqxo0b+vDDD/X666+rVKlSevrpp2+b6e/e7/r16zp69KgWLVqkK1euqEyZMurXr59eeumlgv8PBuC/ZsvNzc01OgSAu4fNZtPq1avVrl07o6MAuINxiQcAAJgOBQUAAJgOa1AA5CuuGgPID8ygAAAA06GgAAAA06GgAAAA06GgAAAA06GgAAAA06GgAAAA06GgAAAA06GgAAAA0/l/yBw9GNTwOtMAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "penguins.groupby(\"Species\").Sex.count().plot(kind=\"bar\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this case it's not very extreme. We have around twice as many Adelie than Chinstrap penguins.\n", "\n", "However, this can mean that we accidentally have almost no Chinstrap penguins in our training data, as it randomly overselects Adelie penguins." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:11.738746Z", "iopub.status.busy": "2022-12-13T01:42:11.738246Z", "iopub.status.idle": "2022-12-13T01:42:11.766733Z", "shell.execute_reply": "2022-12-13T01:42:11.766231Z" }, "tags": [] }, "outputs": [ { "ename": "NameError", "evalue": "name 'y_train' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [6], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[43my_train\u001b[49m\u001b[38;5;241m.\u001b[39mreset_index()\u001b[38;5;241m.\u001b[39mgroupby([\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSpecies\u001b[39m\u001b[38;5;124m\"\u001b[39m])\u001b[38;5;241m.\u001b[39mcount()\n", "\u001b[1;31mNameError\u001b[0m: name 'y_train' is not defined" ] } ], "source": [ "y_train.reset_index().groupby([\"Species\"]).count()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can address this by applying **stratification**.\n", "\n", "That is simply achieved by randomly sampling *within a class** (or strata) rather than randomly sampling from the entire dataframe." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:11.769232Z", "iopub.status.busy": "2022-12-13T01:42:11.769232Z", "iopub.status.idle": "2022-12-13T01:42:11.797561Z", "shell.execute_reply": "2022-12-13T01:42:11.797060Z" }, "tags": [] }, "outputs": [ { "ename": "NameError", "evalue": "name 'features' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [7], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m X, y \u001b[38;5;241m=\u001b[39m penguins[\u001b[43mfeatures\u001b[49m], penguins[target[\u001b[38;5;241m0\u001b[39m]]\n\u001b[0;32m 2\u001b[0m X_train, X_test, y_train, y_test \u001b[38;5;241m=\u001b[39m train_test_split(X, y, train_size\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m.7\u001b[39m, random_state\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m42\u001b[39m, stratify\u001b[38;5;241m=\u001b[39my)\n", "\u001b[1;31mNameError\u001b[0m: name 'features' is not defined" ] } ], "source": [ "X, y = penguins[features], penguins[target[0]]\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=.7, random_state=42, stratify=y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To qualitatively assess the effect of stratification, let's plot class distribution in both _training_ and _test_ sets:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:11.800061Z", "iopub.status.busy": "2022-12-13T01:42:11.800061Z", "iopub.status.idle": "2022-12-13T01:42:11.999103Z", "shell.execute_reply": "2022-12-13T01:42:11.998601Z" } }, "outputs": [ { "ename": "NameError", "evalue": "name 'y_train' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [8], line 3\u001b[0m\n\u001b[0;32m 1\u001b[0m fig, (ax1, ax2) \u001b[38;5;241m=\u001b[39m plt\u001b[38;5;241m.\u001b[39msubplots(\u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m, figsize\u001b[38;5;241m=\u001b[39m(\u001b[38;5;241m12\u001b[39m, \u001b[38;5;241m8\u001b[39m))\n\u001b[1;32m----> 3\u001b[0m \u001b[43my_train\u001b[49m\u001b[38;5;241m.\u001b[39mreset_index()\u001b[38;5;241m.\u001b[39mgroupby(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSpecies\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mcount()\u001b[38;5;241m.\u001b[39mplot(kind\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mbar\u001b[39m\u001b[38;5;124m\"\u001b[39m, ax\u001b[38;5;241m=\u001b[39max1, ylim\u001b[38;5;241m=\u001b[39m(\u001b[38;5;241m0\u001b[39m, \u001b[38;5;28mlen\u001b[39m(y)), title\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mTraining\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m 4\u001b[0m y_test\u001b[38;5;241m.\u001b[39mreset_index()\u001b[38;5;241m.\u001b[39mgroupby(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSpecies\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mcount()\u001b[38;5;241m.\u001b[39mplot(kind\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mbar\u001b[39m\u001b[38;5;124m\"\u001b[39m, ax\u001b[38;5;241m=\u001b[39max2, ylim\u001b[38;5;241m=\u001b[39m(\u001b[38;5;241m0\u001b[39m, \u001b[38;5;28mlen\u001b[39m(y)), title\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mTest\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m 5\u001b[0m plt\u001b[38;5;241m.\u001b[39mshow()\n", "\u001b[1;31mNameError\u001b[0m: name 'y_train' is not defined" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA+AAAAKZCAYAAAA4fUHAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAniUlEQVR4nO3df2zX9Z3A8RcU22pmKx5H+XF1nO6c21RwIL3qjPHSG4mGHX9cxqkBjvjjnJxxNncTROmcN8p5zpBMHJHpuT/mwbaoWQbBc93I4uRCxo/EnaBh6OCWtcLtbDncKLSf+2Oxu47i+Nbv99WCj0fy/YOPn0+/7+87xRfPfr/9fscURVEEAAAAUFFjR3oBAAAA8EEgwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAggQAHAACABAIcAAAAEghwAAAASCDAAQAAIEHJAf6jH/0o5s6dG1OmTIkxY8bE888//wev2bJlS3zyk5+Mmpqa+MhHPhJPP/30MJYKAGQw6wGgMkoO8CNHjsT06dNjzZo1p3T+G2+8ETfccENcd911sWvXrvj85z8ft956a7zwwgslLxYAqDyzHgAqY0xRFMWwLx4zJp577rmYN2/eSc+59957Y+PGjfHTn/504Njf/M3fxNtvvx2bN28e7l0DAAnMegAon3GVvoOtW7dGS0vLoGNz5syJz3/+8ye95ujRo3H06NGBP/f398evfvWr+KM/+qMYM2ZMpZYKAKekKIo4fPhwTJkyJcaO9XYqZj0AZ6JKzPuKB3hnZ2c0NDQMOtbQ0BA9PT3x61//Os4+++wTrmlvb48HH3yw0ksDgPflwIED8Sd/8icjvYwRZ9YDcCYr57yveIAPx7Jly6K1tXXgz93d3XHBBRfEgQMHoq6ubgRXBgARPT090djYGOeee+5IL+W0ZdYDMNpVYt5XPMAnTZoUXV1dg451dXVFXV3dkD8Rj4ioqamJmpqaE47X1dUZygCMGl4q/VtmPQBnsnLO+4r/4lpzc3N0dHQMOvbiiy9Gc3Nzpe8aAEhg1gPAqSk5wP/3f/83du3aFbt27YqI3370yK5du2L//v0R8duXlC1cuHDg/DvuuCP27dsXX/jCF2LPnj3x+OOPx7e+9a245557yvMIAICyMusBoDJKDvCf/OQnccUVV8QVV1wRERGtra1xxRVXxIoVKyIi4pe//OXAgI6I+NM//dPYuHFjvPjiizF9+vT4yle+El//+tdjzpw5ZXoIAEA5mfUAUBnv63PAs/T09ER9fX10d3f7vTAARpy5VH72FIDRphKzyYeXAgAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBgWAG+Zs2amDZtWtTW1kZTU1Ns27btPc9fvXp1fPSjH42zzz47Ghsb45577onf/OY3w1owAFB5Zj0AlF/JAb5hw4ZobW2Ntra22LFjR0yfPj3mzJkTb7311pDnP/PMM7F06dJoa2uL3bt3x5NPPhkbNmyI++67730vHgAoP7MeACqj5AB/9NFH47bbbovFixfHxz/+8Vi7dm2cc8458dRTTw15/ssvvxxXX3113HTTTTFt2rT49Kc/HTfeeOMf/Ek6ADAyzHoAqIySAry3tze2b98eLS0tv/sCY8dGS0tLbN26dchrrrrqqti+ffvAEN63b19s2rQprr/++pPez9GjR6Onp2fQDQCoPLMeACpnXCknHzp0KPr6+qKhoWHQ8YaGhtizZ8+Q19x0001x6NCh+NSnPhVFUcTx48fjjjvueM+XpbW3t8eDDz5YytIAgDIw6wGgcir+LuhbtmyJlStXxuOPPx47duyIZ599NjZu3BgPPfTQSa9ZtmxZdHd3D9wOHDhQ6WUCAMNk1gPAqSnpGfAJEyZEVVVVdHV1DTre1dUVkyZNGvKaBx54IBYsWBC33nprRERcdtllceTIkbj99ttj+fLlMXbsiT8DqKmpiZqamlKWBgCUgVkPAJVT0jPg1dXVMXPmzOjo6Bg41t/fHx0dHdHc3DzkNe+8884Jg7eqqioiIoqiKHW9AEAFmfUAUDklPQMeEdHa2hqLFi2KWbNmxezZs2P16tVx5MiRWLx4cURELFy4MKZOnRrt7e0RETF37tx49NFH44orroimpqbYu3dvPPDAAzF37tyB4QwAjB5mPQBURskBPn/+/Dh48GCsWLEiOjs7Y8aMGbF58+aBN2vZv3//oJ+C33///TFmzJi4//774xe/+EX88R//ccydOze+/OUvl+9RAABlY9YDQGWMKU6D14b19PREfX19dHd3R11d3UgvB4APOHOp/OwpAKNNJWZTxd8FHQAAABDgAAAAkEKAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAmGFeBr1qyJadOmRW1tbTQ1NcW2bdve8/y33347lixZEpMnT46ampq4+OKLY9OmTcNaMABQeWY9AJTfuFIv2LBhQ7S2tsbatWujqakpVq9eHXPmzInXXnstJk6ceML5vb298Zd/+ZcxceLE+M53vhNTp06Nn//853HeeeeVY/0AQJmZ9QBQGWOKoihKuaCpqSmuvPLKeOyxxyIior+/PxobG+Ouu+6KpUuXnnD+2rVr41/+5V9iz549cdZZZw1rkT09PVFfXx/d3d1RV1c3rK8BAOVyps8lsx4AKjObSnoJem9vb2zfvj1aWlp+9wXGjo2WlpbYunXrkNd897vfjebm5liyZEk0NDTEpZdeGitXroy+vr73t3IAoOzMegConJJegn7o0KHo6+uLhoaGQccbGhpiz549Q16zb9+++MEPfhA333xzbNq0Kfbu3Rt33nlnHDt2LNra2oa85ujRo3H06NGBP/f09JSyTABgmMx6AKicir8Len9/f0ycODGeeOKJmDlzZsyfPz+WL18ea9euPek17e3tUV9fP3BrbGys9DIBgGEy6wHg1JQU4BMmTIiqqqro6uoadLyrqysmTZo05DWTJ0+Oiy++OKqqqgaOfexjH4vOzs7o7e0d8pply5ZFd3f3wO3AgQOlLBMAGCazHgAqp6QAr66ujpkzZ0ZHR8fAsf7+/ujo6Ijm5uYhr7n66qtj79690d/fP3Ds9ddfj8mTJ0d1dfWQ19TU1ERdXd2gGwBQeWY9AFROyS9Bb21tjXXr1sU3vvGN2L17d3zuc5+LI0eOxOLFiyMiYuHChbFs2bKB8z/3uc/Fr371q7j77rvj9ddfj40bN8bKlStjyZIl5XsUAEDZmPUAUBklfw74/Pnz4+DBg7FixYro7OyMGTNmxObNmwferGX//v0xduzvur6xsTFeeOGFuOeee+Lyyy+PqVOnxt133x333ntv+R4FAFA2Zj0AVEbJnwM+Enw2KACjiblUfvYUgNFmxD8HHAAAABgeAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAgmEF+Jo1a2LatGlRW1sbTU1NsW3btlO6bv369TFmzJiYN2/ecO4WAEhi1gNA+ZUc4Bs2bIjW1tZoa2uLHTt2xPTp02POnDnx1ltvved1b775ZvzDP/xDXHPNNcNeLABQeWY9AFRGyQH+6KOPxm233RaLFy+Oj3/847F27do455xz4qmnnjrpNX19fXHzzTfHgw8+GBdeeOH7WjAAUFlmPQBURkkB3tvbG9u3b4+WlpbffYGxY6OlpSW2bt160uu+9KUvxcSJE+OWW245pfs5evRo9PT0DLoBAJVn1gNA5ZQU4IcOHYq+vr5oaGgYdLyhoSE6OzuHvOall16KJ598MtatW3fK99Pe3h719fUDt8bGxlKWCQAMk1kPAJVT0XdBP3z4cCxYsCDWrVsXEyZMOOXrli1bFt3d3QO3AwcOVHCVAMBwmfUAcOrGlXLyhAkToqqqKrq6ugYd7+rqikmTJp1w/s9+9rN48803Y+7cuQPH+vv7f3vH48bFa6+9FhdddNEJ19XU1ERNTU0pSwMAysCsB4DKKekZ8Orq6pg5c2Z0dHQMHOvv74+Ojo5obm4+4fxLLrkkXnnlldi1a9fA7TOf+Uxcd911sWvXLi83A4BRxqwHgMop6RnwiIjW1tZYtGhRzJo1K2bPnh2rV6+OI0eOxOLFiyMiYuHChTF16tRob2+P2trauPTSSwddf95550VEnHAcABgdzHoAqIySA3z+/Plx8ODBWLFiRXR2dsaMGTNi8+bNA2/Wsn///hg7tqK/Wg4AVJBZDwCVMaYoimKkF/GH9PT0RH19fXR3d0ddXd1ILweADzhzqfzsKQCjTSVmkx9fAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBAgAMAAEACAQ4AAAAJBDgAAAAkEOAAAACQQIADAABAAgEOAAAACQQ4AAAAJBDgAAAAkECAAwAAQAIBDgAAAAkEOAAAACQQ4AAAAJBgWAG+Zs2amDZtWtTW1kZTU1Ns27btpOeuW7currnmmhg/fnyMHz8+Wlpa3vN8AGDkmfUAUH4lB/iGDRuitbU12traYseOHTF9+vSYM2dOvPXWW0Oev2XLlrjxxhvjhz/8YWzdujUaGxvj05/+dPziF79434sHAMrPrAeAyhhTFEVRygVNTU1x5ZVXxmOPPRYREf39/dHY2Bh33XVXLF269A9e39fXF+PHj4/HHnssFi5ceEr32dPTE/X19dHd3R11dXWlLBcAyu5Mn0tmPQBUZjaV9Ax4b29vbN++PVpaWn73BcaOjZaWlti6despfY133nknjh07Fueff/5Jzzl69Gj09PQMugEAlWfWA0DllBTghw4dir6+vmhoaBh0vKGhITo7O0/pa9x7770xZcqUQYP997W3t0d9ff3ArbGxsZRlAgDDZNYDQOWkvgv6qlWrYv369fHcc89FbW3tSc9btmxZdHd3D9wOHDiQuEoAYLjMegA4uXGlnDxhwoSoqqqKrq6uQce7urpi0qRJ73ntI488EqtWrYrvf//7cfnll7/nuTU1NVFTU1PK0gCAMjDrAaBySnoGvLq6OmbOnBkdHR0Dx/r7+6OjoyOam5tPet3DDz8cDz30UGzevDlmzZo1/NUCABVl1gNA5ZT0DHhERGtrayxatChmzZoVs2fPjtWrV8eRI0di8eLFERGxcOHCmDp1arS3t0dExD//8z/HihUr4plnnolp06YN/P7Yhz70ofjQhz5UxocCAJSDWQ8AlVFygM+fPz8OHjwYK1asiM7OzpgxY0Zs3rx54M1a9u/fH2PH/u6J9a997WvR29sbf/3Xfz3o67S1tcUXv/jF97d6AKDszHoAqIySPwd8JPhsUABGE3Op/OwpAKPNiH8OOAAAADA8AhwAAAASCHAAAABIIMABAAAggQAHAACABAIcAAAAEghwAAAASCDAAQAAIIEABwAAgAQCHAAAABIIcAAAAEggwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAggQAHAACABAIcAAAAEghwAAAASCDAAQAAIIEABwAAgAQCHAAAABIIcAAAAEggwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAggQAHAACABAIcAAAAEghwAAAASCDAAQAAIIEABwAAgAQCHAAAABIIcAAAAEggwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAggQAHAACABAIcAAAAEghwAAAASCDAAQAAIIEABwAAgAQCHAAAABIIcAAAAEggwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAggQAHAACABAIcAAAAEghwAAAASCDAAQAAIIEABwAAgAQCHAAAABIIcAAAAEggwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAggQAHAACABAIcAAAAEghwAAAASCDAAQAAIIEABwAAgAQCHAAAABIIcAAAAEggwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAggQAHAACABAIcAAAAEghwAAAASCDAAQAAIIEABwAAgAQCHAAAABIIcAAAAEggwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAggQAHAACABAIcAAAAEghwAAAASCDAAQAAIIEABwAAgAQCHAAAABIIcAAAAEggwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAggQAHAACABAIcAAAAEghwAAAASCDAAQAAIIEABwAAgAQCHAAAABIIcAAAAEggwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAgwbACfM2aNTFt2rSora2Npqam2LZt23ue/+1vfzsuueSSqK2tjcsuuyw2bdo0rMUCADnMegAov5IDfMOGDdHa2hptbW2xY8eOmD59esyZMyfeeuutIc9/+eWX48Ybb4xbbrkldu7cGfPmzYt58+bFT3/60/e9eACg/Mx6AKiMMUVRFKVc0NTUFFdeeWU89thjERHR398fjY2Ncdddd8XSpUtPOH/+/Plx5MiR+N73vjdw7M///M9jxowZsXbt2lO6z56enqivr4/u7u6oq6srZbkAUHZn+lwy6wGgMrNpXCkn9/b2xvbt22PZsmUDx8aOHRstLS2xdevWIa/ZunVrtLa2Djo2Z86ceP755096P0ePHo2jR48O/Lm7uzsifrsBADDS3p1HJf4M+7Rg1gPAb1Vi3pcU4IcOHYq+vr5oaGgYdLyhoSH27Nkz5DWdnZ1Dnt/Z2XnS+2lvb48HH3zwhOONjY2lLBcAKuq///u/o76+fqSXUVZmPQAMVs55X1KAZ1m2bNmgn6S//fbb8eEPfzj2799/xv1DZyT09PREY2NjHDhwwMv8ysSelpf9LD97Wl7d3d1xwQUXxPnnnz/SSzltmfWV5+99ednP8rOn5WU/y68S876kAJ8wYUJUVVVFV1fXoONdXV0xadKkIa+ZNGlSSedHRNTU1ERNTc0Jx+vr630zlVFdXZ39LDN7Wl72s/zsaXmNHXvmfZqnWX/m8fe+vOxn+dnT8rKf5VfOeV/SV6quro6ZM2dGR0fHwLH+/v7o6OiI5ubmIa9pbm4edH5ExIsvvnjS8wGAkWPWA0DllPwS9NbW1li0aFHMmjUrZs+eHatXr44jR47E4sWLIyJi4cKFMXXq1Ghvb4+IiLvvvjuuvfba+MpXvhI33HBDrF+/Pn7yk5/EE088Ud5HAgCUhVkPAJVRcoDPnz8/Dh48GCtWrIjOzs6YMWNGbN68eeDNV/bv3z/oKfqrrroqnnnmmbj//vvjvvvuiz/7sz+L559/Pi699NJTvs+amppoa2sb8qVqlM5+lp89LS/7WX72tLzO9P00688M9rS87Gf52dPysp/lV4k9LflzwAEAAIDSnXnvHgMAAACjkAAHAACABAIcAAAAEghwAAAASDBqAnzNmjUxbdq0qK2tjaampti2bdt7nv/tb387LrnkkqitrY3LLrssNm3alLTS00Mp+7lu3bq45pprYvz48TF+/PhoaWn5g/v/QVTq9+i71q9fH2PGjIl58+ZVdoGnmVL38+23344lS5bE5MmTo6amJi6++GJ/739PqXu6evXq+OhHPxpnn312NDY2xj333BO/+c1vklY7uv3oRz+KuXPnxpQpU2LMmDHx/PPP/8FrtmzZEp/85CejpqYmPvKRj8TTTz9d8XWebsz68jLry8+sLz/zvrzM+vIZsVlfjALr168vqquri6eeeqr4z//8z+K2224rzjvvvKKrq2vI83/84x8XVVVVxcMPP1y8+uqrxf3331+cddZZxSuvvJK88tGp1P286aabijVr1hQ7d+4sdu/eXfzt3/5tUV9fX/zXf/1X8spHr1L39F1vvPFGMXXq1OKaa64p/uqv/ipnsaeBUvfz6NGjxaxZs4rrr7++eOmll4o33nij2LJlS7Fr167klY9epe7pN7/5zaKmpqb45je/WbzxxhvFCy+8UEyePLm45557klc+Om3atKlYvnx58eyzzxYRUTz33HPvef6+ffuKc845p2htbS1effXV4qtf/WpRVVVVbN68OWfBpwGzvrzM+vIz68vPvC8vs768RmrWj4oAnz17drFkyZKBP/f19RVTpkwp2tvbhzz/s5/9bHHDDTcMOtbU1FT83d/9XUXXeboodT9/3/Hjx4tzzz23+MY3vlGpJZ52hrOnx48fL6666qri61//erFo0SJD+f8pdT+/9rWvFRdeeGHR29ubtcTTTql7umTJkuIv/uIvBh1rbW0trr766oqu83R0KkP5C1/4QvGJT3xi0LH58+cXc+bMqeDKTi9mfXmZ9eVn1pefeV9eZn3lZM76EX8Jem9vb2zfvj1aWloGjo0dOzZaWlpi69atQ16zdevWQedHRMyZM+ek53+QDGc/f98777wTx44di/PPP79SyzytDHdPv/SlL8XEiRPjlltuyVjmaWM4+/nd7343mpubY8mSJdHQ0BCXXnpprFy5Mvr6+rKWPaoNZ0+vuuqq2L59+8BL1/bt2xebNm2K66+/PmXNZxpz6b2Z9eVl1pefWV9+5n15mfUjr1xzaVw5FzUchw4dir6+vmhoaBh0vKGhIfbs2TPkNZ2dnUOe39nZWbF1ni6Gs5+/7957740pU6ac8A32QTWcPX3ppZfiySefjF27diWs8PQynP3ct29f/OAHP4ibb745Nm3aFHv37o0777wzjh07Fm1tbRnLHtWGs6c33XRTHDp0KD71qU9FURRx/PjxuOOOO+K+++7LWPIZ52RzqaenJ37961/H2WefPUIrGx3M+vIy68vPrC8/8768zPqRV65ZP+LPgDO6rFq1KtavXx/PPfdc1NbWjvRyTkuHDx+OBQsWxLp162LChAkjvZwzQn9/f0ycODGeeOKJmDlzZsyfPz+WL18ea9euHemlnba2bNkSK1eujMcffzx27NgRzz77bGzcuDEeeuihkV4aUGFm/ftn1leGeV9eZv3oNOLPgE+YMCGqqqqiq6tr0PGurq6YNGnSkNdMmjSppPM/SIazn+965JFHYtWqVfH9738/Lr/88kou87RS6p7+7Gc/izfffDPmzp07cKy/vz8iIsaNGxevvfZaXHTRRZVd9Cg2nO/RyZMnx1lnnRVVVVUDxz72sY9FZ2dn9Pb2RnV1dUXXPNoNZ08feOCBWLBgQdx6660REXHZZZfFkSNH4vbbb4/ly5fH2LF+PluKk82lurq6D/yz3xFmfbmZ9eVn1pefeV9eZv3IK9esH/Fdr66ujpkzZ0ZHR8fAsf7+/ujo6Ijm5uYhr2lubh50fkTEiy++eNLzP0iGs58REQ8//HA89NBDsXnz5pg1a1bGUk8bpe7pJZdcEq+88krs2rVr4PaZz3wmrrvuuti1a1c0NjZmLn/UGc736NVXXx179+4d+MdNRMTrr78ekydP/kAP43cNZ0/feeedEwbvu//g+e17kVAKc+m9mfXlZdaXn1lffuZ9eZn1I69sc6mkt2yrkPXr1xc1NTXF008/Xbz66qvF7bffXpx33nlFZ2dnURRFsWDBgmLp0qUD5//4xz8uxo0bVzzyyCPF7t27i7a2Nh9N8v+Uup+rVq0qqquri+985zvFL3/5y4Hb4cOHR+ohjDql7unv886og5W6n/v37y/OPffc4u///u+L1157rfje975XTJw4sfinf/qnkXoIo06pe9rW1lace+65xb/9278V+/btK/793/+9uOiii4rPfvazI/UQRpXDhw8XO3fuLHbu3FlERPHoo48WO3fuLH7+858XRVEUS5cuLRYsWDBw/rsfTfKP//iPxe7du4s1a9b4GLLfY9aXl1lffmZ9+Zn35WXWl9dIzfpREeBFURRf/epXiwsuuKCorq4uZs+eXfzHf/zHwH+79tpri0WLFg06/1vf+lZx8cUXF9XV1cUnPvGJYuPGjckrHt1K2c8Pf/jDRUSccGtra8tf+ChW6vfo/2con6jU/Xz55ZeLpqamoqamprjwwguLL3/5y8Xx48eTVz26lbKnx44dK774xS8WF110UVFbW1s0NjYWd955Z/E///M/+QsfhX74wx8O+f/Fd/dw0aJFxbXXXnvCNTNmzCiqq6uLCy+8sPjXf/3X9HWPdmZ9eZn15WfWl595X15mffmM1KwfUxRefwAAAACVNuK/Aw4AAAAfBAIcAAAAEghwAAAASCDAAQAAIIEABwAAgAQCHAAAABIIcAAAAEggwAEAACCBAAcAAIAEAhwAAAASCHAAAABIIMABAAAgwf8Bqbh6EFCLdG8AAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 8))\n", "\n", "y_train.reset_index().groupby(\"Species\").count().plot(kind=\"bar\", ax=ax1, ylim=(0, len(y)), title=\"Training\")\n", "y_test.reset_index().groupby(\"Species\").count().plot(kind=\"bar\", ax=ax2, ylim=(0, len(y)), title=\"Test\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's quickly train a model to evaluate" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.002104Z", "iopub.status.busy": "2022-12-13T01:42:12.001603Z", "iopub.status.idle": "2022-12-13T01:42:12.030607Z", "shell.execute_reply": "2022-12-13T01:42:12.029608Z" }, "tags": [] }, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'sklearn'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [9], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msvm\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m SVC\n\u001b[0;32m 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcompose\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ColumnTransformer\n\u001b[0;32m 3\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mpipeline\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Pipeline\n", "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sklearn'" ] } ], "source": [ "from sklearn.svm import SVC\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n", "\n", "num_transformer = StandardScaler()\n", "cat_transformer = OneHotEncoder(handle_unknown='ignore')\n", "\n", "preprocessor = ColumnTransformer(transformers=[\n", " ('num', num_transformer, num_features),\n", " ('cat', cat_transformer, cat_features)\n", "])\n", "\n", "model = Pipeline(steps=[\n", " ('preprocessor', preprocessor),\n", " ('classifier', SVC()),\n", "])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This difference is not drastic, as we can see in the plot above.\n", "That changes however, when we have minority classes with much less data than the majority class.\n", "\n", "Either way it's worth it to keep in mind that stratification exists. The `stratify=` keyword takes any type of vector as long as it matches the dimension of the dataframe.\n", "\n", "## Cross-Validation\n", "Cross-validation is often considered the gold standard in statistical applications and machine learning.\n", "\n", "Cross-validation splits the data into folds, of which one is held out as the validation set and the rest is used to train.\n", "Subsequently, models are trained on the other folds in a round-robin style. That way we have models that are trained and evaluated on every sample of the dataset.\n", "![Scikit-learn cross validation](https://scikit-learn.org/stable/_images/grid_search_cross_validation.png)\n", "*Scikit-learn cross-validation schema. [[Source](https://scikit-learn.org/stable/modules/cross_validation.html)]*\n", "\n", "Cross-validation is particularly useful when we don't have a lot of data or the data is highly heterogeneous." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.033107Z", "iopub.status.busy": "2022-12-13T01:42:12.033107Z", "iopub.status.idle": "2022-12-13T01:42:12.061074Z", "shell.execute_reply": "2022-12-13T01:42:12.060572Z" }, "tags": [] }, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'sklearn'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [10], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmodel_selection\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m cross_val_score\n\u001b[0;32m 3\u001b[0m scores \u001b[38;5;241m=\u001b[39m cross_val_score(model, X_train, y_train, cv\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m5\u001b[39m)\n\u001b[0;32m 4\u001b[0m scores\n", "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sklearn'" ] } ], "source": [ "from sklearn.model_selection import cross_val_score\n", "\n", "scores = cross_val_score(model, X_train, y_train, cv=5)\n", "scores" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.063573Z", "iopub.status.busy": "2022-12-13T01:42:12.063573Z", "iopub.status.idle": "2022-12-13T01:42:12.092079Z", "shell.execute_reply": "2022-12-13T01:42:12.091579Z" }, "tags": [] }, "outputs": [ { "ename": "NameError", "evalue": "name 'scores' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [11], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[43mscores\u001b[49m\u001b[38;5;241m.\u001b[39mmean()\u001b[38;5;132;01m:\u001b[39;00m\u001b[38;5;124m0.2f\u001b[39m\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m accuracy with a standard deviation of \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mscores\u001b[38;5;241m.\u001b[39mstd()\u001b[38;5;132;01m:\u001b[39;00m\u001b[38;5;124m0.2f\u001b[39m\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n", "\u001b[1;31mNameError\u001b[0m: name 'scores' is not defined" ] } ], "source": [ "print(f\"{scores.mean():0.2f} accuracy with a standard deviation of {scores.std():0.2f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we know there are some folds this support-vector machine will do exceptional on and others it does quite well on only getting a few samples wrong." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model Evaluation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Brilliant! So let's recap for a moment what we have done so far, in preparation for our (final) **Model evaluation**.\n", "\n", "We have:\n", "\n", "- prepared the model pipeline: `sklearn.pipeline.Pipeline` with `preprocessor + model`\n", "- generated **train** and **test** data partitions (with stratification): `(X_train, y_train)` and `(X_test, y_test)`, respectively\n", " - stratification guaranteed that those partitions will retain class distributions\n", "- assessed model performance via **cross validation** (i.e. `cross_val_score`) on `X_train`(!!)\n", " - this had the objective of verifying model consistency on multiple data partitioning\n", "\n", "Now we need the complete our last step, namely \"assess how the model we chose in CV\" (we only had one model, so that was an easy choice :D ) will perform on _future data_!\n", "And we have a _candidate_ as representative for these data: `X_test`.\n", "\n", "Please note that `X_test` has never been used so far (as it should have!). The take away message here is: _generate test partition, and forget about it until the last step!_\n", "\n", "\n", "Thanks to `CV`, We have an indication of how the `SVC` classifier behaves on multiple \"version\" of the training set. We calculated an average score of `0.99` accuracy, therefore we decided this model is to be trusted for predictions on _unseen data_.\n", "\n", "Now all we need to do, is to prove this assertion.\n", "\n", "To do so we need to: \n", " - train a new model on the entire **training set**\n", " - evaluate it's performance on **test set** (using the metric of choice - presumably the same metric we chose in CV!)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.095579Z", "iopub.status.busy": "2022-12-13T01:42:12.095078Z", "iopub.status.idle": "2022-12-13T01:42:12.123083Z", "shell.execute_reply": "2022-12-13T01:42:12.122583Z" } }, "outputs": [ { "ename": "NameError", "evalue": "name 'Pipeline' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [12], line 2\u001b[0m\n\u001b[0;32m 1\u001b[0m \u001b[38;5;66;03m# training\u001b[39;00m\n\u001b[1;32m----> 2\u001b[0m model \u001b[38;5;241m=\u001b[39m \u001b[43mPipeline\u001b[49m(steps\u001b[38;5;241m=\u001b[39m[\n\u001b[0;32m 3\u001b[0m (\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mpreprocessor\u001b[39m\u001b[38;5;124m'\u001b[39m, preprocessor),\n\u001b[0;32m 4\u001b[0m (\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mclassifier\u001b[39m\u001b[38;5;124m'\u001b[39m, SVC()),\n\u001b[0;32m 5\u001b[0m ])\n\u001b[0;32m 6\u001b[0m classifier \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mfit(X_train, y_train)\n", "\u001b[1;31mNameError\u001b[0m: name 'Pipeline' is not defined" ] } ], "source": [ "# training\n", "model = Pipeline(steps=[\n", " ('preprocessor', preprocessor),\n", " ('classifier', SVC()),\n", "])\n", "classifier = model.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.126084Z", "iopub.status.busy": "2022-12-13T01:42:12.126084Z", "iopub.status.idle": "2022-12-13T01:42:12.154089Z", "shell.execute_reply": "2022-12-13T01:42:12.153589Z" } }, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'sklearn'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [13], line 2\u001b[0m\n\u001b[0;32m 1\u001b[0m \u001b[38;5;66;03m# Model evaluation\u001b[39;00m\n\u001b[1;32m----> 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmetrics\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m accuracy_score\n\u001b[0;32m 4\u001b[0m y_pred \u001b[38;5;241m=\u001b[39m classifier\u001b[38;5;241m.\u001b[39mpredict(X_test)\n\u001b[0;32m 5\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mTEST ACC: \u001b[39m\u001b[38;5;124m\"\u001b[39m, accuracy_score(y_true\u001b[38;5;241m=\u001b[39my_test, y_pred\u001b[38;5;241m=\u001b[39my_pred))\n", "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sklearn'" ] } ], "source": [ "# Model evaluation\n", "from sklearn.metrics import accuracy_score\n", "\n", "y_pred = classifier.predict(X_test)\n", "print(\"TEST ACC: \", accuracy_score(y_true=y_test, y_pred=y_pred))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can finally say that we have concluded our model evaluation - with a fantastic score of `0.96` Accuracy on the test set." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Choosing the appropriate Evaluation Metric" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok, now for the mere sake of considering a more realistic data scenario, let's pretend our reference dataset is composed by only samples from two (out of the three) classes we have. In particular, we will crafting our dataset by choosing the most and the least represented classes, respectively. \n", "\n", "The very idea is to explore whether the choice of appropriate metrics could make the difference in our machine learning models evaluation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's recall class distributions in our dataset:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.157591Z", "iopub.status.busy": "2022-12-13T01:42:12.157090Z", "iopub.status.idle": "2022-12-13T01:42:12.185068Z", "shell.execute_reply": "2022-12-13T01:42:12.184567Z" } }, "outputs": [ { "ename": "NameError", "evalue": "name 'y' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [14], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[43my\u001b[49m\u001b[38;5;241m.\u001b[39mreset_index()\u001b[38;5;241m.\u001b[39mgroupby([\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSpecies\u001b[39m\u001b[38;5;124m\"\u001b[39m])\u001b[38;5;241m.\u001b[39mcount()\n", "\u001b[1;31mNameError\u001b[0m: name 'y' is not defined" ] } ], "source": [ "y.reset_index().groupby([\"Species\"]).count()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So let's select samples from the first two classes, `Adelie Penguin` and `Chinstrap penguin`:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.188068Z", "iopub.status.busy": "2022-12-13T01:42:12.187569Z", "iopub.status.idle": "2022-12-13T01:42:12.200571Z", "shell.execute_reply": "2022-12-13T01:42:12.200070Z" } }, "outputs": [], "source": [ "samples = penguins[((penguins[\"Species\"].str.startswith(\"Adelie\")) | (penguins[\"Species\"].str.startswith(\"Chinstrap\")))]" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.203070Z", "iopub.status.busy": "2022-12-13T01:42:12.203070Z", "iopub.status.idle": "2022-12-13T01:42:12.216074Z", "shell.execute_reply": "2022-12-13T01:42:12.215573Z" } }, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "samples.shape[0] == 146 + 68 # quick verification" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To make things even harder for our machine learning model, let's also see if we could get rid of _clearly_ separating features in this toy dataset" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.219073Z", "iopub.status.busy": "2022-12-13T01:42:12.218573Z", "iopub.status.idle": "2022-12-13T01:42:12.247078Z", "shell.execute_reply": "2022-12-13T01:42:12.246578Z" } }, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'seaborn'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [17], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mseaborn\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01msns\u001b[39;00m\n\u001b[0;32m 3\u001b[0m pairplot_figure \u001b[38;5;241m=\u001b[39m sns\u001b[38;5;241m.\u001b[39mpairplot(samples, hue\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSpecies\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n", "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'seaborn'" ] } ], "source": [ "import seaborn as sns\n", "\n", "pairplot_figure = sns.pairplot(samples, hue=\"Species\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "OK so if we get to choose, we could definitely say that in this dataset, the `Flipper Length` in combination with the `Culmen Depth` leads to the hardest classification task for our machine learning model.\n", "\n", "Therefore, here is the plan:\n", "- we select only those to numerical features (_iow_ we will get rid of the `Culmen Lenght` feature)\n", "- we will apply an identical _Model evaluation_ pipeline as we did in our previous example\n", " - Cross Validation + Evaluation on Test set\n", "\n", "The very difference this time is that we will use multiple metrics to evaluate our model to prove our point on _carefully selecting evaluation metrics_." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.250078Z", "iopub.status.busy": "2022-12-13T01:42:12.249578Z", "iopub.status.idle": "2022-12-13T01:42:12.262580Z", "shell.execute_reply": "2022-12-13T01:42:12.262080Z" } }, "outputs": [], "source": [ "num_features = [\"Culmen Length (mm)\", \"Culmen Depth (mm)\", \"Flipper Length (mm)\"]\n", "selected_num_features = num_features[1:]\n", "cat_features = [\"Sex\"]\n", "features = selected_num_features + cat_features" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.264580Z", "iopub.status.busy": "2022-12-13T01:42:12.264580Z", "iopub.status.idle": "2022-12-13T01:42:12.293586Z", "shell.execute_reply": "2022-12-13T01:42:12.293087Z" } }, "outputs": [ { "ename": "NameError", "evalue": "name 'StandardScaler' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [19], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m num_transformer \u001b[38;5;241m=\u001b[39m \u001b[43mStandardScaler\u001b[49m()\n\u001b[0;32m 2\u001b[0m cat_transformer \u001b[38;5;241m=\u001b[39m OneHotEncoder(handle_unknown\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mignore\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m 4\u001b[0m preprocessor \u001b[38;5;241m=\u001b[39m ColumnTransformer(transformers\u001b[38;5;241m=\u001b[39m[\n\u001b[0;32m 5\u001b[0m (\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mnum\u001b[39m\u001b[38;5;124m'\u001b[39m, num_transformer, selected_num_features), \u001b[38;5;66;03m# note here, we will only preprocess selected numerical features\u001b[39;00m\n\u001b[0;32m 6\u001b[0m (\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcat\u001b[39m\u001b[38;5;124m'\u001b[39m, cat_transformer, cat_features)\n\u001b[0;32m 7\u001b[0m ])\n", "\u001b[1;31mNameError\u001b[0m: name 'StandardScaler' is not defined" ] } ], "source": [ "num_transformer = StandardScaler()\n", "cat_transformer = OneHotEncoder(handle_unknown='ignore')\n", "\n", "preprocessor = ColumnTransformer(transformers=[\n", " ('num', num_transformer, selected_num_features), # note here, we will only preprocess selected numerical features\n", " ('cat', cat_transformer, cat_features)\n", "])\n", "\n", "model = Pipeline(steps=[\n", " ('preprocessor', preprocessor),\n", " ('classifier', SVC()),\n", "])" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.296586Z", "iopub.status.busy": "2022-12-13T01:42:12.296087Z", "iopub.status.idle": "2022-12-13T01:42:12.324591Z", "shell.execute_reply": "2022-12-13T01:42:12.324091Z" } }, "outputs": [ { "ename": "NameError", "evalue": "name 'target' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [20], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m X, y \u001b[38;5;241m=\u001b[39m samples[features], samples[\u001b[43mtarget\u001b[49m[\u001b[38;5;241m0\u001b[39m]]\n\u001b[0;32m 2\u001b[0m X_train, X_test, y_train, y_test \u001b[38;5;241m=\u001b[39m train_test_split(X, y, train_size\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m.7\u001b[39m, random_state\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m42\u001b[39m, stratify\u001b[38;5;241m=\u001b[39my) \u001b[38;5;66;03m# we also stratify on classes\u001b[39;00m\n", "\u001b[1;31mNameError\u001b[0m: name 'target' is not defined" ] } ], "source": [ "X, y = samples[features], samples[target[0]]\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=.7, random_state=42, stratify=y) # we also stratify on classes" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.327092Z", "iopub.status.busy": "2022-12-13T01:42:12.327092Z", "iopub.status.idle": "2022-12-13T01:42:12.355597Z", "shell.execute_reply": "2022-12-13T01:42:12.355096Z" } }, "outputs": [ { "ename": "NameError", "evalue": "name 'y_train' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [21], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[43my_train\u001b[49m\u001b[38;5;241m.\u001b[39mreset_index()\u001b[38;5;241m.\u001b[39mgroupby(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSpecies\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mcount()\n", "\u001b[1;31mNameError\u001b[0m: name 'y_train' is not defined" ] } ], "source": [ "y_train.reset_index().groupby(\"Species\").count()" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.358597Z", "iopub.status.busy": "2022-12-13T01:42:12.358097Z", "iopub.status.idle": "2022-12-13T01:42:12.386314Z", "shell.execute_reply": "2022-12-13T01:42:12.385813Z" } }, "outputs": [ { "ename": "NameError", "evalue": "name 'y_test' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [22], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[43my_test\u001b[49m\u001b[38;5;241m.\u001b[39mreset_index()\u001b[38;5;241m.\u001b[39mgroupby(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSpecies\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mcount()\n", "\u001b[1;31mNameError\u001b[0m: name 'y_test' is not defined" ] } ], "source": [ "y_test.reset_index().groupby(\"Species\").count()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In our evaluation pipeline we will be using keep record both **accuracy** (`ACC`) and **matthew correlation coefficient** (`MCC`)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.389314Z", "iopub.status.busy": "2022-12-13T01:42:12.389314Z", "iopub.status.idle": "2022-12-13T01:42:12.417304Z", "shell.execute_reply": "2022-12-13T01:42:12.416804Z" } }, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'sklearn'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [23], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmodel_selection\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m cross_validate\n\u001b[0;32m 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmetrics\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m make_scorer\n\u001b[0;32m 3\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmetrics\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m matthews_corrcoef \u001b[38;5;28;01mas\u001b[39;00m mcc\n", "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sklearn'" ] } ], "source": [ "from sklearn.model_selection import cross_validate\n", "from sklearn.metrics import make_scorer\n", "from sklearn.metrics import matthews_corrcoef as mcc\n", "from sklearn.metrics import accuracy_score as acc\n", "\n", "mcc_scorer = make_scorer(mcc)\n", "acc_scorer = make_scorer(acc)\n", "scores = cross_validate(model, X_train, y_train, cv=5,\n", " scoring={\"MCC\": mcc_scorer, \"ACC\": acc_scorer})\n", "scores" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.420305Z", "iopub.status.busy": "2022-12-13T01:42:12.419805Z", "iopub.status.idle": "2022-12-13T01:42:12.448324Z", "shell.execute_reply": "2022-12-13T01:42:12.447823Z" } }, "outputs": [ { "ename": "NameError", "evalue": "name 'scores' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [24], line 3\u001b[0m\n\u001b[0;32m 1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mnp\u001b[39;00m\n\u001b[1;32m----> 3\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAvg ACC in CV: \u001b[39m\u001b[38;5;124m\"\u001b[39m, np\u001b[38;5;241m.\u001b[39maverage(\u001b[43mscores\u001b[49m[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtest_ACC\u001b[39m\u001b[38;5;124m\"\u001b[39m]))\n\u001b[0;32m 4\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mAvg MCC in CV: \u001b[39m\u001b[38;5;124m\"\u001b[39m, np\u001b[38;5;241m.\u001b[39maverage(scores[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtest_MCC\u001b[39m\u001b[38;5;124m\"\u001b[39m]))\n", "\u001b[1;31mNameError\u001b[0m: name 'scores' is not defined" ] } ], "source": [ "import numpy as np\n", "\n", "print(\"Avg ACC in CV: \", np.average(scores[\"test_ACC\"]))\n", "print(\"Avg MCC in CV: \", np.average(scores[\"test_MCC\"]))" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.451324Z", "iopub.status.busy": "2022-12-13T01:42:12.450824Z", "iopub.status.idle": "2022-12-13T01:42:12.479064Z", "shell.execute_reply": "2022-12-13T01:42:12.478563Z" } }, "outputs": [ { "ename": "NameError", "evalue": "name 'model' is not defined", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [25], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m model \u001b[38;5;241m=\u001b[39m \u001b[43mmodel\u001b[49m\u001b[38;5;241m.\u001b[39mfit(X_train, y_train)\n\u001b[0;32m 3\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mACC: \u001b[39m\u001b[38;5;124m\"\u001b[39m, acc_scorer(model, X_test, y_test))\n\u001b[0;32m 4\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mMCC: \u001b[39m\u001b[38;5;124m\"\u001b[39m, mcc_scorer(model, X_test, y_test))\n", "\u001b[1;31mNameError\u001b[0m: name 'model' is not defined" ] } ], "source": [ "model = model.fit(X_train, y_train)\n", "\n", "print(\"ACC: \", acc_scorer(model, X_test, y_test))\n", "print(\"MCC: \", mcc_scorer(model, X_test, y_test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To see exactly what happened, let's have a look at the **Confusion matrix**" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.482064Z", "iopub.status.busy": "2022-12-13T01:42:12.482064Z", "iopub.status.idle": "2022-12-13T01:42:12.510068Z", "shell.execute_reply": "2022-12-13T01:42:12.509568Z" } }, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'sklearn'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [26], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmetrics\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ConfusionMatrixDisplay\n\u001b[0;32m 2\u001b[0m fig, ax \u001b[38;5;241m=\u001b[39m plt\u001b[38;5;241m.\u001b[39msubplots(figsize\u001b[38;5;241m=\u001b[39m(\u001b[38;5;241m15\u001b[39m, \u001b[38;5;241m10\u001b[39m))\n\u001b[0;32m 3\u001b[0m ConfusionMatrixDisplay\u001b[38;5;241m.\u001b[39mfrom_estimator(model, X_test, y_test, ax\u001b[38;5;241m=\u001b[39max)\n", "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sklearn'" ] } ], "source": [ "from sklearn.metrics import ConfusionMatrixDisplay\n", "fig, ax = plt.subplots(figsize=(15, 10))\n", "ConfusionMatrixDisplay.from_estimator(model, X_test, y_test, ax=ax)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As expected, the model did a pretty bad job in classifying *Chinstrap Penguins* and the `MCC` was able to catch that, whilst `ACC` could not as it only considers correctly classified samples!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Time-series Validation\n", "\n", "But validation can get tricky if time gets involved.\n", "\n", "Imagine we measured the growth of baby penguin Hank over time and wanted to us machine learning to project the development of Hank. Then our data suddenly isn't i.i.d. anymore, since it is dependent in the time dimension.\n", "\n", "Were we to split our data randomly for our training and test set, we would test on data points that lie in between training points, where even a simple linear interpolation can do a fairly decent job.\n", "\n", "Therefor, we need to split our measurements along the time axis\n", "![Scikit-learn time series validation](https://scikit-learn.org/stable/_images/sphx_glr_plot_cv_indices_013.png)\n", "*Scikit-learn Time Series CV [[Source]](https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split).*" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.513069Z", "iopub.status.busy": "2022-12-13T01:42:12.512568Z", "iopub.status.idle": "2022-12-13T01:42:12.541078Z", "shell.execute_reply": "2022-12-13T01:42:12.540577Z" }, "lines_to_next_cell": 2 }, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'sklearn'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [27], line 2\u001b[0m\n\u001b[0;32m 1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mnp\u001b[39;00m\n\u001b[1;32m----> 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmodel_selection\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m TimeSeriesSplit\n\u001b[0;32m 4\u001b[0m X \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39marray([[\u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m], [\u001b[38;5;241m3\u001b[39m, \u001b[38;5;241m4\u001b[39m], [\u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m], [\u001b[38;5;241m3\u001b[39m, \u001b[38;5;241m4\u001b[39m], [\u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m], [\u001b[38;5;241m3\u001b[39m, \u001b[38;5;241m4\u001b[39m]])\n\u001b[0;32m 5\u001b[0m y \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39marray([\u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m3\u001b[39m, \u001b[38;5;241m4\u001b[39m, \u001b[38;5;241m5\u001b[39m, \u001b[38;5;241m6\u001b[39m])\n", "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sklearn'" ] } ], "source": [ "import numpy as np\n", "from sklearn.model_selection import TimeSeriesSplit\n", "\n", "X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])\n", "y = np.array([1, 2, 3, 4, 5, 6])\n", "tscv = TimeSeriesSplit(n_splits=3)\n", "print(tscv)\n", "\n", "for train, test in tscv.split(X):\n", " print(\"%s %s\" % (train, test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Spatial Validation\n", "\n", "Spatial data, like maps and satellite data has a similar problem.\n", "\n", "Here the data is correlated in the spatial dimension. However, we can mitigate the effect by supplying a group. In this simple example I used continents, but it's possible to group by bins on a lat-lon grid as well. \n", "\n", "Here especially, a cross-validation scheme is very important, as it is used to validate against every area on your map at least once." ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "execution": { "iopub.execute_input": "2022-12-13T01:42:12.544078Z", "iopub.status.busy": "2022-12-13T01:42:12.543578Z", "iopub.status.idle": "2022-12-13T01:42:12.572069Z", "shell.execute_reply": "2022-12-13T01:42:12.571568Z" } }, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'sklearn'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[1;32mIn [28], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01msklearn\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmodel_selection\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m GroupKFold\n\u001b[0;32m 3\u001b[0m X \u001b[38;5;241m=\u001b[39m [\u001b[38;5;241m0.1\u001b[39m, \u001b[38;5;241m0.2\u001b[39m, \u001b[38;5;241m2.2\u001b[39m, \u001b[38;5;241m2.4\u001b[39m, \u001b[38;5;241m2.3\u001b[39m, \u001b[38;5;241m4.55\u001b[39m, \u001b[38;5;241m5.8\u001b[39m, \u001b[38;5;241m0.001\u001b[39m]\n\u001b[0;32m 4\u001b[0m y \u001b[38;5;241m=\u001b[39m [\u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m4\u001b[39m, \u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m2\u001b[39m, \u001b[38;5;241m3\u001b[39m, \u001b[38;5;241m4\u001b[39m, \u001b[38;5;241m5\u001b[39m]\n", "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sklearn'" ] } ], "source": [ "from sklearn.model_selection import GroupKFold\n", "\n", "X = [0.1, 0.2, 2.2, 2.4, 2.3, 4.55, 5.8, 0.001]\n", "y = [1, 2, 4, 2, 2, 3, 4, 5]\n", "groups = [\"Europe\", \"Africa\", \"Africa\", \"Africa\", \"America\", \"Asia\", \"Asia\", \"Europe\"]\n", "cv = GroupKFold(n_splits=4)\n", "for train, test in cv.split(X, y, groups=groups):\n", " print(\"%s %s\" % (train, test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion\n", "\n", "A simple random split of the data works on toy problems, but real-world data is rarely i.i.d.\n", "\n", "We looked at different ways that we can evaluate models that violate the i.i.d. assumption and how we can still evaluate their performance on unseen data without obtaining artificially high scores.\n", "\n", "\n", "
\n", "Tip: Artificially high scores from leakage and cheating mean that our scientific finding hold no merit. This is often caught in review and prolongs the review process (which no one wants). But in the worst case can lead to diverting research funds in a wrong direction and paper redactions / corrections.
" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.8" }, "vscode": { "interpreter": { "hash": "d7369b48cea8bb1af6d88d25f2646d14ea11b68d7457d74f06fbf0d68480668d" } } }, "nbformat": 4, "nbformat_minor": 4 }