{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Solving classification problems with CatBoost" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/catboost/tutorials/blob/master/events/pydata_la_oct_21_2018.ipynb)\n", "\n", "In this tutorial we will use dataset Amazon Employee Access Challenge from [Kaggle](https://www.kaggle.com) competition for our experiments. Data can be downloaded [here](https://www.kaggle.com/c/amazon-employee-access-challenge/data)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Libraries installation" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "#!pip install --user --upgrade catboost\n", "#!pip install --user --upgrade ipywidgets\n", "#!pip install shap\n", "#!pip install sklearn\n", "#!pip install --upgrade numpy\n", "#!pip install --upgrade pandas\n", "#!jupyter nbextension enable --py widgetsnbextension" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.11.2\n", "Python 2.7.12\r\n" ] } ], "source": [ "import catboost\n", "print(catboost.__version__)\n", "!python --version" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading the data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The most simple way — read everything in pandas data frame" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import os\n", "import numpy as np\n", "np.set_printoptions(precision=4)\n", "import catboost\n", "from catboost import *\n", "from catboost import datasets" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# import ssl\n", "# ssl._create_default_https_context = ssl._create_unverified_context" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "(train_df, test_df) = catboost.datasets.amazon()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ACTIONRESOURCEMGR_IDROLE_ROLLUP_1ROLE_ROLLUP_2ROLE_DEPTNAMEROLE_TITLEROLE_FAMILY_DESCROLE_FAMILYROLE_CODE
013935385475117961118300123472117905117906290919117908
11171831540117961118343123125118536118536308574118539
21367241445711821911822011788411787926795219721117880
31361355396117961118343119993118321240983290919118322
4142680590511792911793011956911932312393219793119325
\n", "
" ], "text/plain": [ " ACTION RESOURCE MGR_ID ROLE_ROLLUP_1 ROLE_ROLLUP_2 ROLE_DEPTNAME \\\n", "0 1 39353 85475 117961 118300 123472 \n", "1 1 17183 1540 117961 118343 123125 \n", "2 1 36724 14457 118219 118220 117884 \n", "3 1 36135 5396 117961 118343 119993 \n", "4 1 42680 5905 117929 117930 119569 \n", "\n", " ROLE_TITLE ROLE_FAMILY_DESC ROLE_FAMILY ROLE_CODE \n", "0 117905 117906 290919 117908 \n", "1 118536 118536 308574 118539 \n", "2 117879 267952 19721 117880 \n", "3 118321 240983 290919 118322 \n", "4 119323 123932 19793 119325 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Preparing your data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Label values extraction" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "y = train_df.ACTION\n", "X = train_df.drop('ACTION', axis=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Categorical features declaration" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0, 1, 2, 3, 4, 5, 6, 7, 8]\n" ] } ], "source": [ "cat_features = list(range(0, X.shape[1]))\n", "print(cat_features)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looking on label balance in dataset" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('Labels: ', set([0, 1]))\n", "Zero count = 1897, One count = 30872\n" ] } ], "source": [ "print('Labels: ', set(y))\n", "print('Zero count = ' + str(len(y) - sum(y)) + ', One count = ' + str(sum(y)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To train model in CatBoost we need to create wrapper class for data: Pool.\n", "This class stores the data in CatBoost internal format.\n", "\n", "There exists several ways to create pool. \n", "The most simple one: create it from pandas dataframe or numpy array" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "pool1 = Pool(data=X, \n", " label=y,\n", " cat_features=cat_features) #Indicies of categorical columns in X" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This approach is not the most efficient, especially for big dataset: we'll need to copy everything from pandas to our internal format.\n", "\n", "So CatBoost could create Pool direclty from file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lets look how we could load Pool from file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Firstly, lets save data frame to disk" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "dataset_dir = './amazon'\n", "if not os.path.exists(dataset_dir):\n", " os.makedirs(dataset_dir)\n", "\n", "# We will be able to work with files with/without header and with different separators.\n", "train_df.to_csv(os.path.join(dataset_dir, 'train.tsv'), index=False, sep='\\t', header=False)\n", "test_df.to_csv(os.path.join(dataset_dir, 'test.tsv'), index=False, sep='\\t', header=False)\n", "\n", "train_df.to_csv(os.path.join(dataset_dir, 'train.csv'), index=False, sep=',', header=True)\n", "test_df.to_csv(os.path.join(dataset_dir, 'test.csv'), index=False, sep=',', header=True)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1\t39353\t85475\t117961\t118300\t123472\t117905\t117906\t290919\t117908\r\n", "1\t17183\t1540\t117961\t118343\t123125\t118536\t118536\t308574\t118539\r\n" ] } ], "source": [ "!head -n2 amazon/train.tsv" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ACTION,RESOURCE,MGR_ID,ROLE_ROLLUP_1,ROLE_ROLLUP_2,ROLE_DEPTNAME,ROLE_TITLE,ROLE_FAMILY_DESC,ROLE_FAMILY,ROLE_CODE\r\n", "1,39353,85475,117961,118300,123472,117905,117906,290919,117908\r\n", "1,17183,1540,117961,118343,123125,118536,118536,308574,118539\r\n" ] } ], "source": [ "!head -n3 amazon/train.csv" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we have dataset in 2 different formats:\n", "\n", "1) tab-separated without header\n", "\n", "2) comma-separated with header\n", "\n", "\n", "CatBoost, like pandas, could load data from different formats, we just need to pass proper options\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before we load data, we need to set types of each column\n", "\n", "Also, we need to specify columns type. For this CatBoost uses special file, column description\n", "And we have helper-function to easily do this" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "from catboost.utils import create_cd\n", "\n", "feature_names = dict()\n", "for column, name in enumerate(train_df):\n", " if column == 0:\n", " continue\n", " feature_names[column] = name\n", " \n", "create_cd(\n", " label=0, \n", " cat_features=list(range(1, train_df.columns.shape[0])),\n", " feature_names=feature_names,\n", " output_path=os.path.join(dataset_dir, 'train.cd')\n", ")" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\tLabel\t\r\n", "1\tCateg\tRESOURCE\r\n", "2\tCateg\tMGR_ID\r\n", "3\tCateg\tROLE_ROLLUP_1\r\n", "4\tCateg\tROLE_ROLLUP_2\r\n", "5\tCateg\tROLE_DEPTNAME\r\n", "6\tCateg\tROLE_TITLE\r\n", "7\tCateg\tROLE_FAMILY_DESC\r\n", "8\tCateg\tROLE_FAMILY\r\n", "9\tCateg\tROLE_CODE\r\n" ] } ], "source": [ "!cat amazon/train.cd" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "create_cd(\n", " label=0, \n", " cat_features=list(range(1, train_df.columns.shape[0])),\n", " # feature_names=feature_names,\n", " output_path=os.path.join(dataset_dir, 'train_without_names.cd')\n", ")" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\tLabel\t\r\n", "1\tCateg\t\r\n", "2\tCateg\t\r\n", "3\tCateg\t\r\n", "4\tCateg\t\r\n", "5\tCateg\t\r\n", "6\tCateg\t\r\n", "7\tCateg\t\r\n", "8\tCateg\t\r\n", "9\tCateg\t\r\n" ] } ], "source": [ "!cat amazon/train_without_names.cd" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "create_cd(\n", " label=0, \n", " cat_features=list(range(2, train_df.columns.shape[0])),\n", " feature_names=feature_names,\n", " output_path=os.path.join(dataset_dir, 'train_with_num.cd')\n", ")" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\tLabel\t\r\n", "1\tNum\tRESOURCE\r\n", "2\tCateg\tMGR_ID\r\n", "3\tCateg\tROLE_ROLLUP_1\r\n", "4\tCateg\tROLE_ROLLUP_2\r\n", "5\tCateg\tROLE_DEPTNAME\r\n", "6\tCateg\tROLE_TITLE\r\n", "7\tCateg\tROLE_FAMILY_DESC\r\n", "8\tCateg\tROLE_FAMILY\r\n", "9\tCateg\tROLE_CODE\r\n" ] } ], "source": [ "!cat amazon/train_with_num.cd" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "? create_cd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's load pool from file now:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "pool2 = Pool(\n", " data=os.path.join(dataset_dir, 'train.tsv'), \n", " #delimiter=',', \n", " column_description=os.path.join(dataset_dir, 'train.cd'),\n", " # has_header=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Loading pool from file is the fastest way to build Pool if you don't have Pool in RAM yet" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Exercices: load the same pools from csv file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, if you want maximum performance and you data is already in RAM, than in some cases we could do better, than simply passing dataframe to Pool constructor\n", "\n", "We have class FeaturesData that is a fast way to pass data from numpy matrices to catboost" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "? FeaturesData" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "# Fastest way to create a Pool is to create it from numpy matrix. This way should be used if you want fast predictions\n", "# or fastest way to load the data in python.\n", "\n", "X_prepared = X.values.astype(str).astype(object)\n", "# For FeaturesData class categorial features must have type str\n", "\n", "pool3 = Pool(\n", " data=FeaturesData(cat_feature_data=X_prepared, cat_feature_names=list(X)),\n", " label=y.values\n", ")\n" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset shape\n", "dataset 1:(32769, 9)\n", "dataset 2:(32769, 9)\n", "dataset 3:(32769, 9)\n", "\n", "\n", "Column names\n", "dataset 1:\n", "['RESOURCE', 'MGR_ID', 'ROLE_ROLLUP_1', 'ROLE_ROLLUP_2', 'ROLE_DEPTNAME', 'ROLE_TITLE', 'ROLE_FAMILY_DESC', 'ROLE_FAMILY', 'ROLE_CODE']\n", "\n", "dataset 2:\n", "['RESOURCE', 'MGR_ID', 'ROLE_ROLLUP_1', 'ROLE_ROLLUP_2', 'ROLE_DEPTNAME', 'ROLE_TITLE', 'ROLE_FAMILY_DESC', 'ROLE_FAMILY', 'ROLE_CODE']\n", "\n", "dataset 3:\n", "['RESOURCE', 'MGR_ID', 'ROLE_ROLLUP_1', 'ROLE_ROLLUP_2', 'ROLE_DEPTNAME', 'ROLE_TITLE', 'ROLE_FAMILY_DESC', 'ROLE_FAMILY', 'ROLE_CODE']\n" ] } ], "source": [ "print('Dataset shape')\n", "print('dataset 1:' + str(pool1.shape) + '\\ndataset 2:' + str(pool2.shape) + \n", " '\\ndataset 3:' + str(pool3.shape))\n", "\n", "print('\\n')\n", "print('Column names')\n", "print('dataset 1:')\n", "print(pool1.get_feature_names()) \n", "print('\\ndataset 2:')\n", "print(pool2.get_feature_names())\n", "print('\\ndataset 3:')\n", "print(pool3.get_feature_names())\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Split your data into train and validation" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/annaveronika/.local/lib/python2.7/site-packages/sklearn/model_selection/_split.py:2179: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.\n", " FutureWarning)\n" ] } ], "source": [ "from sklearn.model_selection import train_test_split\n", "X_train, X_validation, y_train, y_validation = train_test_split(X, y, train_size=0.8, random_state=1234)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
RESOURCEMGR_IDROLE_ROLLUP_1ROLE_ROLLUP_2ROLE_DEPTNAMEROLE_TITLEROLE_FAMILY_DESCROLE_FAMILYROLE_CODE
1492674463105908117961118225129617118702132654118704118705
794017278120340120342120343119076118834311236118424118836
247687932517733117961118300119984118890125128118398118892
163351733075117961117962120677120357120678118424120359
2743156723745117961118300118360124435118362118363124436
\n", "
" ], "text/plain": [ " RESOURCE MGR_ID ROLE_ROLLUP_1 ROLE_ROLLUP_2 ROLE_DEPTNAME \\\n", "14926 74463 105908 117961 118225 129617 \n", "7940 17278 120340 120342 120343 119076 \n", "24768 79325 17733 117961 118300 119984 \n", "1633 5173 3075 117961 117962 120677 \n", "2743 15672 3745 117961 118300 118360 \n", "\n", " ROLE_TITLE ROLE_FAMILY_DESC ROLE_FAMILY ROLE_CODE \n", "14926 118702 132654 118704 118705 \n", "7940 118834 311236 118424 118836 \n", "24768 118890 125128 118398 118892 \n", "1633 120357 120678 118424 120359 \n", "2743 124435 118362 118363 124436 " ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train.head()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(26215, 9)" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_train.shape" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(6554, 9)" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_validation.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Selecting the objective function" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Possible options for binary classification:\n", "\n", "`Logloss`\n", "\n", "`CrossEntropy` for probabilities in target" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "\n", "model = CatBoostClassifier(\n", " iterations=5,\n", " learning_rate=0.1,\n", " #loss_function='Logloss',\n", " #loss_function='CrossEntropy'\n", ")\n", "\n", "train_pool = Pool(data=X_train, \n", " label=y_train, \n", " cat_features=cat_features)\n", "\n", "validation_pool = Pool(data=X_validation, \n", " label=y_validation, \n", " cat_features=cat_features)\n", "model.fit(\n", " train_pool,\n", " eval_set=validation_pool,\n", " verbose=False\n", ")" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model is fitted: True\n", "Model params:\n", "{'learning_rate': 0.1, 'loss_function': 'Logloss', 'iterations': 5}\n" ] } ], "source": [ "print('Model is fitted: ' + str(model.is_fitted()))\n", "print('Model params:')\n", "print(model.get_params())" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = CatBoostClassifier(\n", " iterations=5,\n", " learning_rate=0.1,\n", " #loss_function='Logloss',\n", " #loss_function='CrossEntropy'\n", ")\n", "\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False\n", ")" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model is fitted: True\n", "Model params:\n", "{'learning_rate': 0.1, 'loss_function': 'Logloss', 'iterations': 5}\n" ] } ], "source": [ "print('Model is fitted: ' + str(model.is_fitted()))\n", "print('Model params:')\n", "print(model.get_params())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stdout of the training" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Learning rate set to 0.5\n", "0:\tlearn: 0.2985954\ttest: 0.2997087\tbest: 0.2997087 (0)\ttotal: 56.7ms\tremaining: 794ms\n", "1:\tlearn: 0.2148559\ttest: 0.2108328\tbest: 0.2108328 (1)\ttotal: 116ms\tremaining: 757ms\n", "2:\tlearn: 0.1841475\ttest: 0.1719133\tbest: 0.1719133 (2)\ttotal: 187ms\tremaining: 748ms\n", "3:\tlearn: 0.1760205\ttest: 0.1608898\tbest: 0.1608898 (3)\ttotal: 260ms\tremaining: 715ms\n", "4:\tlearn: 0.1724111\ttest: 0.1553231\tbest: 0.1553231 (4)\ttotal: 321ms\tremaining: 642ms\n", "5:\tlearn: 0.1676418\ttest: 0.1498856\tbest: 0.1498856 (5)\ttotal: 402ms\tremaining: 603ms\n", "6:\tlearn: 0.1660562\ttest: 0.1478320\tbest: 0.1478320 (6)\ttotal: 476ms\tremaining: 544ms\n", "7:\tlearn: 0.1653090\ttest: 0.1467205\tbest: 0.1467205 (7)\ttotal: 558ms\tremaining: 488ms\n", "8:\tlearn: 0.1649062\ttest: 0.1465720\tbest: 0.1465720 (8)\ttotal: 606ms\tremaining: 404ms\n", "9:\tlearn: 0.1643222\ttest: 0.1463720\tbest: 0.1463720 (9)\ttotal: 685ms\tremaining: 342ms\n", "10:\tlearn: 0.1633503\ttest: 0.1450909\tbest: 0.1450909 (10)\ttotal: 752ms\tremaining: 273ms\n", "11:\tlearn: 0.1632233\ttest: 0.1451206\tbest: 0.1450909 (10)\ttotal: 815ms\tremaining: 204ms\n", "12:\tlearn: 0.1626563\ttest: 0.1453396\tbest: 0.1450909 (10)\ttotal: 879ms\tremaining: 135ms\n", "13:\tlearn: 0.1620230\ttest: 0.1445742\tbest: 0.1445742 (13)\ttotal: 941ms\tremaining: 67.2ms\n", "14:\tlearn: 0.1614001\ttest: 0.1444070\tbest: 0.1444070 (14)\ttotal: 1.01s\tremaining: 0us\n", "\n", "bestTest = 0.1444069686\n", "bestIteration = 14\n", "\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=15,\n", " #verbose=5,\n", " logging_level='Verbose'\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", ")" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Learning rate set to 0.5\n", "\n", "{ROLE_ROLLUP_1} pr2 tb0 type0, border=11 score 72.71470337\n", "{ROLE_ROLLUP_1} pr1 tb0 type0, border=4 score 73.13891169\n", "{ROLE_ROLLUP_1, ROLE_ROLLUP_2} pr2 tb0 type0, border=11 score 72.65449439\n", "{ROLE_ROLLUP_1} pr1 tb0 type0, border=2 score 72.85344413\n", " tensor 3 is redundant, remove it and stop\n", "0:\tlearn: 0.2985954\ttest: 0.2997087\tbest: 0.2997087 (0)\ttotal: 55.5ms\tremaining: 776ms\n", "\n", "{ROLE_TITLE} pr2 tb0 type0, border=13 score 22.73425202\n", "{ROLE_DEPTNAME, ROLE_TITLE} pr2 tb0 type0, border=10 score 23.43007817\n", "{ROLE_DEPTNAME, ROLE_TITLE} pr1 tb0 type0, border=3 score 25.16989885\n", "{ROLE_TITLE} pr0 tb0 type0, border=2 score 25.40915672\n", "{ROLE_TITLE} pr0 tb0 type0, border=14 score 25.17279721\n", " tensor 4 is redundant, remove it and stop\n", "1:\tlearn: 0.2148559\ttest: 0.2108328\tbest: 0.2108328 (1)\ttotal: 118ms\tremaining: 769ms\n", "\n", "{MGR_ID} pr2 tb0 type0, border=11 score 10.02687185\n", "{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=11 score 13.662788\n", "{ROLE_ROLLUP_2} pr2 tb0 type0, border=13 score 15.3551266\n", "{ROLE_ROLLUP_1} pr2 tb0 type0, border=8 score 15.4183659\n", "{MGR_ID, ROLE_FAMILY_DESC} pr1 tb0 type0, border=12 score 15.60179644\n", "{ROLE_TITLE} pr0 tb0 type0, border=3 score 15.46732072\n", "2:\tlearn: 0.1841475\ttest: 0.1719133\tbest: 0.1719133 (2)\ttotal: 187ms\tremaining: 748ms\n", "\n", "{MGR_ID} pr1 tb0 type0, border=6 score 5.040708161\n", "{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=11 score 7.163894\n", "{MGR_ID, ROLE_TITLE, ROLE_FAMILY_DESC} pr2 tb0 type0, border=12 score 7.509251314\n", "{ROLE_TITLE} pr1 tb0 type0, border=3 score 8.40093362\n", "{ROLE_DEPTNAME, ROLE_TITLE} pr2 tb0 type0, border=2 score 8.54481688\n", "{MGR_ID} pr2 tb0 type0, border=5 score 8.489125383\n", "3:\tlearn: 0.1760205\ttest: 0.1608898\tbest: 0.1608898 (3)\ttotal: 253ms\tremaining: 694ms\n", "\n", "{MGR_ID} pr2 tb0 type0, border=9 score 3.878720433\n", "{RESOURCE} pr2 tb0 type0, border=10 score 4.787399099\n", "{MGR_ID, ROLE_FAMILY_DESC} pr2 tb0 type0, border=14 score 5.664541783\n", "{ROLE_CODE} pr2 tb0 type0, border=2 score 5.968804419\n", " tensor 3 is redundant, remove it and stop\n", "4:\tlearn: 0.1724111\ttest: 0.1553231\tbest: 0.1553231 (4)\ttotal: 304ms\tremaining: 607ms\n", "\n", "{RESOURCE} pr2 tb0 type0, border=11 score 2.468191783\n", "{RESOURCE, ROLE_DEPTNAME} pr1 tb0 type0, border=8 score 3.031268761\n", "{ROLE_ROLLUP_1} pr0 tb0 type1, border=1 score 4.007602059\n", "{ROLE_ROLLUP_1, ROLE_DEPTNAME} pr2 tb0 type0, border=6 score 4.169017801\n", "{ROLE_ROLLUP_1, ROLE_DEPTNAME, ROLE_FAMILY} pr2 tb0 type0, border=11 score 4.602341778\n", "{ROLE_ROLLUP_1} pr1 tb0 type0, border=11 score 5.038311598\n", "5:\tlearn: 0.1676418\ttest: 0.1498856\tbest: 0.1498856 (5)\ttotal: 379ms\tremaining: 568ms\n", "\n", "{ROLE_DEPTNAME} pr2 tb0 type0, border=6 score 1.91890204\n", "{RESOURCE, ROLE_DEPTNAME} pr1 tb0 type0, border=10 score 2.908847019\n", "{MGR_ID, ROLE_DEPTNAME} pr2 tb0 type0, border=12 score 3.606018456\n", "{RESOURCE} pr0 tb0 type0, border=13 score 4.307821626\n", "{ROLE_TITLE} pr0 tb0 type0, border=14 score 4.579777128\n", " tensor 4 is redundant, remove it and stop\n", "6:\tlearn: 0.1660562\ttest: 0.1478320\tbest: 0.1478320 (6)\ttotal: 445ms\tremaining: 508ms\n", "\n", "{RESOURCE} pr1 tb0 type0, border=1 score 2.258001596\n", "{RESOURCE, ROLE_DEPTNAME} pr2 tb0 type0, border=5 score 2.860735033\n", "{ROLE_TITLE} pr2 tb0 type0, border=5 score 3.18948996\n", "{ROLE_DEPTNAME, ROLE_TITLE} pr2 tb0 type0, border=13 score 3.944580691\n", "{ROLE_DEPTNAME, ROLE_TITLE, ROLE_FAMILY_DESC} pr2 tb0 type0, border=7 score 4.163339306\n", "{ROLE_ROLLUP_2} pr0 tb0 type0, border=14 score 4.519983493\n", " tensor 5 is redundant, remove it and stop\n", "7:\tlearn: 0.1653090\ttest: 0.1467205\tbest: 0.1467205 (7)\ttotal: 517ms\tremaining: 452ms\n", "\n", "{RESOURCE} pr1 tb0 type0, border=1 score 2.213992054\n", "{RESOURCE} pr2 tb0 type0, border=12 score 2.814380964\n", "{ROLE_CODE} pr2 tb0 type0, border=1 score 3.054719636\n", " tensor 2 is redundant, remove it and stop\n", "8:\tlearn: 0.1649062\ttest: 0.1465720\tbest: 0.1465720 (8)\ttotal: 560ms\tremaining: 373ms\n", "\n", "{RESOURCE} pr2 tb0 type0, border=6 score 2.129653303\n", "{ROLE_ROLLUP_2} pr0 tb0 type0, border=12 score 2.50659707\n", "{ROLE_ROLLUP_2, ROLE_DEPTNAME} pr1 tb0 type0, border=6 score 2.826564801\n", "{RESOURCE, ROLE_DEPTNAME} pr1 tb0 type0, border=7 score 3.864031498\n", "{ROLE_FAMILY} pr2 tb0 type0, border=0 score 3.785986162\n", " tensor 4 is redundant, remove it and stop\n", "9:\tlearn: 0.1643222\ttest: 0.1463720\tbest: 0.1463720 (9)\ttotal: 625ms\tremaining: 312ms\n", "\n", "{ROLE_DEPTNAME} pr1 tb0 type0, border=13 score 1.598079397\n", "{ROLE_DEPTNAME, ROLE_TITLE} pr1 tb0 type0, border=3 score 2.182451775\n", "{ROLE_DEPTNAME, ROLE_TITLE, ROLE_FAMILY_DESC} pr2 tb0 type0, border=12 score 3.45517723\n", "{MGR_ID} pr2 tb0 type0, border=1 score 4.087126108\n", "{ROLE_DEPTNAME, ROLE_TITLE} pr2 tb0 type0, border=9 score 4.330920437\n", "{ROLE_TITLE} pr2 tb0 type0, border=0 score 4.436657706\n", " tensor 5 is redundant, remove it and stop\n", "10:\tlearn: 0.1633503\ttest: 0.1450909\tbest: 0.1450909 (10)\ttotal: 687ms\tremaining: 250ms\n", "\n", "{ROLE_FAMILY_DESC} pr1 tb0 type0, border=4 score 1.470512618\n", "{ROLE_DEPTNAME} pr2 tb0 type0, border=13 score 2.656999431\n", "{ROLE_FAMILY} pr0 tb0 type1, border=1 score 3.102896991\n", "{ROLE_TITLE, ROLE_FAMILY} pr0 tb0 type0, border=14 score 3.5519837\n", " tensor 3 is redundant, remove it and stop\n", "11:\tlearn: 0.1632233\ttest: 0.1451206\tbest: 0.1450909 (10)\ttotal: 739ms\tremaining: 185ms\n", "\n", "{ROLE_ROLLUP_2} pr0 tb0 type0, border=9 score 1.63415229\n", "{ROLE_ROLLUP_2, ROLE_DEPTNAME} pr0 tb0 type0, border=6 score 1.841313561\n", "{ROLE_ROLLUP_2, ROLE_TITLE} pr2 tb0 type0, border=13 score 2.839776224\n", "{ROLE_ROLLUP_2} pr0 tb0 type1, border=6 score 3.034144356\n", "{ROLE_ROLLUP_2, ROLE_DEPTNAME, ROLE_FAMILY_DESC} pr2 tb0 type0, border=9 score 3.381467012\n", "{ROLE_TITLE} pr1 tb0 type0, border=5 score 3.77508779\n", "12:\tlearn: 0.1626563\ttest: 0.1453396\tbest: 0.1450909 (10)\ttotal: 792ms\tremaining: 122ms\n", "\n", "{ROLE_FAMILY_DESC} pr2 tb0 type0, border=14 score 1.643727886\n", "{ROLE_TITLE, ROLE_FAMILY_DESC} pr1 tb0 type0, border=13 score 2.322308423\n", "{MGR_ID, ROLE_TITLE, ROLE_FAMILY_DESC} pr2 tb0 type0, border=6 score 2.494762742\n", "{MGR_ID, ROLE_ROLLUP_2, ROLE_TITLE, ROLE_FAMILY_DESC} pr2 tb0 type0, border=6 score 2.723569126\n", "{MGR_ID, ROLE_TITLE, ROLE_FAMILY_DESC} pr2 tb0 type0, border=1 score 3.439027181\n", "{ROLE_TITLE} pr1 tb0 type0, border=14 score 3.425081938\n", " tensor 5 is redundant, remove it and stop\n", "13:\tlearn: 0.1620230\ttest: 0.1445742\tbest: 0.1445742 (13)\ttotal: 863ms\tremaining: 61.6ms\n", "\n", "{ROLE_ROLLUP_2} pr2 tb0 type0, border=11 score 1.510460943\n", "{ROLE_ROLLUP_2, ROLE_FAMILY_DESC} pr1 tb0 type0, border=2 score 2.020092181\n", "{MGR_ID, ROLE_ROLLUP_2, ROLE_FAMILY_DESC} pr2 tb0 type0, border=6 score 2.705877322\n", "{ROLE_ROLLUP_2, ROLE_DEPTNAME} pr2 tb0 type0, border=9 score 3.767178056\n", "{ROLE_TITLE} pr0 tb0 type0, border=11 score 4.23047388\n", "{ROLE_TITLE, ROLE_FAMILY} pr0 tb0 type0, border=5 score 4.350755703\n", "14:\tlearn: 0.1614001\ttest: 0.1444070\tbest: 0.1444070 (14)\ttotal: 938ms\tremaining: 0us\n", "\n", "bestTest = 0.1444069686\n", "bestIteration = 14\n", "\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=15,\n", " #verbose=5,\n", " logging_level='Info'\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Metrics calculation and graph plotting" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "50383c010c304ab4afe465a9ffee0c10", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=500,\n", " random_seed=63,\n", " learning_rate=0.5\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Eval metric, custom metrics and best trees count" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6fe0b05ae929456c89fbd3046b7de131", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=50,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " eval_metric=\"Accuracy\",\n", " use_best_model=False\n", ")\n", "\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "50" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.tree_count_" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "8829f8544aa144c7a2e94df99bc88e1b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=50,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " custom_loss=['AUC', 'Accuracy']\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "21" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model._tree_count" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Metric hints" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ebf229a449b040e4aa7e2d72147792f4", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "\n", "model = CatBoostClassifier(\n", " iterations=50,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " eval_metric='AUC:hints=skip_train~false' #default\n", ")\n", "\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f10f866248e24cc991b55711dcb57d33", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=50,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " eval_metric='AUC:hints=skip_train~false', #default\n", " metric_period=10\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model comparison" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model1 = CatBoostClassifier(\n", " learning_rate=0.5,\n", " iterations=100,\n", " train_dir='learing_rate_0.5'\n", ")\n", "\n", "model2 = CatBoostClassifier(\n", " learning_rate=0.01,\n", " iterations=100,\n", " train_dir='learing_rate_0.01'\n", ")\n", "\n", "model1.fit(\n", " X_train, y_train,\n", " eval_set=(X_validation, y_validation),\n", " cat_features=cat_features,\n", " verbose=False\n", ")\n", "model2.fit(\n", " X_train, y_train,\n", " eval_set=(X_validation, y_validation),\n", " cat_features=cat_features,\n", " verbose=False\n", ")" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ea689d29144c46b48c7b6cb67d4fd6d7", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from catboost import MetricVisualizer\n", "MetricVisualizer(['learing_rate_0.01', 'learing_rate_0.5']).start()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overfitting detector" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c942cf28d09e49bd90c520160340dc3b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_with_early_stop = CatBoostClassifier(\n", " iterations=200,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " early_stopping_rounds=20\n", ")\n", "model_with_early_stop.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "21\n" ] } ], "source": [ "print(model_with_early_stop.tree_count_)" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6d94edb621784b11bd6d0886277d5c7d", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model_with_early_stop = CatBoostClassifier(\n", " eval_metric='AUC',\n", " iterations=200,\n", " random_seed=63,\n", " learning_rate=0.5,\n", " early_stopping_rounds=20\n", ")\n", "model_with_early_stop.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Cross-validation" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e822f1fa217f40338ca0d0cbf140ec71", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "ename": "CatBoostError", "evalue": "library/json/writer/json_value.cpp:515: Not an array", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mCatBoostError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 17\u001b[0m \u001b[0mplot\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0mstratified\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 19\u001b[0;31m \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 20\u001b[0m )\n", "\u001b[0;32m/home/annaveronika/trunk/arcadia/catboost/python-package/catboost/core.pyc\u001b[0m in \u001b[0;36mcv\u001b[0;34m(pool, params, dtrain, iterations, num_boost_round, fold_count, nfold, inverted, partition_random_seed, seed, shuffle, logging_level, stratified, as_pandas, metric_period, verbose, verbose_eval, plot, early_stopping_rounds, save_snapshot, snapshot_file, snapshot_interval, iterations_batch_size)\u001b[0m\n\u001b[1;32m 2928\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mlog_fixup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mplot_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mplot\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2929\u001b[0m return _cv(params, pool, fold_count, inverted, partition_random_seed, shuffle, stratified,\n\u001b[0;32m-> 2930\u001b[0;31m as_pandas, iterations_batch_size)\n\u001b[0m\u001b[1;32m 2931\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2932\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m_catboost.pyx\u001b[0m in \u001b[0;36m_catboost._cv\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m_catboost.pyx\u001b[0m in \u001b[0;36m_catboost._cv\u001b[0;34m()\u001b[0m\n", "\u001b[0;31mCatBoostError\u001b[0m: library/json/writer/json_value.cpp:515: Not an array" ] } ], "source": [ "from catboost import cv\n", "\n", "params = {}\n", "params['loss_function'] = 'Logloss'\n", "params['iterations'] = 80\n", "params['custom_loss'] = 'AUC'\n", "params['random_seed'] = 63\n", "params['learning_rate'] = 0.5\n", "\n", "cv_data = cv(\n", " params = params,\n", " pool = Pool(X, label=y, cat_features=cat_features),\n", " fold_count=5,\n", " inverted=False,\n", " shuffle=True,\n", " partition_random_seed=0,\n", " plot=True,\n", " stratified=False,\n", " verbose=False\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cv_data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "best_value = np.min(cv_data['test-Logloss-mean'])\n", "best_iter = cv_data['test-Logloss-mean'].idxmin()\n", "\n", "print('Best validation Logloss score, not stratified: {:.4f}±{:.4f} on step {}'.format(\n", " best_value,\n", " cv_data['test-Logloss-std'][best_iter],\n", " best_iter)\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cv_data = cv(\n", " params = params,\n", " pool = Pool(X, label=y, cat_features=cat_features),\n", " fold_count=5,\n", " type = 'Classical',\n", " shuffle=True,\n", " partition_random_seed=0,\n", " plot=True,\n", " stratified=True,\n", " verbose=False\n", ")\n", "\n", "best_value = np.min(cv_data['test-Logloss-mean'])\n", "best_iter = cv_data['test-Logloss-mean'].idxmin()\n", "\n", "print('Best validation Logloss score, not stratified: {:.4f}±{:.4f} on step {}'.format(\n", " best_value,\n", " cv_data['test-Logloss-std'][best_iter],\n", " best_iter)\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Select decision boundary" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2d3e64ab6c4349baa355f12ad740bc89", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=200,\n", " learning_rate=0.03,\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import sklearn\n", "from sklearn import metrics\n", "from catboost.utils import get_roc_curve" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [], "source": [ "? get_roc_curve" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd8FNX6+PHPk4Q0EkpCEelNCFIVKYIUlSIgFlRALypyVUBseBEVbCiKoCBIUxT5KV/lKjYuIghIEaVLAOkdEpBOQggJSfb8/phNsoSQbCCbyW6e9+uV106fZyfJPDPnzJwjxhiUUkqpy/GzOwCllFKFmyYKpZRSOdJEoZRSKkeaKJRSSuVIE4VSSqkcaaJQSimVI00UKs9E5CER+dXuOOwmIlVEJEFE/Atwn9VExIhIQEHt05NEZIuItLuC9fRvsACJvkfh3URkP1AeSAMSgPnAIGNMgp1x+SLnsf63MWaRjTFUA/YBxYwxqXbF4YzFALWNMbs9vJ9qFJLvXFTpHYVvuNMYEwY0BpoAL9sczxWx8yrZV67Q80KPt3KXJgofYoz5B1iAlTAAEJEgEXlfRA6KyFERmSoiIS7z7xKRaBGJF5E9ItLZOb2kiHwmIkdEJFZE3k4vYhGRR0VkhXN4ioi87xqHiPwkIoOdw9eKyHciclxE9onIMy7LvSEis0VkpojEA49m/U7OOL5wrn9ARIaLiJ9LHH+IyEQRiROR7SJyW5Z1c/oOf4jIOBE5CbwhIjVF5DcROSkiJ0Tk/0SklHP5L4EqwP+cxU0vZi0GEpGlIvKWc7tnReRXESnjEs/Dzu9wUkReFZH9InJ7dr9LEQkRkQ+cy8eJyArX3xvwkPN3ekJEhrms10xEVorIGef3nigigS7zjYg8JSK7gF3OaeNF5JDzb2C9iNzisry/iLzi/Ns465xfWUSWOxfZ6DwePZ3Ld3P+PZ0RkT9FpKHLtvaLyFAR2QScE5EA12PgjH2dM46jIjLWuWr6vs4499XS9W/Que71IrJQRE45130lu+OqrpAxRn+8+AfYD9zuHK4EbAbGu8wfB8wBIoBw4H/Au855zYA4oAPWRUNFoK5z3g/Ax0BxoBywBnjSOe9RYIVzuA1wiMxizNLAeeBa5zbXA68BgUANYC/QybnsG0AKcLdz2ZBsvt8XwE/O2KsBO4F+LnGkAs8DxYCezu8T4eZ3SAWeBgKAEKCW81gEAWWxTlAfZnesnePVAAMEOMeXAnuA65zbWwqMcs6rh1U02Np5LN53fvfbL/N7neRcvyLgD9zsjCt9n9Oc+2gEJANRzvVuBFo4v1M1YBvwnMt2DbAQ6+8hxDntX0Ckc50XgH+AYOe8IVh/U3UAce4v0mVbtVy23QQ4BjR3xvyI85gFuRy/aKCyy74zjimwEujjHA4DWmR3nLP5GwwHjjhjD3aON7f7f9OXfmwPQH+u8hdo/aMlAGed/0yLgVLOeQKcA2q6LN8S2Occ/hgYl802yztPPiEu03oDS5zDrv+kAhwE2jjHHwd+cw43Bw5m2fbLwOfO4TeA5Tl8N3/gAlDPZdqTwFKXOA7jTFLOaWuAPm5+h4OX27dzmbuBDVmOdW6JYrjL/IHAfOfwa8DXLvNCnd/tkkSBlTTPA42ymZe+z0pZvnOvy3yH54AfXMYNcGsu3/t0+r6BHcBdl1kua6KYAryVZZkdQFuX4/dYNn+/6YliOfAmUOYy3/lyiaK36+9Jf/L/R8sJfcPdxphFItIW+AooA5zBuioOBdaLSPqygnUCBuvKbl4226uKdYV+xGU9P6w7h4sYY4yIzML6Z10OPAjMdNnOtSJyxmUVf+B3l/FLtumijDOOAy7TDmBdZaeLNc6zhcv8a938DhftW0TKA+OBW7CuSv2wTpp58Y/LcCLWlTHOmDL2Z4xJdBZ5ZacM1pXxnrzuR0SuA8YCTbF+9wFYd3Wusn7v/wD9nDEaoIQzBrD+RnKKw1VV4BERedplWqBzu9nuO4t+wAhgu4jsA940xsx1Y795iVFdAa2j8CHGmGXADKxiDYATWFem1xtjSjl/Shqr4husf9qa2WzqENbVeBmX9UoYY66/zK6/Bu4TkapYdxHfuWxnn8s2Shljwo0xXVzDzuErncAqnqnqMq0KEOsyXlFcMoFz/mE3v0PWfb/jnNbAGFMCq0hGclg+L45gFQ0CVh0EVnFPdk4ASWT/u8nNFGA71tNIJYBXuPg7gMv3cNZHvAg8AJQ2xpTCKr5LX+dyfyPZOQSMzPL7DjXGfJ3dvrMyxuwyxvTGKiZ8D5gtIsVzWsdlvzXcjFFdAU0UvudDoIOINDLGOLDKsseJSDkAEakoIp2cy34G9BWR20TEzzmvrjHmCPAr8IGIlHDOq+m8Y7mEMWYD1sntU2CBMSb9DmINcNZZgRnirBitLyI3ufNFjDFpwDfASBEJdyaiwWTesYB1UnlGRIqJyP1AFDAvr9/BKRyrGC9ORCpilc+7OsqVn5BmA3eKyM3OyuU3uPQEDoDz9zYdGCvWwwD+zgrcIDf2Ew7EAwkiUhcY4MbyqcBxIEBEXsO6o0j3KfCWiNQWS0MRSU9wWY/HNKC/iDR3LltcRLqKSLgbcSMi/xKRss7vn/435HDG5uDyx34uUEFEnhPr4Y1wEWnuzj6VezRR+BhjzHGsCuDXnJOGAruBVWI9WbQIq2ISY8waoC9WhXccsIzMq/eHsYoNtmIVv8wGKuSw66+A252f6bGkAd2wnsLaR2YyKZmHr/Q0Vj3LXmCFc/vTXeavBmo7tz0SuM8Yk16kk9fv8CZwA9ax+Bn4Psv8d4Hhzid6/pOH74AxZovzu8zCurtIwKr4Tb7MKv/BqkReC5zCusJ25//1P1jFf2exTtz/zWX5BVjv3uzEKrZL4uLiobFYyfpXrAT0GVYlOljJ7v85j8cDxph1WHVUE7GO926yeZItB52BLSKSgFUE2MsYc94Yk4j1u/3Dua8WrisZY85iPYRwJ1aR3C6gfR72q3KhL9wpryUij2K9ANfa7ljySkTCsK6aaxtj9tkdj1I50TsKpQqIiNwpIqHOcvf3se4Y9tsblVK500ShVMG5C6ui/TBWcVkvo7f0ygto0ZNSSqkc6R2FUkqpHHndC3dlypQx1apVszsMpZTyKuvXrz9hjCl7Jet6XaKoVq0a69atszsMpZTyKiJyIPelsqdFT0oppXKkiUIppVSONFEopZTKkSYKpZRSOdJEoZRSKkeaKJRSSuXIY4lCRKaLyDER+fsy80VEJojIbhHZJCI3eCoWpZRSV86T71HMwGpu+IvLzL8Dq72b2lid3UxxfiqllMqOMZCWDKlJYNLAOKxPh8uwSYO0FDh/zFrekcKFpAtXtVuPJQpjzHIRqZbDIncBXzgbRVslIqVEpIKzwxmllMpfxmGdQB0p4Ei1fkwqnDsKqeddTrwOl5NuluHEY+AfaA2nb8ORCmcPgl8x8AvIPHE7UjNP3Mc2QEhZELl0niMNjqyE0nXAcQHOHoLkOAgq5Zyf6vLpyPPXHvK/Dmw4nFM3LLmz883silzcQUqMc9oliUJEngCeAKhSpUqBBKeUyqPUZEg6BRfOXnwSdL3SdaTB+ePWCTUtBU7vhLQkQDKXMQ7nei7rHv0LStW8+ATvSIU9c6zpGes4Ll4v/TMpr12f2+DYXxePJ5/Jfrl0waVB/EH8nJ/OYT/ncMJhCK9E/evLMOGPalcVmlc04WGM+QT4BKBp06ba3K0q2hypzuKHZOskm5qUWRyR3XhGUYXz6hhjff6zBkLLO6+wU6wT96ElUKJK5hVz1qvjE5uhRFXntkzmts4VQEFA7O/ZTz+9K2/bCQgF/2IgAVbCwlh3ChVaOk+yfi4//pcOn9oOFZpn3kGkb+fcYShVCwJCnNOdJ2w/f2uZpFMQGZX9PD9/63iGlgX/IOuuJbCktazrMuJv3ZXkYOvW4/z11xH+1a8hAA/3NbQdEkf16iPydpxcD9kVr3n1YoHKLuOVnNOU8j0XEuDcP1bRQtoF6wr33GHrBGAc1gl941QIjrBO2rErILyyc94FSIixtiN+V1T8kCdxe3OeH59Dk0H+QVZiKl3H5cTmf/GwSbO+T/mm1okw/iBUvR38Al2W83NZz3miTjoDpWpknpjTT9SBYRBWKfNqGj+X9Vw+/QOhWGi+HqrCJDExhbffXs6YMX/i7y+0aFGJWrUiEBGqVSt1Vdu2M1HMAQaJyCysSuw4rZ9QhZYx1gn06F8XF6NkLYfeOxfCq1on+6PrILScVRSTej7v+zyzO5s4HNaJzz/QunL1DwL/YAgIdn66jmcZ9vMHxFof51XphTgof6PzpOv8caRA6esuPsGnXwX7+VvLBARfvC0RK8kFBF/5MVZX7JdfdvHUU/PYt88qrurX70YiI0NyWct9HksUIvI10A4oIyIxwOtAMQBjzFRgHtAFqwP2RKCvp2JR6rJO7YRd3wPW0yHs+g5CylnDR1ZZV8fpJ093JRzOHE48Zn2mbyMiyir28Au0rqojr4eQMtYJ15EGEXWhYuvMK+bi11hJwT/QOtkHlnCe8JWC2Nh4nntuAbNnbwWgYcPyTJ3alZYtK+eyZt548qmn3rnMN8BTntq/KuLOn4INH0HMUkiIBcQ60WaU0Rs4tc29baUnifRin+secF5ZZy1r9reKZWrdBWUbO6+8QyDsWggMz7VsWam8euqpefz00w5CQ4sxYkQ7nn22BQEB+f96nFdUZit1ifTnyeMPWI8Tnt4JJ7fCgUVwekfet1fjTijXKLP4pUIL6zMwLPMuQLQhA2W/1FRHRjJ4773bKVbMnw8+6EiVKiU9tk9NFKrwSk22Ps8dgZjlVkI4/IdV/LLjv+5tIyIKGj5uVa6GVXRWbAoZ5evFQq2neJQq5OLikhg+/Dd27jzF/PkPISLUqVOGb7+93+P71kSh7GUcVkXvsY2w5XOrQvjoevfX9wuwKpRrdLNeUqrUBkIiofZ9EF5Ji3uU1zPG8O23W3nuufkcOZKAv78QHf0PTZpc3Ut0eaGJQhUMRyoc32w9g35mNxxcZN0l5EXZRtbVf3BpqNoRrmkGpWt5Jl6lCoE9e04xaNAvzJ9vPQHXsmUlpk7tRsOG5Qs0Dk0UyrMOr4RdP8C6MZdfxi/AqmhOOQcNn7Sery/XCMo0tB73VKoIev/9P3n11SUkJaVSqlQw7713O//+9w34+RX8XbImCuU5C/vDpo8vnX5NM+sN1uDS0HwYhBXcLbRS3iIxMYWkpFT69GnI++93pFy54rbFoolC5Z/zJ62XzLbOhG0zL57X/BWo0xPKNrQnNqUKuePHz7Fjx0lat7basxs6tBXt2lWjTRv7H7bQRKGuztG/4Ke7ISURkk5eOj8gFAadtl4YU0pdwuEwTJ++gRdfXEhAgB/btw8iIiKEoKCAQpEkQBOFuhLGwL55sGig1bxyVpHXQ42uULM7XHuzPnmk1GX8/fcx+vefyx9/WA1pd+hQg8TEFCIi8q/5jfygiUK5LzUZlr0A0ZMunddmtFW0FF5ZE4NSuTh37gIjRixj7NhVpKY6KF++OB9+2JmePa9HCuH/jyYK5R5HKkwqZTVX7arpEGj+slUxrZRyy333fcv8+bsRgYEDmzJy5G2UKlV4G1TURKFyFrfPerx12QuZ08rdAJ2mWxXThfDqR6nCbujQVhw9msCUKV1p3ryS3eHkShOFyuRIg/MnYMcs2P2T1YlNVnUfhK7/V/CxKeWlUlMdfPTRavbvP8P48XcA0K5dNdate8KWdyKuhCYKZZnX59JHWl2VqAbdZ1t9Fyil3LJmTSxPPjmX6Oh/AHjiiRu5/vpyAF6TJEATRdFlDBxZDYd+gxXDLp0fGG69DFeprdXtoxYxKeW2M2eSeOWVxUydug5joGrVkkyc2CUjSXgbTRRFTfwhmBFlNZeRVfFr4IkY7RhHqaswa9bfPPfcfI4ePUdAgB8vvNCSV19tQ/Hi3vsukSaKosI44PAqmNXq4unib1VK3/ym9e6D9rmg1FX59dc9HD16jlatKjNlSlcaNCjYBvw8QROFr4vbD78Ngr0/Xzy946dQ/zEtUlLqKiUnpxIbe5YaNaxHxEeP7sAtt1ThkUcae1U9RE40UfiiszGw+CnYMyf7+R0+hgb9CjYmpXzQb7/tY8CAn/HzEzZu7E9goD9lyoTSt28Tu0PLV5oofEXcPoj9A/b8D3Z+c+n8Bo9Ds5egVI2Cj00pH3P0aAL/+c9CZs7cBEDdumWIiYnPuKvwNZoovNmhpTD/Uavf6OxUbg/txkK5xgUZlVI+y+EwTJu2npdeWsyZM0kEBwcwfPgtDBnSisBA330IRBOFtzqxBb5pf+n0mndZn23eg4g6BRuTUj7unnv+y5w5OwDo1KkmkyZ1oWbNCJuj8jxNFN7q3D+Zw42fghbDrcdblVIec++9dVmzJpbx4ztz//31CmUDfp6gicLbVbkVbptodxRK+aQ5c3YQExPPwIE3AfDww424994owsOLVhe9mii8UVoKzL3f7iiU8lkHD8bxzDO/8NNPOwgK8qdz51rUqFEaESlySQI0UXgf44APXd7wDK9iXyxK+ZiUlDQmTFjN668v5dy5FMLDA3n77VupWrWk3aHZShOFNzm2Eb50eYKpehe4fap98SjlQ1atiuHJJ+eyadNRAO6/vx7jxnWiYsUSNkdmP00U3sS1T4ja90L37+yLRSkf8+qrS9i06SjVq5di4sQudOlS2+6QCg1NFN4i9k84uNgabvk63PyGreEo5e2MMZw9e4ESJaw6h4kT7+CLLzYybFgbQkOL2Rxd4aItwHkL136q6/e1Lw6lfMCOHSe4/fYvuffe/2KMAaBOnTKMHHmbJols6B2FN0g8Adu/soZbvgElqtoajlLeKikplXff/Z1Ro/7gwoU0IiND2L//DNWr+2bTG/lFE0Vhl7UC+5qb7ItFKS+2cOEeBg6cx+7dpwB47LHGjB7dgcjIUJsjK/w8WvQkIp1FZIeI7BaRl7KZX0VElojIBhHZJCJdPBmP1/nt2YuTRKW2ULWDffEo5YWMMTz22E907DiT3btPUa9eWZYvf5TPPrtLk4SbPHZHISL+wCSgAxADrBWROcaYrS6LDQe+McZMEZF6wDygmqdi8ipLnoMNEzLH242DG57V/iOUyiMRoVq1UoSEBPDaa20ZPLilTzfg5wmeLHpqBuw2xuwFEJFZwF2Aa6IwQPpDyiWBwx6Mp/A7Fg3Rk2HXbEg6nTn9ycMQVsG+uJTyMtHR/3DkyFnuuMN6xHXo0Fb06dNQ6yKukCeLnioCh1zGY5zTXL0B/EtEYrDuJp7ObkMi8oSIrBORdcePH/dErIXD6ndh87SLk8SAo5oklHLT2bPJDB68gBtv/IRHHvmRU6fOAxAUFKBJ4irYXZndG5hhjPlARFoCX4pIfWOMw3UhY8wnwCcATZs2NTbEWTDSkq3PxoOgbCO47j4ILmVvTEp5AWMMP/64nWeemU9MTDx+fsKDDzagWDF9AyA/eDJRxAKVXcYrOae56gd0BjDGrBSRYKAMcMyDcRV+VW6D2nfbHYVSXuHAgTMMGvQLc+fuBKBp02v5+ONu3HCD3onnF0+m27VAbRGpLiKBQC8gayfOB4HbAEQkCggGfLhsSSmVn4wx9OjxDXPn7qREiSAmTryDVav6aZLIZx67ozDGpIrIIGAB4A9MN8ZsEZERwDpjzBzgBWCaiDyPVbH9qEl/TVIppS7D4TD4+Qkiwvvvd2Tq1HWMG9eJChXC7Q7NJ3m0jsIYMw+rktp12msuw1uBVp6MQSnlO06eTOSllxYBMG1adwDatatGu3bVbIzK92lNj1Kq0DPG8P/+XzR1607i00838MUXm4iJibc7rCLD7qeeVDpHGhz+0+4olCp0tm07zoABP7Ns2QHAuoOYMqUrlSppPxEFRROF3YyBLTNgwWOZ0/z0rVGljDG89toS3nvvD1JSHJQpE8oHH3SkT5+GiLZQUKA0UdjFkQqr34FtX8HpHZnTQ8tD5Xa2haVUYSEixMaeJSXFweOP38CoUbcTERFid1hFkiYKu3x5A5zYfPG0+3+DKu3tiUepQuDw4bOcOJFIw4blARg9ugP9+jWhVSvtG95OmigKWtIZ+LIJxO+3xsMqQbsPoPodEKiP9qmiKS3NwZQp6xg27DcqVgwnOro/gYH+lCkTSpkymiTspomioC15NjNJlKwB/XZri7CqSPvrryM8+eRc1q2z2gRt06Yq8fHJlCmjTYAXFm4lCueb1VWMMbs9HI/vS3S2TlKhOfReqUlCFVnx8cm8+upvTJy4FofDUKlSCSZM6Mzdd9fVyupCJtdEISJdgbFAIFBdRBoDrxtj7vF0cD6t5euaJFSRZYyhTZvP2bjxKP7+wuDBLXjjjXaEhwfZHZrKhjsv3I0AmgNnAIwx0UAtTwallPJtIsLzz7egWbOKrFv3BB980EmTRCHmTtFTijHmTJZbQW2P6Uqc2QP759sdhVIF7sKFNMaOXYm/vzBkiNVqz8MPN+Jf/2qIv782EFHYuZMotonIA4CfiFQHngFWeTYsH/WXS9emIWXsi0OpAvT77wfo3/9ntm49TlCQPw8/3Ijy5cMQEfz9tfjVG7iTygcBNwIO4HsgGXjWk0H5nNRkWDQwsw/smndB+ab2xqSUh504kchjj/1EmzYz2Lr1OLVrRzB37oOULx9md2gqj9y5o+hkjBkKDE2fICL3YiUNlRtHKnxeB+IPZE5r9KRWZCufZYxhxoxohgxZyMmT5wkM9Ofll1vz0kutCQ7WJ/K9kTu/teFcmhSGZTNNZefo+ouTRO8/4dqW9sWjVAGYOXMzJ0+e59ZbqzN5chfq1NGiVm922UQhIp2wuimtKCJjXWaVwCqGUu5wpGYOP5MIxbStGuV7EhNTiItLokKFcESEyZO7sHbtYR56qIG+E+EDcrqjOAb8DSQBW1ymnwVe8mRQPunamzVJKJ/0yy+7eOqpedSoUZqFC/sgItSpU0bvInzIZROFMWYDsEFE/s8Yk1SAMfmO8ydhVmu7o1DKI2Jj43nuuQXMnr0VgPDwIE6ePK9Nb/ggd+ooKorISKAeEJw+0Rhzncei8nbGwNyesPPbzGkVNWEo35CW5mDSpLUMH/4bZ89eoHjxYowY0Z5nnmlOQIC+E+GL3EkUM4C3gfeBO4C+6At3OVv6/MVJ4pb3oNmL9sWjVD5xOAxt287gjz8OAXD33XUZP74zVaqUtDky5UnuJIpQY8wCEXnfGLMHGC4i64BXPRyb9zq+0fr0D4JnzmmPdcpn+PkJHTvW5ODBOCZO7EL37nXsDkkVAHcSRbKI+AF7RKQ/EAtoxwnuuPcXTRLKqxlj+OabLQQE+NGjRz0Ahg5txeDBLQkLC7Q5OlVQ3EkUzwPFsZruGAmUBB7LcY2izBg48bfdUSh11fbsOcXAgfP49dc9lC0byq23Vqd06RCCggII0vb7ipRcE4UxZrVz8CzQB0BEKnoyKK+2oC+cP2ENi1bsKe+TnJzKmDF/MnLk7yQlpVK6dDAjR95KyZLBua+sfFKOiUJEbgIqAiuMMSdE5HqspjxuBSoVQHze49Ay+KbdxdOuaWZLKEpdqaVL9zNgwM9s325d7PTp05D33+9IuXLFbY5M2emyl7wi8i7wf8BDwHwReQNYAmwE9NFYV6d3Z0kSAk/G6gt2yqukpTkYONBKEnXqRPLbbw/zxRf3aJJQOd5R3AU0MsacF5EI4BDQwBizt2BC8yKubTndPQdq3mlfLErlgcNhSEpKJTS0GP7+fkyZ0pXlyw/w4outCArSBvyUJae/hCRjzHkAY8wpEdmpSeIyDv1mfVa5VZOE8hqbNx+lf/+fqVs3ks8+uwuAtm2r0bZtNXsDU4VOTomihoiktxArWP1lZ7QYa4y516OReYvEY7D6HWu4mD41rAq/c+cuMGLEMsaOXUVqqoN9+05z+vR5SpfWolKVvZwSRY8s4xM9GYjXWvlW5nD7cfbFoZQb/ve/HQwa9AsHD8YhAgMHNmXkyNsoVUqfaFKXl1OjgIsLMhCv5EiFaGf+rNgaSla3Nx6lLiM11UHPnrP5/vttADRufA0ff9yNZs30SXeVO62tuhrGpcmr7j/YF4dSuQgI8KNkySDCwgJ56632DBrUTBvwU27z6F+KiHQWkR0isltEsu3DQkQeEJGtIrJFRL7yZDwe4xcAodr2vipcVq+OYfXqmIzxMWM6sG3bUzz3XAtNEipP3L6jEJEgY0xyHpb3ByYBHYAYYK2IzDHGbHVZpjbwMtDKGHNaRMq5H7qNjm2E1SPh1Da7I1HqEmfOJPHyy4v4+OP11K1bhujo/gQG+hMZqf1EqCuT62WFiDQTkc3ALud4IxH5yI1tNwN2G2P2GmMuALOw3s1w9TgwyRhzGsAYcyxP0dtl/VirGfH0Np1qdrc3HqWwGvD76qvN1K07kalT1+Pv70f37nVIS9Oei9XVceeOYgLQDfgRwBizUUTau7FeRayX9NLFAM2zLHMdgIj8AfgDbxhj5ruxbXulXbA+bxkFjQZAUAl741FF3q5dJxk4cB6LFlmvOrVqVZmpU7tRv7533KSrws2dROFnjDmQpYP0tHzcf22gHVbbUctFpIEx5ozrQiLyBPAEQJUqVfJp11coLQUOLLSGw6toklC2S0lJ49ZbvyAmJp6IiBBGj76dvn2b4Ocnua+slBvcSRSHRKQZYJz1Dk8DO91YLxao7DJeyTnNVQyw2hiTAuwTkZ1YiWOt60LGmE+ATwCaNm1qX+96iSdgStnMcf9itoWilDEGEaFYMX9GjryVJUv2M3r07ZQtq20zqfzlzqMPA4DBQBXgKNDCOS03a4HaIlJdRAKBXsCcLMv8iHU3gYiUwSqKKrzNhOz+MXM4IASqdrQvFlVkHT2aQJ8+P/D228szpj38cCM+//wuTRLKI9y5o0g1xvTK64aNMakiMghYgFX/MN0Ys0VERgDrjDFznPM6ishWrOKsIcaYk3ndV4FxpFifpWpC3+3WY7FKFRCHwzBt2npeemkxZ84kUapUMM8914LwcO1FSHmWO2e6tSKyA/gv8L0x5qy7GzfGzAPmZZn2msuwwbpbGezPfU2QAAAgAElEQVTuNguFqh00SagCtXHjP/Tv/zOrVlnvRXTuXItJk7poklAFwp0e7mqKyM1YRUdvikg0MMsYM8vj0RUmF87C4oF2R6GKmJSUNF5+eTEffriKtDRDhQphjB/fmfvuq0eWB0yU8hi3Xs80xvxpjHkGuAGIx+rQqGiJ/SNzuIS26aQKRkCAHxs2/IPDYXj66WZs2/YU999/vSYJVaByvaMQkTCsF+V6AVHAT8DNHo6r8DHOl5bCroWbhtgbi/JpBw/GkZbmoHr10ogIU6d2JS4umaZNr7U7NFVEuVPQ/jfwP2C0MeZ3D8dT+JVtBHo1pzwgJSWN8eNX8/rrS2nZshILF/ZBRKhdO9Lu0FQR506iqGGM0TYAsO/1DeX7Vq48RP/+P7Np01EAIiJCSExMoXjxQJsjUyqHRCEiHxhjXgC+E5FLzpJFqoe7DRPht6ftjkL5oNOnz/PSS4v45JO/AKhevRSTJnXhjjtq2xyZUplyuqP4r/OzaPdsl3L+4iRRvYt9sSifkpycSuPGH3PwYBzFivkxZMjNDBvWhtBQfeNfFS459XC3xjkYZYy5KFk4X6QrGj3gGZdmrR4/CCUqX35ZpfIgKCiAfv2asHjxPqZM6Uq9emVzX0kpG7jzeOxj2Uzrl9+BFHrFimuSUFclKSmV119fwldfbc6Y9sort7B06SOaJFShllMdRU+sR2Kri8j3LrPCgTPZr+WDtB5f5YOFC/cwcOA8du8+RblyxbnnnrqEhBTTnuaUV8ipjmINcBKr1ddJLtPPAhs8GVShseMbmNvTGjb61JPKu3/+SWDw4AV8/bXVydX115dl6tRuhIRoPYTyHjnVUewD9gGLCi6cQmbtmMzhqAfti0N5nbQ0Bx9/vJ5XXllMXFwyISEBvP56W55/viWBgf52h6dUnuRU9LTMGNNWRE5z8UsEgtWeX4THo7NbekX2PT9DDX3aSbkvLc3w0UdriItLpkuX2kyceAfVq5e2OyylrkhORU/p3Z2WKYhACrXi19gdgfICZ88mk5ZmKFUqmMBAf6ZNu5OjRxO4994obZtJebXL1qS5vI1dGfA3xqQBLYEnAe0dRSknYwzff7+NqKhJvPDCgozprVtXoUcPbeVVeT93Hrn4Easb1JrA51hdlX7l0agKA2Pg5Fa7o1CF3P79Z+jefRY9enxDbOxZ/v77OElJqXaHpVS+cidROJx9Wt8LfGSMeR6o6NmwCoHfX4K0ZOeIXhGqi6WkpPHeeyuoV28Sc+fupESJICZOvIM//3yM4GDt1Er5Fre6QhWR+4E+wN3Oab79bN+pnbB2tDXsFwBl6tsbjypUEhNTaNHiUzZvPgZAr171GTu2IxUqhNscmVKe4U6ieAwYiNXM+F4RqQ587dmwbLZ5WuZwz+Xg79t5UeVNaGgxmja9lsTEFCZP7krHjjXtDkkpj3KnK9S/ReQZoJaI1AV2G2NGej40GzlSrM/aPaBCC3tjUbYzxvDFFxupWTOC1q2rADBuXCcCA/31xTlVJLjTw90twJdALFZh/TUi0scY80fOa/qAiq21k6Iibtu24wwY8DPLlh0gKqoM0dH9CQz0p2TJYLtDU6rAuFP0NA7oYozZCiAiUViJo6knA1PKTufPpzBy5O+MHv0HKSkOypYN5eWXW1OsmLbNpIoedxJFYHqSADDGbBMR3+52Ky3F7giUjebP381TT81j797TADz++A2MGnU7EREhNkemlD3cSRR/ichUYKZz/CF8tVHAhCPwaXWXx2JVUZOQcIE+fX7gxIlE6tcvx9SpXWnVqordYSllK3cSRX/gGeBF5/jvwEcei8gu68bCshcunlb1dntiUQUqLc2Bw2EoVsyfsLBAxo/vTExMPM8/34JixbQBP6VyTBQi0gCoCfxgjBldMCHZIDnu4iTR9D/Qdszll1c+Y/36wzz55FzuuqsOr77aFoAHH2xgc1RKFS6XrZkTkVewmu94CFgoItn1dOcbXOskHt6kSaIIiI9P5tlnf6FZs09Zv/4IX365iZSUtNxXVKoIyumO4iGgoTHmnIiUBeYB0wsmrAK29j3rMzgSyurVpC8zxjB79laefXY+R44k4O8vDB7cgjffbK/FTEpdRk6JItkYcw7AGHNcRHz3ucDoidZnWAV741AedfZsMj17zuaXX3YD0Lx5RaZO7UbjxtqMvFI5ySlR1HDpK1uAmq59Zxtj7vVoZAUl/gCkJlnD9y+2NxblUWFhgSQnp1GyZBCjRt3OE0/ciJ+fvlCpVG5yShQ9soxP9GQgtpn3r8zhwBL2xaE8YvnyA1SoEEbt2pGICNOndyc4OIDy5cPsDk0pr5FTn9m+f3l9eBXErrCGb3oRArRZBl9x4kQiL764kM8/j+a226qzcGEfRISqVUvZHZpSXqdoN5y/5t3M4RsH2xeHyjcOh2HGjGiGDFnIqVPnCQz055ZbqpCWZggI0GImpa6ERyuoRaSziOwQkd0i8lIOy/UQESMiBdt+VNoF6/PmEVC8fIHuWuW/LVuO0a7dDPr1m8OpU+e57bbqbN48gNdfb0dAgO8+i6GUp7l9RyEiQcYYt9u2EBF/YBLQAYgB1orIHNd2o5zLhQPPAqvd3Xa+u0bbN/R2cXFJtGjxGQkJFyhXrjhjx3bkwQcbaH/VSuWDXC+zRKSZiGwGdjnHG4mIO014NMPqu2KvMeYCMAu4K5vl3gLeA5LcDzsf7PgW9s8v0F2q/GeMAaBkyWCGDm1F//43sn37Uzz0UENNEkrlE3fuxycA3YCTAMaYjUB7N9arCBxyGY8hS1/bInIDUNkY83NOGxKRJ0RknYisO378uBu7zsXxzTD3gczxso2ufpuqQMXGxnPffd8wc+amjGnDht3ClCndKF1aW3lVKj+5kyj8jDEHsky76rYOnC/wjQVeyG1ZY8wnxpimxpimZcuWvdpdw16XvPTAEgi79uq3qQpEaqqD8eNXUbfuJL77bhuvv76UtDQHgN5BKOUh7tRRHBKRZoBx1js8Dex0Y71YoLLLeCXntHThQH1gqfMf/Bpgjoh0N8ascyf4K2cVV3B9X6jczrO7Uvlm7dpY+vf/mb/+OgLA3XfXZcKEzvj7a0W1Up7kTqIYgFX8VAU4CixyTsvNWqC2iFTHShC9gAfTZxpj4oAy6eMishT4j+eThAt90skrnDt3gaFDFzF58lqMgSpVSvLRR3fQvXsdu0NTqkjINVEYY45hneTzxBiTKiKDgAWAPzDdGLNFREYA64wxc/IcbX4wBv5Za8uu1ZUJCPBj0aK9+PkJgwe35PXX21K8uG93sqhUYZJrohCRaWSU1WQyxjyR27rGmHlYrc66TnvtMsu2y217V80Y+O1p2P2DNe5XtN83LMz27DlFqVLBREaGEhQUwJdf3kNwcAANGuhdoFIFzZ3C3UXAYufPH0A5wDv7Cj25BaInZY5H9bEvFpWt5ORU3n57OfXrT2Ho0EUZ02+6qaImCaVs4k7R039dx0XkS2CFxyLypOgpmcP/3gslq9sXi7rE0qX7GTDgZ7ZvPwFYTzilpTm0slopm11J2Ut1wPsu7c6fhI2TreHqd2iSKESOHTvHkCEL+eKLjQDUqRPJlCldad9ef0dKFQbu1FGcJrOOwg84BVy23aZCK/V85vCtvtliujc6cSKRqKhJnDp1nqAgf4YNu4UXX2xFUJDWHylVWOT43yjWCw6NyHz/wWHS20zwVmEVoVQNu6NQTmXKhHLXXXWIiYln8uSu1KoVYXdISqksckwUxhgjIvOMMfULKiDl286du8CIEcvo2vU62rSpCsDkyV0JCvLXN6uVKqTcqSWMFpEmHo/Ek/YvhE8q576c8qj//W8H9epNZvToPxk48GccDuvmNDg4QJOEUoXYZe8oRCTAGJMKNMFqInwPcA6r/2xjjLmhgGK8OgcWw3cdM8evucm+WIqoQ4fiePbZ+fzww3YAmjS5ho8/7qb9VSvlJXIqeloD3AB0L6BYPGPte5nDd34LtbN2Ba48JTXVwYQJq3nttSWcO5dCWFggb7/dnqeeaqYdCSnlRXJKFAJgjNlTQLF4hiPV+rx9Clx3n72xFDHx8cm8++4Kzp1LoUePKD78sDOVKpWwOyylVB7llCjKishlO5I2xoz1QDyeU/o6uyMoEs6cSSIkJICgoAAiIkL4+ONuBAX507WrHn+lvFVO9//+QBhWc+DZ/SiVwRjDV19tpk6diYwe/UfG9HvvjdIkoZSXy+mO4ogxZkSBReIJO7+DQ0vsjsLn7dx5koEDf2bx4n0ALF9+EGOMPsmklI/ItY7Cq+2dmzkcEWVfHD4qKSmV995bwTvvrODChTQiIkIYM6YDjz7aWJOEUj4kp0RxW4FF4Wm3T4WwCnZH4VP++SeBNm0+Z9euUwA8+mhjxozpQJkyoTZHppTKb5dNFMaYUwUZiEf5ayc3+a18+eJUrlySgAA/pkzpStu21ewOSSnlIb7b8tqhpbBlht1R+AyHwzBt2nrat6/OdddFIiJ89dW9lC4dQmCgv93hKaU8yHffejq0NHO4fFPbwvAFGzf+Q6tW0+nf/2cGDvyZ9HYhy5cP0yShVBHgu3cU6VoMh7IN7I7CKyUkXOCNN5by4YerSEszXHttOP37a9JVqqjx/UQhesV7JX78cTtPP/0LMTHx+PkJTz/djLffvpUSJYLsDk0pVcB8P1GoPIuNjadXr9kkJ6dx440VmDq1G02bXmt3WEopm/huojjj3U1UFbSUlDQCAvwQESpWLMHIkbcSGOjPwIE3aZ/VShVxvnkG2PYVbJtpDYtvfsX89Oefh7jxxk+YOXNTxrQXXriZp59urklCKeWDieLMHpj3UOa4thh7WadOnefJJ/9Hq1bT2bz5GJMnr8Pbe7pVSuU/3yt6ij+QOdxzGUTWsy+WQsoYw8yZm3jhhV85fjyRYsX8ePHFVgwbdos2vaGUuoTvJYp0ldtDpTZ2R1HoHD2aQO/e37FkyX4A2ratypQpXYmKKmtvYEqpQst3E4XKVqlSwRw5kkCZMqG8/34HHn64kd5FKKVypImiCFi4cA833FCByMhQgoIC+Pbb+6lQIYzISG3ATymVO9+rzFYZjhw5S+/e39Gx40yGDl2UMb1+/XKaJJRSbvOtOwpjYNFAu6OwXVqag48/Xs/LLy8mPj6ZkJAA6tSJ1M6ElFJXxLcSRfx+OL3DGi5R1dZQ7PLXX0fo338ua9ceBqBr19pMnNiFatVK2RyZUspb+VaicKRlDnf42L44bLJ//xmaNZtGWpqhYsVwJky4g3vuqat3EUqpq+LRRCEinYHxgD/wqTFmVJb5g4F/A6nAceAxY8yBSzbkDmPgu07WcMkaRbKzomrVStG3b2PCw4N48812hIdrA35KqavnscpsEfEHJgF3APWA3iKS9e23DUBTY0xDYDYw+op3aNIgbq81XKPbFW/Gm+zff4Y77/yaZcv2Z0z75JM7GTu2kyYJpVS+8eQdRTNgtzFmL4CIzALuAramL2CMWeKy/CrgX1e9V/GHW8df9WYKs5SUNMaOXcmbby7j/PlUTpxIZOXKfgBazKSUyneeTBQVgUMu4zFA8xyW7wf8kt0MEXkCeAKgSpUq+RWfV1qx4iD9+89ly5bjAPTqVZ+xYzvaHJVSypcVispsEfkX0BRom918Y8wnwCcATZs2LZKt1p0+fZ4hQxby2WcbAKhZszSTJ3elY8eaNkemlPJ1nkwUsUBll/FKzmkXEZHbgWFAW2NMsgfj8WoOh+Gnn3ZQrJgfL73Umpdfbk1ISDG7w1JKFQGeTBRrgdoiUh0rQfQCHnRdQESaAB8DnY0xxzwYi1favv0E1auXIigogMjIUP7v/+6lSpWS1K1bxu7QlFJFiMeeejLGpAKDgAXANuAbY8wWERkhIt2di40BwoBvRSRaROZ4Kh5vkpiYwrBhi2nYcAqjR/+RMb1jx5qaJJRSBc6jdRTGmHnAvCzTXnMZvt2T+/dG8+fvZuDAn9m37wwAJ04k2hyRUqqoKxSV2QoOHz7Lc8/N59tvraeHGzQox9Sp3bj55sq5rKmUUp6liaIQ2LnzJE2bfsLZsxcIDS3GG2+05bnnWlCsmL/doSmllCaKwqB27QhuuqkixYsX46OP7qBqVW3ATylVeGiisEF8fDKvvbaEgQNv4rrrIhER5szpRfHiRa99KqVU4ec7ieKXh50Dhfd9PGMMs2dv5dln53PkSALbt59g/nyr1RJNEkqpwso3EsXRDbD9a2u4Qkt7Y7mMvXtPM2jQPH75ZTcALVpU4r339KEvpVTh5xuJYvmQzOF75toXRzYuXEjj/ff/5K23lpOUlEqpUsGMGnUbjz9+I35+2oCfUqrw841EkeJ81+DmNyG4cFUEHzoUx4gRy0hOTuOhhxrwwQcdKV8+zO6wlFLKbd6fKOIPwZGV1nCVwlGUc/r0eUqVCkZEqFkzgvHjO1OrVgS33VbD7tCUUirPPNaER4FZ9VbmcGC4fXFgNdw3ffoGatX6iJkzN2VMf/LJppoklFJey/sTxYWz1mfVjlCmvm1hbNlyjHbtZtCv3xxOnTqfUWmtlFLezruLnk7vhh2zrOHrHwUbendLTEzhrbeW8f77K0lNdVCuXHHGjetE7972JS2llMpP3psoTu+C6ddljhe/psBD2LnzJJ06zWT//jOIQP/+N/LOO7dRunRIgceilFKe4r2J4sCizOHmr0DldgUeQtWqJQkODqBRo/JMndqNFi0qFXgMqvBKSUkhJiaGpKQku0NRRUhwcDCVKlWiWLH869jMexNFuuvug9YjC2RXqakOpk5dR+/e9YmMDCUoKID58x+iYsUSBAR4f3WPyl8xMTGEh4dTrVo1xIZiUVX0GGM4efIkMTExVK9ePd+26/1nt5CC6chnzZpYmjWbxtNP/8LQoZl3M1WrltIkobKVlJREZGSkJglVYESEyMjIfL+L9f47Cg+Li0ti2LDfmDx5LcZAlSolueuuOnaHpbyEJglV0DzxN6eJ4jKMMfz3v1t4/vkF/PNPAgEBfgwe3ILXXmurDfgppYoU7ywzSToDiwd6dBcbNx6ld+/v+OefBG6+uTJ//fUE773XQZOE8ir+/v40btyY+vXrc+edd3LmzJmMeVu2bOHWW2+lTp061K5dm7feegtjMltf/uWXX2jatCn16tWjSZMmvPDCC3Z8hRxt2LCBfv362R1Gjt59911q1apFnTp1WLBgQbbLLF68mBtuuIHGjRvTunVrdu+23sM6ePAg7du3p0mTJjRs2JB586yepTdv3syjjz5aUF/BunL2pp8bb7zRmL3zjHkf62f9eJNfUlPTLhp//vn5Ztq09SYtzZFv+1BFx9atW+0OwRQvXjxj+OGHHzZvv/22McaYxMREU6NGDbNgwQJjjDHnzp0znTt3NhMnTjTGGLN582ZTo0YNs23bNmOMMampqWby5Mn5GltKSspVb+O+++4z0dHRBbrPvNiyZYtp2LChSUpKMnv37jU1atQwqamplyxXu3btjL+XSZMmmUceecQYY8zjjz+ecdy3bNliqlatmrHObbfdZg4cOJDtfrP72wPWmSs873pn0VP6VU9EXbjhmXzZ5JIl+xg4cB4ff9yNNm2qAjB2bKd82bZSfOChuooX3O9/pWXLlmzaZDUt89VXX9GqVSs6duwIQGhoKBMnTqRdu3Y89dRTjB49mmHDhlG3bl3AujMZMGDAJdtMSEjg6aefZt26dYgIr7/+Oj169CAsLIyEhAQAZs+ezdy5c5kxYwaPPvoowcHBbNiwgVatWvH9998THR1NqVJWY561a9dmxYoV+Pn50b9/fw4ePAjAhx9+SKtWrS7a99mzZ9m0aRONGjUCYM2aNTz77LMkJSUREhLC559/Tp06dZgxYwbff/89CQkJpKWlsWzZMsaMGcM333xDcnIy99xzD2+++SYAd999N4cOHSIpKYlnn32WJ554wu3jm52ffvqJXr16ERQURPXq1alVqxZr1qyhZcuLu0MQEeLj4wGIi4vj2muvzXE6wJ133smsWbN48cUXrypGd3hnokhX8uof/zp27BxDhizkiy82AjB27MqMRKGUr0hLS2Px4sUZxTRbtmzhxhtvvGiZmjVrkpCQQHx8PH///bdbRU1vvfUWJUuWZPPmzQCcPn0613ViYmL4888/8ff3Jy0tjR9++IG+ffuyevVqqlatSvny5XnwwQd5/vnnad26NQcPHqRTp05s27btou2sW7eO+vUzW0CoW7cuv//+OwEBASxatIhXXnmF7777DoC//vqLTZs2ERERwa+//squXbtYs2YNxhi6d+/O8uXLadOmDdOnTyciIoLz589z00030aNHDyIjIy/a7/PPP8+SJUsu+V69evXipZdeumhabGwsLVq0yBivVKkSsbGxl6z76aef0qVLF0JCQihRogSrVq0C4I033qBjx4589NFHnDt3jkWLMp+4bNq0KaNGjdJE4UkOh+Gzz/5i6NBFnD6dRFCQP8OHt2HIkJvtDk35ojxc+een8+fP07hxY2JjY4mKiqJDhw75uv1FixYxa9asjPHSpUvnus7999+Pv78/AD179mTEiBH07duXWbNm0bNnz4ztbt26NWOd+Ph4EhISCAvLbKL/yJEjlC1bNmM8Li6ORx55hF27diEipKSkZMzr0KEDERERAPz666/8+uuvNGnSBLDuinbt2kWbNm2YMGECP/zwAwCHDh1i165dlySKcePGuXdw8mDcuHHMmzeP5s2bM2bMGAYPHsynn37K119/zaOPPsoLL7zAypUr6dOnD3///Td+fn6UK1eOw4cP53ss2fHORLHvl6tbfd9p/vWvH/jzz0MAdOxYk0mTulCrVkR+RKdUoRESEkJ0dDSJiYl06tSJSZMm8cwzz1CvXj2WL19+0bJ79+4lLCyMEiVKcP3117N+/fqMYp28cn1EM+sz/cWLF88YbtmyJbt37+b48eP8+OOPDB8+HACHw8GqVasIDg7O8bu5bvvVV1+lffv2/PDDD+zfv5927dplu09jDC+//DJPPvnkRdtbunQpixYtYuXKlYSGhtKuXbts30fIyx1FxYoVOXToUMZ4TEwMFStWvGiZ48ePs3HjRpo3bw5YybNz584AfPbZZ8yfPz/jWCUlJXHixAnKlSuXUcRWELzvqSfHBYieaA0HhF7RJkqUCGLnzpNcc00Ys2b1YP78hzRJKJ8WGhrKhAkT+OCDD0hNTeWhhx5ixYoVGUUZ58+f55lnnskoxhgyZAjvvPMOO3fuBKwT99SpUy/ZbocOHZg0aVLGeHrRU/ny5dm2bRsOhyPjCj07IsI999zD4MGDiYqKyrh6Ty9uSRcdHX3JulFRURlPB4F1R5F+Ep4xY8Zl99mpUyemT5+eUYcSGxvLsWPHiIuLo3Tp0oSGhrJ9+/aM4p+sxo0bR3R09CU/WZMEQPfu3Zk1axbJycns27ePXbt20axZs4uWKV26NHFxcRnHeuHChURFRQFQpUoVFi9eDMC2bdtISkrKuIvauXPnRUVvnuSFiSItc7jVCLdXW7BgN8nJqQBERoYyZ04vtm9/ip496+tLUapISH/E8uuvvyYkJISffvqJt99+mzp16tCgQQNuuukmBg0aBEDDhg358MMP6d27N1FRUdSvX5+9e/dess3hw4dz+vRp6tevT6NGjTKutEeNGkW3bt24+eabqVChQo5x9ezZk5kzZ2YUOwFMmDCBdevW0bBhQ+rVq5dtkqpbty5xcXGcPWt1NfDiiy/y8ssv06RJE1JTUy+7v44dO/Lggw/SsmVLGjRowH333cfZs2fp3LkzqampREVF8dJLL11Ut3Clrr/+eh544AHq1atH586dmTRpUkaxW5cuXTh8+DABAQFMmzaNHj160KhRI7788kvGjBkDwAcffMC0adNo1KgRvXv3ZsaMGRnnqyVLltC1a9erjtEdYow9ZadXqmnj6826Plshsh48uiXX5Q8diuOZZ+bz44/beeut9gwf3qYAolTKugJMvzJUnjFu3DjCw8P597//bXcoBSo5OZm2bduyYsUKAgIurUHI7m9PRNYbY5peyf68744i7tKrmuykpjoYO3YlUVGT+PHH7YSFBRIRoc1/K+VLBgwYQFBQkN1hFLiDBw8yatSobJOEJ3hfZXaqs3Iph/4nVq2KoX//uWzceBSAHj2iGD++MxUrliiICJVSBSQ4OJg+ffrYHUaBq127NrVr1y6w/XlfogDoPAOqZ182t3p1DDff/BnGQLVqpZg48Q66dr0u22WV8jRjjNaBqQLlieoE70wUNbpBSGS2s5o1q0inTrVo0uQahg9vQ2ho/nXeoVReBAcHc/LkSW1qXBUY4+yPIqfHiq+EdyYKF7t2neT55xcwdmwnrrvO+of8+ecH8fPTf0xlr0qVKhETE8Px48ftDkUVIek93OUnr00UycmpjBq1gnffXUFychrBwQHMnv0AgCYJVSgUK1YsX3sZU8ouHn3qSUQ6i8gOEdktIpe8jSIiQSLyX+f81SJSzZ3tLl5yiIYNp/LGG8tITk6jb9/GTJ3aLb/DV0ophQfvKETEH5gEdABigLUiMscYs9VlsX7AaWNMLRHpBbwH9Lx0a5n2nSrF7V1/AiAqqgxTp3bTRvyUUsqDPHlH0QzYbYzZa4y5AMwC7sqyzF3A/3MOzwZuk1xq/U4nhhAc7M8779xKdHR/TRJKKeVhHnszW0TuAzobY/7tHO8DNDfGDHJZ5m/nMjHO8T3OZU5k2dYTQHrD8PWBvz0StPcpA5zIdamiQY9FJj0WmfRYZKpjjAm/khW9ojLbGPMJ8AmAiKy70tfQfY0ei0x6LDLpscikxyKTiKy70nU9WfQUC1R2Ga/knJbtMiISAJQETnowJqWUUnnkyUSxFqgtItVFJBDoBczJsswc4BHn8H3Ab8bbWilUSikf57GiJ2NMqogMAjZa3ZoAAAcmSURBVBYA/sB0Y8wWERmB1cn3HOAz4EsR2Q2cwkomufnEUzF7IT0WmfRYZNJjkUmPRaYrPhZe18y4UkqpguV9zYwrpZQqUJoolFJK5ajQJgpPNf/hjdw4FoNFZKuIbBKRxSLis28h5nYsXJbrISJGRHz20Uh3joWIPOD829giIl8VdIwFxY3/kSoiskRENjj/T7rYEaenich0ETnmfEctu/kiIhOcx2mTiNzg1oaNMYXuB6vyew9QAwgENgL1siwzEJjqHO4F/NfuuG08Fu2BUOfwgKJ8LJzLhQPLgVVAU7vjtvHvojawASjtHC9nd9w2HotPgAHO4XrAfrvj9tCxaAPcAPx9mfldgF8AAVoAq93ZbmG9o/BI8x9eKtdjYYxZYoxJdI6uwnpnxRe583cB8BZWu2FJBRlcAXPnWDwOTDLGnAYwxhwr4BgLijvHwgDpXVyWBA4XYHwFxhizHOsJ0v/f3r2GSF2FcRz//rpYdqEoKbSiLSzt4qWysHxhpkkXMgpRRCuj6EIXtOxFWGTQi8AMMjHtAiqkkZUlJl0IzZK1svASZRYmJUj5wiTMovTXi3M2p2125r+bzs7uPh8Y2Dnzvzxz2P0/c85/9jktuR5Y4GQNcLykntWOW6+J4hTgx5Ln23Jb2W1s/wXsAsqvZtSxFemLUreRPjF0RlX7Ig+lT7P9di0DawdFfi/OBs6WtFrSGklX1Sy62irSF9OACZK2AcuB+2oTWt1p7fUE6CAlPEIxkiYAg4Ch7R1Le5B0CPA0MLGdQ6kXh5Gmny4njTJXSepn+5d2jap9jAPm2Z4h6VLS/2+db3tfewfWEdTriCLKf+xXpC+QNAKYCoyy/UeNYqu1an1xLKlo5EpJW0lzsEs76Q3tIr8X24Cltv+0/T2wmZQ4OpsifXEb8CqA7UbgSFLBwK6m0PWkuXpNFFH+Y7+qfSHpAmAuKUl01nloqNIXtnfZ7mG7wXYD6X7NKNttLoZWx4r8jbxJGk0gqQdpKmpLLYOskSJ98QMwHEDSOaRE0RXXqF0K3Jy//TQY2GV7e7Wd6nLqyQev/EeHU7AvpgPHAIvz/fwfbI9qt6APkoJ90SUU7It3gZGSvgL2Ag/Z7nSj7oJ98SDwgqTJpBvbEzvjB0tJi0gfDnrk+zGPAYcD2J5Duj9zDfAd8Btwa6HjdsK+CiGEcADV69RTCCGEOhGJIoQQQkWRKEIIIVQUiSKEEEJFkShCCCFUFIki1B1JeyWtK3k0VNi2oaVKma0858pcfXR9LnnRpw3HuEvSzfnniZJ6lbz2oqRzD3Ccn0kaWGCfSZKO+r/nDl1XJIpQj/bYHljy2Fqj8463PYBUbHJ6a3e2Pcf2gvx0ItCr5LXbbX91QKLcH+dsisU5CYhEEdosEkXoEPLI4SNJX+THZWW2OU/Sp3kUskHSWbl9Qkn7XEmHVjndKqB33nd4XsNgY671f0Ruf1L71wB5KrdNkzRF0mhSza2X8zm755HAoDzq+Ofinkces9oYZyMlBd0kPSdprdLaE4/ntvtJCWuFpBW5baSkxtyPiyUdU+U8oYuLRBHqUfeSaaclue1n4ErbFwJjgZll9rsLeMb2QNKFelsu1zAWGJLb9wLjq5z/OmCjpCOBecBY2/1IlQzulnQicANwnu3+wBOlO9t+DVhL+uQ/0Paekpdfz/s2GQu80sY4ryKV6Wgy1fYgoD8wVFJ/2zNJJbWH2R6WS3k8AozIfbkWeKDKeUIXV5clPEKXtydfLEsdDszKc/J7SXWLmmsEpko6FXjD9reShgMXAZ/l8ibdSUmnnJcl7QG2kspQ9wG+t705vz4fuAeYRVrr4iVJy4BlRd+Y7R2StuQ6O98CfYHV+bitibMbqWxLaT+NkXQH6e+6J2mBng3N9h2c21fn83Qj9VsILYpEETqKycBPwADSSPg/ixLZXijpE+BaYLmkO0krec23/XCBc4wvLSAo6YRyG+XaQpeQisyNBu4FrmjFe3kFGANsApbYttJVu3CcwOek+xPPAjdKOgOYAlxse6ekeaTCd80JeN/2uFbEG7q4mHoKHcVxwPa8fsBNpOJv/yLpTGBLnm55izQF8wEwWtJJeZsTVHxN8W+ABkm98/ObgA/znP5xtpeTEtiAMvv+Sip7Xs4S0kpj40hJg9bGmQvaPQoMltSXtHrbbmCXpJOBq1uIZQ0wpOk9STpaUrnRWQj/iEQROorZwC2S1pOma3aX2WYM8KWkdaR1KRbkbxo9ArwnaQPwPmlapirbv5Oqay6WtBHYB8whXXSX5eN9TPk5/nnAnKab2c2OuxP4Gjjd9qe5rdVx5nsfM0hVYdeT1sfeBCwkTWc1eR54R9IK2ztI38halM/TSOrPEFoU1WNDCCFUFCOKEEIIFUWiCCGEUFEkihBCCBVFogghhFBRJIoQQggVRaIIIYRQUSSKEEIIFf0NFZL4TjT1MVIAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "eval_pool = Pool(X_validation, y_validation, cat_features=cat_features)\n", "curve = get_roc_curve(model, eval_pool)\n", "(fpr, tpr, thresholds) = curve\n", "\n", "plt.figure()\n", "lw = 2\n", "roc_auc = sklearn.metrics.auc(fpr, tpr)\n", "plt.plot(fpr, tpr, color='darkorange',\n", " lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)\n", "\n", "plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n", "plt.xlim([0.0, 1.0])\n", "plt.ylim([0.0, 1.05])\n", "plt.xlabel('False Positive Rate')\n", "plt.ylabel('True Positive Rate')\n", "plt.title('Receiver operating characteristic')\n", "plt.legend(loc=\"lower right\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [], "source": [ "from catboost.utils import get_fpr_curve\n", "from catboost.utils import get_fnr_curve" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [], "source": [ "? get_fpr_curve" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8VfX9+PHXOzeTEFYARYZgZcsQ4q6Ko9YJ4kBxVOusFltnf45i3dWvtVWr1rr3QIsWJ1oVV9UaHCgggsoIQyAgIyTkJnn//vicm9yEjEu45557b97PPs7jnpVz3jmV+85nnM9HVBVjjDGmKRlBB2CMMSa5WaIwxhjTLEsUxhhjmmWJwhhjTLMsURhjjGmWJQpjjDHNskRhjDGmWZYoTNIRkYUiUi4iG6OWHUSkr4ho1L6FInJ51M+piJR5x5aKyF9FJNTMfU4XkeoG97nLO/aId73do87fWUQ0anuGiFR4P7daRKaKSA+/nosxQbFEYZLVUaraPmpZFnWsk6q2ByYCV4vIoVHHRnjH9gdOAM5o4T4fNbjPpKhja4AbWvj5Sd79dgbaA3+J5ZfbFiKS6fc9jIlmicKkLFX9CJgN7NLIsQXAh8DIbbjFo8BwEdk/hlh+Al5s7n4ikicit4nIIhFZJyIfePvGiEhJg3MXisjB3vo1IvK8iDwhIuuBK70SV5eo83f1SjVZ3vYZIjJXRNaKyHQR2dHbLyLyNxFZKSLrReQrEdni+RkTzRKFSUneF94+wFDg80aODwL2BRZsw202ATcBN8YQTyFwTAv3+wswGtgb6AL8AaiJMZZxwPNAJ+BW4CPg2KjjJwHPq2pYRMYBV3rxdAPeB572zjsE2A8YAHQEJgClMcZg2ihLFCZZvSgiP3nLiw2OrcZVCz0AXK6qb0Ud+0xEyoC5wAzgnhbus2fUfX4SkT0bHP8n0EdEDmvi5+8UkXVeTF2BCxo7SUQycNVgv1fVpaparar/VdXNLcQX8ZGqvqiqNapaDjyFq3pDRAQ40dsH8Bvgz6o6V1WrcMlupFeqCAMFwCBAvHOWxxiDaaMsUZhkdbSqdvKWoxsc66qqnVV1sKre2eDYKFxbwQnAHkA+gIjsG9VgPTvq/I+j7tNJVT+Ovpj3RX69tzTmd6raERgOdAZ6NXFeVyAX+K75X7tJSxps/wvYy2s83w9XMnnfO7YjcEck+eGSqgA9VfVt4C7gbmCliNwnIh1aGZNpIyxRmLSjzhRc9czV3r73oxqsh27lJR/GVfkc08w9v8I1fN/t/YXf0GqgAvhZI8fKgHaRDa+nVreGt2hwv7XAG7iEeBLwjNYNBb0EOLdBAsxT1f96P3unqo4GhuCqoC5r6vcyBixRmPR2M3C2iGy/LRfxqm/+BPy/Fk59FNgOGNvINWqAh4C/el19QyKyl4jkAN8CuSJyhNcY/UcgJ4bQngJ+BRxHXbUTwL3AFSIyFEBEOorI8d76biKyh3efMlzyirWdxLRRlihM2vL+yn+P+PzF/DTQbF2+qlYCdwCTmzjlUuAr4FNcddAtQIaqrgPOx7W5LMV9gZc0cY1o04D+wApV/TIqjhe8az/j9ZL6Goi0sXQA7gfWAotwDdm3xnAv04aJTVxkjDGmOVaiMMYY0yxLFMYYY5plicIYY0yzLFEYY4xpVsoNLta1a1ft27dv0GEYY0xKmTlz5mpVbfh+TkxSLlH07duX4uLioMMwxpiUIiKLWvuzVvVkjDGmWZYojDHGNMsShTHGmGZZojDGGNMsSxTGGGOaZYnCGGNMs3xLFCLykDcv79dNHBcRuVNEFojILBEZ5VcsxhhjWs/P9ygewc2k9VgTxw/DDZHcHzcT2T+8z2atXw9vvgmZmRAKuSWy3ti+WI83OtWMMcakqPJwOfPXzGdj5UY2Vm7cpmv5lihU9T0R6dvMKeOAx7xZuT4WkU4i0qOl+Xvnz4dDDoljoB6RbUs00Z/5+dCnD/Tu7T779IH27VuOIScHunSBwkK3bowxraGqjL5vNHNXz43L9YJ8M7sn9ecBLvH2bZEoROQc4ByA3Nxh7LMPVFdDVZX7jF5vbF8sx1UhHHZLMmjf3iWMyNKtW13i2XFH99mxo0twO+zgEpQxxgDMXjWbuavnkpeZx4jtR5Cflc9bvNXq66XEEB6qeh9wH0BRUZH+5z/xvj7U1MQv+axfD0uWwOLFblmyBMrLW46jvBxKS92ycaNbFsXw0v3ee8OHH277czDGpId3F74LwDGDj+GJY54AQE5rff16kIliKdA7aruXty/hROqqkbKzg4igPlWXbCJJY/VqWLnSJZxFi1zyWbQINmyAZcvgv/+FkhLo1SvoyI0xyWDxusUADO02NC7XCzJRTAMmicgzuEbsdS21T7QVIq5aqWNH2Gmn5s8dPx5efNFVS112GZxwAowenZg4jTHJaU35GgAK2xXG5Xp+do99GvgIGCgiJSJypoj8RkR+453yKvA9sAA32fv5fsWSzu65BwYOdOu33gq77eaSxRNPwLRpsNxSrzFtTml5KQBd8rrE5Xp+9nqa2MJxBX7r1/3bih49YPZseOcduPdemDoVpkxxC0BGBhxwAOy7r2vL2H//5KheM8b4J1KiSPpEYRInFIKDD3bLd9+5ZPHRR64N49134a233ALQsydcdRWceaYlDGPS1dqKtQB0zu0cl+tZokgzP/uZa6uIKC11pY0PP4TXXoN58+D88+FPf4Jx42C77eDCC6Fr1+BiNsbE16bwJgDaZ8fwAlcMbKynNFdYCMcdB3/7G8yZAzfc4F4IXLUKHngAbrwRuneHzZuDjtQYEy9llWUAtMtqF5frWaJoQzIyXLXT6tXw1FNw9dVuvyrssgs8/HDyvHBojGm9SInCEoVptdxcmDgRrr0WnnsO+veHBQvgjDNg553h7393VVbGmNQUSRT52flxuZ64zkepo6ioSIuLi4MOI62Ew/DMM/DnP8Ncb2iYUMiVMgYNgl/9qv5YVQUFMHKkDaRoTDIKV4fJviGbkIQITw4j3j9UEZmpqkWtuaY1ZhuysuDUU+Hkk+Hf/3bvZsyYAV9+6ZZnn93yZwYPhjvugF/8IuHhGmOaEV3tJHH6a84ShamVkeHe9B4/Hn76Cb74Ap580vWUirZggSt5HHooHHWUq7q65BLYfvtg4jbG1CkLu4bseFU7gSUK04ROnWDMGLc0FA673lM33OBKIADPPw/Fxa6XlTEmOPFuyAZrzDatkJXlGsJnzYL773fDnC9c6OYJ+cc/3LoxJhi1DdlZ8StRWKIwrTZ0KJx1lhtXCuCzz9zLfP36wdix8OmnwcZnTFsU73cowBKFiYMDDnBvfl94IRx/vOt++9JLsPvuridVinWsMyalWdWTSVp77+3e/p4yxTV+n3ii23/llW5Awu+/DzY+Y9oKPxqzLVGYuOvTB55+2nWrzchwpY1ddoHHHgs6MmPSn5UoTEqZMMENgf7LX7ppXs88E954I+iojElv1phtUs6gQfDyy65xu6rKJY0994R164KOzJj0ZI3ZJiVlZsILL8DFF7vtTz5xCaOmJti4jElHVvVkUlZGBtx2m5vfOy/PJYuuXeHuuy1hGBNPtY3ZVvVkUtW4cW4ODIC1a2HSJLjmmkBDMiatWInCpIULL4T5893b3QDXXw+HHVY3cq0xpvWsjcKkBRE378WFF7phPwBefx2GDXO9pOwFPWNab1NVfOeiAEsUJkAdOsD06e49i8xMqK5271v06+dm4nvrraAjNCb1WNWTSUt77+1m1DvtNJcwFi2Cm25y1VE//hh0dMaklkjVkzVmm7TToQM88ghs3uyGAQE3nHlkwEFjTGysRGHSXkaGG1jwhRfc9p/+5CZQMsbExhKFaTOOPhpOOgnKytyb3Fdc4UobxpjmlVeVA5CXlRe3a1qiMEnrwQfdKLSbN8PNN7vhQG6+GZYvDzoyY5JXRVUFALmZuXG7piUKk7Ryc92c3Q88AD17upnzrrgC+vZ13WmNMVuKJIq8TCtRmDYiI8ONOrtoEbzyChx+OFRWuh5Rw4bB558HHaExyaU87KqecjJz4nZNSxQmJYRCLklMmwYHHuj2ff01jBoFhx4Kq1YFG58xycKGGTdtXigE//mPe+/iiCPcvunToX9/N5y5MW2ZqlpjtjHghgDp0sXNyz15squCWrfOdat96SU3FevSpUFHaUziRTdkZ0j8vt59TRQicqiIzBORBSJyeSPH+4jIOyLyuYjMEpHD/YzHpBcRuO46+PJLOOYYqKhwEyQNGgS9esHUqUFHaExiRaqd4tmQDT4mChEJAXcDhwFDgIkiMqTBaX8EpqjqrsCJwD1+xWPSlwg89RT89rcwYIBrAAc4+WSbetW0LX68bAf+lih2Bxao6veqWgk8A4xrcI4CHbz1jsAyH+MxaSwnB+66y1U7VVbCUUe5EsaECbBmTdDRGZMYqZgoegJLorZLvH3RrgFOEZES4FXggsYuJCLniEixiBSvsu4tpgWhEDzzjHvfYt0694a3zdFt2oJUTBSxmAg8oqq9gMOBx0W2bIFR1ftUtUhVi7p165bwIE3qadcO/vUvVy01fXrdfN3GpLNUTBRLgd5R2728fdHOBKYAqOpHQC7Q1ceYTBsyahT8+c9u/e23g43FmESIdI1NpUTxKdBfRPqJSDausXpag3MWAwcBiMhgXKKwuiUTN5deCgUFbviPI45wc15cfDFs3Bh0ZMbEX22vpzi+QwGQGderRVHVKhGZBEwHQsBDqjpbRK4DilV1GnAJcL+IXIRr2D5d1SbCNPETCsH48fDYY/Dqq3X7Kyvh7393VVPGpAu/qp58SxQAqvoqrpE6et/VUetzgH38jMGYu+5yw3xUVsLatXDRRXD33W502rfecjPsGZMOUjJRGJMMCgpg4sS67e7d3TsWFRWutPHVV26fMamuNlFkpk4bhTFJ6aSTYMMG9/b2ypWw3Xb1q6WMSVWp2OvJmKTVvj08/7xrwwC44YZg4zEmHiJDjFuiMCZO9tjDjUIL8NFHdfN0G5Oq/Or1ZInCtGkdO8Kdd7r1q69u/lxjkp1VPRnjk7PPdp9z5kB1dbCxGLMtLFEY45PcXOjWDWpq4KabXAO3MaloU5UlCmN8M2qU+7z6aujZ071bsXhxsDEZs7WsRGGMj+66C847D4YPh6oq17g9frxVRZnUYr2ejPHRzjvDPffAF1/ANG9Ess8+c0OVH3cc3H57oOEZExMrURiTACJu0qO774Y+faCkxA1XftFFcM45bhgQY5JVyk2FakwqO/98+OEHeP99KCpy++6/H3bf3fWOMiYZWYnCmATLyICf/xw+/RT+9z/XO+rLL13JwphkZInCmADttht8+KFb//BD15ZhTLKxRGFMwEaNgt//3q0feig88QSEw8HGZEy0VJzhzpi0c/XV7h2LH3+EU0+FYcPgnXdg+fKgIzNtnaraWE/GJIMuXeC99+D66yE7G+bNgwMPhB12qOtWa0wQKqoqAMgJ5ZAh8f1qt0RhzFYKheCPf6ybhzti/HiYNSuwsEwb51f7BFiiMKbVevSAl1+G8nLYcUc3VtS++8Lq1UFHZtoiSxTGJLHcXDdDXkEBrF8PF1/splk1JpEsURiT5IYMqZvP4vHHYeBAN16UMYniV48nsERhTNyce64bWHDgQDfy7EEHwVtvBR2VaSusRGFMCigocAMLfv01nHGGa7s4+GBYsSLoyExb4FfXWLBEYUzcZWa6caF23NFtH3mkSx7G+MlKFMakmIwMuPlmtz5zpnsx74EHrJHb+McShTEp6MQT3VvbEWefDU8/HVw8Jr1ZojAmRY0ZA2++Wbd9xhkwe3Zg4Zg0Vju7XWZAiUJE8kRkYNzvbkwbcPDBMHVq3fbZZ9sUqyb+Ai1RiMhRwBfA6972SBGxUW2M2Qrjx8Nrr7n1jz6C0aNh3bpgYzLpJeheT9cAuwM/AajqF0C/uEdiTJo79FA3pSq4CZCGD3ddaI2Jh6DbKMKq2vBvH417JMa0AX/9q5tKNSvLvZRnjdsmXoJOFLNF5CQgJCL9ReTvwH/jHokxbcTgwfCPf7j1J58MNhaTPoJOFBcAQ4HNwFPAOuD3sVxcRA4VkXkiskBELm/inAkiMkdEZovIU7EGbkwqO/ZYyMmBt992pYxNm4KOyKS6oMd6OkJVr1LV3bzlj8DYln5IRELA3cBhwBBgoogMaXBOf+AKYB9VHQpcuNW/gTEpqFMn+H//z61fcgkUFVl7hdk2QZcorohxX0O7AwtU9XtVrQSeAcY1OOds4G5VXQugqitjuK4xaeGaa+D220EE5s6FyZNh8+agozKpys9EkdnUARE5DDgc6Ckid0Yd6gBUxXDtnsCSqO0SYI8G5wzw7vUhEAKuUdXXY7i2MSlPBH7/e9hpJxg7Fm67DV55BYqLIT8/6OhMqqntHpuZ2O6xy4BioAKYGbVMA34Zp/tnAv2BMcBE4H4R6dTwJBE5R0SKRaR41apVcbq1McnhyCPhtNPc+jffwLhx8MMPwcZkUk8gVU+q+qWqPgrsrKqPRi1TI1VFLVgK9I7a7uXti1YCTFPVsKr+AHyLSxwNY7lPVYtUtahbt24x3NqY1CECjzziBg/Mz3dzWIwaBc8+G3RkJpUE3UbRV0Se93omfR9ZYvi5T4H+ItJPRLKBE3GlkWgv4koTiEhXXFVULNc2Ju2MGgWffOK6z/70kxtU8IMPgo7KpIqgez09DPwD1y5xAPAY8ERLP6SqVcAkYDowF5iiqrNF5DoRifSamg6Uisgc4B3gMlUt3fpfw5j0MHSoG+JjwAC3/ZvfwOefBxuTSQ1+lihEtfmXrEVkpqqOFpGvVHVY9L64RxODoqIiLS4uDuLWxiRMaSnsuScsWOC2jzkGHnzQdas1pjH5N+WzKbyJjVdsJD97y94Q3vd2UWuuHUuJYrOIZADzRWSSiIwH2rfmZsaY2BQWuraKAw9021OnulLGxo3BxmWSk6oGPijg74F2wO+A0cCpwGlxj8QYU0+fPi5ZTPNa9latgt/9Dqpi6Zxu2pSKKjd1Yk4ohwyJ/zRDLV5RVT9V1Y2qWqKqv1bVY3BdZ40xCXDUUa4HlAg8/DDk5UG3bq56yhjwt30CWkgUIrKXiBwnIt297eHeeEwf+hKNMaZREya40kW/fq5EsXq1SxZ33x10ZCYZ+NnjCZpJFCJyK/AQcCzwiojcALwBfEIj7zoYY/x1wAHw3Xfw7rvQsSOowqRJbhKkoqL6y/77uyHMW+irYtKE3yWKJofwAI4AdlXVChHpjBuOYxdVXehLJMaYFonAfvvBmjWw117wv//BZ581fu5778GLL8Jjj7mRak36CjJRVKhqBYCqrhWR+ZYkjEkOGRnw8cfw1VdQWVn/WE0NvPQS/PnPMGWKSy5//7s71rkzZDb3r96kJD97PEHziWKnBnNj94veVtUWhxo3xvhHxE2n2pjdd4ejj4a993YN4ZHhQHr0gMcfh4MOSlycxn9BligaDgl+my8RGGN8MXq0m0HvD3+ADRtcA/jy5XDwwa6949//hoKCoKM08RBYolDVd325ozEmYY47zi3gJkY6/HCYMQPeecd1uz39dAiF4JBDYLvtgozUbIvysL+9nqy20pg2Ii/PJYh581yV1LvvuiVi0iQ49VTYbTdXrWVSR6DvURhj0s/Aga6n1KRJcMopsMMObv9dd8Eee8AuuzTdk8okp9pEkRnMC3chEfmLL3c2xgRmxx1dT6jHH4eSEtc76tBD3bE5c9yMeyttYuKUEWSvJ1S1WkR+7sudjTFJQQSOP94tCxe69zSWLHFtFuecA+28P1Lbt4eLLnJdbK1qKrkE2esp4nOvW+xzQFlkp6pO9SUiY0xg+vaF1193XWvnz4f77qt//IYbXBfbv/wFTjopkBBNI5IhUeQCpcCBUfsUsERhTBoaMgS+/dbNrvfpp27fsmXwwAOwfr3rYnvyye6t8NGjXZWVzVAcLL/HemoxUajqr325szEmqf38526JuPVWN3bUbbfBZZfBHXe4/R06wBlnwFlnuRn6TOIF3utJRHqJyAsistJb/iUivXyJxhiT1ETg0kth+nSXGMaMcaWM22+HESPgj38MOsK2KRmqnh4GngKO97ZP8fb9wpeIjDFJ75BD3AJQXOyGO3/kEbjxRhg0CPbdt/75ItCrlxujysRf4CUKoJuqPqyqVd7yCGA1ksYYwA1r/tBDdW92n3qqaxSPXnbcEY491oY990tt99hMf7rHxpIoSkXkFO+dipCInIJr3DbGGMCVGGbMcIMU9umz5ZKZ6YY879vXVVuZ+EqGEsUZwARgBbAcOA6wBm5jTD2DBsGXX8KiRVsu11/vxpRavNj1kmrXDrp3hyuugHXrgo489fn9wl2Lb2YDx6jqWFXtpqrdVfVoVV3sSzTGmLR0+eVQUQHXXQfZ2W6AwlWr4Oab3XsZ++3n3gg3rVNa7ip5uuR18eX6zSYKVa0GJvpyZ2NMm5KZCZMnu15Sa9bA1Ve7saXKy+H9992MfXvvXX+5/PKgo04N6ypcsaxzbmdfri/aQuuSiPwNyAKepf6b2YEMG1ZUVKTFxcVB3NoYE2eqsGABXHwxvPxy4+cccIAbOuSww2x2vsaoKqHrQihKeHKYzIzGH5KIzFTVotbcI5bHPtL7vC46Nuq/qW2MMVtNBPr3h2nTXPvGpk11x/79bzdUyDvvuKVDBzjhBHesWzeXXDIz3f62PPbU+s3rUZT8rPwmk8S2avaqIpIB/ENVp/hyd2OMwX3RjxxZf9/ee7uJle66C+65x1VZ3X9/3fGbbnKfhYWwYkXbLW38VPETAJ3z/Kl2gpbbKGqAP/h2d2OMacbgwe5lvq++gn/+0y0nnQRdutRN41paCnfeGWycQapNFD61T0Bs3WP/IyKXikhvEekSWXyLyBhjGthlFzfk+TnnuHnAS0tdCeOWW9zxN94INr4gra1YC0Cn3E6+3SOWRHEC8FvgPWCmt1hrsjEmcOPHu8+2/BJfIqqeYhk9tp9vdzfGmG2w006QkwObN7s2jlNPdb2jdtoJcnODji4xIomiY05H3+7RZIlCRP4QtX58g2M3+RaRMcbEKBRyL/GB6zV16aVuqPO8PDjvPPjwQ1i9um5ZuzbYeP2wfvN6ADrkdPDtHs1VPZ0YtX5Fg2OH+hCLMcZstT/8Ab77zjVyDx5ct//ee918Gt261S1durgk8vbbwcUbb5ESRVBtFNLEemPbjV9A5FARmSciC0SkyXcsReRYEVERadXLIMaYtm2nnVwj95w5bna+Bx6AX/7SdZ2NLBEVFXDQQW4okaeeCi7meAm06gn3Ul1j641tb8EbJ+pu4DBgCDBRRIY0cl4B8HvgkxajNcaYFvTvD2ee6eb+jq52UoXnnnPVVQDhsJvStXNn2H9/N5RIKgr6PYoRIrJeRDYAw731yPawGK69O7BAVb9X1UrgGWBcI+ddD9wCVGxt8MYYszWOOw6qquDHH12PqcxM+OkneO89Vy21dGnQEW69dZvdOE+BlChUNaSqHVS1QFUzvfXIdlYM1+4JLInaLvH21RKRUUBvVX2luQuJyDkiUiwixatWrYrh1sYY07Tu3WHqVFfSuPhit6+iAnr3dm+Cf/011NQEG2OsIo3ZHXODqXrylTc8yF+BS1o6V1XvU9UiVS3q1s0m1zPGxEfHjnDbbW7p3dtVT11wAQwb5mblix57KllFRo4NqtfTtloK9I7a7uXtiygAdgFmiMhCYE9gmjVoG2MS7eKLXSnisMNc8gAoKYH8fDcneDLzey4K8DdRfAr0F5F+IpKN6247LXJQVdepaldV7auqfYGPgbGqmuT/txhj0lGHDvDqq67N4tpr6/bvtltyN3avKV8DQNd2XX27h2+JQlWrgEnAdGAuMEVVZ4vIdSIy1q/7GmPMtrr6anjrrbrt995z07f+73/BxdSYGq1J+aonVPVVVR2gqj9T1Ru9fVer6rRGzh1jpQljTLI48EDXZnHmmXX7rrkmsHAatWHzBhSlILvAt7koIMDGbGOMSQUPPOAauwFeew169oRbb3VJJGiJGDkWLFEYY0yLzjjDdakFWLbMDRty5JGuwTvIhLG23CUKP1+2A0sUxhjTok6dXIL46KO6fa++6rrUjhwZ3GCDkRKFn5MWgSUKY4yJSSgEe+7pJkwaPtxVQQHMmgVnnRVMTFaiMMaYJFRQ4IY0LymByZPdvqlTXeni/vthyZLmfz6eIl1jrURhjDFJ6pprYIg31GlJiZuqddSoxN0/kij8fNkOLFEYY0yrZWTA7NmuRHHMMW7f6tVwxRWJaeSOvJXt58t2YInCGGO22fjx8K9/uSHLAW6+2SWRxx7z976rNrlBUgvzCls4c9tYojDGmDiZPbv+9mmnwYUXwuLF/pQwVm9aDUC3fH8HS7VEYYwxcdKjh5vv4s036/bdcYcbiTYrC+65J77Dl68qcyWKbu0sURhjTMoIheDgg+Hzz+H44+v2V1fDb38L550Xv3tZicIYY1LYyJEwZYqrcnrzTejVy+2/7z64pMVZeGITaaOwxmxjjElxBx/s5ruI+Otf3Yx626KyupL1m9cTkpCN9WSMMemgY8f6M+bl5cHDD7f+epFqp8J2hWSIv1/lliiMMSZB8vLqVzudcYZLFq3pEVXbPuFzQzZYojDGmIT6y1/cECARZ5wB55679deJ9Hjyu30CLFEYY0zCDR/uRqMVcdv33w9du9Zvx2jJio0rANiu/XY+RFifJQpjjAlAjx4QDsOuu7rt0lIYNiz2aqiS9SUA9Cro5VOEdSxRGGNMQEIhmDkTHn+8bt+UKbH97NINSwHo1cEShTHGpDUROOUUGDHCbU+aFNvP1ZYoLFEYY0zbcO217nP1avjii5bPjySKnh16+hiVY4nCGGOSwFFH1a2/8krL51vVkzHGtDEZGXWlij/+Eb76qulzw9Vhlm9YjiD0aN/D/9h8v4MxxpiYHHdc3frvftf0eSs2rkBRtmu/HVmhLN/jskRhjDFJYsgQ+Mc/3PqMGU0PSZ7IaiewRGGMMUnlrLPq1qO7zUZLZI8nsERhjDFJJTPTvXgHcPrpbh6LhhavWwxAzwL/ezxAqReOAAAUHElEQVSBJQpjjEk6U6fWrc+cueXxr1a6lu5BXQclJB5LFMYYk2R23hm6d3frt9665fGZy1z2GN1jdELisURhjDFJ6KST3Oerr9bfH64OM2fVHARhxPYjEhKLJQpjjElCF13kPjdtqj+q7Pdrv6daq+ndsTftstolJBZLFMYYk4R6965bv/POuvVI+8Sw7sMSFouviUJEDhWReSKyQEQub+T4xSIyR0RmichbIrKjn/EYY0yqEIFrrnHr//lP3f5ZP84CYPh2wxMWi2+JQkRCwN3AYcAQYKKIDGlw2udAkaoOB54H/s+veIwxJtX8/Ofu84cf3NwVAF/+6KbHS4tEAewOLFDV71W1EngGGBd9gqq+o6qR6cY/BhLz9ogxxqSA/fevW8/OhmXLlI9LPgYS1+MJ/E0UPYElUdsl3r6mnAm81tgBETlHRIpFpHjVqlVxDNEYY5JXZiZcdlnd9sTffs/KspV0a9eNnbvsnLA4kqIxW0ROAYqARnoMg6rep6pFqlrUrVu3xAZnjDEB+r//g/PPd+vvLXoXgH367INEJtxOAD8TxVIgqt2eXt6+ekTkYOAqYKyqbvYxHmOMSUk33eSt9HsbgP37HJjQ+/uZKD4F+otIPxHJBk4EpkWfICK7Av/EJYmVPsZijDEpq2NHmDVLaxNFVkliE0WmXxdW1SoRmQRMB0LAQ6o6W0SuA4pVdRquqqk98JxXjFqsqmO39l7hcJiSkhIqKiri+Bskh9zcXHr16kVWlv9jzhtjkldWj3lQsBw2dueWS4fw2+MTd2/fEgWAqr4KvNpg39VR6wfH4z4lJSUUFBTQt2/fhNbb+U1VKS0tpaSkhH79+gUdjjEmQG//4EoT/HAgaGK/55KiMXtbVVRUUFhYmFZJAkBEKCwsTMuSkjFm67z5/Ztu5YcDWbLEdZd9443E3DstEgWQdkkiIl1/L2NM7DaFN/Hmdy5R7F74S8C9gPfoo4m5f9okCmOMSVcfLv6QsnAZo3qM4uPpfbjlFrd/3rzE3N8SRZyEQiFGjhxZuyxcuJAZM2bQsWNHRo4cyeDBg7n22msB6u0fNGgQl156acDRG2OS2cvfvgzAmB3HIAKjvZeyG5vUyA++Nma3JXl5eXzxxRf19i1cuJB9992Xl19+mbKyMkaOHMlRRx0FULu/vLycXXfdlfHjx7PPPvsEEboxJolV1VTx1NdPATB+8HgABg+uO75+PXTo4G8MaVeiEPFn2Vb5+fmMHj2aBQsW1Nufl5fHyJEjWbp0i3cRjTGGDxZ/wOpNq+nfpT/79HZ/TO6wQ93xSZP8jyHtEkVQysvLa6udxo8fv8Xx0tJSPv74Y4YOHVpv/9q1a5k/fz777bdfokI1xqSQqXPdBNrjB42v17ll4kT3+dZb/seQdlVPqsHct7GqJ4D333+fXXfdlYyMDC6//HKGDh3KjBkzeP/99xkxYgTz58/nwgsvZPvttw8gamNMMquqqeLZ2c8CcPzQ+m/YXXghPP00LFvmZsFr5+Nkd2mXKJJNpC2iqf0//PADe+65JxMmTGDkyJEBRGiMSVYvf/syK8tWMqBwwBbDiu+2W936/Pkwwsfps63qKWD9+vXj8ssv55ZIfzdjjMGVJq5860oAzis6b4t3qkQgUmP99NP+xmKJIgn85je/4b333mPhwoVBh2KMSRKPfPEIc1fPZafOO3Fe0XmNnlNV5T4js9/5RTSoSv1WKioq0uLi4nr75s6dy+Do/mJpJt1/P2NMfZXVlQz4+wAWrVvEU8c8xcRhExs974EH4Oyz3XpLX+UiMlNVi1oTj5UojDEmyTz42YMsWreIwV0HM2HohCbPGzSobt3PIeEsURhjTBL5YsUXXPLGJQBcO+ZaQhmhJs/de++69fnz/YvJEoUxxiSJ8nA5p75wKuVV5Rw7+FiOG3Jcs+dnZECRV5n0zjv+xWWJwhhjksC6inUc8dQRfL3yawYUDuDRox+NafToLl3c5+WX+xebJQpjjEkCF7x2Ae8sfIeu7boydcJU8rPzY/q5s85yn+Xldb2g4s0ShTHGBEhVuej1i3h81uNkh7J5/9fvM7T70JZ/0HN81AvbzzzjQ4BYooibpoYZFxFeeuml2vOOPPJIZsyYAcCYMWMYOHAgI0aMYLfddmt0CBBjTPpasGYBRz97NLd/cjtZGVk8NPYhBnUd1PIPNhDpPX/vvXEO0GOJIk4iYz1Flr59+wLQq1cvbrzxxiZ/7sknn+TLL7/k/PPP57LLLktQtMaYIJVuKuXMf5/JgL8PYNq8aRRkF/DSxJc4efjJrbreuee6z8WL4xhklLQb60mu9WfqUP1T615MHDFiBOFwmDfffJNf/OIXTZ631157ceutt7Y2PGNMCigPl3P7x7fzl4/+wpryNWRmZHLysJO5/oDr6d2xd6uve/TRbpBAv2YrSLtEEZTIMOPgxm964YUXao9dddVVTJ48udlE8frrr3P00Uf7HqcxJrFUlU+XfcoTs57giVlPsLZiLQBj+o7h3iPuZWDXgdt8j8JC91lTA9XVEGr61YtWSbtE0dq//LdVU8OMA7VzTXzwwQdbHDv55JOprKxk48aN1kZhTBrZWLmR1xe8zh2f3MEHi+v+7Y/uMZrJ+01m7MCxMXV/jUX79nXr8+bBkCFxuWyttEsUyeqqq67ihhtuIDOz/iN/8sknGT16NJdddhkXXHABU6dODShCY8y22ly1mf8u+S/PzXmOKbOnUFpeCkBhXiG/GvErThl+CqN6jPLl3oWFUFoKr79uiSJlHXLIIUyePJnly5dvcUxEuP766/nZz37GN998w6BBW9/rwRiTWMs3LOeb1d/wbem3fLb8M4qXFzN31VzKq8przxndYzQThk7g3NHn0jG3o6/xDB/u3s72YygPSxQJdNVVVzFu3LhGj+Xl5XHJJZdw66238uCDDyY4MmNMYzZXbWbJ+iXMXTWXb0u/ZdmGZXy39jtmLp9JyfqSRn9maLehHDXgKE7c5USGbzc8btVLLTn2WJcoXn89/te2RBEnGzdu3GLfmDFjGDNmTO322LFjiR7WPfI+RcQll1ziV3jGmCasLFvJop8WMXf1XJauX8rSDUuZVzqPb0u/Zcm6JSiNt3t2yOnAsO7D6F/Yn0GFg9i7997s0n0XOud1TvBv4PT2Ok1t2hT/a1uiMMakleqaapZtWMbKspWsKV/D2oq1rC1fy9qKtawqW8Xi9YtZsXEFyzcsZ8XGFZSFy5q8VoZk0KdDH/p17sew7sPo1aEXOxTswKgeoxjUdRAZkjyvog0b5j5Xrox/zydLFMaYpKaqVFRVsHrTan4s+5GVZStZVbaKVZtWuXXvc2XZSpZtWMaKjSuo0ZqYr1+QXcDOXXamb6e+9O/Sn+753RnYdSADCgfQt1NfskPZPv528dOvX936a6/BkUfG79ppkyhUNWF1gYmUajMQmvRVozVUVFVQVVNFVU0V1TXVtetVNVVsrt7MpvAmyirL3Ge4rN52ZCkLl7GxciOl5aWUVZZRUVVRu2yu3lxvu6Kqgsrqyq2Odbv87ehR0IMueV3onNu59rNzXmd27LgjPQp60KN9D7Zvvz0dcjqkzXfHoEHwzTfwt79ZothCbm4upaWlFBYWps3/4eCSRGlpKbm5uUGHYlJIdU015VXl9b6cN4U3UR4up7yq3H0hV7kv5I2VG9lQuYENmzfUfkbOqaiqYP3m9ZSWl1K6qZQ15Wuo1upAfqfsUDZd23Wle3732qVbu271Prvnd2eHgh3Yrv12KVMKiLdf/QquvBLefhvWr4cOHeJz3bSYMzscDlNSUkKFn3MBBiQ3N5devXqRlZUVdCgmTmq0hvJwee1f12WVZTF/bqra1OTxSDLYXL3Zt9hzM3PJzMist4QkRGZGJtmhbPKz88nPyqddVjvys91nu8x29bbzs/LJz86nS14XOuR0ICeUQ25m7hZLTqbbnx3KTqq2gGS2fj109HrhTpsGRx1Vd2xb5sxOixJFVlYW/aIr6IxppXB1uNEv4fJw3V/Z0VUokWqVzdWbqaqpIlwddp81YcI1YTZs3sD6zetrl7UVa/mp4ifff492We0aXXIzc2u/mHMyc2if1Z6CnAIKsgtqP/Oz82u/rNtnt6cwr5DCdoV0yevSZv9STxUdOrj3KWbNgq+/rp8otoWviUJEDgXuAELAA6p6c4PjOcBjwGigFDhBVRf6GZNJDjVaU/vFGqmbjlSHhGvC9eq+o+vCwzXhelUnkTrtyurK2uuFa8KEq8O1x8qrymurXWq/5CvLqKyurD03Um/emvrw1sjNzK3313Wjn80da/CXe2Rfu6x25IRy0qoK1myd3XZzieLKK+GKK+JzTd8ShYiEgLuBXwAlwKciMk1V50SddiawVlV3FpETgVuAE5q77qbwJoqXuaqnhtVmDfs7p/NxVSVcE6Y8XE64JkyN1lBdU02N1rh1jVr39ldr1Beu95dvtVbXO15dU137s5H16prq2i/v2s/ov5yrmz4WuV/kyz2yf2t6pSRShmQ0+YUc+Ss7Lyuv9os88kWdE8ohK5RFZkYmWRneZyiL9tnt6ZjTkQ45HWqXLnldCGXEedQ2YzwnngiRd3b/+1/Iydn2a/rWRiEiewHXqOovve0rAFT1z1HnTPfO+UhEMoEVQDdtJijZQZRzfQnZJFjkCzW6PjryhZuVkUUoI7RFXXhWKKuuDjtU9zPZoezaL+esjCyyQllkh7LJy8wjLyuv9jNSBdM+u32987IysuiU24nczFz7a9ykvMb/E07ONoqewJKo7RJgj6bOUdUqEVkHFAKro08SkXOAc7zNzVzD175EnHq60uBZpZKw979yyls+uWUp/SzizJ5FHXsWdVo9nnlKNGar6n3AfQAiUtzarJhu7FnUsWdRx55FHXsWdUSkuOWzGudnn7OlQPSUTb28fY2e41U9dcQ1ahtjjEkSfiaKT4H+ItJPRLKBE4FpDc6ZBpzmrR8HvN1c+4QxxpjE863qyWtzmARMx3WPfUhVZ4vIdUCxqk4DHgQeF5EFwBpcMmnJfX7FnILsWdSxZ1HHnkUdexZ1Wv0sUu7NbGOMMYll78UbY4xpliUKY4wxzUraRCEih4rIPBFZICKXN3I8R0Se9Y5/IiJ9Ex9lYsTwLC4WkTkiMktE3hKRHYOIMxFaehZR5x0rIioiads1MpZnISITvP82ZovIU4mOMVFi+DfSR0TeEZHPvX8nhwcRp99E5CERWSkijb5rJs6d3nOaJSKjYrqwqibdgmv8/g7YCcgGvgSGNDjnfOBeb/1E4Nmg4w7wWRwAtPPWz2vLz8I7rwB4D/gYKAo67gD/u+gPfA509ra7Bx13gM/iPuA8b30IsDDouH16FvsBo4Cvmzh+OPAaIMCewCexXDdZSxS7AwtU9XtVrQSeAcY1OGcc8Ki3/jxwkKTn2AstPgtVfUdVIzPlfox7ZyUdxfLfBcD1uHHD0m/c+TqxPIuzgbtVdS2Aqq5McIyJEsuzUCAyO0NHYFkC40sYVX0P14O0KeOAx9T5GOgkIj1aum6yJorGhv/o2dQ5qloFRIb/SDexPItoZ+L+YkhHLT4LryjdW1VfSWRgAYjlv4sBwAAR+VBEPvZGc05HsTyLa4BTRKQEeBW4IDGhJZ2t/T4BUmQIDxMbETkFKAL2DzqWIIhIBvBX4PSAQ0kWmbjqpzG4UuZ7IjJMVf2fECP5TAQeUdXbvAFLHxeRXVSTdBjjJJOsJQob/qNOLM8CETkYuAoYq6r+TXEWrJaeRQGwCzBDRBbi6mCnpWmDdiz/XZQA01Q1rKo/AN/iEke6ieVZnAlMAVDVj4Bc3ICBbU1M3ycNJWuisOE/6rT4LERkV+CfuCSRrvXQ0MKzUNV1qtpVVfuqal9ce81YVW31YGhJLJZ/Iy/iShOISFdcVdT3iQwyQWJ5FouBgwBEZDAuUaxKaJTJYRrwK6/3057AOlVd3tIPJWXVk/o3/EfKifFZ3Aq0B57z2vMXq+rYwIL2SYzPok2I8VlMBw4RkTlANXCZqqZdqTvGZ3EJcL+IXIRr2D49Hf+wFJGncX8cdPXaY/4EZAGo6r249pnDgQXAJuDXMV03DZ+VMcaYOErWqidjjDFJwhKFMcaYZlmiMMYY0yxLFMYYY5plicIYY0yzLFGYNkNECkXkC29ZISJLvfWfvC6k8b7fGBF5eSt/ZkZjLwiKyOkiclf8ojMmdpYoTJuhqqWqOlJVRwL3An/z1kcCLQ7l4I0AYEybY4nCGCckIvd78za8ISJ5UPsX/u0iUgz8XkS6ici/RORTb9nHO2//qNLK5yJS4F23vYg8LyLfiMiTkRGOReQg77yvvDkEchoGJCK/FpFvReR/wD4Jeg7GbMEShTFOf9yQ3EOBn4Bjo45lq2qRqt4G3IEriezmnfOAd86lwG+9Esq+QLm3f1fgQtwcCDsB+4hILvAIcIKqDsONkHBedDDe0M/X4hLEz72fNyYQliiMcX5Q1S+89ZlA36hjz0atHwzcJSJf4MbN6SAi7YEPgb+KyO+ATt7Q9wD/U9USb5TSL7zrDvTu9613zqO4CWei7QHMUNVV3hwLz2JMQKzO1RgnesTdaiAvarssaj0D2FNVG06KdLOIvIIbR+dDEfllE9e1f3Mm5ViJwpit8wZRk96IyEjv82eq+pWq3oIbzXRQM9eYB/QVkZ297VOBdxuc8wmwv9dTKws4Pl6/gDFbyxKFMVvnd0CRNzH9HOA33v4LReRrEZkFhGlmlkGvNPJr3Gi/X+F6XN3b4JzluFnZPsJVa82N9y9iTKxs9FhjjDHNshKFMcaYZlmiMMYY0yxLFMYYY5plicIYY0yzLFEYY4xpliUKY4wxzbJEYYwxpln/H9BeqRxAj1lKAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.figure()\n", "lw = 2\n", "(thresholds, fpr) = get_fpr_curve(curve=curve)\n", "(thresholds, fnr) = get_fnr_curve(curve=curve)\n", "plt.plot(thresholds, fpr, color='blue', lw=lw, label='FPR')\n", "plt.plot(thresholds, fnr, color='green', lw=lw, label='FNR')\n", "\n", "plt.xlim([0.0, 1.0])\n", "plt.ylim([0.0, 1.05])\n", "plt.xlabel('Threshold')\n", "plt.ylabel('Error Rate')\n", "plt.title('FPR-FNR curves')\n", "plt.legend(loc=\"lower left\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.471901498177\n", "0.987647294429\n" ] } ], "source": [ "from catboost.utils import select_threshold\n", "\n", "print(select_threshold(model=model, data=eval_pool, FNR=0.01))\n", "print(select_threshold(model=model, data=eval_pool, FPR=0.01))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Snapshotting" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [], "source": [ "#!rm 'catboost_info/snapshot.bkp'\n", "\n" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Learning rate set to 0.5\n", "0:\tlearn: 0.2965875\ttest: 0.2955232\tbest: 0.2955232 (0)\ttotal: 59.9ms\tremaining: 1.14s\n", "1:\tlearn: 0.2160431\ttest: 0.2105519\tbest: 0.2105519 (1)\ttotal: 120ms\tremaining: 1.08s\n", "2:\tlearn: 0.1847090\ttest: 0.1727051\tbest: 0.1727051 (2)\ttotal: 195ms\tremaining: 1.11s\n", "3:\tlearn: 0.1761768\ttest: 0.1597829\tbest: 0.1597829 (3)\ttotal: 258ms\tremaining: 1.03s\n", "4:\tlearn: 0.1714326\ttest: 0.1548739\tbest: 0.1548739 (4)\ttotal: 325ms\tremaining: 976ms\n", "5:\tlearn: 0.1694150\ttest: 0.1517186\tbest: 0.1517186 (5)\ttotal: 388ms\tremaining: 905ms\n", "6:\tlearn: 0.1675766\ttest: 0.1498146\tbest: 0.1498146 (6)\ttotal: 459ms\tremaining: 852ms\n", "7:\tlearn: 0.1667392\ttest: 0.1488889\tbest: 0.1488889 (7)\ttotal: 518ms\tremaining: 777ms\n", "8:\tlearn: 0.1664645\ttest: 0.1489624\tbest: 0.1488889 (7)\ttotal: 570ms\tremaining: 697ms\n", "9:\tlearn: 0.1659541\ttest: 0.1484477\tbest: 0.1484477 (9)\ttotal: 636ms\tremaining: 636ms\n", "10:\tlearn: 0.1647458\ttest: 0.1472531\tbest: 0.1472531 (10)\ttotal: 698ms\tremaining: 571ms\n", "11:\tlearn: 0.1647261\ttest: 0.1471785\tbest: 0.1471785 (11)\ttotal: 748ms\tremaining: 498ms\n", "12:\tlearn: 0.1644109\ttest: 0.1466226\tbest: 0.1466226 (12)\ttotal: 806ms\tremaining: 434ms\n", "13:\tlearn: 0.1644007\ttest: 0.1466116\tbest: 0.1466116 (13)\ttotal: 853ms\tremaining: 366ms\n", "14:\tlearn: 0.1641944\ttest: 0.1464603\tbest: 0.1464603 (14)\ttotal: 907ms\tremaining: 302ms\n", "15:\tlearn: 0.1626904\ttest: 0.1452219\tbest: 0.1452219 (15)\ttotal: 967ms\tremaining: 242ms\n", "16:\tlearn: 0.1619736\ttest: 0.1454357\tbest: 0.1452219 (15)\ttotal: 1.03s\tremaining: 183ms\n", "17:\tlearn: 0.1619147\ttest: 0.1453835\tbest: 0.1452219 (15)\ttotal: 1.1s\tremaining: 123ms\n", "18:\tlearn: 0.1619084\ttest: 0.1452848\tbest: 0.1452219 (15)\ttotal: 1.15s\tremaining: 60.8ms\n", "19:\tlearn: 0.1613569\ttest: 0.1452973\tbest: 0.1452219 (15)\ttotal: 1.22s\tremaining: 0us\n", "\n", "bestTest = 0.1452219235\n", "bestIteration = 15\n", "\n", "Shrink model to first 16 iterations.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 57, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#!rm 'catboost_info/snapshot.bkp'\n", "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=20,\n", " save_snapshot=True,\n", " snapshot_file='snapshot.bkp',\n", " snapshot_interval=1,\n", " random_seed=43\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " eval_set=(X_validation, y_validation),\n", " cat_features=cat_features,\n", " verbose=True\n", ")" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Learning rate set to 0.218957\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c7042c21faab470497825be24e3581d8", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "20:\tlearn: 0.1612152\ttest: 0.1452297\tbest: 0.1452219 (15)\ttotal: 1.28s\tremaining: 11.5s\n", "21:\tlearn: 0.1612042\ttest: 0.1452362\tbest: 0.1452219 (15)\ttotal: 1.33s\tremaining: 10.6s\n", "22:\tlearn: 0.1610427\ttest: 0.1452320\tbest: 0.1452219 (15)\ttotal: 1.4s\tremaining: 11.1s\n", "23:\tlearn: 0.1602574\ttest: 0.1443945\tbest: 0.1443945 (23)\ttotal: 1.49s\tremaining: 11.9s\n", "24:\tlearn: 0.1600288\ttest: 0.1443164\tbest: 0.1443164 (24)\ttotal: 1.56s\tremaining: 12.1s\n", "25:\tlearn: 0.1589975\ttest: 0.1435700\tbest: 0.1435700 (25)\ttotal: 1.63s\tremaining: 12.1s\n", "26:\tlearn: 0.1589964\ttest: 0.1435730\tbest: 0.1435700 (25)\ttotal: 1.69s\tremaining: 11.6s\n", "27:\tlearn: 0.1588313\ttest: 0.1433549\tbest: 0.1433549 (27)\ttotal: 1.75s\tremaining: 11.5s\n", "28:\tlearn: 0.1588014\ttest: 0.1433848\tbest: 0.1433549 (27)\ttotal: 1.81s\tremaining: 11.3s\n", "29:\tlearn: 0.1584245\ttest: 0.1431081\tbest: 0.1431081 (29)\ttotal: 1.88s\tremaining: 11.2s\n", "30:\tlearn: 0.1583421\ttest: 0.1431004\tbest: 0.1431004 (30)\ttotal: 1.94s\tremaining: 11.1s\n", "31:\tlearn: 0.1583295\ttest: 0.1430967\tbest: 0.1430967 (31)\ttotal: 2s\tremaining: 10.9s\n", "32:\tlearn: 0.1583294\ttest: 0.1430988\tbest: 0.1430967 (31)\ttotal: 2.04s\tremaining: 10.6s\n", "33:\tlearn: 0.1582839\ttest: 0.1431464\tbest: 0.1430967 (31)\ttotal: 2.1s\tremaining: 10.5s\n", "34:\tlearn: 0.1582705\ttest: 0.1430595\tbest: 0.1430595 (34)\ttotal: 2.14s\tremaining: 10.2s\n", "35:\tlearn: 0.1582696\ttest: 0.1430571\tbest: 0.1430571 (35)\ttotal: 2.19s\tremaining: 9.97s\n", "36:\tlearn: 0.1577099\ttest: 0.1426513\tbest: 0.1426513 (36)\ttotal: 2.26s\tremaining: 10s\n", "37:\tlearn: 0.1577047\ttest: 0.1426538\tbest: 0.1426513 (36)\ttotal: 2.34s\tremaining: 10.1s\n", "38:\tlearn: 0.1576397\ttest: 0.1426637\tbest: 0.1426513 (36)\ttotal: 2.39s\tremaining: 9.97s\n", "39:\tlearn: 0.1576390\ttest: 0.1426667\tbest: 0.1426513 (36)\ttotal: 2.44s\tremaining: 9.81s\n", "40:\tlearn: 0.1576366\ttest: 0.1426616\tbest: 0.1426513 (36)\ttotal: 2.49s\tremaining: 9.63s\n", "41:\tlearn: 0.1575441\ttest: 0.1427510\tbest: 0.1426513 (36)\ttotal: 2.55s\tremaining: 9.59s\n", "42:\tlearn: 0.1573020\ttest: 0.1424677\tbest: 0.1424677 (42)\ttotal: 2.62s\tremaining: 9.61s\n", "43:\tlearn: 0.1572261\ttest: 0.1425183\tbest: 0.1424677 (42)\ttotal: 2.69s\tremaining: 9.59s\n", "44:\tlearn: 0.1571721\ttest: 0.1424962\tbest: 0.1424677 (42)\ttotal: 2.75s\tremaining: 9.48s\n", "45:\tlearn: 0.1567994\ttest: 0.1423557\tbest: 0.1423557 (45)\ttotal: 2.81s\tremaining: 9.47s\n", "46:\tlearn: 0.1565110\ttest: 0.1418114\tbest: 0.1418114 (46)\ttotal: 2.88s\tremaining: 9.41s\n", "47:\tlearn: 0.1557751\ttest: 0.1414036\tbest: 0.1414036 (47)\ttotal: 2.94s\tremaining: 9.35s\n", "48:\tlearn: 0.1557492\ttest: 0.1413917\tbest: 0.1413917 (48)\ttotal: 3s\tremaining: 9.27s\n", "49:\tlearn: 0.1556371\ttest: 0.1415897\tbest: 0.1413917 (48)\ttotal: 3.06s\tremaining: 9.21s\n", "50:\tlearn: 0.1555799\ttest: 0.1416534\tbest: 0.1413917 (48)\ttotal: 3.12s\tremaining: 9.17s\n", "51:\tlearn: 0.1553364\ttest: 0.1415682\tbest: 0.1413917 (48)\ttotal: 3.19s\tremaining: 9.14s\n", "52:\tlearn: 0.1549614\ttest: 0.1414633\tbest: 0.1413917 (48)\ttotal: 3.26s\tremaining: 9.12s\n", "53:\tlearn: 0.1549144\ttest: 0.1414808\tbest: 0.1413917 (48)\ttotal: 3.32s\tremaining: 9.02s\n", "54:\tlearn: 0.1542091\ttest: 0.1411187\tbest: 0.1411187 (54)\ttotal: 3.4s\tremaining: 9.06s\n", "55:\tlearn: 0.1540154\ttest: 0.1411603\tbest: 0.1411187 (54)\ttotal: 3.47s\tremaining: 9.03s\n", "56:\tlearn: 0.1539392\ttest: 0.1412245\tbest: 0.1411187 (54)\ttotal: 3.54s\tremaining: 8.99s\n", "57:\tlearn: 0.1538993\ttest: 0.1411606\tbest: 0.1411187 (54)\ttotal: 3.61s\tremaining: 8.93s\n", "58:\tlearn: 0.1538640\ttest: 0.1411852\tbest: 0.1411187 (54)\ttotal: 3.67s\tremaining: 8.86s\n", "59:\tlearn: 0.1537952\ttest: 0.1411267\tbest: 0.1411187 (54)\ttotal: 3.73s\tremaining: 8.8s\n", "60:\tlearn: 0.1537380\ttest: 0.1411907\tbest: 0.1411187 (54)\ttotal: 3.8s\tremaining: 8.76s\n", "61:\tlearn: 0.1536350\ttest: 0.1409801\tbest: 0.1409801 (61)\ttotal: 3.88s\tremaining: 8.74s\n", "62:\tlearn: 0.1535942\ttest: 0.1410172\tbest: 0.1409801 (61)\ttotal: 3.93s\tremaining: 8.66s\n", "63:\tlearn: 0.1535253\ttest: 0.1410033\tbest: 0.1409801 (61)\ttotal: 4s\tremaining: 8.62s\n", "64:\tlearn: 0.1534290\ttest: 0.1410210\tbest: 0.1409801 (61)\ttotal: 4.07s\tremaining: 8.55s\n", "65:\tlearn: 0.1534215\ttest: 0.1410008\tbest: 0.1409801 (61)\ttotal: 4.14s\tremaining: 8.53s\n", "66:\tlearn: 0.1532798\ttest: 0.1408475\tbest: 0.1408475 (66)\ttotal: 4.22s\tremaining: 8.49s\n", "67:\tlearn: 0.1530738\ttest: 0.1409062\tbest: 0.1408475 (66)\ttotal: 4.28s\tremaining: 8.44s\n", "68:\tlearn: 0.1529922\ttest: 0.1409187\tbest: 0.1408475 (66)\ttotal: 4.35s\tremaining: 8.38s\n", "69:\tlearn: 0.1529770\ttest: 0.1409706\tbest: 0.1408475 (66)\ttotal: 4.44s\tremaining: 8.38s\n", "70:\tlearn: 0.1529625\ttest: 0.1409251\tbest: 0.1408475 (66)\ttotal: 4.5s\tremaining: 8.3s\n", "71:\tlearn: 0.1529499\ttest: 0.1409476\tbest: 0.1408475 (66)\ttotal: 4.57s\tremaining: 8.24s\n", "72:\tlearn: 0.1528810\ttest: 0.1409514\tbest: 0.1408475 (66)\ttotal: 4.63s\tremaining: 8.18s\n", "73:\tlearn: 0.1528579\ttest: 0.1409500\tbest: 0.1408475 (66)\ttotal: 4.69s\tremaining: 8.1s\n", "74:\tlearn: 0.1528168\ttest: 0.1409697\tbest: 0.1408475 (66)\ttotal: 4.75s\tremaining: 8.04s\n", "75:\tlearn: 0.1526691\ttest: 0.1409859\tbest: 0.1408475 (66)\ttotal: 4.82s\tremaining: 7.98s\n", "76:\tlearn: 0.1525093\ttest: 0.1409349\tbest: 0.1408475 (66)\ttotal: 4.88s\tremaining: 7.91s\n", "77:\tlearn: 0.1524918\ttest: 0.1409958\tbest: 0.1408475 (66)\ttotal: 4.94s\tremaining: 7.84s\n", "78:\tlearn: 0.1523623\ttest: 0.1411616\tbest: 0.1408475 (66)\ttotal: 5.01s\tremaining: 7.78s\n", "79:\tlearn: 0.1520972\ttest: 0.1411054\tbest: 0.1408475 (66)\ttotal: 5.07s\tremaining: 7.71s\n", "80:\tlearn: 0.1520786\ttest: 0.1411760\tbest: 0.1408475 (66)\ttotal: 5.14s\tremaining: 7.65s\n", "81:\tlearn: 0.1520778\ttest: 0.1411706\tbest: 0.1408475 (66)\ttotal: 5.19s\tremaining: 7.56s\n", "82:\tlearn: 0.1520648\ttest: 0.1412223\tbest: 0.1408475 (66)\ttotal: 5.26s\tremaining: 7.52s\n", "83:\tlearn: 0.1515509\ttest: 0.1408977\tbest: 0.1408475 (66)\ttotal: 5.33s\tremaining: 7.46s\n", "84:\tlearn: 0.1515241\ttest: 0.1409170\tbest: 0.1408475 (66)\ttotal: 5.41s\tremaining: 7.41s\n", "85:\tlearn: 0.1507357\ttest: 0.1406431\tbest: 0.1406431 (85)\ttotal: 5.5s\tremaining: 7.4s\n", "86:\tlearn: 0.1505739\ttest: 0.1406102\tbest: 0.1406102 (86)\ttotal: 5.57s\tremaining: 7.34s\n", "87:\tlearn: 0.1505528\ttest: 0.1406036\tbest: 0.1406036 (87)\ttotal: 5.65s\tremaining: 7.3s\n", "88:\tlearn: 0.1502959\ttest: 0.1407175\tbest: 0.1406036 (87)\ttotal: 5.72s\tremaining: 7.25s\n", "89:\tlearn: 0.1501328\ttest: 0.1404297\tbest: 0.1404297 (89)\ttotal: 5.79s\tremaining: 7.19s\n", "90:\tlearn: 0.1499854\ttest: 0.1403098\tbest: 0.1403098 (90)\ttotal: 5.86s\tremaining: 7.13s\n", "91:\tlearn: 0.1499249\ttest: 0.1401775\tbest: 0.1401775 (91)\ttotal: 5.93s\tremaining: 7.07s\n", "92:\tlearn: 0.1497086\ttest: 0.1401750\tbest: 0.1401750 (92)\ttotal: 6s\tremaining: 7s\n", "93:\tlearn: 0.1496704\ttest: 0.1401246\tbest: 0.1401246 (93)\ttotal: 6.06s\tremaining: 6.95s\n", "94:\tlearn: 0.1496610\ttest: 0.1401833\tbest: 0.1401246 (93)\ttotal: 6.13s\tremaining: 6.89s\n", "95:\tlearn: 0.1495849\ttest: 0.1401673\tbest: 0.1401246 (93)\ttotal: 6.21s\tremaining: 6.83s\n", "96:\tlearn: 0.1494667\ttest: 0.1400086\tbest: 0.1400086 (96)\ttotal: 6.27s\tremaining: 6.77s\n", "97:\tlearn: 0.1493525\ttest: 0.1401855\tbest: 0.1400086 (96)\ttotal: 6.34s\tremaining: 6.71s\n", "98:\tlearn: 0.1489917\ttest: 0.1401021\tbest: 0.1400086 (96)\ttotal: 6.41s\tremaining: 6.64s\n", "99:\tlearn: 0.1489125\ttest: 0.1401268\tbest: 0.1400086 (96)\ttotal: 6.47s\tremaining: 6.57s\n", "100:\tlearn: 0.1489016\ttest: 0.1401160\tbest: 0.1400086 (96)\ttotal: 6.57s\tremaining: 6.54s\n", "101:\tlearn: 0.1487716\ttest: 0.1400791\tbest: 0.1400086 (96)\ttotal: 6.64s\tremaining: 6.48s\n", "102:\tlearn: 0.1487237\ttest: 0.1400954\tbest: 0.1400086 (96)\ttotal: 6.71s\tremaining: 6.42s\n", "103:\tlearn: 0.1486277\ttest: 0.1401909\tbest: 0.1400086 (96)\ttotal: 6.78s\tremaining: 6.36s\n", "104:\tlearn: 0.1486153\ttest: 0.1401898\tbest: 0.1400086 (96)\ttotal: 6.85s\tremaining: 6.29s\n", "105:\tlearn: 0.1485328\ttest: 0.1401209\tbest: 0.1400086 (96)\ttotal: 6.92s\tremaining: 6.23s\n", "106:\tlearn: 0.1484761\ttest: 0.1401947\tbest: 0.1400086 (96)\ttotal: 6.98s\tremaining: 6.16s\n", "107:\tlearn: 0.1484161\ttest: 0.1402889\tbest: 0.1400086 (96)\ttotal: 7.05s\tremaining: 6.1s\n", "108:\tlearn: 0.1483455\ttest: 0.1404464\tbest: 0.1400086 (96)\ttotal: 7.12s\tremaining: 6.03s\n", "109:\tlearn: 0.1482798\ttest: 0.1403834\tbest: 0.1400086 (96)\ttotal: 7.18s\tremaining: 5.96s\n", "110:\tlearn: 0.1482755\ttest: 0.1403591\tbest: 0.1400086 (96)\ttotal: 7.24s\tremaining: 5.89s\n", "111:\tlearn: 0.1481230\ttest: 0.1405596\tbest: 0.1400086 (96)\ttotal: 7.32s\tremaining: 5.84s\n", "112:\tlearn: 0.1477729\ttest: 0.1405197\tbest: 0.1400086 (96)\ttotal: 7.4s\tremaining: 5.79s\n", "113:\tlearn: 0.1477185\ttest: 0.1405934\tbest: 0.1400086 (96)\ttotal: 7.47s\tremaining: 5.72s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "114:\tlearn: 0.1474968\ttest: 0.1405011\tbest: 0.1400086 (96)\ttotal: 7.54s\tremaining: 5.66s\n", "115:\tlearn: 0.1474954\ttest: 0.1405148\tbest: 0.1400086 (96)\ttotal: 7.63s\tremaining: 5.61s\n", "116:\tlearn: 0.1474155\ttest: 0.1405253\tbest: 0.1400086 (96)\ttotal: 7.69s\tremaining: 5.54s\n", "117:\tlearn: 0.1473274\ttest: 0.1405328\tbest: 0.1400086 (96)\ttotal: 7.77s\tremaining: 5.48s\n", "118:\tlearn: 0.1472799\ttest: 0.1405519\tbest: 0.1400086 (96)\ttotal: 7.83s\tremaining: 5.42s\n", "119:\tlearn: 0.1471431\ttest: 0.1402404\tbest: 0.1400086 (96)\ttotal: 7.91s\tremaining: 5.35s\n", "120:\tlearn: 0.1470829\ttest: 0.1403126\tbest: 0.1400086 (96)\ttotal: 7.98s\tremaining: 5.29s\n", "121:\tlearn: 0.1470043\ttest: 0.1402714\tbest: 0.1400086 (96)\ttotal: 8.04s\tremaining: 5.22s\n", "122:\tlearn: 0.1469235\ttest: 0.1403153\tbest: 0.1400086 (96)\ttotal: 8.12s\tremaining: 5.16s\n", "123:\tlearn: 0.1469211\ttest: 0.1403098\tbest: 0.1400086 (96)\ttotal: 8.18s\tremaining: 5.09s\n", "124:\tlearn: 0.1467363\ttest: 0.1401837\tbest: 0.1400086 (96)\ttotal: 8.25s\tremaining: 5.02s\n", "125:\tlearn: 0.1467291\ttest: 0.1401725\tbest: 0.1400086 (96)\ttotal: 8.31s\tremaining: 4.96s\n", "126:\tlearn: 0.1466647\ttest: 0.1402149\tbest: 0.1400086 (96)\ttotal: 8.38s\tremaining: 4.89s\n", "127:\tlearn: 0.1465627\ttest: 0.1400712\tbest: 0.1400086 (96)\ttotal: 8.45s\tremaining: 4.82s\n", "128:\tlearn: 0.1465278\ttest: 0.1401424\tbest: 0.1400086 (96)\ttotal: 8.52s\tremaining: 4.75s\n", "129:\tlearn: 0.1465071\ttest: 0.1400993\tbest: 0.1400086 (96)\ttotal: 8.58s\tremaining: 4.69s\n", "130:\tlearn: 0.1464897\ttest: 0.1400964\tbest: 0.1400086 (96)\ttotal: 8.66s\tremaining: 4.63s\n", "131:\tlearn: 0.1464020\ttest: 0.1401162\tbest: 0.1400086 (96)\ttotal: 8.73s\tremaining: 4.56s\n", "132:\tlearn: 0.1462656\ttest: 0.1400077\tbest: 0.1400077 (132)\ttotal: 8.8s\tremaining: 4.49s\n", "133:\tlearn: 0.1460190\ttest: 0.1398864\tbest: 0.1398864 (133)\ttotal: 8.86s\tremaining: 4.42s\n", "134:\tlearn: 0.1459637\ttest: 0.1398490\tbest: 0.1398490 (134)\ttotal: 8.92s\tremaining: 4.35s\n", "135:\tlearn: 0.1457154\ttest: 0.1395519\tbest: 0.1395519 (135)\ttotal: 9s\tremaining: 4.29s\n", "136:\tlearn: 0.1457117\ttest: 0.1395426\tbest: 0.1395426 (136)\ttotal: 9.06s\tremaining: 4.23s\n", "137:\tlearn: 0.1456945\ttest: 0.1395723\tbest: 0.1395426 (136)\ttotal: 9.14s\tremaining: 4.16s\n", "138:\tlearn: 0.1456748\ttest: 0.1395792\tbest: 0.1395426 (136)\ttotal: 9.2s\tremaining: 4.09s\n", "139:\tlearn: 0.1455239\ttest: 0.1398163\tbest: 0.1395426 (136)\ttotal: 9.28s\tremaining: 4.03s\n", "140:\tlearn: 0.1455034\ttest: 0.1398620\tbest: 0.1395426 (136)\ttotal: 9.37s\tremaining: 3.97s\n", "141:\tlearn: 0.1454291\ttest: 0.1400755\tbest: 0.1395426 (136)\ttotal: 9.43s\tremaining: 3.9s\n", "142:\tlearn: 0.1454196\ttest: 0.1401048\tbest: 0.1395426 (136)\ttotal: 9.49s\tremaining: 3.84s\n", "143:\tlearn: 0.1453446\ttest: 0.1400991\tbest: 0.1395426 (136)\ttotal: 9.56s\tremaining: 3.77s\n", "144:\tlearn: 0.1453092\ttest: 0.1400823\tbest: 0.1395426 (136)\ttotal: 9.63s\tremaining: 3.7s\n", "145:\tlearn: 0.1452807\ttest: 0.1401418\tbest: 0.1395426 (136)\ttotal: 9.71s\tremaining: 3.64s\n", "146:\tlearn: 0.1452572\ttest: 0.1401681\tbest: 0.1395426 (136)\ttotal: 9.77s\tremaining: 3.57s\n", "147:\tlearn: 0.1452437\ttest: 0.1401694\tbest: 0.1395426 (136)\ttotal: 9.82s\tremaining: 3.5s\n", "148:\tlearn: 0.1452409\ttest: 0.1401696\tbest: 0.1395426 (136)\ttotal: 9.88s\tremaining: 3.43s\n", "149:\tlearn: 0.1452313\ttest: 0.1401945\tbest: 0.1395426 (136)\ttotal: 9.95s\tremaining: 3.36s\n", "150:\tlearn: 0.1452248\ttest: 0.1402280\tbest: 0.1395426 (136)\ttotal: 10s\tremaining: 3.29s\n", "151:\tlearn: 0.1451256\ttest: 0.1401826\tbest: 0.1395426 (136)\ttotal: 10.1s\tremaining: 3.22s\n", "152:\tlearn: 0.1451233\ttest: 0.1401831\tbest: 0.1395426 (136)\ttotal: 10.1s\tremaining: 3.15s\n", "153:\tlearn: 0.1451067\ttest: 0.1402200\tbest: 0.1395426 (136)\ttotal: 10.2s\tremaining: 3.08s\n", "154:\tlearn: 0.1450766\ttest: 0.1402811\tbest: 0.1395426 (136)\ttotal: 10.3s\tremaining: 3.02s\n", "155:\tlearn: 0.1450670\ttest: 0.1402883\tbest: 0.1395426 (136)\ttotal: 10.3s\tremaining: 2.95s\n", "156:\tlearn: 0.1450589\ttest: 0.1402922\tbest: 0.1395426 (136)\ttotal: 10.4s\tremaining: 2.88s\n", "157:\tlearn: 0.1450369\ttest: 0.1404510\tbest: 0.1395426 (136)\ttotal: 10.5s\tremaining: 2.81s\n", "158:\tlearn: 0.1450249\ttest: 0.1404327\tbest: 0.1395426 (136)\ttotal: 10.5s\tremaining: 2.75s\n", "159:\tlearn: 0.1450211\ttest: 0.1404312\tbest: 0.1395426 (136)\ttotal: 10.6s\tremaining: 2.68s\n", "160:\tlearn: 0.1449969\ttest: 0.1404053\tbest: 0.1395426 (136)\ttotal: 10.7s\tremaining: 2.61s\n", "161:\tlearn: 0.1449487\ttest: 0.1403818\tbest: 0.1395426 (136)\ttotal: 10.8s\tremaining: 2.55s\n", "162:\tlearn: 0.1449365\ttest: 0.1403904\tbest: 0.1395426 (136)\ttotal: 10.8s\tremaining: 2.49s\n", "163:\tlearn: 0.1447393\ttest: 0.1403556\tbest: 0.1395426 (136)\ttotal: 10.9s\tremaining: 2.42s\n", "164:\tlearn: 0.1447223\ttest: 0.1403308\tbest: 0.1395426 (136)\ttotal: 11s\tremaining: 2.35s\n", "165:\tlearn: 0.1446902\ttest: 0.1402935\tbest: 0.1395426 (136)\ttotal: 11s\tremaining: 2.29s\n", "166:\tlearn: 0.1446412\ttest: 0.1402642\tbest: 0.1395426 (136)\ttotal: 11.1s\tremaining: 2.22s\n", "167:\tlearn: 0.1445740\ttest: 0.1403223\tbest: 0.1395426 (136)\ttotal: 11.2s\tremaining: 2.15s\n", "168:\tlearn: 0.1445716\ttest: 0.1403393\tbest: 0.1395426 (136)\ttotal: 11.2s\tremaining: 2.08s\n", "169:\tlearn: 0.1445616\ttest: 0.1403364\tbest: 0.1395426 (136)\ttotal: 11.3s\tremaining: 2.02s\n", "170:\tlearn: 0.1445163\ttest: 0.1403374\tbest: 0.1395426 (136)\ttotal: 11.4s\tremaining: 1.95s\n", "171:\tlearn: 0.1444992\ttest: 0.1403069\tbest: 0.1395426 (136)\ttotal: 11.4s\tremaining: 1.88s\n", "172:\tlearn: 0.1444212\ttest: 0.1404872\tbest: 0.1395426 (136)\ttotal: 11.5s\tremaining: 1.81s\n", "173:\tlearn: 0.1443616\ttest: 0.1404046\tbest: 0.1395426 (136)\ttotal: 11.6s\tremaining: 1.75s\n", "174:\tlearn: 0.1441623\ttest: 0.1404320\tbest: 0.1395426 (136)\ttotal: 11.6s\tremaining: 1.68s\n", "175:\tlearn: 0.1441596\ttest: 0.1404399\tbest: 0.1395426 (136)\ttotal: 11.7s\tremaining: 1.61s\n", "176:\tlearn: 0.1441157\ttest: 0.1403950\tbest: 0.1395426 (136)\ttotal: 11.7s\tremaining: 1.54s\n", "177:\tlearn: 0.1438309\ttest: 0.1402648\tbest: 0.1395426 (136)\ttotal: 11.8s\tremaining: 1.48s\n", "178:\tlearn: 0.1437287\ttest: 0.1402467\tbest: 0.1395426 (136)\ttotal: 11.9s\tremaining: 1.41s\n", "179:\tlearn: 0.1437173\ttest: 0.1402203\tbest: 0.1395426 (136)\ttotal: 12s\tremaining: 1.34s\n", "180:\tlearn: 0.1436975\ttest: 0.1402458\tbest: 0.1395426 (136)\ttotal: 12s\tremaining: 1.28s\n", "181:\tlearn: 0.1436947\ttest: 0.1402477\tbest: 0.1395426 (136)\ttotal: 12.1s\tremaining: 1.21s\n", "182:\tlearn: 0.1435053\ttest: 0.1401582\tbest: 0.1395426 (136)\ttotal: 12.2s\tremaining: 1.14s\n", "183:\tlearn: 0.1433447\ttest: 0.1402034\tbest: 0.1395426 (136)\ttotal: 12.2s\tremaining: 1.07s\n", "184:\tlearn: 0.1433105\ttest: 0.1402677\tbest: 0.1395426 (136)\ttotal: 12.3s\tremaining: 1.01s\n", "185:\tlearn: 0.1432248\ttest: 0.1403297\tbest: 0.1395426 (136)\ttotal: 12.4s\tremaining: 943ms\n", "186:\tlearn: 0.1431444\ttest: 0.1402979\tbest: 0.1395426 (136)\ttotal: 12.5s\tremaining: 875ms\n", "187:\tlearn: 0.1431310\ttest: 0.1404082\tbest: 0.1395426 (136)\ttotal: 12.5s\tremaining: 808ms\n", "188:\tlearn: 0.1431025\ttest: 0.1405215\tbest: 0.1395426 (136)\ttotal: 12.6s\tremaining: 740ms\n", "189:\tlearn: 0.1430438\ttest: 0.1404832\tbest: 0.1395426 (136)\ttotal: 12.7s\tremaining: 673ms\n", "190:\tlearn: 0.1429858\ttest: 0.1404577\tbest: 0.1395426 (136)\ttotal: 12.7s\tremaining: 605ms\n", "191:\tlearn: 0.1429675\ttest: 0.1404993\tbest: 0.1395426 (136)\ttotal: 12.8s\tremaining: 537ms\n", "192:\tlearn: 0.1427232\ttest: 0.1404004\tbest: 0.1395426 (136)\ttotal: 12.9s\tremaining: 471ms\n", "193:\tlearn: 0.1427034\ttest: 0.1404061\tbest: 0.1395426 (136)\ttotal: 12.9s\tremaining: 404ms\n", "194:\tlearn: 0.1426123\ttest: 0.1404027\tbest: 0.1395426 (136)\ttotal: 13s\tremaining: 337ms\n", "195:\tlearn: 0.1425407\ttest: 0.1403096\tbest: 0.1395426 (136)\ttotal: 13.1s\tremaining: 269ms\n", "196:\tlearn: 0.1425383\ttest: 0.1403008\tbest: 0.1395426 (136)\ttotal: 13.1s\tremaining: 202ms\n", "197:\tlearn: 0.1425326\ttest: 0.1402927\tbest: 0.1395426 (136)\ttotal: 13.2s\tremaining: 135ms\n", "198:\tlearn: 0.1423886\ttest: 0.1403825\tbest: 0.1395426 (136)\ttotal: 13.3s\tremaining: 67.3ms\n", "199:\tlearn: 0.1421418\ttest: 0.1405326\tbest: 0.1395426 (136)\ttotal: 13.3s\tremaining: 0us\n", "\n", "bestTest = 0.1395425582\n", "bestIteration = 136\n", "\n", "Shrink model to first 137 iterations.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 58, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = CatBoostClassifier(\n", " iterations=200,\n", " save_snapshot=True,\n", " snapshot_file='snapshot.bkp',\n", " snapshot_interval=1,\n", " random_seed=43\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " eval_set=(X_validation, y_validation),\n", " cat_features=cat_features,\n", " verbose=True,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model predictions" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [], "source": [ "? model.predict_proba" ] }, { "cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[0.0381 0.9619]\n", " [0.0072 0.9928]\n", " [0.015 0.985 ]\n", " ...\n", " [0.0099 0.9901]\n", " [0.0159 0.9841]\n", " [0.0243 0.9757]]\n" ] } ], "source": [ "print(model.predict_proba(X=X_validation))" ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1. 1. 1. ... 1. 1. 1.]\n" ] } ], "source": [ "print(model.predict(data=X_validation))" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[3.2289 4.9216 4.1868 ... 4.6093 4.1257 3.693 ]\n" ] } ], "source": [ "raw_pred = model.predict(data=X_validation, prediction_type='RawFormulaVal')\n", "print(raw_pred)" ] }, { "cell_type": "code", "execution_count": 63, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0.9619 0.9928 0.985 ... 0.9901 0.9841 0.9757]\n" ] } ], "source": [ "import math\n", "def sigmoid(x):\n", " return 1 / (1 + math.exp(-x))\n", "probabilities = [sigmoid(x) for x in raw_pred]\n", "print(np.array(probabilities))" ] }, { "cell_type": "code", "execution_count": 64, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[0.0381 0.9619]\n", " [0.0072 0.9928]\n", " [0.015 0.985 ]\n", " ...\n", " [0.0099 0.9901]\n", " [0.0159 0.9841]\n", " [0.0243 0.9757]]\n" ] } ], "source": [ "X_prepared = X_validation.values.astype(str).astype(object)\n", "# For FeaturesData class categorial features must have type str\n", "\n", "fast_predictions = model.predict_proba(X=FeaturesData(cat_feature_data=X_prepared, \n", " cat_feature_names=list(X_validation)))\n", "\n", "print(fast_predictions)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Staged prediction" ] }, { "cell_type": "code", "execution_count": 65, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Iteration 0, predictions:\n", "[[0.3147 0.6853]\n", " [0.3119 0.6881]\n", " [0.3119 0.6881]\n", " ...\n", " [0.2477 0.7523]\n", " [0.2477 0.7523]\n", " [0.3119 0.6881]]\n", "Iteration 1, predictions:\n", "[[0.2677 0.7323]\n", " [0.2415 0.7585]\n", " [0.3053 0.6947]\n", " ...\n", " [0.2419 0.7581]\n", " [0.1878 0.8122]\n", " [0.3053 0.6947]]\n", "Iteration 2, predictions:\n", "[[0.2348 0.7652]\n", " [0.2108 0.7892]\n", " [0.2694 0.7306]\n", " ...\n", " [0.2112 0.7888]\n", " [0.1883 0.8117]\n", " [0.2694 0.7306]]\n" ] } ], "source": [ "predictions_gen = model.staged_predict_proba(data=X_validation, ntree_start=2, ntree_end=8, eval_period=2)\n", "for iteration, predictions in enumerate(predictions_gen):\n", " print('Iteration ' + str(iteration) + ', predictions:')\n", " print(predictions)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Solving MultiClassification problem" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "267f049977264b0581dbacd8f56aa86b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 66, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoostClassifier\n", "model = CatBoostClassifier(\n", " iterations=150,\n", " random_seed=43,\n", " loss_function='MultiClass'\n", " #loss_function='MultiClassOneVsAll'\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " eval_set=(X_validation, y_validation),\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Metric evaluation on a new dataset" ] }, { "cell_type": "code", "execution_count": 67, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0:\tlearn: 0.6569860\ttotal: 41.3ms\tremaining: 8.21s\n", "50:\tlearn: 0.1950260\ttotal: 2.58s\tremaining: 7.52s\n", "100:\tlearn: 0.1700584\ttotal: 5.3s\tremaining: 5.19s\n", "150:\tlearn: 0.1641016\ttotal: 8.09s\tremaining: 2.62s\n", "199:\tlearn: 0.1604074\ttotal: 10.8s\tremaining: 0us\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 67, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=200,\n", " learning_rate=0.03,\n", ")\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " verbose=50\n", ")" ] }, { "cell_type": "code", "execution_count": 68, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "metrics = model.eval_metrics(data=pool1, \n", " metrics=['Logloss','AUC'],\n", " ntree_start=0,\n", " ntree_end=0, \n", " eval_period=1,\n", " plot=True)" ] }, { "cell_type": "code", "execution_count": 69, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "AUC values:\n", "[0.4998 0.538 0.5504 0.5888 0.6503 0.6487 0.6487 0.6601 0.6601 0.6612\n", " 0.6614 0.67 0.6699 0.6697 0.6697 0.6698 0.6698 0.6698 0.6698 0.7265\n", " 0.7338 0.7376 0.7478 0.7478 0.7506 0.7591 0.7642 0.7877 0.8148 0.8265\n", " 0.8353 0.8459 0.8533 0.8564 0.8573 0.8748 0.8874 0.893 0.8948 0.898\n", " 0.9003 0.9052 0.9122 0.9159 0.92 0.9204 0.9209 0.9246 0.9262 0.9266\n", " 0.9279 0.9303 0.9311 0.9312 0.9327 0.9329 0.9337 0.9341 0.9341 0.9349\n", " 0.9354 0.9365 0.9391 0.9411 0.9424 0.9435 0.9446 0.9458 0.9463 0.9471\n", " 0.9478 0.9481 0.9482 0.9483 0.9486 0.9497 0.951 0.9514 0.9515 0.9519\n", " 0.9522 0.9526 0.953 0.9537 0.9543 0.9544 0.9545 0.9548 0.9548 0.9548\n", " 0.9548 0.9548 0.955 0.9551 0.9553 0.9554 0.9557 0.9557 0.9557 0.9557\n", " 0.956 0.9566 0.9566 0.9569 0.9568 0.957 0.9569 0.9572 0.9573 0.9575\n", " 0.9576 0.9576 0.9577 0.958 0.9587 0.9594 0.9595 0.9594 0.9601 0.9609\n", " 0.9609 0.9608 0.9608 0.9612 0.9616 0.9618 0.9622 0.9623 0.9623 0.9625\n", " 0.9625 0.9628 0.9629 0.9629 0.9629 0.963 0.9631 0.9632 0.9635 0.9636\n", " 0.9637 0.9638 0.9638 0.964 0.9641 0.9643 0.9645 0.9645 0.9645 0.9647\n", " 0.9647 0.9647 0.9647 0.9648 0.9648 0.9648 0.965 0.9652 0.9652 0.9653\n", " 0.9653 0.9654 0.9655 0.9655 0.9655 0.9656 0.9656 0.9657 0.9657 0.9657\n", " 0.9658 0.9658 0.9659 0.9659 0.966 0.966 0.966 0.966 0.966 0.966\n", " 0.966 0.966 0.9661 0.9662 0.9663 0.9663 0.9663 0.9663 0.9663 0.9663\n", " 0.9665 0.9664 0.9666 0.9666 0.9666 0.9666 0.9666 0.9667 0.9667 0.9666]\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f6e28c97b9564447b6f30be735949b0b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print('AUC values:')\n", "print(np.array(metrics['AUC']))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Saving the model" ] }, { "cell_type": "code", "execution_count": 70, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 70, "metadata": {}, "output_type": "execute_result" } ], "source": [ "my_best_model = CatBoostClassifier(iterations=10)\n", "my_best_model.fit(\n", " X_train, y_train,\n", " eval_set=(X_validation, y_validation),\n", " cat_features=cat_features,\n", " verbose=False\n", ")" ] }, { "cell_type": "code", "execution_count": 71, "metadata": {}, "outputs": [], "source": [ "my_best_model.save_model('catboost_model.bin')" ] }, { "cell_type": "code", "execution_count": 72, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'loss_function': u'Logloss', 'verbose': 0, 'iterations': 10, 'logging_level': u'Silent'}\n", "0\n" ] } ], "source": [ "my_best_model.load_model('catboost_model.bin')\n", "print(my_best_model.get_params())\n", "print(my_best_model.random_seed_)" ] }, { "cell_type": "code", "execution_count": 73, "metadata": {}, "outputs": [], "source": [ "my_best_model.save_model('catboost_model.json', format='json')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Feature importances" ] }, { "cell_type": "code", "execution_count": 74, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0:\tlearn: 0.6569860\ttotal: 47.6ms\tremaining: 9.47s\n", "50:\tlearn: 0.1950260\ttotal: 2.81s\tremaining: 8.22s\n", "100:\tlearn: 0.1700584\ttotal: 5.64s\tremaining: 5.53s\n", "150:\tlearn: 0.1641016\ttotal: 8.57s\tremaining: 2.78s\n", "199:\tlearn: 0.1604074\ttotal: 11.5s\tremaining: 0us\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 74, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=200,\n", " learning_rate=0.03)\n", "\n", "model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " verbose=50\n", ")" ] }, { "cell_type": "code", "execution_count": 75, "metadata": {}, "outputs": [], "source": [ "fstrs = model.get_feature_importance(prettified=True)" ] }, { "cell_type": "code", "execution_count": 76, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'MGR_ID': 18.398056223683028,\n", " 'RESOURCE': 21.27625707750949,\n", " 'ROLE_CODE': 11.943230282191125,\n", " 'ROLE_DEPTNAME': 15.252806962601875,\n", " 'ROLE_FAMILY': 2.4789173515872176,\n", " 'ROLE_FAMILY_DESC': 9.984073192415531,\n", " 'ROLE_ROLLUP_1': 2.6278918788673646,\n", " 'ROLE_ROLLUP_2': 13.582536028250486,\n", " 'ROLE_TITLE': 4.456231002893895}" ] }, "execution_count": 76, "metadata": {}, "output_type": "execute_result" } ], "source": [ "{feature_name : value for feature_name, value in fstrs}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Shap values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "https://github.com/slundberg/shap" ] }, { "cell_type": "code", "execution_count": 77, "metadata": {}, "outputs": [], "source": [ "def object_predictions(model, obj):\n", " print('Probability of class 1 = {:.4f}'.format(model.predict_proba([obj])[0][1]))\n", " print('Formula raw prediction = {:.4f}'.format(model.predict([obj], prediction_type='RawFormulaVal')[0]))\n", " " ] }, { "cell_type": "code", "execution_count": 78, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "The model has complex ctrs, so the SHAP values will be calculated approximately.\n" ] } ], "source": [ "import shap\n", "explainer = shap.TreeExplainer(model)\n", "shap_values = explainer.shap_values(Pool(X, y, cat_features=cat_features))\n" ] }, { "cell_type": "code", "execution_count": 79, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3.3457409968646434" ] }, "execution_count": 79, "metadata": {}, "output_type": "execute_result" } ], "source": [ "explainer.expected_value" ] }, { "cell_type": "code", "execution_count": 80, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Probability of class 1 = 0.9798\n", "Formula raw prediction = 3.8820\n" ] } ], "source": [ "object_predictions(model, X.iloc[3,:])" ] }, { "cell_type": "code", "execution_count": 81, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "
\n", " Visualization omitted, Javascript library not loaded!
\n", " Have you run `initjs()` in this notebook? If this notebook was from another\n", " user you must also trust this notebook (File -> Trust notebook). If you are viewing\n", " this notebook on github the Javascript has been stripped for security. If you are using\n", " JupyterLab this error is because a JupyterLab extension has not yet been written.\n", "
\n", " " ], "text/plain": [ "" ] }, "execution_count": 81, "metadata": {}, "output_type": "execute_result" } ], "source": [ "shap.initjs()\n", "shap.force_plot(explainer.expected_value, shap_values[3,:], X.iloc[3,:])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above explanation shows features each contributing to push the model output from the base value (the average model output over the training dataset we passed) to the model output. Features pushing the prediction higher are shown in red, those pushing the prediction lower are in blue" ] }, { "cell_type": "code", "execution_count": 82, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Probability of class 1 = 0.6404\n", "Formula raw prediction = 0.5772\n" ] } ], "source": [ "object_predictions(model, X.iloc[91,:])" ] }, { "cell_type": "code", "execution_count": 83, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "
\n", " Visualization omitted, Javascript library not loaded!
\n", " Have you run `initjs()` in this notebook? If this notebook was from another\n", " user you must also trust this notebook (File -> Trust notebook). If you are viewing\n", " this notebook on github the Javascript has been stripped for security. If you are using\n", " JupyterLab this error is because a JupyterLab extension has not yet been written.\n", "
\n", " " ], "text/plain": [ "" ] }, "execution_count": 83, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import shap\n", "shap.initjs()\n", "shap.force_plot(explainer.expected_value, shap_values[91,:], X.iloc[91,:])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get an overview of which features are most important for a model we can plot the SHAP values of every feature for every sample. The plot below sorts features by the sum of SHAP value magnitudes over all samples, and uses SHAP values to show the distribution of the impacts each feature has on the model output. The color represents the feature value (red high, blue low)." ] }, { "cell_type": "code", "execution_count": 84, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlEAAAFZCAYAAAC4x3ouAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3XeYHFeZ6OFfhY7TaXLUaEY52JIll2ycbXAEbGMWcCIYDGa5y8ICu8BdvCbD3YUlLbBLMnFtWK/BYHAOsiwHySUr5zQz0uTQ09O5u8L9o1ujUXCSPZY0fO/ztNRdfarOqepQ35zz1WnFdV2EEEIIIcQrox7vBgghhBBCnIwkiBJCCCGEOAYSRAkhhBBCHAMJooQQQgghjoEEUUIIIYQQx0CCKCGEEEKIYyBBlBBCCCHEMZAgSgghhBDiGEgQJYQQQghxDCSIEkIIIYQ4BhJECSGEEEIcAwmihBBCCCGOgQRRQgghhBDHQIIoIYQQQohjIEGUEEIIIcQxkCBKCCGEEOIYSBAlhBBCCHEMJIgSQgghhDgGEkQJIYQQQhwDCaKEEEIIIY6BBFFCCCGEEMdAgighhBBCiGMgQZQQQgghTgiKonQoinLKYctMRVEuVBTlS4qiXPsytvEFRVG+OXmtPEh/PSoRQgghhHg1XNe97Xi34XDSEyWEEEKIE56iKL9QFOWj5ftRRVHuVhRlm6IojyqK8qvDep+aFUW5r/z8XxRFCU5Gm6QnSgghhBAnkv9VFCU34fGco5S5DYi7rjtPUZQqYA1w94TnDWAZkAAeBG4EfvJaN1SCKHG8uce7AUKIqevee+8F4MorrzzOLTlpKJOy1ch1R37Xj/32hep6h+u6m8YbpCjmUcpcBPw9gOu6I4qi3HPY8w+6rjtaXn8VMPNYmv1SJIgSQgghxKRyjxKbTU60Nm5iT5YNBCajEsmJEkIIIcSkcpUjb6/ScuC9AIqixICrX/UWj4EEUUIIIYSYVC7KEbdX6UtAnaIo24A/ACal/KfXlQznCSGEEGJSvdyeJ9d1246yzCjfXT5hcRq43nXdnKIoEWAl8ONy+S8ctv4hj19LEkQJIYQQYpK95hlQlcD9iqJogB+4w3XdR17rSl6KBFFCCCGEOKm4rjsAnH682yFBlBBCCCEm1WuQSH5CkiBKCCGEEJPqNUgkPyFJECWEEEKISSU9UUIIIYQQx0B6ooQQQgghjoGrSBAlhBBCCPGKTdUfSZUgSgghhBCTaqr2RMnPvgghhBBCHAPpiRJCCCHK7NE0H/rY40xLjuF1bHKKyrx/uIgbL2w63k07qU3VnigJooQQQoiyr7z39zR7PJhts1Bcl4TXR//t63j41xtRFIef//SK493Ek9JUzYmS4TwhhBCibCQYYltDM9FclrTXR6IizKoZc+iPVLKnqpG+3YPHu4knJVdRjrhNBRJECSGEEGXPzpiD4roUNI3hijBQCgBGgiH8VpHP/fNK7KHEcW7lycc9ym0qkCBKCCGEKNMch4KqobguuAdP9Zrr8P7nnmTZSD9v+LdOyBWOYyvFiUJyooQQQoiyi3Zt4YZ1q1jf2MKjsxeys7YRzXF4y9a1nNG1BxWIB4Lc+UcP1187/3g396QxVYbvDidBlBBCiL9a3//8ozy132ZvbSMoLv9nsI+AVcRr2zSNjXLd+mdpHxlGdxwALEWlfXiQnZ1jx7nlJxf52RchhBDiJOVYNp/69ON49w6SjIZwbYu5fX3ogDV7IRd07GAgFOXfLnorneue5b9PPxtXVfnLgtN4x7pn0VyYNjrCkzPmECwW0AqHBgWJZIGv/2Yna7cmKXg8xD0eLmjQ+O4/LDw+O3yCkR8gFkIIIU5GjsvHbvoLG5umc5nTw9yuTszGVnY1TmNvtIr3rFuF7jrjxf938Rm4aillOO3zs7alnQv27mAkFCaWzXDvwqUsiTh07Evyk0/fzx5/hGqryFPtc6n0+kgF/KDpPBF3ufqjT/Dbb51LwKsdr70/IUhPlBBCCHGCcByXtet6KFg2d68Y5KHRALXpMTyFIlHLwlFUWoYH6Pf5cb0+tre38faNz3HH0nNwFYWmxAg16SRLe7sOCaDqkwny2oRTo+vSNjLEcy3tjPn9nNLXzandnfTHK/jbb42QaZ1LTzSGpWoojkvGo3Pt+tXUppJsqW9iZfsczv/8Vj5/bRNvPa3q9T9QJ4qpGUNJEDWZDMNYDlwAXGua5v9MWH4m8CzQaZpmW3nZLOBfgIuBGDACbAJ+Yprm78tlvgDcCuQoXSE6CPwK+KJpmi95xahhGC5wnmmaKw3DuBB4HEgDDlAEtgF3Az8wTTP/6vZeTCUbuvP8fFWCoFdh+c4sw2mbcC5LXSrJmZ27COWyZDwebN1LXziKAyzt6cRjO2yub2J3TR3xQAWzB/uYNTzI7upanm2dweU7NtM+MkhnrBoX0FyXDQ3NzBvsI1TMMxKoIJjP8aeFS3EVhVg2zSl93QyHwnhsm8XdnQxXhFnX1Ep1OonuuuR1nd5wjOkjQ8wf7GHuQC/LZ8wl5QuQ9XjZXVNPRSHP3MEeGsdGUR2XMX+Q7mgV0VyGWDbN5oZm6pJjtMeHWLZvD2ZLO8tnzuOcjp0s7u0i6Q3w2yVnkta9pP0BZgz38871z5H0+/npGRcwEgxxak8XGb+fuD9IqJAnksuSR0XRFM7p2ElNJoXHsrj7lNPxOg7tI6X5h3ojMWYP9bOhoYW814vRtYfGZILfnnYmCW+ArM+Hqyhojk0ol8XSdbIeD7ai0jYyyFVb1uG3iqxunYFm2bxl2wbum7+49PqoGimfn5FAkJzHy+yhPqKZDE/PmIOjKASLRU7ft5emxAjnduwk4/UxUBFhV/n1C+dzTI8Pcc8pp5Px6MQrwtiKim7bLOzr5g1duzi3Yye2qrI/WklO97C+oYUH558GgOs6WKqKg4Kqqiiuy9zefbQkE2S9PnZV16M4DqgKGa+PykyKW55dzvT4EMtnzmNPdT1VmRQ9kRibGlt426bnmR4fxl/fRHHRMjoCFURzWcIjg3htGzweWqwiWyprcFSVtc1t4wnOPdEqGscStI6O0BWt4vFZ8/EXi4z5/Niaiuo6+CwLW9XY2DSN3TX1AOyobWRufzcewF8ssq2h5eAHRYVA3iKWSXPH0rPJeH34rSKpQJDbftfDZadW4tGmaDTxEqZqT5RMcTD5tgIfOmzZh8rLATAM41RgDaVg5kIgAswGvgtcc9i6y03TDJXLvA/4dPn/Y2GbphkyTTMCNANfAD4ALDcMw3uM2xRTTCrv8ME7+7l/S4a716UZTjsUVY3BcJTNjS38bskbCBULtKRTTE+McOa+PZzduYtYLkfW62Vj0zQyPj8+x6YpmWBGfIhLdm3lLds2sGx/BzWZNKf3dDESrOD++afSHh+ifXSY2nSK2UMD5Lx+bli/CkfTeOemNeyubWB3TQPb6ptZOXMe8wf78NtFVEXBZ1vsi1WT8fnZ2tjCXYvP5M6lZ3PL6idxUFjX0kYyEKQvWsna5na2NrSQ8frZ1tBMMhCgq7KaR+cspC9ayYbmVq7ZaDIUDPGHRQZto8Oc3bWbimKRhvQYl+7YRKyQw1EVdtY1saOukeaxBG/ftIa6dJJksILd1fWMBYL0RivZXt9E0LWYO9jHYDjK1vpmXFVl4WAf8wZ7mTUyyKyRQZbt28vapunsrG+ks6qWu087k7b4EH+zwSQTCOBoGq6qYmkajq6T83ixNR1UleFQhGmJOHXpFG/duoH+aIz/OPcSdtQ10h+JEa8IUdR1glaRtNdHQdN5aMFiipqOUw6wnmudgc9xcBSVvdV1pP1+GlNjjAWCPD+tnSdmzueNOzczFIriqBqKomDpOjmPh0t2bSFoFfEXi+R1DwXNwwMLluCoaumm6eR1bymAAlAUtjdOozGZYG1LG2PBIKMVIUaDIQq6h/5wjL/MP436dJKPPfUIV25Zy+zhAc7u3MUZnXuYV04CP727kzn9vagopPxBNjW0kNU9DERidFZWkfL7aB4dpjZ9MBlcs22U8mxFvzn9HLbXNbG+eTp9kRgA1ek0DWMJ/MUiHVW14+sVdJ0ddU3sj1XTEas55LPiAkl/gAfnnkrK68NSFNJeHwA+x+GfHxid1M/qicxVjrxNBRJETb7fA0sMw5gBYBhGGPgb4OcTynwHeM40zfebprnTNE3bNM2caZoPmKb5nqNt1DRN1zTNJ4HNgPFqG1mu72FKQdsSjj0we0WSyaTcP8Hv7xsYYyx3cLgDDu2ZT/v8HN5Xr7sOCjDmC8CES5vH/IHx+1WZ9CHrBC0L3XEJFovjy1Rc4sEgbSND+KwikVyWkWDF+PPDwRBZjwdXUVEdh4KmU9A9B9vpuuysbaA5Mcz+yqpD2lLUNLIeLz3R2IT6QD0wN5CisK+ymozXh61qRHPZQ9pblUmjAJpTKh8PVJSXZ5geH2IgFDniWGU8vvFcG4DBijD1qQQVhYMdv37bIhEI4ioHy22rayKSzx5xmbgC2OrBZUl/gMKEoaimsQRZj3f8WByguS4q4HEOfV0B7HLAc+Dkf0CkvP8jFSGi+dwR7fBbxfF9tRUFW9XIeL1HXtquHNoWgIzXd/RL4BWFpM+PqyhYqkYsl4FyuyuKh3aWj/kDB+tXNeKBCjyWxfl7tpPwVxCwinRW1tA6MkRNOsn8gR5iuSxVmRQpn398Ox7bRnVddNfFUVWqMilimfT4nFGK44y/R/K6TkU+B66L6jj4i0VcSq+X6tjYmoajqriKQtzvZ1N/aW6pE+Fz/UL3J4uLcsRtKpAgavLlgP8Gbi4/vh54AugFMAwjQGnI785XslHDMFTDMC4CTgG2v1aNNU1zJ6VesTe9Vtt8MeFwWO6f4PfntkS5YNbB4AcA1x0/EZ7RuZvhYJBCOTjIeLx0RStxgOZEnEg2M75OUyIOwJjPz/KZ88h4SgFP3B8k4fMTyud4vrmVglpKwu2MVXNadycr2+eQ83jZXVXL/P6e8Wac0refUC6H6jikvT48loVvQhDmKgrn7N3B/fMWMz0+VBoqKrclls3QMjrM0v17J+xW6WQIoNs200eGmDEyyOyBXtY0t5EoB4GK67KybQ55TcNSVSryOZZ2d+ACj82az5b6Zi7ZsQkAp3yyUB2HUD5LXj2YYHxq7z6Wz5hHT6RyfAbn/dFKFvV24i+WTri+YpE37trC5oZmdNs6ePwdF0tR8Fr2+PYaxkbxlsvYisKY18fins7yY3U8EDgQmCR8fmpSYxQ1rTSc6tic0bWbhsQodekxdLu07aKq0lvuoVnWtZsVM+aiuO7BWacdB8VlPJfIY1tUpVNE8jnC5eDLpdTVrtk2tqKU2uK6aI5DU2K0FKhAqXeo3E6vVeTGdc+g2zYuLvsilQAk/AGWz5pPZ6waB9ha20h/JMpEVZkki3u7KOgeVNch6/WVgmQFlnXtYdZgH6f2dOGxHS4uv1aq4xDOZzl93x6q00kKuofhQBBbLQ1ZFhUVXzlYdAFHVdEdl2guRzhfGrK1UdCAzITADEUBTefj55Q+VyfC5/qF7k+WqdoTpbjuS6bSiGNUzol6BLgHeAhopZQL9XmgEvgKcA6wH7jCNM0HyuudBiwvb8YPzDVNs7OcE/U5SnlMAcAL/CfwcdM0D545Xrg9h+dEPWKa5hF5cYZh/A6oMk3zkmPa8VdG3oAnAdtxWbs/T1VQZedAgXu3pKlMJEjt6Mc3nCBm5clX+OlJQsrrpaCozB4ZIJzP0xcK0x2toj8cIZLPM7+/m6zupaOqhlg2xazhQfpCUTy2RdbnY29lLTXJUSyPl5rUGJaqsbFxGl6rSGU2TVU6jaOp5HQP4VwOVIWCpjFzsJ+M18fG+mbiFRWgqMyID1GfiKMqCn0VITJeL/2RSkK5LC1jCULpFMPRCM2JOHsra1nQvQ+v67By5hwChTyW5mNp9x7iPj8Fjw+PXaRtZIjVLW3Eg2HSPh+OouKzCswdGqAvGKI/VoWjuEwfGaYuk2J3pJq834evUCBQLDAcCrOwbz/NyTH2V1ejxQL0FnU8hSL5cBDbcakq5CCTI+vx0zQ2QsrnZ2dNPSndg6rpFFWNgF1kpr9IV14ng4qlarTGB2mNDxMsWmyrrkN3XS7etZmcx8vmmnpCVpH+cIz90UoiuRxt8UE8VpFIPovXsgnlc6yZNoP+YAXNyTFmD/Wjug4dVbXsj1URzmUI5vMMhiJkvX5q0wnyqof+cISW0RHmjPQTKBbpDMeIhyOcv3c7veEY989bhG47TIsPEw8EAIfRYATVddEci7aRQaYn4oxUhOmIVpLyBVAdl5bRYc7p3EUsn+XRmfPIen00jo2S93jxFvP0RmIoLqQCpSB+NBDC1TRi2TRXbNuAoyhsbGghqyi4Xh8FvfR11xwf5sZ1z3Lenu3sqG0gHqjgybY5PNU+m+31TVyyfQOxbJahijA+y2J16wxGghXM7+/BUlR6YpWkvT5022bBQA/b65pQXJfm0RG2NDSXeidcF92xSyG06xJVLFbdOvv4fYhfvkkJb/qaP3zEd31D949O+lBKEstfB6ZpbjIMo5NS4ngd8AClHimAOGADLRPKrwNihmG0APs49E39hGmaF5dzlj4FvIdSQPWSQdQrcKBeIQDQVAWjtfSX9YwaL5ctCAH1wJzj2q7Xx1mTsM2lk7BNgHlHWXbBJNX1clwEwNde1TbeCMBNL7N0IZnjnn/+C88tms5ZZ7Uyt7iZrT/qZiRSS3U+g4OCA0wbHUYF5g328XPjHO5ebJSGYF2X1dNmkfN6UVyXs/fuoC6ZYMwfYLgixFu3rOU3p55B8/AAraPDfHj1CrojMdY1t7G8bQ7T4sN0V1aDomApKnqxgNe1WfXFua/qKJzsZMZy8Wr9GPgZ8CXTNG3DKKUxmaaZMQxjBXAd8NOXuzHTNAvA1w3DuBz4IvCJ16KR5asETwd++VpsTwghXk/esJ93/cff8K7y43vv3cypf9fM+h8OMBCtJJbNMBiK8Im3Xse5HbuwVZV1zdPJebxUFIvgOOS8pTwyV1F4csZcxgJBoJQD9eDcRfgUuPXdbVxknAm8mQ/f8gC7KqpQFQXHdWgeHSbr8RLKZfjWh2ewZHbl8TkYJ5CpOuQgQdTr505KvTtrjvLcJ4EnDcO4ndIfbXsBjdJQ30u5FXjEMIzvmKbZeayNMwzDB5wLfBtYjwRRQogp5Nb7bz5imW2dxbs+tpIK22buQC8DoQgFIBUMjXf/W9rBHDZbURkJVhDJZbnn+SQXGaVpD37048uP2HbRsvDocoqd6iSx/HVSvvrtEdM040d5bh2lK+w8wAogCewG3k/parkXDI7KV+g9Sak36pXSDMNIGYYxRinR/SvAb4ALZJ4oIcRUp+kav/n2uRQtm7jXT+vwAKlABXlNx1JULEUhN+FqR49jk/b4SHu9EE+/yJaRAOoIylFuJz9JLBfHm7wBhRCT5t577wXgyiuvfFnl531pL86E3qcb1qwsTdfg8fLA/MVkPF50q8j9769lVktoUtp8nE1KdNM97SNHfNc37/vPkz6SklBZCCGEKJs70MP2+mYcVeX0fXuZMzyIrUBFoYCjqvx53iJi+cxUDaAmzVSZF+pwEkRNIYZhbAamH+WpTtM05afEhRDiJdSkk7Ru38i8oT4qsxlsRcFbLM0N1ZAc5Z1OD1/7ystJVxUTTZV5oQ4nQdQUIoGSEEK8OruqaqjNZfHbFiPBLN2RGKfv6yj9tl+kih/fdvbxbuJJSXqihBBCiClufmaEPjXA3upaFBd6QxFGfEE0q8gP//08lCk639Fkk54oIYQQYor7r/+8lJVmL5+7sxu1aHFxm8YXPnLh8W7WSU96ooQQQoi/AucajTxhNB7vZkwpMmO5EEIIIcQxmKpz2UgQJYQQQojJJT1RQgghxInp21ffzmOt88j6/NSkxljQ3cWlV7TBtOPdMgHSEyWEEEKckO5/YA+PTJ9Hb6yaQL7A7uo6WkeGeOT+Dk79QAxVn/xekJ2PbeOeH6ymLxjm3EqHRdcsov2Cmaiq/LoaTN2cKHl1hRBCnNR6/+0vKKicv3sbD9z+TVb+8GvUp5MYPR3kNo1Nev2/fPvP+Pg9cTbGGlEKNjv3JLnzu6v42SX/hWPbk16/OH4kiBJCCHFSCxXytMaH+OSKB/BbFiouf/fMo7SMDpO3J/80d0frKSzt2cfCoT7qMmnymoavWGB7XQPbv/HgpNd/MnCPcpsKJIgSQghxUlvTPB1bUeiOVLK2sZUxj5+s7qGoangqJ/80N310hHAxD4CrqswdHuD6daupTCbZ0JWZ9PpPBq6iHHGbCiQnSgghxEktHggxZ6if56bPAqCjqpbT9+3hzwuXsmT/GO6MivGyvUMZtpk9bHq6k+GN/QQsi+5oJf3hCBXJJEm/j8pigcpshk9++1IG7nia5Q93EQ9WoDgOmlUko2oUQ0HivgCRbI4K3XNIeyxVZWt9E3+3ejlfO/fvufZ1PRonJplsUwghhDgBjQQq8Dg2lE/UfZEYWd1LVvfSsybBhXc9xONPWazclmK4IoLquviLBYKKQtHjoS6TIlTIM+oPUJNOclbHLp5qn83X/vFxPrL6Ca7PpMjpHv7fhVdQ1D0odpFpiQRZ3c8j8xfzht3bwXVBUVAdh3P27uDhOafyhq7dBHsHju/BOUHIz74IIYQQr4MVzw2w+5kuzvnDvXSMwJ5Fc7jp9uvwe7Sjlq9PJUjrXkJWEYBpI0M819rOSCjMxuZWVs6cxyfvvo+RM87Db1uM+oPkdQ/B1Bjz+7s5raeLlMfLHxaejopLR1UN1z//NA/OXQyuy7fOu4w91bVEchni/iBF3cP+qnoW9XRxw5qnuXDPdpbPms9gMMSqaTP448Il3PbQPaQ9HuZa6fF2DnTF+cY/PsLC/m76KiLsr6oh5dH51g8uoyp49H2bKqZqT5TiulMlvUucpOQNKIQY99mf7ODP3SpffeBujH17yHh97ItW8XRzG22jQ+yeM4uPfvUiqpui4+vcds0dbGqcxi2rn2B6fJg5Q30oQMrr45NX3sBAKMypvV3sqmkgWCzwxMx51KSSXL/2WT64+onx5OB9kRid1XUAhLMZ/mfxGfREK0n6A+N1hXJZsl7/+OPzd25CB7pi1fzplKXYaikYahse4G+ffhT3vRdywy2nA/Clt/2aK7duJJbLsLJtFgPhKLaqsr62iSA2N107k3MunT2px/dlmJRoZ+vsTx7xXT9/57dO+shKeqLKDMNYDpwFFAEb2At81TTNu47y/ERnmaa50TCMWuD/AZcDUSAJrAM+YJpmb3kbfuBzwPVAMzAG3A/8i2ma+8pl2sp1TzNNc/+E9h2y3DCMLwC3AjlKgcgg8Cvgi6ZpuhPWawRuA94M1ADDwLPAv5qmucYwjJuA24HDsx9/YJrmZ17BIRTiuLMsG11T2bc/gZMt4rigaAp19WHyI0mU0Qx2NEioIYrHo6Gope9wxyl9ZDRVwQWU1zDp1bYdLNvBo6lYRQvdq4/PHeS6Lq4LTiZHwXFRXQXF78Ep2ni9GlnLpdA7Qi6eZc+K3TRfNIfqiA/dpzM2lCSfKuIdGcXKWYSjfqzaKIVwkMrGKJquo2SybFnVxcbfr2e4N0t9apSCojCtMUK314O1e4S6VJLdNfX0h8KMBSpwHRs9l2FGMkF1BTBvOjMqINIcYWNKYQgvrU6Gxa1BKs+bTyDsw3ZdPH4vqqbiui75gsXg3hG8jsWI4sOrFJk2sxbHAcYybFi+h0zRwqqOMntWJU3VfgYHM/TFLdZtGuHvtq7nqi1rGQ5UoAELBnsp6jrPNbdR193HZz73DPsjMQL5HPWpJDWqyg1rn+aSnZtRXbf0l5miECrkueWZx/jhuRdjaToK0B2tpGk0zlu2racmkzrk6qqAVWSwIkRtOsXumnoyXh/OYe+FnO4h5fURLOTx2haO7iGYyzJ7eIAZw4PsrG0ASnlZPzvzAhY/1sFd69JkdB+/XrWCWC4LwCU7t/Dxq28k5fNjKSrb6hpp+NFK1v/oKW74+fVEwl4yBYfl28f4+/sO/XqeVwW/e08dQa+Grp0c14dN1Z4oCaIO9WXTNL9iGIYOfBS4wzCMtaZp7pr4/Aus+xsgASwxTXPAMIw6SgGVC2AYhgb8BWgAbgTWUJpL99+BVYZhLDNNs/sVtne5aZoXG4ahAOcCDwIdwC/KdTYBqykFc28GtgEB4Brg7eU2AOwxTXPWK6z7uBv49Uq0z/4Sx3F5eMFiGm86l46KGA890UMgl+M8ElzzL28k1hLDTeVIve97PNKvMhiOULd0Gn8aDZBUNBZPC3DrZwy27h7jO7/cTi7vcPM72jnXl6Hnlp/wh+oZjPoCdEcrUVpr+dd/WsR/39vFSnOQ9pYQn7llHuEKz0s3+DhJDqf52Uf/hJ3MU1RVglaRgqYRWtzKhz6xjLGb/5PCug72hCsZ8/o5f+8OsrqX5JJZtN7zKZzdfSRu/C56zzC4cMfSs3i+pY2iz8u7b17Em4xqdq3Yw4ofPIWmq1wQzlN337MoTVU48RSKpqJ+52a+2RUgsWI7LcODFFWN55unk/H6OH9JFW+/sIEf/+d6ehMWI/4Auq7yqffM4qzFVXz2O5vZsWeMafEhCsEAff4Q1ekkC/r2MxIMsbWuEUfVwHWpKORLf0e7kPH6cBUFzXF45/pVnNW1h5TXx69OP4eBcARLURkLBAFQXBcFF9zSVUQV+SyhQoG2+CD1qTGu3rQWS4HH5pzCvmgV1ZkUo/4gOa8Pb6EA5WCsbXiA7fVN6I5D+/AA3aEoT8+ci6VqRHIZ/JaFqyjYisKFu7bSUV3HqC/AWCBAUdOpTY2huQ5dsRocVSGne2hOxHnz1vX8Ytl5pPf1Up9M0Dw2iuLCKX37uHjnFkK2xVOts3iutR1XUTirYxcNY3G6qmr5xbLzcHytjMyvYMwfZFFvF9dsep63b9/AQ3NO5U+LDQZCUYKFPO0jg0TyOYqaRjocxZdO8e6f/pahUITfLT6T6fFBZuRzPDBvMaueGeStt93Dz95wAcFikas3reGMrr3srarh85e+jcbkGEPBELWZFAouO2rqURyXs7ox3wVBAAAgAElEQVR24SgKe6vqSAQtNHuAWDqF7th8aPUKHujuIK+VenMs7eAQ15zBPj5/6TU88JNvkNc99EZiZIIhen1+eqnlb1c9gTZhZMUt/7O4p4svPvgHIvkcGY+X2y67hmg2g8dxGAhF2FtZQ3t8iLym8UzrTHZW1ZH0+bFRiORyZD1e8roHS9NwgD019biKQjiXpW14cDwo0lyXqzavJenz8ftTTideESIeDLG5cRqn9nQxEgyV3p9ligury4nwiuty/bpn+PhTD+OoKoW5K7mvdSb/eOX15PUjv1u2jcDi7x6aa+XX4cEP1tMaOzFP65IT9VfENE3LMIyfAN8GTgN2vcQqAGcD15qmOVDexgClnqEDrgfOAxZMCMr2GoZxLbAZ+CLwwWNsrws8aRjGZsCgHEQBXwLSwDWmaR7oQUsBvz6Wek4krusS+KfbSycw4OKtG/jOnxvZXlf65fWC18/aRIaG21dzxW2XYv/sYTbtSNDb3AYu9KzZj7exBScUYe3+PKue6+fOh3sYipe294M7dnHmjkd53F9P0utHc12mjY6wIVDB9369k7VbRgHYumeMex7p5j1Xtx2Pw/CyPPC9p3GTOVTXJVwoogC64zC8uZfOL/+J0OpdKMDM4UFCuSx+y8JvWWQ3dcLPH6Xw0Hp8+wdRgO21DTzf0gaAJ1/g9t9s441Lz+axf38CK28B8GR3lr9J53B29oBS6tnJ/P3P2HrhNZw5NIjm2Gyqbx4fInl8/SjxzjhDQ1kGIzFsVaXgwjfv2MPHLYctnWnQNHojMXJeHwAD4SjVmRS6ZZUCKEXBa5fq99k2cX8Qt9zbY2salqbjsy18WYsrtm/g10vPZiwcLh2gckKwizI+kJH2BfA7DolABe9a/xyq67Kjvpk95aGevkisXF4h7/WiOzYKsKe2AY/jALC7pp6NDS1Ymg6ui9+yQFFQKJ1wV86cCyjkdB1XKbV1f7SKrM938MVzXbqjlXzvvEvGh4k8joNSbuvCvm585f3e0DStVBfw7PRZ/O0zj3LbFe/EKR8H3XUZCoV5bNYCmhNxdlXXcd/8xYTzOSL5HABdlTXMHB7A65QmiBypCLEvVsOKmXNRXJfTuju59Yp3kfeUTux7quuoT40xbXSYC/bsAGDBQC/XbniOH595IXMG+9Dd0vGYP9BLdTqJCvSGoiSCFeOvTyoQoH14kGXdHeOvYVbT0RwHuxxIPTFzHtWZFB7HwVEU8l4f/mJh/FBF8tlD3/iuWzq+Hu/4/gWLBeb099AbrSwVURQenrWAWD5LTtexNB1HU9le14Tm2JzZsYueaBVjfj+Xb9vAJ6+6cfzS/KQ/QMIfwFLU8X20NBVXVXn32qf54TmX0D4yyJyBPrbWNXBex05yuge9kMdWVb7+preUej3L7bh+3Sp01wXbJqN7OLtzF1dtep67TjuTlyNnwWfvi3PHDbUvq/zrbar2RJ0c/YCvM8MwvMBHyg93vMzVVgDfMAzjFsMwlpR7niZ6M7BqQgAFQDm4+R1wxator2oYxkXAKcD2w+q8a0IAdcJJJpPHfH9iOpVylNw+F4ViwSo/cI/4ECsTVikNq0xcALZlHSU7wKVwYJtlTnm9V7Mvk3n/BdMeXbCswgs8Wea4pePwQtyjHLujbsd50a/QA6u7hy2b+Fg5bAtueelELzQMl/V4x+9r5aDp5ThwkpsMr+SkMvHoTVyvoB/8O1gvB28AmnPkLNnjx1JRyHk87KmuLa83oayiUNAO/ds6VMiND48VNX08gILSpfxw6GfpQGUORzrQ8sPbp7iH7kt3JMZtl76NO097A2sbp/HDN1zE3aeczjvXryav63RW1gAcMtR296nG+D7uqq6lqGpkdZ2np88cL7Ojpo4/LjK4d+FSNjS20B+KkPV4S0Nqmo4LpLx+uiMx+kNRdtU0cPXm5/mnJx6gPxylKpMa35bXKpLzelkxYw5d0UrSHi+Z8vssmstx87PL+cldt1OdTVHQdW5Y+yy2pvH3b3s3yz7+Bf5w6rLSfEmA4jrUpA9+dg/0wL3S9155VPpVfq9ODlc58jYVSBB1qM8ZhjEKZIGvAB80TXPD4c9PvE147lpKQ3rvB54Ghg3D+E45DwqgFnih4boeoO4Y2nvBhPY+Bvwc+K8Jz79YnRO1H75fhmHccAztecXCB3oDXuF9RVHIfPV9ZL1e0h4vj8w9lYsua+fy8xrQFAgXcizy5jj3lrMA0D5wMae0V1CfSqDiUr+oiWzQj+K6LGrycuayej70rpnEIh4Cfo0PXzcT/7/exAXpPioKeRxFYX+0kmh1kE+8fz4XnlGHrivMbgvxtjc1v6p9mez7l330bKgoDW2lvD5spTRMVDWvnpmffzv60nZcTWVPVQ0r22aT1zQS/gD5ea3w/jcS+Np7yDdUYSsKswb7Oa27E4CC18P7bpyDqqlc9Inz8QY9BGJ+zpkVBK+O2lYHkQBUhQh892ZmLW1kX2U1lqoyZ7CPinwO1XE479QoH3jvPGKVPmqyKXTbxofDP1zXzvlLa5jTEkB1HOqSCRoyY6UTTmqMhrEEquOg2Da4LgVVw3UcLFUlYBXHIzPFcajMpLAVhTGfnwfnnIKlqoRzmQnRm3vIraKQQ7NtYtk0T8yYgwvMGehl+kipJ60umUAv1+stFtAcB9VxmDXQS1FRcYH24QGWde3CU+4pyng85WAebEXB6NqDr1jEVyzitYqojkNbfJCZQ30ojoO/UCCaSWPs28tHnnkM3bbRHBsXF1wHXJd1jdPoCUcZ9QeYNdhPZSZFNJvhol1b2Bur5v2rnkC3LVTHoT9Uej/MGuzjLVvW0T48iKMoZHXPePCh2zbBQo6k10de04hkUgwHQyzo66aoqqxrms75u7eOl7123SoyHg+dlTWsaJ9DTtfZUtfEr08/mwX9PcQDFdjlQGF3dR3bahtxgZp0krbhAarTSQL5HO3D/QxWhPnDgiWM+gNsaGjB1nVyPh8bm6cTsoq8cfdWFNdldcsMXMBXLFDQdAqqRlFReGZaO1+4+Cq+ee5lfPidN3PLOz/A+6/7EI/NXkigUKAzVsX9cxfRH4piaRpPzJzP/y5axo66Bla1zqA7EmPU72d9cysD4Sj7KqvpqKoZ/zzFgyEW9e5j5mA/7cODnNWxi2g2w2AowrPTZ+Gxirjl4/lcSxsffeoRFNclr+pE8rnxHqy0z0/KdzAxvSKf5fTOPXzp4qvZF61kfySGZlmsaJ/DPacs5eXyavD1Kypf9ffGZCkNmB96mwrk6ryycuL4I+WcqErgZ0DGNM13H/78y9iWl1I+1K+B75qmeZthGHdQSgo/7yjlvwzcbJpmk2EYzcB+YIZpmnsnlJlNqVeswTTN/nJi+bnlnCgv8CngPcAbTNMcK6/TA9xumuatL9LWm4Bbj2NOlLwBX2eu676midMnCsdxyOZtdE1hpG+M4b40+XiWbMBLW1DBKVpkQkEc2yGAQ9viFoqWhWsDlg0eHcV2UDwqiqKgayqO644ngTu2M544feD4OY5zSJI4joOrqOOnCMsB1XVI9ifwhwJYKngCPjSllMReyFsUXQUlk8XOWWiRIP3dCTRNwR/2EVBc/EOjDD2yia1dSabNa6DizFmouSJFp8jQfZsIVFfgqY7ihLx4/Tr+yhBq2EefFiDkFCjuH+F/7tpNrGMfXtshVREgVhOk3/Gwz/aR8/gIZTLUZVOM+v3kNY3T9nUSzqXY1TyN0WiMSEOIdDRCrZInqtg0eGwiHo0Gr0vNW08l5/VTFfaiODY5VyfoV+h4rot4xmFrXmVJDBoWNhPxq2zfNsSq+3dQH1ShrYbpLTEi9RXs3DKMoyus+v4zLO3awzPtswkXC2jlwLMzWsk71z7LE+3zCBXzaK7L7ppadNshVMgTzaQ5o7cTn2XxucvfMT6c25CI86vf/oSU18u/n385z7TPGU/+rksmyOo6lu5BdWyu2vw8z7TPHX9PtQ8NcOsjf2RXTT1Pt8/m96cY1KSTLNvfAUBRURn1ll7PbXVNdEcrx+u9aOdm3rhrC7WpMbyWxVPtc7hx7bNsqWviH6+6npzuoTY1xgfMlTw0awGKbdEbiqApCr/7x9kUNJ1Kv8rqriR/+8dDhyqX1sO/vrWaqF+jukJDUZTX8jM9KV8O6+d9+ojv+sXb/u2k/yKSnKijME0zbhjGB4HdhmFcbZrmH1/h+gXgT4ZhPEIppwrgAeCnhmHMNE1z94Gy5ST2d1G6Sg+gF8gDsyhdjXfALEr5TUfM3Fau7+uGYVxOKbfqE+Wn7gPeYRjGF0/kIT3x+pqKARSAqqpUBEonsMbWKhpbq15yHa/HAx4o/3PkNiccK7V8FdTE43cggBpfrmnlM1DpX10FUIm2VAMwIdupVH/AixcgeLD+9oj/0EINMRpPaaPxKO2bdtbcoywtGc+MWdDIrZcufMFyr7UDA6ezzp4BwLLDnl90WiOLTjtyb9pbIgDU11TQ+aGf0T7YT1HXWDttBolyT9u26gbOqrK4+N/egVZ3cIqDPy/7Kl3RKhYM9ALwiScfZPmMeZze3cFTbbNZ2TabgVCEpV17CRTztIwOk/N4mT3Qi9naTk06jQrMG+ilo6puPGdqxlAf3zv/MlTbZswfJOkPkPQHyHq81CcTAPisApW5HEVdHw+gAOqTCTbUN3P5+TN405eu5ELL5jM3+thR38zS/R24lII417FoHhlif1MD72h0+NQnlh7yHrtsfhV757+KF0RMKgmiXoBpmiOGYXwL+JphGPe+VPly2TuBjUABOB+4CPh6ucgdwM3APYZh3MzBq/O+QWlKhC+U63UMw/g18CXDMPYAe4AZlIKjX06cvuAobgUeMQzjO6ZpdgKfB1YB/2sYxmcp9WT5gauAhS/WQyWEEMfDkkW1zF/+KYb/tJZtP3iY8I7N5ENBbvr9+9Er/EddZ0NdEz7HxnVdIrksl+zYxLm7t+GoKvfOP417FyxGc1wKmsrC/l7C5SHpYD7PSEWYVPkih7bREb5/z695tnUmTWOj+AsF3n3Dh1nS3YmlKOMXIfRFYui2xf955jEu276Rb17wZhoScYaCIVzg4l1b+OSTD7EvVsXjoTO4GNB0jU985Ty+ftszjAWDRPNZqtNJVk2fxY9/8Eb83qk+2ebUJEHUi/supV6d95Yf/0s5GJnoOtM0/0wpv+znQCul90s38E1KUxgcuOLvckqBzm+BJkrzRD0AnHFgnqiyf6A0t9NDlHKlBigln3/5xRprmuaThmE8SSngusk0zW7DMJZRCqYeAqoozRP1DKU5rQ6YYRhG6rDN3Wua5vUvVp8QQkwGf8BD87Vn0HztGS+r/GC0kjdvfp5wPle6wg0IFYusbp7OWbu3oQPqOXPYuDuFr1gklstgKyqdsSrqU4nxIGpbTQPndOzksh2bcIF75y6iKp3i0TmnEMtmqE0m8Nk2LhDJZZk2OoLuOPiKBdY3Ty9dfem6vHPDanTXoSUxQjiXG29n8+x6vn/n217jo3VymCo/OHw4yYkSx5u8AYUQr8qH3/sXPv34fUxPxg8udKEnEsP81mV4/CpXXnklAPu29dO5tZ8dv30e3/5hdlfXYXs85DweTu3upDk+TEsywZrm6bzprg9z5/9sx9w4SjwU4qr1zzFnuJ/lbbOxdI1wLsdAMMz2llY2NLeNV/3xJx/kfWue4sHZC6mr9nLxH/7udT4ir8qkRDtr53/miO/6JVv/9aSPrKQnSgghxEmtMZWgoGmsaZrO6T2d43+aaY5NbsjC03Jwiotp8+qZNq+ec69Z9KLbPPDjKx/7xMSsrtLVvhccVvZt/2cFquPgqCqq47C7sobvnHMxrutQd9VJN4/xpJgqV+MdToIoIYQQJzUXhf2V1ayZ1s6W+iamx4cZqQgRyqbxWEUOprtPjoLuZUl3B9trGwnnc+yrqmWjfzpvf/4ZlrzxuP8W3glhqswLdTgJooQQQpzUQo5F2uejoOsMhyIMhyL4igX6mqZzViE/6TkDS3s7WNE+F0tVsTSNUX+AZR07sSsCVE+rnOTaTw7SEyWEEEKcgM69ejbVX3qOtU3TSfgDBAoFjJ59WC0a7pzJ/xmUr9x1HX/63808+7vnCLg5KmY2sPja+bzp7Qsmve6TxVTtiZLEcnG8yRtQCPGq3X3+N1iwfz+uopLXNJ6YMZebv/1mlu96HmA8sVy8pEkJd1Yv/OcjvuvP2Py1kz60kp4oIYQQJ72/WfFPhzxecuBOOYgSx9dUneJAgighhBBCTKqpOuQgQZQQQgghJpf0RAkhhBB/HSzL4ZaPPYKTLNAYVfj6999yvJt0UpuqPVHqSxcRQggh/noUiw5v/vQaRl0/O1raeDzcynU3P3y8myVOQBJECSGEEBNc+4knmde3n4FoDK9j43Fd9tXUH+9mndRcRTniNhXIcJ4QQggxQQYdFWX8x4w118WZIif940WG84QQQoi/AqF0iu7ooTONL+zvPk6tmRqmak+UBFFCCCHEBAVNozqd4obnn6Z1eJAzOndy5ZZ1fOQbG493005a7lFuU4EM5wkhhBATBHM53rZ1LRsbpxGvCDEWDJLVvQzvGjreTTtpTZWep8NJECWEEEKUrVrXQ0hVSHt9/HnBEpL+ALgulqKi5/O4rosyRQOCySQ/QCyEEEKcJD73fx9D82gsW/bK1vv2f2yhp74JV/eUAigARWEkGKLBKr72Df0rMVV/gFiCKCGEECe1jo5hrvv3Heg+P47r0pIY5oy+bgqazs+GpvPeswdfcht9/Qke+P7TJPxhKooF8rqGZtvYmgaArSi47lTJ5DkepmYU9boEUYZhLAfOAoqADewFvmqa5l0TypwFfL5cTge2Ad8zTfOXE8p8ATjXNM2Lj1LHTcDtQOawp35gmuZnXmb7CoADDANPAd8xTXPNC+zHRGeZprnRMIxfADcC+fJ29pf34b8Mw0hNKO8r/58/sMA0zVB5+xcAF5imuWJCvbuAr5im+YsJyxRgO9AANJmmmZrw3IXA48AW0zQXHrav9wOXA+83TfMXhmG0UXo9Mhya67fBNM2zj3rAhBDHZu0eWLeHMd3L1Wt8WB4vumUTymdpSCV4y6Y15PxBNjU0MxQMMRQMk/HqnDO0n6U7dpL0+mlMjNATq2Q0EGKktpp8uILO1ha8KCxeEKWganQ8tZul6zeR0L08OX02OtAwFkdVFQaCEbKajm5bZD1eopk0tqrRFa2iIZ2kbmyErO4l6/WSDIZwXZe010tOVdGBimKRhT37yHm9JL0+FBTyHg8DoQiOY6Oj4qqgWhYVtkVR0/Hl8yiuC6rCQDDM7JEBUBTGPB7qM2lGKkLsi1ZSl0iwp7oOBQgVckRyGXbXNWGrGqFClunxEVriI2xuakF1bGYPDwCwub4Zv9/P9PgwfeEoF+zrQHcdIM+yni5+81Qrj973R+IVIfIeDx2xalKBIJpjM2+gj3A+R0dlNYrSSIMdx1vIUZtJ0xbfhKvA7qpa+sJR8i6s+NAvcZ/bTdC2sD/7Ds68fgmqJtdovRTpiXr1vmya5lcMw9CBjwJ3GIax1jTNXYZhXAr8Cfg6pSAkC7wV+JFhGDNM0/z8y6xjj2mas15N+wAMw5gOfAh41jCMd5mm+YejlXsBvzRN84OGYajAtZT2c4dpmqEDBQzD+Cmgm6Z501HWHwa+aRjGmaZpvtifPRcBM4AUcD3wk8OetwGPYRjnmKb5VLneVuBMoOco25trmub+F6lPCPFq/OA++L+/AuDLl78Dt20WOlDw+eiKRIlZRS7p3E1FsYC7aQ1fftOVbG9qBdfltN17uGLXlvFN7Usm+PRbr6UvEuONO7dw1caH+N75l7JjdQJvIc9nn1jB4p5OPnzN+4gVslRlM7gobK9qIJLPEbIt8ppOPBQhWizgAeoLWfbWNrBgsJctlTUMhSLsrapFd2yaEnH8QF7TsR2XLdPayHm8VORy9EdiWJqGx7JojQ/SlBrDQWFbbQN1IwliuSwKMBKoYEdlA1mPh62B6URzWaozKUbDUaYlRtBdWDljLigKmm2jZjX2VdeiqKUeoTd07SVcKP3d2T4ySDSXxePYKMBpPV08PHshCX8Qj2Ozua4RVJX2kUEUXNa0zeSKrSl+cdfteB2b3y4+g2+ffxkNyTHGAkF020ZVwFIVlnZ30ZBKsC9WzbREHIDZQ/3cP+cU+ivCjK3s5S17d6HisumLd/D9e7fz0Tuuk0DqJUzVnKjX/VU3TdOidMLXgdPKi38A3Gma5hdN0xw2TTNjmub/AJ8APlfuLXk929hpmuatwK+A/yj3+rzSbTimad5JKSha8gpW/QnQQikwejH/n73zDpOjuPb2W92TZ3PSaldZQplcBIlgsonG4WJjG3xtLGOcE8E2XBtsg42vry8O2NhgwgUDTp+NBQaTERmKIATKWVppc5qd3N31/dGj1Wq1QloUdqWt93n62ZrqqtOnZ2anf33qdNXngUeAuwvlgbgNXwxu4bPAffgi1WAw7Et+/VBv8fVxk3ovKflAAITgP956lXg+B/gDH6lQxG8gBA/NPGwbU6N6ujlp1VIAXh43CSxBZcoPRk9rbWJ8RytdkRjd0RgV6VSvzfGdbYUIDYRdh9pEV+9FoCiXJerkeW7SdIJa+/MkCUFxNtPbJuw6FOdyZIIhABKRKE5huCsfCPQmXFtoxnZ1EHLd3vOsSCcLwse/nMazmd7zaSitoCMW712k1rVtXNsm5Pq+lmbSvQIKoCqVJOh5va+DnkvQ8/Bsm2wwxAuTprFo9FgenXYwj0+ZCcDXnnuUkOcCcOHCVzhp5RLGd7Yxs6mBrliMTDCEYwd4YPYRjOruoiSdoisc4Y26cTSUljOmuxPXsjmiYR1WIWg/u6kBqytJ48q2HX7sBh8ttt8OBPa5iJJShoAvFF4ul1JOBaYA9wzQ/F78//3T95F7/bkfqAemDbajlNKWUn4CqADUILomge8BN0gpwwM1kFJWAx/EH768HThSSnnkAE3vBD4opSyVUtrAJWwfsRpSEomEKZvyyCiPrep9Hc3lesuikGezqaSst04D3ZFI7+vR3V30JRUKsbHMnwxyVE830VyWnoLoaosV4VgWRdk0nhA41taf+UwguI2dXGDrYIQHOJZNcSaNB4QdBwBXbHuZ0Nu83DZYbhdEim/b3mafh8CxRO/Fs++x49lM7/F8sxqrj0jqCYfJ2tsPnOiC38uqR28T59BC4AlBNhBkY3kl4L8vW0gFgjgFewKw+uU6BTyP5ngx/3fkcTw2dTb3HXYsa8qrCOVztPexkw4Eyds2dnxr32HzfXuP5b2FRmy3HQjsSxF1tZSyEz8K8iNgnlLqLaC6sH+76WCVUjmgFajZxWNMlFJ29ts+sRs+bxnequxTd3X/Y/Trc3Ghrhm4HPisUuqZQR73Dvxhuq/tYP9ngC5gvlLqDeAN4NL+jZRSzcDjwEXAWUCjUurNHdh8p995/WaQPr8niouLTdmUR0b5T1fAMVPR5UV8YcHDeFrjCIHtOERyWe45Yi7/mnYwDSVlbCgt57CGdYTyOUK5HJNbGllVXsW60goW14zmqUnTWVZVy+Eb13LE+tX8Y/aRZAMBhOuyuaSUe46Yw7KaOi555RlWl1fREYnRFo2xsryKpngxPcEQ3aEwneEoiVCYVCDI+rJKhPY4d/HrjO7sYGJrk58vVRheywuLznCUZZW1jGtvZXRXB1NaGqlJdBFy8gTzeVqjRXSHI7RF4zRH4yQDQXqCfu7U+tJykqEw0WyOcC5Hc7yY1mgR5ckeStIpVpdXEszlCOXzlBSEXDSbQaPJ2DZPTJnBsqpa0gXx5QFv1o3jickz2VRSjugjuizPw9KaoJNnRuNGQvkc3zv9gzw3fgqLRtXzo1PPIxXyo2muEMxsbCCezRJy8pyxbBGpUIgV1bWkQ4X7WCFoKinloNZGVpf5n8Ga8ioenzKT4+cdw6gx1dt+1vtx2TA49mVO1PWFnKhy4A/4OT1/ALY8NlGPn0zeSyFqVdWnzc5Ysxs5UQMxpvC3b6z2+p3kRN2tlJq3OwdVSrlSyiuA+6SUf+i7rzC0+DngHqXUlgT3PwA/kVJerpTqf0txK3AjsI53j0LNMjlRBsNepLwIHvsBAjinsG3Pwb2lS+l7Z3T0Nq1mAh9614NtDUzvLC/g3XzYMZMHbfW94DgOF92wkE0tWdpjcVYWl9IUDjGuu4tUMEgklWRaawuTujv494zDyIRCoD1GdXdS19PF2K4OOiNRvJ4epkwp5/o5p1Lq5CjtSbAmXoZl+RGS0V0dHNKwlowd5KFph7KsejRVPQksz8MrRPICrsuY7i7Of+W7vf5N3CfvwoHBgTJ81599PsWBUqpDSjkPWCWlPB8/oXw18AngiX7NL8SP2D62b73s5WP4EbJl+/rASqmHpZSv4g/t9eUU/OHPS/pE2QJAEf57+Lt+7R8FbsEXrZ/cex4bDAbDniUQCHD/9wbKVNie6/uUP/6fD9FeXE5PMISTTHDph3Ocd97RO+zbl/d941XeqaljfGcbsVwWAb1PMqpxk8xkm++RA2X4rj9DMk+UUqpdSvlz4AZgPv7Tev+QUq4BfoM/5HcOcBNwo1JqTZ/ulpQy0s9kjj2IlHIsMA/4NPCxnTwltze5HHiZPlMh4CeRL8AXeH25Af/GdRsRpZTSUspzgGjfaRAMBoPhQOW+u7bG+ebPnz+ovpMCGRZ5RbxVNw4tBEHPBU8Td/K0FJUYAfUeMcu+7Hl+gf/03acK8xWdih91uRKwgeXA5Uqp2/v1O5ntny7bErGe1G8+JvDzhnYlov1fUsqr8CNfbcALwFyl1CsDtPt2v7oLlVIP7sIxBoVSaqGU8j58MYeUsgY/ofwjSqnGvm2llDcCS6SUcgA7i/vXDcAyKWVfsdiplBqzw9YGg8FwAHLHf5/A2V9YQEtxCQhB3h6+OVAAACAASURBVA5g4yfMx3LZnfQ2jDSEmYHVMMSYL6DBYNhrbIlEnXfeebvc57zPPcma+nEkwxFiuSz1Xe1kAiE++coCPv/0V/aWq8OFvRIyeuToG7b7rT/zle/u9+Eps+yLwWAwGAx9yHsu05o2cddfbqOoMB3FIwfN4tgnrxhiz/ZfDtS75REhoqSUt+A/5j8QM5VS6/elPwaDwWAYvtRlUhzRsLZXQAHUJbooK+2fjmvYVUxO1H6MUuoy4LKh9sNgMBgMw594NkdJNsP6sgrGdbbjIfjzIUdzyFA7th9jIlEGg8FgMIwALvjGMbzzg4d5evIMqpI9LK+q4aX6CUPt1n6NmeLAYDAYDIYRwInHjePFumo2JfK8VTuGFWVV/NfR9s47GnaImWzTYDAYDIYRwlV3fHioXTigMJEog8FgMBgMhvfAgZpYvi8XIDYYDAaDwWA4YDCRKIPBYDAYdsDCxa3c8PM3sQTYWhOy4Wc/OYGKcjPdwf6OEGI6cAFQq7X+UuF1SGv91q7aMJEog8FgMBiAGz5+Pz88/26++Ml/8uGvP8/6Ne38+vrnCdo2th1AB4KkRJCvXvk8HR39Vx8zvBtaiO22oUQIcQH+OrT1wMWF6iLg54OxY0SUwWAwGEY8Pzv3diZt3MS6imoSsTijWtu49Vv/Jh0vhsIFXwABrcnaAX5ww8tD6/B+hh5gG2J+AJyutb4MCosjwkLg0MEYMSLKYDAYDCOeY9au4vEps8gFgtiAZdlUp5O9AqoXIYg5Odo78kPi5/7KcItEATXAlmE73efvoPSdEVEGg8FgGNF89ptP86/ph5AKhRD4V9FkOMKi0WN8EaW1vwHC87AQwyGSsl8xDCNRr7F1GG8LFwKvDMaISSw3GAwGw4hmnY6Rq6zG9jy6IlEA4tksFdkMuUAQT1hYngfQK7KGPI6ynzEMIk/9+SrwqBDis0BcCPFvYCpwxmCMGBFlMBgMhhHN5tIKSpw8bdEiSjMpgp7LqGQXQc9jVKKLhuIyPCEIau1HUTwXLDOQMxiGQeRpG7TWSwtP450LPAhsAB7UWvcMxo4RUQaDwWA4oPjs5/9NzguQDoXxvDKSoRA3PLuQSD5HcSZJ2NPUTyrlpquOoKcrTTYYYmHdeAA6MzEO2byBYJ/IUyocYUVlDYdt3gBAzgoTck1O1GAYhpEotNYp4M+7Y8OIKIPBYDDs9yxb1cEVP1tEAGguG00+ECDkuoSdPKO7OsiEwqQjUWLaJed6rFnXwwVfe45Rbe0w87BeOz3hCIlwGJ3YOnQXz2WY3tLYJ8lcmITiQTO8RJQQ4ll2ECDTWp+4q3aMiDIYDAbDsEUXErpv+uqDNDUkaI4VM6tlM6OSPbxeP45Y3kGj6YwWsX7yNNzCsBtA3rKxtIcnBGHXoTyTRFu2PxRnhahtbaa+u4OSTJquaByAkmyaiOPw8tiJHN6wHtvzQFhEXMe3KSws7eHYNg0NCerri4fmjdnPGIYLEN/W73Ut8FngnsEYGVIRJaV8GpgD5PHnaVgDXK+U+kufNnOA7xfaBYClwC+VUnf1aXMtcLxS6rQBjvFp4HYg1W/XzUqpq3bXvz3o4zVKqSm7uq9//QC+rgZ+pJT6207O8WzgcuAQwAbeBr6rlHr23foZDCOeTA6+dTs8twTmTIOffhqeXwKvrYKiMNzyb+hKQXEEDh4P37kA5Hb/4vCdu+DPz0NtOe6FJ+A+vghrXSMikSaPhSiNEZo7DfHhOeROmM2jz26mY1ULyeWNrA8WkYxEmNzRSnBFA51WkFVVo7A8l1NXLWFSWwury6vIBIK8MOEgEpEIZyx7m7JMimWVtSweXc+Jq5dhey65QJDiTJonJs/g+PUrydhBllbX8vbosZRkUsTyeTqjMTQC1xIkQxFytk15Okl5KkXYdRiV6GR0ogvb0zw2ZQYR16G5uIREKMoXXnySuu4O7jliLotqxxDQGseyCLou1cluPvbGy1SmenhhwhSenTCV0ckEJZkUQmt6gmE8q5Se8bXMbGqgIp8jFYly2sqltEfjRN08m0vKWF9ZRd6yaS4p899bIXC1JhcIUpXqoTUWJ+x6WPhP29Ulunh4xmFMa2mkqagELQSTWhsZ1dOD1v6cUD2FRPMtWJ5HwHUoyme5/jvPkLEDeLYNQHFxkMNmVxGJ2kQjAWbNqmTatIq9/EXcPxhuCxBrre/qXyeE+BtwB/4cUrvEcIhE/VAp9SMpZQD4MnCvlPINpdRKKeUZwD+BHwOfBNL4SWC/k1JOUkp9fxePsXoggbK7/gHsQR/3BH19/SbwJynlTKXU8nfpUw78CngK6AE+BzwspZyhlNqw9102GPZTPvxjX0ABrGmCF5bA2pbt2yUz0NgJTy2Cl/4bptZv3fejP8HNDwPgtSTILP3z1iEjrbG0JtzSiVi5Ce5+mvmXXcT9ydJC5yJIQ6y1i2xjM5vjpaysrgWt+dn8+xnf2QZANJ/nxlPOIRMMcfnT/+KY9av54+FzUBP9n8TXxk7ivHde56zlbwPwgSULsT2PCz71ZRpKfQFga826yhq08AexEuEIuUCA2q4OLCDu5Al5LsX5HJ4dwLNhfKKT1eXVbC6t5NtPzucz6jkATly7nPfPu4KOeBEA6VCYnkiUBw4+kvKMPwt4XU83Ie2RDYWxPI/yTIoXJk71c5NqRvP0lJkcumk9uUnTWVk1ioD2iORzuJbFJ157nt/NPQ0tBEHXYWxXJ/lAgFfHTkILQSSfY2JbM3WJLjaXlTO7aSPJUJi6RBcAAQ09kQhxJwdCEM3nSAeDuJaN7boFURUi6jmUp1NEHYe2WBEIQaLH5dkXN2MXPqGHH1nHt6+STJ5c9l6/ZQcMwzASNRAN+AGFXWbYDOsqpRzgVnxht2WA+mbgPqXUdUqpNqVUSin1Z+AbwNVSyglD7N+w8rGfr7/BjywdvJO2f1RK/V0p1amUcpRSv8UXU0ftA1dJJBKmbMr7Z/nNNWzDQAKqL44Hr63axo774Ku9Zc8S207sKASW9rbev2tN8LUV25ktymYQQHchYhLL53oFFEB9VweZYAiAWY0NALxTu1XItRSXUJTL9r6O53NsLinrFVAUbG8RUAAB15/gORcIIAQEPf91JO8PeWnAFRbd4ShaCKa0Nm/1N5ejtqer97VdGBoryWa28WHLvEyeZZEOhckGg737u6IxekJhumIxPEtQlMsQ1B4R1+GhWYdT39XOURtWc+7iN4jns2wqLe9NbM4EQ2ggXBies7UmWDifgOsSKtQ7lt27vzrRzfjWJrrCEf8zEoLWmD+MF+j7GQF9c388T7NylX+uw+Z7u5PySEEIcUm/7cvAQ8BLg7EzbESUlDIEfKHwcrmUciowhYHHJ+/F/6aevo/c286/Qt2w8nELBV+/hD+0t3CQfQ8GqoBFe8G17SguLjZlU94/yyfMZBtmjQXrXW63I0GYM30bO/ZFJ/eWLc/rFQ5ojdAaV1hbM18DNu5J298TdUeiaKA8lQStSYXCLK2u7d2/prKaWEEkvVk3DoDDNq3v3T+6q4OWQlQIoCsSpa67k/Htrb11ZekUdkEoaSBfGL4Kui5aQzbgD2r0hMOA/8NneR41yW5sz+Vvh0jcgoh5c/Q4lleN6rXtWDZB1yGS3/q0WyawVTCN7u7koJbGXqEDEHQcwk6OoOtieduKmJwd4KRVS/nOkw9y8esvcskrz2L36Wt5Hi3xom0yipuKSnijdiyJUIjZDes5c8lCxrQ0kRcWXuF8O+JF21wwA4Wn93KW3e/Js62Wg0GLmTN8MTpsvrc7Ke8tNGK7bYi5uN92JvAC8InBGBkOw3lXSykvB4rxL/rzlFJvSSmPK+xv6N9BKZWTUrbiT9u+K0yUUnb2q/uiUure9+pfYV/1HvRxT7DF1xywEvjIlmHHXUFKWQP8DfiZUmr7W16DwbCVey+Hn/0dHn0DTj4ErvywH516czVUl8DP/gGt3VBRBEdPha+cCxP6/Rx8+Rw/snHH41gTRxH9xEm4j7yOtbYJ0Z4grwX5qmJCJ86EsyXnHzKBqtda6F7fQXrxRtYGi0iES6iYFKb8xeVUrVnOsurR/OSks5mzfjXT2hrZFC/hzHde54WJU7l5zsm8XVNHdTKBXLOC5bV11He0saKyhu+fdj7lqR4WTJzGCWtXMHfNUsa1V7CsejQlmRTj21toiReTDEew4sUU5TLEslmKs2nyQuDZAdoiEYoyaaK5DGtLy4nnckxubuTl+olc+f6PENIeD08/BE8LQtkMIc9ldC7H9M0bOX/JGyysG8+aiirWl5ZR3tPDlNYmtLBxLItvPPUQz02eQSYQwNYe2vMIuC7l6RS5QICQ5wuqkkyaQzZvFYl1iU5s7SG0xvI8wq6DYwfZXFLG2M42WmPFrKgeTTYQIOo4zFPPYmvNYQ3r+clp5+EUBucsrSnOZkiG/KheUSZJVzBMJhAA7WHZgtGjYhx5eA3RaIBQ0GLatHLGjDGJ5zD8hvO01ifvvNXOGQ4i6vpCHk858Afg5MLfLbHxevxE7V4KkZaqPm12xprdyInakX/sYR93RB4IDlAfLOzbztf3chApZR3wGPAo8J33YsNgGFHYFlz1EX/bwtEH+RvAh+bsmp0vne1v+EMD1oeO6d0V7tfUAk44qgaOqoGPTOu39zjejY/3lk5513ZfB2CXn/DeIV/arsaPov14wNZHAB9416miX3utgcdvWUxZMoH2NKsrqohoaI0VMaazjapkgqDrImyb9lgRtPk/vW2xOK3xYjzLIpLPEfA86rs6SNlBVpdXE81lcSwL17L8JHutcYXgrqNOwPI8vMKkmulgiID2KMlmcAHPsvnqdyQzZ1Ttzts0YhgGkSeEEJN2pZ3WevWu2hwOIgoApVSHlHIesEpKeT5+svZq/NDaE/2aX4gfM31sqPxTSj0ArNgHPq4FRkspY0qpvk8YTikce7cp5G09AfxdKXX5nrBpMBgMe5Ijj6zn/lvrd7jf7erhS79aQnJNJ2/WjmFq0yZCnsePTv0A6WCIWDZDZbKHolyWsJPHCQaZNr6I5StyRLJpcnYRL42bzKUvP00yFKY9XkTA89Ba4whBNhBEoHERBFwXD20E1CAYJpGolex81R4Nvc8G7JRhI6IAlFLtUsqfAzcA8/GfhvuHlHINfqJ0GjgHuAm4USnVN7PTklJG+pnM7S3/pJTzlVKelHJP+SgG2JfHXwxxBXCTlPIqoBv/lnMecNnunpOUcjrwOHCnUuqa3bVnMBgMQ4FdWsQt1/jPw9z26fv4/dHvo7m0jI5ojHEtjVRkUmgNOdsm53l8/Jw6Ln7/eFxX8+0L/8q/Zh/Oy+Mm8d0zP8IFC1/Gcl0820ZoTbaQmJ+zA+Rsm6DjEM3v0cvLAc9wiERprfd4HviwElEFfoH/ZNunlFJ3SilPBb4HXImvDpcDlyulbu/X72R8AdOXLRHsSVLK/uvhzFdKfZzB0+sfvvB4eE/5OMC+7yilfiKlPAe4EX8OpyL86NS3+s9X9R65Cn848utSyq/3qf+8UuqPe8C+wWAw7FPm3bn1p33+/PkAnHfeeQO2tW1BR3EJoxLdJCJRFteO4braMcQzaWJOHk9YZINBLNclGy7c5woxDCTBfsYwXPZlTyC2zAZrMAwR5gtoMBj2GjsTUQD/Oe8xAkBpMoFn2XRG43zmlQXcd8ScwtIvmqbiUiKOQ86yieay2J7DXX94/745iX3LXlE7d53y6+1+6//zyS8PmbISQgSALwLvw89f7jObiFn2xWAwGAyGXcKbMorc6haWjhrDslF1fp1lMa69hYdnHkZ9V0fvPFZRJ4+tPSxz+zcohuECxP+L/5TF74HrgavxpzG6fzBGRrSIklLeAly0g90zlVLrd7Bvv0FKeQLw8A5236CUumFf+mMwGAzDjbu/fQhfvOifvDx+60PcasxEDt+wGqE1YcfZ2rgwlOcNP1EwrBmGmvPDwByt9XohxHVa618IIf4N/A64dleNjGgRpZS6jD2QnD2cKayBV7TThgaDwTCCiUQt4pkM3TF/IeKKZIKuUITLnnucN+vH0xn3l3axPRcLqKruPwGF4d0YhpGoGLBlabO0ECKmtV4qhDh8MEaGzYzlBoPBYDAMFT+/9Vw+mV/PhOZN1Lc2cUg4zZX3fZTHp82mOxIh4DqEnDy25+ECN/549+fSGknoAbYhZglblzdTwLVCiGsYYPLsd2NER6IMBoPBYNjC5T85nf4T5f32x8dw8TWKSC5H2PM4ZEYpV3zzKMTwi6wMa4ZhJOprwJb1gL4J/BZ/ZZJLB2PEiCiDwWAwGHZAeXmUB28+YajdMOxhtNav9imvAE57L3bMcJ7BYDAYDIa9ynAbzhNCLBRCXCGEGLs7doyIMhgMBoPBsJcRA2xDyrX4OVFLhBDPCCE+L4SoGKwRM5xnMBgMhhFPVyLLSdctxg77T92Fkj28cNPRQ+zVgcNwy4nSWv8d+LsQohh/uoOPAz8XQjyhtf7ArtoxkSiDwWAwjHjOvHYRkWCQikwaLJvO8krO+vxTQ+3WAcNwG87bgtY6AdyLn1j+MnD2YPqbSJTBYDAYRjxRrZFrlmMBqWCI58ZPodRxSeYc4iFzqdxdhlskSviPV54CfAL4ELAOX0z952DsmEiUwWAwGEYMj7/dzkU/fIN7Hlm7Tf3Yrs7eC2Isn6MkmybquRx33bJ97uOByDCMRG0Cbin8PU5rfbjW+r+11ht20m8bjLw2GAwGw4jg2MvfoMLJ0xmL8esFSf7xj6f56y0nAX70aQuuEHRG40xtbSQXDACzhsbhA4jhFokCztdav7K7RkwkymAwGAwjgomJLjYVl9IVjpIIR1hdNYovffkRAJqLinmrdgwrK2t4evJ0OuJF3Dr3FIoy2SH2+sBguEWi9oSAAhOJMhgMBsMIYPq1K7HrxqKFwPI8LCATDNGRDXLfH17HtWwyAU1TcRUAQcchb9u8Mn4SbzckmV0fH9oT2M8ZhpGoPYKJRBkMBoPhgOava6sQloW2LBACz7KwPQ9ba6rTSexbH+P41ctwrQBh1yXsulSmehCeRyoQ5CO3NQ71Kez3DLdI1J7CRKIMBoPBcEBw3g/foqkHPMumJxgkrF1Gd1URcR3sag/PtoGtF/GydIqfPfgnsoEA9T3dPDN1Vu8UkJbWhFyHbCiM5TpDdUoHDgdoJMqIKIPBYDiAyDsu9zyzCTmllEnlQe7/2zIyaYfJs2o46dhawqHgThfPzeQcFqxI8L6D4oRDoXdt+17JZPNEwkEAGjd2kIuG8FzNa2+1cPD0csbVxLn63rVkHI/rPjaO0lCAm/66nNsXW1SmekiGIxwTTrO0zeGYxo2sq6hiQ00dXsRGA0GtiToumUiU5kiUoOfhaU1PMIQTCpIOaTwBH7voizSUlDOpvZl4NkMyHAEgHQwS0Brh5Mlo+PczGznt+Dps2wzgvBf00M9Qvh1CiNOBC4EarfV5QggJlGitn9xlG1ofKEG1nSOlfBqYA+TxV29eA1yvlPpLnzZzgO8X2gWApcAvlVJ39WlzLXC8Umq7BQullJ8GbgdS/XbdrJS6ahd8lMA1wHFAGGgE/gXcqJTaXGgzE/gBcDIQLZzHH4CblFLeAH54QAZ4G/gjcMcO2g3a3z3AyPkCGgzvwo03vMzzjR6JcIRYLsvJS9/i9rmnctqKd6hOJmgsLqWpqIR4Ps/m4lJa48WUJhN8bNGrzG5q4O1RY/jbzMNZV11LyMlTnUyQCoXxELiWhRZQkknTGo1TlM9heR7NxaU4lkXYcahMJwm4Lu3RGLbWxHNZBHDBmy9x5Kb1xPI5NpSW89iU2SwePYacZZEKhYk4eRzLJh0M4hXEmYefKxLL57C0RhS26U2baY7FObphLZ9+7XmKshnuPewY/t8hR9EZiXHUhlVsLK3AtW1ydoC2eBHpYIhYLosHaMumPNlDPhgkb9mM7u4g5HkIrQnns7xTOxbP8kWO5bmUp9OkgkE8y+59n3tCYbJBX7zFclli2Szl6SSTOlrJWzav143FCQR7c3hs18URMLN5M2Enz2v1E8mEQlhaM7q7g6biUhzLxnZdSrNpQJAIR8gHAoX+DlpYlKWTfPT1F3n7onO566OVBKzhJyoK7BXHfnn27dv91n/1X5cM2ZsghPgK8DXgNuA7WutSIcQs4Fat9dxdtTMSJfUPlVJFQCVwJ3CvlHIKgJTyDOAp4EVgElAN3AjcJKW8bhDHWK2UKuq37YqAOh14DlgGHKaUKgHeB7QV/iKlPAR/VtUWYDZQBnwd+CZwxw78KCmcz6/x1wv6yw7aDcpfg8GwZ3jx2Y28viFLc3Ep6VCYtqIS7ph7KuM7WqhLdBHyPMZ1dTCjpZHKdJJZzZsIuw5HN6zlA0sWMqm9lQ8seRO5eT2ubVOZSpIKhXEtG21ZCCBnB+iMxinNZQl7Hu3xYpxAACyLklyGiOsQQFOVTuFaFj3hCAGtKctlObhxI3LjWj70zhv897/+REWqhwBQkssS8jxiTp6Q6xJwXSzABqKOQ8jzCGiNBVhCsGxUHSeuW8lpq5YwsaOV6lQPX3rxKTxh4QQCvDhxGlg2JdksiXCUnkgU17ZJRGMEC0LMs21srYk6eUKeB/hJy5WpFMesW9n7nnrCwrVEr6jqpV/gIB0KkSsInqDnUplObnNnF8tlyQVD2Fr7PgjRm1fVUFaJYwdACNxAgPZ4Me3xol4Bhda4dgDPsmiPF/PPgyVr3tzMzxYk9uTXZ79Ai+23IebrwGla65/g637wgybTBmNkJIooAJRSDnArfrTpsEL1zcB9SqnrlFJtSqmUUurPwDeAq6WUE/ayW78B7lVKXaWUaij4uVkp9UOl1P2FNj/3q9UXCvtySqnHgIuAT0kpjx/IsFKqRyn1/4BPAh8uCLYhJ5FImLIpj/hya3MKZ4BhorJ0epuwgK3933oBRPJ5KlLJbdpXFl5rsePhE6sgItw+4iLkulv3o7G0xrMsNJAJBAk5W3OC4vkcoxNdA9vtM0xoaa+3vKU26Dq0FhUT7JNjFNBe73kBRJ2c73+/SI0G7D7H0IDlbe1XlkkxZ92q3tdBz/WFneOQL5yLKwSgEZ5HyMlTlerxffU0FckeWmNx1lVUkwsEcISgKJ3ioNYmHDvAuvKqwvs2iKt/v2HTZDhCeTrF2rZMb91Qf/f6l/cWGrHdNsQUA1sm1tyim4NAbjBGRqyIklKGgC8UXi6XUk4FpgD3DND8Xvzfgb0mPPoc/953aRMFTmIAH5VSTwMbgbPe7ThKqQX4M7Se+t693XMUFxebsimP+PKpZ06kPJsm7OQBCLguZckEb9fWFy78kLNsugv5Ot3hCG3xOI8dNJvWWBEAjcWlPDl5BmhNS1EJYSfnR120Jl+IRtmeRyoQAK0pyaR7ozKJULjXp5xl4wlB2MmjgYpkgp5wpPcqs7KimuVVtX6UpXAhdIUg2ydp2xOCTCDY28cD0Jp4LkN5soefnnQOzfFi8pbFLce8j5Z44T3Rmq6wn79U1ZMgnPffj6DjgPaPE+gjwEpTPUxt3swRG9fylece4/mJB2F5HtFcjrKML0CT4TBl6SQekLdtgoVzr+vuxPY88pZF2HWY3dRAbUEcaiFwbRsh4NnJ0+lLcSbdW47mslsjW57XWxZ9xF0vWnPG0rdYWV/PVSeX7/J3Y1+X9xbDMBK1APh2v7qv4o9G7TIjMbH8ainl5fgqNA/MU0q9JaU8rrC/oX8HpVROStkK1OziMSZKKTv71X1RKbVDgYQ/dDjg8ftQgR8p31GbTeyajxvxhzO38F78NRgMe4hYPMhvbz6FBx9fz7JOeP/BcYL3PM0/K6bw5LFHM23lSlbFy+mIRol4momBLKckNjLr9UU8OWYiC+vGsaG0jDxQ1t1NJhwk63kctHkDIc9jcXUtTjBIOJvD1h7ZgE0gn6O0qwNt24SyGbotG4Gm2w4gXBdXg6XhhuNO5+ajT2ZcRxNV6TQvjJtE3grQFQyibZuiTJqQ55LxPAS+ACzKZvHwpwfQwkJrz8+5SiZ5atxkalM9fPzCS8kA47s7qOtopT1aRM6yaI7ESFo2lvYo7e4ins/SGQhj2RYxz6MnGCTkupSlk7SHowjHZXRXO1ec+zEWTJ5OOJ+nMtWDBTjCIh0Mc/47z/Gnw47BKVzyXNsmEQgxvqsNrTUdkSjrS8s5ePNG1NiJtMWLKU/1EMtmewVRMJOmzQoSSiapyOcIOw4RJ0djvBQ0OLbNYa0N9FSWs7mqhg/NCLO4Ocfqds3kFct5n+5k9vVnce2YMEXhkRe/GAaRp/58BZgvhPgcUCyEWAYkgHMHY2QkiqjrlVI/klKW4ydjn1z421LYX48/LtpLIWpV1afNzlijlJoySL/6Hn/JDtq04yfE1+9gfx3wxC4cawzbqu334q/BYNiDBEM2Hzp74taKGz/KbOC7AEzdQa+T97pf+wuvrenijdsacEIhmmNFhJ082WCIeDbDgolTOWbdSp46aOvyLdlwmHXlVYzp6qDC6+Hew45lzvqV3Pbn29lQVsHYznaen3gQD04/hNWVo3juf47cBS929DnV7ZFz3J8ZBpGn/jQBRxW28fhDe69orQcII+6YkSeHCyilOoB5wDlSyvOBFcBq/BWd+3MhfpT6sb3oz3JgJfDxd2mTxg9BbuejlPJEfHH08LsdR0p5Av5/9C4/wmkwGAzDnSMnlvL29TNZ+v0pLL52Cm/8aAa3niA4rmExOTvHUwfNIuDktw6/aU1FT4LOcJSmaBHlqSSd0ThPTZnBklG+6OkJRdhQXoX23Hc5smFXGE45UUIIG0gCIa31K1rrv2itXxqsgIKRGYnqRSnVLqX8OXADMB/4MvAPKeUa/CTvNHAOcBP+FANr+nS3pJSRfiYHlZA2AF8E5kspm4BfK6U2SSlHAZfgR4vuB74FPCul/DXwI/zoEe+9kAAAIABJREFU1An4T+bdq5R6diDDUso4fk7XL4AHlFKP7qavBoPBMKw57pRJtCffIef18L03PDJ2gNnNDXRF4tQmuuiMxkiE4tRkE+QdzdryKjaW+5kOCyZOoyscIZ5MMmnGrmZyGHbIMJpsU2vtCiGW46e1bNodWyNaRBX4Bf7Td59SSt0ppTwV+B5wJX7+0XLgcqXU7f36nYwvsvqyJYo0SUrZ02/ffKXUDqNMAEqpxwpP110DLCoMIzYCD1KYvkAp9YaU8lj8eaIWAxFgHfAr/Cf3+rLFDw1k8eeJugF/XoyB2g3KX4PBYNgfCFmaZy4fzxk/XsG4rk7o8lNAA55HVsP8Xx7L9efdxVPTZvdOttlSXEIom6UnHufFS8xw3O4yDCcE/CPwoBDiF/h5wr0umsk2DfsT5gtoMBj2GvPnzwfgvPPO48Rvvcb09hbieX/QYEVlDf/4+TEEbMHDR/yQN8ZNZP6swwGo7e4kksuzsbScZ3966JD5PwTslZDRT8+/e7vf+isfuHgoJ9tcs4NdWms9aVftmEiUwWAwGEYEz/z0cM762osIO0A2EODys8oJ2P51/JVxE/mvJx/kqA1ryNk2i6tHs7q6lg2UDbHXBwbD7W5Zaz1x5612jhFR+xAp5S34k2IOxEyl1Pp96Y/BYDCMJIRt8civjxtw399mHcncDas5Yc0yXq8bz7Lq0WwuKeXUOZUDtjcMDj2McqL2JEZE7UOUUpcBlw21HwaDwWDYlspMis9d8NltLvYVbS1c96GxQ+jVgcNwi0QJITawA7e01uN21Y4RUQaDwWAY8fzuU7Wc80CWXCgMQhDKZnn1hlk772jYX+k/KjQaf0Hi+wdou0OMiDIYDAbDiGfqofUsP9jjh3cuZbMb5DfzpviLDRv2CMNtOE9r/Uz/OiHE08Aj+E/t7xJGRBkMBoPBAAjL4nuXzBxqNw5Ihttw3g7IAoNKODciymAwGAwGw15luEWihBA/6FcVA85mJ6t+9MeIKIPBYDCMGJI9OcK4BIqiQ+3KCGN4iSig/xMDSfwJq+8ejBEjogwGg8FwQPPHf0S5/M21uJbgd3+9k5NXLWFlRTVLv/VRzv3KCUPt3ohgGA7nfUdr3di/UghRi79SyC5hRJTBYDAYDlgaFnTz4pTDwbI4dOMaTlm1hOaiEgSg/+9ZPr8iz+9+eUpve8fxuPQrT1LW0UUiHOXz35TIQ83aebvLcBvOw1/SrWSA+sVAxa4aMSLKYDAYDAcsq9bbUGUBsHDMRKZd+ROmtDZz9eMPUJZN43Rn+OFVT/JfN/pC6nsf/yunNm/ilBWLSYSj/CGVZMWm9STCYQ694TyOOXb8UJ7OfsswjERtp+qEECWANxgjRkQZDAaD4YClVYS2eZ0Nhnhn9BjuOPp93PK3O/nnjMN4o7sIgOaWJALNue+8iRaC4myGa556kOJcFoA/Xp6l9raLGT/dRKYGy3CJRPWZZDMqhOi/SkglcN9g7BkRZTAYDIYDkq9e9A9+9Oy/mXvYMdDvIr62oop7jpxLLhCgO+onmd/ywDqaikpYXlNLKhQmlM8ztbWRXxx3OpPamzlq03quufEN7r7j/UNxOvs1wygSdRF+FOpfwMV96jXQpLVeNhhjRkQZDAaD4YBj2veW8+iCR6lIJZnQ3sLaimp/hxCEnDwrqmv5/vs/TMB1qGpv5VM3LGRFt+DCzQ1EHIdZjQ0I4J2aOnrCYf5y6NGEHIdUMPSuxzUMzHCJRG2ZZFMIUaW1Tu2uPWv3XTIYDAaDYXiQzOT54LwnOHfFOwRcl4jr8MwtP+H3f72DeDZNZaILr086jGMHuPaph3BWNvLgHb9gU2k5y6tGsaK6llQohLYEZ6x4h688/zgZO8ChDeuG8Oz2X/QA21CitU4JIQ4TQnxFCHGdEOIHW7bB2DGRKIPBYDAcEPzvdQtYvayNG155lqiTpziX5TdzT+XuI+ZS393BqJ4ErmUxqb2FukQXi2rHsrmklEW19Vzz9L9ojcV5ZPohzJ99BLbn8dUFj3BY48Ze+zObNxMv5EcZBskwiURtQQhxKfC/wKPAWfiTbJ4BPDAYO0ZEGQwGg2G/ZNWmHn5yx3JY3MCUrjY6o1HWjh7HNz7wCSqTPRy/Zhm/Ov50APK2xezGBl6YcBCNJaVMXvQ6X37+cf5w9AncLU9gVVUtDaXl5AP+ZdG1LB6efiizmzcR8Dxyls2K6lrimd0eATIMD64EztRaPyuE6NBaf0gIcRZw4WCMGBFlMBgMewDP9chnHXpakpTVl5BqTxOvjOFmHYKxd8+jcTJ5spk8YUvgIci4mtbuLJVNLWQry7CFIBWP4nmCYAD++nIb+cZOxuaTrG5IEagrpdsLMubNZdiJHoqbm0m4QVaVVLGqpobNpVV0R6NEnTwHtbUwob2Z9nCU6c2bWTh6HAELBIKVFdUkQyHCrsPM5s14rsfC0WOZ3dzAmO5O/j1lFtlImFhAsyEcI+652BaUtLaTDkdpD0UoS6cY09WBIyzWVNUQzWQ4oWEtLdE460pKKc0msawgQdchb9nUJrpZVltPNJtFrl/N8to6QrkcqVAIV0NtJkXIdbAch1XVtSyvHYMGJrc1Y2uPTSVldMw+nKddB4TgjGWLOHvpWzQVlTC5rZk77/89vz/mZM5a8ibXvf/DIAQawcPTD2HuupWM7eygOJshFwwCILTuzd+Z2NFKY1EpESfP32YfyeaSMspsm5O/8gLxvEM+YBGwLfIuzF2zlHorjzh4EqtGj+LNnjBF3Z381xlljN7QRHZtM6VfOotEMEhLt8vY8gAaQSqT5+3NeSbVhFm7op3WjMuJB5dTmcuxSYRobk5y6PRyVq7soCXtcoLbTe7wKXSlPcaU+z6nci7ZvCZoC+Jha1gunKyH34zlNVrrZwtlTwhhaa0fFkL8cTBGdiqipJRPA3OAPOACa4DrlVJ/6dNmDvD9QrsAsBT4pVLqrj5trgWOV0qdNsAxPg3cDvSX+Dcrpa4ahH9buF8pNa9Pm4nAKuAJpdTp/fr/CLga+JVS6qt96mPAJqAUGKuU2iilnAdcrpSaXmhzD9CjlLqsn83PAdcBByul2vrU/w9wIjBXKdXX3759pwNL8Keg14ADrAT+CfyvUqpngHZ9eUUpdUqhzfuB7wEz8fPfNhXem+v6HG9Goc3JQBHQBDwJ/FgptXogHw0Gw7ZsXtzEA9c8TD7t9NYJzyPuOgitmXLmdI6/4tQB+75132s89KclvF4/gbCT5+vPPMw1Z/0Hn3t5AaesWkIiGmPeBZewvqKK4kyaM5a/zWt141ldNQpPVBCKOkSb83x1wcPcduwpHJNZxec2vk3QdZgRa+Pbsw5lfVklFakeNpWU0VpaTkUmRS4UZsH0g1ldUUM+EOD8t19DLlrHaSvfIeh5/P7o93H70SdyyasL+P5j/gjHR99+jSltzVz2kU/TPn4KHjbJQIjOUWPR2qMqlaS1ahStVaMoyqSY1trE2M52zl65GMt1qUgnKc5m+MQnv0BzcSml6SRvjZ/C7MaN/Pjff6M4l2XD2nK++MGLyQeDZIIh2jvbmN3YgBOO0FlUTH1XOwe1NLK5pIy3R49FA7b2sIGDmjdz7aP/QI2dyPjOdsZ1dQBwxdP/IhkMEc3nyBbEUjyXJWMHcC1BQGtygSDxfI4xne10RyLUdXZwzPpVdETjZAMBgq5LSTrJW3XjOXvpQj7/8tN4QnDLMSdzztI3yQZDRPI5Fjk2z7qjyAQ1LYEyrv1bK5969XmO2LSBpx5awRXnXYhjB/zJiIQF2s8QEkJQkexhUnsL3Tc+xKUvP8OzR87ld8eeTHekp3c4rDSV52tX/IL/Ovs/iAcFnzokwp8XtPjfOWBMZYhfXzKeUWXBvfBNf+/oYaeh2CiEmKC1Xos/8eb5QohWIDcYI7uaWP5DpVQR/hwKdwL3SimnAEgpzwCeAl4EJgHVwI3ATVLK6wY2NyCrlVJF/bZ3FVD9/euzzeu3/3NAB3CqlHLyAP2XAZ+QUkb61H0UX3QMGqXUrcArwO+21EkpTwYuBT65IwHVjwlKqWKgBvgm/sKIr0gpSwdo1/fctwioacDfgV/hfyaVhXNa2cenI4BXgW7gWKAYOAZ4Bzhz0CduMIxQXvo/tY2AArC0Rmg/fXblI0tpXdq0Xb98KsdLd7zKa2MmooUgEwzxu7mnUtOT4JRVSwCoSKf4tHoOgEQkypKaOv8ibFkgBLlAEMeyWTpqDLlAgC8//zixfI6g51GaSbOuopqqZIK2opLe9m/UTyAZjhD0PEYnOgFYUlPHiWuWEfT8uQYvfeUZovk8Fyx8pdffaa1NvDD+IF4aP8U/RyDkOAg0CIvWoq0TQPdEYnRG4zSUVdARjTGho41DGhtYMGk6zcX+z1hXNI7QmvPffr13LqaxXR0ct34VmcJTcBvKKukOR+iKxuiMFdFcXMrj02azqH48WggmtzcTdvz3vqanGwv47dxTCXhury9Bz6U9FudT6nkmtLcwsa2ZuWuW88Csw0mHwr3ttBBMaG9mclszHbEYKytreGbydF6acBA1yQRZO0A+EGDeKwuwtSboeVy48CWyBV8zwRCzmjYRLETFANaXVxF2XDqjMW479iQcO9D73qE1CHojR+3xInrCEZ4+aCYNJWV86rUXiOZz2+QTdcXiNBeXMq6jlWRec9dLXQi2zhy5sS3Hfc/33rsPGzRiu22I+Skwo1D+AXAPfgBhMLplcE/nKaUc4Fb8aNNhheqbgfuUUtcppdqUUiml1J+BbwBXSyknDOYYexopZRD4DPBDfLH0uQGarQVeB/6jT93n8M/1vTIPmCul/M+C8LkL+JZSavlgjCil8kqpZ4HzgVrgqzvpsgUJtCql7ldKOYVtkVKqb6jyl8ACpdTnlVJrlVJaKdWqlLpJKfWbwfj5XkkkEqZsyvt9ORgZ4K6/3zUiEA1u1zeZThEIWATcrRf8iJMnFdp2+C/Z50If9NwB7uo1Zentc3WCrkPYyW93wcoEt/prFYTemK6ObR7fTwWD5G0/D6gvsfy2idVbH13f/nkr2/OwPI+w4/QepzzdP3gOHbH4Nq+b+4gxtCbgefQU3gMtBJ5l+/uEoLmolHGd7aA1asxEFtfUUZTN8OvjTqMnFMYVgoiTY3prI5Pbmvny80/wwbdf58UJU5ANa5nY2ozGn6a6JxShM1ZEaTrNqESCt2vH9gqYtZU1fGSRQguxzeeTs7cd0EmFwjjWtpfWgPYIeB7R/Nb75x09nWZ5HsLziDh58paFJ7a/TJelk73HtQfQIgG2CvrBfp/3Flpsvw0lWus7tdYPF8oPA+VAudb6t4OxMygRJaUMAV8ovFwupZwKTMFXcP25F/9n5PQB9u1LPoC/Ds49+EOGnykIq/7cSkFgSSln4kfV5r/XgyqlWoFPA7/Afy9eV0r9fjfsNeNH/AYeE9ieV4AaKeXtUsoPSCnr++4sCLu5Bd+GjOLiYlM25f2+fOIX5lA7o2Yb4eRZNlY8TNHoEo7+4vGUja/Yrm9ZZRmnXX0aJ3U3UJPsZkpHC5e9/DTFmTS3HXUCjUUlvD2qjjvlcYScPPWd7Yxtb8UDAoWhwlA+h+15jO5q56RVS7j7yLlkAgEygQCbSsr5n/n3MbmtiVg2A0BpOsXH3nwZgHA+z8S2Zo7YsJqcbXPNmR9hcU0dqyqqueGUc/GA75z5H/z22JP4y8GS/ztiDmM72zn3nTdAaxzLImfbeEDQcansSRBwHYKOw/i2ZkKuQ3WiiyU1o7n9qONZXjWKcxYv5JQV74DW1HR3cpF6lmcmTePfU2exvGoUdxx5HK2xIia1NhF0HEoyKTaWlrO+vAoPP2cJvVWCRJw85ZkUM5o20R2J8JmPfpbNxWWsrKzhSx+8mI0lZQQ9j7DrcPimdfxr2sH8z4nv59a/3MGXn3+C7z3xTz7x2gu0xotxLYuyVJJxnW0cuWndNoIvks9x8solTG5t4uY5p7K6vIrlVaO4+4i52I5DwHXA07xaPx6hNbbr1520YjEuvnj8+oJHiGUzvY/4CzRo0Fpjuy51nR0UZ9OcvWQhqWCI60/9AMlQCDwPtEZ4HkduWE3YydNUXMrUqgA3fLiGYNgGAfGIzcmzirnktLr3/H3eWwzDSBRCiEohxMVCiCu11jmgRAgxZjA2djWx/Gop5eX4wz15YJ5S6i0p5XGF/Q39OyilclLKVvzhqF1hopSys1/dF5VSu3KR3+LfFs5USr1UKH8emK+UapVS/h9wA/BB4C/9bDwA/LowDHYp/rClw26glHpUSnkf8DFg6u7YKrBxADurpJR9b2puV0p9Uym1Qkp5LH7k6iZggpRyCXClUuohoAr/J3+7z85gMAyOklHFXPDzD7ynvuOOm8Rlx03qU/NB7ujX5pVtXh26A0tfAfwLspP+EOlEhszGJJnGbuSrTdipJjZ1WBy5YS1Hr12BXLeSB6cezKbySixg4ag6yrIZNsWL6AzHcFyPsc2biIyt4rnTjmdmuou26WOZdvQ8fjw2wndzeYRlE7IF61oytLy5mZ5wkLGlHl36/7d352FyVOXix79vd8+SmUySyUoSCNkICQZkOQRDRAgBRAGBi14IIkQvXFDxXr2CXEXZw+IVxAV+IIKAXBC5IBBkR1BAEV5WI0vIRjbInkxmn+4+vz+qZujp2Xom09Mz3e/nefqZ6qpTVW/V9HS9c86pUzE2+F3YtqOJ3YdNYP1rq2FbPfLf36Fm4lB+0CB84Y31vPHQEj4qL6ekpob7Zu7LAYMaiUdiTC2NUzp2GLPi1by8uomPiooZVr2dpkiM3bdv4ZMfruHJaTNpiMUYv30r9dEYK4eNIImQjMVYPmIUiLC9tIyHZu7PjOceBaC6pIRXd5tE1MMemze0nLmDVi/n4A/ep8kLZck4Ue9JhI99QYSERNhzwzoiwN6rl7P78ftQctYMKocO4vpJFfz9nwcwZWSECRNHcky4Te89yaQnGv0EcCIAY4F/ZPSpOAwImk8ub7NsCjCPk1PmHD2zvefo9i+5rnlKJyKHAvcDCswhaN7bAzgPOC7T7WSaRC1U1Succ5XArQSdkG8FNobLxxN0Jm8R1lqNTCnTlRWqOjXDsu3Glz7TOTcZOAI4FkBV1zvnHiFIrFolUWHSdyfBN9EpBH2DesObwCFhzdTO2hVIb+ye0tG2VfUN4GsAzrkxBJ3/HwgTxc0E/wyNb29dY8zAJCIUlZVQVFbC/mOGAuM4+pjpKSUOgvAS/Jl2tzA7o/1UxKIt03uMG8we4/bouPABrZsEhwN7jK/gxGN6+pUPPwp/1m+r5cKrXyK66SNWjhjN5rLB+GgUvCcRjfLW2An8atahxKNRnp0yg42DhzCyuopXx+/OAeHAmauHDedLb77MLz59JKuLS9h73WogaOocWl8XHG9DPf9v9uHsKC7lW6dNbxXLXNe2rkBEiLbX1lag+kPNU5rrgZO998+IyNZw3t+BWd3ZSLeGOFDVreEdasucc8cT3DG2HDgVeCat+CkEF+mnurOPXnYWQW3Lb1Jqa8qBcufcVFVdmlb+FoI73p5V1WW57s+Vyjk3iiB5vb4n64cJ5MUEzbF7qeqjzrm/AvNpvznWGGP6vdJhZVx79eEAPPLKRu7+1Vu8NGkaiUiESDKB4HlwpqOqtJQRtTVM2bSeZSPHsOCUszjmnTepLi5hVNV2ihJNbC4bTCIa5aFP7MeuVVtJIuyyoyp40Nr0fVgyaheOfOfN3B7wANXfaqKAid775rylOT9opJt5UbfHiVLVLc656wiaxRYB5wIPOudWADcCdcAxBBf7a1R1RcrqkbQ74JqD7nUpHcoXAr9MW/w8QYLV6u4/VV3inDuU7jVxRds5pgZV7ZVR7cPjmEVQ1biBoDN4JusdTnDnwcMEzYAVwPkEQyK8Hhb7T+DPzrkbCe6oXEXQue5UINlXncuNMaY3HHvgKI49cB5zf/AGFTtqWDxuAquHDqc+7JTuPljGhX9axNWHH8tDM/fn/n0OZPy2zczevo3lo8aycVA5gxJxPhxaydrKEQCUNjZyxqsvMnbHNnbbvoUhDXW5PMQBrN9lUW+LyGe990+kzDuCTFtcQz0dbPNnBHffna6qtzvn5hGMNfQ9IEow5sJ5qnpb2npzCZKsVPPDn5Odc9Vpyxap6nx65gRgCMHYSq2awJxzPwUucc79KH0l1fBe4sydGb5SHUjQzrozVoa1ZwmCYQkeAa5T1ap2yqW+X6+qU4AtBJ36fwAMI0ie3iDoL/YhgKq+6pw7kKCZ72WCWroNBLWHV+9k/MYYkxPPXrkvcy5/l2g8HtyJmEwS857TX/srY6ur+NnDdzNr1XJ+8PkvsWHwEMrjcT41uZQHN3rWDRkWDB8B4D3jtm9t2e4u1VWsqBiWo6Ma2PrLA4hTfBd4RET+CAwSkZsJ+kId352NiPe9UmFiTE/ZB9AYkxVf/MUyVq2rp7akhAjw1nU/bFn2YcVQ5nzrRwypq+Xzi1/jyvtPYdn7m/js72uIxz6uXxhSV8cR7y9mVE01eM+fJ0zl8V9k1m9sgMpKtvPdLz/U5rv+2v89PqeZlYiMA04DdgdWA3d579d0vlZr3RriwBhjjBko/u9bUxhWvZXyxkbqi4pZPObj+2jeGT2WPTd8yBmvPM/SypEATJoyglHVVS1DKEgySUm8iaXDR7O2YiirKiq5ZrY9La0nvEibVy6ISMtdDt77dd77H3vvv+m9v7q7CRQMgGfnOeduIsgU27OXqq7qy3h2lnOuhLZ32DV7SlVP7Mt4jDEmn03YsZWJqz/gjgMP4asnn8XJb7xEbVEx9+57EI2RKB/tN5uFLrigRyLCyKptCJ6a4lJKEgmmrV/H9HVr2DSohO+fN4vxn5rcxR5NP7eEoKsPACLygPf+X3q6MWvOM7lmH0BjTNZcf9UfmPDQ+9zyqUN5J3x4cSSZ5OCVS9hYVsHCeYM58Li9Wq2jL67i9//3PjM/UcmpX92PWGENVZCVg/3OaQ+3+a7/6V1f6PMTKyI7vPcVKe+3eO+H93R7/b4myhhjjOmpKTNj/G71fuy5/kPcB8vYMGQoG8uHsGZIJRcsmMKBM9MfRwpuzgTcnAk5iDZ/9aOO5b36j7slUcYYY/LaKZ+r57jjTsh1GAWtHzU5xERkLh/XuKW/x3v/p4w31svBGWOMMca00o9qojYQPEe32ea0957g2bkZsSTKGGOMMQXBez+xN7dnQxwYY4wpOA+8vI7pFy3hkxd2a4Bq00P9ZYiD3mZJlDHGmIKy6LX1/Pfj9cSLS9hRPpQ9LlvR9Upmp3ikzSsfWBJljDGmoHz3D1uJeg+RCIKQKCpi90vSn0dvepOXtq98YH2ijDHGFJRihKZolKJkAg80RSLEonY5zKZ8qXlKZ58aY4wxBaUpGm1phhEg6j1N0o9uws9D+Xp2LYkyxhhTUOIIRXw8MFASISnRXIaU9/KlI3k6S6KMMcYUFEk2AalJk2956LDJjnw9u9ax3BhjTF742jlPcc6/P9ZluWg80aqHjoAlUVmWr0McWE2UMcaYASseTxCLRdn/B/9k2257IsDxZz9LFKiYNIovfqLtOmUSoUaEqPd4IC4RiFqdguk+S6KMMcYMODMvXUxZnScRiTJl/RqqJk9HIhGSwJuTpiHeU9zYxN/+NolrZwfjQG3eXMMBP1+LlA0iGY0R8R4vgvgk0aam3B6QGZDEF2gVpnPuOWA20AQkgBXAQlW9L6XMbODisFwMeBf4uarekVLmEuDTqnpEO/tYQPBMntq0RTeo6gWdxFad8rYk/NnQPENVB4fxPw38GUitvy4DGoF4+P55Vf2cc84Dh6jqC+3sbyWwS8o6zcar6vaO4uwlhfkBNMZk7Jy71vDC0nqKGxsYUVdDbSTC5qHD8ZEIEl7DmqIxvAgJEaI+GYwDBcSamhhaV0txMsGoqm3M3PAhb43djdcmhI9H855BDfUkRJj3iSF8//Ch7F5ZlKtD7Q+y0s521r892ea7/pZbjxrwbXqFXhN1uape4ZyLAecCdzvnXlfVpc65o4CHgauALwN1wLHAzc65yap6cYb7WK6qU7sTlKoObp52zv0aiKnqgg7KPg+kll8KXKGqt3dnn8CZqnpXN9cxxpis+uxPV7K2KujD1FRSypZolE1lgylOJoCgY68AEe+Jh/1sIslksLII8ViMupISmhIJqsaM472xu9EQjTKsthoBqotLaSguprKmhseWNvLkso28cM5oxg0p9Mtj78rXcaKsERhQ1ThwC0FSuW84+wbgHlW9VFU3q2qtqv4e+A5woXNuYm6iNcaYwrGuKqwgDxOk2qLilulUEZ8kmohTEm8i5j0RH9xxF/GeCNAUK6KhqJh4NEp5UyMx74l6T0VDPclIlPLGeqLJBAkPD71T14dHWBjydcRyS6IA51wx8PXw7RLn3DRgKtBezczdBP/4HNlH4eW1HTt22LRN27RNdzhdHJVWdRjNtUyJMJFKAt57kiKISKtBNCPJJIOaGgGIRyIt8yWlG4vgkUSCWDJJQoIyM4d/3LOhv5yHvprOlnx9dl6h94k6iKCvUQVB36hvquqtzrk5wAvADFV9t5111xP0jVqYQZ+oW4H0T+g3VPXuDONstzmvuU+Uql6RNr/d5rwM+kSNIjgHzVap6j6ZxLiTCvMDaIzJyNbaOHN/8gGJRJJYMsH4qm18OKicTUMrifjwUiyCeI94T8wnW9aNJuIUJZPgPfVFxUDzCOVJiuNBolRbVISXCEVNjfjBZVx4ZCWn7lve9wfaf2Qlu/nqWU+3+a7/zS1HDPhMqtAbfReGfaIqCZKdueHPjeHy8QSdyVuEtVYjU8p0ZUV3+0TlyNnWJ8oY099UlsV446IpbeZPPu91IkMraYoIEKE4rI1KIMFdd0BDrJhhO7Zz4Yz3WPj2FErq6qitGEZT6SA7YlggAAAgAElEQVRqY82dx6WlZuqf5+/aZ8dVaPKl5imdNecBqroVOBM4xjl3PPA+sBw4tZ3ipxDUnjzVdxEaY4xJtfwn+7H3CChpauLo8Q3ECZrvkiI0RiKU19YxftN6LpuzkpLhJbx+zX689PODefOKGUETYCRCXXEJdcUlNMRi1EXscphN+donqtBrolqo6hbn3HXAlcAigrv1HnTOrQBuJLg77xjgeuAaVV2RsnrEOVeatsnGPgi7u4rT4kyqan+M0xhjuvSHb0xsmb7unre5960kCeD7J43iJDcBgEWLFrVaR0QAT1Mk1tJBPRmJEonYs/OyK0+ypjSWRLX2M4K7705X1dudc/OAi4DvETxoaQlwnqrelrbeXIIkK9X88OfktHGfABap6nz63jNp798DpofTv3bO3ZS2fLaq/iP7YRljzM75r/l78V8Zfqsm4kl8q0fn+Y+HRTBZkS81T+kKtmO56TfsA2iMyZrmmqjjjjuuZd6Ui5aQLC7BR6MgQiSZpLShjrcvnZarMPuTrKQ7Xzn72Tbf9b+9ee6AT62sJsoYY0xBifgkiBBNqX2qj9rlMJvy5YHD6exTkyNh09lpHSzeS1VX9WU8xhhTKCLRWDDOVNgS46HdATyN6YolUTmiqucA5+Q6DmOMKTSRRBzvi4lHBPGQFPB1DV2vaHosX/tt2D2dxhhjCsq4WBNFyThFTXEkmaC0vo6VV0zvekXTY16kzSsfWBJljDGmoDxz8d6UVVdRnGhicEM9t/3r8FyHlPd8O698YM15xhhjCs4b1+zbdSHTa/Kl5imdJVHGGGOMyap8qXlKZ0mUMcYYY7LLaqKMMcaYgcd72POypZQkksTBBtXMgXytibKO5cYYY/LaBa/sSTQJTbEiIhJhv+/b06z6Wr7enWc1UcYYY/KaRCMQjSJAIhpFYsW5DqngeHsAsTHGGDPweBEaojGSkUjwqJeYPWy4r+XrA4itOc8YY0xei0ciJKJRvAjxaDTX4RQkj7R55QOriTLGGJPXEpHWiVOjJVJ9zmqijDHGmAHJtzxsOPiZp1d00+esJsoYY0zeuuCVaRQVJSlKNhHBkwSsR1Tfy5fmu3SWRBljjMk7n/3ucwyqgejocTSJUBTWREWBWLwpt8GZvGFJlDHGmLxywHeUpopxMBhi8ThRX9Rq+ZiaHTmKrHDly7hQ6XKaRDnnngNmA01AAlgBLFTV+1LKzAYuDsvFgHeBn6vqHSllLgE+rapHtLOPBcBtQG3aohtU9YKdja8XY/yhqk7NdFn6/HZiXQ5coar3d3GM44EbgX2BCcBXVPWuztYxxpie8t7zzHNreODhZdTUJFovS5lOANXFJdQVl1JdXELUe2atXsb7I8ewbkgllXU1DKutIR6LsfvWTeyz9gO2lQziDvdpYuVl7Lp9C1sHlbN9UBmxeJxk7OPL3YwP17B8rzuYvPh6iETg9eVw29Ow20j49heg2OoXepuNWJ49l6vqYGAEcDtwt3OuOTE4CngW+BswGRgFXANc75y7tBv7WK6qg9NenSZQmcTXyzH2htRY7wHudc519XyDJPAkcCqwJsvxGWMK3LN/Wctv71nSJoGCoLt38ysGbBgyjHXDhlNVVs7W8sE8sec+LB+5C/XFJWwrG8yGoZVsKa/g9V0n8diM/fj1wfOoLh/MtvIKtpRXMKZmBxO2byEZSbnUec+qylGsjA6BQ38Am3fAF66AO/4EV/weLrmnj85EYcnXEcv7QxIFgKrGgVsI/nb2DWffANyjqpeq6mZVrVXV3wPfAS50zk3McXz9Ksa0WG8kaP7fu4uyH6rqDar6IsE/f31qx44dNm3TNl1A02vWVpOphljrZrjUh9jGpfXl66OKoTQUfVy+trgYRCiNx4klUr7aRGiKRRlTW01y5XpYswm2pzRU/HNVvzlXuZjOlnwdJ6rfJFHOuWLg6+HbJWENylSgvaaluwn+WTmyj8JrE184r1/F2CyM9ZsETXtv9vX+u6OiosKmbdqmC2h6lhtDpobWt+6FEU2GyZD3lCTirZqIJm3ZyPCUvk7D6upayhYl4q22s72klMUjxhD50qdh+q6w3+RggQicfEi/OVe5mM4WL21f+aA/NPxe6Jw7D6gguOifqapvOefmhMvXpq+gqo3OuU3A6Az3Mck5ty1t3jdU9e6exhcuG9WLMfaG5lgbgaXASaq6tA/3b4wxnZo+rZJrLp/N839dy7N/XktNbesK8CRB/5l6iVBeW8uYrVvwCFWDBnH0e/9gS1kFf5swiZiHiZvXU1lXy9DGevZZ+wHvV47knv0PZlhTA4PiCZKRCD6ZpK50UKt9bC8bzMnfmAUL5gUzHr0I/rwYxo2AfSf1zYkoMPlS85SuPyRRC1X1CudcJXArMDf8uTFcPp6go3aLsKZlZEqZrqxor9P2TsZHL8fYkSagqJ35ReGyNrHu5P6MMSardhlTxpdO3IMvnbhHN9ec3enSo4FvAbPO/BNjBpVQ0VjP26PGtikXS8Q/TqAAykvh866bsZjuyJeap3T9pjlPVbcCZwLHOOeOB94nuMPs1HaKn0Lwz8pTOYwP+ibGlcBY51xZ2vyp4b6NMcakePnXh7PoF3M48/TdaSwqJsHHNVxJYOz2rbkNsADla5+o/lAT1UJVtzjnrgOuBBYB5wIPOudWEHSUrgOOAa4HrlHVFSmrR5xzpWmbbMxWfM65RaqadM71VozSzrIm4GWCZO1659wFQBUwhyChO6c3jitlvwIUhe/jYQd1Y4wZkA6fMwUefw+Ki1tqDATYNqg8l2EVpvzImdroNzVRKX4GjAVOV9XHgHnAZwhqZDYBFwLnqeqFaevNJUhgUl//Gi6b7JyrTnv19D7WlvgAejPGdpadr6pNBEnZUGAxsAX4BfDd9PGqdkLz/iYQjKlVB/ywl7ZtjDE5c82sJQxqat3zobakJEfRFK58rYkS7/N1CCwzQNgH0BiTNYsWLeKyv06iuqwcIWjOizU2sPjy6bkOrb/KSnbz2e++2ua7/olrDxjwmVS/as4zxhhjetuOsvKWZpcI0FRsNVF9LV87lhd0EuWcuwk4rYPFe6nqqr6MJxucc4cAj3Ww+EpVvbIv4zHGmL5m1d25ly/Nd+kKOolS1XPopc7Z/ZWqPg8MznUcxhiTK5JIQDT68XufzGE0hSlfHvOSrj92LDfGGGN6jU8maO7/6z34xvQh9ky2+XZe+cCSKGOMMXntJwcugfp6SCSQuhreu8I6lfe1fH0AcUE35xljjCkM71y5V65DKGj5UvOUzpIoY4wxxmRXntQ8pbMkyhhjjDFZZTVRxhhjzADkmxJccuLdbC8rpzjeyKcOGcWJ5x6W67AKSr70gUpnHcuNMcbktcW3bWVb+WB8JEJDcSkvPb8x1yEVnHx97IvVRBljjMlr1aVlrfrkxKNFOYymMOXriOVWE2WMMSavTdi2ORggKjR6x/YcRlOY8rUmypIoY4wxecs3JDhiyT8Zu30b4j0jqndw/D9ezXVYJk9Yc54xxpi8VXWF8tDMA9h1+xYOW/YOa4ZVYg996Xv52pxnSZQxxpi89Mb4r3MInnVxz0GrltMUiTKqZgdLho/Cht7sW/nSfJfOkihjjDF5558TvsEon6CsqZHdVi1HgOJkAi8Rlo4em+vwCo7VRBljjDH91N7nv0XViBEAlNbU8DARhjbUk0wb5TEpwooRo0kkkkSj1i247+RnFmVJlDHGmAHrV1+5hdMfeoY/lZZR7JM8POOTPDp9H14ZP4EnhlYytK6WOSveJxmNUB8r5vEZ+5AUYdHnr+WEJ87PdfgFI19HLBfv8+fQnHPPAbOBJiABrAAWqup9KWVmAxeH5WLAu8DPVfWOlDKXAJ9W1SPa2ccC4DagNm3RDap6QTfia/Y7VT0zpcwkYBnwjKoembb+FcCFwC9U9T9S5pcB64ChwG6qusY5dyZwnqpOD8vcBVSr6jlp2zwLuBTYW1U3p8y/FvgMcLCqpsbb2/LnA2iMySr/wtvw7Vthew0vFA1nUtUW1gyt5FOrVwDw0m6TuGreF5i0dRNJoLyhnrqS0pb1i+JxGqNRPFBUX48vKQEgFo/zL2ftz7MPLWHoiFLmf/NAKkeV5eAI+4WsVBkd9MN32nzX//2KGQO+eiof6zIvV9XBwAjgduBu59xUAOfcUcCzwN+AycAo4Brgeufcpd3Yx3JVHZz26jSBSo8v5XVm2vKzgK3APOfclHbWfw841TlXmjLvXwmSqG5T1VuAl4Gbm+c55+YC/w58OcsJlDHGZO7ffglL1tGwcQez167kld0mM2v1ypbF0zatZ9LWTUBwcdtcXtGy7MR/KD/+4++5/Ik/UB8rpiIRfLUJEI9GeeDWN9i6qZaV721h0V3/6MODKgw2TtQAo6px4BaC2qZ9w9k3APeo6qWqullVa1X198B3gAudcxNzE23AOVcEfBW4nCBZOqudYiuB14Avpsw7i+BYe+pM4GDn3BnOuaHAHcB3VXXJTmwzIzt27LBpm7Zpm85suq4RCJ7DFvFJ1lcM5cMhQ1vKbCobTGMk2vI+4oPBDMZUbeczy4Ovs4qGek5+82XqYh2PWl5X09A/jjcH09nipe0rH+RtEuWcKwa+Hr5d4pybBkwF7mqn+N0E/5Ac2c6yvvQFYDhBjLcBXw0Tq3S3ECZYzrm9CGrVFvV0p6q6CVgA/IzgXLymqr/q6fa6o6KiwqZt2qZtOrPpH58BxTGK8CwePY7jF7/K9z/3RR6fNpNnps5gR1Ep74wey8aywSSBb/3lKS545hHOeOUvrfoNNEUjJOMJhKA/gfgks4+YCED5kGI+P3/v/nG8OZjOlnyticrHjuUXOufOAyoI+h6dqapvOefmhMvXpq+gqo3OuU3A6Az3Mck5ty1t3jdU9e5uxNfsaFV9KZw+G1ikqpucc3cCVwInAPelbeMh4JfOuT0Jmt1uB+IZxt4uVX3SOXcPcDIwbWe2ZYwx2SCnHII/aTZRYO9EktdvfJrpj7zH0soRTN24gYs+ewJxhKriYmas20JdcTG7bPqoZf1tpYNYNWwEL+4+lUM+ep9P3Ho2tVtq2PvgSQAc+5V9iMYiRCL5cYHvT/Kl5ildPiZRC1X1CudcJXArMDf82fzY7vEEnclbhLVWI1PKdGWFqk7dmfjSZzrnJgNHAMcCqOp659wjBIlVqyQqTPruBL4FnAIc1MNY0r0JHBLWTBljTL8jReFlqwj2/6/Ps/9/BW8P//bLLNtlPADbq0u55a+3MCLe2GrdP874JDphMu6DZcx75NuUV7augSkqjmKyI19qntLlYxIFgKpuDe9QW+acOx54GFgOnAo8k1b8FIJa3af6NspWziJoUvyNc6655rkcKHfOTVXVpWnlbwHeAZ5V1WW57s9ljDG59KfrZ7V6v+mGJNtKyxhZVwPAkpFjeH3cBCprqtlSVNQmgTJZlp85VP4mUQCqusU5dx1Bs9gi4FzgQefcCuBGoA44BrgeuEZVV6SsHkm7Aw6gkSxI6VC+EPhl2uLnCRKsVnf/qeoS59yhtNM82YloO8fUoKo2zIAxJq+M3Hon64edylOTp7HbjipuPvhwALbGBuO9PT3P9I68TqJCPyO4++50Vb3dOTcPuAj4HhAFlhCMp3Rb2npzCZKsVPPDn5Odc9Vpyxap6nx65gRgCPDT1LGaAJxzPwUucc79KH0lVX2hm/s5M3ylOhDQbm7HGGP6vTHb7mb1rKuQYa2b6UrjO9WF1PRAvjbn5dVgm2ZAsg+gMSZr7rv5Xt57NE59cRHVpYPAe2auW8WZfzo316H1V1nJdva95P023/VvXLLHgM+s8naIA2OMMaZ0XBnFiaYggQIQoaq0YEcjzxkb4sB0yTl3E3BaB4v3UtVVfRmPMcYYGFZbzYfDKvESQZJJJm7ZkOuQCo6X/Eia0lkS1YvC59Kd02VBY4wxfWbP9euYtnkDHwwfxeRN6yFpHcv7Wr7227DmPGOMMXltw3GTSSDss24VcYmwePfxuQ6p4HiRNq98YDVRxhhj8lrpnDHMu/pMGjZXUTJiCHNzHVAByteaKEuijDHGFISSEUNyHULBypeap3TWnGeMMcYY0wNWE2WMMcaYrMrXmihLoowxxhiTVdYnyhhjjDGmB/K1Jsr6RBljjDHG9IDVRBljjDEmq/LlMS/pLIkyxhhjTFb5/MyhLIkyxhhjTHZZTZQxxhhjTA9YTZQxxhhjTI/kZxZld+cZY4wxJqu8tH21R0RWisjMvo2u56wmyhhjjDFZla99oqwmyhhjjDFZlWlNVHtE5EAR+ZuIvBX+PDCcf5WInB9O/6uIJEVkdPj+URE5KhvHksqSKGOMMcZklUfavDIhIsXA/cAPvff7AD8C7g/nPwPMC4vOA14CDheRIuAg4IXePo501pxnckpEngBG5jqOzsRisZHxeHxTruPoqYEePwz8Y7D4c8vi75bHvfdH9/ZGP7hgXE/b8/YEGr33zwB4758WkcZw/ovAvWFCNQc4D/gisBZY7L2v3fnIO2dJlMmpbPyx9jbnnKqqy3UcPTXQ44eBfwwWf25Z/PnJe18nIm8B84EPgWeBa4E1BLVUWWfNecYYY4zpr94DikVkLoCIHA4UhfMhSJYuBZ7x3jcQJFAL6KMkymqijDHGGNOfPC0i8ZT3JwI/F5FyoAb4ove+MVz2DHA5HydNzwAHAy/3RaCWRBnTtV/lOoCdNNDjh4F/DBZ/bln8A4T3fmIHi2Z3UP5vpIzk6b3/MfDj3o+sfeK976t9GWOMMcbkDesTZYwxxhjTA9acZ0wa59wNBGOONADVwH+qqrZTbgFwPbAynLVCVU/sozA7lGn8YdkfEXTCBLhdVS/vkyC74Jw7DfgesBfwbVX9ZQflDgMeBZaEsxpU9aA+CbITmcYflj0LuICgSeIx4D9UNdkngXYcUxnwG+AAIA6cp6qPtFPuMPrJ+XfOTQPuAEYAm4HTVfX9tDJR4OfA0YAHrlbVX/d1rO3JMP5LgG8A68JZL6rqN/syTtOaJVHGtPUYwYWvyTl3LHAvMKWDsk+r6hf7LrSMZBS/c+4zwJeA5udU/d0592dV/UvfhdqhN4BTgP/OoOzb/fD274zid85NAi4G9iO4cD4GnAbcme0Au3AeUKWqU51zewDPO+emqmp1O2X7y/m/CbhBVe8Kk9ibgcPTynwZmArsQZCsvO6ce1pVV/ZppO3LJH6AO1X1vL4NzXTEmvOMSaOqj6hqU/j2b8CuzrkB87fSjfhPJvhCrlPVOoIL98l9FWdnVHWxqr4N5LRGpqe6Ef8XgQdVdWNY+3QL/eN3cDLBRZywNkSBz+U0ok4450YD+wP3hLPuAfZ3zo1KK3oycIuqJlV1I/AgwT8SOdWN+E0/M2AuDMbkyLnAHztpXjnUOfeGc+4vzrlj+jKwDHUW/wTgg5T3q4Dd+iSq3jXNOfeac+7vzrkzch1MN/XX30F34uoP5383YK2qJgDCn+toG3N/Pd+Zxg9winPuLefck865du9YM33HmvNMwXHOvUbwZdqeMc1fZM65U4BTgc90UPYR4F5VrXPO7Qc85pybq6rv9HrQKXox/pzJ9Bgy8Bqwm6puD5vGnnbOrVXVp3sl0A70Yvw50VX83dhUTs5/AbsJWBg21R8JPOScm6Gqm3MdWKGyJMoUHFXdv6syzrkTgYXAPFVd38F2NqVMv+6cexGYBWQ1ieqt+An+C9895f0EYPXOR9i1TI4hw+1UpUyvcM49SPAMraxexHsrfnL0O+gqfudcc1wbU+J6tp3t5OT8t2M1MN45F1XVRNiBfBxtz2Xzcb0Svk+vmcqVjOJX1Y9Spp9yzq0m6NP45z6N1rSw5jxj0oSdsa8DPttZh1Pn3PiU6d2BTwFvZT3ALmQaP3AfcLpzbpBzbhBwOvD7Pgix1zjnxjrnJJweDhxF0Kl7oLgfOME5Nyrst3YW/eN3cB9wNkDYsfxA4PH0Qv3l/KvqhnC/88NZ84HXw35Pqe4DznLORcL+RicA/9d3kbYv0/jTvnP2BSby8eNPTA7YYJvGpHHObQQa+fi/cAhqdDY7534NPKyqDzvnrgSOJ7gFHOA6Vb2jj8NtI9P4w7KXECRPEHQyv6QvY+2Ic24+8D9AJcGx1ABHqerbzrnLgHWqepNz7lzg60ATQc36Har6P7mKu1mm8YdlzyYYDgHgSeDcXDcHOufKgdsJ7hpMAN9T1YfCZf3y/DvnphMMEVAJbCUYIuA959yjwEWqqmENzy8Jkj2Aa1S1X4wGnmH8dxAMO5Eg+FxdrKqP5ixoY0mUMcYYY0xPWHOeMcYYY0wPWBJljDHGGNMDlkQZY4wxxvSAJVHGGGOMMT1gSZQxxhhjTA9YEmWM6TYRmSgiXkR2zfJ+zhGR36a8f0xEvtfZOiY7RGSpiCzIsGyffD76goiUhMc+PdexmP7HkihjskhEJovIfSLykYhUi8hqEfmDiBSHyxeIyNJ21uto/pfDi9PF7Sx7TkQawv1sF5HXReSk7BxZ9olIOXAZcEnzPO/957z3P85ZUF0IfzefznUchSAb51pEDhOReOo8730DwZhfOR9/zPQ/lkQZk12PAh8CewIVwGzgCUB6uL2zgS3Av4lItJ3ll3vvBwMjCJ4Ef6+ITOvhvnLtNOAf3vtluQ7EFLx7gMNFZGquAzH9iyVRxmSJiIwgSJ5u8t5v94E13vubwv9uu7u9GcAhwBnAWOBzHZX13seBG4EosHc72/qmiLyRNm+SiCREZGL4/jdhzdkOEXlbRE7tJLZLROTptHnPicgPU97PFJEnRGSjiKwSkatEpKiTQz4BeKqjbaY0GZ0RxlcjIo+KSKWIXC0iG8IawG+mrL8gbJq5QEQ+DMtcmxpHV8ctIvuIyOPhcWxpPm4ReTMs8mRYG/jrDs5VmYj8LNzHJhF5UEQmpCx/Lozp/jCGZSJyfEcnKeWYviMia8J1fiIiI8JtVInIu6m1NiISE5GLRGS5iGwVkWdEZGbK8iIRuS7lHF7Qzn4PEZEXwnOwTES+KyIZ/3MgIieJyJthrembInJi+jGllb+9+Zx2dK5FZGV4XC+E81VEDmxvGynzVorIaSIyDngMiIbrVovIGQDe+yqC5+19IdPjM4XBkihjssR7vxn4J/BrETldRPbqzkWmHf8OvOW9f4SghuvsjgpK0Fz4TYLHcbzZTpG7gekism/KvAXAc977leH7F4B9gWEEzWq3i8hePQlcREYTPCT1AWA8QY3ckcD3O1ltf+DtDDZ/EvBpgofJTgT+DiwjeIDrV4HrU5MUggfQTgAmh3EcB5yfsrzD4xaRseFx/Dnc1y7A1QDe+0+G6x/lvR/svT+zg3h/SvCcxU+FsWwCFknrmsUzgGuBoQSPKblDRMo6OQe7h/FODs/FtwgSguZHzzwA/Cal/PkEj/v5fHgMzwNPiciQcPl/A8cCBwOTwmNteVByeD4eDbc/CjgGOBf4SicxthCRg4H/DfczAvgBcI+IHJTJ+l2c63OA/wSGEzwX79GU4+psm+sI/jFJhNsc7L1PfYzTPwg+k8a0sCTKmOw6DHgO+DbBA0bXi8iP0pKpSSKyLfVFUIvUQkRKCS56zRfCW4HPSduOuxeG668heK7fSd77Nn2rvPdbgYcIkgzCeM4Abkspc6v3frP3PuG9/x3Bw5UP68E5IIz9Te/9zd77Ru/9WuAqPn5uX3sqgaoMtn25935LmLQ+AjR572/x3se9948RPIdsv5TySeB8731d2FT4Y4IEEujyuL8CLPXeX+W9rwmPpVUNXGdEJEJwnn/ovV/rva8h+GzMAGalFL3Xe/9X730S+BVBMrVHJ5uuAy4N43mTIHF+xXv/kvc+AdwFTBWRoWH5rwLXeO/fDWtFLyN4Htsx4fLTw+VLvfd1wHlA6jPCvgHc571/KDxP7xIke539PlMtAO733j8W/p7+CPwB+FqG63fmVu/9q977RuAagnNzbC9st4ogMTOmhSVRxmSR936T9/4H3vv9CWoKvgdcRJi8hFZ474elvgguUqm+BAwmuBhCUAuwEUiv7VgYbmO09/5g7/2iTsL7DXBq2JR1eBjfAxBc7EXkMhF5L2xu2QZ8kqDWoScmAXPSEsXbCGpBOrIV6LIGgaDPWbPatPfN8ypS3m/w3temvF8J7AoZHfdEYEkGMXVkFFACrGie4b2vBjYAu6WU+zBleU04mXoM6TaECVez9PPQfLzN29gtLYYkwXlojmHX8H1qDBtStjcJmJ/2+7yYoJk5E632H1pG63PQUyubJ3zwcNhVhL/fnTSEoD+iMS0siTKmj3jva733txPUbOzbRfF0/07Qv2mxiHxEUNNUSccdzDPxFNBA0Jy1APhdWOsAMJ8gQTsJqAwTuzfpuEP8DqA8bd64lOkPgKfTksWhYSf4jrwO9Kj5sAuj05rGJhKcT+j6uFfSeY1QV09030hwzic2zxCRwcBoYHVG0feO1WkxRML3zTGsTVteTusE+gPgtrTf5xDv/Sd6sv/Q5JT9d/V5go7PdWrcQtB02/z7bbVdEYkRnPtmqYloupkEn0ljWlgSZUyWSNDB+SoJOlQXhZ15TyL4Mn6+G9vZi6Cfy4kEyVfzaxZBTc7nexJf2MxzJ/AfwL+Q0pRH8F93nOCiHxGRrxHUyHTkVWB/ETkgPM5zCWormt0JOBH5moiUhjU+k0Xk6E62+SBwRPePrEsR4BoRGSQikwmaqpr7vnR13HcBe0rQMb1MRIpFJDXGj+gkyQprfO4ELheRcWEydy3wLvByLx1fJm4Hvici08L+cxcCMeCP4fLfAueLyBQRGUTQ5Jl6vbgROEVEjkv5bO8lIodmuP87gJNE5LMiEhWRzxF8Bpubq98gSHaPDT8rJwKfSdtGR+f6ayKyf1jDej5QlnJcrwLzJLiJogRYCKTe3PARQcfy1M8uIlJB8Pf2cIbHZwqEJXxCWQ0AAAHCSURBVFHGZE8jwX+5DxA0A2wEfgj8h/f+vm5s52zgNe/9Iu/9Rymvt4D76KSDeQZ+AxxK0KSYehG/g6CD9lKCWom96CTx894/B1wHPE7QjDQGeDFl+UfAXII77lYSNNX9gaD2oSO/BT4ZJjq96QOCmokVBMf4OEGSAF0cd9j5+DCCTvFrCC66qZ3SLwQuk+COt5s72P93ACW422sVQRPYF8Kktq/8D8Ft+08C6wmac48K70KDoL/aE8BLBOdpFcF5A8B7v5ign9G3CX7fGwgSs4yae733LxL0DfsJwWfhx8Bp3vuXwuXLCDqH/4rgb+do4P60zXR0rn8F/Dzc7snAMd777eGy/yVIhF4jaD5cRfB7bo5rCfD/gJfDZsrmjvLzgWe99+9ncnymcEjQZGyMMf2PiJwDzPHeZ3TXVwbbW0DQqdvG+8lDIrKS4Pd7V1dlu7HNEmAxQaL7Tm9t1+SHWK4DMMaYjnjvbwJuynUcpnCFdy921g/OFDBrzjPGGGOM6QFrzjPGGGOM6QGriTLGGGOM6QFLoowxxhhjesCSKGOMMcaYHrAkyhhjjDGmByyJMsYYY4zpAUuijDHGGGN64P8D6FYogNJjdg4AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "shap.summary_plot(shap_values, X)" ] }, { "cell_type": "code", "execution_count": 85, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAFJCAYAAABHBFN/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XucVXW9//HXVxAv4S2FUrwAkRmpx/LjKfJKXjKRtNPFa4aGqeXpHIuin3iBBIwuiiamoQhqYHoqCzuaWpLmOSf7mKZZisiACZqIoBImF9fvj+93cLHYe2bPzJq9B+b9fDz2Y2av73et9Vnfvfaez3y/37V2yLIMERERESnPZo0OQERERGRTowRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGQ9Gx2AbLxmz56dDR8+vNFhiIiI1FOopZJ6sERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKFrIsa3QMspHaY9JinTwiItJlLRy9S2dsNtRSST1YIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiXr2egANmVmNgc4FDjB3W/NLf8g8H/AQnfvn5YNAi4EjgC2B14G/gxMdfefpjpjgQuAfwIZsAS4ERjn7q1e0WdmGXCwu//OzA4D7gP+AbwJrAaeBH4CTHH3Nzp29CIiIt2XerA631+BMwvLzkzLATCzfYCHiYnOYcC2wLuBK4BPFNad4+69U53PAV9PP9tjrbv3dvdtgX7AWOAMYI6Z9WrnNkVERLo99WB1vp8CZ5vZQHefb2bbAJ8EJgJfSnUmA39w99Nz660F7kqPDaQeqwfM7AnAgOkdCdLd/wncY2afAB4nJm1TO7JNERGR7ko9WJ3vn8CPgM+n5ycBvwWeBzCzrYjDiLPaslEz28zMhgJ7A0+VFay7P03sTTu8rG2KiIh0N+rBqo+pwN1mdjHwBeBiYIdU9nagB7CoubKZ7QfMSU+3BN7j7gvT80PNbDmwFdAL+AFwTcnxPgfsWPI2RUREug31YNWBu/8ZWEicxN6X9Yf9lhGHA3fN1X/U3bcn9k5twfq35f9tKtsGOJ84Z2urkkPeFVha8jZFRES6DSVY9fNDYoI1zd3XNi9095XA/cCJbdmYu69y90uJVxKOKyvIdDXj/sBvytqmiIhId6MhwvqZBfyNOL+p6CvECevTiJPfm4jDhgfWsN0LgHvNbHJuGLHNzGwL4CDgcuBPwIz2bktERKS7U4JVJ+kqvXurlD1qZkZMlu4n3gdrKfAE8TYNVRMnd3/AzB4g9mKNaGNYPcxsBfH2EGuIk+VvBq7UfbBERETaL2RZq/enFKloj0mLdfKIiEiXtXD0Lp2x2dB6Fc3BEhERESmdhgg3Iemmo3tUKFro7u+rdzwiIiLdlRKsTYiSKBERka5BCZa021WDH2b48OGNDkNERKTL0RwsERERkZIpwRIREREpmRIsERERkZIpwRIREREpmRIsERERkZIpwRIREREpmRIsERERkZLpuwil3fRdhCIina+Tvk9P2k/fRSgiIiLSCEqwREREREqmBEtERESkZEqwREREREqmBEtERESkZD0bHUBXYWZzgCHAamAt0ARMcPfbKpTnDXH3x82sD/At4GhgO+A14FHgDHd/Pm1jS2AMcBLQD3gVuBO40N3/lur0T/vezd2fy8W33nIzGwtcAPwTyIAlwI3AOHfPcuvtDFwEHAPsBCwF/g+Y5O4Pm9kIYBqwsnBcU9x9dBuaUERERBIlWOu7xN3Hm1lP4Fxgppk94u7z8uVV1r0ZeAV4v7u/aGZ9iclWBmBmPYBfAu8ETgEeBnYDvgf83swOcPdFbYx3jrsfYWYBOAj4FbAAmJ72uQvwEDHROwZ4EtgK+ATwbykGgPnuPqiN+xYREZEqlGBV4O5rzGwqcDmwHzCvlVUAPgyc4O4vpm28SOxRanYScDAwOJewNZnZCcATwDhgZDvjzYAHzOwJwEgJFvBN4B/AJ9y9uedtBXBTe/YjIiIitVGCVYGZ9QLOSU/n1rja/cB3zGxX4A/AY+6+Nld+DPD7XHIFgLuvNrMfA2d0IN7NgEOBvVk/qTsGmJZLrkRERKQOlGCtb4yZjQK2Ic61Gunuj1UoX8fdt0+/ngD8O3A6cAXwhplNB77h7v8E+gDVhgAXA33bEe+hZracOOzXC/gBcE2uvKV95g1I28n7orvPbEdMIiIi3Z4SrPVNSHOwdgCuB4amn+uVV1rR3VcAlwKXph6wo4lDca8SJ5kvIc65qmSXVA5vTaLfvFBn80I5wG/THKxewFeBzxKTreY6S4iT6VvTpDlYIiIi5dFtGipw92XE+VDDzOy4dqy/yt1/AdxLnMMFcBfwQTN7V75umlD/GeLVhADPA28AxYRnEHE+1YtV9ncpMaEalyv6b+BTZlZM1kRERKQTqQerCnd/2cwuAyaa2ezW6qe6s4DHgVXAIcQesEtTlZnA54HbzezzvHUV4XeIt3UYm/b7ppndBHzTzOYD84GBxMRpRv4WDBVcANxrZpPdfSFwMfB74L/M7BvE+WRbAh8H3ufuF9TaHiIiIlI79WC17ApgZ+C09PxCM1tReBybyjYDbiD2MC0Drga+S7wNA+6+hjhs+AvgFmJv1EPA68C/Nt8HK/lP4qT5u4lDjHcTe8PWm/9V5O4PAA+QerHSbR8OIPaKNW/rr8DxwE9yqw6scFyzam8mERERyQtZ1lKHiEh1e0xarJNHRKSTLRy9S6NDkPWFWiqpB0tERESkZEqwREREREqmBEtERESkZEqwREREREqm2zRIu101+GGGDx/e6DBERES6HPVgiYiIiJRMCZaIiIhIyZRgiYiIiJRMCZaIiIhIyZRgiYiIiJRMCZaIiIhIyZRgiYiIiJRMX/Ys7aYvexbpvvQFxNKN6cueRURERBpBCZaIiIhIyZRgiYiIiJRMCZaIiIhIyZRgiYiIiJRMCZaIiIhIyXrWYydmNgcYAqwG1gJNwAR3vy1XZwhwcarXE3gSuNLdZ+TqjAUOcvcjKuxjBDANWFkomuLuo2uMbxXwJrAUeBCY7O4PVzmOvCHu/riZTQdOAd5I23kuHcM1ZrYiV3+L9PON5gXu3jtt/1DgUHe/P7ffecB4d5+eWxaAp4B3Aru4+4pc2WHAfcBf3P19hWO9EzgaON3dp5tZf+LrsRLI33bhMXf/cMUGExERkRbVJcFKLnH38WbWEzgXmGlmj7j7PDM7CvgFcCkxQXkdOBa41swGuvvFNe5jvrsP6kh8AGa2B3Am8H9m9hl3/1mlelXMcPeRZrYZcALxOOe6e+/mCmZ2HdDT3UdUWH8p8F0z+6C7t3SfqaHAQGAFcBIwtVC+FtjczA509wfTfncHPggsrrC997j7cy3sT0RERGpU9yFCd19DTAZ6AvulxVOAWe4+zt2XuvtKd78VOA8Yk3pZ6hnjQne/ALgR+H7qLWrrNt5091nEhOn9bVh1KrArMWlqyVnAXcBN6fdKriMmis0+D8wiJrAiIiLSSeqeYJlZL+Cc9HSume0JDAJurlB9JvGOqUfWKbyiW4B+wHvauqKZ9TCzk4G3A96GVf8BXARMNLMtKlUwsz7A8cQh0WnA/ma2f4Wq04HjzWw7M+sBnMGGPV0iIiJSsnomWGPMbDmx92Q8MNLdHwP6pPJFxRXcfRXwEtC3xn0MMLPlhcfJHYi5echsx9yyMcV9FNb5bFr2IjAK+Ly7/7aN+72BOPT3H1XKTwdeAWa7+yPAI8AXipXc/UXgXuBU4GPAC+7+aJVtPlE4rqvbGLOIiIgk9ZyDNSHNwdoBuJ44h+h6YEkq70ec2L5O6u3aKVenNU0dmINVya7p59LcsgmtzMG6yd1HdmSn7r7WzL4GzDKz6/NlabjyTOBmd2+ebH898C0zG+XurxU2NxWYBCyk5d6r92kOloiISDkaMQdrGTASGGZmxwFPA/OBSj1NJxKvbLunfhGu5wRiz9pT9d6xu98J/IE4XJj3EeKQ6hlm9oKZvQCMA3pTuQ3vBrYjJrQzOy9iERERaVbPHqx13P1lM7sMmAjMJl5VeLuZNQFXE4cRhwGTgUnu3pRbfTMz27KwyVVlxmdmuxGTwBHACa1czdeZRgG/J3c7B+KE9vuJyV/eROIw4bX5he6emdkwYKv8rRxERESk8zQkwUquIF4leFq6H9PhxN6arwM9gLnAKHefVlhvKBteBdd8xd3Awv2mIM5Tau2KPIALzWw0scdsKfA/wIfd/aEK9b5RWHaiu99Rwz7axN3/ZGaziIkeZtaXOLn9k+7+Qr6umU0C/mpmVmE7f6lhd0+ZWT6RXO7uu1atLSIiIlWFLGtU54xs7PaYtFgnj0g3tXD0Lo0OQaRRarp1k74qR0RERKRkjRwirBszu4Z4q4JKBrv7s/WMR0RERDZtGiKUdps9e3Y2fPjwRochIiJSTxoiFBEREWkEJVgiIiIiJVOCJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJdONRqXd9F2EIo2n7wQUqTvdaFRERESkEZRgiYiIiJRMCZaIiIhIyZRgiYiIiJRMCZaIiIhIyZRgiYiIiJSsZyN3bmZzgCHAamAt0ARMcPfbcnWGABenej2BJ4Er3X1Grs5Y4CB3P6LCPkYA04CVhaIp7j66o/GVGOMF7j6o1rLi8gqxzgfGu/tPWjnGY4BRwL5AD+DPwPnu/kBL64mIiEh1XaEH6xJ37w3sCEwHZppZc9JwFHAf8L/AQKAPMAmYbGbj2rCP+e7eu/BoMbmqJb6SYyxDPtZZwI/NbM9W1tkB+D4wiBj7TOBOM9utUyMVERHZhHWFBAsAd18DTCX2AO2XFk8BZrn7OHdf6u4r3f1W4DxgjJn1b3B8XSrGQqxXE3uk9mml7o/c/Wfuvtzd17j7D4AVwAF1CFVERGST1GUSLDPrBZyTns5NPS+DgJsrVJ9JvJPqkXUKb4P40rIuFWOzFOuXiMOFf2rjuvsAOwGPd0JoIiIi3UJXSLDGmNly4HVgPDDS3R8jDlcBLCqu4O6rgJeAvjXuY4CZLS88Tu5gfJQcYxmaY30OOA74pLvPq3VlM+sL/AT4rrs/3UkxioiIbPIaOsk9meDu481sB+B6YGj6uSSV9yNOGl8n9dDslKvTmqZKE8g7GB8lx1jNamDzCss3T2UbxNqenZjZLsA9wN3A/2vPNkRERCTqCj1YALj7MmAkMMzMjgOeJl4JV6mn6UQgIyYEjYoP6hPjAmBnM9u6sHxQ2neHpXliDwB3uvu57q4vcRYREemArtCDtY67v2xmlwETgdnAucDtZtZEnLT9OjAMmAxMcvem3OqbmdmWhU2u6qz4zGy2u79pZmXFGCqUrQYeIiZyk81sNPAqcCAx2Tu7o8dkZnsB9wLT3f2Cjm5PREREulAPVs4VwM7Aae5+J3A4cAixJ+clYAwwyt3HFNYbSkxu8o/PpLKBZrai8JjV0fgAyoyxQtnX3H01MWHbjniPqpeJt1X4avF+XO00mjjE+Z+F9jmlhG2LiIh0SyHLNBok7bPHpMU6eUQabOHoXRodgkh3E2qp1BV7sEREREQ2al1qDla9mdk1wKlVige7+7P1jKczmNnBwJ1Viie6+8R6xiMiItIdaIhQ2m327NnZ8OHDGx2GiIhIPWmIUERERKQRlGCJiIiIlEwJloiIiEjJlGCJiIiIlEwJloiIiEjJlGCJiIiIlEwJloiIiEjJdB8saTd9VY5I/emrcUQaTvfBEhEREWkEJVgiIiIiJVOCJSIiIlIyJVgiIiIiJVOCJSIiIlIyJVgiIiIiJevZ6ADqyczmAEOA1cBaoAmY4O635eoMAS5O9XoCTwJXuvuMXJ2xwEHufkSFfYwApgErC0VT3H10DTEacAFwILAF8ALw38Akd38+1RkMfBMYCmyVjuN6YLK7v1khjjeBfwJ/Bn4E3FClXpvjFRERkQ11qwQrucTdx5tZT+BcYKaZPeLu88zsKOAXwKXAKcDrwLHAtWY20N0vrnEf8919UFsDM7MjgdnAFcCX3H2Rme0MjAQOBW4xs32BB4Gbgb2BpansBuBfgM9VisPMegNHpW0fA3yyo/GKiIhIZd0xwQLA3deY2VTgcmA/YB4wBZjl7uNyVW81s62B68zsBndf0IlhXQ3MzPccpV6rS3J1LouL/ZzcsnvM7FTgPjOb6u6/K27Y3VcAPzWzl4DfmtmR7n5P5xyGiIhI99Zt52CZWS+gOUmZa2Z7AoOIPUNFM4l3bj2yE+Np3v/MFupsBRxGhRjdfQ7wHPCxlvbj7vcDi4HD2x+tiIiItKQ79mCNMbNRwDbEuVgj3f0xMzswlS8qruDuq1LPT98a9zHAzJYXln3R3asmT0CfavvPeTvQo4U6i6ktxueAHXPP2xOviIiIVNEdE6wJaQ7WDsSJ4UPTzyWpvB9xYvs6qbdrp1yd1jS1Y05Tfv9/rVLnZeLk/H5VyncBfl3DvnYF7ss9b0+8IiIiUkW3HSJ092XEyePDzOw44GlgPnByheonAhnQaXOW3H0ucR7YSS3UeR24nwoxmtkhxMTpzpb2Y2YHExOx33QkXhEREamuO/ZgrePuL5vZZcBE4tV75wK3m1kTccL568AwYDLxNglNudU3M7MtC5tc1cGQvgjMNrO/A1e5+2IzewdwBrGX6Rbgq8ADZnYVMJ7Yq3Uw8SrCme7+QKUNm9nbiHPIrgB+7u53dzBWERERqaJbJ1jJFcB5wGnuPt3MDgcuAr5OnO80Fxjl7tMK6w0lJmB5zb1PA81sRaFstrtX7Z0CcPd7zOwg4n2wHk9Dky8AdxATKNz9ETP7EPE+WH8BtgQWAt8nXmGY1xxHBrxBvA/WROC6KvXaFK+IiIhUFrIsa3QMspHaY9JinTwidbZw9C6NDkGkuwu1VOq2c7BEREREOouGCOvIzK4BTq1SPNjdn61nPCIiItI5NEQo7aYhQpH60xChSMPVNESoHixpt6sGP8zw4cMbHYaIiEiXozlYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiXTjUal3XSjUZFy6SaiIhsFfRehiIiISCMowRIREREpmRIsERERkZIpwRIREREpmRIsERERkZIpwRIREREpWc/WKpjZHGAIsBpYCzQBE9z9tlydIcDFqV5P4EngSnefkaszFjjI3Y+osI8RwDRgZaFoiruPbkN8zW5x95G5OgOAZ4Bfu/uRhfXHA2OA77v7l3PLtwYWA9sBu7n7c2Y2Ehjl7nulOjcDK9z97MI2zwTGAfu4+9Lc8u8BhwAfdvd8vPl19wL+CvwDyIA1wDzgF8Dl7r6iQr28h9z9I6nOR4GLgMHEZHpxaptxuf29N9UZCvQG/g78BrjU3edXilFERERaVmsP1iXu3hvYEZgOzDSzQQBmdhRwH/C/wECgDzAJmGxm4ypvrqL57t678GgxuSrGl3uMLJSfCSwDDjezd1VY/yngZDPbMrfsM8SEpM3cfSrwEHBt8zIzGwp8ATilWnJV0N/dtwH6Al8BjgEeMrPtKtTLH3tzcvUe4GfA94mvyY7pmOblYvoA8AfgVeBDwDbAB4EngKPbfOAiIiIC1NCDlefua8xsKnA5sB/xj/UUYFa+VwS4NfUAXWdmN7j7grICbisz2xw4HbgEOIuYbH2jUG0B8CzwKeDmtOxMYCpwWTt3PRJ4zMw+B9wOzAC+6u5z27KRlIw9YGbHEXsGv0w8ltYY8JK735Jb9nh6NLsSuN/dz8otewmY3JYYRUREZH1tmoNlZr2Ac9LTuWa2JzCIt5KSvJnEu50eWaGsnj4OvJ0Y4zTg9JR0FU0lJlWY2WBib9zs9u7U3V8CRgBXENvij+7+ww5s70ViT+HhNa7yENDXzKaZ2cfNrF++MPWEfTjFJiIiIiWqNcEaY2bLgdeB8cBId3+MOPQEsKi4gruvIvaG9K1xHwPMbHnhcXJb4ss9PpQrOwuYnRKeG4nJ1vEVtvFzYK80tPYF4lDomhr3X5G73w3MIs4RKw5btsdzxKG+vGcKx35Z2vfTxGE/iD1SfzOzJ8xsWFq2EzEB3uC1ExERkY6pdYhwgruPN7MdgOuJE6KvB5ak8n7E4at1Um/XTrk6rWly90E11q0YX3GhmQ0EjgCOBXD3v5vZHcSk67Z8XXdfZWY3Av8OnEici1SGPwEHpwSvo3YFlhaWvavatt39UeAMADN7B/FChJ+mJHIpcRJ9v0rrioiISPu1aYjQ3ZcRe2KGpTlBTwPzgUo9TScS/4Df09EgO+BMYi/NDWb2gpm9QEy4PtI8Sb9gKnEI9FF3f6aOcbbKzPoQE9vftGd9d/87McHqBQx29+XA/wAnlRakiIiIAG2c5A7g7i+nYaiJxDlK5wK3m1kTcDVxGHEYcVhqkrs35VbfrHClHsCqdkXeitzk9gnAVYXiB4jJ13pXKbr7XDM7lLYNm/WocExvuHvWxpArSsfxr8C3gReJE9NrWe8jwHuJt3d4jniF4NeIt3V4JFX7D+C3ZnY18crPZ4EdiAnzm+5+dRnHICIi0t20OcFKrgDOA05z9+lmdjjxXkpfB3oAc4n3i5pWWG8oMQHLa+5BGWhmKwpls929vT0sxwPbEu8dtd6wmpldDow1swuLK7n779q4n5FsOL/qAMDbuJ2iBWaWEe89Ng+4A7jM3V+tUC///O/u/i7gZeIFBucD2xMTq0eBo939eQB3f9jMDiD2bD0EvI2YxN0DfKuD8YuIiHRbIctK6WiRbmiPSYt18oiUaOHoXRodgoi0LtRSSV+VIyIiIlKy9g4R1o2ZXQOcWqV4sLs/W894OsrMtmDDKwGb3ePun6hnPCIiIlI+DRFKu2mIUKRcGiIU2SjUNETY5XuwpOu6avDDDB8+vNFhiIiIdDmagyUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMt1oVNpNNxrtWnSTShGRutB3EYqIiIg0ghIsERERkZIpwRIREREpmRIsERERkZIpwRIREREpmRIsERERkZL1bHQAjWJmc4AhwGpgLdAETHD323J1hgAXp3o9gSeBK919Rq7OWOAgdz+iwj5GANOAlYWiKe4+uoXYVuSebpF+vtG8wN17p/jvBX4L3JmrvzWwCliTnj/g7h8zsww42N1/V2F/C4B35tZp1s/dX6kWp4iIiFTWbROs5BJ3H29mPYFzgZlm9oi7zzOzo4BfAJcCpwCvA8cC15rZQHe/uMZ9zHf3QW0Jyt17N/9uZtcBPd19RJW6DwD5+vOA8e4+vS37BEa6+81tXEdEREQq0BAh4O5rgKnEhHO/tHgKMMvdx7n7Undf6e63AucBY8ysf2OiFRERka5OCRZgZr2Ac9LTuWa2JzAIqNSjM5N4F9cj6xSeiIiIbGS6+xDhGDMbBWxDnIs10t0fM7MDU/mi4gruvsrMXgL61riPAWa2vLDsi+4+s91Rd45rzeyq3PNn3X3fhkUjIiKyEevuCdaENAdrB+B6YGj6uSSV9yNObF8n9XbtlKvTmqa2zsFqkLM0B0tERKQcGiIE3H0ZMBIYZmbHAU8D84GTK1Q/EciAe+oXoYiIiGxMunsP1jru/rKZXQZMBGYTryq83cyagKuJVxEOAyYDk9y9Kbf6Zma2ZWGTq+oQdlv1KsT5prt3xThFREQ2akqw1ncF8SrB09x9upkdDlwEfB3oAcwFRrn7tMJ6Q4kJWN5J6efAwn2tAGa7+0nU368Lz58C9kq/X2dm1xTKh7j7450floiIyKYlZFnW6BhkI7XHpMU6ebqQhaN3aXQIIiLdQailkuZgiYiIiJRMQ4QNkobjTq1SPNjdn61nPCIiIlIeDRFKu82ePTsbPnx4o8MQERGpJw0RioiIiDSCEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkulGo9JuG9t3Eeq7+kREpAS60aiIiIhIIyjBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkvVs5M7NbA4wBFgNrAWagAnufluuzhDg4lSvJ/AkcKW7z8jVGQsc5O5HVNjHCGAasLJQNMXdR3c0vhJjvMDdB9VaVlxeIdb5wHh3/0krx9gPuBrYD9gd+Ky739zSOiIiItKyrtCDdYm79wZ2BKYDM82sOWk4CrgP+F9gINAHmARMNrNxbdjHfHfvXXi0mFzVEl/JMZYhH+ss4Mdmtmcr67wJ3A2cDDzXyfGJiIh0C10hwQLA3dcAU4k9QPulxVOAWe4+zt2XuvtKd78VOA8YY2b9Gxxfl4qxEOvVQA9gn1bqPu/uU9z9QWLPl4iIiHRQl0mwzKwXcE56Ojf1vAwCKg1XzSTe6OvIOoW3QXxpWZeKsVmK9UvE4cI/1Xv/IiIi3V1XSLDGmNly4HVgPDDS3R8jDrUBLCqu4O6rgJeAvjXuY4CZLS88Tu5gfJQcYxmaY30OOA74pLvPq+P+RUREhAZPck8muPt4M9sBuB4Ymn4uSeX9iJPG10k9NDvl6rSmqdIE8g7GR8kxVrMa2LzC8s1T2QaxdnB/IiIi0kFdoQcLAHdfBowEhpnZccDTxCvhKvU0nQhkwD0NjA/qE+MCYGcz27qwfFDat4iIiHQxXaEHax13f9nMLgMmArOBc4HbzayJOGn7dWAYMBmY5O5NudU3M7MtC5tc1Vnxmdlsd3/TzMqKMVQoWw08REzkJpvZaOBV4EBisnd2GceV228ANk/P16TJ8iIiItJGXaYHK+cKYGfgNHe/EzgcOITYk/MSMAYY5e5jCusNJSY3+cdnUtlAM1tReMzqaHwAZcZYoexr7r6amLBtB/wZeBn4PvBZHhCfAAAXWElEQVTV4v24OqB5f7sT7xn2OnBBSdsWERHpdkKWZY2OQTZSe0xavFGdPAtH79LoEEREZOMXaqnUFXuwRERERDZqXWoOVr2Z2TXAqVWKB7v7s/WMpzOY2cHAnVWKJ7r7xHrGIyIi0h1oiFDaTUOEIiLSDdU0RNite7CkY64a/DDDhw9vdBgiIiJdjuZgiYiIiJRMCZaIiIhIyZRgiYiIiJRMCZaIiIhIyZRgiYiIiJRMCZaIiIhIyZRgiYiIiJRMNxqVduvqNxrVjUVFRKQT6LsIRURERBpBCZaIiIhIyZRgiYiIiJRMCZaIiIhIyZRgiYiIiJRMCZaIiIhIyXo2OoAymdkcYAiwGlgLNAET3P22XJ0hwMWpXk/gSeBKd5+RqzMWOMjdj6iwjxHANGBloWiKu49uQ3zNbnH3kbk6A4BngF+7+5GF9ccDY4Dvu/uXc8u3BhYD2wG7uftzZjYSGOXue6U6NwMr3P3swjbPBMYB+7j70tzy7wGHAB9293y8IiIi0opNsQfrEnfvDewITAdmmtkgADM7CrgP+F9gINAHmARMNrNxbdjHfHfvXXi0mFwV48s9RhbKzwSWAYeb2bsqrP8UcLKZbZlb9hligtVm7j4VeAi4tnmZmQ0FvgCcouRKRESk7TbFBAsAd18DTCX2Uu2XFk8BZrn7OHdf6u4r3f1W4DxgjJn1b0y0kZltDpwOXEJMpM6sUG0B8EfgU7llZxKPtb1GAh82s8+Z2XbADOCr7j63A9sUERHptjbZBMvMegHnpKdzzWxPYBBwc4XqM4l3Zj2yQlk9fRx4OzHGacDpKekqmkpKvsxsMLE3bnZ7d+ruLwEjgCuIbfFHd/9he7cnIiLS3W2KCdYYM1sOvA6MB0a6+2PE4UCARcUV3H0V8BLQt8Z9DDCz5YXHyW2JL/f4UK7sLGB2SnhuJCZbx1fYxs+BvczsPcShvOnAmhr3X5G73w3MIs4RKw5bioiISBtsUpPckwnuPt7MdgCuB4amn0tSeT/ixPZ1Um/XTrk6rWly90Edia+40MwGAkcAxwK4+9/N7A5i0nVbvq67rzKzG4F/B04EPtjOWIr+BBycEjwRERFpp02xBwsAd19G7IkZZmbHAU8D84FKPU0nAhlwT/0i3MCZxGHKG8zsBTN7gZhwfaR5kn7BVOIQ6KPu/kwd4xQREZFWbIo9WOu4+8tmdhkwkThH6VzgdjNrAq4mDiMOAyYDk9y9Kbf6ZoUr9QBWdUacucntE4CrCsUPEJOv9a5SdPe5ZnYoFYY8W9CjwjG94e5ZG0MWERGRFmzSCVZyBfEqwdPcfbqZHQ5cBHwd6AHMJd4valphvaHEBCzvpPRzoJmtKJTNdveTaJ/jgW2By/P3ogIws8uBsWZ2YXEld/9dG/czkg3nVx0AeBu3IyIiIi0IWabOC2mfPSYt7tInz8LRuzQ6BBER2fSEWiptsnOwRERERBqlOwwR1o2ZXQOcWqV4sLs/W894REREpDE0RCjtpiFCERHphmoaIlQPlrTbVYMfZvjw4Y0OQ0REpMvRHCwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkinBEhERESmZEiwRERGRkoUsyxodg2yktthiiz+vWrXqn42OY1PTs2fPndasWfNSo+PY1Khdy6c27Rxq185RYru+lGXZ0a3ur4QdSTe1zz77/NPdrdFxbGrMzNWu5VO7lk9t2jnUrp2j3u2qIUIRERGRkinBEhERESmZEizpiB82OoBNlNq1c6hdy6c27Rxq185R13bVJHcRERGRkqkHS0RERKRkuopQNmBmewIzgB2BpcBp7v50oU4P4ErgaCADvuXu17VW1p2V0K5jgS8Ci1P1B939S/WJvmuqsU2PAiYC+wDfd/dRuTKdqxWU0K5j0bm6gRrb9ULgRGAtsBo4391/lcq2Bm4A9gfWAKPc/Y76HUHXU0KbTgeOAJpv33Cbu08oIzb1YEkl1wBT3H1PYApwbYU6pwCDgHcDQ4CxZta/hrLurKPtCnCju++XHt3+Dxa1tel8YCTwnQplOlcr62i7gs7VSmpp14eAA9x9X+AM4MdmtlUqGwW86u6DgOHAdWbWuw5xd2UdbVOI/1g1n6ulJFegBEsKzKwv8AFgVlo0C/iAmfUpVD0BmOrub7r7EuB24NM1lHVLJbWr5NTapu4+z90fJf7HX6T2LiipXaWgDe36K3dfmZ4+BgRi7wzE8/XaVO9pwIGPdXLoXVZJbdpplGBJ0W7AIndfC5B+Lk7L83YHFuaeP5ur01JZd1VGuwKcaGaPmdndZjakMwPeCNTapi3RubqhMtoVdK4WtaddTwOecffn0nOdr+sro00BvmJmj5vZ7Wb23rKCU4IlsvG4BhiQurm/A/zczDr9vzCRdtC52kFmdihwCXBSo2PZVFRp0zHAIHffB/gpcFeam9lhSrCk6G9Av+YTLP3cJS3PexbYI/d891ydlsq6qw63q7u/4O6r0+/3pOV7d3LcXVmtbdoSnasb6nC76lytqOZ2TT1+NwPHu/tTuSKdr+vrcJu6+yJ3fzP9fiPQG9i1jOCUYMl63P1F4FHeyvBPAh5J81PybgPONLPN0nj38cB/1VDWLZXRrmbWr7mSme0H9AeeoptqQ5u2ROdqQRntqnN1Q7W2q5kdAPwY+JS7/7GwmduAs1K9dwMHAHd1ZtxdWRltWjhXP0q80nBRGfHpNg1SydnADDO7CFhGHLPGzP4buMjdHbgJ+CDQfDnsN929Kf3eUll31tF2nWhm+xM/AFYBn3X3F+p5AF1Qq21qZgcBtwDbAsHMTgQ+ny7T1rlaWUfbVedqZbV8BlwNbAVca7bue4k/6+6PE4dbp5vZPGLbfsHdX6vzMXQ1HW3TGWb2DuBN4FXg4+5eyoUbupO7iIiISMk0RCgiIiJSMiVYIiIiIiVTgiUiIiJSMiVYIiIiIiVTgiUiIiJSMiVY0q2EED4aQngg9/ywEMKCBoZUNyGE6SGE60rcXv8QQpZ73ieEsDCEsFMN654dQriprFg2BiGEg0MIyxsdR3cUQji1Le/zst8r0rLOem+043X/VgjhkrL2rwRLuo0QQgAuBy5upd45IYQ/hxBeDSEsCyF4COGEXPmCEMKpFdbbYHmI5qZt9S6UHRZCyEIIK9JjcQjhhhDC2zt2pI2RZdkSYCatt+/bgG8CY+sQVpeRZdkDWZZt3+g4qgkhjA0h3NvoOLqDzmrrEMKcEMIFZW+3sxXfGw08FycBXwoh9Gu1Zg2UYEl3chTQC7ivWoUQwknEBOHzwHbEr104j3gDu/YYCgwk3sSu0neKrc2yrHeWZb2Bg4AhwOR27qsrmAacHkLYtoU6pwKPZ1n2TJ1iWk8IoUcIQZ99IrKeLMuWAXeS7pbfUfqQkU6RenMuCCHcl3pnHg8h7BtCOCmEMC+E8EoI4boQQs/cOruHEP4rhPBCCOH5EMIPQwjb5MonhhDmp+09E0L4z1xZ/9Qb9NkQwl9CCK+FEO4OIeycC+t44N6s5bvrfhi4P8uy32fR6+m/q7vb2RRnEb/K4iZaedNmWTYfuAN4f7EshNAztcnxheXTQwg3pN8PDyH8PvW6LQkh3BJC6Fttf6m9Dso9PyyEsCb3vGcI4fzUA7c8hPBgCMEqb23dMTwNvAQc0UK144F7CrH8RwjhyfS6PRtCuDSE0COVfSeEcHuh/mGp7tvS871DCL9Kx928/uaprPnc+HwI4S/ASqBvCOHEEMKfUu/i8yGEa5u3l9Z7ZwhhdjpX56b1sxBC/1ydM1Nv5yshhEdCCEdVO+gK7Ts9hHBTCGFaat9F6f2xXwjhD+n47gsh7JJbZ0EI4aIQwu/S+8BDCAfkyls8B0IIm6fX9Km0/WdCCJ8KsYf2fOCw8FaP6sAqx3Fo2scr6TU7K1d2WAhhTQjhhLTtV0IIt+bfxxW2157Pin1DCL9Jxzk/rd8jV/6vqW1WhBB+R/wnJ7/PrUMI3w0hNIUQXg4h3BVCGFQtxgox7xhCuDHEz6oXQggzQq7nORR6s3Pn4K7V2jqEMCId7+h0Pr4YQvhehfN419x2R4QQ5qXfrwIOBi5M26z41UQh9g79OoQwKZ0jS0MIXwkh7JHa9LUQwsMhhPfm1unQeyV3rk/NnesbnDfp9xbbp3As6w3llvS630P8jOq4LMv00KP0B7CA+PUj7wU2J37J5jPAD4G3Eb+k9EXglFR/S2AecehoK2AH4L+BabltnkrsUQrAR4DXgY+msv5ARkxQdiJ+fceDwNTc+r8HvlyI8zBgQe75p4F/AuOBw4Htqxzbqa0tB/oAbwD/RkyaMmD/wr7X5J4PIn5f27Qqbfpt4Pbc897ACuDg9Pwg4neT9QTeCdwPzMrVnw5cl3ueAQe1EM+E1GYDgR7EXr2XgB3ybV4hztnA+BbOjb8DHy8s+yQwIL227091zkplg4lft9InV38GcH36vS+wlJjA9gL6AQ5cVDg3fp3apVc6no8B7yP+ozkI+AtwaW4fvwZ+ks6lvsCctJ3+qfxM4jn7L2kbx6TXY1CV4y6273TiOTwsrX92Wv8XxC+b3Rr4DeufwwuAxcD+6Ti+ASwBtq3xHJiUjnPf1Na7AvumsrHEf0Bael8PSDGPSPv4EPAy8OncMWbA9cTz8x3Ez4ExJX5WbJfOjwuBLdJ684Gv5cqXprbpldrjBdZ/n/+I+FnxjlRnHPAksHml90qFmO8inuc7pMcvgV+28FnQP7XLrtXaOrXpamAK8TPwXcBc4PxK28itMy/3fA5wQSuv4di0n5G89T5YC9xbeA3uya3T0ffKdOJ58/G0jX9LMexR5b1RrX3mFZate53KeN1Tnf2JIw69WmrHWh51/aOrR/d5pA+Yr+WeH5PecPk/krcCl6ffPwU8U9jG/sQEpUeVffwX8O30e/OHzwG58i8Bj+SezwVGFLZxWP4NmJYdC/yU+CG+ljikuHfh2P4BLC883mT9D9WvE/8wNH9o/xG4trDvLK27DGgCrqFCUpfqv5eYaPRNz88A5rbwGhwLvJh7vu7DKD2vmmAR//i+BhxS2ObjzcdI9QTrR8DVLcS1CjislfPnu8Ctuee/B85Lv2+T2v/A9HwU8JvC+p8kfRjnzo1DWtnnucBD6fdd0zoDc+WHs/4fjT8DpxW2MZsqf+ConGDl/yhvnbb/6dyyL7L+ObwAuCT3PADPAie3dg6kuiuAYVXqjqX1BOt84MHCskuBXxXO6fz7/DvAz1rY5gLa9llxMvA30le9pWVnAU+l309JbZIvn0B6nxP/AcuA3XPlmwGvkN4PtJBgEf/Jy4B355a9Jy3bOXdM7Umw3gC2zi0bSXqPF7eRW6c9CdYThWUvVngNlpX4XplO7lxPy5YAx1V5b1Rrn5YSrA6/7mnZu1O9vi21Yy0PfdmzdKbnc7+vJM43WlJY1jx0MADYPWx4JUlG/E98UQjhy8Reg12Jfyy2Ik6qrrbPf+S2DzGJaWluUNxhlt1B/C+HEMJexC8KvSOEMCBL70Bi78rN+fVC7mqVEEJIsd6cZdnqtPh64FshhFFZljV/QevarMaJz1mW/TWE8EdiT95lwOnADbl97g9MJPaobE1so94VNlWLndK6s0PuSkHif7e7Vl5lnW2JyWI1G7wOIc59+wqxt6wn8b/L/8tVuQE4h3iRwmeA57IsezCVDQAOLJw7gfjfed6Cwj6PBC4C9iL2hPQg/qGB2AsG8QO72cLC9gYAU0IIV+aW9QSeo3brztcsy1bG02aD901xeG1Bbp0shPAs6TVp5RzoQ+wRmtuG+Ip2Y8PX9hnguNzz4vu8+D6spC2fFbsBC3PvxeYYdku/71qhPB/zgPTzsdTezTbPbaMlzXXy23wmV/Y87fdilmUrc88X0Pr7rT2KMa6khfOuhPdKpX3Wcl60RVmv+7a89Y9vh2gOlnQVC4n/qW1feGyZZdmiEMKBxOGNs4CdUlIym/gHpFaPEIebapZl2ZPEP+p7EIcCavURYlf6Gc3zNIjd0b2J/4G31w3AiDRv4EPAjbmyW4i9ZHtmWbYtlSfV560g/sFttkvu95eIH4BHFF6Pt2VZ9q1Wtrs3sa2rWe91CCHsRhySGE/sAdiOOEySf21vAfYMIXyA+J/sDbmyhcT/dvNxbpfFCwfy3sztsxdwe9ru7qm9Ruf2uSj93D23fv735v2eUdhv7yzLzmnh2MvQv/mXlMjvzltJXUvnwBLiH853V9num1WW5/0tv/9kYFpeL38D9gjr/5XMx7CoQnn/3O/Nf/zfXXjtts6ybFaN+y9uc2Ch7DWqv7egelv3DSFsXYi7+bVt/qesPdttt5LeK21V6TiKbQrrH39Zr/vexB6+Ve2MfR0lWNJV3AH0CnEC7jYh6hdC+EQq35Y4XLcEyEIIw4jzAtridmLXdVUhhDNCCJ8O6V5OaULp2cBfsix7uQ37Oos4/2UvYL/02JuYGHyhjXHn3UJM3K4kzpFYlCvbltjd/VoIYXfiXISWPAx8LoTQK01G/UpzQfov8ArguyGEdwOEEHqHeB+x4of6Oinx60Ocz1HN7aw/Cb438bNoCbA6hPAh4LP5FbIsWw78jJiEfYg4B6vZjYCl127LEMJmaVLs0S3E0Iv4n/iyLMteDyEMJg57NO/vOeJwy7fS+dgHKF7+fjkwNsRJ6SGEsFUI4aDU69mZzgghfCDEyc9fI/ZU/TKVVT0H0mt6NfDtEC8KCCFOut43VXmB2Ivcq4V9zwL2DyGcFuJFEP9KPNevL/UIW/ZL4mt3fjp330P8g98cwx3Ec+prIU7q/wBx/iAAWZa9SOz5vjqky/FDCNuHED4RCrdSqSTLssXA3cD30no7AN8D7syyrLmX5mHgpPSe6UOcL5ZXra03Ayalc2kgcfh7RtrvUlJSH+KVsPsQe8mL2615sn6NynivtFWl9nmUmIAem97jnwAOyZWX9bofSfyM6jAlWNIlpG7xjxB7Np4k/pH4NTExAfgV8Q/pQ8TelU8R/+C2xa+ANSGEw1qos4w4FPXXEMI/iHN/lhPnstQkxKu2jge+m2XZC/kHsRfu/aGVq/GqybLsFeJxf4x4S4S8LxDnbLxGnEN2WyubO5f4YfwycY7L9EL5xcDPgZ+HEF4lTkQ+m5Y/N84Apqc4q7kJ+Jf0B4Qsy/6a29dyYlJQqSfhBuJx/yr3h4zUrkOJbb6A+Br+jMIVRHlZlq0gvs7fDiGsIPaYFYebTyYmL88RL5hobs830jamEi88uCHt81niH9LNWzj2MvyQmGAvA04gzqlqbu/WzoExxNf69lRnDm/9Qb6N2APzQohXeg0orEuWZU3E+TnnEicU3wRcmGXZrWUdXGvSsR5FTNL/zlufDZel8uXECwdOILbRlcAPCps5k3hByZwQwmvEuYWfJg4N1eJUYvs9Rfy8Wg6cliu/gPgP4fPENr6lsH61tl5IPN+aiJ89dxHPsWafI34WvZKOt5jYXk78Z2N5COGJGo+lRWW8V9phg/bJ4m1d/oN4/r8MHE2cWN8cZ4df9xDC9sTz+5p2xr2esP5wpcimLfVqnJ9l2SHp+WHEhKB/I+PaGKVer6Ysy0J63od49Z4V5s9UWvds4iT1z7ZUrysJIXyUmARulTXogzPEeX4XFOf/ycYvhDCC+NqW3QNVd13hvdIeIYRLifP/SrlZqya5S7eSZdldxP8KpWQpqdqjxrrXUNJ/iZ0lhLAfcS7I48QJsuOBH29MfzBE6mFTea9kWfb/ytyehgilu1vAxn3n9EZaTpy4v6nagTjMtgL4HfAYcYhCRNan90oFGiIUERERKZl6sERERERKpgRLREREpGRKsERERERKpgRLREREpGRKsERERERKpgRLREREpGT/HwHNlylgTqzdAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "shap.summary_plot(shap_values, X, plot_type=\"bar\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Hyperparameter tunning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Training speed" ] }, { "cell_type": "code", "execution_count": 86, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "80373e00f5534d56b89daaf137b6a7bf", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 86, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoost\n", "fast_model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=150,\n", " learning_rate=0.01,\n", " boosting_type='Plain',\n", " bootstrap_type='Bernoulli',\n", " subsample=0.5,\n", " rsm=0.5,\n", " one_hot_max_size=20,\n", " leaf_estimation_iterations=2,\n", " max_ctr_complexity=1)\n", "\n", "fast_model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " verbose=False,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Accuracy" ] }, { "cell_type": "code", "execution_count": 87, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "83c5a9d1574b4f498df146f5339d8d99", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 87, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tunned_model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=1000,\n", " learning_rate=0.03,\n", " l2_leaf_reg=3,\n", " bagging_temperature=1,\n", " random_strength=1,\n", " one_hot_max_size=2,\n", " leaf_estimation_method='Newton'\n", ")\n", "tunned_model.fit(\n", " X_train, y_train,\n", " cat_features=cat_features,\n", " verbose=False,\n", " eval_set=(X_validation, y_validation),\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Training the model after parameter tunning" ] }, { "cell_type": "code", "execution_count": 88, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Learning rate set to 0.044153\n", "0:\tlearn: 0.6405883\ttotal: 37.9ms\tremaining: 38.7s\n", "100:\tlearn: 0.1584501\ttotal: 5.77s\tremaining: 52.6s\n", "200:\tlearn: 0.1518465\ttotal: 11.5s\tremaining: 46.9s\n", "300:\tlearn: 0.1486421\ttotal: 17.5s\tremaining: 41.8s\n", "400:\tlearn: 0.1463685\ttotal: 23.8s\tremaining: 36.8s\n", "500:\tlearn: 0.1446445\ttotal: 30.1s\tremaining: 31.2s\n", "600:\tlearn: 0.1431370\ttotal: 36.5s\tremaining: 25.5s\n", "700:\tlearn: 0.1417911\ttotal: 42.7s\tremaining: 19.5s\n", "800:\tlearn: 0.1408268\ttotal: 48.8s\tremaining: 13.4s\n", "900:\tlearn: 0.1398684\ttotal: 55.1s\tremaining: 7.34s\n", "1000:\tlearn: 0.1390739\ttotal: 1m 1s\tremaining: 1.23s\n", "1020:\tlearn: 0.1389030\ttotal: 1m 2s\tremaining: 0us\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 88, "metadata": {}, "output_type": "execute_result" } ], "source": [ "best_model = CatBoostClassifier(\n", " random_seed=63,\n", " iterations=int(tunned_model.tree_count_ * 1.2),\n", ")\n", "best_model.fit(\n", " X, y,\n", " cat_features=cat_features,\n", " verbose=100\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Feature evaluation" ] }, { "cell_type": "code", "execution_count": 89, "metadata": {}, "outputs": [], "source": [ "from catboost.eval.catboost_evaluation import *\n", "learn_params = {'iterations': 20, # 2000\n", " 'learning_rate': 0.5, # we set big learning_rate,\n", " # because we have small\n", " # #iterations\n", " 'random_seed': 0,\n", " 'verbose': False,\n", " 'loss_function' : 'Logloss',\n", " 'boosting_type': 'Plain'}\n", "evaluator = CatboostEvaluation('amazon/train.tsv',\n", " fold_size=10000, # <= 50% of dataset\n", " fold_count=20,\n", " column_description='amazon/train.cd',\n", " partition_random_seed=0,\n", " #working_dir=... \n", ")\n", "\n", "result = evaluator.eval_features(learn_config=learn_params,\n", " eval_metrics=['Logloss', 'Accuracy'],\n", " features_to_eval=[6, 7, 8])" ] }, { "cell_type": "code", "execution_count": 90, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PValueScoreQuantile 0.005Quantile 0.995Decision
Features: 60.0002191.5395420.8560902.174448GOOD
Features: 80.0929630.363555-0.3215350.947839UNKNOWN
Features: 70.1353570.289296-0.2961040.847476UNKNOWN
\n", "
" ], "text/plain": [ " PValue Score Quantile 0.005 Quantile 0.995 Decision\n", "Features: 6 0.000219 1.539542 0.856090 2.174448 GOOD\n", "Features: 8 0.092963 0.363555 -0.321535 0.947839 UNKNOWN\n", "Features: 7 0.135357 0.289296 -0.296104 0.847476 UNKNOWN" ] }, "execution_count": 90, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost.eval.evaluation_result import *\n", "logloss_result = result.get_metric_results('Logloss')\n", "logloss_result.get_baseline_comparison(\n", " ScoreConfig(ScoreType.Rel, overfit_iterations_info=False)\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Calculate predictions for the contest" ] }, { "cell_type": "code", "execution_count": 91, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Predictoins:\n", "[[0.3846 0.6154]\n", " [0.0154 0.9846]\n", " [0.0106 0.9894]\n", " ...\n", " [0.0077 0.9923]\n", " [0.0641 0.9359]\n", " [0.0147 0.9853]]\n" ] } ], "source": [ "X_test = test_df.drop('id', axis=1)\n", "test_pool = Pool(data=X_test, cat_features=cat_features)\n", "contest_predictions = best_model.predict_proba(test_pool)\n", "print('Predictoins:')\n", "print(contest_predictions)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prepare the submission" ] }, { "cell_type": "code", "execution_count": 92, "metadata": {}, "outputs": [], "source": [ "f = open('submit.csv', 'w')\n", "f.write('Id,Action\\n')\n", "for idx in range(len(contest_predictions)):\n", " line = str(test_df['id'][idx]) + ',' + str(contest_predictions[idx][1]) + '\\n'\n", " f.write(line)\n", "f.close()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Submit your solution [here](https://www.kaggle.com/c/amazon-employee-access-challenge/submit).\n", "Good luck!!!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bonus\n", "### Solving MultiClassification problem via Ranking" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For multiclass problems with many classes sometimes it's better to solve classification problem using ranking.\n", "To do that we will build a dataset with groups.\n", "Every group will represent one object from our initial dataset.\n", "But it will have one additional categorical feature - possible class value.\n", "Target values will be equal to 1 if the class value is equal to the correct class, and 0 otherwise.\n", "Thus each group will have exactly one 1 in labels, and some zeros.\n", "You can put all possible class values in the group or you can try setting only hard negatives if there are too many labels.\n", "We'll show this approach on an example of binary classification problem." ] }, { "cell_type": "code", "execution_count": 93, "metadata": {}, "outputs": [], "source": [ "from copy import deepcopy\n", "def build_multiclass_ranking_dataset(X, y, cat_features, label_values=[0,1], start_group_id=0):\n", " ranking_matrix = []\n", " ranking_labels = []\n", " group_ids = []\n", "\n", " X_train_matrix = X.values\n", " y_train_vector = y.values\n", "\n", " for obj_idx in range(X.shape[0]):\n", " obj = list(X_train_matrix[obj_idx])\n", "\n", " for label in label_values:\n", " obj_of_given_class = deepcopy(obj)\n", " obj_of_given_class.append(label)\n", " ranking_matrix.append(obj_of_given_class)\n", " ranking_labels.append(float(y_train_vector[obj_idx] == label)) \n", " group_ids.append(start_group_id + obj_idx)\n", " \n", " final_cat_features = deepcopy(cat_features)\n", " final_cat_features.append(X.shape[1]) # new feature that we are adding should be categorical.\n", " return Pool(ranking_matrix, ranking_labels, cat_features=final_cat_features, group_id = group_ids)" ] }, { "cell_type": "code", "execution_count": 94, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "65c2a9bef94b40c3b9c0f9ec8f975280", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self=u'stretch', height=u'500px'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 94, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost import CatBoost\n", "params = {'iterations':150, 'learning_rate':0.01, 'l2_leaf_reg':30, 'random_seed':0, 'loss_function':'QuerySoftMax'}\n", "\n", "groupwise_train_pool = build_multiclass_ranking_dataset(X_train, y_train, cat_features, [0,1])\n", "groupwise_eval_pool = build_multiclass_ranking_dataset(X_validation, y_validation, cat_features, [0,1], X_train.shape[0])\n", "\n", "model = CatBoost(params)\n", "model.fit(\n", " X=groupwise_train_pool,\n", " verbose=False,\n", " eval_set=groupwise_eval_pool,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Doing predictions with ranking mode" ] }, { "cell_type": "code", "execution_count": 96, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('Raw values:', array([-1.3969, 1.4067]))\n", "('Probabilities', array([0.0571, 0.9429]))\n" ] } ], "source": [ "import math\n", "\n", "obj = list(X_validation.values[0])\n", "ratings = []\n", "for label in [0,1]:\n", " obj_with_label = deepcopy(obj)\n", " obj_with_label.append(label)\n", " rating = model.predict([obj_with_label])[0]\n", " ratings.append(rating)\n", "print('Raw values:', np.array(ratings))\n", "\n", "def soft_max(values):\n", " return [math.exp(val) / sum([math.exp(val) for val in values]) for val in values]\n", "\n", "print('Probabilities', np.array(soft_max(ratings)))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" }, "widgets": { "state": { "1057714ebc614324aa3ba2cf69408966": { "views": [ { "cell_index": 17 } ] }, "8381e9eed05f4a03905ae8a56d7ab4ea": { "views": [ { "cell_index": 48 } ] }, "f49684e8c5c44241bfe2c7f577f5cb41": { "views": [ { "cell_index": 53 } ] } }, "version": "1.2.0" } }, "nbformat": 4, "nbformat_minor": 2 }