{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.163789Z", "start_time": "2018-07-31T02:46:34.410255Z" } }, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import warnings\n", "import matplotlib.pyplot as plt\n", "warnings.filterwarnings('ignore')\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Decision Tree and Ensemble Learning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Decision Tree\n", "\n", "A decision tree may be described as a set of decisions/categorization steps that help you to classify your data into predefined classes. This helps you to solve a classification problem. \n", "\n", "For example, consider the following data set related to playing tennis in a specific day based on weather condition parameters. Based on the provided parameters, we are need to come up with a decision tree that helps us to decide whether the player plays tennis on a specific day based on four weather related parameters including outlook, temperature, humidity and wind condition." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.202766Z", "start_time": "2018-07-31T02:46:35.165787Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
OutlookTemperatureHumidityWindPlayTennis
0SunnyHotHighFalseNo
1SunnyHotHighTrueNo
2OvercastHotHighFalseYes
3RainyMildHighFalseYes
4RainyCoolNormalFalseYes
5RainyCoolNormalTrueNo
6OvercastCoolNormalTrueYes
7SunnyMildHighFalseNo
8SunnyCoolNormalFalseYes
9RainyMildNormalFalseYes
10SunnyMildNormalTrueYes
11OvercastMildHighTrueYes
12OvercastHotNormalFalseYes
13RainyMildHighTrueNo
\n", "
" ], "text/plain": [ " Outlook Temperature Humidity Wind PlayTennis\n", "0 Sunny Hot High False No\n", "1 Sunny Hot High True No\n", "2 Overcast Hot High False Yes\n", "3 Rainy Mild High False Yes\n", "4 Rainy Cool Normal False Yes\n", "5 Rainy Cool Normal True No\n", "6 Overcast Cool Normal True Yes\n", "7 Sunny Mild High False No\n", "8 Sunny Cool Normal False Yes\n", "9 Rainy Mild Normal False Yes\n", "10 Sunny Mild Normal True Yes\n", "11 Overcast Mild High True Yes\n", "12 Overcast Hot Normal False Yes\n", "13 Rainy Mild High True No" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_dict = {\n", " 'Outlook' : ['Sunny', 'Sunny', 'Overcast', 'Rainy', 'Rainy', 'Rainy', 'Overcast', 'Sunny', 'Sunny','Rainy', 'Sunny', 'Overcast', 'Overcast', 'Rainy']\n", " ,'Temperature': ['Hot', 'Hot', 'Hot', 'Mild', 'Cool', 'Cool', 'Cool', 'Mild', 'Cool', 'Mild','Mild','Mild', 'Hot', 'Mild']\n", " ,'Humidity' : ['High', 'High', 'High', 'High', 'Normal', 'Normal', 'Normal', 'High','Normal','Normal', 'Normal', 'High', 'Normal', 'High']\n", " ,'Wind': ['False', 'True', 'False', 'False', 'False', 'True', 'True', 'False', 'False', 'False', 'True', 'True', 'False', 'True']\n", " ,'PlayTennis': ['No', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes', 'Yes', 'Yes', 'No']\n", "}\n", "tennis_data = pd.DataFrame(data_dict, columns=data_dict.keys())\n", "tennis_data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following decision three helps us to use a set of decisions to solve our classification problem.\n", "![alt text](Figures/DT1.jpg \"Decison tree for tennis play problem\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Order of decision; why Outlook is the first question/decision?\n", "\n", "The main question is why outlook was chosen as the first decision level. The answer is that the amount of **Information Grain** that is resolved by this decision is higher that the other decisions. This can be quantified using entropy which is a measure of randomness. Higher entropy means higher randomness and the amount of reduction in randomness based on a decision is called information gain." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.210759Z", "start_time": "2018-07-31T02:46:35.204764Z" } }, "outputs": [ { "data": { "text/latex": [ "\n", "Entropy = $-\\sum_{i=1}^{n} P_i\\times Log_b(P_i)$" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%latex\n", "\n", "Entropy = $-\\sum_{i=1}^{n} P_i\\times Log_b(P_i)$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note: corresponding units of entropy are the bits for b = 2, nats for b = e, and bans for b = 10 where b is the base of the logarithm function" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.217756Z", "start_time": "2018-07-31T02:46:35.212759Z" } }, "outputs": [], "source": [ "def entropy_calculate(prob_list):\n", " \n", " entropy = 0\n", " for item in prob_list:\n", " entropy -= item * np.log2(item) \n", " return entropy" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Entropy of the entire system\n", "Entropy of the entire system:\n", "14 observations: 9 Yes and 5 No" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.235745Z", "start_time": "2018-07-31T02:46:35.220753Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Probabilities of No and Yes are 0.357, 0.643 respectively\n", "Entire syetems entropy is 0.940 bits\n" ] } ], "source": [ "cases,counts = np.unique(tennis_data.PlayTennis,return_counts=True)\n", "P = [count/len(tennis_data) for count in counts]\n", "print('Probabilities of %s and %s are %.3f, %.3f respectively'%(cases[0],cases[1],P[0],P[1]))\n", "\n", "entropy_entire = entropy_calculate(P)\n", "\n", "print('Entire syetems entropy is %.3f bits'%entropy_entire)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Information Gain\n", "\n", "Lets calculate reduction in entropy for each decision. For each decision, entropy is calculated for each case under that decision and the a probability weighted average is calculated." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Outlook decision" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.243740Z", "start_time": "2018-07-31T02:46:35.238744Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "For outlook:\n", "\tProbabality of Overcast is 0.286\n", "\tProbabality of Rainy is 0.357\n", "\tProbabality of Sunny is 0.357\n" ] } ], "source": [ "cases_outlook,counts_outlook= np.unique(tennis_data.Outlook,return_counts=True)\n", "P_outlook = [count/len(tennis_data) for count in counts_outlook]\n", "print('For outlook:')\n", "for case, prob in zip(cases_outlook,P_outlook):\n", " print('\\tProbabality of %s is %.3f'%(case, prob))" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.260729Z", "start_time": "2018-07-31T02:46:35.245739Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Entropy for Overcast is 0.00\n", "Entropy for Rainy is 0.97\n", "Entropy for Sunny is 0.97\n", "\n", "Entropy at Outlook decision level is 0.694\n", "\n", "Information gain is 0.247\n" ] } ], "source": [ "entropy_outlook={}\n", "total_entropy_outlook=0\n", "for case, prob in zip(cases_outlook,P_outlook):\n", " cases,counts = np.unique(tennis_data.PlayTennis[tennis_data.Outlook==case],return_counts=True)\n", " P = [count/len(tennis_data[tennis_data.Outlook==case]) for count in counts]\n", " entropy_outlook[case]=entropy_calculate(P)\n", " total_entropy_outlook += entropy_calculate(P)*prob\n", "\n", "for case, entropy in entropy_outlook.items():\n", " print('Entropy for %s is %.2f'%(case,entropy))\n", "print('\\nEntropy at Outlook decision level is %.3f'%total_entropy_outlook)\n", "print('\\nInformation gain is %.3f'%(entropy_entire- total_entropy_outlook))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Temperature Decision" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.268726Z", "start_time": "2018-07-31T02:46:35.262729Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "For temperature:\n", "\tProbabality of Cool is 0.286\n", "\tProbabality of Hot is 0.286\n", "\tProbabality of Mild is 0.429\n" ] } ], "source": [ "cases_temperature,counts_temperature= np.unique(tennis_data.Temperature,return_counts=True)\n", "P_temperature = [count/len(tennis_data) for count in counts_temperature]\n", "print('For temperature:')\n", "for case, prob in zip(cases_temperature,P_temperature):\n", " print('\\tProbabality of %s is %.3f'%(case, prob))" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.290710Z", "start_time": "2018-07-31T02:46:35.271723Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Entropy for Cool is 0.81\n", "Entropy for Hot is 1.00\n", "Entropy for Mild is 0.92\n", "\n", "Entropy at Temperature decision level is 0.911\n", "\n", "Information gain is 0.029\n" ] } ], "source": [ "entropy_temperature={}\n", "total_entropy_temperature=0\n", "for case, prob in zip(cases_temperature,P_temperature):\n", " cases,counts = np.unique(tennis_data.PlayTennis[tennis_data.Temperature==case],return_counts=True)\n", " P = [count/len(tennis_data[tennis_data.Temperature==case]) for count in counts]\n", " entropy_temperature[case]=entropy_calculate(P)\n", " total_entropy_temperature += entropy_calculate(P)*prob\n", "\n", "for case, entropy in entropy_temperature.items():\n", " print('Entropy for %s is %.2f'%(case,entropy))\n", "print('\\nEntropy at Temperature decision level is %.3f'%total_entropy_temperature)\n", "print('\\nInformation gain is %.3f'%(entropy_entire- total_entropy_temperature))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Wind Decision" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.298705Z", "start_time": "2018-07-31T02:46:35.292709Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "For wind:\n", "\tProbabality of False is 0.571\n", "\tProbabality of True is 0.429\n" ] } ], "source": [ "cases_wind,counts_wind= np.unique(tennis_data.Wind,return_counts=True)\n", "P_wind = [count/len(tennis_data) for count in counts_wind]\n", "print('For wind:')\n", "for case, prob in zip(cases_wind,P_wind):\n", " print('\\tProbabality of %s is %.3f'%(case, prob))" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.312698Z", "start_time": "2018-07-31T02:46:35.300704Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Entropy for False is 0.81\n", "Entropy for True is 1.00\n", "\n", "Entropy at Wind decision level is 0.892\n", "\n", "Information gain is 0.048\n" ] } ], "source": [ "entropy_wind={}\n", "total_entropy_wind=0\n", "for case, prob in zip(cases_wind,P_wind):\n", " cases,counts = np.unique(tennis_data.PlayTennis[tennis_data.Wind==case],return_counts=True)\n", " P = [count/len(tennis_data[tennis_data.Wind==case]) for count in counts]\n", " entropy_wind[case]=entropy_calculate(P)\n", " total_entropy_wind += entropy_calculate(P)*prob\n", "\n", "for case, entropy in entropy_wind.items():\n", " print('Entropy for %s is %.2f'%(case,entropy))\n", "print('\\nEntropy at Wind decision level is %.3f'%total_entropy_wind)\n", "print('\\nInformation gain is %.3f'%(entropy_entire- total_entropy_wind))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Humidity Decision" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.320692Z", "start_time": "2018-07-31T02:46:35.314696Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "For humidity:\n", "\tProbabality of High is 0.500\n", "\tProbabality of Normal is 0.500\n" ] } ], "source": [ "cases_humidity,counts_humidity= np.unique(tennis_data.Humidity,return_counts=True)\n", "P_humidity = [count/len(tennis_data) for count in counts_humidity]\n", "print('For humidity:')\n", "for case, prob in zip(cases_humidity,P_humidity):\n", " print('\\tProbabality of %s is %.3f'%(case, prob))" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.336683Z", "start_time": "2018-07-31T02:46:35.322691Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Entropy for High is 0.99\n", "Entropy for Normal is 0.59\n", "\n", "Entropy at Humidity decision level is 0.788\n", "\n", "Information gain is 0.152\n" ] } ], "source": [ "entropy_humidity={}\n", "total_entropy_humidity=0\n", "for case, prob in zip(cases_humidity,P_humidity):\n", " cases,counts = np.unique(tennis_data.PlayTennis[tennis_data.Humidity==case],return_counts=True)\n", " P = [count/len(tennis_data[tennis_data.Humidity==case]) for count in counts]\n", " entropy_humidity[case]=entropy_calculate(P)\n", " total_entropy_humidity += entropy_calculate(P)*prob\n", "\n", "for case, entropy in entropy_humidity.items():\n", " print('Entropy for %s is %.2f'%(case,entropy))\n", "print('\\nEntropy at Humidity decision level is %.3f'%total_entropy_humidity)\n", "print('\\nInformation gain is %.3f'%(entropy_entire- total_entropy_humidity))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As shown above, by choosing outlook as the first decision/question, the highest reduction in entropy/randomness is achieved that corresponds to the highest **Information Gain**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Titanic Example" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.577533Z", "start_time": "2018-07-31T02:46:35.338683Z" } }, "outputs": [], "source": [ "import urllib3\n", "import pandas as pd\n", "import certifi\n", "import re\n", "from bs4 import BeautifulSoup\n", "import warnings\n", "warnings.filterwarnings('ignore')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Loading data using BeautifulSoup and urlib3 as the htm client" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:35.583529Z", "start_time": "2018-07-31T02:46:35.579533Z" } }, "outputs": [], "source": [ "html_address = 'https://www.encyclopedia-titanica.org/titanic-passengers-and-crew/'" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:44.086026Z", "start_time": "2018-07-31T02:46:35.587527Z" } }, "outputs": [], "source": [ "http = urllib3.PoolManager(maxsize=10, cert_reqs='CERT_REQUIRED',ca_certs=certifi.where())\n", "r = http.request('GET', html_address) " ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:49.267820Z", "start_time": "2018-07-31T02:46:44.088025Z" } }, "outputs": [], "source": [ "soup = BeautifulSoup(r.data, 'html.parser')" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:59.244638Z", "start_time": "2018-07-31T02:46:49.269816Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameAgeClass/DeptTicketJoinedJobBoat [Body]Unnamed: 7
0ABBING, Mr Anthony423rd Class Passenger5547£7 11sSouthamptonBlacksmithNaNNaN
1ABBOTT, Mrs Rhoda Mary 'Rosa'393rd Class PassengerCA2673£20 5sSouthamptonNaNANaN
2ABBOTT, Mr Rossmore Edward163rd Class PassengerCA2673£20 5sSouthamptonJeweller[190]NaN
3ABBOTT, Mr Eugene Joseph133rd Class PassengerCA2673£20 5sSouthamptonScholarNaNNaN
4ABBOTT, Mr Ernest Owen21Victualling CrewNaNSouthamptonLounge Pantry StewardNaNNaN
\n", "
" ], "text/plain": [ " Name Age Class/Dept Ticket \\\n", "0 ABBING, Mr Anthony 42 3rd Class Passenger 5547£7 11s \n", "1 ABBOTT, Mrs Rhoda Mary 'Rosa' 39 3rd Class Passenger CA2673£20 5s \n", "2 ABBOTT, Mr Rossmore Edward 16 3rd Class Passenger CA2673£20 5s \n", "3 ABBOTT, Mr Eugene Joseph 13 3rd Class Passenger CA2673£20 5s \n", "4 ABBOTT, Mr Ernest Owen 21 Victualling Crew NaN \n", "\n", " Joined Job Boat [Body] Unnamed: 7 \n", "0 Southampton Blacksmith NaN NaN \n", "1 Southampton NaN A NaN \n", "2 Southampton Jeweller [190] NaN \n", "3 Southampton Scholar NaN NaN \n", "4 Southampton Lounge Pantry Steward NaN NaN " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "table = soup.find('table')\n", "data = pd.read_html(str(table),flavor='bs4')[0] # Note that the flavor cotains information about the encoding as well\n", "http.clear()\n", "data.head()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:59.257632Z", "start_time": "2018-07-31T02:46:59.246637Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameAgeClass/DeptBoat [Body]
0ABBING, Mr Anthony423rd Class PassengerNaN
1ABBOTT, Mrs Rhoda Mary 'Rosa'393rd Class PassengerA
2ABBOTT, Mr Rossmore Edward163rd Class Passenger[190]
3ABBOTT, Mr Eugene Joseph133rd Class PassengerNaN
4ABBOTT, Mr Ernest Owen21Victualling CrewNaN
\n", "
" ], "text/plain": [ " Name Age Class/Dept Boat [Body]\n", "0 ABBING, Mr Anthony 42 3rd Class Passenger NaN\n", "1 ABBOTT, Mrs Rhoda Mary 'Rosa' 39 3rd Class Passenger A\n", "2 ABBOTT, Mr Rossmore Edward 16 3rd Class Passenger [190]\n", "3 ABBOTT, Mr Eugene Joseph 13 3rd Class Passenger NaN\n", "4 ABBOTT, Mr Ernest Owen 21 Victualling Crew NaN" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_trim = data[['Name', 'Age', 'Class/Dept', 'Boat [Body]']]\n", "data_trim.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Processing data\n", "\n", "Special characters are removed and then the age is change to float and ages below 1 year are converted to a representative float number" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:59.609419Z", "start_time": "2018-07-31T02:46:59.259628Z" } }, "outputs": [], "source": [ "data_trim['Name'] = data_trim['Name'].map(str).apply(lambda x: x.encode('utf-8').decode('ascii', 'ignore'))\n", "data_trim['Boat [Body]'] = data_trim['Boat [Body]'].map(str).apply(lambda x: x.encode('utf-8').decode('ascii', 'ignore'))\n", "def process_age(value):\n", " if 'm' in value:\n", " return float(re.findall(r'-?\\d+\\.?\\d*',value)[0])/12\n", " else:\n", " return(float(value)) \n", "data_trim['Age'] = data_trim['Age'].map(str).apply(process_age)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:59.622405Z", "start_time": "2018-07-31T02:46:59.611411Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameAgeClass/DeptBoat [Body]
0ABBING, Mr Anthony42.03rd Class Passengernan
1ABBOTT, Mrs Rhoda Mary 'Rosa'39.03rd Class PassengerA
2ABBOTT, Mr Rossmore Edward16.03rd Class Passenger[190]
3ABBOTT, Mr Eugene Joseph13.03rd Class Passengernan
4ABBOTT, Mr Ernest Owen21.0Victualling Crewnan
\n", "
" ], "text/plain": [ " Name Age Class/Dept Boat [Body]\n", "0 ABBING, Mr Anthony 42.0 3rd Class Passenger nan\n", "1 ABBOTT, Mrs Rhoda Mary 'Rosa' 39.0 3rd Class Passenger A\n", "2 ABBOTT, Mr Rossmore Edward 16.0 3rd Class Passenger [190]\n", "3 ABBOTT, Mr Eugene Joseph 13.0 3rd Class Passenger nan\n", "4 ABBOTT, Mr Ernest Owen 21.0 Victualling Crew nan" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_trim.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Categorize passengers and crews" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:59.717345Z", "start_time": "2018-07-31T02:46:59.624404Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameAgeClass/DeptBoat [Body]Crew/Passenger
0ABBING, Mr Anthony42.03rd Class PassengernanPassenger
1ABBOTT, Mrs Rhoda Mary 'Rosa'39.03rd Class PassengerAPassenger
2ABBOTT, Mr Rossmore Edward16.03rd Class Passenger[190]Passenger
3ABBOTT, Mr Eugene Joseph13.03rd Class PassengernanPassenger
4ABBOTT, Mr Ernest Owen21.0Victualling CrewnanCrew
\n", "
" ], "text/plain": [ " Name Age Class/Dept Boat [Body] \\\n", "0 ABBING, Mr Anthony 42.0 3rd Class Passenger nan \n", "1 ABBOTT, Mrs Rhoda Mary 'Rosa' 39.0 3rd Class Passenger A \n", "2 ABBOTT, Mr Rossmore Edward 16.0 3rd Class Passenger [190] \n", "3 ABBOTT, Mr Eugene Joseph 13.0 3rd Class Passenger nan \n", "4 ABBOTT, Mr Ernest Owen 21.0 Victualling Crew nan \n", "\n", " Crew/Passenger \n", "0 Passenger \n", "1 Passenger \n", "2 Passenger \n", "3 Passenger \n", "4 Crew " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def categorizer(value):\n", " if 'PASSENGER' in value.upper():\n", " return 'Passenger'\n", " else:\n", " return 'Crew'\n", "data_trim['Crew/Passenger'] = data_trim['Class/Dept'].map(str).apply(categorizer)\n", "data_trim.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Check passenger ticket class" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:59.826278Z", "start_time": "2018-07-31T02:46:59.720346Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameAgeClass/DeptBoat [Body]Crew/PassengerClass
0ABBING, Mr Anthony42.03rd Class PassengernanPassenger3rd
1ABBOTT, Mrs Rhoda Mary 'Rosa'39.03rd Class PassengerAPassenger3rd
2ABBOTT, Mr Rossmore Edward16.03rd Class Passenger[190]Passenger3rd
3ABBOTT, Mr Eugene Joseph13.03rd Class PassengernanPassenger3rd
4ABBOTT, Mr Ernest Owen21.0Victualling CrewnanCrewCrew
\n", "
" ], "text/plain": [ " Name Age Class/Dept Boat [Body] \\\n", "0 ABBING, Mr Anthony 42.0 3rd Class Passenger nan \n", "1 ABBOTT, Mrs Rhoda Mary 'Rosa' 39.0 3rd Class Passenger A \n", "2 ABBOTT, Mr Rossmore Edward 16.0 3rd Class Passenger [190] \n", "3 ABBOTT, Mr Eugene Joseph 13.0 3rd Class Passenger nan \n", "4 ABBOTT, Mr Ernest Owen 21.0 Victualling Crew nan \n", "\n", " Crew/Passenger Class \n", "0 Passenger 3rd \n", "1 Passenger 3rd \n", "2 Passenger 3rd \n", "3 Passenger 3rd \n", "4 Crew Crew " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def get_passenger_class(value):\n", " if 'PASSENGER' in value.upper():\n", " return value.split(' ')[0]\n", " else:\n", " return 'Crew'\n", "data_trim['Class'] = data_trim['Class/Dept'].map(str).apply(get_passenger_class)\n", "data_trim.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Check Adult/Child" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:46:59.922218Z", "start_time": "2018-07-31T02:46:59.828277Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameAgeClass/DeptBoat [Body]Crew/PassengerClassAdult/Child
0ABBING, Mr Anthony42.03rd Class PassengernanPassenger3rdAdult
1ABBOTT, Mrs Rhoda Mary 'Rosa'39.03rd Class PassengerAPassenger3rdAdult
2ABBOTT, Mr Rossmore Edward16.03rd Class Passenger[190]Passenger3rdChild
3ABBOTT, Mr Eugene Joseph13.03rd Class PassengernanPassenger3rdChild
4ABBOTT, Mr Ernest Owen21.0Victualling CrewnanCrewCrewAdult
\n", "
" ], "text/plain": [ " Name Age Class/Dept Boat [Body] \\\n", "0 ABBING, Mr Anthony 42.0 3rd Class Passenger nan \n", "1 ABBOTT, Mrs Rhoda Mary 'Rosa' 39.0 3rd Class Passenger A \n", "2 ABBOTT, Mr Rossmore Edward 16.0 3rd Class Passenger [190] \n", "3 ABBOTT, Mr Eugene Joseph 13.0 3rd Class Passenger nan \n", "4 ABBOTT, Mr Ernest Owen 21.0 Victualling Crew nan \n", "\n", " Crew/Passenger Class Adult/Child \n", "0 Passenger 3rd Adult \n", "1 Passenger 3rd Adult \n", "2 Passenger 3rd Child \n", "3 Passenger 3rd Child \n", "4 Crew Crew Adult " ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def check_adult(age):\n", " if age > 18:\n", " return 'Adult'\n", " else:\n", " return 'Child'\n", "data_trim['Adult/Child'] = data_trim['Age'].apply(check_adult)\n", "data_trim.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Check Gender" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:00.023158Z", "start_time": "2018-07-31T02:46:59.925218Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameAgeClass/DeptBoat [Body]Crew/PassengerClassAdult/ChildGender
0ABBING, Mr Anthony42.03rd Class PassengernanPassenger3rdAdultMale
1ABBOTT, Mrs Rhoda Mary 'Rosa'39.03rd Class PassengerAPassenger3rdAdultFemale
2ABBOTT, Mr Rossmore Edward16.03rd Class Passenger[190]Passenger3rdChildMale
3ABBOTT, Mr Eugene Joseph13.03rd Class PassengernanPassenger3rdChildMale
4ABBOTT, Mr Ernest Owen21.0Victualling CrewnanCrewCrewAdultMale
\n", "
" ], "text/plain": [ " Name Age Class/Dept Boat [Body] \\\n", "0 ABBING, Mr Anthony 42.0 3rd Class Passenger nan \n", "1 ABBOTT, Mrs Rhoda Mary 'Rosa' 39.0 3rd Class Passenger A \n", "2 ABBOTT, Mr Rossmore Edward 16.0 3rd Class Passenger [190] \n", "3 ABBOTT, Mr Eugene Joseph 13.0 3rd Class Passenger nan \n", "4 ABBOTT, Mr Ernest Owen 21.0 Victualling Crew nan \n", "\n", " Crew/Passenger Class Adult/Child Gender \n", "0 Passenger 3rd Adult Male \n", "1 Passenger 3rd Adult Female \n", "2 Passenger 3rd Child Male \n", "3 Passenger 3rd Child Male \n", "4 Crew Crew Adult Male " ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def check_gender(name):\n", " firstname = name[name.index(',')+2:]\n", " salutation = firstname.split(' ')[0]\n", " if salutation.upper() in ['MR', 'MASTER']:\n", " return 'Male'\n", " else:\n", " return 'Female'\n", "\n", "data_trim['Gender'] = data_trim['Name'].map(str).apply(check_gender)\n", "data_trim.head() " ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:00.046143Z", "start_time": "2018-07-31T02:47:00.025158Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NameAgeClass/DeptBoat [Body]Crew/PassengerClassAdult/ChildGenderSurvival
0ABBING, Mr Anthony42.03rd Class PassengernanPassenger3rdAdultMale0
1ABBOTT, Mrs Rhoda Mary 'Rosa'39.03rd Class PassengerAPassenger3rdAdultFemale1
2ABBOTT, Mr Rossmore Edward16.03rd Class Passenger[190]Passenger3rdChildMale0
3ABBOTT, Mr Eugene Joseph13.03rd Class PassengernanPassenger3rdChildMale0
4ABBOTT, Mr Ernest Owen21.0Victualling CrewnanCrewCrewAdultMale0
\n", "
" ], "text/plain": [ " Name Age Class/Dept Boat [Body] \\\n", "0 ABBING, Mr Anthony 42.0 3rd Class Passenger nan \n", "1 ABBOTT, Mrs Rhoda Mary 'Rosa' 39.0 3rd Class Passenger A \n", "2 ABBOTT, Mr Rossmore Edward 16.0 3rd Class Passenger [190] \n", "3 ABBOTT, Mr Eugene Joseph 13.0 3rd Class Passenger nan \n", "4 ABBOTT, Mr Ernest Owen 21.0 Victualling Crew nan \n", "\n", " Crew/Passenger Class Adult/Child Gender Survival \n", "0 Passenger 3rd Adult Male 0 \n", "1 Passenger 3rd Adult Female 1 \n", "2 Passenger 3rd Child Male 0 \n", "3 Passenger 3rd Child Male 0 \n", "4 Crew Crew Adult Male 0 " ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def check_survival(value):\n", " if value.strip()=='nan' or '[' in value:\n", " return 0\n", " else:\n", " return 1\n", "data_trim['Survival'] = data_trim['Boat [Body]'].map(str).apply(check_survival)\n", "data_trim.head() " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Check the importance of each feature" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:00.052140Z", "start_time": "2018-07-31T02:47:00.048142Z" } }, "outputs": [], "source": [ "def entropy_calculate(prob_list):\n", " \n", " entropy = 0\n", " for item in prob_list:\n", " entropy -= item * np.log2(item) \n", " return entropy" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:00.062132Z", "start_time": "2018-07-31T02:47:00.055138Z" } }, "outputs": [], "source": [ "def information_gain(feature, data, initial_entropy):\n", " print('Information gain analysis for {}'.format(feature))\n", " probability_l1 = data.groupby([feature])['Survival'].count()/len(data)\n", " entropy_feature=0\n", " for item, probabality in probability_l1.items():\n", " print('\\tProbability of {}:'.format(item), probabality)\n", " probability_l2 = data[data[feature]==item].groupby(['Survival'])['Survival'].count()/len(data[data[feature]==item])\n", "\n", " entropy_feature += entropy_calculate(probability_l2) * probabality\n", " print('\\n\\tEntropy of feature: %.3f'%(entropy_feature))\n", " print('\\tInformation gain: %.3f'%(initial_entropy - entropy_feature))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Information gain analysis" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:01.033787Z", "start_time": "2018-07-31T02:47:00.064131Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Initial entropy: 0.815\n" ] } ], "source": [ "probability_initial = data_trim.groupby(['Survival'])['Survival'].count()/len(data_trim)\n", "initial_entropy = entropy_calculate(probability_initial)\n", "print('Initial entropy: %.3f'%(initial_entropy))" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:01.049778Z", "start_time": "2018-07-31T02:47:01.035786Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Information gain analysis for Crew/Passenger\n", "\tProbability of Crew: 0.4543987086359968\n", "\tProbability of Passenger: 0.5456012913640033\n", "\n", "\tEntropy of feature: 0.772\n", "\tInformation gain: 0.043\n" ] } ], "source": [ "information_gain(feature='Crew/Passenger', data=data_trim, initial_entropy=initial_entropy)" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:01.073762Z", "start_time": "2018-07-31T02:47:01.051777Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Information gain analysis for Class\n", "\tProbability of 1st: 0.14124293785310735\n", "\tProbability of 2nd: 0.11824051654560129\n", "\tProbability of 3rd: 0.2861178369652946\n", "\tProbability of Crew: 0.4543987086359968\n", "\n", "\tEntropy of feature: 0.738\n", "\tInformation gain: 0.077\n" ] } ], "source": [ "information_gain(feature='Class', data=data_trim, initial_entropy=initial_entropy)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:01.093749Z", "start_time": "2018-07-31T02:47:01.075761Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Information gain analysis for Adult/Child\n", "\tProbability of Adult: 0.880548829701372\n", "\tProbability of Child: 0.11945117029862792\n", "\n", "\tEntropy of feature: 0.813\n", "\tInformation gain: 0.003\n" ] } ], "source": [ "information_gain(feature='Adult/Child', data=data_trim, initial_entropy=initial_entropy)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:01.110740Z", "start_time": "2018-07-31T02:47:01.095750Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Information gain analysis for Gender\n", "\tProbability of Female: 0.23284907183212267\n", "\tProbability of Male: 0.7671509281678773\n", "\n", "\tEntropy of feature: 0.702\n", "\tInformation gain: 0.113\n" ] } ], "source": [ "information_gain(feature='Gender', data=data_trim, initial_entropy=initial_entropy)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Pick features and lable to construct the decision tree" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:01.128729Z", "start_time": "2018-07-31T02:47:01.112739Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AgeCrew/PassengerClassAdult/ChildGenderSurvival
042.0Passenger3rdAdultMale0
139.0Passenger3rdAdultFemale1
216.0Passenger3rdChildMale0
313.0Passenger3rdChildMale0
421.0CrewCrewAdultMale0
\n", "
" ], "text/plain": [ " Age Crew/Passenger Class Adult/Child Gender Survival\n", "0 42.0 Passenger 3rd Adult Male 0\n", "1 39.0 Passenger 3rd Adult Female 1\n", "2 16.0 Passenger 3rd Child Male 0\n", "3 13.0 Passenger 3rd Child Male 0\n", "4 21.0 Crew Crew Adult Male 0" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "training_data = data_trim[['Age', 'Crew/Passenger', 'Class', 'Adult/Child', 'Gender','Survival']]\n", "training_data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Process categorical variables to numerical" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:01.447534Z", "start_time": "2018-07-31T02:47:01.130727Z" } }, "outputs": [], "source": [ "category_map ={}\n", "for column in ['Crew/Passenger','Class','Adult/Child','Gender']:\n", " category_map[column] = dict( enumerate(training_data[column].astype('category').cat.categories) )\n", " training_data[column] = training_data[column].astype('category').cat.codes" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:01.460523Z", "start_time": "2018-07-31T02:47:01.449530Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AgeCrew/PassengerClassAdult/ChildGenderSurvival
042.012010
139.012001
216.012110
313.012110
421.003010
\n", "
" ], "text/plain": [ " Age Crew/Passenger Class Adult/Child Gender Survival\n", "0 42.0 1 2 0 1 0\n", "1 39.0 1 2 0 0 1\n", "2 16.0 1 2 1 1 0\n", "3 13.0 1 2 1 1 0\n", "4 21.0 0 3 0 1 0" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "training_data.head()" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:01.548469Z", "start_time": "2018-07-31T02:47:01.462521Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of valid records: 2449\n" ] } ], "source": [ "training_data.dropna(inplace=True)\n", "training_data.reset_index(drop=True, inplace=True)\n", "print('Total number of valid records: {}'.format(len(training_data)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Split to train and test data" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.092501Z", "start_time": "2018-07-31T02:47:01.550467Z" } }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.099493Z", "start_time": "2018-07-31T02:47:02.094497Z" } }, "outputs": [], "source": [ "train, test = train_test_split(training_data, test_size=0.2)\n", "train.reset_index(drop=True, inplace=True)\n", "test.reset_index(drop=True, inplace=True)" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.107489Z", "start_time": "2018-07-31T02:47:02.101491Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of records used for training: 1959\n", "Total number of records used for testin: 490\n" ] } ], "source": [ "print('Total number of records used for training: {}\\nTotal number of records used for testin: {}'.format(len(train),len(test)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Build a decision tree" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.156457Z", "start_time": "2018-07-31T02:47:02.110487Z" } }, "outputs": [], "source": [ "from sklearn.tree import DecisionTreeClassifier" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.165453Z", "start_time": "2018-07-31T02:47:02.158458Z" } }, "outputs": [], "source": [ "X = train[['Age', 'Crew/Passenger', 'Class', 'Adult/Child', 'Gender']]\n", "y = train[['Survival']]" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.171451Z", "start_time": "2018-07-31T02:47:02.167451Z" } }, "outputs": [], "source": [ "clf = DecisionTreeClassifier(max_leaf_nodes=20, criterion='entropy')" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.183442Z", "start_time": "2018-07-31T02:47:02.174448Z" } }, "outputs": [], "source": [ "clf = clf.fit(X, y)" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.192436Z", "start_time": "2018-07-31T02:47:02.186439Z" } }, "outputs": [ { "data": { "text/plain": [ "DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=None,\n", " max_features=None, max_leaf_nodes=20,\n", " min_impurity_decrease=0.0, min_impurity_split=None,\n", " min_samples_leaf=1, min_samples_split=2,\n", " min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n", " splitter='best')" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- class_weight: A dictionary to provide user defined weights for each feature\n", "\n", "- criterion: The criterion is the method used to decide with label and what decision needs to be made. gini impurity and information gain/entropy are the two main methods\n", "\n", "- max_depth: It determines the maximum depth/layer from the root of the tree to the finest leaf\n", "\n", "- max_features: if you want you can ask the algorithm to use a limited number of features based on their importance. This could be a dimension reduction approach that can simplify the tree\n", "\n", "- max_leaf_nodes: The max number of end nodes that is another stopping criteria for the decision tree\n", "\n", "- min_impurity_split: The minimum amount of impurity split percentage that result in generating another branch. In case of information gain, this will be measured by the amount/percentage of reduction in entropy.\n", "\n", "- min_samples_leaf: Minimum number of samples that should exist in a subset created by a decision. If less than this number, the branch gets pruned and the previous level of the tree will be kept\n", "\n", "- min_samples_split: Minimum number of samples in a tree level to consider further splitting and decision making\n", "\n", "- min_weight_fraction_leaf: similar to min_samples_leaf but expressed as a fraction of whole data\n", "\n", "- random_state, presort and splitter are performance related settings and do not affect the algorithm" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.200431Z", "start_time": "2018-07-31T02:47:02.193434Z" } }, "outputs": [ { "data": { "text/plain": [ "array([0.20502926, 0. , 0.28082862, 0. , 0.51414211])" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.feature_importances_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As can be seen above, the importance analysis confirms the result of information gain analysis that Gender is the most important feature. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Tree visualization" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.205429Z", "start_time": "2018-07-31T02:47:02.203430Z" } }, "outputs": [], "source": [ "from sklearn import tree" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.222417Z", "start_time": "2018-07-31T02:47:02.207426Z" } }, "outputs": [], "source": [ "with open ('Outputs/Titanic.dot', 'w') as f:\n", " f = tree.export_graphviz(clf, feature_names=['Age', 'Crew/Passenger', 'Class', 'Adult/Child', 'Gender'], out_file=f)" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.249399Z", "start_time": "2018-07-31T02:47:02.224416Z" } }, "outputs": [], "source": [ "# conda install -c conda-forge python-graphviz\n", "from graphviz import Source" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.449277Z", "start_time": "2018-07-31T02:47:02.251400Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "Tree\r\n", "\r\n", "\r\n", "0\r\n", "\r\n", "Gender <= 0.5\r\n", "entropy = 0.822\r\n", "samples = 1959\r\n", "value = [1456, 503]\r\n", "\r\n", "\r\n", "1\r\n", "\r\n", "Class <= 1.5\r\n", "entropy = 0.982\r\n", "samples = 464\r\n", "value = [195, 269]\r\n", "\r\n", "\r\n", "0->1\r\n", "\r\n", "\r\n", "True\r\n", "\r\n", "\r\n", "2\r\n", "\r\n", "Age <= 12.0\r\n", "entropy = 0.626\r\n", "samples = 1495\r\n", "value = [1261, 234]\r\n", "\r\n", "\r\n", "0->2\r\n", "\r\n", "\r\n", "False\r\n", "\r\n", "\r\n", "3\r\n", "\r\n", "Class <= 0.5\r\n", "entropy = 0.737\r\n", "samples = 231\r\n", "value = [48, 183]\r\n", "\r\n", "\r\n", "1->3\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "4\r\n", "\r\n", "Class <= 2.5\r\n", "entropy = 0.95\r\n", "samples = 233\r\n", "value = [147, 86]\r\n", "\r\n", "\r\n", "1->4\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "21\r\n", "\r\n", "Age <= 38.5\r\n", "entropy = 0.581\r\n", "samples = 137\r\n", "value = [19, 118]\r\n", "\r\n", "\r\n", "3->21\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "22\r\n", "\r\n", "Age <= 16.0\r\n", "entropy = 0.891\r\n", "samples = 94\r\n", "value = [29, 65]\r\n", "\r\n", "\r\n", "3->22\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "25\r\n", "\r\n", "Age <= 8.0\r\n", "entropy = 0.326\r\n", "samples = 67\r\n", "value = [4, 63]\r\n", "\r\n", "\r\n", "21->25\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "26\r\n", "\r\n", "entropy = 0.75\r\n", "samples = 70\r\n", "value = [15, 55]\r\n", "\r\n", "\r\n", "21->26\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "29\r\n", "\r\n", "entropy = 0.0\r\n", "samples = 1\r\n", "value = [1, 0]\r\n", "\r\n", "\r\n", "25->29\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "30\r\n", "\r\n", "entropy = 0.267\r\n", "samples = 66\r\n", "value = [3, 63]\r\n", "\r\n", "\r\n", "25->30\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "23\r\n", "\r\n", "entropy = 0.0\r\n", "samples = 12\r\n", "value = [0, 12]\r\n", "\r\n", "\r\n", "22->23\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "24\r\n", "\r\n", "Age <= 53.0\r\n", "entropy = 0.937\r\n", "samples = 82\r\n", "value = [29, 53]\r\n", "\r\n", "\r\n", "22->24\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "35\r\n", "\r\n", "entropy = 0.904\r\n", "samples = 75\r\n", "value = [24, 51]\r\n", "\r\n", "\r\n", "24->35\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "36\r\n", "\r\n", "entropy = 0.863\r\n", "samples = 7\r\n", "value = [5, 2]\r\n", "\r\n", "\r\n", "24->36\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "9\r\n", "\r\n", "Age <= 38.5\r\n", "entropy = 0.991\r\n", "samples = 176\r\n", "value = [98, 78]\r\n", "\r\n", "\r\n", "4->9\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "10\r\n", "\r\n", "Age <= 27.5\r\n", "entropy = 0.585\r\n", "samples = 57\r\n", "value = [49, 8]\r\n", "\r\n", "\r\n", "4->10\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "13\r\n", "\r\n", "Age <= 20.5\r\n", "entropy = 1.0\r\n", "samples = 155\r\n", "value = [79, 76]\r\n", "\r\n", "\r\n", "9->13\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "14\r\n", "\r\n", "entropy = 0.454\r\n", "samples = 21\r\n", "value = [19, 2]\r\n", "\r\n", "\r\n", "9->14\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "37\r\n", "\r\n", "entropy = 0.977\r\n", "samples = 73\r\n", "value = [43, 30]\r\n", "\r\n", "\r\n", "13->37\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "38\r\n", "\r\n", "entropy = 0.989\r\n", "samples = 82\r\n", "value = [36, 46]\r\n", "\r\n", "\r\n", "13->38\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "11\r\n", "\r\n", "entropy = 0.0\r\n", "samples = 31\r\n", "value = [31, 0]\r\n", "\r\n", "\r\n", "10->11\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "12\r\n", "\r\n", "entropy = 0.89\r\n", "samples = 26\r\n", "value = [18, 8]\r\n", "\r\n", "\r\n", "10->12\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "5\r\n", "\r\n", "Class <= 1.5\r\n", "entropy = 1.0\r\n", "samples = 45\r\n", "value = [22, 23]\r\n", "\r\n", "\r\n", "2->5\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "6\r\n", "\r\n", "Class <= 0.5\r\n", "entropy = 0.599\r\n", "samples = 1450\r\n", "value = [1239, 211]\r\n", "\r\n", "\r\n", "2->6\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "17\r\n", "\r\n", "entropy = 0.592\r\n", "samples = 14\r\n", "value = [2, 12]\r\n", "\r\n", "\r\n", "5->17\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "18\r\n", "\r\n", "entropy = 0.938\r\n", "samples = 31\r\n", "value = [20, 11]\r\n", "\r\n", "\r\n", "5->18\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "7\r\n", "\r\n", "Age <= 49.5\r\n", "entropy = 0.893\r\n", "samples = 142\r\n", "value = [98, 44]\r\n", "\r\n", "\r\n", "6->7\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "8\r\n", "\r\n", "Class <= 1.5\r\n", "entropy = 0.551\r\n", "samples = 1308\r\n", "value = [1141, 167]\r\n", "\r\n", "\r\n", "6->8\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "15\r\n", "\r\n", "entropy = 0.955\r\n", "samples = 109\r\n", "value = [68, 41]\r\n", "\r\n", "\r\n", "7->15\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "16\r\n", "\r\n", "entropy = 0.439\r\n", "samples = 33\r\n", "value = [30, 3]\r\n", "\r\n", "\r\n", "7->16\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "19\r\n", "\r\n", "entropy = 0.276\r\n", "samples = 126\r\n", "value = [120, 6]\r\n", "\r\n", "\r\n", "8->19\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "20\r\n", "\r\n", "Age <= 52.5\r\n", "entropy = 0.574\r\n", "samples = 1182\r\n", "value = [1021, 161]\r\n", "\r\n", "\r\n", "8->20\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "27\r\n", "\r\n", "Age <= 19.5\r\n", "entropy = 0.582\r\n", "samples = 1158\r\n", "value = [997, 161]\r\n", "\r\n", "\r\n", "20->27\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "28\r\n", "\r\n", "entropy = 0.0\r\n", "samples = 24\r\n", "value = [24, 0]\r\n", "\r\n", "\r\n", "20->28\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "31\r\n", "\r\n", "entropy = 0.366\r\n", "samples = 100\r\n", "value = [93, 7]\r\n", "\r\n", "\r\n", "27->31\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "32\r\n", "\r\n", "Age <= 33.5\r\n", "entropy = 0.599\r\n", "samples = 1058\r\n", "value = [904, 154]\r\n", "\r\n", "\r\n", "27->32\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "33\r\n", "\r\n", "entropy = 0.653\r\n", "samples = 661\r\n", "value = [550, 111]\r\n", "\r\n", "\r\n", "32->33\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "34\r\n", "\r\n", "entropy = 0.495\r\n", "samples = 397\r\n", "value = [354, 43]\r\n", "\r\n", "\r\n", "32->34\r\n", "\r\n", "\r\n", "\r\n", "\r\n", "\r\n" ], "text/plain": [ "" ] }, "execution_count": 50, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s = Source.from_file('Outputs/Titanic.dot')\n", "s.render() # saves a pdf file\n", "s" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Evaluate the tree" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Using train and test data sets" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.455273Z", "start_time": "2018-07-31T02:47:02.451277Z" } }, "outputs": [], "source": [ "from sklearn.metrics import accuracy_score" ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.463268Z", "start_time": "2018-07-31T02:47:02.457271Z" } }, "outputs": [], "source": [ "predictions = clf.predict(test.drop(['Survival'], axis=1))" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.471264Z", "start_time": "2018-07-31T02:47:02.465266Z" } }, "outputs": [ { "data": { "text/plain": [ "0.8224489795918367" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "accuracy_score(test['Survival'],predictions)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is possible to tune the decision tree parameters based on the accuracy measure. For example the max_leaf_nodes can be tuned to find the maximum accuracy." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### K-fold cross validation" ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.476261Z", "start_time": "2018-07-31T02:47:02.473263Z" } }, "outputs": [], "source": [ "from sklearn.model_selection import cross_val_predict, cross_val_score" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.485255Z", "start_time": "2018-07-31T02:47:02.478260Z" } }, "outputs": [], "source": [ "X = training_data[['Age', 'Crew/Passenger', 'Class', 'Adult/Child', 'Gender']]\n", "y = training_data[['Survival']]" ] }, { "cell_type": "code", "execution_count": 56, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.514239Z", "start_time": "2018-07-31T02:47:02.487255Z" } }, "outputs": [ { "data": { "text/plain": [ "array([0.82244898, 0.81428571, 0.79795918, 0.78367347, 0.81799591])" ] }, "execution_count": 56, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cross_val_score(clf, X, y, cv=5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Ensemble learning to avoid over fitting" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ensemble learning relies on using multiple instances of a machine learning algorithm (i.e. decision tree) that are trained with different subsets of the available training data. Also, the algorithm parameters and number of used features may be different for each instance.\n", "\n", "Each instance might be over/under fitted based on used data. However, the ensemble response is determined based on a majority rule decision.\n", "\n", "In other words, an ensemble may be defined as a collection of models (even different types) that solve the same problem. However, the final answer is achieved by a combining the response of all the models.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Random Forest\n", "\n", "A collection of decision trees are built and trained independently and the combination provides the final response.\n", "\n", "- Each tree in ensemble is built from a different subset of the training set (using Bagging (bootstrap aggregating) technique)\n", "\n", "- Each tree in ensemble is built using a different subset of features (using random subspace technique)\n" ] }, { "cell_type": "code", "execution_count": 90, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.541219Z", "start_time": "2018-07-31T02:47:02.516237Z" } }, "outputs": [], "source": [ "from sklearn.ensemble import RandomForestClassifier" ] }, { "cell_type": "code", "execution_count": 91, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.549215Z", "start_time": "2018-07-31T02:47:02.543219Z" } }, "outputs": [ { "data": { "text/plain": [ "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n", " max_depth=None, max_features='auto', max_leaf_nodes=None,\n", " min_impurity_decrease=0.0, min_impurity_split=None,\n", " min_samples_leaf=1, min_samples_split=2,\n", " min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n", " oob_score=False, random_state=None, verbose=0,\n", " warm_start=False)" ] }, "execution_count": 91, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf = RandomForestClassifier()\n", "clf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Most of the parameters are related to settings of a decision tree. Settings related to the ensemble learning (i.e. the random forest) are explained below.\n", "\n", "- bootstrap: Using the bootstrap technique to randomly select different subsets of data for each tree\n", "- n_estimators: number of trees" ] }, { "cell_type": "code", "execution_count": 92, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy : 0.81\n" ] } ], "source": [ "clf.fit(X = train[['Age', 'Crew/Passenger', 'Class', 'Adult/Child', 'Gender']], y=train[['Survival']])\n", "predictions = clf.predict(test.drop(['Survival'], axis=1))\n", "print('Accuracy : %.2f'%(accuracy_score(test['Survival'], predictions)))" ] }, { "cell_type": "code", "execution_count": 93, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Importance for Age : 0.450\n", "Importance for Crew/Passenger : 0.052\n", "Importance for Class : 0.185\n", "Importance for Adult/Child : 0.009\n", "Importance for Gender : 0.304\n" ] } ], "source": [ "for feature, importance in zip(['Age', 'Crew/Passenger', 'Class', 'Adult/Child', 'Gender'],clf.feature_importances_):\n", " print('Importance for {} : {:.3f}'.format(feature,importance))" ] }, { "cell_type": "code", "execution_count": 94, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:02.557209Z", "start_time": "2018-07-31T02:47:02.552214Z" } }, "outputs": [], "source": [ "# Using the train/test data\n", "def checkAccuracy(n_estimators=10):\n", " clf = RandomForestClassifier(n_estimators=n_estimators, criterion='entropy')\n", " X = train[['Age', 'Crew/Passenger', 'Class', 'Adult/Child', 'Gender']]\n", " y=train[['Survival']]\n", " return np.mean(cross_val_score(clf, X, y, cv=5).mean())" ] }, { "cell_type": "code", "execution_count": 95, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:05.325497Z", "start_time": "2018-07-31T02:47:02.596187Z" }, "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Number of treesAccuracyScore
010.773856
1110.789191
2210.792767
3310.789701
4410.788175
\n", "
" ], "text/plain": [ " Number of trees AccuracyScore\n", "0 1 0.773856\n", "1 11 0.789191\n", "2 21 0.792767\n", "3 31 0.789701\n", "4 41 0.788175" ] }, "execution_count": 95, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result ={'Number of trees':[], 'AccuracyScore': []}\n", "for n_tree in np.arange(1, 500, 10):\n", " result['Number of trees'].append(n_tree)\n", " result['AccuracyScore'].append(checkAccuracy(n_estimators=n_tree))\n", " \n", "result_df = pd.DataFrame.from_dict(result)\n", "result_df.head()" ] }, { "cell_type": "code", "execution_count": 96, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:05.557352Z", "start_time": "2018-07-31T02:47:05.327496Z" } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 96, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmkAAAFACAYAAADnOuuxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xl8VNXd+PHPyUYChEACsm+yIySsioqA4gLVqqhV0FZttdr+qrY+z2MffGrV2lWftj5q7UKr1VoLKlXqglvFCCIIiOxrQJCwhiWBBBKynN8f507unclMcme5s8D3/Xrlxb137sy9w01mvvec8/0epbVGCCGEEEIkl7REn4AQQgghhGhKgjQhhBBCiCQkQZoQQgghRBKSIE0IIYQQIglJkCaEEEIIkYQkSBNCCCGESEISpAkhhBBCJCEJ0oQQQgghkpAEaUIIIYQQSSgj0ScQCx07dtR9+vTx9BhVVVW0adPG02OIyMi1SW5yfZKXXJvkJtcneUV7bT777LODWutOLe13SgRpffr0YcWKFZ4eo7i4mEmTJnl6DBEZuTbJTa5P8pJrk9zk+iSvaK+NUmqnm/2ku1MIIYQQIglJkCaEEEIIkYQkSBNCCCGESEKnxJg0IYQQ4lRXW1tLaWkp1dXViT6V015eXh4bN25scb/s7Gx69OhBZmZmRMeRIE0IIYRIAaWlpeTm5tKnTx+UUok+ndPasWPHyM3NbXYfrTWHDh2itLSUvn37RnQc6e4UQgghUkB1dTUFBQUSoKUIpRQFBQVRtXxKkCaEEEKkCAnQUku010uCNCGEEEKIJCRBmoiN+pNQdyLRZyGEEMJjr732GkopNm3alOhTaVFDQwP33HMPw4YNY/jw4YwdO5Yvvvgi0aflmgRpInqV2+G1rubnyJpEn40QQggPzZ49m/HjxzNnzhzPjlFfXx+T13nppZfYs2cPa9asYe3atbz22mu0b98+qtesq6uLybm5IUGaiN62v8LJw1BbAdufS/TZCCGE8EhlZSWLFy/mmWee8QvSHnvsMYYPH05RUREzZ84EoKSkhIsvvpiioiJGjRrFtm3bKC4u5oorrmh83l133cVzzz0HmCkeH3nkEcaPH88rr7zCn//8Z8aOHUtRURHXXnstx48fB2D//v1MmzaNoqIiioqK+OSTT/jxj3/ME0880fi6P/rRj3jyySfZu3cvXbt2JS3NhDs9evSgQ4cOALzzzjuMGjWKoqIiJk+eDMDhw4e5+uqrKSwsZNy4caxZYxoeHn74Ye644w4uvfRSbr75Zurr67nvvvsYO3YshYWF/OlPf/Lk/1tKcIjoHd1gL1ftSNhpCCHEaeMfHiYQ3KhDPjRv3jymTJnCwIEDyc/PZ+XKlezfv5958+bx6aef0rp1aw4fPgzATTfdxMyZM5k2bRrV1dU0NDSwa9euZg+dnZ3Nxx9/DMChQ4f49re/DcADDzzAM888w913380999zDxIkTee2116ivr6eyspJu3bpxzTXX8P3vf5+GhgbmzJnDsmXLOHHiBOPHj2fRokVMnjyZr3/964wcOZKysjK+/e1vs3DhQvr27dt4zg899BAjR45k3rx5LFiwgJtvvplVq1YB8Nlnn/Hxxx+Tk5PDk08+SV5eHsuXL6empobzzz+fSy+9NOJSG6FIkCaid9QxLkGCNCGEOGXNnj2bH/zgBwBMnz6d2bNn09DQwDe/+U1at24NQH5+PseOHWP37t1MmzYNMMGXGzfccEPj8rp163jggQcoLy+nsrKSyy67DIAFCxbwt7/9DYD09HTy8vLIy8ujoKCAzz//nP379zNy5EgKCgoA2Lx5MwsWLGDBggVMnjyZV155hePHjzNhwoTGoCo/Px+Ajz/+mH/+858AXHTRRRw6dIiKigoArrzySnJychrPYcOGDcydOxeAiooKtm7dKkGaSDINdXBsq71etTNx5yKEEMIzhw4dYsGCBaxbtw6lFPX19SiluPbaa5uUmtA6eGtcRkYGDQ0NjeuBNcTatGnTuHzrrbcyb948ioqKeO655yguLm72/G6//Xaee+459u3bx7e+9a3G7a1atWLq1KlMnTqVzp07M2/ePC655JKg5TGCnbdvP+e5aa156qmnGgNHr8iYNBGdym3QUGuvnzwMtccSdz5CCHE6uFF79xPC3Llzufnmm9m5cyc7duxg165d9O3bl/z8fJ599tnGMWOHDx+mXbt29OjRg3nz5gFQU1PD8ePH6d27Nxs2bKCmpoaKigo++OCDkMc7duwYXbt2pba2lhdffLFx++TJk/nDH/4AmASDo0ePAjBt2jTeeecdli9f3hg8rVy5kj179gAm03PNmjX07t2bc889l48++qgx09PX3TlhwoTGYxUXF9OxY0fatWvX5Nx851Bba77/tmzZQlVVVUtXLWwSpCXKFy/A8v+X+i1PFUHmLkv19ySEEKKJ2bNnN3Zf+lx77bXs2bOHK6+8kjFjxjBixAh+/etfA/DCCy/w5JNPUlhYyHnnnce+ffvo2bMn119/PYWFhdx0002MHDky5PF++tOfcs4553DJJZcwePDgxu1PPPEEH374IcOHD2f06NGsX78egKysLC688EKuv/560tPTAThw4ABf/epXGTZsGIWFhWRkZHDXXXfRqVMnZs2axTXXXENRUVFjN+vDDz/MihUrKCwsZObMmTz//PNBz+2WW25h6NChjBo1imHDhnHnnXd6kvWpQjVJppIxY8boFStWeHqM4uJiJk2aFJsXq9gAbw0DNHS7Aia9EZvXTYT1v4TV/+O/beKb0P3yuJ1CTK+NiDm5PslLrk1yC7w+GzduZMiQIYk7oSTX0NDAqFGjeOWVVxgwYICnx3Izd6dPsOumlPpMaz2mpee6aklTSk1RSm1WSpUopWYGefxxpdQq62eLUqrc8dijSql11s8Nju0vWq+5Tin1rFIq09o+SSlV4Xi9B92cY0rZ92/ACo73vZ/aRWCDtqTtiPtpCCGEOH1t2LCB/v37M3nyZM8DtHhqMXFAKZUOPA1cApQCy5VSr2utG+suaK3vdex/NzDSWr4cGAWMAFoBHyml3tZaHwVeBL5uPe0fwO3AH6z1RVpru5DKqabsE3u5oQYOLoYuFyfufKJxVLo7hRBCJNbQoUPZvn17ok8j5ty0pJ0NlGitt2utTwJzgKua2X8GMNtaHgp8pLWu01pXAauBKQBa6/naAiwDekT6JlLOwcX+6/v+nZjziJbW/uU3fKQlTQghPHEqDFE6nUR7vdyU4OgOOKvPlQLnBNtRKdUb6AsssDatBh5SSv0WaA1cCGwIeE4m8A3g+47N5yqlVgN7gP/SWq8Pcqw7gDsAOnfu3GJqbrQqKytjcoxW9Qc493ip37ZjW1/js/IpUb92vLWqP8C5dZVNth/du5aVHl8Pp1hdG+ENuT7JS65Ncgu8Pm3btqW0tJS8vLyg5SNE/NTX13PsWPOVDLTWVFRUUFVVFfHfmZsgLdhvQqjQcDowV2tdb53ge0qpscAnQBmwBAhMf/g9sFBrvchaXwn01lpXKqW+AswDmnQwa61nAbPAJA54Pfg1ZgNsd74E+/035dZuZdK5w6FVQfSvH09737PfS+uecNzE8u3Sj8R1MLIng5+rD0JWHqRlxvZ1T0Mxvz71J+FEKbQ9M3avGWtaw5HPoXUvyO6Y6LMJSRIHXKjaBTndIC097ocOvD61tbWUlpaye/fuuJ+L8FddXe2qQG92djZFRUVkZkb2XeImSCsFejrWe2BauIKZDnzPuUFr/XPg5wBKqX8AjZVPlVIPAZ2AOx37H3Usz1dK/V4p1VFrfdDFuSY/53i0Rhr2fwi9rov76UTFmTTQZTJ88XfQdVC93yRDZOQk7tyi8cWLsPRmE3hetiKpv2RPO3Un4J3RZizk8Idh+EOJPqPg1j4M6x6BnK4wZSXkdEn0GYlIrPwv2PQbaNMXxv4BunlbuLQlmZmZMa9oLyJTXFzcbPmQWHEzJm05MEAp1VcplYUJxF4P3EkpNQjogGkt821LV0oVWMuFQCHwnrV+O3AZMENr3eB4ThdlteMqpc62zvFQZG8vCR10BGn5juzbVByX5kwayBsGrR3DCo9/Gf/ziZWSP4JuMAkQ22Yl+myE05637N+7rX8wLVbJaMffzb8n9sK6nyb2XERkGmph6+/NctUXUDwFFt8IJ/Yl9rzEaaXFIE1rXQfcBbwLbARe1lqvV0o9opS60rHrDGCO9h8llwksUkptwHRNft16PYA/Ap2BJQGlNq4D1llj0p4EputTZaRkXZXpAvEZ9oC9nJJBmiNpoN0QaNPbXk/VDE+toXytvV7yZxOwieSw61V7uXq/+fJMNiePQKUjy6xklv+6SA1HVkF9QHmknbPhzSHmmsrngogDV3N3aq3nA/MDtj0YsP5wkOdVYzI8g71m0GNrrX8H/M7NeaWcQ8vBDNeDvLOg6xRIbw31x830SpVfQNsUasr2a0kbAm36AB+Z9VTN8Dy+C2or7PWqHSaA7nppwk5JWOprYPeb/tvKFiff2LTDK/3XdR2seRDO+3tizkdExjk0pVUB1FgdOrXlsOxO+OJvMPZP0P6sxJyfOC3ItFDx5Ozq7HgepLeCMybY2/aFnsMs6dQchuoDZjk927SinQotaeVrmm4rkS7PpLB/AdQFZFMFHeOZYIc/a7ptxz/gSJDfLZG8nJ/Xwx+BC9+Dtv3sbWWL4Z2RsPqB1C5ILpKaBGnx5PxC6XS++ddZxDaVujydrWi5g0Cl+QdplTvifkox4ezq9Cn9l4xDSQbOrk6fg0kepGW0sRY0rP5RQk5HREBrE4T5dDoPul4CX1kLZ/0PKKsjqKEW1v8c5hem1k22SBkSpMWLbmjakgb+Qdr+D1JnnENFQFcnWN2dluOnQEuasv48dB1sfy4hpyMsDfUmWA5UvhZOVjTdnkjOIG30kzRWMdrzpv8Xv0hex3fBCavMRUZbyBtuLedA0c9h6ir7MxygsgQWXAxLvwUNsZ9kW5y+JEiLl6ObzYBigFYdIbe/WW4/HFp1Mss1B4N3tyUjZ0taO1+Qdip0dzpa0vp/x17eJgkECXVwMdSUmeXsLtC+yHpAw6FPE3ZaTZwsN+NLwdTY63MT9LnRfnzV/cmbkSpszl6PjuOa1khrfxZcsgjG/hEy8+zt2/8KW56KzzmK04IEafES2Irmqxat0kyNMZ9U6fIM1pLWuieNrQbHd5uio6mkvsYE0z7DH4asDma5crsZEyUSY9dr9nKPq6HTeHs9mcalObO384aZcafDf2J3j5Utgr3vJObchHvOqfucLWZOKg0G3AlXbIKe19jbk7k0jEg5EqTFS7DxaD6pOC4tWEtaepapzA2ANlXhU8nRTaZrE0zxyuxO0Pdm+3FJIEgMrf3Ho/Wc5v83FDgXbiI5uzrzR5t/c/tB/zvs7av/R1plk11zn9eBcrrAuX+DzHZm/dhWOPCRd+cmTisSpMVLc3dmziDtwELTopPM6o7b3ZkqDXIds3alcpens6uzvTUGpd+37W27XoMTAXN6Ce8d+dwujpzZHs6YZAZy+xxcasasJYNgQRqYmojp1gwcR1bBzpfje17CvdpKKF9trSgoCDpVtb+MNtDn6/a63NCJGJEgLR6qD9rdaGmZ/h/eYAKbttYYtfoT5ksnmR3dTOP0rW37mS4dH2fyQKplePoFaYXWv2fZd9K6Dr54Pv7ndbpztqJ1v8K02LbuZbfa1lVCxbrEnFugUEFaTlcY9AN7fc2PTWagSD6Hltn1LNsPM3P4uuFsLd31T/O5L0SUJEiLh0OOoKvDqOBzWqbSuLRgXZ0+Kd2S5kja8LWkAfRzfPjGYwaCyi/sJJNkVVdlbibKPoGyJWb54KdwcJn5ObQcDq2Aw5/RpnZ7dBlvpY7xaL6xP0oFdHkmwbi0kxWmqwvMGDTn7xDA0PtMSyCYbMBtz8b3/KLRUA9Ht54eY638xg+30NXp1KEICs42yw0nTbFbIaIkQVo8uBnfkErj0oIlDfj4BWk74nI6MROsuxOg19f8v1z3F3t3Dht/A6/3Mz/JGOTqBtj6J3itB7x3Lrx/Prx/nll+bxy8d475efdseHcsvDOGsWW3wXvnRZZIcnQzVGwwy+k50NUxwbVz2EAylLZwJg20H2aKPDtldYCzZtrr6x4xQweS3f4P4a2h8OZA+Pj6RJ+N9/w+r0MkDYTS/057edus0yOoFZ6SIC0e3GQKdb6QxszIw8uSr/aTU+CcnU7O7s5kDDJCqTls10VKa+U/zi4jB/p+w173arzJnrfh8/sAbVrStj3jzXEiVb4W3h8Py79jpsYJx+HlsO0v4R/TmdXZdQpktLbX/YK0JGhJc3Z1dhgVfJ+Bd5uuT4ATe2BLEs+AV3MIln4TPrgIjm0x23bNTe7PpmjpBji4xF4P9XkdSu8bICPXLB/dbMYYCxEFCdK81lBrxjj4dDw3+H6tCiDf+mDXDXCg2PNTi9ip2N3pbEXLGwppAVPL9nckEJS+CtVlsT3+sW2w+EYax/pB8Ar7iVBXBZ//N7w9yv8LLKe7GVRdcI7p5skfC/ljrJ/RJlBxBrvrHjGvFQ6/rM5r/B/rMMJurar6Ak7sDe+1Yy3UeDSnjNYwzDHt8YZfmdpqyURr2P43eHNw8CLOyTL+zwsVG+0bkOwzwp8XNqMN9JUEAhE7EqR57cgqqK82y236QOtuofdNhS7Phjr7rhogb7D/484g7fiu5Mm6a0mwpAGn9sPtALuhNrYJBHVVsGha09apivVwdEvw58TL7vnw1jDY+JhdniQtE876EXx1K1y21Pr5FKYsgynLrZ8VMPUzmLqamrSO5nnV+2HzE+6PXbXLtMCBGePV/XL/x9Oz7DFAkPjWNDdBGkC/2+w5IE8egY3/6+15hePoVlhwCSy9xRTX9nEWbE2VgtuRCByP5qtnGQ6/BIK59sTsQkRAgjSvhVNvJxWCtMrtdlZaTje7NpBPRo65AwXzpX5iT3zPL1KhkgacnB++JTEab6I1LL3NDhLTsvyDROeg+Xg6vseMP/rocv+xhZ0uMFPiFP0seAJMoIwcduTeaq9veNT9l1bpPHu580V2YWEnZ3dUIpMHao/aNy8qPXig75OWCYU/tdc3/V/i54atPwnrfg7zh5vp6Xxa94SJb8BZ99vbgs1ve6o4GMV4NJ8OI0yrMkgCgYiaBGleczMerfHx8814KDDjvo4nYTHY5ro6fVqnYPJAqKQBp17X2y0KsSpYuem38OVL9vrY38Pg/7TXd8U5SGuohy1Pw1tD4MtX7O1Z+XDOM3BxsekODsO+1lMgd6BZqT1qAjU3AgvYBpMs49IOO2caOKvlALb3DfbUVvXHYd3PvDu3lpQthndGwZoHoMGq0ajSYNC9cPkGU/bEGXSeykFaWRif183x4oZOnJYkSPOS1v5/9C3dmWXk+E93s++D0PsmSnOZnT5t+9jLqTAuTTf4j7MJ1QqS0Tq2BSv3fQCrfmiv9/+O6QrrfoVpjQEzL+Xx3dEdx62TR0y25oq7TDDl0/cWM/VNv2/Zk86HQat0Mym1z5anWr4BqT4IZb5B1wp6XBV8P+cYzyOf2UML4s1tV6ePSoOiX9jr22aZVup40pp+FU+bZJCK9fb2DqPgsmUw+reQ2dZsc964lK+NPujQGtb+BD6+Aaq+jO61fKoPwidfh8/+I7KSL9VldgmVtCx31zGU3tPNxOxgbrjLPo78tRKhYhMsngHrfwl1J6J/Pd1gEqEW35j4YQkpRoI0Lx3fZXf3ZbQ1c/m1JNm7PN20pKVa8kDVDlMQFaBVR8juHHpfZwJBNAUrq3bC4hvsmmsdz4XR1nitVvlWtq/F2e3npdUP+E9WnjsQJi+Ac58zU2RFo+e19pdefTWsfaT5/Xe/7v9/48uIDJTdEdoNMssNtaY2WyKEG6QBdJtq35Q11MKSm+M73+2Of9Czaq69ntEGRv3WjC8MfA853e0yNLUV5rMtGgeKYe3D8OXLsOw70b2Wz6ofwo4XYfPjUPKn8J/vTIrJH+NfpDtcmW2hz032eiolEDTUw8fXwc45Zgqz+YXRNRiUr4X3zodPb4eds2Hhlf43gaJZEqR5yXnH0HFc04zBYAKDtGRrJnfTkuZXhmOHl2cTG4FJA80NFo5Fwcq6E7DwGntsVnYXGD/XDIT36eHo3otHlmdDvRnk7DP0fvjKGv9gMRpKwYhf2evbn/WfzD7QriAFbENJhnFpRyII0pSCkf9rt06WLYaV98b+3EIp+aO93OUS07U5+N7gn1NKQQdnl2eUyQMHHC1Le9+JfnaSk+UmqPAp+VP4n53h9Hq44ezy/PIVU+YnFex40b9ltbIEFlwMn9wcXlZ73XFYNdNkhTsLutccMvUghSsSpHkpnPFoPh1G2gOkq/fZhTyTgdbN10jzSbWWtCMukgacnB++4Ras1NrUGTuy0qyrDLhgbtOs3x5X28sHPvI+Q+zgEqg+YJazzzAD26NpSQimy8XQ2ZpZQ9ebqZGCqT0G+96z10ONR/NJdJBWe8zOwlXp9lgzNzqOgyJH8Lr19/GZiaBig90FpzLg3BegTa/mn5MX0OUZDWdQi46+JuCOF82Uej7la/1bhd2IdKaBUPJHmRY5MGP9UiGBoL4G1jpKxDiHN+x4wYxV3f5cy595e941WeEbHrWzwnHc/G76jf15I5olQZqX/FrSXAZpaekmk80nmbo8T+yGumNmObN96G7BVAvSKlwkDTj1CihYWbbI/bG2PO3/YT3myeBZv627QcE4s6zrYfcb7o8RCWcWaY+rze+hF0b80l7+8pXg3ZN73jatlGACnpZqVTn//8o+iX/r85HPaaxvlzfUXdar05D/Mr9TPsu/a6bW8lLJn+3lHldDTjNd/D6B49Ki4eweBtj+TORTh2kdvHsznC7G+pNmKjOfUPUsw5VqCQQlf7I/s1t1hMs3+f9u+gocL5gcvDzQif1m3FnxFFO70KfTBfCVtfbvUF2VySYWLZIgzSu1lVC+2lpR5o7ZrWQdlxbY1RmqWzAwSPN6rstotVQjLVCT8SZ/Dr2v04FF/t1ZZ37TJAuE4uzm8zLLU2v/LtUeLbRcRaNgLPS8zl5f/T9N93GT1enUbpDd+lxTBsdKojvHcB1eaS9HMthcKRj3jP0F1nASFl1jvvC8UF/tX+fPGUg0p32MujurDzRNHDmxF/a8FdnrHVpm/w0rx83FzjnuZ0c4stLObG3bz13Q6oZfAsHG5Ji+LJTaY/5Zxmc9AO0GwPg5MGm+/+f6/g/NWLW1PzUBrm4wn4NvDjbjznyyOsA5fzFZ4e3P8k+WKflD9N3cpwEJ0rxyeLlpAQEzj19WXvP7OzmDtAPFdl2yRHOTNACmdprvS7OhJrmbtetOOIrzKlM+wY3A8SZli02r0KHl9iTjBz+1JiFfAvsWwMdfs5v+88eYchvNjX9zBih73zWBvxfKV9tjBzPb+bfkeqHoZ/aX6b73/Qcl11f7f1m3NB4NTJeMs+Uj3l2eftNBRZgRmNEGJsyz/25O7Da/L14kEnz5T5PJC5xI7wpdJrt7XntH4tPRzZGfW2Armk+kg+udz+v7DTuYrD8BO//h7jXCqWcZjsxc6HOjvZ7MCQSbfmtucgBa94IBjhvIblPh8vWm1df3t9tgdY2+PQL+PQGW3eFfkLvPTVZW+G12t2m3y+1epYZaWPuQ9+8rxUmQ5pVo6u207WfftdRV+k8rlUjO8WihkgZ8UmWi9aMb7Za+tv3854ZsTv5I//Em7483E4q/e7Y9yfh746xJyM8z3QPVVstIq05wwatNJ+AOlNvf0bpSYwZYe8HZStftCv8EBi+0G2RaEX1W3W93A+37wM60bdvPXUY0JHZcWiSZncG0PRPOn+NIJFgEn/9n88+JxDY7UNjb+nL3ZVUyc6FNX7Os6/w/D8Lh/P/q9hV7ec/b4ZfjOFnhnzDQ/07/G6itLhMIDkYwNMUtvxu6l5MzgaC6DDb+2l4vfKTpmNSMNibRZcoK+7MPmrYQtj0TLnwXzvu7XdjcJzCB6IsXoPwUnmYsBlz9dSqlpiilNiulSpRSM4M8/rhSapX1s0UpVe547FGl1Drr5wbH9r5KqU+VUluVUi8ppbKs7a2s9RLr8T7Rv80EiGQ8mo9SydnlWeGyJQ1SZ6J1Z1dnBxddnU797wz/eCodxr8MbXq62z8eWZ7hdi/GwvCH7MLNh5fbY+JKA7I63U7L4zcuLY5dSrWVdrCi0kz2bzS6XurfJbTld8Hnz4xUxSZ70m+Vwb7WU8N7vt+4tAi7PJ1BWq+vmcxSIKIEgp3/MMWAfedWcI5pwUm3xgWWr4bDLZRlCbeeZbh889iCudna8ffYvn4srP+FfXOUN9S/HmSgDiPg0qWmZJCvKxdMAsrQ++Er68zvcShnXOAIzjWs/lHUp38qazFIU0qlA08DU4GhwAyllF/Jca31vVrrEVrrEcBTwKvWcy8HRgEjgHOA+5RSvnmEHgUe11oPAI4At1nbbwOOaK37A49b+6UW3eBfcyeSP/rOSRikHXVRfsMnVZIHnF80eS6SBpzO/CYMvMdk5HYYaT6I80dbP2PM1DAFZ9uTkJ8xEc6fDZ0nuT+Gs7tvz1sm+yqWjpXYhXzTs6HrlNi+fiite8Cgu+311T8y7630X/a2cMbGFYy1u2Eq1sdv0vIjq2hMGmg31H1LbHOG/NAELz7LvuM/qD0a25wJA1dyMj0/vOfHInkgsOXR2dK0/Vn3CQRam5Yyn353mKA+q72ZHcSnpS7Gqh0mkx5Md7/bIQ/hSOYEgqqdJqvYp+gXLScOpaXDoHtM2Zb+d5ixd1NXwohfuEuccd6I7H5dCtw2w01L2tlAidZ6u9b6JDAHCFH+G4AZgG/k4FDgI611nda6ClgNTFFKKeAiwFeY6XnAV3PgKmsd6/HJ1v6p4+gmu28++wx7MuVwdHGMCzq41AzqTKSTR+zuurRW/lM/BZMqtdKiaUlLS4cxT5gPp6krzYTiU1ZYP8vNhOOXfWpPQn5xsf+XrxvtC+0uptqjsH9BeM9vibOrs8uldoX5eBg605779egmM6bFN6l3TlfoeI7718poY+7wfQ4uDb3IpFtJAAAgAElEQVRvLMWqq9NJKTjnWburt6HGJBJEO7YzMGGgn8uEAadokweqy+xCuOk5pkW++5V2t9jxUvfd+odX2MlZ6dnQ19H64wyKds5uvniqX6/HuRHNqtGiPjPM7yiYmwjnTXyirX3YzqYuGGeuh1ttesLZfzI3n24y4306FEFvx1i9VTOTK3BNIi6qq9IdcJaXLsW0ijWhlOoN9AV83ySrgYeUUr8FWgMXAhuAAqBc68YCKqXWcfyOp7WuU0pVWPv7lXZXSt0B3AHQuXNniouLXbyVyFVWVro+RteqN7FqoFPGQNZ/FNkcj2My+tG2bhvoOtZ88DsOZ8coLTwC7U6uw2qwpzKtOysWNl92ouOJY/hGEx36ciVrq4o9O7dwrk2g8w6swDcC69PNxzmxLbLX8VI/xtITk86+Z9nTbGkfZomHZowsew5fSsumqiHs8+DvqLnr0yv7a5xZa3VxOUqT7FZj2frRwqDPCaV/TW96YIKmHSv+wY52LYz5i4HBR96ii7W89VAuu2P4/5fdaiaj1XfI1JVwvJTyNy9ldcGv0crNx3ZTZxz/gKFWvb3q9M4s3ZxJZVV4fzuta09glXKmZv8KloT5fvOrl+EL8yrS+vL5QlOr7cyMyfSy7u0PLv0l6wpavlkYWP5rfNUF92VNZNMnq+wHtWZsRh/a1O2Auio2v/cwe9sEDz4GlL/c+OXzRWVXdnr0XTIwaxLd6kxSzL6PH2FThyYjh5qI5rPNjda1XzC27G+NFcw+5wYqIvy+Cld23Vc4m5dIox7KFrHm3cc4nB3GjVmCeX1tfNz8tQdrxQoV8k4H5mpt0hq11u8ppcYCnwBlwBKgroXXdHU8rfUsYBbAmDFj9KRJk5p5C9ErLi7G9TGWPg9W5nenwVcyaajL5wVaebUp+gcUdjoAoyN8nVjYtq0xTG7bbSyTxrdwLofbwTumKGJBdqX7/7sIhHVtnKrL4FWT5UZ6DudcNMO7+mDRKMuE918GoFv9MrpNuCA253l8D8yziiWrdAZPvo/BrQqif90AzV6f2jHwxpt2K62l+7i76d4lxHNC2bkfFpvxdX3a7KaPx58JALz1PbBqqA44ZzoDYpkZCLCnAIq/Amjan1zNxNw3TOttJP79k8bF7LPuZtKwi8L/22mog5fvhIYaWjUcZNK5hWYaM7fWLQZr3HxenwuZNNY69rEe8IYJ0jqeXMqks/ubLvFQao/Ca8WNq13GP0iXwGElm++Fz74PwKD0hQya9NvgrzX/B2ANa+s79ib6hvt759ahtvCuCdK6nPyILufNtrN5Q4j4s82thU8AVuJU16mMvPAH3h0rmOWfNHa1FjbMhon3edOS6QHPr43Fzf9GKeAc5dwD2BNi3+nYXZ0AaK1/bo1XuwQTgG3FfN23V6rxltD5mo3Hsx7Po/HPOkX4pXNHMQjVmTywP8GTrbuZDsopsLszGZuynV2decOSM0AD0wWTbbXX1JTFLnvROSfoGRPBgwCtRZltYVjAzANZHcz5hMuZoHPoU/djm+qqTIbp5/eFN5l0XVVA0sCI5vePRLcpAZPTPwnbI6hcf3SLKecDZuyeM7s2HGkZZmC5T7jj0kJ1D+f2d8xG0dDyrAs7Zpv/fzBjyIIVn+3zdTuD+vBnwUt/1B61i1mrNDN21Cv5o83YVTBdz/+eCP++0Px8cJH1M9nxczGFh34I2/7qzfkcXOr/GTDiF6H39cqwByDdGsdZvhp2vhT/c0hyboK05cAAKxszCxOIvR64k1JqENAB01rm25aulCqwlguBQuA9rbUGPgR8VS1vAXwjhl+31rEeX2DtnxqqD9p1t9KyohuncsYFkJZplsvXQvn65vf3ktsaaT5ZHezMn7oqOJmEcXZ5mNNBJYpKgx6OYaCxyvL0m2UgTlmdwfT7tj3uDqD7V+3f+3C06Wm3vtRVuQsgtIYlt8CGX5kSBKv+2/3xjqyyy7e0G2yPOYq1oTP9CwAvv9O/gK4bzoSB7l9tOg1ZOKJJHmhuDJ/fdGt/MfPJhuJMBuh/R/As4Fb5/v9vwYpOH/zUvobtC02ZEa8o5f8ey9eawPlAsSkOu/9DM+a08ecD8muWw6ffMrUWY0lrMw7Mp/cMb24yWpLTFQY7Wu/WPOBNbcAU1mKQZo0buwt4F9gIvKy1Xq+UekQp5ezknwHMCQioMoFFSqkNmK7JrzvGof038B9KqRLMmDNf7vUzQIG1/T+Aljvuk4mzlSN/dMu1sJqT0cY/227NA5G/VrTCbUlTKvlrpYU700AiObM8S1+LvmWy5jDsL3a8/tUhd/VcehaMecrc1KRnm6yxSHUMsxTHhl/Brn/a6yV/hMrt7o4ViyK2bigF4/5qZx3WV8PCaeaG0I36Gv8yHm5nGAgl0uSB6jI4btVBS8/2b5EDMz1Vq05m+fguU8A5mMOf2XPfpmc3Xy7C+V53vNi0ILSX9dGC6XNTy9OcBbM6xgPr975n5gQGUzqj8JHYvXa4htxnd/tWbjdThIlGrkagaq3nA/MDtj0YsP5wkOdVYzI8g73mdmgcgxr4nDBT4JJIrP/oh//EnrexdJ5pog5niqlYqDthB1kqDXIHuntemz4mkwlMmnesst9iJVVa0gDOmASZeVBbYf4vj3xuJnCO1O437dkPCs5ufvxPPHS/HK7YbAK1aFp5Op0HX1pdJgc/gUF3hd53z9tNazQ11MKah+C8F1o+lheZnaFktoULXjMFk2srTLCz+AZTNDSthY/x0nl21mzrXiaLNxqRtqQ5/7/aFzU97/QsOPMWu6jqtlnQ/Ss04WxF6/m15sfEdRpvWjmPbjJ1wHbOgf632497NdNAKJm5Zg7Lw59ZM9I4Aq/GIMz6t7aChkXXm4H1BxaarNduYda1C0Y3wOr77fX+3zbdzYmS1d7UV1v1Q7O+9hHoe7N3LdMpJjVG6KWSWI1H88kfaWrQ+CQiVfnYZho/ONqc2bQSdSjJXCutod4OICH5g7T0LNNN5RPtXJ6BRWOTQds+0QVo4H7mgWMlZiLoxhpng+3HdrwIR1y0EMUzSAMzj+J5L9KYW7V/gf+XbSjOoKbf7dGPvXS2pFWscz83r5v/r37ftpd3vwnHd/s/XlsJOxxTPfX/Ns0K7GJ0/l801PuXwohHSxqYWnpnXGDqJXa+0P7pcpH1M9n89LyGva2vsJ+36v7YzIP85SvmJg9MGZTAMaGJMPAuyLH+9qv3weanEns+SUSCtFiqP2mqp/vE6o++8KemSRpME3WobgCvhNvV6eNMHki2iXQrt5u5/cAMys/ulNjzccM5G0BpFOPS6qr8f4cSOR4t1joU2QORq3Y2/ZIH80W/8Gq7lmHrHjC52EyJBbiqgl5X5RinqeI3nqf75TD8YXt9469hx5yQu3N0q11bT6VBvwgTBpyyO0Orjma5rtL9UAY3QVq7gabVGExL0/aAQfM759iV8dsNNi1lLel7s2mhBfP5fMQq1VGxHuqs+pM5Xf1vKpPEztxvBAysb+Zau9FQC6sdw2YG/cC890TLyDGzkPhseLRxftnTnQRpsXTkczNeBMy4g5wuze/vVm5//zvGVTNjc0fllnOOPjdJAz7OD73jSdaSlkpdnT5dL7Onu6nYYCa5jsTed+0ANW+o+WI8VaRlmu5bn8DWNK1h6TftVtS0VmYe1ZzOVnab1Uq150048HHo4xxZ7Z80EM8iwMMe8C84+ultoVv+tv3FXu52eWy6tZWKrMvTbctjYAKB87POTcJAoFYF0PNax2tYCQR+Q1POdz8FWRydTC8IGFj/4+gG1m97BipLzHJWBxj6w+hOMJbO/CbkDjDLteUmUIsVreHEvvh+b8aIBGmx5OUg1GE/TlyqcjjTQTklc3dnKiUN+GS0MYGaT6Rdns7n9UiSrs5Ycg4zCJxuZsOjsGuuvT72D2ZKKTCBh3MQenODtePd1emk0uDcv9ljQ+uPw6JpTSfurj/p3xIVbcKAk1/ygIsgrfqgnTSQ1qpp0oBTz2l2OZiqnbD3fbN8+HO7pyIty7SQueWXQPB30xIa66EpXgkcWO8MvMNRXQZr7Vp5DJ1pxoMli7RMKPyZvb75CdMSHK36k/DRV+G1rrDk5uQsB9UMCdJiydnilD8mtq+dyFTlcCZWd0rm7s4KZ5CWIi1p4D9+LJJSHPUnzVifxtc7hbo6fZw3SM4Mzz3vwOr/sdcHfK9p91/hT+zyH2WLzXypwSQySAPIyoMJr9llbiq3wyc3+Zet2P0vU1cPTAtaLOdlDXeidb9M2KLmS6ykZ0PfW+z1bVbrmbOMSM/rwqvrd8ZERyvNUdj5Mhx0/G7EazxaJLLaw1mO39t1jzTNUm1JQ51JNPHNUZrTzYwDSza9rrMno6+vhkVXRz8l4sp77b/jHS+azNYUIkFaLDnnh/NijFOs7qjC0VBn130D/wHWLck+wy5BUlsOJytie27ROJKC3Z0A3a+wxyceXg5Vu5rfP9CBYnssVpvednHNU4mzsOmRz6HuuJUoMIPGRIFOF8Dox5s+t21f6P8de33V/cHrdR1x1ClLVNZy3lA41zEX5953YK1jXI+za/DM21rOAg3r2GF2dx4JM6h1JhCUvg7HtsEXf7e3hdsqGJhAsPFRu9RKenby/x0M+B7kWJNXVe83rUzh+PyHpg4bAArOnmUSGJKNSoNz/mxaW8EM61j6zchbv7Y94z95PFgt5KnT7SlBWiw5g7QMD4oi+lKVfdY9Ylfd9krlF/bkuzldzR28W0qZlH+fZOnyrKuCym1mWaU13/WSbLI6mEwwH2fFcDd2BRSwTcJxOFFrlW+3+Oo6Uw9u4TQ7OM3pDuNfCd2ac9aPHJNhr4Od//B/vO6E+fIATNJAAr/ge17j/5mw/ufmGh/bBvv+bbapNOj3rdget/1ZNI7fO7al5Zkawm15zBsMZ0wwy7rOTDDvG+SfO9B+LBx9b7GvuXM8Z/5Ykz2dzDJyAhJGHgNrHtYWffEibHbckAz/iUlASVb5o0wQ6bPrn6aeYbgOLoXl/6/p9iOrTEtqipAgLZaczbKZ7bw5xsC7orujCle4Mw0E8pseKkmCtPL1NLao5A6MruBwIvhleYYxLk03+Ad1p2JXp49zjNEnN5lgC/wTBULJ6QyD/9NeX/OgKQjrU77aqnEFtBsU36SBYAp/6j9WccnN/qU5uk6FNr2aPi8aGW2gbT+zrBv8PyeCiaR7uJ+zOr+j5dttwkCg7E7Bx2Am83g0pzNvNb9vYBoE3AQuhz+HZY66cD2uhmEtZC4ngzNvhoF32+urf2SGK7h1Yh8sutZuYGg/HAY6imSn0MwGEqTFkrMlzavpRQLvqDY86v6OKhJRB2lJmDyQquPRfHpcTWMrxoGP3FeeP7jUHpPSqpN/df5TjfO9+VrQwCQKdGxSQ7upIf/pGLy+A0r+ZD+W6PFogdLS4bx/2JXs6ypNLSyfWCYMOHVwmTxQc8j+209rZc+c0JJe1zadgDwty3+8WriC1VVLlb+DtAwodMzjuvmp5oc7VB80CSW+igPtBpvu8RSZwJxRv3G0mGozXOFYScvPqz8JH18HJ6zpwLM6wIR5Zrxp43ChbfEZLhQDKXK1UkRdHFrSILI7qkhFWiPNp20fe9mLqaHqa+hz9FlYfpf7YDUVMzudcrras07oBvjSZdO931ydVyXvhPKxEKx1JFiiQCiZ7Uy3p8+6n9kt5ckWpIHp4r3gVbtEi09ON+gWpGp/LOS5TB7wm2mg0P28rIEJBGC6d7M7uj/HQJ0vtFsAfYJNzp6sel5jJ6U11MC6nwTfz5co4AuOM9uZQMXL76VYS8uE81+2y8bUllvDFlpImvjs+3bCkEqD818yNzBZ7U1Gq088hgvFgARpseTXkubhH0O4d1TRiLYlrbWHLWlaw4rv0afyBdj6NCy8yl0TdirWSAvkzPJccResuNv/9y+Q1v7ZoKdSAdtgcgdClmO6oE7jYdRvw3uNAd+F1j3Nck0ZbLLG9fhlKkYxNVesdSiCcwLmPewX44QBJ7e10qIJagNbvqJtFVRp/q+ZOzC6oC/elIIRjpvy7X+Fik1N91s10y5iDHDu3+0b+1SS09ncfDQmEqwzE86HSiQo+YuZf9en6FfQ9RJ7feDd/sOFNv2fN+cdQxKkxZJzTJoXiQNOPa8xA16h+TuqaGgdfUual5Osl/zRZO/4lC026dbN0Tr1W9IA+nzDVH4HQMOW38GbQ6H0X8H3L19rZ7Nl5JppZ05lStljWtoNhvFzwx8cnp7tP/H0xv81N0PO6cTykywrsM8MuwUwu4sJNL3itlZaNEFa3lDodb1ZPmOCPRtBNM68zW6d6Xd78/smoy6ToYsVeOgGWBMwxmzHbNj0G3t9+MPQ46ukrIKxcLYj8PryFfO3GOjgUljxPXu993QY8l/++0STgJEgEqTFSn2NCZbAlEjwejC62zuqaJzYY3fhZuaZD/1w+XV3xrAl7cAiWHFP0+1bfw/bng39vOr99mTTGW2TcioYV3I6w6VL/WtfndhtpjtadG3T6ZCcXZ3dL3c//2oqG/4QXPkFTF3dfKJAc/p8w87+rauET260kwZyByZn91HRz+CKTXDFBm+n/Gl7pt29Wr3PFEsNJtru4fNnw+Ub4KIPYpONnN3RvN4Vm2DofdG/XiIU/cJe3vUqHFxmlo+sMjNQ+HS/Mjnm5ozWmbea4Qo+q+/3r3d2Yq/JAG5MFCiEc/4S/PclcLjQ+l96ddYxIUFarPhldubGp7RBl4uav6OKVmBXZ0QZVV3tul41ZaZmVbSOl5qBoboOgGOZA/ynfVn+XTj4afDnOrs684alziDaYNr2gUnz4bzZpiadz65X4a2hsOX3dj2g06mr00cp838UTXmFtHT/oQVljqmikmU8WjDtBjUddB9raenmb8gnWGtazSG7BT0ty33SgJNKM634sey2zcxNze4/n4Ix0Otr9vrqmeb/euE0e8q3doPMzBSp/BnnNPpxU98QzOfa4unWHMwnYdF1JlADM8xhwjy7jE6gwOFCW34HVV96e+5ROEWuXhKIV9JAoBGOuwDnHVUsOFvmIunqBPNB3iaGtdLqq2HhNVB9wKy36sS6Dj81WUu+MTINJ81d1Yl9TZ/v/CLpkKJdnU5KQZ/pcPlGM/7Ip/aoafp/f7wpBuoLTtNaQbepiTnXVNXjKigY13R7Mgdp8dLSzAOHHUV/2xcmfz2yVFL4M1BW8s/+D83fui8gzsiFC+aFV9cy2aVlmvqGvjFlJ4+YnoPl37GnZFRpMP4lU5S6OYHDhdY+7NlpR0uCtFjxupBtKPmj7TEb0Px8g+GKNmnAJ1ZlOLQ2rWS++ftUBox/hZqMzuauacI8u/XgxB7T2haYSODXkpaiSQPBtMo3zfuTi+05HQEOLjEJFT5dLvGuPMypKnBogY8EaS0nDyRjJuypot1A/xsz57SE571gCgKfahoTCaxgv3yt//y0Ix6DLhe3/DqBf9NfPO8oUJ1cJEiLlXgUsg0l8I5qxV3RT8G09z3/LrJIW9IgdkHalqdh+3P2+qjfQueJ9nrbM+H8OXbzfrBEglOtJS1Q54nwldUw7MHgpQ5O5QK2Xuo8sencl8k+lVA8tJQ8IEGat4Y92HT887AHTevvqarj2abeYaDeN8Lg/3D/Ol0ugi6XmmXdYArmJiEJ0mIlXuU3gmk3wD9Laevv4a0hZuqLcFvVqg/A4pvgw8vswqfRTn3jN+vAjsheY/9HsNIxwfyZtwafILjrpVDk6AJ2JhI01PnfLaVq+Y2WpGebwo1TV0EnR6HOtEwzkFhEpugXNBYRbl90anUlRcr5N1Sxruk8pxKkeat1dxj0fXu92xUmYeZU1+9b/pnLHUaYOT/DHTftHC5UOg/KlsTm/GLIowI6p6F4zDbQnJH/awY/7n3brJ/Ya4oZbv8rjH3arkYeitZm38//y/T1+2R1gDFPmw+DSEXbklb1JXz8NTurLn+suZMK9Qc55D7z5eAr8rr8u2bAcmY7OwM3p7v3A6sTLW8oXLzQBKm7/gl9v5FaNaGSTf5IU9l/9xswJIw79lNZdieT9V29zwxYr9xubhoBag5D1RdmOS3LP8lAxM7wR4A08/k47IFTJ1GgJaP+z3yGn9hrepMimTA+fxT0ugG+fMmsr55phowk0ZzGp8nVjINEJQ40HjMXJr1lqis70+73vgNvnQXrfxG60OvRzfDBRSZ12xmg9b7RpKn3mRHduUVTK63uhEkCqLHS+7PPgAmvNl/iRCkY92zTRIJ979v7pGp9tHCpNOh/O1z4NvS5MdFnk/r6TIfzX5RWISe/1jRHl+eRlf77SNKAN9KzYMQvYOSjp9d40/QsKPq5+axv3S3y1yn6mV2B4MBC852ZRCRIi5VEJQ44KQW9rzeZfgPvorFrpr7a9Le/PcJ0G/rU18DaR2B+IRwotre36QuT3jFfRs7SDpGKdJJ1rWHZnXaXicowRUl9hSibEyyRYKWj9eNU7eoUIt6cf0tHHIk50tUpUkFuf/9ZKFbdb5cuSgISpMVKIhMHAmXlwZin4LJl/tPWHN0IH0yCJbfC7rfg7ZGw9iG7AKBKhyE/hMvXQbfLYnc+rXvYTfAn9prg0I3NT8KOF+z10U/AGRe4P25gIoF2jJeRIE2I2HC2Sjtb0iRIE6li2I8h3eouLV8NO+ck9nwcJEiLlUQmDoRSMAYu+9QEN87WvS+eh4+u8C+xUXA2TPnMNJlH0rffnLRMu7YNGo67mGd0/4fw+X/a6/1ui2yKm8BEAp/TpbtTCK9JS5pIdTldYbAjMW3Nj93NAx0HroI0pdQUpdRmpVSJUmpmkMcfV0qtsn62KKXKHY89ppRar5TaqJR6Uhm5jv1XKaUOKqX+z9r/VqVUmeOx1JhcrS5gxoFkkZYBg+6BKzb6V+X3yWgLo5+ESz4xEzR7JZzkgapd8PH1dstXwTkmeSHSwZxD7jODQ31UhpnPUQgRvXZD7Nbqym1QV2XGtvrmik3LlKQBkfyG/NDMVgDmd7dkVmLPx9JikKaUSgeeBqYCQ4EZSqmhzn201vdqrUdorUcATwGvWs89DzgfKASGAWOBiVrrY779refs9D3H8pLj8b9E/zbjwG9MWpK0pDm17g4XzIWJb5oxZwA9rjbB26C7zcwAXnIbpNVXm7knffNrZneGC/4Z3VyTSsG4Z6DTeLPe73YZxCxErGTkOAooa1PmxjnTQN7w02OuWJHasvLgrPvNcno21Fcl9nwsbkpwnA2UaK23Ayil5gBXAaHK884AfIVaNJANZGFGsWcC+507K6UGAGcAi8I9+aSS6BIcbnW/3EwLVHM4vuUY3NRK09oU4g2YUSCq8h8+GW3g4o/MvJ+te0b/ekIIW/vhdsX78jVmHkkf6eoUqWLgXVC9Hwb9IDbfOzHgJkjrDjgHEZUC5wTbUSnVG+gLLADQWi9RSn0I7MUEab/TWm8MeNoMTMuZs+rqtUqpCcAW4F6tdZNBTEqpO4A7ADp37kxxcbGLtxK5ysrKZo8x8lApvtKWn6/bRsVWb88n1XStqsY3nfG+bZ+y6XBxkH3eYFDFM43rW3O/y+4N9bCh6b5OLV2bpraHsa+IVvjXR8RLrK5N72O5+GZLLF33Nln1B/HlhW8+2Ja9cv0jIn87iXA5LNsKbG12r3hdGzdBWrCBQKHK2E8H5mptBhMppfoDQwBfzYT3lVITtNYLA57zDcf6G8BsrXWNUuo7wPPARU1OQOtZwCyAMWPG6EmTJrl4K5ErLi6m2WPMV2CNMxx59gRTAVnY9p6ED38DQJfcaroE/l8eXAr/fspe7/N1Bpz7BANcjENr8dqIhJLrk7xidm1KK2ChmdmjR9sjZlypZdC4GxlUMCb6Y5yG5G8necXr2rhJHCgFnP1DPYA9IfadDsx2rE8DlmqtK7XWlcDbwDjfg0qpIiBDa92YBqS1PqS19tVo+DOQGm3lyVSCIxk1VyvtxH5YdB001Jr19kVw9p+SquqzEKIZzgzPQ8v9kwak3I0QEXMTpC0HBiil+iqlsjCB2OuBOymlBgEdAOfkV18CE5VSGUqpTGAi4OzunIF/UIdSylEunysD9k9eyZ44kGhtetnLx0vNPJpgArPFN8CJ3WY9q4OZUSDWZUCEEN5p08dkioN/pnveMEkaECIKLQZpWus64C7gXUzA9LLWer1S6hGllHO25hnAnICxZXOBbcBaYDWwWmv9huPx6wkI0oB7rJIdq4F7gFvDfE/xp3XqJA4kSnq2meMPTGkNX1D2+X/DAd8sCMrMjdjSPKNCiOSi0oKX2ZCkASGi4mqCda31fGB+wLYHA9YfDvK8euDOZl63ybex1vp+4H4355U0GmpAWy1DaVly5xhKm95mImYwXZ5ln8Dmx+3HC38K3aYk5tyEENHpUAiHlvpvkyBNiKjIjAOxIK1o7jhrpe1+w0zo7tPjKrtGjRAi9eQFGXsmQZoQUXHVkiZa4EwakPFooTmTBzb+2l7OHQjjnrerlgshUk9ggoDKkKQBIaIk34qxIC1p7jhb0nwy2sCE10y1ZyFE6goMyNoPM2NRhRARkyAtFuqk/IYrwYK0cc9B3tCm24UQqaVVPuQ4qrRLV6cQUZMgLRb8WtIkSAvJ2d0JZkLbXtcl5FSEEB5oX2gvS5AmRNQkSIsFvxpp0t0ZUrvB9kwM3a6Aop8n9nyEELE14LumgG1Od+h1faLPRoiUJ4kDsSCzDbiTlg6XfAJHN0KHkTKjgBCnmh5fhWv2m8K2aZmJPhshUp4EabEgiQPuZeRA/qhEn4UQwitZHRJ9BkKcMqS7MxYkcUAIIYQQMSZBWixI4oAQQgghYkyCtFiQxAEhhBBCxJgEabEgiQNCCCGEiDEJ0mJBEgeEEEIIEWMSpMWCJA4IIYQQIsYkSIsFGZMmhBBCiBiTIC0WZEyaEEIIIWJMgrRYkBIcQgghhIgxCdKipYJhcfwAABr4SURBVDXUSeKAEEIIIWJLgrRo1Z8A3WCW07NlvjohhBBCxIQEadGSpAEhhBBCeECCtGhJ0oAQQgghPCBBWrTqJGlACCGEELHnKkhTSk1RSm1WSpUopWYGefxxpdQq62eLUqrc8dhjSqn1SqmNSqknlVLK2l5svabveWdY21sppV6yjvWpUqpPbN6qR2S2ASGEEEJ4IKOlHZRS6cDTwCVAKbBcKfW61nqDbx+t9b2O/e8GRlrL5wHnA4XWwx8DE4Fia/0mrfWKgEPeBhzRWvdXSk0HHgVuCP+txYmzuzNDWtKEEEIIERtuWtLOBkq01tu11ieBOcBVzew/A5htLWsgG8gCWgGZwP4WjncV8Ly1PBeY7Gt9S0rSkiaEEEIID7gJ0roDuxzrpda2JpRSvYG+wAIArfUS4ENgr/XzrtZ6o+Mpf7W6On/sCMQaj6e1rgMqgALX7yjeZN5OIYQQQnigxe5OIFgrlg6x73Rgrta6HkAp1R8YAvSwHn9fKTVBa70Q09W5WymVC/wT+AbwN7fHU0rdAdwB0LlzZ4qLi128lchVVlYGPUbPY6voZy1/ubec7R6fh2gq1LURyUGuT/KSa5Pc5Pokr3hdGzdBWinQ07HeA9gTYt/pwPcc69OApVrrSgCl1NvAOGCh1no3gNb6mFLqH5hu1b85jleqlMoA8oDDgQfSWs8CZgGMGTNGT5o0ycVbiVxxcTFBj7H637DeLPY68yx6Dff2PERTIa+NSApyfZKXXJvkJtcnecXr2rjp7lwODFBK9VVKZWECsdcDd1JKDQI6AEscm78EJiqlMpRSmZikgY3WekfreZnAFcA66zmvA7dYy9cBC7TWoVruEk/m7RRCCCGEB1psSdNa1yml7gLeBdKBZ7XW65VSjwArtNa+gG0GMCcgoJoLXASsxXRZvqO1fkMp1QZ41wrQ0oF/A3+2nvMM8IJSqgTTgjY96nfpJUkcEEIIIYQH3HR3orWeD8wP2PZgwPrDQZ5XD9wZZHsVMDrEsaqBr7k5r6QgiQNCCCGE8IDMOBAtmbtTCCGEEB6QIC1aMnenEEIIITwgQVq0JHFACCGEEB6QIC1akjgghBBCCA9IkBYtSRwQQgghhAckSIuG1gETrLdN3LkIIYQQ4pQiQVo06qponLEqvTWkuapoIoQQQgjRIgnSoiHj0YQQQgjhEQnSoiHj0YQQQgjhEQnSoiHlN4QQQgjhEQnSoiGzDQghhBDCIxKkRUNmGxBCCCGERyRIi4YkDgghhBDCIxKkRUMSB4QQQgjhEQnSoiGJA0IIIYTwiARp0ZDEASGEEEJ4RIK0aEjigBBCCCE8IkFaNCRxQAghhBAekSAtGpI4IIQQQgiPSJAWDRmTJoQQQgiPSJAWDRmTJoQQQgiPSJAWjTopwSGEEEIIb0iQFg1JHBBCCCGER1wFaUqpKUqpzUqpEqXUzCCPP66UWmX9bFFKlTsee0wptV4ptVEp9aQyWiul3lJKbbIe+5Vj/1uVUmWO17s9Nm/VA9LdKYQQQgiPZLS0g1IqHXgauAQoBZYrpV7XWm/w7aO1vtex/93ASGv5POB8oNB6+GNgIrAM+LXW+kOlVBbwgVJqqtb6bWu/l7TWd0X97rykG6Cu0l7PaJu4cxFCCCHEKcdNS9rZQInWervW+iQwB7iqmf1nALOtZQ1kA1lAKyAT2K+1Pq61/hDAes2VQI/I3kKCBAZoSnqOhRBCCBE7LbakAd2BXY71UuCcYDsqpXoDfYEFAFrrJUqpD4G9gAJ+p7XeGPCc9sBXgSccm69VSk0AtgD3aq2dx/c97w7gDoDOnTtTXFzs4q1ErrKy0u8YrerLONdarmnIZonHxxehBV4bkVzk+iQvuTbJTa5P8orXtXETpKkg23SIfacDc7XW9QBKqf7AEOxWsveVUhO01gutxzMwrW5Paq23W/u8AczWWtcopb4DPA9c1OQEtJ4FzAIYM2aMnjRpkou3Erni4mL8jlGxAd4yi63aFuD18UVoTa6NSCpyfZKXXJvkJtcnecXr2rjpoysFejrWewB7Quw7HburE2AasFRrXam1rgTeBsY5Hp8FbNVa/59vg9b6kNa6xlr9MzDaxTnGnzNpIEOSBoQQQggRW26CtOXAAKVUX2uQ/3Tg9cCdlFKDgA7AEsfmL4GJSqkMpVQmJmlgo7X/z4A84AcBr9PVsXqlb/+kI+U3hBBCCOGhFrs7tdZ1Sqm7gHeBdOBZrfV6pdQjwAqttS9gmwHM0Vo7u0LnYroq12K6SN/RWr+hlOoB/AjYBKxUSoEZr/YX4B6l1JVAHXAYuDUG7zP2ZN5OIYQQQnjIzZg0tNbzgfkB2x4MWH84yPPqgTuDbC8l+Fg3tNb3A/e7Oa+Eknk7hRBCCOEhqRsRKSlkK4QQQggPSZAWqVqZt1MIIYQQ3pEgLVKSOCCEEEIID0mQFilJHBBCCCGEhyRIi5QkDgghhBDCQxKkRUoSB4QQQgjhIQnSIiWJA0IIIYTwkARpkZLEASGEEEJ4SIK0SEnigBBCCCE8JEFapCRxQAghhBAekiAtUpI4IIQQQggPSZAWiYY6qD9urSjIaJPQ0xFCCCHEqUeCtEjUVdrLmbmggs4VL4QQQggRMQnSIiHlN4QQQgjhMQnSIiFJA0IIIYTwmARpkZCkASGEEEJ4TIK0SEghWyGEEEJ4TIK0SEghWyGEEEJ4TIK0SEjigBBCCCE8JkFaJCRxQAghhBAekyAtEpI4IIQQQgiPSZAWiTpJHBBCCCGEtyRIi4S0pAkhhBDCY66CNKXUFKXUZqVUiVJqZpDHH1dKrbJ+tiilyh2PPaaUWq+U2qiUelIpM4eSUmq0Umqt9ZrO7flKqfeVUlutfzvE6s3GjIxJE0IIIYTHWgzSlFLpwNPAVGAoMEMpNdS5j9b6Xq31CK31COAp4FXruecB5wOFwDBgLDDRetofgDuAAdbPFGv7TOADrfUA4ANrPblIS5oQQgghPOamJe1soERrvV1rfRKYA1zVzP4zgNnWsgaygSygFZAJ7FdKdQXaaa2XaK018Dfgaus5VwHPW8vPO7YnjzopwSGEEEIIb2W42Kc7sMuxXgqcE2xHpVRvoC+wAEBrvUQp9SGwF1DA77TWG5VSY6zXcb5md2u5s9Z6r/X8vUqpM0Ic6w5MSxydO3emuLjYxVuJXGVlZeMxRh/eg6+Tc8XqzVRmeXpo0QLntRHJR65P8pJrk9zk+iSveF0bN0GaCrJNh9h3OjBXa10PoJTqDwwBeliPv6+UmgCcCOM1g9JazwJmAYwZM0ZPmjQpnKeHrbi4mMZjvK6h0iyOGXcRtBvg6bFF8/yujUg6cn2Sl1yb5CbXJ3nF69q46e4sBXo61nsAe0LsOx27qxNgGrBUa12pta4E3gbGWa/Zw7Gf8zV93aFY/x5wcY7xJXN3CiGEEMJjboK05cAApVRfpVQWJhB7PXAnpdQgoAOwxLH5S2CiUipDKZWJSRrYaHVnHlNKjbOyOm8G/mU953XgFmv5Fsf25CFzdwohhBDCYy0GaVrrOuAu4F1gI/Cy1nq9UuoRpdSVjl1nAHOsRACfucA2YC2wGlittX7Deuy7wF+AEmuft63tvwIuUUptBS6x1pNH/UmorzbLKg3ScxJ7PkIIIYQ4JbkZk4bWej4wP2DbgwHrDwd5Xj1wZ4jXXIEpyxG4/RAw2c15JYSzFS2jHahgQ/aEEEIIIaIjMw6ES2qkCSGEECIOJEgLlyQNCCGEECIOJEgLlyQNCCGEECIOJEgLl8zbKYQQQog4kCAtXDImTQghhBBxIEFauGpl3k4hhBBCeE+CtHBJ4oAQQggh4kCCtHBJ4oAQQggh4kCCtHBJ4oAQQggh4kCCtHBJ4oAQQggh4kCCtHDJmDQhhBBCxIEEaeGS7E4hhBBCxIEEaeGSxAEhhBBCxIEEaeGSxAEhhBBCxIEEaeGSxAEhhBBCxIEEaeGqk8QBIYQQQnhPgrRwSeKAEEIIIeJAgrRw1NdAQ61ZTsuEtFaJPR8hhBBCnLIkSAtHYNKAUok7FyGEEEKc0iRIC4eU3xBCCCFEnEiQFg6ZbUAIIYQQcSJBWjik/IYQQggh4kSCtHBIIVshhBBCxImrIE0pNUUptVkpVaKUmhnk8ceVUqusny1KqXJr+4WO7auUUtVKqautxxY5tu9RSs2ztk9SSlU4Hnswlm84KlJ+QwghhBBxktHSDkqpdOBp4BKgFFiulHpda73Bt4/W+l7H/ncDI63tHwIjrO35QAnwnvXYBY7n/BP4l+Owi7TWV0T+tjwiiQNCCCGEiBM3LWlnAyVa6+1a65PAHOCqZvafAcwOsv064G2t9XHnRqVULnARMM/dKSeQdHcKIYQQ/7+9+w+Wq6zvOP7+5IagA4z80hQDJaipxVFEjIij1UitResU2gGaVMefM1SrVJ1qC52WKjNOtY7oWK2VUkQ7KqVYakQ6wEAutR2VIIRfUhQx1RQUWlS4giEh3/5xzpXN5iZ3k3DPHnPfr5k7u89zzp7z7H4nm+8+P85RR2btSQOWAN8fKG8Anj/TjkkOB44Arp5h80rgnBnqfwe4qqoGMiBekORG4C7gXVV16wznOg04DWDx4sVMTk7O/k52w9TUFOvvv5mlbXn9/9zH+gfm9pwazdTU1JzHX7vO+PSXsek349NfXcVmlCRtpiu21nb2XQlcXFWPbHWA5BDgWcDlM7xmFXDeQPl64PCqmkrySpoetmXbNKDqXOBcgOXLl9eKFStmeRu7Z3JykqX7HQi3N+Wly57F0iPn9pwazeTkJHMdf+0649NfxqbfjE9/dRWbUYY7NwCHDZQPpenhmslKZh7qPBW4pKo2DVYmOYhmOPXL03VVdX9VTbXPLwP2SnLwCO2cey4ckCRJHRklSVsLLEtyRJJFNInY6uGdkjwdOAD46gzH2N48tVOAS6vqZwPH+aWkud9SkmPbNv7fCO2cey4ckCRJHZl1uLOqNid5G81Q5QRwflXdmuRs4Lqqmk7YVgEXVtVWQ6FJltL0xF0zw+FXAu8fqjsZeEuSzcBDwMrhY46NCwckSVJHRpmTNj3seNlQ3VlD5fds57XraRYfzLRtxQx1HwM+Nkq7OucdByRJUke848DO8N6dkiSpIyZpO8M5aZIkqSMmaTvDOWmSJKkjJmmjqnK4U5IkdcYkbUQLeBimr9G7YG+Y2Hu8DZIkSXs0k7QRTWz56aMFe9EkSdIcM0kb0cJ66NGCiwYkSdIcM0kb0VY9aS4akCRJc8wkbUQL68FHC/akSZKkOWaSNqIJkzRJktQhk7QRuXBAkiR1ySRtRC4ckCRJXTJJG5ELByRJUpdM0kY0YU+aJEnqkEnaiBaWc9IkSVJ3TNJGNLHF1Z2SJKk7Jmkj8jppkiSpSyZpI9rqOmkuHJAkSXPMJG1EDndKkqQumaSNyIUDkiSpSyZpI7InTZIkdckkbUQuHJAkSV0ySRtF1dYXs3XhgCRJmmMjJWlJTkhye5I7kpwxw/YPJ1nX/n0ryY/b+pcO1K9L8rMkJ7XbLkjy3YFtR7f1SfLR9lw3JTnmsXzDu+SRBwlbmucTj4cFC8fbHkmStMebNdtIMgF8HPgNYAOwNsnqqvrm9D5V9c6B/U8HntPWrwGmk68DgTuAKwYO/+6qunjolK8AlrV/zwc+0T6Oz6b7H33uogFJktSBUXrSjgXuqKo7q+ph4ELgxB3svwr4/Az1JwP/VjU4uWtGJwKfqcbXgP2THDJCO+fOpgcefb7Q+WiSJGnujTJutwT4/kB5A9vp2UpyOHAEcPUMm1cC5wzVvS/JWcBVwBlVtXE751sC3D10rtOA0wAWL17M5OTkCG9l1+z38O08t33+wMbwjTk8l3be1NTUnMZfu8f49Jex6Tfj019dxWaUJC0z1NV29l0JXFxVj2x1gKYn7FnA5QPVZwI/ABYB5wJ/Cpw96vmq6tz2dSxfvrxWrFixwzexW36w5edp534HPJk5PZd22uTkpDHpMePTX8am34xPf3UVm1GGOzcAhw2UDwXu2s6+K5l5qPNU4JKq2jRdUVV3t0OaG4FP0Qyr7uz5urF5YLjTy29IkqQOjJKkrQWWJTkiySKaRGz18E5Jng4cAHx1hmNsM09tep5ZkgAnAbe0m1YDr21XeR4H/KSqthrq7NzgwgEvvyFJkjow63BnVW1O8jaaocoJ4PyqujXJ2cB1VTWdsK0CLqyqrYYmkyyl6Rm7ZujQn03yRJrhzXXAm9v6y4BX0qwEfRB4wy68r8fWJnvSJElSt0a64FdVXUaTPA3WnTVUfs92XrueZuL/cP3x29m/gLeO0q7OeAkOSZLUMe84MIqtkjR70iRJ0twzSRuFCwckSVLHTNJG4cIBSZLUMZO0UbhwQJIkdcwkbRQuHJAkSR0zSRuFc9IkSVLHTNJG4Zw0SZLUMZO0UXgJDkmS1DGTtFG4cECSJHXMJG02VbBofzZnH8gCWLjvuFskSZLmAZO02SRw0vf4j0MuhZWbYMHEuFskSZLmAZO0nRE/LkmS1A2zDkmSpB4ySZMkSeohkzRJkqQeMkmTJEnqIZM0SZKkHjJJkyRJ6iGTNEmSpB4ySZMkSeohkzRJkqQeSlWNuw27Lcm9wH/P8WkOBv53js+hXWNs+s349Jex6Tfj01+7G5vDq+qJs+20RyRpXUhyXVUtH3c7tC1j02/Gp7+MTb8Zn/7qKjYOd0qSJPWQSZokSVIPmaSN7txxN0DbZWz6zfj0l7HpN+PTX53ExjlpkiRJPWRPmiRJUg+ZpEmSJPWQSdoskpyQ5PYkdyQ5Y9ztmY+SnJ/kniS3DNQdmOTKJN9uHw9o65Pko228bkpyzPhavudLcliSNUluS3Jrkre39cZnzJI8Lsm1SW5sY/Petv6IJF9vY/NPSRa19Xu35Tva7UvH2f75IslEkhuSXNqWjU9PJFmf5OYk65Jc19Z1+t1mkrYDSSaAjwOvAJ4BrEryjPG2al66ADhhqO4M4KqqWgZc1ZahidWy9u804BMdtXG+2gz8cVUdCRwHvLX9N2J8xm8jcHxVPRs4GjghyXHAB4APt7H5EfCmdv83AT+qqqcBH27309x7O3DbQNn49MtLq+rogWuidfrdZpK2Y8cCd1TVnVX1MHAhcOKY2zTvVNW/A/cNVZ8IfLp9/mngpIH6z1Tja8D+SQ7ppqXzT1XdXVXXt88foPnPZgnGZ+zaz3iqLe7V/hVwPHBxWz8cm+mYXQz8epJ01Nx5KcmhwG8B57XlYHz6rtPvNpO0HVsCfH+gvKGt0/gtrqq7oUkUgCe19cZsTNrhl+cAX8f49EI7lLYOuAe4EvgO8OOq2tzuMvj5/zw27fafAAd12+J55yPAnwBb2vJBGJ8+KeCKJN9Iclpb1+l328LdPcAebqZfKV6zpN+M2Rgk2Rf4AvCOqrp/Bz/wjU+HquoR4Ogk+wOXAEfOtFv7aGw6lORVwD1V9Y0kK6arZ9jV+IzPC6vqriRPAq5M8l872HdO4mNP2o5tAA4bKB8K3DWmtmhrP5zuSm4f72nrjVnHkuxFk6B9tqr+pa02Pj1SVT8GJmnmDe6fZPoH+uDn//PYtNufwLbTDPTYeSHw20nW00ylOZ6mZ8349ERV3dU+3kPzI+dYOv5uM0nbsbXAsna1zSJgJbB6zG1SYzXwuvb564AvDtS/tl1pcxzwk+muaT322jkx/wDcVlXnDGwyPmOW5IltDxpJHg+8jGbO4Brg5Ha34dhMx+xk4OryaudzpqrOrKpDq2opzf8tV1fVqzE+vZBknyT7TT8HXg7cQsffbd5xYBZJXknz62YCOL+q3jfmJs07ST4PrAAOBn4I/CXwr8BFwC8D3wNOqar72qThYzSrQR8E3lBV142j3fNBkhcBXwFu5tF5NX9GMy/N+IxRkqNoJjZP0Pwgv6iqzk7yFJqemwOBG4DXVNXGJI8D/pFmXuF9wMqqunM8rZ9f2uHOd1XVq4xPP7RxuKQtLgQ+V1XvS3IQHX63maRJkiT1kMOdkiRJPWSSJkmS1EMmaZIkST1kkiZJktRDJmmSJEk9ZJImaaySVJIPDZTfleQ9j9GxL0hy8ux77vZ5TklyW5I1Q/VLk/z+XJ9f0p7JJE3SuG0EfjfJweNuyKAkEzux+5uAP6yqlw7VLwVmTNIGriovSTMySZM0bpuBc4F3Dm8Y7glLMtU+rkhyTZKLknwryfuTvDrJtUluTvLUgcO8LMlX2v1e1b5+IskHk6xNclOSPxg47pokn6O5QO9we1a1x78lyQfaurOAFwF/l+SDQy95P/BrSdYleWeS1yf55yRfAq5oX//ugXa8d+Bcr2nfz7okn2zbPNF+Jre07djmM5O05/CXnKQ++DhwU5K/3onXPJvmhuH3AXcC51XVsUneDpwOvKPdbynwEuCpwJokTwNeS3Pblucl2Rv4zyRXtPsfCzyzqr47eLIkTwY+ADwX+BFwRZKT2qv4H09zxfjhK4yf0dZPJ4evB14AHNVepfzlwLL2nAFWJ3kxcC/wezQ3eN6U5G+BVwO3Akuq6pnt8fbfic9L0i8YkzRJY1dV9yf5DPBHwEMjvmzt9L3xknyHtmeKpgdscNjxoqraAnw7yZ3Ar9Lch++ogV66J9AkSw8D1w4naK3nAZNVdW97zs8CL6a5RdnOuLKqpm+M/fL274a2vG/bjqNoksG1zd1meDzNjZy/BDwlyd8AXx54z5L2QCZpkvriI8D1wKcG6jbTTsto7423aGDbxoHnWwbKW9j6u2343ndF02t1elVdPrihvYfiT7fTvsz6DkYzePwAf1VVnxxqx+nAp6vqzG0akTwb+E3grcCpwBsfo3ZJ6hnnpEnqhbZ36SKaSfjT1tP0KAGcCOy1C4c+JcmCdp7aU4DbgcuBtyTZCyDJryTZZ5bjfB14SZKD20UFq4BrZnnNA8B+O9h+OfDGJPu27ViS5EnAVcDJ7XOSHJjk8HZxxYKq+gLwF8Axs5xf0i8we9Ik9cmHgLcNlP8e+GKSa2kSl+31cu3I7TTJ1GLgzVX1syTn0cxVu77tobsXOGlHB6mqu5OcCayh6QG7rKq+OMu5bwI2J7kRuIBmLtvgMa9IciTw1XZYcwp4TVV9M8mf08x7WwBsouk5ewj4VFsHsE1Pm6Q9R6qGRwIkSZI0bg53SpIk9ZBJmiRJUg+ZpEmSJPWQSZokSVIPmaRJkiT1kEmaJElSD5mkSZIk9dD/A1xFNeRqOek0AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "result_df.plot(x='Number of trees', y='AccuracyScore', figsize=(10,5), grid=True, lw=3, color='orange')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As can be seen above, adding more trees provides a better classification accuracy on average. However, this doe not mean that adding more tress will necessarily always result in a better accuracy depending on the cross validation population." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Gradient Boosted Trees\n", "\n", "Each tree is built based on learnings of the previous tree\n", "\n", "Each tree is trained using a different subset of data. However, the Boosting + Gradient Descent techniques are used to pool the subsets. After the bootstrap for one tree, Gradient Descent adjusts the probability of a data point being in the next training set and this continues sequentially. In this context, while the first bootstrap is based on uniform sampling the next one is modified based on what was learned from the previous three classification results. Data points that are classified correctly get lower weight for the next bootstrap. In this way, the likelihood of picking data that are not classified with previous tree increases. Finally, each tree is given a different weight for the final weighting for the majority rule decision.\n", "\n", "I random forest, there is a \"democracy\" that all trees have the same voting weights. However, in the gradient boosted approach, trees that have a better performance/accuracy get a higher weight. Gradient descent is one of the technique that can be used to quantify the probability of choosing a data point in the train subset based on the result of the previous tree.\n", "\n", "\n", "For window installation:\n", "\n", "```shell\n", "conda install -c anaconda py-xgboost\n", "```\n", "\n", "A rule of thumb is to use a lower number of tree when dealing with a small training data set. " ] }, { "cell_type": "code", "execution_count": 64, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:55:18.294710Z", "start_time": "2018-07-31T02:55:18.291713Z" } }, "outputs": [], "source": [ "from xgboost.sklearn import XGBClassifier" ] }, { "cell_type": "code", "execution_count": 65, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:55:18.905034Z", "start_time": "2018-07-31T02:55:18.900037Z" } }, "outputs": [ { "data": { "text/plain": [ "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n", " colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0,\n", " max_depth=3, min_child_weight=1, missing=None, n_estimators=100,\n", " n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,\n", " reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,\n", " silent=True, subsample=1)" ] }, "execution_count": 65, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf = XGBClassifier()\n", "clf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- colsample_bylevel: Fraction of features used by each tree\n", "\n", "- learning_rate: Controls the probability manipulation of data records based on the previous tree. Increasing this factor results in an increased focus on data records that were miss-classified.\n", "\n", "- subsample: Fraction of the provided training data set used for each tree. Lowering this number results in a more distinct difference between training data sets used for each tree\n", "\n", "- gamma: Minimum reduction in error to split the tree further\n", "\n", "- n_estimators: Number of trees" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy : 0.82\n" ] } ], "source": [ "clf.fit(X = train[['Age', 'Crew/Passenger', 'Class', 'Adult/Child', 'Gender']], y=train[['Survival']])\n", "predictions = clf.predict(test.drop(['Survival'], axis=1))\n", "print('Accuracy : %.2f'%(accuracy_score(test['Survival'], predictions)))" ] }, { "cell_type": "code", "execution_count": 67, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Importance for Age : 0.629\n", "Importance for Crew/Passenger : 0.057\n", "Importance for Class : 0.221\n", "Importance for Adult/Child : 0.000\n", "Importance for Gender : 0.093\n" ] } ], "source": [ "for feature, importance in zip(['Age', 'Crew/Passenger', 'Class', 'Adult/Child', 'Gender'],clf.feature_importances_):\n", " print('Importance for {} : {:.3f}'.format(feature,importance))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Hyper Parameter Tuning\n", "\n", "Considering the different parameters that affect the performance of a decision tree and also parameters that control the ensemble learning, there are many combinations that should be checked to find the best accuracy for the train/test data set. Hyper parameter tuning uses optimization technique to find an optimal combination." ] }, { "cell_type": "code", "execution_count": 68, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:07.350556Z", "start_time": "2018-07-31T02:47:06.970478Z" } }, "outputs": [], "source": [ "# conda install -c jaikumarm hyperopt\n", "from hyperopt import fmin, tpe, hp, STATUS_OK, Trials" ] }, { "cell_type": "code", "execution_count": 69, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:07.360549Z", "start_time": "2018-07-31T02:47:07.352552Z" } }, "outputs": [], "source": [ "# provide the optimization range/population \n", "control = {\n", " 'n_estimators': hp.quniform('n_estimators',100,1000,1)\n", " ,'learning_rate': hp.quniform('learning_rate', 0.1,0.4,0.1)\n", " ,'max_depth': hp.quniform('max_depth',1,15,1)\n", " ,'min_child_weight': hp.quniform('min_child_weight',1,6,1)\n", " ,'subsample': hp.quniform('subsample',0.5,1,0.05)\n", " ,'gamma': hp.quniform('gamma', 0.5,1,0.05)\n", " ,'colsample_bytree': hp.quniform('colsample_bytree',0.5,1,0.05)\n", " ,'nthread':5\n", " ,'silent':1 \n", "}" ] }, { "cell_type": "code", "execution_count": 70, "metadata": {}, "outputs": [], "source": [ "def scorevalue(clf):\n", " X = train[['Age', 'Crew/Passenger', 'Class', 'Adult/Child', 'Gender']]\n", " y=train[['Survival']]\n", " return np.mean(cross_val_score(clf, X, y, cv=5).mean())" ] }, { "cell_type": "code", "execution_count": 71, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:07.368542Z", "start_time": "2018-07-31T02:47:07.362545Z" } }, "outputs": [], "source": [ "# Define a score that gets minimized and provides what you need\n", "def score(params):\n", " params['n_estimators'] = int(params['n_estimators'])\n", " params['max_depth'] = int(params['max_depth'])\n", " clf = XGBClassifier(**params)\n", " return {'loss':1-scorevalue(clf), 'status':STATUS_OK}" ] }, { "cell_type": "code", "execution_count": 72, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:47:07.374538Z", "start_time": "2018-07-31T02:47:07.370540Z" } }, "outputs": [], "source": [ "trails=Trials()" ] }, { "cell_type": "code", "execution_count": 73, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:51:09.629027Z", "start_time": "2018-07-31T02:47:07.376537Z" } }, "outputs": [], "source": [ "best_param_set = fmin(score,control,algo=tpe.suggest,trials=trails,max_evals=250)" ] }, { "cell_type": "code", "execution_count": 74, "metadata": { "ExecuteTime": { "end_time": "2018-07-31T02:51:09.639022Z", "start_time": "2018-07-31T02:51:09.632024Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Optimized parameters: \n", "colsample_bytree = 1.0\n", "gamma = 1.0\n", "learning_rate = 0.2\n", "max_depth = 10.0\n", "min_child_weight = 5.0\n", "n_estimators = 583.0\n", "subsample = 1.0\n" ] } ], "source": [ "print('Optimized parameters: ')\n", "for key, value in best_param_set.items():\n", " print('{} = {}'.format(key,value))" ] }, { "cell_type": "code", "execution_count": 75, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Optimized Accuracy: 0.816\n" ] } ], "source": [ "print('Optimized Accuracy: {:.3f}'.format(1-score(best_param_set)['loss']))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" }, "toc": { "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "toc_cell": false, "toc_position": { "height": "774px", "left": "0px", "right": "1452.91px", "top": "111px", "width": "293px" }, "toc_section_display": true, "toc_window_display": true } }, "nbformat": 4, "nbformat_minor": 2 }