{ "cells": [ { "cell_type": "markdown", "id": "bf4f17ca", "metadata": {}, "source": [ "# Visualisation in examples (Python – plotnine)" ] }, { "cell_type": "markdown", "id": "acab7550", "metadata": {}, "source": [ "Here are examples of graphs in Python by [plotnine](https://plotnine.org/) package. (It is probably necessary to install it.)" ] }, { "cell_type": "markdown", "id": "c6060570", "metadata": {}, "source": [ "First we read packages, setup the environment, read and adjust data." ] }, { "cell_type": "code", "execution_count": null, "id": "ea2bf1de", "metadata": {}, "outputs": [], "source": [ "### Setup\n", "# Import libraries\n", "import pandas as pd\n", "import numpy as np\n", "import plotnine as gg # can be imported as `pn` or `p9`, no convention yet\n", "\n", "# classes for special types\n", "from pandas.api.types import CategoricalDtype\n", "\n", "# Reading and adjusting data\n", "K = pd.read_csv(\"application_train.csv\")\n", "\n", "# random sample of 2000 rows\n", "K = K.sample(n=2000, axis=0)\n", "\n", "# sample of columns\n", "K.columns = K.columns.str.lower() # column names to lowercase\n", "K = K[['sk_id_curr', 'target', 'name_contract_type', 'code_gender', 'cnt_children', 'amt_credit',\n", " 'occupation_type', 'name_education_type', 'days_birth', 'own_car_age', 'ext_source_1']]" ] }, { "cell_type": "markdown", "id": "522875cc", "metadata": {}, "source": [ "## 1. Introduction\n", "\n", "Each graph has the same layer structure common for the *grammar of graphics*:\n", "* data,\n", "* mapping,\n", "* geometry,\n", "* and optionally facets, statistics, coordinates and theme.\n", "\n", "Here is a simple example of graph with some layers (notice that there are two geometry layers):" ] }, { "cell_type": "code", "execution_count": null, "id": "4b4df002", "metadata": {}, "outputs": [], "source": [ "# data frame with squared numbers\n", "D = pd.DataFrame({'x': np.arange(6), 'y': [i**2 for i in np.arange(6)]})\n", " \n", "(\n", " gg.ggplot(data=D, mapping=gg.aes(x='x', y='y')) # data and mapping (aestetics) layers\n", " + gg.geom_point() # geometry layer\n", " + gg.geom_line(color='red') # another geometry layer\n", " + gg.scale_x_continuous(breaks=np.arange(6)) # coordinates layer\n", " + gg.theme_light() # theme layer\n", ")" ] }, { "cell_type": "markdown", "id": "0892af80", "metadata": {}, "source": [ "## 2. Distributions of individual variables\n", "\n", "We did an exploratory analysis by numerical means: frequencies, means, standard deviations etc. Now we support it by visualisation." ] }, { "cell_type": "markdown", "id": "e0305cc3", "metadata": {}, "source": [ "### 2.1 Categorial variable\n", "\n", "Before we make a graph, let's make a frequency table (we combine absolute and relative frequencies)." ] }, { "cell_type": "code", "execution_count": null, "id": "2dfe3f07", "metadata": {}, "outputs": [], "source": [ "# absolute and relative frequency table\n", "freqtab = K.groupby(\"name_education_type\").agg(count=(\"sk_id_curr\", \"count\")) # absolute frequencies (counts)\n", "freqtab[\"count_rel\"] = freqtab[\"count\"] / sum(freqtab[\"count\"]) # relative frequencies\n", "freqtab" ] }, { "cell_type": "markdown", "id": "e002deb6", "metadata": {}, "source": [ "It is reasonable to make the variable `name_education_type` ordered. Then we can compute cumulative frequencies." ] }, { "cell_type": "code", "execution_count": null, "id": "e6011dea", "metadata": {}, "outputs": [], "source": [ "# making a new categorical ordered variable\n", "cat_type = CategoricalDtype(categories=[\"Lower secondary\", \"Secondary / secondary special\",\n", " \"Incomplete higher\", \"Higher education\"],\n", " ordered=True)\n", "K[\"education\"] = K[\"name_education_type\"].astype(cat_type)\n", "\n", "# frequency table\n", "# - grouping varible as a column, not as index, more useful for graphs\n", "freqtab = K.groupby(\"education\", observed=True, as_index=False).agg(count=(\"sk_id_curr\", \"count\")) # absolute frequencies (counts)\n", "freqtab[\"count_cum\"] = freqtab[\"count\"].cumsum() # cumulative frequencies\n", "freqtab[\"count_rel\"] = freqtab[\"count\"] / sum(freqtab[\"count\"]) # relative frequencies\n", "freqtab[\"count_relcum\"] = freqtab[\"count_rel\"].cumsum() # cumulative relative frequencies\n", "freqtab" ] }, { "cell_type": "markdown", "id": "546133dd", "metadata": {}, "source": [ "The visualisation of frequencies is simple – we use a layer method for barplot: *stat_count* (basic frequency barplot), *geom_bar* (similar but more consistent with geometry layer), *geom_col* (more general, suitable for using values in other columns). Variable name is mapped to *x*, horizontal bars are made by *coord_flip* (method in coordinates layer)." ] }, { "cell_type": "code", "execution_count": null, "id": "e2f3749b", "metadata": {}, "outputs": [], "source": [ "# graph of absolute frequencies\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"education\")) # mapping education to x axis\n", " + gg.stat_count() # statistics layer, making geometry too\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "b50803e9", "metadata": {}, "outputs": [], "source": [ "# horizontal bars will be better due to long annotations\n", "# additionally, we use another way of making barplot (geometry layer)\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"education\")) # mapping education to x but axes will be switched\n", " + gg.geom_bar(stat=\"count\") # geometry layer, making statistics\n", " + gg.coord_flip() # switching X and Y axes\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "1e6f9faf", "metadata": {}, "outputs": [], "source": [ "# for relative and cumulative frequencies, we need to use geom_col instead of geom_bar and aggregated table `freqtab`\n", "# relative frequencies - different only at x axis (ratios instead of counts)\n", "(\n", " gg.ggplot(data=freqtab, mapping=gg.aes(x=\"education\", y=\"count_rel\")) # mapping education to x axis\n", " + gg.geom_col()\n", " + gg.coord_flip() # switching X and Y axes\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "082ec53e", "metadata": {}, "outputs": [], "source": [ "# cumulative frequencies - stacked bar\n", "# - we need to create a dummy variable for x axis mapping\n", "freqtab[\"hlp\"] = [\"\"] * len(freqtab)\n", "\n", "(\n", " gg.ggplot(data=freqtab, mapping=gg.aes(x=\"hlp\", y=\"count_rel\", fill=\"education\"))\n", " + gg.geom_col()\n", " + gg.coord_flip()\n", " + gg.theme(aspect_ratio=0.25) # reducing height of the stacked graph\n", ")\n", "# for stacked absolute frequencies, use \"count\" instead of \"count_rel\"" ] }, { "cell_type": "markdown", "id": "4ab8a647", "metadata": {}, "source": [ "If we want to annotate and fine-tune the graph, we may use coordinates and theme layers (and colors as parameters in geometry layers, too)." ] }, { "cell_type": "code", "execution_count": null, "id": "c98bc87e", "metadata": {}, "outputs": [], "source": [ "(\n", " gg.ggplot(data=freqtab, mapping=gg.aes(x=\"education\", y=\"count_rel\")) # mapping education to x axis\n", " + gg.geom_col()\n", " + gg.scale_y_continuous(breaks=np.linspace(0, 1, 11))\n", " + gg.labs(\n", " x = \"Education\",\n", " y = \"Share\",\n", " title = \"Relative frequency of education grades\",\n", " subtitle = \"\",\n", " caption = f\"HomeCredit dataset - sample of {K.shape[0]:,.0f} records\".replace(\",\", \" \")\n", " )\n", " + gg.coord_flip() # switching X and Y axes\n", " + gg.theme_light()\n", " + gg.theme(\n", " plot_title=gg.element_text(face=\"bold\"),\n", " plot_caption=gg.element_text(color=\"gray\", size=7),\n", " panel_border=gg.element_rect(color=\"white\"),\n", " panel_grid_minor_x=gg.element_blank(),\n", " aspect_ratio=1/2 \n", " )\n", ")" ] }, { "cell_type": "markdown", "id": "02b232c4", "metadata": {}, "source": [ "### 2.2 Numerical variable\n", "\n", "Again, before we start to plot graphs, let's calculate some statistical characteristics of variable `days_birth` (days of lifetime, for some reason with minus sign)." ] }, { "cell_type": "code", "execution_count": null, "id": "6e4b11a2", "metadata": {}, "outputs": [], "source": [ "# computing statistical characteristics of distribution\n", "print(\"Min and max: \", \"%.1f\" % K[\"days_birth\"].min(), \"|\", \"%.1f\" % K[\"days_birth\"].max())\n", "print(\"Mean: \", \"%.1f\" % K[\"days_birth\"].mean())\n", "print(\"Median: \", \"%.1f\" % K[\"days_birth\"].median())\n", "print(\"Std. dev.: \", \"%.1f\" % K[\"days_birth\"].std())\n", "# decils\n", "hlp_10s = [i/10.0 for i in range(0, 11)]\n", "hlp_qs = K[\"days_birth\"].quantile(hlp_10s)\n", "print(\"Decils:\\n\", hlp_qs)\n", "# IQR\n", "hlp_qs = K[\"days_birth\"].quantile([0.25, 0.75])\n", "print(\"IQR: \", hlp_qs.iloc[1]-hlp_qs.iloc[0])" ] }, { "cell_type": "markdown", "id": "49de6aa5", "metadata": {}, "source": [ "We make bunch of graphs with different level of detail and smoothing. One should distinct discrete and continuous variables. The former can be treated as categorical, the latter not. So is the difference between barplot and histogram.\n", "\n", "If the variable is numeric but with few unique values, we can treat it as categorial – we prefer barplot, not histogram." ] }, { "cell_type": "code", "execution_count": null, "id": "a994fc8c", "metadata": {}, "outputs": [], "source": [ "# right way: each value in variable is treated individually - good for discrete variables, not for continuous ones\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"cnt_children\")) # mapping variable to x axis\n", " + gg.geom_bar(stat=\"count\") # geometry layer, making statistics\n", ")\n", "# Note: we may use stat_count instead of geom_bar as above." ] }, { "cell_type": "code", "execution_count": null, "id": "15042ab2", "metadata": {}, "outputs": [], "source": [ "# not so pretty - interval of x values is binned by default\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"cnt_children\")) # mapping variable to x axis\n", " + gg.geom_histogram() # number of bins can be changed by `bins` or `binwidth` parameters\n", ")" ] }, { "cell_type": "markdown", "id": "494cf815", "metadata": {}, "source": [ "Continuous numeric variable can be plotted many ways depending on required completeness of information." ] }, { "cell_type": "code", "execution_count": null, "id": "5c70363a", "metadata": {}, "outputs": [], "source": [ "# histogram\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"days_birth\")) # mapping variable to x axis\n", " + gg.geom_histogram(bins=6) # number of bins can be changed by `bins` or `binwidth` parameters\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "6fea0bd0", "metadata": {}, "outputs": [], "source": [ "# for less smoothing, use bigger number of bins or narrower binwidth:\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"days_birth\")) # mapping variable to x axis\n", " + gg.geom_histogram(binwidth=730.5) # number of bins can be changed by `bins` or `binwidth` parameters\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "c18bdd87", "metadata": {}, "outputs": [], "source": [ "# frequency barplot is a bad idea\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"days_birth\")) # mapping variable to x axis\n", " + gg.geom_bar(stat=\"count\") # geometry layer, making statistics\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "6fe76484", "metadata": {}, "outputs": [], "source": [ "# smoothed information - density estimation with rug\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"days_birth\")) # mapping variable to x axis\n", " + gg.geom_density() # level of smoothing may be controlled by `n` parameter\n", " + gg.geom_rug()\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "4ec2d5ff", "metadata": {}, "outputs": [], "source": [ "# different bandwith for estimation - less smoothing\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"days_birth\")) # mapping variable to x axis\n", " + gg.geom_density(adjust=0.5)\n", " + gg.geom_rug()\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "6271c939", "metadata": {}, "outputs": [], "source": [ "# ecdf with rug and an additional line (lower quartile)\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"days_birth\")) # mapping variable to x axis\n", " + gg.stat_ecdf() # not geom_ecdf!\n", " + gg.geom_rug()\n", " + gg.geom_hline(\n", " yintercept=0.25,\n", " color=\"red\", # set line colour\n", " size=1, # set line thickness\n", " linetype=\"dashed\", # set line type\n", " )\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "66e495d1", "metadata": {}, "outputs": [], "source": [ "# boxplot\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(y=\"days_birth\")) # mapping variable to x axis\n", " + gg.geom_boxplot()\n", " + gg.theme(axis_text_x=gg.element_blank(), aspect_ratio=3) # suppressing axis x, making the box narrower\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "c8024b05", "metadata": {}, "outputs": [], "source": [ "# catplot, stripplot, swarmplot - not available in plotnine yet (as far as I know...)" ] }, { "cell_type": "markdown", "id": "93c6b1d2", "metadata": {}, "source": [ "## 3. Relationships of variables\n", "\n", "Method for analysis and plotting are different depending on type (categorial or numerical) of both variables. \n", "\n", "* If one of variables is categorial, the basic strategy is to split the data into categories by this variable and to study distribution of the other variable for each category (and to compare distributions among various categories).\n", "* If both variables are numerical, then we use bivariate plots and compute statistics like correlation.\n", "* For a numerical variable, one may consider binning it and getting this way a categorial variable." ] }, { "cell_type": "markdown", "id": "d01de385", "metadata": {}, "source": [ "### 3.1 Categorial vs. categorial\n", "\n", "In this case we usually compute a contingency table (2-D frequency table)." ] }, { "cell_type": "code", "execution_count": null, "id": "921e6914", "metadata": {}, "outputs": [], "source": [ "# contingency table with absolute frequencies\n", "pd.crosstab(K[\"education\"], K[\"code_gender\"])" ] }, { "cell_type": "code", "execution_count": null, "id": "ef30d3ff", "metadata": {}, "outputs": [], "source": [ "# for relative frequencies in contingency table, use parameter normalize:\n", "pd.crosstab(K[\"education\"], K[\"code_gender\"], normalize=\"columns\") # relative by columns" ] }, { "cell_type": "markdown", "id": "dab1e664", "metadata": {}, "source": [ "Visualisation of contingency table, similarly to frequency table, can be done by various ways:\n", "\n", "* dotplot (only if both variables are ordered categorial or discrete numerical, so they may can be treated as numbers; can be done by *factor* function)\n", "* barplot (only for categorial, so discrete numerical variables need to be *factor*ed)\n", " - put combinations of categories beside one by one\n", " - stacked within each category as absolute counts\n", " - stacked within each category as relative counts (all stacked bars sum up to 1)" ] }, { "cell_type": "code", "execution_count": null, "id": "fa669dfe", "metadata": {}, "outputs": [], "source": [ "# dotplot\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"cnt_children\", y=\"factor(code_gender)\")) # factor makes `code_gender` ordered\n", " + gg.geom_count() # counts frequencies\n", " + gg.scale_size_continuous(range=[1, 15]) # defines range of points sizes\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "c871f311", "metadata": {}, "outputs": [], "source": [ "# barplot of absolute frequencies\n", "# combinations of categories side by side (\"dodge\" position)\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"factor(code_gender)\", fill=\"factor(cnt_children)\")) # factor makes it ordered category\n", " + gg.geom_bar(position=\"dodge\") # geometry layer, puts bars side by side\n", " + gg.geom_vline(xintercept=1.5, color=\"grey\", linetype=\"dotted\") # dividing line between main categories\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "9e3e713d", "metadata": {}, "outputs": [], "source": [ "# barplot of absolute frequencies - stacked\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"factor(code_gender)\", fill=\"factor(cnt_children)\")) # factor makes it ordered category\n", " + gg.geom_bar() # stacked by default\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "801cb13f", "metadata": {}, "outputs": [], "source": [ "# cumulative (relative) frequencies - stacked bar\n", "# needs making an auxiliary table (like at 2.1)\n", "hlp_df = pd.crosstab(K[\"cnt_children\"], K[\"code_gender\"], normalize=\"columns\").reset_index()\n", "hlp_df = pd.melt(hlp_df, id_vars=\"cnt_children\", var_name=\"code_gender\", value_name=\"prop\")\n", "\n", "(\n", " gg.ggplot(data=hlp_df, mapping=gg.aes(x=\"code_gender\", y=\"prop\", fill=\"factor(cnt_children)\"))\n", " + gg.geom_col()\n", " + gg.theme(aspect_ratio=2)\n", ")\n", "# mapping cnt_children without *factor*ing leads to slightly different output - try yourself" ] }, { "cell_type": "markdown", "id": "d5abb212", "metadata": {}, "source": [ "Another idea is to make *heatmap* – replace each cell in a contingency table by color tone according to the value in the cell. This is good for plotting absolute frequencies but may be confusing for relative ones." ] }, { "cell_type": "code", "execution_count": null, "id": "1a1647ad", "metadata": {}, "outputs": [], "source": [ "# heatmap for categories\n", "hlp_df = pd.crosstab(K[\"cnt_children\"], K[\"code_gender\"]).reset_index()\n", "hlp_df = pd.melt(hlp_df, id_vars=\"cnt_children\", var_name=\"code_gender\", value_name=\"n\")\n", "\n", "(\n", " gg.ggplot(data=hlp_df, mapping=gg.aes(x=\"code_gender\", y=\"factor(cnt_children)\", fill=\"n\"))\n", " + gg.geom_tile(width=0.95, height=0.95) # makes spaces between tiles\n", " + gg.theme(aspect_ratio=2)\n", ")" ] }, { "cell_type": "markdown", "id": "6e4f817a", "metadata": {}, "source": [ "### 3.2 Categorial vs. numerical\n", "\n", "We can split the data by categorial variable and compute statistics by categories, e. g. mean, median or SD, and compare them. Computing is easy by pandas *groupby* and *agg* methods." ] }, { "cell_type": "code", "execution_count": null, "id": "1bf6630f", "metadata": {}, "outputs": [], "source": [ "# statistics by categories\n", "# it is possible to use user defined functions\n", "def quartileH(x): # upper quartile\n", " return x.quantile(q=0.75)\n", "\n", "K.groupby(\"code_gender\").agg({\"ext_source_1\": [\"mean\", \"median\", \"std\", \"max\", \"min\", quartileH]})" ] }, { "cell_type": "markdown", "id": "b76ac0e9", "metadata": {}, "source": [ "There are many ways how to do splitting by categories when plotting:\n", "\n", "* multiple lines (curves), possibly overlapping\n", "* use one axis for categories (sections inside one graph), distribution graph in each section separately\n", "* split figure to separate graphs (*facetting*)\n", "\n", "Grouping by categories is done in mapping, sometimes by parameters *x* and *y*, sometimes by using *fill*. It depends on the graph type." ] }, { "cell_type": "code", "execution_count": null, "id": "bef5cdcd", "metadata": {}, "outputs": [], "source": [ "# histogram by categories - stacked bars by default\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"ext_source_1\", fill=\"code_gender\")) # mapping variable to x axis\n", " + gg.geom_histogram(bins=10, na_rm=True) # number of bins can be changed by `bins` or `binwidth` parameters\n", ")\n", "# Note: na_rm=True omits all NA values without warning." ] }, { "cell_type": "code", "execution_count": null, "id": "11841bb4", "metadata": {}, "outputs": [], "source": [ "# histogram by categories - dodged bars\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"ext_source_1\", fill=\"code_gender\")) # mapping variable to x axis\n", " + gg.geom_histogram(bins=10, na_rm=True, position=\"dodge\") # number of bins can be changed by `bins` or `binwidth` parameters\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "8a723a2d", "metadata": {}, "outputs": [], "source": [ "# histogram by categories in facets and normalized\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"ext_source_1\", y=gg.after_stat(\"ncount\"))) # mapping normalization to y axis\n", " + gg.geom_histogram(bins=10, na_rm=True) # number of bins can be changed by `bins` or `binwidth` parameters\n", " + gg.facet_wrap(\"code_gender\") # splitting to facets\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "2c6e138b", "metadata": {}, "outputs": [], "source": [ "# density by categories\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"ext_source_1\", fill=\"code_gender\")) # mapping variable to x axis\n", " + gg.geom_density(alpha=0.8, na_rm=True) # opacity of density can be controlled by `alpha` parameter\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "90c08922", "metadata": {}, "outputs": [], "source": [ "# violin plot - density by categories\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"code_gender\", y=\"ext_source_1\", fill=\"code_gender\")) # mapping variable to x axis\n", " + gg.geom_violin(style=\"left-right\", na_rm=True) # make half-violin for each group\n", ")\n", "# Note: adding `fill` parameter causes assigning colors to groups." ] }, { "cell_type": "code", "execution_count": null, "id": "027ca328", "metadata": {}, "outputs": [], "source": [ "# boxplots by categories\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"code_gender\", y=\"ext_source_1\")) # mapping variable to x axis\n", " + gg.geom_boxplot(na_rm=True)\n", " + gg.theme(aspect_ratio=2) # suppressing axis x, making the box narrower\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "2999a5a9", "metadata": {}, "outputs": [], "source": [ "# mean +/- 1 standard deviation by categories\n", "# needs making an auxiliary table\n", "hlp_df = K.groupby(\"code_gender\").agg({'ext_source_1':['mean', 'std']}).reset_index()\n", "hlp_df.columns = ['code_gender', 'ext_mean', 'ext_sd']\n", "hlp_df[\"ext_low\"] = hlp_df[\"ext_mean\"]-hlp_df[\"ext_sd\"]\n", "hlp_df[\"ext_high\"] = hlp_df[\"ext_mean\"]+hlp_df[\"ext_sd\"]\n", "\n", "(\n", " gg.ggplot(data=hlp_df, mapping=gg.aes(x=\"code_gender\", y=\"ext_mean\")) # mapping variable to x axis and mean to y axis\n", " + gg.geom_col(alpha=0.7)\n", " + gg.geom_linerange(gg.aes(x=\"code_gender\", ymin=\"ext_low\", ymax=\"ext_high\"))\n", ")" ] }, { "cell_type": "markdown", "id": "83b72bda", "metadata": {}, "source": [ "### 3.3 Numerical vs. numerical\n", "\n", "The basic statistics for this case is a correlation coefficient. For plotting we use *relplot* or *displot* method with two basic cases:\n", "\n", "* for each x value there can be more observations – *scatterplot* (a cloud of points), heatmap, contourplot\n", "* for each x value there is only one observation or we want to aggregate over y axis – *lineplot* (time series)" ] }, { "cell_type": "code", "execution_count": null, "id": "400f1398", "metadata": {}, "outputs": [], "source": [ "# correlation between two columns\n", "print(\"Correlation: \", K[\"days_birth\"].corr(K[\"amt_credit\"]))\n", "\n", "# correlation matrix between every pair of numerical variables\n", "corrmat = K.corr(numeric_only=True)\n", "corrmat" ] }, { "cell_type": "code", "execution_count": null, "id": "d82da79a", "metadata": {}, "outputs": [], "source": [ "# correlation as a heatmap\n", "corrmat[\"var1\"] = corrmat.index\n", "corrmat = pd.melt(corrmat, id_vars=\"var1\", var_name=\"var2\", value_name=\"corr\")\n", "\n", "(\n", " gg.ggplot(data=corrmat, mapping=gg.aes(x=\"var1\", y=\"var2\", fill=\"corr\"))\n", " + gg.geom_tile(width=0.95, height=0.95) # makes spaces between tiles\n", " + gg.theme(aspect_ratio=1)\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "77425b58", "metadata": {}, "outputs": [], "source": [ "# scatterplot with regression lines by groups\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"ext_source_1\", y=\"days_birth\", fill=\"code_gender\"))\n", " + gg.geom_point(na_rm=True) # layer with points\n", " + gg.geom_smooth(na_rm=True, size=1) # regression line\n", " # + gg.geom_rug() # optionally, we can plot rug of each variable on the margin\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "74e3fa09", "metadata": {}, "outputs": [], "source": [ "# frequency heatmap\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"ext_source_1\", y=\"days_birth\"))\n", " + gg.geom_bin2d(na_rm=True, binwidth=(0.1, 1000)) # binwidth parameter is a tuple (for x, for y)\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "bf390edf", "metadata": {}, "outputs": [], "source": [ "# contour plot\n", "(\n", " gg.ggplot(data=K, mapping=gg.aes(x=\"ext_source_1\", y=\"days_birth\"))\n", " + gg.geom_density_2d(na_rm=True)\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "97c42e57", "metadata": {}, "outputs": [], "source": [ "# lineplot\n", "hlp_df = K.groupby('own_car_age').agg({'days_birth': \"mean\"}).reset_index()\n", "\n", "(\n", " gg.ggplot(data=hlp_df, mapping=gg.aes(x=\"own_car_age\", y=\"days_birth\")) # car age to x axis, mean age to y axis\n", " + gg.geom_line()\n", ")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.18" } }, "nbformat": 4, "nbformat_minor": 5 }