---
title: "Data Understanding"
subtitle: "The Importance of Thinking"
author: "[Anna Michálková](mailto:anna.michalkova@profinit.eu), [Jan Hučín](mailto:jan.hucin@profinit.eu)"
---



## Outline

* What is DU
    - Goals
    - Three sources of DU
    - Why is it important?
    - What to find out - basics
* DU steps in a project
    - 1. Data & Existing Knowledge Collection
    - 2. First looks into the data
    - 3. Data description (profiling)
    - 4. Data exploration

...and many examples along the road


# What is Data Understanding (DU)?

## Goals


:::: {.columns}
::: {.column width="50%"}

- Increase **familiarity** with the data via **profiling and exploration**
- Identify **data quality** problems and potential issues
- Discover **initial insights** into the data
- Find interesting **paterns**
- Generate **hypotheses** for further phases of analysis
- ...

:::

::: {.column width="10%"}
:::


::: {.column width="40%"}
![](img/crisp_du.png){height="15em"}
:::

::::




## Three sources of understanding

<div style="height: 60px;"></div>

::: {.fragment}

#### 1. Existing knowledge base

What others know about the data

:::

<div style="height: 60px;"></div>

::: {.fragment}

#### 2. Data themselves

What we can find from the data

:::

<div style="height: 60px;"></div>

::: {.fragment}

#### 3. Own critical thinking

To compare and verify outcomes of previous sources

:::


## Why is it important?


:::: {.columns}
::: {.column width="70%"}

#### Make informed decisions about how to proceed

- Helps with **justification for the business**  
- What should we **focus** on? 
    - Don't waste resources: many data sets, columns or rows may be irrelevant
    - Set right **order of next tasks**: from core to the rest & from simple to difficult
- **Key inputs for next CRISP phases** 
    - Which data preprocessing is needed? 
    - Which modelling techniques will be suitable? 
    
        [(no. of obs., no. of potential features, NAs, dtypes, relationships between target and variables or variables themselves…)]{.p-grey}

:::

::: {.column width="30%"}

![](img/garbage_in_out_xkcd.png){height="20em"}

:::

::::

---

<div style="height: 90px;"></div>

#### Tackling problems easier

- Prevents **misinterpretation** and incorrect conclusions
- **Early detection of problematic** columns → saved effort in mining featues, crafting models depending on them
- Something doesn't work as expected → it's **difficult to trace down the issue without DU** - where to start?


## Example: loan application

:::: {.columns}
::: {.column width="50%"}

#### About

- UK FinTech StartUp
- Consumer loans via aggregating websites 
    - Form: Fill info about you
    - API calls to credit bureau, income verification...
    - Competitors offer rate → customer selects among offers
- **Task**: predict offer of the startup is clicked, optimize price to improve click rate

:::

::: {.column width="10%"}

:::

::: {.column width="40%"}
![](img/mcf_selection.png)
:::

::::

---

<div style="height: 90px;"></div>


#### Importance of DU

- First model iteration (quick and dirty) - much added value from employment sector and position features
- **Reason?** <span class="fragment">Data leak</span>

<span class="fragment">
![](img/mcf_leak.png)
</span>

::: {.notes}
Risks:

- Business was already informed "there is something valuable"
- Developed techniques for semantic lumping...
:::

## What to find out - basics {.smaller}


::: {.fragment}

::: {.callout-tip icon=false}
Documenting findings already during your exploration in a reproducible way >>> Generating random stats and plots and then trying to make sense of them retrospectively. 

More in week 4
:::

:::

::: {.fragment}

#### Properties of a **data set**

:::

::: {.incremental}
- File **format** [(DB table, csv, binary, audiovisual...)]{.p-grey}
- **Population** of entities → what the data set covers and what not?
- **Data creation pipeline**
    - Original data source
    - Owner and users with access  
    - Process of verification, correction, cleaning, consolidation 
    - Anonymization / pseudonymization
    - → where ask for repeated ingest? how much can we trust the data?
:::

::: {.notes}
- Popul. examples - customers of company (all /exising/new), evolution of application statuses...
- Orig. data source examples - 1:1 view over DB table, several DB tables combined, scrapped web page... → DQ
- Owner - where to go if I need refinements, more info
- Corrections etc. - very raw data / std. mart, corrections may happen in first 1-2 days, delay (validity x avail. on day x+2) → DQ
- Consolidation example 
    - parameter change in a loan may mean new product ID internally but from customer's POV, it's still the same loan
    - multi-currency accounts - internally individual accounts, from customer's POV one account with multiple bins
- Anony - some columns will be masked from security reasons, pseudony example - hash name-parts to guess people in marriage
:::

---

<div style="height: 90px;"></div>

#### Properties of a **data set** (cont.)

::: {.incremental}
- **Size**
    - small / big data (does it fit on the RAM, HDD)
    - no. of rows, columns
    - → data processing engine
- **Historization**
    - fixed / changes in time (valid date available?)
    - → what can be used for modelling?
:::

::: {.fragment}

#### **Data sets relations**

- Relation keys (possibility of missings, uniqueness...)
- Units relation stats (no. of records for basic entity)  

:::

--- 

<div style="height: 90px;"></div>


#### Properties of **variables** in a data set

::: {.incremental}
- **Meaning** (incl. code sheets)
- **Value origin** - exact value / estimated / model result?
- **Types** (numeric / categorical / time / text...) and **formats** (decimal places, encoding...)
- **Missings** - possible or not, encoding
- **Errors, outliers and specificities**
- **Relevance** to the addressed problem
- **Summary statistics**
- Distribution **visualization** - more next week
:::

::: {.fragment}

#### **Variables relations**

- Correlations and dependencies 
- Special focus: relation to **target variable**

:::


# DU steps in a project


## 1. Data & Existing Knowledge Collection {.smaller}

<span class="fragment">
![](img/meme_monkey_side_eye.png){.absolute top=3em right=-10em width=15em}
</span>

- Gather **access** to all relevant **data sources** (databases, files etc.)
    <span class="fragment">
    - Security concerns
    - Anonymization / pseudonymization
    - Collection difficulties...
    </span>
- Get all **documentation** available
    <span class="fragment">
    - We don't have documentation
    - It hasn't been updated for years
    </span>
- **Meet people** who work with the data
    <span class="fragment">
    - We don't have time to explain...
    - It's clear from the names what the data mean...
    </span>


::: {.notes}
- security, anony - makes delays
- coll. diff. ex. 
    - we would need results of price sensitivity AB test but that is costly and delays project
    - data from app usage would be nice but it's not so easy to get to them
:::

---

<div style="height: 90px;"></div>

#### Using AI

::: {.fragment}

- Transcriptions & meeting summaries
- Documentation - extraction, summarization

:::

::: {.fragment}

::: {.callout-warning icon=false}
Be careful about **security and GDPR**.

Be doubtful - check outcomes and don't rely on them blindly.

:::
:::


## 2. First looks into the data

- It pays off to spend some time just looking at the data
    - Confrontation of first impression we got during BU or collecting existing data knowledge
    - Inspiration for sanity checks
    - Helps to formulate first hypotheses 
- Use small random sample (SRSWOR, stratified...)


## Example: loan application

- BU stage: after scoring there are two options which may lead to decline or approval
    - auto stream
    - manual stream - loan expert evaluates the customer

. . .

- First check of the Application Status Table - it's a bit more complicated...


:::: {.columns}
::: {.column width="35%"}


| time     | state        |
|---------:|:-------------|
| 10:00:00 | scoring      |
| 10:00:02 | manual check |
| 12:00:00 | scoring      |


:::

::: {.column width="20%"}
:::


::: {.column width="45%"}


| time     | state        |
|---------:|:-------------|
| 10:00:00 | scoring      |
| 10:00:02 | waits for income verif docs |
| 10:15:00 | scoring      |
| 10:15:02 | income verif failed      |
| 10:15:03 | waits for manual income verif  |


:::

::::

## 3. Data description (profiling)



:::: {.columns}
::: {.column width="50%"}

- Basic info learnable from the data itself (see next slide for examples)
- Automation tools:
    - [ydata_profiling](https://docs.profiling.ydata.ai/latest/)
    - [data_prep](https://docs.dataprep.ai/)

:::

::: {.column width="20%"}
:::

::: {.column width="30%"}
![ydata_logo](img/ydata_logo.png){height="4em"}
![data_prep_logo](img/data_prep_logo.png){height="4em"}
:::

::::

::: {.callout-tip}
- Focus on overall and 1D descriptions
- Use sample if necessary
- Subset colums if necessary
:::


## Variable types

:::: {.columns}
::: {.column width="50%"}

- **Numeric (quantitative)**
    - By metric type
        - *Continuous* - uncountable <span class="fragment">[(height...)]{.p-grey}</span>
        - *Discrete* - countable <span class="fragment">[(count of transactions...)]{.p-grey}</span>
    - By divisibility
        - *Ratio* - ratio of values makes sense <span class="fragment">[(weight, age...)]{.p-grey}</span>
        - *Interval* - ratio doesn't make sense <span class="fragment">[(temp. in °C, cal. year...)]{.p-grey}</span>
- **Categorical (qualitative)**
    - *Nominal* - no natural ordering <span class="fragment">[(marital status, city...)]{.p-grey}</span>
    - *Ordinal* - categories can be sorted and compared <span class="fragment">[(education...)]{.p-grey}</span>



:::

::: {.column width="50%"}

![](./img/var_types.png)

:::
::::

--- 

<div style="height: 90px;"></div>

From exploration perspective

- Summary stats or visualization tech. for numerics are the same (no matter cont./discrete)
- **Special types**
    - *Binary* (boolean) - categorical (nominal or ordinal), can be reformulated as indicator → behaves a bit like numeric (stats as mean make sense) <span class="fragment">[(gender, open/closed...)]{.p-grey}</span>
    - *Date/Time* - numeric but many usual stats are irrelevant
    - *Keys* - usually repr. as discrete continuous but  primary interest is uniqueness and missings


## Variable distributions

:::: {.columns}
::: {.column width="40%"}

- discrete / continuous
    - Num.: discrete with infinite support (Poisson...) or continuous (normal, exponential...) distribution
    - Cat.: discrete distribution with finite support  (Bernoulli, categorical)
- symmetrical / skewed
- heavy tailed
- unimodal / multimodal

::: {.notes}
- lognormal = log of it is normal
- chiqs = sum of ind. normal squares, chi_k = Gamma(k/2, 1/2)
- t = normal / chisq
- B(a, b) = X/(X+Y), X=Gamma(a, lambda), Y=Gamma(b, lambda) ind.
- Exp(lambda) = Gamma(1, lambda)
- Weibull - (W/lambda)^k is Exp(1)
- B(n, p) ~ N(np, np(1-p))
- B(n, p) ~ Po(np) (Poisson limit theorem: As n approaches ∞ and p approaches 0 with the product np held fixed, the Binomial(n, p) distribution approaches the Poisson distribution with expected value λ = np)
- bin. = SRSWR, HG = SRSWOR
- geom = discrete time till event
- neg. binomial - waiting not till first event but r-th
- categorical = Mult_k(1, p)

:::

:::

::: {.column width="60%"}
![](img/var_dist.png){height="15em"}
:::
::::

## Summary statistics


- **Location** - what is typical value? where is the main mass of the distribution located? 
    - Num.: [mean, median, trimmed mean...]{.p-grey}
    - Cat.: [median, mode]{.p-grey}
- **Variability** - how much can values differ? how spread are they?
    - Num.: [SD, IQR, MAD, CV, range...]{.p-grey}
    - Cat.: [levels count]{.p-grey}
- **Distribution**
    - Num.: [quantiles, skewness, kurtosis]{.p-grey}
    - Cat.: [proportions per level]{.p-grey}
- **Other artifacts**
    - % of NAs, distinct, zeroes, positive/negative, infinite
    - monotonicity
    - outliers (extreme values)
    - common values




## {.scrollable}

![](img/ydata_ds_overview.png)

:::: {.columns}
::: {.column width="50%"}

::: {.fragment}
![](img/ydata_num.png) 
![](img/ydata_num_statistics.png) 
![](img/ydata_num_extremes.png)
:::
:::

::: {.column width="50%"}
::: {.fragment}
![](img/ydata_cat.png) 
![](img/ydata_cat_cats.png)
:::
:::

::::

::: aside
[More examples](https://docs.profiling.ydata.ai/latest/getting-started/examples/)
:::

::: {.notes}
- Overall: nrow, ncol, n dupl, var types count
- Nums: 
    - location - mean, median
    - variability/scale - IQR, SD, MAD, CV
    - othr moments - skewness, kurtosis
    - % NAs, zeroes, unique, inf, neg.
    - dist - quantiles
    - extremes, common values
- Cats:
    - location - median, mode
    - % NAs
    - dist - nunique, level counts/props

:::


## 4. Data exploration

- aka **EDA** (Exploratory Data Analysis)
- Digging deeper
    - **Specificities** of variables
    - **Relationships between variables**
    - Investigation whether **hypotheses** could be true
    - Finding **patterns**
- Using more **advanced visualizations** (and possibly tables)
    - More next week
- Later: focusing on **specific parts of data**


---

<div style="height: 90px;"></div>


#### Using AI

:::: {.columns}
::: {.column width="50%"}

::: {.fragment}

- **Currently** 
    - Tools automate routine aggregations and plotting (good old automation, no genAI)
    - AI can help drafting ideas: generating more hypotheses
    - Interpretation is up to you

![](img/du_makes_more_data_xkcd.png)

:::
:::


::: {.column width="50%"}

::: {.fragment}

- **Future?**
    - Describe what table / plot says in human language
    - Pin interesting observations
    - Validate against existing knowledge
    - Recommend cleaning steps
    - ...

:::
:::
::::



## Example: loan application


#### EDA - possible status changes on aggregated level 

- **Inspired by first look** into the data → **questions**: 
    - How frequent is manual check? How does it affect duration of application process? Does it have effect on acceptance?
    - Where to put income verification in the process? Does it always happen the same?
    - What about the rest of the process (before and after scoring part), is it simplier?


![](img/ex_applications_alluvial.png)

---


- **Confrontation of three sources of DU**
    - BU / meetings to gather existing knowledge: some status transitions are not possible
    - Data tell us otherwise - and doesn't seem to be random error
    - Our own thinking → brainstorm ideas why that might be
- **Inputs for further steps**
    - Second loop of BU
        - Are our findings valid, why initial impression was different?
        - Why income verification has multiple different ways to be done?
    - Data selection based on quality
        - What can be mined without leaks?
        - Are there any applications we should drop?

![](img/ex_applications_alluvial.png)


<!-- # Extra

## Data Quality & Stability Checks

**TBD**

- GE
- evidently -->
