Python toolkit [1]

• Data manipulation
  • Pandas 
  • Apache Spark (pyspark) 
  • Numpy
• Data visualization
  • matplotlib
  • seaborn
  • seaborn.objects
  • plotly
  • plotnine

Python toolkit [2]

• Stat and Machine learning
  • scikit-learn (and scikit-* family) 
  • statsmodels
  • many others: prophet (time-series data), h2o, …
• Gradient boosting
  • catboost
  • lightgbm
  • Xgboost
• MLops
  • mlflow
  • metaflow

Python toolkit [3]

• NLP
  • nltk
  • spacy
  • gensim
• Vision
  • opencv (cv2)
  • scikit-image 
• Graphs
  • networkX
  • snap-stanford

Pandas basics

• import pandas as pd
• pd.Series
  • 1d labeled array
  • May contain mixed data types
• pd.DataFrame
  • 2d array, aka table
  • a dict like container of pd.Series
• Axes
  • 0 – columns (column-wise)
  • 1 – index (row-wise)

Pandas indices and columns

• Index
  • aka primary key for pd.Series and pd.DataFrame (indicating rows)
  • without UNIQUE constraint!
  • df.index – access the index
  • df.set_index(col_name), df.reset_index()
• Columns - column index of a pd.DataFrame
  • df.columns
  • Change colnames via assigning:
    • df.columns = ["a", "b", "c"]
    • df.columns = [c.lower() for c in df.columns]

DataFrame - example

Data types

Pandas I/O

• Files
  • json, csv, tsv –> pd.read_csv() / pd.read_json()
  • parquet - (pyarrow / fastparquet) –> pd.read_parquet()
  • excel - (openpyxl) –> pd.read_excel()
  • pickle  –> pd.read_pickle()
• Database
  • sqlalchemy –> create_engine –> pandas
    • pyodbc string needed
    • e.g.: pd.read_sql_table(tabname, con=engine) or pd.read_sql_query()
• Writing
  • to_*() family of functions (csv, json, parquet, excel)
  • use compression to reduce file-size and speedup IO (.gz, .zip)

Getting the right data

• By columns
  • A single column df[“age”]  –> pd.Series
  • Many columns df[[“Age”, “Sex”]]  –> pd.DataFrame
  • By index of a column df.iloc[:, 0], df.iloc[:, [2,3]], df.iloc[:, 2:6]
• By rows
  • By index value df.loc[0], df.loc[0:5]
  • By integer df.iloc[0], df.iloc[0:3]
• Subset
  • By condition df[df["age"] >= 30]
  • By multiple conditions df[(df["age"] >= 30) & (df["sex"] == "female")]
  • Boolean indexing operators & (and), | (or), ~ (not)
  • Tip: .query() method

DataFrame operations & attributes

• df.head(), df.tail(), df.sample(n=12) - a quick glimps at data

• df.columns, df.shape - data dimmensions

• df.count()

• df.describe() - summary stats for numeric columns

• df["sex"].value_counts() - frequency table

• df.sort_index() or df.sort_values("column", ascending=True)

  • You can sort by multiple columns – df.sort_values(["a", "b"])

• df[“val”].astype("newtype") - change dtype

DataFrame operations & attributes

• df.drop(labels, axis="columns")

• df.drop_duplicates()

  • You can specify which columns check for duplicities via ‘subset’

• df.rename({"A": "B"}, axis="columns") - rename

• df["sex"].map({"male": 0, "female": 1}) - relevel

• df["age"].replace(0, 999) - replace

• pd.cut(df["age"], bins=[17, 21, 35, 45, 100])

DataFrame operations & attributes

• df.assign() - ❗ Deriving new columns
• syntax of params:
  • new_column_name = fun, where fun is a function operating on the df
• You can work with columns derived on preceding rows of .assign

import numpy as np 

df[["fare", "alive"]].assign(
    is_alive = lambda d: np.where(d["alive"] == "yes", 1, 0),
    fare_sq = lambda d: d["fare"]**2,
    fare_sq_sqrt = lambda d: d["fare_sq"]**(1/2)
).head()

DataFrame piping

df2["is_male"] = df2["gender"] == 'male'
male_proportion_by_class = df2.groupby("pclass").agg({"is_male": "mean"})
male_proportion_by_class.sort_values().head(1)

vs

(
    df
    .assign(is_male = lambda d: d["sex"] == 'male')
    .groupby("pclass")
    .agg({"is_male": "mean"})
    .sort_values()
    .head(1)
)

Missing data - example

Missing data

• Missing completly at random
  • errors during data collection or data processing, in a non systematic way
• Missing at random
  • Missings caused by known facts only (e.g., not having a wife –> unknown wife’s age)
• Missing not at random - Missings caused by unkown variables, too (e.g., rich people not motivated enough to fill a poll –> bias)
• Typical strategies to deal with missing data
  • Drop column - df.drop(columns=["colA"])
  • Drop rows  - df.dropna(subset=["colA"])
  • Impute with a constant (mean, mode, 0), e.g.: df.fillna({"colA": 0})
  • Impute with a model 

Missings in Pandas

• None – python general representation of missing value

• np.nan – Numpy’s NaN is usually used in pandas 

• Note: By default, NaNs are excluded from aggregate functions 

• To check whether a value is missing

  • df["age"].isna() or df["age"].notna()

• We can drop rows with missing values 

  • df.dropna(subset=["age"])

• We can fill missing values with a constant 

  • df["age"].fillna(val)

Summary tables

• How to produce a summary table for two or more categorical variables?
  • pd.cross_table – frequency table
  • df.pivot_table – any aggregation fn

Combining tables - merge Docs

• Database like join – df.merge 
• Either on (or a pair left_on, right_on) defines colname(s) to proceed the merge with
  • You can use left_index = True / right_index as well
• how defines the type of the join
  • left, right, outer, inner, cross

Combining tables - concat Docs

• We can also glue dataframes together by rows/columns using pd.concat 

Aggregations

• Split a data into groups and compute summary statistics for each group 
  • Tip: use as_index=False to keep the grouping col(s)

(
    df
    .groupby([group_key, another_one])
    .agg(example = ("column_name", "agg_function"))
)

• Aggregate functions
  • min, max, mean, nunique, sum, size, count, var, sem (standard error of the mean), descrie
  • first, last, nth