Unit-2: Data Understanding and Preparation

 

Introduction -

Data understanding and preparation are foundational steps in the data science workflow (often formalized in CRISP-DM) where we carefully inspect and clean data before modeling. In the data understanding phase one explores the raw data (using summary tables and graphs) to assess its structure and quality. This helps spot problems (like wrong types or missing values) before extensive modeling. 

Reading Data from Various Sources

Data can come in many formats. Common sources include flat files (CSV, TSV), spreadsheets (Excel), relational databases and APIs (often returning JSON or XML). For example, in R-programme one can use to load data easily

Reading Data from Various Sources in R

Reading CSV

# Reading a CSV file
install.packages("readr")

library(readr)

data <- read_csv("path/to/your/file.csv")
Or
# Import a CSV file
data <- read.csv("data.csv")

Reading Excel

library(readxl)

# For reading Excel files (.xls, .xlsx)

install.packages("readxl")

data <- read_excel("data.xlsx")

Reading from Database

library(DBI)
con <- dbConnect(RSQLite::SQLite(), "mydatabase.sqlite")
data <- dbGetQuery(con, "SELECT * FROM tablename")
dbDisconnect(con)

Reading from Web API (JSON)

library(jsonlite)

data <- fromJSON("https://api.example.com/data")

Data Visualization for Exploration

Visualizing data helps reveal patterns, distributions and anomalies that raw tables hide. Common charts for initial exploration include histograms, scatter plots and box plots. For example, a histogram of ages shows how many observations fall into each age bin, while a scatter plot of height vs. weight can reveal a positive trend. Box plots overlay information about spread and outliers in one view. By plotting such charts, analysts can quickly see the shape of each distribution and relationships between variables. In practice, exploratory plots are used to visually check assumptions and to identify features of interest.

Histograms – show the distribution of one numeric variable. Useful to see skew or multimodality. 

R-Code

hist(data$age, main="Age Distribution", xlab="Age", col="skyblue")

Scatter plots – This plot two variables (x vs. y) to assess correlation. Clustering or linear patterns become apparent.

R-Code

plot(data$height, data$weight, main="Height vs Weight", xlab="Height", ylab="Weight")

Box plots – Its display median, quartiles and extreme values of a variable. Points beyond the whiskers often flag potential outliers.

R-Code

boxplot(data$income, main="Income Boxplot", ylab="Income")

Pair Plot - A pair plot (or scatterplot matrix) displays scatter plots between each pair of selected numerical variables. This helps in identifying linear relationships, correlations, clusters, or outliers among features. The diagonal typically shows the distribution (such as a histogram) of individual variables, making it a compact tool for multivariate exploration.

R-Code

pairs(data[,c("height", "weight", "income")])

Each plot gives complementary insight: e.g. a box plot for a variable can highlight outliers that a histogram might only hint at, while scatterplots make it easy to see if two features move together. Combining tables and plots is a key part of early data exploration.

Understanding Distributions and Summary Statistics

Describing a dataset statistically is crucial. The distribution of each variable can be summarized by measures of central tendency and spread and by shape descriptors:

  • Mean (average) – sum of values divided by count. It indicates the “center” of data.

  • Median – middle value when data are ordered. It is robust to outliers (less affected by extreme values than the mean).

  • Mode – the most frequent value in the data. Useful for categorical or discrete data.

These central tendency measures tell us where most data lie. In other words: “the mean is the average… the median is the middle value… the mode is the most frequent value”.

  • Variance – average of the squared deviations from the mean; it quantifies how spread out the values are. High variance means data are widely spread.

  • Standard Deviation (SD) – square root of variance; also measures spread, but in the same units as the data. A larger SD means more variability.

Variance and SD measure dispersion. In fact, “Variance measures the dispersion of a dataset, indicating how much the values differ from the mean. It is the average of the squared differences from the mean”. The standard deviation is simply its square root, giving an intuitive scale of typical deviation from the mean.

Skewness and kurtosis describe the shape of the distribution beyond these basics. Skewness measures asymmetry: a right-skewed (positively skewed) distribution has a long tail to the right. A symmetric (normal) distribution has skewness is 0. So, skewness tells us if the bulk of the data lies to one side of the mean.

Kurtosis measures tail “tailedness” or peakedness. Informally, it indicates how heavy or light the tails are relative to a normal curve. As one source explains, “Kurtosis is the measure of the thickness or heaviness of the tails of a distribution”. High kurtosis (leptokurtic) means more data in tails (outliers more likely); low kurtosis (platykurtic) means lighter tails.

Together, mean/median/mode, variance/SD and skewness/kurtosis provide a descriptive profile of each variable’s distribution – telling us where data tend to cluster and how variable and asymmetric it is.

In R- 

Summary Statistics

summary(data)

Mean, Median, Mode

mean(data$income)
median(data$income)
Mode <- function(x) {
  ux <- unique(x)
  ux[which.max(tabulate(match(x, ux)))]
}
Mode(data$income)

Variance and Standard Deviation

var(data$income)
sd(data$income)

Skewness and Kurtosis

library(e1071)
skewness(data$income)
kurtosis(data$income)

Identifying Relationships Among Variables

Beyond single-variable summaries, we also examine how variables relate. Two key concepts are covariance and correlation. Covariance between X and Y measures how they vary together: 

formally Cov(X,Y)=E[(X–μ)(Y–μ)]. 

A positive covariance means X and Y tend to increase together; negative means one increases while the other decreases. However, covariance is scale-dependent and hard to interpret directly.

The Pearson correlation coefficient (Pearson’s r) standardizes covariance to range −1…+1. In statistics it is defined as the covariance divided by the product of the standard deviations. In other words, it measures linear correlation:

“the Pearson correlation coefficient… measures linear correlation between two sets of data. It is the ratio between the covariance of two variables and the product of their standard deviations; thus… it is essentially a normalized measurement of the covariance, such that the result always has a value between −1 and 1.”.

A value near +1 indicates a strong positive linear relationship (points lie roughly along an upward line), near −1 indicates a strong negative linear relationship, and near 0 indicates little linear relation. In Python/pandas, one can quickly compute a correlation matrix with df.corr() to see all pairwise Pearson correlations.

A scatter plot matrix (pairplot) is also commonly used: it creates a grid of scatter plots for every pair of numeric variables (often with histograms on the diagonal). This visual summary lets us eyeball relationships or clusters in multi-dimensional data. For example, if several variable pairs show tight linear patterns on the scatter matrix, we know those pairs have high correlation. In pandas one can use pd.plotting.scatter_matrix(df) or Seaborn’s pairplot to generate this.

Handling Missing Data

Real datasets often have missing values. These must be detected and handled, as many analyses and models require complete data. A typical first step is to detect missing entries (for example, in pandas df.isnull().sum() counts nulls per column). Missingness can be random or systematic; understanding its pattern is important.

Once identified, common strategies are:

  • Deletion: Remove any rows (or even whole columns) with missing values. This is simple but can discard a lot of data. As noted, “deleting rows or columns containing missing values… can result in significant data loss”. This approach is best when missingness is rare.

  • Imputation: Fill in missing values with plausible replacements. Simple imputation replaces a missing numeric entry with the mean (or median) of that column, or a missing categorical entry with the mode. For example, mean-imputation is easy (df[col].fillna(df[col].mean())), though it can misrepresent variance. Median imputation is more robust to outliers. More advanced imputation methods include forward/backward filling in time series, K-nearest neighbors (filling based on similar rows), regression models, or even iterative/multiple imputation techniques. The choice depends on context and how much missingness there is.

  • Other techniques: In some cases, missingness itself carries information, so one can add an indicator variable flagging missing entries. Or one can predict missing values using models (e.g. training a regression to predict missing salary from other features).

As one author warns, ignoring missing data can bias results: “Missing data can also alter key statistical properties… Mean, variance, and correlation can become distorted, leading to inaccurate conclusions. Therefore, handling missing data correctly is crucial”. Proper handling (whether by deletion or imputation) yields more reliable summary statistics and models. Pandas provides convenient methods (df.fillna(), df.dropna(), etc.) and scikit-learn has imputation transformers, but the key is to decide on a method appropriate for the dataset.

Data Segmentation

Data segmentation means breaking the dataset into meaningful groups or subsets for analysis. In a supervised setting (with labeled data), segmentation can be as simple as grouping by class label or another key attribute (e.g. customer segments by region). This allows separate analysis of each segment or training specialized models per group. In contrast, unsupervised segmentation (clustering) discovers groups without labels. A common approach is k-means clustering or hierarchical clustering on the features, which partitions data into clusters that are similar internally. 

In other words: “Data segmentation is the process of breaking down a dataset into discrete groups according to specific standards or attributes”. In supervised segmentation, the “standards” are the known labels or categories; in unsupervised, the algorithm learns to group data points into homogeneous clusters.

For instance, in clustering we might compute distances between points and assign them to k groups (k-means) or use DBSCAN to find dense regions. The resulting segments (clusters) should exhibit intra-group similarity and inter-group differences. Segmentation can improve downstream tasks by allowing models to focus on one kind of pattern at a time. In practice, one often tries several techniques (e.g. k-means, Gaussian mixtures, etc.) and uses metrics like the silhouette score or domain knowledge to choose the number of clusters. In all cases, segmentation helps manage heterogeneity by analyzing one segment at a time.

Outlier Detection

Outliers are individual observations that deviate markedly from the rest of the data. They may arise from data entry errors, measurement issues or true rare events. Detecting outliers is important because they can skew analyses (especially the mean and standard deviation) or adversely affect models.

Common methods include:

  • Z-score method: Compute the z-score (standard score) for each data point, which is (value – mean)/SD. Observations with |z| typically above 3 are often flagged as outliers because in a normal distribution about 99.7% of data lie within 3 SD of the mean).

  • Interquartile Range (IQR) method: Use the box-plot rule. Calculate Q1 (25th percentile) and Q3 (75th percentile), compute IQR = Q3−Q1. Then define “fences” at Q1−1.5·IQR and Q3+1.5·IQR. Any value outside these fences is considered an outlier. In other words: “Any observations that are more than 1.5 IQR below Q1 or more than 1.5 IQR above Q3 are considered outliers”. This rule is exactly how box plots typically mark outliers (the points beyond whiskers).

  • Visual inspection: Plotting can also reveal outliers. For example, a box plot will show potential outliers as individual points; a scatter plot may reveal points far from any cluster of others; a histogram may show isolated bars. Points identified as outliers by any method can then be examined individually.

Once detected, one must decide how to handle outliers: possibly remove them, transform the data, or keep them if they carry important information. Care is needed because an outlier in one context (rare but legitimate event) might simply be noise in another. Nonetheless, the 1.5·IQR rule or z-scores give a systematic way to identify candidates for further review.


Comments

Popular posts from this blog

Model Paper

Unit-4: Data Preparation and Model Management

Unit-3: Model Development and Techniques