DI04000061: Introduction Machine Learning
Computer Engineering

4th Semester

Computer Engineering

DI04000061: ML

4th Semester

1 / 455

Outline
1

Introduction to Machine Learning

2

Python libraries for Machine Learning

3

Preparing to Model and Evaluation

4

Preparing to Model and Evaluation

5

Supervised Machine Learning

6

Supervised Machine Learning

7

Unsupervised Machine Learning and Generative AI
Computer Engineering

DI04000061: ML

4th Semester

2 / 455

What is Learning? I

Defining Human Learning
Human learning is the cognitive process of acquiring new understanding, knowledge, behaviors, skills, values, attitudes, and preferences through experience, study,
or instruction.

Defining Machine Learning (Tom Mitchell, 1997)
A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T , as
measured by P, improves with experience E .

Task (T ): Image classification, email spam filtering, medical diagnosis.
Performance (P): Accuracy, precision, recall, mean squared error.
Experience (E ): Dataset of labeled examples, historical interactions.

Computer Engineering

DI04000061: ML

4th Semester

3 / 455

Human Learning vs. Machine Learning Paradigm I

Human Learning

Machine Learning

Input: Sensory input, observation, reasoning, contextual experience.
Process: Synaptic plasticity, intuition, abstraction, continuous
lifelong learning.
Data Efficiency: Learns concepts from very few examples (few-shot /
one-shot).
Generalization: Transfers knowledge across completely different
domains effortlessly.

Computer Engineering

DI04000061: ML

Input: Structured numerical vectors, matrices, text, images, or audio
tensors.
Process: Mathematical optimization, loss minimization, gradient
descent.
Data Efficiency: Requires large statistical samples for reliable
generalization.
Generalization: Domain-specific; highly vulnerable to distribution
shifts.

4th Semester

4 / 455

Traditional Programming vs. Machine Learning I
Traditional Paradigm (Rule-Based Software)
Engineers write explicit rules (algorithms) that operate on input data to produce outputs.
Data + Rules (Code) −→ Answers

Machine Learning Paradigm (Data-Driven Software)
Algorithms process data alongside known answers (labels) to automatically discover the underlying rules.
Data + Answers (Labels) −→ Rules (Model)

Key Distinction
In traditional programming, logic is explicitly hardcoded. In ML, logic is inferred statistically from empirical data distributions.

Computer Engineering

DI04000061: ML

4th Semester

5 / 455

Mathematical Formalization of Machine Learning I

Formal Setup
Let X denote the Input Space (feature vectors) and Y denote the Output Space (labels or target values).
Unknown Target Function: f : X → Y (the true underlying relationship)
Training Dataset: D = {(x1 , y1 ), (x2 , y2 ), . . . , (xN , yN )} drawn i.i.d. from distribution P(X , Y )
Hypothesis Set: H = {h1 , h2 , . . . , hm } candidate functions approximating f
Learning Algorithm A: Maps dataset D to select final hypothesis g ∈ H such that g ≈ f

Goal of Machine Learning
The objective is not merely to memorize D, but to find g that achieves minimal expected error on unseen samples sampled from P(X , Y ) (Generalization).

Computer Engineering

DI04000061: ML

4th Semester

6 / 455

Taxonomy of Machine Learning I

Supervised

Unsupervised

Reinforcement

Data: Labeled pairs (x, y )

Data: Unlabeled inputs {x}

Data: State-Action-Reward tuples

Goal: Predict target y given input x

Goal: Discover underlying structure or
pattern

Goal: Learn policy π(a|s) to maximize
cumulative reward

Tasks:

Tasks:

Tasks:

Classification
(y ∈ {0, 1})
Regression (y ∈ R)

Computer Engineering

Clustering
Dimensionality
Reduction
Density Estimation

Robotics control
Game playing
(AlphaGo)

DI04000061: ML

4th Semester

7 / 455

Applications: Computer Vision & Natural Language Processing I

Computer Vision (CV)
Extracting meaningful semantic information from digital images and videos.
Medical Diagnostics: Automated detection of tumors from MRI/CT scans.
Autonomous Mobility: Pedestrian and lane detection for self-driving vehicles.
Facial Recognition: Biometric security and authentication systems.

Natural Language Processing (NLP)
Enabling machines to understand, interpret, generate, and process human language.
Large Language Models (LLMs): Contextual text generation, code synthesis, reasoning.
Machine Translation: Real-time neural translation across hundreds of languages.
Sentiment Analysis: Financial market analysis and customer opinion mining.

Computer Engineering

DI04000061: ML

4th Semester

8 / 455

Applications: Finance, Healthcare & Smart Systems I

Financial Engineering & Fintech
Fraud Detection: Real-time anomaly detection on millions of credit card transactions.
Algorithmic Trading: Predictive high-frequency modeling of equity price dynamics.
Credit Scoring: Assessing loan default risk using multi-source feature sets.

Healthcare & Precision Medicine
Drug Discovery: Predicting molecular binding affinities to accelerate drug design.
Genomics: Analyzing DNA sequencing data to identify disease markers.
Personalized Treatment: Recommending tailored therapeutics based on patient profiles.

Computer Engineering

DI04000061: ML

4th Semester

9 / 455

Code Example: Classical Supervised ML Workflow I
i m p o r t numpy a s np
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
from s k l e a r n . l i n e a r m o d e l i m p o r t L o g i s t i c R e g r e s s i o n
from s k l e a r n . m e t r i c s i m p o r t a c c u r a c y s c o r e
# 1. Generate Synthetic Dataset ( Features X, Labels y )
np . random . s e e d ( 4 2 )
X = np . random . r a n d n ( 2 0 0 , 2 )
y = (X [ : , 0 ] + X [ : , 1 ] > 0 ) . a s t y p e ( i n t )
# 2 . P a r t i t i o n i n t o T r a i n and T e s t S e t s
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42
)
# 3 . I n s t a n t i a t e and F i t H y p o t h e s i s Model ( g i n H)
model = L o g i s t i c R e g r e s s i o n ( )
model . f i t ( X t r a i n , y t r a i n )
# 4 . E v a l u a t e P e r f o r m a n c e P on Unseen Data E
y p r e d = model . p r e d i c t ( X t e s t )
accuracy = accuracy score ( y test , y pred )
p r i n t ( f ” T e s t A c c u r a c y (P ) : { a c c u r a c y ∗ 1 0 0 : . 2 f}%” )

Computer Engineering

DI04000061: ML

4th Semester

10 / 455

Summary and Core Takeaways I

Key Concepts Covered
1

Definition of ML: Mitchell’s (T , P, E ) framework formalizes machine learning.

2

Paradigm Shift: Moving from manually engineered rules to data-driven statistical learning.

3

Core Goal: Achieving robust generalization on unseen data, rather than overfitting or memorizing training samples.

4

Pervasive Impact: ML powers critical domains including healthcare, finance, CV, NLP, and autonomous navigation.

Looking Ahead: Lecture 1.2
In the next lecture, we will explore the foundational mathematical building blocks of machine learning: Linear Algebra, Multivariate Calculus, and Probability
Theory.

Computer Engineering

DI04000061: ML

4th Semester

11 / 455

Taxonomy of Machine Learning I

The Three Core Paradigms
Machine Learning algorithms are broadly categorized based on the nature of the learning signal or feedback available during training.

Supervised

Unsupervised

Reinforcement

Labeled training data

Unlabeled data

Decision agent

Predict targets (y )

Discover patterns

Environment interaction

Direct feedback

No explicit feedback

Reward signals

Regression & Classification

Clustering & Reduction

Policy optimization

Computer Engineering

DI04000061: ML

4th Semester

12 / 455

Supervised Learning: Mathematical Formulation I

Formal Definition
Given a training dataset of N input-output pairs:
D = {(x
where x

(i)

d

∈ X ⊆ R represents feature vectors and y

(i)

(1)

,y

(1)

), (x

(2)

,y

(2)

), . . . , (x

(N)

,y

(N)

)}

∈ Y represents target labels.

Learning Goal
Learn a predictive function hθ : X → Y parameterized by θ such that hθ (x) accurately predicts y on unseen data by minimizing empirical risk:

θ

∗

= arg min
θ

N
1 X

N i=1



(i)
(i)
L y , hθ (x ) + λR(θ)

where L(·, ·) is a loss function and R(θ) is a regularization penalty.

Computer Engineering

DI04000061: ML

4th Semester

13 / 455

Supervised Learning: Classification vs. Regression I

Classification Task

Regression Task

Target Space: Discrete label set Y = {1, 2, . . . , K } or binary
Y ∈ {0, 1}.

Target Space: Continuous values Y ⊆ R.
Objective: Fit continuous surface estimating quantitative values.

Objective: Learn decision boundaries separating classes.

Loss Function: Mean Squared Error (MSE):

Loss Function: Categorical Cross-Entropy Loss:

L=−

K
X

L=
yk log(ŷk )

X (i)
1 N
(i) 2
(y − ŷ )

N i=1

k=1

Examples: Spam filtering, medical diagnosis, image recognition.

Computer Engineering

DI04000061: ML

Examples: Real estate valuation, stock forecasting, temperature
prediction.

4th Semester

14 / 455

Supervised Learning: Scikit-Learn Example I

from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
from s k l e a r n . e n s e m b l e i m p o r t R a n d o m F o r e s t C l a s s i f i e r
from s k l e a r n . m e t r i c s i m p o r t a c c u r a c y s c o r e
i m p o r t numpy a s np
# 1. Generate s y n t h e t i c dataset ( Features X, Labels y )
X = np . random . r a n d n ( 1 0 0 0 , 1 0 )
y = (X [ : , 0 ] + X [ : , 1 ] > 0 ) . a s t y p e ( i n t )
# 2 . S p l i t d a t a s e t i n t o t r a i n i n g and t e s t i n g s e t s
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42
)
# 3 . I n s t a n t i a t e and t r a i n s u p e r v i s e d model
model = R a n d o m F o r e s t C l a s s i f i e r ( n e s t i m a t o r s =100)
model . f i t ( X t r a i n , y t r a i n )
# 4. Evaluate pr e d i c t io n s
y p r e d = model . p r e d i c t ( X t e s t )
p r i n t ( f ” A c c u r a c y : { a c c u r a c y s c o r e ( y t e s t , y p r e d ) : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

15 / 455

Unsupervised Learning: Principles & Tasks I

Formal Definition
Given an unlabeled dataset of N observations:
D = {x

(1)

,x

(2)

,...,x

(N)

},

x

(i)

d

∈R

The goal is to model the underlying probability density function p(x) or discover intrinsic geometric structures without ground-truth targets.

Clustering

Dimensionality Reduction

Group data into homogeneous clusters based on similarity metrics (e.g.,
Euclidean distance).

Compress high-dimensional data Rd → Rk (k ≪ d) while preserving
variance.

K-Means Clustering

Principal Component Analysis (PCA)

Hierarchical Clustering

t-SNE & UMAP

DBSCAN

Autoencoders

Computer Engineering

DI04000061: ML

4th Semester

16 / 455

Unsupervised Learning: Key Applications I

Practical Use Cases
Customer Segmentation: Grouping customers by purchasing behavior for targeted marketing.
Anomaly Detection: Identifying fraudulent transactions or fault conditions by detecting low-density data regions.
Data Visualization: Projecting high-dimensional embeddings (e.g., word vectors) to 2D/3D space.
Feature Extraction: Learning compact representations prior to downstream supervised modeling.

Key Challenge
Evaluating unsupervised models is inherently complex due to the absence of ground-truth labels. Metrics rely on internal cohesion and separation (e.g., Silhouette
Score).

Computer Engineering

DI04000061: ML

4th Semester

17 / 455

Unsupervised Learning: Scikit-Learn Example I

from s k l e a r n . c l u s t e r i m p o r t KMeans
from s k l e a r n . d e c o m p o s i t i o n i m p o r t PCA
i m p o r t numpy a s np
# 1. Unlabeled feature matrix
X = np . random . r a n d n ( 5 0 0 , 2 0 )
# 2 . D i m e n s i o n a l i t y R e d u c t i o n : Compress 20D t o 2D
pca = PCA( n c o m p o n e n t s =2)
X r e d u c e d = pca . f i t t r a n s f o r m (X)
p r i n t ( f ” E x p l a i n e d V a r i a n c e R a t i o : {pca . e x p l a i n e d v a r i a n c e r a t i o }” )
# 3. Clustering : Discover 3 latent c l u s t e r s
kmeans = KMeans ( n c l u s t e r s =3 , r a n d o m s t a t e =42 , n i n i t =10)
c l u s t e r l a b e l s = kmeans . f i t p r e d i c t ( X r e d u c e d )
# Output c l u s t e r c e n t r o i d s i n 2D s p a c e
p r i n t ( ” C l u s t e r C e n t e r s : \ n” , kmeans . c l u s t e r c e n t e r s )

Computer Engineering

DI04000061: ML

4th Semester

18 / 455

Reinforcement Learning: Agent-Environment Interface I

Core Paradigm
An agent learns optimal decision-making behavior through trial-and-error interaction with a dynamic environment.

Markov Decision Process (MDP)

Optimization Objective

Formulated as a tuple (S, A, P, R, γ):

Find policy π(a|s) maximizing expected return:

State Space S: Current state st .
Action Space A: Action at executed.

Gt =

Transition Prob.: P(st+1 |st , at ).

∞
X
k
γ rt+k+1
k=0

Reward Function: R(st , at ) → rt+1 .
Value Function:

Discount Factor: γ ∈ [0, 1).

π

V (s) = Eπ [Gt | st = s]

Computer Engineering

DI04000061: ML

4th Semester

19 / 455

Reinforcement Learning: Characteristics & Applications I

The Exploration-Exploitation Dilemma
Exploration: Trying novel actions to discover potential high long-term rewards.
Exploitation: Leveraging current knowledge to maximize immediate expected rewards.
Balancing this trade-off is central to RL algorithms (e.g., ϵ-greedy action selection).

Modern Applications of RL
Autonomous Systems: Self-driving path planning, robotic manipulation.
Complex Games: DeepMind AlphaGo, AlphaZero, OpenAI Five.
LLM Alignment: Reinforcement Learning from Human Feedback (RLHF) used in state-of-the-art AI models.

Computer Engineering

DI04000061: ML

4th Semester

20 / 455

Comparison of Paradigms & Hybrid Methods I

Taxonomy Comparison Matrix
Paradigm
Supervised
Unsupervised
Reinforcement

Data Input
Labeled (x, y )
Unlabeled (x)
Environment states

Feedback
Direct (Loss)
None
Delayed (Reward)

Primary Goal
Predict y from x
Discover patterns / p(x)
Maximize cumulative return

Hybrid Learning Paradigms
Semi-Supervised Learning: Small set of labeled data + large volume of unlabeled data.
Self-Supervised Learning: Model creates artificial labels from input data structure (e.g., Masked Language Modeling in BERT).
Transfer Learning: Reusing representations learned on large datasets for domain-specific tasks.

Computer Engineering

DI04000061: ML

4th Semester

21 / 455

Topic 1.6: Tools & Technology Stack for ML I

Modern Machine Learning Infrastructure
The ML software ecosystem spans data wrangling, algorithmic modeling, deep learning, and hardware acceleration.

Machine Learning Stack

Core Data Science Stack
Python: Primary programming language for ML/DL.

Scikit-Learn: Industry standard for classical ML algorithms.

NumPy: Vectorized matrix computations.

XGBoost / LightGBM: Gradient boosted decision trees.

Pandas: High-performance tabular data structures (DataFrames).

PyTorch / TensorFlow: Deep learning framework with automatic
differentiation.

Matplotlib / Seaborn: Exploratory data visualization.

Jupyter / Google Colab: Interactive environments.

Computer Engineering

DI04000061: ML

4th Semester

22 / 455

Scikit-Learn: Design Principles & API Pattern I

Unified Estimator API
Scikit-Learn enforces a consistent object-oriented interface across all machine learning algorithms:
Estimator (model.fit(X, y)): Learns model parameters from data.
Transformer (transformer.transform(X)): Preprocesses or scales features (e.g., StandardScaler).
Predictor (model.predict(X)): Generates predictions on new feature arrays.

Design Advantage
Allows seamless pipeline construction (Pipeline([scaler, model])), preventing data leakage between training and validation folds.

Computer Engineering

DI04000061: ML

4th Semester

23 / 455

Deep Learning Ecosystem: PyTorch vs. TensorFlow I

PyTorch (Meta AI)

TensorFlow / Keras (Google)

Dynamic Computational Graph (Eager execution by default).

Static / Hybrid Graph execution via tf.function.

Pythonic syntax, intuitive debugging.

Robust production deployment ecosystem (TF Serving, TF Lite,
TF.js).

Dominant framework in modern academic research and LLM
ecosystem (Hugging Face).

High-level Keras API for rapid prototyping.

Native GPU tensor operations via CUDA.

Computer Engineering

Strong enterprise production track record.

DI04000061: ML

4th Semester

24 / 455

Complete Python ML Pipeline Example I
from
from
from
from
from
from

s k l e a r n . d a t a s e t s import l o a d i r i s
s k l e a r n . m o d e l s e l e c t i o n import t r a i n t e s t s p l i t
s k l e a r n . p r e p r o c e s s i n g import S t an d ar d Sc a l er
s k l e a r n . p i p e l i n e import m a k e p i p e l i n e
s k l e a r n . l i n e a r m o d e l import L o g i s t i c R e g r e s s i o n
s k l e a r n . m e t r i c s import c l a s s i f i c a t i o n r e p o r t

# 1 . Load d a t a s e t
i r i s = l o a d i r i s ()
X , y = i r i s . data , i r i s . t a r g e t
# 2 . T r a i n−t e s t s p l i t
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 5 , r a n d o m s t a t e =42
)
# 3 . C r e a t e s t a n d a r d i z e d ML p i p e l i n e
pipeline = make pipeline (
StandardScaler () ,
L o g i s t i c R e g r e s s i o n ( m a x i t e r =200)
)
# 4. Train p i p e l i n e & Evaluate
p i p e l i n e . f i t ( X train , y t r a i n )
y pred = p i p e l i n e . predict ( X test )
p r i n t ( c l a s s i f i c a t i o n r e p o r t ( y t e s t , y p r e d , t a r g e t n a m e s= i r i s . t a r g e t n a m e s ) )

Computer Engineering

DI04000061: ML

4th Semester

25 / 455

Summary & Key Takeaways I

Lecture 1.2 Summary
Supervised Learning: Requires labeled pairs (x, y ) for classification and regression tasks.
Unsupervised Learning: Extracts intrinsic representations or clusters from unlabeled data x.
Reinforcement Learning: Optimizes decision-making policy via agent-environment reward interaction.
Tooling Stack: Python ecosystem provides unified tools: NumPy/Pandas for data, Scikit-Learn for classical models, and PyTorch/TensorFlow for deep
learning.

Computer Engineering

DI04000061: ML

4th Semester

26 / 455

Lecture Overview & Learning Objectives I

Course: DI04000061 (Introduction Machine Learning)
Unit 1: Introduction to Machine Learning
Topic: 1.3 Benefits of Machine Learning
Learning Objectives:
1

Differentiate between traditional algorithmic programming and data-driven learning paradigms.

2

Identify key advantages of Machine Learning in handling complex, high-dimensional, and non-linear patterns.

3

Understand how adaptability and continuous model updates drive business and technical value.

4

Analyze real-world applications demonstrating ML efficiency over rule-based systems.

Computer Engineering

DI04000061: ML

4th Semester

27 / 455

Paradigm Shift: Traditional Programming vs. Machine Learning I

Traditional Software Engineering

Machine Learning Paradigm

Inputs: Explicit Rules (Code) + Data.

Inputs: Historical Data + Answers (Targets).

Output: Answers / Decisions.

Output: Learned Model / Rules (f : X → y ).

Limitation: Requires humans to manually program every rule, edge
case, and logical branch.

Advantage: Algorithmic discovery of complex relationships directly
from empirical data.

Brittle: Fails when problem complexity scales or data patterns shift.

Scalable: Automatically updates with new data streams.

Computer Engineering

DI04000061: ML

4th Semester

28 / 455

1. Solving Problems Beyond Human Rule Crafting I

The Limits of Hard-Coded Logic
For perception and cognitive tasks (e.g., Computer Vision, Natural Language Processing), writing explicit rules is intractable:
Rules for recognizing a cat =⇒ Millions of pixel combinations, lighting, angles, breeds.
Automatic Feature Discovery: Deep Neural Networks extract hierarchical features (edges → textures → object parts) without manual feature engineering.
Handling Non-Linearity: ML models approximate complex non-linear functions y = f (x) over arbitrary vector spaces.
High Dimensionality: Operates effectively in high-dimensional feature spaces (x ∈ Rd where d ≫ 106 ).

Computer Engineering

DI04000061: ML

4th Semester

29 / 455

Code Comparison: Rule-Based vs. Machine Learning I

Spam Detection: Hardcoded Rules vs Naive Bayes
Traditional rule-based systems decay quickly as spammers change tactics; ML adapts effortlessly.

# −−− T r a d i t i o n a l Rule−Based Approach −−−
def i s s p a m r u l e s ( e m a i l t e x t ) :
k e y w o r d s = [ ”BUY NOW” , ”FREE MONEY” , ”CLICK HERE” , ”WINNER” ]
s c o r e = sum ( 1 f o r kw i n k e y w o r d s i f kw i n e m a i l t e x t . u p p e r ( ) )
r e t u r n s c o r e >= 2 # B r i t t l e : m i s s e s s u b t l e spam , f l a g s s a f e m a i l
# −−− Machine L e a r n i n g Approach −−−
from s k l e a r n . f e a t u r e e x t r a c t i o n . t e x t i m p o r t T f i d f V e c t o r i z e r
from s k l e a r n . n a i v e b a y e s i m p o r t M u l t i n o m i a l N B
# ML l e a r n s s t a t i s t i c a l word a s s o c i a t i o n s d i r e c t l y from d a t a
vectorizer = TfidfVectorizer ()
X train = vectorizer . fit transform ( corpus emails )
c l f = MultinomialNB ( )
c l f . f i t ( X train , y l a b e l s ) # Au tomat ical ly g e n e r a l i z e s to unseen t e x t

Computer Engineering

DI04000061: ML

4th Semester

30 / 455

2. Adaptability and Continuous Improvement I

Dynamic Environment Adaptation
Traditional software requires manual code refactoring and redeployment when system environment parameters change. Machine learning models continuously
adapt through retraining.
Online & Incremental Learning: Updating weights θ (t+1) = θ (t) − η∇L(θ (t) ) as new streaming data arrives.
Concept Drift Handling: Detecting statistical shifts in target distribution P(y |x) over time and automatically retraining models.
Feedback Loops: Reinforcement Learning agents continuously improve policy π(a|s) via environmental rewards.

Computer Engineering

DI04000061: ML

4th Semester

31 / 455

3. Scalability, Efficiency, and Hyper-Personalization I

Scalability & Automation

Hyper-Personalization

Automated decision systems process millions of queries per second
(e.g., credit scoring, fraud detection).

Recommendation systems (Collaborative Filtering & Matrix
Factorization) deliver personalized content feeds to billions of users.

Reduces operational overhead by replacing manual inspection with
high-throughput inference engines.

Optimization:
P
T
2
2
2
minP,Q
(u,i)∈R (Rui − pu qi ) + λ(∥pu ∥ + ∥qi ∥ ).

Computer Engineering

DI04000061: ML

4th Semester

32 / 455

Practical Python Example: Retraining ML Model on Stream Data I

i m p o r t numpy a s np
from s k l e a r n . l i n e a r m o d e l i m p o r t S G D C l a s s i f i e r
# I n i t i a l i z e i n c r e m e n t a l SGD model ( O n l i n e L e a r n i n g )
model = S G D C l a s s i f i e r ( l o s s= ’ l o g l o s s ’ , r a n d o m s t a t e =42)
# Simulated streaming batches of user i n t e r a c t i o n data
c l a s s e s = np . a r r a y ( [ 0 , 1 ] ) # 0 : No C l i c k , 1 : C l i c k
f o r b a t c h i d x i n range ( 5 ) :
X b a t c h = np . random . r a n d n ( 1 0 0 , 1 0 ) # 100 i n c o m i n g r e q u e s t s
y b a t c h = np . random . r a n d i n t ( 0 , 2 , s i z e =100)
# P a r t i a l f i t u p d a t e s model p a r a m e t e r s w i t h o u t f u l l d a t a s e t r e l o a d
model . p a r t i a l f i t ( X b a t c h , y b a t c h , c l a s s e s=c l a s s e s )
p r i n t ( f ” Batch { b a t c h i d x +1} p r o c e s s e d . Model u p d a t e d . ” )

Computer Engineering

DI04000061: ML

4th Semester

33 / 455

4. Key Benefits Across High-Impact Domains I

Healthcare & Diagnostics:

Automated radiology scan classification (CNNs achieving expert level sensitivity).
Drug discovery: Predicting protein folding structures (AlphaFold).
Finance & Risk Management:

Real-time anomaly detection in transaction processing streams.
Algorithmic trading and automated credit scoring systems.
Autonomous Systems:

Sensor fusion (LiDAR, Radar, Camera) for self-driving vehicles.
Robotics trajectory planning and obstacle avoidance.

Computer Engineering

DI04000061: ML

4th Semester

34 / 455

Quantitative Comparison: Rule-Based vs. Machine Learning I

Dimension
Development
Maintainability
Pattern Complexity
Execution Speed
Generalization

Rule-Based Systems
High manual logic coding
Degrades with complexity
Linear & simple logic
Instant deterministic rules
Poor (fails on edge cases)

Machine Learning
Data curation & model tuning
High; retrain on new data
High non-linear interactions
Inference latency (∼ms)
High generalization capability

Table: Comparison Matrix of Software Engineering Paradigms

Computer Engineering

DI04000061: ML

4th Semester

35 / 455

Summary & Key Takeaways I

Core Takeaway
Machine Learning transforms software engineering from explicit instruction writing to statistical estimation from data, enabling systems to solve previously
intractable perception, prediction, and automation tasks.

Primary Benefits:
1
2
3
4

Generalization: Capability to make accurate predictions on unseen data.
Automation: Replaces complex human heuristics with data-driven models.
Adaptability: Seamlessly adjusts to shifting environments via model updates.
Scalability: Processes massive multi-dimensional datasets beyond human comprehension.

Computer Engineering

DI04000061: ML

4th Semester

36 / 455

Lecture Overview & Learning Objectives I

Course: DI04000061 (Introduction Machine Learning)
Unit 1: Introduction to Machine Learning
Topic: Lecture 1.4: Challenges of Machine Learning
Learning Objectives:
1

Identify primary data-centric challenges including data scarcity, noise, non-representative sampling, and high dimensionality.

2

Understand model-centric challenges such as overfitting, underfitting, and the bias-variance tradeoff.

3

Analyze deployment and operational challenges including data drift, concept drift, and model interpretability.

4

Implement Python code to simulate and mitigate key challenges in machine learning workflows.

Computer Engineering

DI04000061: ML

4th Semester

37 / 455

Taxonomy of Machine Learning Challenges I

The Three Pillars of ML Obstacles
Developing effective machine learning systems requires overcoming challenges spanning data quality, algorithm design, and real-world operational deployment.

1. Data Challenges

2. Model Challenges

3. Operational Challenges

Insufficient data quantity

Overfitting vs. underfitting

Data & concept drift

Poor data quality & noise

Hyperparameter tuning

Black-box opacity (XAI)

Non-representative data

High dimensionality

Deployment latency

Irrelevant features

Computational complexity

Fairness & bias

Computer Engineering

DI04000061: ML

4th Semester

38 / 455

Data Quality and Quantity Issues I
Data-Centric Bottlenecks
Machine learning algorithms are fundamentally driven by data: “Garbage in, garbage out.”

Insufficient Data Quantity:

Complex models (e.g., Deep Networks) require 105 –108 samples to generalize well.
Data acquisition can be prohibitively expensive or limited by rare domain occurrences.
Non-Representative Training Data:

Sampling Bias: Training distribution Ptrain (X ) differs from deployment distribution Ptest (X ).
Results in poor generalization on out-of-distribution real-world instances.
Poor Data Quality (Noise & Outliers):

Sensor errors, corrupted logs, and missing values degrade boundary estimation and model
stability.

Computer Engineering

DI04000061: ML

4th Semester

39 / 455

Python Example: Impact of Noise and Outliers I
Simulating Corrupted Target Data
Demonstration of how random label noise degrades linear regression boundary estimations.

i m p o r t numpy a s np
from s k l e a r n . l i n e a r m o d e l i m p o r t L i n e a r R e g r e s s i o n
# Generate s y n t h e t i c clean dataset
np . random . s e e d ( 4 2 )
X = np . l i n s p a c e ( 0 , 1 0 , 5 0 ) . r e s h a p e ( −1 , 1 )
y clean = 2.5 ∗ X. squeeze () + 5.0
# Add h i g h−m a g n i t u d e n o i s e / o u t l i e r s t o 10% o f l a b e l s
y n o i s y = y c l e a n . copy ( )
n o i s e i d x = np . random . c h o i c e ( 5 0 , s i z e =5 , r e p l a c e=F a l s e )
y n o i s y [ n o i s e i d x ] += np . random . n o r m a l ( 0 , 5 0 , s i z e =5)
# F i t m o d e l s on c l e a n v s n o i s y d a t a
m o d e l c l e a n = L i n e a r R e g r e s s i o n ( ) . f i t (X , y c l e a n )
m o d e l n o i s y = L i n e a r R e g r e s s i o n ( ) . f i t (X , y n o i s y )
p r i n t ( f ” C l e a n S l o p e : { m o d e l c l e a n . c o e f [ 0 ] : . 2 f } , I n t e r c e p t : { m o d e l c l e a n . i n t e r c e p t : . 2 f }” )
p r i n t ( f ” N o i s y S l o p e : { m o d e l n o i s y . c o e f [ 0 ] : . 2 f } , I n t e r c e p t : { m o d e l n o i s y . i n t e r c e p t : . 2 f }” )

Computer Engineering

DI04000061: ML

4th Semester

40 / 455

Model Generalization: Overfitting vs. Underfitting I
Fundamental Tradeoff
The ultimate goal of machine learning is to minimize expected generalization error (risk):
R(f ) = E(X ,Y )∼P [L(f (X ), Y )]

Underfitting (High Bias):

Model capacity is insufficient to capture underlying relationships.
Characterized by high training error AND high test error.
Overfitting (High Variance):

Model memorizes training samples including idiosyncratic noise.
Characterized by low training error, but high test error.
Mitigation Strategies:

Regularization (L1 , L2 ), Cross-Validation, Early Stopping, Pruning.

Computer Engineering

DI04000061: ML

4th Semester

41 / 455

Python Example: Polynomial Overfitting I
Demonstrating Model Capacity vs. Overfitting
Fitting high-degree polynomials leads to severe variance on unseen test points.

i m p o r t numpy a s np
from s k l e a r n . p r e p r o c e s s i n g i m p o r t P o l y n o m i a l F e a t u r e s
from s k l e a r n . l i n e a r m o d e l i m p o r t L i n e a r R e g r e s s i o n
from s k l e a r n . m e t r i c s i m p o r t m e a n s q u a r e d e r r o r
X train
y train
X test
y test

= np . a r r a y ( [ 1 , 2 , 3 , 4 , 5 ] ) . r e s h a p e ( −1 , 1 )
= np . a r r a y ( [ 2 . 1 , 3 . 9 , 6 . 2 , 8 . 1 , 1 0 . 2 ] )
= np . a r r a y ( [ 2 . 5 , 4 . 5 ] ) . r e s h a p e ( −1 , 1 )
= np . a r r a y ( [ 5 . 0 , 9 . 1 ] )

# High d e g r e e p o l y n o m i a l ( D e g r e e 4 ) o v e r f i t s 5 p o i n t s e x a c t l y
p o l y = P o l y n o m i a l F e a t u r e s ( d e g r e e =4)
X train poly = poly . f i t t r a n s f o r m ( X train )
X test poly = poly . transform ( X test )
model = L i n e a r R e g r e s s i o n ( ) . f i t ( X t r a i n p o l y , y t r a i n )
t r a i n r m s e = np . s q r t ( m e a n s q u a r e d e r r o r ( y t r a i n , model . p r e d i c t ( X t r a i n p o l y ) ) )
t e s t r m s e = np . s q r t ( m e a n s q u a r e d e r r o r ( y t e s t , model . p r e d i c t ( X t e s t p o l y ) ) )
p r i n t ( f ” T r a i n RMSE : { t r a i n r m s e : . 4 f } | T e s t RMSE : { t e s t r m s e : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

42 / 455

The Curse of Dimensionality I

High-Dimensional Geometric Paradoxes
As feature dimension d increases, the volume of space grows exponentially, making data points extremely sparse.
Data Sparsity: To maintain uniform sample density, required sample size N grows exponentially: N ∝ e d .
Distance Concentration: In high-dimensional spaces, pairwise Euclidean distances become nearly uniform:

lim

d→∞

dmax − dmin
dmin

=0

Distances lose discriminatory power for algorithms like k-NN, K-Means, and SVMs.
Solutions:

Dimensionality Reduction: PCA, t-SNE, UMAP, Autoencoders.
Feature Selection: Variance thresholding, mutual information, L1 Lasso.

Computer Engineering

DI04000061: ML

4th Semester

43 / 455

Python Example: Distance Concentration Effect I

Measuring Distance Range Across Dimensions
Observing how relative distance contrast collapses as feature dimension d grows.

i m p o r t numpy a s np
from s c i p y . s p a t i a l . d i s t a n c e i m p o r t p d i s t
d e f d i s t a n c e c o n t r a s t ( dim , n s a m p l e s =500):
# Sample u n i f o r m random v e c t o r s i n h y p e r c u b e [ 0 , 1 ] ˆ dim
X = np . random . u n i f o r m ( 0 , 1 , s i z e =( n s a m p l e s , dim ) )
d i s t a n c e s = p d i s t (X , m e t r i c= ’ e u c l i d e a n ’ )
d min , d max = np . min ( d i s t a n c e s ) , np . max ( d i s t a n c e s )
r e t u r n ( d max − d m i n ) / d m i n
f o r d i n [ 2 , 10 , 100 , 1 0 0 0 ] :
contrast = distance contrast (d)
p r i n t ( f ” D i m e n s i o n {d : 4 d} −> R e l a t i v e D i s t a n c e C o n t r a s t : { c o n t r a s t : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

44 / 455

Production Challenges: Distribution Drift I

Dynamic Real-World Environments
Models assume stationary distributions P(X , Y ). Real-world data distributions evolve over time.

Covariate Shift (Data Drift):

Input distribution changes: Pt1 (X ) ̸= Pt2 (X ), while relationship P(Y |X ) remains constant.
Example: User demographic shift in an e-commerce platform over time.
Concept Drift:

Target relationship changes: Pt1 (Y |X ) ̸= Pt2 (Y |X ).
Example: Financial fraud patterns evolving post-security updates.
Prior Probability Shift:

Label distribution changes: Pt1 (Y ) ̸= Pt2 (Y ).

Computer Engineering

DI04000061: ML

4th Semester

45 / 455

Model Interpretability vs. Performance I

The Accuracy-Explainability Spectrum
Tradeoff between high predictive accuracy and human-understandable decision rules.

White-Box Models (High Interpretability, Lower Capacity):

Linear/Logistic Regression, Decision Trees, Naive Bayes.
Explicit coefficients / rules allow transparent domain auditing.
Black-Box Models (Low Interpretability, High Capacity):

Deep Neural Networks, Gradient Boosted Trees, Random Forests.
High capacity for complex non-linear structures, but decision paths are opaque.
Explainable AI (XAI) Frameworks:

Local explanation models: SHAP (Shapley Additive exPlanations), LIME.

Computer Engineering

DI04000061: ML

4th Semester

46 / 455

Summary: Overcoming ML Challenges I

Challenge
Data Scarcity
Noise / Outliers
Overfitting
Curse of Dim.
Data/Concept Drift
Black-Box Opacity

Computer Engineering

Root Cause
High labeling costs
Sensor errors, corrupted data
Model over-capacity
Sparse feature space
Non-stationary distributions
Complex parameters

DI04000061: ML

Standard Solutions
Data Augmentation, Transfer Learning
Robust Scalers, Isolation Forests, Cleaning
L1 /L2 Regularization, Cross-Validation
PCA, Feature Selection, Autoencoders
Continuous Monitoring, MLOps Retraining
SHAP, LIME, Feature Importance Plots

4th Semester

47 / 455

Lecture Overview & Learning Objectives I

Course: DI04000061 (Introduction Machine Learning)
Unit 2: Python Libraries for Machine Learning
Topic: Lecture 2.1: NumPy — Key Concepts and Features
Learning Objectives:
1

Understand the fundamental role of NumPy as the foundation for scientific computing and Machine Learning in Python.

2

Analyze the internal architecture of the ndarray object, including memory layout, strides, and data types (dtype).

3

Contrast NumPy vectorized operations with standard Python loops in terms of performance and memory efficiency.

4

Master the mechanics and formal rules of NumPy Broadcasting for multi-dimensional operations.

5

Differentiate between memory views and deep copies to avoid subtle bug mutations in data processing.

Computer Engineering

DI04000061: ML

4th Semester

48 / 455

Introduction to NumPy in Machine Learning I

What is NumPy?
NumPy (Numerical Python) is the core library for scientific computing in Python. It provides a high-performance N-dimensional array object (ndarray) and tools
for working with these arrays.
Ecosystem Foundation: Serves as the numerical backbone for SciPy, Pandas, Scikit-Learn, PyTorch, and TensorFlow.
Machine Learning Data Representation:

Feature Matrix (X ): 2D array of shape (N, d) where N is samples and d is features.
Target Vector (y ): 1D array of shape (N, ) containing ground-truth labels.
Model Parameters (w, b): Weights and bias vectors updated via matrix algebra.
Performance Engine: Core routines written in C/Fortran for low-level execution speed.

Computer Engineering

DI04000061: ML

4th Semester

49 / 455

Computer Engineering

DI04000061: ML

4th Semester

50 / 455

Why Python Standard Lists are Inefficient for ML
Python lists store pointers to objects scattered across non-contiguous memory locations. Each
element lookup incurs dynamic type checking and pointer dereferencing overhead.
Contiguous Memory Allocation: NumPy arrays allocate a single, contiguous block of
memory for elements of uniform data type (dtype).
Cache Locality: Contiguous storage maximizes CPU L1/L2 cache hit rates during
sequential iteration.
SIMD Hardware Acceleration: Enables Single Instruction, Multiple Data (SIMD)
processor instructions for parallel numerical execution.

Feature
Data Types
Memory Layout
Execution
Element Operations
Computer Engineering

Python List
Heterogeneous
Non-contiguous (Pointers)
Interpreted Loop
Requires explicit loops
DI04000061: ML

NumPy ndarray
Homogeneous (dtype)
Contiguous Block
Compiled C / SIMD
Vectorized Ufuncs
4th Semester

51 / 455

Object

Computer Engineering

DI04000061: ML

4th Semester

51 / 455

Key Attributes of ndarray
An ndarray is a multidimensional container of items of the same type and size, defined by a
small set of metadata.

shape: Tuple of integers indicating the size of the array along each dimension (e.g.,
(500, 10)).
ndim: Total number of dimensions (axes) in the array.
dtype: Data type object describing the layout of bytes (e.g., float64, int32).
size: Total number of elements, equal to the product of shape elements.
itemsize & nbytes: Byte size of each element and total array memory usage
(size × itemsize).
strides: Tuple of bytes to step in each dimension when traversing the array in memory.
Computer Engineering

DI04000061: ML

4th Semester

52 / 455

Python Code: Creating Arrays & Inspecting Attributes I

Demonstration of Array Attributes
Creating arrays and examining internal structural properties.

i m p o r t numpy a s np
# C r e a t e a 2D f e a t u r e m a t r i x ( 3 s a m p l e s , 4 f e a t u r e s )
X = np . a r r a y ( [ [ 1 . 5 , 2 . 3 , 0 . 0 , 4 . 1 ] ,
[3.1 , 0.5 , 2.2 , 1.9] ,
[ 0 . 8 , 4 . 0 , 1 . 1 , 3 . 5 ] ] , d t y p e=np . f l o a t 6 4 )
p r i n t ( ” Shape :
” , X. shape )
# Output : ( 3 , 4 )
p r i n t ( ” D i m e n s i o n s : ” , X . ndim )
# Output : 2
p r i n t ( ” Data Type : ” , X . d t y p e )
# Output : f l o a t 6 4
p r i n t ( ” I t e m S i z e : ” , X . i t e m s i z e )# Output : 8 b y t e s
p r i n t ( ” T o t a l B y t e s : ” , X . n b y t e s ) # Output : 3 ∗ 4 ∗ 8 = 96 b y t e s
p r i n t ( ” S t r i d e s : ” , X . s t r i d e s ) # Output : ( 3 2 , 8 )

Computer Engineering

DI04000061: ML

4th Semester

52 / 455

Key Concept: Vectorization & Universal Functions I

Vectorization
The process of performing operations on whole arrays simultaneously without explicit Python for loops.
Universal Functions (ufuncs): Functions that operate element-wise on ndarray objects, implemented as fast compiled C loops.
Mathematical Expressiveness: Allows writing mathematical formulas cleanly, matching linear algebra notation.

Example: Logistic Regression Activation (Sigmoid)
Mathematical equation applied vectorially to a logit vector z ∈ RN :

ŷ = σ(z) =

1
1 + e −z

In NumPy, this executes across all N elements simultaneously without a single Python loop.

Computer Engineering

DI04000061: ML

4th Semester

53 / 455

Python Code: Vectorization vs. Python Loops I
Performance Benchmark
Comparing element-wise computation speed between Python lists and NumPy arrays.

i m p o r t numpy a s np
import time
s i z e = 1 000 000
# Pure Python Loop
l i s t a , l i s t b = l i s t ( range ( s i z e ) ) , l i s t ( range ( s i z e ) )
t0 = time . time ()
py result = [ a + b for a , b in zip ( l i s t a , l i s t b )]
t py = time . time () − t0
# NumPy V e c t o r i z e d A d d i t i o n
a r r a , a r r b = np . a r a n g e ( s i z e ) , np . a r a n g e ( s i z e )
t0 = time . time ()
np result = arr a + arr b
t np = time . time () − t0
p r i n t ( f ” Python Loop Time : { t p y : . 4 f } s e c ” )
p r i n t ( f ”NumPy V e c t o r Time : { t n p : . 4 f } s e c ” )
p r i n t ( f ” Speedup F a c t o r :
{ t p y / t n p : . 1 f }x ” )

Computer Engineering

DI04000061: ML

4th Semester

54 / 455

Key Concept: Broadcasting Rules I

Definition
Broadcasting describes how NumPy treats arrays of different shapes during arithmetic operations. It expands the smaller array to match the larger array without
making unnecessary data copies.

Formal Broadcasting Rules
To determine if two arrays are compatible for broadcasting:
1

Compare their shapes element-wise, starting from the trailing (rightmost) dimensions and moving left.

2

Two dimensions are compatible if:

They are equal, OR
One of them is 1.
3

If neither condition is met, NumPy raises a ValueError:

Computer Engineering

operands could not be broadcast together.

DI04000061: ML

4th Semester

55 / 455

Python Code: Broadcasting in Machine Learning I
Practical Example: Feature Normalization (Mean Subtraction)
Subtracting feature mean vector µ ∈ Rd from data matrix X ∈ RN×d .

i m p o r t numpy a s np
# Data m a t r i x X : 3 s a m p l e s , 2 f e a t u r e s ( Shape : 3 , 2 )
X = np . a r r a y ( [ [ 1 0 0 . 0 , 2 . 0 ] ,
[200.0 , 4.0] ,
[300.0 , 6.0]])
# F e a t u r e −w i s e mean v e c t o r ( Shape : 2 , )
mu = np . mean (X , a x i s =0) # Output s h a p e : ( 2 , ) −> [ 2 0 0 . 0 , 4 . 0 ]
# B r o a d c a s t i n g s u b t r a c t s ( 2 , ) from ( 3 , 2 ) by e x p a n d i n g a x i s 0
X c e n t e r e d = X − mu
p r i n t ( ” C e n t e r e d M a t r i x : \ n” , X c e n t e r e d )
# Output :
# [[ −100.
−2.]
# [
0.
0.]
# [ 100.
2.]]

Computer Engineering

DI04000061: ML

4th Semester

56 / 455

Memory Management: Views vs. Deep Copies I

Views vs. Copies
View: A new array object pointing to the same memory buffer. Modifying data in a view mutates the original array!
Copy: A complete duplication of data into a new memory location.

i m p o r t numpy a s np
a r r = np . a r r a y ( [ 1 0 , 2 0 , 3 0 , 4 0 , 5 0 ] )
# B a s i c s l i c i n g c r e a t e s a VIEW
sub view = arr [ 1 : 4 ]
s u b v i e w [ 0 ] = 999
p r i n t ( ” O r i g i n a l Array modified : ” , a r r )
# Output : [ 10 999 30 40 5 0 ]
# E x p l i c i t c o p y a l l o c a t e s NEW memory
a r r c o p y = a r r . copy ( )
a r r c o p y [ 0 ] = −1
p r i n t ( ” O r i g i n a l A r r a y u n t o u c h e d : ” , a r r [ 0 ] ) # Output : 10

Computer Engineering

DI04000061: ML

4th Semester

57 / 455

Summary: NumPy Best Practices for ML I

Eliminate Python Loops: Replace iterative code with vectorized expressions and built-in ufuncs for order-of-magnitude speedups.
Master Shape Mechanics: Always check X.shape and ensure broadcasting dimensions align before performing element-wise operations.
Optimize Data Types: Choose minimal necessary precision (float32 vs float64) to reduce memory usage and accelerate GPU transfers.
Beware of Unintended Side Effects: Use .copy() when extracting sub-matrices to avoid implicit mutation of training datasets.

Computer Engineering

DI04000061: ML

4th Semester

58 / 455

Lecture Overview & Learning Objectives I

Course: DI04000061 (Introduction Machine Learning)
Unit 2: Python Libraries for Machine Learning
Topic: 2.1.1 Creating and Accessing Array — 2.4.2 Functions
Learning Objectives:
1

Master fundamental NumPy array creation techniques: array(), zeros(), ones(), and arange().

2

Understand array dimensionality, memory layout, and efficient reshaping using reshape().

3

Utilize Scikit-Learn’s built-in dataset loaders (load *()) to import benchmark datasets.

4

Implement reproducible dataset partitioning using train test split() with stratification.

5

Construct end-to-end data preprocessing pipelines for machine learning models.

Computer Engineering

DI04000061: ML

4th Semester

59 / 455

Why NumPy for Machine Learning?
The core data structure of NumPy is ndarray (N-dimensional array), providing contiguous memory storage and C-optimized vectorized computation essential for
linear algebra operations in ML models.
Creation from Python Sequences: Convert lists or tuples into homogeneous numerical vectors or matrices.
Key Structural Attributes:

shape: Tuple representing dimensions along each axis.
ndim: Number of array dimensions (axes).
dtype: Uniform data type of array elements (e.g., float64, int32).
i m p o r t numpy a s np
# 1D A r r a y ( T a r g e t V e c t o r y )
y a r r = np . a r r a y ( [ 0 , 1 , 0 , 1 , 1 ] )
p r i n t ( f ” y s h a p e : { y a r r . s h a p e } , ndim : { y a r r . ndim } , d t y p e : { y a r r . d t y p e }” )
# 2D A r r a y ( F e a t u r e M a t r i x X : 3 s a m p l e s , 2 f e a t u r e s )
X a r r = np . a r r a y ( [ [ 1 . 5 , 2 . 3 ] ,
[3.1 , 4.0] ,
[0.8 , 1.2]])
p r i n t ( f ”X s h a p e : { X a r r . s h a p e } , ndim : { X a r r . ndim}” )
and np.ones()

Motivation in Machine Learning
Pre-allocating arrays filled with zeros or ones is crucial for initializing model weights, bias vectors, gradient placeholders, and target masks.
Computer Engineering

DI04000061: ML

4th Semester

61 / 455

np.zeros(shape, dtype=float): Returns a new array of given shape filled with 0.0.
np.ones(shape, dtype=float): Returns a new array of given shape filled with 1.0.
i m p o r t numpy a s np
# I n i t i a l i z i n g b i a s v e c t o r b i n Rˆ5 w i t h z e r o s
b i a s = np . z e r o s ( 5 , d t y p e=np . f l o a t 3 2 )
p r i n t ( ” Bias Vector : ” , b i a s )
# I n i t i a l i z i n g a 3 x4 m a t r i x o f o n e s ( e . g . , dummy f e a t u r e m a t r i x )
o n e s m a t r i x = np . o n e s ( ( 3 , 4 ) , d t y p e=np . f l o a t 6 4 )
p r i n t ( ” Ones M a t r i x Shape : ” , o n e s m a t r i x . s h a p e )
# Adding an i n t e r c e p t column ( b i a s f e a t u r e ) f i l l e d
N samples = 4
i n t e r c e p t c o l = np . o n e s ( ( N s a m p l e s , 1 ) )
p r i n t ( ” I n t e r c e p t Column : \ n” , i n t e r c e p t c o l )

with 1 s

Definition
np.arange([start,] stop[, step,], dtype=None) creates 1D arrays with evenly spaced values within the half-open interval [start, stop).

Parameters:

start: Start of interval (inclusive, default is 0).
stop: End of interval (exclusive).
step: Spacing between values.
Use Cases: Generating sample indices, synthetic 1D feature grids, and time-step sequences.
Computer Engineering

DI04000061: ML

4th Semester

62 / 455

i m p o r t numpy a s np
# I n t e g e r s e q u e n c e from 0 t o 9
i n d i c e s = np . a r a n g e ( 1 0 )
print (” Indices : ” , indices )
# F e a t u r e g r i d w i t h s t e p s i z e 0 . 5 from −2 t o 2 ( s t o p i s
g r i d = np . a r a n g e ( −2.0 , 2 . 5 , 0 . 5 )
print (” Feature Grid : ” , grid )

exclusive )

# Even numbers s e q u e n c e
e v e n s = np . a r a n g e ( 0 , 1 0 , 2 )
p r i n t ( ” Evens : ” , evens )

Dimension Transformation
np.reshape(a, newshape) gives a new shape to an array without changing its underlying data memory. The total element count N =
invariant.

Q

shape dimi must remain

Inferring Dimensions (-1): One dimension parameter can be −1, allowing NumPy to automatically compute the required length.
Scikit-Learn Formatting: Converting 1D arrays of shape (N, ) to 2D matrices of shape (N, 1) via reshape(-1, 1).
i m p o r t numpy a s np
# 1D a r r a y o f 12 e l e m e n t s
a = np . a r a n g e ( 1 2 ) # s h a p e : ( 1 2 , )
# R e s h a p e i n t o 2D m a t r i x ( 3 s a m p l e s , 4 f e a t u r e s )
X 2d = a . r e s h a p e ( 3 , 4 )
p r i n t ( ” R e s h a p e d ( 3 , 4 ) : \ n” , X 2d )
Computer Engineering

DI04000061: ML

4th Semester

63 / 455

# R e s h a p e 1D v e c t o r t o 2D Column V e c t o r (N, 1 ) f o r S c i k i t −L e a r n
y c o l = a . r e s h a p e ( −1 , 1 )
p r i n t ( f ” O r i g i n a l s h a p e : {a . s h a p e } −> Column V e c t o r s h a p e : { y c o l . s h a p e }” )
Functions

Built-in Benchmark Datasets
sklearn.datasets provides toy datasets pre-packaged for quick algorithm experimentation and prototyping. Common loaders include load iris(),
load digits(), and load breast cancer().

Bunch Object: Returns a dictionary-like structure containing:

data: Feature matrix X ∈ RN×d .
target: Target vector y ∈ RN .
feature names, target names, DESCR: Metadata descriptions.
Direct Extraction: Setting return X y=True returns (X , y ) directly as NumPy arrays.
from s k l e a r n . d a t a s e t s i m p o r t l o a d i r i s , l o a d b r e a s t c a n c e r
# Load I r i s d a t a s e t a s (X , y ) t u p l e d i r e c t l y
X i r i s , y i r i s = l o a d i r i s ( r e t u r n X y=True )
p r i n t ( f ” I r i s X s h a p e : { X i r i s . s h a p e } , y s h a p e : { y i r i s . s h a p e }” )
# Load B r e a s t C a n c e r d a t a s e t o b j e c t t o i n s p e c t m e t a d a t a
cancer data = load breast cancer ()
print (” Features : ” , cancer data . feature names [ : 3 ] )
p r i n t ( ” Target C l a s s e s : ” , cancer data . target names )
Computer Engineering

DI04000061: ML

4th Semester

65 / 455

Purpose in Supervised Learning
To measure generalization performance and detect overfitting, datasets are partitioned into non-overlapping training and testing subsets: (Xtrain , ytrain ) and
(Xtest , ytest ).

Key Arguments in sklearn.model selection.train test split:

*arrays: Arrays (X , y ) with matching first dimension length N.
test size: Proportion of dataset for test split (e.g., 0.2 for 20%).
random state: Integer seed ensuring reproducible pseudo-random shuffling.
shuffle: Boolean flag (default True).
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
from s k l e a r n . d a t a s e t s i m p o r t l o a d i r i s
X , y = l o a d i r i s ( r e t u r n X y=True )
# S p l i t d a t a s e t : 80% T r a i n i n g , 20% T e s t i n g
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42
)
p r i n t ( f ” O r i g i n a l X : {X . s h a p e }” )
p r i n t ( f ” T r a i n X : { X t r a i n . s h a p e } , T e s t X : { X t e s t . s h a p e }” )

Computer Engineering

DI04000061: ML

4th Semester

66 / 455

Advanced Splitting: Stratification I

The Imbalanced Dataset Challenge
Random splitting on imbalanced or multi-class datasets can produce class distribution mismatches between training and test sets, biasing model evaluation.
Stratified Sampling (stratify=y): Forces the split to maintain the exact target class proportions in both training and testing partitions.
i m p o r t numpy a s np
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
# I m b a l a n c e d t a r g e t v e c t o r : 90% C l a s s 0 , 10% C l a s s 1
X dummy = np . a r a n g e ( 1 0 0 ) . r e s h a p e ( 1 0 0 , 1 )
y dummy = np . a r r a y ( [ 0 ] ∗ 9 0 + [ 1 ] ∗ 1 0 )
# S t r a t i f i e d s p l i t to p r e s e r v e 9:1 c l a s s r a t i o
X tr , X te , y t r , y t e = t r a i n t e s t s p l i t (
X dummy , y dummy , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42 , s t r a t i f y =y dummy
)
p r i n t ( ” T r a i n C l a s s 1 Count : ” , np . sum ( y t r == 1 ) ) # E x a c t l y 8
p r i n t ( ” T e s t C l a s s 1 Count : ” , np . sum ( y t e == 1 ) ) # E x a c t l y 2

Computer Engineering

DI04000061: ML

4th Semester

66 / 455

End-to-End ML Data Preparation Pipeline I
Workflow Integration
Combining array creation, shape inspection, dataset loading, reshaping, and splitting into a unified, clean machine learning ingestion pipeline.

i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t l o a d d i g i t s
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
# 1 . Load benchmark d i g i t s d a t a s e t
d i g i t s = l o a d d i g i t s ()
X , y = d i g i t s . data , d i g i t s . t a r g e t
# 2. Inspect o r i g i n a l dimensions
p r i n t ( f ”Raw F e a t u r e M a t r i x : {X . s h a p e }” ) # ( 1 7 9 7 , 6 4 )
# 3 . P a r t i t i o n i n t o T r a i n (70%) , V a l (15%) , T e s t (15%)
X t r a i n , X temp , y t r a i n , y t em p = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 3 , r a n d o m s t a t e =42 , s t r a t i f y =y
)
X val , X test , y val , y t e s t = t r a i n t e s t s p l i t (
X temp , y temp , t e s t s i z e = 0 . 5 , r a n d o m s t a t e =42 , s t r a t i f y =y t em p
)
p r i n t ( f ” T r a i n : { X t r a i n . s h a p e } | V a l : { X v a l . s h a p e } | T e s t : { X t e s t . s h a p e }” )

Computer Engineering

DI04000061: ML

4th Semester

67 / 455

Summary of Functions & Best Practices I

Function
np.array()
np.zeros() / ones()
np.arange()
np.reshape()
load *()
train test split()

Library / Module
numpy
numpy
numpy
numpy
sklearn.datasets
sklearn.model selection

Primary Purpose in ML
Convert sequences to N-dimensional arrays
Pre-allocate memory for weights & masks
Generate numerical ranges & grid indices
Re-dimension arrays (e.g. 1D to 2D column vector)
Load benchmark datasets (X , y )
Partition data into train/test subsets

Key Best Practice
Always set random state when calling train test split() to ensure reproducible model training and evaluation across experiments!

Computer Engineering

DI04000061: ML

4th Semester

68 / 455

Overview of Array Stacking & Splitting I

Role in Machine Learning Workflows
In machine learning, data often comes from disparate sources or requires transformation during preprocessing. Stacking and splitting operations are critical for
feature assembly, mini-batch generation, dataset concatenation, and train-validation-test partitioning.

Array Stacking: Joining existing arrays together along an existing or newly created dimension.

np.vstack(): Stack arrays vertically (row-wise, axis 0).
np.hstack(): Stack arrays horizontally (column-wise, axis 1).
np.dstack(): Stack arrays depth-wise along third axis (axis 2).
np.stack(): Join arrays along a new dimension.
Array Splitting: Dividing a single array into multiple sub-arrays along specified axes.

np.split(): Flexible splitting by equal parts or index locations.
np.vsplit() / np.hsplit(): Shortcut functions for row and column splitting.

Computer Engineering

DI04000061: ML

4th Semester

69 / 455

Vertical and Horizontal Stacking I

Dimensionality Constraints
np.vstack(): Concatenates along axis 0. All input arrays must have matching dimensions except along axis 0.
np.hstack(): Concatenates along axis 1 (for 2D arrays). All input arrays must have matching dimensions except along axis 1.

i m p o r t numpy a s np
a = np . a r r a y ( [ [ 1 , 2 ] , [ 3 , 4 ] ] )
b = np . a r r a y ( [ [ 5 , 6 ] , [ 7 , 8 ] ] )
# V e r t i c a l S t a c k i n g ( Row−w i s e c o n c a t e n a t i o n )
v s t a c k e d = np . v s t a c k ( ( a , b ) ) # Output s h a p e : ( 4 , 2 )
# H o r i z o n t a l S t a c k i n g ( Column−w i s e c o n c a t e n a t i o n )
h s t a c k e d = np . h s t a c k ( ( a , b ) ) # Output s h a p e : ( 2 , 4 )
p r i n t ( ” v s t a c k e d : \ n” , v s t a c k e d )
p r i n t ( ” h s t a c k e d : \ n” , h s t a c k e d )

Computer Engineering

DI04000061: ML

4th Semester

70 / 455

Difference Between Concatenation and Stacking
Unlike np.concatenate() or np.vstack()/np.hstack() which operate on existing axes, np.stack() creates a new axis and joins arrays along it.
Given k arrays of shape (M, N):
np.stack((a, b), axis=0) → Output shape (k, M, N).
np.stack((a, b), axis=-1) → Output shape (M, N, k).

i m p o r t numpy a s np
# Two 2D g r a y s c a l e image f r a m e s o f s h a p e ( 2 8 , 2 8 )
img1 = np . o n e s ( ( 2 8 , 2 8 ) )
img2 = np . z e r o s ( ( 2 8 , 2 8 ) )
# S t a c k a l o n g new a x i s 0 ( Batch d i m e n s i o n f o r model i n p u t )
b a t c h = np . s t a c k ( ( img1 , img2 ) , a x i s =0) # Shape : ( 2 , 2 8 , 2 8 )
# S t a c k a l o n g new a x i s −1 ( C h a n n e l d i m e n s i o n , e . g . , d u a l−c h a n n e l )
c h a n n e l s = np . s t a c k ( ( img1 , img2 ) , a x i s =−1) # Shape : ( 2 8 , 2 8 , 2 )
p r i n t ( ” Batch s h a p e : ” , b a t c h . s h a p e )
p r i n t ( ” Channels shape : ” , channels . shape )
& np.column stack()

Depth Stacking & Feature Column Assembly
np.dstack(): Stacks 2D matrices depth-wise along axis 2 (3rd dimension). Essential for building multi-channel image tensors (RGB).
np.column stack(): Takes 1D arrays and stacks them as columns into a 2D matrix (ideal for feature matrices X ).

Computer Engineering

DI04000061: ML

4th Semester

72 / 455

i m p o r t numpy a s np
# 1D F e a t u r e c o l u m n s −> 2D F e a t u r e M a t r i x X
f 1 = np . a r r a y ( [ 1 . 5 , 2 . 3 , 3 . 1 ] )
f 2 = np . a r r a y ( [ 1 0 . 0 , 2 0 . 0 , 3 0 . 0 ] )
X = np . c o l u m n s t a c k ( ( f 1 , f 2 ) )
p r i n t ( ” F e a t u r e M a t r i x X ( s h a p e : ” , X . shape , ” ) : \ n” , X)
# Stacking R, G, B planes
r p l a n e = np . f u l l ( ( 2 , 2 ) ,
g p l a n e = np . f u l l ( ( 2 , 2 ) ,
b p l a n e = np . f u l l ( ( 2 , 2 ) ,

i n t o 3D RGB image t e n s o r
255)
128)
0)

r g b i m a g e = np . d s t a c k ( ( r p l a n e , g p l a n e , b p l a n e ) )
p r i n t ( ”RGB Image s h a p e : ” , r g b i m a g e . s h a p e ) # ( 2 , 2 , 3 )

Equal Partition vs. Index-based Partition
Equal Split (indices or sections = N): Splits array into N equal-sized sub-arrays along specified axis. Requires array size along that axis to be divisible
by N.
Index-based Split (indices or sections = [i, j]): Splits array at specified indices into slices arr[:i], arr[i:j], and arr[j:].

i m p o r t numpy a s np
d a t a = np . a r a n g e ( 1 2 )

# [ 0 , 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11]

# Equal s p l i t i n t o 3 p a r t s
p a r t 1 , p a r t 2 , p a r t 3 = np . s p l i t ( data , 3 )
p r i n t ( ” Equal s p l i t part1 : ” , part1 ) # [ 0 , 1 , 2 , 3]
# Custom s p l i t a t i n d e x p o s i t i o n s 3 and 7
Computer Engineering

DI04000061: ML

4th Semester

73 / 455

s1 , s2 , s 3 = np . s p l i t ( data , [ 3 , 7 ] )
p r i n t ( ” S l i c e 1 ( 0 . . 2 ) : ” , s1 ) # [ 0 , 1 , 2]
p r i n t ( ” S l i c e 2 ( 3 . . 6 ) : ” , s2 ) # [ 3 , 4 , 5 , 6]
p r i n t ( ” S l i c e 3 ( 7 . . 1 1 ) : ” , s3 ) # [ 7 , 8 , 9 , 10 , 11]

& np.hsplit()

Row-wise and Column-wise Splitting
np.vsplit(arr, N): Splits array vertically (along axis 0, row-wise). Equivalent to np.split(arr, N, axis=0).
np.hsplit(arr, N): Splits array horizontally (along axis 1, column-wise). Equivalent to np.split(arr, N, axis=1).

i m p o r t numpy a s np
m a t r i x = np . a r a n g e ( 1 6 ) . r e s h a p e ( 4 , 4 )
# V e r t i c a l S p l i t : d i v i d e r o w s i n t o two e q u a l ( 2 , 4 ) m a t r i c e s
t o p h a l f , b o t t o m h a l f = np . v s p l i t ( m a t r i x , 2 )
# H o r i z o n t a l S p l i t : d i v i d e c o l u m n s i n t o two e q u a l ( 4 , 2 ) m a t r i c e s
l e f t h a l f , r i g h t h a l f = np . h s p l i t ( m a t r i x , 2 )
p r i n t ( ”Top h a l f s h a p e : ” , t o p h a l f . s h a p e )
p r i n t ( ” L e f t h a l f shape : ” , l e f t h a l f . shape )

Computer Engineering

# (2 , 4)
# (4 , 2)

DI04000061: ML

4th Semester

75 / 455

ML Practical Workflow: Feature/Target Splitting I
Separating Feature Matrix X from Target Vector y
When loading datasets stored as a single matrix [X | y ], splitting is used to isolate predictors X from target labels y .

i m p o r t numpy a s np
# S i m u l a t e d d a t a s e t : 5 rows , 3 f e a t u r e c o l u m n s + 1 t a r g e t column
d a t a s e t = np . a r r a y ( [
[1.2 , 0.5 , 3.1 , 0] ,
[2.1 , 1.1 , 4.0 , 1] ,
[0.9 , 0.2 , 2.5 , 0] ,
[3.4 , 1.8 , 5.2 , 1] ,
[2.8 , 1.4 , 4.8 , 1]
])
# S p l i t h o r i z o n t a l l y a t column i n d e x 3
X , y = np . h s p l i t ( d a t a s e t , [ 3 ] )
p r i n t ( ” F e a t u r e M a t r i x X ( s h a p e ” , X . sha pe , ” ) : \ n” , X)
p r i n t ( ” T a r g e t V e c t o r y ( s h a p e ” , y . shape , ” ) : \ n” , y . r a v e l ( ) )

Computer Engineering

DI04000061: ML

4th Semester

75 / 455

Summary & Cheat Sheet I
Summary of Stacking & Splitting Functions

Operation
np.vstack()
np.hstack()
np.dstack()
np.stack()
np.vsplit()
np.hsplit()

Axis / Dimension
Axis 0 (Rows)
Axis 1 (Columns)
Axis 2 (Depth)
New Axis
Axis 0 (Rows)
Axis 1 (Columns)

Primary Use Case
Combining rows / dataset instances
Combining feature columns
Stacking channels (e.g. RGB)
Adding batch/channel dimension
Train/Test row partitioning
Separating features X and labels y

Common Errors to Avoid
ValueError:

all input array dimensions must match...: Ensure array shapes match on non-stacked axes.

ValueError:

array split does not result in an equal division: Use index lists [i, j] for unequal splits instead of an integer.

Computer Engineering

DI04000061: ML

4th Semester

76 / 455

Lecture Overview & Learning Objectives I

Course: DI04000061 (Introduction Machine Learning)
Unit 2: Python Libraries for Machine Learning
Topic: 2.1.3 Maths Functions: add(), subtract(), multiply(), divide(), power(), sqrt()
Learning Objectives:
1

Understand NumPy’s universal functions (ufuncs) for fast, element-wise mathematical operations.

2

Master fundamental arithmetic functions: np.add(), np.subtract(), np.multiply(), and np.divide().

3

Apply non-linear transformations using np.power() and np.sqrt().

4

Leverage broadcasting and in-place memory allocation (out parameter) for optimized machine learning code.

5

Build core machine learning components (loss functions, Euclidean distance, feature scaling) using NumPy math functions.

Computer Engineering

DI04000061: ML

4th Semester

77 / 455

NumPy Universal Functions (ufuncs) I
What is a Universal Function (ufunc)?
A ufunc is a vectorized wrapper around compiled C loops that operates on ndarray objects element-by-element. They provide high computational efficiency
compared to standard Python loops.

Operator Overloading: Standard Python operators map directly to NumPy ufuncs:

+ → np.add()
- → np.subtract()
* → np.multiply()
/ → np.divide()
** → np.power()
Key Advantages of Explicit Function Calls:

Support for optional arguments like out (in-place modification) and where (conditional
operations).
Functional pipeline integration (e.g., passing functions as callbacks or map targets).

Computer Engineering

DI04000061: ML

4th Semester

78 / 455

Mathematical Definition
For matrices A, B ∈ Rm×n , element-wise addition yields matrix C where:
Ci,j = Ai,j + Bi,j

∀ i ∈ {1, . . . , m}, j ∈ {1, . . . , n}

i m p o r t numpy a s np
# Define input feature vectors / matrices
A = np . a r r a y ( [ [ 1 . 0 , 2 . 0 ] , [ 3 . 0 , 4 . 0 ] ] )
B = np . a r r a y ( [ [ 5 . 0 , 6 . 0 ] , [ 7 . 0 , 8 . 0 ] ] )
# Explicit function call
C f u n c = np . add (A , B)
# Equivalent operator syntax
C op = A + B
p r i n t ( ” F u n c t i o n Output : \ n” , C f u n c )
# [ [ 6.
8.]
# [10. 12.]]

Mathematical Definition & Machine Learning Context
Computes Ci,j = Ai,j − Bi,j . Subtraction is essential in machine learning for calculating residual errors (y − ŷ ) and performing gradient descent updates
(w (t+1) = w (t) − η∇L).

i m p o r t numpy a s np
Computer Engineering

DI04000061: ML

4th Semester

80 / 455

# Ground t r u t h v a l u e s ( y ) and model p r e d i c t i o n s ( y h a t )
y t r u e = np . a r r a y ( [ 1 0 0 . 0 , 1 5 0 . 0 , 2 0 0 . 0 ] )
y p r e d = np . a r r a y ( [ 1 1 0 . 0 , 1 4 0 . 0 , 2 0 5 . 0 ] )
# Compute p r e d i c t i o n r e s i d u a l e r r o r s ( r e s i d u a l s )
r e s i d u a l s = np . s u b t r a c t ( y p r e d , y t r u e )
p r i n t ( ” R e s i d u a l s ( y h a t − y ) : ” , r e s i d u a l s ) # Output : [ 1 0 . −10.

5.]

# G r a d i e n t D e s c e n t P a r a m e t e r Update S t e p
w e i g h t s = np . a r r a y ( [ 0 . 5 , −0.2 , 1 . 1 ] )
g r a d i e n t s = np . a r r a y ( [ 0 . 0 4 , −0.01 , 0 . 1 2 ] )
l r = 0.01 # Learning rate
u p d a t e d w e i g h t s = np . s u b t r a c t ( w e i g h t s , np . m u l t i p l y ( l r , g r a d i e n t s ) )
p r i n t ( ” Updated W e i g h t s : ” , u p d a t e d w e i g h t s ) # Output : [ 0 . 4 9 9 6 −0.1999 1 . 0 9 8 8 ]

Hadamard Product (A ⊙ B)
np.multiply() computes the element-wise (Hadamard) product, not matrix multiplication (A · B).
(A ⊙ B)i,j = Ai,j · Bi,j

i m p o r t numpy a s np
A = np . a r r a y ( [ [ 1 , 2 ] , [ 3 , 4 ] ] )
B = np . a r r a y ( [ [ 1 0 , 2 0 ] , [ 3 0 , 4 0 ] ] )
# Hadamard ( e l e m e n t−w i s e ) m u l t i p l i c a t i o n
hadamard = np . m u l t i p l y (A , B)
p r i n t ( ” Element−w i s e ( np . m u l t i p l y ) : \ n” , hadamard )
Computer Engineering

DI04000061: ML

4th Semester

81 / 455

# [ [ 10 4 0 ]
# [ 90 1 6 0 ] ]
# Note : M a t r i x p r o d u c t u s e s np . matmul ( ) o r @ o p e r a t o r
matrix prod = A @ B # Matrix product : [ [ 7 0 , 100] , [150 , 220]]

Mathematical Definition & Special Values
Computes true division Ci,j = Ai,j /Bi,j . When dividing by zero, NumPy returns IEEE 754 floating-point values (inf, -inf, nan) and issues runtime warnings.

i m p o r t numpy a s np
f e a t u r e s = np . a r r a y ( [ 1 0 . 0 , 2 5 . 0 , 5 0 . 0 ] )
s c a l e f a c t o r s = np . a r r a y ( [ 2 . 0 , 5 . 0 , 1 0 . 0 ] )
# Feature normalization s c a l i n g
s c a l e d f e a t u r e s = np . d i v i d e ( f e a t u r e s , s c a l e f a c t o r s )
print (” Scaled Features : ” , s c a l e d f e a t u r e s ) # [ 5 . 5. 5 . ]
# H a n d l i n g D i v i s i o n by Z e r o s a f e l y
x = np . a r r a y ( [ 1 . 0 , 0 . 0 , − 2 . 0 ] )
y = np . a r r a y ( [ 0 . 0 , 0 . 0 ,
0.0])
# U s i n g np . e r r s t a t e t o h a n d l e o r s u p p r e s s w a r n i n g s
w i t h np . e r r s t a t e ( d i v i d e= ’ i g n o r e ’ , i n v a l i d = ’ i g n o r e ’ ) :
r e s = np . d i v i d e ( x , y )
p r i n t ( ” D i v i s i o n by Z e r o R e s u l t : ” , r e s ) # [ i n f
nan − i n f ]

Computer Engineering

DI04000061: ML

4th Semester

83 / 455

Mathematical Definition
p

Computes yi = xi element-wise. Both the base x and exponent p can be scalars or arrays.
b

b

If A = [a1 , a2 ], B = [b1 , b2 ] =⇒ np.power(A, B) = [a11 , a22 ]
i m p o r t numpy a s np
# S c a l a r e x p o n e n t : S q u a r i n g d i f f e r e n c e s f o r MSE
e r r o r s = np . a r r a y ( [ − 2 . 0 , 0 . 5 , 3 . 0 ] )
s q u a r e d e r r o r s = np . power ( e r r o r s , 2 )
p r i n t ( ” Squared E r r o r s : ” , s q u a r e d e r r o r s ) # [ 4 .
# A r r a y e x p o n e n t : Element−s p e c i f i c p o w e r s
b a s e s = np . a r r a y ( [ 2 . 0 , 3 . 0 , 4 . 0 ] )
e x p o n e n t s = np . a r r a y ( [ 1 . 0 , 2 . 0 , 3 . 0 ] )
p o w e r s = np . power ( b a s e s , e x p o n e n t s )
p r i n t ( ” Element−w i s e E x p o n e n t i a t i o n : ” , p o w e r s )

0.25 9.

# [ 2.

]

9. 64.]

Mathematical Definition & Domain Restrictions
Computes yi =

√
xi = xi0.5 for non-negative inputs (xi ≥ 0). Input of negative values produces nan (Not a Number) for real floating-point dtypes.

i m p o r t numpy a s np
# Computing v a r i a n c e and s t a n d a r d d e v i a t i o n s t e p
v a r i a n c e s = np . a r r a y ( [ 4 . 0 , 1 6 . 0 , 2 5 . 0 , 1 0 0 . 0 ] )
s t d d e v s = np . s q r t ( v a r i a n c e s )
p r i n t ( ” Standard D e v i a t i o n s : ” , s t d d e v s ) # [ 2.
4.
Computer Engineering

5. 10.]

DI04000061: ML

4th Semester

84 / 455

# Domain w a r n i n g e x a m p l e : n e g a t i v e numbers
n e g v a l u e s = np . a r r a y ( [ 9 . 0 , −4.0 , 1 6 . 0 ] )
w i t h np . e r r s t a t e ( i n v a l i d = ’ i g n o r e ’ ) :
s q r t n e g = np . s q r t ( n e g v a l u e s )
p r i n t ( ” Sqrt with Negative Input : ” , sqrt neg )

Computer Engineering

# [ 3 . nan

4.]

DI04000061: ML

4th Semester

85 / 455

Broadcasting with Math Functions I
Broadcasting Principle
Broadcasting allows NumPy to execute element-wise operations on arrays of different shapes without making unneeded copies of data in memory.
Rule 1: If arrays differ in rank, prepends 1s to the smaller shape.
Rule 2: Arrays are compatible along a dimension if sizes match or one size is 1.
i m p o r t numpy a s np
# F e a t u r e m a t r i x X : 3 s a m p l e s , 2 f e a t u r e s ( s h a p e : 3 x2 )
X = np . a r r a y ( [ [ 1 0 . 0 , 1 0 0 . 0 ] ,
[20.0 , 200.0] ,
[30.0 , 300.0]])
# F e a t u r e mean v e c t o r mu : 2 f e a t u r e s ( s h a p e : 2 , ) −> b r o a d c a s t t o ( 3 , 2 )
mu = np . a r r a y ( [ 2 0 . 0 , 2 0 0 . 0 ] )
# Mean−c e n t e r i n g f e a t u r e s u s i n g np . s u b t r a c t and b r o a d c a s t i n g
X c e n t e r e d = np . s u b t r a c t (X , mu)
p r i n t ( ” C e n t e r e d Data : \ n” , X c e n t e r e d )
# [ [ − 1 0 . −100.]
# [
0.
0.]
# [ 10.
100.]]

Computer Engineering

DI04000061: ML

4th Semester

85 / 455

Parameter

Why Use out?
By default, np.add(A, B) allocates a new array in memory to store results. Pass the out argument to write results into an existing array, avoiding allocation
overhead on large datasets.

i m p o r t numpy a s np
# Pre−a l l o c a t e l a r g e a r r a y s
A = np . o n e s ( ( 1 0 0 0 , 1 0 0 0 ) , d t y p e=np . f l o a t 6 4 )
B = np . f u l l ( ( 1 0 0 0 , 1 0 0 0 ) , 2 . 0 , d t y p e=np . f l o a t 6 4 )
# 1 . S t a n d a r d a p p r o a c h : A l l o c a t e s new memory
C = np . add (A , B) # C r e a t e s b r a n d new ( 1 0 0 0 , 1 0 0 0 ) a r r a y
# 2 . I n−p l a c e m o d i f i c a t i o n : O v e r w r i t e s a r r a y A d i r e c t l y
np . add (A , B , o u t=A)
p r i n t ( ”A a f t e r i n−p l a c e a d d i t i o n f i r s t e l e m e n t : ” , A [ 0 , 0 ] )

# 3.0

# 3 . W r i t e o u t p u t t o p r e−a l l o c a t e d m a t r i x C
np . m u l t i p l y (A , 0 . 5 , o u t=C)
p r i n t ( ”C o u t p u t a f t e r i n−p l a c e m u l t i p l y : ” , C [ 0 , 0 ] )

# 1.5

Computer Engineering

DI04000061: ML

4th Semester

87 / 455

ML Case Study: Euclidean Distance & RMSE I
Formulas
Euclidean Distance: d(u, v) =

qP

d (u − v )2 =
i
i=1 i

Root Mean Squared Error: RMSE =

q

p

sum(power(subtract(u, v), 2))

1 PN (y − ŷ )2
i
i=1 i
N

i m p o r t numpy a s np
# F e a t u r e v e c t o r s f o r two d a t a p o i n t s u and v
u = np . a r r a y ( [ 3 . 0 , 4 . 0 , 5 . 0 ] )
v = np . a r r a y ( [ 1 . 0 , 1 . 0 , 1 . 0 ] )
# Compute E u c l i d e a n d i s t a n c e u s i n g NumPy u f u n c s
d i f f = np . s u b t r a c t ( u , v )
s q d i f f = np . power ( d i f f , 2 )
e u c l i d e a n d i s t = np . s q r t ( np . sum ( s q d i f f ) )
p r i n t ( f ” E u c l i d e a n D i s t a n c e : { e u c l i d e a n d i s t : . 4 f }” )

# 5.3852

# Compute RMSE f o r p r e d i c t i o n s y h a t and t a r g e t s y
y t r u e = np . a r r a y ( [ 3 . 0 , −0.5 , 2 . 0 , 7 . 0 ] )
y p r e d = np . a r r a y ( [ 2 . 5 ,
0.0 , 2.1 , 7.8])
rmse = np . s q r t ( np . mean ( np . power ( np . s u b t r a c t ( y t r u e , y p r e d ) , 2 ) ) )
p r i n t ( f ”RMSE : {rmse : . 4 f }” ) # 0 . 4 9 5 0

Computer Engineering

DI04000061: ML

4th Semester

87 / 455

Summary of Math Functions in Machine Learning I

Function
np.add()
np.subtract()
np.multiply()
np.divide()
np.power()
np.sqrt()

Operator
+
*
/
**
N/A

Primary ML Applications
Bias addition, vector combination
Residual errors (y − ŷ ), gradient update
Hadamard product, feature weighting, masks
Min-max normalization, Standard score calculation
Polynomial features, MSE loss computation
Standard deviation, RMSE, Euclidean distance

Key Takeaway
Combining vectorised arithmetic ufuncs with NumPy’s broadcasting capabilities allows writing high-performance, readable ML algorithms without explicit Python
loops.

Computer Engineering

DI04000061: ML

4th Semester

88 / 455

Overview of Statistical Functions in NumPy I

Role of Descriptive Statistics in Machine Learning
Statistical functions provide essential summary metrics for understanding data distributions, detecting outliers, and performing feature preprocessing.

Central Tendency: Measures that locate the center of a distribution.

np.mean(): Arithmetic average of elements.
np.median(): Middle value separating upper and lower halves.
Dispersion / Spread: Measures that describe data variability.

np.var(): Average squared deviation from the mean.
np.std(): Square root of variance (same units as data).
Multidimensional Aggregation: All functions accept an axis parameter to compute statistics along specific dimensions (rows or columns).

Computer Engineering

DI04000061: ML

4th Semester

89 / 455

Mathematical Definition
The mean µ of a dataset X = {x1 , x2 , . . . , xN } is defined as:
µ=

N
1 X

N i=1

xi

i m p o r t numpy a s np
d a t a = np . a r r a y ( [ [ 1 0 , 2 0 , 3 0 ] ,
[ 4 0 , 50 , 6 0 ] ] )
# G l o b a l mean a c r o s s a l l e l e m e n t s
p r i n t ( ” G l o b a l Mean : ” , np . mean ( d a t a ) )

# Output : 3 5 . 0

# Mean a l o n g a x i s 0 ( column−w i s e mean a c r o s s r o w s )
p r i n t ( ” A x i s 0 Mean : ” , np . mean ( data , a x i s =0)) # Output : [ 2 5 . 3 5 . 4 5 . ]
# Mean a l o n g a x i s 1 ( row−w i s e mean a c r o s s c o l u m n s )
p r i n t ( ” A x i s 1 Mean : ” , np . mean ( data , a x i s =1)) # Output : [ 2 0 . 5 0 . ]

Definition & Properties
Median: The middle element in a sorted, 1D numerical array.
If N is even, it is the average of the two middle values.
Robustness: Unlike the mean, the median is resistant to extreme outliers.

i m p o r t numpy a s np
Computer Engineering

DI04000061: ML

4th Semester

91 / 455

# D a t a s e t c o n t a i n i n g a s e v e r e o u t l i e r ( e . g . , in co me d a t a )
i n c o m e s = np . a r r a y ( [ 4 5 0 0 0 , 4 8 0 0 0 , 5 2 0 0 0 , 5 0 0 0 0 , 5 0 0 0 0 0 0 ] )
m e a n v a l = np . mean ( i n c o m e s )
m e d i a n v a l = np . median ( i n c o m e s )
# $1 , 0 3 9 , 0 0 0 . 0 0 ( Skewed )
p r i n t ( f ”Mean Income :
${ m e a n v a l : , . 2 f }” )
p r i n t ( f ” Median Income : ${ m e d i a n v a l : , . 2 f }” ) # $52 , 0 0 0 . 0 0
( Robust )

Mathematical Definition & Degrees of Freedom
Variance measures the average squared distance from the mean:
1

2

σ =

N
X

N − ddof i=1

2

(xi − µ)

ddof=0 (Default): Population Variance (divides by N).
ddof=1: Sample Variance (divides by N − 1, Bessel’s correction).

i m p o r t numpy a s np
s a m p l e s = np . a r r a y ( [ 1 2 . 5 , 1 4 . 2 , 1 1 . 8 , 1 5 . 0 , 1 3 . 1 ] )
# P o p u l a t i o n v a r i a n c e ( d d o f =0)
p o p v a r = np . v a r ( s a m p l e s , d d o f =0)
# Sample v a r i a n c e ( d d o f =1 , u n b i a s e d e s t i m a t o r f o r s a m p l e d a t a )
s a m p l e v a r = np . v a r ( s a m p l e s , d d o f =1)
Computer Engineering

DI04000061: ML

4th Semester

92 / 455

p r i n t ( f ” P o p u l a t i o n V a r i a n c e : { p o p v a r : . 4 f }” ) # Output : 1 . 2 5 0 4
p r i n t ( f ” Sample V a r i a n c e :
{ s a m p l e v a r : . 4 f }” ) # Output : 1 . 5 6 3 0

Definition & Interpretation
Standard deviation σ is the square root of the variance:

σ=

q

v
u
u
u
Var(X ) = t

1

N
X

N − ddof i=1

(xi − µ)2

Interpretable in the original units of measurement. In a Normal distribution:
≈ 68.27% of values lie within µ ± 1σ.
≈ 95.45% of values lie within µ ± 2σ.

i m p o r t numpy a s np
s c o r e s = np . a r r a y ( [ 8 8 , 9 2 , 7 9 , 9 5 , 8 5 ] )
s t d p o p = np . s t d ( s c o r e s )
# P o p u l a t i o n Std Dev
s t d s a m p l e = np . s t d ( s c o r e s , d d o f =1) # Sample Std Dev
p r i n t ( f ”Mean : {np . mean ( s c o r e s ) : . 2 f }” )
p r i n t ( f ” P o p u l a t i o n Std : { s t d p o p : . 2 f } | Sample Std : { s t d s a m p l e : . 2 f }” )

Computer Engineering

DI04000061: ML

4th Semester

94 / 455

ML Application: Feature Standardization (Z-score) I
Z-Score Normalization Formula
Standardizing features to have zero mean (µ = 0) and unit variance (σ = 1):

z =

x −µ
σ

Essential for gradient-based optimizers and distance-based algorithms (KNN, SVM, PCA).

i m p o r t numpy a s np
# Feature matrix X:
X = np . a r r a y ( [ [ 1 . 0 ,
[2.0 ,
[3.0 ,
[4.0 ,

4 samples , 2 f e a t u r e s
200.0] ,
500.0] ,
300.0] ,
800.0]])

# Compute column−w i s e s t a t i s t i c s
mean X = np . mean (X , a x i s =0)
s t d X = np . s t d (X , a x i s =0)
# A p p l y Z−s c o r e t r a n s f o r m a t i o n
X s c a l e d = (X − mean X ) / s t d X
p r i n t ( ” S t a n d a r d i z e d F e a t u r e s : \ n” , np . round ( X s c a l e d , 2 ) )
Computer Engineering

DI04000061: ML

4th Semester

94 / 455

ML Application: Feature Standardization (Z-score) II

p r i n t ( ” S c a l e d Mean : ” , np . round ( np . mean ( X s c a l e d , a x i s =0) , 2 ) ) # [ 0 . ,
p r i n t ( ” S c a l e d Std : ” , np . round ( np . s t d ( X s c a l e d , a x i s =0) , 2 ) ) # [ 1 . ,

Computer Engineering

DI04000061: ML

0.]
1.]

4th Semester

95 / 455

Summary of NumPy Statistics Functions I

Function
np.mean()
np.median()
np.var()
np.std()

Mathematical Concept
Arithmetic Mean (µ)
Median (50th Percentile)
Variance (σ 2 )
Standard Deviation (σ)

Key Parameter / Property
Affected by outliers, axis
Robust to outliers, axis
Uses ddof=0 (Pop) / ddof=1 (Sample)
Expressed in original data units

Key Takeaway for ML Pipelines
Always verify parameter settings (axis and ddof) when normalizing training sets vs test sets to prevent data leakage and ensure unbiased scaling estimators!

Computer Engineering

DI04000061: ML

4th Semester

96 / 455

Introduction to Pandas in Machine Learning I

What is Pandas?
Pandas is an open-source Python library built on top of NumPy, designed for fast, flexible, and expressive data manipulation and tabular data analysis.

Role in the ML Pipeline:

Data Ingestion: Loading structured datasets from CSV, JSON, SQL, Parquet, and Excel.
Data Cleaning: Handling missing values, duplicate entries, and incorrect data types.
Feature Engineering: Creating derived variables, binning, and one-hot encoding.
Exploratory Data Analysis (EDA): Computing summary statistics, correlations, and
distributions.
Core Data Structures:

Series: 1D homogeneous labeled array.
DataFrame: 2D heterogeneous tabular data structure with labeled axes (rows and columns).

Computer Engineering

DI04000061: ML

4th Semester

97 / 455

Core Data Structures: Series and DataFrames I

i m p o r t p a n d a s a s pd
i m p o r t numpy a s np
# C r e a t i n g a S e r i e s ( 1D)
s = pd . S e r i e s ( [ 1 0 , 2 0 , 3 0 , 4 0 ] , i n d e x =[ ’ a ’ , ’ b ’ , ’ c ’ , ’ d ’ ] )
# C r e a t i n g a DataFrame ( 2D) from a d i c t i o n a r y
data = {
’ Age ’ : [ 2 5 , 3 0 , 3 5 , 4 0 ] ,
’ Salary ’ : [50000.0 , 64000.0 , 71000.0 , 88000.0] ,
’ P u r c h a s e d ’ : [ F a l s e , True , True , F a l s e ]
}
d f = pd . DataFrame ( d a t a )
print ( df )
pri nt (” Index : ” , df . index )
p r i n t ( ” Columns : ” , d f . c o l u m n s )
p r i n t ( ” U n d e r l y i n g V a l u e s : \ n” , d f . v a l u e s )

Computer Engineering

DI04000061: ML

4th Semester

98 / 455

Data Ingestion & Initial Exploration I

Essential Data Inspection Methods
Before modeling, always inspect dataset shapes, column types, and basic statistical summaries.

# L o a d i n g t a b u l a r d a t a from CSV
d f = pd . r e a d c s v ( ’ h o u s i n g d a t a . c s v ’ )
# Shape and d i m e n s i o n s
p r i n t ( ” D a t a s e t Shape : ” , d f . s h a p e )

# Returns ( n samples , n f e a t u r e s )

# I n s p e c t f i r s t few r o w s
p r i n t ( d f . head ( 3 ) )
# Data t y p e s and non−n u l l v a l u e c o u n t s
df . i n f o ()
# Summary s t a t i s t i c s ( mean , s t d , min , max ,
print ( df . describe ())

Computer Engineering

quartiles )

DI04000061: ML

4th Semester

99 / 455

Indexing and Selection: loc vs iloc I

.loc[]: Selection based on row and column labels.
.iloc[]: Selection based on integer positions (0-indexed).
Boolean Indexing: Filtering rows using conditional expressions.
# S e l e c t s p e c i f i c r o w s and c o l u m n s by l a b e l
s u b s e t 1 = d f . l o c [ 0 : 2 , [ ’ Age ’ , ’ S a l a r y ’ ] ]
# S e l e c t by i n t e g e r p o s i t i o n
subset2 = df . i l o c [ 0 : 3 , 0 : 2 ]

indices

# F i l t e r i n g rows with a boolean c o n d i t i o n
high earners = df [ df [ ’ Salary ’ ] > 60000.0]
# M u l t i−c o n d i t i o n b o o l e a n f i l t e r i n g (& f o r AND, | f o r OR)
t a r g e t g r o u p = d f [ ( d f [ ’ Age ’ ] >= 3 0 ) & ( d f [ ’ P u r c h a s e d ’ ] == True ) ]

Computer Engineering

DI04000061: ML

4th Semester

100 / 455

Handling Missing Data in ML Datasets I

Impact of Missing Values
Most machine learning algorithms (e.g., linear models, SVMs, neural networks) fail if input matrices contain missing values (NaN or None).

# Count m i s s i n g v a l u e s p e r column
p r i n t ( d f . i s n a ( ) . sum ( ) )
# S t r a t e g y 1 : Drop r o w s w i t h m i s s i n g v a l u e s
d f c l e a n = d f . d r o p n a ( a x i s =0)
# S t r a t e g y 2 : Drop c o l u m n s w i t h e x c e s s i v e m i s s i n g d a t a (> 50%)
d f f i l t e r e d = d f . d r o p n a ( t h r e s h=l e n ( d f ) ∗ 0 . 5 , a x i s =1)
# S t r a t e g y 3 : I m p u t e m i s s i n g n u m e r i c a l v a l u e s w i t h median
m e d i a n s a l a r y = d f [ ’ S a l a r y ’ ] . median ( )
df [ ’ Salary ’ ] = df [ ’ Salary ’ ] . f i l l n a ( median salary )

Computer Engineering

DI04000061: ML

4th Semester

101 / 455

Data Transformation & Feature Engineering I

Modifying and Creating Features
Pandas supports efficient vectorized element-wise operations and custom mappings.

# V e c t o r i z e d e l e m e n t−w i s e f e a t u r e e n g i n e e r i n g
d f [ ’ S a l a r y P e r Y e a r A g e ’ ] = d f [ ’ S a l a r y ’ ] / d f [ ’ Age ’ ]
# Custom f e a t u r e t r a n s f o r m a t i o n u s i n g . a p p l y ( )
d e f c a t e g o r i z e a g e ( age ) :
r e t u r n ’ Young ’ i f age < 32 e l s e ’ S e n i o r ’
d f [ ’ Age Group ’ ] = d f [ ’ Age ’ ] . a p p l y ( c a t e g o r i z e a g e )
# C a t e g o r i c a l mapping u s i n g a d i c t i o n a r y
d f [ ’ T a r g e t ’ ] = d f [ ’ P u r c h a s e d ’ ] . map({ True : 1 , F a l s e : 0})
# Dropping u n n e c e s s a r y columns
d f = d f . d r o p ( c o l u m n s =[ ’ P u r c h a s e d ’ ] )

Computer Engineering

DI04000061: ML

4th Semester

102 / 455

Grouping and Aggregation I

Split-Apply-Combine Paradigm
GroupBy splits data into groups based on key columns, applies aggregate functions, and combines results.

# Compute mean s a l a r y g r o u p e d by Age Group
g r o u p m e a n s = d f . g r o u p b y ( ’ Age Group ’ ) [ ’ S a l a r y ’ ] . mean ( )
p r i n t ( group means )
# Computing m u l t i p l e a g g r e g a t e s t a t i s t i c s s i m u l t a n e o u s l y
s t a t s = d f . g r o u p b y ( ’ Age Group ’ ) . agg ({
’ S a l a r y ’ : [ ’ mean ’ , ’ s t d ’ , ’ min ’ , ’ max ’ ] ,
’ Age ’ : ’ c o u n t ’
})
print ( stats )

Computer Engineering

DI04000061: ML

4th Semester

103 / 455

Combining Datasets: Merging and Concatenation I

Concatenation (pd.concat): Stacking DataFrames vertically (axis=0) or horizontally (axis=1).
Merging (pd.merge): Database-style joins (Inner, Left, Right, Outer) on key columns.
# C o n c a t e n a t i n g r o w s o f two DataFrames
d f c o m b i n e d = pd . c o n c a t ( [ d f 1 , d f 2 ] , a x i s =0 , i g n o r e i n d e x=True )
# R e l a t i o n a l i n n e r j o i n on k e y column ’ u s e r i d ’
d f m e r g e d = pd . merge (
l e f t=d f u s e r p r o f i l e ,
r i g h t=d f u s e r t r a n s a c t i o n s ,
on= ’ u s e r i d ’ ,
how= ’ i n n e r ’
)

Computer Engineering

DI04000061: ML

4th Semester

104 / 455

Pandas Integration with ML Pipelines I

Preparing Data for Scikit-Learn
Machine learning models require clean numerical feature matrices (X ∈ Rn×d ) and target vectors (y ∈ Rn ).

# One−Hot E n c o d i n g f o r c a t e g o r i c a l f e a t u r e s
d f e n c o d e d = pd . g e t d u m m i e s ( df , c o l u m n s =[ ’ Age Group ’ ] ,

d r o p f i r s t =True )

# S e p a r a t e F e a t u r e M a t r i x X and T a r g e t V e c t o r y
X = d f e n c o d e d . d r o p ( c o l u m n s =[ ’ T a r g e t ’ ] )
y = df encoded [ ’ Target ’ ]
# E x t r a c t i n g NumPy a r r a y s f o r S c i k i t −L e a r n
X mat = X . to numpy ( )
y v e c = y . to numpy ( )
p r i n t ( ” F e a t u r e m a t r i x s h a p e X : ” , X mat . s h a p e )
p r i n t ( ” Target v e c t o r shape y : ” , y vec . shape )

Computer Engineering

DI04000061: ML

4th Semester

105 / 455

Lecture Overview & Learning Objectives I

Course: DI04000061 (Introduction Machine Learning)
Unit 2: Python Libraries for Machine Learning
Topic: 2.2.1 Data structure: Series(), DataFrame()
Learning Objectives:
1

Understand the core architectural differences between 1D pd.Series and 2D pd.DataFrame.

2

Master techniques for creating, indexing, and manipulating Pandas data structures.

3

Utilize index alignment and vectorized operations for efficient feature computation.

4

Apply Series and DataFrames to construct Machine Learning feature matrices (X ) and target vectors (y ).

Computer Engineering

DI04000061: ML

4th Semester

106 / 455

Role of Pandas in Machine Learning Pipelines I

Why Pandas for Machine Learning?
Pandas provides high-performance, easy-to-use data structures and data analysis tools built on top of NumPy. It bridges the gap between raw data storage (CSV,
SQL, JSON) and numerical ML algorithms.

Data Ingestion & Preprocessing: Loading structured datasets into tabular form.
Handling Heterogeneous Types: Managing numerical, categorical, string, and timestamp data seamlessly within a single container.
Explicit Indexing: Label-based data alignment preventing data misalignment errors during transformations.
ML Integration: Direct conversion to NumPy arrays (X ∈ Rn×d ) required by algorithms (Scikit-Learn, PyTorch, TensorFlow).

Computer Engineering

DI04000061: ML

4th Semester

107 / 455

Pandas Series: 1D Labeled Array I
Definition
A pd.Series is a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects).

Components:

Data Values: Contiguous block of memory stored as a 1D NumPy array (s.values).
Index Labels: Axis labels governing data retrieval and alignment (s.index).
Data Type: Uniform dtype across all elements (s.dtype).
i m p o r t p a n d a s a s pd
i m p o r t numpy a s np
# C r e a t i n g S e r i e s from a L i s t w i t h Custom I n d e x
w e i g h t s = pd . S e r i e s ( [ 6 8 . 5 , 7 4 . 2 , 5 9 . 0 ] ,
i n d e x =[ ’ P a t i e n t A ’ , ’ P a t i e n t B ’ , ’ P a t i e n t C ’ ] ,
name= ’ W e i g h t k g ’ )
print ( weights )
68.5
# Patient A
# Patient B
74.2
# Patient C
59.0
# Name : W e i g h t k g , d t y p e : f l o a t 6 4

Computer Engineering

DI04000061: ML

4th Semester

108 / 455

Series: Creation Methods & Index Alignment I

Creation from Dictionary: Dictionary keys automatically become index labels.
Automatic Index Alignment: Operations align on index labels, not position.
# S e r i e s from D i c t i o n a r y
i n c o m e 2 0 2 2 = pd . S e r i e s ({ ’ A l i c e ’ : 8 5 0 0 0 , ’ Bob ’ : 9 2 0 0 0 , ’ C h a r l i e ’ : 78000})
i n c o m e 2 0 2 3 = pd . S e r i e s ({ ’ Bob ’ : 9 8 0 0 0 , ’ C h a r l i e ’ : 8 1 0 0 0 , ’ D a v i d ’ : 105000})
# V e c t o r i z e d A d d i t i o n with Automatic Label Alignment
growth = income 2023 − income 2022
p r i n t ( growth )
# Alice
NaN ( M i s s i n g i n 2 0 2 3 )
# Bob
6000.0
( Aligned )
# Charlie
3000.0
( Aligned )
# David
NaN ( M i s s i n g i n 2 0 2 2 )
# dtype : f l o a t 6 4

Computer Engineering

DI04000061: ML

4th Semester

109 / 455

Pandas DataFrame: 2D Tabular Structure I

Definition
A pd.DataFrame is a 2D labeled, size-mutable, and potentially heterogeneous tabular data structure with aligned rows and columns.
Conceptualized as a container of aligned pd.Series objects sharing a common row index.
Dual Axes: Axis 0 represents rows (samples N), Axis 1 represents columns (features d).
# C o n s t r u c t i n g DataFrame from D i c t i o n a r y o f Equal−L e n g t h L i s t s
data = {
’ Age ’ : [ 2 5 , 3 0 , 3 5 , 4 0 ] ,
’ Salary ’ : [50000.0 , 64000.0 , 82000.0 , 110000.0] ,
’ Target Purchased ’ : [0 , 1 , 1 , 0]
}
d f = pd . DataFrame ( data , i n d e x =[ ’ U s e r 1 ’ , ’ U s e r 2 ’ , ’ U s e r 3 ’ , ’ U s e r 4 ’ ] )
p r i n t ( d f . s h a p e ) # ( 4 , 3 ) −> 4 rows , 3 c o l u m n s

Computer Engineering

DI04000061: ML

4th Semester

110 / 455

DataFrame: Essential Structural Inspection I

Always inspect data structures before feeding them to ML models to check for data types and missing values.
# Key i n s p e c t i o n a t t r i b u t e s and methods
print ( df . dtypes )
# R e t u r n s d a t a t y p e o f e a c h f e a t u r e column
p r i n t ( df . shape )
# R e t u r n s t u p l e ( num rows , n u m c o l s )
p r i n t ( df . columns )
# I n d e x o b j e c t c o n t a i n i n g column names
# Summary S t a t i s t i c s f o r N u m e r i c a l Columns
print ( df . describe ())
#
Age
Salary
Target Purchased
# count
4.000000
4.000000
4.000000
# mean
32.500000
76500.000000
0.500000
# std
6.454972
25916.532690
0.577350
# min
25.000000
50000.000000
0.000000
# max
40.000000
110000.000000
1.000000

Computer Engineering

DI04000061: ML

4th Semester

111 / 455

Subsetting & Accessing Data: loc vs iloc I

.loc[]: Purely label-based indexing (Row label, Column label).
.iloc[]: Purely integer-positional indexing (0-based integer offsets).
# 1 . Column S e l e c t i o n
a g e s e r i e s = d f [ ’ Age ’ ]
s u b d f = d f [ [ ’ Age ’ , ’ S a l a r y ’ ] ]

# R e t u r n s pd . S e r i e s
# R e t u r n s pd . DataFrame

# 2 . L a b e l−b a s e d S l i c i n g w i t h . l o c
r o w u s e r 2 = d f . l o c [ ’ U s e r 2 ’ , [ ’ Age ’ , ’ S a l a r y ’ ] ]
# 3. P o s i t i o n a l S l i c i n g with . i l o c
sub matrix = df . i l o c [ 0 : 2 , 0 : 2 ]

# F i r s t 2 rows ,

f i r s t 2 columns

# 4 . C o n d i t i o n a l ( B o o l e a n ) F i l t e r i n g f o r Data C l e a n i n g
h i g h e a r n e r s = df [ df [ ’ S ala ry ’ ] > 70000]

Computer Engineering

DI04000061: ML

4th Semester

112 / 455

Comparison: pd.Series vs pd.DataFrame I

Property
Dimensions
Data Homogeneity
ML Representation
Axis Labels
Conversion

pd.Series
1D (Vector)
Single dtype across all elements
Target Vector y ∈ RN or single Feature
Single Index (Row labels)
s.to frame() converts to DataFrame

pd.DataFrame
2D (Matrix/Table)
Heterogeneous (different dtype per column)
Feature Matrix X ∈ RN×d
Row Index (Axis 0) & Column Index (Axis 1)
Extracting single column yields Series

Table: Architectural Comparison of Series and DataFrame

Computer Engineering

DI04000061: ML

4th Semester

113 / 455

Machine Learning Workflow Integration I

Constructing Feature Matrix X and Target Vector y
In supervised ML, input features must be isolated from the target label.

# S p l i t t i n g DataFrame i n t o F e a t u r e M a t r i x (X) and T a r g e t V e c t o r ( y )
X = d f . d r o p ( c o l u m n s =[ ’ T a r g e t P u r c h a s e d ’ ] ) # 2D DataFrame (N s a m p l e s , d f e a t u r e s )
y = df [ ’ Target Purchased ’ ]
# 1D S e r i e s (N s a m p l e s )
# E x t r a c t i n g raw NumPy a r r a y s f o r S c i k i t −L e a r n / PyTorch model f i t t i n g
X mat = X . to numpy ( )
# Shape : ( 4 , 2 ) , d t y p e : f l o a t 6 4
y v e c = y . to numpy ( )
# Shape : ( 4 , ) ,
dtype : int64
p r i n t ( f ”X t y p e : { t y p e (X) } , Shape : {X . s h a p e }” )
p r i n t ( f ” y t y p e : { t y p e ( y ) } , Shape : {y . s h a p e }” )

Computer Engineering

DI04000061: ML

4th Semester

114 / 455

Lecture Summary & Best Practices I

Key Takeaways
Series: 1D labeled array designed for single variables/targets.
DataFrame: 2D tabular container for multidimensional datasets (X ).
Indexing Alignment: Arithmetic operations automatically align on index labels.
Explicit Indexing: Prefer .loc and .iloc over chained indexing (df[][]) to avoid copy warnings.

Best Practice Note
Avoid using inplace=True in modern Pandas code as it is being deprecated. Always reassign returned DataFrames (e.g., df = df.drop(...)) for cleaner code
execution.

Computer Engineering

DI04000061: ML

4th Semester

115 / 455

Data Manipulation Essentials in Pandas I

Overview
Data manipulation is a fundamental step in the Machine Learning pipeline, enabling data exploration, cleaning, indexing, and transformation before model training.
Data Structure & Inspection: Examining shape, column titles, and quick previews.
Indexing & Selection: Retrieving specific rows and columns using labels.
Handling Missing Data: Detecting and removing null or missing values.
Data Cleaning: Removing duplicate entries and unnecessary features.
Aggregations & Sorting: Summary statistics and reordering datasets.

Computer Engineering

DI04000061: ML

4th Semester

116 / 455

, columns, head(), and tail()

Dataset Structure & Quick Preview
Before applying ML algorithms, inspect the dimensions and structure of the dataset.
shape: Attribute returning a tuple (num rows, num cols).
columns: Attribute containing an index of all column labels.
head(n): Method returning the first n rows (default n = 5).
tail(n): Method returning the last n rows (default n = 5).
i m p o r t p a n d a s a s pd
d a t a = { ’ Name ’ : [ ’ A l i c e ’ , ’ Bob ’ , ’ C h a r l i e ’ ] ,
’ Age ’ : [ 2 5 , 3 0 , 3 5 ] ,
’ Score ’ : [ 8 5 . 5 , 92.0 , 78.0]}
d f = pd . DataFrame ( d a t a )
p r i n t ( df . shape )
p r i n t ( df . columns )
p r i n t ( d f . head ( 2 ) )

# (3 , 3)
# I n d e x ( [ ’ Name ’ , ’ Age ’ ,
# Returns f i r s t 2 rows

’ S c o r e ’ ] , d t y p e =’ o b j e c t ’ )

Label-Based Indexer: loc[]
Accesses a group of rows and columns by label(s) or a boolean array.
Syntax: df.loc[row indexer, column indexer]
Slicing with labels includes both the start and stop endpoints.
Computer Engineering

DI04000061: ML

4th Semester

118 / 455

Supports boolean selection for conditional filtering.
d f i d x = d f . s e t i n d e x ( ’ Name ’ )
# A c c e s s s p e c i f i c e n t r y by l a b e l
a g e b o b = d f i d x . l o c [ ’ Bob ’ , ’ Age ’ ]

# 30

# C o n d i t i o n a l i n d e x i n g : S e l e c t r o w s w h e r e S c o r e > 80
h i g h s c o r e s = d f . l o c [ d f [ ’ S c o r e ’ ] > 8 0 , [ ’ Name ’ , ’ S c o r e ’ ] ]
print ( high scores )
and sum()

Detecting Missing Values
Real-world ML datasets often contain missing values (NaN or None).

isnull(): Returns a boolean DataFrame indicating True where values are null.
sum(): When chained (df.isnull().sum()), counts total missing values per column.
i m p o r t numpy a s np
r a w d a t a = { ’A ’ : [ 1 , np . nan , 3 ] ,
’B ’ : [ np . nan , 5 , 6 ] ,
’C ’ : [ 7 , 8 , np . nan ] }
d f n u l l = pd . DataFrame ( r a w d a t a )
# Count m i s s i n g v a l u e s p e r column
p r i n t ( d f n u l l . i s n u l l ( ) . sum ( ) )
# Output : A : 1 , B : 1 , C : 1
Computer Engineering

DI04000061: ML

4th Semester

120 / 455

and drop()

Removing Missing Values & Dropping Features
Clean datasets by discarding incomplete records or removing irrelevant features.
dropna(axis=0/1, how=’any’/’all’): Drops rows (axis=0) or columns (axis=1) containing missing values.
drop(labels, axis=0/1): Removes specified labels from rows or columns.
# Remove a l l r o w s w i t h a t l e a s t one m i s s i n g v a l u e
d f c l e a n = d f n u l l . dropna ( )
# Drop s p e c i f i c column ( ’ C ’ ) from DataFrame
d f n o c = d f n u l l . d r o p ( c o l u m n s =[ ’C ’ ] )
# Drop row by i n d e x l a b e l ( e . g . , i n d e x 0 )
d f n o r o w 0 = d f n u l l . d r o p ( i n d e x =0)

Identifying Redundant Data
Duplicate entries can distort statistical analysis and model evaluations.
duplicated(keep=’first’/’last’/False): Returns a boolean Series flagging duplicate rows.
Parameter keep=’first’ marks duplicates except for the first occurrence.
Combine with boolean indexing to inspect duplicate rows.
d u p d a t a = { ’ ID ’ : [ 1 0 1 , 1 0 2 , 1 0 2 , 1 0 3 ] ,
’ F e a t u r e ’ : [ ’A ’ , ’B ’ , ’B ’ , ’C ’ ] }
d f d u p = pd . DataFrame ( d u p d a t a )
Computer Engineering

DI04000061: ML

4th Semester

121 / 455

# Get b o o l e a n mask o f d u p l i c a t e r o w s
p r i n t ( df dup . duplicated ())
# F i l t e r and d i s p l a y d u p l i c a t e r o w s
p r i n t ( df dup [ df dup . duplicated ( ) ] )
, max(), and sum()

Numerical Aggregations
Pandas provides fast summary statistic methods across Series or DataFrames.
sum(): Returns sum of values over requested axis (axis=0 column-wise, axis=1 row-wise).
min(): Computes minimum value along specified axis.
max(): Computes maximum value along specified axis.
s c o r e s = pd . DataFrame ({
’ Midterm ’ : [ 7 5 , 8 8 , 9 2 ] ,
’ F i n a l ’ : [ 8 0 , 85 , 95]
})
p r i n t ( ” Column Min : ” , s c o r e s . min ( ) )
p r i n t ( ” Column Max : ” , s c o r e s . max ( ) )
p r i n t ( ” T o t a l Sum p e r S t u d e n t : \ n” , s c o r e s . sum ( a x i s =1))

Sorting Datasets
Reorder rows based on values in one or more columns.
Computer Engineering

DI04000061: ML

4th Semester

123 / 455

sort values(by, ascending=True, inplace=False): Sorts DataFrame by specified column(s).

Supports sorting by multiple columns with individual ordering directions.

d f s t u d e n t s = pd . DataFrame ({
’ Name ’ : [ ’ A l i c e ’ , ’ Bob ’ , ’ C h a r l i e ’ , ’ D a v i d ’ ] ,
’ Group ’ : [ ’B ’ , ’A ’ , ’A ’ , ’B ’ ] ,
’ Score ’ : [ 8 8 , 95 , 92 , 85]
})
# S o r t p r i m a r i l y by Group ( a s c e n d i n g ) , s e c o n d a r i l y by S c o r e ( d e s c e n d i n g )
df sorted = df students . sort values (
by =[ ’ Group ’ , ’ S c o r e ’ ] ,
a s c e n d i n g =[ True , F a l s e ]
)
print ( df sorted )

Computer Engineering

DI04000061: ML

4th Semester

124 / 455

Summary: Data Manipulation in ML Workflow I

Preprocessing Pipeline Roadmap
Connecting Pandas data manipulation operations to the ML pipeline:
1

Exploration: Use shape, columns, head(), and tail() for initial dataset assessment.

2

Cleaning: Detect missing entries with isnull().sum(), clean with dropna(), and filter duplicates with duplicated().

3

Feature Engineering: Isolate features/labels via loc[], drop irrelevant variables using drop().

4

Analysis & Ordering: Summarize features with min(), max(), sum(), and structure results with sort values().

Computer Engineering

DI04000061: ML

4th Semester

124 / 455

Introduction to CSV Handling in Pandas I

What is a CSV File?
CSV (Comma-Separated Values): A plain-text tabular data format widely used in machine learning datasets.
Records are stored line-by-line; data fields are separated by a delimiter (commonly a comma).

Pandas I/O Capabilities
pd.read csv(): Parses flat CSV files into a Pandas DataFrame.
DataFrame.to csv(): Exports DataFrame records to disk as a CSV file.
Utilizes an optimized C-parsing engine for fast data loading in ML workflows.

Computer Engineering

DI04000061: ML

4th Semester

125 / 455

Reading CSV Files: Core Parameters of read csv() I
Essential Arguments
filepath or buffer: File path, URL string, or file-like object.
sep / delimiter: Character separator (default is ’,’; e.g., ’\t’, ’;’).
header: Line index for column titles (default 0; set to None if no header exists).
names: List of custom column names to assign.

i m p o r t p a n d a s a s pd
# Basic loading with header i n f e r e n c e
d f = pd . r e a d c s v ( ’ h o u s i n g . c s v ’ )
# Custom d e l i m i t e r and e x p l i c i t column names
d f c u s t o m = pd . r e a d c s v (
’ data . t x t ’ ,
s e p= ’ ; ’ ,
h e a d e r=None ,
names =[ ’ Age ’ , ’ Income ’ , ’ T a r g e t ’ ]
)

Computer Engineering

DI04000061: ML

4th Semester

126 / 455

read csv(): Indexing, Column Selection, and Dtypes I

Optimizing Import Performance
index col: Column(s) to set as row labels in the DataFrame.
usecols: Load only a specific subset of columns to save memory.
dtype: Explicitly declare column data types to avoid inference overhead.

# S e l e c t columns , s e t i n d e x , and s p e c i f y e x a c t d t y p e s
d f = pd . r e a d c s v (
’ dataset . csv ’ ,
i n d e x c o l= ’ ID ’ ,
u s e c o l s =[ ’ ID ’ , ’ Age ’ , ’ S a l a r y ’ , ’ P u r c h a s e d ’ ] ,
d t y p e={ ’ Age ’ : ’ i n t 3 2 ’ , ’ S a l a r y ’ : ’ f l o a t 6 4 ’ , ’ P u r c h a s e d ’ :
)

’ c a t e g o r y ’}

print ( df . dtypes )

Computer Engineering

DI04000061: ML

4th Semester

127 / 455

read csv(): Handling Missing Data and Dates I

Data Cleaning at Load Time
na values: Custom scalar, string, or list of values to recognize as NaN.
keep default na: Boolean determining whether to append custom na values to standard NaN strings.
parse dates: Automatically convert date-like string columns into datetime64 objects.

# Custom m i s s i n g v a l u e s e n t i n e l v a l u e s and d a t e p a r s i n g
d f = pd . r e a d c s v (
’ s a l e s l o g . csv ’ ,
n a v a l u e s =[ ’NA ’ , ’ m i s s i n g ’ , ’ −999 ’ , ’N/A ’ ] ,
k e e p d e f a u l t n a=True ,
p a r s e d a t e s =[ ’ Date Time ’ ]
)
p r i n t ( d f . i s n a ( ) . sum ( ) )

Computer Engineering

DI04000061: ML

4th Semester

128 / 455

Handling Large Datasets: Chunking with read csv() I

Memory-Efficient Batch Loading
Loading multi-gigabyte CSV files all at once can exceed system memory.
chunksize: Returns an iterable TextFileReader object yielding DataFrame chunks of specified size.

# P r o c e s s a l a r g e CSV f i l e
c h u n k s i z e = 50000
total filtered = 0

iteratively

i n chunks o f 50 ,000 rows

f o r chunk i n pd . r e a d c s v ( ’ b i g d a t a . c s v ’ , c h u n k s i z e=c h u n k s i z e ) :
# F i l t e r r o w s p e r chunk w i t h o u t l o a d i n g e n t i r e f i l e i n t o RAM
f i l t e r e d c h u n k = chunk [ chunk [ ’ s c o r e ’ ] > 0 . 8 ]
t o t a l f i l t e r e d += l e n ( f i l t e r e d c h u n k )
p r i n t ( f ” T o t a l m a t c h i n g r e c o r d s : { t o t a l f i l t e r e d }” )

Computer Engineering

DI04000061: ML

4th Semester

129 / 455

Writing CSV Files: DataFrame.to csv() I

Key Export Arguments
path or buf: Destination file path or write stream.
index: Set to False to prevent writing row index numbers as an un-named column.
header: Write column titles (default True).
na rep: String representation for missing NaN values (default ’’).

i m p o r t p a n d a s a s pd
d f = pd . DataFrame ({
’ ID ’ : [ 1 0 1 , 1 0 2 , 1 0 3 ] ,
’ Name ’ : [ ’ A l i c e ’ , ’ Bob ’ , ’ C h a r l i e ’ ] ,
’ S c o r e ’ : [ 8 5 . 5 , None , 9 2 . 0 ]
})
# S t a n d a r d e x p o r t e x c l u d i n g auto−g e n e r a t e d i n t e g e r i n d e x
d f . t o c s v ( ’ p r o c e s s e d d a t a . c s v ’ , i n d e x=F a l s e , n a r e p= ’NaN ’ )

Computer Engineering

DI04000061: ML

4th Semester

130 / 455

to csv(): Advanced Export Features I

Selective Output, Encoding, and Compression
columns: Specify a subset of columns to write out.
encoding: Character encoding format (e.g., ’utf-8’, ’latin1’, ’utf-8-sig’).
compression: Compress output directly on write (’gzip’, ’zip’, ’bz2’).

# E x p o r t s e l e c t e d c o l u m n s d i r e c t l y t o a g z i p p e d CSV w i t h UTF−8 e n c o d i n g
df . to csv (
’ o u t p u t s e l e c t e d . c s v . gz ’ ,
c o l u m n s =[ ’ ID ’ , ’ S c o r e ’ ] ,
i n d e x=F a l s e ,
e n c o d i n g= ’ u t f −8 ’ ,
c o m p r e s s i o n= ’ g z i p ’
)

Computer Engineering

DI04000061: ML

4th Semester

131 / 455

Complete Workflow: Load, Process, and Save I

Practical Machine Learning Data Pipeline
An end-to-end workflow reading raw data, filtering missing targets, selecting features, and writing clean output.

i m p o r t p a n d a s a s pd
# 1 . Load raw CSV w i t h m i s s i n g v a l u e s e n t i n e l s and d a t e p a r s i n g
d f = pd . r e a d c s v ( ’ r a w d a t a . c s v ’ , n a v a l u e s =[ ’ ? ’ ] , p a r s e d a t e s =[ ’ t i m e s t a m p ’ ] )
# 2 . C l e a n m i s s i n g l a b e l s and s e l e c t r e l e v a n t f e a t u r e s
c l e a n d f = d f . d r o p n a ( s u b s e t =[ ’ t a r g e t ’ ] ) . c o p y ( )
f e a t u r e s = [ ’ timestamp ’ , ’ f e a t u r e 1 ’ , ’ f e a t u r e 2 ’ , ’ t a r g e t ’ ]
final df = clean df [ features ]
# 3 . E x p o r t p r o c e s s e d d a t a s e t f o r downstream m o d e l i n g
f i n a l d f . t o c s v ( ’ c l e a n d a t a s e t . c s v ’ , i n d e x=F a l s e )
p r i n t ( ” P i p e l i n e complete : c l e a n d a t a s e t . csv created s u c c e s s f u l l y . ” )

Computer Engineering

DI04000061: ML

4th Semester

132 / 455

Introduction to Matplotlib & Core Architecture I

What is Matplotlib?
The foundational 2D visualization library in Python’s scientific computing stack.
Provides full control over figure elements, rendering publication-quality graphics.

Matplotlib Architecture: Object-Oriented vs. Pyplot
Pyplot API (pyplot): State-based interface (similar to MATLAB); quick for simple scripts.
Object-Oriented (OO) API: Explicitly creates Figure (canvas) and Axes (plot window); recommended for robust ML visualization pipelines.

import m a t p l o t l i b . p y p l o t as p l t
i m p o r t numpy a s np
# O b j e c t−O r i e n t e d a p p r o a c h ( P r e f e r r e d i n ML w o r k f l o w s )
f i g , ax = p l t . s u b p l o t s ( f i g s i z e =(8 , 4 ) )
ax . s e t t i t l e ( ” O b j e c t−O r i e n t e d I n t e r f a c e Example ” )

Computer Engineering

DI04000061: ML

4th Semester

133 / 455

Line Plots: Tracking Model Training Curves I

Applications in Machine Learning
Monitoring loss function minimization over training epochs.
Detecting overfitting (divergence between training and validation loss).
Tracking hyperparameter tuning metrics across iterations.

e p o c h s = np . a r a n g e ( 1 , 1 1 )
t r a i n l o s s = [0.9 , 0.6 , 0.4 , 0.3 , 0.25 , 0.2 , 0.18 , 0.15 , 0.14 , 0.12]
val loss
= [0.95 , 0.65 , 0.48 , 0.38 , 0.35 , 0.36 , 0.39 , 0.42 , 0.45 , 0.48]
f i g , ax = p l t . s u b p l o t s ( f i g s i z e =(7 , 3 . 5 ) )
ax . p l o t ( e p o c h s , t r a i n l o s s , l a b e l= ’ T r a i n i n g L o s s ’ , c o l o r= ’ b l u e ’ , m a r k e r= ’ o ’ )
ax . p l o t ( e p o c h s , v a l l o s s , l a b e l= ’ V a l i d a t i o n L o s s ’ , c o l o r= ’ r e d ’ , l i n e s t y l e = ’−− ’ )
ax . s e t x l a b e l ( ’ Epochs ’ )
ax . s e t y l a b e l ( ’ C r o s s−E n t r o p y L o s s ’ )
ax . s e t t i t l e ( ’ T r a i n i n g v s V a l i d a t i o n L o s s ( O v e r f i t t i n g D e t e c t i o n ) ’ )
ax . l e g e n d ( )
ax . g r i d ( True , l i n e s t y l e = ’ : ’ , a l p h a =0.6)

Computer Engineering

DI04000061: ML

4th Semester

134 / 455

Scatter Plots: Visualizing Feature Relationships I

EDA and Clustering Inspection
ax.scatter(): Plots bivariate data points to analyze feature correlation.
Useful for visualizing 2D projection of classes, cluster assignments, and decision boundaries.
Color-coding by class labels (c=y) and scaling markers by feature magnitude.

# G e n e r a t e s y n t h e t i c 2− f e a t u r e c l a s s i f i c a t i o n
X = np . random . r a n d n ( 1 0 0 , 2 )
y = (X [ : , 0 ] + X [ : , 1 ] > 0 ) . a s t y p e ( i n t )

dataset

f i g , ax = p l t . s u b p l o t s ( f i g s i z e =(7 , 3 . 5 ) )
s c a t t e r = ax . s c a t t e r (X [ : , 0 ] , X [ : , 1 ] , c=y , cmap= ’ coolwarm ’ , e d g e c o l o r s= ’ k ’ , a l p h a =0.8)
ax . s e t x l a b e l ( ’ F e a t u r e 1 ( $ x 1 $ ) ’ )
ax . s e t y l a b e l ( ’ F e a t u r e 2 ( $ x 2 $ ) ’ )
ax . s e t t i t l e ( ’ 2D F e a t u r e D i s t r i b u t i o n by C l a s s L a b e l ’ )
c b a r = f i g . c o l o r b a r ( s c a t t e r , ax=ax )
cbar . s e t l a b e l ( ’ C l a s s Target ’ )

Computer Engineering

DI04000061: ML

4th Semester

135 / 455

Histograms & Box Plots: Data Distribution Analysis I

Assessing Feature Distributions
Histograms (ax.hist): Inspect normality, skewness, multimodality, and zero-inflation.
Box Plots (ax.boxplot): Identify median, interquartile range (IQR), and empirical outliers.

f e a t u r e d a t a = np . random . n o r m a l ( l o c =50 , s c a l e =15 , s i z e =500)
f i g , ( ax1 , ax2 ) = p l t . s u b p l o t s ( 1 , 2 , f i g s i z e =(9 , 3 . 5 ) )
# Histogram with p r o b a b i l i t y d e n s i t y n o r m a l i z a t i o n
ax1 . h i s t ( f e a t u r e d a t a , b i n s =25 , c o l o r= ’ s k y b l u e ’ , e d g e c o l o r= ’ b l a c k ’ , d e n s i t y=True )
ax1 . s e t t i t l e ( ’ F e a t u r e H i s t o g r a m & D e n s i t y ’ )
ax1 . s e t x l a b e l ( ’ F e a t u r e V a l u e ’ )
# Box p l o t f o r o u t l i e r d e t e c t i o n
ax2 . b o x p l o t ( f e a t u r e d a t a , p a t c h a r t i s t=True , b o x p r o p s=d i c t ( f a c e c o l o r= ’ l i g h t g r e e n ’ ) )
ax2 . s e t t i t l e ( ’ F e a t u r e Box P l o t ’ )
ax2 . s e t y l a b e l ( ’ V a l u e S c a l e ’ )

Computer Engineering

DI04000061: ML

4th Semester

136 / 455

Bar Charts: Class Imbalance & Feature Importance I

Categorical Visualization in ML
Visualizing class frequencies to detect data imbalance before model training.
Displaying relative feature importances from decision trees or random forests.

f e a t u r e s = [ ’ Age ’ , ’ Income ’ , ’ C r e d i t S c o r e ’ , ’ E d u c a t i o n ’ , ’ Z i p c o d e ’ ]
importance = [0.35 , 0.28 , 0.20 , 0.12 , 0.05]
f i g , ax = p l t . s u b p l o t s ( f i g s i z e =(7 , 3 . 5 ) )
# H o r i z o n t a l bar chart f o r readable f e a t u r e l a b e l s
y p o s = np . a r a n g e ( l e n ( f e a t u r e s ) )
ax . b a r h ( y p o s , i m p o r t a n c e , a l i g n= ’ c e n t e r ’ , c o l o r= ’ t e a l ’ , a l p h a =0.85)
ax . s e t y t i c k s ( y p o s )
ax . s e t y t i c k l a b e l s ( f e a t u r e s )
ax . i n v e r t y a x i s ( ) # L a b e l s r e a d top−to−bottom
ax . s e t x l a b e l ( ’ G i n i I m p o r t a n c e S c o r e ’ )
ax . s e t t i t l e ( ’ Random F o r e s t F e a t u r e I m p o r t a n c e ’ )

Computer Engineering

DI04000061: ML

4th Semester

137 / 455

Multi-Panel Layouts: Creating Subplot Dashboards I
Subplot Management with plt.subplots()
plt.subplots(nrows, ncols): Generates a grid of Axes objects.
Essential for building multi-metric diagnostics (e.g., Loss, Accuracy, Precision, Recall).
Use plt.tight layout() to automatically adjust spacing and prevent overlapping labels.

f i g , a x e s = p l t . s u b p l o t s ( 2 , 2 , f i g s i z e =(8 , 4 . 5 ) )
a x e s [ 0 , 0 ] . p l o t ( [ 1 , 2 , 3 ] , [ 2 , 4 , 9 ] , ’ b−’ )
axes [0 , 0 ] . s e t t i t l e ( ’ Training Loss ’ )
axes [0 , 1 ] . plot ([1 , 2 , 3] , [ 0 . 5 , 0.8 , 0.95] ,
axes [ 0 , 1 ] . s e t t i t l e ( ’ V a l i d a t i o n Accuracy ’ )

’ g−’ )

a x e s [ 1 , 0 ] . h i s t ( np . random . r a n d n ( 1 0 0 ) , c o l o r= ’ p u r p l e ’ )
axes [1 , 0 ] . s e t t i t l e ( ’ Residuals ’ )
a x e s [ 1 , 1 ] . s c a t t e r ( np . random . r a n d ( 2 0 ) , np . random . r a n d ( 2 0 ) , c o l o r= ’ o r a n g e ’ )
axes [1 , 1 ] . s e t t i t l e ( ’ P r e d i c t i o n s vs Actuals ’ )
plt . tight layout ()

Computer Engineering

DI04000061: ML

4th Semester

138 / 455

Matrix Visualizations: Heatmaps & Confusion Matrices I
Visualizing 2D Data Matrices
ax.imshow() renders numeric 2D arrays as color-encoded heatmaps.
Widely used for confusion matrix evaluation and feature correlation matrices.

# 3 x3 C o n f u s i o n M a t r i x Example
c o n f m a t r i x = np . a r r a y ( [ [ 4 5 , 3 , 2 ] ,
[ 1 , 38 , 5 ] ,
[ 4, 2, 50]])
f i g , ax = p l t . s u b p l o t s ( f i g s i z e =(5 , 4 ) )
c a x = ax . imshow ( c o n f m a t r i x , cmap= ’ B l u e s ’ )
# Add n u m e r i c l a b e l s t o e a c h m a t r i x c e l l
f o r i i n range ( c o n f m a t r i x . shape [ 0 ] ) :
f o r j i n range ( c o n f m a t r i x . shape [ 1 ] ) :
ax . t e x t ( j , i , s t r ( c o n f m a t r i x [ i , j ] ) , ha= ’ c e n t e r ’ , va= ’ c e n t e r ’ ,
c o l o r= ’ w h i t e ’ i f c o n f m a t r i x [ i , j ] > 25 e l s e ’ b l a c k ’ )
f i g . c o l o r b a r ( cax )
ax . s e t x t i c k s ( [ 0 , 1 , 2 ] ) ; ax . s e t y t i c k s ( [ 0 , 1 , 2 ] )
ax . s e t x t i c k l a b e l s ( [ ’ C l a s s 0 ’ , ’ C l a s s 1 ’ , ’ C l a s s 2 ’ ] )
ax . s e t y t i c k l a b e l s ( [ ’ C l a s s 0 ’ , ’ C l a s s 1 ’ , ’ C l a s s 2 ’ ] )
ax . s e t x l a b e l ( ’ P r e d i c t e d L a b e l ’ ) ; ax . s e t y l a b e l ( ’ True L a b e l ’ )

Computer Engineering

DI04000061: ML

4th Semester

139 / 455

Customization & Saving Publication-Quality Figures I

Exporting and Styling Figures
Built-in Styles: plt.style.use(’seaborn-v0 8’) or ’ggplot’ for pre-configured aesthetics.
Text Annotations: ax.annotate() to highlight critical points (e.g., minimum loss).
High-Resolution Export: fig.savefig() for vector (PDF, SVG) or high-DPI raster (PNG) formats.

# Custom s t y l i n g and v e c t o r e x p o r t
p l t . s t y l e . u s e ( ’ s e a b o r n −v 0 8−w h i t e g r i d ’ )
f i g , ax = p l t . s u b p l o t s ( f i g s i z e =(6 , 3 . 5 ) )
x = np . l i n s p a c e ( 0 , 1 0 , 1 0 0 )
ax . p l o t ( x , np . s i n ( x ) , l a b e l=r ’ $\ s i n ( x ) $ ’ , c o l o r= ’ d a r k b l u e ’ , lw =2)
# Annotation f o r key p o i n t
ax . a n n o t a t e ( ’ Peak ’ , xy =(np . p i / 2 , 1 ) , x y t e x t =(np . p i /2 + 1 , 0 . 8 ) ,
a r r o w p r o p s=d i c t ( f a c e c o l o r= ’ b l a c k ’ , s h r i n k = 0 . 0 5 ) )
# E x p o r t w i t h t i g h t b o u n d i n g box t o remove e x t r a p a d d i n g
f i g . s a v e f i g ( ’ s i n e p l o t . p d f ’ , d p i =300 , b b o x i n c h e s= ’ t i g h t ’ )

Computer Engineering

DI04000061: ML

4th Semester

140 / 455

Summary: Key Matplotlib Functions for Machine Learning I

Visualization Type
Line Plot
Scatter Plot
Histogram
Box Plot
Bar Chart
Heatmap

Matplotlib Method
ax.plot()
ax.scatter()
ax.hist()
ax.boxplot()
ax.bar() / ax.barh()
ax.imshow()

ML Use Case
Learning curves, loss tracking
Cluster analysis, 2D projections
Feature distribution, skewness
Outlier detection, IQR range
Feature importance, class balance
Confusion matrix, correlation

Best Practices
Always label axes (set xlabel, set ylabel) and include legends.
Prefer the Object-Oriented API (fig, ax) over stateful plt.* calls.
Export vector graphics (.pdf, .svg) for academic reports.

Computer Engineering

DI04000061: ML

4th Semester

141 / 455

Overview of Matplotlib Visualization Functions I
Role of Data Visualization in Machine Learning
Visualizing data is a fundamental step in Exploratory Data Analysis (EDA), model evaluation, and diagnostic reporting. Matplotlib provides lower-level, highly
customizable plotting functions.

Basic Trend & Relationship Functions:

plt.plot(): Line plots for continuous functions, time series, and loss curves.
plt.scatter(): Scatter plots for feature relationships, clustering, and classification data.
Distribution & Summary Functions:

plt.bar() & plt.pie(): Categorical distributions and class composition.
plt.hist(): Histograms for univariate continuous feature distributions.
plt.boxplot(): Box plots for identifying outliers and quartile statistics.
Annotation & Exporting Functions:

plt.title(), plt.xlabel(), plt.ylabel(), plt.grid(): Chart formatting.
plt.show() & plt.savefig(): Rendering and saving figure artifacts.
Computer Engineering

DI04000061: ML

4th Semester

142 / 455

and Annotations

Tracking Continuous Metrics
plt.plot() connects numerical data points with lines. It is widely used to visualize training loss curves, evaluation accuracy over iterations, and time-series data.

import m a t p l o t l i b . p y p l o t as p l t
i m p o r t numpy a s np
e p o c h s = np . a r a n g e ( 1 , 1 1 )
t r a i n l o s s = [0.9 , 0.7 , 0.5 , 0.35 , 0.25 , 0.18 , 0.14 , 0.11 , 0.09 , 0.08]
val loss
= [0.95 , 0.75 , 0.55 , 0.42 , 0.33 , 0.28 , 0.26 , 0.25 , 0.26 , 0.27]
p l t . f i g u r e ( f i g s i z e =(7 , 4 ) )
p l t . p l o t ( e p o c h s , t r a i n l o s s , l a b e l= ’ T r a i n i n g L o s s ’ , c o l o r= ’ b l u e ’ , l i n e s t y l e = ’− ’ , m a r k e r= ’ o ’ )
p l t . p l o t ( e p o c h s , v a l l o s s , l a b e l= ’ V a l i d a t i o n L o s s ’ , c o l o r= ’ r e d ’ , l i n e s t y l e = ’−− ’ , m a r k e r= ’ s ’ )
p l t . t i t l e ( ’ Model T r a i n i n g v s V a l i d a t i o n L o s s ’ )
p l t . x l a b e l ( ’ Epochs ’ )
p l t . y l a b e l ( ’ C r o s s−E n t r o p y L o s s ’ )
p l t . g r i d ( True )
plt . legend ()
for Feature Inspection

Bivariate Feature Correlation & Clustering
plt.scatter() plots individual data points without connecting lines. Essential for visualizing feature correlation, decision boundaries, and cluster assignments.

import m a t p l o t l i b . p y p l o t as p l t
i m p o r t numpy a s np
Computer Engineering

DI04000061: ML

4th Semester

144 / 455

# S y n t h e t i c f e a t u r e d a t a f o r two c l a s s e s
np . random . s e e d ( 4 2 )
x1 = np . random . r a n d n ( 1 0 0 )
x2 = 2 ∗ x1 + np . random . r a n d n ( 1 0 0 )
l a b e l s = np . random . c h o i c e ( [ 0 , 1 ] , s i z e =100)
p l t . f i g u r e ( f i g s i z e =(7 , 4 ) )
s c a t t e r = p l t . s c a t t e r ( x1 , x2 , c=l a b e l s , cmap= ’ coolwarm ’ , s =50 , a l p h a = 0 . 8 , e d g e c o l o r s= ’ k ’ )
p l t . t i t l e ( ’ 2D F e a t u r e R e l a t i o n s h i p S c a t t e r P l o t ’ )
p l t . x l a b e l ( ’ F e a t u r e 1 ( $x 1$ ) ’ )
p l t . y l a b e l ( ’ F e a t u r e 2 ( $x 2$ ) ’ )
p l t . c o l o r b a r ( s c a t t e r , l a b e l= ’ C l a s s L a b e l ’ )
p l t . g r i d ( True , l i n e s t y l e = ’ : ’ )
and plt.pie()

Analyzing Discrete Features and Class Imbalance
plt.bar(): Represents categorical variables with rectangular bars proportional to values.
plt.pie(): Displays relative proportions of a whole as slices of a pie.

import m a t p l o t l i b . p y p l o t as p l t
classes = [ ’ Class A’ , ’ Class B’ , ’ Class C ’ , ’ Class D’ ]
counts = [450 , 300 , 150 , 100]
f i g , ( ax1 , ax2 ) = p l t . s u b p l o t s ( 1 , 2 , f i g s i z e =(9 , 4 ) )
# Bar C h a r t f o r C l a s s C o u n t s
ax1 . b a r ( c l a s s e s , c o u n t s , c o l o r =[ ’#1f 7 7 b 4 ’ , ’#f f 7 f 0 e ’ , ’#2c a 0 2 c ’ , ’#d62728 ’ ] )
Computer Engineering

DI04000061: ML

4th Semester

145 / 455

ax1 . s e t t i t l e ( ’ C l a s s F r e q u e n c y D i s t r i b u t i o n ’ )
ax1 . s e t y l a b e l ( ’ Count ’ )
# Pie Chart f o r C l a s s P r o p o r t i o n s
ax2 . p i e ( c o u n t s , l a b e l s =c l a s s e s , a u t o p c t= ’ %1.1 f%%’ , s t a r t a n g l e =90 , e x p l o d e = ( 0 . 1 , 0 , 0 , 0 ) )
ax2 . s e t t i t l e ( ’ C l a s s P r o p o r t i o n S h a r e ’ )
plt . tight layout ()

Histogram Analysis in Feature Preprocessing
plt.hist() bins continuous values to estimate probability density distributions. Crucial for detecting skewed distributions, multimodality, and variance issues.

import m a t p l o t l i b . p y p l o t as p l t
i m p o r t numpy a s np
# Feature with normal d i s t r i b u t i o n
f e a t u r e d a t a = np . random . n o r m a l ( l o c =50 , s c a l e =10 , s i z e =1000)
p l t . f i g u r e ( f i g s i z e =(7 , 4 ) )
n , b i n s , p a t c h e s = p l t . h i s t ( f e a t u r e d a t a , b i n s =25 , c o l o r= ’ s k y b l u e ’ , e d g e c o l o r= ’ b l a c k ’ , a l p h a =0.7)
p l t . t i t l e ( ’ Feature Value D i s t r i b u t i o n ’ )
p l t . x l a b e l ( ’ Feature Values ’ )
p l t . y l a b e l ( ’ Frequency ’ )
p l t . g r i d ( a x i s= ’ y ’ , a l p h a =0.75)

Computer Engineering

DI04000061: ML

4th Semester

147 / 455

Five-Number Summary & Outlier Identification
Box plots summarize statistical distributions via five key numbers:
Median (Q2 ): Middle horizontal line.
Interquartile Range (IQR = Q3 − Q1 ): Box length from 25th to 75th percentile.
Whiskers: Extend to 1.5 × IQR beyond Q1 and Q3 .
Outliers: Individual points plotted beyond whisker limits.

import m a t p l o t l i b . p y p l o t as p l t
i m p o r t numpy a s np
d a t a 1 = np . random . n o r m a l ( 1 0 0 , 1 0 , 2 0 0 )
d a t a 2 = np . c o n c a t e n a t e ( [ np . random . n o r m a l ( 9 0 , 2 0 , 2 0 0 ) , [ 1 6 0 , 1 7 0 , 2 0 ] ] ) # w i t h o u t l i e r s
p l t . f i g u r e ( f i g s i z e =(7 , 3 . 5 ) )
p l t . b o x p l o t ( [ da ta 1 , d a t a 2 ] , l a b e l s =[ ’ F e a t u r e 1 ’ , ’ F e a t u r e 2 ( O u t l i e r s ) ’ ] , p a t c h a r t i s t=True )
p l t . t i t l e ( ’ F e a t u r e C o m p a r i s o n Box P l o t ’ )
p l t . y l a b e l ( ’ Value Scale ’ )
p l t . g r i d ( a x i s= ’ y ’ )
and plt.savefig()

Workflow for Saving High-Resolution Figures
plt.grid(): Toggles grid lines for easier numeric alignment.
plt.savefig(): Exports plots to disk (PNG, PDF, SVG). Call before plt.show()!
plt.show(): Displays the plot interface and flushes the active figure memory buffer.

import m a t p l o t l i b . p y p l o t as p l t
Computer Engineering

DI04000061: ML

4th Semester

148 / 455

i m p o r t numpy a s np
x = np . l i n s p a c e ( 0 , 1 0 , 1 0 0 )
p l t . f i g u r e ( f i g s i z e =(6 , 3 . 5 ) )
p l t . p l o t ( x , np . s i n ( x ) , l a b e l= ’ s i n ( x ) ’ , c o l o r= ’ p u r p l e ’ )
p l t . t i t l e ( ’ S i n e Wave S i g n a l ’ )
p l t . x l a b e l ( ’ Time ( t ) ’ )
p l t . y l a b e l ( ’ Amplitude ’ )
p l t . g r i d ( True )
plt . legend ()
# Save f i g u r e t o f i l e BEFORE c a l l i n g show ( )
p l t . s a v e f i g ( ’ s i n e w a v e . png ’ , d p i =300 , b b o x i n c h e s= ’ t i g h t ’ )
p l t . show ( )

Computer Engineering

DI04000061: ML

4th Semester

149 / 455

Summary of Matplotlib Functions I

Function
plot()
scatter()
bar()
hist()
boxplot()
pie()
savefig()

Primary Machine Learning Use Case
Loss curves, time-series, metric trends
2D feature correlation, cluster visualization
Class imbalance, feature importance counts
Feature density distribution, skew detection
Outlier detection, quantile analysis
Class proportion breakdown
Exporting figures for research papers/reports

Key Parameters
linestyle, marker, color
s, c, cmap, alpha
x, height, color
bins, density, alpha
vert, patch artist
autopct, explode
dpi, bbox inches

Best Practices in ML Visualization
Always label axes (xlabel, ylabel), add a descriptive title, include a grid or legend when appropriate, and execute savefig() before show() to prevent saving
blank figures!

Computer Engineering

DI04000061: ML

4th Semester

149 / 455

Overview of Scikit-learn in Python ML I

What is Scikit-learn?
Scikit-learn (sklearn) is the industry-standard Python library for classical machine learning algorithms, built on top of NumPy, SciPy, and Matplotlib.
Unified Interface: Provides a consistent, object-oriented API across disparate model families (linear models, trees, support vector machines, clustering).
Standardized Data Model:

Feature Matrix X : A 2D array of shape (nsamples , nfeatures ) containing input covariates.
Target Vector y : A 1D array of shape (nsamples , ) containing target labels or continuous
responses.
Scope: Supervised learning (classification, regression), unsupervised learning (clustering, dimensionality reduction), and preprocessing/model selection.

Computer Engineering

DI04000061: ML

4th Semester

150 / 455

Scikit-learn API Design Principles I

Core Architectural Principles (Buitinck et al., 2013)
Consistency: All objects adhere to a minimalist, uniform interface.
Inspection: Constructor parameters and learned model attributes are public.
Non-factoring of objects: Data is represented as standard NumPy arrays or SciPy matrices rather than proprietary classes.

Estimators: Any object that learns from data via fit(X, y).

Learned parameters end with a trailing underscore (e.g., model.coef , scaler.mean ).
Transformers: Objects that modify or filter data via transform(X) or fit transform(X).
Predictors: Objects capable of producing inferences on new data via predict(X) or predict proba(X).

Computer Engineering

DI04000061: ML

4th Semester

151 / 455

Data Preprocessing and Leakage Prevention I
The Fit-Transform Paradigm
To prevent data leakage, transformation parameters (e.g., feature mean µ and standard deviation σ) must be computed solely on the training split, then applied
to both train and test splits.

i m p o r t numpy a s np
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
# S y n t h e t i c f e a t u r e m a t r i x ( 5 s a m p l e s , 2 f e a t u r e s ) and t a r g e t
X = np . a r r a y ( [ [ 1 0 0 . 0 , 0 . 1 ] , [ 2 0 0 . 0 , 0 . 4 ] , [ 1 5 0 . 0 , 0 . 2 ] ,
[300.0 , 0.8] , [250.0 , 0.5]])
y = np . a r r a y ( [ 0 , 0 , 0 , 1 , 1 ] )
# S p l i t dataset p r i o r to s c a l i n g
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42
)
# F i t s c a l e r ON TRAINING DATA ONLY
scaler = StandardScaler ()
X train scaled = scaler . fit transform ( X train )
# T r a n s f o r m t e s t d a t a u s i n g t r a i n i n g p a r a m e t e r s (mu , s i g m a )
X test scaled = s c a l e r . transform ( X test )

Computer Engineering

DI04000061: ML

4th Semester

152 / 455

Supervised Learning Workflow: Classification I
Model Lifecycle
Instantiate Model −→ Fit on Training Data −→ Predict on Unseen Data −→ Evaluate Metrics

from
from
from
from

s k l e a r n . d a t a s e t s import l o a d i r i s
s k l e a r n . m o d e l s e l e c t i o n import t r a i n t e s t s p l i t
s k l e a r n . l i n e a r m o d e l import L o g i s t i c R e g r e s s i o n
s k l e a r n . m e t r i c s import a c c u r a c y s c o r e , c l a s s i f i c a t i o n r e p o r t

# Load benchmark I r i s d a t a s e t
i r i s = l o a d i r i s ()
X tr , X te , y t r , y t e = t r a i n t e s t s p l i t (
i r i s . d at a , i r i s . t a r g e t , t e s t s i z e = 0 . 3 , r a n d o m s t a t e =42
)
# I n s t a n t i a t e estimator with hyperparameters
c l f = L o g i s t i c R e g r e s s i o n ( m a x i t e r =200 , C=1.0)
c l f . f i t ( X tr , y t r )
# I n f e r e n c e and q u a n t i t a t i v e e v a l u a t i o n
y pred = c l f . p r e d i c t ( X te )
p r i n t ( f ” A c c u r a c y : { a c c u r a c y s c o r e ( y t e , y p r e d ) : . 4 f }” )
p r i n t ( ” L e a r n e d C o e f f i c i e n t s Shape : ” , c l f . c o e f . s h a p e )

Computer Engineering

DI04000061: ML

4th Semester

153 / 455

Model Selection: Cross-Validation & Hyperparameter Tuning I
Cross-Validation & Grid Search
GridSearchCV evaluates hyperparameter combinations using K -Fold cross-validation to optimize model capacity while preventing overfitting.

from s k l e a r n . e n s e m b l e i m p o r t R a n d o m F o r e s t C l a s s i f i e r
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t G r i d S e a r c h C V
# Base e s t i m a t o r
r f = R a n d o m F o r e s t C l a s s i f i e r ( r a n d o m s t a t e =42)
# Define hyperparameter search grid
param grid = {
’ n estimators ’ : [50 , 100] ,
’ m a x d e p t h ’ : [ 3 , 5 , None ]
}
# 5−F o l d C r o s s−V a l i d a t i o n G r i d S e a r c h
g r i d s e a r c h = G r i d S e a r c h C V ( e s t i m a t o r=r f , p a r a m g r i d=p a r a m g r i d ,
c v =5 , s c o r i n g= ’ a c c u r a c y ’ )
g r i d s e a r c h . f i t ( X tr , y t r )
p r i n t ( ” Best Parameters : ” , g r i d s e a r c h . best params )
p r i n t ( f ” B e s t CV A c c u r a c y : { g r i d s e a r c h . b e s t s c o r e : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

154 / 455

Building Robust Machine Learning Pipelines I
Encapsulation with sklearn.pipeline.Pipeline
Pipeline chains preprocessing steps and an estimator into a single unified object, preventing data leakage during cross-validation and simplifying production
deployment.

from
from
from
from

s k l e a r n . p i p e l i n e import P i p e l i n e
s k l e a r n . impute import SimpleImputer
s k l e a r n . p r e p r o c e s s i n g import S t an d ar d Sc a l er
s k l e a r n . svm i m p o r t SVC

# D e f i n e o r d e r e d s e q u e n c e o f t r a n s f o r m e r s and f i n a l
pipeline = Pipeline ([
( ’ i m p u t e r ’ , S i m p l e I m p u t e r ( s t r a t e g y= ’ median ’ ) ) ,
( ’ scaler ’ , StandardScaler ()) ,
( ’ c l a s s i f i e r ’ , SVC( k e r n e l= ’ r b f ’ , C= 1 . 0 ) )
])
# Single f i t call triggers
p i p e l i n e . f i t ( X tr , y t r )

sequential

estimator

f i t t r a n s f o r m and f i n a l

fit

# Predict automatically transforms test features before inference
y p r e d p i p e = p i p e l i n e . p r e d i c t ( X te )
p r i n t ( f ” P i p e l i n e T e s t A c c u r a c y : { p i p e l i n e . s c o r e ( X t e , y t e ) : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

155 / 455

Summary of Key Scikit-learn Modules I

Module
preprocessing
model selection
linear model
ensemble
metrics
pipeline

Key Classes / Functions
StandardScaler, OneHotEncoder
train test split, GridSearchCV
LinearRegression, LogisticRegression
RandomForestClassifier, HistGradientBoostingClassifier
accuracy score, confusion matrix
Pipeline, make pipeline

Primary Purpose
Feature transformation & scaling
Validation & tuning
Linear baseline estimators
Tree ensemble models
Evaluation & diagnostics
Workflow composition

Best Practice
Always construct full workflows using Pipeline objects to guarantee zero data leakage during validation and cross-validation!

Computer Engineering

DI04000061: ML

4th Semester

156 / 455

Overview of Machine Learning Activities I

The Lifecycle of an ML Project
Developing a machine learning solution is an iterative engineering process involving structured steps from raw data to deployed models.
Problem Definition: Identifying business or research goals, target variables, and success metrics.
Data Ingestion & Exploration: Understanding distributions, anomalies, missingness, and feature relationships.
Data Preprocessing: Data cleaning, missing value imputation, categorical encoding, and feature scaling.
Feature Engineering: Extracting domain-specific representations to enhance model predictive power.
Model Building & Validation: Selection of algorithms, validation strategy, metric calculation, and error diagnostics.
Deployment & Monitoring: Productionizing model pipelines and tracking data/concept drift over time.

Computer Engineering

DI04000061: ML

4th Semester

157 / 455

Data Preprocessing & Cleaning I

Key Data Cleaning Steps
Raw real-world data is uncurated and noisy. Systematic cleaning is essential to prevent biased or failing models.

1

Handling Missing Values:

Imputation via mean, median, mode, or model-based methods (e.g., K -NN imputation).
Dropping missing features/records when missingness threshold is excessively high (> 50%).
2

Outlier Identification & Remediation:

Statistical detection using Interquartile Range (IQR = Q3 − Q1 ) or Z-score (|Z | > 3).
Winsorization (capping values) or truncation.
3

Deduplication & Data Consistency:

Removing duplicate observations and fixing non-standard string formats.

Computer Engineering

DI04000061: ML

4th Semester

158 / 455

Feature Encoding and Feature Scaling I
Categorical Encoding
Transforms non-numeric attributes into quantitative representations:
One-Hot Encoding: Creates binary indicator columns for nominal features (prevents artificial ordering).
Ordinal Encoding: Maps ordered categories to sequential integers (e.g., Low → 0, Medium → 1, High → 2).

Feature Scaling
Prevents features with larger magnitudes from dominating distance-based algorithms and gradient updates:
Min-Max Normalization: Rescales data to [0, 1] range:
x − xmin

′

x =

xmax − xmin

Standardization (Z-score): Centers data to zero mean and unit variance:
′

x =

Computer Engineering

x −µ
σ

∼ N (0, 1)

DI04000061: ML

4th Semester

159 / 455

Feature Engineering & Feature Selection I

Feature Engineering
Creating informative variables using domain knowledge to simplify learning:
Interaction terms (X1 × X2 ), feature ratios, and polynomial expansions.
Time-based features (e.g., day of week, hour, elapsed time).
Domain transformations (e.g., TF-IDF for text, Fourier transform for signal data).

Feature Selection Techniques
Selecting an optimal subset of attributes to improve model interpretability and reduce variance:
Filter Methods: Variance thresholding, Pearson correlation, ANOVA F -test, Mutual Information.
Wrapper Methods: Recursive Feature Elimination (RFE), Forward/Backward selection.
Embedded Methods: L1 Regularization (Lasso), Tree-based feature importance scores.

Computer Engineering

DI04000061: ML

4th Semester

160 / 455

Validation Strategies & Data Splitting I

Data Partitioning Scheme
To accurately estimate model generalization to unseen data:
Training Set (60 − 80%): Used to estimate parameters (weights and biases).
Validation Set (10 − 20%): Used for hyperparameter tuning and model selection.
Test Set (10 − 20%): Held out for unbiased final performance assessment.

Cross-Validation (CV)
Mitigates sample bias inherent to a single train-test split:
K -Fold Cross-Validation: Partitions dataset into K equal subsets; iterates K times training on K − 1 folds and evaluating on the remaining fold.
Stratified K -Fold: Preserves class ratio distributions across every fold (critical for imbalanced tasks).

Computer Engineering

DI04000061: ML

4th Semester

161 / 455

Python ML Workflow Example I
i m p o r t numpy a s np
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t , c r o s s v a l s c o r e
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r , OneHotEncoder
from s k l e a r n . compose i m p o r t C o l u m n T r a n s f o r m e r
from s k l e a r n . p i p e l i n e i m p o r t P i p e l i n e
from s k l e a r n . e n s e m b l e i m p o r t R a n d o m F o r e s t C l a s s i f i e r
# D e f i n e n u m e r i c a l and c a t e g o r i c a l f e a t u r e s
n u m c o l s = [ ’ age ’ , ’ i nc om e ’ , ’ c r e d i t s c o r e ’ ]
c a t c o l s = [ ’ education ’ , ’ occupation ’ ]
# Preprocessing pipeline
p r e p r o c e s s o r = C o l u m n T r a n s f o r m e r ( t r a n s f o r m e r s =[
( ’ num ’ , S t a n d a r d S c a l e r ( ) , n u m c o l s ) ,
( ’ c a t ’ , OneHotEncoder ( d r o p= ’ f i r s t ’ ) , c a t c o l s )
])
# F u l l Machine L e a r n i n g P i p e l i n e
p i p e l i n e = P i p e l i n e ( s t e p s =[
( ’ preprocessor ’ , preprocessor ) ,
( ’ c l a s s i f i e r ’ , R a n d o m F o r e s t C l a s s i f i e r ( n e s t i m a t o r s =100 , r a n d o m s t a t e =42))
])
# C r o s s−V a l i d a t i o n E v a l u a t i o n
# X train , y t r a i n defined beforehand
s c o r e s = c r o s s v a l s c o r e ( p i p e l i n e , X t r a i n , y t r a i n , c v =5 , s c o r i n g= ’ a c c u r a c y ’ )
p r i n t ( f ”5−F o l d CV A c c u r a c y : { s c o r e s . mean ( ) : . 4 f } +/− { s c o r e s . s t d ( ) : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

162 / 455

Summary of Machine Learning Activities I

Key Lessons
Data preparation and feature engineering comprise up to 80% of practical machine learning effort.
Data Leakage Caution: Always fit scalers, encoders, and feature selectors only on training data folds.
Continuous diagnostic feedback loops guide whether to gather more data, engineer new features, or alter hyperparameter space.

Next Steps in Unit 3
Lecture 3.2: Model Evaluation Metrics (Confusion Matrix, Precision-Recall, ROC-AUC, MSE, MAE).
Lecture 3.3: Bias-Variance Tradeoff & Hyperparameter Optimization.

Computer Engineering

DI04000061: ML

4th Semester

163 / 455

Taxonomy of Data in Machine Learning I
The Fundamental Structure of Datasets
Machine learning models operate on feature vectors x = (x1 , x2 , . . . , xd )T . Understanding the mathematical nature of each attribute xi is essential for selecting
appropriate algorithms, preprocessing techniques, and distance metrics.

Numerical (Quantitative) Data: Expresses quantities, measurements, or counts. Mathematical operations (addition, multiplication) are inherently
meaningful.

Continuous: Real-valued measurements (x ∈ R).
Discrete: Integer-valued counts (x ∈ Z≥0 ).
Categorical (Qualitative) Data: Expresses characteristics, labels, or group memberships. Represents qualitative attributes rather than numerical
magnitudes.

Nominal: Distinct categories without natural ordering.
Ordinal: Categories with a well-defined sequential order or rank.
Specialized Types: Datetime series, text/embeddings, images/tensors, and graph structures.

Computer Engineering

DI04000061: ML

4th Semester

164 / 455

Numerical (Quantitative) Data I

Characteristics of Numerical Features
Numerical variables represent quantifiable amounts where distance and magnitude carry concrete mathematical meaning.

Continuous Data:

Can take any real value within a continuous interval.
Examples: Height (175.4 cm), House Price ($450, 250), Temperature (23.6◦ C).
Measured using floating-point representations in computational implementations.
Discrete Data:

Takes countable, distinct values (typically integers).
Examples: Number of children (2), Website clicks per minute (142), Credit transactions per
day (5).
Expresses finite or countably infinite step values.

Computer Engineering

DI04000061: ML

4th Semester

165 / 455

Numerical (Quantitative) Data II

Stevens’ Levels of Measurement for Numerical Data
Interval Scale: Equal differences between values, but no true zero point (e.g., Temperature in ◦ C or ◦ F; 0◦ C does not mean zero heat). Ratios are
invalid (40◦ C is not ”twice as hot” as 20◦ C).
Ratio Scale: Equal intervals and a meaningful absolute zero (e.g., Income, Weight, Kelvin temperature). Ratios are valid (100 kg is twice 50 kg).

Computer Engineering

DI04000061: ML

4th Semester

166 / 455

Numerical Preprocessing: Scaling & Transformations I
Why Scaling Matters
Distance-based algorithms (K -NN, SVM, K -Means) and gradient descent updates are sensitive to feature scales. Unscaled features with large ranges dominate
loss functions and distance computations.
1

Standardization (Z-score Normalization): Centers features to zero mean (µ = 0) and unit variance (σ = 1):
′

x =

x −µ
σ

Recommended for models assuming Gaussian-distributed data (e.g., Logistic Regression, Linear Regression, PCA).
2

Min-Max Normalization: Rescales features linearly into a fixed bounded range [0, 1]:
′

x =

x − xmin
xmax − xmin

Sensitive to extreme outliers which compress the standard feature range.
3

Log Transformations: Applies x ′ = log(x + 1) to handle right-skewed numerical distributions (e.g., income, house prices) and stabilize variance.

Computer Engineering

DI04000061: ML

4th Semester

167 / 455

Categorical (Qualitative) Data I
Characteristics of Categorical Features
Categorical variables represent qualitative values belonging to discrete classes or categories. Direct arithmetic operations (+, ×) on string labels are undefined.

Nominal Data (Unordered Categories):

Categories have no intrinsic order, rank, or hierarchy.
Examples: Eye Color (Blue, Green, Brown), Country (USA, India, Germany), Marital Status
(Single, Married).
Allowed operations: Equality testing (= vs ̸=). Relational comparisons (>, <) are invalid.
Ordinal Data (Ordered Categories):

Categories follow a clear, natural sequence or ranking, but differences between ranks are
non-uniform or unquantifiable.
Examples: Education Level (High School ¡ Bachelor’s ¡ Master’s ¡ PhD), Customer
Satisfaction (Low ¡ Medium ¡ High).
Allowed operations: Order relations (>, <, =). Exact mathematical differences (X1 − X2 )
are undefined.
Computer Engineering

DI04000061: ML

4th Semester

168 / 455

Categorical Encoding Strategies I
Converting Categories to Machine Learning Input
Algorithms require numeric matrix input X ∈ Rn×d . Categorical features must be encoded appropriately to reflect their underlying structure.

Ordinal Encoding:

Maps each category to a unique integer preserving rank (e.g., Low → 0, Med → 1, High
→ 2).
Warning : Applying this to nominal data introduces false ordinal assumptions (e.g., Red= 0,
Green= 1, Blue= 2 implies Blue > Red).
One-Hot Encoding (OHE):

Expands a nominal feature with K categories into K binary columns (0 or 1).
Dummy Variable Trap: For linear models, drop one category (K − 1 columns) to avoid
PK
perfect multicollinearity ( i=1 Di = 1).
Target / Frequency Encoding:

Replaces high-cardinality categories (e.g., ZIP codes) with target means or class frequencies
to prevent extreme dimensionality explosion.
Computer Engineering

DI04000061: ML

4th Semester

169 / 455

Summary Comparison of Data Types I
Comparative Matrix
Data Type
Continuous
Discrete
Nominal
Ordinal

Key Feature
Real values R
Count integers Z≥0
Unordered sets
Ranked sequence

Valid Ops
+, −, ×, /
+, −, ×, /
=, ̸=
=, ̸=, <, >

Standard Preprocessing
Standardization / Min-Max
Scaling / Binning
One-Hot / Target Encoding
Ordinal Integer Encoding

Key ML Design Takeaways
Tree-based Models (e.g., Decision Trees, Random Forests): Invariant to monotonic numeric scaling; natively handle ordinal features.
Linear & Distance Models (e.g., SVM, Linear Regression, K -NN): Highly sensitive to unscaled numerical features and invalid nominal encoding.
Data Leakage Principle: Compute encoding mappings and scaling parameters (µ, σ) using only the training set fold!

Computer Engineering

DI04000061: ML

4th Semester

170 / 455

Python Implementation: Feature Preprocessing I
i m p o r t p a n d a s a s pd
i m p o r t numpy a s np
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r , MinMaxScaler , OneHotEncoder , O r d i n a l E n c o d e r
# Sample d a t a s e t w i t h mixed d a t a t y p e s
d f = pd . DataFrame ({
’ ag e ’ : [ 2 5 , 4 5 , 3 5 , 5 0 ] ,
# Continuous numerical
’ income ’ : [ 5 0 0 0 0 , 120000 , 85000 , 9 5 0 0 0 ] , # Continuous n u m e r i c a l
’ e d u c a t i o n ’ : [ ’ High S c h o o l ’ , ’PhD ’ , ’ B a c h e l o r ’ , ’ M a s t e r ’ ] , # O r d i n a l
’ c i t y ’ : [ ’NY ’ , ’ P a r i s ’ , ’NY ’ , ’ Tokyo ’ ] # Nominal
})
# 1. Numerical Scaling
scaler = StandardScaler ()
d f [ [ ’ a g e s c a l e d ’ , ’ i n c o m e s c a l e d ’ ] ] = s c a l e r . f i t t r a n s f o r m ( d f [ [ ’ age ’ , ’ i nc om e ’ ] ] )
# 2 . O r d i n a l Encoding ( P r e s e r v i n g Order )
e d u o r d e r = [ [ ’ High S c h o o l ’ , ’ B a c h e l o r ’ , ’ M a s t e r ’ , ’PhD ’ ] ]
o r d e n c = O r d i n a l E n c o d e r ( c a t e g o r i e s=e d u o r d e r )
df [ ’ edu encoded ’ ] = ord enc . f i t t r a n s f o r m ( df [ [ ’ education ’ ] ] )
# 3 . One−Hot E n c o d i n g f o r Nominal Data
ohe = OneHotEncoder ( s p a r s e o u t p u t=F a l s e , d r o p= ’ f i r s t ’ )
c i t y o h e = ohe . f i t t r a n s f o r m ( d f [ [ ’ c i t y ’ ] ] )
p r i n t ( ” S c a l e d & Encoded Data s h a p e : ” , d f . s h a p e )

Computer Engineering

DI04000061: ML

4th Semester

171 / 455

Dimensions of Data Quality in Machine Learning I

The Foundation of Robust Models
Data quality dictates the upper bound of model performance (”Garbage In, Garbage Out”). High-quality training datasets must satisfy six core dimensions before
modeling.
Completeness: Extent to which required values are present (lack of missing entries).
Accuracy: Agreement between recorded data values and true ground-truth entities.
Consistency: Equivalence of attributes across disparate data sources and schemas.
Validity: Conformance to defined domain constraints, ranges, formats, and data types.
Uniqueness: Absence of unintended duplicate records or redundant observations.
Timeliness: Freshness and temporal relevance of the dataset relative to inference time.

Impact of Poor Data Quality
Systematic noise, unhandled missingness, and invalid attributes lead to biased parameter estimation, inflated variance, severe data leakage, and silent failures in
deployment.

Computer Engineering

DI04000061: ML

4th Semester

172 / 455

Taxonomy of Data Defects and Noise I
Common Data Anomalies in ML Pipelines
Real-world datasets suffer from heterogeneous defects originating during collection, logging, and storage.

1

Structural Anomalies:

Heterogeneous representations (e.g., "NY", "New York", "ny").
Inconsistent temporal formats, mixed metric units, and trailing whitespace.
2

Measurement & Sensor Noise:

Gaussian/random noise added to continuous features during acquisition.
Miscalibrated hardware or transmission errors causing corrupted readings.
3

Label Noise:

Incorrect or ambiguous ground-truth labels assigned by human annotators.
Adversarial or systemic biases in historical target label assignment.

Computer Engineering

DI04000061: ML

4th Semester

173 / 455

Missing Data Mechanisms: MCAR, MAR, and MNAR I

Statistical Taxonomy of Missingness
Let Y = (Yobs , Ymis ) represent the dataset and M be the missingness indicator matrix. The mechanism determines the appropriate remediation strategy.
Missing Completely at Random (MCAR):
P(M | Yobs , Ymis ) = P(M)
Missingness is independent of both observed and unobserved data. Unbiased under listwise deletion, but reduces statistical power.
Missing at Random (MAR):
P(M | Yobs , Ymis ) = P(M | Yobs )
Missingness depends systematically on observed data, but not on the missing values themselves. Requires model-based imputation (e.g., K -NN, MICE).
Missing Not at Random (MNAR):
P(M | Yobs , Ymis ) depends on Ymis
Missingness depends directly on the unobserved value. Requires modeling the missingness mechanism or explicit missing indicators (M).

Computer Engineering

DI04000061: ML

4th Semester

174 / 455

Missing Value Remediation Techniques I
Imputation Strategies
Deletion Methods:

Listwise Deletion: Drops rows containing missing values. Safe only under MCAR and low
missingness (< 5%).
Feature Dropping : Removes features exceeding high missingness thresholds (> 40 − 50%).
Univariate Imputation:

Replaces missing entries with global statistics (Mean for Gaussian, Median for skewed
continuous, Mode for categorical).
Multivariate Imputation:

K -Nearest Neighbors (KNN Imputer): Imputes using weighted average of K nearest sample
profiles.
MICE (Iterative Imputer): Models each feature with missing values as a function of all other
features in a round-robin chain.

Computer Engineering

DI04000061: ML

4th Semester

175 / 455

Outlier Detection and Remediation Strategies I
Univariate vs. Multivariate Outlier Detection
Outliers represent extreme anomalies that distort distance computations, loss functions, and gradient steps.

Univariate Statistical Rules:

Z-Score Thresholding : Identifies points with |Z | = x−µ
> 3 (assumes Gaussian
σ
distribution).
Interquartile Range (IQR) Rule: Classifies values outside [Q1 − 1.5 · IQR, Q3 + 1.5 · IQR] as
outliers.
Multivariate Detection:

Isolation Forest: Isolates anomalies by randomly selecting features and splitting values;
outliers require fewer partition splits.
Local Outlier Factor (LOF): Measures local density deviation relative to neighboring
instances.

Remediation Methods
Winsorization (clipping extreme values to 1st and 99th percentiles), Trimming (removing outlier samples), or applying Log/Box-Cox Transformations.
Computer Engineering

DI04000061: ML

4th Semester

176 / 455

Remediation for Class Imbalance I
Handling Severe Class Skew
In tasks such as fraud detection or medical diagnosis, target classes are highly imbalanced (1 : 100 or 1 : 1000).

Resampling Techniques:

Random Undersampling : Reduces majority class instances (risk of losing valuable
information).
SMOTE (Synthetic Minority Over-sampling Technique): Synthesizes new minority samples
along feature-space lines connecting nearest minority neighbors:
xnew = xi + λ(xzi − xi ),

λ ∼ Uniform(0, 1)

Cost-Sensitive Learning:

Adjusts class weights wc inversely proportional to class frequencies during loss calculation:
wc =
Computer Engineering

N
K · Nc

DI04000061: ML

4th Semester

177 / 455

Python Implementation: Data Remediation Pipeline I
i m p o r t numpy a s np
i m p o r t p a n d a s a s pd
from s k l e a r n . e x p e r i m e n t a l i m p o r t e n a b l e i t e r a t i v e i m p u t e r
from s k l e a r n . i m p u t e i m p o r t I t e r a t i v e I m p u t e r , S i m p l e I m p u t e r
from s k l e a r n . e n s e m b l e i m p o r t I s o l a t i o n F o r e s t
from s k l e a r n . p r e p r o c e s s i n g i m p o r t R o b u s t S c a l e r
# 1 . M i s s i n g V a l u e R e m e d i a t i o n v i a MICE ( I t e r a t i v e I m p u t e r )
m i c e i m p u t e r = I t e r a t i v e I m p u t e r ( m a x i t e r =10 , r a n d o m s t a t e =42)
X i m p u t e d = m i c e i m p u t e r . f i t t r a n s f o r m ( X raw )
# 2. Outlier Detection via I s o l a t i o n Forest
i s o f o r e s t = I s o l a t i o n F o r e s t ( c o n t a m i n a t i o n = 0 . 0 5 , r a n d o m s t a t e =42)
o u t l i e r m a s k = i s o f o r e s t . f i t p r e d i c t ( X i m p u t e d ) # −1 f o r o u t l i e r s , 1 f o r

inliers

# F i l t e r out i d e n t i f i e d anomalies
X c l e a n = X i m p u t e d [ o u t l i e r m a s k == 1 ]
y c l e a n = y r a w [ o u t l i e r m a s k == 1 ]
# 3 . Robust S c a l i n g to a t t e n u a t e r e m a i n i n g heavy t a i l s
s c a l e r = RobustScaler ()
X scaled = s c a l e r . f i t t r a n s f o r m ( X clean )
p r i n t ( f ” O r i g i n a l s h a p e : {X raw . s h a p e } , C l e a n e d s h a p e : { X s c a l e d . s h a p e }” )

Computer Engineering

DI04000061: ML

4th Semester

178 / 455

Summary and Best Practices I

Key Takeaways
Data quality assessment must precede feature engineering and model training.
Match the missingness mechanism (MCAR, MAR, MNAR) to the appropriate remediation strategy.
Combine statistical outlier detection with robust scaling techniques to guard distance metrics.

Critical Rules for ML Pipelines
Prevent Data Leakage: Compute all imputation parameters (mean, median, MICE models) and scaling factors strictly on training sets, then apply them
to validation/test sets.
Continuous Monitoring: Track data quality metrics continuously post-deployment to identify data drift and pipeline failures early.

Computer Engineering

DI04000061: ML

4th Semester

179 / 455

Lecture Overview & Learning Objectives I

Course: DI04000061 (Introduction Machine Learning)
Unit 3: Preparing to Model and Evaluation
Topic: 3.4 Data Pre-Processing: Dimensionality reduction, Feature subset selection
Learning Objectives:
1

Understand the Curse of Dimensionality and its computational and mathematical impact on ML models.

2

Master Feature Extraction techniques (PCA, LDA) to project high-dimensional data into lower-dimensional sub-spaces.

3

Evaluate Feature Subset Selection paradigms: Filter, Wrapper, and Embedded methods.

4

Implement dimensionality reduction and feature selection pipelines using Python and scikit-learn.

Computer Engineering

DI04000061: ML

4th Semester

180 / 455

The Curse of Dimensionality I

What is the Curse of Dimensionality?
As the number of features (dimensions d) increases, the volume of the feature space grows exponentially, causing the available data points to become extremely
sparse.
Geometric Sparsity: In high dimensions, almost all volume of a hypercube is concentrated in its corners, and data points lie near the boundary.
Distance Concentration Phenomenon:
lim

d→∞

Distmax − Distmin
Distmin

→0

Euclidean distance loses its discriminative power because all points become nearly equidistant.
Overfitting Risk: High capacity for noise fitting when d ≫ N (d features relative to N samples).
Computational Burden: Matrix inversions and distance calculations scale as O(d 3 ) or O(N · d 2 ).

Computer Engineering

DI04000061: ML

4th Semester

181 / 455

Dimensionality Reduction: Extraction vs. Selection I

Taxonomy of Dimensionality Reduction
Reducing feature space dimension d → k (k ≪ d) can be achieved via two fundamental paradigms:

1. Feature Extraction (Projection)

2. Feature Subset Selection

Creates a new set of k composite features by combining original variables:

Selects a subset of k features directly from the original d attributes:
S ⊂ {x1 , x2 , . . . , xd },

zj = f (x1 , x2 , . . . , xd )

|S| = k

Pros: Captures global structure, compresses information efficiently.

Pros: Retains original domain semantics and interpretability.

Cons: Loss of original feature interpretability.

Cons: Ignores complex non-linear feature combinations.

Examples: PCA, LDA, t-SNE, UMAP.

Examples: Filter, Wrapper, Embedded.

Computer Engineering

DI04000061: ML

4th Semester

182 / 455

Principal Component Analysis (PCA): Theory I

Unsupervised Linear Feature Extraction
PCA finds orthogonal axes (Principal Components) that maximize data variance or, equivalently, minimize reconstruction mean squared error.
1

Mean Centering: Center dataset X ∈ RN×d such that µ = 0.

2

Covariance Matrix Computation:
Σ=

3

T

d×d

X X ∈R

Eigen-Decomposition: Compute eigenvectors vi and eigenvalues λi :
Σvi = λi vi ,

4

1
N−1

λ1 ≥ λ2 ≥ · · · ≥ λd ≥ 0

Projection: Matrix transformation to top k principal directions Wk = [v1 , . . . , vk ]:
N×k

Z = XWk ∈ R

Computer Engineering

DI04000061: ML

4th Semester

183 / 455

PCA: Explained Variance & Singular Value Decomposition I
Explained Variance Ratio (EVR)
The proportion of total dataset variance retained by the i-th principal component:
Pk

λi
,
d
λ
j=1 j

λi
d
λ
j=1 j

Cumulative Variance = Pi=1

EVRi = P

Rule of Thumb: Retain enough components k to preserve 85% − 95% cumulative variance.

SVD Implementation (Numerical Efficiency)
In practice, PCA uses Singular Value Decomposition on centered matrix X :
X = UΣV

T

Right singular vectors V are the principal components (eigenvectors of X T X ).
s2

i .
Singular values si relate to eigenvalues via λi = N−1

Avoids computing explicit d × d covariance matrix Σ.
Computer Engineering

DI04000061: ML

4th Semester

184 / 455

Linear Discriminant Analysis (LDA) I
Supervised Linear Dimension Reduction
Unlike PCA (unsupervised variance maximization), LDA projects data to maximize class separability by finding directions that maximize the ratio of between-class
variance to within-class variance.
Within-Class Scatter Matrix (SW ):
C X
X

SW =

(x − µc )(x − µc )

T

c=1 x∈Cc

Between-Class Scatter Matrix (SB ):
SB =

C
X

T

Nc (µc − µ)(µc − µ)

c=1

Fisher’s Criterion Optimization:
J(w ) = arg max
w

w T SB w
w T SW w

−1

=⇒ SW SB w = λw

Dimensional Constraint: Maximum k = min(d, C − 1) components, where C is the number of classes.

Computer Engineering

DI04000061: ML

4th Semester

185 / 455

Python Example: PCA and LDA with Scikit-Learn I

i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t l o a d i r i s
from s k l e a r n . d e c o m p o s i t i o n i m p o r t PCA
from s k l e a r n . d i s c r i m i n a n t a n a l y s i s i m p o r t L i n e a r D i s c r i m i n a n t A n a l y s i s a s LDA
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r
# 1 . Load and s t a n d a r d s c a l e f e a t u r e s
X , y = l o a d i r i s ( r e t u r n X y=True )
X s c a l e d = S t a n d a r d S c a l e r ( ) . f i t t r a n s f o r m (X)
# 2 . PCA : U n s u p e r v i s e d ( T a r g e t y i s NOT u s e d )
pca = PCA( n c o m p o n e n t s =2)
X pca = pca . f i t t r a n s f o r m ( X s c a l e d )
p r i n t ( f ”PCA E x p l a i n e d V a r i a n c e R a t i o : {pca . e x p l a i n e d v a r i a n c e r a t i o }” )
# Output : [ 0 . 7 2 9 6 2 1 4 5 0 . 2 2 8 5 0 7 6 2 ] −> P r e s e r v e s 95.8% v a r i a n c e
# 3 . LDA : S u p e r v i s e d ( T a r g e t y I S r e q u i r e d )
l d a = LDA( n c o m p o n e n t s =2)
X lda = lda . f i t t r a n s f o r m ( X scaled , y )
p r i n t ( f ”LDA E x p l a i n e d V a r i a n c e R a t i o : { l d a . e x p l a i n e d v a r i a n c e r a t i o }” )
# Output : [ 0 . 9 9 1 2 1 2 6
0 . 0 0 8 7 8 7 4 ] −> M a x i m i z e s c l a s s s e p a r a t i o n

Computer Engineering

DI04000061: ML

4th Semester

186 / 455

Feature Subset Selection: Filter Methods I
Core Principle
Filter methods evaluate individual features based on statistical properties independent of any machine learning model.
Variance Thresholding: Removes features with variance below a specified threshold (σ 2 < ϵ). Extremely useful for removing constant or near-constant
features.
Pearson Correlation Coefficient (r ):
Cov(Xj , Y )
r (Xj , Y ) =
σ X σY
j

Filters features highly correlated with target Y while eliminating collinear features (r (Xi , Xj ) > 0.9).
Statistical Tests:

ANOVA F -test: Measures linear dependency between numerical features and categorical
target.
Chi-Square (χ2 ): Evaluates independence between categorical features and categorical
target.
Mutual Information (MI): Non-parametric score capturing non-linear relationships:
ZZ
p(x, y )
I (X ; Y ) =
p(x, y ) log
dxdy
p(x)p(y )
Computer Engineering

DI04000061: ML

4th Semester

187 / 455

Feature Subset Selection: Wrapper Methods I
Core Principle
Wrapper methods use a specific ML algorithm as a black-box evaluator to search the space of feature subsets based on predictive performance.

1

Sequential Forward Selection (SFS):

Starts with empty feature set ∅. Iteratively adds feature that maximizes model score.
2

Sequential Backward Elimination (SBE):

Starts with full feature set F . Iteratively removes feature whose removal causes smallest drop
in performance.
3

Recursive Feature Elimination (RFE):

Trains model on current features, ranks feature importance (e.g. coefficients wj or tree
importances), prunes lowest rank features, and repeats.

Trade-offs
Pros: Evaluates feature interactions; tailored to specific model.
Cons: High computational complexity O(2d ); prone to overfitting.

Computer Engineering

DI04000061: ML

4th Semester

188 / 455

Feature Subset Selection: Embedded Methods I

Core Principle
Embedded methods perform feature selection naturally as part of the model learning/optimization process.
L1 Regularization (Lasso Regression): Adds penalty term proportional to absolute sum of weights:

min
w

1
2N

2

∥y − Xw ∥2 + α∥w ∥1

Due to geometry of L1 norm diamond constraint, coefficients of irrelevant features are driven to exactly zero, performing automated variable selection.
Tree-Based Feature Importance:

Mean Decrease in Impurity (MDI): Measures cumulative reduction of Gini impurity or
Entropy brought by splitting on feature Xj across all trees in Random Forest/GBDT.
Thresholding feature importance scores selects informative features.

Computer Engineering

DI04000061: ML

4th Semester

189 / 455

Python Example: Feature Subset Selection I

from s k l e a r n . d a t a s e t s i m p o r t m a k e c l a s s i f i c a t i o n
from s k l e a r n . f e a t u r e s e l e c t i o n i m p o r t S e l e c t K B e s t , f c l a s s i f , RFE
from s k l e a r n . l i n e a r m o d e l i m p o r t L o g i s t i c R e g r e s s i o n , LassoCV
X , y = m a k e c l a s s i f i c a t i o n ( n s a m p l e s =200 , n f e a t u r e s =20 , n i n f o r m a t i v e =5 , r a n d o m s t a t e =42)
# 1 . F i l t e r Method : S e l e c t t o p 5 f e a t u r e s u s i n g ANOVA F−t e s t
s e l e c t o r f i l t e r = S e l e c t K B e s t ( s c o r e f u n c= f c l a s s i f , k=5)
X f i l t e r = s e l e c t o r f i l t e r . f i t t r a n s f o r m (X , y )
# 2 . Wrapper Method : R e c u r s i v e F e a t u r e E l i m i n a t i o n ( RFE )
model = L o g i s t i c R e g r e s s i o n ( )
r f e = RFE ( e s t i m a t o r=model , n f e a t u r e s t o s e l e c t =5)
X r f e = r f e . f i t t r a n s f o r m (X , y )
# 3 . Embedded Method : L a s s o ( L1 r e g u l a r i z a t i o n )
l a s s o = LassoCV ( c v =5). f i t (X , y )
s e l e c t e d m a s k = ( l a s s o . c o e f != 0 )
p r i n t ( ”Non−z e r o c o e f f i c i e n t s c o u n t : ” , s e l e c t e d m a s k . sum ( ) )

Computer Engineering

DI04000061: ML

4th Semester

190 / 455

Comparison of Dimensionality Reduction Methods I

Method
PCA
LDA
Filter
Wrapper
Embedded

Type
Extraction
Extraction
Selection
Selection
Selection

Supervised?
No
Yes
Varies
Yes
Yes

Preserves Semantics?
No (Linear combinations)
No (Linear combinations)
Yes (Original features)
Yes (Original features)
Yes (Original features)

Computational Cost
Low (O(d 3 ) or SVD)
Low (O(d 3 ))
Very Low (O(d))
High (O(k · Train))
Medium (Integrated)

Practical Recommendation Matrix
High d, linear model, tabular interpretability required → Lasso / Filter (MI).
Multi-class classification, low-dim representation → LDA.
Multicollinearity reduction, dense continuous features → PCA.
Maximum predictive power, moderate d ≤ 50 → RFE (Wrapper).

Computer Engineering

DI04000061: ML

4th Semester

191 / 455

Predictive vs. Descriptive Modeling Overview I

Taxonomy of Machine Learning Models
Model selection begins by identifying whether the primary operational objective is prediction of unobserved outcomes or description of underlying data structures.

Predictive Modeling

Descriptive Modeling

Objective: Forecast target label Y given feature vector X .

Objective: Discover patterns, groupings, or representations in X .

Supervision: Requires labelled dataset D = {(xi , yi )}N
i=1 .

Supervision: Operates on unlabelled data D = {xi }N
i=1 .

Focus: Generalization performance on unseen data.

Focus: Interpretability, actionable insights, structure.

Examples: Spam detection, price forecasting, disease diagnosis.

Examples: Customer segmentation, topic modeling, anomaly
detection.

Computer Engineering

DI04000061: ML

4th Semester

192 / 455

Mathematical Foundations of Predictive Modeling I
Formal Problem Setup
Given an input space X ⊂ Rd and output space Y, learn a mapping function f : X → Y minimizing expected risk:
Z
R(f ) = E(X ,Y )∼P [L(Y , f (X ))] =

L(y , f (x)) dP(x, y )
X ×Y

where L(y , ŷ ) is a task-specific loss function (e.g., Mean Squared Error or Cross-Entropy).

Key Characteristics
Empirical Risk Minimization (ERM): Approximates R(f ) using training dataset Dtrain :

fˆ = arg min

X
1 N

f ∈H N

L(yi , f (xi )) + λΩ(f )

i=1

Out-of-Sample Validation: Evaluated rigorously on holdout/test sets to detect overfitting.
Trade-off: Often trades mathematical interpretability for superior predictive accuracy (e.g., Deep Learning, Gradient Boosting).

Computer Engineering

DI04000061: ML

4th Semester

193 / 455

Structure Discovery in Descriptive Modeling I

Formal Problem Setup
Given unlabelled sample vectors {x1 , x2 , . . . , xN } ⊂ Rd , fit a model g (X ) to estimate joint probability density p(X ), project dimensions, or partition space into
K latent clusters:
K
[
C = {C1 , C2 , . . . , CK }, where
Ck = D and Ci ∩ Cj = ∅ (∀i ̸= j)
k=1

Primary Tasks & Objectives
Clustering: Partitioning instances based on similarity metrics (e.g., K -Means, Hierarchical, DBSCAN).
Dimensionality Reduction: Projecting high-dimensional data onto lower-dimensional manifolds while preserving variance or distance (e.g., PCA, t-SNE,
UMAP).
Association Rule Mining: Uncovering co-occurrence relationships (A =⇒ B) evaluated via Support, Confidence, and Lift.
Density Estimation: Modeling p(x) directly (e.g., Gaussian Mixture Models, Kernel Density Estimation).

Computer Engineering

DI04000061: ML

4th Semester

194 / 455

Predictive vs. Descriptive Modeling Comparison I

Side-by-Side Comparison
Dimension
Primary Goal
Target Label
Validation

Predictive Modeling
Predict target outcome Y
Mandatory (yi ∈ Y)
Ground-truth metrics (R 2 , F1, ROC-AUC)

Overfitting Risk

High risk; guarded by test splits & regularization
Linear/Logistic Reg., Random Forest, XGBoost, Neural Nets

Algorithms

Computer Engineering

DI04000061: ML

Descriptive Modeling
Describe latent patterns in X
Absent / Unsupervised
Internal
metrics
(Silhouette,
Inertia,
AIC/BIC)
Evaluated via stability & domain plausibility
K -Means, PCA, GMM, Apriori, Hierarchical
Clustering

4th Semester

195 / 455

Model Selection Framework & Complexity Penalties I

Occam’s Razor in Model Selection
When multiple candidate models perform similarly, prefer the simplest model with fewer parameters to maximize interpretability and reduce generalization error.

Akaike Information Criterion (AIC)

Bayesian Information Criterion (BIC)

Penalizes model complexity based on parameter count k and log-likelihood L̂:

Applies a stronger penalty for large sample size N:
BIC = k ln(N) − 2 ln(L̂)

AIC = 2k − 2 ln(L̂)

Lower AIC indicates a superior balance between fit and parsimony.

Computer Engineering

DI04000061: ML

Heavily penalizes complex models as N increases.

4th Semester

196 / 455

Hybrid Paradigms: Synergy of Both Approaches I
Integrated Workflows in Practical Data Science
In real-world engineering, predictive and descriptive modeling are frequently combined to form hybrid pipelines:

1

Descriptive Preprocessing for Predictive Tasks:

Using Principal Component Analysis (PCA) to reduce multicollinearity before training a
regression model.
Applying K-Means Clustering to generate cluster membership IDs as categorical features for
a gradient boosted tree.
2

Predictive Models for Descriptive Insights:

Inspecting Feature Importance or SHAP (SHapley Additive exPlanations) values from a
random forest to understand physical relationships.
Autoencoders (Predictive reconstruction loss) used for Anomaly Detection (Descriptive
structure analysis).

Computer Engineering

DI04000061: ML

4th Semester

197 / 455

Python Implementation: Predictive vs. Descriptive Pipelines I
i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t m a k e c l a s s i f i c a t i o n
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
from s k l e a r n . e n s e m b l e i m p o r t R a n d o m F o r e s t C l a s s i f i e r
from s k l e a r n . c l u s t e r i m p o r t KMeans
from s k l e a r n . m e t r i c s i m p o r t a c c u r a c y s c o r e , s i l h o u e t t e s c o r e
# Generate s y n t h e t i c dataset
X , y = m a k e c l a s s i f i c a t i o n ( n s a m p l e s =500 , n f e a t u r e s =10 ,
n i n f o r m a t i v e =5 , r a n d o m s t a t e =42)
# −−− 1 . P r e d i c t i v e Model ( S u p e r v i s e d C l a s s i f i c a t i o n ) −−−
X t r , X t e , y t r , y t e = t r a i n t e s t s p l i t (X , y , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42)
p r e d m o d e l = R a n d o m F o r e s t C l a s s i f i e r ( n e s t i m a t o r s =50 , r a n d o m s t a t e =42)
pred model . f i t ( X tr , y t r )
y pred = pred model . p r e d i c t ( X te )
p r i n t ( f ” P r e d i c t i v e Model A c c u r a c y : { a c c u r a c y s c o r e ( y t e , y p r e d ) : . 4 f }” )
# −−− 2 . D e s c r i p t i v e Model ( U n s u p e r v i s e d C l u s t e r i n g ) −−−
d e s c m o d e l = KMeans ( n c l u s t e r s =2 , r a n d o m s t a t e =42 , n i n i t =10)
c l u s t e r l a b e l s = d e s c m o d e l . f i t p r e d i c t (X)
s i l s c o r e = s i l h o u e t t e s c o r e (X , c l u s t e r l a b e l s )
p r i n t ( f ” D e s c r i p t i v e S i l h o u e t t e S c o r e : { s i l s c o r e : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

198 / 455

Summary: Model Selection Guidelines I

Decision Matrix for Model Selection
Choose Predictive Modeling when:

Ground-truth labels Y are available and target accuracy is paramount.
The primary goal is automated decision-making on incoming unseen instances.
Choose Descriptive Modeling when:

Data is unlabelled and objective is exploratory analysis or pattern discovery.
Understanding underlying domain relationships takes precedence over precise prediction.

Key Takeaway
Model selection should be dictated by the business/scientific question, data availability, and the required balance between interpretability and generalization
performance.

Computer Engineering

DI04000061: ML

4th Semester

199 / 455

Supervised Model Training Workflow I

The Fundamental Goal
In supervised learning, the objective is to learn a mapping function f : X → Y using empirical data such that it generalizes accurately to unseen samples.

The Pitfall of Resubstitution Error
Evaluating a model on the exact same data used to train it leads to optimistic bias and fails to detect overfitting.
Training Error (Etrain ): Empirical loss computed on training samples.
Generalization Error (Etest ): Expected loss on new, unseen samples drawn from the true underlying distribution P(X , Y ).
Data Partitioning: Disjoint splitting of datasets to simulate evaluation on unseen data.

Computer Engineering

DI04000061: ML

4th Semester

200 / 455

The Holdout Method I

Data Partitioning Strategy
The dataset D is randomly partitioned into two or three non-overlapping subsets:
Training Set (Dtrain ): Used to learn parameters θ (typically 60% − 80%).
Validation Set (Dval ): Used for hyperparameter tuning and model selection (10% − 20%).
Test Set (Dtest ): Reserved strictly for final, unbiased evaluation (10% − 20%).

Advantages

Disadvantages

Computationally efficient (O(1) training runs).

High variance depending on split.

Simple and intuitive implementation.

Reduces data available for training.

Computer Engineering

DI04000061: ML

4th Semester

201 / 455

Holdout Method in Python I
Implementation using Scikit-Learn
Demonstrating train-validation-test split with target stratification:

from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
from s k l e a r n . d a t a s e t s i m p o r t l o a d i r i s
# Load d a t a s e t
X , y = l o a d i r i s ( r e t u r n X y=True )
# F i r s t s p l i t : T r a i n+V a l (80%) and T e s t (20%)
X temp , X t e s t , y temp , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 0 , r a n d o m s t a t e =42 , s t r a t i f y =y
)
# Second s p l i t : T r a i n (60% t o t a l ) and V a l (20% t o t a l )
X train , X val , y t r a i n , y v a l = t r a i n t e s t s p l i t (
X temp , y temp , t e s t s i z e = 0 . 2 5 , r a n d o m s t a t e =42 , s t r a t i f y =y t em p
)
p r i n t ( f ” Train : { X t r a i n . shape [ 0 ] } , Val : {X val . shape [ 0 ] } , Test : { X t e s t . shape [ 0 ] } ” )

Computer Engineering

DI04000061: ML

4th Semester

202 / 455

K-Fold Cross-Validation I
Core Mechanism
K -Fold Cross-Validation mitigates split dependency by evaluating model performance across K disjoint validation folds.
1

Randomly partition dataset D into K equal-sized subsets: D1 , D2 , . . . , DK .

2

For each fold k ∈ {1, 2, . . . , K }:

Train model on D \ Dk .
Compute validation error metric Ek on fold Dk .
3

Calculate aggregated metric score:
Ē =

K
1 X

K k=1

Ek

Key Benefit
Every data point is used exactly once for validation and K − 1 times for model training.

Computer Engineering

DI04000061: ML

4th Semester

203 / 455

Variations of Cross-Validation I

Stratified K -Fold Cross-Validation
Preserves target class distribution proportions across all K folds.
Essential for imbalanced classification to avoid fold bias or empty class representation.

Leave-One-Out Cross-Validation (LOOCV)
Extreme case of K -Fold CV where K = N (N is dataset size).
In each fold, N − 1 samples are used for training, 1 sample for validation.
Pros: Low estimation bias; deterministic (no random seed variance).
Cons: High computational overhead (N model fits); high error variance.

Computer Engineering

DI04000061: ML

4th Semester

204 / 455

Cross-Validation in Python I
Scikit-Learn Implementation
Comparing standard K -Fold and Stratified K -Fold cross-validation:

i m p o r t numpy a s np
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t KFold , S t r a t i f i e d K F o l d ,
from s k l e a r n . l i n e a r m o d e l i m p o r t L o g i s t i c R e g r e s s i o n
from s k l e a r n . d a t a s e t s i m p o r t l o a d b r e a s t c a n c e r

cross val score

X , y = l o a d b r e a s t c a n c e r ( r e t u r n X y=True )
c l f = L o g i s t i c R e g r e s s i o n ( m a x i t e r =10000)
# 5−F o l d C r o s s V a l i d a t i o n
k f = KFold ( n s p l i t s =5 , s h u f f l e =True , r a n d o m s t a t e =42)
s c o r e s k f = c r o s s v a l s c o r e ( c l f , X , y , c v=k f , s c o r i n g= ’ a c c u r a c y ’ )
# 5−F o l d S t r a t i f i e d C r o s s V a l i d a t i o n
s k f = S t r a t i f i e d K F o l d ( n s p l i t s =5 , s h u f f l e =True , r a n d o m s t a t e =42)
s c o r e s s k f = c r o s s v a l s c o r e ( c l f , X , y , c v=s k f , s c o r i n g= ’ a c c u r a c y ’ )
p r i n t ( f ”K−F o l d A c c u r a c y : {np . mean ( s c o r e s k f ) : . 4 f } +/− {np . s t d ( s c o r e s k f ) : . 4 f }” )
p r i n t ( f ” S t r a t i f i e d K−F o l d : {np . mean ( s c o r e s s k f ) : . 4 f } +/− {np . s t d ( s c o r e s s k f ) : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

205 / 455

Comparison & Practical Guidelines I
Trade-off Summary
Method
Holdout Split
K -Fold CV (K = 5, 10)
Stratified K -Fold
LOOCV (K = N)

Compute Cost
Low (O(1))
Moderate (O(K ))
Moderate (O(K ))
High (O(N))

Bias
High
Low
Low
Very Low

Variance
High
Moderate
Low-Moderate
High

Best Practices
Use Holdout for massive datasets (e.g., Deep Learning) where retraining is expensive.
Standardize on Stratified 5- or 10-Fold CV for small-to-medium tabular datasets.
Prevent Data Leakage: Perform preprocessing (e.g., feature scaling, imputation) strictly inside each cross-validation fold using pipelines.

Computer Engineering

DI04000061: ML

4th Semester

206 / 455

Introduction to Performance Evaluation I

Why Simple Accuracy is Often Not Enough
Classification accuracy measures the proportion of correct predictions:

Accuracy =

Number of Correct Predictions
Total Predictions

Class Imbalance Problem: In fraud detection (99% legitimate, 1% fraud), a naive model predicting all transactions as ”legitimate” achieves 99% accuracy
while failing completely.
Asymmetric Misclassification Costs: Failing to detect a disease (False Negative) is often far more critical than a false alarm (False Positive).

The Solution
A Confusion Matrix breaks down predictions into correct and incorrect classifications for each specific class, revealing hidden performance flaws.

Computer Engineering

DI04000061: ML

4th Semester

207 / 455

Structure of a Binary Confusion Matrix I
A 2 × 2 table comparing Actual Ground Truth against Predicted Classes:

2*Actual Class

Positive (1)
Negative (0)

Predicted Class
Positive (1)
Negative (0)
True Positive (TP)
False Negative (FN)
(Hit / Correct)
(Type II Error / Miss)
False Positive (FP)
True Negative (TN)
(Type I Error / False Alarm)
(Correct Rejection)

Actual Condition: Rows represent true labels.
Predicted Condition: Columns represent model output predictions.
Total Samples: N = TP + TN + FP + FN.

Computer Engineering

DI04000061: ML

4th Semester

208 / 455

Understanding the Four Core Outcomes I
True Positive (TP):

Actual = Positive, Predicted = Positive.
Example: A sick patient correctly identified as having the illness.
True Negative (TN):

Actual = Negative, Predicted = Negative.
Example: A healthy individual correctly identified as healthy.
False Positive (FP) — Type I Error:

Actual = Negative, Predicted = Positive.
Example: Healthy patient falsely diagnosed with illness (False Alarm).
False Negative (FN) — Type II Error:

Actual = Positive, Predicted = Negative.
Example: Sick patient incorrectly declared healthy (Missed Detection).

Computer Engineering

DI04000061: ML

4th Semester

209 / 455

Basic Derived Metrics: Accuracy & Error Rate I
Classification Accuracy
Measures the overall fraction of correct predictions across all classes:

Accuracy =

TP + TN
TP + TN + FP + FN

Classification Error Rate (Misclassification Rate)
Measures the overall proportion of incorrect predictions:

Error Rate =

FP + FN
TP + TN + FP + FN

= 1 − Accuracy

Caution
Accuracy is reliable ONLY when class distributions are balanced and error costs are symmetric across classes.

Computer Engineering

DI04000061: ML

4th Semester

210 / 455

Precision and Recall (Sensitivity) I

Precision (Positive Predictive Value)

Recall (Sensitivity / True Positive Rate)

Out of all instances predicted as Positive, how many were actually Positive?

Out of all actual Positive instances, how many did the model correctly catch?

Precision =

TP

Recall =

TP + FP

TP
TP + FN

Goal: Minimize False Positives.

Goal: Minimize False Negatives.

Critical in spam filtering, search results.

Critical in medical diagnosis, fraud detection.

Computer Engineering

DI04000061: ML

4th Semester

211 / 455

Specificity and F1-Score I
Specificity (True Negative Rate)
Out of all actual Negative instances, how many were correctly identified as Negative?

Specificity =

TN
TN + FP

FP .
False Positive Rate (FPR) = 1 − Specificity = TN+FP

F1-Score (Harmonic Mean)
Combines Precision and Recall into a single metric:

F1 = 2 ·

Precision · Recall
Precision + Recall

=

2 · TP
2 · TP + FP + FN

Penalizes extreme imbalances between Precision and Recall.
Essential metric for evaluating imbalanced datasets.

Computer Engineering

DI04000061: ML

4th Semester

212 / 455

Multi-Class Confusion Matrix Extension I
For a classification problem with K > 2 classes (e.g., K = 3: Class A, B, C):

3*Actual Class

Class A
Class B
Class C

Class A
CAA (TPA )
CBA
CCA

Predicted Class
Class B
CAB
CBB (TPB )
CCB

Class C
CAC
CBC
CCC (TPC )

Diagonal Elements (Cii ): Correct classifications for each class.
Off-Diagonal Elements (Cij , i ̸= j): Misclassifications where actual class i is predicted as class j.
Per-Class Evaluation: Precision and Recall can be computed using One-vs-Rest strategy.

Computer Engineering

DI04000061: ML

4th Semester

213 / 455

Python Implementation: Confusion Matrix I
import m a t p l o t l i b . p y p l o t as p l t
from s k l e a r n . m e t r i c s i m p o r t (
confusion matrix , ConfusionMatrixDisplay ,
classification report , accuracy score
)
# Ground t r u t h l a b e l s and p r e d i c t i o n s
y true = [1 , 0 , 1 , 1 , 0 , 1 , 0 , 0 , 1 , 0]
y pred = [1 , 0 , 1 , 0 , 0 , 1 , 1 , 0 , 1 , 0]
# Compute c o n f u s i o n m a t r i x
cm = c o n f u s i o n m a t r i x ( y t r u e , y p r e d )
tn , f p , f n , t p = cm . r a v e l ( )
p r i n t ( f ”TP : { t p } , TN : { t n } , FP : { f p } , FN : { f n }” )
# D e t a i l e d m e t r i c s summary r e p o r t
p r i n t ( c l a s s i f i c a t i o n r e p o r t ( y t r u e , y p r e d , t a r g e t n a m e s =[ ’ Neg ’ , ’ Pos ’ ] ) )
# Visual p l o t of the Confusion Matrix
d i s p = C o n f u s i o n M a t r i x D i s p l a y ( c o n f u s i o n m a t r i x=cm ,
d i s p . p l o t ( cmap=p l t . cm . B l u e s )
p l t . t i t l e (” Confusion Matrix Evaluation ”)

Computer Engineering

d i s p l a y l a b e l s =[ ’ Neg ’ , ’ Pos ’ ] )

DI04000061: ML

4th Semester

214 / 455

Summary: Choosing the Right Metric I

Scenario / Objective
Balanced classes, equal costs
Minimize False Alarms (FP)
Minimize Missed Detections (FN)
Imbalanced datasets

Key Metric
Accuracy
Precision
Recall
F1-Score

Focus Area
Overall Correctness
Quality of Positives
Quantity of Positives
Harmonic Balance of P & R

Key Takeaways
Never rely on Accuracy alone on imbalanced datasets.
Always analyze the Confusion Matrix to understand specific error distributions.
Select evaluation metrics based on the real-world business/domain cost of errors.

Computer Engineering

DI04000061: ML

4th Semester

215 / 455

Improving Model Performance: Overview I

Why Isn’t the Baseline Model Enough?
Initial baseline models often suffer from underfitting (high bias), overfitting (high variance), or data-related issues such as class imbalance and uninformative
features.

Model-Centric Strategies

Data-Centric Strategies

Hyperparameter Tuning: Grid Search, Random Search, Bayesian
Optimization.
Regularization: L1 /L2 penalties, Dropout, Early Stopping.

Class Imbalance: SMOTE, undersampling, cost-sensitive learning.
Feature Engineering: Selection, transformations, interactions.
Data Cleaning: Outlier handling, missing data imputations.

Ensembling: Bagging, Boosting, Stacking.

Computer Engineering

DI04000061: ML

4th Semester

216 / 455

Hyperparameter Optimization I

Optimization Strategies
Q
Grid Search Exhaustively evaluates all parameter combinations over a defined grid. Guarantees finding the best grid point but scales poorly: O( ki ).
Random Search Samples combinations randomly from specified statistical distributions. Highly efficient in high-dimensional hyperparameter spaces.
Bayesian Optimization Constructs a probabilistic surrogate model (e.g., Gaussian Process) to balance exploration and exploitation of the search space.

Validation Leakage Warning
Over-tuning hyperparameters on the validation set can cause validation set overfitting. Always preserve a separate test set for final evaluation.

Computer Engineering

DI04000061: ML

4th Semester

217 / 455

Code: Hyperparameter Tuning in Scikit-Learn I
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t RandomizedSearchCV
from s k l e a r n . e n s e m b l e i m p o r t R a n d o m F o r e s t C l a s s i f i e r
from s c i p y . s t a t s i m p o r t r a n d i n t
# 1. Define hyperparameter d i s t r i b u t i o n search space
param dist = {
’ n e s t i m a t o r s ’ : r and int (50 , 300) ,
’ m a x d e p t h ’ : [ None , 1 0 , 2 0 , 3 0 ] ,
’ m i n s a m p l e s s p l i t ’ : r a n d i n t (2 , 11)
}
# 2 . C o n f i g u r e Randomized S e a r c h C r o s s−V a l i d a t i o n
r f = R a n d o m F o r e s t C l a s s i f i e r ( r a n d o m s t a t e =42)
r a n d s e a r c h = RandomizedSearchCV (
e s t i m a t o r=r f ,
p a r a m d i s t r i b u t i o n s=p a r a m d i s t ,
n i t e r =20 , c v =5 , s c o r i n g= ’ f 1 m a c r o ’ , n j o b s=−1
)
rand search . f i t ( X train , y t r a i n )
p r i n t ( f ” B e s t H y p e r p a r a m e t e r s : { r a n d s e a r c h . b e s t p a r a m s }” )
p r i n t ( f ” B e s t V a l i d a t i o n S c o r e : { r a n d s e a r c h . b e s t s c o r e : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

218 / 455

Handling Class Imbalance I

The Imbalance Challenge
When target classes are skewed (e.g., 99% negative vs. 1% positive), standard loss functions default to predicting majority classes, yielding high accuracy but
near-zero recall.

Resampling Strategies

Algorithmic Adjustments

Oversampling: Duplicates minority instances (risk of overfitting).
Undersampling: Discards majority instances (loss of information).

Cost-Sensitive Learning: Scale loss via inverse class frequencies
N .
wc = K ·N

SMOTE: Synthetic Minority Over-sampling Technique via k-NN
interpolation.

Threshold Tuning: Shift decision threshold off 0.5 using Precision-Recall
curves.

Computer Engineering

c

DI04000061: ML

4th Semester

219 / 455

Code: Addressing Class Imbalance I

from i m b l e a r n . o v e r s a m p l i n g i m p o r t SMOTE
from s k l e a r n . l i n e a r m o d e l i m p o r t L o g i s t i c R e g r e s s i o n
from s k l e a r n . m e t r i c s i m p o r t c l a s s i f i c a t i o n r e p o r t
# Method A : Cost−S e n s i t i v e L e a r n i n g v i a C l a s s W e i g h t s
m o d e l w e i g h t e d = L o g i s t i c R e g r e s s i o n ( c l a s s w e i g h t= ’ b a l a n c e d ’ )
model weighted . f i t ( X train , y t r a i n )
# Method B : SMOTE R e s a m p l i n g ( f i t o n l y on t r a i n i n g s e t ! )
smote = SMOTE( r a n d o m s t a t e =42)
X r e s , y r e s = smote . f i t r e s a m p l e ( X t r a i n , y t r a i n )
model smote = L o g i s t i c R e g r e s s i o n ( )
model smote . f i t ( X res , y r e s )
# E v a l u a t i o n on p r i s t i n e v a l i d a t i o n s e t
y p r e d = model smote . p r e d i c t ( X val )
print ( c l a s s i f i c a t i o n r e p o r t ( y val , y pred ))

Computer Engineering

DI04000061: ML

4th Semester

220 / 455

Ensemble Learning Methods I

Combining Base Learners for Superior Performance
1

Bagging (Bootstrap Aggregating): Trains independent parallel estimators on bootstrap samples and averages predictions. Primary goal: Reduce
Variance. Example: Random Forest.

2

Boosting: Sequential ensemble where subsequent models fit residual errors of prior estimators. Primary goal: Reduce Bias. Examples: AdaBoost,
Gradient Boosting, XGBoost, LightGBM.

3

Stacking: Trains heterogeneous base models and feeds their predictions into a meta-learner.

Computer Engineering

DI04000061: ML

4th Semester

221 / 455

Code: Boosting and Stacking Ensembles I

from
from
from
from

s k l e a r n . ensemble import G r a d i e n t B o o s t i n g C l a s s i f i e r ,
s k l e a r n . l i n e a r m o d e l import L o g i s t i c R e g r e s s i o n
s k l e a r n . t r e e import D e c i s i o n T r e e C l a s s i f i e r
s k l e a r n . svm i m p o r t SVC

StackingClassifier

# 1. Gradient Boosting C l a s s i f i e r
g b m o d e l = G r a d i e n t B o o s t i n g C l a s s i f i e r ( n e s t i m a t o r s =100 , l e a r n i n g r a t e =0.1)
gb model . f i t ( X t r a i n , y t r a i n )
# 2 . Heterogeneous S t a c k i n g Ensemble
base estimators = [
( ’ dt ’ , D e c i s i o n T r e e C l a s s i f i e r ( max depth =4)) ,
( ’ s v c ’ , SVC( p r o b a b i l i t y=True ) )
]
stack model = S t a c k i n g C l a s s i f i e r (
e s t i m a t o r s=b a s e e s t i m a t o r s ,
f i n a l e s t i m a t o r =L o g i s t i c R e g r e s s i o n ( )
)
stack model . f i t ( X train , y t r a i n )

Computer Engineering

DI04000061: ML

4th Semester

222 / 455

Feature Selection & Refinement I

Feature Selection Taxonomy
Filter Methods Evaluate feature subsets using statistical properties independent of the model (χ2 , Mutual Information, ANOVA). Fast and scalable.
Wrapper Methods Use predictive performance of a target model to evaluate feature subsets (Recursive Feature Elimination - RFE). Computationally intensive.
Embedded Methods Perform feature selection naturally during training (L1 Lasso regularization, Tree feature importances).

Feature Transformation Best Practices
Log/Power transformations for highly skewed feature distributions.
Engineering domain-specific interactions and ratios (Xi · Xj , Xi /Xj ).

Computer Engineering

DI04000061: ML

4th Semester

223 / 455

Summary: Systematic Performance Improvement Workflow I

Iterative Model Improvement Recipe
1

Establish Baseline: Simple model + default parameters + task-appropriate metric (F1, ROC-AUC).

2

Diagnose Error Patterns: Inspect confusion matrix, learning curves, and residuals (bias vs. variance).

3

Data Feature Engineering: Address imbalance (SMOTE/weights), handle outliers, select relevant features.

4

Explore Models: Benchmark diverse model families (Linear, Tree-based, Neural).

5

Hyperparameter Tuning: Apply Random/Bayesian Search to top-performing models.

6

Ensemble Predictions: Combine top models via Stacking/Blending for maximum predictive performance.

Computer Engineering

DI04000061: ML

4th Semester

224 / 455

Comprehensive Framework for Model Evaluation I
Beyond Classification Accuracy
While accuracy is intuitive, it can be deeply misleading in real-world applications (e.g., imbalanced datasets where 99% of samples belong to the majority class).
Precision: Measure of exactness — proportion of positive identifications that were correct:

Precision =

TP
TP + FP

Recall (Sensitivity): Measure of completeness — proportion of actual positives correctly identified:

Recall =

TP
TP + FN

F1 -Score: Harmonic mean of precision and recall, balancing both metrics:

F1 = 2 ×

Computer Engineering

Precision × Recall
Precision + Recall

DI04000061: ML

=

2TP
2TP + FP + FN

4th Semester

225 / 455

ROC and Precision-Recall Curves I

Receiver Operating Characteristic (ROC) Curve
Plots the True Positive Rate (TPR) against the False Positive Rate (FPR) across varying classification thresholds:

TPR =

TP
TP + FN

,

FPR =

FP
FP + TN

Area Under Curve (AUC-ROC): Quantifies overall ranking performance (0.5 = random guess, 1.0 = perfect classifier).

Precision-Recall (PR) Curve
Recommended for highly skewed/imbalanced datasets where negative samples dominate:
Focuses strictly on the positive class without being inflated by a large number of True Negatives.
Area Under PR Curve (PR-AUC) evaluates performance under heavy class imbalance.

Computer Engineering

DI04000061: ML

4th Semester

226 / 455

Model Diagnostics: Bias vs. Variance I
The Bias-Variance Decomposition
Total expected generalization error decomposes into three fundamental components:
2

Expected Error = Bias + Variance + Irreducible Error

High Bias (Underfitting):

Model is overly simple and fails to capture underlying patterns.
Symptoms: High training error and high validation error.
Remedies: Add more features, decrease regularization, use a more complex model.
High Variance (Overfitting):

Model memorizes training noise and fails to generalize.
Symptoms: Low training error but high validation error.
Remedies: Gather more data, select feature subset, increase regularization.

Computer Engineering

DI04000061: ML

4th Semester

227 / 455

Hyperparameter Optimization Strategies I
Automated Model Tuning
Hyperparameters govern algorithm behavior and model capacity. They are set prior to training rather than learned from data.

1

Grid Search CV:

Exhaustively evaluates all Cartesian combinations of specified hyperparameter values.
Guaranteed to find the best configuration in grid, but computationally expensive.
2

Randomized Search CV:

Samples a fixed number of parameter combinations randomly from specified distributions.
Drastically reduces runtime while exploring high-dimensional search spaces effectively.
3

Bayesian Optimization:

Uses probabilistic surrogate models (e.g., Gaussian Processes) to sample promising
hyperparameter regions sequentially.

Computer Engineering

DI04000061: ML

4th Semester

228 / 455

Ensemble Techniques for Performance Improvement I
Combining Weak Learners into Strong Models
Ensembling aggregates predictions from multiple base models to reduce variance, bias, or both.

Bagging (Bootstrap Aggregating):

Trains base models independently in parallel on bootstrap samples of training data.
Primary Goal: Variance reduction (e.g., Random Forests).
Boosting:

Trains base models sequentially, each focusing on correcting errors made by previous
iterations.
Primary Goal: Bias and variance reduction (e.g., AdaBoost, XGBoost, LightGBM).
Stacking:

Combines heterogeneous base classifiers via a meta-learner trained on out-of-fold predictions.

Computer Engineering

DI04000061: ML

4th Semester

229 / 455

Python Pipeline: Grid Search & Evaluation I
i m p o r t numpy a s np
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t , G r i d S e a r c h C V , S t r a t i f i e d K F o l d
from s k l e a r n . m e t r i c s i m p o r t c l a s s i f i c a t i o n r e p o r t , r o c a u c s c o r e
from s k l e a r n . e n s e m b l e i m p o r t R a n d o m F o r e s t C l a s s i f i e r
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r
from s k l e a r n . p i p e l i n e i m p o r t P i p e l i n e
# 1. Pipeline Definition
pipe = Pipeline ([
( ’ scaler ’ , StandardScaler ()) ,
( ’ r f ’ , R a n d o m F o r e s t C l a s s i f i e r ( r a n d o m s t a t e =42))
])
# 2. Hyperparameter Search Grid
param grid = {
’ r f n e s t i m a t o r s ’ : [ 5 0 , 100 , 200] ,
’ r f m a x d e p t h ’ : [ None , 1 0 , 2 0 ] ,
’ r f m i n s a m p l e s s p l i t ’ : [2 , 5]
}
# 3 . G r i d S e a r c h w i t h S t r a t i f i e d C r o s s−V a l i d a t i o n
c v = S t r a t i f i e d K F o l d ( n s p l i t s =5 , s h u f f l e =True , r a n d o m s t a t e =42)
g r i d s e a r c h = G r i d S e a r c h C V ( p i p e , p a r a m g r i d , c v=cv , s c o r i n g= ’ r o c a u c ’ , n j o b s =−1)
grid search . f i t ( X train , y t r a i n )
# 4 . E v a l u a t i o n on Held−Out T e s t S e t
best model = grid search . best estimator
y pred proba = best model . predict proba ( X test ) [ : , 1]
Computer Engineering

DI04000061: ML

4th Semester

230 / 455

Python Pipeline: Grid Search & Evaluation II

p r i n t ( f ” B e s t ROC−AUC : { r o c a u c s c o r e ( y t e s t , y p r e d p r o b a ) : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

231 / 455

Unit 3 Review & End-to-End Workflow Checklist I

Preparing to Model & Evaluation Checklist
1

Data Cleaning & Remediation: Impute missingness, mitigate outliers, handle noise.

2

Preprocessing: Scale numerical attributes, encode categorical variables properly.

3

Validation Protocol: Establish strict train/validation/test split or Stratified K -Fold CV.

4

Metric Alignment: Choose evaluation metrics aligned with business objectives (F1 , ROC-AUC, MAE).

5

Diagnostic Check: Plot learning curves to distinguish underfitting vs. overfitting.

6

Optimization: Apply hyperparameter tuning and ensemble techniques for performance boosts.

Golden Rule of Model Building
Prevent Data Leakage: Always fit transformers, scalers, and encoders strictly on the training partition within cross-validation loops!

Computer Engineering

DI04000061: ML

4th Semester

232 / 455

Introduction to Supervised Learning I

What is Supervised Learning?
Supervised learning is a machine learning paradigm where an algorithm learns a mapping function f : X → Y from a labeled training dataset
D = {(x (1) , y (1) ), (x (2) , y (2) ), . . . , (x (N) , y (N) )}.

Core Goal

Fundamental Task Types

Given a new, unseen input vector x∗ ∈ X , predict the corresponding true
target label y ∗ ∈ Y with high accuracy and generalization.

Computer Engineering

DI04000061: ML

Regression: Continuous target space (Y ⊆ Rk ).
Classification: Discrete categorical space (Y ∈ {1, . . . , C }).

4th Semester

233 / 455

Mathematical Setup & Empirical Risk Minimization I

Supervised Learning Framework
Let X ∈ RN×d be the feature design matrix and y ∈ RN be the vector of ground-truth target outputs.
Hypothesis Space H: The set of candidate functions fw (x) parameterized by w.
Loss Function L(fw (x), y ): Measures discrepancy between prediction and ground truth.

Empirical Risk Minimization (ERM)
Since the true underlying distribution P(X , Y ) is unknown, we minimize the empirical risk (training loss):

ŵ = arg min Remp (w) = arg min
w

Computer Engineering

w

N
1 X

N i=1

DI04000061: ML



(i)
(i)
L fw (x ), y

4th Semester

234 / 455

Overview of Regression Problems I

Definition of Regression
Regression models model the relationship between a continuous outcome variable y ∈ R and one or more predictor variables x ∈ Rd .

Real-World Applications

Probabilistic View
Assuming target is generated by deterministic model plus additive Gaussian noise:

Finance: Stock price estimation, credit scoring.
Real Estate: Property valuation based on square footage and location.

y = f (x) + ϵ,

Healthcare: Predicting patient response doses and survival duration.

2

ϵ ∼ N (0, σ )

Predicting f (x) corresponds to modeling the conditional expectation
E[Y | X = x].

Computer Engineering

DI04000061: ML

4th Semester

235 / 455

Simple Linear Regression & Ordinary Least Squares (OLS) I

Simple Linear Regression Model
With a single feature x ∈ R, we assume a linear functional form:
ŷ = f (x) = w1 x + w0
where w1 is the slope coefficient and w0 is the bias (intercept).

Mean Squared Error (MSE) Objective
We find parameters (w0 , w1 ) by minimizing the sum of squared residuals:

J(w0 , w1 ) =

Computer Engineering

N 
2
1 X
(i)
(i)
y − (w1 x + w0 )

2N i=1

DI04000061: ML

4th Semester

236 / 455

Simple Linear Regression & Ordinary Least Squares (OLS) II

Analytical Closed-Form Solution
∂J = 0 and ∂J = 0 yields:
Setting derivatives ∂w
∂w
0

1

PN
w1 =

Computer Engineering

(i)
− x̄)(y (i) − ȳ )
Cov(x, y )
i=1 (x
,
=
PN
(i) − x̄)2
(x
Var(x)
i=1

DI04000061: ML

w0 = ȳ − w1 x̄

4th Semester

237 / 455

Multiple Linear Regression & Normal Equation I
Matrix Formulation
For d input features, represent inputs as augmented vectors x = [1, x1 , x2 , . . . , xd ]T ∈ Rd+1 and weights w = [w0 , w1 , . . . , wd ]T :
ŷ = Xw,

N×(d+1)

where X ∈ R

Matrix Loss Minimization
J(w) =

1
2N

2

∥Xw − y∥2 =

1
2N

T

(Xw − y) (Xw − y)

1 XT (Xw − y).
Gradient with respect to w: ∇w J(w) = N

The Normal Equation
Setting ∇w J(w) = 0 yields the analytical solution:
w

∗

T

−1 T

= (X X)

X y

3

Computational complexity is O(d ) due to matrix inversion.

Computer Engineering

DI04000061: ML

4th Semester

238 / 455

Optimization via Gradient Descent I

Why Gradient Descent?
When feature count d is very large (d > 104 ), computing (XT X)−1 becomes computationally intractable or non-invertible due to multicollinearity.

Iterative Parameter Update
Initialize w
α > 0:

(0)

Gradient Descent Variants

randomly or with zeros. Update iteratively using learning rate
w

(t+1)

=w

(t)

(t)

−

For MSE loss:
w

(t+1)

=w

Computer Engineering

− α∇w J(w

(t)

Batch GD: Uses all N samples per iteration (exact gradient).
Stochastic GD (SGD): Uses 1 random sample per step (fast, noisy).

)

Mini-batch GD: Uses batch size B ∈ [32, 256] (optimal balance).

α T
(t)
X (Xw − y)

N

DI04000061: ML

4th Semester

239 / 455

Code: Linear Regression from Scratch & Scikit-Learn I

i m p o r t numpy a s np
from s k l e a r n . l i n e a r m o d e l i m p o r t L i n e a r R e g r e s s i o n
from s k l e a r n . m e t r i c s i m p o r t m e a n s q u a r e d e r r o r
# G e n e r a t e s y n t h e t i c d a t a s e t : y = 3∗ x1 + 2∗ x2 + 4 + n o i s e
np . random . s e e d ( 4 2 )
X = np . random . r a n d n ( 1 0 0 , 2 )
y = 3 ∗ X [ : , 0 ] + 2 ∗ X [ : , 1 ] + 4 + np . random . r a n d n ( 1 0 0 ) ∗ 0 . 1
# Method 1 : C l o s e d−form v i a Normal E q u a t i o n
X b = np . c [ np . o n e s ( ( 1 0 0 , 1 ) ) , X ] # Add b i a s term column
w a n a l y t i c a l = np . l i n a l g . i n v ( X b . T @ X b ) @ X b . T @ y
# Method 2 : S c i k i t −L e a r n E s t i m a t o r
model = L i n e a r R e g r e s s i o n ( f i t i n t e r c e p t =True )
model . f i t (X , y )
p r i n t ( f ” A n a l y t i c a l W e i g h t s : { w a n a l y t i c a l }” )
p r i n t ( f ” S k l e a r n I n t e r c e p t : {model . i n t e r c e p t : . 4 f } , C o e f f s : {model . c o e f }” )
y p r e d = model . p r e d i c t (X)
p r i n t ( f ” T r a i n i n g MSE : { m e a n s q u a r e d e r r o r ( y , y p r e d ) : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

240 / 455

Polynomial Regression & Non-linear Features I

Extending Linear Models to Non-linear Data
If relationship between x and y is non-linear, we can apply a non-linear feature transformation ϕ(x):
2

3

p T

ϕ(x) = [1, x, x , x , . . . , x ]
The model remains linear in its parameters w:
2

ŷ = w0 + w1 x + w2 x + · · · + wp x

p

T

= w ϕ(x)

Overfitting Risk with High-Degree Polynomials
Low degree p (e.g., p = 1): High bias / underfitting (fails to capture trend).
High degree p (e.g., p = 15): High variance / overfitting (fits noise).
Solution: Cross-validation to select p, combined with feature scaling.

Computer Engineering

DI04000061: ML

4th Semester

241 / 455

Code: Polynomial Regression Pipeline I

from s k l e a r n . p r e p r o c e s s i n g i m p o r t P o l y n o m i a l F e a t u r e s , S t a n d a r d S c a l e r
from s k l e a r n . p i p e l i n e i m p o r t P i p e l i n e
from s k l e a r n . l i n e a r m o d e l i m p o r t L i n e a r R e g r e s s i o n
# 1 . C r e a t e a s c i k i t −l e a r n P i p e l i n e
poly regression = Pipeline ([
( ’ p o l y f e a t u r e s ’ , P o l y n o m i a l F e a t u r e s ( d e g r e e =3 , i n c l u d e b i a s=F a l s e ) ) ,
( ’ scaler ’ , StandardScaler ()) ,
( ’ linear reg ’ , LinearRegression ())
])
# 2 . F i t p i p e l i n e on t r a i n i n g d a t a
p o l y r e g r e s s i o n . f i t (X , y )
# 3 . P r e d i c t and e v a l u a t e
y p o l y p r e d = p o l y r e g r e s s i o n . p r e d i c t (X)
mse = m e a n s q u a r e d e r r o r ( y , y p o l y p r e d )
p r i n t ( f ” 3 r d D e g r e e P o l y n o m i a l MSE : {mse : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

242 / 455

Evaluation Metrics for Regression Models I

Quantitative Metrics Overview
Let yi be true targets, ŷi be predictions, and ȳ be target mean.

R-Squared (R 2 ) Score

Absolute & Squared Errors
1 PN |y − ŷ | Robust to outliers.
MAE (Mean Absolute Error): N
i
i=1 i
1 PN (y − ŷ )2 Penalizes large errors
MSE (Mean Squared Error): N
i
i=1 i

heavily.

RMSE (Root MSE):

√
MSE Interpretable in target units.

2

R =1−

SSres
SStot

PN

(yi − ŷi )2
N (y − ȳ )2
i=1 i

= 1 − Pi=1

R 2 = 1.0: Perfect predictions.
R 2 = 0.0: Performance equals baseline mean predictor ȳ .
R 2 < 0.0: Predictor performs worse than mean predictor.

Computer Engineering

DI04000061: ML

4th Semester

243 / 455

Summary: Supervised Learning & Regression Fundamentals I

Key Takeaways
1

Supervised Paradigm: Mapping feature inputs to targets via labeled pairs D = {(x(i) , y (i) )}.

2

Linear Regression: Fundamental baseline predicting continuous outcomes by minimizing squared errors.

3

Optimization: Closed-form Normal Equation (XT X)−1 XT y vs. iterative Gradient Descent.

4

Non-linearity: Polynomial feature maps ϕ(x) extend linear capacity without changing linear parameterization.

5

Evaluation: Utilize RMSE, MAE, and R 2 to assess generalization quality.

Computer Engineering

DI04000061: ML

4th Semester

244 / 455

Brief Overview of Supervised Machine Learning I

What is Supervised Machine Learning?
Supervised learning is an algorithm design paradigm where a model learns a mapping function f : X → Y from a dataset of labeled training examples.

Dataset Structure: D = {(x (1) , y (1) ), (x (2) , y (2) ), . . . , (x (N) , y (N) )}, where:

x (i) ∈ Rd represents the d-dimensional input feature vector.
y (i) represents the ground-truth target label.
Primary Objective: Generalize to accurately predict target outputs ŷ for novel, unseen input samples xnew .
Core Paradigms:

Classification: Target space Y is discrete (e.g., categorical labels {0, 1}).
Regression: Target space Y is continuous (e.g., real numbers R).

Computer Engineering

DI04000061: ML

4th Semester

245 / 455

Core Components of the Supervised Learning Pipeline I

3. Empirical Risk Minimization

1. Data Representation
Feature Matrix: X ∈ RN×d containing N samples and d features.

Define a loss metric L(y , hθ (x)) measuring prediction discrepancy.

Target Vector: Y ∈ RN containing ground-truth values.
θ̂ = arg min
θ

2. Model Hypothesis Class
Select a parameterized function family hθ (x) (e.g., linear, polynomial, neural
net).

N
1 X

N i=1



(i)
(i)
L y , hθ (x )

4. Evaluation & Generalization
Assess model performance on a separate validation/test set to ensure no
overfitting.

Computer Engineering

DI04000061: ML

4th Semester

246 / 455

Defining Regression Analysis I

Definition of Regression Analysis
Regression analysis is a fundamental statistical and machine learning method used to model the relationship between a continuous dependent target variable
y ∈ R and one or more independent feature variables x ∈ Rd .
Generative Assumption:
y = f (x) + ϵ
where f (x) = E[Y | X = x] is the true systematic function, and ϵ represents random irreducible noise with E[ϵ] = 0 and Var(ϵ) = σ 2 .
Goal of Regression: Construct an estimator fˆ(x) such that the expected risk or deviation between y and ŷ = fˆ(x) is minimized.

Computer Engineering

DI04000061: ML

4th Semester

247 / 455

Taxonomy & Real-World Applications of Regression I

Types of Regression Models
Simple Linear Regression: Single predictor x ∈ R, linear mapping y = β0 + β1 x.
P
Multiple Linear Regression: Multiple features x ∈ Rd , y = β0 + dj=1 βj xj .
Non-linear / Polynomial Regression: Models non-linear interactions via basis function expansions (e.g., x 2 , sin(x)).

Practical Applications
Real Estate Valuation: Estimating house prices based on size, location, and age.
Finance & Economics: Forecasting stock returns, inflation rates, or credit risk metrics.
Healthcare: Predicting patient blood pressure or drug response levels based on clinical features.

Computer Engineering

DI04000061: ML

4th Semester

248 / 455

Measuring Prediction Error in Regression I
Residual: The difference between observed outcome y (i) and predicted value ŷ (i) :

e

(i)

=y

(i)

− ŷ

(i)

Common Regression Loss Functions
Mean Squared Error (MSE):
MSE =


X  (i)
1 N
(i) 2
y − ŷ

N i=1

Penalizes larger errors disproportionately due to squaring.
√
MSE (interpretable in original target units).

Root Mean Squared Error (RMSE): RMSE =
Mean Absolute Error (MAE):

MAE =

N
1 X

N i=1

y

(i)

− ŷ

(i)

More robust to extreme outliers compared to MSE.

Computer Engineering

DI04000061: ML

4th Semester

249 / 455

Python Code Example: Supervised Regression I
i m p o r t numpy a s np
from s k l e a r n . l i n e a r m o d e l i m p o r t L i n e a r R e g r e s s i o n
from s k l e a r n . m e t r i c s i m p o r t m e a n s q u a r e d e r r o r
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
# 1. Generate s y n t h e t i c dataset : y = 2.5 ∗ X + 4.0 + noise
np . random . s e e d ( 4 2 )
X = 2 ∗ np . random . r a n d ( 1 0 0 , 1 )
y = 2 . 5 ∗ X + 4 . 0 + np . random . r a n d n ( 1 0 0 , 1 ) ∗ 0 . 5
# 2 . S p l i t d a t a s e t i n t o t r a i n i n g (80%) and t e s t i n g (20%) s e t s
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42
)
# 3 . I n i t i a l i z e and f i t S u p e r v i s e d L i n e a r R e g r e s s i o n model
model = L i n e a r R e g r e s s i o n ( )
model . f i t ( X t r a i n , y t r a i n )
# 4 . P r e d i c t on t e s t s e t and e v a l u a t e p e r f o r m a n c e
y p r e d = model . p r e d i c t ( X t e s t )
mse = m e a n s q u a r e d e r r o r ( y t e s t , y p r e d )
p r i n t ( f ” S l o p e ( b e t a 1 ) : {model . c o e f [ 0 ] [ 0 ] : . 4 f }” )
p r i n t ( f ” I n t e r c e p t ( b e t a 0 ) : {model . i n t e r c e p t [ 0 ] : . 4 f }” )
p r i n t ( f ” T e s t S e t MSE : {mse : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

250 / 455

Lecture Summary & Next Steps I

Key Takeaways
1

Supervised Learning uses paired feature-target observations (x, y ) to construct predictive mapping functions.

2

Regression Analysis addresses continuous outcome targets by modeling conditional expected values E[Y | X = x].

3

Optimization algorithms solve regression problems by minimizing loss metrics like MSE or MAE.

Looking Ahead
In subsequent lectures, we will explore:
Closed-form solutions via Ordinary Least Squares (OLS) Normal Equations.
Iterative optimization via Gradient Descent.
Regularization techniques (Ridge, Lasso) to combat overfitting.

Computer Engineering

DI04000061: ML

4th Semester

251 / 455

Lecture 4.3: Overview & Learning Objectives I

Unit 4: Supervised Machine Learning
Supervised Learning learns a mapping function f : X → Y from labeled training pairs {(xi , yi )}N
i=1 , where xi ∈ X are input features and yi ∈ Y are target
outputs.

4.1.2 Learning Steps

4.3.2 Types of Regression

Problem Formulation & Target Definition

Simple & Multiple Linear Regression

Data Acquisition & Preprocessing

Polynomial Regression

Model Selection & Hypothesis Space

Regularized Regression (L1 , L2 , ElasticNet)

Empirical Risk Minimization

Tree-based & Non-Parametric Regression

Validation, Testing & Deployment

Support Vector Regression (SVR)

Computer Engineering

DI04000061: ML

4th Semester

252 / 455

Learning Steps in Supervised Machine Learning (Part 1) I
Step-by-Step Execution Pipeline
1

Problem Formulation & Target Definition:

Determine the target output domain: Continuous (y ∈ R for Regression) or Categorical
(y ∈ {C1 , . . . , Ck } for Classification).
Establish task metrics (e.g., MSE, R 2 , Accuracy, F1-score) and business constraints.
2

Data Collection & Preprocessing:

Gather historical data samples {(xi , yi )}. Clean noise, deal with missing values and outliers.
Perform feature engineering: scaling (StandardScaler, MinMaxScaler), encoding, and
dimensionality selection.
3

Dataset Partitioning:

Split data into Train (60 − 80%), Validation (10 − 20%), and Test (10 − 20%) sets to
evaluate out-of-sample generalization.

Computer Engineering

DI04000061: ML

4th Semester

253 / 455

Learning Steps in Supervised Machine Learning (Part 2) I
Model Selection, Optimization, and Evaluation
1

Hypothesis Space & Model Selection:

Choose candidate functional family fθ (x) ∈ H parameterized by weights θ (e.g., linear
combination, decision trees, neural network).
2

Model Training & Optimization:

Fit parameter set θ via optimization algorithm (e.g., OLS closed-form, Gradient Descent) by
minimizing training loss:
N

θ̂ = arg min
θ

3

1 X
L(fθ (xi ), yi ) + λΩ(θ)
N
i=1

Validation, Hyperparameter Tuning & Testing:

Tune structural parameters using Cross-Validation on the validation set.
Evaluate final model generalization capability on the isolated test dataset.
Computer Engineering

DI04000061: ML

4th Semester

254 / 455

Mathematical Framework: Empirical Risk Minimization I

Formal Setup of Supervised Learning
Given training dataset D = {(x1 , y1 ), . . . , (xN , yN )} sampled i.i.d. from joint distribution P(X , Y ):

Loss Function L(y , ŷ )

Empirical Risk Minimization (ERM)

Measures prediction discrepancy:

Minimizes average training error:

Squared Loss (Regression):
Remp (θ) =

2

L(y , ŷ ) = (y − ŷ )
Absolute Loss (Regression):

N
1 X

N i=1

L(yi , fθ (xi ))

Goal: Minimize true generalization risk R(θ) = E(x,y )∼P [L(y , fθ (x))].

L(y , ŷ ) = |y − ŷ |

Computer Engineering

DI04000061: ML

4th Semester

255 / 455

Introduction to Regression Analysis I

What is Regression?
Regression is a primary subfield of supervised learning where the objective is to predict a continuous numerical target y ∈ R given input vector x ∈ Rp .

Core Objectives

Typical Applications

Prediction: Estimate unknown continuous values for unseen inputs x ∗ .

Housing valuation based on physical features.

Inference: Analyze direct relationships, coefficient magnitude, and
feature impact on y .

Temperature & climate forecasting.

Computer Engineering

DI04000061: ML

Customer Lifetime Value (CLV) estimation.

4th Semester

256 / 455

Taxonomy & Types of Regression Models I

Regression Family
Linear Models
Polynomial Basis
Regularized Linear
Regularized Linear
Regularized Linear
Non-Parametric
Ensemble Trees
Kernel Methods

Specific Model Type
Simple / Multiple Linear
Polynomial Regression
Ridge (L2 Penalty)
Lasso (L1 Penalty)
ElasticNet (L1 + L2 )
Decision Tree Regressor
Random Forest Regressor
Support Vector Regression

Key Operational Trait
Linear mapping, OLS analytical solution
Non-linear curve fitting via power features
Weight shrinkage, handles multicollinearity
Sparse coefficients, feature selection
Balanced sparsity & feature grouping
Piecewise step-constant spatial partitions
Aggregates trees to reduce variance
Insensitive margin tube (ϵ), kernel trick

Model Selection Trade-off
High interpretability (Linear/Ridge) vs. high flexibility & expressive power (Trees/SVR).

Computer Engineering

DI04000061: ML

4th Semester

257 / 455

Linear and Polynomial Regression I
1. Simple & Multiple Linear Regression
Assumes linear relationship between predictor variables X and continuous outcome y :
y = β0 + β1 x1 + β2 x2 + · · · + βp xp + ϵ = X β + ϵ

Ordinary Least Squares (OLS) estimates: β̂ = (X T X )−1 X T y .
Highly interpretable; vulnerable to outliers and strong feature correlation (multicollinearity).

2. Polynomial Regression
Models non-linear relationships by creating degree-d polynomial combinations:
2

d

y = β0 + β1 x + β2 x + · · · + βd x + ϵ

Remains linear with respect to parameters β, allowing standard OLS fitting.
High degree polynomials (d ≫ 3) risk severe overfitting and boundary instability.

Computer Engineering

DI04000061: ML

4th Semester

258 / 455

Regularized Regression: Ridge, Lasso, and ElasticNet I

Regularization penalizes large weight coefficients to prevent overfitting and control model complexity:

Lreg (β) =

Ridge (L2 Penalty)
2

Ω(β) = λ∥β∥2 = λ

1
2N

2

∥y − X β∥2 + Ω(β)

ElasticNet

Lasso (L1 Penalty)

p
X

2

βj

j=1

Ω(β) = λ∥β∥1 = λ

p
X

2

|βj |

Ω(β) = λ1 ∥β∥1 + λ2 ∥β∥2

j=1

Convex combination of L1 and L2 .
Shrinks weights smoothly toward zero.

Drives irrelevant weights exactly to zero.

Stabilizes estimates under multicollinearity.

Performs automatic feature selection.

Computer Engineering

DI04000061: ML

Retains correlated feature groups while
sparse.

4th Semester

259 / 455

Non-Parametric & Advanced Regression Methods I

Decision Tree & Ensemble Regressors

Support Vector Regression (SVR)

Decision Tree Regressor: Divides feature space into orthogonal regions
Rm and predicts mean regional response ȳm .

Uses an ϵ-insensitive loss function: ignores errors smaller than threshold
ϵ.

Random Forest Regressor: Averages predictions across an ensemble of
decorrelated trees to reduce variance.

Formulates objective:

Captures complex non-linear feature interactions naturally.
min

1

w ,b 2

2

∥w ∥ + C

N
X
∗
(ξi + ξi )
i=1

Applies kernel trick (RBF, Polynomial) for non-linear mappings into
higher-dimensional feature space.

Computer Engineering

DI04000061: ML

4th Semester

260 / 455

Python Implementation: Comparing Regression Models I
i m p o r t numpy a s np
from s k l e a r n . l i n e a r m o d e l i m p o r t L i n e a r R e g r e s s i o n , Ridge , L a s s o
from s k l e a r n . p r e p r o c e s s i n g i m p o r t P o l y n o m i a l F e a t u r e s
from s k l e a r n . p i p e l i n e i m p o r t m a k e p i p e l i n e
from s k l e a r n . m e t r i c s i m p o r t m e a n s q u a r e d e r r o r , r 2 s c o r e
# 1 . G e n e r a t e S y n t h e t i c Data
np . random . s e e d ( 4 2 )
X = np . l i n s p a c e ( −3 , 3 , 1 0 0 ) . r e s h a p e ( −1 , 1 )
y = 0 . 5 ∗ X . s q u e e z e ( ) ∗ ∗ 2 + X . s q u e e z e ( ) + 2 + np . random . n o r m a l ( 0 , 0 . 5 , 1 0 0 )
# 2. F i t Simple Linear Regression
l i n r e g = L i n e a r R e g r e s s i o n ( ) . f i t (X , y )
# 3 . F i t Polynomial R e g r e s s i o n ( Degree 2)
p o l y r e g = m a k e p i p e l i n e ( P o l y n o m i a l F e a t u r e s ( d e g r e e =2) , L i n e a r R e g r e s s i o n ( ) )
p o l y r e g . f i t (X , y )
# 4 . F i t R e g u l a r i z e d Ridge & Lasso R e g r e s s i o n
r i d g e r e g = R i d g e ( a l p h a = 1 . 0 ) . f i t (X , y )
l a s s o r e g = L a s s o ( a l p h a = 0 . 1 ) . f i t (X , y )
# 5. Evaluate Performance
y p r e d p o l y = p o l y r e g . p r e d i c t (X)
p r i n t ( f ” P o l y MSE : { m e a n s q u a r e d e r r o r ( y , y p r e d p o l y ) : . 4 f }” )
p r i n t ( f ” P o l y R2 S c o r e : { r 2 s c o r e ( y , y p r e d p o l y ) : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

261 / 455

Summary: Model Selection Guide for Regression I

Practical Model Selection Guidelines
1

Linear baseline & high interpretability needed: → Simple / Multiple Linear Regression (OLS).

2

Multicollinearity present or high-dimensional features: → Ridge Regression (L2 regularization).

3

High-dimensional space requiring sparse feature selection: → Lasso Regression (L1 ) or ElasticNet.

4

Non-linear continuous curve with single/few features: → Polynomial Regression.

5

Complex tabular dataset with strong non-linear interactions: → Random Forest / Gradient Boosted Trees.

6

Complex non-linear boundary with low sample density: → Support Vector Regression (SVR with RBF kernel).

Best Practice
Always evaluate regression models on out-of-sample test datasets using metrics like MSE, RMSE, MAE, and R 2 , and perform residual error analysis.

Computer Engineering

DI04000061: ML

4th Semester

262 / 455

Lecture 4.4: Overview & Learning Objectives I
Unit 4: Supervised Machine Learning
Supervised Machine Learning algorithms map input features x ∈ X to labeled target outputs y ∈ Y using training pairs D = {(xi , yi )}N
i=1 .

4.3.2 Real World Applications
Healthcare: Diagnosis & Risk Prediction
Finance: Credit Scoring & Fraud Detection
Industry: House Pricing & Demand Forecast
Tech: Email Spam Filtering & NLP

4.3.3 Linear Regression
Classification of Linear Models
Scalar, Vector, Matrix Formulation
Geometry of Positive & Negative Slopes
Parameter Estimation via OLS & Code

Computer Engineering

DI04000061: ML

4th Semester

263 / 455

4.1.3 Real-World Applications: Healthcare & Finance I

Healthcare & Medical Diagnostics
Disease Detection: Classification algorithms analyze imaging (MRI, X-rays) to detect malignant tumors or diabetic retinopathy.
Patient Readmission: Logistic regression predicts 30-day readmission likelihood based on patient EHR metrics.
Genomic Medicine: Regression models predict therapeutic drug efficacy from patient genetic markers.

Finance & Banking Systems
Credit Risk Scoring: Estimate loan default probability (y ∈ {0, 1}) using income, debt ratio, and credit history.
Fraud Detection: Supervised classifiers detect unauthorized credit card transactions in real-time.
Algorithmic Trading: Predict continuous asset prices (ŷ ∈ R) using economic indicator vectors.

Computer Engineering

DI04000061: ML

4th Semester

264 / 455

4.1.3 Real-World Applications: Industry & E-Commerce I

Real Estate & Supply Chain
Automated Valuation Models (AVM): Multiple linear regression estimates house market values based on area, rooms, and location.
Demand Forecasting: Regression models forecast product demand for optimal warehouse inventory allocation.

Cybersecurity
Spam Filtering: Classifies emails as Spam or Ham using text feature vectors.

E-Commerce & Marketing
Customer Lifetime Value (CLV): Models future revenue generated per user over a time horizon.
Churn Prediction: Binary classification identifies customers at high risk of subscription cancellation.
Dynamic Pricing: Predicts optimal pricing strategies dynamically based on competitor data and demand.

Computer Engineering

DI04000061: ML

4th Semester

265 / 455

4.3.3 Linear Regression: Fundamentals & Concept I
Definition
Linear Regression is a parametric supervised learning model designed to establish a linear relationship between input features x ∈ Rd and a continuous target
variable y ∈ R.

Core Characteristics
Target Domain: Continuous real values (y ∈ R).
Functional Form: Linear combination of input parameters.
High Interpretability: Weights quantify feature influence directly.
Efficiency: Fast to compute analytical or numerical solutions.

Learning Goal
Fit a line (2D) or hyperplane (d-D) that minimizes prediction residuals:
ei = yi − ŷi
Generalize accurately to unseen test instances.

Computer Engineering

DI04000061: ML

4th Semester

266 / 455

4.3.3 Types of Linear Regression I
Taxonomy of Linear Regression Models
Linear regression algorithms are categorized based on feature dimensionality, basis function transformations, and regularization constraints.

Dimensionality & Basis Functions
Simple Linear Regression: One predictor variable (x ∈ R1 ). Fits a 2D line.
Multiple Linear Regression: Multiple predictor variables (x ∈ Rd ). Fits a hyperplane.
Polynomial Regression: Incorporates non-linear terms (x 2 , x 3 ). Linear in weights w.

Regularization Variants
Ordinary Least Squares (OLS): Unconstrained squared error minimization.
Ridge Regression (L2 ): Adds penalty λ∥w∥22 to address multicollinearity.
Lasso Regression (L1 ): Adds penalty λ∥w∥1 ; drives weights to 0 for feature selection.
ElasticNet: Combines L1 and L2 penalties.

Computer Engineering

DI04000061: ML

4th Semester

267 / 455

4.3.3 Mathematical Formulation of Linear Regression I

1. Simple Linear Regression (Scalar Form)
For a single input xi , the response variable is modeled as:
yi = w0 + w1 xi + ϵi ,

2

ϵi ∼ N (0, σ )

where w0 represents the y-intercept, w1 is the slope, and ϵi is the random error term.

2. Multiple Linear Regression (Vector Form)
For d input features xi = [xi1 , xi2 , . . . , xid ]T :

ŷi = w0 + w1 xi1 + w2 xi2 + · · · + wd xid = w0 +

d
X

wj xij

j=1

Computer Engineering

DI04000061: ML

4th Semester

268 / 455

4.3.3 Mathematical Formulation of Linear Regression II

3. Matrix Notation (Compact System)
Augmenting feature vector with xi0 = 1, we write for all N data instances:
y = Xw + ϵ,

Computer Engineering

where X ∈ R

N×(d+1)

, w∈R

DI04000061: ML

d+1

, y∈R

N

4th Semester

269 / 455

4.3.3 Geometric Interpretation: Positive vs. Negative Slope I

Positive Slope (w1 > 0)

Negative Slope (w1 < 0)

Direct linear relationship: y increases as x increases.

Inverse linear relationship: y decreases as x increases.

y

y
w0

y = w0 + w1 x

y = w 0 + w1 x

Slope w1 < 0

w0

Slope w1 > 0

Example: House Price vs. Living Area.

Computer Engineering

x

x

Example: Used Car Price vs. Mileage.

DI04000061: ML

4th Semester

270 / 455

4.3.3 Parameter Estimation: Loss Function & Optimization I

Mean Squared Error (MSE) Cost Function
Parameters w are estimated by minimizing the average empirical loss J(w):

J(w) =

N
1 X
1
2
T
2
∥y − Xw∥2
(yi − w xi ) =
2N

2N i=1

Closed-Form Solution: Normal Equation
Solving ∇w J(w) = 0 yields:
w

∗

T

−1 T

= (X X)

X y

Exact global minimum.
Computationally expensive for d > 104 due to (XT X)−1 .

Computer Engineering

DI04000061: ML

4th Semester

271 / 455

4.3.3 Parameter Estimation: Loss Function & Optimization II

Iterative Solution: Gradient Descent
Iterative parameter update rule:
w

(t+1)

=w

(t)

− α∇w J(w

(t)

)

α T
(t)
(t+1)
(t)
X (y − Xw )
w
=w +
N
Efficiently scales to large-scale data (N, d ≫ 105 ).

Computer Engineering

DI04000061: ML

4th Semester

272 / 455

Python Implementation: Linear Regression with Scikit-Learn I

Computer Engineering

DI04000061: ML

4th Semester

273 / 455

Python Implementation: Linear Regression with Scikit-Learn II
Hands-on Python Example
i m p o r t numpy a s np
from s k l e a r n . l i n e a r m o d e l i m p o r t L i n e a r R e g r e s s i o n
from s k l e a r n . m e t r i c s i m p o r t m e a n s q u a r e d e r r o r , r 2 s c o r e
# 1. Generate s y n t h e t i c dataset ( x : feature , y : t a r g e t )
np . random . s e e d ( 4 2 )
X = 2 ∗ np . random . r a n d ( 1 0 0 , 1 ) # 100 s a m p l e s , 1 f e a t u r e
y = 4 + 3 ∗ X + np . random . r a n d n ( 1 0 0 , 1 ) # y = 4 + 3 x + n o i s e
# 2 . I n s t a n t i a t e and t r a i n L i n e a r R e g r e s s i o n model
model = L i n e a r R e g r e s s i o n ( )
model . f i t (X , y )
# 3. Extract f i t t e d parameters ( slope & i n t e r c e p t )
w0 = model . i n t e r c e p t [ 0 ] # I n t e r c e p t ( w 0 )
w1 = model . c o e f [ 0 ] [ 0 ]
# Slope ( w 1 )
p r i n t ( f ” F i t t e d L i n e : y = {w0 : . 2 f } + {w1 : . 2 f }∗x ” )
# 4 . P r e d i c t and e v a l u a t e p e r f o r m a n c e
y p r e d = model . p r e d i c t (X)
mse = m e a n s q u a r e d e r r o r ( y , y p r e d )
r2 = r 2 s c o r e (y , y pred )
p r i n t ( f ”MSE : {mse : . 4 f } | Rˆ2 S c o r e : { r 2 : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

274 / 455

Lecture 4.4: Summary & Key Takeaways I

Core Concepts Summary
1

Real-World Applications: Supervised ML powers critical applications in Healthcare, Finance, Cybersecurity, Real Estate, and E-Commerce.

2

Linear Regression Paradigm: Parametric model fitting a line/hyperplane y = wT x + ϵ to continuous targets.

3

Model Taxonomy: Includes Simple, Multiple, Polynomial, and Regularized variants (Ridge L2 , Lasso L1 , ElasticNet).

4

Slope Geometry: Positive slope (w1 > 0) indicates direct relationship; negative slope (w1 < 0) indicates inverse relationship.

5

Parameter Fitting: Solved via OLS Normal Equations (XT X)−1 XT y or iteratively via Gradient Descent.

Computer Engineering

DI04000061: ML

4th Semester

275 / 455

Supervised Machine Learning: Core Paradigm I

Definition & Goal
Supervised Learning algorithms learn a mapping function f : X → Y from labeled training pairs D = {(x1 , y1 ), (x2 , y2 ), . . . , (xN , yN )}, where xi ∈ Rd are
features and yi are ground-truth targets.

Regression Tasks

Classification Tasks

Target: Continuous variable (yi ∈ R).

Target: Discrete class label (yi ∈ {1, . . . , C }).

Examples: House price prediction, temperature forecasting, stock
index estimation.

Examples: Spam detection, medical diagnosis, image recognition.

Computer Engineering

DI04000061: ML

4th Semester

276 / 455

Advantages of Supervised Machine Learning I

Key Strengths
High Predictive Accuracy Explicit ground-truth guidance enables optimization of precise loss functions tailored to target tasks.
Clear Performance Metrics Direct evaluation against ground truth using standard quantitative metrics (Accuracy, F1-Score, MSE, R 2 ).
Interpretability & Control Parametric models (e.g., Linear Regression, Decision Trees) allow direct analysis of feature importance and decision boundaries.
Task-Specific Optimization Models optimize directly for specific business objectives or clinical/engineering outcomes.

Computer Engineering

DI04000061: ML

4th Semester

277 / 455

Disadvantages & Limitations of Supervised Machine Learning I

Key Challenges
High Labeling Cost Requires large volumes of accurately annotated data, often demanding expensive domain expertise or human annotation.
Risk of Overfitting Complex models may memorize training noise rather than generalizable patterns (Bias-Variance Tradeoff).
Closed-World Assumption Models struggle with Out-Of-Distribution (OOD) data and unseen target categories at test time.
Data Bias Propagation Historical biases present in training labels are directly encoded and amplified by the model.

Operational Bottleneck
Data annotation is often the single biggest operational bottleneck in real-world ML deployment pipelines.

Computer Engineering

DI04000061: ML

4th Semester

278 / 455

Trade-off Matrix: Supervised Machine Learning I

Methodological Comparison
Dimension
Data Req.
Primary Goal
Evaluation
Complexity

Supervised ML
Labeled pairs (X , y )
Map inputs to targets
Objective ground-truth metrics
High data prep, low ambiguity

Computer Engineering

Unsupervised ML
Unlabeled features X
Discover hidden structures
Heuristic / internal metrics
Low data prep, high evaluation ambiguity

DI04000061: ML

4th Semester

279 / 455

Introduction to Simple Linear Regression I

What is Simple Linear Regression?
Simple Linear Regression (SLR) models the relationship between a single independent continuous predictor variable X and a dependent target variable Y using a
linear function.

Mathematical Formulation
For a dataset with N observations {(x1 , y1 ), . . . , (xN , yN )}:
yi = w0 + w1 xi + ϵi = β0 + β1 xi + ϵi

w0 (β0 ): y-intercept — expected value of Y when X = 0.
w1 (β1 ): Slope coefficient — expected change in Y per unit increase in X .
ϵi : Unobserved random error term (ϵi ∼ N (0, σ 2 )).

Computer Engineering

DI04000061: ML

4th Semester

280 / 455

Simple Linear Regression: Objective Function I
Predictive Model & Residuals
Given estimates (ŵ0 , ŵ1 ), the predicted target is ŷi = ŵ0 + ŵ1 xi . The residual (error) for sample i is:
ei = yi − ŷi = yi − (ŵ0 + ŵ1 xi )

Ordinary Least Squares (OLS) Objective
Find parameters (w0 , w1 ) that minimize the Sum of Squared Errors (SSE):

J(w0 , w1 ) = SSE(w0 , w1 ) =

N
X
i=1

2

ei =

N
X

yi − (w0 + w1 xi )


2

i=1

Why Square the Errors?
1. Penalizes larger errors more heavily than small errors.
2. Prevents positive and negative residuals from canceling out.
3. Yields a convex, smooth, and twice-differentiable objective function.

Computer Engineering

DI04000061: ML

4th Semester

281 / 455

Deriving the OLS Analytical Solution I
First-Order Conditions (Normal Equations)
Set partial derivatives of J(w0 , w1 ) with respect to w0 and w1 to zero:
∂J
∂w0

∂J
∂w1

= −2

= −2

N
X

N
X

i=1

i=1

(yi − w0 − w1 xi ) = 0 =⇒

N
X

xi (yi − w0 − w1 xi ) = 0 =⇒

i=1

N
X

(yi − w0 − w1 xi ) = 0

xi (yi − w0 − w1 xi ) = 0

i=1

Closed-Form Expressions
Solving the system of equations yields:
PN
ŵ1 =
1
where x̄ = N

P

1
xi and ȳ = N

P

Computer Engineering

Cov(X , Y )
i=1 (xi − x̄)(yi − ȳ )
,
=
PN
Var(X )
(x − x̄)2
i=1 i

ŵ0 = ȳ − ŵ1 x̄

yi are the sample means.

DI04000061: ML

4th Semester

282 / 455

Code: Implementation of OLS from Scratch & Scikit-Learn I

i m p o r t numpy a s np
from s k l e a r n . l i n e a r m o d e l i m p o r t L i n e a r R e g r e s s i o n
# Synthetic dataset : y = 3 + 2.5 ∗ x + noise
np . random . s e e d ( 4 2 )
X = np . a r r a y ( [ 1 , 2 , 3 , 4 , 5 ] , d t y p e=np . f l o a t 6 4 )
y = np . a r r a y ( [ 5 . 1 , 8 . 2 , 1 0 . 3 , 1 3 . 1 , 1 5 . 7 ] , d t y p e=np . f l o a t 6 4 )
# 1 . C l o s e d−form OLS c a l c u l a t i o n from s c r a t c h
x b a r , y b a r = np . mean (X ) , np . mean ( y )
w 1 s c r a t c h = np . sum ( ( X − x b a r ) ∗ ( y − y b a r ) ) / np . sum ( ( X − x b a r ) ∗ ∗ 2 )
w0 scratch = y bar − w1 scratch ∗ x bar
p r i n t ( f ” S c r a t c h −> I n t e r c e p t ( w0 ) : { w 0 s c r a t c h : . 4 f } , S l o p e ( w1 ) : { w 1 s c r a t c h : . 4 f }” )
# 2 . E q u i v a l e n t S c i k i t −L e a r n I m p l e m e n t a t i o n
r e g = L i n e a r R e g r e s s i o n ( ) . f i t (X . r e s h a p e ( −1 , 1 ) , y )
p r i n t ( f ” S k l e a r n −> I n t e r c e p t : { r e g . i n t e r c e p t : . 4 f } , S l o p e : { r e g . c o e f [ 0 ] : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

283 / 455

Assumptions of Simple Linear Regression I

Classical Gauss-Markov Assumptions
To guarantee that OLS estimators are BLUE (Best Linear Unbiased Estimators):
1

Linearity: The relationship between X and Y is linear in parameters.

2

Independence: Residuals ϵi are mutually independent (Cov(ϵi , ϵj ) = 0).

3

Homoscedasticity: Constant error variance (Var(ϵi ) = σ 2 for all i).

4

Normality of Errors: ϵi ∼ N (0, σ 2 ) (required for hypothesis testing & confidence intervals).

Diagnostics
Always perform residual diagnostics (e.g., Residual vs. Fitted plot, Q-Q plot) to verify these assumptions before deploying predictions!

Computer Engineering

DI04000061: ML

4th Semester

284 / 455

Evaluating Simple Linear Regression I

Goodness-of-Fit Metric: Coefficient of Determination (R 2 )
R 2 measures the proportion of total variance in Y explained by feature X :

2

R =1−

SSres
SStot

Interpretation

PN

(yi − ŷi )2
N (y − ȳ )2
i=1 i

= 1 − Pi=1

Additional Error Metrics
1
MAE: N

R 2 = 1: Perfect model fit.
2

R = 0: Model performs no better than baseline mean ȳ .
R 2 < 0: Model performs worse than the sample mean baseline.

Computer Engineering

DI04000061: ML

P

|yi − ŷi |

1 P(y − ŷ )2
MSE: N
i
i

RMSE:

√
MSE

4th Semester

285 / 455

Lecture Overview & Learning Objectives I

Course: DI04000061 (Introduction Machine Learning)
Unit 4: Supervised Machine Learning
Topics:

4.2 Classification Fundamentals & Taxonomy
4.3.5 Real-World Applications of Regression Analysis
Learning Objectives:
1

Formalize the Classification Paradigm and distinguish binary, multiclass, and multi-label settings.

2

Differentiate between Discriminative and Generative classification approaches.

3

Master core classification algorithms and decision boundary mechanics.

4

Analyze real-world regression applications across finance, healthcare, and engineering.

5

Implement classification and regression pipelines using Python and scikit-learn.

Computer Engineering

DI04000061: ML

4th Semester

286 / 455

Introduction to Classification I

What is Classification?
Classification is a supervised learning paradigm where the algorithm learns a mapping function f : X → Y to assign an input feature vector x ∈ Rd to a discrete
categorical label y ∈ Y.
Target Space Y: A finite, discrete set of classes {c1 , c2 , . . . , cK }.
Decision Surface / Boundary: The geometric partition in feature space Rd separating regions assigned to different classes:
d

{x ∈ R | P(Y = ci | x) = P(Y = cj | x)}
Operational Goal: Minimize expected misclassification loss or maximize classification accuracy on unseen test data.

Computer Engineering

DI04000061: ML

4th Semester

287 / 455

Classification Problem Taxonomy I
1. Binary Classification

2. Multiclass Classification

Target set: Y = {0, 1} or {−1, +1}.

Target set: Y = {1, 2, . . . , K } (K > 2).

Output: Single probability score p = P(Y = 1 | x).

Output: Categorical distribution p ∈ [0, 1]K with

Real-World Examples:

Real-World Examples:

Email Spam Filtering (Spam vs.
Legitimate).
Credit Card Fraud Detection.
Medical Diagnosis (Disease
Positive/Negative).

PK

k=1 pk = 1.

Optical Character Recognition (MNIST
digits 0–9).
Satellite Image Land Cover
Classification.
Document Topic Categorization.

Multi-Label Classification
In Multi-Label Classification, an instance can simultaneously belong to multiple classes (e.g., an article tagged with both Machine Learning and Healthcare),
where y ∈ {0, 1}K .

Computer Engineering

DI04000061: ML

4th Semester

288 / 455

Discriminative vs. Generative Classification I
1. Discriminative Models
Directly model the conditional distribution P(Y | X) or learn decision boundaries separating classes.
Focus: Maximizing separation efficiency between classes.
Examples: Logistic Regression, Support Vector Machines (SVM), Decision Trees, Neural Networks.
Advantage: Often achieves higher accuracy when large labeled datasets are available.

2. Generative Models
Model the joint distribution P(X, Y ) = P(X | Y )P(Y ) by estimating class-conditional likelihoods P(X | Y ) and class priors P(Y ).
Inference: Applies Bayes’ Rule to compute posterior class probabilities:

P(Y = ck | x) = P

P(x | Y = ck )P(Y = ck )
K P(x | Y = c )P(Y = c )
j
j
j=1

Examples: Naive Bayes, Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA).

Computer Engineering

DI04000061: ML

4th Semester

289 / 455

Overview of Key Classification Algorithms I

Logistic Regression: Uses the sigmoid function σ(z) =

1
1+e −z

to model posterior probability:

T

P(Y = 1 | x) = σ(w x + b)
Trained by minimizing Binary Cross-Entropy loss.
Support Vector Machines (SVM): Finds the maximum-margin hyperplane separating classes. Utilizes kernel functions K (x, x′ ) to project data into
higher-dimensional spaces for non-linear separation.
Decision Trees & Random Forests: Partition feature space hierarchically based on impurity reduction (Gini Impurity, Information Gain). Ensembles
aggregate multiple trees to reduce variance.
k-Nearest Neighbors (k-NN): Instance-based non-parametric classifier that assigns labels based on majority vote among k nearest neighbors under a
distance metric (e.g., Euclidean distance).

Computer Engineering

DI04000061: ML

4th Semester

290 / 455

Python Implementation: Classification Pipeline I
i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t m a k e c l a s s i f i c a t i o n
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
from s k l e a r n . l i n e a r m o d e l i m p o r t L o g i s t i c R e g r e s s i o n
from s k l e a r n . m e t r i c s i m p o r t a c c u r a c y s c o r e , c l a s s i f i c a t i o n r e p o r t
# 1. Synthesize binary c l a s s i f i c a t i o n dataset
X , y = m a k e c l a s s i f i c a t i o n ( n s a m p l e s =1000 , n f e a t u r e s =10 ,
n i n f o r m a t i v e =5 , r a n d o m s t a t e =42)
# 2 . S p l i t d a t a s e t i n t o t r a i n i n g (80%) and t e s t i n g (20%) s e t s
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 0 , r a n d o m s t a t e =42
)
# 3 . I n i t i a l i z e and t r a i n L o g i s t i c R e g r e s s i o n model
c l f = L o g i s t i c R e g r e s s i o n ( s o l v e r= ’ l b f g s ’ , C=1.0)
c l f . f i t ( X train , y t r a i n )
# 4 . P r e d i c t and e v a l u a t e p e r f o r m a n c e
y pred = c l f . predict ( X test )
acc = a c c u r a c y s c o r e ( y t e s t , y pred )
p r i n t ( f ” T e s t S e t A c c u r a c y : { a c c ∗ 1 0 0 : . 2 f}%” )
print ( c l a s s i f i c a t i o n r e p o r t ( y test , y pred ))

Computer Engineering

DI04000061: ML

4th Semester

291 / 455

Real-World Regression: Finance & Economics I

1. Asset Pricing & Market Volatility Forecasting
Financial Volatility Modeling: Regression models predict continuous asset volatility metrics based on macroeconomic factors, interest rates, and historical
trading volumes.
Capital Asset Pricing Model (CAPM):
Ri − Rf = αi + βi (Rm − Rf ) + ϵi
Quantifies market risk sensitivity (βi ) of security i.

2. Real Estate Valuation & Credit Risk
Hedonic Housing Models: Estimates continuous home prices (y ∈ R+ ) using structural features (square footage, bedrooms), location indices, and local
market trends.
Credit Limit Determination: Regression algorithms forecast optimal credit line amounts for bank customers as a function of annual income, debt ratio,
and credit history score.

Computer Engineering

DI04000061: ML

4th Semester

292 / 455

Real-World Regression: Healthcare & Medicine I

1. Physiological Monitoring & Diagnostics
Continuous Blood Pressure Estimation: Predicts continuous systolic and diastolic pressure (mmHg) using features derived from photoplethysmogram
(PPG) and electrocardiogram (ECG) waveforms.
Pharmacokinetic Dosing Models: Predicts drug plasma concentration over time to establish customized, optimal dosage amounts for patients.

2. Healthcare Management & Disease Trajectory
Hospital Length-of-Stay (LOS) Prediction: Estimates total days a hospitalized patient will occupy a bed based on clinical vitals, age, and laboratory
markers.
Cognitive Decline Scoring: Fits continuous progression indices (e.g., MMSE scores in Alzheimer’s patients) against genetic indicators and age.

Computer Engineering

DI04000061: ML

4th Semester

293 / 455

Real-World Regression: Engineering & Smart Grids I

1. Predictive Maintenance & Industrial IoT
Remaining Useful Life (RUL) Estimation: Models remaining operational cycles of industrial machinery and turbofan engines before maintenance is
required:
RUL = f (vibration, temperature, pressure, operating hours)
Manufacturing Yield Optimization: Predicts chemical product output yield percentage based on reactor temperature, pressure, and catalyst concentration.

2. Smart Grid & Energy Demand Forecasting
Electrical Load Forecasting: Predicts hourly power demand (MW) on regional power grids based on weather forecasts, time of day, and industrial activity.
Renewable Generation: Forecasts solar photovoltaic power output based on solar irradiance, ambient temperature, and humidity.

Computer Engineering

DI04000061: ML

4th Semester

294 / 455

Comparative Analysis: Classification vs. Regression I

Domain
Real Estate
Healthcare

Regression Task (y ∈ R)
Predict home price in USD ($450,000).
Predict blood glucose level (mg/dL).

Finance

Forecast exact stock return percentage
(+3.4%).
Estimate distance to obstacle (12.5 m).

Automotive
Retail

Predict
customer
($1, 200).

lifetime

spend

Classification Task (y ∈ Y)
Predict status: Sold vs. Unsold.
Classify patient: Diabetic vs.
NonDiabetic.
Predict market direction: Up vs. Down.
Identify object type: Pedestrian, Car,
Sign.
Predict customer churn risk: High, Med,
Low.

Design Principle
Formulate your ML problem as regression when output precision on a continuous scale is needed, and as classification when categorical decision boundaries are
required.

Computer Engineering

DI04000061: ML

4th Semester

295 / 455

Python Implementation: Real-World Regression Pipeline I
i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t f e t c h c a l i f o r n i a h o u s i n g
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
from s k l e a r n . e n s e m b l e i m p o r t R a n d o m F o r e s t R e g r e s s o r
from s k l e a r n . m e t r i c s i m p o r t m e a n s q u a r e d e r r o r , r 2 s c o r e , m e a n a b s o l u t e e r r o r
# 1 . Load r e a l −w o r l d c o n t i n u o u s t a r g e t d a t a s e t
housing = f e t c h c a l i f o r n i a h o u s i n g ()
X , y = h o u s i n g . data , h o u s i n g . t a r g e t # T a r g e t : Median h o u s e v a l u e ( $100k )
# 2 . T r a i n−t e s t s p l i t (80% t r a i n , 20% t e s t )
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 0 , r a n d o m s t a t e =42
)
# 3 . I n s t a n t i a t e and f i t Random F o r e s t R e g r e s s o r
r e g = R a n d o m F o r e s t R e g r e s s o r ( n e s t i m a t o r s =100 , r a n d o m s t a t e =42)
reg . f i t ( X train , y t r a i n )
# 4 . P r e d i c t and e v a l u a t e c o n t i n u o u s t a r g e t p e r f o r m a n c e
y pred = reg . p r e d i c t ( X test )
rmse = np . s q r t ( m e a n s q u a r e d e r r o r ( y t e s t , y p r e d ) )
mae = m e a n a b s o l u t e e r r o r ( y t e s t , y p r e d )
r2 = r 2 s c o r e ( y test , y pred )
p r i n t ( f ” Root Mean S q u a r e d E r r o r (RMSE ) : ${rmse ∗ 1 0 0 0 0 0 : . 2 f }” )
p r i n t ( f ”Mean A b s o l u t e E r r o r (MAE) : ${mae ∗ 1 0 0 0 0 0 : . 2 f }” )
p r i n t ( f ”R−s q u a r e d S c o r e ( R2 ) : { r 2 : . 4 f }” )
Computer Engineering

DI04000061: ML

4th Semester

296 / 455

Python Implementation: Real-World Regression Pipeline II

Computer Engineering

DI04000061: ML

4th Semester

297 / 455

Summary & Key Takeaways I

Core Concepts Covered
Classification Paradigm: Supervised learning task for mapping continuous inputs x ∈ Rd to discrete categorical targets y ∈ {c1 , . . . , cK }.
Model Taxonomy:

Discriminative (P(Y | X ) directly) vs. Generative (P(X | Y )P(Y ) via Bayes’ Rule).
Binary, Multiclass, and Multi-Label task formulations.
Real-World Regression Applications: Continuous value estimation spans finance (CAPM, house pricing), medicine (blood pressure, hospital LOS), and
engineering (RUL, load forecasting).
Practical Implementation: Scikit-learn provides unified interfaces for training, evaluating, and deploying both classification and regression models.

Computer Engineering

DI04000061: ML

4th Semester

298 / 455

Introduction to Classification I

What is Classification?
Classification is a core supervised machine learning task where the objective is to learn a decision mapping function f : X → Y from an input feature space
X ⊆ Rd to a discrete categorical label space Y.

Labeled Training Dataset: D = {(x (1) , y (1) ), (x (2) , y (2) ), . . . , (x (N) , y (N) )}, where:

(i)

(i)

(i)

x (i) = [x1 , x2 , . . . , xd ]T ∈ Rd denotes the d-dimensional feature vector.
y (i) ∈ C = {c1 , c2 , . . . , cK } represents the discrete target class label.
Primary Goal: Accurately predict the discrete class label ŷ ∈ C for novel, unseen input feature vectors xnew .
Classification vs. Regression:

Regression: Predicts continuous outputs (y ∈ R).
Classification: Predicts qualitative, discrete categories (y ∈ {c1 , . . . , cK }).

Computer Engineering

DI04000061: ML

4th Semester

299 / 455

Taxonomy of Classification Problems I
Binary Classification

Multi-label Classification

Target Space: Y ∈ {0, 1} or {−1, +1}.

Target Space: y ∈ {0, 1}K .

Two mutually exclusive outcomes (Positive vs. Negative class).

Each instance can simultaneously belong to multiple classes.

Examples:

Examples: Image tag generation (”cat”, ”outdoor”, ”daylight”),
Article topic categorization.

Email classification (Spam /
Non-Spam)
Medical diagnosis (Disease / Healthy)
Credit risk (Default / Non-Default)

Structured Output Classification
Target is a structured entity (sequence, tree, or graph).
Examples: Part-of-Speech (POS) tagging in Natural Language
Processing.

Multi-class Classification
Target Space: Y ∈ {1, 2, . . . , K } with K > 2.
Each instance belongs to exactly one category.
Examples: Optical Character Recognition (digits 0 − 9), Iris species
classification.

Computer Engineering

DI04000061: ML

4th Semester

300 / 455

Decision Boundaries & Mathematical Formalization I

Decision Surface / Boundary
A decision boundary is a hyper-surface in feature space Rd partitioning regions assigned to different target classes.
d

Sij = {x ∈ R | P(Y = ci | X = x) = P(Y = cj | X = x)}
Linear Decision Boundaries: Separated by a linear hyperplane (w T x + b = 0). Algorithms include Logistic Regression and Linear Support Vector
Machines (SVM).
Non-linear Decision Boundaries: Complex surfaces constructed by non-linear models such as Decision Trees, Kernel SVMs, and Neural Networks.
Decision Rule (Bayes Optimal Classifier):

∗

h (x) = arg max P(Y = ck | X = x)
ck ∈C

0-1 Loss Metric: L0−1 (y , ŷ ) = I(y ̸= ŷ ) penalizes any misclassification with unit cost.

Computer Engineering

DI04000061: ML

4th Semester

301 / 455

Key Performance Metrics for Classification I
The Binary Confusion Matrix
Actual Positive (y = 1)
Actual Negative (y = 0)

Predicted Positive (ŷ = 1)
True Positive (TP)
False Positive (FP)

Accuracy & Precision

Recall & F1-Score
Recall (Sensitivity): Positive coverage

Accuracy: Proportion of correct predictions

Acc =

TP + TN

Rec =

TP + TN + FP + FN

Prec =

TP
TP + FN

F1-Score: Harmonic mean

Precision: Fidelity of positive calls

Computer Engineering

Predicted Negative (ŷ = 0)
False Negative (FN)
True Negative (TN)

TP

F1 = 2 ·

TP + FP

DI04000061: ML

Prec · Rec
Prec + Rec

4th Semester

302 / 455

Python Implementation: Building a Classifier I
i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t m a k e c l a s s i f i c a t i o n
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
from s k l e a r n . l i n e a r m o d e l i m p o r t L o g i s t i c R e g r e s s i o n
from s k l e a r n . m e t r i c s i m p o r t a c c u r a c y s c o r e , c l a s s i f i c a t i o n r e p o r t
# 1. Synthesize binary c l a s s i f i c a t i o n dataset
X, y = m a k e c l a s s i f i c a t i o n (
n s a m p l e s =500 , n f e a t u r e s =4 , n c l a s s e s =2 , r a n d o m s t a t e =42
)
# 2 . T r a i n−t e s t s p l i t (80% t r a i n , 20% t e s t )
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42
)
# 3 . I n i t i a l i z e and t r a i n b i n a r y L o g i s t i c R e g r e s s i o n
model = L o g i s t i c R e g r e s s i o n ( )
model . f i t ( X t r a i n , y t r a i n )

classifier

# 4 . I n f e r e n c e and p e r f o r m a n c e e v a l u a t i o n
y p r e d = model . p r e d i c t ( X t e s t )
p r i n t ( f ” A c c u r a c y : { a c c u r a c y s c o r e ( y t e s t , y p r e d ) : . 4 f }” )
print ( c l a s s i f i c a t i o n r e p o r t ( y test , y pred ))

Computer Engineering

DI04000061: ML

4th Semester

303 / 455

Real-World Applications & Core Takeaways I

Real-World Applications of Classification
Healthcare & Medicine: Tumor malignancy diagnosis from diagnostic imaging.
Computer Vision: Facial identification, autonomous vehicle traffic sign recognition.
Natural Language Processing: Spam filtering, customer sentiment analysis.
Finance & Banking: Credit card transaction fraud detection and credit score rating.

Summary of Key Concepts
Definition: Learning a mapping f : X → Y from feature vectors to categorical labels.
Decision Boundary: Geometry separating class regions in input space.
Metric Selection: Tailor metrics (Accuracy vs. Precision/Recall/F1) to handle class imbalance and asymmetric error costs.

Computer Engineering

DI04000061: ML

4th Semester

304 / 455

Overview of Classification Tasks I

Classification Definition
Classification is a supervised learning paradigm where the goal is to learn a mapping function f : X → Y from feature vectors x ∈ Rd to categorical labels y ∈ Y.

Binary Classification

Multi-class Classification

Label Space: Y = {0, 1} or Y = {−1, +1}.

Label Space: Y = {1, 2, . . . , K } where K > 2.

Goal: Partition feature space into two mutually exclusive regions.

Goal: Assign each sample to exactly one of K discrete categories.

Examples: Spam vs. Non-Spam, Tumor Benign vs. Malignant.

Examples: Handwritten digit recognition (0 − 9), Crop disease
identification.

Key Distinction
In multi-class classification, classes are mutually exclusive (each instance belongs to exactly one class). This differs from multi-label classification where an
instance can belong to multiple classes simultaneously.

Computer Engineering

DI04000061: ML

4th Semester

305 / 455

Binary Classification: Mathematical Framework I

Probabilistic Output & Hypothesis Space
For a binary target y ∈ {0, 1}, a probabilistic binary classifier models the posterior distribution P(Y = 1 | x).
Model Hypothesis: p̂(x) = σ(f (x)) =

1
1+e −f (x)

∈ [0, 1].

Decision Rule:

(
ŷ = h(x) =

1
0

if P(Y = 1 | x) ≥ τ
if P(Y = 1 | x) < τ

(default threshold τ = 0.5)

Binary Loss Function: Log Loss / Cross-Entropy
Given N samples, parameters are optimized by minimizing the Binary Cross-Entropy (BCE) loss:

LBCE (w ) = −

Computer Engineering

N
1 X

N i=1

[yi log(p̂i ) + (1 − yi ) log(1 − p̂i )]

DI04000061: ML

4th Semester

306 / 455

Binary Classification: Decision Boundaries I

Linear vs. Non-linear Boundaries
The decision boundary is the hypersurface in Rd where the classifier is indifferent between classes (p̂(x) = 0.5 ⇐⇒ f (x) = 0).

Linear Decision Boundary

Non-linear Decision Boundary

T

Form: w x + b = 0.

Form: Complex surface in feature space.

Separates classes using a straight line (d = 2) or hyperplane (d ≥ 3).

Captures non-linear dependencies without manual feature engineering.

Algorithms: Logistic Regression, Linear SVM, Perceptron.

Algorithms: Kernel SVM, Decision Trees, Neural Networks, k-NN.

Computer Engineering

DI04000061: ML

4th Semester

307 / 455

Multi-class Classification: Problem Setup I
Target Representation & Softmax Extension
For K > 2 classes, targets are represented using one-hot encoding: yi ∈ {0, 1}K where

PK

k=1 yik = 1.

Softmax Activation Function
To output a valid probability distribution over K classes from raw logits z = [z1 , z2 , . . . , zK ]T :

P(Y = k | x) = σ(z)k = P

e zk

K e zj
j=1

such that

K
X

P(Y = k | x) = 1

k=1

Categorical Cross-Entropy Loss
Optimization objective across N training examples:
LCCE (W ) = −

Computer Engineering

N X
K
1 X

N i=1 k=1

yik log(p̂ik )

DI04000061: ML

4th Semester

308 / 455

Decomposing Multi-class: One-vs-Rest (OvR) I
Strategy Principle
Also known as One-vs-All (OvA). Reduces a K -class problem into K independent binary classification tasks.

Training Phase

Inference Phase
Pass test sample x through all K classifiers.

Train K separate binary classifiers f1 , f2 , . . . , fK .

Assign label with the maximum confidence score:
For classifier fk :

Class k samples → Positive (+1).
All other K − 1 classes → Negative (0).

ŷ = arg

max

k∈{1,...,K }

fk (x)

Limitation of OvR
Can introduce severe class imbalance during binary training (1 vs. K − 1 ratio) and scale calibration mismatches across unstandardized classifiers.

Computer Engineering

DI04000061: ML

4th Semester

309 / 455

Decomposing Multi-class: One-vs-One (OvO) I
Strategy Principle
Constructs a dedicated binary classifier for every unique pair of classes, completely isolating pair-wise decision boundaries.

Model Complexity

Voting & Decision

Number of binary classifiers:

Each classifier fij votes for either class i or class j.
Final prediction uses majority voting:

M =

 
K
2

=

K (K − 1)
2

ŷ = arg max
k

X

I(fij (x) = k)

i<j

For K = 10 classes ⇒ 45 classifiers.
For K = 100 classes ⇒ 4, 950 classifiers.

Advantage
Each classifier is trained on a small subset of data (only instances belonging to classes i and j), making training computationally efficient per classifier.

Computer Engineering

DI04000061: ML

4th Semester

310 / 455

Native Multi-class Classification Algorithms I
Direct Multi-class Modeling
Instead of binary decomposition wrappers (OvR/OvO), several algorithms natively handle K > 2 classes within their mathematical formulation.
Multinomial Logistic Regression Uses vector-valued linear outputs with Softmax normalization and cross-entropy optimization.
Decision Trees & Random Forests Calculates split criteria (Gini Impurity, Information Gain) over categorical target distribution across all K classes:

Gini(p) = 1 −

K
X
2
pk
k=1

k-Nearest Neighbors (k-NN) Computes class frequencies among the k nearest neighbors and assigns the mode class.
Naive Bayes Computes joint class probabilities using Bayes’ theorem:

P(Y = k | x) ∝ P(Y = k)

d
Y

P(xj | Y = k)

j=1

Computer Engineering

DI04000061: ML

4th Semester

311 / 455

Evaluation Metrics: Multi-class Averaging I

Extending Binary Metrics (Precision, Recall, F1)
In multi-class settings, per-class metrics must be aggregated to provide a global performance summary.

Macro-Averaging

Micro & Weighted Averaging

Unweighted mean across all classes:

Micro: Aggregates total TP, FP, FN across all classes first.
Weighted: Class-frequency weighted mean:

F1macro =

X
1 K
K k=1

F1k
F1weighted =

K
X
Nk
k=1

N

F1k

Treats all classes equally; highlights performance on minority classes.

Computer Engineering

DI04000061: ML

4th Semester

312 / 455

Python Implementation: Scikit-Learn Pipelines I
i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t m a k e c l a s s i f i c a t i o n
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
from s k l e a r n . l i n e a r m o d e l i m p o r t L o g i s t i c R e g r e s s i o n
from s k l e a r n . m u l t i c l a s s i m p o r t O n e V s R e s t C l a s s i f i e r , O n e V s O n e C l a s s i f i e r
from s k l e a r n . m e t r i c s i m p o r t c l a s s i f i c a t i o n r e p o r t
# G e n e r a t e s y n t h e t i c 4− c l a s s c l a s s i f i c a t i o n d a t a s e t
X , y = m a k e c l a s s i f i c a t i o n ( n s a m p l e s =1000 , n f e a t u r e s =10 ,
n c l a s s e s =4 , n i n f o r m a t i v e =6 , r a n d o m s t a t e =42)
X t r , X t e , y t r , y t e = t r a i n t e s t s p l i t (X , y , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42)
# 1 . N a t i v e Softmax M u l t i−c l a s s R e g r e s s i o n
c l f s o f t m a x = L o g i s t i c R e g r e s s i o n ( m u l t i c l a s s= ’ m u l t i n o m i a l ’ , s o l v e r= ’ l b f g s ’ )
c l f s o f t m a x . f i t ( X tr , y t r )
# 2 . E x p l i c i t One−vs−R e s t Meta−e s t i m a t o r
clf ovr = OneVsRestClassifier ( LogisticRegression ())
c l f o v r . f i t ( X tr , y t r )
# 3 . E x p l i c i t One−vs−One Meta−e s t i m a t o r
clf ovo = OneVsOneClassifier ( LogisticRegression ())
c l f o v o . f i t ( X tr , y t r )
# Evaluation
p r i n t ( c l a s s i f i c a t i o n r e p o r t ( y te , c l f s o f t m a x . p r e d i c t ( X te ) ) )

Computer Engineering

DI04000061: ML

4th Semester

313 / 455

Comparative Summary: Multi-class Strategies I

Methodology Comparison Matrix
Property
No. of Models
Sub-dataset Size
Train Time Complexity
Output Type
Imbalance Sensitivity
Best Suited For

Computer Engineering

One-vs-Rest (OvR)
K
Full dataset (N)
O(K · Tbinary )
Independent scores
High (1 vs K − 1)
Fast baseline, K ≤ 10

DI04000061: ML

One-vs-One (OvO)
K (K −1)
2
Subset (Ni + Nj )
O(K 2 · Tsub )

Pairwise votes
Low (Pairwise)
SVMs, Kernels

Native (Softmax)
1
Full dataset (N)
Directly optimized
Calibrated probs
Handled via Loss
Neural Nets, Trees

4th Semester

314 / 455

Lecture Overview & Learning Objectives I

Course: DI04000061 (Introduction Machine Learning)
Unit 4: Supervised Machine Learning
Topic 4.2.3: k-Nearest Neighbor (kNN) Algorithm
Learning Objectives:
1

Understand the core concepts of Instance-Based Learning and Lazy Learning.

2

Master the mathematical formulation of distance metrics (Euclidean, Manhattan, Minkowski, Cosine).

3

Understand the step-by-step working mechanism of k-NN for classification and regression.

4

Analyze hyperparameter selection (k) and its impact on the Bias-Variance Tradeoff.

5

Evaluate the main advantages, limitations, computational complexity, and mitigation strategies (KD-Tree, Ball-Tree).

6

Implement a complete k-NN pipeline in Python using scikit-learn.

Computer Engineering

DI04000061: ML

4th Semester

315 / 455

Introduction to k-Nearest Neighbor (kNN) I

What is k-NN?
The k-Nearest Neighbor (k-NN) algorithm is a non-parametric, instance-based supervised learning method used for both classification and regression.

Core Intuition: ”Birds of a feather flock together.” Data points that are close to each other in feature space are likely to belong to the same class or have
similar target values.
Instance-Based Learning: The algorithm does not construct an explicit internal model or target function during training; instead, it stores the entire
training dataset.
d
Formal Setup: Given a dataset D = {(xi , yi )}N
i=1 where xi ∈ R and yi ∈ Y, for a query point x0 , k-NN finds the set Nk (x0 ) of k closest training
points.

Computer Engineering

DI04000061: ML

4th Semester

316 / 455

Need for k-NN: Lazy vs. Eager Learning I

Eager Learning Paradigm

Lazy Learning (k-NN) Paradigm

Examples: Decision Trees, Logistic Regression, Neural Networks.
Training Phase: Construct an explicit generalization model f (x; w )
before receiving test queries. High training time complexity
O(epochs · N · d).
Inference Phase: Fast prediction O(d) by evaluating f (x0 ). Training
data can be discarded.

Computer Engineering

DI04000061: ML

Training Phase: Zero training work (O(1) time). Simply store
training instances D in memory.
Inference Phase: Computation is deferred to prediction time.
Evaluates distance to all stored points (O(N · d)).
Non-Parametric: No prior assumptions about the probability
distribution of data.

4th Semester

317 / 455

Distance Metrics in k-NN I
Measuring Proximity in Feature Space
The performance of k-NN depends heavily on the choice of distance metric d(x, z) between vectors x, z ∈ Rd :
Euclidean Distance (L2 Norm): Most common for continuous variables.
v
u
d
uX
u
d2 (x, z) = t
(xj − zj )2 = ∥x − z∥2
j=1

Manhattan Distance (L1 Norm / City-Block): Preferred for high-dimensional or grid-like data.

d1 (x, z) =

d
X

|xj − zj | = ∥x − z∥1

j=1

Minkowski Distance (Lp Norm Generalized):

dp (x, z) = 

d
X

1
p

|xj − zj | 

p

(p = 1 ⇒ Manhattan, p = 2 ⇒ Euclidean)

j=1

Computer Engineering

DI04000061: ML

4th Semester

318 / 455

Distance Metrics in k-NN II

Cosine Distance: Measures orientation angle regardless of magnitude (useful for text vectors).

dcos (x, z) = 1 −

Computer Engineering

x ·z
∥x∥2 ∥z∥2

DI04000061: ML

4th Semester

319 / 455

Working Mechanism of k-NN I
Algorithmic Workflow for Query Point x0
1

Parameter Choice: Select integer k ≥ 1 and distance metric d(·, ·).

2

Distance Computation: Compute distance d(x0 , xi ) for all i = 1, 2, . . . , N.

3

Neighbor Identification: Identify the set Nk (x0 ) containing the k training points with smallest distances.

4

Target Prediction:

Majority Vote (Classification):
X

ŷ0 = arg max
c∈Y

I(yi = c)

i∈Nk (x0 )

Distance-Weighted Majority Vote: Assign higher weights wi = d(x0 ,x1i )2 +ϵ to closer
neighbors.
Mean Average (Regression):
1 X
ŷ0 =
yi
k
i∈Nk (x0 )

Computer Engineering

DI04000061: ML

4th Semester

320 / 455

Selecting the Value of k: Hyperparameter Tuning I
Impact of k on Model Capacity & Decision Surface
Large k (e.g., k → N)

Small k (e.g., k = 1)
Highly flexible, complex decision boundary.

Smooth, overly simplified decision boundary.

Low Bias, High Variance.

High Bias, Low Variance.

Sensitive to noise, outliers, and mislabeled data.

Predicts majority class everywhere as k → N.

Risk of Overfitting.

Risk of Underfitting.

Best Practices for Choosing k
Use an odd value for k in binary classification to prevent tie votes.
√
Rule of thumb starter value: k = ⌊ N⌋.
Grid Search with Cross-Validation: Select k that minimizes validation error across candidate values k ∈ {1, 3, 5, 7, . . . , 25}.

Computer Engineering

DI04000061: ML

4th Semester

321 / 455

Critical Requirement: Feature Scaling I
Sensitivity to Unscaled Features
Because k-NN relies strictly on geometric distances, features with larger numeric ranges dominate distance calculations, rendering small-scale features irrelevant.

Example Illustrating Feature Dominance
Consider two features: Income ($20, 000 − $150, 000) vs. Age (18 − 65 years). Without scaling, a difference of $1, 000 in income completely overwhelms a
40-year difference in age during Euclidean distance evaluation.

Scaling Solutions
Standardization (Z-Score Normalization): (Recommended when data is Gaussian)

xstd =

x −µ
σ

⇒

µ = 0, σ = 1

Min-Max Normalization: (Scales values strictly to [0, 1])
xnorm =

Computer Engineering

x − xmin
xmax − xmin

DI04000061: ML

4th Semester

322 / 455

Advantages of k-NN Algorithm I

Key Strengths
1

Simplicity & Intuition: Very easy to understand, explain, and implement with minimal mathematical complexity.

2

Zero Training Cost: Instant training step (O(1)) since data points are stored as-is without optimization loops.

3

No Distributional Assumptions: Non-parametric algorithm that naturally handles arbitrary multi-modal distributions and complex decision surfaces.

4

Natural Multi-Class Extension: Inherently supports multi-class classification and multi-output regression without requiring One-vs-Rest wrappers.

5

Dynamic Data Updates: New training samples can be continuously added to memory without retraining the model.

Computer Engineering

DI04000061: ML

4th Semester

323 / 455

Disadvantages & Practical Challenges I

Key Limitations
1

High Computational Overhead at Prediction: Searching N points in d dimensions takes O(N · d) time per test sample. Prohibitive for real-time
applications on large datasets.

2

High Memory Consumption: Must store entire dataset in RAM (O(N · d) space complexity).

3

Curse of Dimensionality: In high-dimensional spaces (d ≫ 10), the distance between any pair of points converges to nearly the same value, causing loss
of discriminative power.

4

Sensitivity to Outliers & Class Imbalance: Outliers can skew predictions; majority classes dominate random voting.

5

Sensitivity to Irrelevant Features: Uninformative features corrupt distance metrics unless pruned or scaled.

Computer Engineering

DI04000061: ML

4th Semester

324 / 455

Computational Efficiency: Indexing Structures I

Accelerating Neighbor Search beyond Brute-Force O(N · d)
To avoid computing distance to all N samples, specialized tree structures partition space hierarchically:

KD-Tree (k-dimensional Tree)

Ball Tree

Recursively splits data along axis-aligned hyperplanes.

Partitions data into nested hyperspheres (”balls”).

Search Complexity: Average O(d · log N).

Search Complexity: Efficient in higher dimensions (d > 20) where
axis-aligned splits fail.

Performance degrades when dimension d > 20 (falls back to
brute-force speed).

Computer Engineering

More expensive to construct than KD-Tree.

DI04000061: ML

4th Semester

325 / 455

Python Implementation: Complete k-NN Pipeline I
i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t m a k e c l a s s i f i c a t i o n
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t , G r i d S e a r c h C V
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r
from s k l e a r n . n e i g h b o r s i m p o r t K N e i g h b o r s C l a s s i f i e r
from s k l e a r n . p i p e l i n e i m p o r t P i p e l i n e
# 1. Generate Synthetic C l a s s i f i c a t i o n Dataset
X , y = m a k e c l a s s i f i c a t i o n ( n s a m p l e s =500 , n f e a t u r e s =4 ,
n i n f o r m a t i v e =3 , r a n d o m s t a t e =42)
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42
)
# 2 . B u i l d P i p e l i n e ( S c a l i n g + k−NN C l a s s i f i e r )
pipeline = Pipeline ([
( ’ scaler ’ , StandardScaler ()) ,
( ’ knn ’ , K N e i g h b o r s C l a s s i f i e r ( m e t r i c= ’ m i n k o w s k i ’ , p =2)) # E u c l i d e a n
])
# 3 . H y p e r p a r a m e t e r Tuning f o r o p t i m a l ’ k ’ v i a 5−F o l d C r o s s V a l i d a t i o n
p a r a m g r i d = { ’ k n n n n e i g h b o r s ’ : l i s t ( r a n g e ( 1 , 2 1 , 2 ) ) } # Odd numbers 1 t o 19
g r i d = G r i d S e a r c h C V ( p i p e l i n e , p a r a m g r i d , c v =5 , s c o r i n g= ’ a c c u r a c y ’ )
grid . f i t ( X train , y t r a i n )
# 4 . E v a l u a t e B e s t Model
p r i n t ( f ” Optimal k : { g r i d . b e s t p a r a m s [ ’ k n n n n e i g h b o r s ’]} ” )
p r i n t ( f ” T e s t A c c u r a c y : { g r i d . s c o r e ( X t e s t , y t e s t ) : . 4 f }” )
Computer Engineering

DI04000061: ML

4th Semester

326 / 455

Python Implementation: Complete k-NN Pipeline II

Computer Engineering

DI04000061: ML

4th Semester

327 / 455

Summary & Practical Recommendations I

k-NN Algorithm Summary Table
Aspect
Model Type
Training Complexity
Prediction Complexity
Primary Preprocessing
Hyperparameter k
Best Suited For

Characteristics / Recommendation
Non-parametric, Instance-based, Lazy Learner
O(1) time, O(N · d) storage space
O(N · d) brute-force; O(d log N) with KD-Tree
Mandatory Feature Standardization / Normalization
Tune via Cross-Validation; use odd numbers for binary targets
Low-dimensional (d ≤ 20), small-to-medium dataset sizes

Golden Rule for Practical Deployment
Always pair k-NN with a feature scaling preprocessor (e.g. StandardScaler) and reduce dimensions (e.g. via PCA) if feature dimension d is large.

Computer Engineering

DI04000061: ML

4th Semester

328 / 455

Introduction to Support Vector Machines (SVM) I

What is a Support Vector Machine?
Support Vector Machines (SVMs) are powerful supervised learning algorithms used primarily for classification and regression. The core philosophy of SVM is to
construct an optimal decision hyperplane that maximizes the margin of separation between classes in feature space.

Key Characteristics

Why Maximize Margin?

Maximum Margin Classifier: Finds the decision boundary furthest
from closest samples.

Generalization: Larger margins provide maximum safety margin
against noise and unseen test instances.

Sparsity: Decision boundary depends strictly on a subset of samples
called Support Vectors.

Robustness: Implements structural risk minimization rather than
purely empirical risk.

Convexity: Solves a convex quadratic program guaranteed to reach
global optimum.

Stability: Prevents boundary from overfitting to isolated training
points.

Computer Engineering

DI04000061: ML

4th Semester

329 / 455

Geometric Foundations: Hyperplanes & Distance I
Hyperplane Geometry
In a d-dimensional feature space Rd , an affine hyperplane separating two classes is defined by:
T

f (x) = w x + b = 0
where w ∈ Rd is the weight vector normal (orthogonal) to the hyperplane, and b ∈ R is the bias offset.

Perpendicular Distance of a Point to Hyperplane
The signed Euclidean distance r from any arbitrary vector xi ∈ Rd to the decision plane w T x + b = 0 is:

r =

w T xi + b
∥w ∥2

Given binary targets yi ∈ {−1, +1}, the correct geometric distance ri from xi to the boundary is:

ri =

Computer Engineering

yi (w T xi + b)
∥w ∥2

DI04000061: ML

4th Semester

330 / 455

Functional vs. Geometric Margins I

Functional Margin

Geometric Margin

For a sample (xi , yi ), the functional margin is:

The geometric margin normalizes by the L2 norm of w :

T

γ̂i = yi (w xi + b)

γi =

Measures algebraic correctness and confidence.

yi (w T xi + b)
∥w ∥2

=

γ̂i
∥w ∥2

Represents true geometric distance in feature space.

Scale Sensitivity: Scaling (w , b) → (αw , αb) inflates γ̂i arbitrarily
without shifting geometry.

Completely invariant to scaling parameters by α > 0.

Canonical Hyperplane Convention
To resolve scale ambiguity, we set the functional margin of the closest point(s) to 1: mini γ̂i = 1. Consequently, the geometric margin of closest points becomes
γ = ∥w1∥ .
2

Computer Engineering

DI04000061: ML

4th Semester

331 / 455

Hard-Margin SVM: Optimization Formulation I
Primal Optimization Problem
1
For a linearly separable dataset D = {(xi , yi )}N
i=1 with yi ∈ {−1, +1}, maximizing the margin ∥w ∥

min
w ,b

subject to

Convex Quadratic Program

1
2

2

is equivalent to minimizing 12 ∥w ∥22 :

2

∥w ∥2
T

yi (w xi + b) ≥ 1,

∀i = 1, . . . , N

Total Margin Width

Strictly convex quadratic objective function.

Positive boundary: w T x + b = +1.

N linear inequality constraints.

Negative boundary: w T x + b = −1.

Guaranteed single global optimum; solvable via QP solvers.

Total margin distance between boundary boundaries:

M =

Computer Engineering

DI04000061: ML

2
∥w ∥2

4th Semester

332 / 455

Support Vectors & Margin Geometry I

Definition of Support Vectors
Support Vectors are the subset of training examples xi that lie exactly on the marginal hyperplanes where constraint equality is met:
T

yi (w xi + b) = 1

Limitations of Hard-Margin

Properties of Support Vectors
Non-support vector points can be removed or moved without altering
(w ∗ , b ∗ ).
The model parameters w depend only on support vectors.

Extremely sensitive to single outliers or noise.
Strictly fails (infeasible QP) if dataset has any non-linear overlap.
Motivates soft-margin relaxation with slack variables.

Memory-efficient and fast decision boundary evaluations.

Computer Engineering

DI04000061: ML

4th Semester

333 / 455

Soft-Margin SVM: Handling Non-Separable Data I

Slack Variables (ξi )
To handle non-linearly separable data, we introduce non-negative slack variables ξi ≥ 0 allowing samples to violate the strict margin constraint:
ξi = 0: Point lies outside or directly on the correct margin (yi f (xi ) ≥ 1).
0 < ξi ≤ 1: Point is within the margin but correctly classified (0 ≤ yi f (xi ) < 1).
ξi > 1: Point is misclassified on the wrong side of hyperplane (yi f (xi ) < 0).

Soft-Margin Primal Formulation
1

min

w ,b,ξ

subject to

2

2

∥w ∥2 + C

N
X

ξi

i=1
T

yi (w xi + b) ≥ 1 − ξi ,

ξi ≥ 0,

∀i = 1, . . . , N

where C > 0 is the regularization trade-off hyperparameter.

Computer Engineering

DI04000061: ML

4th Semester

334 / 455

Hinge Loss Formulation I

Unconstrained Risk Minimization
By setting ξi = max (0, 1 − yi f (xi )), the Soft-Margin SVM problem can be rewritten as unconstrained loss minimization with L2 regularization:

min
w ,b

N
X



1
2
T
∥w ∥2
max 0, 1 − yi (w xi + b) +
2C
i=1
{z
}
|
Hinge Loss Sum

Hinge Loss Function
LHinge (z) = max(0, 1 − z)

Sparsity vs. Log-Loss
Log Loss (Logistic Regression) is strictly positive everywhere ⇒ every
sample affects w .

where z = yi f (xi )

Hinge Loss is zero for z ≥ 1 ⇒ leads to sparse support vectors.
Zero penalty for points with margin score z ≥ 1.
Linear penalty for margin score z < 1.

Computer Engineering

DI04000061: ML

4th Semester

335 / 455

Hyperparameter C & Feature Scaling I

Role of Hyperparameter C
Hyperparameter C dictates the trade-off between maximizing margin width and penalizing slack errors

P

ξi :

Large C (Hard Margin Behavior)

Small C (Soft Margin Behavior)

Heavy penalty on violations ⇒ narrow margin.

Light penalty on violations ⇒ wider margin.

Fits training data aggressively.

Allows more margin errors for better smoothness.

Low bias, high variance (risk of overfitting).

High bias, low variance (risk of underfitting).

Crucial Necessity: Feature Standardization
Because SVM computes distance ∥w ∥2 , unscaled features with large numerical ranges will dominate distance calculations and distort the margin. Always apply
StandardScaler before fitting SVMs!

Computer Engineering

DI04000061: ML

4th Semester

336 / 455

Python Implementation: Linear SVM Pipeline I
i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t m a k e c l a s s i f i c a t i o n
from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r
from s k l e a r n . p i p e l i n e i m p o r t m a k e p i p e l i n e
from s k l e a r n . svm i m p o r t SVC
from s k l e a r n . m e t r i c s i m p o r t c l a s s i f i c a t i o n r e p o r t , a c c u r a c y s c o r e
# 1. Generate s y n t h e t i c binary c l a s s i f i c a t i o n dataset
X , y = m a k e c l a s s i f i c a t i o n ( n s a m p l e s =500 , n f e a t u r e s =4 , n i n f o r m a t i v e =2 ,
n r e d u n d a n t =0 , n c l u s t e r s p e r c l a s s =1 ,
c l a s s s e p = 1 . 5 , r a n d o m s t a t e =42)
# Map l a b e l s {0 , 1} t o {−1, +1} s t a n d a r d SVM f o r m a t
y svm = np . w h e r e ( y == 0 , −1, 1 )
# 2 . Train / Test s p l i t
X train , X test , y train , y t e s t = t r a i n t e s t s p l i t (
X , y svm , t e s t s i z e = 0 . 2 , r a n d o m s t a t e =42
)
# 3 . B u i l d p i p e l i n e : F e a t u r e S t a n d a r d i z a t i o n + L i n e a r SVC
svm pipeline = make pipeline (
StandardScaler () ,
SVC( k e r n e l= ’ l i n e a r ’ , C= 1 . 0 , r a n d o m s t a t e =42)
)
svm pipeline . f i t ( X train , y t r a i n )

Computer Engineering

DI04000061: ML

4th Semester

337 / 455

Python Implementation: Linear SVM Pipeline II

# 4 . Model E v a l u a t i o n
y pred = svm pipeline . predict ( X test )
p r i n t ( f ” T e s t A c c u r a c y : { a c c u r a c y s c o r e ( y t e s t , y p r e d ) : . 4 f }” )
print ( c l a s s i f i c a t i o n r e p o r t ( y test , y pred ))

Computer Engineering

DI04000061: ML

4th Semester

338 / 455

Extracting Decision Boundaries & Support Vectors I
# E x t r a c t p i p e l i n e components
s c a l e r = s v m p i p e l i n e . named steps [ ’ s t a n d a r d s c a l e r ’ ]
model = s v m p i p e l i n e . n a m e d s t e p s [ ’ s v c ’ ]
# A c c e s s h y p e r p l a n e p a r a m e t e r s : wˆT x + b = 0
w = model . c o e f [ 0 ]
# Weight v e c t o r n o r m a l t o b o u n d a r y
b = model . i n t e r c e p t [ 0 ]
# Bias o f f s e t
# Extract support vector p r o p e r t i e s
s v i n d i c e s = model . s u p p o r t
s u p p o r t v e c t o r s = model . s u p p o r t v e c t o r s
n s v p e r c l a s s = model . n s u p p o r t
p r i n t ( f ” Weight v e c t o r (w ) : {np . r o u n d (w , 4)} ” )
p r i n t ( f ” B i a s ( b ) : {b : . 4 f }” )
p r i n t ( f ” S u p p o r t V e c t o r C o u n t s ( c l a s s −1 / +1): { n s v p e r c l a s s }” )
# Compute e x a c t g e o m e t r i c m a r g i n M = 2 / | | w | |
w norm = np . l i n a l g . norm (w)
m a r g i n w i d t h = 2 . 0 / w norm
p r i n t ( f ” G e o m e t r i c Margin Width (M) : { m a r g i n w i d t h : . 4 f }” )
# C a l c u l a t e t e s t f u n c t i o n a l m a r g i n s y ∗ (wˆT x + b )
X test scaled = s c a l e r . transform ( X test )
s c o r e s = model . d e c i s i o n f u n c t i o n ( X t e s t s c a l e d )
functional margins = y test ∗ scores
p r i n t ( f ”Min T e s t F u n c t i o n a l Margin : { f u n c t i o n a l m a r g i n s . min ( ) : . 4 f }” )

Computer Engineering

DI04000061: ML

4th Semester

339 / 455

Linear SVM vs. Logistic Regression I
Methodological Comparison
Property
Loss Function
Optimization
Sparsity
Prediction
Outliers
Feature Scaling

Linear SVM
Hinge Loss max(0, 1 − yf (x))
Quadratic Programming (Convex)
Sparse (SVs only)
Margin / Distance Score
Robust (distant points ignored)
Essential

Logistic Regression
Log Loss log(1 + e −yf (x) )
Gradient Descent / L-BFGS
Dense (All instances affect w )
Calibrated Probability
Sensitive (all points pull boundary)
Highly Recommended

Key Takeaways
Linear SVM maximizes geometric margin M = ∥w2∥

2

to achieve strong generalization.

Soft-margin SVM uses slack variables (ξi ) and parameter C to handle non-separable data.
Hinge loss creates exact zeros for points outside the margin, resulting in a sparse support vector model.

Computer Engineering

DI04000061: ML

4th Semester

340 / 455

Introduction to Unsupervised Learning I

What is Unsupervised Learning?
(i)
Unlike Supervised Learning where dataset D = {(x (i) , y (i) )}N
i=1 contains ground-truth labels y , Unsupervised Learning operates on unlabeled data:

D = {x

(1)

,x

(2)

,...,x

(N)

},

x

(i)

d

∈R

The core objective is to uncover underlying patterns, structures, probability densities, or low-dimensional representations within D.

Key Paradigms

Modern Frontier: Generative AI

Clustering: Grouping similar instances into clusters S1 , . . . , SK .
Dimensionality Reduction: Projecting d-dim space to k-dim space
(k ≪ d).
Density Estimation: Modeling the data distribution p(x).

Computer Engineering

DI04000061: ML

Generative Modeling: Learning pθ (x) to sample novel synthetic data
xnew ∼ pθ (x).
Latent Representation: Mapping complex inputs into structured
latent spaces z ∈ Rk .

4th Semester

341 / 455

Taxonomy of Unsupervised & Generative Learning I

Mathematical Taxonomy
Unsupervised Learning problems can be broadly classified by their mathematical goals:
Partitioning & Grouping Minimize intra-cluster distance: arg min{S }

PK

k=1

k

P

2
x∈Sk ∥x − µk ∥ .

Manifold Learning Learn mapping z = f (x) preserving local/global geometry (e.g., PCA, t-SNE, Autoencoders).
PN
(i)
i=1 log pθ (x ) or minimize KL divergence DKL (pdata ∥ pθ ).

Probabilistic Modeling Maximize log-likelihood maxθ

The Generative AI Paradigm Shift
Classical unsupervised learning focuses on discriminating patterns (e.g., cluster assignment). Generative AI extends this to synthesis: learning latent variable
distribution p(z) and conditional generator pθ (x | z) to model marginal density:
Z
p(x) =

Computer Engineering

pθ (x | z)p(z) dz

DI04000061: ML

4th Semester

342 / 455

Overview of Classical vs. Generative Approaches I

2gray!10white

Attribute
Primary Goal
Output Type

Classical Unsupervised
Cluster, reduce dimensionality, detect
anomalies
Labels, feature scores, projection matrices

Key Algorithms

K -Means, PCA, GMM, DBSCAN, t-SNE

Latent Space

Geometric subspace (e.g., principal axes)

Computer Engineering

DI04000061: ML

Generative AI
Model data distribution p(x) and synthesize
samples
High-fidelity synthetic data,
embeddings,
text/image
VAEs, GANs, Diffusion Models, Autoregressive
Transformers
Continuous probabilistic distribution p(z)

4th Semester

343 / 455

Case Study Context: Port Decarbonization & Green Hydrogen I

Problem Statement: Maritime Port Decarbonization
Maritime ports are major emission hubs (CO2 , NOx , PM). Transitioning to Green Hydrogen (H2 ) requires:
1

Operational Profiling: Unsupervised clustering of vessel duty cycles (tugs, ferries, cargo vessels) from raw sensor streams without labels.

2

Generative Forecasting: Simulating adoption trajectories and forecasting port-wide emission reductions under stochastic operational profiles.

Stage 1: Duty Profile Discovery

Stage 2: Generative Forecasting

Apply unsupervised clustering (K -Means/GMM) on vessel operational
metrics: dwell time, peak power demand (kW), daily fuel burn rate (L/h).

Use generative scenario generation to forecast CO2 abatement under varying
adoption rates and green hydrogen production capacity.

Computer Engineering

DI04000061: ML

4th Semester

344 / 455

Case Study: Unsupervised Vessel Duty Cycle Clustering I
i m p o r t numpy a s np
i m p o r t p a n d a s a s pd
from s k l e a r n . c l u s t e r i m p o r t KMeans
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r
# S i m u l a t e d p o r t t e l e m e t r y : [ Peak Power (kW) , D a i l y D w e l l ( h r s ) , I d l e R a t i o ]
X raw = np . a r r a y ( [
[4500 , 14.2 , 0.65] , [4800 , 15.1 , 0.70] , [4200 , 13.8 , 0.62] , # Container
[1200 ,
4.5 , 0.20] , [1350 ,
5.0 , 0.25] , [1100 ,
4 . 1 , 0 . 1 8 ] , # Tugs
[ 800 , 2 2 . 0 , 0 . 8 5 ] , [ 850 , 2 1 . 5 , 0 . 8 8 ] , [ 790 , 2 3 . 1 , 0 . 8 2 ] # A u x i l i a r y
])
# Step 1 : Feature S t a n d a r d i z a t i o n
scaler = StandardScaler ()
X s c a l e d = s c a l e r . f i t t r a n s f o r m ( X raw )
# S t e p 2 : U n s u p e r v i s e d K−Means C l u s t e r i n g
kmeans = KMeans ( n c l u s t e r s =3 , r a n d o m s t a t e =42 , n i n i t =10)
c l u s t e r l a b e l s = kmeans . f i t p r e d i c t ( X s c a l e d )
p r i n t ( ” D i s c o v e r e d V e s s e l Duty C l u s t e r s : ” , c l u s t e r l a b e l s )
p r i n t ( ” C l u s t e r C e n t r o i d s ( S c a l e d ) : \ n” , kmeans . c l u s t e r c e n t e r s )

Computer Engineering

DI04000061: ML

4th Semester

345 / 455

Case Study: Mathematical Formulation for Emission Abatement I
Baseline Emission Model
For vessel cluster k ∈ {1, . . . , K }, baseline annual emissions Ek0 (metric tons CO2 /yr) are computed from cluster centroids:


peak
0
−6
Ek = Nk · Pk
· LFk · hk · EFdiesel × 10
peak

where Nk is vessel count, Pk

is peak power, LFk is load factor, hk is operational hours, and EFdiesel ≈ 690 g CO2 /kWh.

Generative Hydrogen Adoption & Emission Reduction
Let ηk (t) ∼ Beta(αk (t), βk (t)) be a generative stochastic adoption curve for cluster k at year t. The net port-wide emission reduction ∆E (t) is:

∆E (t) =

K
X
0
Ek · ηk (t) ·

1−

k=1

For green hydrogen produced via renewables, EFH2 ≈ 0, yielding ∆E (t) =

Computer Engineering

EFH2

!

EFdiesel

PK

0
k=1 Ek · ηk (t).

DI04000061: ML

4th Semester

346 / 455

Case Study: Generative Forecasting of Emission Reductions I
# G e n e r a t i v e s c e n a r i o s i m u l a t i o n f o r G r e e n H2 a d o p t i o n and CO2 f o r e c a s t i n g
np . random . s e e d ( 4 2 )
y e a r s = np . a r a n g e ( 2 0 2 5 , 2 0 3 6 )
n u m s i m u l a t i o n s = 1000
# B a s e l i n e a n n u a l e m i s s i o n s p e r c l u s t e r ( M e t r i c Tons CO2)
E b a s e l i n e = np . a r r a y ( [ 4 5 0 0 0 , 1 2 0 0 0 , 8 0 0 0 ] ) # C o n t a i n e r , Tugs , A u x i l i a r y
# S i m u l a t e s t o c h a s t i c a d o p t i o n t r a j e c t o r i e s o v e r 10 y e a r s
def g e n e r a t e s c e n a r i o s ( n sims , n y e a r s ) :
# G e n e r a t i v e t r a j e c t o r y u s i n g s i g m o i d w i t h random a d o p t i o n r a t e s
t = np . l i n s p a c e ( −3 , 3 , n y e a r s )
b a s e s i g m o i d = 1 / ( 1 + np . e xp(− t ) )
# Add G a u s s i a n l a t e n t n o i s e t o s i m u l a t e a d o p t i o n v a r i a n c e
n o i s e = np . random . n o r m a l ( 0 , 0 . 0 5 , s i z e =( n s i m s , n y e a r s ) )
t r a j e c t o r i e s = np . c l i p ( b a s e s i g m o i d + n o i s e , 0 , 1 )
return t r a j e c t o r i e s
adoption sims = g e n e r a t e s c e n a r i o s ( num simulations , len ( years ))
a n n u a l r e d u c t i o n s = a d o p t i o n s i m s ∗ np . sum ( E b a s e l i n e )
m e a n r e d u c t i o n = np . mean ( a n n u a l r e d u c t i o n s , a x i s =0)
c i 9 5 l o w e r = np . p e r c e n t i l e ( a n n u a l r e d u c t i o n s , 2 . 5 , a x i s =0)
c i 9 5 u p p e r = np . p e r c e n t i l e ( a n n u a l r e d u c t i o n s , 9 7 . 5 , a x i s =0)
p r i n t ( f ” Year 2030 E x p e c t e d CO2 Abatement : { m e a n r e d u c t i o n [ 5 ] : . 2 f } M e t r i c Tons ” )
p r i n t ( f ”95% C I : [ { c i 9 5 l o w e r [ 5 ] : . 2 f } , { c i 9 5 u p p e r [ 5 ] : . 2 f } ] ” )

Computer Engineering

DI04000061: ML

4th Semester

347 / 455

Summary & Key Takeaways I

Core Takeaways
Unsupervised Paradigm: Extracts latent patterns, density estimations, and clusters without human labeling.
Generative AI Evolution: Transitions from descriptive modeling to predictive scenario generation and probabilistic sample synthesis.
Real-World Impact: Applied to port decarbonization by discovering vessel duty clusters (K -Means) and generating stochastic green hydrogen adoption
trajectories for CO2 abatement forecasting.

Next Steps in Unit 5
In upcoming lectures, we will explore:
Lecture 5.2: Clustering Deep Dive (K -Means, Hierarchical Clustering, DBSCAN).
Lecture 5.3: Dimensionality Reduction & PCA.
Lecture 5.4: Generative AI Architectures (VAEs, GANs, Diffusion Models).

Computer Engineering

DI04000061: ML

4th Semester

348 / 455

The Unlabeled Data Explosion & Annotation Bottleneck I
The Real-World Data Paradox
In modern enterprise and scientific domain applications, raw, unannotated data is generated at petabyte scale continuously. However, human annotation is
exceptionally slow, expensive, and non-scalable.

Supervised Data Bottleneck

Unlabeled Data Abundance

High Cost: Requires expert annotators (e.g., radiologists, legal
scholars, domain experts).

Zero Marginal Cost: Server logs, satellite imagery, web text,
streaming sensor telemetry.

Human Fatigue & Error: Inter-annotator disagreement skews label
quality.

Continuous Ingestion: Streaming real-time data without human
intervention.

Scale Constraint: Less than 1% of global digital data contains
ground-truth labels y .

Massive Scale: Enables statistical learning of complex distributions
p(x).

Core Motivating Imperative
Supervised learning models fail to utilize over 99% of available world data. Unsupervised learning unlocks value directly from unannotated observations
X = {x1 , x2 , . . . , xN }.

Computer Engineering

DI04000061: ML

4th Semester

349 / 455

Mathematical Motivation: Density & Latent Modeling I
Shifting from Conditional p(y |x) to Data Density p(x)
While supervised learning estimates conditional boundaries p(y |x), unsupervised learning models the underlying data-generating distribution p(x) or maps data
into a structured latent space Z.

Marginal Density Integration

Optimization Comparison

Data points x ∈ Rd are generated via hidden latent factors z ∈ Rk :

Supervised Loss:

Z
p(x | z; θ)p(z)dz

p(x) =

min

Z

θ

Learning p(x) allows detection of low-density regions (anomalies) and
high-density regions (clusters).

X
1 N
N i=1

Unsupervised Likelihood:

max
θ

Computer Engineering

L(yi , f (xi ; θ))

DI04000061: ML

N
1 X

N i=1

log p(xi ; θ)

4th Semester

350 / 455

Four Primary Drivers for Unsupervised Learning I

Why Unsupervised Techniques are Essential in Modern ML
Unsupervised learning addresses core computational, economic, and operational challenges that supervised methods cannot solve.
1. Label Scarcity & Acquisition Costs Annotating specialized datasets (e.g., gene sequencing, seismic scans, pathology slides) requires thousands of
domain-expert hours, making supervised pipelines economically infeasible.
2. Discovery of Unknown Patterns Supervised models can only learn pre-defined class categories. Unsupervised models discover novel sub-populations, taxonomy
shifts, and emerging behavior patterns without prior human hypotheses.
3. Novelty & Zero-Day Anomaly Detection Fraudulent transactions, cyber intrusion vectors, and equipment failure modes are inherently rare or completely
unseen; supervised models lack target training labels yanomaly .
4. Feature Compression & Manifold Learning Real-world data resides in ultra-high dimensions (d ≫ 103 ). Unsupervised compression extracts compact latent
codes z ∈ Rk (k ≪ d) to defeat the curse of dimensionality.

Computer Engineering

DI04000061: ML

4th Semester

351 / 455

Use Case 1: Pattern Discovery & Customer Segmentation I

Unlocking Structure in Unlabeled Behavioral Data
Organizations collect user interaction logs without explicit tags describing behavioral personas. Unsupervised clustering groups instances based on geometric
similarity in feature space.

E-Commerce & Fintech

Precision Medicine

Feature Inputs: Recency, Frequency, Monetary value (RFM), dwell
time, click paths.

Feature Inputs: High-throughput RNA sequencing expression levels
(d > 20,000).

Outcome: Discover distinct buyer personas (e.g., impulse buyers,
bargain hunters, high-value loyalists) for targeted campaigns.

Outcome: Stratify patients into disease sub-types to tailor drug
therapies without requiring manual disease labels.

Geometric Criterion
Optimal partitions maximize inter-cluster distance Dbetween while minimizing intra-cluster variance Wwithin .

Computer Engineering

DI04000061: ML

4th Semester

352 / 455

Use Case 2: Anomaly & Outlier Detection I

The Extreme Class Imbalance Problem
In critical safety and security systems, anomalous events constitute less than 0.01% of total observations. Supervised classifiers suffer from catastrophic
false-negative rates due to severe class imbalance.

Financial Fraud Monitoring

Industrial IoT & Predictive Maintenance

Fraud tactics continuously adapt to bypass known supervised rules.
Unsupervised density models identify transactions residing in
low-density tails of p(x).

Machine breakdown data is rarely observed prior to catastrophic
failure.
Unsupervised models construct baseline normal operational
boundaries and trigger alerts upon deviation.

Zero-Day Vulnerability Defense
Because unsupervised anomaly detection models normal baseline behavior p(x), it successfully flags brand-new (zero-day) attack patterns that have never occurred
in history.

Computer Engineering

DI04000061: ML

4th Semester

353 / 455

Use Case 3: Dimensionality Reduction & Visualization I

Taming High-Dimensional Feature Spaces
High-dimensional datasets (d > 100) suffer from numerical instability, distance concentration effects, and severe computational overhead.

The Distance Concentration Effect

Benefits of Low-Dimensional Mapping

As dimensionality d → ∞, the ratio of maximum to minimum Euclidean
distance between any two random points approaches 1:

lim

dmax − dmin

d→∞

dmin

Visualization: Projecting d-dimensional features into R2 or R3 via
PCA/t-SNE for human inspection.
Noise Filtering: Discarding low-variance components removes
measurement noise.

=0

Downstream Acceleration: Reduces train time for downstream ML
models.

This renders nearest-neighbor metrics ineffective without dimensionality
reduction.

Computer Engineering

DI04000061: ML

4th Semester

354 / 455

Use Case 4: Self-Supervised Learning & Foundation Models I

Unsupervised Pre-training: The Bedrock of Generative AI
Modern Foundation Models (LLMs, Vision Transformers, Generative Diffusion Models) are trained using unsupervised / self-supervised objectives over massive
unannotated datasets.

Self-Supervised Pre-training

Downstream Fine-Tuning

Create pretext tasks from raw data (e.g., mask next token in text,
mask image patches).
Objective: Predict masked components using contextual
representations z.
Learns universal domain representations without any manual labels.

Transfer pre-trained weights to specific downstream tasks.
Requires only a small fraction (less than 1%) of labeled data
(Few-Shot / Zero-Shot learning).
Dramatically outperforms models trained strictly supervised from
scratch.

Generative Modeling Core
Generative AI models (GANs, VAEs, Diffusion) learn the true underlying distribution pdata (x) to sample completely new, realistic data instances xnew ∼ pmodel (x).

Computer Engineering

DI04000061: ML

4th Semester

355 / 455

Comparative Analysis: Learning Paradigms I

Systematic Comparison of Core Machine Learning Paradigms
Understanding when unsupervised learning is required relative to supervised and self-supervised approaches.

Dimension
Target Input
Primary Goal

Supervised
Feature X + Label y
Map X → y

Unsupervised
Raw Feature X only
Discover p(x), clusters, latent Z

Data Scale
Human Overhead
Primary Failure

Limited by label cost
Extremely High
Overfitting / Label Noise

Unlimited raw data
Zero / Minimal
Evaluation ambiguity

Computer Engineering

DI04000061: ML

Self-Supervised
Raw Feature X (Self-labeled)
Pre-train representations Z via pretext tasks
Unlimited raw data
Zero for pre-training
Pretext-task misalignment

4th Semester

356 / 455

Python Implementation: Feature Reduction & Anomaly Detection I
i m p o r t numpy a s np
i m p o r t p a n d a s a s pd
from s k l e a r n . d a t a s e t s i m p o r t m a k e b l o b s
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r
from s k l e a r n . d e c o m p o s i t i o n i m p o r t PCA
from s k l e a r n . e n s e m b l e i m p o r t I s o l a t i o n F o r e s t
# 1 . S i m u l a t e h i g h−d i m e n s i o n a l u n l a b e l e d o p e r a t i o n a l d a t a ( 1 0 0 0 s a m p l e s , 20 f e a t u r e s )
X raw ,
= m a k e b l o b s ( n s a m p l e s =950 , n f e a t u r e s =20 , c e n t e r s =3 , c l u s t e r s t d = 1 . 2 , r a n d o m s t a t e =42)
# Add 50 e x t r e m e u n l a b e l e d a n o m a l o u s p o i n t s ( z e r o −day e v e n t s )
a n o m a l i e s = np . random . u n i f o r m ( l ow =−10, h i g h =10 , s i z e =(50 , 2 0 ) )
X c o m p l e t e = np . v s t a c k ( [ X raw , a n o m a l i e s ] )
# 2 . P r e p r o c e s s & s t a n d a r d i z e f e a t u r e s ( Z e r o mean , u n i t v a r i a n c e )
scaler = StandardScaler ()
X scaled = s c a l e r . f i t t r a n s f o r m ( X complete )
# 3 . U n s u p e r v i s e d D i m e n s i o n a l i t y R e d u c t i o n v i a PCA ( R e t a i n 90% v a r i a n c e )
pca = PCA( n c o m p o n e n t s =0.90)
X pca = pca . f i t t r a n s f o r m ( X s c a l e d )
p r i n t ( f ” O r i g i n a l f e a t u r e s : { X s c a l e d . s h a p e [ 1 ] } −> Reduced f e a t u r e s : {X pca . s h a p e [ 1 ] } ” )
p r i n t ( f ” C u m u l a t i v e V a r i a n c e R a t i o : {np . sum ( pca . e x p l a i n e d v a r i a n c e r a t i o ) : . 4 f }” )
# 4 . U n s u p e r v i s e d Anomaly D e t e c t i o n u s i n g I s o l a t i o n F o r e s t
i s o f o r e s t = I s o l a t i o n F o r e s t ( c o n t a m i n a t i o n = 0 . 0 5 , r a n d o m s t a t e =42)
o u t l i e r p r e d i c t i o n s = i s o f o r e s t . f i t p r e d i c t ( X pca ) # −1 f o r anomaly , 1 f o r n o r m a l
n u m a n o m a l i e s = np . sum ( o u t l i e r p r e d i c t i o n s == −1)
Computer Engineering

DI04000061: ML

4th Semester

357 / 455

Python Implementation: Feature Reduction & Anomaly Detection II

p r i n t ( f ” D e t e c t e d U n s u p e r v i s e d O u t l i e r s : { n u m a n o m a l i e s } o u t o f { l e n ( X c o m p l e t e )} s a m p l e s ” )

Computer Engineering

DI04000061: ML

4th Semester

358 / 455

Python Implementation: Customer Segmentation Pipeline I
i m p o r t numpy a s np
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r
from s k l e a r n . c l u s t e r i m p o r t KMeans
from s k l e a r n . m e t r i c s i m p o r t s i l h o u e t t e s c o r e , c a l i n s k i h a r a b a s z s c o r e
# S i m u l a t e u n l a b e l e d c u s t o m e r b e h a v i o r a l t e l e m e t r y ( Recency , F r e q u e n c y , Monetary , D w e l l )
np . random . s e e d ( 4 2 )
t e l e m e t r y = np . random . r a n d n ( 6 0 0 , 4 ) ∗ np . a r r a y ( [ 1 5 , 5 , 2 0 0 , 1 2 ] ) + np . a r r a y ( [ 3 0 , 1 0 , 5 0 0 , 2 5 ] )
# Standardize feature matrix
scaler = StandardScaler ()
X norm = s c a l e r . f i t t r a n s f o r m ( t e l e m e t r y )
# E v a l u a t e o p t i m a l c l u s t e r c o u n t (K) w i t h o u t g r o u n d t r u t h l a b e l s
sil scores = []
k range = range (2 , 7)
for k in k range :
km = KMeans ( n c l u s t e r s=k , i n i t = ’ k−means++ ’ , n i n i t =10 , r a n d o m s t a t e =42)
l a b e l s = km . f i t p r e d i c t ( X norm )
s c o r e = s i l h o u e t t e s c o r e ( X norm , l a b e l s )
s i l s c o r e s . append ( s c o r e )
c h s c o r e = c a l i n s k i h a r a b a s z s c o r e ( X norm , l a b e l s )
p r i n t ( f ”K={k} | S i l h o u e t t e : { s c o r e : . 4 f } | C a l i n s k i −H a r a b a s z : { c h s c o r e : . 2 f }” )
o p t i m a l k = k r a n g e [ np . argmax ( s i l s c o r e s ) ]
p r i n t ( f ” O p t i m a l d i s c o v e r e d c u s t o m e r s e g m e n t s ( u n s u p e r v i s e d ) : K={o p t i m a l k }” )

Computer Engineering

DI04000061: ML

4th Semester

359 / 455

Strategic Framework: When to Deploy Unsupervised Learning I
Decision Protocol for ML Engineers
Use this workflow to determine whether unsupervised learning is the mandatory or optimal choice for a problem domain.

1

Is target annotation y unavailable or prohibitively expensive?

Yes: Deploy Unsupervised Clustering / Self-Supervised Representation Learning.
2

Is the objective to detect rare, unseen, or zero-day anomalies?

3

Is feature dimension d extremely large, causing model overfitting or high latency?

Yes: Deploy Unsupervised Density Estimation (Isolation Forest, GMMs, One-Class SVM).
Yes: Apply Unsupervised Dimensionality Reduction (PCA, UMAP, Autoencoders) as a
preprocessing module.
4

Is the objective to explore data structure without pre-conceived biases?

Yes: Perform Unsupervised Clustering and Manifold Visualization.

Computer Engineering

DI04000061: ML

4th Semester

360 / 455

End-to-End Operational Pipeline I
The Core Unsupervised Workflow
Unsupervised learning algorithms transform an unannotated raw feature matrix X ∈ RN×d into structured patterns, latent representations, or clusters without
target labels yi .

Stage 1: Ingestion & Preprocessing

Stage 3: Optimization & Partitioning

Data Normalization: Apply Z-score or MinMax scaling to prevent
high-magnitude features from dominating distance functions.

Iterative Optimization: Minimize Within-Cluster Sum of Squares
(WCSS) or maximize expected log-likelihood via EM.

Imputation & Cleaning: Handle missing attributes and remove severe
noise points.

Matrix Factorization: Extract top eigenvectors or singular values via
SVD/PCA.

Stage 2: Distance & Structure Matrix

Stage 4: Latent Mapping & Validation

Compute pairwise distance matrix D ∈ RN×N or covariance matrix
C ∈ Rd×d .
Construct similarity kernels or graph adjacency matrices for non-linear
structures.

Computer Engineering

DI04000061: ML

Project instances into low-dimensional space Z ∈ RN×k (k ≪ d).
Evaluate partition quality using internal metrics (Silhouette,
Davies-Bouldin).

4th Semester

361 / 455

Data Preprocessing & Scale Sensitivity I
Why Preprocessing is Critical in Unsupervised Learning
Unlike supervised learning where decision trees or target loss functions can compensate for scale differences, unsupervised algorithms rely strictly on geometric
metrics in Rd or covariance structures.

Min-Max Normalization

Standardization (Z-Score)

Scales feature range strictly into [0, 1]:

Transforms features to zero mean (µ = 0) and unit variance
(σ 2 = 1):
xij − µj
zij =
σj

′

xij =

Use Case: Essential for PCA, K -Means, Gaussian Mixture Models,
and gradient-based learning.

xij − xmin,j
xmax,j − xmin,j

Use Case: Preferred when distance bounds are fixed or preserving
zero values in sparse matrices is required.

Sensitivity to Outliers
Unsupervised algorithms lack ground-truth supervision to mask or ignore extreme values. Outliers distort mean vectors µk , shrink main clusters, and skew
principal eigenvectors.

Computer Engineering

DI04000061: ML

4th Semester

362 / 455

Distance & Similarity Metrics: The Algorithmic Engine I
Geometric Metrics in Feature Space Rd
The choice of distance metric defines the topology of the feature space and governs how similarity between unannotated data points is evaluated.
Minkowski Metric (Lp -norm) Generalized metric for feature vectors x, y ∈ Rd :

d(x, y) = 

d
X

1/p
p

|xj − yj | 

j=1

Euclidean (p = 2): Isotropic distance metric; assumes spherical clusters.
Manhattan (p = 1): Grid-based distance; robust to moderate high-dimensional noise.
Cosine Similarity & Distance Evaluates angular orientation rather than spatial magnitude:
Scos (x, y) =

x·y

,

∥x∥2 ∥y∥2

dcos (x, y) = 1 − Scos (x, y)

Used extensively in text mining and sparse document embeddings.
Mahalanobis Distance Scale-invariant metric accounting for feature correlation:
dM (x, y) =

q

(x − y)T Σ−1 (x − y)

Where Σ is the sample covariance matrix.
Computer Engineering

DI04000061: ML

4th Semester

363 / 455

Mathematical Formulation of Unsupervised Objectives I

Optimization Objectives Across Key Paradigms
Unsupervised algorithms operate by minimizing an objective function J over parameters Θ given data X.

1. Clustering: Within-Cluster Sum of Squares (WCSS / Inertia)
Given K cluster centers µ = {µ1 , . . . , µK } and cluster assignments C :

JWCSS (C , µ) =

K X
X

2

∥xi − µk ∥2 =

k=1 i∈Ck

N X
K
X

2

rik ∥xi − µk ∥2

i=1 k=1

where rik ∈ {0, 1} is a binary indicator of point i belonging to cluster k.

Computer Engineering

DI04000061: ML

4th Semester

364 / 455

Mathematical Formulation of Unsupervised Objectives II

2. Dimensionality Reduction: Reconstruction Loss Minimization
Finding an orthogonal projection matrix W ∈ Rd×k (k < d):

JPCA (W) =

X
1 N
N i=1

T

2

T

∥xi − WW xi ∥2 = Tr(ΣX ) − Tr(W ΣX W)

Minimizing reconstruction loss is mathematically equivalent to maximizing projected variance Tr(WT ΣX W).

Computer Engineering

DI04000061: ML

4th Semester

365 / 455

The Expectation-Maximization (EM) Framework I
General Probabilistic Latent Variable Optimization
When data generation depends on unobserved latent variables Z , direct maximum marginal likelihood estimation maxθ
intractable.

Expectation Step (E-Step)

Update parameter vector θ by maximizing the Q-function:
θ

(t)

)

Construct the expected complete-data log-likelihood (lower bound Q):

Q(θ | θ

(t)

)=

N
X

i=1 log p(xi | θ) is analytically

Maximization Step (M-Step)

Compute posterior probabilities over latent variable Z given current
parameters θ (t) :
qi (z) = P(Zi = z | xi ; θ

PN

(t+1)

= arg max Q(θ | θ

(t)

θ

)

Re-estimate cluster parameters (means µk , covariances Σk , and
mixture weights πk ).

Eqi [log p(xi , Zi | θ)]

i=1

Computer Engineering

DI04000061: ML

4th Semester

366 / 455

The Expectation-Maximization (EM) Framework II

Monotonic Convergence Property
Every iteration of the EM algorithm guarantees L(θ (t+1) ) ≥ L(θ (t) ), converging to a local log-likelihood maximum.

Computer Engineering

DI04000061: ML

4th Semester

367 / 455

Step-by-Step Mechanics: Centroid & Density Updating I

Iterative Partitioning Dynamics: Hard vs. Soft Assignment
Unsupervised iterative optimization proceeds by updating assignment scores and cluster parameters in alternating turns.

Hard Assignment (K -Means)

Soft Assignment (GMM - EM)
Assign: Compute responsibility γik ∈ [0, 1]:

Assign: Each point belongs to exactly one cluster center:
c

(i)

= arg

min

k∈{1,...,K }

2

πk N (xi | µk , Σk )
γik = P
K π N (x | µ , Σ )
i
j
j
j=1 j

∥xi − µk ∥2

Update: Centroid vector re-estimation:
Update: Soft mean update using Nk =
I(c (i) = k)xi
µk = Pi=1
N I(c (i) = k)
i=1

PN

i=1 γik :

PN

Computer Engineering

µk =

DI04000061: ML

N
1 X

Nk i=1

γik xi

4th Semester

368 / 455

Working of Dimensionality Reduction: SVD & Eigen-Decomposition I

Spectral Mechanics of Linear Representation Learning
Dimensionality reduction transforms high-dimensional features into a lower-dimensional orthogonal subspace that preserves maximal variance.

Covariance Eigen-Decomposition

Singular Value Decomposition (SVD)

Compute mean-centered matrix Xc = X − x̄.

Directly factorize mean-centered matrix Xc :

Form covariance matrix:

C=

1
N−1

Xc = UΣV
T
d×d
Xc Xc ∈ R

U ∈ RN×N : Left singular vectors (principal components scaled).
q
Σ ∈ RN×d : Singular values σj = (N − 1)λj .

Solve eigenvalue problem Cvj = λj vj .
Sort eigenvalues λ1 ≥ λ2 ≥ · · · ≥ λd to form projection matrix
Wk = [v1 , . . . , vk ].

Computer Engineering

T

DI04000061: ML

Transformed data: Zk = Xc Vk = Uk Σk ∈ RN×k .

4th Semester

369 / 455

Python Implementation: Complete Unsupervised Pipeline I
i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t m a k e b l o b s
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r
from s k l e a r n . d e c o m p o s i t i o n i m p o r t PCA
from s k l e a r n . c l u s t e r i m p o r t KMeans
from s k l e a r n . m e t r i c s i m p o r t s i l h o u e t t e s c o r e , d a v i e s b o u l d i n s c o r e
# 1 . G e n e r a t e s y n t h e t i c h i g h−d i m e n s i o n a l u n l a b e l e d d a t a
X raw ,
= m a k e b l o b s ( n s a m p l e s =500 , n f e a t u r e s =10 , c e n t e r s =4 , c l u s t e r s t d = 1 . 5 , r a n d o m s t a t e =42)
# 2 . Step 1 : P r e p r o c e s s i n g & S t a n d a r d i z a t i o n
scaler = StandardScaler ()
X s c a l e d = s c a l e r . f i t t r a n s f o r m ( X raw )
# 3 . S t e p 2 : D i m e n s i o n a l i t y R e d u c t i o n v i a PCA
pca = PCA( n c o m p o n e n t s =2)
X pca = pca . f i t t r a n s f o r m ( X s c a l e d )
p r i n t ( f ” E x p l a i n e d V a r i a n c e R a t i o : {pca . e x p l a i n e d v a r i a n c e r a t i o }” )
# 4 . S t e p 3 : C l u s t e r i n g v i a K−Means O p t i m i z a t i o n
kmeans = KMeans ( n c l u s t e r s =4 , i n i t = ’ k−means++ ’ , n i n i t =10 , r a n d o m s t a t e =42)
c l u s t e r l a b e l s = kmeans . f i t p r e d i c t ( X pca )
# 5 . S t e p 4 : I n t e r n a l Model E v a l u a t i o n
s i l s c o r e = s i l h o u e t t e s c o r e ( X pca , c l u s t e r l a b e l s )
d b s c o r e = d a v i e s b o u l d i n s c o r e ( X pca , c l u s t e r l a b e l s )
p r i n t ( f ” S i l h o u e t t e S c o r e : { s i l s c o r e : . 3 f } | D a v i e s−B o u l d i n I n d e x : { d b s c o r e : . 3 f }” )
Computer Engineering

DI04000061: ML

4th Semester

370 / 455

Python Implementation: Complete Unsupervised Pipeline II

Computer Engineering

DI04000061: ML

4th Semester

371 / 455

Python Implementation: Custom K-Means Iteration Mechanics I
i m p o r t numpy a s np
d e f c u s t o m k m e a n s (X , k =3 , m a x i t e r s =100 , t o l =1e −4):
n samples , n f e a t u r e s = X. shape
# S t e p 1 : I n i t i a l i z e c e n t r o i d s r a n d o m l y from d a t a p o i n t s
i d x = np . random . c h o i c e ( n s a m p l e s , k , r e p l a c e=F a l s e )
c e n t r o i d s = X[ idx ]
for

i t e r a t i o n i n range ( m a x i t e r s ) :
# S t e p 2 : C a l c u l a t e E u c l i d e a n d i s t a n c e s (N, K)
d i s t a n c e s = np . l i n a l g . norm (X [ : , np . n e w a x i s , : ] − c e n t r o i d s [ np . n e w a x i s , : ,

:] ,

a x i s =2)

# S t e p 3 : Hard a s s i g n m e n t s t e p ( E−s t e p e q u i v a l e n t )
l a b e l s = np . a r g m i n ( d i s t a n c e s , a x i s =1)
# S t e p 4 : R e c a l c u l a t e c e n t r o i d s (M−s t e p e q u i v a l e n t )
n e w c e n t r o i d s = np . a r r a y ( [ X [ l a b e l s == j ] . mean ( a x i s =0) f o r j i n r a n g e ( k ) ] )
# S t e p 5 : Check f o r c o n v e r g e n c e
c e n t r o i d s h i f t = np . l i n a l g . norm ( n e w c e n t r o i d s − c e n t r o i d s )
centroids = new centroids
if centroid shift < tol :
p r i n t ( f ” C o n v e r g e d a t i t e r a t i o n { i t e r a t i o n + 1}” )
break
return centroids ,

labels

Computer Engineering

DI04000061: ML

4th Semester

372 / 455

Evaluating Unsupervised Learning Models (Without Ground Truth) I

Internal Validation Criteria
Because true labels y are unavailable, unsupervised model performance is evaluated using geometric compactness (intra-cluster variance) and separability
(inter-cluster distance).

Silhouette Coefficient

Davies-Bouldin & Calinski-Harabasz

For sample i, let a(i) be mean intra-cluster distance and b(i) be
mean nearest-cluster distance:

s(i) =

b(i) − a(i)
max(a(i), b(i))

∈ [−1, 1]

DB =

S = 1: Ideal separation.

X
1 K

max

K i=1 j̸=i

σi + σj

!

d(µi , µj )

Lower scores indicate superior separation.

S = 0: Overlapping clusters.

Calinski-Harabasz: Ratio of between-cluster to within-cluster
dispersion (higher is better).

S < 0: Incorrect assignment.

Computer Engineering

Davies-Bouldin Index (DB): Measures average similarity ratio
between each cluster and its most similar one:

DI04000061: ML

4th Semester

373 / 455

Challenges, Pitfalls & Operational Best Practices I

Core Practical Challenges in Unsupervised Learning
Unsupervised workflows present unique operational hurdles due to the lack of explicit target supervision.
Curse of Dimensionality As feature dimensionality d increases, volume grows exponentially, making data sparse. Pairwise Euclidean distances converge to equal
values (dmax /dmin → 1), deteriorating clustering quality.
Best Practice: Always apply dimensionality reduction (PCA, UMAP) or feature selection before clustering high-dimensional spaces.
Local Minima & Sensitivity to Initialization Objectives like WCSS in K -Means or non-convex log-likelihoods in EM are non-convex and sensitive to initial seeds.
Best Practice: Use probabilistic initialization (e.g., K -Means++) and run multiple random restarts.
Determining Hyperparameters (K or Latent k) Deciding optimal cluster count K without ground-truth.
Best Practice: Utilize Elbow curves (WCSS knee), Scree plots, and Silhouette profile analysis.

Computer Engineering

DI04000061: ML

4th Semester

374 / 455

Overview: Unsupervised Learning Taxonomy I
Defining Unsupervised Machine Learning
Unlike supervised learning, unsupervised learning operates on unlabeled datasets D = {x1 , x2 , . . . , xN } where xi ∈ Rd . The primary goal is to uncover
underlying structures, patterns, latent variables, or joint probability distributions p(x) without explicit target labels.

1. Clustering
Objective: Partition data into homogeneous subgroups (clusters).
Core Principle: Maximize intra-cluster similarity, minimize inter-cluster similarity.
Paradigms: Partitioning (K -Means), Hierarchical (Agglomerative), Density-Based (DBSCAN).

2. Association Rule Mining
Objective: Discover conditional dependencies (X =⇒ Y ) in itemsets.
Core Principle: Extract rules satisfying support and confidence thresholds.
Paradigms: Level-wise Search (Apriori), Tree-based Mining (FP-Growth).

Computer Engineering

DI04000061: ML

4th Semester

375 / 455

Clustering: Problem Formulation & Distance Metrics I

Mathematical Formulation of Clustering
Given dataset D = {x1 , . . . , xN }, partition D into K disjoint clusters C = {C1 , C2 , . . . , CK } such that:
K
[

Ck = D

and

Ci ∩ Cj = ∅,

∀i ̸= j

k=1

Minkowski Distance Metric
General metric for x, y ∈ Rd :

dp (x, y ) = 

d
X

1/p
p

|xj − yj | 

j=1

p = 1: Manhattan Distance (L1 norm).
p = 2: Euclidean Distance (L2 norm).

Computer Engineering

DI04000061: ML

4th Semester

376 / 455

Clustering: Problem Formulation & Distance Metrics II

Specialized Distance Metrics
Cosine Similarity:
Simcos (x, y ) =

xT y
∥x∥2 ∥y ∥2

Mahalanobis Distance:
dMah (x, y ) =

q

(x − y )T Σ−1 (x − y )

Accounts for correlation across features (Σ).

Computer Engineering

DI04000061: ML

4th Semester

377 / 455

Partitioning Clustering: K -Means Algorithm I

Objective Function: Within-Cluster Sum of Squares (WCSS)
K -Means seeks cluster assignments C and centroids µ = {µ1 , . . . , µK } that minimize the Total Inertia:

J(C, µ) =

K
X

X

2

∥xi − µk ∥2

k=1 xi ∈Ck

where µk = |C1 |
k

P

xi ∈Ck xi is the mean (centroid) of cluster Ck .

Computer Engineering

DI04000061: ML

4th Semester

378 / 455

Partitioning Clustering: K -Means Algorithm II

Lloyd’s Iterative Optimization Algorithm
(0)

(0)

1

Initialization: Select K initial cluster centroids µ1 , . . . , µK .

2

Assignment Step: Assign each data point to its nearest centroid:
(t)

Ck
3

=

n

(t) 2

o

Update Step: Recompute centroids based on newly assigned points:
(t+1)

µk

4

(t) 2

xi : ∥xi − µk ∥2 ≤ ∥xi − µj ∥2 , ∀j ̸= k

(t+1)

Convergence: Repeat steps 2–3 until centroids stabilize (µk

Computer Engineering

=

1

X

(t)

|Ck |

xi

(t)
xi ∈C
k

(t)

= µk ).

DI04000061: ML

4th Semester

379 / 455

K -Means++ Initialization & Methodological Limitations I
K -Means++ Initialization
Mitigates poor local minima by spreading initial centroids:
1

Choose first centroid µ1 uniformly at random from dataset D.

2

For each x ∈ D, compute shortest distance to existing centroids: D(x) = mink ∥x − µk ∥2 .

3

Select next centroid µj with probability proportional to D(x)2 :

P(x) = P

D(x)2

′ 2
x ′ ∈D D(x )

4

Repeat until K centroids are chosen. Guarantees an O(log K ) approximation bound.

Limitations of K -Means
Spherical Shape Assumption: Struggles with non-globular or complex non-convex cluster topologies.
Pre-specified K : Requires user to define cluster count K a priori.
Sensitivity to Outliers: Centroid mean computation is susceptible to extreme values.
Scale Sensitivity: Requires strict feature standardization (z-score normalization).

Computer Engineering

DI04000061: ML

4th Semester

380 / 455

Hierarchical Clustering: Agglomerative vs. Divisive I
Hierarchical Taxonomy
Hierarchical clustering constructs a nested tree of clusters called a Dendrogram.
Agglomerative (Bottom-Up): Starts with N singletons; iteratively merges nearest pair.
Divisive (Top-Down): Starts with one root cluster containing all points; recursively splits.

Cluster Linkage Criteria d(A, B)
Single Linkage (Minimum distance):
dmin (A, B) =

min

d(x, y )

max

d(x, y )

x∈A,y ∈B

Complete Linkage (Maximum distance):
dmax (A, B) =

x∈A,y ∈B

Average Linkage (Mean pairwise distance):
davg (A, B) =

Computer Engineering

1

X X

|A||B| x∈A y ∈B

DI04000061: ML

d(x, y )

4th Semester

381 / 455

Hierarchical Clustering: Agglomerative vs. Divisive II

Ward’s Minimum Variance
Minimizes total within-cluster variance growth when merging clusters A and B:

∆ESS(A, B) =

|A||B|
|A| + |B|

2

∥µA − µB ∥2

Tends to create balanced, spherical clusters.
Most widely used linkage in practical data science pipelines.

Computer Engineering

DI04000061: ML

4th Semester

382 / 455

Density-Based Clustering: DBSCAN I
Density-Based Spatial Clustering of Applications with Noise
DBSCAN identifies clusters as dense continuous regions separated by regions of low density. It effectively discovers arbitrarily shaped clusters and identifies noise
points.

Core Definitions (ε, MinPts)
ε-Neighborhood: Nε (x) = {y ∈ D : d(x, y ) ≤ ε}.
Core Point: Point x with |Nε (x)| ≥ MinPts.
Border Point: Point y ∈
/ Core, but y ∈ Nε (p) for some core point p.
Noise Point: Any point that is neither a core point nor a border point.

Density-Reachability
Directly Density-Reachable: y ∈ Nε (x) where x is a Core Point.
Density-Reachable: Chain of core points connecting x to y .
Density-Connected: Points x and y are both density-reachable from a common core point o.

Computer Engineering

DI04000061: ML

4th Semester

383 / 455

Validation Metrics for Clustering I

Internal Validation (No Ground Truth)
Silhouette Coefficient:
s(i) =

b(i) − a(i)
max(a(i), b(i))

where a(i) is mean intra-cluster distance, b(i) is mean distance to nearest cluster. s(i) ∈ [−1, +1].
Davies-Bouldin Index:
DB =

X
1 K

max

K k=1 j̸=k

σk + σj

!

d(µk , µj )

Lower values indicate better clustering compactness and separation.

Computer Engineering

DI04000061: ML

4th Semester

384 / 455

Validation Metrics for Clustering II

External Validation (With Ground Truth)
Adjusted Rand Index (ARI): Measures agreement between true class labels Y and cluster assignments C , adjusted for random chance:

ARI =

RI − E[RI]
max(RI) − E[RI]

Normalized Mutual Information (NMI):
NMI(Y , C ) =

2I (Y ; C )
H(Y ) + H(C )

Ranges from 0 (uncorrelated) to 1 (perfect recovery).

Computer Engineering

DI04000061: ML

4th Semester

385 / 455

Association Rule Mining: Problem Formulation I
Transactional Database Framework
Let I = {i1 , i2 , . . . , im } be a universe of items. Let T = {t1 , t2 , . . . , tN } be a database of transactions, where each transaction tk ⊆ I . An Association Rule
is an implication of the form:
X =⇒ Y where X , Y ⊂ I and X ∩ Y = ∅
X is the Antecedent (Left-Hand Side), Y is the Consequent (Right-Hand Side).

1. Support
Frequency of itemset in dataset:
Supp(X ) =

|{tk ∈ T : X ⊆ tk }|
|T |

2. Confidence
Conditional probability P(Y |X ):
Conf(X =⇒ Y ) =

Computer Engineering

Supp(X ∪ Y )
Supp(X )

DI04000061: ML

4th Semester

386 / 455

Association Rule Mining: Problem Formulation II

3. Lift
Ratio of observed to expected co-occurrence:
Lift(X =⇒ Y ) =

Computer Engineering

Supp(X ∪ Y )
Supp(X ) · Supp(Y )

DI04000061: ML

4th Semester

387 / 455

The Apriori Algorithm & Monotonicity Principle I

The Apriori Property (Downward Closure / Anti-Monotonicity)
Theorem: If an itemset X is frequent (Supp(X ) ≥ min sup), then all of its non-empty subsets S ⊂ X must also be frequent.
Pruning Rule: If an itemset S is infrequent, all of its supersets X ⊃ S are guaranteed to be infrequent and can be pruned immediately without scanning the
database!

Level-Wise Search Strategy
1

Frequent 1-Itemsets (L1 ): Scan database to identify all 1-itemsets satisfying min sup.

2

Candidate Generation (Ck+1 ): Join Lk with Lk (Lk ▷◁ Lk ) to construct candidate (k + 1)-itemsets.

3

Candidate Pruning: Remove any candidate c ∈ Ck+1 if any k-subset of c is not in Lk .

4

Database Scan: Count actual support of remaining candidates in Ck+1 to form Lk+1 .

5

Termination: Stop when no new frequent itemsets are generated (Lk+1 = ∅).

Computer Engineering

DI04000061: ML

4th Semester

388 / 455

FP-Growth: Candidate-Free Pattern Mining I
Motivation: Overcoming Apriori Bottlenecks
Apriori suffers from candidate generation explosion (2|I | search space) and requires multiple full database scans (k passes for length-k itemsets).

FP-Tree Data Structure
Encodes database into a compact tree using 2 passes:
Pass 1: Calculate item support; discard infrequent items; sort items in descending support order.
Pass 2: Read transactions sequentially and insert sorted items into the FP-Tree, sharing common prefixes.

Divide-and-Conquer Mining
Builds a Header Table linking item nodes via pointers.
Constructs Conditional Pattern Bases for each item starting from suffix nodes.
Mines conditional FP-Trees recursively.
Avoids candidate generation entirely!

Computer Engineering

DI04000061: ML

4th Semester

389 / 455

Comparative Analysis: Clustering vs. Association Rules I

Methodological Comparison
Dimension
Input Format
Primary Goal
Key Metrics
Primary Output
Key Algorithms
Search Space

Clustering Analysis
Feature vectors x ∈ Rd
Group similar observations
Euclidean distance, WCSS, Silhouette
Partition assignments (C1 , . . . , CK )
K -Means, Hierarchical, DBSCAN
Partitioning space K N

Computer Engineering

DI04000061: ML

Association Rule Mining
Transactional item sets t ⊆ I
Discover co-occurrence rules
Support, Confidence, Lift
Rules of form X =⇒ Y
Apriori, FP-Growth, ECLAT
Combinatorial itemset space 2|I |

4th Semester

390 / 455

Python Implementation: Clustering Techniques I
i m p o r t numpy a s np
from s k l e a r n . c l u s t e r i m p o r t KMeans , DBSCAN
from s k l e a r n . m e t r i c s i m p o r t s i l h o u e t t e s c o r e
from s k l e a r n . d a t a s e t s i m p o r t m a k e b l o b s
# 1. Generate s y n t h e t i c dataset
X , y t r u e = m a k e b l o b s ( n s a m p l e s =500 , c e n t e r s =4 ,
c l u s t e r s t d = 0 . 6 0 , r a n d o m s t a t e =42)
# 2 . P a r t i t i o n i n g C l u s t e r i n g : K−Means++
kmeans = KMeans ( n c l u s t e r s =4 , i n i t = ’ k−means++ ’ ,
n i n i t =10 , r a n d o m s t a t e =42)
l a b e l s k m = kmeans . f i t p r e d i c t (X)
s c o r e k m = s i l h o u e t t e s c o r e (X , l a b e l s k m )
# 3 . D e n s i t y −Based C l u s t e r i n g : DBSCAN
d b s c a n = DBSCAN( e p s = 0 . 4 , m i n s a m p l e s =5)
l a b e l s d b = d b s c a n . f i t p r e d i c t (X)
p r i n t ( f ”K−Means S i l h o u e t t e S c o r e : { s c o r e k m : . 4 f }” )
p r i n t ( f ”DBSCAN D e t e c t e d C l u s t e r s : { l e n ( s e t ( l a b e l s d b ) ) − ( 1 i f −1 i n l a b e l s d b
p r i n t ( f ”DBSCAN O u t l i e r P o i n t s : { l i s t ( l a b e l s d b ) . c o u n t (−1)}” )

Computer Engineering

DI04000061: ML

e l s e 0)} ” )

4th Semester

391 / 455

Python Implementation: Association Rule Mining I
i m p o r t p a n d a s a s pd
from m l x t e n d . p r e p r o c e s s i n g i m p o r t T r a n s a c t i o n E n c o d e r
from m l x t e n d . f r e q u e n t p a t t e r n s i m p o r t a p r i o r i , a s s o c i a t i o n r u l e s
# 1 . Sample T r a n s a c t i o n a l D a t a s e t
d a t a s e t = [ [ ’ M i l k ’ , ’ Onion ’ , ’ Nutmeg ’ , ’ Eggs ’ , ’ Y o g u r t ’ ] ,
[ ’ D i l l ’ , ’ Onion ’ , ’ Nutmeg ’ , ’ Eggs ’ , ’ Y o g u r t ’ ] ,
[ ’ M i l k ’ , ’ A p p l e ’ , ’ Eggs ’ ] ,
[ ’ M i l k ’ , ’ Corn ’ , ’ Y o g u r t ’ ] ,
[ ’ Corn ’ , ’ Onion ’ , ’ Eggs ’ , ’ I c e cream ’ ] ]
# 2 . One−h o t Encode T r a n s a c t i o n s
te = TransactionEncoder ()
te ary = te . f i t ( dataset ) . transform ( dataset )
d f = pd . DataFrame ( t e a r y , c o l u m n s=t e . c o l u m n s )
# 3 . Mine F r e q u e n t I t e m s e t s ( Min S u p p o r t = 60%)
f r e q u e n t i t e m s e t s = a p r i o r i ( d f , m i n s u p p o r t = 0 . 6 , u s e c o l n a m e s=True )
# 4 . G e n e r a t e A s s o c i a t i o n R u l e s ( Min C o n f i d e n c e = 70%)
r u l e s = a s s o c i a t i o n r u l e s ( f r e q u e n t i t e m s e t s , m e t r i c=” c o n f i d e n c e ” , m i n t h r e s h o l d =0.7)
print ( r u l e s [ [ ’ antecedents ’ , ’ consequents ’ , ’ support ’ , ’ confidence ’ , ’ l i f t ’ ] ] )

Computer Engineering

DI04000061: ML

4th Semester

392 / 455

Summary & Key Takeaways I

Core Conceptual Takeaways
Unsupervised Learning Objectives: Focuses on discovering intrinsic geometry (Clustering) or item correlation structure (Association Rules) without target
supervision.
Clustering Selection Guide:

Use K -Means for large-scale datasets with compact, convex clusters.
Use Hierarchical when tree structure/dendrogram is required.
Use DBSCAN when noise handling and arbitrary shape discovery are needed.
Association Rule Mining Strategy: Utilize Lift > 1.0 to filter out uninformative rules resulting from high marginal probabilities. Prefer FP-Growth over
Apriori for high transaction density.

Next Lecture Preview
Lecture 5.5: Dimensionality Reduction — Principal Component Analysis (PCA), Singular Value Decomposition (SVD), and Manifold Learning (t-SNE, UMAP).

Computer Engineering

DI04000061: ML

4th Semester

393 / 455

Landscape of Unsupervised Learning in Industry I

The Industry Reality: Abundance of Unlabeled Data
Over 90% of real-world enterprise data (system logs, sensor telemetry, customer interactions, unstructured text) is unlabeled. Unsupervised learning extracts
actionable insights without expensive manual annotation.

Core Application Domains

Strategic Value Drivers

Customer Analytics: RFM segmentation, behavioral profiling.

Exploratory Data Analysis: Discovering hidden data structures.

Cybersecurity & Finance: Fraud detection, network intrusion alerts.

Feature Preprocessing: Dimensionality reduction for downstream
models.

Bioinformatics: Genomic subgrouping, disease discovery.

Automated Alerting: Detecting rare events and operational
anomalies.

Computer Engineering

DI04000061: ML

4th Semester

394 / 455

Domain 1: Customer Segmentation & Market Basket Analysis I

RFM Customer Segmentation
Clustering customers by Recency, Frequency, and Monetary value using K -Means or Gaussian Mixture Models (GMMs) to tailor marketing strategies.

Market Basket Analysis

Association Metrics

Identifies product co-occurrence patterns in retail transactions using
Association Rule Mining (Apriori / FP-Growth).

For itemsets A =⇒ B:
Support: P(A ∩ B) =

freq(A,B)
N

Confidence: P(B|A) =

Support(A∩B)
Support(A)

P(A∩B)

Lift: P(A)P(B) =

Computer Engineering

DI04000061: ML

Confidence(A =⇒ B)
Support(B)

4th Semester

395 / 455

Domain 2: Anomaly & Fraud Detection Systems I

Unsupervised Outlier Detection Paradigm
In financial transactions and network traffic, anomalies are extremely rare and dynamic, making supervised classification prone to severe class imbalance and
out-of-date rules.

Primary Algorithms

Industrial Applications

Isolation Forest: Isolates anomalies by randomly partitioning feature
space; outliers require fewer splits.
Local Outlier Factor (LOF): Measures local density deviation relative
to k-nearest neighbors.

Credit card fraud detection in real-time stream processing.
Predictive maintenance in IoT sensor telemetry (e.g., turbine failure).
Zero-day network intrusion detection.

Autoencoders: Reconstruction error ∥x − x̂∥22 > τ signals abnormal
behavior.

Computer Engineering

DI04000061: ML

4th Semester

396 / 455

Domain 3: High-Dimensional Visualization & Feature Extraction I

Curse of Dimensionality Mitigation
High-dimensional datasets (e.g., gene expression profiles with 20,000+ features or image embeddings) suffer from data sparsity and heavy computational overhead.

Linear vs. Non-linear Reduction

Genomics & Image Compression

PCA (Principal Component Analysis): Orthogonal linear projection
preserving global variance.

Single-cell RNA sequencing (scRNA-seq) visualization via UMAP.
Eigenfaces for facial recognition preprocessing via SVD:

t-SNE / UMAP: Non-linear manifold learning preserving local
neighborhood distances.
X = UΣV

Computer Engineering

DI04000061: ML

T

4th Semester

397 / 455

Domain 4: Topic Modeling & Unstructured Text Clustering I

Latent Dirichlet Allocation (LDA)
LDA is a generative probabilistic model that represents documents as random mixtures over latent topics, where each topic is characterized by a distribution over
words.

Generative Process

Modern Text Applications

1

For each document d, sample topic proportions θd ∼ Dir(α).

Legal Discovery: Clustering millions of contract clauses.

2

For each word wd,n , sample topic assignment
zd,n ∼ Multinomial(θd ).

Customer Feedback: Grouping support tickets automatically.

3

Sample word wd,n ∼ Multinomial(βzd,n ).

Computer Engineering

BERTopic: Combining Transformer embeddings (SBERT) with
HDBSCAN.

DI04000061: ML

4th Semester

398 / 455

Domain 5: Image Segmentation & Computer Vision I

Pixel-Level Unsupervised Grouping
Partitioning images into semantically meaningful regions without pixel-level ground truth masks.

Techniques

Real-World Scenarios

Color Quantization: K -Means clustering on RGB/Lab color channels
to compress color palettes.

Medical MRI tumor boundary candidate generation.

Normalized Cuts: Graph-based spectral partitioning of pixel similarity
matrices.

Image compression for embedded devices.

Satellite imagery land-cover classification (forest vs. urban).

SLIC Superpixels: Localized K -Means in 5D space (R, G , B, x, y ).

Computer Engineering

DI04000061: ML

4th Semester

399 / 455

Python Implementation: Customer Segmentation & Outlier Detection I
End-to-End Pipeline in Python
Demonstrates feature scaling, dimensionality reduction, K -Means clustering, and Isolation Forest anomaly filtering.

i m p o r t numpy a s np
from s k l e a r n . p r e p r o c e s s i n g i m p o r t S t a n d a r d S c a l e r
from s k l e a r n . d e c o m p o s i t i o n i m p o r t PCA
from s k l e a r n . c l u s t e r i m p o r t KMeans
from s k l e a r n . e n s e m b l e i m p o r t I s o l a t i o n F o r e s t
# 1. Feature Engineering & Scaling
X raw = np . random . r a n d ( 1 0 0 0 , 5 ) # Recency , F r e q u e n c y , Monetary , e t c .
scaler = StandardScaler ()
X s c a l e d = s c a l e r . f i t t r a n s f o r m ( X raw )
# 2 . D i m e n s i o n a l i t y R e d u c t i o n (PCA)
pca = PCA( n c o m p o n e n t s =2)
X pca = pca . f i t t r a n s f o r m ( X s c a l e d )
# 3 . Customer S e g m e n t a t i o n (K−Means )
kmeans = KMeans ( n c l u s t e r s =4 , r a n d o m s t a t e =42)
c l u s t e r l a b e l s = kmeans . f i t p r e d i c t ( X pca )
# 4 . Anomaly / F r a u d D e t e c t i o n ( I s o l a t i o n F o r e s t )
i s o f o r e s t = I s o l a t i o n F o r e s t ( c o n t a m i n a t i o n = 0 . 0 5 , r a n d o m s t a t e =42)
a n o m a l i e s = i s o f o r e s t . f i t p r e d i c t ( X s c a l e d ) # −1: O u t l i e r , 1 : Normal
Computer Engineering

DI04000061: ML

4th Semester

400 / 455

Python Implementation: Customer Segmentation & Outlier Detection II

Computer Engineering

DI04000061: ML

4th Semester

401 / 455

Evaluation & Operational Challenges in Production I

The Evaluation Dilemma: No Ground Truth Labels
Without true labels y , model validation relies on internal cluster validation indices and domain-specific downstream metrics.

Validation Metrics

Production Challenges

Silhouette Coefficient:

s(i) =

Concept Drift: Cluster centers shift over time as user behavior
changes.
b(i) − a(i)
max(a(i), b(i))

Scalability: Distance matrix computation O(N 2 d) requires
mini-batch or approximate nearest neighbors (e.g., Faiss, Annoy).

∈ [−1, 1]

Davies-Bouldin Index: Ratio of intra-cluster scatter to inter-cluster
separation (lower is better).

Interpretability: Translating high-dimensional latent clusters into
business logic.

Calinski-Harabasz Index: Ratio of between-cluster to within-cluster
dispersion.

Computer Engineering

DI04000061: ML

4th Semester

402 / 455

Summary: Real-World Unsupervised Learning Taxonomy I

Comparative Industry Map
Application
Customer Clustering
Fraud Detection
Genomic Analysis
Topic Extraction
Color Quantization

Core Algorithm
K -Means / GMM
Isolation Forest
PCA / UMAP
LDA / BERTopic
MiniBatch K -Means

Key Metric
Silhouette Score
Precision@K / Recall
Variance Explained
Topic Coherence
Mean Squared Error

Industry Impact
Personalized Marketing
Financial Loss Reduction
Disease Subtype Discovery
Automated Document Routing
Image Compression

Bridge to Generative AI
Unsupervised representation learning forms the foundation of modern Generative AI: Autoencoders (VAEs) and Self-Supervised Learning (Contrastive Learning,
Masked Autoencoders) map unstructured data into continuous latent spaces.

Computer Engineering

DI04000061: ML

4th Semester

403 / 455

Learning Paradigms: Overview I

The Fundamental Divide in Machine Learning
Machine Learning paradigms are primarily categorized by the nature of the training data and the presence or absence of explicit target supervisory signals.

Unsupervised Learning

Supervised Learning
Dataset: D = {(xi , yi )}N
i=1 containing feature-label pairs.

Dataset: D = {xi }N
i=1 containing unannotated feature vectors.

Goal: Learn functional mapping f : X → Y to predict target y from
features x.

Goal: Uncover intrinsic data structure, latent patterns, or underlying
distribution p(x).

Feedback: Direct error signal computed against ground truth targets
yi .

Feedback: Indirect signal (e.g., reconstruction error, density
maximization).

Core Distinction
Supervised learning answers ”What is the correct output for this input?”, whereas unsupervised learning answers ”What is the underlying structure of this data?”.

Computer Engineering

DI04000061: ML

4th Semester

404 / 455

Mathematical & Probabilistic Formulations I

Supervised Formulation: Empirical Risk Minimization (ERM)
Given N samples, supervised learning estimates parameters θ by minimizing empirical risk over conditional distribution p(y | x; θ):

θ

∗

= arg min
θ

N
1 X

N i=1

L(yi , f (xi ; θ))

⇐⇒

θ

∗

= arg max
θ

N
X

log p(yi | xi ; θ)

i=1

Unsupervised Formulation: Density Estimation & Latent Modeling
Unsupervised learning models marginal likelihood p(x; θ), often introducing latent variables z ∈ Rk :
Z
p(x | z; θ)p(z)dz

p(x; θ) =

=⇒

θ

∗

Z

Computer Engineering

DI04000061: ML

= arg max
θ

N
X

log p(xi ; θ)

i=1

4th Semester

405 / 455

Mathematical & Probabilistic Formulations II

Reconstruction Formalism (Autoencoders / PCA)
Learn encoder fϕ : X → Z and decoder gθ : Z → X :
min

X
1 N

θ,ϕ N

Computer Engineering

xi − gθ fϕ (xi )


 2
2

i=1

DI04000061: ML

4th Semester

406 / 455

Taxonomy of Tasks I

Supervised Tasks

Unsupervised Tasks
Clustering: Partitioning X into K discrete subsets.

Classification: Categorical target Y = {1, . . . , K }.

K -Means, Hierarchical, DBSCAN,
GMM.

Logistic Regression, SVM, Random
Forest, Neural Networks.

Dimensionality Reduction: Projection to Rk (k ≪ d).

Regression: Continuous target Y = Rm .

PCA, t-SNE, UMAP, Autoencoders.

Ridge, Lasso, Gradient Boosting
Regressors.

Generative Modeling: Learning p(x).

VAEs, GANs, Diffusion Models.

Task Divergence
Supervised models map fixed inputs to pre-defined outputs; unsupervised models explore unlabelled spaces to infer representations, structures, or generation
mechanisms.

Computer Engineering

DI04000061: ML

4th Semester

407 / 455

Systematic Paradigm Comparison I

Feature Comparison Matrix
Dimension
Input Data
Primary Goal
Target Labels
Evaluation
Data Cost
Algorithmic Complexity

Computer Engineering

Supervised Learning
Labeled pairs (xi , yi )
Predict target y for unseen x
Required (yi ∈ Y)
Objective metrics (Accuracy, MSE)
High (Manual annotation needed)
Driven by loss convergence

DI04000061: ML

Unsupervised Learning
Unlabeled features xi
Discover structural patterns in X
Absent (∅)
Heuristic / Internal metrics (Silhouette)
Low (Raw data collected abundantly)
Driven by space/density exploration

4th Semester

408 / 455

Evaluation Paradigms & Validation Metrics I
Supervised Validation
Ground truth labels enable direct loss and metric computation:
Classification: Accuracy, Precision, Recall, F1 -score, ROC-AUC.
Regression: Mean Squared Error (MSE), R 2 coefficient of determination.

Unsupervised Validation (No Ground Truth)
Performance must be assessed via data-intrinsic structural properties:
Silhouette Coefficient: Evaluates cluster cohesion a(i) vs. separation b(i):

s(i) =

b(i) − a(i)
max{a(i), b(i)}

∈ [−1, 1]

Reconstruction Error: Mean distance between original and reconstructed vectors.
External Benchmarking: Adjusted Rand Index (ARI), Normalized Mutual Information (NMI) (when synthetic ground truth exists).

Computer Engineering

DI04000061: ML

4th Semester

409 / 455

Bridging the Gap: Semi-Supervised & Self-Supervised Learning I
Semi-Supervised Learning (SSL)
N

N

L alongside large unlabeled dataset D = {x } U (N ≫ N ).
Leverages small labeled dataset DL = {(xi , yi )}i=1
j j=1
U
U
L

Combined Objective: L = Lsupervised (DL ) + λLunsupervised (DU ).
Techniques: Pseudo-labeling, consistency regularization, graph-based label propagation.

Self-Supervised Learning (SSL)
Transforms unsupervised data into supervised learning tasks by creating pretext tasks directly from raw data without human labeling.
Masked Pretext Tasks: Predict masked words (BERT) or masked image patches (MAE).
Contrastive Learning: Pull augmented views of same image together, push different images apart (SimCLR, CLIP).

Modern Workflow Standard
Self-supervised pre-training on massive unlabeled data followed by supervised fine-tuning on small target datasets.

Computer Engineering

DI04000061: ML

4th Semester

410 / 455

Python Implementation: Workflow Comparison I
Comparing Supervised (Random Forest) vs. Unsupervised (K-Means)
i m p o r t numpy a s np
from s k l e a r n . d a t a s e t s i m p o r t m a k e b l o b s
from s k l e a r n . e n s e m b l e i m p o r t R a n d o m F o r e s t C l a s s i f i e r
from s k l e a r n . c l u s t e r i m p o r t KMeans
from s k l e a r n . m e t r i c s i m p o r t a c c u r a c y s c o r e , s i l h o u e t t e s c o r e
# Generate s y n t h e t i c d a t a s e t : X ( f e a t u r e s ) , y ( ground t r u t h l a b e l s )
X , y = m a k e b l o b s ( n s a m p l e s =500 , c e n t e r s =3 , n f e a t u r e s =4 , r a n d o m s t a t e =42)
# −−− 1 . SUPERVISED LEARNING WORKFLOW −−−
# U s e s BOTH f e a t u r e s X and l a b e l s y
r f m o d e l = R a n d o m F o r e s t C l a s s i f i e r ( n e s t i m a t o r s =50 , r a n d o m s t a t e =42)
r f m o d e l . f i t (X [ : 4 0 0 ] , y [ : 4 0 0 ] )
# Training with t a r g e t s y
y p r e d = r f m o d e l . p r e d i c t (X [ 4 0 0 : ] )
# Exact metric a g a i n s t y
acc = a c c u r a c y s c o r e ( y [ 4 0 0 : ] , y pred )
p r i n t ( f ” S u p e r v i s e d T e s t A c c u r a c y : { a c c : . 4 f }” )
# −−− 2 . UNSUPERVISED LEARNING WORKFLOW −−−
# U s e s ONLY f e a t u r e s X ( no g r o u n d t r u t h l a b e l s y ! )
kmeans = KMeans ( n c l u s t e r s =3 , r a n d o m s t a t e =42 , n i n i t =10)
c l u s t e r l a b e l s = kmeans . f i t p r e d i c t (X)
# Discovery of structure
s i l s c o r e = s i l h o u e t t e s c o r e (X , c l u s t e r l a b e l s ) # I n t e r n a l v a l i d a t i o n
p r i n t ( f ” U n s u p e r v i s e d S i l h o u e t t e S c o r e : { s i l s c o r e : . 4 f }” )
Computer Engineering

DI04000061: ML

4th Semester

411 / 455

Python Implementation: Workflow Comparison II

Computer Engineering

DI04000061: ML

4th Semester

412 / 455

Real-World Applications & Practical Decision Guide I

Clinical Healthcare Application

E-Commerce & Retail Application

Supervised: Train classifier on labeled MRI scans (Tumor vs. Benign)
to assist automated diagnostic triage.

Supervised: Predict customer churn probability y ∈ [0, 1] based on
historical transaction features.

Unsupervised: Cluster multi-omic genomic sequences to discover
previously unknown disease sub-phenotypes.

Unsupervised: Segment customer base into behavioral personas using
purchasing pattern clustering.

Decision Tree for Method Selection
1

High-quality labeled target available? → Choose Supervised Learning.

2

No target labels available, goal is exploration / grouping? → Choose Unsupervised Learning.

3

Abundant raw data, scarce labels? → Choose Self-Supervised Pre-training + Supervised Fine-tuning.

Computer Engineering

DI04000061: ML

4th Semester

413 / 455

Discriminative vs. Generative Modeling I

Discriminative Models
Estimate conditional probability distribution P(Y | X ) or decision boundary f (X ) → Y .
Goal: Distinguish between categories given input features (Classification / Regression).
Examples: Logistic Regression, Support Vector Machines, Random Forests, Standard CNNs.

Generative Models
Estimate joint distribution P(X , Y ) or marginal data distribution P(X ).
Goal: Learn underlying probability structure to generate new synthetic samples x̂ ∼ P(X ).
Examples: Naive Bayes, GMMs, Variational Autoencoders (VAEs), GANs, Diffusion Models.

Computer Engineering

DI04000061: ML

4th Semester

414 / 455

Taxonomy of Generative Modeling I

Problem Formulation
Given dataset D = {x1 , x2 , . . . , xN } sampled independently from unknown data distribution pdata (x), fit a parametric model pθ (x).

Density Estimation Paradigms
Explicit Density Estimation: Define parametric model pθ (x) and directly maximize log-likelihood

PN

i=1 log pθ (xi ).

Tractable Density: Normalizing Flows, Autoregressive Models (e.g., PixelCNN, GPT).
Approximate Density: Variational Autoencoders (VAEs, maximizing ELBO).
Implicit Density Estimation: Train generator Gθ (z) mapping noise z ∼ p(z) to realistic samples without explicit calculation of pθ (x) (e.g., GANs).

Computer Engineering

DI04000061: ML

4th Semester

415 / 455

Variational Autoencoders (VAEs): Architecture I

Latent Variable Model
Assume observed data x is generated by unobserved continuous latent variable z ∼ p(z) = N (0, I ) via conditional likelihood pθ (x | z).

Intractability of Direct Maximum Likelihood
Marginalizing over latent space pθ (x) =

R

pθ (x | z)p(z) dz and computing exact posterior pθ (z | x) =

pθ (x|z)p(z)
are computationally intractable.
pθ (x)

Encoder-Decoder Structure
2
Probabilistic Encoder qϕ (z | x): Approximates intractable posterior pθ (z | x), mapping input x to parameters (µϕ (x), σϕ
(x)).

Probabilistic Decoder pθ (x | z): Reconstructs data x given latent sample z.

Computer Engineering

DI04000061: ML

4th Semester

416 / 455

Evidence Lower Bound (ELBO) & Reparameterization I

ELBO Objective
Using Jensen’s Inequality / KL divergence, log marginal likelihood is bounded below:
log pθ (x) ≥ LELBO (θ, ϕ; x) = Eq (z|x) [log pθ (x | z)] −
ϕ
|
{z
}
Reconstruction Loss

DKL (qϕ (z | x) ∥ p(z))
|
{z
}
Latent Regularization (KL Divergence)

The Reparameterization Trick
Sampling z ∼ N (µϕ , Σϕ ) breaks backpropagation (non-differentiable).
Isolate stochasticity by sampling standard normal noise ϵ ∼ N (0, I ).
Compute continuous mapping:
z = µϕ (x) + σϕ (x) ⊙ ϵ
Allows gradient computation via chain rule:

Computer Engineering

∂L = ∂L ∂z .
∂ϕ
∂z ∂ϕ

DI04000061: ML

4th Semester

417 / 455

Generative Adversarial Networks (GANs) I

Minimax Two-Player Game
Simultaneously train two competing neural networks:
Generator Gθ (z): Maps random noise z ∼ pz (z) to synthetic sample Gθ (z).
Discriminator Dϕ (x): Classifies whether input x comes from real data distribution pdata (x) (D ≈ 1) or generated sample (D ≈ 0).

Minimax Objective Function
min max V (D, G ) = Ex∼p
G

D

data (x)

[log Dϕ (x)] + Ez∼pz (z) [log(1 − Dϕ (Gθ (z)))]

pdata (x)
.
data (x)+pg (x)
∗
1
At global equilibrium (pg = pdata ): D (x) = 2 and V (D ∗ , G ) = − log 4.

Optimal Discriminator: D ∗ (x) = p

Computer Engineering

DI04000061: ML

4th Semester

418 / 455

GAN Training Challenges & Extensions I

Training Dynamics & Failure Modes
Mode Collapse: Generator outputs low variety (collapses to generating single convincing sample).
Vanishing Gradients: Early in training, D easily rejects generated samples (D(G (z)) ≈ 0), leading to zero gradient for G .
Non-Saturating Loss Fix: Train G to maximize Ez [log D(G (z))] instead of minimizing Ez [log(1 − D(G (z)))].

Wasserstein GAN (WGAN)
Replaces Jensen-Shannon Divergence with Earth Mover’s (Wasserstein-1) Distance:
min max Ex∼pdata [D(x)] − Ez∼pz [D(G (z))]
G D∈DL

Requires 1-Lipschitz continuity enforced via weight clipping or gradient penalty (WGAN-GP).

Computer Engineering

DI04000061: ML

4th Semester

419 / 455

Diffusion & Autoregressive Models I
Denoising Diffusion Probabilistic Models (DDPM)
Forward Process (Noising): Slowly add Gaussian noise to sample x0 → x1 → · · · → xT .
Reverse Process (Denoising): Train network ϵθ (xt , t) to predict noise added at step t:
h
i
2
Lsimple (θ) = Et,x0 ,ϵ ∥ϵ − ϵθ (xt , t)∥
Powers state-of-the-art image generators (Stable Diffusion, Midjourney).

Autoregressive Models (Transformers)
Factorize joint sequence distribution using chain rule:

p(x1 , x2 , . . . , xT ) =

T
Y

p(xt | x1 , . . . , xt−1 )

t=1

Dominant paradigm in Large Language Models (LLMs like GPT-4, LLaMA).

Computer Engineering

DI04000061: ML

4th Semester

420 / 455

Python Implementation: PyTorch VAE Module I
import torch
i m p o r t t o r c h . nn a s nn
i m p o r t t o r c h . nn . f u n c t i o n a l a s F
c l a s s VAE( nn . Module ) :
def
init
( s e l f , i n p u t d i m =784 , l a t e n t d i m =20):
super ( ) .
init
()
s e l f . f c 1 = nn . L i n e a r ( i n p u t d i m , 4 0 0 )
s e l f . f c m u = nn . L i n e a r ( 4 0 0 , l a t e n t d i m )
s e l f . f c l o g v a r = nn . L i n e a r ( 4 0 0 , l a t e n t d i m )
s e l f . f c 3 = nn . L i n e a r ( l a t e n t d i m , 4 0 0 )
s e l f . f c 4 = nn . L i n e a r ( 4 0 0 , i n p u t d i m )
d e f r e p a r a m e t e r i z e ( s e l f , mu , l o g v a r ) :
s t d = t o r c h . ex p ( 0 . 5 ∗ l o g v a r )
eps = torch . r a n d n l i k e ( std )
r e t u r n mu + e p s ∗ s t d
def forward ( s e l f , x ) :
h = F. relu ( s e l f . fc1 (x ))
mu , l o g v a r = s e l f . f c m u ( h ) , s e l f . f c l o g v a r ( h )
z = s e l f . r e p a r a m e t e r i z e (mu , l o g v a r )
recon x = torch . sigmoid ( s e l f . fc4 (F . r e l u ( s e l f . fc3 ( z ) ) ) )
r e t u r n r e c o n x , mu , l o g v a r
d e f v a e l o s s ( r e c o n x , x , mu , l o g v a r ) :
BCE = F . b i n a r y c r o s s e n t r o p y ( r e c o n x , x , r e d u c t i o n= ’ sum ’ )
KLD = −0.5 ∗ t o r c h . sum ( 1 + l o g v a r − mu . pow ( 2 ) − l o g v a r . e xp ( ) )
Computer Engineering

DI04000061: ML

4th Semester

421 / 455

Python Implementation: PyTorch VAE Module II

r e t u r n BCE + KLD

Computer Engineering

DI04000061: ML

4th Semester

422 / 455

Python Implementation: PyTorch GAN Step I
d e f t r a i n g a n s t e p (G , D, opt G , opt D , r e a l i m g s , l a t e n t d i m =100):
batch size = real imgs . s i z e (0)
device = real imgs . device
c r i t e r i o n = t o r c h . nn . BCELoss ( )
r e a l l a b e l s = t o r c h . o n e s ( b a t c h s i z e , 1 , d e v i c e=d e v i c e )
f a k e l a b e l s = t o r c h . z e r o s ( b a t c h s i z e , 1 , d e v i c e=d e v i c e )
# 1 . T r a i n D i s c r i m i n a t o r : max l o g (D( x ) ) + l o g ( 1 − D(G( z ) ) )
opt D . z e r o g r a d ( )
z = t o r c h . r a n d n ( b a t c h s i z e , l a t e n t d i m , d e v i c e=d e v i c e )
f a k e i m g s = G( z )
l o s s D r e a l = c r i t e r i o n (D( r e a l i m g s ) , r e a l l a b e l s )
l o s s D f a k e = c r i t e r i o n (D( f a k e i m g s . d e t a c h ( ) ) , f a k e l a b e l s )
loss D = loss D real + loss D fake
l o s s D . b a c kw ard ( )
opt D . s t e p ( )
# 2 . T r a i n G e n e r a t o r : max l o g (D(G( z ) ) )
opt G . z e r o g r a d ( )
l o s s G = c r i t e r i o n (D( f a k e i m g s ) , r e a l l a b e l s )
l o s s G . b a c kward ( )
opt G . s t e p ( )
return loss D . item ( ) , l o s s G . item ()

Computer Engineering

DI04000061: ML

4th Semester

423 / 455

Comparison of Generative AI Paradigms I

Model Class
VAEs
GANs
Diffusion
Autoregressive

Sampling Speed
Fast
Fast
Slow (Multi-step)
Sequential

Sample Quality
Moderate (Blurry)
High (Sharp)
State-of-the-Art
High

Likelihood Access
Approximate (ELBO)
Implicit (None)
Exact / Lower Bound
Exact Likelihood

Summary Takeaways
Generative models learn P(X ) or P(X , Y ) to synthesize novel data samples.
VAEs leverage variational inference and reparameterization trick for stable latent representations.
GANs use zero-sum game dynamics for high-frequency perceptual fidelity.
Diffusion & Autoregressive Transformers currently dominate multimodal AI synthesis.

Computer Engineering

DI04000061: ML

4th Semester

424 / 455

Lecture Overview & Learning Objectives I

Course: DI04000061 (Introduction Machine Learning)
Unit 5: Unsupervised Machine Learning and Generative AI
Topic: 5.2.1 Define Generative AI
Learning Objectives:
1

Define Generative AI and contrast it with traditional Discriminative AI models.

2

Formalize the mathematical objectives of generative modeling (P(X ) vs P(Y | X )).

3

Classify generative approaches into Explicit vs. Implicit density estimation.

4

Survey foundational model paradigms: VAEs, GANs, Diffusion Models, and Autoregressive LLMs.

5

Understand evaluation metrics (e.g., FID, Perplexity) and challenges like Mode Collapse.

6

Implement a fundamental generative density sampling pipeline in Python.

Computer Engineering

DI04000061: ML

4th Semester

425 / 455

What is Generative AI? I

Formal Definition
Generative AI refers to a class of machine learning models that learn the underlying joint probability distribution P(X , Y ) or marginal distribution P(X ) of a
dataset in order to generate new, synthetic data instances x̂ ∼ Pmodel (X ) that resemble real data x ∼ Pdata (X ).

Discriminative AI

Generative AI

Learns conditional probability P(Y | X ).

Learns distribution P(X ) or P(X , Y ).

Maps inputs to labels (decision boundary).

Models data density to sample new data.

Task: Classification, Detection, Regression.

Task: Synthesis, Inpainting, Translation.

Question: ”Is this image a cat or a dog?”

Question: ”Generate a new realistic image of a cat.”

Computer Engineering

DI04000061: ML

4th Semester

426 / 455

Mathematical Foundations: Discriminative vs. Generative I
Probabilistic Formulations
Given input data x ∈ X and target labels y ∈ Y:
Discriminative Objective: Maximizes conditional log-likelihood:

max
θ

N
X

log Pθ (yi | xi )

i=1

Generative Objective (Joint Likelihood): Maximizes joint likelihood:

max
θ

N
X

log Pθ (xi , yi ) = max
θ

i=1

N
X

[log Pθ (xi | yi ) + log P(yi )]

i=1

Unsupervised Generative Objective: Models raw marginal density P(X ):
max Ex∼P
θ

Computer Engineering

data

[log Pθ (x)]

DI04000061: ML

4th Semester

427 / 455

Mathematical Foundations: Discriminative vs. Generative II

Core Challenge of Generative Modeling
Real-world data (e.g., 1024 × 1024 images) live in extremely high-dimensional spaces (R3,145,728 ). Explicitly computing high-dimensional density P(x) is
intractable; generative models rely on deep neural networks to approximate or sample from this manifold.

Computer Engineering

DI04000061: ML

4th Semester

428 / 455

Taxonomy of Generative Models I
Generative models are broadly categorized based on how they handle the data density P(X ):

1. Explicit Density Models

2. Implicit Density Models

Model exact or approximate density function Pθ (x) explicitly.

Do not define explicit density P(x); instead, provide a sampling mechanism
x = G (z) from latent noise z.

Tractable Density:

Autoregressive
Models: Factorize
Qd
P(x) = j=1 P(xj | x<j ) (e.g., GPT,
PixelCNN).

Generative Adversarial Networks (GANs): Game-theoretic
optimization between Generator and Discriminator.
Diffusion / Score-Based Models: Denoising iterative stochastic
processes (DDPMs, SGM).

Approximate Density:

Variational Autoencoders (VAEs):
Maximize Evidence Lower Bound
(ELBO).
Normalizing Flows: Exact
transformation via invertible maps.

Computer Engineering

DI04000061: ML

4th Semester

429 / 455

Foundational Generative Architectures I
1. Variational Autoencoders (VAEs)
Encoder maps data to latent distribution parameters (µ, σ); Decoder reconstructs data. Optimizes the Evidence Lower Bound (ELBO):
LELBO (θ, ϕ) = Eq (z|x) [log pθ (x | z)] − DKL qϕ (z | x) || p(z)
ϕ




2. Generative Adversarial Networks (GANs)
Minimax game between Discriminator Dϕ and Generator Gθ :
min max V (D, G ) = Ex∼P
G

D

data

[log D(x)] + Ez∼pz [log(1 − D(G (z)))]

3. Diffusion Models (DDPM)
Progressive noise addition q(xt | xt−1 ) and learned reverse denoising pθ (xt−1 | xt ):
h
i
2
Lsimple (θ) = Et,x0 ,ϵ ∥ϵ − ϵθ (xt , t)∥

Computer Engineering

DI04000061: ML

4th Semester

430 / 455

Generative AI Modalities & Applications I
Text & Code Generation:

Architectures: Transformer Decoders (Autoregressive LLMs: GPT-4, LLaMA).
Applications: Conversational agents, automated code generation, summarization.
Computer Vision (Image & Video Synthesis):

Architectures: Latent Diffusion Models (Stable Diffusion), StyleGAN.
Applications: Text-to-image synthesis, photorealistic video rendering, image inpainting.
Audio & Speech Generation:

Architectures: WaveNet, Audio Spectrogram Transformers.
Applications: Text-to-speech (TTS), voice cloning, automated music composition.
Science & Tabular Data:

Applications: De novo molecular design, protein folding prediction, synthetic data creation
for privacy compliance.

Computer Engineering

DI04000061: ML

4th Semester

431 / 455

Python Implementation: Generative Density Sampling I
Fitting a Gaussian Mixture Model (GMM) as a Generative Model
A Gaussian Mixture Model models P(x) =

PK

k=1 πk N (x | µk , Σk ) and enables sampling new data.

i m p o r t numpy a s np
from s k l e a r n . m i x t u r e i m p o r t G a u s s i a n M i x t u r e
# 1. Generate s y n t h e t i c multimodal t r a i n i n g data
np . random . s e e d ( 4 2 )
c l u s t e r 1 = np . random . n o r m a l ( l o c =−3.0, s c a l e = 0 . 8 , s i z e =(250 , 2 ) )
c l u s t e r 2 = np . random . n o r m a l ( l o c = 3 . 0 , s c a l e = 1 . 2 , s i z e =(250 , 2 ) )
X t r a i n = np . v s t a c k ( [ c l u s t e r 1 , c l u s t e r 2 ] )
# 2 . T r a i n G e n e r a t i v e Model ( E x p l i c i t D e n s i t y E s t i m a t o r )
gmm = G a u s s i a n M i x t u r e ( n c o m p o n e n t s =2 , c o v a r i a n c e t y p e= ’ f u l l ’ , r a n d o m s t a t e =42)
gmm. f i t ( X t r a i n )
# 3 . Sample new s y n t h e t i c d a t a p o i n t s from l e a r n e d P(X)
X s y n t h e t i c , y s y n t h e t i c c o m p o n e n t s = gmm. s a m p l e ( n s a m p l e s =100)
p r i n t ( f ” L e a r n e d Component Means : \ n{gmm. means }” )
p r i n t ( f ” G e n e r a t e d S y n t h e t i c S a m p l e s Shape : { X s y n t h e t i c . s h a p e }” )

Computer Engineering

DI04000061: ML

4th Semester

432 / 455

Evaluating Generative Models & Key Challenges I

Evaluation Metrics

Key Technical Challenges

Fréchet Inception Distance (FID): Quantifies distance between
feature representations of real and generated samples:
2

FID = ∥µr − µg ∥ + Tr(Σr + Σg − 2(Σr Σg )

1/2

Training Instability: Vanishing gradients or non-convergence in
min-max optimizations.

)

Hallucination: LLMs generating factually incorrect yet confident
content.

Perplexity: Evaluates language model prediction likelihood.

Computational Demands: High memory and throughput
requirements for inference/training.

Inception Score (IS): Evaluates sample quality and diversity.

Computer Engineering

Mode Collapse: GAN generator produces limited variety of samples,
missing modes of Pdata .

DI04000061: ML

4th Semester

433 / 455

Summary & Paradigm Comparison I

Feature
Primary Goal
Output

Discriminative AI
Predict label Y given X
Scalar / Category Class

Math Formulation
Key Models
Primary Risk

P(Y | X )
Logistic Reg., SVM, ResNet
Overfitting / Decision error

Generative AI
Model data distribution P(X )
New
Complex
Sample
(Text/Image)
P(X ) or P(X , Y )
VAE, GAN, Diffusion, GPT
Mode Collapse / Hallucination

Key Takeaway
Generative AI represents a fundamental evolution from pattern recognition to pattern creation, enabling machines to model the underlying structure of
high-dimensional data spaces and sample novel realizations.

Computer Engineering

DI04000061: ML

4th Semester

434 / 455

Fundamental Paradigm: Discriminative vs. Generative I

Discriminative Modeling
Focuses on learning the conditional distribution P(Y | X ) or a decision boundary f (X ) → Y .
Goal: Predict label Y given feature set X .
Examples: Logistic Regression, SVMs, ResNet classifiers.

Generative Modeling
Focuses on estimating the joint distribution P(X , Y ) or data distribution P(X ).
Goal: Model the underlying probability distribution of high-dimensional data X to sample novel instances x̃ ∼ Pθ (X ).
Key Challenge: High-dimensional probability estimation (X ∈ RD where D is large).

Computer Engineering

DI04000061: ML

4th Semester

435 / 455

Core Components of Generative Workflows I

Generative models transform simple noise distributions into complex target data distributions through three key mechanisms:
1

Latent Space Representation (Z):

Maps raw high-dimensional data X to a lower-dimensional manifold Z ∼ N (0, I).
Captures semantic attributes (e.g., lighting, pose, tone).
2

Density Estimation / Mapping Function (gθ ):

Neural network parameterizes the transformation: gθ : Z → X.
Explicit density models estimate pθ (x); implicit models directly generate samples.
3

Sampling & Decoding:

Draw seed z ∼ p(z) and compute xgen = gθ (z).

Computer Engineering

DI04000061: ML

4th Semester

436 / 455

1. Variational Autoencoders (VAEs) I

Probabilistic Formulation
VAEs optimize the Evidence Lower Bound (ELBO) using an Encoder qϕ (z | x) and Decoder pθ (x | z):
LELBO (θ, ϕ; x) = Eq (z|x) [log pθ (x | z)] − DKL (qϕ (z | x) ∥ p(z))
ϕ

The Reparameterization Trick
2
Direct sampling z ∼ N (µϕ , σϕ
) blocks backpropagation.

Solution: Isolate stochasticity by sampling standard noise ϵ ∼ N (0, I):
z = µϕ (x) + σϕ (x) ⊙ ϵ
Enables gradient flow through deterministic network operations.

Computer Engineering

DI04000061: ML

4th Semester

437 / 455

2. Generative Adversarial Networks (GANs) I

Adversarial Game Theory
GANs operate via a zero-sum game between Generator Gθ and Discriminator Dϕ :
min max V (D, G ) = Ex∼pdata [log D(x)] + Ez∼pz [log(1 − D(G (z)))]
G

D

Discriminator (Dϕ ): Tries to distinguish real data x from fake data G (z).
Generator (Gθ ): Maps noise z to fake samples, aiming to fool D.
Training Dynamics: Reaches Nash Equilibrium when pg = pdata and D(x) = 0.5.
Key Challenges: Mode collapse, vanishing gradients, training instability.

Computer Engineering

DI04000061: ML

4th Semester

438 / 455

3. Diffusion Models: Iterative Denoising I

Forward and Reverse Processes
1

Forward Process (q): Gradually adds Gaussian noise to data x0 over T steps:
q(xt | xt−1 ) = N (xt ;

2

p

1 − βt xt−1 , βt I)

Reverse Process (pθ ): Neural network parameterizes noise removal:
pθ (xt−1 | xt ) = N (xt−1 ; µθ (xt , t), Σθ (xt , t))

Training Objective
Predict the added noise ϵ at step t given noisy sample xt :
h
i
2
Lsimple (θ) = Et,x0 ,ϵ ∥ϵ − ϵθ (xt , t)∥

Computer Engineering

DI04000061: ML

4th Semester

439 / 455

4. Autoregressive Models & LLMs I

Chain Rule of Probability
Decomposes the joint probability of a sequence x = (x1 , x2 , . . . , xT ) into conditional probabilities:

P(x) =

T
Y

P(xt | x1 , x2 , . . . , xt−1 ) =

t=1

T
Y

P(xt | x<t )

t=1

Mechanism: Predicts the next element (token) given all preceding context.
Architecture: Transformer Decoders using Causal Self-Attention masks.
Inference Strategies:

Greedy Decoding: Select arg max P(xt | x<t ).
Top-k / Nucleus (Top-p) Sampling: Sample from high-probability subset.
i /T )
Temperature Scaling: Adjust logits via P(xi ) = Pexp(z
exp(zj /T ) .

Computer Engineering

DI04000061: ML

4th Semester

440 / 455

Comparative Analysis of Generative Paradigms I

Model Family
VAE
GAN
Diffusion
Autoregressive

Likelihood
Explicit (ELBO)
Implicit
Explicit
Exact

Sampling Speed
Fast (1 step)
Fast (1 step)
Slow (T steps)
Slow (Sequential)

Sample Quality
Moderate / Blurry
High (Sharp)
State-of-the-Art
High (Text/Audio)

Generative Trilemma
Ideal generative models seek three properties simultaneously:
1

High sample quality.

2

Fast sampling speed.

3

Mode coverage (diversity).

Existing architectures trade off between these three attributes.

Computer Engineering

DI04000061: ML

4th Semester

441 / 455

Python: VAE Reparameterization & Sampling I
import torch
i m p o r t t o r c h . nn a s nn
c l a s s VAESampler ( nn . Module ) :
def
init
( s e l f , l a t e n t d i m =20):
super ( ) .
init
()
s e l f . latent dim = latent dim
d e f r e p a r a m e t e r i z e ( s e l f , mu , l o g v a r ) :
”””
R e p a r a m e t e r i z a t i o n T r i c k : z = mu + s t d ∗ e p s i l o n
”””
s t d = t o r c h . ex p ( 0 . 5 ∗ l o g v a r )
e p s = t o r c h . r a n d n l i k e ( s t d ) # N( 0 , I )
r e t u r n mu + e p s ∗ s t d
d e f d e c o d e l a t e n t n o i s e ( s e l f , d e c o d e r , n u m s a m p l e s =16):
# Sample d i r e c t l y from s t a n d a r d p r i o r N( 0 , I )
z = t o r c h . r a n d n ( num samples , s e l f . l a t e n t d i m )
with torch . no grad ( ) :
generated samples = decoder ( z )
return generated samples

Computer Engineering

DI04000061: ML

4th Semester

442 / 455

Python: GAN Loss Functions I

import torch
i m p o r t t o r c h . nn . f u n c t i o n a l a s F
def compute gan losses ( d i s c r i m i n a t o r , generator , real imgs , z ) :
# 1 . T r a i n D i s c r i m i n a t o r : Maximize l o g (D( x ) ) + l o g ( 1 − D(G( z ) ) )
fake imgs = generator ( z )
d real logits = discriminator ( real imgs )
d f a k e l o g i t s = d i s c r i m i n a t o r ( fake imgs . detach ( ) )
loss D real = F. binary cross entropy with logits (
d r e a l l o g i t s , torch . ones like ( d r e a l l o g i t s ))
loss D fake = F. binary cross entropy with logits (
d fake logits , torch . z e r o s l i k e ( d f a k e l o g i t s ))
loss D = ( l o s s D r e a l + loss D fake ) / 2.0
# 2 . T r a i n G e n e r a t o r : Maximize l o g (D(G( z ) ) )
d f a k e l o g i t s g = discriminator ( fake imgs )
loss G = F. binary cross entropy with logits (
d fake logits g , torch . ones like ( d f a k e l o g i t s g ))
return loss D , loss G

Computer Engineering

DI04000061: ML

4th Semester

443 / 455

Evaluating Generative AI Models I

Fréchet Inception Distance (FID)
Measures distance between feature distributions of real (r ) and generated (g ) images extracted from an Inception-v3 network:


2
1/2
FID = ∥µr − µg ∥ + Tr Σr + Σg − 2(Σr Σg )

Lower FID indicates higher visual quality and closer distribution match.

Inception Score (IS) & Precision/Recall
Inception Score: Evaluates sharpness P(y | x) and diversity P(y ).
Precision vs. Recall: Precision measures quality (fidelity of samples); Recall measures coverage of real data modes.

Computer Engineering

DI04000061: ML

4th Semester

444 / 455

Summary: Working of Generative AI I

Core Goal: Model data distribution P(X) to sample new, realistic data points.
Latent Space: Low-dimensional structured representation driving generative controllers.
Diverse Mechanics:

VAE: ELBO optimization via reparameterization trick.
GAN: Minimax adversarial game between G and D.
Diffusion: Iterative noise estimation and removal over discrete timestamps.
Autoregressive: Sequentially sampling next tokens conditioned on past tokens.
Evaluation: Quantitative evaluation via metrics like FID, IS, and Precision/Recall balance.

Computer Engineering

DI04000061: ML

4th Semester

445 / 455

Overview & Domain Taxonomy of Generative AI I

Generative AI synthesizes novel, high-dimensional data points x̃ ∼ Pmodel (X) across diverse modalities:

Taxonomy of Generative Modalities
Text & Language: Autoregressive language modeling, dialog, summarization, and translation.
Vision & Graphics: Image synthesis, video generation, 3D asset generation, and neural rendering.
Code & Software: Code completion, automated bug fixing, unit test generation, and docstring synthesis.
Science & Healthcare: Protein design, molecular generation, materials discovery, and EHR synthesis.

Operational Paradigms
Unconditional: Sample directly from latent space z ∼ N (0, I) → x.
Conditional (P(X | C)): Guided synthesis using prompt, class label, or layout constraint C.

Computer Engineering

DI04000061: ML

4th Semester

446 / 455

Text Generation & Decoding Strategies I
Autoregressive Parameterization
Text generation factors joint likelihood over tokens w = (w1 , w2 , . . . , wT ) via the chain rule:

P(w) =

T
Y

P(wt | w1 , w2 , . . . , wt−1 ; θ)

t=1

Sampling Techniques for Text Synthesis
Given predicted logits zt , token probabilities are computed via Temperature Scaling:
exp(zt,i /T )
P(wt = i | w<t ) = P
j exp(zt,j /T )

Temperature (T ): T → 0 approaches greedy decoding; higher T increases diversity.
Top-k Sampling: Truncates candidate pool to the top k most probable tokens.
Top-p (Nucleus) Sampling: Selects smallest set of tokens whose cumulative probability ≥ p.

Computer Engineering

DI04000061: ML

4th Semester

447 / 455

Retrieval-Augmented Generation (RAG) I
RAG addresses static knowledge cutoffs and hallucinations by coupling parametric memory (LLM) with non-parametric vector databases.

RAG Algorithmic Workflow
1

Query Embedding: Embed user query q ∈ Rd via encoder EQ (q).

2

Dense Retrieval: Fetch top-k relevant document chunks Dk using cosine similarity:

sim(q, di ) =

3

EQ (q)⊤ ED (di )
∥EQ (q)∥∥ED (di )∥

Augmented Generation: Condition LLM on retrieved context Dk :
|Y |

P(y | q) =

Y

P(yt | y<t , q, Dk )

t=1

Key Advantage
Enables domain-specific QA and real-time knowledge integration without expensive model fine-tuning.

Computer Engineering

DI04000061: ML

4th Semester

448 / 455

Computer Vision & Image Synthesis I

Latent Diffusion Models (LDMs)
LDMs perform diffusion in a compressed latent space z = E(x), drastically reducing computational cost:
Conditioning Mechanism: Cross-attention integrates prompt embeddings τθ (y):

Attention(Q, K , V ) = softmax

QK ⊤
√
d

!

(i)

V
(i)

Q = WQ ϕi (zt ), K = WK τθ (y )

Key Vision Applications
Text-to-Image / Video: Generating photorealistic artwork and videos from natural language.
Inpainting & Outpainting: Restoring corrupted regions or expanding image boundaries.
Spatial Control (ControlNet): Adding structural conditioning (e.g., edge maps, depth maps, human poses) to diffusion models.

Computer Engineering

DI04000061: ML

4th Semester

449 / 455

Code Generation & Software Engineering I

Code-Specialized Language Models
Models trained on source code (e.g., StarCoder, CodeLlama) leverage structured syntax (ASTs) and high context lengths.
Task Scenarios: Autocompletion, text-to-code translation, docstring generation, unit test creation.
Fill-in-the-Middle (FIM): Trained to predict missing code given prefix and suffix: P(middle | prefix, suffix).

Evaluation Benchmark Metric: pass@k
Measures the probability that at least one of k generated code samples passes all functional unit tests:

pass@k = Eproblems 1 −





n−c
k
 
n
k



where n is total generated samples per problem (n ≥ k) and c is the number of correct samples (c ≤ n).

Computer Engineering

DI04000061: ML

4th Semester

450 / 455

Python Implementation: Generative Pipelines I
Generating Text and Images using Hugging Face Ecosystem
import torch
from t r a n s f o r m e r s i m p o r t p i p e l i n e
from d i f f u s e r s i m p o r t S t a b l e D i f f u s i o n P i p e l i n e
# 1 . Text G e n e r a t i o n w i t h an LLM p i p e l i n e
t e x t g e n = p i p e l i n e ( ” t e x t −g e n e r a t i o n ” , model=” g p t 2 ” )
prompt = ” U n s u p e r v i s e d l e a r n i n g d i s c o v e r s h i d d e n p a t t e r n s by ”
t e x t o u t p u t = t e x t g e n ( prompt , m a x l e n g t h =50 , n u m r e t u r n s e q u e n c e s =1)
p r i n t ( ” G e n e r a t e d Text : ” , t e x t o u t p u t [ 0 ] [ ’ g e n e r a t e d t e x t ’ ] )
# 2 . Text−to−Image G e n e r a t i o n w i t h L a t e n t D i f f u s i o n
m o d e l i d = ” runwayml / s t a b l e −d i f f u s i o n −v1−5”
pipe = StableDiffusionPipeline . from pretrained (
m o d e l i d , t o r c h d t y p e=t o r c h . f l o a t 1 6
)
p i p e = p i p e . t o ( ” cuda ” )
i m a g e p r o m p t = ”A h i g h−t e c h n e u r a l n e t w o r k v i s u a l i z a t i o n ,
image = p i p e ( i m a g e p r o m p t ) . i m a g e s [ 0 ]
image . s a v e ( ” g e n e r a t e d n e t w o r k . png ” )

Computer Engineering

futuristic

DI04000061: ML

style”

4th Semester

451 / 455

Generative AI in Science & Healthcare I

De Novo Molecular & Protein Design
Generative architectures create novel physical structures satisfying specific biological constraints:
Molecular Graph Generation: VAEs and Flow models generate target small molecules parameterized as SMILES strings or molecular graphs.
Protein Backbone Design: Diffusion models (e.g., RFdiffusion) generate de novo 3D protein structures tailored for receptor binding.

Healthcare Data Privacy & Augmentation
Synthetic EHR Generation: Generative adversarial networks generate privacy-preserving synthetic patient medical records.
Medical Image Augmentation: Diffusion models generate synthetic MRI/CT scans for rare disease classification tasks.

Computer Engineering

DI04000061: ML

4th Semester

452 / 455

Quantitative Evaluation of Generative Models I
Evaluating generative models requires assessing sample quality, diversity, and fidelity.

Image Evaluation Metrics
Fréchet Inception Distance (FID): Measures distance between real (r ) and generated (g ) feature distributions in Inception-v3 latent space:


2
1/2
FID = ∥µr − µg ∥2 + Tr Σr + Σg − 2(Σr Σg )
Inception Score (IS): Evaluates sample quality and diversity via conditional class distributions.
LPIPS: Perceptual similarity distance based on deep feature representations.

Text Evaluation Metrics
Perplexity (PPL): Exponentiated average negative log-likelihood per token.
BERTScore & LLM-as-a-Judge: Semantic similarity matching using contextual embeddings or strong evaluator LLMs (e.g., GPT-4).

Computer Engineering

DI04000061: ML

4th Semester

453 / 455

Safety, Alignment & Open Technical Challenges I

Human Alignment Strategies
RLHF: Aligning model responses using a Reward Model Rψ (x, y) and PPO optimization:


max E(x,y)∼D Rψ (x, y) − βDKL (πθ (y | x) ∥ πref (y | x))
θ

Direct Preference Optimization (DPO): Eliminates explicit reward modeling by optimizing directly on preference pairs (yw ≻ yl ).

Critical Challenges
Hallucination & Factuality: Generating plausible but factually incorrect assertions.
Mode Collapse: Loss of output diversity in GANs and diffusion processes.
Copyright & Provenance: Watermarking generated media (e.g., SynthID) to track authenticity.

Computer Engineering

DI04000061: ML

4th Semester

454 / 455