Unit-3: Model Development and Techniques

 

Meaning of Data Partitioning


The process of splitting a dataset into more manageable, smaller pieces in order to improve efficiency, scalability and performance is known as data partitioning.

  • It can be accomplished by either vertical partitioning, which separates data into columns, or horizontal partitioning, which divides data into rows according to particular criteria.
  • This method is especially helpful in databases, big data processing frameworks and machine learning applications since it enables quicker query execution, simpler management of massive datasets and better resource use.

Need for Data Partitioning


Data partitioning is essential for the following reasons:

  1. Performance Improvement: By breaking data into smaller segments, systems can access only the relevant partitions, leading to faster query execution and reduced load times.
  2. Scalability: As datasets grow, partitioning allows for easier management and distribution across multiple servers or storage systems, enabling horizontal scaling.
  3. Efficient Resource Utilization: It helps optimize the use of resources by allowing systems to focus processing power on specific partitions rather than the entire dataset.
  4. Enhanced Manageability: Smaller partitions are easier to back up, restore, and maintain, facilitating better data governance and maintenance practices.

Methods of Data Partitioning


The following are important methods used for Data Partitioning:

1. Horizontal Partitioning

In this technique, the dataset is divided based on rows or records. Each partition contains a subset of rows, and the partitions are typically distributed across multiple servers or storage devices. Horizontal partitioning is often used in distributed databases or systems to improve parallelism and enable load balancing.


Horizontal Partitioning

Advantages: 
  1. Large datasets can be processed in parallel to horizontal partitioning, which divides data among multiple computers or storage devices. So, improve the scalability. 
  2. Data partitioning allows for the equitable distribution of workload across multiple nodes, preventing bottlenecks and improving system performance i.e., load balancing. 
  3. Data isolation and fault tolerance are enhanced since each partition can be controlled separately. Even if one partition fails, the others can continue to function.
Disadvantages: 
 
  1. Horizontal partitioning can make join operations across multiple partitions more complex and potentially slower, as data needs to be fetched from different nodes. 
  2. If the distribution of data is uneven or if some partitions receive more queries or updates than others, it can result in data skew, impacting performance and load balancing.

2. Vertical Partitioning

Vertical partitioning separates the dataset according to columns or attributes, in contrast to horizontal partitioning. Each partition in this method has a subset of columns for every row. When certain columns are visited more frequently than others or when different columns have different access patterns, vertical partitioning might be helpful.

Vertical Partitioning

Advantages: 

  1. By placing frequently accessed columns in a separate partition, vertical partitioning can enhance query performance by reducing the amount of data read from storage.
  2. When a query only requires a subset of columns, vertical partitioning allows retrieving only the necessary data, saving storage and I/O resources.
  3. With vertical partitioning, adding or removing columns becomes easier, as the changes only affect the respective partitions.
Disadvantages:
  1. Vertical partitioning can lead to more complex query execution plans, as queries may need to access multiple partitions to gather all the required data.
  2. Joining data from different partitions can be more complex and potentially slower, as it involves retrieving data from different partitions and combining them.
  3. Vertical partitioning may not be as effective for datasets that continuously grow in terms of the number of columns, as adding new columns may require restructuring the partitions.


3. Key-based Partitioning

This approach divides the data according to a specific attribute value or key. Each partition of the dataset contains all the information related to a particular key value. In distributed databases or systems, key-based partitioning is frequently used to spread the data uniformly and enable effective key lookup-based data retrieval.

Key-based Partitioning

Advantages 
  1. Even data distribution: Key-based partitioning ensures that data with the same key value is stored in the same partition, enabling efficient data retrieval by key lookups.
  2. Scalability: Key-based partitioning can distribute data evenly across partitions, allowing for better parallelism and improved scalability.
  3. Load balancing: By distributing data based on key values, the workload is balanced across multiple partitions, preventing hotspots and optimizing performance.
Disadvantages
  1. Skew and hotspots: If the key distribution is uneven or if certain key values are more frequently accessed than others, it can lead to data skew or hotspots, impacting performance and load balancing.
  2. Limited query flexibility: Key-based partitioning is most efficient for queries that primarily involve key lookups. Queries that span multiple keys or require range queries may suffer from increased complexity and potentially slower performance.
  3. Partition management: Managing partitions based on key values requires careful planning and maintenance, especially when the dataset grows or the key distribution changes.

4. Range Partitioning

The dataset is divided using range partitioning based on a preset range of values. For example, if your dataset has timestamps, you can divide it according to a specific time range. Range partitioning might be useful when you have data with natural ordering and wish to distribute it evenly based on the range of values.

Range Partitioning


Advantages
  • Natural ordering: Range partitioning is suitable for datasets with a natural ordering based on a specific attribute. It allows for efficient data retrieval based on ranges of values.
  • Even data distribution: By dividing the dataset based on ranges, range partitioning can distribute the data evenly across partitions, ensuring load balancing and optimal performance.
  • Simplified query planning: Range partitioning simplifies query planning when queries primarily involve range-based conditions, as the system knows which partition(s) to access based on the range specified.
Disadvantages
  • Uneven data distribution: If the data distribution is not evenly distributed across ranges, it can lead to data skew and impact load balancing and query performance.
  • Data growth challenges: As the dataset grows, the ranges may need to be adjusted or new partitions added, requiring careful management and potentially affecting existing queries and data distribution.
  • Joins and range queries: Range partitioning can introduce complexity when performing joins across partitions or when queries involve multiple non-contiguous ranges, potentially leading to performance challenges.


5. Hash-based Partitioning

The technique of employing a hash function to analyze data and determine which division it belongs to is known as hash partitioning. After being fed the data, the hash function generates a hash value that is used to classify the data into a particular division. Hashing-based partitioning can help in load balancing and speedy data retrieval by dividing data among partitions at random.

Hash-based Partitioning


Advantages
  • Even data distribution: Hash-based partitioning provides a random distribution of data across partitions, ensuring even data distribution and load balancing.
  • Scalability: Hash-based partitioning enables scalable parallel processing by evenly distributing data across multiple nodes.
  • Simpleness: Hash-based partitioning does not depend on any particular data properties or ordering, and it is relatively easy to implement.
Disadvantages 
  • Key-based queries: Hash-based partitioning is not suitable for efficient key-based lookups, as the data is distributed randomly across partitions. Key-based queries may require searching across multiple partitions.
  • Load balancing challenges: In some cases, the distribution of data may not be perfectly balanced, resulting in load imbalances and potential performance issues.
  • Partition management: Hash-based partitioning may require adjustments to the number of partitions or hash functions as the dataset grows or the system requirements change, necessitating careful management and potential data redistribution.

6. Round-Robin Partitioning

Data is cyclically and equally distributed among partitions in round-robin partitioning. Regardless of the properties of the data, each split is sequentially assigned the next accessible data item. Implementing round-robin partitioning is simple and can offer a minimal degree of load balancing.


Advantages 
  • Simple implementation: Round-robin partitioning is straightforward to implement, as it assigns data items to partitions in a cyclic manner without relying on any specific data characteristics.
  • Basic load balancing: Round-robin partitioning can provide a basic level of load balancing, ensuring that data is distributed across partitions evenly.
  • Scalability: It is made possible by round-robin partitioning, which divides the data into several parts and permits parallel processing.
Disadvantages
  • Uneven data distribution or a number of partitions that are not a multiple of the total number of data items may cause round-robin partitioning to produce unequal partition sizes.
  • Inefficient data retrieval: Round-robin partitioning does not consider any data characteristics or access patterns, which may result in inefficient data retrieval for certain queries.
  • Limited query optimization: Round-robin partitioning does not optimize for specific query patterns or access patterns, potentially leading to suboptimal query performance.


Sourse: Data Partitioning Techniques in System Design - GeeksforGeeks


Model Selection

Model selection is a key step in Data Modelling because it affects how well a system can learn from data and make accurate predictions. Different models have different ways of processing data and choosing the right one ensures that the system works efficiently. A simple model cannot capture details and has poor accuracy, while a model too complex might overfit that is doing very well on training data but fails on new data. The goal is to find a model that learns patterns effectively without being too simple or too complex.

  1. Proper model selection involves experimenting with different models and comparing their performance using evaluation metrics such as accuracy, precision, recall or mean squared error. These metrics help in determining which model is best suited for a given task.
  2. Apart from performance metrics, other factors such as training time, dataset size and available computing power also play a crucial role in choosing the right model.
  3. Selecting an appropriate model not only improves prediction accuracy but also enhances efficiency, making the system faster and more reliable.


Steps in Model Selection



1. Understanding Problem Type - Before selecting a model, it is important to first analyze the problem what we are trying to solve. The initial step is to determine whether it is a regression problem, Classification problem or Clustering problem. 
  • If the goal is to predict continuous values like house prices, it falls under regression problem 
  • If the task involves predicting categorical labels, such as distinguishing between spam and non-spam emails, it falls under classification problem. 
  • If the objective is to group similar data points, like segmenting customers based on behavior, then it is a clustering problem. 
2. Selecting Suitable Models

After understanding the problem, one then choose a best model that should solve the problem. Different types of models work better for different kinds of problems:
  • For Regression: Linear Regression, Decision Trees, Random Forest, Neural Networks.
  • For Classification: Logistic Regression, Support Vector Machines (SVM), k-Nearest Neighbors (k-NN), Neural Networks.For Clustering: k-Means, Hierarchical Clustering, DBSCAN) 

3. Data Characteristics - Another important point is about the nature of the dataset itself. One has to check for missing values, the number of numerical and categorical variables and the distribution of data. Understanding the type of problem and the dataset helps in choosing the most suitable model.

4. Interpretability - The other important point is can we need to understand the model prediction, or is simply getting accurate predictions? 

5. Computational Resources - Understand the processing power and time to have available for training, testing and deployment.  

One must rank each one according to how well it does the job. The most common method is to split the dataset into two parts.
  • Training Set: The data used to train a machine learning model by learning patterns and relationships.
  • Testing Set: This checks how well a model performs over new, unseen data. 

6. Performance Metrics

The traditional performance metrics such as accuracy, precision, recall, F1-score for classification; Mean Squared Error, R-squared for regression 

Here we discuss some of recent used performance metrics in machine learning models

a) Grid Search 
One of the simplest and most commonly used model selection techniques is grid search. In this approach, systematically different combinations of hyperparameters are tried and that gives the best performance chosen. It can be effective, but the main drawback will be computationally intensive, especially for complex models and many parameters. 
b) Random Search 
Similar to grid search, random search doesn't check all possible combinations. Instead, it randomly chooses a subset of the hyperparameter combinations. The random search method often runs much faster than the grid search method and yet achieves equally good results. 
c) Bayesian Optimization 
Bayesian optimization is a smarter approach to model selection. Instead of just randomly searching for the best hyperparameters, it uses probability models to predict which parameters are likely to perform best and focuses on evaluating those. This method is efficient and often finds better results than grid or random search. 

d) Cross-Validation
This method involves using cross-validation to evaluate multiple models and selecting the one with the best average performance. Instead of relying on a single train-test split, cross-validation divides the dataset into multiple parts and trains the model on different subsets. This helps to ensure that the model’s performance is not just due to a specific split of data. By averaging the results from different splits, we get how well the model will perform on new, unseen data. 
This approach reduces the risk of overfitting and helps in selecting a good model. 

(Source: Model Selection for Machine Learning - GeeksforGeeks) 

Model Development Techniques 

A) Linear Regression 
(Predicting Continuous Outcomes) 

Linear regression is a foundational statistical method and a supervised machine learning algorithm employed to model the linear relationship between a continuous outcome variable and one or more explanatory variables. It is specifically used when the variable to be predicted is continuous in nature.  The primary purpose of linear regression is to identify a "best fit line" that passes through the observed data points. This line is then used to predict continuous values of an outcome (dependent or response) variable based on the values of one or more explanatory (independent or predictor) variables. The model quantifies how much and in what direction (positive or negative) the outcome variable is expected to change as the explanatory variable changes.For a simple linear regression model involving                                       
                                 y= a+b1​x1​
Where: y represents the predicted label or the output of the model. a is the bias of the model, which is analogous to the y-intercept in a standard algebraic linear equation (y=mx+c). In machine learning, bias is a parameter calculated during training.   b1​ is the weight of the feature, akin to the slope in the algebraic equation. x1​ represents the feature, which serves as the input to the model. 

Linear regression can be readily extended to include two or more explanatory variables; a model is known as multiple linear regression. In this case, each feature possesses its own distinct weight. The equation for a model with multiple features would be:                        
                            y=a+b1​x1​+b2​x2​+⋯+bn​xn 
Assumptions: For reliable predictions, linear regression models rely on certain assumptions: 
  • Linearity: It assumes that there is a linear relationship between the outcome variable and the explanatory variables.   
  • No Collinearity: In multiple linear regression, it is assumed that the explanatory variables are not highly correlated with each other (i.e., no multicollinearity). High correlation between predictors can make it difficult to reliably determine the individual effect of each variable on the outcome.    

B) Logistic Regression: 
(Classifying Categorical Outcomes) 

Logistic regression is a supervised machine learning algorithm primarily used for classification problems, where the goal is to predict a discrete or categorical outcome (e.g., Yes/No, 0/1). Unlike linear regression, which predicts continuous values, logistic regression predicts the probability that an input belongs to a specific class. The model is: 

or    
Where,          
Ln - Natural Log,    
Ln (P/1-P) - log odds ratio of non out obtaining the event / output (Y=0)    P - probability of obtaining the output (Y=1),   
alpha- coefficient and beta's are coefficients of independent variables in the model​ 

Categorical Output: The key distinction from linear regression is its application when the outcome variable is categorical, such as determining whether a loan is approved or not, or classifying an email as spam or not spam.   

Probability Prediction: The model outputs a probability value, typically ranging between 0 and 1. A threshold, often set at 0.5, is then used to classify the outcome into one of the predefined categories (e.g., "approved" or "not approved").   

Sigmoid (Logistic) Function: Instead of fitting a straight line, logistic regression employs an S-shaped curve, known as the sigmoid or logistic function. 
This function maps any real-valued input into a probability between 0 and 1, making it perfectly suited for modeling probabilities which are inherently bounded within this range.   

Log-Odds (Logit Function): Since probabilities are constrained between 0 and 1, directly modeling them with a linear function is problematic. Logistic regression addresses this by working with the logarithm of the odds, known as the logit function. This transformation maps the probability scale from to the entire real number line (−∞, ∞), allowing a linear relationship to be assumed between the independent variables and the log-odds of the outcome. 
The equation is often expressed as:  
                  
                     log(p(x)/(1−p(x)))=β0​+β1​⋅x1​.  
Exponentiating and rearranging this yields the sigmoid function, ensuring the predicted value remains within the range. 

Odds Ratio: The odds ratio is used to interpret the effect of the model's coefficients. It indicates how the odds of the outcome change for a one-unit increase in an input variable. A positive coefficient implies that the log-odds and, consequently, the probability of the outcome increase with the input. A negative coefficient indicates a decrease, while a zero coefficient suggests no effect.
  
Maximum Likelihood Estimation (MLE): The coefficients of the logistic regression model are typically estimated using Maximum Likelihood Estimation. MLE aims to find the parameter values that maximize the "likelihood" of observing the given dataset under the logistic regression model. Gradient descent, an iterative optimization algorithm, is commonly used to find these parameters by maximizing the log-likelihood function.   
Types Logistic regression can be broadly categorized into three main types based on the nature of the dependent variable : 
  • Binomial Logistic Regression: This is the most common type, used when the dependent variable has only two possible categories (e.g., Yes/No, Pass/Fail, 0/1). It is applied to binary classification problems.   
  • Multinomial Logistic Regression: Employed when the dependent variable has three or more possible categories that do not have a natural order. An example includes classifying animals into categories like "cat," "dog," or "sheep". 
  • Ordinal Logistic Regression: This type is used when the dependent variable has three or more categories that possess a natural order or ranking (e.g., ratings such as "low," "medium," and "high")

Assumptions                                                                                

For logistic regression to yield reliable and stable results, certain assumptions should be met:  

  1. Independent Observations: Each data point is assumed to be independent of the others, implying no correlation or dependence between input samples.  
  2. Binary Dependent Variable: For binomial logistic regression, the dependent variable must strictly be binary. 
  3. Linearity of Log-Odds: The model assumes a linear relationship between the independent variables and the log odds of the dependent variable. This means that predictors affect the log odds in a linear fashion.  
  4. No Outliers: The dataset should ideally not contain extreme outliers, as they can significantly distort the estimation of the logistic regression coefficients.  
  5. Large Sample Size: A sufficiently large sample size is generally required to produce reliable and stable results                                                                          

Applications  

Logistic regression is widely utilized for prediction and classification problems across various sectors:  

Fraud Detection: Models can identify data anomalies that are predictive of fraudulent activities, which is particularly valuable for banking and financial institutions. 

Disease Prediction: In medicine, this approach can predict the likelihood of disease or illness in a given population, enabling healthcare organizations to implement preventive care strategies for at-risk individuals.  

Churn Prediction: It helps identify behaviors indicative of customer or employee churn. For instance, HR teams can predict if high-performing employees are at risk of leaving, prompting discussions about workplace culture or compensation. Sales organizations can identify clients at risk of taking their business elsewhere, allowing for proactive retention strategies.

C) Decision Trees 

(Flowchart-like Decisions)

Decision Trees are supervised learning algorithms that can be applied to both classification and regression tasks. Their hierarchical, flowchart-like structure makes them highly interpretable, mimicking human decision-making processes.  

A decision tree operates on the principle of recursively splitting a dataset into increasingly "pure" subsets based on the values of input features. Ideally, all items within a resulting group belong to the same class.

The structure of a decision tree includes: 

Root Node: This is the starting point of the tree, representing the entire dataset.  

Internal Nodes: These nodes represent attribute tests or decision points, where the data is split based on the value of a particular feature.   

Branches: These lines connect nodes and represent the various attribute values or the outcomes of a decision test. 

Leaf Nodes: These are the terminal nodes of the tree, where no further splitting occurs. Each leaf node represents a final decision or prediction.

Steps in Flow chart 

The process of building a decision tree involves a series of steps that resemble a flowchart:

Start with the Entire Dataset: The entire dataset is initially considered as the root node of the decision tree.  

Choose the Best Question (Attribute): At each node, the algorithm selects the "best" attribute to divide the dataset. This selection is based on attribute selection measures that quantify how well a feature separates data into distinct groups.  

Split the Data into Subsets: The dataset is then divided into groups corresponding to the answers to the chosen question. Each answer leads to a new branch and a new subset of data.  

Continue Splitting Recursively: This process of asking questions and splitting data continues recursively for each new subset. The splitting stops when a subset becomes "pure" (meaning all instances belong to the same class) or when no more attributes remain to split on.  

Assign Final Decisions (Leaf Nodes): Once the splitting stops, the terminal nodes (leaf nodes) are assigned labels, which become the predictions of the tree.  

 Evolution Methods 

a) Information Gain: This measure quantifies how much the uncertainty (entropy) of the dataset decreases after a split. A feature that results in the highest Information Gain is chosen, as it creates clearer and more homogeneous groups. Entropy itself measures the impurity or disorder of a collection of examples; higher entropy indicates greater uncertainty.

b) Gini Index (Gini Impurity): The Gini Index measures the probability of incorrectly classifying a randomly chosen element from the dataset if it were randomly labeled according to the distribution of labels in the subset. A lower Gini Index indicates a more homogeneous or pure distribution within a node, making it a preferred splitting criterion. It is often faster to compute than entropy.

Pruning (Preventing Overfitting)

Pruning is a crucial technique employed to prevent overfitting in decision trees. Overfitting occurs when a tree becomes excessively deep and learns the training data too well, including its noise and irrelevant patterns. This memorization leads to poor performance on new, unseen data.  

Mechanism: Pruning reduces the complexity of the tree by removing branches that have little predictive power or that are likely to represent noise rather than generalizable patterns.  Benefit: By simplifying the tree, pruning improves the model's ability to generalize better to new data, making it more robust. It also contributes to a simpler and faster-to-deploy model.  

Advantages and Disadvantages

Decision trees are widely used due to their distinct advantages, but they also come with certain limitations:

Advantages:

Easy to Understand & Interpret: Their visual, flowchart-like structure makes the decision-making process straightforward to follow and explain. 

Versatility: They can be effectively applied to both classification and regression problems.  

No Need for Feature Scaling: Unlike many other machine learning models, decision trees do not typically require features to be scaled or normalized.   

Handles Non-linear Relationships: They are capable of capturing complex, non-linear relationships between features and outcomes.  

Disadvantages:

Overfitting: They are prone to overfitting the training data if the tree is allowed to grow too deep, leading to poor generalization. 

Instability: Small changes in the training data can sometimes lead to significant alterations in the tree's structure and, consequently, its predictions.  

Bias: They can exhibit a bias towards features that have a large number of distinct categories.  

Difficulty in Capturing Complex Interactions: Decision trees may struggle to capture highly intricate interactions between multiple features.  

The repeated emphasis on decision trees' "interpretability" and their "flowchart-like" nature is not merely a theoretical advantage; it translates directly into practical benefits. In domains such as medical diagnosis or loan approval, where understanding the rationale behind a decision is as crucial as the decision itself, decision trees provide a transparent, step-by-step reasoning process. This transparency fosters trust in the model's output and empowers domain experts to validate and act upon the derived insights, highlighting interpretability as a key driver for real-world adoption and accountability in machine learning. 

Applications

Decision trees find diverse applications across various fields due to their simplicity and interpretability:  

  • Loan Approval in Banking: Banks utilize decision trees to assess loan applications based on factors like credit score, income, and employment status, enabling quick and reliable approval or rejection decisions.  

  • Medical Diagnosis: In healthcare, they assist in diagnosing diseases by modeling symptoms and patient data, providing a clear diagnostic pathway.  

  • Customer Churn Prediction: Companies employ decision trees to predict whether a customer will leave or stay, based on behavioral patterns and purchase history, allowing for proactive retention strategies.   

  • Fraud Detection: In finance, they are used to identify suspicious activities, such as credit card fraud, by analyzing past transaction patterns.  

  • Recommendation Systems: They can suggest products or content to users based on their preferences and past interactions.

D) Support Vector Machines (SVM)

(Finding Optimal Boundaries)

Support Vector Machines (SVMs) are powerful supervised machine learning algorithms used for both classification and regression tasks. Their core objective is to identify the "best boundary," known as a hyperplane, that effectively separates different classes in the data with the largest possible margin.

Principles

  • Hyperplane: In SVM, a hyperplane serves as the decision boundary that separates different classes within the feature space. For linearly separable data, this boundary is represented by a linear equation, such as .   

  • Support Vectors: These are the data points that are closest to the hyperplane. They are critically important because they are the only data points that influence the position and orientation of the hyperplane and, consequently, the margin. SVM's robustness to outliers stems from its characteristic of focusing solely on these support vectors to define the optimal boundary.  

  • Margin: The margin is defined as the distance between the hyperplane and the support vectors from each class. The primary goal of SVM is to maximize this margin. A wider margin generally indicates better classification performance and improved generalization to new, unseen data.  

  • Hard vs. Soft Margin:

    • Hard Margin: A "hard margin" refers to a maximum-margin hyperplane that perfectly separates the data points without any misclassifications. This approach is ideal when the data is perfectly linearly separable. 

    • Soft Margin: In real-world scenarios where data is often not perfectly separable or contains outliers, a "soft margin" is introduced. This approach allows for some misclassifications by incorporating "slack variables," which balance the objective of maximizing the margin with a penalty for misclassifications. This makes the model more flexible and robust to noisy data.  

A significant strength of SVM lies in its ability to handle data that is not linearly separable (i.e., cannot be divided by a simple straight line in 2D or a flat plane in higher dimensions). 

  • SVM employs a technique known as the "kernel trick." Kernel functions implicitly map the original data into a higher-dimensional space where it can become linearly separable, without explicitly computing the coordinates in that transformed space. This allows SVM to efficiently work with non-linear data. 

  • Common Kernel Types:

    • Linear Kernel: This is the simplest kernel, suitable for data that is already linearly separable in its original feature space. It is computationally efficient.  

    • Polynomial Kernel: Useful for data where the relationship between features is polynomial, mapping the data into a higher-dimensional space defined by polynomial combinations of the original features.   

    • Radial Basis Function (RBF) Kernel (Gaussian Kernel): This is a very popular and effective kernel for capturing more complex, non-linear relationships in the data. The gamma (γ) parameter within the RBF kernel defines how far the influence of a single training example reaches.  

The "kernel trick" represents a brilliant conceptual advancement. It enables SVM to solve problems that are inherently non-linear in their original data space by implicitly transforming them into a higher-dimensional space where a linear separation becomes feasible. This illustrates a powerful problem-solving paradigm in machine learning: sometimes, the most effective solution is not to devise a more complex boundary within the original space, but rather to transform the problem into a new space where a simpler (linear) solution can be applied. This highlights the profound importance of feature transformations and the power of mathematical abstraction in machine learning.

Advantages and Limitations

SVMs are celebrated for their strengths but also have certain limitations:

Advantages:

  • High Accuracy: Particularly effective in high-dimensional or non-linear problems.  
  • Robustness: The underlying optimization problem in SVM is convex, which guarantees that any local optimum found is also the global optimum, leading to stable and reliable solutions. Versatility: Through the use of kernel methods, SVMs can effectively model and handle non-linear relationships within data. 
  • Effective in High Dimensions: SVMs perform well even when the number of features (dimensions) in the dataset exceeds the number of training samples. 
  • Sparse Solutions: SVM naturally leads to sparse solutions, meaning that only the support vectors (a subset of the training data) influence the model. This can result in more interpretable models in many cases.                                                                                                           
  • Limitations
  • Scalability: SVMs can face challenges when dealing with very large datasets due to computational and memory inefficiencies, as training time can increase significantly with the number of samples. 
  • Parameter Tuning: The performance of an SVM model is highly dependent on the careful selection and tuning of its hyperparameters, particularly the choice of kernel and parameters like C (penalty parameter) and gamma (γ). This often requires extensive cross-validation to find optimal settings.  
  • Interpretability: While the concept of support vectors provides some insight, understanding the precise influence of each individual feature on the decision can be challenging, especially with complex kernels.

Applications

SVMs have a wide range of real-world applications across various domains :  

  • Medical Diagnosis: Extensively used in healthcare to distinguish between malignant and benign tumors based on imaging and genetic data. They have shown promise in classifying patterns in MRI scans for early detection of Alzheimer's disease.  

  • Financial Forecasting: Utilized in the financial industry for predicting market trends and assessing credit risks, by capturing non-linear patterns that traditional regression techniques might miss.  

  • Image Recognition: A cornerstone of modern computer vision, providing robust performance in applications like facial recognition, object detection, and handwriting recognition (e.g., on the MNIST dataset), due to their effectiveness in handling high-dimensional image features.

E) Discriminant Analysis 

(Group Separation)

Discriminant Analysis is a statistical technique employed to classify observations into predefined groups and to identify which features are most significant in distinguishing between these groups. It is particularly effective in fields such as pattern recognition, machine learning and data classification for predicting categorical outcomes.

The core purpose of discriminant analysis is to establish a decision boundary that effectively separates various classes based on input features. A key objective is to determine which features contribute most significantly to differentiating between these classes.

Discriminant Function, a mathematical equation that combines multiple variables in a way that best discriminates between the categories.

Example: Imagine a scenario where a school wants to classify students into those likely to pass or fail an exam based on past performance, study hours and health conditions. Discriminant Analysis can help create a model that determines the likelihood of each outcome, facilitating targeted interventions.

There are four main types of Discriminant Analysis: 

Linear Discriminant Analysis (LDA) 

Quadratic Discriminant Analysis (QDA)

Multiple Discriminant Analysis (MDA)

Gaussian Discriminant Analysis (GDA)

While both aim at class separation, they differ in how they compute the decision boundary.

i) Linear Discriminant Analysis (LDA)

Linear Discriminant Analysis (LDA) assumes that data from different classes are generated from Gaussian distributions that share the same covariance matrix but possess different means. This fundamental assumption leads to the creation of a linear decision boundary between classes.

Discriminant Analysis, especially LDA, is not merely a classifier but also a powerful dimensionality reduction technique that preserves class discriminatory information. This implies that DA is fundamentally about finding the optimal projection of data that maximizes the separation between classes. This transformation simplifies the data, making it easier for subsequent classifiers to perform well and also aids significantly in data visualization. This highlights that sometimes the key to solving a complex classification problem lies in intelligently reshaping the feature space rather than solely relying on a complex algorithm.

The magic of Linear Discriminant Analysis lies in its ability to transform features in a dataset from a high-dimensional space to a lower-dimensional space without losing the essence of the original dataset. This transformation is based on linear combinations of features that provide the best separation between classes.

At the heart of LDA lies the concept of maximising the ratio of between-class variance to within-class variance in any particular dataset, leading to optimal separability. The formula can be expressed as:

            between-class variance / within-class variance 
By aiming for a high ratio, LDA ensures that the differences between groups are highlighted while the similarities within each group are minimised.

Implementing LDA involves calculating the mean and variance for each class, followed by computing the between-class and within-class scatter matrices. The eigenvectors of the ratio of these scatter matrices then form the directions of the linear discriminants.

Assumptions: The primary assumption of LDA is that data from different classes are normally distributed with equal covariance matrices.  

ii) Quadratic Discriminant Analysis (QDA)

Quadratic Discriminant Analysis (QDA) is an extension of Linear Discriminant Analysis (LDA) that allows for the separation of observations with a quadratic decision surface rather than a linear one. This method is particularly useful when classes exhibit distinct covariance structures, making it a versatile tool in statistical classification and machine learning applications. 

The hallmark of Quadratic Discriminant Analysis lies in its ability to accurately model and separate classes that have different variance-covariance structures. Unlike its linear counterpart, QDA does not assume homogeneity of variances across groups. This flexibility allows QDA to capture more complex relationships between variables, offering a nuanced approach to classification problems. Another key feature of QDA includes its capacity to work well with nonlinear relationships among variables. Since QDA models the decision boundary in a quadratic form, it can effectively manage datasets where class separability requires a more intricate boundary.

Quadratic Decision Surface: It is a surface created in multivariable space by a quadratic equation that separates different classes within a dataset. The equation of a quadratic decision surface can be expressed as
with A, B, C, D, E, and F being constants that define the curvature and position of the surface.

Example: Consider a dataset comprising two features that represent the scores of students in mathematics and science, with the target variable being the distinction between pass and fail. If the boundary separating the passes from the fails follows a curved pattern, then QDA can effectively model this non-linear boundary, accurately classifying the students.

iii) Multiple Discriminant Analysis (MDA) 

Multiple Discriminant Analysis (MDA) stands out as a statistical technique aimed at class discrimination and dimensionality reduction, leveraging linear combinations of predictors. By focusing on maximising the separation between multiple classes, MDA serves as a robust method for pattern recognition and classification problems.

Multiple Discriminant Analysis extends the capabilities of Linear Discriminant Analysis (LDA) to scenarios where there are more than two classes to predict. The essence of MDA is to find axes that maximise the separation between these multiple classes while also minimising the variance within each class. At the heart of MDA lies the calculation of eigenvalues and eigenvectors from the scatter matrices—both within-class and between-class. The eigenvectors corresponding to the largest eigenvalues are the directions that ensure maximum class separability.

Scatter matrices: In the context of MDA, the within-class scatter matrix measures the variance within each class, while the between-class scatter matrix quantifies the separation between different classes.
Example: Consider a study aiming to classify consumer products into three categories based on features like price, quality, and utility scores. MDA would identify the combinations of these features that best differentiate the categories, helping in the creation of a predictive model.

iv) Gaussian Discriminant Analysis (GDA): 

Gaussian Discriminant Analysis (GDA) is a powerful statistical technique for classifying datasets when the assumptions about normal distribution in features across classes hold true. By leveraging properties of the Gaussian (or normal) distribution, GDA provides a framework for understanding how classes differ and how to predict class membership for new observations.

Gaussian Discriminant Analysis works under the assumption that the features from each class in the dataset are drawn from a Gaussian distribution. This implies that for each class, features conform to a bell curve-patterned distribution, characterized by mean and covariance  The main goal of GDA is to estimate these parameters and use them to determine the most likely class for a given observation.

Gaussian Distribution: Also known as the normal distribution, it’s a function that illustrates how the values of a variable are distributed. It is symmetric around its mean, showing that data near the mean are more frequent in occurrence than data far from the mean.

Example: If you were to look at the heights of people within a certain population, you would likely see that most individuals cluster around the average height (the mean) with decreasing numbers of people being either much taller or much shorter. This pattern of distribution forms the familiar 'bell curve' associated with Gaussian distributions.

GDA operates under the assumption that features from each class are drawn from a Gaussian (normal) distribution. LDA is a special case of GDA when the covariance matrices are assumed to be equal across classes, leading to a linear decision boundary. QDA is another special case of GDA when class-specific covariance matrices are allowed, resulting in a quadratic decision boundary.


Applications

Discriminant Analysis enhances machine learning models by improving accuracy and facilitating data visualization. It optimizes class separability and can reduce feature space without significant information loss, which is vital for computational efficiency and model performance.  

  • Healthcare: LDA is used to classify patient results into diagnostic categories, such as distinguishing between benign and malignant tumor samples.  

  • Finance: QDA is employed in credit scoring models to differentiate risk profiles, helping banks manage risk by categorizing loan applicants. MDA is used to predict corporate bankruptcies by analyzing financial ratios.  

  • Marketing: Companies utilize discriminant analysis to segment markets and tailor products or services to specific customer groups, maximizing outreach and satisfaction. For instance, LDA can identify differences in shopping patterns between loyal and churning customers to formulate targeted retention strategies.  

  • Facial Recognition & Bioinformatics: LDA's ability to simplify complex datasets while preserving class discriminatory information makes it a preferred choice in these fields

  • Environmental Science: Researchers apply MDA to classify areas based on pollution levels, helping in environmental protection efforts.

Example- 

Suppose we have measurements of different iris flowers (e.g., sepal length, sepal width, petal length, petal width) and you want to classify them into one of three species: Setosa, Versicolor and Virginica. LDA can help us find the best way to separate these species using their measurements.
  • Original Features (Variables):

    • Sepal Length (cm)

    • Sepal Width (cm)

    • Petal Length (cm)

    • Petal Width (cm)

  • Target Classes (Species):

    • Setosa

    • Versicolor

    • Virginica

Procedure of LDA: To transform these 4 original features into a smaller number of new, artificial features (called "Linear Discriminants" or "LDs") that best separate the three flower species. For a classification problem with K classes, LDA can at most create (K-1) linear discriminants. In our case (3 classes), LDA can create up to 2 Linear Discriminants.

Execution LDA (Conceptual Code/Process):

  1. Load the Data: Load the Iris dataset.

  2. Separate Features (X) and Target (y):

    • X = [Sepal Length, Sepal Width, Petal Length, Petal Width]

    • y = [Setosa, Versicolor, Virginica]

  3. Apply LDA: One would use an LDA algorithm .

Output: The LDA algorithm will generate new transformed data.

S.NoSepal LengthSepal WidthPetal LengthPetal WidthSpecies
15.13.51.40.2Setosa
24.931.40.2Setosa
3...............
46.33.362.5Virginica
55.82.75.11.9Virginica
................
.

After applying LDA with n_components =2, the output would look like this (conceptual numerical values):

         
S.NoLD1 (Linear Discriminant 1)LD2 (Linear Discriminant 2)Species
1-2.710.49Setosa
2-2.690.17Setosa
..........
.2.050.65Virginica
.1.58-0.11Virginica
..........

Explanation

Dimensionality Reduction: If you notice that your original 4 features (Sepal Length, Sepal Width, Petal Length, Petal Width) have been reduced to 2 new features (LD1, LD2). This is a core output of LDA. These LDs are linear combinations of the original features. 
For instance:

LD1=c1 SepalLength+c2 * SepalWidth+c3* PetalLength+c4 * PetalWidth 

LD2=d1 SepalLength+d2 * SepalWidth+d3* PetalLength+d4 * PetalWidt

One can also often retrieve measures of how much "discriminatory power" each linear discriminant captures. The first discriminant (LD1) typically captures the most variance that separates the classes, the second (LD2) captures the next most, and so on.

Proportion of Variance Explained by LD1: 0.991 (meaning LD1 captures 99.1% of the information useful for separating classes)

Proportion of Variance Explained by LD2: 0.009 (meaning LD2 captures the remaining 0.9%)

LD1 is overwhelmingly the most important dimension for distinguishing between your iris species, and LD2 adds a small but significant amount of additional discriminatory information.

R-Code:
install.packages(c("MASS", "tidyverse", "caret", "pROC", "klaR", "mvtnorm"))
library(MASS)
library(tidyverse)
library(caret)
library(pROC)
library(klaR)
library(mvtnorm)
library(readxl)
iris <- read_excel("C:/Users/Administrator/Desktop/iris.xlsx")
View(iris)
# After loading, it is good practice to perform initial data exploration to understand 
# the structure and summary statistics of the dataset. 
# The str() function provides a concise summary of the data's structure, including variable types, 
# while summary() offers descriptive statistics for each variable.
str(iris)
summary(iris)
##Splitting Data into Training and Testing Sets
 # To evaluate a model's performance on unseen data and to prevent overfitting, 
 it is essential to split the dataset into training and testing sets. 
 # The caret package provides the createDataPartition() function, 
 # which is particularly useful as it performs stratified sampling. 
 # This ensures that the proportion of classes in both the training and testing sets is maintained, 
 # preventing bias in model training and evaluation. For reproducibility, 
 # it is crucial to set a seed for the random number generator using
set.seed(123)
train_index <- createDataPartition(iris$species, p = 0.8, list = FALSE)
train_data <- iris[train_index, ]
test_data <- iris[-train_index, ]
library(MASS) # Ensure MASS is loaded
preproc_param <- train_data %>% preProcess(method = c("center", "scale"))
train_transform <- preproc_param %>% predict(train_data)
test_transform <- preproc_param %>% predict(test_data)
# Assuming train_transform is prepared as above
model_lda <- lda(species ~., data = train_transform)
print(model_lda)

F) Neural Networks: Brain-Inspired Learning

Neural Networks (NNs), particularly Artificial Neural Networks (ANNs), are computational models inspired by the structure and function of the human brain. They consist of interconnected layers of artificial neurons that learn from data, powering a wide array of advanced applications in machine learning.

Neural networks are composed of a large number of interconnected processing elements, often referred to as nodes or artificial neurons. These nodes are linked to other nodes via connections that carry "weights," which hold information about the input signal. A typical ANN architecture comprises three main types of layers:

  • Input Layer: This is the initial layer that receives the raw input data or features from the source.   
  • Hidden Layers: These are intermediate layers responsible for processing the input data and learning intricate patterns. Each neuron in a hidden layer computes a weighted sum of its inputs from the previous layer, adds a bias term and then passes this result through an activation function, which introduces non-linearity to the network. 
  • Output Layer: This is the final layer of the network, which produces the network's prediction or output.
Neural networks learn adaptively and efficiently by understanding how to solve a specific problem from the training data they receive. The learning process involves several key steps:
  1. Forward Propagation: Input data flows through the network from the input layer, through the hidden layers, to the output layer. Each neuron calculates its output based on the weighted sum of its inputs and its activation function. 

  2. Loss Calculation: Once a prediction is generated by the output layer, it is compared to the actual desired output. The discrepancy between the predicted and actual output is quantified as "loss" (or error).   

  3. Back Propagation: To minimize this calculated loss, the network employs a process called back propagation. This involves propagating the error backward through the network, from the output layer to the input layer, to determine how much each weight and bias contributed to the error.  

  4. Gradient Descent: The weights and biases of the connections are then adjusted iteratively using an optimization algorithm, most commonly gradient descent. This algorithm essentially moves in the direction opposite to the gradient of the loss function, progressively minimizing the loss and improving the model's accuracy.   

This iterative cycle of forward propagation, loss calculation, and back propagation with weight adjustments allows the neural network to learn complex patterns and relationships within the data, refining its ability to make accurate predictions.

Applications

Neural networks are employed across a wide range of domains due to their ability to perform complex computations quickly and respond rapidly:  

  • Image and Speech Recognition: Powering applications like self-driving cars and voice assistants by identifying patterns in visual and auditory data. 

  • Natural Language Processing (NLP): Used for analyzing and understanding human language, enabling sentiment analysis, chatbots, language translation, and text generation.  

  • Financial Forecasting: Employed to predict financial trends, including stock prices and currency exchange rates.  

  • Medical Diagnosis: Assisting in diagnosing diseases from medical images, predicting patient outcomes, and aiding in drug discovery.  

  • Recommendation Systems: Driving recommendation engines that suggest products, movies, or content based on user preferences.  

  • Gaming: Used to develop intelligent AI opponents, adaptive game environments, and procedural content generation.

G) Bayesian Networks: Probabilistic Relationships

Bayesian Networks are graphical models that represent probabilistic relationships among a set of variables using directed acyclic graphs (DAGs). They are invaluable tools for data analysis and decision-making by explicitly modeling uncertainty and dependencies. 

The structure of a Bayesian Netwok:   

  • Nodes: Each node in the graph represents a variable (e.g., a symptom, a disease, a financial indicator).   

  • Edges: Directed edges between nodes indicate direct dependencies or causal relationships. An arrow from node A to node B signifies that A directly influences B.  

  • Conditional Probability Distributions (CPDs): The strength of these relationships is quantified using conditional probability distributions. For each node, a CPD defines the probability of that node's state given the states of its parent nodes (the nodes with incoming edges).

Bayesian Networks provide a robust framework for reasoning under uncertainty: 

  • Modeling Relationships: They represent variables and their dependencies, allowing for a structured understanding of how different factors interact probabilistically.  

  • Inference: A key capability is inference, where the network can deduce the probability of certain outcomes given new evidence or incomplete information. This process utilizes Bayes' theorem and various algorithms, such as Variable Elimination or Belief Propagation, to update probabilities across the network.  

  • Updating Beliefs: As new data or evidence becomes available, Bayesian Networks can dynamically update the probabilities of various nodes. This adaptive nature enables informed and dynamic decision-making.  

  • Handling Missing Data: Bayesian Networks are inherently designed to handle missing data effectively. They can infer missing values based on the existing relationships and available information within the network, enhancing the robustness of analysis. 

Unlike many other machine learning models that focus primarily on direct prediction, Bayesian Networks explicitly represent probabilistic relationships and conditional dependencies. The use of directed edges suggests a notion of influence or causality. This means that these networks are not merely predicting what will happen, but they also provide a framework for understanding why (or at least, the probabilistic relationships that lead to it). This makes them particularly powerful for scenarios where understanding the underlying generative process or the impact of interventions (e.g., in medical diagnosis or risk assessment) is crucial, moving beyond simple correlation to a more nuanced understanding of complex system behavior.

Bayesian Networks are powerful tools for modeling probabilistic relationships and performing inference across diverse domains :  

  • Medical Diagnosis: They help diagnose diseases by modeling the relationships between symptoms, conditions, and test results, assisting healthcare professionals in making informed decisions. 

  • Risk Assessment: In finance and insurance, they are used to assess risks by analyzing the probability of different outcomes and their interdependencies, leading to better decision-making and strategic planning.  

  • Machine Learning: Employed in artificial intelligence for classification tasks, allowing for the integration of prior knowledge and improving predictive accuracy.  

  • Natural Language Processing: Bayesian Networks aid in understanding and processing human languages by modeling the relationships between words and their meanings.  

  • Genetics: In bioinformatics, they help understand genetic interactions and the likelihood of traits in populations. 

  • Supply Chain Management: They can optimize inventory and logistics by assessing probabilities of demand and supply fluctuations, thereby enhancing operational efficiency.

H) Cox Regression: Survival Analysis

Cox Regression, formally known as the Cox Proportional Hazards (CPH) model, is a statistical method predominantly used in survival time analysis. Its purpose is to determine the influence of various independent variables on the time until a specific event occurs (known as a time-to-event outcome).

Cox regression is specifically designed to analyze survival times of subjects, which could refer to the time until an event such as disease progression, equipment failure, or customer churn. It allows researchers and analysts to determine how multiple independent variables—which can be continuous, binary, or categorical—affect the duration until such an event.

Assumptions

Like any statistical model, Cox regression relies on several important assumptions for its results to be valid:

Proportional Hazards Assumption: As detailed above, the hazard ratio for any covariate must remain constant over time. 

Independence Assumption: The survival times of distinct individuals in the study population are assumed to be independent of each other, given the values of the predictor variables. This means one individual's survival time should not influence another's. 

Uninformative Censoring: Censoring occurs when a subject's event time is not fully observed (e.g., they drop out of the study). The assumption of uninformative censoring means that those who are censored have the same risk of experiencing the study endpoint as those who continue to be followed. For example, a patient moving to another city leading to censoring is typically uninformative, whereas censoring due to a severe side effect of treatment would be informative and problematic.   

Linearity Assumption: The relationship between the predictor variables and the logarithm of the hazard rate is assumed to be linear.  

No Multicollinearity: There should be no perfect multicollinearity among the predictor variables, meaning they should not be highly correlated with each other.  

No Outliers: Extreme outliers that significantly deviate from the overall data pattern can distort the estimated coefficients and are assumed not to be present.  

No Effect Modification: It is assumed that there is no interaction between the predictor variables, meaning the effect of one predictor on the outcome does not depend on the level of another predictor.

Unlike traditional regression models that predict a static outcome, Cox Regression is specifically designed for "survival time analysis," focusing on when an event occurs and what factors influence that timing. The proportional hazards assumption is a critical nuance, implying that the relative risk associated with a predictor remains constant over the observation period. This highlights the model's utility in domains where the duration until an event and the factors affecting that duration, are of primary interest, moving beyond simple classification to a more dynamic understanding of risk over time.

Applications

Cox regression is widely applied in various fields where time-to-event data is analyzed :  

  • Medical Research: Frequently used to evaluate the effects of different treatments or patient characteristics on patient survival times, or to predict disease progression. 

  • Reliability Engineering: Applied to predict the time until mechanical components fail.

  • Customer Churn Analysis: Used to model the time until a customer unsubscribes from a service.


I) Association Rules

(Discovering Item Relationships)

Association rule learning is an unsupervised machine learning technique used in data mining. Its primary goal is to identify frequent patterns, connections, and dependencies among different groups of items, often referred to as "item sets," within large datasets, particularly transactional databases.

The fundamental purpose of association rules is to uncover interesting relationships between variables without requiring labeled outputs, which is characteristic of unsupervised learning.

A common and illustrative application is

Market Basket Analysis, where retailers analyze customer purchase patterns to identify which products are frequently bought together (e.g., "customers who buy bread also buy butter"). The insights gained from association rules help businesses make data-driven decisions, such as recommending complementary products to customers or optimizing store layouts for increased sales.

Apriori Algorithm: Principles and Steps

The Apriori algorithm is a foundational and widely used method for generating association rules. It operates by identifying frequent itemsets (combinations of items that frequently appear together) and then uses these itemsets to derive association rules.   

  • Apriori Property: A key principle underlying the Apriori algorithm is the "Apriori property," which states that "if an itemset is infrequent, then all of its supersets must also be infrequent." This property is crucial for efficiently reducing the search space by eliminating infrequent itemsets early in the process, thereby improving computational efficiency.  

  • Steps: The Apriori algorithm typically involves iterative steps to generate frequent itemsets and association rules :  

    1. Frequent Itemsets Generation: The algorithm first identifies unique items (referred to as 1-itemsets) and their frequencies within the dataset. It then combines items that appear together above a specified minimum support threshold into candidate itemsets and filters out those that are infrequent.   

    2. Expand and Prune Itemsets: Using the Apriori property, the algorithm iteratively combines frequent itemsets from the previous step to form larger candidate itemsets. Any combinations that are found to have a lower probability (i.e., are infrequent) are pruned, further reducing the search space.  

    3. Repeat: Steps 1 and 2 are repeated until all frequent itemsets that meet the defined support threshold are generated exhaustively. Once these frequent itemsets are identified, the strength of the generated associations and relationships can be investigated.

Metrics

To evaluate the strength and interestingness of association rules, three key metrics are used

Support: This metric measures how frequently an item or an itemset appears in the dataset. It is defined as the ratio of the number of transactions containing the itemset to the total number of transactions.  

Confidence: Confidence quantifies how often the consequent (the item(s) on the right-hand side of the rule) appears when the antecedent (the item(s) on the left-hand side) is present. It is expressed as P(B|A), the conditional probability of B given A. A minimum confidence threshold is typically applied to filter out weak associations.  

Lift: Lift measures the strength of association between the antecedent and the consequent, indicating whether there is a real relationship or if their co-occurrence is merely coincidental. A lift value greater than 1 suggests a positive association (the items are more likely to be bought together than independently), while a lift value less than 1 indicates a negative association, and a lift of 1 suggests no association

Association rules extend beyond simple correlations by identifying frequent co-occurrences and the strength of implication between items. The metrics of Support, Confidence, and particularly Lift, are designed to filter out trivial relationships and highlight truly synergistic or surprising patterns. This enables businesses to uncover "hidden synergies" in customer behavior, such as effective product bundles or cross-selling opportunities, which might not be apparent from individual product sales data. This directly impacts strategic decisions like optimizing store layouts or personalizing recommendations, demonstrating how unsupervised learning can extract actionable intelligence without requiring explicit labels.

Applications

Association rules, particularly those generated by the Apriori algorithm, have diverse applications :   

  • Market Basket Analysis: As described, this is the most common application, used by retailers to understand customer purchasing habits.  

  • Fraud Detection: In banking and finance, the algorithm can detect suspicious transactions by identifying unusual patterns that deviate from normal behavior.   

  • Medical Diagnosis: Association rules can assist in healthcare by finding relationships between symptoms, treatments, and diseases, aiding in diagnostic processes.  

  • Web Usage Mining: Identifying frequent browsing patterns on websites to enhance navigation and improve user experience.



👈     👍     👉


CARP - Classification and Regression Prediction Model
R-script
## First install the library rpart
library(rpart)
## Read your data file (iris flower data set)
library(readxl)
Iris_Flower_Data_set <- read_excel("C:/Users/Administrator/Downloads/Iris-Flower Data set.xlsx")
View(Iris_Flower_Data_set)
### Build Classification Tree
fit_class=rpart(Species ~ Sepal.Length + Sepal.Width + Petal.Length + Petal.Width,
                   data = iris, method = "class")
print(fit_class)

## plot
plot(fit_class, uniform = TRUE, branch = 0.5, compress = TRUE,
     margin = 0.1, main = "Classification Tree (Iris)")
text(fit_class, use.n = TRUE, all = TRUE, fancy = FALSE, cex = 0.8)

## Predict
pred_class <- predict(fit_class, iris, type = "class")

# Confusion Matrix
table(Predicted = pred_class, Actual = iris$Species)



### Regression Tree

# Read or import the data set i.e. Boston Housing Price

library(readxl)
BostonHousing <- read_excel("C:/Users/Administrator/Desktop/BostonHousing.xlsx")
View(BostonHousing)

# If you need to run regression install MASS


library(MASS)

# Build Regression Tree
fit_reg <- rpart(medv ~ ., data = BostonHousing, method = "anova")

# Print tree
print(fit_reg)

# Plot tree
rpart.plot(fit_reg, type = 3, fallen.leaves = TRUE)

# Predict
pred_reg <- predict(fit_reg, BostonHousing)
print(pred_reg)

# Evaluate with RMSE
rmse <- sqrt(mean((BostonHousing$medv - pred_reg)^2))
cat("RMSE:", rmse, "\n")

## Check whether model is good or bad based on RMSE
# If we get RMSE is less than the standard deviation of target or predicted variable then 
we say the model is reasonably good

sd(BostonHousing$medv, na.rm = FALSE)

## Check in other way our model is over or under estimate using ME

ME=mean(pred_reg-BostonHousing$medv)
print(ME)

# If you get ME is positive then the model tends to overestimate
# If you get ME is negative then the model tends to underestimate



Comments

Popular posts from this blog

Model Paper

Unit-4: Data Preparation and Model Management