IIT M FOUNDATION ET1 EXAM QPF1 S1 03 Sep - 03 Sep 2023

Concepts this paper tests

Auto-generated summary from the source site. Handy as a checklist; the lessons above explain each idea from scratch.

Here’s a structured summary of the core topics, concepts, principles, and equations essential for the Diploma Level: Machine Learning Practice (Computer-Based Exam). The summary is organized by topic domains and includes key equations where applicable.

1. Data Preprocessing

Key Concepts

• Scaling/Normalization:

  • StandardScaler: Standardizes features to have mean = 0 and variance = 1.

    • Formula:

where $\mu$ = mean, $\sigma$ = standard deviation.

• MaxAbsScaler: Scales features by their maximum absolute value (range: $[-1, 1]$).

  • Formula:

• MinMaxScaler: Scales features to a specified range (default: $[0, 1]$).

  • Formula:

• Encoding Categorical Data:

  • LabelEncoder: Converts categorical labels to numerical values (e.g., ["cat", "dog"] → [0, 1]).
  • OneHotEncoder: Converts categorical features into a binary matrix (one-hot encoding).
  • MultiLabelBinarizer: Converts multi-label categorical data into a binary matrix (e.g., ["large", "cold"] → [1, 0, 1, 0]).

• Handling Missing Data:

  • SimpleImputer: Replaces missing values with mean, median, or mode.

    • Example: strategy='mean' replaces NaN with the column mean.

• Polynomial Features:

  • PolynomialFeatures: Generates polynomial/interaction features (e.g., $x_1, x_2, x_1^2, x_1x_2$).

    • Parameter: degree=n (max polynomial degree).
    • Parameter: interaction_only=True (only interaction terms, no higher-order polynomials).

• Feature Union & Pipelines:

  • FeatureUnion: Combines multiple feature transformations (e.g., scaling + polynomial features).
  • Pipeline: Sequentially applies transformations and estimators (e.g., impute → scale → model).

2. Dimensionality Reduction

Key Concepts

• PCA (Principal Component Analysis):

  • Reduces dimensionality by projecting data onto principal components (orthogonal axes of max variance).
  • Parameter: n_components=k (number of principal components to keep).
  • Explained variance ratio: Indicates how much variance each PC captures.

3. Supervised Learning

Key Algorithms & Concepts

A. Linear Models

1. Linear Regression:

   • Equation:

• Ordinary Least Squares (OLS): Minimizes sum of squared residuals.

• Ridge (L2) & Lasso (L1) Regularization:

  • Ridge: Penalizes large coefficients ($L2$ norm).

• Lasso: Penalizes coefficients to sparsity ($L1$ norm).

• Elastic Net: Combines $L1$ and $L2$.

1. Logistic Regression:

   • Sigmoid function:

• Regularization: C (inverse of $\alpha$) controls strength.

• Solvers:

  • liblinear: Good for small datasets, supports $L1/L2$.
  • lbfgs: Default, fast for large datasets (only $L2$).
  • sgd: Stochastic Gradient Descent (supports partial_fit).

B. Naive Bayes

• GaussianNB: Assumes features follow a normal distribution.

  • Predicts posterior probabilities $P(y|X)$.
  • Uses Bayes' Theorem:

C. Support Vector Machines (SVM)

• SVC (Support Vector Classifier):

  • Kernel Trick: Maps data to higher dimensions for linear separation.

    • Kernels: linear, rbf (Gaussian), poly, sigmoid.
  • Regularization: C controls margin hardness.

D. Decision Trees & Ensembles

1. DecisionTreeClassifier:

   • Splitting Criteria:

     • Gini Impurity:

• Entropy:

• Hyperparameters:

  • max_depth: Limits tree depth.
  • min_samples_split: Min samples required to split a node.
  • min_samples_leaf: Min samples required in a leaf.

1. Random Forest:

   • Bagging (Bootstrap Aggregating): Trains multiple trees on random subsets of data/features.

   • Hyperparameters:

     • n_estimators: Number of trees.
     • max_features: Features considered for splitting (default: sqrt(n_features)).
     • bootstrap: Whether to use bootstrapping.

1. AdaBoost:

   • Boosting: Sequentially trains weak learners (e.g., shallow trees), weighting misclassified samples higher.

   • Hyperparameters:

     • n_estimators: Number of weak learners.
     • learning_rate: Shrinks contribution of each tree.

1. Gradient Boosting (GBM, XGBoost, LightGBM):

   • Boosting: Builds trees to correct errors of previous trees.

   • Hyperparameters:

     • learning_rate: Step size shrinkage.
     • n_estimators: Number of boosting stages.

E. Neural Networks (MLP)

• MLPClassifier (Multi-Layer Perceptron):

  • Architecture:

    • Input layer → Hidden layers (with activation functions) → Output layer.

  • Activation Functions:

    • ReLU: $f(x) = \max(0, x)$
    • Sigmoid: $f(x) = \frac{1}{1 + e^{-x}}$
    • Tanh: $f(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$

  • Hyperparameters:

    • hidden_layer_sizes: Number of neurons per layer (e.g., (100, 50)).
    • alpha: $L2$ regularization strength.
    • activation: relu, tanh, logistic.

  • Overfitting Control:

    • Increase alpha (regularization).
    • Reduce hidden_layer_sizes.

F. k-Nearest Neighbors (k-NN)

• KNeighborsClassifier:

  • Distance Metrics: euclidean, manhattan, minkowski.

  • Hyperparameters:

    • n_neighbors: Number of neighbors to consider.
    • weights: uniform (all neighbors equal) or distance (weight by inverse distance).

  • Decision Boundary:

    • Low n_neighbors → complex boundaries (prone to overfitting).
    • High n_neighbors → smooth boundaries (prone to underfitting).

G. Perceptron

• Linear classifier for binary classification.
• Update Rule:

where $\eta$ = learning rate.

4. Unsupervised Learning

Key Algorithms

A. Clustering

1. K-Means:

   • Objective: Minimize within-cluster sum of squares (WCSS).

   • Steps:

     1. Initialize centroids randomly.
     2. Assign points to nearest centroid.
     3. Update centroids as mean of assigned points.
     4. Repeat until convergence.

   • Hyperparameters:

     • n_clusters: Number of clusters ($k$).
     • Elbow Method: Plot WCSS vs. $k$ to find optimal $k$.

1. Hierarchical Clustering:

   • Linkage Methods:

     • Single Linkage: Min distance between clusters.
     • Complete Linkage: Max distance between clusters.
     • Average Linkage: Average distance between clusters.
     • Ward’s Linkage: Minimizes variance when merging clusters.

1. DBSCAN (Density-Based):

   • Groups points based on density.

   • Hyperparameters:

     • eps: Max distance between points to be considered neighbors.
     • min_samples: Min points to form a dense region.

B. Dimensionality Reduction

• PCA (Principal Component Analysis):

  • Transforms data into orthogonal components ordered by variance.
  • Explained Variance Ratio: Proportion of variance explained by each PC.

5. Model Evaluation & Metrics

Key Concepts

• Confusion Matrix: | | Predicted Positive | Predicted Negative | |---------------|--------------------|--------------------| | Actual Positive | True Positive (TP) | False Negative (FN) | | Actual Negative | False Positive (FP) | True Negative (TN) |

• Classification Metrics:

  • Accuracy:

• Precision (for class 1):

• Recall (Sensitivity):

• F1-Score:

• Regression Metrics:

  • Mean Squared Error (MSE):

• R² Score:

where $\text{SS}_{\text{res}} = \sum (y_i - \hat{y}_i)^2$, $\text{SS}_{\text{tot}} = \sum (y_i - \bar{y})^2$.

• Bias-Variance Tradeoff:

  • Underfitting (High Bias): Model is too simple.
  • Overfitting (High Variance): Model is too complex.
  • Regularization: Reduces variance by penalizing complexity.

6. Hyperparameter Tuning

Key Methods

• GridSearchCV:

  • Exhaustively searches over a parameter grid.
  • Uses cross-validation to evaluate performance.

• RandomizedSearchCV:

  • Samples random combinations from a parameter distribution.
  • More efficient than GridSearchCV for large spaces.

7. Incremental Learning

Key Concepts

• Partial Fit:

  • Allows models to be trained incrementally (useful for large datasets).

  • Supported by:

    • SGDClassifier/SGDRegressor
    • PassiveAggressiveClassifier
    • Perceptron
    • MultinomialNB/BernoulliNB

8. Ensemble Methods

Key Concepts

• Bagging (Bootstrap Aggregating):

  • Trains multiple models on random subsets of data.
  • Reduces variance (e.g., RandomForest, BaggingClassifier).

• Boosting:

  • Sequentially trains models to correct errors of previous models.
  • Reduces bias (e.g., AdaBoost, GradientBoosting).

• Stacking:

  • Combines multiple models using another model (meta-learner).

9. Text Feature Extraction

Key Methods

• CountVectorizer:

  • Converts text into a matrix of token counts.

• TF-IDF (Term Frequency-Inverse Document Frequency):

  • Weights terms by importance in a document relative to a corpus.
  • Formula:

where:

• $\text{TF}(t, d)$ = Term frequency in document $d$.
• $N$ = Total number of documents.
• $\text{DF}(t)$ = Number of documents containing term $t$.

10. Model Interpretation

Key Concepts

• Feature Importance:

  • Decision Trees/Random Forest: Based on Gini/entropy reduction.
  • Linear Models: Coefficients indicate feature impact.
• SHAP Values: Explain individual predictions by approximating Shapley values from game theory.

Graph 1: Data Preprocessing Workflow

Graph 2: Supervised Learning Algorithms

Graph 3: Unsupervised Learning

Graph 4: Model Evaluation & Tuning

Graph 5: Ensemble Methods

Graph 6: Neural Networks (MLP)

Graph 7: Text Feature Extraction