IIT M IMPROVEMENT FN EXAM QDD1 01 Sep - 01 Sep 2024

Concepts this paper tests

Auto-generated summary from the source site. Handy as a checklist; the lessons above explain each idea from scratch.

Here’s a structured summary of core topics, concepts, principles, and equations essential for the Diploma Level: Machine Learning Practice (Computer-Based Exam). The exam covers Pandas, Scikit-Learn, ML algorithms, hyperparameter tuning, evaluation metrics, and clustering. Key areas are data preprocessing, model training, evaluation, and interpretation of results.

1. Core Topics & Concepts

A. Data Handling with Pandas

Key Operations:

1. DataFrame Basics:

   • pd.DataFrame(data, columns)
   • df.shape → (rows, columns)
   • df.isna().sum().sum() → Total missing values.
   • df.value_counts() → Frequency of unique values in a column.
   • df.iloc[:, n] → Select column by index.
   • df.query("condition") → Filter rows based on a condition.

1. Handling Missing Data:

   • df.dropna() → Remove rows/columns with missing values.
   • Imputation (e.g., KNNImputer).

1. Descriptive Statistics:

   • df.mean() → Column-wise mean (ignores NaN by default).

Equations (if applicable):

• Mean (for non-missing values):

B. Machine Learning Algorithms

1. Linear Models

• Linear Regression:

  • Predicts continuous output.
  • Equation:

• Evaluation: $R^2$ score (coefficient of determination).

• Ridge/Lasso Regression:

  • Regularization:

    • Ridge (L2):

• Lasso (L1):

• Key Parameters:

  • alpha: Regularization strength (higher = more penalty).
  • max_iter: Maximum iterations for convergence.

• Logistic Regression:

  • Sigmoid Function:

• Class Weights:

  • class_weight='balanced' → Adjusts weights inversely proportional to class frequencies.

• Regularization:

  • C (inverse of alpha): Smaller C = stronger regularization.

• Perceptron:

  • Linear classifier for binary classification.
  • Update Rule:

• Key Parameters:

  • tol: Tolerance for stopping criteria.
  • shuffle: Whether to shuffle data between epochs.

• SGDClassifier/Regressor:

  • Stopping Criteria: max_iter (default: 1000).
  • Loss Functions: hinge (SVM), log (logistic regression).

2. Tree-Based Models

• Decision Trees:

  • Splitting Criteria:

    • Classification: Gini impurity or entropy.
    • Regression: MSE (mean squared error).

  • Key Parameters:

    • max_depth: Maximum depth of the tree.
    • min_samples_split: Minimum samples required to split a node.
    • min_samples_leaf: Minimum samples required at a leaf node.

• Random Forest:

  • Ensemble of decision trees.

  • Key Parameters:

    • n_estimators: Number of trees.
    • bootstrap: Whether to use bootstrapping for sample selection.

• Bagging & Boosting:

  • Bagging (e.g., BaggingClassifier):

    • Trains models in parallel on bootstrapped samples.

    • Key Parameters:

      • max_samples: Number of samples per subset.
      • bootstrap: Whether to sample with replacement.

  • Boosting (e.g., AdaBoost, Gradient Boosting):

    • Sequentially corrects errors from previous models.

3. Support Vector Machines (SVM)

• Key Concepts:

  • Support Vectors: Data points closest to the hyperplane.
  • Margin Maximization:

• Kernel Trick: Transforms data into higher dimensions for linear separation.

  • Key Parameters:
• C: Regularization parameter (trade-off between margin and classification error).
• kernel: linear, rbf, poly.

4. Nearest Neighbors (KNN)

• Key Concepts:

  • Distance Metrics: Euclidean, Manhattan.
  • Voting: Majority vote (classification) or average (regression).

• Key Parameters:

  • n_neighbors: Number of neighbors to consider.
  • weights: uniform or distance (closer neighbors have more influence).

5. Clustering

• K-Means:

  • Objective: Minimize within-cluster sum of squares (inertia).
  • Equation:

• Key Parameters:

  • n_clusters: Number of clusters (k).
  • init: Initialization method (random, k-means++).
  • n_init: Number of times to run with different centroid seeds.

• Agglomerative Clustering:

  • Hierarchical clustering (bottom-up).
  • Linkage Methods: ward, complete, average.
  • Handles Outliers: Assigns them to the nearest cluster.

6. Dummy Classifier

• Strategies:

  • most_frequent: Always predicts the majority class.
  • stratified: Predicts class labels based on training distribution.
• Use Case: Baseline model for comparison.

C. Model Evaluation

1. Metrics

• Classification:

  • Accuracy:

• Precision:

• Recall (Sensitivity):

• F1-Score:

• Confusion Matrix: Table of TP, TN, FP, FN.

• Regression:

  • Mean Squared Error (MSE):

• R² Score:

where:

• $\text{SS}_{\text{res}} = \sum (y_i - \hat{y}_i)^2$ (residual sum of squares),
• $\text{SS}_{\text{tot}} = \sum (y_i - \bar{y})^2$ (total sum of squares).

2. Cross-Validation

• Leave-One-Out Cross-Validation (LOOCV):

  • Trains n models (where n = number of samples).
  • Use Case: Small datasets.

• k-Fold Cross-Validation:

  • Splits data into k folds, trains on k-1 folds, validates on the remaining fold.

D. Feature Engineering

1. Polynomial Features

• PolynomialFeatures:

  • Generates polynomial and interaction features.

  • Key Parameters:

    • degree: Highest polynomial degree.
    • include_bias: Whether to include a bias term (intercept).

2. Feature Scaling

• Standardization (StandardScaler):

• Normalization (MinMaxScaler):

• Impacted Algorithms:

  • Distance-based (KNN, SVM, K-Means).
  • Gradient-based (Linear Regression, Logistic Regression, Neural Networks).

3. Feature Selection

• Variance Threshold: Removes low-variance features.
• Univariate Selection: Selects best features based on statistical tests (e.g., ANOVA F-value).

E. Hyperparameter Tuning

1. Grid Search

• GridSearchCV:

  • Exhaustively searches over a parameter grid.

  • Key Parameters:

    • param_grid: Dictionary of hyperparameters to tune.
    • cv: Number of cross-validation folds.

• Example:

  param_grid = {
      'n_estimators': [10, 20, 30],
      'max_features': [2, 3, 4],
      'bootstrap': [True, False]
  }

  • Total Combinations: $3 \times 3 \times 2 = 18$.

2. Random Search

• RandomizedSearchCV:

  • Samples hyperparameters randomly from a distribution.

F. Ensemble Methods

1. Voting Classifier

• Hard Voting: Majority vote of class labels.

• Soft Voting: Average of predicted probabilities.

  VotingClassifier(estimators=[('lr', clf1), ('rf', clf2)], voting='soft')

2. Bagging vs. Boosting

G. Data Leakage & Best Practices

• Data Snooping:

  • Peeking at test data during training → biased estimates.
  • Consequence: Overly optimistic performance on test sets.

• Avoiding Leakage:

  • Split data before preprocessing (e.g., scaling, imputation).
  • Use pipelines (sklearn.pipeline.Pipeline).

H. Dimensionality Reduction

1. PCA (Principal Component Analysis)

• Goal: Project data into lower dimensions while preserving variance.
• Equation:

Eigenvectors of the covariance matrix = principal components.

2. Key Equations Summary

Graph 2: Unsupervised Learning

Graph 3: Model Evaluation

Graph 4: Data Preprocessing

Graph 5: Hyperparameter Tuning

Graph 6: Ensemble Methods

4. Exam Tips

1. Pandas:

   • Know how to count non-missing values (df.shape[0]*df.shape[1] - df.isna().sum().sum()).
   • Use value_counts() for frequency tables.

1. Scikit-Learn:

   • Initialization: Check default parameters (e.g., SGDRegressor(max_iter=1000)).
   • Pipelines: Use make_pipeline for sequential preprocessing + modeling.
   • Evaluation: model.score(X, y) returns $R^2$ for regression and accuracy for classification.

1. Algorithms:

   • Decision Trees: max_depth controls overfitting.
   • SVM: C controls regularization; higher C = less regularization.
   • KNN: n_neighbors affects bias-variance tradeoff (high n = smoother boundaries).

1. Clustering:

   • K-Means: inertia_ = within-cluster sum of squares.
   • Agglomerative: Assigns outliers to the nearest cluster.

1. Evaluation Metrics:

   • Precision/Recall Tradeoff: High precision = fewer FP; high recall = fewer FN.
   • Dummy Classifier: Baseline for comparison (e.g., strategy='most_frequent').

1. Avoid Common Mistakes:

   • Data Leakage: Never fit scalers/imputers on the entire dataset before train-test split.
   • Parameter Names: alpha in Ridge/Lasso vs. C in LogisticRegression/SVM (they are inverses!).
   • Voting Classifier: voting='soft' requires estimators with predict_proba.