IIT M IMPROVEMENT AN EXAM QDD3 22 - 22 Dec 2024

Concepts this paper tests

Auto-generated summary from the source site. Handy as a checklist; the lessons above explain each idea from scratch.

Here’s a structured summary of core topics, concepts, principles, and equations essential for the exam, followed by Mermaid knowledge graphs to visualize relationships.

Core Topics & Concepts

1. Data Preprocessing

Key Concepts:

• One-Hot Encoding (OneHotEncoder)

  • Converts categorical features into binary columns.
  • Example: ['apple', 'banana'] → [1, 0] (apple), [0, 1] (banana).

  • Equation:

    • If a categorical feature has k unique values, one-hot encoding produces k binary columns.
    • Total columns after encoding = k (categorical) + m (numerical).

• Feature Scaling (MinMaxScaler, StandardScaler)

  • MinMaxScaler: Scales features to [0, 1] range.

    • Equation:

• StandardScaler: Standardizes features to have mean=0 and std=1.

  • Equation:

• Where $\mu$ = mean, $\sigma$ = standard deviation.

• Handling Missing Data (dropna)

  • thresh=N: Drops rows with fewer than N non-null values.
  • Example: df.dropna(thresh=15) keeps rows with ≥15 non-null values.

• Polynomial Features (PolynomialFeatures)

  • Generates interaction terms and polynomial features.
  • Example: degree=2 for [a, b] → [1, a, b, a², ab, b²].
  • Output shape: If input has n features, output has $\binom{n + \text{degree}}{\text{degree}}$ features.

2. Model Evaluation

Key Metrics:

• Mean Absolute Error (MAE)

  • Equation:

• R² Score (Coefficient of Determination)

  • Equation:

• Accuracy, Precision, Recall, F1-Score, ROC-AUC

  • Used for classification tasks.
  • ROC-AUC: Area under the Receiver Operating Characteristic curve (for binary classification).

• Cross-Validation (cross_val_score)

  • Evaluates model performance across multiple folds.
  • Example: cross_val_score(model, X, y, cv=5, scoring='roc_auc').

3. Supervised Learning Algorithms

Key Models:

• Linear Regression (LinearRegression)

  • Equation:

• fit_intercept=True: Includes $\beta_0$ (bias term).

• Decision Trees (DecisionTreeClassifier)

  • Hyperparameters:

    • max_depth: Maximum depth of the tree.
    • min_samples_split: Minimum samples required to split a node.
    • min_samples_leaf: Minimum samples required at a leaf node.
  • Equation (Gini Impurity):

where $p_i$ = proportion of class $i$.

• Random Forest (RandomForestClassifier)

  • Hyperparameters:

    • n_estimators: Number of trees.
    • max_features: Number of features to consider for splits (default: sqrt(n_features)).

  • Effect of max_features:

    • Higher max_features → lower variance (more correlated trees).
    • Lower max_features → higher variance (more diverse trees).

• Gradient Boosting (GradientBoostingClassifier)

  • Hyperparameters:

    • learning_rate: Shrinkage factor for each tree’s contribution.
    • n_estimators: Number of boosting stages.
  • Equation (Boosting Update):

where $\nu$ = learning rate, $h_m(x)$ = weak learner.

• Support Vector Machines (SVM, SVC)

  • Kernel Trick: Transforms data into higher dimensions for linear separation.

  • Hyperparameters:

    • C: Regularization parameter (trade-off between margin and misclassification).
    • kernel: linear, rbf, poly.

• k-Nearest Neighbors (k-NN, KNeighborsClassifier)

  • Prediction: Majority vote (classification) or average (regression) of k nearest neighbors.
  • Distance Metrics: Euclidean, Manhattan, Minkowski.

• Neural Networks (MLPClassifier, MLPRegressor)

  • Hyperparameters:

    • hidden_layer_sizes: Tuple specifying neurons per layer (e.g., (5, 3) = 2 hidden layers).
    • alpha: L2 regularization strength (higher alpha → stronger regularization).

  • Effect of alpha:

    • Higher alpha → reduces overfitting (simpler model).

4. Unsupervised Learning

Key Models:

• k-Means Clustering (KMeans)

  • Hyperparameters:

    • n_clusters: Number of clusters.
    • init: Initialization method (k-means++ or random).
    • max_iter: Maximum iterations.

  • Inertia: Sum of squared distances to nearest cluster center.

    • Equation:

• Optimal k: Elbow method (plot inertia vs. k).

• Principal Component Analysis (PCA)

  • Dimensionality Reduction: Projects data onto orthogonal axes (principal components).
  • Variance Explained: First PC captures maximum variance.

5. Ensemble Methods

Key Techniques:

• Bagging (BaggingClassifier)

  • Hyperparameters:

    • n_estimators: Number of base models.
    • max_samples: Fraction of samples per base model (e.g., 0.5 = 50%).

  • Effect of max_samples:

    • Lower max_samples → higher diversity (reduces overfitting).

• Boosting (AdaBoostClassifier, GradientBoostingClassifier)

  • AdaBoost: Weights misclassified samples higher in subsequent iterations.
  • Gradient Boosting: Fits new models to residuals of previous models.

• Random Forest vs. Gradient Boosting: | Feature | Random Forest | Gradient Boosting | |-----------------------|----------------------------------------|-------------------------------------| | Bias-Variance | High variance, low bias | Low bias, high variance | | Training | Parallel (bagging) | Sequential (boosting) | | Hyperparameters | n_estimators, max_features | learning_rate, n_estimators |

6. Model Selection & Hyperparameter Tuning

Key Methods:

• Grid Search (GridSearchCV)

  • Exhaustively searches all parameter combinations.

  • Example:

    param_grid = {'n_estimators': [50, 100], 'learning_rate': [0.1, 0.01]}
    grid = GridSearchCV(GradientBoostingClassifier(), param_grid, cv=3)

• Randomized Search (RandomizedSearchCV)

  • Samples random parameter combinations (faster than grid search).

  • Example:

    param_dist = {'alpha': loguniform(1e-4, 1e0), 'penalty': ['l1', 'l2']}
    random_search = RandomizedSearchCV(SGDRegressor(), param_dist, n_iter=10)

• Cross-Validation Strategies:

  • StratifiedKFold: Preserves class distribution in folds (for classification).
  • KFold: Standard k-fold split.

7. Feature Selection & Dimensionality Reduction

Key Techniques:

• Variance Threshold (VarianceThreshold)

  • Removes low-variance features.
  • Example: VarianceThreshold(threshold=0.1).

• PCA (Principal Component Analysis)

  • Reduces dimensions while preserving variance.
  • Equation (Covariance Matrix):

• Eigenvectors of $\Sigma$ = principal components.

• Lasso Regression (L1 Regularization)

  • Performs feature selection by shrinking coefficients to zero.
  • Equation (Lasso Loss):

8. Clustering Evaluation

Key Metrics:

• Inertia (k-Means): Lower inertia → tighter clusters.

• Silhouette Score: Measures cluster cohesion and separation.

  • Equation:

where $a$ = mean intra-cluster distance, $b$ = mean nearest-cluster distance.

9. Incremental Learning (partial_fit)

Use Cases:

• Streaming Data: Update model incrementally without retraining on full dataset.
• Large Datasets: Train in mini-batches (e.g., SGDClassifier).

• Example:

  model = SGDClassifier()
  model.partial_fit(X_batch, y_batch, classes=[0, 1])

2. Supervised Learning

3. Unsupervised Learning

4. Ensemble Methods

5. Model Evaluation & Tuning

Key Equations Summary

Exam Tips

1. Focus on Equations: Memorize key formulas (e.g., MAE, R², Gini, scaling).

2. Hyperparameters: Know the effect of parameters like:

   • n_estimators (ensemble methods),
   • max_depth (decision trees),
   • alpha (regularization),
   • learning_rate (boosting).
3. Preprocessing: Understand when to use StandardScaler vs. MinMaxScaler.
4. Evaluation Metrics: Match metrics to problems (e.g., ROC-AUC for imbalanced classification).
5. Clustering: Know how to interpret inertia and silhouette scores.
6. Incremental Learning: Recognize use cases for partial_fit (streaming data, large datasets).