IIT M FOUNDATION FN EXAM FDF2 24 Dec 2023 - 24 Dec 2023

Concepts this paper tests

Auto-generated summary from the source site. Handy as a checklist; the lessons above explain each idea from scratch.

Here’s a structured summary of core topics, concepts, principles, and equations essential for the Diploma Level: Machine Learning Practice (Computer-Based Exam). The summary is organized by themes (e.g., preprocessing, model evaluation, algorithms) with key equations, Python snippets, and conceptual notes. --- ---

1. Data Preprocessing

Key Concepts

• Standardization/Normalization:

  • Required for distance-based algorithms (e.g., KNN, SVM, neural networks).
  • StandardScaler: Transforms data to have mean = 0 and std = 1. Formula:

• MinMaxScaler: Scales data to a range (e.g., [0, 1]). Formula:

• Handling Missing Data:

  • SimpleImputer: Replaces missing values with mean/median/mode.

    from sklearn.impute import SimpleImputer
    si = SimpleImputer(strategy="mean")
    si.fit_transform(data)

  • Output: si.statistics_ gives the computed means/medians.

• Categorical Encoding:

  • OneHotEncoder: Converts categorical features to binary columns.

    from sklearn.preprocessing import OneHotEncoder
    ohe = OneHotEncoder(sparse_output=False)
    encoded = ohe.fit_transform(data[['category']])

  • LabelEncoder: Converts categories to integers (not recommended for features).

    from sklearn.preprocessing import LabelEncoder
    le = LabelEncoder()
    y_encoded = le.fit_transform(y)

• Feature Selection:

  • SelectKBest: Selects top-k features based on statistical tests (e.g., chi-squared, ANOVA).

    from sklearn.feature_selection import SelectKBest, chi2
    skb = SelectKBest(chi2, k=3)
    X_selected = skb.fit_transform(X, y)

    • skb.scores_: Scores for each feature.
    • skb.pvalues_: p-values for hypothesis testing.

2. Model Evaluation

Key Metrics

• Regression Metrics:

  • R² (R-squared): Higher = better (max = 1).

• RMSE (Root Mean Squared Error): Lower = better.

• MAE (Mean Absolute Error): Lower = better.

• Classification Metrics:

  • Accuracy: $(TP + TN) / (TP + TN + FP + FN)$.
  • Precision: $TP / (TP + FP)$.
  • Recall: $TP / (TP + FN)$.
  • F1-Score: Harmonic mean of precision/recall.

• Cross-Validation:

  • GridSearchCV: Exhaustive search over hyperparameters.

    from sklearn.model_selection import GridSearchCV
    param_grid = {'C': [0.1, 1, 10], 'kernel': ['linear', 'rbf']}
    grid_search = GridSearchCV(SVC(), param_grid, cv=5)
    grid_search.fit(X, y)

  • RandomizedSearchCV: Random sampling of hyperparameters.

    from sklearn.model_selection import RandomizedSearchCV
    param_dist = {'C': loguniform(1e-4, 1e0), 'kernel': ['linear', 'rbf']}
    random_search = RandomizedSearchCV(SVC(), param_dist, n_iter=10, cv=3)

  • LeaveOneOut (LOO): Uses n-1 samples for training, 1 for testing.

    from sklearn.model_selection import LeaveOneOut, cross_val_score
    loo = LeaveOneOut()
    scores = cross_val_score(model, X, y, cv=loo)

3. Supervised Learning Algorithms

Linear Models

• Linear Regression:

  • Predicts continuous output: $\hat{y} = w^T x + b$.
  • Ridge (L2 Regularization):

• Higher $\alpha$ → more regularization (prevents overfitting).

  • Lasso (L1 Regularization):

• Can shrink coefficients to 0 (feature selection).

• Logistic Regression:

  • Predicts probabilities: $P(y=1) = \sigma(w^T x + b)$, where $\sigma$ is the sigmoid.
  • Loss: Log loss (cross-entropy).

• SGDClassifier/Regressor:

  • Supports hinge loss (SVM), log loss (logistic), squared loss (linear regression).

  • Key Parameters:

    • loss: 'hinge', 'log', 'squared_error'.
    • penalty: 'l1', 'l2', 'elasticnet'.
    • alpha: Regularization strength.
    • warm_start: Reuses previous solution for faster training.

  • Example:

    from sklearn.linear_model import SGDClassifier
    clf = SGDClassifier(loss="hinge", penalty="l2", alpha=0.001)

Support Vector Machines (SVM)

• SVC/SVR:

  • Kernel Trick: Maps data to higher dimensions for linear separation.

    • Kernels: 'linear', 'poly', 'rbf', 'sigmoid'.

  • Regularization (C):

    • Low C → simpler boundary (underfitting).
    • High C → complex boundary (overfitting risk).

  • Example:

    from sklearn.svm import SVC
    clf = SVC(kernel='rbf', C=1.0)

Decision Trees

• Hyperparameters:

  • max_depth: Maximum tree depth.
  • min_samples_split: Minimum samples to split a node.
  • min_samples_leaf: Minimum samples in a leaf.
  • Overfitting Risk: Deeper trees → higher variance.
• Pruning: Reduces tree complexity to avoid overfitting.

• Example:

  from sklearn.tree import DecisionTreeClassifier
  clf = DecisionTreeClassifier(max_depth=3, min_samples_leaf=5)

Neural Networks (MLP)

• MLPClassifier/Regressor:

  • Architecture: Input layer → hidden layers → output layer.

  • Key Parameters:

    • hidden_layer_sizes: Tuple specifying neurons per layer (e.g., (10, 5)).
    • activation: 'logistic', 'tanh', 'relu'.
    • alpha: L2 regularization strength.
    • solver: 'adam', 'sgd', 'lbfgs'.
  • Total Parameters: For 1 hidden layer with h neurons, n features, o outputs:

• Example:

  from sklearn.neural_network import MLPClassifier
  clf = MLPClassifier(hidden_layer_sizes=(10,), alpha=0.01)

Ensemble Methods

• Bagging (e.g., Random Forest):

  • Trains multiple models on bootstrapped samples.

  • RandomForestClassifier:

    from sklearn.ensemble import RandomForestClassifier
    clf = RandomForestClassifier(n_estimators=100, max_depth=5)

  • BaggingClassifier:

    from sklearn.ensemble import BaggingClassifier
    bag_clf = BaggingClassifier(KNeighborsClassifier(), n_estimators=30)

• Boosting (e.g., AdaBoost, Gradient Boosting):

  • Sequentially corrects errors of previous models.

• Voting Classifier:

  • Combines predictions from multiple models (hard/soft voting).

  • Example:

    from sklearn.ensemble import VotingClassifier
    eclf = VotingClassifier(
        estimators=[('lr', LogisticRegression()), ('svm', SVC())],
        voting='soft'
    )

4. Unsupervised Learning

Clustering

• K-Means:

  • Partitions data into k clusters by minimizing inertia (within-cluster sum of squares).

  • Elbow Method: Plot inertia vs. k to find optimal k.

    from sklearn.cluster import KMeans
    inertias = []
    for k in range(1, 11):
        kmeans = KMeans(n_clusters=k)
        kmeans.fit(X)
        inertias.append(kmeans.inertia_)
    plt.plot(range(1, 11), inertias)

  • Types of Clustering:

    • Centroid-based: K-Means.
    • Density-based: DBSCAN.
    • Connectivity-based: Hierarchical clustering.

Dimensionality Reduction

• PCA (Principal Component Analysis):

  • Projects data to lower dimensions while preserving variance.

• t-SNE:

  • Non-linear technique for visualization.

5. Model-Specific Notes

K-Nearest Neighbors (KNN)

• Distance Metrics: Euclidean, Manhattan, Minkowski.

• Hyperparameters:

  • n_neighbors: Number of neighbors to consider.
  • weights: 'uniform' or 'distance' (closer neighbors have more influence).
• Scaling: Critical (features should be standardized).

Naive Bayes

• GaussianNB: Assumes features follow a normal distribution.
• MultinomialNB: For discrete counts (e.g., text classification).

Text Vectorization

• CountVectorizer:

  • Converts text to a matrix of token counts.
  • max_features: Limits vocabulary size.

  • Example:

    from sklearn.feature_extraction.text import CountVectorizer
    vectorizer = CountVectorizer(max_features=1000)
    X = vectorizer.fit_transform(texts)

6. Practical Tips

• Overfitting/Underfitting:

  • Overfitting: High training score, low test score.

    • Solutions: Regularization, pruning, cross-validation.

  • Underfitting: Low training/test scores.

    • Solutions: Increase model complexity, add features.

• Hyperparameter Tuning:

  • Use GridSearchCV or RandomizedSearchCV.
  • Log-uniform distributions (e.g., loguniform(1e-4, 1e0)) for parameters like alpha.

• Neural Networks:

  • Hidden Layer Sizes: Start with a single layer (e.g., 10 neurons), then experiment.

  • Activation Functions:

    • Output layer: 'identity' (regression), 'logistic' (binary classification).

• Imbalanced Data:

  • Use class weights (class_weight='balanced') or resampling (SMOTE).

7. Python Snippets Cheat Sheet

8. Common Pitfalls

1. Forgetting to scale data for KNN/SVM/neural networks.
2. Ignoring class imbalance in classification tasks.
3. Using LabelEncoder for features (use OneHotEncoder instead).
4. Not setting random_state for reproducibility.
5. Overlooking feature importance in tree-based models.

6. Misinterpreting predict vs. predict_proba:

   • predict: Returns class labels.
   • predict_proba: Returns class probabilities.

Graph 2: Supervised Learning Algorithms

Graph 3: Model Evaluation

Graph 4: Unsupervised Learning

Graph 5: Neural Networks

Key Equations Summary