Home / Formulas / MLP

MLP formula sheet

Collected automatically from the lessons. Print it, or read it the night before.

Week 1 - Your first end-to-end ML project

• RMSE $=\sqrt{\frac1n\sum(y-\hat y)^2}$; MAE $=\frac1n\sum|y-\hat y|$
• Standardise $\frac{x-\mu}\sigma$; min-max $\frac{x-\min}{\max-\min}$
• train_test_split(X, y, test_size=0.25, random_state=..., stratify=y)
• SimpleImputer(strategy="mean"|"median"|"most_frequent"|"constant"); learned statistics_
• OneHotEncoder(handle_unknown="ignore"), OrdinalEncoder(), LabelEncoder() (targets)

Week 2 - The project continued: pipelines and tuning

• Fits $=\text{combinations}\times\text{cv}\;(+1)$; list-of-dict grids add combinations
• RMSE from negated MSE scores: np.sqrt(-scores)
• Pipeline([(name, step), ...]), make_pipeline(...); tune with "step__param"
• ColumnTransformer([(name, transformer, columns)], remainder="drop"|"passthrough")
• cross_val_score(est, X, y, cv=5, scoring=...); LeaveOneOut(); KFold(n_splits=k)

Week 3 - Linear regression and gradient descent in sklearn

• MSE cost $J=\frac1n\|X\mathbf w-\mathbf y\|^2$; gradient $\frac2nX^T(X\mathbf w-\mathbf y)$
• invscaling: $\eta=\texttt{eta0}/t^{\texttt{power\_t}}$; adaptive: divide by 5 when stalled; constant: $\eta=\texttt{eta0}$
• SGDRegressor defaults: squared_error, l2, alpha=1e-4, max_iter=1000, tol=1e-3, invscaling, eta0=0.01, power_t=0.25
• SGDClassifier defaults: hinge, learning_rate="optimal"

Week 4 - Polynomial regression and regularisation

• Polynomial feature count: $\binom{n+d}{d}$ (with bias), minus 1 without
• Ridge: $\|\mathbf y-X\mathbf w\|^2+\alpha\|\mathbf w\|^2$; $\mathbf w=(X^TX+\alpha I)^{-1}X^T\mathbf y$
• Lasso (sklearn): $\frac1{2n}\|\mathbf y-X\mathbf w\|^2+\alpha\|\mathbf w\|_1$
• Defaults: alpha=1.0 (Ridge, Lasso, ElasticNet), l1_ratio=0.5 (ElasticNet)
• More regularisation: alpha $\uparrow$; C $\downarrow$

Week 5 - Logistic regression in sklearn

• $\sigma(z)=\frac1{1+e^{-z}}$; softmax $p_k=\frac{e^{z_k}}{\sum e^{z_j}}$
• Log loss $-\frac1n\sum[y\ln p+(1-y)\ln(1-p)]$
• predict_proba $\to(m,K)$; decision_function $\to(m,)$ binary, $(m,K)$ multi-class; coef_ $\to(1,d)$ or $(K,d)$

Week 6 - Binary classifiers and how to judge them

• accuracy $=\frac{TP+TN}{n}$; precision $=\frac{TP}{TP+FP}$; recall (TPR) $=\frac{TP}{TP+FN}$; specificity $=\frac{TN}{TN+FP}$; FPR $=\frac{FP}{FP+TN}$
• $F_1=\frac{2PR}{P+R}$
• confusion_matrix(y_true, y_pred) $=[[TN,FP],[FN,TP]]$
• precision_recall_curve, roc_curve, roc_auc_score, classification_report

Week 7 - Multiclass classifiers

• OvR classifiers $=K$; OvO classifiers $=\frac{K(K-1)}2$ (10 classes: 10 vs 45)
• precision_score(y, pred, average="macro"|"weighted"|"micro")
• confusion_matrix(y_true, y_pred): rows = true, columns = predicted

Week 8 - SVMs in sklearn

• Poly kernel $(\gamma\langle\mathbf x,\mathbf x'\rangle+\texttt{coef0})^{\texttt{degree}}$; RBF $e^{-\gamma\|\mathbf x-\mathbf x'\|^2}$; gamma="scale" $=1/(n_{feat}\,\mathrm{Var}(X))$
• Margin $=2/\|\mathbf w\|$
• Defaults: SVC(C=1, kernel="rbf", degree=3, gamma="scale", coef0=0); LinearSVC(C=1, penalty="l2", loss="squared_hinge"); SVR(epsilon=0.1)

Week 9 - Decision trees and ensembles

• Gini $=1-\sum p_k^2$; entropy $=-\sum p_k\log_2p_k$; cost $=\frac{m_L}mG_L+\frac{m_R}mG_R$
• Defaults: tree criterion="gini", max_depth=None, min_samples_leaf=1; BaggingClassifier(n_estimators=10, bootstrap=True); RandomForestClassifier(n_estimators=100, max_features="sqrt")
• OOB fraction $\approx0.37$ per tree; oob_score=True $\to$ oob_score_

Week 10 - Random forests, boosting and tuning

• AdaBoost (sklearn, 2 classes): $\alpha_t=\eta\ln\frac{1-\varepsilon_t}{\varepsilon_t}$; textbook $\frac12\ln\frac{1-\varepsilon_t}{\varepsilon_t}$
• Gradient boosting: $F_m=F_{m-1}+\eta\,h_m(\text{residuals})$
• Fits $=\prod(\text{list lengths})\times\text{cv}$; range(a, b, step) excludes $b$

Week 11 - Neural networks in sklearn

• Parameters: $\sum_\ell(m_\ell n_\ell+n_\ell)$ over layers; n_layers_ = input + hidden + output
• Defaults: hidden_layer_sizes=(100,), activation="relu", solver="adam", alpha=1e-4, learning_rate_init=1e-3, max_iter=200, tol=1e-4, n_iter_no_change=10

Week 12 - Unsupervised learning in sklearn

• Inertia $=\sum_i\|\mathbf x_i-\boldsymbol\mu_{z_i}\|^2$; silhouette in $[-1,1]$
• DBSCAN defaults eps=0.5, min_samples=5; noise label $-1$
• GMM parameters (full): $K(d+\frac{d(d+1)}2)+K-1$; BIC $=-2\ln L+p\ln n$ (lower is better)
• PCA: variance explained $=\sum_{\le k}\lambda/\sum\lambda$; reconstruction error $=\sum_{>k}\lambda$