Thanks, everyone, for being such active students, asking questions during lecture and during the break. It makes it really fun to teach the class so keep the questions coming!

I decided to put together a short summary of things that have come up in class here and will try to do the same for the upcoming weeks as well.

Data leakage

In class, we talked about data leakage quite a lot. Data leakage means that the test data that should be used only for providing performance quantifiers has been contaminated by some kind of analysis, meaning the test error rates or accuracies are overly optimistic.

Now, sometime data leakage is obvious such as when you use the same data to train and test. Other times it a bit more subtle, as when you explore and visualize the entire data to decide on data tranformations and then split the data into training and testing.

If you want to be completely free from data leakage, the test data should be split off from the rest of the data before you do anything else.

Cross-validation

The training data will be used to fit models but also to tune them. In class we discussed what goes inside the CV loop and what goes outside. The test data has already been split off of course. Now, anything that is data driven should really go inside the CV loop. For example, for your project you want to choose how many PCs to use. This tuning clearly goes inside the CV loop in order to select the number of PCs based on the CV error. However, where is the PC computed? On the full training data or inside each fold?

The PCs are data driven. Having access to the full training data to do the projection does use the leave-one-out fold so the CV errors will be optimistic and could affect tuning. To be completely free from data leakage here you should compute the PC on each training fold. In order to compute the validation error (and the test error) you need to project the validation and test data onto these fold specific PCs.

Using the notation from class, X=UDV' (svd) or Cov(X)=VD^2V'. On training data, compute PC(X_tr) = V_tr D_tr^2 V´_tr. Note, the PCs are X_trV_tr = U_tr D_tr, i.e. the V_tr matrix projects the data onto the low-dimensional "meta-features" for each fold. To create test PC and validation PCs you transform the data as PC(X_te) = X_te V_tr, PC(X_val) = X_val V_tr.

After you have used CV to select the optimal number of PCs you should update your classifier model on the full training data. That is, compute PCs on the full training data, using the number of PCs that the CV process suggested, train your classifiers. Then, project the test data onto the training PCs as above and evaluate your test error.

Finally, CV provides both the mean (across folds) validation error as well as the standard deviation. Remember to look at the standard deviations as well to determine if the choice of hyperparameter is "easy" or "hard" (how flat is the CV-error curve?).

Pre-filtering/dimension reduction.

In class we talked about not necessarily doing one or the other. You can of course pre-filter the data to remove noise feature that appear to be completely unrelated to the classes and then apply PCs. Note, do not touch the test data.

We talked about the types of pre-filtering mechanisms you can use. For multiclass data you can run into problems with the ANOVA-filtered feature perhaps mainly reflecting the separation of one class from the rest. What you can do then is use class-pairwise testing or run C tests: one-against-all to ensure that you include features that separate all classes.

Imbalanced classes

We discussed that it is important to look at different performance metrics when you have class imbalance since the global test error rate might be uninformative (e.g. just always predicting the majority class). We can use other metrics like kappa, sensitivity, F1 etc for validation and we can also choose to train our models using different metrics.

Standardizing data

PCA is scale dependent and one should usually standardize the data prior to using PCA. However, during class we discussed if this could perhaps create a problem for us at times. If there are outliers in the data or some features are skewed, then the underpinning logic of using the covariance matrix is a bit off (means and sds are not good summary statistics in this setting). If so, you should consider variable transformations first, e.g. logs or square roots or perhaps removing outliers/truncation.

On the other side of the spectrum, if some features have low variance then standardizing them might blow up the feature (dividing by ~0). Now, perhaps these features are just noise and should be pre-filtered away. However, you can also use a regularized estimate of the standard deviation, sd+delta, where delta is a small fudge factor, to stabilize the standardization procedure.

In class we discussed how this is essentially what is done in ridge regression (X'X+lambda I) replaced (X'X), as well as in regularized discriminant analysis.

Side note on random PCA

I mentioned in class that one can reduce dimension through random projects rather than diagonalizing the covariance matrix (PCA). Here are some fun papers on this topic:

VerySparseRandomProjections.pdf,

PCAvsRandomProjectionAndKNN.pdf

and also an introduction to the topic at scikit-learn: scikit-learn

Random projects and deep learning

Finally, for those that are curious, here is a brand new paper on the connection between random features and deep learning. The references in the introduction of the paper give a nice snapshot of the research field.

RainbowDeepNetworks.pdf