Instructions (don’t skip!!):

This homework is intended to be a stepping stone towards your final project while promoting friendly competition. You will be given two different data sets, and for each your challenge is to create the best performing modeling pipeline for a certain performance criteria.

For both data sets, your approach should incorporate:

  1. Feature engineering
  2. Pipelines and pre-processing steps (such as scaling, transformation, PCA, etc.)
  3. Model selection and hyperparameter tuning.

Additionally, you may only use functions that are native to Python, or contained in the sklearn, XGBoost, numpy, scipy, matplotlib or pandas libraries, but you may go beyond the functions in these libraries that we’ve explicitly seen in our labs, with the one exception being TF-IDF functions from sklearn, which are not allowed on this assignment (but are something you would be encouraged to use on the final project).

You do not need to perform a train-test split (unless you want to). And you are encouraged to use GridSearchCV() or RandomizedSearchCV() to help you decide upon a model, but once you’ve chosen a final model you must create a single code block that evaluates only the selected modeling pipeline using cross_val_predict(). This is because I don’t want to re-run your entire grid search to check the performance of your model.

All models will be evaluated via their cross-validated performance using two undisclosed random seeds to assign folds (to prevent favoring models that worked best on the default seed’s fold assignments). If you’d like to try out your model(s) using different random states you should note that you must pass an object created using Kfold() to cross_val_predict().

Finally, for each data set you must submit a paragraph explaining your modeling philosophy and choices - this paragraph will contribute to your score on the assignment. You should write in a scientific tone and describe each choice you made when crafting your modeling approach. Your paragraph will be evaluated on it’s clarity, level of detail, and technical correctness.

\(~\)

Question #1 (Spam Classification Challenge)

The goal of this challenge is to obtain the best cross-validated F1 score for the SMS spam classification data set. You can find the data below:

https://remiller1450.github.io/data/sms_spam.txt

These data are unstructured and you should use features engineered from the message text to predict the ‘label’.

Benchmarks:

\(~\)

Question #2 (Drugged Driving Detection Challenge)

The goal of this challenge is to obtain the best ROC-AUC score on a time-series classification task. The data provided below are a collection of 60-second driving samples from a simulated driving experiment where subjects drove on a curvy stretch of a 4-lane divided highway with a posted speed limit of 70 miles per hour. The driving simulator records driver inputs and vehicle states at a rate of 60 Hz, so each sample (defined by a unique value of Full_Sample_ID) consists of 3600 time steps.

https://remiller1450.github.io/data/drugdetection.csv

These data are from a high-fidelity driving simulator experiment where subjects were dosed on one of the following experimental conditions:

The goal of this application is to use vehicle input patterns to accurately predict target, which is 1 for the active drug conditions (ie: 'YM' and 'ZM') and 0 for the placebo condition, for each Full_Sample_ID.

Your approach should consider at least one (but does not need to use all) of the following predictors:

Because each sample contains 3600 time steps, you should use feature engineering to assemble a data set with 1 row per sample and then predict the variable ‘target’ from this data set.

Benchmarks: