Applied Machine Learning for Tabular Data

My experience with tabular data textbooks is that they introduce many different techniques, in a relatively un-opinionated way, which doesn't help give you intuitions on what your strategy for real-world problems should be. In practice, almost all tabular data problems I've encountered boil down to:

1. Make sure you don't have any impossible nonsense or data leakage in your features or target

2. Split your data in an intelligent way (temporally for time-series, group-wise for hierarchical data)

3. Try a really simple linear regression / logistic regression model to get a "dumb" baseline for your accuracy/error metric and make sure it is reasonable

4. If you need interpretability, consider a GAM, else, throw it into XGBoost and you'll get state of the art results

Like others say, usually just go for XGBoost, which is increasingly even shown (proven?) in literature, eg: https://arxiv.org/abs/2106.03253

You can also start with scikit-learn's `LogisticRegressionCV`, if a linear model is more palatable to your audience.

The bigger challenge is to reliably estimate how good your model is.

It's not about getting the best performance – it's about getting the "real" performance. How will your model _really_ do on future unseen datasets?

The answer to this challenge is cross-validation. But what are the questions?

There are 2 very different questions for which the answer is cross-validation.

One is, which hyper parameters to use with your model?

The second is, what is the generalization performance of the model?

This requires 2 separate applications (loops) of cross-validation. The authors of this book talk about this in terms of having a "validation set" and a "test set" (Sometimes, these terms are switched around, and there is also "holdout set". It's critical to know how you, and the rest of your team, are using these terms in your modeling)

A robust way to implement these 2 CV's is with nested cross-validation, readily available in many packages – and also, should be "fast enough" using modern computers.

One exercise that remains is: with nested CV, which model do you pick as your "production" model?

That is also a bit tricky. Reading things like the following can help: https://stats.stackexchange.com/q/65128/207989

EDIT: for those inclined, here is a paper on why you need 2 loops of CV: "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation" by Cawley and Talbot: https://jmlr.csail.mit.edu/papers/volume11/cawley10a/cawley1...

Spoiler: just use xgboost and you’re done

I've been teaching tabular data modeling to many clients. My usual model of choice these days is XGBoost. Thought, we can often go back to a strong linear model based on our insights from XGBoost.

I appreciate the section on independent component analysis (ICA). It's not well known and very underused. In my experience, it usually works great on heterogeneous tabular data - which PCA usually does poorly on.

Wow, those are some names I’ve not seen in awhile. I wonder how LLMs do at R?

Too many texts is annoying to read and understand. Bad representation.

What do you suggest to learn XGBoost?