While deep learning has enabled tremendous progress on text and image datasets, it does not seem to demonstrate superiority on tabular data when compared with tree-based methods. We explore the paper Why do tree-based models still outperform deep learning on tabular data? in this post to gain some intuition as to why that is the case. In other words, we seek to understand which inductive biases make trees well-suited for tabular data. Results show that neural networks struggle to learn irregular patterns of the target function and are often not able to handle uninformative features present in some tabular datasets.
Neural Nets are Biased Towards Smooth Solutions
A key reason for NN underperformance compared to tree-based methods like Random Forests on certain tabular data tasks is because NNs struggle with creating precise decision boundaries for irregular or jagged patterns (especially when categorical feature are involved as opposed to strictly numeric features). In the benchmark datasets that the researchers prepared, tree-based methods outperformed neural networks in all cases but the outperformance was especially clear in Figure 2, where both numerical and categorical features are involved. Moreover, the authors noted that “Tree-based models are superior for every random search budget, and the performance gap stays wide even after a large number of random search iterations. This does not take into account that each random search iteration is generally slower for NNs than for tree-based models.”
It is hypothesized that the inability to fit irregular / jagged decision boundaries arises because Neural Networks rely on gradients, which are inherently designed for smooth, differentiable search spaces. When the decision boundaries involve sharp, irregular, or non-smooth functions, gradients fail to effectively optimize the model, leading to poor representation of these patterns. In contrast, tree-based methods excel in capturing such irregularities due to their hierarchical, recursive splitting process, which adapts naturally to complex, jagged patterns. If a smoothing kernel is applied on the train set, then the discrepancy in performance disappears, but this is mostly from a degradation of tree performance due to the information loss from smoothing. See below
In addition, tabular data is often sparse and exhibits clustering, where relationships are highly localized. Neural networks tend to underperform in these cases unless the dataset is massive and sufficiently representative. Tree-based models excel in such scenarios because they recursively condition on localized data structures, enabling them to fit these irregular patterns effectively.
Uninformative Features Affect Neural Nets More Strongly
Another significant factor, particularly relevant for large datasets encoding multiple relationships, is the impact of irrelevant features on model performance. Neural Networks appear highly sensitive to the inclusion of irrelevant or noisy features, often resulting in degraded performance and inefficient resource utilization during training. The authors of the study evaluated model performance when random or less important features were added to, or removed from, the dataset. Their findings revealed two noteworthy insights:
Removing a substantial number of features narrowed the performance gap between tree-based methods and Neural Networks. This highlights a key strength of tree-based models—their inherent ability to remain relatively unaffected by the inclusion of less relevant features.
Conversely, adding random features resulted in a much sharper decline in Neural Network performance compared to tree-based models. This effect may be partly mitigated in models with attention mechanisms, such as transformers, which appear to exhibit a stronger level of robustness to noisy features.
The resilience of tree-based methods can be attributed to the fundamental design of Decision Trees, which utilize concepts like Information Gain and Entropy to iteratively select the most informative features at each split. This systematic feature selection process ensures that Decision Trees and Random Forests focus on the most relevant paths, insulating them from the detrimental effects of irrelevant features.
Disclaimer
The information provided on TheSubstack (the "Substack") is strictly for informational and educational purposes only and should not be considered as investment or financial advice. The author is not a licensed financial advisor or tax professional and is not offering any professional services through this Substack. Investing in financial markets involves substantial risk, including possible loss of principal. Past performance is not indicative of future results. The author makes no representations or warranties about the completeness, accuracy, reliability, suitability, or availability of the information provided.
This Substack may contain links to external websites not affiliated with the author, and the accuracy of information on these sites is not guaranteed. Nothing contained in this Substack constitutes a solicitation, recommendation, endorsement, or offer to buy or sell any securities or other financial instruments. Always seek the advice of a qualified financial advisor before making any investment decisions.