Could Decision Trees Help With GenAI?

What if decision trees based ML algorithms could be a part of the solution for the overconfident and hallucinating Generative AI systems?

Everyone who follows me is keenly aware of my decision trees ML boosterism (pun totally intended). Decision trees based algorithms have stood the test of time as overall the most accurate and most robust approach to tabular data modeling. And yet, I will be the first one to admit that decision trees are not without their issues. I’ve been transparent about those in several of my tweets, and I’m likely to write a separate post about these shortcomings at some point soon, so I don’t want to delve too deeply into that whole topic. I will just focus on one alleged shortcoming in this post: the (purported) inability of decision trees to extrapolate beyond their training distribution. I qualify the preceding sentence with “purported”, as I have not observed this issue myself, probably largely due to the kinds of problems/datasets that I’ve worked with thus far. Nonetheless, based on my understanding of how decision trees work, there are lots of very theoretically sound reasons for why this is indeed the case. I will take it as a given in this post.

One of the main issues in Generative AI these days is the tendency of GenAI systems to “hallucinate” - come up with plausible-seeming yet completely wrong generated content. When it comes to creative uses of GenAI this is not that much of a problem, and in fact it can be a virtue - coming up with completely novel and imaginative creations is big part of the appeal of this use case. However, in much more constrained domains where output accuracy is non-negotiable (coding, research, medical diagnosis, etc.) such hallucinations can be very deleterious, to say the least.

Another big issue with going well beyond the bounds of the training data distribution is that of AI-alignment. This, again, is a very thorny topic that’s completely beyond the scope of this short post. However, form my understating, many of the problems of misaligned AI can also be traced to AI systems taking creative freedom with the output that they provide.

I am pretty sure that many, if not all of these “hallucination” issues can be tackled with some ingenious hacks within the existing deep learning framework, which is most likely what will end up. However, what if it’s possible to get at the same point by starting from scratch by training the ML models that are intrinsically incapable of hallucinating? I have no proof of this, but based on what I’ve said above and my intuition, I have a sense that tree-based ML models could come in handy in this domain. One of their biggest “setbacks” could come to be seen as a desirable regularizer for at least some GenAI use cases.