The measuring stick

Why would anyone start with a model they already know is too simple?

I asked that, and the answer turned out to be more interesting than the question.

We start simple on purpose. Not as a rough first draft to be replaced, but as a measuring stick. The simple version tells us what the problem looks like when nobody has been clever about it yet. Everything built afterwards has to beat that, and the size of the win is worth knowing.

Often the clever model wins out. Sometimes it does not.

One of my projects is a churn model, the kind that predicts which customers are about to leave. The plan was the clever version. The simple one was only there to be beaten. It was not. The two came out the same, and the honest recommendation was the simple one, because it does the same job and I can explain it to the person paying for it.

Without the measuring stick I would have shipped the clever version, and nobody would ever have known it was not needed.

Finding out whether the clever thing is worth it is one of the fun things about this work.

Finding out is also the part that gets skipped, because skipping it is invisible.

If the clever thing cannot beat the obvious thing, it is not clever.