Being skeptical of a model's math claim

When I look at a model’s announced success figure, it doesn’t seem very likely to be true. As they stand, these models aren’t that good at arithmetic; so a number like “99.5%” looks overstated.

The general lesson: don’t take a model’s benchmark claim at face value without questioning which task it was measured on and the model’s known weaknesses.