Flutterby™! : Gemini lying about arithmetic

This discussion about Google Gemini 2.5 giving a wrong answer with some arithmetic is interesting, especially in discussions about trying it multiple times, and how sometimes it'll use an actual calculator and get the correct answer, and how sometimes it'll say it used a calculator and still give incorrect answers.

Via.

I've just started passing the "okay, you can use external tools" flag to various LLM APIs (I'm using an abstraction layer written by a coworker), and this sense of what it tells us what it's doing vs what it's actually doing, or not doing, is only gonna get worse.