Like many of us, I've been playing with LLM based chat agents (such as ChatGPT) to see where they excel and where they get tripped up.