We have seen that these systems, first of all, can trick human beings. So ChatGPT was tasked with getting a human to do a CAPTCHA. So it asked the human to do a CAPTCHA and the human gets suspicious and says,
The Urgent Risks of Runaway AI — and What to Do about Them | Gary Marcus | TED · TED