AI hallucinations

Language models produce confident answers whether or not the thing they are describing exists.

For most uses that is an accuracy annoyance. It becomes a security problem the moment somebody acts on it. A model invents a plausible-sounding software package; a developer installs it; somebody who noticed that models invent that particular name has registered it and filled it with something malicious. The same applies to invented settings and invented commands. The failure is not really the model making something up, it is a process that treats generated output as checked.