AI Chatbots Poisoned for Disinformation and Phishing
Threat actors are poisoning AI chatbot responses from ChatGPT, Gemini, and Google AI Overview by seeding the web with malicious links and data. This manipulation enables large-scale disinformation and phishing campaigns targeting users.
AI chatbots depend on continuously scraped web content to answer queries. Threat actors exploit this dependency by injecting malicious links and deceptive data into the indexed web. When a chatbot retrieves this poisoned material, it may present it as legitimate, unwittingly amplifying disinformation. This technique targets the trust users place in AI-generated answers, turning a convenience into a vector for phishing. The attack surface is broad, affecting major platforms like ChatGPT, Gemini, and Google AI Overview, as their responses are only as reliable as the sources they ingest.
The poisoning of AI responses could significantly erode public trust in digital information. Individuals seeking quick answers may be steered toward fraudulent sites, increasing susceptibility to phishing. Organizations relying on AI for research or customer support may face reputational damage and security breaches. The scale of impact could be vast, as these tools are used by millions daily. However, the severity depends on detection and mitigation efforts by providers, which may need to implement stricter source validation to counter this evolving threat.