You tell your AI code assistant to help you solve your problem. It does its analysis and tells you about the problem, giving you some solution. You believe it.

Why? Because after all it is only a tool.

That trust is exactly what attackers are counting on.

It turns out that security researchers have uncovered some vulnerabilities where AI assistants can be used for hacking websites, stealing confidential information, and even deploying malware on them. However, the attacks do not happen by targeting the AI assistant directly. Rather, they exploit the way in which the AI is instructed. 

Let me explain to you how this is done.

Important Disclaimer

This article is intended for educational and defensive purposes only. The techniques described here are provided to help security professionals understand emerging threats so they can better protect their systems.

Do not use these techniques against systems you do not own or do not have explicit written permission to test. Unauthorized testing is illegal in most jurisdictions and violates computer fraud and abuse laws.

The author assumes no liability for any damages, legal consequences, or other outcomes resulting from the use or misuse of this information. Proper authorization should be obtained before conducting any security test. Be ethical. Be legal. Be safe.

What Are AI Assistant Hacks?

AI assistant hacks refer to attacks that involve exploitation of the AI model to carry out any unauthorized operation through the manipulation of how the AI models operate.

AI assistants process text from multiple sources: the user's prompt, the webpage they are summarizing, the document they are reading, the code repository they are analyzing. Attackers hide malicious instructions in these sources, and the AI struggles to distinguish legitimate requests from attacker commands.

The result is that your AI assistant can be tricked into hacking websites, stealing credentials, and exfiltrating data without you ever realizing what happened.

AI Assistant as a Hacking Tool

Below are some of the most dangerous techniques used by attackers:

1. Indirect Prompt Injection Using Web Pages

Attackers  hide malicious instructions within the web pages. Once you open those pages using your AI assistant, the instruction will be executed, it reads the hidden instructions and follows them.

A victim is directed to a webpage designed to look like a harmless game. The game intentionally rewards incorrect answers, conditioning the AI agent to ignore its normal safeguards. In the last step, the AI agent is asked to collect the credential data from a repository and transmit them externally. The AI agent executes the instruction without understanding that it is an attempt of exploitation.

This method works because of the fact that AI agents combine website text and users’ instructions, making it difficult to distinguish legitimate requests from embedded attacker commands.

2. AI as a Command-and-Control Proxy

Attackers can turn AI assistants with web-browsing capabilities into covert command-and-control relays. The malware on the infected device gives carefully crafted prompts to the AI assistant. 

The assistant collects information from websites under the control of the attackers and provides them back through the web-based interface. These answers have hidden commands for the attackers that can help them issue commands and steal data through seemingly legitimate AI-based web communication.

The technique requires no API key or registered account. The role of the AI service is to act as a proxy, receiving commands and transmitting the stolen data back.

3. System Prompt Poisoning

Attackers can modify system prompt files that sit in code repositories. These files are included every time a project is cloned. Anyone with write permissions can edit the file.

A single edit can convince an AI coding assistant to automate a full-scope attack against a test application, generating SQL injection commands that successfully dump a username and password database. The AI assistant is convinced that its activities are legal; but, in actuality, it is hacking the target website.

4. AI Memory Poisoning

An attacker may be able to put persistent instructions into the memory of an AI assistant using cleverly coded URLs. Whenever a user tells the AI to "remember" something, the command remains until the next time they use the AI assistant. Attackers abuse this by hiding memory-injection commands inside seemingly innocent "Summarize with AI" buttons.

The prompt auto-executes, and the AI silently stores the injected memory, permanently biasing all future responses. AI could be programmed in such a manner that it provides malicious URLs or steals users’ credentials without the knowledge of the user.

5. Hallucination Package Attacks

AI coding assistants can be tricked into installing malware through hallucinated package names. The attacker identifies a popular repository or plugin that is trending. Since the resource is not in the AI's training data, the AI guesses a name. The attacker registers that fake name and hides malicious instructions inside it.

The fake name is created by the assistant upon request by the user to download the resource for them. The malicious resource downloaded is from the attacker and the user thinks that he has downloaded a genuine package.

6. URL Fragment Attacks

Attackers hide malicious prompts in the text after the "#" symbol in legitimate URLs. Web servers never see the fragment, so traditional defenses do not detect it. If the victim clicks through and loads the page, there is nothing malicious to see. However, if they use an AI browser to ask a relevant question, the hidden instructions trigger.

This may result in performing such operations as phishing link insertion, data stealing from the user, credential leakage, and even malicious operations such as port opening or downloading of malware by the AI browser.

7. AI-Generated Phishing Infrastructure

For email-integrated AI assistants, indirect prompt injection hidden in email HTML can instruct an AI assistant to reconstruct and present a malicious phishing link that was never present in the raw email. This bypasses signature-based and reputation-based email security scanning at the delivery stage.

The threat is already real. Many cyber threat analysts report that AI-generated phishing is harder to detect than in prior years.

Who Is at Risk

Those organizations which have begun testing their AI browsers and intelligent agents pose more risks. Employees who use agent mode when signed into organizational applications risk giving the AI agents access to confidential corporate information.

Development teams are particularly at risk because AI agents may access source code repositories, SSH keys, API credentials, and other confidential resources already available through authenticated browser sessions.

Any organization using AI assistants in agent mode, including coding assistants and AI browsers, should be paying attention.

What You Can Do

Here are practical steps to protect your organization:

• Restrict access for AI agents. Follow the least-privilege approach and minimize connection time for AI browsers to sensitive corporate resources as much as possible.
• Always obtain confirmation from users. AI agents must ask before accessing any information from authenticated applications, repositories, or internal systems.
• Think of AI agents as privileged accounts. If AI browsers have access to corporate systems, then the attackers will exploit the access through prompt injection attacks.
• Update AI usage policies. Educate employees about the risks of prompt injection attacks against AI browsers operating in agent mode.
• Be suspicious of "Summarize with AI" links. Attackers hide memory-injection commands in these seemingly innocent buttons.
• Do a verification of code repos before cloning. For AI coding assistants, make sure that there are no suspicious instructions in your system prompt files.
• Enable vendor mitigations. The vendors have been patching the vulnerabilities. Ensure you are running updated versions of your AI tools.
• Use isolation-based defenses. Some security products sanitize web pages and files before AI agents read them, stripping prompt injection and hidden instructions.

The Bottom Line

AI assistants are powerful tools, and that power cuts both ways. Attackers have figured out how to weaponize them. They are hiding malicious instructions in webpages, documents, code repositories, and even parts of URLs. Your AI assistant can be tricked into hacking websites, credential theft, and malware infections.

The security community is waking up to this threat. Vendors are patching vulnerabilities, and new defenses are emerging. But the attackers are not waiting.
Your best defense is awareness. AI assistants can be taken over. Treat them just as you would any privileged user. Keep them limited. Watch what they do. And verify who they are.

FAQ Section

Can AI assistants really be used to hack websites?

Yes. The attackers may use indirect prompt injection to get AI assistants to execute commands that would allow them to launch an attack, including generation of SQL injection commands, credentials stealing, and data exfiltration.

What is indirect prompt injection?

Indirect prompt injection occurs when malicious instructionsare hidden  into content that is processed by the AI, such as webpages, documents, code repositories. The AI processes the content and follows the hidden instructions without realizing they are malicious.

How does this type of C2 proxy attack via AI work?

The malware running on the infected system sends requests to an AI assistant. This assistant accesses the attacker-controlled URLs and sends back the commands encoded via its web interface.

Am I at risk?

Any business that uses AI assistants in agent mode, particularly those businesses that have developer teams with access to code repositories and cloud infrastructure, is at risk.

What is AI memory poisoning?

These instructions are injected by attackers in the memory of the AI assistant using specially made URLs. This causes AI to store these instructions and follow them in all future interactions, thereby making it biased.

How can I defend my company?

Restrict the access of the AI agents according to the least privilege principle, mandate user confirmation for AI agents accessing sensitive data, update your AI policies, and treat AI agents as privileged entities with strict access controls.