3 AI Security Risks to Consider

an artificial intelligence illustration on the wall

The rush to integrate Artificial Intelligence (AI) into product lines and solutions has organizations seemingly leaping before they look. Without a doubt, the generative AI future looks bright, but as with any new technology, the security of the solutions must be taken into account to minimize risk to your organization.

Let’s consider just a few of the risks that we are seeing today and what they mean.

  1. Improper data governance.

Many of the publicly available models are trained on publicly available datasets. Therefore there is little concern for data leakage when asking Anthropic to describe a generic security event. However, as a user feeds the model data (such as a company-specific audit report) and asks for a summary, there is a potential for data exposure. This scenario becomes more concerning with retrieval augmented generation (RAG) models, such as Microsoft’s Copilot, which can leverage all data accessible within the user boundary of the Microsoft cloud. Querying this information, and asking for interpretations or insights, while potentially helpful, may also result in data leakage. For instance, Microsoft stores both prompt queries and responses within M365. This means a prompt response may contain private data and reside in secondary locations. (See more about deletion of copilot data here: https://learn.microsoft.com/en-us/purview/ediscovery-search-and-delete-copilot-data).

2. Prompt manipulation.

Most people in the IT security world have heard stories about prompt manipulations. This may include asking for something dangerous or sensitive that is properly denied, only to have the same information become available through a slight change in the question. This type of manipulation has been widely documented. The game is simply a new form of social engineering. The requester simply disarms the bots pre-defined defense by pretending to have a non-malicious intent. A more complex manipulation comes from stringing together metadata provided by responses. By using multiple seemingly distinct queries, larger datasets may be inadvertently exposed.

3. Inaccurate responses and hallucinations.

While certainly impressive at times, today’s generative AI is largely at the skill level of an intern or apprentice. Able to search articles and websites, string together a response, and often miss the mark. After numerous tests, CTInfoSec has seen that the popular LLM’s can only perform basic tasks accurately (this is based on what is allowed in prompts not seen as dangerous). It has often been the case that the information returned remains at a high level and at times, when the initial prompt is incorrect, further incorrect responses build from there. Gen AI is many times confidently incorrect, or verbosely hollow.

This field already shows incredible promise, and there are certainly ways to leverage AI safely and effectively today. However, as we have outlined here, there are pitfalls to be aware of, and as this technology evolves, both strategic testing and architecting will be important components to its security.