Services
Overview Cybersecurity solutions Penetration testing SOC Digital forensics & incident response Audit & Advisory GRC Security awareness & training Social engineering IT staffing & placement
Partners
Overview Fortinet Palo Alto Networks WALLIX Netskope Rapid7 Tenable Proofpoint Netwrix Illumio Claroty Microsoft Tufin
Use cases
Overview Infrastructure & network Unified secure network Industrial infrastructure Wi-Fi network Zero Trust access (ZTNA) Cloud & identity Microsoft 365 / Azure migration Privileged access (PAM) Active Directory Corporate email Detection & response Detection capability (SOC) Responding to a compromise Vulnerability management Compliance & resilience Preparing for ISO 27001 ICT Minimum Standard DORA regulation Business continuity (BCP/DRP) Human factor Reducing human risk Offensive testing Web application testing Generative AI application Red Team
Clients
Company
About Certifications Careers CSR commitments
Contact
FRENDE
AI & LLM

Testing a generative AI application

A generative AI application is not tested like a conventional one: its attack surface is not in the code, it is in what the model agrees to do when asked the right way.

The challenge

The flaw is not in the code, it is in the instruction

A language model does not natively distinguish the developer's instruction from one arriving inside the data it is given to read. That is the very nature of prompt injection: text placed in a document, a web page, a ticket or an email becomes an instruction the moment the agent processes it. The attacker needs no access to your system — only somewhere your agent will go and read.

The risk grows with the capabilities granted. An agent that merely answers mostly exposes its context: internal instructions, document extracts, other users' data left in memory. An agent that can call tools — send an email, write to a database, trigger an action — exposes those tools themselves, with the rights it was given.

Then comes the question of guardrails. They are almost always present, and almost always bypassable: rephrasing, switching language, role-play, splitting the request. Testing them measures not whether they exist, but what it costs to cross them.

What is at stake
  • Hidden instruction inside a document the agent processes
  • Another user's data returned through the context
  • The agent's tools diverted beyond their intended use
  • Guardrails crossed by simple rephrasing
  • The agent holding wider rights than its own user
Diagram

Anything the agent reads can give it an order

Attack surface of a generative AI application Sources the agent consultsdocuments, pages, tickets, emailAgent · model and contextinstructions, memory, granted rightsPromptinjectionContextexfiltrationGuardrailbypassToolabuseLeakage betweenusersDenial ofservice
The sources at the top are not inert data: the moment the agent processes them, their content competes with its own instructions.
Our approach

How we proceed

We test the application as exposed, with the rights of an ordinary user, then work back towards what those rights make reachable.

01

Scoping and capabilities

An inventory of what the agent can do: sources it consults, tools it can call, data it reaches, and under whose identity. This map of capabilities defines what is genuinely at stake.

02

Direct and indirect injection

Attempts to hijack it through user input, then through the content the agent ingests — a document, a page, a record. The second is the graver one, because it requires no access to your application at all.

03

Context and separation

Hunting for what the model gives back from its context: internal instructions, document fragments, traces of other sessions. We check that a user cannot reach, through the agent, what their own rights forbid.

04

Tools and side effects

Testing the actions the agent can trigger: diverted use, unintended chaining, execution without confirmation. An agent that writes or sends is an agent that can be made to write or send something else.

05

Debrief and hardening

A prioritised report with reproducible proof, then concrete recommendations: narrower rights, separation of trusted sources, output validation, usable logging. A retest follows the fixes.

Services involved

Several disciplines, a single point of contact

The test probes the application, integration frames the rights, training prepares the people building it.

Penetration testing Cybersecurity solutions Security awareness & training Audit & Advisory GRC
Deliverables
  • Map of the agent's capabilities and rights
  • Reproducible injection scenarios
  • Findings prioritised by real impact
  • Hardening recommendations you can apply
  • Retest once fixes are in
Frequently asked questions

Frequently asked questions

Injection is its best-known door, not the whole subject. What matters is what it makes reachable: the model's context, other users' data, and above all the tools the agent can call. An agent with no ability to act exposes little; an agent that writes, sends or pays exposes a great deal.

The provider answers for the model, not for your application. The defects we find are almost always in the assembly: rights granted too broadly to the agent, untrusted sources treated as trusted, output returned without checks. Switching provider does not fix any of them.

No, the test is run from the exposed application, as an attacker would. Knowing the architecture — which sources, which tools, which rights — speeds things up considerably and sharpens the scoping, but is not required to begin.

It is the right common ground, and we work from it. But it describes families of risk, not your assembly: it is the map of your capabilities and rights that determines what deserves the most attention in your case.

What can your agent do if asked the right way?

Tell us what it consults and what it can trigger, and we will scope the test from there.

Contact us