So, when is the right time to start prompt testing? And what are the prerequisites before you begin? Let’s dig into that a bit:

Imagine a dev team building an AI application. As part of their pre-launch checks, they bring in a penetration testing team to perform prompt testing.

The pen testers jump in and quickly realize the app is wide open to malicious prompts. They manage to jailbreak it, expose system prompts and templates, and get it to perform actions way outside its intended purpose.

They report these issues to the dev team and recommend fixes. At first glance, it looks like they did a great job — and they absolutely did! But here’s the question: was that really the best use of their time? Did you actually get the full value of their expertise? Probably not.

Prompt testing or penetration testing is supposed to be one of the final steps in your security review. By the time you reach this point, your app should already be hardened with all basic safeguards in place.

The real value of pen testers lies in their ability to find those tricky bugs — things buried deep in business logic or obscure parts of code that require real expertise to uncover. But if they’re spending their time identifying obvious flaws or missing controls, then something is broken in your secure software lifecycle.

Some might argue, “Well, we can just retest after we fix the issues.” Sure, that’s possible — but it’s expensive. Depending on how many issues were found, the fix verification might require a full retest, which takes more time and money — and could delay your launch.

Here are my recommendations to make sure GenAI apps are built securely from the start:

1. Developer training: Developers working on GenAI apps should get the right training on secure development. This should go hand-in-hand with any prompt engineering training.
2. Secure defaults: One of the easiest ways to build secure apps is to use default libraries or CDKs that already have basic guardrails baked in. For example, you could provide standard libraries with built-in safety mechanisms.
3. Security as part of the SDLC: Make security part of your software development lifecycle. That means every app should go through a formal security review — including threat modeling and a review of controls.
4. Make security a part of the software development lifecycle: create a secure development process where any application goes through a security review process which includes threat modeling and review of controls.
5. Define prompt testing prerequisites: Set clear expectations about what should be in place before prompt testing begins.

By following this process, you’ll get way more out of your testing efforts. Your pen testers can focus on uncovering complex vulnerabilities instead of spending time flagging basic issues that should’ve been caught earlier.

Disclaimer: The views expressed in this post are my own and do not reflect those of my employer.