I’ve recently passed my OSWE exam and I found myself wanting more. While CVE hunting and actual appsec work are on the table, I wanted something smaller in scope that I can use to keep my skills sharp in the meantime. I am a huge fan of the DayZero podcast and of their regular challenges so I wanted to create something similar.
As a result, I have built a daily appsec puzzle generator available here. (Still in beta, there are some issues that need fixing)
If you are wondering how I did it, keep reading.
I started with chatgpt. Chatgpt is amazingly powerful and incredibly affordable. About two days of experiments, close to 200 requests, and I incurred a total cost of just 10 cents while also learning a lot about prompt engineering and parsing its output.
The thought process was that I could ask chatgpt to create a vulnerable code snippet, based on a specific vulnerability. For example, I could ask ChatGPT to generate a code snippet that receives user input without proper input validation or sanitization and it would generate an example like:
$username = $_POST['username'];
$sql = "SELECT * FROM users WHERE username = '$username'";
$result = mysqli_query($conn, $sql);
if ($result) {
while($row = mysqli_fetch_assoc($result)) {
echo "Welcome " . $row["username"];
}
}Which is technically correct, but also mostly nonsensical as this kind of application would very likely not exist in a real scenario.
So the next step was to try and add it to an existing web application. After a lot of tweaking with the prompts and the responses, I got some promising results.
Unfortunately, there is a natural fuzziness that comes from chatgpt results. There’s always some entropy in the way it generates responses, even when the prompt specified a fixed format.
However, just as I had released the beta for the daily appsec puzzle using chagpt, I got access to GPT4!
While it is significantly more expensive than chatgpt, I have been able to get reliable results in the format I want:
The solutions are also automatically generated and uploaded the next day.
Overall, I’ve been very happy with the results and it’s one of the things I check every day to test myself. Everything is entirely automated, the vulnerabilities are pulled from the official CWE MITRE website and the language is picked at random, so as long as my raspberry pi doesn’t lose WiFi signal, I can get my daily appsec fix.
I genuinely believe that AI models such as chatgpt and GPT4 can accelerate learning in almost all industries. Since you can ask them to explain their througt process, they can act as individual teachers to fit your needs. You can even send the challenges to chatgpt to get help on identifying the vulnerability!
I’m excited to see how much further I can take this project (appsec labs?) or how others can build on top of existing appsec solutions to further push this industry forward.
