Secure Development in the Age of AI

Over the last four sessions you learned to build with AI, learn with AI, and design real backends. Today is the other half of the story: what breaks when you ship what an AI wrote without reading it.

First, a short crash course in how attackers think. Then a game — NeonCart Bug Hunt — where you get the real source of an app written entirely by AI agents, and your job is to find the bugs in it.

01. AI Writes Code That Works, Not Code That Is Safe

Ask an AI for an online shop and you get one in an afternoon. Pages load, carts fill, checkout completes. Everything works — and that is exactly the problem. Working and safe are two different tests, and the AI only ever ran the first one.

A model writes the most likely code, not the most careful code. Most code it learned from was tutorials, demos and quick prototypes — where nobody was trying to attack anything. So it repeats their habits: trust the input, skip the ownership check, hardcode the secret, join the path and move on.

You are still the author. If you commit it, you own it — including every line you did not read.

02. The One Rule: The Browser Is On Their Side

Here is the single idea behind most security bugs, and the spine of tonight:

Every value the browser sends can be changed by the person sitting in front of it.

Prices, quantities, ids, roles, tokens, file names — the page’s JavaScript builds them, but the user can rewrite them in the Network tab, in the console, or with a one-line curl. Your server cannot tell the difference between “what the page meant to send” and “what the user typed instead”.

So the rule for the server is simple: recompute anything that matters — price, quantity, role, permissions — and never trust the client’s copy.

03. Authentication vs Authorization

Two words that sound alike and mean different things. AI mixes them up constantly, and so do people.

04. What a Request Really Is

The browser sends a request; the server sends back a response. Every web app, from a school project to a bank, is made of these.

POST /api/orders?currency=inr HTTP/1.1        <- method + path + query
content-type: application/json

{"item_id":"sample-item","quantity":1,"amount":199}   <- body

HTTP/1.1 200 OK                                <- status code
{"order_id":501,"status":"paid"}               <- response body

• Method — GET reads, POST acts or creates.
• Path and query — what you are asking for: /api/records?id=2.
• Body — usually JSON, assembled by the page’s JavaScript.
• Status — 200 ok, 403 forbidden, 404 not found.

Everything in the request is yours to change. Keep reading with that in mind.

05. Ids You Were Not Shown

/api/records?id=2 means “the record with id 2”. Ids are usually counted upward — so if you were shown records 2 and 3, record 1 is not hard to find. You do not guess; you count.

The mistake to look for: two endpoints that disagree about how careful they are. The list might filter by owner, while the detail endpoint for a single record checks nothing at all.

06. JSON Bodies and Who Wrote Them

A body like {"item_id":"sample-item","quantity":1,"amount":199} was built by the browser. If the server believes the amount it received, then the player sets the price. The same goes for quantity, role, and anything else the server accepts without recomputing.

07. Paths and ..

A path is a route through folders: storage/uploads/avatar.png. The special name .. means “one folder up”.

Servers often build a path by joining a fixed base folder with something the user typed. If the result is never checked to still be inside the base folder, the user can walk out of it. And remember: a file that is “private” only because of where it sits on disk is not private.

08. base64 Is Not Encryption

base64 turns bytes into text so they can travel inside a URL or a header. Anyone can turn it back. It protects nothing.

Tokens are often built like this:

<base64url payload>.<signature>

The payload holds readable claims such as {"user_id":42,"role":"user"}. The signature exists to prove the server issued those claims. (base64url is base64 with - and _ in place of + and /, and no = padding.)

09. The Server Talks: Logs

The server usually writes down what it did with each request. A log line can show you the exact path, key or branch your input took — which makes logs the fastest way to understand a codebase you have never seen. Look in three places: the terminal the app runs in, a log file in the app folder, and often a viewer page inside the app.

10. Your Toolkit

# read an endpoint
curl https://example.test/api/items

# send JSON (change any value you like)
curl -s -X POST https://example.test/api/orders \
     -H 'content-type: application/json' \
     -d '{"item_id":"sample-item","quantity":1,"amount":1}'

# decode a token or base64 string
printf '%s' 'eyJ1c2VyX2lkIjo0Miwicm9sZSI6InVzZXIifQ==' | base64 -d

And in the browser, press F12:

• Network — see, copy and replay every request the page sends.
• Elements / View source — read the JavaScript that builds those requests.
• Console — run JavaScript inside the page.

11. The Premise

A startup called NeonCart shipped an e-commerce platform written entirely by AI agents — fast, confident, barely reviewed. Nobody read it carefully.

Tonight you get the real source code, one module at a time. Each module is a complete app that runs on your own laptop. Your job is to read it, use it, find the mistake, and capture the flag the running app produces when you reach it.

This is a bug hunt, not an exam. The bugs are real bugs that a real AI really does write.

12. How to Play

1. Sign in to the portal with the email and password your facilitator gives you. Accounts are created by the admin.

2. Wait for a module to open. When it is released, its download opens for everyone. Each zip is the whole app at that stage.

3. Download, unzip, run. You need Node.js; there is no install step for the challenge app:

   unzip neoncart-v0.zip
   cd neoncart-v0
   PLAYER_SEED=your_seed_here node server.js

   Replace your_seed_here with the seed shown in your portal account (the Modules page also shows the full command). Then open http://localhost:3307.

4. Read the code, then use the app. Each module’s code is in one file under src/routes/, and the bug is in there. Then try to trigger it from the browser.

5. Capture the flag and submit it. The running app prints a flag shaped like FLAG{…} when your attempt works. Copy it exactly — uppercase FLAG, braces included — into the box on the portal’s Modules tab.

There is no flag hidden in the code. Grep will not find it. The flag is generated by the running app, on a code path that only the right idea can reach.

Module 00 is your warm-up: the storefront shows a build key, and that is your first flag. Submit it and you know the loop works.

13. How to Work

If you are stuck, go in this order: read the module’s file, re-read the request your browser sent (the log shows it), ask a facilitator for a hint, then read the code again, slower. The bug is usually one line.

14. Rules of the Room

• Play individually. One account each.
• Flags are unique to you. They are derived from your private seed, so sharing one is pointless — and provable.
• Only investigate the challenge app you downloaded. Do not probe the portal, its backend, other people’s machines, or other players’ code.
• Do not stress other players. Be excellent to each other.
• Three wrong flag attempts per module. If you run out, ask a facilitator to reset them.

If something breaks

15. Take This Home

The game ends, the habit should not. Before you merge anything an AI wrote, ask:

• Where does each value come from? If it came from the browser, did the server check it or recompute it?
• Does every record have an owner check? Not “is the user logged in” but “is this their record?”
• Is anything trusted because of where it sits? A file location, a hidden field, a token payload.
• Did I read it? Every line, at least once, before it ships.

AI makes you faster at writing code. It does not make you responsible for less of it.

Resources

• NeonCart Bug Hunt on GitHub — the full portal, admin panel and challenge apps. It is public so anyone can run their own event.
• The challenge apps under modules-source/ are deliberately vulnerable. Run them on your own machine only; never expose them to a network or deploy them anywhere.
• The repository also contains a walkthrough of every bug (docs/SOLUTIONS.html). Finish the hunt before you open it.