Case 02 · AI tooling
From my project to the whole design team
I built two AI tools for my own project: one turns a live screen into a prototype, the other writes the documentation engineering builds from. Then I reworked both so any designer can use them, on any project.
- Adoptedthe page tracer, now used across the design team
- 80+questions pinned to the screens they’re about
- 110+clickable walkthroughs in my hand-off documents
Illustrative · NDA
The product and the framework are under NDA, so the names here are generic and the pictures are redrawn in the same fictional supermarket chain as the other cases. The tools, the decisions and the lessons are mine.
Problem
Two jobs fill a designer’s week: rebuilding screens that already exist as prototypes, and writing a hand-off that engineering can actually follow. The first comes with every product that’s already live; the second with every feature, on a brand-new product too.
I built an AI assistant for each, inside my own project. Both worked for me. The harder question came next: would they work for a designer who isn’t me, on a project that isn’t mine?
The UI Inspector had already made that trip to the rest of the team. These two took more work, because each had my project’s details built in: its paths, roles, components and colours.
The page tracer
Point it at a page of the live product, and it rebuilds that page as a prototype using our framework’s stack: the same structure, content, states and behaviour, in the product’s own visual values.
Our prototypes never depend on a product’s code: each one is a copy in plain HTML, CSS and JavaScript, and the tracer is what makes that copy.
Most new work on a product that already exists starts from a screen that’s already there. Rebuilt by hand, that screen takes hours, and a guessed colour or spacing looks just as finished as the real one. The tracer measures instead of guessing, and builds with the framework’s own components, so the result is ready to change.
Draft first
One fast pass gives a complete prototype of the page, ready to share. It’s the default, and often all a feature needs.
Refine one region
Point at the part that’s off. It measures that region against the live page, fixes it and checks again.
Complete, with proof
On request, it measures every state and compares the prototype with the product, property by property. It says what that will cost before it starts, and waits for a yes.
It renders, it works
Two checks come before any measure of fidelity: the page draws, and its controls respond. Then it looks at the screenshot itself.
It stops instead of guessing
When it can’t see the page, it stops and says why, rather than inventing what it can’t measure.
Every failure, a rule
Each run that went wrong became a written rule it reads when something looks off: 69 so far.
How I taught it to stop guessing: teaching an agent to see like a designer
The hand-off assistant
It writes the hand-off from the feature’s knowledge base, the matrix, the logs and the meeting notes, and puts it inside the prototype, where people already are.
Every feature gets two documents. What’s New is a guide for stakeholders that opens with who can do what on each screen. Dev and QA is the spec engineering and QA build from: numbered requirements, the user stories under each topic, and the business rules. Both state the current rule, never its history.
It works the same on a brand-new product: everything comes from the knowledge base and the prototype, not from a live app.
Nobody opens a document in the middle of a demo. So the documents open the prototype.
Inside the prototype
A panel beside the prototype
The documents open in a docked panel next to the live prototype, never over it, so a rule is read against the screen it describes. It remembers the document and the place.
Walkthroughs you can click
Every procedure is a walkthrough. Each step opens the prototype where it happens, as that role and in the state it needs, with a Click here marker on what to press.
Made to be read
Every topic opens with an In short line. An outline says where you are, Find highlights in place, Back undoes every jump, and a filter shows only what’s still open.
Stories where they belong
Under each topic, the user stories it fulfils, found through its requirements, so a developer reads the rule and the story together.
Copy it anywhere
Copy a document as Markdown, or as rich text for a document editor, or open it in a tab of its own.
A check before it ships
One run checks the whole set: steps that open a state the prototype can’t stage, questions pinned to screens that don’t exist, links that land nowhere, a role spelled two ways.
Flows and rules as pictures
A workflow view
One switch draws every flow as a picture: a lane per role, each hand-off on its line, a rejection dashed on its way back. Each step in the picture opens the prototype there.
Maps first
Every document opens with maps drawn from the feature’s rules: the process across roles, the status lifecycle, who can do what, a decision table, and how work moves across the prototypes.
Drill down
A role or a status opens its own page inside the map: who it receives from and hands off to, its steps flow by flow, and its rules, each linked to the passage where it’s written.
Questions on the screen
Every open point that needs an answer from a room becomes a question, written to be read out loud. One file feeds two places that can never disagree: a pin on the screen and a board for the work item.
A pin on the control
The question sits on the button or field it’s about, and several on one control share a pin, with a count. A link to a question opens the screen with that question showing.
A card that stands on its own
An id that never changes, so it can be quoted in meetings; a kind (open, confirm or attention); an owner role, never a person; the context behind Show context; and a link to the passage in the spec.
A board for every work item
One page lists every question of a ticket or an epic, a tab per related prototype, opening on what’s still unanswered. Each card opens its screen and its source in the spec, and every pin links back to the board.
Answers that land everywhere
Typed on the pin or on the board, it’s the same answer. Export answers hands them to the step that applies each one to the matrix, the logs, the prototype and both documents.
Nothing is deleted
An answered question stays on the board, read-only, until the prototype shows the answer. Then it moves to History, and its pin goes away.
From my project to the team
Page tracer
- Jun · A pipeline9 agents trace a live page into a prototype.
- Aug · One agentCut back to one agent that stops instead of guessing.
- Sep · Its lessons travelMy failure log becomes tracing rules the whole team can read.
- Oct · For the teamI hand it over as one package, and every designer on the team gets it.
Hand-off assistant
- Jul · The first specOne agent writes the developer spec from the matrix, the prototype and the meeting notes.
- Aug · Questions on the screenOpen points become questions pinned to the control they’re about, and the stakeholder guide opens with who can do what.
- Sep · Flows you can clickEvery procedure becomes a walkthrough, and the answers flow back into every document.
- Oct · Any projectMaps and the workflow view arrive, and it’s rebuilt so another project can run it with its own components, roles and colours.
What made them portable
-
01
Project values out of the code
Paths, roles, components, wording and colours moved into one profile and one theme per project. The acceptance test fits in a sentence: another project’s profile and theme, no code changes.
-
02
The rules in one place
The rules moved out of the agent and into a skill it reads only when a task needs them. The agent just runs the steps, and every project reads the same rules.
-
03
Generic pieces
The panel, the walkthroughs, the pictures and the questions became generic components, with each project’s colours coming from its theme.
-
04
Handed over as one package
I bundled the tracer, its rules and its lessons into a single package, so it could reach the rest of the team as it is, with nothing for anyone to install.
What people changed
The documents got better when other people used them.
- A manager found the questions board overwhelming. It now shows less at once.
- Answered questions used to disappear. Now they’re kept.
- Not everyone reads a whole topic: that’s where the In short line came from.
- In September I put together a ten-minute talk for the other designers, from the matrix to the questions on the screen.
AI layer
Where AI assisted
- Writing the code. I built both tools with an AI coding agent. The rules and the design are mine.
- Drafting the documents from the matrix, the prototype and the meeting notes.
- Turning failed runs into written rules, for both tools.
Where I overruled it
- Change notes. The drafts narrated every reversal; I kept the history in the design log instead.
- Warning marks. The drafts carried far more of them than there were open points; now each one marks a point that is still open.
- Guessing. The first tracer filled in what it couldn’t see; I made it stop instead.
Outcome
The hand-off assistant is ready for any project. Next, I want to see it run on a second one, with that project’s own profile and theme.
The idea, the design and the code are mine; teammates helped bring them to the team. There are no team-wide usage numbers yet.
That was the summary.