S8 to S9
Foundation
Express intent
Intent became structured, reviewable and actionable.
Six sprints / Product evolution
From chat interface
to delegated agency.
Six months across mobile and desktop — one continuous shift: from expressing intent, to understanding long-running work, to controlling an agent acting in the real world.
● ●
Pine product designSprint 8 to 13
S10 to S12
Task operating system
See, steer and repeat work
Tasks gained history, continuity and repeatability.
S13
Delegated agency
Watch and control Computer Use
The interface became a control room for an agent acting in the real world.
S08
Intent foundation
Chat input, inline forms, Need Actions, verification and three-way calling.
Input and controlS09
Task feedback
To-do, task and call summaries, voice states, artifacts and task cards.
Status and evidenceS10
Continuity
Google Calendar, Task Notebook, task sharing across web and mobile.
Memory and ecosystemS11
Ambient feedback
Full-screen voice and toast patterns for low-attention moments.
Presence and pacingS12
Standing delegation
Scheduled tasks, Replay Highlight, email review and repeatable Task Runs.
Repeatability and trustS13
Agent control room
Computer Use across desktop, mobile, pop-up and side view.
Agency and oversightWhat shipped
A cross-platform system spanning intent capture, live execution, human approval, artifacts and memory.
What I learned
AI product design is the design of uncertainty — probability legible, latency meaningful, autonomy bounded, failure recoverable.
01
Background & goal
AN AGENT THAT ACTS IN THE REAL WORLD
Pine gets things done by acting — real phone calls, bookings, forms, driving a computer. Its output is not an answer; it is an outcome in the real world.
The core tension
DELEGATION ANXIETY IS THE REAL CEILING
The more capable the agent, the less the user sees — and the more it feels like losing control. Delegation anxiety, not capability, is the real adoption ceiling.
- Role
- Product Designer — Interaction & UI
- Scope
- Sprints 8–13, mobile app and desktop web
- Focus
- Agent status, feedback and control systems
- Surfaces
- Voice · chat · computer use
6 sprintsSprints 8–13 owned end-to-end
100+screens across mobile & web
3 surfacesvoice · chat · computer use
1 systemstatus, control and evidence
What actually stops people — three anxieties behind every hand-off
01
“Did it understand me?”
Voice is fast but lossy — did it mishear a name, a date, an amount, and act on the wrong thing?
Miscommitment02
“What is it doing right now?”
Minutes of silence while an AI makes real calls in your name. The black box is where trust dies.
The black box03
“Can I still stop it?”
Money, identity, permissions — handed over with no visible brake reads as loss of control.
No exit02
Analysis · how trust forms around an agent
TRUST IS BUILT — OR LOST — ACROSS THE TASK LIFECYCLE
Each stage of a delegated task has one dominant question and one failure mode. These three questions became the backbone of every design decision that follows.
User anxiety across a delegated task
– – – without design support ——— with the trust system (touchpoints marked)
DelegateExecuteDeliver
Stage 1 · Delegate
“What exactly is it about to do?”
The blind hand-off — a task submitted into a void.
→ Task cards set the goal, plan and what Pine may need, before execution.
Stage 2 · Execute
“What is it doing right now?”
The black box — minutes of silence while an AI acts in your name.
→ Continuous visibility: thinking states, live progress, a real-time window.
Stage 3 · Deliver
“What did I actually get?”
The unverifiable claim — “done!” with nothing to check.
→ Structured proof: summaries, artifacts and next steps to verify and reuse.
03
Design goals
MAKE DELEGATION FEEL SAFE
01
Visibility
See it
The agent is observable at every moment — from a one-line status to a full live screen, at the fidelity you choose.
02
Control
Steer it
Decisions that matter return to the human. Pine pauses at the right moments, without breaking the flow.
03
Deliverables
Get it
Every task ends in a structured, verifiable result you can check, keep and act on.
04
Strategy 1 · See it
MAKE THE AGENT'S WORK VISIBLE — AT THE FIDELITY YOU CHOOSE
Visibility is a system of altitudes — a glanceable status for most moments, a full live window for the ones that matter. Two surfaces carry it: thinking states, and Computer Use.
A · Thinking states — reasoning you can glance, not a wall of logs
Collapsed by default — one calm line: “I'm working.”
Tap to expand — the live step list.
Done — thinking folds away; the answer stays the hero.
Why collapsed by default
Trust needs calm, not spectacle. Showing every token reads as noise and makes errors louder.
B · Computer Use — a window, not a log
One live session at three altitudes, progressively disclosed. Read-only by design — always watchable, always rewindable.
L1 · Glance
Banner — one line + live thumbnail. Enough to know it's running.
L2 · Monitor
Panel — live screen, step 4/5, timer, playback.
L3 · Inspect
Full screen — the whole session, scrubbable like a recording.
The ladder rule
Each level answers one question and offers one step deeper. Glance → Monitor → Inspect — no dead-ends, no forced next.
05
UI anatomy · one surface, dissected
WHY THE LIVE PANEL EARNS TRUST
Every element answers a specific fear. Nothing here is decoration.
Core screen / Computer view
01
Read-only by design
Watching is free; intervening is a separate, deliberate act — the agent can't be nudged by accident.
02
Time made visible
The elapsed thinking timer turns silence into progress. Nothing ever feels stuck.
03
Bounded progress
Step 4 / 5 — a numbered plan makes the wait finite. Users forgive slowness they can measure.
04
The past is auditable
A playback scrubber, not a log — every action replayable, accountability built in.
05
One step deeper
A single, obvious exit to full-screen inspection. The disclosure ladder never dead-ends.
06
Iteration · showing the model's mind, twice
WE SHIPPED THE WRONG TRANSPARENCY FIRST
Our first experiment streamed the model's full reasoning into chat — maximum transparency, catastrophic experience. Trust comes from legible reasoning, not visible reasoning.
V1 · Experiment
Full reasoning, always expanded — hesitations amplified, hallucinated sub-steps became visible promises.
→
Shipped
One calm summary line, expandable on demand. Transparency became a choice, not a burden.
What this taught us about LLM UX
Output length isn't under the designer's control — the UI must be. Summarise by default; disclose by intent.
The first principle · why these patterns exist at all
Voice for the gist,
UI for the rest.
Voice is the fastest way in — you speak ~4× faster than you type. But reading beats listening for anything dense, exact, or private. Pine is a multimodal loop; my job was the half voice can't carry.
🗣
Talk to decide
Discussion, options, judgment. Voice keeps the conversation flowing — the gist, not every detail read back aloud.
👆
Tap to confirm
Exact constraints — a booking time, a hard no-go window — become a scannable card. You tap through the spec instead of reciting it.
⌨
Type in private
An SSN or password must never land in the voice channel. Pine stops you mid-sentence and opens a secure typed field instead.
Fast thinking
Keeps the conversation flowing
Real-time, low-latency. Tracks pauses, hesitation, interruptions, back-channels — the live voice loop.
Slow thinking
Gets things done
Runs alongside: plans, retrieves, calls tools, manages long tasks — surfaced as the screens in this study.
07
Anatomy of one task · where my screens live in time
“LOWER MY XFINITY BILL” — TWO MINDS, ONE TIMELINE
One sentence sets off two parallel minds — fast thinking holds the conversation, slow thinking runs the work. Every screen I designed is where background work surfaces to the human.
Voice · start
“Lower my Xfinity bill”
User speaks. Tone and urgency captured instantly.
Voice · mid-task
“I'll call them now”
Voice confirms intent; conversation keeps flowing.
Delegate
Task card appears
Goal, plan and what Pine may need — set before acting.
◆ My screenExecute
Live Computer-Use view
Read-only window into the real call, step 4 / 5.
◆ My screenHuman handoff
Verification call
The provider needs a human — three-way call card.
◆ My screenDeliver
Result card
Outcome, next steps — verifiable and keepable.
◆ My screenFast thinking — live voiceSlow thinking — background work + my UI
08
Strategy 2 · Steer it
HAND THE WHEEL BACK AT THE MOMENTS THAT MATTER
Earned autonomy, not full autonomy. Five human-in-the-loop patterns, each placed where a wrong move would be irreversible: personal data, standing permissions, identity.
P1 · Structured ask
In-chat forms — Pine asks once, structured and inline, not an interrogation over ten messages.
P2 · Approval gate
Need-action cards — routines pause for explicit human sign-off before running.
P3 · Human handoff
Three-way verification — when a bank insists on a human, Pine dials you in, then takes the call back.
P4 · Send review
Outbound email review — nothing leaves in your name without your eyes on it first.
P5 · Scoped access
Google Calendar — permissions in context, scoped to the task, never a blanket grant.
The interruption policy — when does Pine stop and ask?
Interruptions are spent like a budget — one policy decides every pause.
Pine needs to act
→
Reversible · in-scope · low-stakes?
YES →
Act autonomously — and log every step
NO ↓ what is at stake?
Needs your personal data→P1 · Structured ask — an inline form, once
Moves money or makes a commitment→P2 · Approval gate — explicit sign-off
Institution insists on a human→P3 · Human handoff — three-way call
Speaks in your name (email, messages)→P4 · Send review — you see it before it leaves
Touches a connected account→P5 · Scoped access — permission in context
The pause rule
Pine interrupts only when regret would be irreversible — money, identity, standing permissions. Everything else keeps flowing.
09
Strategy 3 · Get it
END EVERY TASK WITH PROOF, NOT A PROMISE
An agent's word is worthless without receipts. Every run ends in a record the user can verify, keep, and act on.
R1 · Kickoff
Task card at start — goal, plan, what Pine may need. Expectations before execution.
R2 · Task summary
One structured record per task — what was done, what changed, what's next.
R3 · Call record
Every call ends in an auditable to-do trail — outcomes become checkable items.
One state machine
New → Started → In progress → Done → Continue. One card family, learned once, read every task at a glance.
R4 · Cross-surface — the same result system scales to desktop web
10
When trust compounds · standing delegation
FROM ONE TASK TO A STANDING RELATIONSHIP
Once the see–steer–get loop proves itself, users graduate to standing delegation — scheduled tasks, a notebook that remembers, replay highlights that keep long runs auditable. Trust compounding into a relationship.
Scheduled tasks — recurring work Pine runs without being asked twice.
Schedule it
Cadence, scope and limits set once, editable anytime.
Remember it
Every run leaves structured memory Pine builds on.
Audit it
Key moments of long runs, auto-marked for review.
Task Notebook on desktop — the standing record of everything Pine has done for you
12
Principles · what conversational AI demands from design
FOUR RULES I NOW DESIGN BY
01
Design for probability, not the happy path
LLM output is a distribution, not a result. Every surface needs a “partially right” state — not just success and failure.
02
Latency is a material
Pacing is to AI products what typography is to print. We design the wait, not just the answer.
03
Context is the real UI
What the model remembers, touches, and forgets shapes trust more than any single screen.
04
Failure is a first-class flow
Misheard calls, wrong forms, refused requests — repair paths get the same design care as the golden path.
13
Beyond features · reflection
WHAT DESIGNING FOR AN AGENT TAUGHT ME
Systems, not screens
THE QUIET WORK
Icon library, button components, chat spacing rules kept consistent across sprints — the quiet work that makes 100+ screens feel like one product.
Status is the product
FEEDBACK IS THE CORE LOOP
Users don't evaluate the model; they evaluate what they can see of it. The status system is the product.
Design the pauses
WHEN NOT TO AUTOMATE
The hardest decisions were about when NOT to automate — where the agent stops and asks.