Experimenting with the UserTesting API

Question

I work in AI-assisted IDEs like Cursor, and I noticed a pattern in my own commits: text adjustments, layout refinements, assistant conversation tweaks were already responses to user feedback, yet none of them got tested after I shipped them. Usability testing lived in a separate system entirely, needing its own coordination and templates to launch. So I asked: what if the design activity already happening in Git could feed directly into test scaffolding, instead of testing staying a separate track from the changes themselves?

Experiment

I prototyped a pipeline that turns commit messages into usability test prompts:

1. Commit analysis. A script parses commit messages for design-relevant changes using keyword logic (add, refactor, adjust, and similar).
2. Test prompt inference. Each qualifying commit becomes a scaffold for a test task. For example, update chatbot fallback logic becomes "Observe how users respond to the assistant when it fails to provide a direct answer."
3. Drafting via the UserTesting API (planned). The design calls for OAuth2 authentication, fetching available workspaces and templates, then creating draft test plans from the inferred prompts.
4. AI-assisted copy. Cursor and Copilot help turn raw commit text into clear, participant-facing task language.

A custom MCP test server is running to host this scaffolding, and I structured the output as a schema, covering summary, test prompts, a confidence score, and blindspot flags, so a human reviewer can approve or edit before anything goes live. You can view the schema at /commit-eval-schema.json. ¹Decision 01Every commit needed a consistent, reviewable shape before it could turn into a real test. Chose: I defined a structured output format (summary, prompts, confidence score, blindspot flags) so results could plug into a dashboard or ticket instead of arriving as free text. Traded: A rigid schema catches vague commits like 'minor tweaks' and multi-part commits that lose context, but it also means low-confidence output gets logged rather than acted on automatically.

The end-to-end flow: a commit lands, gets parsed and normalized, runs through the AI system prompt to generate a summary and test prompts, gets filtered by confidence score, then passes to the MCP server to format a payload matching the UserTesting API and post it as a draft test. A human still approves before anything reaches a participant.

Learning

I'm still in discussions with UserTesting.com's vendor team to get the API access this needs for full integration, so the draft-creation and results-fetching steps remain conceptual rather than shipped. What I validated is the earlier half of the pipeline: commit parsing, prompt inference, and the structured schema all run today against real commit history.

The bigger shift this project clarified for me: usability testing doesn't have to be a separate track from development. If test prompts can be inferred from the same commits that already describe a design change, the gap between shipping a change and validating it shrinks to nearly nothing.

Next up: finishing the UserTesting API integration for draft creation, then closing the loop by fetching completed test results and tying them back to the commit that triggered them.