I led the project to align the automation workflow builder with customers’ expectations and usage of agents. I made a bet that marketers would shift from implementing changes themselves to reviewing changes from an agent. The project turned power user behavior into productized experiences for all customers. I sketched the vision, pitched the project, validated the research, scoped the roadmap, and shipped improvements.

The bet

I felt marketing automation would experience the same loop we experience with AI tools like Claude Code. I pitched a vision where marketers provide the intent, the agent orchestrates, and the marketer reviews those changes. The marketer is no longer required to implement the workflow themselves. Some of our customers were already making this transition, using an AI tool and our API to build messaging workflows.

This touched on a truth in our product where we offer unrivaled power and flexibility. If you dream up a workflow automation, you can build it. That power comes with complexity. Many customers relied on internal experts to build their automations. The data showed not everyone who views a workflow is comfortable editing it, and even fewer are comfortable starting it. This was a chance to give everyone access to an expert.

Validating and shaping

While most customers were using the agent, only a small percentage used it from the workflow builder. The percentages mirrored the share of people who felt comfortable starting an automation. There was an opportunity to make the agent more accessible to all with stronger entry points.

I analyzed the thousands of agent conversations that originated from the workflow builder to understand how the power users leveraged the agent. That research was distilled into four themes: build, edit, explain, and review. Diving deeper into the “review” theme, I identified an “oh crap” rate. This rate measured how many people started an automation, stopped it, then restarted it within a day. The overwhelming majority made edits to their trigger before restarting the automation. Starting an automation is a high-stress moment. There’s a fear of too many (or too few) people triggering your automation. The last thing you want is to learn about a configuration mistake from your customers. Decreasing the “oh crap” rate was a good indicator that customers were better able to configure automations correctly.

The project was deliberately bound to six weeks focused on three milestones: clean up the UI to account for the “co-pilot” experience with the agent, make the agent’s changes visible in the UI, and add entry points to the agent relevant to the four themes.

The agent becomes visible

Before this work, there was no visual feedback when the agent made a change to your automation. You were presented with a toast directing you to reload the page. While we had to de-scope a more robust draft/publish flow, we could still visualize a change made by the agent.

Typically, a person would drag an action onto the canvas from a panel on the right. When the agent made a change, the action didn’t have an origin. I added a small scale-in animation (0.95 → 1.00), typewriter effect, and animated thinking state to communicate the action arriving on the canvas. Trust was critical because we were now communicating who was making a change. We also made sure to update the sticky notes so if the agent added a note to the canvas, it would be clear who wrote it.

Review with the agent

We had an existing “review” panel where customers could check configuration before starting their automation. It was something that our biggest customers valued as they could double-check the configuration while viewing the workflow before committing to starting the automation. The review provided what I would call “linting” checks. It validated if it was technically possible to start the automation, not whether it was configured correctly for the use case. About 10% of the agent conversations analyzed had customers asking for more: “ok, it’s ready, do a sanity check for me.”

What was needed was a “code review” check; an experience that checked the current configuration against the use case. The vision was to integrate this into the product, but leveraging the agent was a first step we could take. I wrote a “review” skill for the agent that audited structure, technical configuration, timing, and content. Most importantly, I incorporated it into the existing review panel to meet customers where they are. Clicking review sends a short chat message to open the agent while background instructions trigger the new skill.

One design decision worth noting is that the skill calls out what needs attention and what looks correct. This came up in the conversation analysis as customers wanted a visual confirmation to show which items were checked. While it leads to a more text-heavy output, that was preferred over ambiguity of a terse response. We can now monitor skill usage to see which checks are most important to improve and incorporate into the product.

Build with the agent

Customers were asking the agent to build entire automations. This was also an opportunity to address the “oh crap” rate because customers were required to start with defining their trigger. What if customers could describe their intent and have the agent build the automation? For someone without the expertise, the agent, combined with follow-up questions, could steer them in the right direction. A better starting point could help prevent future mistakes. Of course, a power user could build the workflow as they normally would.

Explain with the agent

The most interesting finding was customers using the agent to explain the automation. Only a small percentage of individuals confidently understood how the automation works (the ones who were comfortable starting an automation). Because teams use Customer.io, it’s easy to see why an explanation would be necessary. Perhaps marketers are viewing an automation built by another person or team. Perhaps they inherited the automation. Perhaps they were the ones who originally built it, but it was built a year ago. I designed entry points from the context menu to provide a shortcut to what customers were already asking the agent.

Going forward

This is the tip of the iceberg when it comes to bringing AI agents into this area of the product. Customers’ workflows will certainly change as agents can be leveraged to surface insights, suggest experiments, and more. We’ll need to be clear about where the human remains in the loop. For now, we can use conversations as a data source to pair with usage data. We can use them to understand the intent and emotion behind their behavior. The uncertain future shouldn’t stop us from meeting customers where they are and helping them level up their skills and understanding today.