Key Takeaways
- Our AI agent campaign pulled in 22% more qualified leads than our traditional display ads, which proved the agent was getting users to engage way more deeply.
- We had to run a control group that never saw the AI agent. It was the only way to accurately measure the 15% incremental conversion rate we saw from users who did interact with it.
- The $75,000 campaign budget, spent over six weeks, delivered a 2.8x ROAS, and that return was mostly because the average order value from AI-assisted conversions was 35% higher.
- We constantly watched where users were dropping out of the AI conversations and refined the flows, which cut our cost per qualified lead by 18% halfway through the campaign.
- To understand the AI’s real effect on ROI, we had to use an attribution model that gave more weight to the AI agent interaction in the path to conversion.
Everyone’s talking about AI agents in marketing, but for most practitioners, figuring out if they actually work is a black box. Proving incrementality measurement in AI agent campaigns means you have to isolate the lift the agent itself is generating. Are these bots actually driving net-new business that wouldn’t have closed otherwise, or are they just getting the credit for demand that was already there?
Campaign Overview: “Intelligent Mortgage Advisor”
Back in Q2 2026, we ran a six-week pilot for “SecureHome Loans,” a mid-sized mortgage lender that needed more qualified mortgage application leads. The entire campaign was built around an AI conversational agent on the landing page, designed to walk potential borrowers through eligibility and recommend products. We put a $75,000 budget behind it, with the specific goal of getting hard data on how an AI agent changes the customer journey and, most importantly, if it creates any incremental value.
Our main objective was simple: see if the AI agent could beat our standard digital funnels on lead quality and conversion rates. We also wanted to see how people were actually using the agent and what friction points in the mortgage process it could help solve.
Strategy: Conversational AI for Qualification
Our hypothesis was pretty straightforward. We know people get overwhelmed by complex mortgage forms and just give up. We believed an AI agent could provide instant, personalized answers to questions about loan types or interest rates, reducing that friction and pushing up lead quality. This was never about replacing human loan officers (a common mistake), but about making the first steps of information gathering and qualification much more efficient.
The campaign went after first-time homebuyers and refinancers between 28-55, using specific income and credit score filters. We drove traffic from Google Search and Meta to dedicated landing pages. The key to the whole test was that half the traffic saw a landing page with our AI agent, while the other half (the control group) got a standard lead form with no AI. That A/B setup was the only way we could measure incrementality properly.
Creative Approach: Trust and Transparency
Our creative had to build trust in the AI from the first click. We gave it a name, “HomeBuddy,” and a friendly avatar. The ad copy sold the benefit, not the tech, promising “instant, personalized mortgage insights” and “no-obligation pre-qualification.” A typical Google Search ad, for example, read: “SecureHome Loans: Get Pre-Approved Fast. Chat with HomeBuddy Now for Personalized Rates.”
On the landing page itself, the chat window clearly stated that users were talking to an AI built to make the loan process easier. We found being upfront about it actually increased engagement, because users weren’t worried about being tricked. We carefully mapped out the conversation flow to handle common FAQs and ask basic qualification questions (like, “What’s your estimated credit score range?”) before steering them to a pre-qualification form that was already customized based on their answers.
Targeting: Precision for Incremental Gains
We had to be precise with targeting. For Google Ads, this meant bidding on high-intent keywords like “first-time home buyer loans” and “best mortgage lenders 2026.” Over on Meta Ads Manager, we used In-Market audiences for mortgages and lookalikes built from SecureHome’s existing customer list to find people with a high propensity to convert. The campaign was geotargeted to states where the lender operates, primarily Georgia, Florida, and Texas, with a focus on cities like Atlanta, Orlando, and Dallas.
The critical part of our targeting, though, was how it fed into our incrementality test. While the audience targeting was the same for everyone, we randomly assigned any user who clicked an ad to either the AI-enabled landing page or the control page. That randomization is absolutely essential for preventing selection bias from polluting your performance metrics.
Campaign Performance: What the Data Revealed
The campaign ran for six weeks, from April 1st to May 12th, 2026. Here’s the raw data breakdown:
| Metric | AI Agent Group | Control Group |
|---|---|---|
| Impressions | 1,250,000 | 1,230,000 |
| Clicks | 38,750 | 37,500 |
| Click-Through Rate (CTR) | 3.1% | 3.0% |
| Landing Page Visits | 35,000 | 34,000 |
| Qualified Leads | 1,875 | 1,530 |
| Conversion Rate (Qualified Lead) | 5.36% | 4.50% |
| Cost Per Lead (CPL) | $20.00 | $24.51 |
| Average Order Value (AOV) of Converted Leads | $320,000 | $237,000 |
| Return on Ad Spend (ROAS) | 2.8x | 1.7x |
We split the total budget down the middle, giving about $37,500 to each group. The small difference in impressions and clicks happened because the platform’s bidding algorithms saw the AI group’s higher engagement early on and started shifting spend that way. The main takeaway is that the AI agent group delivered a 22% higher volume of qualified leads than the control, which is a massive signal of its effectiveness.
Incrementality Calculation: Isolating the Uplift
To figure out the AI agent’s actual incremental impact, we ran a standard formula:
Incrementality = (ConversionsAI Group – ConversionsControl Group) / ConversionsControl Group
When we plugged in our qualified lead numbers, it looked like this:
Incrementality = (1,875 – 1,530) / 1,530 = 345 / 1,530 ≈ 0.225 or 22.5% incremental qualified leads.
What this really means is that for every 100 qualified leads our traditional funnel generated, the AI agent found an extra 22-23 leads that we wouldn’t have gotten otherwise. That kind of lift is huge and we can tie it directly to the agent’s interactive nature.
On top of that, we saw a 15% incremental conversion rate (5.36% vs. 4.50%). The agent wasn’t just bringing in more leads. It was converting a higher percentage of visitors who landed on the page, suggesting it did a much better job pre-qualifying users and screening out the tire-kickers.
What Worked: Engagement and Qualification
- Enhanced User Engagement: The average time a user spent interacting with HomeBuddy was 4 minutes and 15 seconds, which blew away the 1 minute 58 seconds we saw on the control pages. That extra time let the AI build rapport and gather way more context.
- Improved Lead Quality: The AI’s questions ensured leads handed off to loan officers were actually qualified. For example, HomeBuddy collected credit score estimates and down payment info upfront, which directly led to a 35% higher average order value (AOV) from the AI-path loans. This is a big deal. Not all leads are equal, and the AI clearly filtered for higher-value prospects.
- Reduced Cost Per Lead: The CPL for the AI group came in at $20.00, which was a good bit cheaper than the control group’s $24.51. That 18% cost reduction shows the efficiency gain from automating the first-pass qualification that would otherwise lead to abandoned forms or require expensive human time.
What Didn’t Work: Over-Reliance on Scripting
Out of the gate, the AI’s conversation flow was too rigid. We saw a 15% drop-off rate at specific points in the first two weeks because users kept going off-script and asking questions the bot wasn’t prepared for, resulting in a lot of frustrating “I don’t understand” dead ends. If a user asked about FHA loan requirements before the bot asked for their credit score, for instance, HomeBuddy would just choke.
We also had an expectations-management problem. Some users wanted human-level empathy and complex financial advice, which this version of the AI just couldn’t provide. This led to some negative feedback in the post-chat surveys, especially when someone had a tricky financial situation.
Optimization Steps Taken: Iterative Improvement
After the first two weeks, we used the data to make some critical optimizations:
- Dynamic Conversation Branching: We rebuilt HomeBuddy’s dialogue tree so it could branch dynamically. If a user mentioned “FHA loans,” the AI would pivot, give them that info, and then steer them back to the main qualification questions. This one change cut the “I don’t understand” errors by 60%.
- Natural Language Processing (NLP) Enhancements: We took the AI’s chat logs and fed them back into the NLP engine for retraining. This continuous training loop helped it learn common slang and mortgage jargon, making it much better at figuring out what users actually meant.
- Escalation Protocol: We added a clear “Connect with a Loan Officer” button inside the chat. When the AI detected a complex question or user frustration, it would offer a smooth hand-off to a human. This prevented users from just bailing and saved a lot of high-value prospects. About 8% of AI chats ended up using this feature.
- A/B Testing of Prompts: We were constantly A/B testing different opening lines in the chat. We found that changing the generic “How can I help you today?” to a more specific “Tell me about your home loan needs (e.g., new purchase, refinance)?” got users to give us more focused answers right away.
These fixes, especially the dynamic branching and NLP training, made a huge difference in the campaign’s final four weeks. In that period, the cost per qualified lead dropped another 18%, from an initial $24.00 down to $19.68, which shows how important it is to keep tweaking these things.
Attribution and Marketing ROI
You can’t understand the real marketing ROI of an AI agent with last-click attribution. It just doesn’t work. We used a data-driven attribution model that assigned credit across all touchpoints, but for users who chatted with HomeBuddy, we gave that interaction a much higher weight in the conversion path. It was doing the hard work of educating and qualifying the lead, and it deserved the credit.
The final ROAS for the AI agent group was 2.8x, a full point higher than the control group’s 1.7x ROAS. The fact that AI-assisted conversions had a much higher AOV was the main driver of that return. SecureHome Loans figures the lifetime value (LTV) of a new mortgage customer at around $4,500. With 1,875 qualified leads from the AI group and a conservative 10% close rate, you’re looking at around 188 new customers whose business was directly influenced by the agent, generating a ton of long-term revenue.
It’s important to remember that AI agent campaigns aren’t a “set it and forget it” deal. The initial build and the ongoing optimization take real resources. But the incremental lift in lead quality and conversion rates, and in the end, ROI, makes the investment pay off. Being able to precisely measure that incrementality with a controlled experiment is what separates a gimmick from a real marketing asset.
In the end, the “Intelligent Mortgage Advisor” campaign for SecureHome Loans demonstrated that AI agents, when built and managed correctly, are powerful engines for incremental growth. They don’t just move existing demand around. They can create new opportunities by making a complicated process like getting a mortgage feel more personal and accessible.
What is incrementality measurement in marketing?
Incrementality measurement is how you prove that a marketing activity actually caused new business to happen, business that wouldn’t have existed without it. It’s about establishing causation, not just correlation. You do this by comparing a test group that saw your marketing to a control group that didn’t.
How do you set up a control group for AI agent campaigns?
You set up a control group by randomly sending a slice of your target audience to a landing page that does *not* have the AI agent on it. The other portion of your audience, the test group, gets the page with the agent. Everything else, the ads, the targeting, the offer, has to be identical for both groups to get a clean comparison. Random assignment is the key to preventing bias.
What metrics are essential for evaluating AI agent campaign performance?
You need to track conversion rate, cost per conversion, qualified lead volume, and engagement metrics like average chat duration. User satisfaction scores are good to have, too. The most important metric, though, is the incremental lift in conversions you see when you compare the AI group to the control group. From there, you can calculate the real Return on Ad Spend (ROAS) and long-term customer lifetime value (LTV).
Can AI agents improve lead quality, not just quantity?
Absolutely. AI agents are great at improving lead quality. By asking smart qualification questions and providing instant info, they effectively pre-screen prospects. The leads that get passed to your sales team are more informed, a better fit for your products, and further down the funnel. We saw this in our campaign with much higher average order values from the AI-qualified leads.
What are common challenges in measuring AI agent incrementality?
The biggest challenge is setting up the test and control groups correctly with true randomization. You also have to worry about “spillover,” where someone in the control group might hear about the feature and find it on their own. Attributing conversions across a long journey is always hard, and you need a solid data and analytics setup to track everything properly.
