What KPIs should a dealership track for its AI agent?

Median first-response time (split day vs. after-hours), engagement rate, qualification completion, appointments set per 100 leads, show rate, sales from AI-set appointments, and cost per appointment and per sale versus your pre-AI baseline.

How long should a dealership test an AI before judging it?

60 to 90 days — the first weeks include configuration fixes, and the appointment-to-sale lag consumes close to a month. Keep ad spend and staffing steady during the test so the comparison means something.

What are vanity metrics in dealership AI reporting?

Total messages sent, conversations handled, open rates and self-scored accuracy. They measure activity, not outcomes. If a report leads with volume and buries appointments, shows and sold, be skeptical.

How to Measure Whether Your Dealership AI Is Actually Working (KPIs + Framework)

Every AI vendor sends a monthly report, and every report looks great: thousands of messages handled, hundreds of "engagements," impressive response times. None of that tells you whether the thing is paying for itself. Activity is what the software does; outcomes are what you buy it for. Here is a measurement framework a dealer can run without a data analyst — and the traps that make most AI evaluations meaningless.

Step 1: Capture your baseline first

The most common mistake is turning the AI on and then wondering what changed. Before launch (or, failing that, from CRM history), record at least one typical month of:

Leads received, by source and by hour of day
Median time to first real response (autoresponders don't count)
Appointments set, appointments shown, units sold from internet leads
What you spend on lead handling: BDC salaries or fees, overtime, tools

Without this snapshot, every later number floats in space. With it, the evaluation becomes subtraction.

Step 2: The funnel KPIs that matter

Measure the AI on the same funnel you'd use to judge a human BDC:

Median first-response time — overall, and specifically for nights and weekends. This is the AI's core mechanical job; it should be seconds, at every hour.
Engagement rate — of leads that received a response, how many replied at least once? A fast answer nobody responds to suggests tone or relevance problems.
Qualification completion — what share of engaged leads answered the key questions (document, income, down payment)? This measures conversation quality, not just speed.
Appointments set per 100 leads — the headline number, tracked separately for business hours and after-hours. After-hours appointments are usually revenue that didn't exist before.
Show rate — booked appointments that walked in. A high set rate with a collapsing show rate means the AI books soft appointments; check whether it qualifies before offering times and runs reminder cadences.
Sold from AI-set appointments — the end of the chain. Attribute honestly: the AI set the appointment; your floor closed the deal. Both things are true, and both need to work.
Cost per appointment and per sale — total AI cost divided by outcomes, side by side with the same math for your previous process.

Step 3: Run a fair test

Give it 60–90 days. The first two weeks include configuration wobbles; the appointment-to-sale lag alone eats a month.
Don't change everything at once. If you launch AI, double your ad budget and hire a closer in the same month, no one can say what worked. Hold other variables as steady as you can.
Mind seasonality. Compare against the same months last year, or at least acknowledge tax season and winter aren't interchangeable.
Read transcripts weekly. Ten random conversations, every week. Numbers tell you whether it's working; transcripts tell you why or why not — and they'll surface fixable configuration issues (wrong hours, missing financing program, awkward phrasing) faster than any dashboard.
Tell your floor about the test. If salespeople don't log outcomes from AI-set appointments in the CRM, your sold column will undercount and the whole evaluation tilts against the tool. Measurement is a store-wide habit, not a vendor feature.

The vanity metrics to ignore

Some numbers exist to make reports look good: total messages sent, "conversations handled," AI response accuracy self-scores, open rates. A store doesn't deposit messages. If a vendor's report leads with volume and buries appointments and shows, that ordering is information.

Red flags in the measurement itself

You can only see the vendor's dashboard. Insist on raw exports — every conversation, every appointment, timestamps included — so you can check their math against your CRM and door log.
Attribution greed. Some tools claim credit for any sale to anyone who ever touched the bot. Define attribution up front: an AI-set appointment that showed and sold counts; a customer who ignored the bot and walked in doesn't.
No after-hours split. If the reporting can't separate 2 PM performance from 2 AM performance, you can't see the AI's most distinctive contribution.

A simple monthly scorecard

One page, six lines: leads in, median response time (day/night), appointments set, show rate, sold, cost per sale — this month, last month, and pre-AI baseline. If after ninety days the AI-era columns don't beat the baseline on appointments and cost per sale, either the configuration needs work or the product does. Both are fixable, but only if you're measuring outcomes instead of admiring activity. The dealers who get burned by AI aren't the ones who bought the wrong tool — they're the ones who never checked.

Want to see this working on your own inventory? UCallNow builds AI sales agents, BDC teams, Facebook Marketplace auto-posting and dealer websites for dealerships across the United States — in English and Spanish. Try SOPHIA live or see every solution and price.