Calculate recommendation rates without hiding uncertainty

1. Define one eligible observation

For a particular brand and question, count a completed, classifiable response as one eligible observation. A timeout belongs in the availability metric. An answer that cannot be classified belongs in an unknown bucket until reviewed. Keep direct brand questions separate from organic discovery questions; otherwise the question mix can manufacture an apparent improvement. Persist the prompt panel version with the counts, and compare cohorts only when model version, language and retrieval conditions are compatible. The example uses an interval as a descriptive warning against overconfidence, not a significance test for correlated AI answers.

2. Implement a bounded interval

A rate without its sample count is hard to assess. The Wilson calculation below returns an interval inside zero and one, including when every observed response recommends a brand or none do. It also makes an empty sample explicit. It assumes a binomial interpretation; repeated answers from related prompts or identical models are not automatically independent trials. Use the result to communicate the sample size and method, and preserve the underlying question-level records for a more appropriate analysis when dependence matters.

import math

def wilson(wins, total, z=1.96):
    if not 0 <= wins <= total:
        raise ValueError('invalid counts')
    if total == 0:
        return None
    p = wins / total
    denominator = 1 + z*z/total
    center = (p + z*z/(2*total)) / denominator
    margin = z * math.sqrt(p*(1-p)/total + z*z/(4*total*total)) / denominator
    return max(0, center-margin), min(1, center+margin)

assert wilson(0, 0) is None
low, high = wilson(3, 10)
assert 0 <= low <= .3 <= high <= 1
assert wilson(0, 10)[0] >= 0
print({'rate': .3, 'interval': [round(low, 3), round(high, 3)]})

3. Preserve paired questions across dates

Suppose a later run recommends a brand four times rather than three. First inspect which question changed. If the two runs used different questions, you cannot attribute the difference to model behavior. For a fixed panel, store transitions per question: absent to recommended, recommended to absent and unchanged. This exposes whether one noisy prompt drives the entire line. A chart should mark a provider/model change as a break in the series. Do not interpolate across a failed collection day as though an observation occurred.

4. Build tests around misleading dashboards

Test zero eligible responses, all successes, all failures and a provider outage. Verify that unavailable attempts do not enter the recommendation denominator and that question-panel changes produce separate cohorts. A dashboard should display raw counts next to a percentage and disclose the extraction rule. Do not label a one-answer movement as revenue growth. The measurements describe observed answers under specified settings; proving a change caused additional purchases requires a different experiment and actual conversion data. Keep that distinction visible in exports as well as in charts.