A method for measuring what agents say about your stock

01

The procedure is the output

Applies to marketplace onlyown storefronthybrid

This lesson publishes a procedure rather than findings. A result is assessable through the design that produced it, and a design stated after collection cannot be distinguished afterwards from one selected to fit what was found.

The source of a figure is part of the same question. Anyone selling software in this area, including the company publishing this course, has a commercial position on the answer, and the check available to a reader is a method specified in enough detail to be run independently. A figure arriving without one cannot be checked against anything.

02

What is fixed before the first query

Applies to marketplace onlyown storefronthybrid

The following are written into a dated document and not amended once results begin arriving.

1. The exact prompts or queries — the wording rather than the theme — and how many times each will run.
2. The scoring rubric, including what counts as the item appearing and what counts as a partial match.
3. The sampling plan: which days, which times, which surfaces, which locale, and which account state.
4. What counts as a negative result, in the same detail as a positive one.
5. The limitations already known at the time of writing.

The fourth entry is the one the rest depends on. Where an unfavorable outcome has no written description in advance, the design retains flexibility in the direction the results later require, and that flexibility is not visible in the finished write-up.

03

Sampling a system that moves

Applies to marketplace onlyown storefronthybrid

These systems differ between days, between accounts, and between sessions, and they are personalized in ways a seller cannot fully suppress. A single observation therefore describes one session rather than a small quantity of the behavior.

Observations are repeated across days rather than within one afternoon. Repetition inside a single session measures that session, which is the difference between the two designs.

04

Scoring blind, and reporting disagreement

Applies to marketplace onlyown storefronthybrid

Scoring happens after the collector knows what the run produced. Three steps separate the two: collection and scoring are performed as distinct passes, identifying context is stripped from the responses before scoring, and a second person scores a subset so the rate of disagreement between the two can be reported.

That disagreement rate is published rather than resolved privately. A rubric two people apply differently is a rubric one person applies differently across two weeks, and its size tells a reader how much of the reported figure is separable from the scorer.

The recommendation research cited here names factual consistency among the objectives its benchmark scores, which describes what that benchmark measures rather than a property of a deployed system.¹ The transferable part is the sequence: the thing being scored is defined first, and scoring is performed against the definition.

05

First question: whether stock appears

Applies to marketplace onlyown storefronthybrid

Pointed at the visibility question, the procedure produces a bounded statement: for this prompt set, on these dates, on these surfaces, items of this kind appeared this often. That is a description of a sample, and it is recorded as one.

Two limits are properties of the design. It observes rather than intervenes, so it does not support a causal statement, and it does not extend past the prompts and dates sampled. Both are written into the result rather than appended to it.

The academic work in this area was conducted under laboratory conditions rather than against a storefront, and it does not describe how any commercial surface currently orders what it presents.² A first-party observation with a stated method covers a different question from the one that literature asked.

06

Second question: how long a sold item stays buyable

Applies to marketplace onlyown storefronthybrid

The same procedure answers a second question: after a one-of-one item sells, how long does each surface continue to present it as purchasable? An appearance in an answer and a false availability value are different outcomes, and this measures the second.

1. Sample real sold items across the channels and surfaces in use.
2. Record the sale timestamp from the seller’s own records, to the minute.
3. Check each surface on a fixed schedule and record the first check at which the item is no longer offered.
4. Publish the distribution rather than the average alone, with the method and the sample size beside it.

The third row is left unfinished because that is what was observed. “Longer than the sampling period” is the statement the data carries, and an average would require values from outside it. The fourth row is filed under the first question rather than the second: an item that never appeared has no staleness window, and a zero there is not a staleness result.

One distinction separates the measurement from a product claim. The quantity here is the externally observable window on surfaces the seller does not control. A vendor’s internal synchronization latency is a different quantity, measured inside a system, and it is not what this procedure observes.

07

Publishing it in a re-runnable form

Applies to marketplace onlyown storefronthybrid

The pre-registration, the raw observations, the rubric, the disagreement rate, and the dates are published together. Raw data without the rubric and a headline without the data are each unusable for the same reason.

The decision to publish is made in advance and applies to both outcomes. A seller reporting that their items appeared in none of forty queries has published a result with a method attached; a favorable rate published without one is a figure that cannot be checked. Whether to publish is settled at the design stage, since settling it afterwards is settling it with the result in hand.

08

What a run of this size supports

Applies to marketplace onlyown storefronthybrid

The cited recommendation research evaluates factual consistency within a defined benchmark.¹ The cold-start literature reports laboratory conditions rather than a live commercial surface.²

Neither publishes a merchant-side visibility rate or a staleness window, which is why a seller running this procedure is producing a primary observation. What it supports is a statement about the prompts, dates, and surfaces sampled, and the sample size is part of the statement rather than a caveat on it.