The Missing Answer Is Part of the Result

Two reports of a room can differ for an interesting reason. A cook notices something to prepare food with; a wheelchair user notices an obstructed route. They can also differ because one prompt asked for furniture and another asked for hazards. Or because one response contained no objects at all, and the analysis counted that absence as maximal disagreement.

My earlier vision-language study managed to put these possibilities into the same score. The corrected paper, released on 12 September, separates them. It makes a smaller claim, and leaves a more useful experiment behind.

What zero was doing

The historical study compared descriptions produced under seven persona prompts. One of its measures was word Jaccard similarity: the overlap between two sets of words relative to their union. Low overlap looked like evidence that changing the context changed what the model saw as usable.

But 363 of the 3,213 parsed responses contained empty object lists. Those responses affected 2,037 of the 9,244 comparisons. The implementation assigned zero overlap to every affected pair, including pairs for which there was no substantive report to compare. The methods said empty responses were excluded. The code included them.

That is a consequential disagreement between a paragraph and a function. A comparison of two different descriptions and a comparison involving an empty report both received the same numerical verdict. The average could not tell the reader which had happened.

Conditioning on nonempty reports raised pooled word Jaccard from 0.095 to 0.121. It raised sentence cosine similarity from 0.415 to 0.511. These are measurements of the historical data under a different treatment of missing content. They are not estimates of how much meaning survived.

That last distinction cost another claim. One minus a similarity score is not automatically a percentage of semantic change. A cosine is a relationship between vectors in an embedding space. Giving it a percentage sign does not turn the space into a measuring jug for meaning.

Deleting the empty rows is not the whole repair

An empty object list is not necessarily a broken answer. It might be a legitimate response that nothing meeting the request was found. It might arise from generation, instruction-following or parsing. The raw response matters, and those explanations should not be silently merged.

Removing empty reports changes the question too. We are now asking how much the answers differ among the cases where answers were available. If availability depends on the persona or the image, that selected subset need not represent the original experiment. The conditional estimate is useful, provided its condition travels with it.

The same problem reached the tensor decomposition. An earlier factor had acquired a functional label: a chef-specific pattern, interpreted as structure in the model's representation. Its concentrated loading did not survive analyses that treated the empty cells differently. A stable numerical object can be a stable consequence of the input convention. Bootstrapping it repeatedly cannot adjudicate the interpretation attached to it.

The revision withdraws that functional-manifold interpretation.

Ask the same question first

The new experiment holds the task still. Each configuration receives 48 images absent from the original pilot, four personas, a shared three-object question, two wordings and two requested seeds. Across Qwen3.5-9B and Ollama's llava:13b, that is 1,536 requests. Raw outputs are retained, and the analysis treats the image as the unit around which the comparisons belong.

The result is less convenient than a general story about context. Persona-associated variation does not uniformly exceed variation associated with wording or sampling. The configurations differ. The complete-image analyses also have fewer images than were requested, so availability remains part of the interpretation rather than an inconvenience tidied away before reporting.

This measures reports. It does not show that a model computes meaning before shape, that its internal representations divide along our preferred functional categories, or that its descriptions would guide successful action in a room. Those are further questions with further experiments attached.

What I would carry into another evaluation is the separation: whether a report exists, what it selects, and how it is worded. Each can change while the others remain stable. A single similarity average is capable of concealing all three.

The useful output of this revision is a way to keep them apart. Before explaining why two answers disagree, establish that there are two answers, that they answered the same question, and that the metric means what the sentence says it means.