← All notes

RESEARCH & FIELD NOTES/

When Should an LLM Ask Before Acting? From Protocol Experiments to Property-Preserving Real-Estate Image Editing

We compare One-shot, pre-task clarification (PPN), and iterative disclosure and repair (IDR): PPN sits between the other two in cost, but its quality advantage did not replicate across models.

I did not start this research with an abstract benchmark. I started with a very concrete problem.

Part of my work involves real estate. Property photos are not always well lit, shot from a good angle, or taken after the space has been fully cleaned. To get a listing online quickly, I sometimes ask ChatGPT to help edit the images.

But if I simply say, “Make this photo look better for an Airbnb or Zillow listing,” the model may improvise. It can add furniture, plants, and decorations, or turn an empty room into a staged interior that has never existed.

That is not what I want. The actual request may be much more specific: preserve the empty-room state, remove clutter, improve exposure and white balance, correct the perspective, and make the image easier to present without changing the facts of the property.

The problem becomes harder when the same room has left, center, and right views. If the three images are edited independently, doors, windows, fixed fixtures, furniture, and spatial relationships may no longer agree.

So the question is not simply “How do we write a better prompt?” It is:

When a user’s request is incomplete, when should an LLM ask a few questions before acting, and when should it just start?

The short answer

In the current experiment, using 12 frozen tasks and two generator models, the most stable result was not that PPN produced the best quality. It was the cost and latency ordering:

One-shot was the cheapest, PPN was in the middle, and IDR was the most expensive. PPN’s quality advantage did not replicate consistently across models.

The current evidence does not support forcing every request through PPN. A more sensible product direction is an adaptive clarification layer: execute simple, well-specified tasks directly, and ask a small number of high-value questions when ambiguity, constraints, or external risk make them worthwhile.

This was a limited Stage A controlled experiment, not a universal claim about every model or task. The real-estate image scenario is the motivation and product direction for the research; it has not yet been directly validated by this text-only experiment.

Three ways to collaborate

To study when information should be disclosed, I represented the same task in three protocols. The protocols used the same final facts, hidden constraints, and acceptance criteria. The only thing that changed was when information became available.

One-shot: provide everything and act

The user provides the complete specification at the beginning, and the model executes the task directly. This is closest to an ordinary, explicit instruction.

PPN: ask before acting

PPN stands for Pre-task Prompt Negotiation. The model first identifies missing information that could affect the result and asks a set of pre-execution questions. After receiving the answers, it executes the task against the confirmed specification.

The goal is not to make the model chat more. It is to make the task contract explicit before work begins.

IDR: act first, then repair through dialogue

IDR stands for Iterative Disclosure and Repair. The model creates an initial result from incomplete information. The user then adds requirements step by step, and the model revises the result.

This resembles everyday conversation more closely, but it can repeatedly resend context, regenerate outputs, and spend tokens repairing problems that might have been prevented earlier.

How the experiment worked

Stage A used 12 frozen synthetic tasks covering:

  • ordinary and technical writing;
  • Python programming and bug fixing;
  • data analysis;
  • JSON extraction;
  • publishing plans;
  • decision analysis;
  • customer communication.

Each task and protocol was run three times with two generator models:

  • deepseek-v4-flash;
  • kimi-k2.6.

That made 12 tasks × 3 protocols × 3 repetitions, or 108 protocol runs per model. An independent qwen3.7-plus blind judge scored the outputs with a shared rubric. I also added deterministic acceptance checks for JSON structure, required fields, dates, arithmetic, and parts of the generated code.

“Independent” here means that Qwen did not generate the answers. It was still a model judge, not a human gold standard. A stronger study would need human blind ratings and a real-user experiment.

The most stable result: IDR is expensive, PPN sits in the middle

DeepSeek batch

  • One-shot: 1,632 tokens on average, 13.41 seconds;
  • PPN: 2,530 tokens, 19.55 seconds;
  • IDR: 10,349 tokens, 47.30 seconds.

Kimi batch

  • One-shot: 2,154 tokens on average, 9.81 seconds;
  • PPN: 2,863 tokens, 12.40 seconds;
  • IDR: 11,661 tokens, 34.76 seconds.

Across both models, PPN used roughly 75% fewer total tokens than IDR. It also reduced latency and repair overhead, although it still cost more than One-shot because the questions had to be asked and answered first.

This may be the most reliable conclusion from the current experiment:

If a task really needs clarification, asking focused questions before execution is usually cheaper than generating a result first and repairing it through multiple rounds. If the task is already clear, pre-task clarification is just extra overhead.

Because I did not have a verifiable per-request dollar rate, I did not force token usage into a dollar-cost estimate.

The quality results do not show that PPN is generally better

DeepSeek: One-shot was highest

The independent judge’s average quality scores were:

  • One-shot: 88.50;
  • IDR: 87.81;
  • PPN: 84.58.

PPN scored 3.92 points below One-shot on average, but the bootstrap interval crossed zero, so the difference was not reliable statistical evidence. Composite success rates were 72.2% for PPN, 75.0% for One-shot, and 83.3% for IDR. IDR had one protocol-availability failure; the failed record was retained and included according to the pre-specified rule instead of being silently removed.

Kimi: PPN was descriptively higher, but still not significant

The Kimi batch produced these average quality scores:

  • PPN: 73.31;
  • IDR: 68.36;
  • One-shot: 65.83.

This time the direction was reversed: PPN received the highest descriptive score. But the paired comparison between PPN and One-shot gave p=0.422, and the comparison with IDR gave p=0.570. Neither supports the claim that PPN has been proven better.

The cross-generator model × protocol interaction also did not provide reliable evidence. The models showed different descriptive patterns, but the current sample cannot establish that a particular model is especially suited to PPN.

Does PPN work better as tasks become more complex?

The current data does not justify that conclusion either. On the lower-complexity tasks, One-shot was often strongest. On some medium-complexity tasks, IDR received higher quality scores, but at a large token and latency cost. PPN did not consistently lead on highly ambiguous tasks.

The value of clarification probably depends on several factors:

  • whether the missing information would actually change the result;
  • whether the constraints are easy to misunderstand;
  • how expensive a failed first attempt would be to repair;
  • whether the user is willing to answer questions;
  • whether the task can trigger an external side effect.

The deterministic checks did not change this conclusion. They can validate JSON, fields, dates, and arithmetic, but they cannot decide whether open-ended writing sounds natural or prove that a room’s structure is preserved in an image. Their correlation with the Qwen quality scores was approximately 0.466, so they are useful supporting evidence, not a replacement for human evaluation.

What product direction does this support?

The results do not support a mandatory “PPN for every request” workflow. They support an adaptive clarification layer that decides whether asking is worth the cost.

The policy could be understood in three broad cases:

  • Simple, explicit, low risk: execute directly instead of asking questions for their own sake;
  • A consequential ambiguity: ask only the high-value question that can change the result;
  • Many constraints, multiple collaborators, or publication risk: confirm the critical parameters before execution and leave an auditable record.

The important product artifact is not a longer prompt. It is a structured Edit Contract or task contract that defines the goal, allowed changes, forbidden changes, and acceptance criteria.

A real-estate editor should preserve property facts

For real-estate images, I would rather build a property-preserving listing editor than a generic tool that simply makes pictures “look better.”

The default mode should be conservative:

  • remove clutter and obvious distractions;
  • improve exposure, white balance, shadows, and noise;
  • correct lens distortion, perspective, and vertical lines;
  • preserve the empty or furnished state of the room;
  • do not automatically add furniture, plants, windows, doors, or change the layout.

The system may need to ask only a few questions:

1. Should the property remain empty, or is virtual staging allowed? 2. Which objects may be removed or added? 3. Does “make the view level” mean correcting perspective and vertical lines, or reconstructing a higher or lower camera angle?

The third question matters. Perspective correction is a relatively low-risk image transformation. Changing the camera height or inventing a new viewpoint can change the apparent facts of the property. Those operations should not be hidden behind the same phrase, “make it level.”

Multi-view listing images need another layer of constraints

Three images of the same room, taken from the left, center, and right, should not be treated as three unrelated images.

A better workflow would be:

1. identify that the images belong to the same room; 2. select one image as an anchor view; 3. build a room-level scene manifest for fixed fixtures, doors, windows, walls, floors, and existing objects; 4. apply the same allowed and forbidden changes across the three views; 5. run a cross-image consistency check after editing; 6. output before-and-after comparisons and a Change Log.

Exposure, white balance, denoising, cropping, and perspective correction should use deterministic image tools whenever possible rather than a free-form generative model. When generative editing is necessary, it should be paired with masks and an explicit list of things that must not change.

That is why “ask one question before generating” is not enough. Multi-image editing also needs scene-level state management and result validation.

Related work is already moving in this direction

This research is not the first attempt to make LLMs ask clarifying questions. Existing work has studied selective clarification, question generation, interactive intent acquisition, and help-seeking in agents. For example:

I therefore do not present PPN as a completely new clarification paradigm. More precisely, this study treats information timing as a reproducible protocol variable. Under matched tasks, facts, constraints, and acceptance criteria, it compares direct execution, batched pre-task clarification, and iterative repair while measuring quality, success rate, tokens, latency, repair burden, and protocol availability.

What has this study not shown?

The experiment has clear boundaries:

  • only 12 synthetic tasks;
  • only two generator models;
  • quality was judged mainly by one model-based evaluator;
  • no human blind rating yet;
  • no real-user comparison between ordinary chat and an adaptive clarification layer;
  • no completed real-estate image benchmark;
  • text-protocol results cannot prove that a vision model preserves consistency across multiple views of a room.

The next stage needs two parallel tracks. One should compare ordinary chat with an adaptive clarification layer using real users, measuring completion rate, quality, total time, user input, clarification burden, repair, and abandonment. The other should build a listing-image benchmark for empty-room preservation, property-fact fidelity, cross-view consistency, and listing readiness.

The conclusion

When a task is already clear, One-shot is often the more sensible default. When a consequential ambiguity exists, asking before execution may be cheaper than repairing a failed attempt. PPN may be a useful middle option, but the current evidence does not show that it is uniformly better than direct execution or iterative repair.

For products, the more useful question is not “How do we make the model always ask first?” It is:

How do we make the system decide when asking is worth it, which question has the highest value, and when it should stop asking and start acting?

For real-estate images, that decision needs one additional boundary: the image can become cleaner and easier to present, but “looking better” must not turn into inventing a property that does not exist.

RESEARCH & FIELD NOTES/

LLM 什么时候应该先问再做?从协议实验到房产图片编辑

我们比较 One-shot、执行前澄清(PPN)和迭代式对话修复(IDR):PPN 的成本位于两者之间,但质量优势没有跨模型复现。

我做这个研究,并不是从一个抽象的 benchmark 开始的,而是从一个很具体的使用痛点开始的。

我的工作和房地产有关。房子的照片有时光线不好、角度不理想,现场也不一定已经收拾干净。为了尽快把房源上线,我会让 ChatGPT 帮忙处理图片。

但如果只说一句“把这张图变好看,做成适合 Airbnb 或 Zillow listing 的样子”,模型很容易自行发挥:添加家具、植物和装饰,甚至把空房改成了重新装修过的样板间。

这不是我想要的结果。我的真实要求可能是:保留空房状态,只清理杂物、改善曝光和白平衡、校正透视,让画面更适合展示,但不要改变房产事实。

如果同一个客厅还有左、中、右三个角度,问题会更复杂。三张图片分别处理以后,门窗、固定设施、家具和空间关系可能互相对不上。

所以,问题不只是“怎样写出一个更好的 prompt”,而是:

当用户的请求还不完整时,LLM 应该什么时候先问几个问题,什么时候直接开始做?

先说结论

在当前的 12 个冻结任务、两个生成模型的实验中,最稳定的结果不是“PPN 质量最好”,而是成本和延迟的排序:

One-shot 最便宜,PPN 居中,IDR 最昂贵;PPN 的质量优势没有跨模型稳定复现。

因此,当前证据不支持让所有请求都强制经过 PPN。更合理的产品方向是一个自适应澄清层:简单而明确的任务直接执行;歧义高、约束多或有外部风险的任务,先问少量真正有价值的问题。

这是一项受限的 Stage A 控制实验,不是对所有模型、所有任务的普遍结论。房地产图片场景目前是研究动机和产品方向,还没有被这轮文本实验直接验证。

我比较了三种协作方式

为了研究“信息什么时候告诉模型”这件事,我把同一项任务设计成三种协议。三种协议使用相同的最终事实、隐藏约束和验收标准,只改变信息披露的时间。

One-shot:信息一次给全,直接执行

用户在任务开始时一次性提供完整规格,模型直接完成任务。这是最接近普通明确指令的方式。

PPN:先问,再做

PPN 是 Pre-task Prompt Negotiation 的缩写。模型先识别可能影响结果的缺失信息,提出一组执行前问题;得到确认以后,再按照确认后的规格执行。

它的目标不是让模型多聊天,而是在动手之前把任务合同说清楚。

IDR:先做,再通过对话修复

IDR 是 Iterative Disclosure and Repair 的缩写。模型先根据不完整的信息生成一个初稿,用户随后逐步补充要求,模型再修改结果。

这种方式更接近日常对话,但它可能反复重传上下文、重复生成,并把一部分成本花在修复本可以提前避免的问题上。

实验怎么做的?

当前 Stage A 使用了 12 个冻结的合成任务,覆盖以下类型:

  • 普通写作和技术写作;
  • Python 编程和 bug 修复;
  • 数据分析;
  • JSON 信息抽取;
  • 发布计划;
  • 决策分析;
  • 客户沟通。

每个任务、每种协议运行 3 次,分别使用两个生成模型:

  • deepseek-v4-flash
  • kimi-k2.6

每个模型都是 12 个任务 × 3 种协议 × 3 次重复,也就是 108 个协议运行。质量由独立的 qwen3.7-plus blind judge 按统一 rubric 评分,同时加入了 JSON 格式、必需字段、日期、算术和部分代码结构等 deterministic acceptance checks。

这里的“独立”是指 Qwen 没有参与生成答案,但它仍然是模型裁判,不是人工金标准。正式研究还需要人工 blind rating 和真实用户实验。

最稳定的发现:IDR 很贵,PPN 位于中间

DeepSeek 批次

  • One-shot:平均 1,632 tokens,13.41 秒;
  • PPN:平均 2,530 tokens,19.55 秒;
  • IDR:平均 10,349 tokens,47.30 秒。

Kimi 批次

  • One-shot:平均 2,154 tokens,9.81 秒;
  • PPN:平均 2,863 tokens,12.40 秒;
  • IDR:平均 11,661 tokens,34.76 秒。

PPN 相比 IDR 少用了大约 75% 的总 Token,两组模型的方向几乎一致。它也明显减少了最终延迟和 repair 开销,但比 One-shot 仍然要付出额外的提问成本。

这可能是当前实验最可靠的结果:

如果任务确实需要澄清,集中在执行前问清楚,通常比先生成、再多轮修复更节省;但如果任务本来就已经说得很清楚,前置澄清也会变成额外开销。

由于没有可核验的逐请求美元费率,这项研究没有把 Token 强行换算成美元成本。

质量结果没有支持“PPN 普遍更好”

DeepSeek:One-shot 最高

独立 judge 的平均质量分数为:

  • One-shot:88.50;
  • IDR:87.81;
  • PPN:84.58。

PPN 比 One-shot 平均低 3.92 分,但 bootstrap 区间跨过零,差异没有达到可靠的统计证据。PPN 的复合成功率为 72.2%,One-shot 为 75.0%,IDR 为 83.3%。IDR 有一次 protocol availability failure,失败记录被保留并按预先规定纳入统计,而不是静默删除。

Kimi:PPN 描述性更高,但仍不显著

Kimi 批次的平均质量分数为:

  • PPN:73.31;
  • IDR:68.36;
  • One-shot:65.83。

这一次质量方向反过来了,PPN 的分数最高。但 PPN 与 One-shot 的配对比较为 p=0.422,与 IDR 的比较为 p=0.570,都不足以支持“PPN 已经被证明更好”。

跨两个生成模型的 model × protocol interaction 也没有可靠证据。换句话说,模型之间确实出现了不同的描述性方向,但当前样本还不能证明“某种模型特别适合 PPN”。

任务越复杂,PPN 就越有效吗?目前也不能这样说

在这组任务里,低复杂度任务通常是 One-shot 表现最好;一些中等复杂度任务中,IDR 的质量分数较高,但它付出了很大的 Token 和延迟代价;高歧义任务里,PPN 也没有稳定领先。

这意味着“任务越歧义,PPN 越有效”这个直觉并没有得到当前数据的支持。更可能的情况是,是否值得澄清取决于多个因素的组合:

  • 缺失的信息是否真的会改变最终结果;
  • 任务约束是否容易被误解;
  • 直接执行失败后的返工代价有多高;
  • 用户是否愿意回答问题;
  • 任务是否会触发外部副作用。

自动验收检查也没有改变这个结论。它们可以检查 JSON、字段、日期和算术,但不能判断开放式写作是否自然,更不能证明图片里的房间结构真实。自动分数与 Qwen 质量分数的相关系数约为 0.466,因此它们适合作为辅助证据,而不是人工评分的替代品。

这项研究真正支持什么产品方向?

研究结果不支持一个“所有请求先 PPN”的强制流程,而更支持一个 adaptive clarification layer:系统先判断当前请求是否值得提问,再决定执行方式。

可以把判断逻辑理解成三种情况:

  • 简单、明确、低风险:直接执行,不要为了形式而追问;
  • 有关键歧义:只问会改变结果的高价值问题;
  • 高约束、多人协作或有外部发布风险:执行前确认关键参数,并留下可检查的记录。

这里的关键不是生成一段更长的 prompt,而是先形成一个结构化的 Edit Contract 或任务合同:目标是什么,哪些变化允许,哪些变化禁止,最后怎样验收。

回到房产图片:产品不应该只是“让图片更好看”

如果把这个思路用于房地产图片,我更希望产品是一个 property-preserving listing editor,也就是“房产事实保真的房源图片编辑器”。

默认模式应当保守一些:

  • 清理杂物和明显干扰;
  • 改善曝光、白平衡、阴影和噪点;
  • 校正镜头畸变、透视和垂直线;
  • 尽量保留空房或原有家具状态;
  • 不自动增加家具、植物、窗户、门或改变房间布局。

系统真正需要问的,可能只有几个问题:

1. 这套房子要保留空房状态,还是允许虚拟布置? 2. 哪些物体允许删除或新增? 3. “调整为平视”是校正透视和垂直线,还是要重新生成一个更高或更低的相机视角?

第三个问题尤其重要。透视校正是相对低风险的图像变换;改变相机高度、重新构造视角,则可能改变房屋的视觉事实,不能把两者混成一句“调平视”。

多角度房源图片还需要另一层约束

同一个房间的左、中、右三张照片,不能当成三张互不相关的图片分别处理。更合理的流程是:

1. 先识别它们属于同一个房间; 2. 选择一张 anchor view 作为参考; 3. 建立 room-level scene manifest,记录固定设施、门窗、墙体、地面和原有物体; 4. 对三张图片共享同一份允许修改和禁止修改的约束; 5. 批量处理后检查跨图一致性; 6. 输出原图/结果对比和 Change Log。

曝光、白平衡、降噪、裁切、透视等操作,能用确定性图像工具完成的,应尽量不要交给自由发挥的生成模型。需要生成式编辑时,也应该配合 mask 和禁止变化清单。

这也是为什么“先问一句再生成”并不能解决所有问题。多图编辑还需要场景级状态管理和结果验收。

相关研究已经走在前面

这项研究并不是第一次提出“让 LLM 澄清”。已有工作已经研究了选择性澄清、澄清问题生成、交互式意图获取以及 agent 的 help-seeking。例如:

所以,本文不把 PPN 包装成全新的澄清范式。更准确的说法是:本研究把信息披露时机变成了一个可重复的协议变量,在匹配的任务、事实、约束和验收标准下,对比直接执行、执行前批量澄清和迭代式对话修复,并联合测量质量、成功率、Token、延迟、返工负担和协议可用性。

当前研究还没有证明什么?

这轮实验有明确的边界:

  • 只有 12 个合成任务;
  • 只有两个生成模型;
  • 质量主要由一个模型 judge 评估;
  • 还没有人工 blind rating;
  • 还没有真实用户的普通聊天 vs adaptive clarification 实验;
  • 还没有完成房地产图片 benchmark;
  • 文本任务中的协议结果,不能直接证明视觉模型已经能保持多角度房间一致性。

下一阶段需要两条线同时推进:一条是普通聊天与 adaptive clarification layer 的真实用户实验,测量完成率、质量、总时间、用户输入量、澄清负担、返工和放弃率;另一条是面向房源图片的 benchmark,专门评估空房状态保持、房产事实保真、跨视角一致性和 listing readiness。

最后的判断

如果任务已经说得很清楚,One-shot 往往是更合理的默认方式。如果任务存在关键歧义,执行前澄清通常比反复返工更节省。PPN 可能是一个有用的中间方案,但目前没有证据表明它在质量上普遍优于直接执行或迭代修复。

对产品而言,最值得继续研究的不是“怎样让模型永远先问”,而是:

怎样让系统判断什么时候值得问、应该问哪一个问题,以及什么时候应该停止提问并开始执行。

对于房产图片,这个判断还必须加上一条底线:图片可以变得更干净、更适合展示,但不能因为“变好看”而虚构一套并不存在的房子。

END OF NOTEContinue reading ↗