arxiv:2610.24520

HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory

Published on Oct 20

· Submitted by

Junming Zhang on Oct 28

Shanghai Jiao Tong University

Upvote

Authors:

Junming Zhang ,

Rubin Wei ,

Xu Cheng ,

Ruoxi Cheng ,

Abstract

HABIT-Bench evaluates whether LLM agent memory systems can infer implicit user habits from sparse dialogue evidence and apply them correctly across long-term interactions.

Generated by thinkingmachines/Inkling-Small

In long-term interactions with large language model (LLM) agents, users reveal habits through their choices and actions. These habits capture implicit user information that agents must infer across conversations. However, individual observations may provide only partial evidence about a habit or when it applies. Retaining past interactions alone does not ensure that agents can infer habits and apply them appropriately. We introduce HABIT-Bench to evaluate how well agent memory systems infer habits from weak evidence and apply them to later requests. We construct synthetic user histories from task-oriented dialogue data and controlled habit graphs. The histories span hundreds of sessions across food, finance, software, and travel. Four-choice tasks assess agents' ability to infer user habits and apply them appropriately. They include temporary exceptions, changes in habits, and suggestions that users have not accepted. Sampled tasks undergo independent human review. We evaluate eight memory methods with Qwen3-235B-A22B and GLM-5.3-Flash. The best-performing memory method achieves 33.61\% macro accuracy across the four domains. Performance remains limited even when annotated evidence is supplied directly. These results expose a persistent gap between access to past interactions and the ability to infer and appropriately use user habits.

View arXiv page View PDF Project page GitHub 4 Add to collection

Community

hollyming

Paper author Paper submitter 1 day ago

Hi everyone! I'm one of the authors of HABIT-Bench 👋

Can an agent learn your habits without being explicitly told—and know when not to apply them?

Remembering a past choice is not enough. An agent needs to distinguish a recurring habit from a one-off exception, recognize when habits change, and avoid treating its own suggestions as the user's preferences.

We introduce HABIT-Bench to evaluate this ability across food, finance, software, and travel. It uses synthetic user histories spanning hundreds of sessions. The tasks require agents to piece together scattered, incomplete evidence and select an appropriate action for a new request—not simply recall something the user explicitly stated.

We evaluate eight memory methods with Qwen3-235B-A22B and GLM-5.3-Flash. The best memory-method configuration reaches 33.61% macro accuracy on four-choice tasks, compared with 25% for random guessing. Even when annotated evidence is provided directly, substantial errors remain: finding relevant history is only part of the challenge.

Remembering what happened is not the same as understanding what should guide the next decision.

💻 Code: https://github.com/LUMIA-Group/habit-bench
🤗 Dataset: https://huggingface.co/datasets/hollyming/HABIT-Bench

We'd love to hear your thoughts, discuss the findings, and see how other memory systems perform on HABIT-Bench!

alphabell-research

about 14 hours ago

Table 4 shows a useful diagnostic gap: with Qwen3-235B, Oracle Evidence scores 39.26% to 58.25% across domains, while Oracle Habit State scores 98.41% to 99.26%. The latter also exposes the required action, so the gap combines state inference and answer information. A next slice could score the model's predicted active habit state separately from the final four-choice action. I would track each habit's binding status (user confirmed, suggestion only, temporary exception, superseded) with source turns. Can the released probes support that intermediate score?

hollyming

Paper author about 13 hours ago

Thanks for the suggestion! Yes—the released probes and their accompanying gold annotations provide a basis for evaluating intermediate habit states separately from the final four-choice answer.

In gold//probe_key.jsonl, Food provides habit graphs with default, boundary, and exception rules, together with evidence components such as user selection, ratified reuse, and rejected suggestions. Finance and Software provide target habits, target-state times, policy signatures, and separate decisive, temporal-context, and nonbinding evidence. Travel includes habit graphs and compositional state annotations. These session references link back to the original user–assistant dialogue in data//lifelines.jsonl. The gold annotations are for scoring, not model input. Dataset schema (https://huggingface.co/datasets/hollyming/HABIT-Bench#files-and-schema)

Human review and adjudication were also part of our validation process to check that the intended answers are supported by the evidence. Regarding Table 4, Oracle Habit State intentionally supplies the resolved state and required action, as described in §4.3; its high accuracy is therefore consistent with its role as a privileged diagnostic reference. The released annotations allow researchers to examine the underlying state and supporting evidence rather than rely only on final-choice accuracy

alphabell-research

about 13 hours ago

@hollyming Thanks for the schema pointers. Keeping gold annotations scoring-only is exactly right. I would report three stages: evidence recall by session ID, active habit state at probe time, and final choice. Then split P(state correct | evidence hit) and P(choice correct | state correct) by exception and unaccepted-suggestion probes. That would tell builders whether to fix retrieval, temporal state updates, or action selection. The released gold fields look rich enough to define this without exposing labels to the agent.

hollyming

Paper author about 13 hours ago

Thanks for the thoughtful suggestion! We agree that this three-stage breakdown would help builders distinguish retrieval, state-tracking, and action-selection errors, making the evaluation more actionable.