Sequential-EQA — Beyond Episodic Evaluation

The setup

One scene, many questions — should the robot start from scratch every time?

Episodic evaluation wipes the agent's memory after every question, forcing a full re-exploration for each new query. Sequential evaluation keeps memory on across questions in the same scene. The question we ask: does simply leaving memory on actually help the robot answer better?

Episodic evaluationmemory reset

Every question re-explores the whole scene.

Sequential evaluationmemory preserved

Later questions navigate directly to what was already seen.

Q1 path Q2 path Q3 path start answer found

Figure 1. With reset memory (top), the agent repeats the full exploration loop for Q1, Q2, and Q3. With preserved memory (bottom), Q1 still explores, but Q2 and Q3 reuse the accumulated map to reach targets directly — if the memory actually retained the right evidence.

Abstract

Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes. However, real-world robots operate continuously and must accumulate, retain, and selectively reuse information acquired from prior interactions. Despite this practical requirement, the architectural mechanisms needed to support sequential memory in EQA remain underexplored.

In this work, we investigate how different memory architectures behave when EQA agents are evaluated sequentially — with multiple questions answered in the same scene while memory is carried forward across queries. We find that simply preserving existing memory is often insufficient. Agents that retain only traversability information, such as 2D occupancy maps, remember where the robot explored but not the visual-semantic evidence needed for later questions. Agents trained on short-horizon episodic data face a different challenge: continuous, multi-query histories put their inherited context out of distribution rather than forming a reusable scene representation.

To overcome this bottleneck, we highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation. Extensive experiments in simulation show that this form of memory breaks the accuracy–efficiency tradeoff in sequential settings, simultaneously achieving higher answer accuracy and lower navigation cost. We further validate these findings on a real-world mobile robot, demonstrating that spatially grounded visual memory is critical for continuous, intelligent operation in physical environments.

+33.3%

Memory advantage of 3D-Mem — the only architecture that answers more questions correctly with memory on.

+53.3%

Step advantage of 3D-Mem — navigation cost cut by more than half on subsequent queries.

4

Memory architectures evaluated, spanning VLM-agent and VLA paradigms.

498

Questions across 50 OpenEQA / HM3D scenes, regrouped into sequences.

Key takeaways

Memory persistence is not the same thing as knowledge accumulation.

Persistence ≠ Accumulation

Merely retaining state across sequential queries does not guarantee that useful environmental knowledge is being accumulated.

Spatial grounding is critical

Only spatially grounded memory reliably turns accumulated observations into simultaneous gains in answer accuracy and navigation efficiency.

Efficiency ≠ success

Shorter paths do not imply better reasoning. Unstructured memory can bias an agent to stop early in familiar regions before finding the right evidence.

Structure compounds

Structuring observations geometrically lets the agent's representation become progressively more useful over later queries in the sequence.

The protocol

Sequential-EQA: a minimal-intervention diagnostic.

We convert a widely used episodic benchmark (OpenEQA) into a sequential one without touching the environments, questions, or model weights. Questions from the same scene are grouped into one shuffled-but-fixed-order sequence, and the agent's internal memory is carried across them with no truncation or summarization.

At each question boundary we do exactly three things: inherit the terminal memory state as the next initial state, update the query and re-initialize any goal-directed planner, and keep all weights frozen. Any performance difference between episodic and sequential conditions is therefore attributable solely to the structure of the inherited representation — not to learned sequential behavior.

Benchmark: 50 scenes · 498 questions, sequences of 6–14 (most 10–13). Foundation model: Qwen3-VL 8B-Instruct (FP8) on A5000 GPUs.

Metrics

SRSuccess rate under episodic resets — memory cleared each question.
SR_memSuccess rate with memory — state persists across the sequence.
MAMemory advantage = SR_mem − SR. Does memory help it answer better?
PL / PL_memPath length (meters) without / with memory reuse.
SAStep advantage = (PL − PL_mem) / PL. Does memory help it move less?

Evaluated architectures

Four memory designs, two paradigms.

VLM-agent methods keep a frozen vision-language model and engineer memory around it; VLA methods fine-tune the model to output actions, with memory implicit in a short-horizon hidden state.

Method	Paradigm	What is stored	Retrieval mechanism
ExploreEQA	VLM-Agent	Occupancy + frontier scores	Frontier ranking by VLM
MemoryEQA	VLM-Agent	RGB + pose + language description + embedding	Entropy-based adaptive retrieval
3D-Memwinner	VLM-Agent	3D point cloud + visual embeddings	Spatially indexed 3D lookup
UniNavid	VLA	Transformer hidden state	Implicit attention

Table 1. Only 3D-Mem fuses observations into a single spatially consistent representation. ExploreEQA keeps geometry but no semantics; MemoryEQA keeps rich but unfused pose-tagged events; UniNavid keeps only a short implicit window.

Main results

Everyone moves less. Only 3D-Mem also answers better.

Method	SR	SR_mem	MA	PL	PL_mem	SA
ExploreEQA	43.8	46.5	+2.7	84.3	84.3	0.0
MemoryEQA	61.0	62.4	+1.4	43.6	43.9	−0.5
3D-Mem	25.5	58.8	+33.3	5.6	2.6	+53.3
UniNavid	36.4	37.3	+0.9	12.2	12.4	−1.5

Table 2. Success rates under episodic (SR) and sequential (SR_mem) conditions, memory advantage (MA), path lengths (PL / PL_mem) and step advantage (SA). Three of four architectures show near-zero MA; 3D-Mem gains +33.3 points of accuracy while cutting navigation cost by 53.3%.

Accuracy vs. navigation cost

Each arrow runs from episodic (hollow) to sequential (filled). Up-and-left is better.

ExploreEQA MemoryEQA 3D-Mem UniNavid episodic sequential

Figure 2. The x-axis is total path length in meters (log scale, lower is better); the y-axis is success rate. Persisting memory shifts most methods horizontally at best — only 3D-Mem climbs toward the high-accuracy, low-cost corner. A table view of the same numbers is in Table 2 above.

Real-robot validation

The bottleneck survives contact with the physical world.

We deploy a Unitree Go2 quadruped with an Intel RealSense D435i depth camera and onboard LiDAR L2, and run all four methods episodically and sequentially across a furnished indoor lab, an open lobby, and a long hallway (plus an outdoor rooftop terrace for UniNavid). Each trial is five spatially dependent questions spanning object identification, counting, and localization.

Physical sensing noise and actuation drift amplify weak memory rather than mitigating it. Without a structured map, the agent has no stable reference for deciding whether a later query can be answered from prior evidence — UniNavid re-navigates to the same chair as if the first encounter never happened, and rotates erratically before answering. Only the spatially grounded architecture improves when memory is preserved.

Success rate: episodic → sequential

Method	Episodic	Sequential
3D-Mem	20%	40%
MemoryEQA	40%	47%
ExploreEQA	33%	26%
UniNavid	15%	15%

3D-Mem doubles its success rate with memory on, mirroring simulation. ExploreEQA's dip is within the small-sample noise of five-question trials.

Citation

Cite this work

@inproceedings{cai2026beyond,
  title     = {Beyond Episodic Evaluation: Memory Architectural Bottlenecks
               in Sequential Embodied Question Answering},
  author    = {Cai, Zikui and Janga, Kaushal and Dao, Tan Dat and Lee, Seungjae
               and Dass, Shivin and Seo, Mingyo and Yue, Kaiyu and Kang, Mintong
               and Pillai, Nandhu and Hoover, Monte and Palnitkar, Aadi
               and Rawal, Ruchit and Zheng, Ruijie and Li, Bo and Zhu, Yuke
               and Mart{\'i}n-Mart{\'i}n, Roberto and Goldstein, Tom
               and Huang, Furong},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots
               and Systems (IROS)},
  year      = {2026}
}