Sequential-EQA — Beyond Episodic Evaluation

The setup

One scene, many questions — should the robot start from scratch every time?

Episodic evaluation wipes the agent's memory after every question, forcing a full re-exploration for each new query. Sequential evaluation keeps memory on across questions in the same scene. The question we ask: does simply leaving memory on actually help the robot answer better?

Episodic evaluationmemory reset

Every question re-explores the whole scene.

Q1full exploration

Q2full exploration

Q3full exploration

Sequential evaluationmemory preserved

Later questions navigate directly to what was already seen.

Q1full exploration

Sequential Q2 — direct navigation reusing memory

Q2direct — reuses memory

Sequential Q3 — direct navigation reusing memory

Q3direct — reuses memory

Q1 path Q2 path Q3 path start answer found

Figure 1. Comparison between episodic and sequential evaluation paradigms. With memory reset (top), the agent repeats the full exploration loop for Q1, Q2 and Q3 as if the environment were novel each time. With memory preserved (bottom), Q1 still explores, but Q2 and Q3 navigate directly to previously observed targets, eliminating redundant re-exploration.

Abstract

Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes. However, real-world robots operate continuously and must accumulate, retain, and selectively reuse information acquired from prior interactions. Despite this practical requirement, the architectural mechanisms needed to support sequential memory in EQA remain underexplored.

In this work, we investigate how different memory architectures behave when EQA agents are evaluated sequentially — with multiple questions answered in the same scene while memory is carried forward across queries. We find that simply preserving existing memory is often insufficient. Agents that retain only traversability information, such as 2D occupancy maps, remember where the robot explored but not the visual-semantic evidence needed for later questions. Agents trained on short-horizon episodic data face a different challenge: continuous, multi-query histories put their inherited context out of distribution rather than forming a reusable scene representation.

To overcome this bottleneck, we highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation. Extensive experiments in simulation show that this form of memory breaks the accuracy–efficiency tradeoff in sequential settings, simultaneously achieving higher answer accuracy and lower navigation cost. We further validate these findings on a real-world mobile robot, demonstrating that spatially grounded visual memory is critical for continuous, intelligent operation in physical environments.

+33.3%

Memory advantage of 3D-Mem — the only architecture that answers more questions correctly with memory on.

+53.3%

Step advantage of 3D-Mem — navigation cost cut by more than half on subsequent queries.

4

Memory architectures evaluated, spanning VLM-agent and VLA paradigms.

498

Questions across 50 OpenEQA / HM3D scenes, regrouped into sequences.

Key takeaways

Memory persistence is not the same thing as knowledge accumulation.

Persistence ≠ Accumulation

Merely retaining state across sequential queries does not guarantee that useful environmental knowledge is being accumulated.

Spatial grounding is critical

Only spatially grounded memory reliably turns accumulated observations into simultaneous gains in answer accuracy and navigation efficiency.

Efficiency ≠ success

Shorter paths do not imply better reasoning. Unstructured memory can bias an agent to stop early in familiar regions before finding the right evidence.

Structure compounds

Structuring observations geometrically lets the agent's representation become progressively more useful over later queries in the sequence.

The protocol

Sequential-EQA: a minimal-intervention diagnostic.

We convert a widely used episodic benchmark (OpenEQA) into a sequential one without touching the environments, questions, or model weights. Questions from the same scene are grouped into one shuffled-but-fixed-order sequence, and the agent's internal memory is carried across them with no truncation or summarization.

At each question boundary we do exactly three things: inherit the terminal memory state as the next initial state, update the query and re-initialize any goal-directed planner, and keep all weights frozen. Any performance difference between episodic and sequential conditions is therefore attributable solely to the structure of the inherited representation — not to learned sequential behavior.

Benchmark: 50 scenes · 498 questions, sequences of 6–14 (most 10–13). Foundation model: Qwen3-VL 8B-Instruct (FP8) on A5000 GPUs.

Metrics

SRSuccess rate under episodic resets — memory cleared each question.
SR_memSuccess rate with memory — state persists across the sequence.
MAMemory advantage = SR_mem − SR. Does memory help it answer better?
PL / PL_memPath length (meters) without / with memory reuse.
SAStep advantage = (PL − PL_mem) / PL. Does memory help it move less?

Evaluated architectures

Four memory designs, two paradigms.

VLM-agent methods keep a frozen vision-language model and engineer memory around it; VLA methods fine-tune the model to output actions, with memory implicit in a short-horizon hidden state.

Method	Paradigm	What is stored	Retrieval mechanism
ExploreEQA	VLM-Agent	Occupancy + frontier scores	Frontier ranking by VLM
MemoryEQA	VLM-Agent	RGB + pose + language description + embedding	Entropy-based adaptive retrieval
3D-Memwinner	VLM-Agent	3D point cloud + visual embeddings	Spatially indexed 3D lookup
UniNavid	VLA	Transformer hidden state	Implicit attention

Table 1. Only 3D-Mem fuses observations into a single spatially consistent representation. ExploreEQA keeps geometry but no semantics; MemoryEQA keeps rich but unfused pose-tagged events; UniNavid keeps only a short implicit window.

Main results

Everyone moves less. Only 3D-Mem also answers better.

Method	SR	SR_mem	MA	PL	PL_mem	SA
ExploreEQA	43.8	46.5	+2.7	84.3	84.3	0.0
MemoryEQA	61.0	62.4	+1.4	43.6	43.9	−0.5
3D-Mem	25.5	58.8	+33.3	5.6	2.6	+53.3
UniNavid	36.4	37.3	+0.9	12.2	12.4	−1.5

Table 2. Success rates under episodic (SR) and sequential (SR_mem) conditions, memory advantage (MA), path lengths (PL / PL_mem) and step advantage (SA). Three of four architectures show near-zero MA; 3D-Mem gains +33.3 points of accuracy while cutting navigation cost by 53.3%.

Accuracy vs. navigation cost

Each arrow runs from episodic (hollow) to sequential (filled). Up-and-left is better.

ExploreEQA MemoryEQA 3D-Mem UniNavid episodic sequential

Figure 2. The x-axis is total path length in meters (log scale, lower is better); the y-axis is success rate. Persisting memory shifts most methods horizontally at best — only 3D-Mem climbs toward the high-accuracy, low-cost corner. A table view of the same numbers is in Table 2 above.

Detailed analysis

Where the gain comes from — and where it fails to compound.

Aggregate metrics show average gains but do not reveal whether memory becomes more useful as a sequence progresses. Two views make the difference visible: per-method means with standard errors, and the same metrics broken down by query position.

Bar charts comparing mean answer score, navigation time and navigation distance between episodic and sequential evaluation across four methods, plus a joint efficiency-vs-accuracy scatter plot. — **Figure 4.** From left to right: mean answer score (1–5; ↑), mean navigation time (seconds; ↓), mean navigation distance (meters; ↓), and a joint efficiency-vs-accuracy scatter plot. Light bars denote episodic (no memory) evaluation; dark bars denote sequential (with memory) evaluation; error bars indicate standard error. Across most methods, memory persistence changes navigation cost more readily than answer quality — only 3D-Mem moves toward the desirable region of higher accuracy and lower navigation cost.

Two-panel line chart of mean answer score and mean navigation time as a function of query position within a scene sequence, comparing episodic and sequential evaluation. — **Figure 5.** Panel A shows mean answer score (1–5; ↑, log scale); Panel B shows mean navigation time (seconds; ↓, log scale). The x-axis is query position i within a scene sequence; each position averages over all sequences with *N_k* ≥ i, not only sequences of length 13. Δ denotes sequential minus episodic performance and is color-coded by sign: green for Δ > 0 and red for Δ < 0. Because metric direction differs across panels, green indicates improvement only for Panel A, whereas a red/negative Δ in Panel B indicates an improvement, since lower time is better.

Lower cost is not evidence of accumulation

Navigation time generally decreases under memory reuse, since even weak memory can reduce repeated exploration or bias the agent toward familiar regions. It may equally reflect premature stopping or over-reliance on incomplete history.

Only one method improves monotonically

ExploreEQA, UniNavid and MemoryEQA show no stable improvement across query positions; gains and losses alternate as history accumulates. Only 3D-Mem shows sustained positive accuracy Δ, consistent with a representation that becomes more complete over time.

Qualitative analysis

Three questions, one apartment, two different agents.

Trajectories for three consecutive questions in the same scene, under episodic (left) and sequential (right) evaluation. Q1 is blue, Q2 orange, Q3 purple.

3D-Mem trajectories for three questions in the same apartment under episodic and sequential evaluation, showing reduced path length and correct answers under sequential evaluation. — **Figure 6a.** Under episodic evaluation the agent re-explores large portions of the apartment for every question, accumulating substantial redundant path length, and answers only 1 of 3 correctly — asked what color the smoke detector is, it returns an answer about a coat closet. With a persistent 3D map, path length drops sharply across Q2 and Q3 and all three questions are answered correctly, including the smoke detector query, which the agent now answers from observations already fused into its 3D representation during the first query's exploration.

UniNavid trajectories for three questions on an MP3D scene under episodic and sequential evaluation, showing a correction on Q2 with minimal change in navigation efficiency. — **Figure 6b.** UniNaviD trajectories on another MP3D scene. The agent answers 2 of 3 questions correctly episodically; sequential state inheritance corrects Q2, yielding 3 of 3. Unlike 3D-Mem, navigation efficiency changes minimally between the two settings — the improvement is not accompanied by a shorter path.

Real-robot validation

The bottleneck survives contact with the physical world.

We deploy a Unitree Go2 quadruped with an Intel RealSense D435i depth camera and onboard LiDAR L2, and run all four methods episodically and sequentially across a furnished indoor lab, an open lobby, and a long hallway (plus an outdoor rooftop terrace for UniNavid). Each trial is five spatially dependent questions spanning object identification, counting, and localization.

Physical sensing noise and actuation drift amplify weak memory rather than mitigating it. Without a structured map, the agent has no stable reference for deciding whether a later query can be answered from prior evidence — UniNavid re-navigates to the same chair as if the first encounter never happened, and rotates erratically before answering. Only the spatially grounded architecture improves when memory is preserved.

Success rate: episodic → sequential

Method	Episodic	Sequential
3D-Mem	20%	40%
MemoryEQA	40%	47%
ExploreEQA	33%	26%
UniNavid	15%	15%

3D-Mem doubles its success rate with memory on, mirroring simulation. ExploreEQA's dip is within the small-sample noise of five-question trials.

UniNavid on the Unitree Go2, indoor and outdoor

real-uninavid

Indoor · lobby

Q1“What color is the pot next to the black chair?”

Q2“How many yellow chairs are in this room?”

The agent re-navigates to the chair for the second question as if the first encounter never occurred.

Outdoor · rooftop terrace

Q1“What's the color of the cone?”

Q2“What's the object behind the cone?”

The agent repeats its navigation pattern for the second query rather than reusing what it already saw.

Figure 7. Real-world deployment of the Unitree Go2; each panel plays the recorded trial. In both environments UniNavid fails to leverage previous knowledge. Indoors (left) the agent re-navigates to the chair for the second question as if the first encounter never occurred; outdoors (right) it repeats its navigation pattern for the second query. In both cases the agent rotates erratically and fails to answer correctly, confirming that the sequential memory reuse bottleneck generalizes to physical deployment.

Citation

Cite this work

@inproceedings{cai2026beyond,
  title     = {Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering},
  author    = {Cai, Zikui and Janga, Kaushal and Dao, Tan Dat and Lee, Seungjae and Dass, Shivin and Seo, Mingyo and Yue, Kaiyu and Kang, Mintong and Pillai, Nandhu and Hoover, Monte and Palnitkar, Aadi and Rawal, Ruchit and Zheng, Ruijie and Li, Bo and Zhu, Yuke and Mart{\'i}n-Mart{\'i}n, Roberto and Goldstein, Tom and Huang, Furong},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year      = {2026}
}

Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering