Core idea
Reflexion was introduced by Shinn, Cassano, Berman, Gopinath, Narasimhan, and Yao in Reflexion: Language Agents with Verbal Reinforcement Learning. The authors frame the problem as follows: large language models can act in environments such as games, compilers, APIs, and search systems, but conventional reinforcement learning usually requires many samples and expensive parameter updates. Reflexion instead “reinforces” the agent by adding self-generated textual feedback to memory: a failed trajectory is evaluated, summarized into a useful lesson, and replayed as context on the next attempt. arXiv
The loop is simple enough to describe in one sentence:
attempt → evaluate → reflect → store reflection → retry with reflection in context.
That simplicity is the point. Reflexion treats the prompt and memory buffer as part of the policy. The underlying model weights remain fixed; what changes is the agent’s situational context. Shinn et al. explicitly describe this as a new paradigm of verbal reinforcement that parameterizes the policy by the pair of fixed LLM parameters and a mutable memory encoding. arXiv
This makes Reflexion a central entry in the lineage from Chain of Thought and ReAct toward self-improving agent scaffolds. ReAct interleaves reasoning traces and actions so a model can think, act, observe, and continue; Reflexion adds a cross-attempt memory channel so failed trajectories can be compressed into reusable lessons. arXiv
Algorithmic design
Reflexion has three functional modules plus memory:
| Component | Role | Typical instantiation |
|---|---|---|
| Actor | Produces task outputs, reasoning traces, or environment actions | LLM prompted as ReAct, Chain of Thought, or code generator |
| Evaluator | Scores the trajectory or output | Environment success signal, exact-match metric, compiler/test result, heuristic, or another LLM |
| Self-reflection model | Converts feedback and trajectory into verbal guidance | LLM prompted to diagnose failure and propose a better strategy |
| Memory | Stores reflections and sometimes recent trajectories | Sliding buffer of one to three reflections in the original paper |
The Actor is the agent’s policy at inference time. It receives the task, environment observations, few-shot examples, and the current memory. In ALFWorld, the actor is a ReAct-style agent that alternates between thoughts and actions. In HotPotQA, the actor can be either a Chain-of-Thought reasoner over provided context or a ReAct agent that searches through a Wikipedia API. In programming, the actor writes code, generates tests, observes failures, and tries again. arXiv
The Evaluator supplies the reinforcement signal. In the paper, this signal is deliberately flexible: exact-match grading for question answering, environment success or heuristic failure detection for ALFWorld, and compiler or self-generated unit-test feedback for code. The important design choice is that the evaluator need not output a dense reward model. A binary success/failure signal can be enough if the self-reflection model can turn it into useful language. arXiv
The Self-Reflection model reads the failed trajectory, the reward or feedback signal, and any prior memory. It then writes a compact natural-language explanation of what went wrong and how the next attempt should differ. Shinn et al. call this reflection more informative than a scalar reward because it can point to specific mistakes, alternative actions, or revised plans. arXiv
The memory system distinguishes short-term and long-term context. The trajectory itself is short-term memory: the detailed sequence of thoughts, actions, observations, code, or answers generated during one attempt. The reflection buffer is long-term episodic memory: compressed lessons carried across attempts. In practice, Shinn et al. bounded this buffer to a small sliding window, usually one to three reflections, because of LLM context-length limits. arXiv
A compact formalization is:
τt∼πθ,mt(x)\tau_t \sim \pi_{\theta, m_t}(x)τt∼πθ,mt(x) rt=E(τt)r_t = E(\tau_t)rt=E(τt) ρt=R(x,τt,rt,mt)\rho_t = R(x, \tau_t, r_t, m_t)ρt=R(x,τt,rt,mt) mt+1=truncate(mt∪ρt)m_{t+1} = \operatorname{truncate}(m_t \cup \rho_t)mt+1=truncate(mt∪ρt) Here θ\thetaθ is fixed model weight state, mtm_tmt is the reflection memory, EEE is the evaluator, RRR is the reflection generator, and τt\tau_tτt is the trajectory on trial ttt. Reflexion’s “learning” happens through mtm_tmt, not through θ\thetaθ. This is why it is attractive when fine-tuning is unavailable, too expensive, or blocked by proprietary model access.
What “verbal reinforcement learning” means
The phrase Verbal Reinforcement Learning is useful but easy to overread. Reflexion is not reinforcement learning in the narrow sense of estimating gradients through reward signals and updating a policy network. It is closer to test-time policy improvement through language-mediated memory. The authors explicitly say Reflexion reinforces agents “not by updating weights, but instead through linguistic feedback,” and describe the reflection as a kind of “semantic gradient signal.” arXiv
The analogy to RL is still meaningful. There is an agent, an environment, trajectories, reward-like feedback, and iterative policy improvement. But the update rule is symbolic and contextual: the agent writes advice to itself. The stored advice can say things like “I searched the wrong container first,” “I confused the comparison relation,” or “my implementation failed on the empty-list case.” That kind of feedback is more structured than a scalar reward, but it is also less formally grounded. It can be wrong, vague, or overfit to one failed attempt.
This positions Reflexion between three traditions:
| Tradition | What changes? | Feedback source | Reflexion’s relation |
|---|---|---|---|
| Prompting | Context | Human-written prompt or examples | Reflexion dynamically writes its own prompt fragments |
| Reinforcement learning | Model parameters or policy table | Reward signal | Reflexion uses reward-like signals but updates memory, not weights |
| Debugging/postmortems | Human strategy | Execution trace and outcome | Reflexion automates the postmortem and reuses it on the next trial |
The methodological claim is therefore modest but important: a fixed LLM plus a feedback-aware memory scaffold can improve over repeated attempts, even when model weights are inaccessible. This matters for API-based agents, closed models, safety-constrained deployments, and fast engineering loops where retraining is not practical.
Empirical results
Shinn et al. evaluate Reflexion on three task families: sequential decision-making, multi-hop reasoning, and programming. The main reported gains are an absolute 22% improvement on ALFWorld, 20% on HotPotQA, and 11% on HumanEval Python. arXiv
| Benchmark | Task type | Baseline context | Reflexion result | Main lesson |
|---|---|---|---|---|
| ALFWorld | Text-based household decision-making | ReAct-style acting over 134 environments | 130/134 tasks solved with heuristic self-evaluation | Reflection helps recover from long-trajectory planning and object-location mistakes |
| HotPotQA | Multi-hop question answering | CoT or ReAct over 100 sampled questions | Model-specific gains, such as GPT-4 CoT with ground-truth context improving from 0.68 to 0.80 | Reflection can repair reasoning/search mistakes when exact-match feedback is available |
| HumanEval | Code generation | GPT-4 zero-shot baseline at 80.1 pass@1 | 91.0 pass@1 on Python HumanEval | Self-generated tests plus reflection can improve code without using hidden tests for selection |
ALFWorld
ALFWorld is a text-grounded household environment aligned with embodied tasks from ALFRED. It gives agents goals like finding objects, moving objects, manipulating objects, or using one object with another. The benchmark is useful for Reflexion because tasks require multi-step action sequences, but the environment can also provide a clear success/failure signal. arXiv
In the Reflexion ALFWorld setup, the actor is a ReAct agent. ReAct supplies interleaved “think” and “act” traces; Reflexion adds cross-trial memory. The authors run 134 ALFWorld environments across six task types and use either an LLM classifier or a simple heuristic to decide when the agent is stuck. The heuristic detects repeated action/observation loops or excessively long trajectories. arXiv
The headline result is that ReAct + Reflexion solves 130 of 134 ALFWorld tasks using the simple heuristic, with learning occurring over as many as 12 consecutive trials. The paper’s analysis emphasizes two recurrent failure modes: agents believe they possess an item they do not actually have, or they search inefficiently through many possible containers and surfaces. Reflections convert long failed trajectories into concise “self-hints” about where the mistaken assumption or bad search order occurred. arXiv
A representative ALFWorld reflection identifies that the task “examine the mug with the desklamp” should have prioritized locating the desklamp rather than repeatedly trying to use it incorrectly. On the next trial, the agent goes directly to the relevant desk, picks up the mug, and successfully uses the desklamp. This example captures Reflexion’s strongest case: the failure is localizable, the environment can be reset, and the lesson is short enough to carry forward. arXiv
HotPotQA
HotPotQA is a Wikipedia-based multi-hop QA dataset with roughly 113,000 question-answer pairs. It was designed to require reasoning across multiple supporting documents and includes sentence-level supporting facts for explainability. arXiv
In the Reflexion paper, HotPotQA is used in two modes. The first isolates reasoning: the agent receives ground-truth context and uses Chain-of-Thought reasoning. The second tests retrieval plus reasoning: a ReAct agent searches a Wikipedia API and then answers. The evaluator uses exact match between the produced answer and the ground truth, which gives the reflection model a binary success signal. arXiv
The model-specific appendix table shows that Reflexion improves several HotPotQA variants: CoT with ground-truth context using text-davinci-003 improves from 0.60 to 0.77; the same setup with GPT-3.5 improves from 0.57 to 0.71; GPT-4 improves from 0.68 to 0.80; and ReAct-style GPT-4 improves from 0.39 to 0.51. arXiv
The important methodological detail is that non-Reflexion baselines did not improve just by retrying with sampling. Shinn et al. report that ReAct-only, CoT-only, and CoT-with-ground-truth-context-only approaches failed to probabilistically improve on tasks in later attempts, while Reflexion improved by carrying explicit lessons across attempts. arXiv
HumanEval and code generation
HumanEval is a functional-correctness benchmark for code synthesis from docstrings, introduced with OpenAI’s Codex work. It evaluates whether generated programs pass unit tests, making it a natural environment for feedback-driven code agents. arXiv
In the Reflexion programming setup, the agent generates code, generates its own tests, executes those tests, and reflects on failures before trying again. The authors argue that the method remains eligible for pass@1-style reporting because it does not use hidden benchmark tests for selection; instead, it uses self-generated tests and public execution feedback. arXiv
The main HumanEval Python result is 91.0 pass@1 for Reflexion versus 80.1 for the GPT-4 baseline reported by the paper. Reflexion also improves HumanEval Rust on the hardest translated subset from 60.0 to 68.0, MBPP Rust from 70.9 to 75.4, and LeetcodeHard Python from 7.5 to 15.0. arXiv
The programming results also expose a failure mode. On MBPP Python, Reflexion underperforms the GPT-4 baseline: 77.1 versus 80.1. The authors trace this to self-generated test unreliability. If internal tests pass an incorrect solution, the agent can prematurely stop; if bad tests fail a correct solution, the agent may edit working code. The paper reports a false-positive test-execution rate of 16.3% on MBPP Python versus 1.4% on HumanEval Python, explaining much of the discrepancy. arXiv
This is one of the cleanest lessons from Reflexion: verbal self-improvement is only as good as the evaluator. In code, tests are a strong evaluator when they cover the real specification; they are a weak or harmful evaluator when they encode the wrong behavior.
Why Reflexion mattered
Reflexion’s methodological contribution is that it made Self-Improving Agents concrete without requiring parameter updates. Earlier agent scaffolds had memory, planning, or tool use, but Reflexion emphasized a specific and reusable loop: distill task feedback into natural-language lessons, preserve them in episodic memory, and reuse them on the same task. The official repository includes code, demos, and logs for ALFWorld, HotPotQA, programming, and WebShop experiments, making the method easy to reproduce conceptually even if API costs and model access make exact reproduction harder. GitHub
For AI engineering, the practical appeal is obvious. Many production systems use closed models, frozen open-weight models, or deployment settings where fine-tuning every task is too slow. Reflexion lets the harness learn a little even when the model cannot. The harness can collect traces, run validators, produce postmortems, and prepend those postmortems to later calls.
The intellectual appeal is subtler. Reflexion reframes “learning” as a property of the whole agent system rather than just the neural network. A frozen model plus mutable memory can exhibit trial-by-trial adaptation. That adaptation is weaker than true parameter learning, but it is inspectable: the reflection buffer is readable. Shinn et al. note that this can make agent behavior more diagnosable than black-box RL policy updates. arXiv
Relation to other self-improvement loops
ReAct
ReAct is the immediate predecessor. ReAct interleaves reasoning and acting: the model writes thoughts, chooses actions, receives observations, and continues. It improves interpretability and helps reduce hallucination by letting the model interact with external tools or environments, including Wikipedia APIs, ALFWorld, and WebShop. arXiv
Reflexion can be seen as ReAct plus memory across attempts. ReAct improves the current trajectory; Reflexion improves the next trajectory by adding a postmortem from the previous one.
Self-Refine
Self-Refine is a nearby but different loop. It generates an initial output, asks the same LLM to produce feedback, and then asks it to refine the output. It requires no supervised data, no additional training, and no reinforcement learning, and the authors report roughly 20% absolute average improvement across evaluated tasks. arXiv
The distinction is that Self-Refine is primarily output-level iterative editing, often within a single generation task. Reflexion is agentic and episodic: it uses environment feedback, failed trajectories, and a memory buffer across attempts.
Tree of Thoughts
Tree of Thoughts generalizes Chain-of-Thought by searching over coherent intermediate “thoughts” rather than committing to one left-to-right chain. It allows the model to explore multiple reasoning paths, self-evaluate, backtrack, and select promising branches. In the Game of 24, the paper reports GPT-4 Chain-of-Thought solving 4% of tasks while Tree of Thoughts solves 74%. arXiv
Tree of Thoughts and Reflexion address different failure modes. ToT combats premature commitment during one attempt by searching over alternatives. Reflexion combats repeated failure across attempts by carrying forward lessons. ToT is breadth/depth in search space; Reflexion is memory over trials.
Language Agent Tree Search
Language Agent Tree Search is the clearest follow-up synthesis. LATS combines planning, acting, reasoning, external feedback, and self-reflection using a Monte Carlo Tree Search-like framework. The paper explicitly positions LATS as a unification of methods such as ReAct, Reflexion, Tree of Thoughts, and Reasoning via Planning. arXiv
LATS uses LLMs as agents, value functions, and optimizers; it expands possible reasoning/action trajectories as a tree and uses environment feedback plus self-reflection to guide search. The public LATS project table reports GPT-3.5 results where LATS outperforms Reflexion on HotPotQA exact match, HumanEval pass@1, and WebShop score: 0.71 vs 0.51 on HotPotQA, 83.8 vs 68.1 on HumanEval, and 75.9 vs 64.2 on WebShop. lapisrocks.github.io
The conceptual lesson is that reflection and search are complementary. Reflexion learns from past failed complete trajectories; LATS tries to avoid bad trajectories by exploring alternatives before committing.
AdaPlanner
AdaPlanner is another feedback-driven agent loop. Instead of storing self-reflections after whole attempts, it refines self-generated plans during execution in response to environmental feedback. The paper targets the weakness of static plans in sequential decision-making and reports improvements on ALFWorld and MiniWoB++ while using fewer samples than some baselines. arXiv
AdaPlanner and Reflexion share the belief that feedback should modify the agent’s future behavior in language. They differ in granularity: AdaPlanner modifies plans inside the execution loop; Reflexion writes episodic lessons between attempts.
ExpeL and cross-task experience
ExpeL pushes the memory idea beyond retrying the same task. It autonomously gathers experiences, extracts natural-language knowledge, and recalls both insights and past examples at inference time. The authors explicitly motivate the method partly by the fact that frontier model weights are often proprietary and unavailable, creating a need for learning from experience without parametric updates. AAAI Publications
Where Reflexion is same-task retry memory, ExpeL is closer to cross-task experiential memory. That distinction matters for scaling: long-horizon agents rarely get to repeat the exact same initial state. They need abstractions that transfer across related tasks.
Re-ReST and reflection as training data
Re-ReST — Reflection-Reinforced Self-Training — moves reflection from inference-time memory into training-time data generation. It uses a reflector to improve low-quality agent samples using environmental feedback, then trains the agent on the resulting higher-quality trajectories. The paper reports that self-training improves over few-shot baselines by 7.6% on HotPotQA and 28.4% on ALFWorld, and that adding Re-ReST further boosts performance by 2.0% and 14.1% respectively. ACL Anthology
This is a natural continuation of Reflexion. Reflexion asks: can a frozen model improve by remembering reflections? Re-ReST asks: can reflections help generate better data so that an open model can actually be fine-tuned? Its authors also stress a limitation of inference-time reflection: reliable feedback may be unavailable at test time, so their method uses the reflector during training and deploys only the agent model at inference. ACL Anthology
FireAct and LUMOS
FireAct and LUMOS represent the contrasting path: improve agents through parameter updates. FireAct investigates fine-tuning language models into agents and reports that fine-tuning Llama2-7B with 500 GPT-4-generated trajectories yields a 77% HotPotQA performance increase. LUMOS trains open-source modular agents with planning, grounding, and execution modules using unified data across interactive tasks. arXiv
These works do not invalidate Reflexion; they clarify its niche. Reflexion is especially useful when the model cannot be fine-tuned. Fine-tuning approaches are more compelling when you control weights, can collect enough trajectories, and need amortized performance rather than repeated test-time calls.
Conditions where Reflexion works best
Reflexion is most effective when five conditions hold.
First, the task must permit repeated attempts. ALFWorld can reset the environment. Code can be regenerated. HotPotQA questions can be re-answered. A one-shot irreversible environment is much less forgiving.
Second, feedback must be reliable enough to diagnose failure. Exact match, compiler errors, unit tests, and environment success signals are useful. Purely aesthetic or preference-based tasks are harder unless the evaluator is strong.
Third, failures must be compressible into actionable language. “I searched the wrong cabinet” is useful. “My global strategy is flawed in a way I cannot identify” is not.
Fourth, the memory buffer must stay relevant. Shinn et al. use small bounded memory because context windows are finite. Later memory-agent surveys describe memory design as a write–manage–read loop and identify trustworthy reflection, continual consolidation, causally grounded retrieval, learned forgetting, and multimodal embodied memory as open challenges. arXiv
Fifth, the actor must be capable enough to use the reflection. Shinn et al. report that smaller or weaker models may not benefit; in an additional HumanEval Python experiment with starchat-beta, baseline and Reflexion both average 0.26 pass@1, and the authors conclude that specifying useful self-corrections appears to be an emergent quality of stronger models. arXiv
Limitations
Reflection quality is a bottleneck
Reflexion depends on the reflection being true, specific, and actionable. If the self-reflection model misdiagnoses the failure, the memory buffer becomes a source of error. This is not hypothetical: the paper’s programming analysis shows that flawed self-generated tests can produce false positives or false negatives, and those labels condition the next reflection. arXiv
This aligns with broader self-correction research. Huang et al. find that LLMs struggle to self-correct reasoning without external feedback and can even degrade after self-correction. Kamoi et al.’s critical survey similarly concludes that prompted self-correction is not generally demonstrated except in tasks suited to it, while self-correction works better when reliable external feedback is available. arXiv
Reflexion is therefore strongest when it is not purely introspective. The evaluator matters. A model reflecting on its own reasoning without a trustworthy external signal is much less reliable than a model reflecting on a failed unit test, exact-match answer, or environment observation.
It can compound errors
A bad reflection can become a bad instruction. If the agent stores “I should always search the drawer first” after a coincidental success, that heuristic may hurt later attempts. If the agent wrongly concludes that its code failed because of edge-case handling when the real bug is a misunderstood specification, subsequent edits can move farther from the correct solution.
This is a general risk in Agent Memory: memory is not automatically knowledge. It is unverified state. The memory write path needs filtering, contradiction handling, and forgetting. Later surveys of autonomous agent memory identify contradiction handling, write-path filtering, latency budgets, privacy, and trustworthy reflection as engineering realities rather than solved details. arXiv
It can get stuck in local minima
Shinn et al. explicitly acknowledge that Reflexion can fall into non-optimal local minima. The WebShop appendix is the clearest example. In a 100-environment WebShop experiment, ReAct + Reflexion did not significantly outperform ReAct, and the authors stopped after four trials because the agent showed no sign of improvement. They conclude that Reflexion struggles on tasks requiring significant diversity and exploration. arXiv
This is a serious limitation. Reflexion’s default loop improves a single trajectory family by reflecting on its failures. It does not inherently solve exploration. If the agent never tries a qualitatively different query, tool, plan, or decomposition, reflection may only polish a bad strategy.
It increases inference cost
Every failed attempt can add calls for acting, evaluation, reflection, and retry. In code, the loop also runs tests or compilers. In web or embodied tasks, it may require tool calls and environment resets. Reflexion trades training cost for test-time cost. That trade is often favorable for small task batches or closed-model deployments, but less favorable for high-throughput systems where fine-tuning or distillation can amortize the improvement.
It is not enough for long-horizon agency
Long-horizon agents need more than same-task postmortems. They need persistent memory across sessions, retrieval, skill abstraction, uncertainty estimation, plan repair, safety constraints, and robust evaluation. Reflexion supplies one ingredient: verbal lessons from failed attempts. It does not by itself solve long-term credit assignment, hierarchical planning, or open-ended exploration.
Open question: does verbal RL scale?
The central open question is whether verbal-RL approaches scale from repeatable benchmark tasks to long-horizon agentic work. The evidence is mixed.
The positive case is strong enough to take seriously. Reflexion shows that small textual memories can improve fixed-model agents on ALFWorld, HotPotQA, and HumanEval. LATS shows that reflection can be combined with tree search and value estimation. ExpeL and Re-ReST show that experience extraction and reflection can be generalized into cross-task memory or training data. ACL Anthology+3arXiv+3arXiv+3
The negative case is equally important. Reflexion fails to significantly improve WebShop in the authors’ appendix, self-correction without external feedback is empirically weak, and current agent-evaluation surveys still identify long-horizon planning, robustness, cost-efficiency, and scalable fine-grained evaluation as major gaps. arXiv+2arXiv+2
The 2026 work on experiential reflective learning is a useful snapshot of the frontier. On Gaia2, a simulated mobile long-horizon benchmark with 12 applications and 101 tools, ERL improves a ReAct baseline from 48.3% to 56.1% overall success and improves over the strongest prior method in that comparison. That is meaningful progress, but also a reminder that even reflection-augmented agents remain far from robust completion on long-horizon tasks. arXiv
The likely answer is that verbal RL scales as a component, not as a complete solution. Reflections are useful when they are grounded, retrieved selectively, tested against future experience, and combined with search, planning, tool feedback, and memory management. They are brittle when treated as magic introspection.
Engineering interpretation
For practical AI systems, Reflexion suggests a design pattern:
-
Log complete trajectories.
-
Evaluate them with the strongest available verifier.
-
Generate concise failure analyses.
-
Store only lessons likely to transfer.
-
Retrieve a small number of relevant lessons on the next attempt.
-
Delete or down-rank lessons contradicted by later evidence.
The most important engineering decision is not the reflection prompt. It is the feedback channel. A Reflexion loop around a weak verifier can make the system more confidently wrong. A Reflexion loop around a strong verifier can turn failures into usable search gradients.
This is why code is a natural domain for Reflexion: compilers and tests provide relatively grounded feedback. It is also why open-ended web tasks are harder: the evaluator is weaker, the action space is larger, and the correct strategy may require exploration rather than local repair.
Selected primary sources
| Source | Why it matters |
|---|---|
| Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning | Original Reflexion algorithm, ALFWorld/HotPotQA/HumanEval results, limitations, and code release. arXiv |
| Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models | Immediate predecessor for interleaved thought/action trajectories used inside Reflexion. arXiv |
| Yao et al., Tree of Thoughts | Parallel search-based inference method; useful contrast with Reflexion’s memory-based improvement. arXiv |
| Zhou et al., Language Agent Tree Search | Follow-up synthesis combining planning, acting, reasoning, external feedback, and self-reflection. arXiv |
| Huang et al. and Kamoi et al. on self-correction | Important counterweight: self-correction is unreliable without strong external feedback. arXiv |
| Dou et al., Re-ReST | Follow-up that uses reflection to improve self-training data for language agents. ACL Anthology |
| Zhao et al., ExpeL | Extends the experience-memory idea beyond same-task retries into cross-task experiential learning. AAAI Publications |
Companion entries
Core theory: Language Agents, Verbal Reinforcement Learning, Agent Memory, Episodic Memory, Policy Improvement, Credit Assignment, Semantic Gradients
Agent loops: ReAct, Reflexion, Self-Refine, Tree of Thoughts, Language Agent Tree Search, AdaPlanner, ExpeL, Re-ReST
Benchmarks: HumanEval, HotPotQA, ALFWorld, WebShop, MBPP, Gaia2
Engineering practice: Agent Trace Logging, Evaluator Design, Self-Generated Unit Tests, Failure Reflection Prompts, Memory Retrieval, Agent Harnesses, Test-Time Adaptation
Counterarguments and limits: LLM Self-Correction Limits, Evaluator Reliability, Memory Poisoning, Error Compounding, Long-Horizon Planning, Exploration in Language Agents