Serendipity in Recommender Systems: Useful Surprise, Not Random Noise

Abstract. Serendipity in recommendation means helping a person discover something unexpected and useful. It is not the same as novelty, diversity, randomness, or relevance, though it draws on all of them. A system optimized only for predicted preference can become repetitive and self-reinforcing. A system optimized only for surprise becomes noise. For learning and news products, the design problem is to create qualified departures from the user’s current profile and then measure whether those departures broadened understanding.

Coverage note: sources checked through August 2026.

1. The concept

A recommendation can be:

relevant but obvious;
novel but irrelevant;
diverse but unhelpful;
surprising but unpleasant;
unexpected and valuable.

Only the last category is clearly serendipitous.

The recommender-systems literature does not have one universally accepted formula, but reviews converge on a family of components: unexpectedness, novelty, relevance or usefulness, and often diversity. 1, 2

This yields a practical definition:

A serendipitous recommendation is one the user was unlikely to find or expect, but later judges worthwhile.

The “later judges” clause matters. Surprise can be predicted from behavior; usefulness is partly revealed by what the item enables.

2. What the word originally meant, and why it constrains the design

The recommender literature inherited "serendipity" from somewhere specific, and the original sense contains a constraint that the engineering definition tends to drop.

The word is Horace Walpole's, coined in 1754 from a Persian tale whose princes were "always making discoveries, by accidents and sagacity, of things they were not in quest of." Both nouns in that phrase are load-bearing. Accident supplies the encounter; sagacity supplies the recognition. A recommender can manufacture the first and cannot manufacture the second.

Robert Merton gave the idea its analytical form for empirical research, and his three conditions transfer almost directly to a feed. The serendipity pattern, in his account, is the fairly common experience of observing an unanticipated, anomalous, and strategic datum that becomes the occasion for developing a new theory or extending an existing one. Each term does work:

Unanticipated — the observation arrives as a by-product of research aimed at something else, and bears on theories that were not under test when the work began.
Anomalous — it is surprising, because it seems inconsistent with prevailing theory or with other established facts, and that seeming inconsistency provokes curiosity.
Strategic — it permits implications for a broader theory. Here Merton is explicit that this third property is not a property of the datum: it refers "rather to what the observer brings to the datum than to the datum itself," because detecting the universal in the particular requires a theoretically sensitized observer. 11

That last clause is the design constraint. Strategic value is jointly produced by the item and the reader's preparation, so serendipity is not a property a ranker can compute over a catalogue. Two consequences follow that the component list in section 1 does not capture on its own:

The same item is serendipitous for one reader and noise for another, and the difference is often not taste but context. A reader midway through a problem can use an anomalous result; the same reader last month could not. A measure that scores items rather than item-reader-moment triples is measuring something else.
Preparation can be supported, not merely exploited. If recognition requires a prepared observer, then explanation, framing, and surrounding context are part of the serendipity mechanism rather than presentation polish on top of it. The explanation lever in section 5 is doing causal work, not decorating.

This is also the honest reason serendipity resists the metric treatment in section 8. Merton's third condition is unobservable at ranking time by construction.

3. Why accuracy is not enough

Traditional recommender evaluation often asks whether the system ranks items the user clicks, rates, or consumes. Accuracy matters. A feed full of irrelevant material is not redeemed by diversity.

The field said this about itself early. McNee, Riedl, and Konstan's "Being accurate is not enough" argued in 2006 that accuracy metrics had actively hurt recommender research — that lists optimized for predicted rating converge on similar, obvious items users have already seen, and that usefulness is a property of the list-in-context, not of per-item prediction error. 6 Serendipity research is in large part the program of taking that critique seriously.

But past behavior is an incomplete target:

it reflects what the old system already exposed;
clicks mix curiosity, outrage, and genuine value;
interests change;
new items lack interaction history;
people cannot click a field they have never encountered;
a learning product sometimes should recommend prerequisites or challenges rather than favorites.

An accuracy-only system can form a feedback loop:

show familiar items
      ↓
user interacts with familiar items
      ↓
model gains more evidence for familiar items
      ↓
show even more familiar items

This is one source of over-specialization. A systematic review of filter-bubble research in recommender systems answers its own title question in the affirmative — it finds evidence that filter bubbles do occur, driven by algorithmic bias and by cognitive bias together, since confirmation bias and its relatives taint the interaction data the ranker then learns from — and recommends diversity- and exposure-oriented interventions as the countermeasure, 4, which is why diversity and exposure remain real design choices in news and recommendation. 3

The news-consumption literature complicates that picture, and this article's own title term comes from it. Fletcher and Nielsen's "automated serendipity" study compared, across four countries, the news repertoires of people who reach news through search engines with those who do not, and found the search-engine users drew on more sources and were more likely to use both left- and right-leaning outlets — both in all four countries. The third finding, more balanced repertoires, is the one to state precisely: the paper's own regression result is that “apart from in the US, using search engines to search for news topics is in fact positively associated with having a smaller gap between the number of left-leaning and right-leaning online news sources” — three of the four, with the US the exception, even though the abstract and conclusion generalise it to all four. Little support for the filter-bubble idea in that domain, then, and some evidence against it, with one country's balance result not carrying. 16 Filter bubbles are a demonstrated risk inside a recommender; whether they dominate a person's news diet depends on the access path, which is the same scoping section 12 applies to the Spotify numbers.

4. Serendipity versus nearby objectives

Objective	Question	Can be high without serendipity?
Relevance	Does it fit known interests or needs?	Yes—an obvious item
Novelty	Has the user seen it before?	Yes—a new but irrelevant item
Diversity	Are items different from each other?	Yes—a varied list of poor matches
Unexpectedness	Does it depart from predictions?	Yes—pure surprise
Coverage	Does the system expose the catalog broadly?	Yes—without helping one user
Serendipity	Was the unexpected item useful?	No—the usefulness is essential

The distinction affects implementation. Randomly inject items and novelty rises. Spread items across topics and list diversity rises. Neither guarantees a fortunate discovery.

5. Where serendipity can enter the pipeline

Candidate generation

Retrieve not only nearest neighbours but also items connected through a bridge concept, complementary skill, contrasting viewpoint, or shared method.

Re-ranking

Reserve part of the list for items that are relevant under a broader model than short-term click prediction.

Exploration

Use a controlled exploration policy to learn whether the user has latent interests. This resembles a contextual bandit — the formulation Li et al. made standard for news recommendation, where the system trades off exploiting known preferences against exploring to learn new ones, evaluated offline against logged data 7 — but the reward should not be only a click.

Editorial injection

Human editors can designate high-value items that deserve broad exposure. This is especially useful when interaction data are sparse or biased.

Explanations

Tell the user why the departure may be useful:

Outside your usual technical stream, but it explains the policy constraint behind two tools you follow.

An explanation turns apparent randomness into a testable invitation.

6. Serendipity for learning

In a learning system, personalization and curriculum pull in different directions.

The learner’s current interests are valuable entry points. They are not a complete map of what the learner may need or come to value.

A serendipitous learning item can:

connect two subjects through a shared concept;
introduce a prerequisite the learner did not know to ask for;
provide a counterargument to a familiar view;
reveal an application in another domain;
interrupt a narrow sequence with a broader orientation.

The best unit may be a bridge, not a random topic:

current interest → shared concept → adjacent field

For example:

AI model routing → exploration/exploitation → recommender systems
agent memory     → episodic/semantic split → cognitive architectures
benchmark drift → delayed test sets       → editorial claim review

The bridge gives the recommendation a reason.

7. Serendipity for news and current information

News recommenders face special constraints:

stories expire quickly;
many outlets repeat the same event;
importance and personal relevance differ;
source diversity matters;
recommendations can shape public attention;
breaking reports are uncertain.

A personalized AI-news feed can separate three lanes:

Core: important developments everyone in scope should see.
Personal: items closely matched to declared interests.
Serendipity: a small number of explained discoveries.

This prevents personalization from deciding the entire agenda. It also allows the system to measure each lane separately.

The lane structure has a normative argument behind it, not just a product one. Helberger's analysis of news recommenders distinguishes the democratic roles a recommender can serve — liberal (serve individual preferences), participatory (map the diversity of ideas and opinions in society, and meet readers' differing information needs and styles), deliberative (expose diverse and challenging viewpoints, and re-create common spaces in a fragmented environment), with a fourth, critical recommender sketched rather than developed (push readers toward marginalized voices) — and shows they imply different recommender designs rather than one neutral algorithm. 8 The core lane is the participatory commitment, the serendipity lane a small deliberative one; a feed that is all personal lane has silently chosen the liberal model without saying so.

8. Measuring serendipity

Offline metrics often estimate unexpectedness relative to a baseline recommender and multiply or combine it with relevance. The formal treatments make the baseline-dependence explicit: Adamopoulos and Tuzhilin define unexpectedness as departure from a user-specific set of expected items and combine it with quality, precisely so that "unexpected" is measured against what this user would have predicted rather than against global popularity. 9 The result depends heavily on the baseline: unexpected relative to popularity is different from unexpected relative to a personalized model.

User evaluation is therefore important. Useful questions include:

Was this item unexpected?
Was it worthwhile?
Would you likely have found it without the recommendation?
Did it change what you explored next?
Do you want more items connected in this way?

Longer-term measures are better than immediate clicks:

second-item exploration in the new topic;
saved or revisited items;
changes in explicit interests;
knowledge gain;
breadth without loss of completion;
regret or “why was this shown?” dismissals.

An industrial 2025 paper reported that an LLM-assisted serendipity framework improved exposure and interaction with designated serendipitous items in an online commerce setting. The system used profile generation, alignment to human serendipity judgments, and production constraints. The result is evidence that serendipity can be operationalized at scale, not proof that its metric captures educational or civic value. 5

9. Cold start and sparse feedback

Serendipity is hard to estimate for a new user because unexpectedness is relative to an expectation.

A cold-start system can rely on:

explicit interests;
a chosen learning stream;
current page or session;
broadly valuable editorial selections;
item-to-item concept bridges;
a small amount of declared negative feedback.

It should not manufacture a detailed profile from a few clicks.

For an anonymous learning product, serendipity can be session-relative:

This item is outside the stream you chose but connects through a concept in the page you just completed.

That is enough to offer a useful departure without a durable personal model.

Sparse feedback also argues for categorical reasons rather than implicit behavior alone. “Too advanced” and “not interested” have different implications for the next recommendation.

10. Multi-objective ranking

A simple re-ranker can score several qualities:

score =
  relevance
  + editorial importance
  + learning value
  + diversity contribution
  + serendipity contribution
  - quality risk
  - repetition
  - difficulty mismatch

This is not a universal formula, and the serendipity term is an estimate standing in for something section 2 says a ranker cannot compute over a catalogue: whether a reader was prepared for this item. The sketch shows that the estimate belongs inside a constrained objective rather than bolted on after accuracy ranking — not that the underlying quantity has been made observable.

Hard constraints can protect the surface:

every item clears a source-quality bar;
the common core is not displaced;
at most one distant item appears in a short digest;
no sensitive inferred trait is used;
an explanation bridge exists;
recent dismissals are respected without permanently suppressing a field.

Evaluation should report each component. One aggregate score can hide a system that increases “serendipity” by lowering relevance.

11. Editorial serendipity

Editors can create discovery without personal surveillance.

Examples include:

a rotating “concept crossing” card;
one work from a different evidence type;
a historical precursor to a current release;
a critique beside a dominant view;
a tool or project connected to an abstract idea;
a reader-selectable “surprise me, but explain why.”

Editorial serendipity is common to everyone or to a chosen stream. It provides a baseline against which algorithmic personalization should be compared.

If a model-driven recommender does not outperform a thoughtful rotating slot on useful discovery, the added profile and complexity have not earned their place.

12. What happens on a real platform when recommendations change

Most of the argument above is normative or offline. Two large-scale studies on the same platform supply harder evidence, and — usefully — they differ in shape while agreeing on the mechanism. Neither is a study of a system deliberately diversifying; both are studies of what personalization itself does to diversity. That is a different question from the one section 8's industrial result answers — a framework built to raise serendipitous exposure, evaluated online against baselines — and the two are worth keeping apart: one measures what a deliberate intervention buys, these two measure what the default does. On the default, they are the largest evidence available.

Anderson and colleagues studied consumption diversity on Spotify observationally, using an embedding of millions of songs derived from listening behavior to quantify how musically diverse each user is. Two findings matter here. High consumption diversity is strongly associated with important long-term metrics including conversion and retention — diversity is not simply a tax paid for social benefit; it moves with outcomes the business already cares about. And algorithmically-driven listening through recommendations is associated with reduced consumption diversity. Those two findings are observational and do not settle direction of causation; what they establish is that the two quantities are related in a way that makes "diversity versus engagement" too simple a story. The same paper's randomized arm points at the mechanism rather than the causation question: across 540,000 free-tier users for one week, ranking by relevance rather than popularity raised streams by about 10% for generalist listeners and about 26% for specialists — recommendations, in the authors' reading, are most effective precisely for the users whose listening is already narrowest. 12 That is the paper's central tension, and it is the one this article is about — though the sentence it invites has to be said carefully. The experiment shows relevance ranking helps specialists more; the observational analysis shows algorithmic listening travels with narrower consumption. Neither shows that the ranker narrowed those specialists. "The system works best on the people it narrows most" is the natural reading of the two together, and it is an inference, not a result: establishing it would take an experiment that measures diversity as an outcome rather than streams.

Holtz and colleagues ran the study built end-to-end around the diversity question. Both arms received podcast recommendations aimed solely at increasing podcast consumption; the treatment arm's recommendations were personalized on music-listening history, while the control arm received the most popular podcasts among users in their demographic group. Personalization increased the average number of podcast streams per user — and decreased individual-level diversity while increasing aggregate diversity across users. 13 The authors name this an engagement-diversity trade-off, and the two-directional result is the part worth carrying: a personalized system can make each user narrower while making the population broader. A platform reporting catalogue coverage as evidence of diversity may be reporting exactly the pattern that narrowed its individual users.

Four things this changes about the design in section 18:

Specify which diversity. Individual-level and aggregate diversity moved in opposite directions in a controlled experiment. A diversity target that does not say which one is unfalsifiable.
The engagement trade-off is real but not total. The randomized study found personalization raising consumption and narrowing individuals; the observational study found diverse users converting and retaining better. Both can hold if costs and benefits land on different time horizons, which argues for measuring serendipity interventions over months rather than sessions.
The control arm matters as much as the treatment. "Most popular within demographic group" is itself a policy with its own diversity profile. Comparing personalization against it answers a narrower question than comparing against chronological or editorial ordering.
Effects are platform- and domain-specific. Both studies concern music and podcasts on one service. Nothing here licenses a numeric expectation for news or learning content.

13. Exploration has a price, and it can be quoted

Section 5 lists exploration as a place serendipity can enter the pipeline. The bandit framing behind that lever is worth stating fully, because it is the one part of this subject with an explicit cost accounting rather than a vocabulary.

A contextual bandit chooses among items given context, observes reward only for the item it chose, and must balance exploiting what it believes against exploring to learn more. 7 Framed this way, a serendipitous recommendation is a probe: an action taken partly for its information value rather than its expected immediate reward. That reframing is what makes the cost nameable — the regret from a probe is the gap between the expected reward of the optimal action and that of the action taken. Two things follow that the framing is often used to skip. The benchmark is the optimal action, not the best one the system currently believes in, so a system's own confidence does not bound its regret. And the quantity is counterfactual: the reward of the action not taken is never observed, so regret is estimated rather than measured, which is exactly the difficulty the next paragraph is about.

The practical obstacle is that exploration is expensive to evaluate honestly, because logged data records rewards only for actions the deployed system happened to take. Li and colleagues addressed this with a replay methodology for offline evaluation of contextual-bandit algorithms that is fully data-driven rather than simulator-based, and provably unbiased, with empirical results on a large news-recommendation dataset from the Yahoo! front page. 14 The reason to know this exists is defensive: a serendipity feature evaluated against a simulator inherits the simulator's model of the user, and the modeling bias introduced there runs in the direction the designer already believes.

Three design implications:

Budget exploration explicitly. "Ten percent of slots are probes" is a stated price. "The ranker sometimes surfaces surprising things" is not, and cannot be tuned or defended.
Log the probe as a probe. An exploratory impression indistinguishable from an exploitative one in the logs makes every later analysis of serendipity's effect impossible.
Randomized logging is an asset with a shelf life. Unbiased offline evaluation depends on randomization being present in the log. A system that never explores cannot later evaluate whether it should have.

14. Serendipity as a process, not an event

Section 8 already reaches past the click, asking for second-item exploration and for saved or revisited items. Research on how people actually experience serendipity in their work goes further still: the encounter is one stage of several, and the later ones, where a find is acted on or quietly is not, are where the value is realized or lost, and are the ones a recommender sees nothing of.

McCay-Peet and Toms interviewed twelve professionals and academics in depth about their own work-related serendipitous experiences, and consolidated prior models into a single process model built on four elements plus one that runs through them: Trigger, Connection, Follow-up, and Valuable Outcome, with an Unexpected Thread running through one or more of the first four. Two further components matter for anything trying to instrument this. A Delay may sit inside Connection — the interval in which a person has met the trigger but has not yet recognised what it connects to — which is why a metric that closes the episode at the end of the session can miss it entirely. And Perception of Serendipity is a separate construct in the model, fed by awareness of the other elements rather than identical to them: the episode can have every structural part and still not be experienced as serendipitous. The study also identifies factors of the individual and their environment that facilitate each element. 15

Mapped onto a feed, the split is unflattering:

Element	What it is	What a recommender does about it
Trigger	The unexpected encounter	Almost all of the engineering effort
Connection	Linking it to something the person already holds	Nothing, usually
Follow-up	Acting on it — reading further, saving, pursuing	A click, then the session ends
Valuable outcome	Something useful actually results	Not observed at all

Two consequences worth acting on. First, the design levers in section 5 that look like polish — explanation, framing, adjacency to what the reader was already doing — are interventions on the Connection stage, which is the stage a bare unexpectedness score cannot reach. Second, follow-up is the earliest stage a system can honestly instrument: saves, returns to an item days later, and onward navigation are observable, whereas the valuable outcome usually is not. That makes the metric caution in section 8 concrete rather than merely epistemic — a system that measures only triggers is measuring the stage it already spends its effort on, and measuring it with the instrument least able to catch its own failures. Section 16's first two failure modes, randomness disguised as discovery and manufactured surprise, are both trigger-stage failures, and neither shows up in a trigger count.

15. User controls

A single global setting may be enough:

focused;
balanced;
exploratory.

More advanced controls could choose:

adjacent versus distant discoveries;
technical versus social bridges;
counterpoints;
unfamiliar sources;
topic reset.

The user should also be able to explain a dismissal:

not relevant;
already knew this;
too far afield;
poor source;
wrong level;
not now.

These reasons improve the model more than a generic dislike and preserve the distinction between topic, timing, source, and difficulty.

16. Failure modes

Randomness disguised as discovery

The system injects low-probability items and calls the result serendipity.

Manufactured surprise

The system under-models the user, then claims obvious recommendations are unexpected.

Click-based usefulness

Curiosity clicks count as value even when the item disappoints.

Stereotype escape

The system uses sensitive traits to decide what would be “surprising,” reinforcing a profile it should not have built.

Forced balance

Every topic receives symmetric treatment even when evidence quality is asymmetric.

Exploration debt

Too much exploration makes the product feel unreliable and increases cognitive load.

Hidden agenda

Sponsored or institutionally preferred material is presented as serendipitous discovery. Editorial and commercial interventions must be labeled.

Feedback-loop blindness

The evaluation dataset was produced by the previous recommender, so offline “relevance” rewards the old exposure pattern. Simulation work names this algorithmic confounding: training on interaction data shaped by the deployed recommender homogenizes user behavior and degrades utility over successive generations, while looking fine on the confounded metrics. 10 Serendipity evaluation is the most exposed to this failure, because the behavior it wants to measure is exactly the behavior the old exposure pattern suppressed.

17. When serendipity should yield

Serendipity is not appropriate in every slot.

It should usually yield when:

the user is completing a safety-critical procedure;
a prerequisite sequence must remain ordered;
the reader explicitly chose a focused, time-limited mode;
the source quality is uncertain;
an accessibility or difficulty constraint would make the item unusable;
the common core already exceeds the available attention budget;
the system cannot explain the bridge;
an exploration choice would expose sensitive inferred information.

A product can move discovery to a boundary: after a task, between modules, or in a separate “explore” lane. This preserves user intent while keeping a route beyond the immediate goal.

Serendipity should also yield to correction. If a user says an item is already familiar, the system should not reinterpret the click as successful surprise. If the user says “not now,” that is not necessarily a permanent topic rejection.

18. A simple design

For a reading stream:

Ask the user to choose broad entry points explicitly.
Build a high-quality core set independent of personalization.
Rank a personal lane using declared interests and local progress.
Generate serendipity candidates through concept bridges.
Apply quality, source, and difficulty filters.
Reserve a bounded share—perhaps one card, not half the feed—and state which diversity it is meant to move: the individual reader's, or the population's across readers.
Explain the bridge.
Gather explicit unexpectedness and usefulness feedback.
Measure later exploration and return over months, not sessions, against a non-personalized control edition.
Keep a common edition available.

What sections 12's studies supply is the shape of the effect behind steps 6 and 9 — that personalization can narrow the individual while broadening the population, and that the difference shows up over long horizons rather than in a session. They supply neither number: "perhaps one card" is not a dosage either paper measured, and the months-long horizon is longer than Anderson's one-week experiment. Both are parameters to test in the domain at hand.

The percentage is a product parameter to test, not a universal constant.

19. Bottom line

Serendipity is the part of recommendation that respects the possibility that a person’s future interests are larger than their recorded past.

It should not replace relevance, editorial judgment, or curriculum. It should create a small, legible route beyond them. The product succeeds when the reader says not merely “I did not expect that,” but “I am glad I found it.”

Recommendation: Recommender Systems, Information Overload and AI Curation, Exploration–Exploitation Tradeoff, Diversity in Recommender Systems

Learning: Personalized AI Learning Systems, Learner Modeling for Adaptive AI, Need for Cognition

Risks: Filter Bubbles, Feedback Loops, Algorithm Aversion, Automation Bias