Every time you open the For You feed, X has to pick ~50 posts out of hundreds of millions. It does this in two stages — cheaply narrow the field, then expensively rank what's left — all driven by a Grok-based transformer that learns what you like from your engagement history. This page walks through the whole machine, one box at a time.
You can't run a heavyweight ML model on hundreds of millions of posts per request — that would be far too slow. So the system splits the job in two: a fast, approximate retrieval stage that finds a few hundred promising posts, followed by a slower, accurate ranking stage that orders them precisely.
home-mixer/ is the Rust conductor that orchestrates everything;
phoenix/ holds the JAX/Haiku ML models (the “brain”);
thunder/ is an in-memory store of recent posts from accounts you follow.
Before anything is ranked, the feed gathers a pool of candidate posts from two very different sources.
Recent posts from accounts you already follow. Thunder keeps these in memory — it consumes post create/delete events off Kafka and keeps per-user stores trimmed to a recent retention window.
Posts from people you don't follow, surfaced by an ML model that searches a global corpus for content similar to what you've engaged with.
Both pools are merged into one list and flow into the same ranking, filtering, and scoring steps from here on.
How do you find relevant posts among hundreds of millions fast? You turn both the user and every post into vectors in the same space, then the most relevant posts are simply the ones whose vector points in the same direction as yours.
[D][N, D]top‑K highest scores → your retrieved posts.
Source: phoenix/recsys_retrieval_model.py — see
build_user_representation, CandidateTower, and
_retrieve_top_k.
Retrieval gave us a few hundred candidates but only a rough similarity score. Now a bigger Grok-based transformer looks at each candidate carefully and predicts how likely you are to take each action on it.
The model packs three things into a single token sequence and runs the transformer over all of it at once:
[1] token[S] tokens[C] tokensEach token isn't a word — it's a post (or author, or action) turned into a vector via hash-based embeddings. The transformer reads your history to understand your taste, then evaluates each candidate against it.
Rather than one vague “relevance” number, the model's final layer predicts a separate probability for every kind of engagement — run through a sigmoid into 0–1 probabilities:
Note the red ones — the model also predicts how likely you are to dislike a post. Section 6 shows how those pull a score down.
Source: phoenix/recsys_model.py (PhoenixModel) and
phoenix/runners.py.
Here's the most important design detail in the ranker. Inside the transformer, attention controls which tokens can “look at” which. The ranker uses a special mask so that candidates can see you and your history, but never each other.
Each row is a token doing the looking (a query); each column is a token being looked at (a key). Blue = allowed, gray = blocked, gold = a candidate attending to itself.
A post's score depends only on you, never on which other posts happen to share its batch. That makes scores consistent (the same post always scores the same) and cacheable, and it lets candidates be scored in parallel without interfering.
Source: phoenix/grok.py → make_recsys_attn_mask.
The model handed us ~15 probabilities per post. To sort the feed we need a single number. The Weighted Scorer multiplies each probability by a tunable weight and adds them up.
Positive actions get positive weights and push a post up; negative actions get negative weights and push it down. Illustrative weights:
After the weighted score, two more scorers adjust it:
The 2nd, 3rd, … post from the same author gets multiplied by a decaying factor, so no single account can flood your feed.
mult(pos) = (1−floor) × decaypos + floor
Out-of-network posts get multiplied by a tunable factor — the dial that balances familiar in-network content against fresh discovery.
Source: home-mixer/scorers/ — weighted_scorer.rs,
author_diversity_scorer.rs, oon_scorer.rs.
Home Mixer runs everything above as an ordered series of stages. Here is the whole journey of a feed request:
Fetch context about you: recent engagement history, who you follow, muted keywords, topics, served history.
Pull posts from Thunder (in-network) and Phoenix Retrieval (out-of-network), plus ads, who-to-follow, etc.
Enrich each candidate with post text, media, author info, video duration, subscription status.
Drop posts that are duplicates, too old, your own, from blocked/muted authors, contain muted keywords, or were already seen.
Phoenix Scorer (ML predictions) → Weighted Scorer (combine) → Author Diversity Scorer → OON Scorer.
Sort by final score, keep the top K candidates.
Final visibility checks (deleted / spam / violence / gore) and conversation-thread dedup.
Cache request info for next time, then return the ranked feed over gRPC.
candidate-pipeline/ crate is a reusable framework defining the
Source, Hydrator, Filter,
Scorer, Selector, and SideEffect traits.
Four ideas that shape the whole system.
There are no manual “this post is good because X” rules. The transformer learns relevance entirely from your raw engagement sequence.
The attention mask keeps each candidate's score independent of its batch — consistent, cacheable, parallelizable. (Section 5.)
Users, posts and authors map through several hash functions into fixed-size embedding tables. Bounds memory and gracefully handles brand-new IDs.
Predicting many actions instead of one score means the feed can be re-tuned by changing weights — no model retrain required.
Most feeds treat recency as a soft preference — older content can still surface if the relevance signal outweighs the age penalty. X treats it as structural. Four independent layers of the system each erase your post from consideration as it ages, and they stack.
AgeFilter's threshold aren't published, so those bars
are illustrative.
The demo corpus shipped with the open release is ~537K posts from a 6-hour slice. Production is larger but the same shape: indexing every post ever made into a hot vector index is infeasible, so the out-of-network discovery path only searches a recent window of the firehose. Once your post falls out, strangers can't be shown it through retrieval at all.
Thunder — the in-network store — explicitly trims posts older than a retention period. After the cutoff your post isn't down-weighted for your followers; it's just gone from the in-network candidate pool. No database fallback, no slow path. It just stops being a candidate.
recsys_model.py defines
POST_AGE_MAX_MINUTES = 4800 — that's 80 hours, ~3.3
days. The model bins post age into 1-hour buckets up to that ceiling
and learns an embedding per bucket. Because training data is dominated
by engagement that happens in the first few hours after posting, the
model absorbs “old = uninteresting” as a learned prior.
Nobody told it recency matters; it figured that out from the engagement
distribution and now imposes it on every future post.
AgeFilter is a hard cutoffBefore scoring even runs, candidates older than a threshold are dropped outright. By the time gates 1–3 have done their work this one is usually redundant — but it makes sure nothing slips through.
Engagement optimization isn't a quirk of X — it's the genre. TikTok, Reels, YouTube Shorts, LinkedIn all do roughly the same thing for the same reason: clicks, dwell, replies are what you can measure cheaply at scale, so they become what you optimize. The blind spots that creates are common to all of them.
Because the model trains on engagement it has already shaped, it can't separate “you engaged because the post was good” from “you engaged because the algorithm pushed it onto you.” The two look identical in the training data. So the system spirals toward whatever it happened to start showing:
The arc on top is where the damage is done. Whatever the model shows you tomorrow is conditioned on what it showed you yesterday, and the labels can't disentangle the two. Dwell-time makes this worse: rage-stopping, curiosity, and confusion all generate dwell, and the model rewards them the same as genuine interest.