Reinforcement Learning-Rewards and punishment
How reinforcement learning actually works
Every time you skip a YouTube video, scroll past an Instagram reel, or close the app mid-scroll, something changes. Not just for that one video. For the AI model that decides what you see next.
That’s reinforcement learning (RL), and it’s a genuinely different way of training a model than the supervised learning most of us picture when we hear “AI.” There’s no labelled dataset saying “this is the correct answer.” There’s an agent, an environment, and a feedback loop that runs constantly, the model tries something, watches what happens, and adjusts.
RL runs on four ideas.
State, what the system currently knows (about you, about the market, about the moment)
Action, what it decides to do (recommend a reel, match a driver)
Reward or punishment, what happens as a result
Policy update, the system nudges its future behavior based on that signal
And then it repeats. Immediately. Every scroll, every ride request, every skip is another lap through that loop.
Instagram
When you open Instagram, content is suggested to you; from accounts, posts, or reels you may be interested in. One of the underlying AI systems decides what to show you and in what order, by predicting what you’re most likely to engage with.
That prediction leans on a mix of signals. Explicit ones, like whether you tapped “interested” or “not interested” on a reel, if you shared a reel, saved it, watched it completely. And implicit ones, who you follow, what kind of creator content you linger on, whether you rewatch something.
At a high level, the system:
Gathers inventory: pulls in a portion of the public content available on Instagram, photos and reels included.
Leverages signals: watch history, saves, shares, interested/not-interested taps, and a long tail of other inputs.
Ranks content: scores everything it has against what it knows about you.
Surfaces the top of that ranking: content predicted to hold the most value for you gets shown first.
Now, scoring every single reel against every single user directly would be absurdly expensive; millions of items, billions of users, recalculated constantly. So the system does something cleverer: it learns to represent both users and content as vectors, lists of numbers, in the same mathematical space. Finding a good recommendation becomes a matter of finding which content-vectors sit closest to your user-vector.
This is the two-tower model, and it’s the workhorse behind modern retrieval systems, Meta has publicly described using it for Instagram Explore, and versions of it are industry standard.
The user tower compresses everything the system knows about you- recent watch history, likes, device, time of day; into one vector. Think of it as a GPS coordinate for your current interests, except in 128 dimensions instead of two.
The item tower does the same for each reel: topic, creator, audio, engagement stats, age; compressed into a vector in that same space.
Here’s where the reward/punishment framing comes in. If you watch a reel all the way through, that’s a strong positive signal. The model treats it close to a reward. If you skip fast, or worse, close the app entirely, that’s a negative signal; closer to a punishment, because it risks losing your session altogether. Based on these outcomes, the system nudges its weights, this reel type worth more, that one worth less and repeats, at scale, across hundreds of millions of users, every few milliseconds.
The ranking step described above: two towers, retrieval, scoring- is mostly large-scale supervised prediction: “given this user and this content, what’s the probability of engagement?” The genuinely reinforcement-learning part shows up one layer deeper, in how platforms handle the sequential, long-term version of that problem. YouTube’s own researchers published on exactly this: they built a production recommender on REINFORCE, a policy-gradient RL algorithm, specifically to optimize for long-term satisfaction rather than the next single click, and to deliberately balance exploration (showing you something a little outside your lane, to learn if you’d like it) against exploitation (showing you more of what already works). That exploration piece is also the platform’s defense against the filter-bubble problem, if you only ever exploit, the model locks you into a narrower and narrower loop.
What should a PM actually take from this
This is the part that matters more than the mechanics, because as an AI PM, you’re rarely the one writing the RL algorithm. You’re the one deciding what the reward means.
Reward design is a product decision, not a data science detail. If “reward = watch time,” you will get watch time, and you may also get rage-bait, clickbait, and a feed that keeps people scrolling but leaves them worse off. The PM’s job is defining what “good” actually looks like long-term, not just what’s easy to measure short-term.
Exploration vs. exploitation is a tradeoff you own, not just an engineering parameter. Too much exploitation, filter bubbles and stale feeds. Too much exploration, an irrelevant, frustrating feed. Where that dial sits is a product call.
The reward horizon is a product call too. Instagram’s lesson generalizes, optimizing for the next single action (the next match, the next click) can actively hurt the metric you actually care about a few steps later. Ask what the true north star is, and over what time window.
Cold start doesn’t go away with RL. A new user has no history for the state representation to work with. Someone still has to decide the fallback, heuristics, popularity, demographic priors, until real signal accumulates.
Because the output is probabilistic, evals matter more than they do in a typical dashboard review. You can’t eyeball whether an RL system is “working” the way you’d review a feature launch. You need a way to catch reward hacking or drift before it shows up as a PR problem.
Shailesh Sharma Technomanagers
Sources:
https://transparency.meta.com/features/explaining-ranking/ig-feed-recommendations/
Really enjoyed this, Sonal. The point around rewards shaping behavior is so important, especially because in RL the challenge is not just making the system learn, but making sure it learns the right thing. This was a crisp and helpful explanation.