Drafting by numbers

Every August, right before the new English Premier League (EPL) season begins, fantasy football enthusiasts do the same thing: draft a squad based on vibes, last season’s highlight reels, and whatever default ranking the fantasy football mobile app states. I wanted to see how far I could get by actually modeling it — scraping every recorded on-pitch action from three seasons of top flight football across several European competitions, building a statistical model to predict each player’s match performance and match appearances, and using it to draft a real team in a real league.

This post is the technical walkthrough: what the pipeline does end to end, where it broke, and how I found out. The code is public at epl-fantasy-draft-stats-tool if you want to run it yourself.

The problem, concretely

Sleeper is an app that lets you and your friends run your own competition built entirely around real English Premier League (EPL) players. Before the season starts, each manager drafts a squad — and because each real player can only belong to one fantasy team, the draft order and who you pick matter. From there, a fixed scoring formula converts each player’s actual on-pitch performance into points every match: goals, assists, shots on target, key passes, tackles, clean sheets, and more each have a set point value, all driven entirely by what the players do in real EPL matches.

So the modeling question is well-defined: for every draftable EPL player, predict their expected match performance — their output across every stat Sleeper’s scoring system rewards — and predict how many matches they’ll actually appear in, since an injury-prone or rotation-risk player scores zero on games he doesn’t play. Convert both into points using Sleeper’s real formula, and rank every player by the result. That ranked list is what you draft from on the day.

Getting the data

There’s no clean API for this. I used soccerdata’s WhoScored scraper to pull full event streams — every pass, shot, tackle, card, substitution — for every match across 8 leagues (Premier League plus 7 feeder leagues players transfer in from) and 3 seasons. That’s roughly 8,800 matches and, once flattened, millions of individual event rows.

Two engineering details mattered more than expected:

• event_id is only unique within a single match, per team — not globally. Two different teams in the same match commonly reuse the same event_id for unrelated events. Any join has to key on the composite (game_id, team_id, event_id), not event_id alone. Missing this silently corrupts joins rather than erroring.
• The field that looks like it tells you who got the assist can’t be trusted. Each goal event includes a reference to a player who appears to be the assist provider, but that reference is sometimes just wrong. The only reliable way to find the real assist is to look at the actual pass immediately before the goal and check whether that specific pass was tagged as a key pass.

Both of these are the kind of bug that doesn’t throw an exception — it just quietly gives you a wrong number that looks plausible. They only surfaced when I spot-checked specific players’ known stat lines against reality — comparing what my pipeline generated for a player against their actual stats in the Sleeper app — which turned out to be the single most useful debugging technique in this whole project.

Shot quality: building an xG model from scratch

Not every shot is equally likely to become a goal — a tap-in from six yards and a speculative strike from 30 shouldn’t count the same. That’s the idea behind “expected goals” (xG): a probability, from 0 to 1, that a given shot results in a goal, based on the conditions it was taken under. (“Expected assists,” xA, is the same idea applied to the pass that created the chance.) Before predicting anything about a player, I built my own xG model: a logistic regression trained on 222,000 shots pooled across all 8 leagues, using pre-shot geometry (distance, angle), body part, situation (open play, corner, penalty, direct free kick), and whether the chance was tagged a “big chance.” I scored it with AUC — a standard 0-to-1 measure of how well a model tells goals and misses apart, where 0.5 is a coin flip and 1.0 is perfect — and got 0.805, respectable for a from-scratch model built on event-stream data rather than a commercial provider’s tracking data.

AUC only says the model separates goals from misses — it doesn’t say the probabilities themselves are trustworthy. Calibration checks the second claim: of every shot the model called “20% likely,” did roughly 20% of them actually go in?

The output — xg/xa per shot — feeds into the model as an input feature, not a scoring category (Sleeper scores real goals and assists, not expected ones). xg_exp, a player’s rolling expected-goals rate, carries more weight in the goals model than any other input — genuinely used, not just present. Yet accuracy barely moved (R², which measures how much of the real variation in goals the model can explain, from 0 to 1, went 0.077 → 0.079). Reason: goal-scoring is mostly finishing luck no pre-shot feature can predict, and xG overlaps with shot-volume history the model already had. A feature can matter and still not move the needle.

Features and leakage discipline

Every player-match gets ~150 engineered features. That number isn’t for show: predicting 16 different stats means capturing many different kinds of signal — recent form, opponent tendencies, team style, injury recency, fixture congestion, set-piece role, and more. Each feature is one hypothesis about what drives a specific stat, kept only if it actually holds up in testing. The one rule governing all of them: a feature can only see matches before the one it’s predicting. Break that rule and you get “data leakage” — the model accidentally sees information from the future, which makes it look far more accurate in testing than it will ever be on a real, unplayed match.

That sounds obvious, but it’s easy to get subtly wrong. Every rolling feature here is built to shift one match backward before any average is calculated, so a player’s “form” going into a match never includes that match itself.

Late in the project I found a leak one level removed from the main model. The main model is properly tested on a season it never saw during training. But one of its inputs, xG, comes from a separate, earlier model — and that earlier model had been trained on every season, including the one used to test the main model. So the test season’s information was leaking in indirectly, through that one feature, even though the main model itself was set up correctly. The fix was to build a second, throwaway version of the xG model, trained only on the older seasons, used only for testing. The numbers barely moved once fixed — but that’s not the point. The point is that I now have a repeatable check for this specific kind of indirect leak, rather than assuming the setup was correct because the results looked plausible.

Training: 16 models, not one

Rather than one model predicting fantasy points directly, the pipeline trains 16 separate models — one per stat: 13 for outfielders (goals, assists, tackles, and so on), 3 for goalkeepers (saves, high claims, smothers). Each one predicts a player’s expected count for that single stat in a typical match. Those predictions get converted into Sleeper points using the real scoring formula, then multiplied by a separately-estimated number of expected appearances.

All 16 are ridge regression models — linear regression with a built-in penalty that stops it over-trusting any single feature, which keeps predictions stable on noisy, real-world data. Two reasons I chose this over one big model or a fancier algorithm, both tested rather than assumed:

1. Interpretability. A per-stat linear model lets me see exactly which features drive, say, tackles won vs. goals, and sanity-check that against real football knowledge.

   Every feature is standardized before the ridge fit, so its coefficient is directly comparable to every other feature’s in that same model — “1 standard deviation of this feature moves the prediction by this many goals/tackles/etc.” Two labels recur below: L5/L10 means a rolling average over the player’s last 5 or 10 matches, and × mins means that rate multiplied by the player’s average minutes played — turning a per-90 rate into an expected per-match count, since ridge is additive and can’t form that product from the rate and minutes features on its own. Here’s what actually drives each of two very different stats, ranked by the size of that effect:

   Two of these bars are partial effects, not marginal ones, and worth reading carefully: Career avg minutes played and Tackle rate (L10, raw) both correlate positively with actual tackles won on their own (r=0.17 and r=0.18), but go negative here once the model already has 3–4 near-duplicate rolling/interaction features in play — minutes_played_exp correlates up to 0.80 with other minutes features already in the model, and tackles_won_roll10 correlates 0.91 with tackles_won_exp alone. Ridge is splitting shared variance across correlated inputs, not claiming that more career minutes causes fewer tackles.

2. It beat the alternatives. I tested two tree-based methods (gradient boosting and random forest) against the same ridge model, same features, same test data. Both lost on accuracy. Two different algorithms losing to the same simple model is a stronger result than either loss alone — I stopped looking for a better algorithm after that.

To know whether any of this actually works, every model is tested on a season it never trained on — never a random shuffle, since that would let it peek at the future. The check that matters most: take the model’s pre-season ranking of every player and compare it to a ranking built the same way but using each player’s real, already-known results from that season. If the two lists agree at the top, the model is doing its job — the whole point of ranking players is knowing who to pick first.

This check caught a real mistake. For most of the project I’d only been comparing against a much simpler ranking — season total points alone — instead of the composite ranking I actually use to make picks. I’d chosen that composite deliberately, on intuition: no single number tells you everything worth knowing about a player, so it weighs a few different angles on him at once — consistency, quality per match, and season total — rather than reducing him to one stat. Once I checked the model against that real composite instead, accuracy among the top 20 players jumped from 45% to 65%. The model hadn’t gotten better — I’d simply been checking its work against the wrong answer the whole time.

Where the Model Falls Short

To check whether the trained models were actually worth their complexity, I compared all 16 of them against simple, non-modeled baselines — a league-wide average, a player’s own career average, his form over his last 5 matches, and just his last match — on the same held-out season. I checked two things: how close each prediction lands to the real number, and whether it puts players in the right order relative to each other, since that’s what actually matters for drafting. Result: 14 of the 16 stats comfortably beat every baseline on both counts — that’s the large majority, and it’s why the model is worth using at all. But 2 didn’t, both goalkeeper stats: for smothers, the trained model loses to a plain career average on both counts — it’s less accurate and gets more players out of order. For high_claims, it’s more accurate than the baseline, but still gets more players out of order. That’s documented plainly in the project’s README under “Known Limitations” — not something I’d rather people didn’t notice.

One more limitation, documented rather than hidden, and worth actually explaining rather than waving at: ridge regression doesn’t just fit a pattern to the data — it fits a pattern, then deliberately shrinks it back toward “just predict the average,” by an amount controlled by a single knob. Turn that knob down and the model trusts whatever pattern it found in full; turn it up and the model barely trusts the pattern at all, and its prediction collapses toward each player’s plain historical average. You don’t set that knob by hand — you try a wide range of settings (12 values, 0.01 to 100,000) and keep whichever one predicts best on matches the model didn’t train on. Most of the 16 stats settle in the 500–1,000 range. Three don’t: goals (5,000), assists (10,000), and penalties drawn (50,000) all land 5–50x higher than the rest, meaning they lean far more heavily toward “just predict the average” than any other stat in the model. That’s not me under-tuning it — it’s the data itself showing that, for these three stats, leaning on recent form, opponent, or anything else actually made predictions worse than just using a player’s long-run average. The likely reason: whether a player scores in one specific match comes down heavily to finishing luck that no feature can see coming, so the safest bet really is close to his average — and this is how the model discovered that on its own, rather than me assuming it going in. And player positions come from the raw match data, which occasionally disagrees with the position Sleeper itself lists a player under — a mismatch that affects more than just that player’s own rank, since positions are also used to compare him against his peers.

Using it for a real draft

I drafted from pick #3 in a 10-team snake league using the model’s rankings as the base layer, then overrode it where I had information the model structurally can’t see: a new manager, a transfer rumor, a role change. Here’s every player I picked, in draft order, alongside where the model’s composite ranking puts them today and where they rank on each of the three things that composite blends — quality per 90 minutes played, quality per match appearance, and total points for the season. These are current numbers, from the same retrained model used throughout this piece, not a draft-day snapshot — three fixes landed after I drafted (the leakage fix, the ridge alpha-grid extension, a season of real data materializing for transferred players), so some of these ranks have moved since pick day:

A couple of picks need context the table can’t show. Minteh, at pick 11, was already ruled out for 3 months with an injury when I took him — a deliberate call, not an oversight; I was willing to stash one medium-term injury player on the bench for the upside once he’s back. Porro, at pick 6, was a pedigree bet at the time: an attacking full-back with a strong Sleeper points finish to the 25/26 season under Tottenham’s new manager, taken well ahead of where the model had him on draft day. The model has since come around to the same view — his composite rank has moved from 52 on draft day to 10 now, as a season of data under the new manager materialized. That’s not the model being wrong on draft day and right now; it’s the model correctly having less to go on back then.

Not every pick was model-led. Palestra and Tzolis were pure vibe picks — taken well below where the model would have had them, on reasoning the model has no way to represent. Tzolis specifically had no season data at all on draft day; he’s since played enough matches to get a real composite rank (222, still well below where he was picked at #12) rather than the blank the model gave him then. Sangaré and Muharemović sat somewhere in between: the model liked them enough to be in the conversation, but the actual call leaned as much on gut feel as on their ranking.

Leave those two vibe picks out, and a pattern shows up in the rest: in a 10-team league, the player taken at your Nth personal pick should, on average, have a composite rank of roughly 10×N if every manager in the league were drafting purely by the numbers. Compare each of my other 15 picks against that benchmark and 14 beat it — Bruno at pick 1 (composite rank 1, benchmark ~10), Fatawu at pick 17 (composite rank 103, benchmark ~170), everything in between. The one exception is Sangaré, the gut-feel pick: composite rank 165 against a benchmark of ~150, a 15-rank miss. On average, my picks landed 36 composite ranks better than that benchmark. That’s not proof the model is right — it’s that I trusted the model for most of my picks, took its advice, and can point to a real, computable number showing I came out ahead of the benchmark for it, even after a retrain moved several of these ranks around. Whatever happens over the season, that’s a squad I feel confident about going in.

What I’d point to if you’re evaluating this as engineering work

• Leakage discipline treated as infrastructure, not vigilance — every rolling feature is built to shift one match backward before it’s calculated, so it can never see the match it’s predicting, and a dedicated code path exists specifically to catch leakage in evaluation, separate from production training.
• Model choice backed by a real held-out comparison, not intuition — two algorithm families tested and rejected with numbers, not assumed inferior.
• Limitations measured and published, including the two cases where the shipped model loses to doing nothing.
• Bugs found by checking predictions against ground truth, not just unit tests — own goals silently scored as regular goals, goals-against attributed regardless of substitution timing, a home/away join silently broken for 16+ clubs from a name-matching mismatch. All caught by picking real players and diffing model output against what actually happened.

Full pipeline, docs, and code: epl-fantasy-draft-stats-tool. docs/MODEL_STATE.md has the line-by-line verification of every claim above against the actual code, if you want to check my work.