How I measured AI skill demand: methods

This is the companion to “What AI is doing to Data Science jobs”. That post stays with the findings; this one documents how they were produced — the pipeline, the taxonomy, the statistics, and the judgement calls. I have tried to be honest about the limits as well as the method.

• The pipeline
• The taxonomy
• Measuring demand
• Small samples and significance
• Skill co-occurrence
• Does AI replace skills?
• Scoring myself, and why it is not a ranking
• Limitations
• Provenance and AI use

The pipeline

A job ad is unstructured text. Turning thousands of them into comparable numbers takes several stages.

1. Extract. A language model reads each ad and returns the discrete skills it asks for, with the evidence span it came from.
2. Normalise. The same skill is written many ways (Python, Py3, python3). A normalisation step maps every raw phrase onto one canonical concept, so counts are not split across spellings. There are about 11,300 concepts.
3. Type, categorise, facet. Each concept is given a type (one of nine: tool, library, language, method, and so on), a category (one of 27 skill families), and a set of facets — the most important being AI relatedness (core AI / AI-adjacent / AI-independent).
4. Analyse. An analytics layer joins ads to concepts and computes demand, change over time, and co-occurrence, served from a small BigQuery warehouse.

The taxonomy

The 27 categories and the facet definitions are frozen — decided once and not adjusted to fit the results. This matters: it stops the categories being quietly tuned until the story looks neat. Types, categories, and facets are all defined at the concept level, independent of any specific occupation or year.

Measuring demand

The core measure is penetration: for a given skill (or category, or facet value) in one occupation-year, the share of that cell’s ads that mention it at least once. The denominator is every ad in the cell — including ads that mention the skill zero times — so a skill’s penetration is never inflated by quietly dropping the ads that never mentioned it.

Small samples and significance

The 2021 samples are small — 100 to 150 ads per role — so raw percentages can mislead. Two safeguards:

• Wilson confidence intervals on every penetration figure. The Wilson interval is a range that, across many samples, contains the true rate about 95% of the time. I use it rather than the textbook p ± 1.96·√(p(1−p)/n) (Wald) interval, which behaves badly at small sample sizes and near 0% or 100% — it can even produce bounds below 0 or above 1.
• A two-proportion test on every 2021→2026 change. It asks: if there were really no change, how likely is a gap this large from sampling noise alone? I only call a change real when that probability is below 5%. Changes that fail the test are reported as “no clear change”, not dressed up.

Skill co-occurrence

The knowledge graph and the replacement analysis both rest on skill adjacency — how often two skills appear in the same ad, adjusted for how common each is on its own. For each pair of skills I compute a per-cell co-occurrence statistic that is high only when the two genuinely go together, not merely when both happen to be common. A keep threshold on that statistic stops the graph collapsing into a hairball of “both skills are just common” links.

Does AI replace skills?

It is tempting to read the 2021→2026 shift as AI replacing old skills. I resist that, for a specific reason. The data is cross-sectional: two separate snapshots of different ads, not the same jobs followed over time. Because no ad appears in both years, I can never watch one ad drop Hadoop and add dbt — I only see the aggregate rates move. Inferring an individual-level swap from group-level totals is a known trap (the ecological inference problem).

So “replacement” stays a hypothesis, flagged only when three things line up in the same role: an old skill falls significantly, a new skill rises significantly, and the new skill moves into the old one’s neighbourhood (the skills it used to appear alongside). Even then the language is “consistent with replacement”, never “caused”. True proof would need longitudinal data — the same roles at the same employers tracked over years — which job ads cannot give.

Scoring myself, and why it is not a ranking

For the personal section I scored my own level on each role’s top skills (0 aware, 1 developing, 2 advanced), using my resume and portfolio projects as evidence, and combined that with demand into a single “match” percentage per role. For example, dbt appears in 21% of Data Engineer ads, but I have not used it in production, so I scored it 0 (aware); SQL, which I have used daily for years, scored 2 (advanced).

I deliberately stopped short of turning that into a single “best job for me” number, and it is worth being explicit about why. A demand-match score only measures overlap with today’s advertised skills. It is blind to three things that matter at least as much: which tasks I actually want to do, how exposed each role is to automation, and the kind of team and place I want to work in. Ranking the roles on the score alone would give a confident answer to the wrong question. So the score is presented as one input — the “match” — and set beside a qualitative “fit”, rather than collapsed into a league table.

Limitations

• One source, one country. All ads are from LinkedIn, US roles. Other boards and other countries may differ.
• Small 2021 samples. Three roles, ~100–150 ads each; hence the confidence intervals and tests.
• Coverage. 2–4% of skill mentions map to no concept; about 2% of concepts are “held” (unclassified) and excluded.
• Ads are not jobs. An ad reflects what an employer chose to write, not the real day-to-day work — and ads under-list human skills badly (communication appears in only 10–15% of ads), so this method cannot speak to those.
• Company is a rough proxy for sector. The “AI demand beyond tech” point is indicative, not a sector analysis.

Provenance and AI use

The extraction, typing, categorisation, and facet classification steps use Claude Haiku, at temperature 0 for the facet step. Every model-produced row is marked as model-made and was reviewed before use; model outputs are never recorded as human decisions. The analytics layer (demand, change, adjacency, the reflection scoring) is deterministic Python plus SQL — no model in the loop.

The pipeline and method are reproducible, but the exact figures depend on the specific job-ad sample, which I do not republish. Tools: Python and BigQuery for the analysis; the interactive graph is a static export rendered with sigma.js.

Want to replicate it? I have published a minimal, runnable version of the pipeline — the stage code, the taxonomy and facets as a worked example, the input and output data contracts, and a run guide — at github.com/reggiealderson/labour-market-skill-graph. Bring your own job ads in the documented schema and an Anthropic API key, and it produces the same kind of skill graph you see here. The raw job-ad dataset is not included.

Back to the findings.