How Does a RAG Reranker Really Work?

Stay Ahead, Stay ONMINE

How Does a RAG Reranker Really Work?

When RAG retrieval disappoints, the advice AI engineers hear today is almost always “add a reranker”. Ask why a reranker works, and the answer usually stays at the architecture level: it is a cross-encoder, it applies attention over the query and the passage together, it is fine-tuned on relevance labels. All of that is true, and none of it says what the model actually learned. Push one level down, to terms a business partner could check, and the explanation usually stops.That gap matters. A team that cannot say in plain terms what the reranker does cannot defend the choice to use one, and cannot spot the cases where a keyword lookup would beat it for a fraction of the cost.This article gives the honest answer, the one you can hand to your business partner without waving hands. The reranker is not smarter than the embeddings step below it. It runs the same mechanism (statistical token association from training data), just conditioned differently (on the query-passage pair rather than each text independently). Once you see that, the “when to use a reranker” question stops being “add it because the tutorial did” and becomes “add it only when this specific tradeoff is worth paying for”.🧭 New to the series? Start with the map: Prompt, Context, Loop sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.This article sits in Part I, alongside the embeddings triptych (2A / 2B / 2C). – Image by author📓 Try the reranker on your own PDF at doc-intel/notebooks-vol1. The companion notebook loads a cross-encoder, applies it to a keyword-filtered top-K, and shows both the score and the tokens driving it. Change the query, watch which keywords carry the ranking.1. What data scientists say, and why it isn’t enoughAsk three data scientists what a reranker does and you get three answers, roughly:“It’s a cross-encoder. It scores the query-passage pair jointly and gives a relevance score.” Technically true, but the words cross-encoder and relevance are hiding what the model actually learned.“It applies attention over both texts, so it sees the interaction between them.” True at the architecture level, but architecture does not tell you what the model is doing with that attention.“It’s trained on relevance labels, so it learns which passages answer which questions.” Very close, but “learns which passages answer” is the wrong verb. The model does not learn to answer. It learns which tokens co-occurred.None of the three is wrong. All three are incomplete in a way that matters when you have to decide whether to keep the reranker in your pipeline, whether to fine-tune it on your corpus, or whether to replace it with something cheaper.The rest of this article walks that answer down to the mechanism, then names three consequences that change how you architect enterprise RAG.2. What actually happens inside a rerankerThe reranker is a specific kind of transformer, trained on a specific kind of data, that produces a specific kind of number. Each of those three pieces matters.2.1 The architecture: cross-encoder, not bi-encoderAn embedder (bi-encoder) reads the query alone, produces one vector. Reads a passage alone, produces one vector. Compares the two vectors by cosine. Each text is embedded independently, and the model never sees them together during scoring.A reranker (cross-encoder) reads the query and the passage together, as one concatenated input: [CLS] query [SEP] passage [SEP]. It runs BERT-style attention over the joint input, where every token can attend to every other token. It outputs a single relevance score.That “reads them together” is the whole architectural difference. Bi-encoder: two vectors, one comparison operation. Cross-encoder: one forward pass, one score. The joint attention is why the reranker feels smarter, and why it is 30 to 100 times slower per query.2.2 The training data: MS MARCO and its cousinsWhere does the reranker learn its scoring? From query-passage relevance pairs labeled by humans. The canonical dataset is MS MARCO (Bajaj et al. 2016, one million real Bing search queries with human-graded passage relevance). Others: Natural Questions (Google search + Wikipedia paragraphs), BEIR (a benchmark aggregator), TREC.Every training example is a triple: (query, passage, relevance_label). The model sees millions of these, and its weights adjust so that pairs labeled relevant get higher scores than pairs labeled not relevant.That is the sole learning signal. The model is never shown a question and asked to compose an answer; it is shown pairs, and it optimizes for a score that separates relevant pairs from non-relevant ones.Which raises the honest question: what pattern actually separates them in the training data?2.3 What the model really learns: keyword co-occurrence at the pair levelHere is the level down that rarely gets explained.The model looks at millions of (query, passage, relevance) triples and asks: what patterns in the joint token stream predict the relevance label? The dominant pattern is not “answering”. It is which query tokens tend to co-occur with which passage tokens in high-relevance pairs.Concretely, in MS MARCO the query “how to cancel my subscription” is labeled relevant against passages containing cancel, subscription, unsubscribe, terminate, end your membership. Millions of examples reinforce that when the query contains cancel, passages containing terminate or unsubscribe tend to be labeled relevant. The reranker’s weights absorb that association.So the “smart” reranker is doing keyword linking, at the query-passage pair level. It is a learned association table between query token neighborhoods and passage token neighborhoods, dressed up as a neural network score.The embedder does the same thing, but at each text independently. The reranker does it conditioned on the pair. Same mechanism, different conditioning.Second-order signals the reranker also picks up: positional patterns (a term appearing early in the passage often correlates with relevance), syntactic structure (subject-verb-object relations that link query tokens to passage tokens), the presence of definitional phrasing (“X is Y”). Those help, but they are second-order; the dominant signal is keyword co-occurrence.Why this frame matters: once you see the mechanism, the “will it work on my corpus?” question has a clear answer. If your corpus vocabulary and query vocabulary look like MS MARCO (general English, common web topics), the trained associations transfer, and the reranker feels magical. If your corpus vocabulary is specialized (insurance contracts, medical records, regulatory filings), the trained associations do not cover your domain, and the reranker inherits the same out-of-vocabulary failures as the embedder below it. No amount of “but it’s a cross-encoder” fixes that.3. The mechanism, shown: where the reranker wins, where it hits a wallSection 2 made a claim: the reranker is a learned association table between question-language and answer-language. That claim is testable. Take a handful of candidates, score them with three embedders (MiniLM, ada-002, text-embedding-3-large) and three cross-encoders (bge-base, bge-large, ms-marco-MiniLM), and read each row.3.1 Where it wins: the answer that does not repeat the questionAsk “What is the maximum coverage amount?” against three passages: the answer (“Cover is capped at 50,000 euros per year”), an echo that repeats the question’s words without answering (“The maximum coverage amount can be found in the benefits schedule”), and a distractor.Every embedder ranks the echo first; both bge rerankers flip the answer to the top. – Image by authorEvery embedder puts the echo first. It shares maximum, coverage, amount with the question, so its vector sits close. The answer shares almost nothing lexically, so it lands second or third. The two bge rerankers flip it: they read the question and the answer together, recognize that a “capped at X per year” passage answers a “maximum coverage amount” question, and lift it to #1. This is the reranker doing its one real job, bridging the question’s words to the answer’s words.It is not a one-off. The same flip reproduces on plain factoids:Same shape, general-knowledge version. bge lifts the answer over the echo, ms-marco keeps the echo on top. – Image by authorAcross a dozen queries of this shape (who wrote a play, the boiling point of water, the speed of light, the first president, plus the enterprise trio of deductible, notice period, coverage) the two bge rerankers rescue the answer to #1 where every embedder ranked an echo above it. The win is real and repeatable, on exactly one shape: a short factual answer that does not repeat the question, sitting behind an echo that does.Two honest caveats sit in the same two figures. First, not every reranker does it: ms-marco-MiniLM keeps the echo on top in both cases, the same lexical bias an embedder has. Second, when a strong embedder already answers the question (text-embedding-3-large gets several of these on its own), the reranker adds nothing over just using a better embedder.3.2 Where it hits a wall: your private vocabularyNow the case that decides the enterprise question. Ask “what’s the rule on contractor overtime?” where the answer uses the company’s own term, “non-employee labor compensated beyond 40h/week”, and never the word contractor.The answer never says “contractor”, it says “non-employee labor”. Every model, embedder and reranker alike, ranks it last. – Image by authorEvery column, embedder and reranker, ranks the answer last. The surface match (“Contractors are paid on a per-project basis”) wins. The reranker never saw contractor map to non-employee labor in MS MARCO, so its association table has no entry for it. The cross-attention it runs is real, but it can only fire on associations it learned, and this one it never learned.3.3 To clear that wall, you must already know the answerThe fix the literature offers is fine-tuning: feed the reranker labeled (question, passage, relevant) triples from your own domain until it learns that contractor maps to non-employee labor. But look at what labeling one of those triples requires. Someone who knows the domain has to point at the right passage and say this one answers the question. To point at it, they had to recognize that “non-employee labor beyond 40h/week” is what the answer looks like. That recognition is the answer keywords.So the training label and the dictionary entry carry the same information. For a “maximum coverage amount” question, labeling the answer means knowing the answer contains capped at, up to, a currency, per year. Writing the expert dictionary means typing exactly that: {capped at, up to, maximum, €, per year}. For the contractor case, labeling the pairs means knowing that contractor equals non-employee labor in this company, and the dictionary entry is that one line.The difference is the cost and the shape. The reranker needs hundreds of labeled pairs to generalize the mapping statistically, a retraining run, and it stays a black box scoring 0.83. The dictionary needs one line, fires deterministically, and shows the exact keyword that matched under audit. If you already know the answer well enough to label the data, you already know the answer keywords, and writing them down is the cheaper, auditable path. The reranker’s statistical learning only pays when the mapping is too broad to enumerate, which is the open web, not a bounded enterprise domain.4. Why the answer matters in enterpriseThree consequences flow from the honest answer, and each of them changes an architecture decision you may have made without noticing.4.1 The audit trail is opaqueA relevance score of 0.83 from a reranker is not defensible under scrutiny. A regulator asking why was this passage returned? gets “the reranker gave it 0.83” as an answer. That is not an audit trail. It is a black box that produced a number.Contrast with a keyword filter: the retrieved passage contains force majeure and pandemic. That statement is inspectable, replayable, and defensible. If the retrieval was wrong, you can trace which keyword was missing from the dictionary and add it. If a reranker was wrong, you shrug at the score and move on, or you retrain the whole thing.For enterprise use cases where retrieval decisions have compliance or contractual consequences (insurance underwriting, legal discovery, medical records, regulatory reporting), opacity is not a small tradeoff; it is a disqualifier.4.2 The cost is realA cross-encoder is 30 to 100 times slower per query than a bi-encoder. If your bi-encoder scores 1000 candidates in 20 ms, the reranker scores the same 1000 in 600 ms to 2 seconds. In practice, you do not rerank 1000 candidates: you take the bi-encoder’s top-20 or top-50 and rerank only those, which puts the added latency back in the 15 to 100 ms range, depending on the depth and the model.That is fine at low query volume. At 100 queries per second sustained, the reranker cost is a real operational line item: more GPU capacity, longer p99 latencies, more infrastructure to keep warm. The value it adds has to justify that cost, and that only happens when its trained associations genuinely cover your vocabulary. On out-of-domain enterprise corpora, it often does not.4.3 The vocabulary gap will show upEvery failure mode catalogued for embeddings on out-of-domain enterprise vocabulary applies to the reranker too, because it was trained on the same distribution (general web search). Force majeure and act of God are equivalent in an insurance contract but land in different neighborhoods in the reranker’s learned associations, because it saw them in different training contexts. Rescission was rare in MS MARCO. ShieldPro Elite was not there at all.Fine-tuning the reranker on your domain corpus helps, but only up to a point. You need labeled query-passage pairs from your domain to fine-tune, which is exactly what enterprise teams rarely have. And even a fine-tuned reranker inherits the same underlying mechanism: it still learns token associations, just from your smaller domain corpus, and the number of examples you can label rarely matches the millions MS MARCO provides.5. What to do instead, and when to keep the rerankerGiven the mechanism and the enterprise consequences, the question becomes: what earns the reranker’s slot in your pipeline?The default in enterprise RAG (per the series’ recommendation): a curated keyword dictionary maintained by domain experts. The expert already knows that force majeure equals act of God in this contract, that rescission is the formal term for what the user called cancellation, that ShieldPro Elite is the top-tier homeowners plan. Encoding that once in a versioned YAML dictionary and running keyword-based retrieval on top gives you:Auditable retrieval (the matched keywords are inspectable)Low latency (no LLM in the hot path, no GPU cost)Durability across model releases (the dictionary outlives every reranker version)Explainability to the business (they can read the dictionary)The reranker earns its slot in four specific cases. The first three are runtime slots, the fourth is not.In-domain distribution. Your corpus vocabulary and query vocabulary genuinely look like MS MARCO (general web, common English, high-frequency topics). Consumer FAQs, public-service portals, e-commerce help. The reranker’s trained associations transfer. Use it.Semantic re-ranking of a keyword-filtered top-K. After the keyword dictionary filters the corpus down to 20 candidates, the reranker can order them by contextual relevance. This is the same role Article 2C section 5.3 assigns to bi-encoder embeddings, and a cross-encoder does it more accurately at the cost of extra latency. Worth it when the top-K is small and the ordering matters.Compliance scenarios where the reranker’s score itself is the audit artefact. If your compliance framework requires “the model scored this passage above threshold X”, the score is the artefact, and the reranker fits the requirement.Offline, to discover what belongs in the dictionary. Run the reranker over a sample of real questions and read what it pulls up. Where it surfaces a mapping the dictionary does not have yet, you have a candidate alias. An expert confirms it or throws it out, and only the confirmed line ships. The model does the searching, the expert does the deciding, and what reaches production is the validated line, never the score. Article 2C gives embeddings the same treatment, and Article 16D runs this loop continuously at corpus scale, a failed search proposing the alias and an expert confirming it.The fourth case is the one that reframes the other three. Both paths do the same job, and the diagram below puts them side by side.The same table twice: learned on someone else’s corpus, or written by people who know the words. – Image by authorOutside those four cases, the reranker mostly adds cost: impressive in a demo, expensive in production, opaque under audit, and unable to compensate for the trained associations it does not have.One equivalence sits underneath all of it, and it is worth stating in a single line. A reranker is a keyword-association table that someone else trained on someone else’s corpus. Writing your own dictionary is the same job, done by the people who actually know the vocabulary, at a fraction of the cost and in a form an auditor can read. That equivalence stays invisible as long as the model is treated as magic. Open the box, as Section 2.3 did, and the choice makes itself: use the model to find candidate links, use the expert to validate them, and let the validated table be what production runs on.6. Sources and further readingThe reranker literature is dense and largely optimistic. Reading it against the article’s frame (“cross-encoders learn keyword association at the pair level, not comprehension”) is more useful than reading it as an unqualified endorsement.Same direction as the article:Nogueira & Cho, Passage Re-ranking with BERT, 2019 (arXiv:1901.04085). The paper that introduced cross-encoder reranking with BERT and set the pattern most current rerankers follow. Reads honestly about what the model learns.Khattab & Zaharia, ColBERT, SIGIR 2020 (arXiv:2004.12832). Late-interaction retrieval. Explicitly designed to preserve token-level signal that both embedders and cross-encoders lose, which is the strongest architectural signal that the token-level pattern is what actually matters.Different angle, different context:Bajaj et al., MS MARCO, 2016 (arXiv:1611.09268). The training data that shapes what almost every commercial reranker actually knows. Worth skimming to see the query and passage distribution the reranker’s associations come from.Muennighoff et al., MTEB: Massive Text Embedding Benchmark, EACL 2023 (arXiv:2210.07316). Includes reranker leaderboards. The leaderboard is measured on in-distribution benchmarks, which is exactly the case where the reranker looks good. It says less about what happens on your out-of-domain enterprise corpus.

That gap matters. A team that cannot say in plain terms what the reranker does cannot defend the choice to use one, and cannot spot the cases where a keyword lookup would beat it for a fraction of the cost.

This article gives the honest answer, the one you can hand to your business partner without waving hands. The reranker is not smarter than the embeddings step below it. It runs the same mechanism (statistical token association from training data), just conditioned differently (on the query-passage pair rather than each text independently). Once you see that, the “when to use a reranker” question stops being “add it because the tutorial did” and becomes “add it only when this specific tradeoff is worth paying for”.

🧭 New to the series? Start with the map: Prompt, Context, Loop sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.

*This article sits in Part I, alongside the embeddings triptych (2A / 2B / 2C). – Image by author*

📓 Try the reranker on your own PDF at doc-intel/notebooks-vol1. The companion notebook loads a cross-encoder, applies it to a keyword-filtered top-K, and shows both the score and the tokens driving it. Change the query, watch which keywords carry the ranking.

1. What data scientists say, and why it isn’t enough

Ask three data scientists what a reranker does and you get three answers, roughly:

“It’s a cross-encoder. It scores the query-passage pair jointly and gives a relevance score.” Technically true, but the words cross-encoder and relevance are hiding what the model actually learned.
“It applies attention over both texts, so it sees the interaction between them.” True at the architecture level, but architecture does not tell you what the model is doing with that attention.
“It’s trained on relevance labels, so it learns which passages answer which questions.” Very close, but “learns which passages answer” is the wrong verb. The model does not learn to answer. It learns which tokens co-occurred.

None of the three is wrong. All three are incomplete in a way that matters when you have to decide whether to keep the reranker in your pipeline, whether to fine-tune it on your corpus, or whether to replace it with something cheaper.

The rest of this article walks that answer down to the mechanism, then names three consequences that change how you architect enterprise RAG.

2. What actually happens inside a reranker

The reranker is a specific kind of transformer, trained on a specific kind of data, that produces a specific kind of number. Each of those three pieces matters.

2.1 The architecture: cross-encoder, not bi-encoder

An embedder (bi-encoder) reads the query alone, produces one vector. Reads a passage alone, produces one vector. Compares the two vectors by cosine. Each text is embedded independently, and the model never sees them together during scoring.

A reranker (cross-encoder) reads the query and the passage together, as one concatenated input: [CLS] query [SEP] passage [SEP]. It runs BERT-style attention over the joint input, where every token can attend to every other token. It outputs a single relevance score.

That “reads them together” is the whole architectural difference. Bi-encoder: two vectors, one comparison operation. Cross-encoder: one forward pass, one score. The joint attention is why the reranker feels smarter, and why it is 30 to 100 times slower per query.

2.2 The training data: MS MARCO and its cousins

Where does the reranker learn its scoring? From query-passage relevance pairs labeled by humans. The canonical dataset is MS MARCO (Bajaj et al. 2016, one million real Bing search queries with human-graded passage relevance). Others: Natural Questions (Google search + Wikipedia paragraphs), BEIR (a benchmark aggregator), TREC.

Every training example is a triple: (query, passage, relevance_label). The model sees millions of these, and its weights adjust so that pairs labeled relevant get higher scores than pairs labeled not relevant.

That is the sole learning signal. The model is never shown a question and asked to compose an answer; it is shown pairs, and it optimizes for a score that separates relevant pairs from non-relevant ones.

Which raises the honest question: what pattern actually separates them in the training data?

2.3 What the model really learns: keyword co-occurrence at the pair level

Here is the level down that rarely gets explained.

The model looks at millions of (query, passage, relevance) triples and asks: what patterns in the joint token stream predict the relevance label? The dominant pattern is not “answering”. It is which query tokens tend to co-occur with which passage tokens in high-relevance pairs.

Concretely, in MS MARCO the query “how to cancel my subscription” is labeled relevant against passages containing cancel, subscription, unsubscribe, terminate, end your membership. Millions of examples reinforce that when the query contains cancel, passages containing terminate or unsubscribe tend to be labeled relevant. The reranker’s weights absorb that association.

So the “smart” reranker is doing keyword linking, at the query-passage pair level. It is a learned association table between query token neighborhoods and passage token neighborhoods, dressed up as a neural network score.

The embedder does the same thing, but at each text independently. The reranker does it conditioned on the pair. Same mechanism, different conditioning.

Second-order signals the reranker also picks up: positional patterns (a term appearing early in the passage often correlates with relevance), syntactic structure (subject-verb-object relations that link query tokens to passage tokens), the presence of definitional phrasing (“X is Y”). Those help, but they are second-order; the dominant signal is keyword co-occurrence.

Why this frame matters: once you see the mechanism, the “will it work on my corpus?” question has a clear answer. If your corpus vocabulary and query vocabulary look like MS MARCO (general English, common web topics), the trained associations transfer, and the reranker feels magical. If your corpus vocabulary is specialized (insurance contracts, medical records, regulatory filings), the trained associations do not cover your domain, and the reranker inherits the same out-of-vocabulary failures as the embedder below it. No amount of “but it’s a cross-encoder” fixes that.

3. The mechanism, shown: where the reranker wins, where it hits a wall

Section 2 made a claim: the reranker is a learned association table between question-language and answer-language. That claim is testable. Take a handful of candidates, score them with three embedders (MiniLM, ada-002, text-embedding-3-large) and three cross-encoders (bge-base, bge-large, ms-marco-MiniLM), and read each row.

3.1 Where it wins: the answer that does not repeat the question

Ask “What is the maximum coverage amount?” against three passages: the answer (“Cover is capped at 50,000 euros per year”), an echo that repeats the question’s words without answering (“The maximum coverage amount can be found in the benefits schedule”), and a distractor.

*Every embedder ranks the echo first; both bge rerankers flip the answer to the top. – Image by author*

Every embedder puts the echo first. It shares maximum, coverage, amount with the question, so its vector sits close. The answer shares almost nothing lexically, so it lands second or third. The two bge rerankers flip it: they read the question and the answer together, recognize that a “capped at X per year” passage answers a “maximum coverage amount” question, and lift it to #1. This is the reranker doing its one real job, bridging the question’s words to the answer’s words.

It is not a one-off. The same flip reproduces on plain factoids:

*Same shape, general-knowledge version. bge lifts the answer over the echo, ms-marco keeps the echo on top. – Image by author*

Across a dozen queries of this shape (who wrote a play, the boiling point of water, the speed of light, the first president, plus the enterprise trio of deductible, notice period, coverage) the two bge rerankers rescue the answer to #1 where every embedder ranked an echo above it. The win is real and repeatable, on exactly one shape: a short factual answer that does not repeat the question, sitting behind an echo that does.

Two honest caveats sit in the same two figures. First, not every reranker does it: ms-marco-MiniLM keeps the echo on top in both cases, the same lexical bias an embedder has. Second, when a strong embedder already answers the question (text-embedding-3-large gets several of these on its own), the reranker adds nothing over just using a better embedder.

3.2 Where it hits a wall: your private vocabulary

Now the case that decides the enterprise question. Ask “what’s the rule on contractor overtime?” where the answer uses the company’s own term, “non-employee labor compensated beyond 40h/week”, and never the word contractor.

*The answer never says “contractor”, it says “non-employee labor”. Every model, embedder and reranker alike, ranks it last. – Image by author*

Every column, embedder and reranker, ranks the answer last. The surface match (“Contractors are paid on a per-project basis”) wins. The reranker never saw contractor map to non-employee labor in MS MARCO, so its association table has no entry for it. The cross-attention it runs is real, but it can only fire on associations it learned, and this one it never learned.

3.3 To clear that wall, you must already know the answer

The fix the literature offers is fine-tuning: feed the reranker labeled (question, passage, relevant) triples from your own domain until it learns that contractor maps to non-employee labor. But look at what labeling one of those triples requires. Someone who knows the domain has to point at the right passage and say this one answers the question. To point at it, they had to recognize that “non-employee labor beyond 40h/week” is what the answer looks like. That recognition is the answer keywords.

So the training label and the dictionary entry carry the same information. For a “maximum coverage amount” question, labeling the answer means knowing the answer contains capped at, up to, a currency, per year. Writing the expert dictionary means typing exactly that: {capped at, up to, maximum, €, per year}. For the contractor case, labeling the pairs means knowing that contractor equals non-employee labor in this company, and the dictionary entry is that one line.

The difference is the cost and the shape. The reranker needs hundreds of labeled pairs to generalize the mapping statistically, a retraining run, and it stays a black box scoring 0.83. The dictionary needs one line, fires deterministically, and shows the exact keyword that matched under audit. If you already know the answer well enough to label the data, you already know the answer keywords, and writing them down is the cheaper, auditable path. The reranker’s statistical learning only pays when the mapping is too broad to enumerate, which is the open web, not a bounded enterprise domain.

4. Why the answer matters in enterprise

Three consequences flow from the honest answer, and each of them changes an architecture decision you may have made without noticing.

4.1 The audit trail is opaque

A relevance score of 0.83 from a reranker is not defensible under scrutiny. A regulator asking why was this passage returned? gets “the reranker gave it 0.83” as an answer. That is not an audit trail. It is a black box that produced a number.

Contrast with a keyword filter: the retrieved passage contains force majeure and pandemic. That statement is inspectable, replayable, and defensible. If the retrieval was wrong, you can trace which keyword was missing from the dictionary and add it. If a reranker was wrong, you shrug at the score and move on, or you retrain the whole thing.

For enterprise use cases where retrieval decisions have compliance or contractual consequences (insurance underwriting, legal discovery, medical records, regulatory reporting), opacity is not a small tradeoff; it is a disqualifier.

4.2 The cost is real

A cross-encoder is 30 to 100 times slower per query than a bi-encoder. If your bi-encoder scores 1000 candidates in 20 ms, the reranker scores the same 1000 in 600 ms to 2 seconds. In practice, you do not rerank 1000 candidates: you take the bi-encoder’s top-20 or top-50 and rerank only those, which puts the added latency back in the 15 to 100 ms range, depending on the depth and the model.

That is fine at low query volume. At 100 queries per second sustained, the reranker cost is a real operational line item: more GPU capacity, longer p99 latencies, more infrastructure to keep warm. The value it adds has to justify that cost, and that only happens when its trained associations genuinely cover your vocabulary. On out-of-domain enterprise corpora, it often does not.

4.3 The vocabulary gap will show up

Every failure mode catalogued for embeddings on out-of-domain enterprise vocabulary applies to the reranker too, because it was trained on the same distribution (general web search). Force majeure and act of God are equivalent in an insurance contract but land in different neighborhoods in the reranker’s learned associations, because it saw them in different training contexts. Rescission was rare in MS MARCO. ShieldPro Elite was not there at all.

Fine-tuning the reranker on your domain corpus helps, but only up to a point. You need labeled query-passage pairs from your domain to fine-tune, which is exactly what enterprise teams rarely have. And even a fine-tuned reranker inherits the same underlying mechanism: it still learns token associations, just from your smaller domain corpus, and the number of examples you can label rarely matches the millions MS MARCO provides.

5. What to do instead, and when to keep the reranker

Given the mechanism and the enterprise consequences, the question becomes: what earns the reranker’s slot in your pipeline?

The default in enterprise RAG (per the series’ recommendation): a curated keyword dictionary maintained by domain experts. The expert already knows that force majeure equals act of God in this contract, that rescission is the formal term for what the user called cancellation, that ShieldPro Elite is the top-tier homeowners plan. Encoding that once in a versioned YAML dictionary and running keyword-based retrieval on top gives you:

Auditable retrieval (the matched keywords are inspectable)
Low latency (no LLM in the hot path, no GPU cost)
Durability across model releases (the dictionary outlives every reranker version)
Explainability to the business (they can read the dictionary)

The reranker earns its slot in four specific cases. The first three are runtime slots, the fourth is not.

In-domain distribution. Your corpus vocabulary and query vocabulary genuinely look like MS MARCO (general web, common English, high-frequency topics). Consumer FAQs, public-service portals, e-commerce help. The reranker’s trained associations transfer. Use it.
Semantic re-ranking of a keyword-filtered top-K. After the keyword dictionary filters the corpus down to 20 candidates, the reranker can order them by contextual relevance. This is the same role Article 2C section 5.3 assigns to bi-encoder embeddings, and a cross-encoder does it more accurately at the cost of extra latency. Worth it when the top-K is small and the ordering matters.
Compliance scenarios where the reranker’s score itself is the audit artefact. If your compliance framework requires “the model scored this passage above threshold X”, the score is the artefact, and the reranker fits the requirement.
Offline, to discover what belongs in the dictionary. Run the reranker over a sample of real questions and read what it pulls up. Where it surfaces a mapping the dictionary does not have yet, you have a candidate alias. An expert confirms it or throws it out, and only the confirmed line ships. The model does the searching, the expert does the deciding, and what reaches production is the validated line, never the score. Article 2C gives embeddings the same treatment, and Article 16D runs this loop continuously at corpus scale, a failed search proposing the alias and an expert confirming it.

The fourth case is the one that reframes the other three. Both paths do the same job, and the diagram below puts them side by side.

*The same table twice: learned on someone else’s corpus, or written by people who know the words. – Image by author*

Outside those four cases, the reranker mostly adds cost: impressive in a demo, expensive in production, opaque under audit, and unable to compensate for the trained associations it does not have.

One equivalence sits underneath all of it, and it is worth stating in a single line. A reranker is a keyword-association table that someone else trained on someone else’s corpus. Writing your own dictionary is the same job, done by the people who actually know the vocabulary, at a fraction of the cost and in a form an auditor can read. That equivalence stays invisible as long as the model is treated as magic. Open the box, as Section 2.3 did, and the choice makes itself: use the model to find candidate links, use the expert to validate them, and let the validated table be what production runs on.

6. Sources and further reading

The reranker literature is dense and largely optimistic. Reading it against the article’s frame (“cross-encoders learn keyword association at the pair level, not comprehension”) is more useful than reading it as an unqualified endorsement.

Same direction as the article:

Nogueira & Cho, Passage Re-ranking with BERT, 2019 (arXiv:1901.04085). The paper that introduced cross-encoder reranking with BERT and set the pattern most current rerankers follow. Reads honestly about what the model learns.
Khattab & Zaharia, ColBERT, SIGIR 2020 (arXiv:2004.12832). Late-interaction retrieval. Explicitly designed to preserve token-level signal that both embedders and cross-encoders lose, which is the strongest architectural signal that the token-level pattern is what actually matters.

Different angle, different context:

Bajaj et al., MS MARCO, 2016 (arXiv:1611.09268). The training data that shapes what almost every commercial reranker actually knows. Worth skimming to see the query and passage distribution the reranker’s associations come from.
Muennighoff et al., MTEB: Massive Text Embedding Benchmark, EACL 2023 (arXiv:2210.07316). Includes reranker leaderboards. The leaderboard is measured on in-distribution benchmarks, which is exactly the case where the reranker looks good. It says less about what happens on your out-of-domain enterprise corpus.

Stay Ahead

Explore More Insights

Stay ahead with more perspectives on cutting-edge power, infrastructure, energy, bitcoin and AI solutions. Explore these articles to uncover strategies and insights shaping the future of industries.

Cisco bulks up its AI infrastructure portfolio with Supermicro’s liquid-cooled servers

“This expansion enables customers to easily manage complex, high-density AI clusters alongside non-AI workloads. Customers will also now be able to deploy rack-to-fabric liquid cooling, featuring Cisco liquid-cooled AI networking systems alongside Supermicro’s liquid-cooled servers. This unlocks trillion-parameter training and high-throughput inference use cases with platforms including Nvidia Vera Rubin NVL72 and Nvidia

Taking your temperature from the inside

Oral and forehead thermometers may not accurately capture a person’s core body temperature, and the few ingestible temperature sensors on the market are so big they are hard to swallow and risk obstructing the GI tract. But MIT engineers created one that can send continuous temperature updates at a size

Cisco taps Teleport for infrastructure identity management tech

Cisco is continuing to embed identity management capabilities deeper into its product portfolio by teaming with Teleport, a security vendor headquartered in Oakland, Calif., that’s focused on identity-based infrastructure access management. Cisco is investing in and partnering with Teleport as part of its efforts to bring infrastructure identity everywhere, Matt Caulfield,

Google adds AI-powered assessments to Migration Center to speed up cloud migration planning

“This tool is fundamentally bounded by its inputs such as infrastructure inventory, price of compute and storage, but it cannot see application dependencies, compliance obligations, or real-world performance behavior, each of which requires separate, deliberate discovery,” said Manoj Chandra Jha, principal analyst at Nord-IQ Research. “Thus, enterprises should treat AI-generated

DOE and SBA Launch SBIC-E Initiative to Unleash Private Capital for American Innovation and Small Businesses

WASHINGTON—The U.S. Department of Energy (DOE) and the U.S. Small Business Administration (SBA) today signed a Memorandum of Agreement establishing the Small Business Investment Company-Energy (SBIC-E) Initiative, a new strategic partnership advancing President Trump’s commitment to supporting America’s small businesses, strengthening domestic manufacturing and supply chains, and ensuring the United States leads in the technologies critical to our national and economic security. The new SBIC-E Initiative brings together DOE’s scientific and technical expertise with SBA’s proven Small Business Investment Company (SBIC) Program, which currently has $58 billion in combined portfolio value. Since 1958, the SBIC Program has invested $147 billion in American small businesses, and since 1995, SBIC-backed businesses have created or supported 10.6 million jobs. “America’s small businesses drive American innovation and affordable, reliable energy access,” said U.S. Secretary of Energy Chris Wright. “By partnering with the Small Business Administration, the Energy Department is committing to invest its resources in American small businesses that will create jobs, strengthen our domestic manufacturing base, and unleash American energy production.” Through DOE’s Office of Technology Commercialization (OTC), the Department will identify strategic technology priorities, provide technical and commercialization expertise, and help engage the investment community. SBA, through its Office of Investment and Innovation, will administer the initiative and encourage the formation and growth of investment funds focused on those priorities. SBIC-E adds another tool to that effort by connecting innovators with private capital to help promising technologies grow, scale, and build here at home. “President Trump is establishing American energy dominance, ending the Green New Scam, and putting our nations’ producers and innovators back in control at the dawn of a new era of energy reliability and abundance,” said SBA Administrator Kelly Loeffler. “Through this partnership, the SBA and Department of Energy are strengthening access to capital in the private sector to

Energy Department Announces $500 Million Award to Revitalize American Steelmaking

WASHINGTON—The U.S. Department of Energy (DOE) today announced a $500 million award to support a $1 billion investment at Cleveland-Cliffs’ Middletown Works facility in Middletown, Ohio. Vice President JD Vance and U.S. Energy Secretary Chris Wright visited Middletown Works today to highlight the Trump Administration’s commitment to American steelworkers and the resurgence of American manufacturing. The investment will modernize American steelmaking, protect 2,300 American jobs, and strengthen the domestic steel supply chain. The project advances President Trump’s commitment to put American workers first, bring investment back to American communities, and strengthen the industries critical to America’s economic and national security. Cleveland-Cliffs determined that the business case for the original project scope no longer made sense given customers’ unwillingness to pay a “green premium” for steel. Working with DOE, Cleveland-Cliffs identified a viable alternative that will upgrade and improve the efficiency of the existing coal-fired blast furnace while also capturing and commercializing co-product blast furnace gas (BFG). “President Trump is rebuilding America’s industrial base,” said Secretary Wright. “This investment puts American workers and American manufacturing first. It will modernize one of our nation’s critical steelmaking facilities, protect thousands of jobs, and strengthen our domestic steel production—keeping Ohio at the heart of American manufacturing and strengthening our national security.” The investment will modernize critical steelmaking operations at Middletown Works by rebuilding and upgrading the plant’s main coal-fired ironmaking furnace, deploying AI to optimize furnace operations and improve energy efficiency, and building an on-site facility to convert steel mill process gases into electricity. Follow-on investments will turn industrial byproducts into materials for concrete used in regional infrastructure. “This landmark investment at Middletown Works will secure a reliable domestic supply of high-purity steel while protecting thousands of quality jobs in Ohio,” said Assistant Secretary of Energy Audrey Robertson. “DOE is proud to partner with Cleveland-Cliffs to reduce America’s dependence on foreign products

Energy Secretary Keeps Critical Generation Available in Mid-Atlantic

WASHINGTON—U.S. Secretary of Energy Chris Wright today issued an emergency order to address critical grid reliability issues facing the Mid-Atlantic region of the United States. The emergency order directs PJM Interconnection L.L.C. (PJM), in coordination with Constellation Energy Corporation, to ensure Units 3 and 4 of the Eddystone Generating Station in Pennsylvania remain available to operate and to employ economic dispatch to minimize costs for the American people. The units were originally slated to shut down on May 31, 2025. “The energy sources that perform when you need them most are the most valuable,” Secretary Wright said. “During recent Mid-Atlantic heat waves, coal, natural gas, and nuclear kept the lights and air conditioners on. President Trump and the Energy Department are committed to keeping critical generation available when demand is highest, reducing the risk of blackouts and ensuring Americans have affordable, reliable, and secure power—regardless of whether the wind is blowing or the sun is shining.” As outlined in DOE’s Resource Adequacy Report, power outages could increase by 100 times in 2030 if the U.S. continues to take reliable power offline. This order is in effect beginning on August 23, 2026, through November 20, 2026. ###

Energy Department Announces $500 Million to Secure America’s Critical Mineral and Battery Supply Chains

WASHINGTON—The U.S. Department of Energy’s (DOE) Office of Critical Minerals and Energy Innovation (CMEI) today announced $500 million for seven selected projects to expand critical mineral and material processing, battery manufacturing, and recycling capacity in the United States. In accordance with President Trump’s Executive Order, Unleashing American Energy, the selected projects advance the President’s agenda to strengthen America’s domestic critical minerals and materials supply chains, reduce reliance on foreign sources, bolster national security, and advance American energy dominance. “For too long, America has depended on foreign actors for critical materials essential to modern life that underpin our economy, energy security, and national security,” said U.S. Secretary of Energy Chris Wright. “President Trump is reversing that dependence by securing our critical supply chains, unleashing American industry, and bringing critical materials production and processing back to the United States.” “DOE is taking decisive action to secure the critical supply chains necessary to power our nation,” said Assistant Secretary of Energy Audrey Robertson. “These projects underscore DOE’s commitment to driving innovation, reducing reliance on foreign sources, and promoting American energy dominance.” This is the third round of funding from DOE’s Battery Materials Processing and Battery Manufacturing and Recycling programs, which support battery materials processing, recycling, and manufacturing projects. These include demonstration projects, construction of commercial-scale facilities, and retrofitting or retooling existing facilities. Critical minerals and materials are essential to American industry, energy production, and national security. Expanding domestic capacity will help ensure the resources America needs are processed, manufactured, and recycled in the United States. Information on the selected projects is available here and here.

bp lets Shah Deniz compression automation contract

bp has let a contract to Emerson to deliver automation technologies for the Shah Deniz Compression project offshore Azerbaijan. Emerson will provide integrated control and safety systems aimed at enhancing production, safety, and reliability on the new offshore compression platform. The contract includes systems to provide process control, safety shutdown, fire and gas detection, and power management. Together, these systems deliver real-time visibility and remote control of critical operations, Emerson said. The $2.9 billion Shah Deniz Compression project, which includes an electrically powered, normally unattended offshore production platform, is a next stage development of the Caspian Sea Shah Deniz field. Designed to access low-pressure gas reserves and maximize overall recovery, the platform will be equipped with four 11 Mw compressors and serve as the central compression hub for gas from the Shah Deniz Alpha and Bravo platforms. The platform will operate remotely from bp’s onshore Sangachal terminal 55 km south of Baku. The project is expected to enable about 50 billion cu m of additional gas and about 25 million bbl of condensate production and export. Construction is scheduled to be completed in 2029, with first gas compression expected from the Shah Deniz Alpha platform in 2029 and from the Shah Deniz Bravo platform in 2030. The agreement follows a previous automation contract bp signed with Emerson for the Azeri Central East and Shah Deniz Stage 2 developments. bp is operator at Shah Deniz (29.99%) with partners Lukoil (19.99%), TPAO (19%), Cenub Qaz Dehlizi (16.02%), NICO (10%), and MVM (5%).

Federal court voids Texas GulfLink license over agency’s ‘serious procedural errors’

The ruling voids the license, halting all construction or progress. Sentinel Midstream declined comment on the ruling and would not answer questions about the status of construction. GulfLink, sited about 30 miles offshore Freeport, Tex., is designed to export up to 1 million b/d via Very Large Crude Carriers (VLCCs) to the government of Japan and Freeport Commodities. The project involves a 44-mile, 42-in. OD pipeline and was scheduled to begin operations around 2028. The estimated $2.1 billion investment was funded as part of a broader trade agreement between the US and Japan. The legal battle stems from a specific rule in the Deepwater Port Act of 1974 that dictates that the federal government can only permit one crude oil deepwater port, including any supporting infrastructure, within a single designated “application area.” Because the competing SPOT project’s pipeline route physically overlaps and intersects GulfLink’s lines, the plaintiff—Citizens for Clean Air & Clean Water in Brazoria County (Better Brazoria), represented by Earthjustice—successfully argued that MARAD violated the “one port” rule when issuing GulfLink’s license in February. The three-judge panel found that MARAD “improperly drew” the map designing the project’s official boundaries to exclude the pipelines and approved two overlapping projects in the same zone instead of only licensing one. The court wrote that the scope of the error made vacatur, not the less serious remand without vacatur, the appropriate remedy. Vacatur deems the license invalid and is used when the court finds “serious procedural errors” that cannot be easily explained or fixed with minor changes. Remand without vacatur sends the decision back to the agency for corrections but leaves the current license in place in the meantime. SPOT project status The $2.5-3-billion SPOT project, developed by Enterprise Products Partners in partnership with Enbridge Inc., also lies about 30 miles from Freeport. Designed to handle VLCCs,

IBM unveils dual-architecture processor to run Arm-native apps on Z mainframes

“These caches have enormously low latency, and that is one of the key reasons and key engineering choices to support the performance and scalability of enterprise workloads, very data-intensive workloads like databases and transactions,” Jacobi said. “In addition, we have an on-chip data processing unit for IO acceleration and dedicated AI accelerators as well as accelerators for data compression, cryptography and data sorting.” One of the biggest takeaways from this processor announcement is that the enormous catalog of software already built for Arm becomes accessible on a mainframe without anyone having to port it first, notes Matt Kimball, senior datacenter analyst at Moor Insights & Strategy, in a research note about the news. Still, “this is a 2027 conversation, and with no date, supported software list, or Arm licensing treatment, the work now is inventory and scenario planning rather than financial modeling,” Kimball wrote.

PJM’s New Data Center Power Equation

PJM Interconnection has now filed one of the most consequential proposed changes yet in the relationship between data centers and the electric grid. Rather than simply treating a new hyperscale or AI facility like any other customer whose demand will be backed through regional capacity procurement, PJM is proposing a framework under which the largest new loads would need to be supported by new capacity, have their needs covered through the Reliability Backstop Procurement, or face potential curtailment when the regional power system is short of supply. The approach has been developing since PJM launched its Critical Issue Fast Path process for large loads in 2025, but it became substantially more concrete in late July and August 2026. PJM filed its proposed Reliability Backstop Procurement with FERC on July 31 and began accepting applications that day for its FERC-approved Expedited Interconnection Track. On Aug. 13, PJM filed its proposed Interim Resource Adequacy Service, or IRAS, along with the Large Load Registry that would support it. The immediate numbers explain the urgency. PJM’s July 2026 capacity auction for the 2028/2029 delivery year procured 138,318 MW of unforced capacity through the centralized auction. Even after including Fixed Resource Requirement resources, however, PJM came up 6,831 MW short of its reliability requirement. The auction cleared at the FERC-approved $325/MW-day price cap. It was the second consecutive auction in which the PJM region failed to procure its full reliability requirement, something that had not happened before these two auctions. That gap is occurring while demand continues to accelerate. PJM’s 2026 long-term forecast projects summer peak demand growing at an average 3.6% annually over the next decade, compared with just 0.3% in the comparable forecast issued in 2021. Summer peak demand is projected to rise by nearly 66 GW over 10 years. Data centers are

Zayo, NVIDIA Build the Long-Haul Backbone for Distributed AI

The data center industry’s increasingly power-first approach to site selection has created a follow-on question: Once the megawatts are found, is there enough network infrastructure to make the site useful at AI scale? Zayo and NVIDIA are putting real infrastructure behind that question. Zayo said it is working with NVIDIA to expand network capacity supporting AI factories across North America, including an 8,000-route-mile program targeting some of the fastest-growing AI corridors in the United States. The project encompasses six new long-haul routes along with overbuilds of existing network across 10 high-demand corridors. The announcement arrives as AI data center development moves beyond the largest established hubs toward markets where power and land may be more readily available, but fiber capacity cannot necessarily be taken for granted. That geography is increasingly important. NVIDIA has separately developed “scale-across” networking technology designed to allow AI infrastructure distributed among different buildings — or even data centers separated by hundreds of kilometers — to operate as a more unified computing environment. Put together, the developments suggest that networking is becoming inseparable from the AI factory buildout itself. Power may determine where the next generation of AI infrastructure can be built. Fiber will increasingly determine how effectively those sites can participate in the larger AI ecosystem. Fiber Follows the Power Zayo CEO Steve Smith said AI demand is changing both where network infrastructure is needed and how aggressively capacity must be deployed ahead of development. “AI is fundamentally reshaping where and how network infrastructure needs to be built across the U.S.,” Smith said. The company’s 8,000-mile program is more nuanced than that top-line number might suggest. Zayo disclosed in April that the expansion includes approximately 3,000 route miles across six new long-haul routes, plus more than 5,000 route miles of overbuilds across 10 existing corridors. Zayo

Southern’s 17 GW Pipeline Puts AI Power Demand Into Utility Math

The headline number from Southern Company’s latest earnings report is hard to miss: electricity use by data centers across the utility’s system increased 55% in the second quarter compared with a year earlier. But the more consequential numbers may be the ones sitting behind it. Southern now has more than 1.2 GW of operating data center load, up by more than 500 MW from a year ago. At the same time, its electric utilities have signed contracts and large-load agreements totaling more than 17 GW by the mid-2030s, with another 8 GW in late-stage development and a prospective pipeline of large industrial and data center projects exceeding 75 GW. That leaves an enormous gap between the data center megawatts consuming electricity today and the load Southern has contractually positioned itself to serve during the next decade. For the data center industry, that gap may be the most important part of Southern’s second-quarter story. It offers a look at how utilities are beginning to convert the AI infrastructure boom from forecasts and campus announcements into contracts, generation procurement, transmission investment and eventually energized capacity. From Contracts to Megawatts Southern added roughly 6 GW of contracted large load during the quarter alone. Alabama Power signed three projects representing about 3 GW, while Georgia Power reached a 25-year agreement to serve OpenAI’s planned project in Effingham County near Savannah. That facility is expected to require approximately 3.2 GW and begin taking electric service in phases in 2028. The numbers nevertheless require an important distinction. Seventeen gigawatts contracted does not mean 17 GW will suddenly appear on Southern’s grid. Large data center campuses ramp gradually, often over several years, and Southern executives acknowledged that actual customer ramp schedules do not always match the assumptions made when projects are first approved. CEO Chris Womack said

PORTS-Pike Takes Shape as an 8-GW AI Infrastructure Model

Back on March 31, 2026, we discussed we discussed SoftBank and SB Energy’s plans to redevelop the former Portsmouth Gaseous Diffusion Plant site near Piketon as a 10-GW artificial intelligence data center campus supported by almost an equal amount of new power generation. At the time, the plan called for as much as 10 GW of new generation, including 9.2 GW of natural gas capacity, along with approximately $4.2 billion of high-voltage transmission infrastructure developed with AEP Ohio. An initial 800-MW data center phase was targeted for service in 2028. The March story was notable because Pike County appeared to offer a preview of a new model for building hyperscale infrastructure: develop the generation, transmission and data center simultaneously rather than wait for an increasingly congested regional grid to deliver multiple gigawatts of capacity. Not to mention the reuse of a brownfield site with the encouragement of the federal government. Since then, almost every important part of the project has moved forward, and on August 17, the most consequential missing pieces fell into place. NVIDIA announced that it will become the exclusive AI compute infrastructure provider for the PORTS-Pike Technology Campus. OpenAI will be the data center customer, signing a 20-year lease with SB Energy for approximately 8 GW of IT capacity. NVIDIA will invest another $1.5 billion in SB Energy and provide credit support for the land, power and shell infrastructure behind an initial 4.25 GW of IT load, with an option covering approximately another 3.75 GW. The Securities and Exchange Commission filing accompanying the announcement makes the financial commitment even more significant. NVIDIA disclosed that its aggregate payment obligation associated with its initial commitment is capped at $105 billion. That is not a conventional capital commitment to spend $105 billion building the campus, nor is it simply a

Nvidia scales back financing guarantee for OpenAI data center

Nvidia is scaling back a proposed financial guarantee tied to a massive OpenAI data center project in Ohio, reducing its initial commitment from as much as $250 billion to less than $120 billion, according to report in the Wall Street Journal. Earlier this month, Nvidia announced partnerships with major financial firms including Apollo Global Management, BlackRock, Blackstone, Brookfield Asset Management, Goldman Sachs and KKR, aimed at mobilizing more than $500 billion in capital for AI computing infrastructure. The change represents a significant restructuring of Nvidia’s role in financing the planned facility, which is being developed by SB Energy, a subsidiary of SoftBank. Under the revised arrangement, Nvidia would guarantee financing for the project’s first phase, representing roughly 5 gigawatts of capacity, or half of the total proposed capacity. Financing for the remaining capacity would be considered separately at a later stage.

Microsoft will invest $80B in AI data centers in fiscal 2025

And Microsoft isn’t the only one that is ramping up its investments into AI-enabled data centers. Rival cloud service providers are all investing in either upgrading or opening new data centers to capture a larger chunk of business from developers and users of large language models (LLMs). In a report published in October 2024, Bloomberg Intelligence estimated that demand for generative AI would push Microsoft, AWS, Google, Oracle, Meta, and Apple would between them devote $200 billion to capex in 2025, up from $110 billion in 2023. Microsoft is one of the biggest spenders, followed closely by Google and AWS, Bloomberg Intelligence said. Its estimate of Microsoft’s capital spending on AI, at $62.4 billion for calendar 2025, is lower than Smith’s claim that the company will invest $80 billion in the fiscal year to June 30, 2025. Both figures, though, are way higher than Microsoft’s 2020 capital expenditure of “just” $17.6 billion. The majority of the increased spending is tied to cloud services and the expansion of AI infrastructure needed to provide compute capacity for OpenAI workloads. Separately, last October Amazon CEO Andy Jassy said his company planned total capex spend of $75 billion in 2024 and even more in 2025, with much of it going to AWS, its cloud computing division.

John Deere unveils more autonomous farm machines to address skill labor shortage

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Self-driving tractors might be the path to self-driving cars. John Deere has revealed a new line of autonomous machines and tech across agriculture, construction and commercial landscaping. The Moline, Illinois-based John Deere has been in business for 187 years, yet it’s been a regular as a non-tech company showing off technology at the big tech trade show in Las Vegas and is back at CES 2025 with more autonomous tractors and other vehicles. This is not something we usually cover, but John Deere has a lot of data that is interesting in the big picture of tech. The message from the company is that there aren’t enough skilled farm laborers to do the work that its customers need. It’s been a challenge for most of the last two decades, said Jahmy Hindman, CTO at John Deere, in a briefing. Much of the tech will come this fall and after that. He noted that the average farmer in the U.S. is over 58 and works 12 to 18 hours a day to grow food for us. And he said the American Farm Bureau Federation estimates there are roughly 2.4 million farm jobs that need to be filled annually; and the agricultural work force continues to shrink. (This is my hint to the anti-immigration crowd). John Deere’s autonomous 9RX Tractor. Farmers can oversee it using an app. While each of these industries experiences their own set of challenges, a commonality across all is skilled labor availability. In construction, about 80% percent of contractors struggle to find skilled labor. And in commercial landscaping, 86% of landscaping business owners can’t find labor to fill open positions, he said. “They have to figure out how to do

2025 playbook for enterprise AI success, from agents to evals

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More 2025 is poised to be a pivotal year for enterprise AI. The past year has seen rapid innovation, and this year will see the same. This has made it more critical than ever to revisit your AI strategy to stay competitive and create value for your customers. From scaling AI agents to optimizing costs, here are the five critical areas enterprises should prioritize for their AI strategy this year. 1. Agents: the next generation of automation AI agents are no longer theoretical. In 2025, they’re indispensable tools for enterprises looking to streamline operations and enhance customer interactions. Unlike traditional software, agents powered by large language models (LLMs) can make nuanced decisions, navigate complex multi-step tasks, and integrate seamlessly with tools and APIs. At the start of 2024, agents were not ready for prime time, making frustrating mistakes like hallucinating URLs. They started getting better as frontier large language models themselves improved. “Let me put it this way,” said Sam Witteveen, cofounder of Red Dragon, a company that develops agents for companies, and that recently reviewed the 48 agents it built last year. “Interestingly, the ones that we built at the start of the year, a lot of those worked way better at the end of the year just because the models got better.” Witteveen shared this in the video podcast we filmed to discuss these five big trends in detail. Models are getting better and hallucinating less, and they’re also being trained to do agentic tasks. Another feature that the model providers are researching is a way to use the LLM as a judge, and as models get cheaper (something we’ll cover below), companies can use three or more models to

OpenAI’s red teaming innovations define new essentials for security leaders in the AI era

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More OpenAI has taken a more aggressive approach to red teaming than its AI competitors, demonstrating its security teams’ advanced capabilities in two areas: multi-step reinforcement and external red teaming. OpenAI recently released two papers that set a new competitive standard for improving the quality, reliability and safety of AI models in these two techniques and more. The first paper, “OpenAI’s Approach to External Red Teaming for AI Models and Systems,” reports that specialized teams outside the company have proven effective in uncovering vulnerabilities that might otherwise have made it into a released model because in-house testing techniques may have missed them. In the second paper, “Diverse and Effective Red Teaming with Auto-Generated Rewards and Multi-Step Reinforcement Learning,” OpenAI introduces an automated framework that relies on iterative reinforcement learning to generate a broad spectrum of novel, wide-ranging attacks. Going all-in on red teaming pays practical, competitive dividends It’s encouraging to see competitive intensity in red teaming growing among AI companies. When Anthropic released its AI red team guidelines in June of last year, it joined AI providers including Google, Microsoft, Nvidia, OpenAI, and even the U.S.’s National Institute of Standards and Technology (NIST), which all had released red teaming frameworks. Investing heavily in red teaming yields tangible benefits for security leaders in any organization. OpenAI’s paper on external red teaming provides a detailed analysis of how the company strives to create specialized external teams that include cybersecurity and subject matter experts. The goal is to see if knowledgeable external teams can defeat models’ security perimeters and find gaps in their security, biases and controls that prompt-based testing couldn’t find. What makes OpenAI’s recent papers noteworthy is how well they define using human-in-the-middle