FACTS Benchmark Suite: Systematically evaluating the factuality of large language models

Stay Ahead, Stay ONMINE

FACTS Benchmark Suite: Systematically evaluating the factuality of large language models

Large language models (LLMs) are increasingly becoming a primary source for information delivery across diverse use cases, so it’s important that their responses are factually accurate.In order to continue improving their performance on this industry-wide challenge, we have to better understand the types of use cases where models struggle to provide an accurate response and better measure factuality performance in those areas.The FACTS Benchmark SuiteToday, we’re teaming up with Kaggle to introduce the FACTS Benchmark Suite. It extends our previous work developing the FACTS Grounding Benchmark, with three additional factuality benchmarks, including:A Parametric Benchmark that measures the model’s ability to access its internal knowledge accurately in factoid question use-cases.A Search Benchmark that tests a model’s ability to use Search as a tool to retrieve information and synthesize it correctly.A Multimodal Benchmark that tests a model’s ability to answer prompts related to input images in a factually correct manner.We are also updating the original FACTS grounding benchmark with Grounding Benchmark – v2, an extended benchmark to test a model’s ability to provide answers grounded in the context of a given prompt.Each benchmark was carefully curated to produce a total of 3,513 examples, which we are making publicly available today. Similar to our previous release, we are following standard industry practice and keeping an evaluation set held-out as a private set. The FACTS Benchmark Suite Score (or FACTS Score) is calculated as the average accuracy of both public and private sets across the four benchmarks. Kaggle will oversee the management of the FACTS Benchmark Suite. This includes owning the private held-out sets, testing the leading LLMs on the benchmarks, and hosting the results on a public leaderboard. More details about the FACTS evaluation methodology can be found in our tech report.Benchmark overviewParametric BenchmarkThe FACTS Parametric benchmark assesses the ability of models to accurately answer factual questions, without the aid of external tools like web search. All the questions in the benchmark are “trivia style” questions driven by user interest that can be answered via Wikipedia (a standard source for LLM pretraining). The resulting benchmark consists of a 1052-item public set and a 1052-item private set.

Large language models (LLMs) are increasingly becoming a primary source for information delivery across diverse use cases, so it’s important that their responses are factually accurate.

In order to continue improving their performance on this industry-wide challenge, we have to better understand the types of use cases where models struggle to provide an accurate response and better measure factuality performance in those areas.

The FACTS Benchmark Suite

Today, we’re teaming up with Kaggle to introduce the FACTS Benchmark Suite. It extends our previous work developing the FACTS Grounding Benchmark, with three additional factuality benchmarks, including:

A Parametric Benchmark that measures the model’s ability to access its internal knowledge accurately in factoid question use-cases.
A Search Benchmark that tests a model’s ability to use Search as a tool to retrieve information and synthesize it correctly.
A Multimodal Benchmark that tests a model’s ability to answer prompts related to input images in a factually correct manner.

We are also updating the original FACTS grounding benchmark with Grounding Benchmark – v2, an extended benchmark to test a model’s ability to provide answers grounded in the context of a given prompt.

Each benchmark was carefully curated to produce a total of 3,513 examples, which we are making publicly available today. Similar to our previous release, we are following standard industry practice and keeping an evaluation set held-out as a private set. The FACTS Benchmark Suite Score (or FACTS Score) is calculated as the average accuracy of both public and private sets across the four benchmarks. Kaggle will oversee the management of the FACTS Benchmark Suite. This includes owning the private held-out sets, testing the leading LLMs on the benchmarks, and hosting the results on a public leaderboard. More details about the FACTS evaluation methodology can be found in our tech report.

Benchmark overview

Parametric Benchmark

The FACTS Parametric benchmark assesses the ability of models to accurately answer factual questions, without the aid of external tools like web search. All the questions in the benchmark are “trivia style” questions driven by user interest that can be answered via Wikipedia (a standard source for LLM pretraining). The resulting benchmark consists of a 1052-item public set and a 1052-item private set.

Stay Ahead

Explore More Insights

Stay ahead with more perspectives on cutting-edge power, infrastructure, energy, bitcoin and AI solutions. Explore these articles to uncover strategies and insights shaping the future of industries.

Cybersecurity skills matter more than headcount in an AI era: ISC2 study

Organizations have experienced oversights in cybersecurity processes and procedures (26%), been forced to put underqualified or inexperienced people into roles to cover them (25%), are lacking the time or resources to train cybersecurity staff (25%), and are dealing with misconfigured systems (24%), according to this year’s study. The report also

TotalEnergies farms out 40% participating interest in certain licenses offshore Nigeria to Chevron

@import url(‘https://fonts.googleapis.com/css2?family=Inter:[email protected]&display=swap’); a { color: var(–color-primary-main); } .ebm-page__main h1, .ebm-page__main h2, .ebm-page__main h3, .ebm-page__main h4, .ebm-page__main h5, .ebm-page__main h6 { font-family: Inter; } body { line-height: 150%; letter-spacing: 0.025em; font-family: Inter; } button, .ebm-button-wrapper { font-family: Inter; } .label-style { text-transform: uppercase; color: var(–color-grey); font-weight: 600; font-size: 0.75rem; } .caption-style

AI-driven network management gains enterprise trust

The way the full process works is that the raw data feed comes in, and machine learning is used to identify an anomaly that could be a possible incident. That’s where the generative AI agents step up. In addition to the history of similar issues, the agents also look for

Short memory supply forces Micron to abandon consumer market, prioritize enterprise

Along with that, they take a good hard look at how they are using their scarce resources, checking to see that they are selling into the most profitable markets and asking if they should be redirecting their output to more profitable businesses, Handy said. That’s what happened to Crucial. If

Exxon CFO to Retire

Kathy Mikells, the first outsider to join Exxon Mobil Corp.’s inner circle of top executives, will retire next year as she battles a serious but non-life threatening health condition. Mikells, 60, joined Exxon in 2021 when the Texas oil giant was under pressure from shareholders to improve financial performance, diversify its leadership and build a low-carbon business. She will retire on Feb. 1 to focus on her health, Chief Executive Officer Darren Woods said in a presentation to investors Tuesday. She will be replaced by Neil Hansen, a 25-year Exxon veteran who leads the company’s global business solutions division. “In recent months Kathy has undergone a series of procedures and surgeries to address a debilitating but thankfully non-life threatening health issue,” Woods said. “While her condition has improved, it has become clear to her, and I think the rest of us, that she needs to focus fully on her recovery.” Mikells came to Exxon from beverage titan Diageo Plc and became the first female senior vice president on the company’s management committee, which oversees day-to-day operations and sets strategy. She was also the first executive to hold such a high position at Exxon without a background in oil, natural gas or chemicals. “Things are absolutely getting better, but it has been slow going, and I still have a lot of work to do with my doctors to one day get back to my usual self,” Mikells said. “I know my colleagues on the call today will recognize the sincerity of my disappointment in needing to leave this great company.” Mikells modernized Exxon’s finance function and was the first manager to hold the formal title of chief financial officer. She overhauled investor communications and provided more granular, forward-looking information to analysts and the market in general. Hansen has been prepared for

Eni Announces ‘Significant’ Find Offshore Indonesia

Eni announced, in a statement sent to Rigzone recently, a “significant gas discovery” in the Konta-1 exploration well off the coast of East Kalimantan in Indonesia. “Estimates indicate 600 billion cubic feet of gas initially in place (GIIP) with a potential upside beyond one trillion cubic feet,” Eni said in the statement. The Konta-1 discovery is situated in the Muara Bakau PSC, Eni highlighted in the statement, pointing out that this is operated by the company with an 88.334 percent participating interest. Saka Energi holds the remaining 11.666 percent stake. “Konta-1 was drilled to a depth of 4,575 meters [15,009 feet] in 570 meters [1,870 feet] water depth, encountering gas in four separate sandstone reservoirs of Miocene age with good petrophysical properties that have been subject to an extensive data acquisition campaign,” Eni said in the statement. “A well production test (DST) has been successfully performed in one of the reservoirs and it flowed up to 31 million standard cubic feet per day of gas and approximately 700 barrels per day of condensate,” Eni added. “Based on the DST results the well has an estimated potential for a multi-pool gas rate of up to 80 million standard cubic feet per day of gas and about 1,600 barrels per day of condensate,” it continued. Eni noted in the statement that preliminary estimates indicate a discovered volume of 600 billion cubic feet of gas in place in the four reservoirs hit by the well trajectory. “Additional reservoir segments in the Konta Prospect area, not penetrated by the well, but with similar gas signature, may bring the overall volumes beyond one trillion cubic feet GIIP,” it added. Eni highlighted in the statement that the Konta discovery is sitting nearby existing facilities and adjacent to existing discoveries, “providing significant synergies for the development”. The

Phase 1 of Varco’s Sizing John BESS in Liverpool Reaches Full Operation

Varco Energy and Fluence Energy Inc said the first phase of their Sizing John Battery Energy Storage System (BESS) in the United Kingdom is now in full commercial operation and has entered phase 2. “The 57 MW/137.5 MWh project is located within the Mersey Ring east of Liverpool, a region known for its acute grid constraints”, a joint statement said. Sizing John Phase 1 has “one of the longest durations [2.4 hours] of any operational battery project in the UK”, the companies said. “Its longer duration will help lower overall energy costs by addressing growing congestion in the region and reducing price volatility created by rising renewable generation”. Phase 2, which will continue to involve intelligent energy storage and optimization software provider Fluence, will add 85.5 MW/201 MWh. The partners expect to start up phase 2 next year. “On energization of phase 2, the Sizing John project will rank among the largest battery energy storage systems in the UK”, the companies said. “Phase 2 will incorporate the Fluence-supplied next-generation Gridstack Pro 5000 with advanced grid-forming capabilities, providing critical support to the UK’s grid by actively regulating voltage and frequency, providing essential regional grid stability”, they said. Varco director Richard Whitmore said, “The addition of grid-forming capabilities will set a new standard for regional grid support, especially in the wake of recent Iberian Peninsula blackout”. Brian Perusse, Fluence managing director for the UK and Ireland, said, “Sizing John is a key step in bringing longer-duration storage and additional grid-forming capabilities to the UK, technologies that play a vital role in improving system resilience, unlocking greater renewable integration and reducing costs to consumers”. Sizing John is the second project by Varco, a BESS asset owner and operator backed by the Adaptogen Capital Battery Storage Fund, to start operation, according to the statement. Native

Kinder Morgan Expects to Ride on LNG, Power Demand Growth

Kinder Morgan Inc has announced adjusted earnings per share (EPS) guidance of $1.37 for 2026, with the North American pipeline operator encouraged by growth in the liquefied natural gas (LNG) and power sectors. That is an increase of eight percent versus its adjusted EPS forecast for 2025. For 2026 adjusted earnings before interest, taxes, depreciation and amortization (EBITDA), Kinder Morgan expects $8.7 billion, up four percent compared to its guidance for 2025. The outlook reflects “continued execution on expansion projects in our natural gas pipelines business segment”, chief executive Kim Dang said in an online statement. “We are projecting an annualized dividend of $1.19 for 2026, marking the ninth consecutive year of dividend increases”, Dang added. “Our year-end 2026 net debt-to-adjusted EBITDA ratio is forecast at 3.8 times, remaining at the low end of our 3.5x-4.5x target range and preserving flexibility for opportunistic investments”. The Houston, Texas-based owner of oil and gas pipelines and terminals, which also produces oil and renewable natural gas, plans nearly $3.4 billion in discretionary capital next year, “substantially funded from internally generated cash flow”. Kinder Morgan president Tom Martin said, “We expect to continue benefiting from strong natural gas market fundamentals, supporting growth on our existing transportation and storage assets and creating expansion opportunities”. For the first nine months of 2025, Kinder Morgan recorded $0.91 in EPS adjusted for nonrecurring items. That was up 10 percent from the same period in 2024, according to its third-quarter report October 22. Adjusted EBITDA for January-September 2025 totaled $6.12 billion, up four percent year-on-year. Volumes transported via its gas and liquid pipelines rose year-over-year. Gas transport volumes exceeded 46 trillion British thermal units a day, while it delivered 2.12 million barrels per day of liquids (crude oil, condensate and refined products). Revenue totaled $12.43 billion, up from $11.11 billion for the

Gas Deals Take Center Stage in Q4

After being overshadowed by oil focused transactions, gas deals have taken center stage in the final quarter of 2025 as strong current pricing and a bullish outlook for the commodity motivates buyers. That’s what Andrew Dittmar, Principal Analyst at Enverus Intelligence Research (EIR), said in a statement sent to Rigzone recently, adding that “the latest round of deals include Antero Resources acquiring West Virginia producer HG Energy’s upstream assets for $2.8 billion plus the purchase of its midstream infrastructure by Antero Midstream for $1.1 billion”. “Concurrently Antero and Antero Midstream are divesting their Ohio Utica position to a partnership of Infinity Natural Resources and Northern Oil and Gas [NOG] for a total of $1.2 billion, with the upstream portion garnering $800 million and the balance from midstream,” Dittmar continued. In the statement, Dittmar said the deals “have an obvious strategic rationale for Antero as the company blocks up its core operating region in West Virginia and adds the second largest private E&P in the Marcellus by remaining inventory”. “HG’s more than 400 remaining locations compliment Antero’s legacy position with comparable quality. The acquisition of HG by Antero had a sense of inevitability given their relative positions within the play and likely just needed the right time and commodity price environment for the two companies to come together,” he added. “Along with the strategic fit between leasehold positions, the deal offers the opportunity to add further midstream infrastructure to Antero Midstream’s holdings,” he continued. Dittmar highlighted in the statement that Antero “says it has identified $950 million in cumulative synergies (P-10 over 10 years) with more than half coming from drilling and completion savings and development optimization including longer laterals”, noting that “that is the type of strategic fit that investors want to see in acquisitions”. Looking at HG “and their

Kinder Morgan Ups Profit Growth Projection for 2026

Kinder Morgan Inc has bumped up its adjusted earnings per share (EPS) guidance for next year to $1.37, with the North American pipeline operator encouraged by growth in the liquefied natural gas (LNG) and power sectors. That is an increase of eight percent from its forecast in the third quarter. For 2026 adjusted earnings before interest, taxes, depreciation and amortization (EBITDA), Kinder Morgan now expects $8.7 billion, up four percent versus its Q3 guidance. The upward revisions reflect “continued execution on expansion projects in our natural gas pipelines business segment”, chief executive Kim Dang said in an online statement. “We are projecting an annualized dividend of $1.19 for 2026, marking the ninth consecutive year of dividend increases”, Dang added. “Our year-end 2026 net debt-to-adjusted EBITDA ratio is forecast at 3.8 times, remaining at the low end of our 3.5x-4.5x target range and preserving flexibility for opportunistic investments”. The Houston, Texas-based owner of oil and gas pipelines and terminals, which also produces oil and renewable natural gas, plans nearly $3.4 billion in discretionary capital next year, “substantially funded from internally generated cash flow”. Kinder Morgan president Tom Martin said, “We expect to continue benefiting from strong natural gas market fundamentals, supporting growth on our existing transportation and storage assets and creating expansion opportunities”. For the first nine months of 2025, Kinder Morgan recorded $0.91 in EPS adjusted for nonrecurring items. That was up 10 percent from the same period in 2024, according to its third-quarter report October 22. Adjusted EBITDA for January-September 2025 totaled $6.12 billion, up four percent year-on-year. Volumes transported via its gas and liquid pipelines rose year-over-year. Gas transport volumes exceeded 46 trillion British thermal units a day, while it delivered 2.12 million barrels per day of liquids (crude oil, condensate and refined products). Revenue totaled $12.43 billion,

Most significant networking acquisitions of 2025

Cisco makes two AI deals: EzDubs and NeuralFabric Last month Cisco completed its acquisition of EzDubs, a privately held AI software company with speech-to-speech translation technology. EzDubs translates conversations across 31 languages and will accelerate Cisco’s delivery of next-generation features, such as live voice translation that preserves the characteristics of speech, the vendor stated. Cisco plans to incorporate EzDubs’ technology in its Cisco Collaboration portfolio. Also in November, Cisco bought AI platform company NeuralFabric, which offers a generative AI platform that lets organizations develop domain-specific small language models using their own proprietary data. Coreweave buys Core Scientific Nvidia-backed AI cloud provider CoreWeave acquired crypto miner Core Scientific for about $9 billion, giving it access to 1.3 gigawatts of contracted power to support growing demand for AI and high-performance computing workloads. CoreWeave said the deal augments its vertical integration by expanding its owned and operated data center footprint, allowing it to scale GPU-powered services for enterprise and research customers. F5 picks up three: CalypsoAI, Fletch and MantisNet F5 acquired Dublin, Ireland-based CalypsoAI for $180 million. CalypsoAI’s platform creates what the company calls an Inference Perimeter that protects across models, vendors, and environments. F5 says it will integrate CalypsoAI’s adaptive AI security capabilities into its F5 Application Delivery and Security Platform (ADSP). F5’s ADSP also stands to gain from F5’s acquisition of agentic AI and threat management startup Fletch. Fletch’s technology turns external threat intelligence and internal logs into real-time, prioritized insights; its agentic AI capabilities will be integrated into ADSP, according to F5. Lastly, F5 grabbed startup MantisNet to enhance cloud-native observability in F5’s ADSP. MantisNet leverages extended Berkeley Packet Filer (eBPF)-powered, kernel-level telemetry to provide real-time insights into encrypted protocol activity and allow organizations “to gain visibility into even the most elusive traffic, all without performance overhead,” according to an F5 blog

Aviz Networks launches enterprise-grade community SONiC distribution

First, the company enabled FRR (Free Range Routing) features that exist in the community code but aren’t consistently implemented across different ASICs. VRRP (Virtual Router Redudancy Protocol) provides router redundancy for high availability. Spanning tree variants prevent network loops in layer 2 topologies. MLAG allows two switches to act as a single logical device for link aggregation. EVPN enhancements support layer 2 and layer 3 VPN services over VXLAN overlays. These protocols work differently depending on the underlying silicon, so Aviz normalized their implementation across Broadcom, Nvidia, Cisco and Marvell chips. Second, Aviz fixed bugs discovered in production deployments. One customer deployed community SONiC with OpenStack and started migrating virtual machines between hosts. The network fabric couldn’t handle the workload and broke. Aviz identified the failure modes and patched them. Third, Aviz built a software component that normalizes monitoring data across vendors. Broadcom’s Tomahawk ASIC generates different telemetry formats than Nvidia’s Spectrum or Cisco’s Silicon One. Network operators need consistent data for troubleshooting and capacity planning. The software collects ASIC-specific logs and network operating system telemetry, then translates them into a standardized format that works the same way regardless of which silicon vendor’s chips are running in the switches. Validated for enterprise deployment scenarios The distribution supports common enterprise network architectures. IP CLOS provides the leaf-spine topology used in modern data centers for predictable latency and scalability. EVPN/VXLAN creates layer 2 and layer 3 overlay networks that span physical network boundaries. MLAG configurations provide link redundancy without spanning tree limitations. Aviz provides validated runbooks for these deployments across data center, edge and AI fabric use cases.

US approves Nvidia H200 exports to China, raising questions about enterprise GPU supply

Shifting demand scenarios What remains unclear is how much demand Chinese firms will actually generate, given Beijing’s recent efforts to steer its tech companies away from US chips. Charlie Dai, VP and principal analyst at Forrester, said renewed H200 access is likely to have only a modest impact on global supply, as China is prioritizing domestic AI chips and the H200 remains below Nvidia’s latest Blackwell-class systems in performance and appeal. “While some allocation pressure may emerge, most enterprise customers outside China will see minimal disruption in pricing or lead times over the next few quarters,” Dai added. Neil Shah, VP for research and partner at Counterpoint Research, agreed that demand may not surge, citing structural shifts in China’s AI ecosystem. “The Chinese ecosystem is catching up fast, from semi to stack, with models optimized on the silicon and software,” Shah said. Chinese enterprises might think twice before adopting a US AI server stack, he said. Others caution that even selective demand from China could tighten global allocation at a time when supply of high-end accelerators remains stretched, and data center deployments continue to rise.

What does Arm need to do to gain enterprise acceptance?

But in 2017, AMD released the Zen architecture, which was equal if not superior to the Intel architecture. Zen made AMD competitive, and it fueled an explosive rebirth for a company that was near death a few years prior. AMD now has about 30% market share, while Intel suffers from a loss of technology as well as corporate leadership. Now, customers have a choice of Intel or AMD, and they don’t have to worry about porting their applications to a new platform like they would have to do if they switched to Arm. Analysts weigh in on Arm Tim Crawford sees no demand for Arm in the data center. Crawford is president of AVOA, a CIO consultancy. In his role, he talks to IT professionals all the time, but he’s not hearing much interest in Arm. “I don’t see Arm really making a dent, ever, into the general-purpose processor space,” Crawford said. “I think the opportunity for Arm is special applications and special silicon. If you look at the major cloud providers, their custom silicon is specifically built to do training or optimized to do inference. Arm is kind of in the same situation in the sense that it has to be optimized.” “The problem [for Arm] is that there’s not necessarily a need to fulfill at this point in time,” said Rob Enderle, principal analyst with The Enderle Group. “Obviously, there’s always room for other solutions, but Arm is still going to face the challenge of software compatibility.” And therein lies what may be Arm’s greatest challenge: software compatibility. Software doesn’t care (usually) if it’s on Intel or AMD, because both use the x86 architecture, with some differences in extensions. But Arm is a whole new platform, and that requires porting and testing. Enterprises generally don’t like disruption —

Intel decides to keep networking business after all

That doesn’t explain why Intel made the decision to pursue spin-off in the first place. In July, NEX chief Sachin Katti issued a memo that outlined plans to establish key elements of the Networking and Communications business as a stand-alone company. It looked like a done deal, experts said. Jim Hines, research director for enabling technologies and semiconductors at IDC, declined to speculate on whether Intel could get a decent offer but noted NEX is losing ground. IDC estimates Intel’s market share in overall semiconductors at 6.8% in Q3 2025, which is down from 7.4% for the full year 2024 and 9.2% for the full year 2023. Intel’s course reversal “is a positive for Intel in the long term, and recent improvements in its financial situation may have contributed to the decision to keep NEX in house,” he said. When Tan took over as CEO earlier this year, prioritized strengthening the balance sheet and bringing a greater focus on execution. Divest NEX was aligned with these priorities, but since then, Intel has secured investments from the US Government, Nvidia and SoftBank that have reduced the need to raise cash through other means, Hines notes. “The NEX business will prove to be a strategic asset for Intel as it looks to protect and expand its position in the AI datacenter market. Success in this market now requires processor suppliers to offer a full-stack solution, not just silicon. Scale-up and scale-out networking solutions are a key piece of the package, and Intel will be able to leverage its NEX technologies and software, including silicon photonics, to develop differentiated product offerings in this space,” Hines said.

At the Crossroads of AI and the Edge: Inside 1623 Farnam’s Rising Role as a Midwest Interconnection Powerhouse

That was the thread that carried through our recent conversation for the DCF Show podcast, where Severn walked through the role Farnam now plays in AI-driven networking, multi-cloud connectivity, and the resurgence of regional interconnection as a core part of U.S. digital infrastructure. Aggregation, Not Proximity: The Practical Edge Severn is clear-eyed about what makes the edge work and what doesn’t. The idea that real content delivery could aggregate at the base of cell towers, he noted, has never been realistic. The traffic simply isn’t there. Content goes where the network already concentrates, and the network concentrates where carriers, broadband providers, cloud onramps, and CDNs have amassed critical mass. In Farnam’s case, that density has grown steadily since the building changed hands in 2018. At the time an “underappreciated asset,” the facility has since become a meeting point for more than 40 broadband providers and over 60 carriers, with major content operators and hyperscale platforms routing traffic directly through its MMRs. That aggregation effect feeds on itself; as more carrier and content traffic converges, more participants anchor themselves to the hub, increasing its gravitational pull. Geography only reinforces that position. Located on the 41st parallel, the building sits at the historical shortest-distance path for early transcontinental fiber routes. It also lies at the crossroads of major east–west and north–south paths that have made Omaha a natural meeting point for backhaul routes and hyperscale expansions across the Midwest. AI and the New Interconnection Economy Perhaps the clearest sign of Farnam’s changing role is the sheer volume of fiber entering the building. More than 5,000 new strands are being brought into the property, with another 5,000 strands being added internally within the Meet-Me Rooms in 2025 alone. These are not incremental upgrades—they are hyperscale-grade expansions driven by the demands of AI traffic,

Microsoft will invest $80B in AI data centers in fiscal 2025

And Microsoft isn’t the only one that is ramping up its investments into AI-enabled data centers. Rival cloud service providers are all investing in either upgrading or opening new data centers to capture a larger chunk of business from developers and users of large language models (LLMs). In a report published in October 2024, Bloomberg Intelligence estimated that demand for generative AI would push Microsoft, AWS, Google, Oracle, Meta, and Apple would between them devote $200 billion to capex in 2025, up from $110 billion in 2023. Microsoft is one of the biggest spenders, followed closely by Google and AWS, Bloomberg Intelligence said. Its estimate of Microsoft’s capital spending on AI, at $62.4 billion for calendar 2025, is lower than Smith’s claim that the company will invest $80 billion in the fiscal year to June 30, 2025. Both figures, though, are way higher than Microsoft’s 2020 capital expenditure of “just” $17.6 billion. The majority of the increased spending is tied to cloud services and the expansion of AI infrastructure needed to provide compute capacity for OpenAI workloads. Separately, last October Amazon CEO Andy Jassy said his company planned total capex spend of $75 billion in 2024 and even more in 2025, with much of it going to AWS, its cloud computing division.

John Deere unveils more autonomous farm machines to address skill labor shortage

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Self-driving tractors might be the path to self-driving cars. John Deere has revealed a new line of autonomous machines and tech across agriculture, construction and commercial landscaping. The Moline, Illinois-based John Deere has been in business for 187 years, yet it’s been a regular as a non-tech company showing off technology at the big tech trade show in Las Vegas and is back at CES 2025 with more autonomous tractors and other vehicles. This is not something we usually cover, but John Deere has a lot of data that is interesting in the big picture of tech. The message from the company is that there aren’t enough skilled farm laborers to do the work that its customers need. It’s been a challenge for most of the last two decades, said Jahmy Hindman, CTO at John Deere, in a briefing. Much of the tech will come this fall and after that. He noted that the average farmer in the U.S. is over 58 and works 12 to 18 hours a day to grow food for us. And he said the American Farm Bureau Federation estimates there are roughly 2.4 million farm jobs that need to be filled annually; and the agricultural work force continues to shrink. (This is my hint to the anti-immigration crowd). John Deere’s autonomous 9RX Tractor. Farmers can oversee it using an app. While each of these industries experiences their own set of challenges, a commonality across all is skilled labor availability. In construction, about 80% percent of contractors struggle to find skilled labor. And in commercial landscaping, 86% of landscaping business owners can’t find labor to fill open positions, he said. “They have to figure out how to do

2025 playbook for enterprise AI success, from agents to evals

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More 2025 is poised to be a pivotal year for enterprise AI. The past year has seen rapid innovation, and this year will see the same. This has made it more critical than ever to revisit your AI strategy to stay competitive and create value for your customers. From scaling AI agents to optimizing costs, here are the five critical areas enterprises should prioritize for their AI strategy this year. 1. Agents: the next generation of automation AI agents are no longer theoretical. In 2025, they’re indispensable tools for enterprises looking to streamline operations and enhance customer interactions. Unlike traditional software, agents powered by large language models (LLMs) can make nuanced decisions, navigate complex multi-step tasks, and integrate seamlessly with tools and APIs. At the start of 2024, agents were not ready for prime time, making frustrating mistakes like hallucinating URLs. They started getting better as frontier large language models themselves improved. “Let me put it this way,” said Sam Witteveen, cofounder of Red Dragon, a company that develops agents for companies, and that recently reviewed the 48 agents it built last year. “Interestingly, the ones that we built at the start of the year, a lot of those worked way better at the end of the year just because the models got better.” Witteveen shared this in the video podcast we filmed to discuss these five big trends in detail. Models are getting better and hallucinating less, and they’re also being trained to do agentic tasks. Another feature that the model providers are researching is a way to use the LLM as a judge, and as models get cheaper (something we’ll cover below), companies can use three or more models to

OpenAI’s red teaming innovations define new essentials for security leaders in the AI era

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More OpenAI has taken a more aggressive approach to red teaming than its AI competitors, demonstrating its security teams’ advanced capabilities in two areas: multi-step reinforcement and external red teaming. OpenAI recently released two papers that set a new competitive standard for improving the quality, reliability and safety of AI models in these two techniques and more. The first paper, “OpenAI’s Approach to External Red Teaming for AI Models and Systems,” reports that specialized teams outside the company have proven effective in uncovering vulnerabilities that might otherwise have made it into a released model because in-house testing techniques may have missed them. In the second paper, “Diverse and Effective Red Teaming with Auto-Generated Rewards and Multi-Step Reinforcement Learning,” OpenAI introduces an automated framework that relies on iterative reinforcement learning to generate a broad spectrum of novel, wide-ranging attacks. Going all-in on red teaming pays practical, competitive dividends It’s encouraging to see competitive intensity in red teaming growing among AI companies. When Anthropic released its AI red team guidelines in June of last year, it joined AI providers including Google, Microsoft, Nvidia, OpenAI, and even the U.S.’s National Institute of Standards and Technology (NIST), which all had released red teaming frameworks. Investing heavily in red teaming yields tangible benefits for security leaders in any organization. OpenAI’s paper on external red teaming provides a detailed analysis of how the company strives to create specialized external teams that include cybersecurity and subject matter experts. The goal is to see if knowledgeable external teams can defeat models’ security perimeters and find gaps in their security, biases and controls that prompt-based testing couldn’t find. What makes OpenAI’s recent papers noteworthy is how well they define using human-in-the-middle

Stay Ahead, Stay ONMINE