Mastering the Poisson Distribution: Intuition and Foundations

Stay Ahead, Stay ONMINE

Mastering the Poisson Distribution: Intuition and Foundations

You’ve probably used the normal distribution one or two times too many. We all have — It’s a true workhorse. But sometimes, we run into problems. For instance, when predicting or forecasting values, simulating data given a particular data-generating process, or when we try to visualise model output and explain them intuitively to non-technical stakeholders. Suddenly, things don’t make much sense: can a user really have made -8 clicks on the banner? Or even 4.3 clicks? Both are examples of how count data doesn’t behave. I’ve found that better encapsulating the data generating process into my modelling has been key to having sensible model output. Using the Poisson distribution when it was appropriate has not only helped me convey more meaningful insights to stakeholders, but it has also enabled me to produce more accurate error estimates, better Inference, and sound decision-making. In this post, my aim is to help you get a deep intuitive feel for the Poisson distribution by walking through example applications, and taking a dive into the foundations — the maths. I hope you learn not just how it works, but also why it works, and when to apply the distribution. If you know of a resource that has helped you grasp the concepts in this blog particularly well, you’re invited to share it in the comments! Outline Examples and use cases: Let’s walk through some use cases and sharpen the intuition I just mentioned. Along the way, the relevance of the Poisson Distribution will become clear. The foundations: Next, let’s break down the equation into its individual components. By studying each part, we’ll uncover why the distribution works the way it does. The assumptions: Equipped with some formality, it will be easier to understand the assumptions that power the distribution, and at the same time set the boundaries for when it works, and when not. When real life deviates from the model: Finally, let’s explore the special links that the Poisson distribution has with the Negative Binomial distribution. Understanding these relationships can deepen our understanding, and provide alternatives when the Poisson distribution is not suited for the job. Example in an online marketplace I chose to deep dive into the Poisson distribution because it frequently appears in my day-to-day work. Online marketplaces rely on binary user choices from two sides: a seller deciding to list an item and a buyer deciding to make a purchase. These micro-behaviours drive supply and demand, both in the short and long term. A marketplace is born. Binary choices aggregate into counts — the sum of many such decisions as they occur. Attach a timeframe to this counting process, and you’ll start seeing Poisson distributions everywhere. Let’s explore a concrete example next. Consider a seller on a platform. In a given month, the seller may or may not list an item for sale (a binary choice). We would only know if she did because then we’d have a measurable count of the event. Nothing stops her from listing another item in the same month. If she does, we count those events. The total could be zero for an inactive seller or, say, 120 for a highly engaged seller. Over several months, we would observe a varying number of listed items by this seller — sometimes fewer, sometimes more — hovering around an average monthly listing rate. That is essentially a Poisson process. When we get to the assumptions section, you’ll see what we had to assume away to make this example work. Other examples Other phenomena that can be modelled with a Poisson distribution include: Sports analytics: The number of goals scored in a match between two teams. Queuing: Customers arriving at a help desk or customer support calls. Insurance: The number of claims made within a given period. Each of these examples warrants further inspection, but for the remainder of this post, we’ll use the marketplace example to illustrate the inner workings of the distribution. The mathy bit … or foundations. I find opening up the probability mass function (PMF) of distributions helpful to understanding why things work as they do. The PMF of the Poisson distribution goes like: Where λ is the rate parameter, and 𝑘 is the manifested count of the random variable (𝑘 = 0, 1, 2, 3, … events). Very neat and compact. The probability mass function of the Poisson distribution, for a few different lambdas. Contextualising λ and k: the marketplace example In the context of our earlier example — a seller listing items on our platform — λ represents the seller’s average monthly listings. As the expected monthly value for this seller, λ orchestrates the number of items she would list in a month. Note that λ is a Greek letter, so read: λ is a parameter that we can estimate from data. On the other hand, 𝑘 does not hold any information about the seller’s idiosyncratic behaviour. It’s the target value we set for the number of events that may happen to learn about its probability. The dual role of λ as the mean and variance When I said that λ orchestrates the number of monthly listings for the seller, I meant it quite literally. Namely, λ is both the expected value and variance of the distribution, indifferently, for all values of λ. This means that the mean-to-variance ratio (index of dispersion) is always 1. To put this into perspective, the normal distribution requires two parameters — 𝜇 and 𝜎², the average and variance respectively — to fully describe it. The Poisson distribution achieves the same with just one. Having to estimate only one parameter can be beneficial for parametric inference. Specifically, by reducing the variance of the model and increasing the statistical power. On the other hand, it can be too limiting of an assumption. Alternatives like the Negative Binomial distribution can alleviate this limitation. We’ll explore that later. Breaking down the probability mass function Now that we know the smallest building blocks, let’s zoom out one step: what is λᵏ, 𝑒^⁻λ, and 𝑘!, and more importantly, what is each of these components’ function in the whole? λᵏ is a weight that expresses how likely it is for 𝑘 events to happen, given that the expectation is λ. Note that “likely” here does not mean a probability, yet. It’s merely a signal strength. 𝑘! is a combinatorial correction so that we can say that the order of the events is irrelevant. The events are interchangeable. 𝑒^⁻λ normalises the integral of the PMF function to sum up to 1. It’s called the partition function of exponential-family distributions. In more detail, λᵏ relates the observed value 𝑘 to the expected value of the random variable, λ. Intuitively, more probability mass lies around the expected value. Hence, if the observed value lies close to the expectation, the probability of occurring is larger than the probability of an observation far removed from the expectation. Before we can cross-check our intuition with the numerical behaviour of λᵏ, we need to consider what 𝑘! does. Interchangeable events Had we cared about the order of events, then each unique event could be ordered in 𝑘! ways. But because we don’t, and we deem each event interchangeable, we “divide out” 𝑘! from λᵏ to correct for the overcounting. Since λᵏ is an exponential term, the output will always be larger as 𝑘 grows, holding λ constant. That is the opposite of our intuition that there is maximum probability when λ = 𝑘, as the output is larger when 𝑘 = λ + 1. But now that we know about the interchangeable events assumption — and the overcounting issue — we know that we have to factor in 𝑘! like so: λᵏ 𝑒^⁻λ / 𝑘!, to see the behaviour we expect. Now let’s check the intuition of the relationship between λ and 𝑘 through λᵏ, corrected for 𝑘!. For the same λ, say λ = 4, we should see λᵏ 𝑒^⁻λ / 𝑘! to be smaller for values of 𝑘 that are far removed from 4, compared to values of 𝑘 that lie close to 4. Like so: inline code: 4²/2 = 8 is smaller than 4⁴/24 = 10.7. This is consistent with the intuition of a higher likelihood of 𝑘 when it’s near the expectation. The image below shows this relationship more generally, where you see that the output is larger as 𝑘 approaches λ. The probability mass function without the normalising component e^-lambda. The assumptions First, let’s get one thing off the table: the difference between a Poisson process, and the Poisson distribution. The process is a stochastic continuous-time model of points happening in given interval: 1D, a line; 2D, an area, or higher dimensions. We, data scientists, most often deal with the one-dimensional case, where the “line” is time, and the points are the events of interest — I dare to say. These are the assumptions of the Poisson process: The occurrence of one event does not affect the probability of a second event. Think of our seller going on to list another item tomorrow indifferently of having done so already today, or the one from five days ago for that matter. The point here is that there is no memory between events. The average rate at which events occur, is independent of any occurrence. In other words, no event that happened (or will happen) alters λ, which remains constant throughout the observed timeframe. In our seller example, this means that listing an item today does not increase or decrease the seller’s motivation or likelihood of listing another item tomorrow. Two events cannot occur at exactly the same instant. If we were to zoom at an infinite granular level on the timescale, no two listings could have been placed simultaneously; always sequentially. From these assumptions — no memory, constant rate, events happening alone — it follows that 1) any interval’s number of events is Poisson-distributed with parameter λₜ and 2) that disjoint intervals are independent — two key properties of a Poisson process. A Note on the distribution:The distribution simply describes probabilities for various numbers of counts in an interval. Strictly speaking, one can use the distribution pragmatically whenever the data is nonnegative, can be unbounded on the right, has mean λ, and reasonably models the data. It would be just convenient if the underlying process is a Poisson one, and actually justifies using the distribution. The marketplace example: Implications So, can we justify using the Poisson distribution for our marketplace example? Let’s open up the assumptions of a Poisson process and take the test. Constant λ Why it may fail: The seller has patterned online activity; holidays; promotions; listings are seasonal goods. Consequence: λ is not constant, leading to overdispersion (mean-to-variance ratio is larger than 1, or to temporal patterns. Independence and memorylessness Why it may fail: The propensity to list again is higher after a successful listing, or conversely, listing once depletes the stock and intervenes with the propensity of listing again. Consequence: Two events are no longer independent, as the occurrence of one informs the occurrence of the other. Simultaneous events Why it may fail: Batch-listing, a new feature, was introduced to help the sellers. Consequence: Multiple listings would come online at the same time, clumped together, and they would be counted simultaneously. Balancing rigour and pragmatism As Data Scientists on the job, we may feel trapped between rigour and pragmatism. The three steps below should give you a sound foundation to decide on which side to err, when the Poisson distribution falls short: Pinpoint your goal: is it inference, simulation or prediction, and is it about high-stakes output? List the worst thing that can happen, and the cost of it for the business. Identify the problem and solution: why does the Poisson distribution not fit, and what can you do about it? list 2-3 solutions, including changing nothing. Balance gains and costs: Will your workaround improve things, or make it worse? and at what cost: interpretability, new assumptions introduced and resources used. Does it help you in achieving your goal? That said, here are some counters I use when needed. When real life deviates from your model Everything described so far pertains to the standard, or homogenous, Poisson process. But what if reality begs for something different? In the next section, we’ll cover two extensions of the Poisson distribution when the constant λ assumption does not hold. These are not mutually exclusive, but neither they are the same: Time-varying λ: a single seller whose listing rate ramps up before holidays and slows down afterward Mixed Poisson distribution: multiple sellers listing items, each with their own λ can be seen as a mixture of various Poisson processes Time-varying λ The first extension allows λ to have its own value for each time t. The PMF then becomes Where the number of events 𝐾(𝑇) in an interval 𝑇 follows the Poisson distribution with a rate no longer equal to a fixed λ, but one equal to: More intuitively, integrating over the interval 𝑡 to 𝑡 + 𝑖 gives us a single number: the expected value of events over that interval. The integral will vary by each arbitrary interval, and that’s what makes λ change over time. To understand how that integration works, it was helpful for me to think of it like this: if the interval 𝑡 to 𝑡₁ integrates to 3, and 𝑡₁ to 𝑡₂ integrates to 5, then the interval 𝑡 to 𝑡₂ integrates to 8 = 3 + 5. That’s the two expectations summed up, and now the expectation of the entire interval. Practical implication One may want to modeling the expected value of the Poisson distribution as a function of time. For instance, to model an overall change in trend, or seasonality. In generative model notation: Time may be a continuous variable, or an arbitrary function of it. Process-varying λ: Mixed Poisson distribution But then there’s a gotcha. Remember when I said that λ has a dual role as the mean and variance? That still applies here. Looking at the “relaxed” PMF*, the only thing that changes is that λ can vary freely with time. But it’s still the one and only λ that orchestrates both the expected value and the dispersion of the PMF*. More precisely, 𝔼[𝑋] = Var(𝑋) still holds. There are various reasons for this constraint not to hold in reality. Model misspecification, event interdependence and unaccounted for heterogeneity could be the issues at hand. I’d like to focus on the latter case, as it justifies the Negative Binomial distribution — one of the topics I promised to open up. Heterogeneity and overdispersionImagine we are not dealing with one seller, but with 10 of them listing at different intensity levels, λᵢ, where 𝑖 = 1, 2, 3, …, 10 sellers. Then, essentially, we have 10 Poisson processes going on. If we unify the processes and estimate the grand λ, we simplify the mixture away. Meaning, we get a correct estimate of all sellers on average, but the resulting grand λ is naive and does not know about the original spread of λᵢ. It still assumes that the variance and mean are equal, as per the axioms of the distribution. This will lead to overdispersion and, in turn, to underestimated errors. Ultimately, it inflates the false positive rate and drives poor decision-making. We need a way to embrace the heterogeneity amongst sellers’ λᵢ. Negative binomial: Extending the Poisson distributionAmong the few ways one can look at the Negative Binomial distribution, one way is to see it as a compound Poisson process — 10 sellers, sounds familiar yet? That means multiple independent Poisson processes are summed up to a single one. Mathematically, first we draw λ from a Gamma distribution: λ ~ Γ(r, θ), then we draw the count 𝑋 | λ ~ Poisson(λ). In one image, it is as if we would sample from plenty Poisson distributions, corresponding to each seller. A negative Binomial distribution arises from many Poisson distributions. The more exposing alias of the Negative binomial distribution is Gamma-Poisson mixture distribution, and now we know why: the dictating λ comes from a continuous mixture. That’s what we needed to explain the heterogeneity amongst sellers. Let’s simulate this scenario to gain more intuition. Gamma mixture of lambda. First, we draw λᵢ from a Gamma distribution: λᵢ ~ Γ(r, θ). Intuitively, the Gamma distribution tells us about the variety in the intensity — listing rate — amongst the sellers. On a practical note, one can instill their assumptions about the degree of heterogeneity in this step of the model: how different are sellers? By varying the levels of heterogeneity, one can observe the impact on the final Poisson-like distribution. Doing this type of checks (i.e., posterior predictive check), is common in Bayesian modeling, where the assumptions are set explicitly. Gamma-Poisson mixture distribution versus homogenous Poisson distribution. Τhe dashed line reflects λ, which is 4 for both distributions. In the second step, we plug the obtained λ into the Poisson distribution: 𝑋 | λ ~ Poisson(λ), and obtain a Poisson-like distribution that represents the summed subprocesses. Notably, this unified process has a larger dispersion than expected from a homogeneous Poisson distribution, but it is in line with the Gamma mixture of λ. Heterogeneous λ and inference A practical consequence of introducing flexibility into your assumed distribution is that inference becomes more challenging. More parameters (i.e., the Gamma parameters) need to be estimated. Parameters act as flexible explainers of the data, tending to overfit and explain away variance in your variable. The more parameters you have, the better the explanation may seem, but the model also becomes more susceptible to noise in the data. Higher variance reduces the power to identify a difference in means, if one exists, because — well — it gets lost in the variance. Countering the loss of power Confirm whether you indeed need to extend the standard Poisson distribution. If not, simplify to the simplest, most fit model. A quick check on overdispersion may suffice for this. Pin down the estimates of the Gamma mixture distribution parameters using regularising, informative priors (think: Bayes). During my research process for writing this blog, I learned a great deal about the connective tissue underlying all of this: how the binomial distribution plays a fundamental role in the processes we’ve discussed. And while I’d love to ramble on about this, I’ll save it for another post, perhaps. In the meantime, feel free to share your understanding in the comments section below 👍. Conclusion The Poisson distribution is a simple distribution that can be highly suitable for modelling count data. However, when the assumptions do not hold, one can extend the distribution by allowing the rate parameter to vary as a function of time or other factors, or by assuming subprocesses that collectively make up the count data. This added flexibility can address the limitations, but it comes at a cost: increased flexibility in your modelling raises the variance and, consequently, undermines the statistical power of your model. If your end goal is inference, you may want to think twice and consider exploring simpler models for the data. Alternatively, switch to the Bayesian paradigm and leverage its built-in solution to regularise estimates: informative priors. I hope this has given you what you came for — a better intuition about the Poisson distribution. I’d love to hear your thoughts about this in the comments! Unless otherwise noted, all images are by the author.Originally published at https://aalvarezperez.github.io on January 5, 2025.

I’ve found that better encapsulating the data generating process into my modelling has been key to having sensible model output. Using the Poisson distribution when it was appropriate has not only helped me convey more meaningful insights to stakeholders, but it has also enabled me to produce more accurate error estimates, better Inference, and sound decision-making.

In this post, my aim is to help you get a deep intuitive feel for the Poisson distribution by walking through example applications, and taking a dive into the foundations — the maths. I hope you learn not just how it works, but also why it works, and when to apply the distribution.

If you know of a resource that has helped you grasp the concepts in this blog particularly well, you’re invited to share it in the comments!

Outline

Examples and use cases: Let’s walk through some use cases and sharpen the intuition I just mentioned. Along the way, the relevance of the Poisson Distribution will become clear.
The foundations: Next, let’s break down the equation into its individual components. By studying each part, we’ll uncover why the distribution works the way it does.
The assumptions: Equipped with some formality, it will be easier to understand the assumptions that power the distribution, and at the same time set the boundaries for when it works, and when not.
When real life deviates from the model: Finally, let’s explore the special links that the Poisson distribution has with the Negative Binomial distribution. Understanding these relationships can deepen our understanding, and provide alternatives when the Poisson distribution is not suited for the job.

Example in an online marketplace

I chose to deep dive into the Poisson distribution because it frequently appears in my day-to-day work. Online marketplaces rely on binary user choices from two sides: a seller deciding to list an item and a buyer deciding to make a purchase. These micro-behaviours drive supply and demand, both in the short and long term. A marketplace is born.

Binary choices aggregate into counts — the sum of many such decisions as they occur. Attach a timeframe to this counting process, and you’ll start seeing Poisson distributions everywhere. Let’s explore a concrete example next.

Consider a seller on a platform. In a given month, the seller may or may not list an item for sale (a binary choice). We would only know if she did because then we’d have a measurable count of the event. Nothing stops her from listing another item in the same month. If she does, we count those events. The total could be zero for an inactive seller or, say, 120 for a highly engaged seller.

Over several months, we would observe a varying number of listed items by this seller — sometimes fewer, sometimes more — hovering around an average monthly listing rate. That is essentially a Poisson process. When we get to the assumptions section, you’ll see what we had to assume away to make this example work.

Other examples

Other phenomena that can be modelled with a Poisson distribution include:

Sports analytics: The number of goals scored in a match between two teams.
Queuing: Customers arriving at a help desk or customer support calls.
Insurance: The number of claims made within a given period.

Each of these examples warrants further inspection, but for the remainder of this post, we’ll use the marketplace example to illustrate the inner workings of the distribution.

The mathy bit

… or foundations.

I find opening up the probability mass function (PMF) of distributions helpful to understanding why things work as they do. The PMF of the Poisson distribution goes like:

Where λ is the rate parameter, and 𝑘 is the manifested count of the random variable (𝑘 = 0, 1, 2, 3, … events). Very neat and compact.

Graph: The probability mass function of the Poisson distribution, for a few different lambdas. — The probability mass function of the Poisson distribution, for a few different lambdas.

Contextualising λ and k: the marketplace example

In the context of our earlier example — a seller listing items on our platform — λ represents the seller’s average monthly listings. As the expected monthly value for this seller, λ orchestrates the number of items she would list in a month. Note that λ is a Greek letter, so read: λ is a parameter that we can estimate from data. On the other hand, 𝑘 does not hold any information about the seller’s idiosyncratic behaviour. It’s the target value we set for the number of events that may happen to learn about its probability.

The dual role of λ as the mean and variance

When I said that λ orchestrates the number of monthly listings for the seller, I meant it quite literally. Namely, λ is both the expected value and variance of the distribution, indifferently, for all values of λ. This means that the mean-to-variance ratio (index of dispersion) is always 1.

To put this into perspective, the normal distribution requires two parameters — 𝜇 and 𝜎², the average and variance respectively — to fully describe it. The Poisson distribution achieves the same with just one.

Having to estimate only one parameter can be beneficial for parametric inference. Specifically, by reducing the variance of the model and increasing the statistical power. On the other hand, it can be too limiting of an assumption. Alternatives like the Negative Binomial distribution can alleviate this limitation. We’ll explore that later.

Breaking down the probability mass function

Now that we know the smallest building blocks, let’s zoom out one step: what is λᵏ, 𝑒^⁻λ, and 𝑘!, and more importantly, what is each of these components’ function in the whole?

λᵏ is a weight that expresses how likely it is for 𝑘 events to happen, given that the expectation is λ. Note that “likely” here does not mean a probability, yet. It’s merely a signal strength.
𝑘! is a combinatorial correction so that we can say that the order of the events is irrelevant. The events are interchangeable.
𝑒^⁻λ normalises the integral of the PMF function to sum up to 1. It’s called the partition function of exponential-family distributions.

In more detail, λᵏ relates the observed value 𝑘 to the expected value of the random variable, λ. Intuitively, more probability mass lies around the expected value. Hence, if the observed value lies close to the expectation, the probability of occurring is larger than the probability of an observation far removed from the expectation. Before we can cross-check our intuition with the numerical behaviour of λᵏ, we need to consider what 𝑘! does.

Interchangeable events

Had we cared about the order of events, then each unique event could be ordered in 𝑘! ways. But because we don’t, and we deem each event interchangeable, we “divide out” 𝑘! from λᵏ to correct for the overcounting.

Since λᵏ is an exponential term, the output will always be larger as 𝑘 grows, holding λ constant. That is the opposite of our intuition that there is maximum probability when λ = 𝑘, as the output is larger when 𝑘 = λ + 1. But now that we know about the interchangeable events assumption — and the overcounting issue — we know that we have to factor in 𝑘! like so: λᵏ 𝑒^⁻λ / 𝑘!, to see the behaviour we expect.

Now let’s check the intuition of the relationship between λ and 𝑘 through λᵏ, corrected for 𝑘!. For the same λ, say λ = 4, we should see λᵏ 𝑒^⁻λ / 𝑘! to be smaller for values of 𝑘 that are far removed from 4, compared to values of 𝑘 that lie close to 4. Like so: inline code: 4²/2 = 8 is smaller than 4⁴/24 = 10.7. This is consistent with the intuition of a higher likelihood of 𝑘 when it’s near the expectation. The image below shows this relationship more generally, where you see that the output is larger as 𝑘 approaches λ.

Graph: The probability mass function without the normalising component e^-lambda. — The probability mass function without the normalising component e^-lambda.

The assumptions

First, let’s get one thing off the table: the difference between a Poisson process, and the Poisson distribution. The process is a stochastic continuous-time model of points happening in given interval: 1D, a line; 2D, an area, or higher dimensions. We, data scientists, most often deal with the one-dimensional case, where the “line” is time, and the points are the events of interest — I dare to say.

These are the assumptions of the Poisson process:

The occurrence of one event does not affect the probability of a second event. Think of our seller going on to list another item tomorrow indifferently of having done so already today, or the one from five days ago for that matter. The point here is that there is no memory between events.
The average rate at which events occur, is independent of any occurrence. In other words, no event that happened (or will happen) alters λ, which remains constant throughout the observed timeframe. In our seller example, this means that listing an item today does not increase or decrease the seller’s motivation or likelihood of listing another item tomorrow.
Two events cannot occur at exactly the same instant. If we were to zoom at an infinite granular level on the timescale, no two listings could have been placed simultaneously; always sequentially.

From these assumptions — no memory, constant rate, events happening alone — it follows that 1) any interval’s number of events is Poisson-distributed with parameter λₜ and 2) that disjoint intervals are independent — two key properties of a Poisson process.

A Note on the distribution:
The distribution simply describes probabilities for various numbers of counts in an interval. Strictly speaking, one can use the distribution pragmatically whenever the data is nonnegative, can be unbounded on the right, has mean λ, and reasonably models the data. It would be just convenient if the underlying process is a Poisson one, and actually justifies using the distribution.

The marketplace example: Implications

So, can we justify using the Poisson distribution for our marketplace example? Let’s open up the assumptions of a Poisson process and take the test.

Constant λ

Why it may fail: The seller has patterned online activity; holidays; promotions; listings are seasonal goods.
Consequence: λ is not constant, leading to overdispersion (mean-to-variance ratio is larger than 1, or to temporal patterns.

Independence and memorylessness

Why it may fail: The propensity to list again is higher after a successful listing, or conversely, listing once depletes the stock and intervenes with the propensity of listing again.
Consequence: Two events are no longer independent, as the occurrence of one informs the occurrence of the other.

Simultaneous events

Why it may fail: Batch-listing, a new feature, was introduced to help the sellers.
Consequence: Multiple listings would come online at the same time, clumped together, and they would be counted simultaneously.

Balancing rigour and pragmatism

As Data Scientists on the job, we may feel trapped between rigour and pragmatism. The three steps below should give you a sound foundation to decide on which side to err, when the Poisson distribution falls short:

Pinpoint your goal: is it inference, simulation or prediction, and is it about high-stakes output? List the worst thing that can happen, and the cost of it for the business.
Identify the problem and solution: why does the Poisson distribution not fit, and what can you do about it? list 2-3 solutions, including changing nothing.
Balance gains and costs: Will your workaround improve things, or make it worse? and at what cost: interpretability, new assumptions introduced and resources used. Does it help you in achieving your goal?

That said, here are some counters I use when needed.

When real life deviates from your model

Everything described so far pertains to the standard, or homogenous, Poisson process. But what if reality begs for something different?

In the next section, we’ll cover two extensions of the Poisson distribution when the constant λ assumption does not hold. These are not mutually exclusive, but neither they are the same:

Time-varying λ: a single seller whose listing rate ramps up before holidays and slows down afterward
Mixed Poisson distribution: multiple sellers listing items, each with their own λ can be seen as a mixture of various Poisson processes

Time-varying λ

The first extension allows λ to have its own value for each time t. The PMF then becomes

Where the number of events 𝐾(𝑇) in an interval 𝑇 follows the Poisson distribution with a rate no longer equal to a fixed λ, but one equal to:

More intuitively, integrating over the interval 𝑡 to 𝑡 + 𝑖 gives us a single number: the expected value of events over that interval. The integral will vary by each arbitrary interval, and that’s what makes λ change over time. To understand how that integration works, it was helpful for me to think of it like this: if the interval 𝑡 to 𝑡₁ integrates to 3, and 𝑡₁ to 𝑡₂ integrates to 5, then the interval 𝑡 to 𝑡₂ integrates to 8 = 3 + 5. That’s the two expectations summed up, and now the expectation of the entire interval.

Practical implication
One may want to modeling the expected value of the Poisson distribution as a function of time. For instance, to model an overall change in trend, or seasonality. In generative model notation:

Time may be a continuous variable, or an arbitrary function of it.

Process-varying λ: Mixed Poisson distribution

But then there’s a gotcha. Remember when I said that λ has a dual role as the mean and variance? That still applies here. Looking at the “relaxed” PMF*, the only thing that changes is that λ can vary freely with time. But it’s still the one and only λ that orchestrates both the expected value and the dispersion of the PMF*. More precisely, 𝔼[𝑋] = Var(𝑋) still holds.

There are various reasons for this constraint not to hold in reality. Model misspecification, event interdependence and unaccounted for heterogeneity could be the issues at hand. I’d like to focus on the latter case, as it justifies the Negative Binomial distribution — one of the topics I promised to open up.

Heterogeneity and overdispersion
Imagine we are not dealing with one seller, but with 10 of them listing at different intensity levels, λᵢ, where 𝑖 = 1, 2, 3, …, 10 sellers. Then, essentially, we have 10 Poisson processes going on. If we unify the processes and estimate the grand λ, we simplify the mixture away. Meaning, we get a correct estimate of all sellers on average, but the resulting grand λ is naive and does not know about the original spread of λᵢ. It still assumes that the variance and mean are equal, as per the axioms of the distribution. This will lead to overdispersion and, in turn, to underestimated errors. Ultimately, it inflates the false positive rate and drives poor decision-making. We need a way to embrace the heterogeneity amongst sellers’ λᵢ.

Negative binomial: Extending the Poisson distribution
Among the few ways one can look at the Negative Binomial distribution, one way is to see it as a compound Poisson process — 10 sellers, sounds familiar yet? That means multiple independent Poisson processes are summed up to a single one. Mathematically, first we draw λ from a Gamma distribution: λ ~ Γ(r, θ), then we draw the count 𝑋 | λ ~ Poisson(λ).

In one image, it is as if we would sample from plenty Poisson distributions, corresponding to each seller.

A negative Binomial distribution arises from many Poisson distributions.

The more exposing alias of the Negative binomial distribution is Gamma-Poisson mixture distribution, and now we know why: the dictating λ comes from a continuous mixture. That’s what we needed to explain the heterogeneity amongst sellers.

Let’s simulate this scenario to gain more intuition.

First, we draw λᵢ from a Gamma distribution: λᵢ ~ Γ(r, θ). Intuitively, the Gamma distribution tells us about the variety in the intensity — listing rate — amongst the sellers.

On a practical note, one can instill their assumptions about the degree of heterogeneity in this step of the model: how different are sellers? By varying the levels of heterogeneity, one can observe the impact on the final Poisson-like distribution. Doing this type of checks (i.e., posterior predictive check), is common in Bayesian modeling, where the assumptions are set explicitly.

*Gamma-Poisson mixture distribution versus homogenous Poisson distribution. Τhe dashed line reflects λ, which is 4 for both distributions.*

In the second step, we plug the obtained λ into the Poisson distribution: 𝑋 | λ ~ Poisson(λ), and obtain a Poisson-like distribution that represents the summed subprocesses. Notably, this unified process has a larger dispersion than expected from a homogeneous Poisson distribution, but it is in line with the Gamma mixture of λ.

Heterogeneous λ and inference

A practical consequence of introducing flexibility into your assumed distribution is that inference becomes more challenging. More parameters (i.e., the Gamma parameters) need to be estimated. Parameters act as flexible explainers of the data, tending to overfit and explain away variance in your variable. The more parameters you have, the better the explanation may seem, but the model also becomes more susceptible to noise in the data. Higher variance reduces the power to identify a difference in means, if one exists, because — well — it gets lost in the variance.

Countering the loss of power

Confirm whether you indeed need to extend the standard Poisson distribution. If not, simplify to the simplest, most fit model. A quick check on overdispersion may suffice for this.
Pin down the estimates of the Gamma mixture distribution parameters using regularising, informative priors (think: Bayes).

During my research process for writing this blog, I learned a great deal about the connective tissue underlying all of this: how the binomial distribution plays a fundamental role in the processes we’ve discussed. And while I’d love to ramble on about this, I’ll save it for another post, perhaps. In the meantime, feel free to share your understanding in the comments section below 👍.

Conclusion

The Poisson distribution is a simple distribution that can be highly suitable for modelling count data. However, when the assumptions do not hold, one can extend the distribution by allowing the rate parameter to vary as a function of time or other factors, or by assuming subprocesses that collectively make up the count data. This added flexibility can address the limitations, but it comes at a cost: increased flexibility in your modelling raises the variance and, consequently, undermines the statistical power of your model.

If your end goal is inference, you may want to think twice and consider exploring simpler models for the data. Alternatively, switch to the Bayesian paradigm and leverage its built-in solution to regularise estimates: informative priors.

I hope this has given you what you came for — a better intuition about the Poisson distribution. I’d love to hear your thoughts about this in the comments!

Unless otherwise noted, all images are by the author.
Originally published at https://aalvarezperez.github.io on January 5, 2025.

Stay Ahead

Explore More Insights

Stay ahead with more perspectives on cutting-edge power, infrastructure, energy, bitcoin and AI solutions. Explore these articles to uncover strategies and insights shaping the future of industries.

Fluent Bit vulnerabilities could enable full cloud takeover

Attackers could flood monitoring systems with false or misleading events, hide alerts in the noise, or even hijack the telemetry stream entirely, Katz said. The issue is now tracked as CVE-2025-12969 and awaits a severity valuation. Almost equally troubling are other flaws in the “tag” mechanism, which determines how the records are

NFL, AWS drive football modernization with cloud, AI

AWS Next Gen Stats: Initially used for player participation tracking (replacing manual photo-taking), Next Gen Stats uses sensors to capture center-of-mass and contact information, which is then used to generate performance insights. Computer vision: Computer vision was initially insufficient, but the technology has improved greatly over the past few years.

Apstra founder launches Aria to tackle AI networking performance

Aria’s technical approach differs from incumbent vendors in its focus on end-to-end path optimization rather than individual switch performance. Karam argues that traditional networking vendors think of themselves primarily as switch companies, with software efforts concentrated on switch operating systems rather than cluster-wide operational models. “It’s no longer just about

FCC reversal removes federal cyber safeguards targeting telecom weaknesses post-Salt Typhoon attacks

The decision included a Notice of Proposed Rulemaking (NPRM) requiring telecom companies to create, update, and implement cybersecurity risk management plans, and certify them annually. However, this week the FCC claimed that the Declaratory Ruling “misconstrued” CALEA, calling it “flawed,” and “unlawful and ineffective.” According to the agency, their action

Energy Secretary Secures Grid Reliability in Mid-Atlantic Ahead of Winter

Emergency order increases grid stability, lowers energy costs, and minimizes the risk of energy shortfalls in the Mid-Atlantic region of the United States ahead of cold winter months. WASHINGTON—U.S. Secretary of Energy Chris Wright today issued an emergency order to minimize the risk of blackouts in the Mid-Atlantic region of the United States. Secretary Wright’s order directs PJM Interconnection (PJM), in coordination with Constellation Energy, to ensure Units 3 and 4 of the Eddystone Generating Station in Pennsylvania remain available for operation and to take every step to minimize costs for the American people. The production of electricity from the units will continue to be critical to maintaining reliability in PJM over the coming winter months.  “Thanks to President Trump’s leadership, the Department of Energy is using all tools available to keep the lights on and heat running for the American people,” said Energy Secretary Wright. “This emergency order is needed to strengthen grid reliability and will help provide affordable, reliable, and secure power when Americans need it most.” As outlined in DOE’s Resource Adequacy Report, power outages could increase by 100 times in 2030 if the U.S. continues to take reliable power offline. Secretary Wright ordered that the two Eddystone Generating Station units remain online past their planned retirement date in a May 30, 2025 emergency order. Keeping these units operational over the summer strengthened energy security in the PJM region, as demonstrated when PJM called on the Eddystone Units to generate electricity during heat waves that hit the region in June and July.  A subsequent order was issued on August 28, 2025. PJM’s service area will continue to face emergency conditions both in the near and long term. In January 2025, PJM reached a new record peak for winter demand, exceeding the previous winter peak set in 2015. This order is in effect

Oil Closes the Day Near Month Low

Oil fell as signs of progress in peace talks between Ukraine and Russia buoyed expectations that Moscow’s supply will stay online. West Texas Intermediate futures fell 1.5% to settle near $58 a barrel, the lowest in a month, as talks to end the war in Ukraine show signs of progress. Crude had dropped sharply earlier in the session after ABC News reported Kyiv agreed to the terms of a revised peace deal aimed at ending Russia’s nearly four-year war. Ukraine’s President Volodymyr Zelenskiy said talks on a peace plan are continuing with the US. There are key points still to be resolved between US and Ukraine, including the thorniest issues, according to a person familiar with the matter. A White House spokesperson signaled optimism around the efforts while warning some details still need to be sorted as Russia’s position on the plan remains unclear. Moscow and Ukraine carried out airstrikes overnight. “Flat crude prices got hammered on news that Ukraine appeared open to the broad contours of the US-proposed peace plan,” said Rory Johnston of Commodity Context. The physical market tells a different story, with “prompt Brent timespreads continuing to strengthen, indicating continued tightness” in near-term supply. An end to the war would have significant ramifications for the oil market. Russia is one of the world’s top producers and its flows are heavily sanctioned by the US, European Union and UK. In October the US announced new sanctions on its two major producers. It’s still far from certain, though, that Russia will accept a revised peace plan that cut several points from an initial proposal following input from European officials. “For energy markets, this means volatility is far from over,” said Jorge Leon, head of geopolitical analysis at Rystad Energy A/S. “Prices reacted swiftly to the initial optimism for an

OPEC+ Again Faces Thorny Issue of How Much It Can Pump

OPEC+ nations gathering this weekend are once again grappling with the thorny question of how much oil they’re physically able to pump. In May, the Organization of the Petroleum Exporting Countries and its allies launched a new assessment of members’ “maximum sustainable capacity” to help set production quotas in 2027. With output levels for the months ahead already set, delegates say this longer-term review will likely be one area of focus at Sunday’s meeting. The process looks increasingly necessary, as the struggle by some OPEC+ members to increase supplies as much as agreed this year indicates they may be nearing output limits. Clarifying their full capacity would help align quotas more closely with reality — and make any future cutbacks more credible. OPEC’s readiness to make new curbs could be tested in 2026 amid signs of a swelling global oil surplus and downward pressure on crude prices, which have slumped to near $60 a barrel in London. In a report on Monday, JPMorgan Chase & Co. indicated that the alliance may need to slash output next year to avert a plunge into the $40s. But the capacity assessment also poses an area of friction for the organization, as some countries push for a higher estimate of their abilities and others refuse to admit they can’t produce as much as claimed. In 2023, discord over the process led to the exit of long-term OPEC member Angola. While group leader Saudi Arabia is capable of boosting output significantly, the outlook for other nations is less clear-cut. The United Arab Emirates and Iraq have been eager to expand capacity, but some members like Russia are challenged by international sanctions. The review will be conducted with the assistance of several energy consulting firms, which in the past have included Wood Mackenzie and IHS, which is

NatGas Immediate Term Volatility Risks Remain High

In an EBW Analytics Group report sent to Rigzone by the EBW team on Tuesday, Eli Rubin, an energy analyst at the company, warned that, for the U.S. natural gas price, “immediate-term volatility risks remain high into December expiration”. Rubin highlighted in the report that yesterday’s December options expiry “saw the front-month falter 13.6 cents before recovering 10.5 cents into the close”. “While daily fundamental signals appear supportive, downside offers confirmation of last week’s January contract bearish triple-top technical pattern at $4.80 [per million British thermal units (MMBtu)],” he said. “DTN’s outsized day over day weather gain is partially catching up to other meteorologists previously anticipating a colder early December,” Rubin added. “Further, while more expansive cold to open the month, Week 3 became milder (particularly across the South) amid shifts in the Pacific North America (PNA) teleconnection – and early-morning price action seems to be reacting to a possibility of ‘seeing beyond’ the early-December cold,” Rubin continued. Rubin went on to state in the report that daily LNG feedgas “may be touching another all-time high this morning as gas production continues to show strength”. “We repeat our analysis that while immediate-term pricing appears to have run ahead of fundamentals, the medium to longer term outlook could see narrowing storage surpluses to lead renewed mid-winter upside potential,” he added. The EBW report pointed out that the December natural gas contract closed at $4.549 per MMBtu on Monday. This marked a 3.1 cent, or 0.7 percent drop from Friday’s close, the report outlined. In the report, EBW predicted a “volatility risks elevated” trend for the NYMEX front-month natural gas contract price over the next 7-10 days and a “jagged path higher” trend over the next 30-45 days. In a separate report sent to Rigzone by the EBW team on Monday, Rubin

Aramco Weighs Raising Billions From Its Biggest Disposals Yet

Saudi Aramco is considering plans to raise billions of dollars by selling a range of assets, people familiar with the matter said, deals that could rank as its most significant disposals ever. The firm is weighing the sale of a stake in its oil export and storage terminals as part of the plans, the people said, declining to be identified as the information is confidential. Banks have been asked to pitch for roles on feasibility studies for the disposals, which could fetch more than $10 billion, they said. Aramco is eying options including raising fresh equity from the deal, the people said. It could also pursue a structure similar to the recent $11 billion lease transaction with a group led by BlackRock Inc.’s Global Infrastructure Partners for assets linked to the Jafurah gas project, they said. That sale drew interest from firms around the world and bankers have since pitched several asset disposal plans given increasing demand from investors, one of the people said. Aramco’s terminals business is seen as a lucrative asset and the company could kick off a formal sales process as soon as early next year, the person said. At the same time, the oil giant is considering selling part of its real estate portfolio, some of the people said. Those assets will also likely be worth billions of dollars and will be seen as attractive at a time when the kingdom is advancing plans to allow foreign ownership. Discussions are at an early stage and no final decisions have been made. Aramco declined to comment. Aramco’s main oil storage and export infrastructure is located at Ras Tanura on the Persian Gulf and the company has similar terminals on the Red Sea. Internationally, the firm owns stakes in product terminals in the Netherlands and leases crude as well as product

Natural gas sees ‘largest year-over-year drop’ in California as solar surges

Listen to the article 2 min This audio is auto-generated. Please let us know if you have feedback. California’s natural gas generation has continued a several-year decline in 2025, while the state’s utility-scale solar keeps rising, according to a new report from the Energy Information Administration. Natural gas is still the dominant energy source in the state overall, but solar is starting to close the gap. For the first eight months of this year, utility-scale solar generation totaled 40.3 billion kilowatt hours in California, and natural gas accounted for 45.5 BkWh. As of the second quarter of this year, California had a total of 49 GW of solar capacity installed, according to the Solar Energy Industries Association. Optional Caption Courtesy of Energy Information Administration While solar’s performance from January to August 2025 was nearly double its generation for the same period in 2020, natural gas supplied 18% less than it did in the same period in 2020, EIA said. California’s natural gas generation peaked above 2020 levels in 2021 “due to drought-spurred reduced hydroelectric output, but natural gas generation has fallen since then,” EIA said. “The largest year-over-year drop occurred this year, when natural gas generation declined 9.5 BkWh, or 17%, compared with 2024.” In the midday hours between noon and 5 p.m., when solar generation is highest, natural gas generation decreases, EIA said. In the midday hours of May and June this year, solar generation accounted for 18.8 GW, compared to 10.2 GW in 2020, according to data from the California Independent System Operator. “During peak evening hours between 5:00 p.m. and 9:00 p.m., generation from batteries charged by excess solar generation during midday rose from an average of less than 1 GW in May and June 2022 to 4.9 GW in 2025, displacing natural gas generation during that period,”

Networks, AI, and metaversing

Our first, conservative, view says that AI’s network impact is largely confined to the data center, to connect clusters of GPU servers and the data they use as they crunch large language models. It’s all “horizontal” traffic; one TikTok challenge would generate way more traffic in the wide area. WAN costs won’t rise for you as an enterprise, and if you’re a carrier you won’t be carrying much new, so you don’t have much service revenue upside. If you don’t host AI on premises, you can pretty much dismiss its impact on your network. Contrast that with the radical metaverse view, our third view. Metaverses and AR/VR transform AI missions, and network services, from transaction processing to event processing, because the real world is a bunch of events pushing on you. They also let you visualize the way that process control models (digital twins) relate to the real world, which is critical if the processes you’re modeling involve human workers who rely on their visual sense. Could it be that the reason Meta is willing to spend on AI, is that the most credible application of AI, and the most impactful for networks, is the metaverse concept? In any event, this model of AI, by driving the users’ experiences and activities directly, demands significant edge connectivity, so you could expect it to have a major impact on network requirements. In fact, just dipping your toes into a metaverse could require a major up-front network upgrade. Networks carry traffic. Traffic is messages. More messages, more traffic, more infrastructure, more service revenue…you get the picture. Door number one, to the AI giant future, leads to nothing much in terms of messages. Door number three, metaverses and AR/VR, leads to a message, traffic, and network revolution. I’ll bet that most enterprises would doubt

Microsoft’s Fairwater Atlanta and the Rise of the Distributed AI Supercomputer

Microsoft’s second Fairwater data center in Atlanta isn’t just “another big GPU shed.” It represents the other half of a deliberate architectural experiment: proving that two massive AI campuses, separated by roughly 700 miles, can operate as one coherent, distributed supercomputer. The Atlanta installation is the latest expression of Microsoft’s AI-first data center design: purpose-built for training and serving frontier models rather than supporting mixed cloud workloads. It links directly to the original Fairwater campus in Wisconsin, as well as to earlier generations of Azure AI supercomputers, through a dedicated AI WAN backbone that Microsoft describes as the foundation of a “planet-scale AI superfactory.” Inside a Fairwater Site: Preparing for Multi-Site Distribution Efficient multi-site training only works if each individual site behaves as a clean, well-structured unit. Microsoft’s intra-site design is deliberately simplified so that cross-site coordination has a predictable abstraction boundary—essential for treating multiple campuses as one distributed AI system. Each Fairwater installation presents itself as a single, flat, high-regularity cluster: Up to 72 NVIDIA Blackwell GPUs per rack, using GB200 NVL72 rack-scale systems. NVLink provides the ultra-low-latency, high-bandwidth scale-up fabric within the rack, while the Spectrum-X Ethernet stack handles scale-out. Each rack delivers roughly 1.8 TB/s of GPU-to-GPU bandwidth and exposes a multi-terabyte pooled memory space addressable via NVLink—critical for large-model sharding, activation checkpointing, and parallelism strategies. Racks feed into a two-tier Ethernet scale-out network offering 800 Gbps GPU-to-GPU connectivity with very low hop counts, engineered to scale to hundreds of thousands of GPUs without encountering the classic port-count and topology constraints of traditional Clos fabrics. Microsoft confirms that the fabric relies heavily on: SONiC-based switching and a broad commodity Ethernet ecosystem to avoid vendor lock-in and accelerate architectural iteration. Custom network optimizations, such as packet trimming, packet spray, high-frequency telemetry, and advanced congestion-control mechanisms, to prevent collective

Land & Expand: Hyperscale, AI Factory, Megascale

Land & Expand is Data Center Frontier’s periodic roundup of notable North American data center development activity, tracking the newest sites, land plays, retrofits, and hyperscale campus expansions shaping the industry’s build cycle. October delivered a steady cadence of announcements, with several megascale projects advancing from concept to commitment. The month was defined by continued momentum in OpenAI and Oracle’s Stargate initiative (now spanning multiple U.S. regions) as well as major new investments from Google, Meta, DataBank, and emerging AI cloud players accelerating high-density reuse strategies. The result is a clearer picture of how the next wave of AI-first infrastructure is taking shape across the country. Google Begins $4B West Memphis Hyperscale Buildout Google formally broke ground on its $4 billion hyperscale campus in West Memphis, Arkansas, marking the company’s first data center in the state and the anchor for a new Mid-South operational hub. The project spans just over 1,000 acres, with initial site preparation and utility coordination already underway. Google and Entergy Arkansas confirmed a 600 MW solar generation partnership, structured to add dedicated renewable supply to the regional grid. As part of the launch, Google announced a $25 million Energy Impact Fund for local community affordability programs and energy-resilience improvements—an unusually early community-benefit commitment for a first-phase hyperscale project. Cooling specifics have not yet been made public. Water sourcing—whether reclaimed, potable, or hybrid seasonal mode—remains under review, as the company finalizes environmental permits. Public filings reference a large-scale onsite water treatment facility, similar to Google’s deployments in The Dalles and Council Bluffs. Local governance documents show that prior to the October announcement, West Memphis approved a 30-year PILOT via Groot LLC (Google’s land assembly entity), with early filings referencing a typical placeholder of ~50 direct jobs. At launch, officials emphasized hundreds of full-time operations roles and thousands

The New Digital Infrastructure Geography: Green Street’s David Guarino on AI Demand, Power Scarcity, and the Next Phase of Data Center Growth

As the global data center industry races through its most frenetic build cycle in history, one question continues to define the market’s mood: is this the peak of an AI-fueled supercycle, or the beginning of a structurally different era for digital infrastructure? For Green Street Managing Director and Head of Global Data Center and Tower Research David Guarino, the answer—based firmly on observable fundamentals—is increasingly clear. Demand remains blisteringly strong. Capital appetite is deepening. And the very definition of a “data center market” is shifting beneath the industry’s feet. In a wide-ranging discussion with Data Center Frontier, Guarino outlined why data centers continue to stand out in the commercial real estate landscape, how AI is reshaping underwriting and development models, why behind-the-meter power is quietly reorganizing the U.S. map, and what Green Street sees ahead for rents, REITs, and the next wave of hyperscale expansion. A ‘Safe’ Asset in an Uncertain CRE Landscape Among institutional investors, the post-COVID era was the moment data centers stepped decisively out of “niche” territory. Guarino notes that pandemic-era reliance on digital services crystallized a structural recognition: data centers deliver stable, predictable cash flows, anchored by the highest-credit tenants in global real estate. Hyperscalers today dominate new leasing and routinely sign 15-year (or longer) contracts, a duration largely unmatched across CRE categories. When compared with one-year apartment leases, five-year office leases, or mall anchor terms, the stability story becomes plain. “These are AAA-caliber companies signing the longest leases in the sector’s history,” Guarino said. “From a real estate point of view, that combination of tenant quality and lease duration continues to position the asset class as uniquely durable.” And development returns remain exceptional. Even without assuming endless AI growth, the math works: strong demand, rising rents, and high-credit tenants create unusually predictable performance relative to

The Flexential Blueprint: New CEO Ryan Mallory on Power, AI, and Bending the Physics Curve

In a coordinated leadership transition this fall, Ryan Mallory has stepped into the role of CEO at Flexential, succeeding Chris Downie. The move, described as thoughtful and planned, signals not a shift in direction, but a reinforcement of the company’s core strategy, with a sharpened focus on the unprecedented opportunities presented by the artificial intelligence revolution. In an exclusive interview on the Data Center Frontier Show Podcast, Mallory outlined a confident vision for Flexential, positioning the company at the critical intersection of enterprise IT and next-generation AI infrastructure. “Flexential will continue to focus on being an industry and market leader in wholesale, multi-tenant, and interconnection capabilities,” Mallory stated, affirming the company’s foundational strengths. His central thesis is that the AI infrastructure boom is not a monolithic wave, but a multi-stage evolution where Flexential’s model is uniquely suited for the emerging “inference edge.” The AI Build Cycle: A Three-Act Play Mallory frames the AI infrastructure market as a three-stage process, each lasting roughly four years. We are currently at the tail end of Stage 1, which began with the ChatGPT explosion three years ago. This phase, characterized by a frantic rush for capacity, has led to elongated lead times for critical infrastructure like generators, switchgear, and GPUs. The capacity from this initial build-out is expected to come online between late 2025 and late 2026. Stage 2, beginning around 2026 and stretching to 2030, will see the next wave of builds, with significant capacity hitting the market in 2028-2029. “This stage will reveal the viability of AI and actual consumption models,” Mallory notes, adding that air-cooled infrastructure will still dominate during this period. Stage 3, looking ahead to the early 2030s, will focus on long-term scale, mirroring the evolution of the public cloud. For Mallory, the enduring nature of this build cycle—contrasted

Centersquare Launches $1 Billion Expansion to Scale an AI-Ready North American Data Center Platform

A Platform Built for Both Colo and AI Density The combined Evoque–Cyxtera platform entered the market with hundreds of megawatts of installed capacity and a clear runway for expansion. That scale positioned Centersquare to offer both traditional enterprise colocation and the higher-density, AI-ready footprints increasingly demanded through 2024 and 2025. The addition of these ten facilities demonstrates that the consolidation strategy is gaining traction, giving the platform more owned capacity to densify and more regional optionality as AI deployment accelerates. What’s in the $1 Billion Package — and Why It Matters 1) Lease-to-Own Conversions in Boston & Minneapolis Centersquare’s decision to purchase two long-operated but previously leased sites in Boston and Minneapolis reduces long-term occupancy risk and gives the operator full capex control. Owning the buildings unlocks the ability to schedule power and cooling upgrades on Centersquare’s terms, accelerate retrofits for high-density AI aisles, deploy liquid-ready thermal topologies, and add incremental power blocks without navigating landlord approval cycles. This structural flexibility aligns directly with the platform’s “AI-era backbone” positioning. 2) Eight Additional Data Centers Across Six Metros The acquisitions broaden scale in fast-rising secondary markets—Tulsa, Nashville, Raleigh—while deepening Centersquare’s presence in Dallas and expanding its Canadian footprint in Toronto and Montréal. Dallas remains a core scaling hub, but Nashville and Raleigh are increasingly important for enterprises modernizing their stacks and deploying regional AI workloads at lower cost and with faster timelines than congested Tier-1 corridors. Tulsa provides a network-adjacent, cost-efficient option for disaster recovery, edge aggregation, and latency-tolerant compute. In Canada, Toronto and Montréal offer strong enterprise demand, attractive economics, and grid advantages—including Québec’s hydro-powered, low-carbon energy mix—that position them well for AI training spillover and inference workloads requiring reliable, competitively priced power. 3) Self-Funded With Cash on Hand In the current rate environment, funding the entire $1 billion package

Microsoft will invest $80B in AI data centers in fiscal 2025

And Microsoft isn’t the only one that is ramping up its investments into AI-enabled data centers. Rival cloud service providers are all investing in either upgrading or opening new data centers to capture a larger chunk of business from developers and users of large language models (LLMs). In a report published in October 2024, Bloomberg Intelligence estimated that demand for generative AI would push Microsoft, AWS, Google, Oracle, Meta, and Apple would between them devote $200 billion to capex in 2025, up from $110 billion in 2023. Microsoft is one of the biggest spenders, followed closely by Google and AWS, Bloomberg Intelligence said. Its estimate of Microsoft’s capital spending on AI, at $62.4 billion for calendar 2025, is lower than Smith’s claim that the company will invest $80 billion in the fiscal year to June 30, 2025. Both figures, though, are way higher than Microsoft’s 2020 capital expenditure of “just” $17.6 billion. The majority of the increased spending is tied to cloud services and the expansion of AI infrastructure needed to provide compute capacity for OpenAI workloads. Separately, last October Amazon CEO Andy Jassy said his company planned total capex spend of $75 billion in 2024 and even more in 2025, with much of it going to AWS, its cloud computing division.

John Deere unveils more autonomous farm machines to address skill labor shortage

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Self-driving tractors might be the path to self-driving cars. John Deere has revealed a new line of autonomous machines and tech across agriculture, construction and commercial landscaping. The Moline, Illinois-based John Deere has been in business for 187 years, yet it’s been a regular as a non-tech company showing off technology at the big tech trade show in Las Vegas and is back at CES 2025 with more autonomous tractors and other vehicles. This is not something we usually cover, but John Deere has a lot of data that is interesting in the big picture of tech. The message from the company is that there aren’t enough skilled farm laborers to do the work that its customers need. It’s been a challenge for most of the last two decades, said Jahmy Hindman, CTO at John Deere, in a briefing. Much of the tech will come this fall and after that. He noted that the average farmer in the U.S. is over 58 and works 12 to 18 hours a day to grow food for us. And he said the American Farm Bureau Federation estimates there are roughly 2.4 million farm jobs that need to be filled annually; and the agricultural work force continues to shrink. (This is my hint to the anti-immigration crowd). John Deere’s autonomous 9RX Tractor. Farmers can oversee it using an app. While each of these industries experiences their own set of challenges, a commonality across all is skilled labor availability. In construction, about 80% percent of contractors struggle to find skilled labor. And in commercial landscaping, 86% of landscaping business owners can’t find labor to fill open positions, he said. “They have to figure out how to do

2025 playbook for enterprise AI success, from agents to evals

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More 2025 is poised to be a pivotal year for enterprise AI. The past year has seen rapid innovation, and this year will see the same. This has made it more critical than ever to revisit your AI strategy to stay competitive and create value for your customers. From scaling AI agents to optimizing costs, here are the five critical areas enterprises should prioritize for their AI strategy this year. 1. Agents: the next generation of automation AI agents are no longer theoretical. In 2025, they’re indispensable tools for enterprises looking to streamline operations and enhance customer interactions. Unlike traditional software, agents powered by large language models (LLMs) can make nuanced decisions, navigate complex multi-step tasks, and integrate seamlessly with tools and APIs. At the start of 2024, agents were not ready for prime time, making frustrating mistakes like hallucinating URLs. They started getting better as frontier large language models themselves improved. “Let me put it this way,” said Sam Witteveen, cofounder of Red Dragon, a company that develops agents for companies, and that recently reviewed the 48 agents it built last year. “Interestingly, the ones that we built at the start of the year, a lot of those worked way better at the end of the year just because the models got better.” Witteveen shared this in the video podcast we filmed to discuss these five big trends in detail. Models are getting better and hallucinating less, and they’re also being trained to do agentic tasks. Another feature that the model providers are researching is a way to use the LLM as a judge, and as models get cheaper (something we’ll cover below), companies can use three or more models to

OpenAI’s red teaming innovations define new essentials for security leaders in the AI era

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More OpenAI has taken a more aggressive approach to red teaming than its AI competitors, demonstrating its security teams’ advanced capabilities in two areas: multi-step reinforcement and external red teaming. OpenAI recently released two papers that set a new competitive standard for improving the quality, reliability and safety of AI models in these two techniques and more. The first paper, “OpenAI’s Approach to External Red Teaming for AI Models and Systems,” reports that specialized teams outside the company have proven effective in uncovering vulnerabilities that might otherwise have made it into a released model because in-house testing techniques may have missed them. In the second paper, “Diverse and Effective Red Teaming with Auto-Generated Rewards and Multi-Step Reinforcement Learning,” OpenAI introduces an automated framework that relies on iterative reinforcement learning to generate a broad spectrum of novel, wide-ranging attacks. Going all-in on red teaming pays practical, competitive dividends It’s encouraging to see competitive intensity in red teaming growing among AI companies. When Anthropic released its AI red team guidelines in June of last year, it joined AI providers including Google, Microsoft, Nvidia, OpenAI, and even the U.S.’s National Institute of Standards and Technology (NIST), which all had released red teaming frameworks. Investing heavily in red teaming yields tangible benefits for security leaders in any organization. OpenAI’s paper on external red teaming provides a detailed analysis of how the company strives to create specialized external teams that include cybersecurity and subject matter experts. The goal is to see if knowledgeable external teams can defeat models’ security perimeters and find gaps in their security, biases and controls that prompt-based testing couldn’t find. What makes OpenAI’s recent papers noteworthy is how well they define using human-in-the-middle