Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Well-funded French AI startup Mistral is content to go its own way. In a sea of competing reasoning models, the company today introduced Mistral OCR, a new Optical Character Recognition (OCR) API designed to provide advanced document understanding capabilities. The API extracts content—including handwritten notes, typed text, images, tables, and equations—from unstructured PDFs and images with high accuracy, presenting in a structured format. Structured data is information that is organized in a predefined manner, typically using rows and columns, making it easy to search and analyze. Common examples include names, addresses, and financial transactions stored in databases or spreadsheets.  In contrast, unstructured data lacks a specific format or structure, making it more challenging to process and analyze. This category encompasses a wide range of data types, such as emails, social media posts, videos, images, and audio files. Since unstructured data doesn’t fit neatly into traditional databases, specialized tools and techniques, like natural language processing and machine learning, are often employed to extract meaningful insights from it.  Understanding the distinction between these data types is crucial for businesses aiming to effectively manage and leverage their information assets. With multilingual support, fast processing speeds, and integration with large language models for document understanding, Mistral OCR is positioned to assist organizations in making their documentation AI-ready. Given that, according to Mistral’s blog post announcing the new API, 90% of all business information is unstructured, the new API should be a huge boon to organizations seeking to digitize and catalog their data for use in AI applications or internal/external knowledge bases. A new gold standard for OCR Mistral OCR aims to improve how organizations process and analyze complex documents. Unlike traditional OCR solutions that primarily

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Anthropic has launched a significant overhaul to its developer platform today, introducing team collaboration features and extended reasoning capabilities for its Claude AI assistant that aim to solve major pain points for organizations implementing artificial intelligence solutions. The upgraded Anthropic Console now enables cross-functional teams to collaborate on AI prompts—the text instructions that guide AI models—while also supporting the company’s latest Claude 3.7 Sonnet model with new controls for complex problem-solving. “We built our shareable prompts to help our customers and developers work together effectively on prompt development,” an Anthropic spokesperson told VentureBeat. “What we learned from talking to customers was that prompt creation rarely happens in isolation. It’s a team effort involving developers, subject matter experts, product managers, and QA folks all trying to get the best results.” The move addresses a growing challenge for enterprises adopting AI: coordinating prompt engineering work across technical and business teams. Before this update, companies often resorted to sharing prompts through documents or messaging apps, creating version control issues and knowledge silos. How Claude’s new thinking controls balance advanced AI power with budget-friendly cost management The updated platform also introduces “extended thinking controls” for Claude 3.7 Sonnet, allowing developers to specify when the AI should use deeper reasoning while setting budget limits to control costs. “Claude 3.7 Sonnet gives you two modes in one model: standard mode for quick responses and extended thinking mode when you need deeper problem-solving,” the spokesperson told VentureBeat. “In extended thinking mode, Claude takes time to work through problems step-by-step, similar to how humans approach complex challenges.” This dual approach helps companies balance performance with expenditure—a key consideration as AI implementation costs come under greater scrutiny amid widespread adoption.

Despite the AI hype, many tech companies still rely heavily on machine learning to power critical applications, from personalized recommendations to fraud detection. 

I’ve seen firsthand how undetected drifts can result in significant costs — missed fraud detection, lost revenue, and suboptimal business outcomes, just to name a few. So, it’s crucial to have robust monitoring in place if your company has deployed or plans to deploy machine learning models into production.

Undetected Model Drift can lead to significant financial losses, operational inefficiencies, and even damage to a company’s reputation. To mitigate these risks, it’s important to have effective model monitoring, which involves:

Tracking model performance

Monitoring feature distributions

Detecting both univariate and multivariate drifts

A well-implemented monitoring system can help identify issues early, saving considerable time, money, and resources.

In this comprehensive guide, I’ll provide a framework on how to think about and implement effective Model Monitoring, helping you stay ahead of potential issues and ensure stability and reliability of your models in production.

What’s the difference between feature drift and score drift?

Score drift refers to a gradual change in the distribution of model scores. If left unchecked, this could lead to a decline in model performance, making the model less accurate over time.

On the other hand, feature drift occurs when one or more features experience changes in the distribution. These changes in feature values can affect the underlying relationships that the model has learned, and ultimately lead to inaccurate model predictions.

Simulating score shifts

To model real-world fraud detection challenges, I created a synthetic dataset with five financial transaction features.

The reference dataset represents the original distribution, while the production dataset introduces shifts to simulate an increase in high-value transactions without PIN verification on newer accounts, indicating an increase in fraud.

Each feature has different underlying distributions:

Transaction Amount: Log-normal distribution (right-skewed with a long tail)

Account Age (months): clipped normal distribution between 0 to 60 (assuming a 5-year-old company)

Time Since Last Transaction: Exponential distribution

Transaction Count: Poisson distribution

Entered PIN: Binomial distribution.

To approximate model scores, I randomly assigned weights to these features and applied a sigmoid function to constrain predictions between 0 to 1. This mimics how a logistic regression fraud model generates risk scores.

As shown in the plot below:

Drifted features: Transaction Amount, Account Age, Transaction Count, and Entered PIN all experienced shifts in distribution, scale, or relationships.

Distribution of drifted features (image by author)

Stable feature: Time Since Last Transaction remained unchanged.

Distribution of stable feature (image by author)

Drifted scores: As a result of the drifted features, the distribution in model scores has also changed.

Distribution of model scores (image by author)

This setup allows us to analyze how feature drift impacts model scores in production.

Detecting model score drift using PSI

To monitor model scores, I used population stability index (PSI) to measure how much model score distribution has shifted over time.

PSI works by binning continuous model scores and comparing the proportion of scores in each bin between the reference and production datasets. It compares the differences in proportions and their logarithmic ratios to compute a single summary statistic to quantify the drift.

Python implementation:

# Define function to calculate PSI given two datasets
def calculate_psi(reference, production, bins=10):
# Discretize scores into bins
min_val, max_val = 0, 1
bin_edges = np.linspace(min_val, max_val, bins + 1)

# Calculate proportions in each bin
ref_counts, _ = np.histogram(reference, bins=bin_edges)
prod_counts, _ = np.histogram(production, bins=bin_edges)

ref_proportions = ref_counts / len(reference)
prod_proportions = prod_counts / len(production)

# Avoid division by zero
ref_proportions = np.clip(ref_proportions, 1e-8, 1)
prod_proportions = np.clip(prod_proportions, 1e-8, 1)

# Calculate PSI for each bin
psi = np.sum((ref_proportions – prod_proportions) * np.log(ref_proportions / prod_proportions))

return psi

# Calculate PSI
psi_value = calculate_psi(ref_data[‘model_score’], prod_data[‘model_score’], bins=10)
print(f”PSI Value: {psi_value}”)

Below is a summary of how to interpret PSI values:

PSI < 0.1: No drift, or very minor drift (distributions are almost identical).

0.1 ≤ PSI < 0.25: Some drift. The distributions are somewhat different.

0.25 ≤ PSI < 0.5: Moderate drift. A noticeable shift between the reference and production distributions.

PSI ≥ 0.5: Significant drift. There is a large shift, indicating that the distribution in production has changed substantially from the reference data.

Histogram of model score distributions (image by author)

The PSI value of 0.6374 suggests a significant drift between our reference and production datasets. This aligns with the histogram of model score distributions, which visually confirms the shift towards higher scores in production — indicating an increase in risky transactions.

Detecting feature drift

Kolmogorov-Smirnov test for numeric features

The Kolmogorov-Smirnov (K-S) test is my preferred method for detecting drift in numeric features, because it is non-parametric, meaning it doesn’t assume a normal distribution.

The test compares a feature’s distribution in the reference and production datasets by measuring the maximum difference between the empirical cumulative distribution functions (ECDFs). The resulting K-S statistic ranges from 0 to 1:

0 indicates no difference between the two distributions.

Values closer to 1 suggest a greater shift.

Python implementation:

# Create an empty dataframe
ks_results = pd.DataFrame(columns=['Feature', 'KS Statistic', 'p-value', 'Drift Detected'])

# Loop through all features and perform the K-S test
for col in numeric_cols:
ks_stat, p_value = ks_2samp(ref_data[col], prod_data[col])
drift_detected = p_value < 0.05

# Store results in the dataframe
ks_results = pd.concat([
ks_results,
pd.DataFrame({
'Feature': [col],
'KS Statistic': [ks_stat],
'p-value': [p_value],
'Drift Detected': [drift_detected]
})
], ignore_index=True)

Below are ECDF charts of the four numeric features in our dataset:

ECDFs of four numeric features (image by author)

Let’s look at the account age feature as an example: the x-axis represents account age (0-50 months), while the y-axis shows the ECDF for both reference and production datasets. The production dataset skews towards newer accounts, as it has a larger proportion of observations have lower account ages.

Chi-Square test for categorical features

To detect shifts in categorical and boolean features, I like to use the Chi-Square test.

This test compares the frequency distribution of a categorical feature in the reference and production datasets, and returns two values:

Chi-Square statistic: A higher value indicates a greater shift between the reference and production datasets.

P-value: A p-value below 0.05 suggests that the difference between the reference and production datasets is statistically significant, indicating potential feature drift.

Python implementation:

# Create empty dataframe with corresponding column names
chi2_results = pd.DataFrame(columns=['Feature', 'Chi-Square Statistic', 'p-value', 'Drift Detected'])

for col in categorical_cols:
# Get normalized value counts for both reference and production datasets
ref_counts = ref_data[col].value_counts(normalize=True)
prod_counts = prod_data[col].value_counts(normalize=True)

# Ensure all categories are represented in both
all_categories = set(ref_counts.index).union(set(prod_counts.index))
ref_counts = ref_counts.reindex(all_categories, fill_value=0)
prod_counts = prod_counts.reindex(all_categories, fill_value=0)

# Create contingency table
contingency_table = np.array([ref_counts * len(ref_data), prod_counts * len(prod_data)])

# Perform Chi-Square test
chi2_stat, p_value, _, _ = chi2_contingency(contingency_table)
drift_detected = p_value < 0.05

# Store results in chi2_results dataframe
chi2_results = pd.concat([
chi2_results,
pd.DataFrame({
'Feature': [col],
'Chi-Square Statistic': [chi2_stat],
'p-value': [p_value],
'Drift Detected': [drift_detected]
})
], ignore_index=True)

The Chi-Square statistic of 57.31 with a p-value of 3.72e-14 confirms a large shift in our categorical feature, Entered PIN. This finding aligns with the histogram below, which visually illustrates the shift:

Distribution of categorical feature (image by author)

Detecting multivariate shifts

Spearman Correlation for shifts in pairwise interactions

In addition to monitoring individual feature shifts, it’s important to track shifts in relationships or interactions between features, known as multivariate shifts. Even if the distributions of individual features remain stable, multivariate shifts can signal meaningful differences in the data.

By default, Pandas’ .corr() function calculates Pearson correlation, which only captures linear relationships between variables. However, relationships between features are often non-linear yet still follow a consistent trend.

To capture this, we use Spearman correlation to measure monotonic relationships between features. It captures whether features change together in a consistent direction, even if their relationship isn’t strictly linear.

To assess shifts in feature relationships, we compare:

Reference correlation (ref_corr): Captures historical feature relationships in the reference dataset.

Production correlation (prod_corr): Captures new feature relationships in production.

Absolute difference in correlation: Measures how much feature relationships have shifted between the reference and production datasets. Higher values indicate more significant shifts.

Python implementation:

# Calculate correlation matrices
ref_corr = ref_data.corr(method='spearman')
prod_corr = prod_data.corr(method='spearman')

# Calculate correlation difference
corr_diff = abs(ref_corr – prod_corr)

Example: Change in correlation

Now, let’s look at the correlation between transaction_amount and account_age_in_months:

In ref_corr, the correlation is 0.00095, indicating a weak relationship between the two features.

In prod_corr, the correlation is -0.0325, indicating a weak negative correlation.

Absolute difference in the Spearman correlation is 0.0335, which is a small but noticeable shift.

The absolute difference in correlation indicates a shift in the relationship between transaction_amount and account_age_in_months.

There used to be no relationship between these two features, but the production dataset indicates that there is now a weak negative correlation, meaning that newer accounts have higher transaction accounts. This is spot on!

Autoencoder for complex, high-dimensional multivariate shifts

In addition to monitoring pairwise interactions, we can also look for shifts across more dimensions in the data.

Autoencoders are powerful tools for detecting high-dimensional multivariate shifts, where multiple features collectively change in ways that may not be apparent from looking at individual feature distributions or pairwise correlations.

An autoencoder is a neural network that learns a compressed representation of data through two components:

Encoder: Compresses input data into a lower-dimensional representation.

Decoder: Reconstructs the original input from the compressed representation.

To detect shifts, we compare the reconstructed output to the original input and compute the reconstruction loss.

Low reconstruction loss → The autoencoder successfully reconstructs the data, meaning the new observations are similar to it has seen and learned.

High reconstruction loss → The production data deviates significantly from the learned patterns, indicating potential drift.

Unlike traditional drift metrics that focus on individual features or pairwise relationships, autoencoders capture complex, non-linear dependencies across multiple variables simultaneously.

Python implementation:

ref_features = ref_data[numeric_cols + categorical_cols]
prod_features = prod_data[numeric_cols + categorical_cols]

# Normalize the data
scaler = StandardScaler()
ref_scaled = scaler.fit_transform(ref_features)
prod_scaled = scaler.transform(prod_features)

# Split reference data into train and validation
np.random.shuffle(ref_scaled)
train_size = int(0.8 * len(ref_scaled))
train_data = ref_scaled[:train_size]
val_data = ref_scaled[train_size:]

# Build autoencoder
input_dim = ref_features.shape[1]
encoding_dim = 3
# Input layer
input_layer = Input(shape=(input_dim, ))
# Encoder
encoded = Dense(8, activation="relu")(input_layer)
encoded = Dense(encoding_dim, activation="relu")(encoded)
# Decoder
decoded = Dense(8, activation="relu")(encoded)
decoded = Dense(input_dim, activation="linear")(decoded)
# Autoencoder
autoencoder = Model(input_layer, decoded)
autoencoder.compile(optimizer="adam", loss="mse")

# Train autoencoder
history = autoencoder.fit(
train_data, train_data,
epochs=50,
batch_size=64,
shuffle=True,
validation_data=(val_data, val_data),
verbose=0
)

# Calculate reconstruction error
ref_pred = autoencoder.predict(ref_scaled, verbose=0)
prod_pred = autoencoder.predict(prod_scaled, verbose=0)

ref_mse = np.mean(np.power(ref_scaled – ref_pred, 2), axis=1)
prod_mse = np.mean(np.power(prod_scaled – prod_pred, 2), axis=1)

The charts below show the distribution of reconstruction loss between both datasets.

Distribution of reconstruction loss between actuals and predictions (image by author)

The production dataset has a higher mean reconstruction error than that of the reference dataset, indicating a shift in the overall data. This aligns with the changes in the production dataset with a higher number of newer accounts with high-value transactions.

Summarizing

Model monitoring is an essential, yet often overlooked, responsibility for data scientists and machine learning engineers.

All the statistical methods led to the same conclusion, which aligns with the observed shifts in the data: they detected a trend in production towards newer accounts making higher-value transactions. This shift resulted in higher model scores, signaling an increase in potential fraud.

In this post, I covered techniques for detecting drift on three different levels:

Model score drift: Using Population Stability Index (PSI)

Individual feature drift: Using Kolmogorov-Smirnov test for numeric features and Chi-Square test for categorical features

Multivariate drift: Using Spearman correlation for pairwise interactions and autoencoders for high-dimensional, multivariate shifts.

These are just a few of the techniques I rely on for comprehensive monitoring — there are plenty of other equally valid statistical methods that can also detect drift effectively.

Detected shifts often point to underlying issues that warrant further investigation. The root cause could be as serious as a data collection bug, or as minor as a time change like daylight savings time adjustments.

There are also fantastic python packages, like evidently.ai, that automate many of these comparisons. However, I believe there’s significant value in deeply understanding the statistical techniques behind drift detection, rather than relying solely on these tools.

What’s the model monitoring process like at places you’ve worked?

Want to build your AI skills?

👉🏻 I run the AI Weekender and write weekly blog posts on data science, AI weekend projects, career advice for professionals in data.

Resources

Peer Global Inc announced today that it has raised $10.5 million in its latest round of funding for its metaverse game engine, which it plans to use to build out its team and to accelerate AI product development. In addition to the funding, the company also launched its personal planets feature — an in-engine feature that allows users to create their own 3D social hubs. According to founder Tony Tran, this new form of social engagement is intended to be a tonic to more addictive, static forms of social media.

Peer’s total investment numbers sit at $65.5 million, all from angel investors. The Family Office of Tommy Mai is the sole investor in this round of funding. The company will build out its AI features, which make up the backbone of its persistent world. AI is also one of the tools that the company offers for developers who wish to build their experiences on Peer. Within the game’s engine, all games and experiences would be connected to each other.

Mai said in a statement, “Websites, social networks, and digital brand experiences today are flat. People have short attention spans. AI will push everything into spatial experiences, and Peer is leading the way. We’re really excited about the potential for this technology and think Tony and team are the ones to get this right.”

Peer wants to redefine the social experience

Speaking with GamesBeat, Tran said that Peer reshapes those same social engagement forces into something that gets users going outside and engaging with the world around them. “We use location sharing dynamically within the platform, to create a living map where people can see each other moving in real-time, sparking spontaneous interactions and collaboration rather than passive consumption. This transforms the energy of traditional FOMO into something constructive towards exploration, discovery, and shared experiences. Instead of feeling left out, users are invited into the action, whether it’s meeting up with friends, joining an event, or co-creating within the AI-driven world.”

Tran also notes the advantages of using AI for developers: “What we have is a social interface where AI can create to its maximum potential—generating games, characters, and entire experiences on demand—for mass consumption. Peer leverages AI to bring the visual side of the metaverse to life in a way that other experiences can’t. All other metaverses exist in isolation, where in Peer, AI acts as the connective tissue. It links people, places, and experiences in real time, forming an instant information layer that keeps everything fluid, responsive, and intelligent.”

According to Tran, Peer plans to launch its location-based mechanics in the near-future, as well as nascent monetization mechanics, such as digital property sales and premium experiences. For the long-term, the company plans to make the Peer experience accessible on any device. To build it at scale, they plan to offer subscription tiers, AI-based advertising and a full digital economy.

Tran told GamesBeat, “Peer’s AI integration allows for dynamic, procedurally generated environments, meaning developers can create living worlds that adapt to player actions… Peer gives developers a platform to build games that are not just played but lived in—unlocking new possibilities for immersive, connected gameplay.”

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More One goal for an agentic future is for AI agents from different organizations to freely and seamlessly talk to one another. But getting to that point requires interoperability, and these agents may have been built with different LLMs, data frameworks and code. To achieve interoperability, developers of these agents must agree on how they can communicate with each other. This is a challenging task.  A group of companies, including Cisco, LangChain, LlamaIndex, Galileo and Glean, have now created AGNTCY, an open-source collective with the goal of creating an industry-standard agent interoperability language. AGNTCY aims to make it easy for any AI agent to communicate and exchange data with another. Uniting AI Agents “Just like when the cloud and the internet came about and accelerated applications and all social interactions at a global scale, we want to build the Internet of Agents that accelerate all of human work at a global scale,” said Vijoy Pandey, head of Outshift by Cisco, Cisco’s incubation arm, in an interview with VentureBeat.  Pandey likened AGNTCY to the advent of the Transmission Control Protocol/Internet Protocol (TCP/IP) and the domain name system (DNS), which helped organize the internet and allowed for interconnections between different computer systems.  “The way we are thinking about this problem is that the original internet allowed for humans and servers and web farms to all come together,” he said. “This is the Internet of Agents, and the only way to do that is to make it open and interoperable.” Cisco, LangChain and Galileo will act as AGNTCY’s core maintainers, with Glean and LlamaIndex as contributors. However, this structure may change as the collective adds more members.  Standardizing a fast-moving industry AI agents cannot be

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Thomas Wolf, co-founder of AI company Hugging Face, has issued a stark challenge to the tech industry’s most optimistic visions of artificial intelligence, arguing that today’s AI systems are fundamentally incapable of delivering the scientific revolutions their creators promise. In a provocative blog post published on his personal website this morning, Wolf directly confronts the widely circulated vision of Anthropic CEO Dario Amodei, who predicted that advanced AI would deliver a “compressed 21st century” where decades of scientific progress could unfold in just years. “I’m afraid AI won’t give us a ‘compressed 21st century,’” Wolf writes in his post, arguing that current AI systems are more likely to produce “a country of yes-men on servers” rather than the “country of geniuses” that Amodei envisions. The exchange highlights a growing divide in how AI leaders think about the technology’s potential to transform scientific discovery and problem-solving, with major implications for business strategies, research priorities, and policy decisions. From straight-A student to ‘mediocre researcher’: Why academic excellence doesn’t equal scientific genius Wolf grounds his critique in personal experience. Despite being a straight-A student who attended MIT, he describes discovering he was a “pretty average, underwhelming, mediocre researcher” when he began his PhD work. This experience shaped his view that academic success and scientific genius require fundamentally different mental approaches — the former rewarding conformity, the latter demanding rebellion against established thinking. “The main mistake people usually make is thinking Newton or Einstein were just scaled-up good students,” Wolf explains. “A real science breakthrough is Copernicus proposing, against all the knowledge of his days — in ML terms we would say ‘despite all his training dataset’ — that the earth may orbit the sun