Mastering Prompt Engineering with Functional Testing: A Systematic Guide to Reliable LLM Outputs

Stay Ahead, Stay ONMINE

Mastering Prompt Engineering with Functional Testing: A Systematic Guide to Reliable LLM Outputs

Creating efficient prompts for large language models often starts as a simple task… but it doesn’t always stay that way. Initially, following basic best practices seems sufficient: adopt the persona of a specialist, write clear instructions, require a specific response format, and include a few relevant examples. But as requirements multiply, contradictions emerge, and even minor modifications can introduce unexpected failures. What was working perfectly in one prompt version suddenly breaks in another. If you have ever felt trapped in an endless loop of trial and error, adjusting one rule only to see another one fail, you’re not alone! The reality is that traditional prompt optimisation is clearly missing a structured, more scientific approach that will help to ensure reliability. That’s where functional testing for prompt engineering comes in! This approach, inspired by methodologies of experimental science, leverages automated input-output testing with multiple iterations and algorithmic scoring to turn prompt engineering into a measurable, data-driven process. No more guesswork. No more tedious manual validation. Just precise and repeatable results that allow you to fine-tune prompts efficiently and confidently. In this article, we will explore a systematic approach for mastering prompt engineering, which ensures your Llm outputs will be efficient and reliable even for the most complex AI tasks. Balancing precision and consistency in prompt optimisation Adding a large set of rules to a prompt can introduce partial contradictions between rules and lead to unexpected behaviors. This is especially true when following a pattern of starting with a general rule and following it with multiple exceptions or specific contradictory use cases. Adding specific rules and exceptions can cause conflict with the primary instruction and, potentially, with each other. What might seem like a minor modification can unexpectedly impact other aspects of a prompt. This is not only true when adding a new rule but also when adding more detail to an existing rule, like changing the order of the set of instructions or even simply rewording it. These minor modifications can unintentionally change the way the model interprets and prioritizes the set of instructions. The more details you add to a prompt, the greater the risk of unintended side effects. By trying to give too many details to every aspect of your task, you increase as well the risk of getting unexpected or deformed results. It is, therefore, essential to find the right balance between clarity and a high level of specification to maximise the relevance and consistency of the response. At a certain point, fixing one requirement can break two others, creating the frustrating feeling of taking one step forward and two steps backward in the optimization process. Testing each change manually becomes quickly overwhelming. This is especially true when one needs to optimize prompts that must follow numerous competing specifications in a complex AI task. The process cannot simply be about modifying the prompt for one requirement after the other, hoping the previous instruction remains unaffected. It also can’t be a system of selecting examples and checking them by hand. A better process with a more scientific approach should focus on ensuring repeatability and reliability in prompt optimization. From laboratory to AI: Why testing LLM responses requires multiple iterations Science teaches us to use replicates to ensure reproducibility and build confidence in an experiment’s results. I have been working in academic research in chemistry and biology for more than a decade. In those fields, experimental results can be influenced by a multitude of factors that can lead to significant variability. To ensure the reliability and reproducibility of experimental results, scientists mostly employ a method known as triplicates. This approach involves conducting the same experiment three times under identical conditions, allowing the experimental variations to be of minor importance in the result. Statistical analysis (standard mean and deviation) conducted on the results, mostly in biology, allows the author of an experiment to determine the consistency of the results and strengthens confidence in the findings. Just like in biology and chemistry, this approach can be used with LLMs to achieve reliable responses. With LLMs, the generation of responses is non-deterministic, meaning that the same input can lead to different outputs due to the probabilistic nature of the models. This variability is challenging when evaluating the reliability and consistency of LLM outputs. In the same way that biological/chemical experiments require triplicates to ensure reproducibility, testing LLMs should need multiple iterations to measure reproducibility. A single test by use case is, therefore, not sufficient because it does not represent the inherent variability in LLM responses. At least five iterations per use case allow for a better assessment. By analyzing the consistency of the responses across these iterations, one can better evaluate the reliability of the model and identify any potential issues or variation. It ensures that the output of the model is correctly controlled. Multiply this across 10 to 15 different prompt requirements, and one can easily understand how, without a structured testing approach, we end up spending time in trial-and-error testing with no efficient way to assess quality. A systematic approach: Functional testing for prompt optimization To address these challenges, a structured evaluation methodology can be used to ease and accelerate the testing process and enhance the reliability of LLM outputs. This approach has several key components: Data fixtures: The approach’s core center is the data fixtures, which are composed of predefined input-output pairs specifically created for prompt testing. These fixtures serve as controlled scenarios that represent the various requirements and edge cases the LLM must handle. By using a diverse set of fixtures, the performance of the prompt can be evaluated efficiently across different conditions. Automated test validation: This approach automates the validation of the requirements on a set of data fixtures by comparison between the expected outputs defined in the fixtures and the LLM response. This automated comparison ensures consistency and reduces the potential for human error or bias in the evaluation process. It allows for quick identification of discrepancies, enabling fine and efficient prompt adjustments. Multiple iterations: To assess the inherent variability of the LLM responses, this method runs multiple iterations for each test case. This iterative approach mimics the triplicate method used in biological/chemical experiments, providing a more robust dataset for analysis. By observing the consistency of responses across iterations, we can better assess the stability and reliability of the prompt. Algorithmic scoring: The results of each test case are scored algorithmically, reducing the need for long and laborious « human » evaluation. This scoring system is designed to be objective and quantitative, providing clear metrics for assessing the performance of the prompt. And by focusing on measurable outcomes, we can make data-driven decisions to optimize the prompt effectively. Step 1: Defining test data fixtures Selecting or creating compatible test data fixtures is the most challenging step of our systematic approach because it requires careful thought. A fixture is not only any input-output pair; it must be crafted meticulously to evaluate the most accurate as possible performance of the LLM for a specific requirement. This process requires: 1. A deep understanding of the task and the behavior of the model to make sure the selected examples effectively test the expected output while minimizing ambiguity or bias. 2. Foresight into how the evaluation will be conducted algorithmically during the test. The quality of a fixture, therefore, depends not only on the good representativeness of the example but also on ensuring it can be efficiently tested algorithmically. A fixture consists of: • Input example: This is the data that will be given to the LLM for processing. It should represent a typical or edge-case scenario that the LLM is expected to handle. The input should be designed to cover a wide range of possible variations that the LLM might have to deal with in production. • Expected output: This is the expected result that the LLM should produce with the provided input example. It is used for comparison with the actual LLM response output during validation. Step 2: Running automated tests Once the test data fixtures are defined, the next step involves the execution of automated tests to systematically evaluate the performance of the LLM response on the selected use cases. As previously stated, this process makes sure that the prompt is thoroughly tested against various scenarios, providing a reliable evaluation of its efficiency. Execution process 1. Multiple iterations: For each test use case, the same input is provided to the LLM multiple times. A simple for loop in nb_iter with nb_iter = 5 and voila! 2. Response comparison: After each iteration, the LLM response is compared to the expected output of the fixture. This comparison checks whether the LLM has correctly processed the input according to the specified requirements. 3. Scoring mechanism: Each comparison results in a score: ◦ Pass (1): The response matches the expected output, indicating that the LLM has correctly handled the input. ◦ Fail (0): The response does not match the expected output, signaling a discrepancy that needs to be fixed. 4. Final score calculation: The scores from all iterations are aggregated to calculate the overall final score. This score represents the proportion of successful responses out of the total number of iterations. A high score, of course, indicates high prompt performance and reliability. Example: Removing author signatures from an article Let’s consider a simple scenario where an AI task is to remove author signatures from an article. To efficiently test this functionality, we need a set of fixtures that represent the various signature styles. A dataset for this example could be: Example InputExpected OutputA long articleJean LeblancThe long articleA long articleP. W. HartigThe long articleA long articleMCZThe long article Validation process: Signature removal check: The validation function checks if the signature is absent from the rewritten text. This is easily done programmatically by searching for the signature needle in the haystack output text. Test failure criteria: If the signature is still in the output, the test fails. This indicates that the LLM did not correctly remove the signature and that further adjustments to the prompt are required. If it is not, the test is passed. The test evaluation provides a final score that allows a data-driven assessment of the prompt efficiency. If it scores perfectly, there is no need for further optimization. However, in most cases, you will not get a perfect score because either the consistency of the LLM response to a case is low (for example, 3 out of 5 iterations scored positive) or there are edge cases that the model struggles with (0 out of 5 iterations). The feedback clearly indicates that there is still room for further improvements and it guides you to reexamine your prompt for ambiguous phrasing, conflicting rules, or edge cases. By continuously monitoring your score alongside your prompt modifications, you can incrementally reduce side effects, achieve greater efficiency and consistency, and approach an optimal and reliable output. A perfect score is, however, not always achievable with the selected model. Changing the model might just fix the situation. If it doesn’t, you know the limitations of your system and can take this fact into account in your workflow. With luck, this situation might just be solved in the near future with a simple model update. Benefits of this method Reliability of the result: Running five to ten iterations provides reliable statistics on the performance of the prompt. A single test run may succeed once but not twice, and consistent success for multiple iterations indicates a robust and well-optimized prompt. Efficiency of the process: Unlike traditional scientific experiments that may take weeks or months to replicate, automated testing of LLMs can be carried out quickly. By setting a high number of iterations and waiting for a few minutes, we can obtain a high-quality, reproducible evaluation of the prompt efficiency. Data-driven optimization: The score obtained from these tests provides a data-driven assessment of the prompt’s ability to meet requirements, allowing targeted improvements. Side-by-side evaluation: Structured testing allows for an easy assessment of prompt versions. By comparing the test results, one can identify the most effective set of parameters for the instructions (phrasing, order of instructions) to achieve the desired results. Quick iterative improvement: The ability to quickly test and iterate prompts is a real advantage to carefully construct the prompt ensuring that the previously validated requirements remain as the prompt increases in complexity and length. By adopting this automated testing approach, we can systematically evaluate and enhance prompt performance, ensuring consistent and reliable outputs with the desired requirements. This method saves time and provides a robust analytical tool for continuous prompt optimization. Systematic prompt testing: Beyond prompt optimization Implementing a systematic prompt testing approach offers more advantages than just the initial prompt optimization. This methodology is valuable for other aspects of AI tasks: 1. Model comparison: ◦ Provider evaluation: This approach allows the efficient comparison of different LLM providers, such as ChatGPT, Claude, Gemini, Mistral, etc., on the same tasks. It becomes easy to evaluate which model performs the best for their specific needs. ◦ Model version: State-of-the-art model versions are not always necessary when a prompt is well-optimized, even for complex AI tasks. A lightweight, faster version can provide the same results with a faster response. This approach allows a side-by-side comparison of the different versions of a model, such as Gemini 1.5 flash vs. 1.5 pro vs. 2.0 flash or ChatGPT 3.5 vs. 4o mini vs. 4o, and allows the data-driven selection of the model version. 2. Version upgrades: ◦ Compatibility verification: When a new model version is released, systematic prompt testing helps validate if the upgrade maintains or improves the prompt performance. This is crucial for ensuring that updates do not unintentionally break the functionality. ◦ Seamless Transitions: By identifying key requirements and testing them, this method can facilitate better transitions to new model versions, allowing fast adjustment when necessary in order to maintain high-quality outputs. 3. Cost optimization: ◦ Performance-to-cost ratio: Systematic prompt testing helps in choosing the best cost-effective model based on the performance-to-cost ratio. We can efficiently identify the most efficient option between performance and operational costs to get the best return on LLM costs. Overcoming the challenges The biggest challenge of this approach is the preparation of the set of test data fixtures, but the effort invested in this process will pay off significantly as time passes. Well-prepared fixtures save considerable debugging time and enhance model efficiency and reliability by providing a robust foundation for evaluating the LLM response. The initial investment is quickly returned by improved efficiency and effectiveness in LLM development and deployment. Quick pros and cons Key advantages: Continuous improvement: The ability to add more requirements over time while ensuring existing functionality stays intact is a significant advantage. This allows for the evolution of the AI task in response to new requirements, ensuring that the system remains up-to-date and efficient. Better maintenance: This approach enables the easy validation of prompt performance with LLM updates. This is crucial for maintaining high standards of quality and reliability, as updates can sometimes introduce unintended changes in behavior. More flexibility: With a set of quality control tests, switching LLM providers becomes more straightforward. This flexibility allows us to adapt to changes in the market or technological advancements, ensuring we can always use the best tool for the job. Cost optimization: Data-driven evaluations enable better decisions on performance-to-cost ratio. By understanding the performance gains of different models, we can choose the most cost-effective solution that meets the needs. Time savings: Systematic evaluations provide quick feedback, reducing the need for manual testing. This efficiency allows to quickly iterate on prompt improvement and optimization, accelerating the development process. Challenges Initial time investment: Creating test fixtures and evaluation functions can require a significant investment of time. Defining measurable validation criteria: Not all AI tasks have clear pass/fail conditions. Defining measurable criteria for validation can sometimes be challenging, especially for tasks that involve subjective or nuanced outputs. This requires careful consideration and may involve a difficult selection of the evaluation metrics. Cost associated with multiple tests: Multiple test use cases associated with 5 to 10 iterations can generate a high number of LLM requests for a single test automation. But if the cost of a single LLM call is neglectable, as it is in most cases for text input/output calls, the overall cost of a test remains minimal. Conclusion: When should you implement this approach? Implementing this systematic testing approach is, of course, not always necessary, especially for simple tasks. However, for complex AI workflows in which precision and reliability are critical, this approach becomes highly valuable by offering a systematic way to assess and optimize prompt performance, preventing endless cycles of trial and error. By incorporating functional testing principles into Prompt Engineering, we transform a traditionally subjective and fragile process into one that is measurable, scalable, and robust. Not only does it enhance the reliability of LLM outputs, it helps achieve continuous improvement and efficient resource allocation. The decision to implement systematic prompt Testing should be based on the complexity of your project. For scenarios demanding high precision and consistency, investing the time to set up this methodology can significantly improve outcomes and speed up the development processes. However, for simpler tasks, a more classical, lightweight approach may be sufficient. The key is to balance the need for rigor with practical considerations, ensuring that your testing strategy aligns with your goals and constraints. Thanks for reading!

If you have ever felt trapped in an endless loop of trial and error, adjusting one rule only to see another one fail, you’re not alone! The reality is that traditional prompt optimisation is clearly missing a structured, more scientific approach that will help to ensure reliability.

That’s where functional testing for prompt engineering comes in! This approach, inspired by methodologies of experimental science, leverages automated input-output testing with multiple iterations and algorithmic scoring to turn prompt engineering into a measurable, data-driven process.

No more guesswork. No more tedious manual validation. Just precise and repeatable results that allow you to fine-tune prompts efficiently and confidently.

In this article, we will explore a systematic approach for mastering prompt engineering, which ensures your Llm outputs will be efficient and reliable even for the most complex AI tasks.

Balancing precision and consistency in prompt optimisation

Adding a large set of rules to a prompt can introduce partial contradictions between rules and lead to unexpected behaviors. This is especially true when following a pattern of starting with a general rule and following it with multiple exceptions or specific contradictory use cases. Adding specific rules and exceptions can cause conflict with the primary instruction and, potentially, with each other.

What might seem like a minor modification can unexpectedly impact other aspects of a prompt. This is not only true when adding a new rule but also when adding more detail to an existing rule, like changing the order of the set of instructions or even simply rewording it. These minor modifications can unintentionally change the way the model interprets and prioritizes the set of instructions.

The more details you add to a prompt, the greater the risk of unintended side effects. By trying to give too many details to every aspect of your task, you increase as well the risk of getting unexpected or deformed results. It is, therefore, essential to find the right balance between clarity and a high level of specification to maximise the relevance and consistency of the response. At a certain point, fixing one requirement can break two others, creating the frustrating feeling of taking one step forward and two steps backward in the optimization process.

Testing each change manually becomes quickly overwhelming. This is especially true when one needs to optimize prompts that must follow numerous competing specifications in a complex AI task. The process cannot simply be about modifying the prompt for one requirement after the other, hoping the previous instruction remains unaffected. It also can’t be a system of selecting examples and checking them by hand. A better process with a more scientific approach should focus on ensuring repeatability and reliability in prompt optimization.

From laboratory to AI: Why testing LLM responses requires multiple iterations

Science teaches us to use replicates to ensure reproducibility and build confidence in an experiment’s results. I have been working in academic research in chemistry and biology for more than a decade. In those fields, experimental results can be influenced by a multitude of factors that can lead to significant variability. To ensure the reliability and reproducibility of experimental results, scientists mostly employ a method known as triplicates. This approach involves conducting the same experiment three times under identical conditions, allowing the experimental variations to be of minor importance in the result. Statistical analysis (standard mean and deviation) conducted on the results, mostly in biology, allows the author of an experiment to determine the consistency of the results and strengthens confidence in the findings.

Just like in biology and chemistry, this approach can be used with LLMs to achieve reliable responses. With LLMs, the generation of responses is non-deterministic, meaning that the same input can lead to different outputs due to the probabilistic nature of the models. This variability is challenging when evaluating the reliability and consistency of LLM outputs.

In the same way that biological/chemical experiments require triplicates to ensure reproducibility, testing LLMs should need multiple iterations to measure reproducibility. A single test by use case is, therefore, not sufficient because it does not represent the inherent variability in LLM responses. At least five iterations per use case allow for a better assessment. By analyzing the consistency of the responses across these iterations, one can better evaluate the reliability of the model and identify any potential issues or variation. It ensures that the output of the model is correctly controlled.

Multiply this across 10 to 15 different prompt requirements, and one can easily understand how, without a structured testing approach, we end up spending time in trial-and-error testing with no efficient way to assess quality.

A systematic approach: Functional testing for prompt optimization

To address these challenges, a structured evaluation methodology can be used to ease and accelerate the testing process and enhance the reliability of LLM outputs. This approach has several key components:

Data fixtures: The approach’s core center is the data fixtures, which are composed of predefined input-output pairs specifically created for prompt testing. These fixtures serve as controlled scenarios that represent the various requirements and edge cases the LLM must handle. By using a diverse set of fixtures, the performance of the prompt can be evaluated efficiently across different conditions.
Automated test validation: This approach automates the validation of the requirements on a set of data fixtures by comparison between the expected outputs defined in the fixtures and the LLM response. This automated comparison ensures consistency and reduces the potential for human error or bias in the evaluation process. It allows for quick identification of discrepancies, enabling fine and efficient prompt adjustments.
Multiple iterations: To assess the inherent variability of the LLM responses, this method runs multiple iterations for each test case. This iterative approach mimics the triplicate method used in biological/chemical experiments, providing a more robust dataset for analysis. By observing the consistency of responses across iterations, we can better assess the stability and reliability of the prompt.
Algorithmic scoring: The results of each test case are scored algorithmically, reducing the need for long and laborious « human » evaluation. This scoring system is designed to be objective and quantitative, providing clear metrics for assessing the performance of the prompt. And by focusing on measurable outcomes, we can make data-driven decisions to optimize the prompt effectively.

Step 1: Defining test data fixtures

Selecting or creating compatible test data fixtures is the most challenging step of our systematic approach because it requires careful thought. A fixture is not only any input-output pair; it must be crafted meticulously to evaluate the most accurate as possible performance of the LLM for a specific requirement. This process requires:

1. A deep understanding of the task and the behavior of the model to make sure the selected examples effectively test the expected output while minimizing ambiguity or bias.

2. Foresight into how the evaluation will be conducted algorithmically during the test.

The quality of a fixture, therefore, depends not only on the good representativeness of the example but also on ensuring it can be efficiently tested algorithmically.

A fixture consists of:

• Input example: This is the data that will be given to the LLM for processing. It should represent a typical or edge-case scenario that the LLM is expected to handle. The input should be designed to cover a wide range of possible variations that the LLM might have to deal with in production.

• Expected output: This is the expected result that the LLM should produce with the provided input example. It is used for comparison with the actual LLM response output during validation.

Step 2: Running automated tests

Once the test data fixtures are defined, the next step involves the execution of automated tests to systematically evaluate the performance of the LLM response on the selected use cases. As previously stated, this process makes sure that the prompt is thoroughly tested against various scenarios, providing a reliable evaluation of its efficiency.

Execution process

1. Multiple iterations: For each test use case, the same input is provided to the LLM multiple times. A simple for loop in nb_iter with nb_iter = 5 and voila!

2. Response comparison: After each iteration, the LLM response is compared to the expected output of the fixture. This comparison checks whether the LLM has correctly processed the input according to the specified requirements.

3. Scoring mechanism: Each comparison results in a score:

◦ Pass (1): The response matches the expected output, indicating that the LLM has correctly handled the input.

◦ Fail (0): The response does not match the expected output, signaling a discrepancy that needs to be fixed.

4. Final score calculation: The scores from all iterations are aggregated to calculate the overall final score. This score represents the proportion of successful responses out of the total number of iterations. A high score, of course, indicates high prompt performance and reliability.

Example: Removing author signatures from an article

Let’s consider a simple scenario where an AI task is to remove author signatures from an article. To efficiently test this functionality, we need a set of fixtures that represent the various signature styles.

A dataset for this example could be:

Example Input	Expected Output
A long article Jean Leblanc	The long article
A long article P. W. Hartig	The long article
A long article MCZ	The long article

Validation process:

Signature removal check: The validation function checks if the signature is absent from the rewritten text. This is easily done programmatically by searching for the signature needle in the haystack output text.
Test failure criteria: If the signature is still in the output, the test fails. This indicates that the LLM did not correctly remove the signature and that further adjustments to the prompt are required. If it is not, the test is passed.

The test evaluation provides a final score that allows a data-driven assessment of the prompt efficiency. If it scores perfectly, there is no need for further optimization. However, in most cases, you will not get a perfect score because either the consistency of the LLM response to a case is low (for example, 3 out of 5 iterations scored positive) or there are edge cases that the model struggles with (0 out of 5 iterations).

The feedback clearly indicates that there is still room for further improvements and it guides you to reexamine your prompt for ambiguous phrasing, conflicting rules, or edge cases. By continuously monitoring your score alongside your prompt modifications, you can incrementally reduce side effects, achieve greater efficiency and consistency, and approach an optimal and reliable output.

A perfect score is, however, not always achievable with the selected model. Changing the model might just fix the situation. If it doesn’t, you know the limitations of your system and can take this fact into account in your workflow. With luck, this situation might just be solved in the near future with a simple model update.

Benefits of this method

Reliability of the result: Running five to ten iterations provides reliable statistics on the performance of the prompt. A single test run may succeed once but not twice, and consistent success for multiple iterations indicates a robust and well-optimized prompt.
Efficiency of the process: Unlike traditional scientific experiments that may take weeks or months to replicate, automated testing of LLMs can be carried out quickly. By setting a high number of iterations and waiting for a few minutes, we can obtain a high-quality, reproducible evaluation of the prompt efficiency.
Data-driven optimization: The score obtained from these tests provides a data-driven assessment of the prompt’s ability to meet requirements, allowing targeted improvements.
Side-by-side evaluation: Structured testing allows for an easy assessment of prompt versions. By comparing the test results, one can identify the most effective set of parameters for the instructions (phrasing, order of instructions) to achieve the desired results.
Quick iterative improvement: The ability to quickly test and iterate prompts is a real advantage to carefully construct the prompt ensuring that the previously validated requirements remain as the prompt increases in complexity and length.

By adopting this automated testing approach, we can systematically evaluate and enhance prompt performance, ensuring consistent and reliable outputs with the desired requirements. This method saves time and provides a robust analytical tool for continuous prompt optimization.

Systematic prompt testing: Beyond prompt optimization

Implementing a systematic prompt testing approach offers more advantages than just the initial prompt optimization. This methodology is valuable for other aspects of AI tasks:

1. Model comparison:

◦ Provider evaluation: This approach allows the efficient comparison of different LLM providers, such as ChatGPT, Claude, Gemini, Mistral, etc., on the same tasks. It becomes easy to evaluate which model performs the best for their specific needs.

◦ Model version: State-of-the-art model versions are not always necessary when a prompt is well-optimized, even for complex AI tasks. A lightweight, faster version can provide the same results with a faster response. This approach allows a side-by-side comparison of the different versions of a model, such as Gemini 1.5 flash vs. 1.5 pro vs. 2.0 flash or ChatGPT 3.5 vs. 4o mini vs. 4o, and allows the data-driven selection of the model version.

2. Version upgrades:

◦ Compatibility verification: When a new model version is released, systematic prompt testing helps validate if the upgrade maintains or improves the prompt performance. This is crucial for ensuring that updates do not unintentionally break the functionality.

◦ Seamless Transitions: By identifying key requirements and testing them, this method can facilitate better transitions to new model versions, allowing fast adjustment when necessary in order to maintain high-quality outputs.

3. Cost optimization:

◦ Performance-to-cost ratio: Systematic prompt testing helps in choosing the best cost-effective model based on the performance-to-cost ratio. We can efficiently identify the most efficient option between performance and operational costs to get the best return on LLM costs.

Overcoming the challenges

The biggest challenge of this approach is the preparation of the set of test data fixtures, but the effort invested in this process will pay off significantly as time passes. Well-prepared fixtures save considerable debugging time and enhance model efficiency and reliability by providing a robust foundation for evaluating the LLM response. The initial investment is quickly returned by improved efficiency and effectiveness in LLM development and deployment.

Quick pros and cons

Key advantages:

Continuous improvement: The ability to add more requirements over time while ensuring existing functionality stays intact is a significant advantage. This allows for the evolution of the AI task in response to new requirements, ensuring that the system remains up-to-date and efficient.
Better maintenance: This approach enables the easy validation of prompt performance with LLM updates. This is crucial for maintaining high standards of quality and reliability, as updates can sometimes introduce unintended changes in behavior.
More flexibility: With a set of quality control tests, switching LLM providers becomes more straightforward. This flexibility allows us to adapt to changes in the market or technological advancements, ensuring we can always use the best tool for the job.
Cost optimization: Data-driven evaluations enable better decisions on performance-to-cost ratio. By understanding the performance gains of different models, we can choose the most cost-effective solution that meets the needs.
Time savings: Systematic evaluations provide quick feedback, reducing the need for manual testing. This efficiency allows to quickly iterate on prompt improvement and optimization, accelerating the development process.

Challenges

Initial time investment: Creating test fixtures and evaluation functions can require a significant investment of time.
Defining measurable validation criteria: Not all AI tasks have clear pass/fail conditions. Defining measurable criteria for validation can sometimes be challenging, especially for tasks that involve subjective or nuanced outputs. This requires careful consideration and may involve a difficult selection of the evaluation metrics.
Cost associated with multiple tests: Multiple test use cases associated with 5 to 10 iterations can generate a high number of LLM requests for a single test automation. But if the cost of a single LLM call is neglectable, as it is in most cases for text input/output calls, the overall cost of a test remains minimal.

Conclusion: When should you implement this approach?

Implementing this systematic testing approach is, of course, not always necessary, especially for simple tasks. However, for complex AI workflows in which precision and reliability are critical, this approach becomes highly valuable by offering a systematic way to assess and optimize prompt performance, preventing endless cycles of trial and error.

By incorporating functional testing principles into Prompt Engineering, we transform a traditionally subjective and fragile process into one that is measurable, scalable, and robust. Not only does it enhance the reliability of LLM outputs, it helps achieve continuous improvement and efficient resource allocation.

The decision to implement systematic prompt Testing should be based on the complexity of your project. For scenarios demanding high precision and consistency, investing the time to set up this methodology can significantly improve outcomes and speed up the development processes. However, for simpler tasks, a more classical, lightweight approach may be sufficient. The key is to balance the need for rigor with practical considerations, ensuring that your testing strategy aligns with your goals and constraints.

Thanks for reading!

Stay Ahead

Explore More Insights

Stay ahead with more perspectives on cutting-edge power, infrastructure, energy, bitcoin and AI solutions. Explore these articles to uncover strategies and insights shaping the future of industries.

IBM X-Force: Stealthy attacks on the rise, toolkits targeting AI emerge

Last year, X-Force predicted that once AI technologies “establish market dominance—when a single technology approaches 50% market share or when the market consolidates to three or fewer technologies—attackers will be incentivized to invest in attack toolkits” that target AI models and solutions. “Are we there yet? Not quite, but adoption

VMware (quietly) brings back its free ESXi hypervisor

By many accounts, Broadcom’s handling of the VMware acquisition was clumsy and caused many enterprises to reevaluate their relationship with the vendor The move to subscription models was tilted in favor of larger customers and longer, three-year licenses. Because the string of bad publicity and VMware’s competitors pounced, offering migration

CoreWeave offers cloud-based Grace Blackwell GPUs for AI training

Cloud services provider CoreWeave has announced it is offering Nvidia’s GB200 NVL72 systems, otherwise known as “Grace Blackwell,” to customers looking to do intensive AI training. CoreWeave said its portfolio of cloud services are optimized for the GB200 NVL72, including CoreWeave’s Kubernetes Service, Slurm on Kubernetes (SUNK), Mission Control, and

Kyndryl launches private cloud services for enterprise AI deployments

Kyndryl’s AI Private Cloud environment includes services and capabilities around containerization, data science tools, and microservices to deploy and manage AI applications on the private cloud. The service supports AI data foundations and MLOps/LLMOps services, letting customers manage their AI data pipelines and machine learning operation, Kyndryl stated. These tools facilitate

US Energy Expands Carbon Capture Assets With New Acquisition

U.S. Energy Corporation strengthened its industrial gas and carbon capture platform in Montana by acquiring a privately held company for $0.2 million. With the acquisition, U.S. Energy secured approximately 2,300 net acres with carbon dioxide (CO2) rights that are highly contiguous to its existing position across Montana’s Kevin Dome structure. Additionally, the acquisition includes an active Class II injection well to sequester CO2 captured from U.S. Energy’s upcoming industrial gas processing facility, the company said in a media release. The permitted well advances the company’s carbon capture, utilization, and storage (CCUS) initiatives within its industrial gas development platform, U.S. Energy said. The Class II injection well is a key part of U.S. Energy’s plan to store CO2 from its upcoming gas processing facility. The well has active permits from the U.S. Environmental Protection Agency (EPA) under the Safe Drinking Water Act’s Underground Injection Control Program (UIC), ensuring compliance with regulations for safe CO2 storage, the company said. U.S. Energy added that the acquisition adds CCUS-ready infrastructure and supports its strategy to develop low-emission gas operations while establishing itself as a U.S. supplier of clean helium and other essential gases. “This acquisition marks a meaningful milestone forward in our efforts to integrate carbon sequestration into our industrial gas platform” Ryan Smith, Chief Executive Officer of U.S. Energy, said. “The addition of permitted injection infrastructure and strategic acreage strengthens our position across the Kevin Dome and accelerates our ability to deliver clean, domestically sourced helium while sequestering CO₂ at scale. We are committed to executing a responsible growth strategy that aligns with global demand for lower-carbon energy solutions”. The acquisition enhances U.S. Energy’s control over a contiguous acreage block in the Kevin Dome, a geological formation recognized for its helium-rich and CO₂-dominated gas systems. The company plans to present a Monitoring, Reporting,

Carney, Poilievre Scrap Over Energy and Housing in Canada Debate

Liberal Party Leader Mark Carney argued that he represents change from Justin Trudeau’s nine years in power as he fended off attacks from his rivals during the final televised debate of Canada’s election. “Look, I’m a very different person from Justin Trudeau,” Carney said in response to comments from Conservative Leader Pierre Poilievre, his chief opponent in the election campaign that concludes April 28. Carney’s Liberals lead by several percentage points in most polls, marking a stunning reversal from the start of this year, when Trudeau was still the party’s leader and Poilievre’s Conservatives were ahead by more than 20 percentage points in some surveys. Trudeau’s resignation and US President Donald Trump’s economic and sovereignty threats against Canada have upended the race. Poilievre sought to remind Canadians of their complaints about the Liberal government, while Carney tried to distance himself from Trudeau’s record. Poilievre argued that Carney was an adviser to Trudeau’s Liberals during a time when energy projects were stymied and the cost of living soared — especially housing prices. Carney, 60, responded that he has been prime minister for just a month, and pointed to moves he made to reverse some of Trudeau’s policies, such as scrapping the carbon tax on consumer fuels. As for inflation, Carney noted that it was well under control when he was governor of the Bank of Canada. “I know it may be difficult, Mr. Poilievre,” Carney told him. “You spent years running against Justin Trudeau and the carbon tax and they’re both gone.” “Well, you’re doing a good impersonation of him, with the same policies,” Poilievre shot back. Trudeau announced in January that he was stepping down as prime minister and Carney was sworn in as his replacement on March 14. He triggered an election nine days later. “The question you have

Gunvor, Adnoc Shortlisted for Shell South Africa Unit

Abu Dhabi National Oil Co. and Swiss commodities trading firm Gunvor are among companies that have been shortlisted to buy Shell Plc’s downstream assets in South Africa, according to people familiar with the matter. The two companies are strong contenders for the assets that are valued at about $1 billion, said the people, who asked not to be identified as the information is private. Previous potential bidders including Trafigura’s Puma Energy, Sasol Ltd. and South Africa’s PetroSA are no longer in the running, two of the people said. “While Adnoc Distribution regularly reviews opportunities for domestic and international growth, we don’t comment on market speculation,” Adnoc’s fuel retail unit said. Shell, Gunvor, Trafigura and Sasol declined to comment. PetroSA did not immediately reply to a request for comment. Shell has been looking to offload the assets, which include about 600 fuel stations and trading operations in Africa’s biggest economy, as part of a broader strategy to focus on regions and businesses that offer higher returns. The assets are attractive for trading firms since they ensure demand for fuels that they can then supply. Adnoc and other Middle East oil companies such as Saudi Aramco have been expanding their trading arms as they look to break into new markets. Shell is working with adviser Rothschild & Co and a winner could be announced in the coming weeks, the people said. Talks are continuing and there’s no certainty there will be a final sale, they said. Saudi Aramco has also been involved in the process, but it wasn’t immediately clear if it was still in the running, the people said. Aramco declined to comment. A deal would give the buyer about 10% of South Africa’s fuel stations. The market in the country has changed significantly in recent years with trader Glencore Plc acquiring

ICYMI: Trump Administration Adds Two DOE Critical Minerals Projects to Federal Permitting Dashboard

ICYMI— The Federal Permitting Improvement Steering Council (Permitting Council) today announced increased transparency and accountability for the federal permitting of two Department of Energy (DOE) critical minerals projects. The projects — Michigan Potash and the South West Arkansas Project — are part of the first wave of critical minerals projects added to the Permitting Dashboard in response to President Trump’s Executive Order, Immediate Measures to Increase American Mineral Production. Once completed, both DOE-supported projects will help meet President Trump’s commitment to bolster domestic production of America’s vast mineral resources, support more American jobs and reduce reliance on foreign supply chains. The Michigan Potash Project, supported by DOE’s Loan Programs Office, is projected to produce the largest American-based source of high-quality potash fertilizer and food-grade salt using mechanical vapor recompression technology and geothermal heat from subsurface brine. Once completed, this project will reduce reliance on potash imports, support American farmers, improve food security, and create 200 permanent and 400 construction sector jobs. DOE announced a conditional commitment for a loan guarantee of up to $1.26 billion to Michigan Potash in January 2025. The South West Arkansas Project, under DOE’s Office of Manufacturing and Energy Supply Chains, supports the construction of a world-class Direct Lithium Extraction facility that will produce battery-grade lithium carbonate from lithium-rich brine in North America. Once completed, this project will help secure the domestic lithium supply chain and is expected to create roughly 100 direct long-term jobs and 300 construction sector jobs. These additions to the Federal Permitting Dashboard reflect the Administration’s commitment to strengthen domestic supply chains for critical minerals and materials, reduce dependence on foreign sources, and advance President Trump’s bold agenda for American energy dominance through a more secure, affordable, and reliable U.S. energy system. The Department looks forward to working with federal partners, project

EVOL: Courting wood, grid zombies and Easter wake loss

This week, Wood provided updates on Sidara’s proposed £250 million takeover, NESO declared war on zombies in the grid queue, and Equinor and Orsted warned of the impacts of wake loss. Aberdeen-headquartered Wood received a non-binding takeover bid from Dubai-based rival Sidara worth £250m, a significant drop-off compared to last year’s £1.5 billion bid. Our reporters discuss this, Wood’s shares being suspended and the impacts of yet another Scottish company being bought over by international competitors. Next up, the UK’s National Energy System Operator (NESO) unveiled plans to get rid of ‘zombies’ from the grid queue in a collaboration with regulator Ofgem. This could see up to 360GW of projects on the current queue have their contracts downgraded because they are not ready. What does this mean, and is it a result of too much dithering from the UK? Finally, European energy giants Equinor and Orsted have said offshore wind revenues could take a £363m hit due to other projects getting in the way of their turbines. Although those in the Tour de France peloton don’t mind the frontrunner taking the brunt of the wind resistance, turbine operators do. Does the industry need to share its survey results so that everyone can benefit from the North Sea breeze? Listen to Energy Voice Out Loud on your podcast platform of choice.

Trump administration moves to curb energy regulation; BLM nominee stands down

The Trump administration issued two policy directives Apr. 10 to curb energy regulations, the same day the president’s choice to lead the Bureau of Land Management (BLM) pulled her nomination. Kathleen Sgamma, former head of Western Energy Alliance (WEA), an oil and gas trade association, withdrew her nomination after a memo was leaked on X that included critical remarks following the Jan. 6, 2021, attack on the US Capitol. In the memo to WEA executives, Sgamma said she was “disgusted” by Trump “spreading misinformation” on Jan. 6 and “dishonoring the vote of the people.” The Senate was to conduct a confirmation hearing Apr. 10. Prior to her withdrawal, industry had praised the choice of Sgamma to head the agency that determines the rules for oil and gas operations on federal lands. Deregulation On the deregulation front, the Interior Department said it would no longer require BLM to prepare environmental impact statements (EIS) for about 3,244 oil and gas leases in seven western states. The move comes in response to two executive orders by President Donald Trump in January to increase US oil and gas production “by reducing regulatory barriers for oil and gas companies” and expediting development permits, Interior noted (OGJ Online, Jan. 21, 2025). Under the policy, BLM would no longer have to prepare an EIS for oil and gas leasing decisions on about 3.5 million acres across Colorado, New Mexico, North Dakota, South Dakota, Utah, and Wyoming. BLM currently manages over 23 million acres of federal land leased for oil and gas development. The agency said it will look for ways to comply with the National Environmental Policy Act (NEPA), a 1970 law that requires federal agencies to assess the potential environmental impacts of their proposed actions. In recent years, courts have increasingly delayed lease sales and projects,

The Rise of AI Factories: Transforming Intelligence at Scale

AI Factories Redefine Infrastructure The architecture of AI factories reflects a paradigm shift that mirrors the evolution of the industrial age itself—from manual processes to automation, and now to autonomous intelligence. Nvidia’s framing of these systems as “factories” isn’t just branding; it’s a conceptual leap that positions AI infrastructure as the new production line. GPUs are the engines, data is the raw material, and the output isn’t a physical product, but predictive power at unprecedented scale. In this vision, compute capacity becomes a strategic asset, and the ability to iterate faster on AI models becomes a competitive differentiator, not just a technical milestone. This evolution also introduces a new calculus for data center investment. The cost-per-token of inference—how efficiently a system can produce usable AI output—emerges as a critical KPI, replacing traditional metrics like PUE or rack density as primary indicators of performance. That changes the game for developers, operators, and regulators alike. Just as cloud computing shifted the industry’s center of gravity over the past decade, the rise of AI factories is likely to redraw the map again—favoring locations with not only robust power and cooling, but with access to clean energy, proximity to data-rich ecosystems, and incentives that align with national digital strategies. The Economics of AI: Scaling Laws and Compute Demand At the heart of the AI factory model is a requirement for a deep understanding of the scaling laws that govern AI economics. Initially, the emphasis in AI revolved around pretraining large models, requiring massive amounts of compute, expert labor, and curated data. Over five years, pretraining compute needs have increased by a factor of 50 million. However, once a foundational model is trained, the downstream potential multiplies exponentially, while the compute required to utilize a fully trained model for standard inference is significantly less than

Google’s AI-Powered Grid Revolution: How Data Centers Are Reshaping the U.S. Power Landscape

Google Unveils Groundbreaking AI Partnership with PJM and Tapestry to Reinvent the U.S. Power Grid In a move that underscores the growing intersection between digital infrastructure and energy resilience, Google has announced a major new initiative to modernize the U.S. electric grid using artificial intelligence. The company is partnering with PJM Interconnection—the largest grid operator in North America—and Tapestry, an Alphabet moonshot backed by Google Cloud and DeepMind, to develop AI tools aimed at transforming how new power sources are brought online. The initiative, detailed in a blog post by Alphabet and Google President Ruth Porat, represents one of Google’s most ambitious energy collaborations to date. It seeks to address mounting challenges facing grid operators, particularly the explosive backlog of energy generation projects that await interconnection in a power system unprepared for 21st-century demands. “This is our biggest step yet to use AI for building a stronger, more resilient electricity system,” Porat wrote. Tapping AI to Tackle an Interconnection Crisis The timing is critical. The U.S. energy grid is facing a historic inflection point. According to the Lawrence Berkeley National Laboratory, more than 2,600 gigawatts (GW) of generation and storage projects were waiting in interconnection queues at the end of 2023—more than double the total installed capacity of the entire U.S. grid. Meanwhile, the Federal Energy Regulatory Commission (FERC) has revised its five-year demand forecast, now projecting U.S. peak load to rise by 128 GW before 2030—more than triple the previous estimate. Grid operators like PJM are straining to process a surge in interconnection requests, which have skyrocketed from a few dozen to thousands annually. This wave of applications has exposed the limits of legacy systems and planning tools. Enter AI. Tapestry’s role is to develop and deploy AI models that can intelligently manage and streamline the complex process of

Podcast: Vaire Computing Bets on Reversible Logic for ‘Near Zero Energy’ AI Data Centers

The AI revolution is charging ahead—but powering it shouldn’t cost us the planet. That tension lies at the heart of Vaire Computing’s bold proposition: rethinking the very logic that underpins silicon to make chips radically more energy efficient. Speaking on the Data Center Frontier Show podcast, Vaire CEO Rodolfo Rossini laid out a compelling case for why the next era of compute won’t just be about scaling transistors—but reinventing the way they work. “Moore’s Law is coming to an end, at least for classical CMOS,” Rossini said. “There are a number of potential architectures out there—quantum and photonics are the most well known. Our bet is that the future will look a lot like existing CMOS, but the logic will look very, very, very different.” That bet is reversible computing—a largely untapped architecture that promises major gains in energy efficiency by recovering energy lost during computation. A Forgotten Frontier Unlike conventional chips that discard energy with each logic operation, reversible chips can theoretically recycle that energy. The concept, Rossini explained, isn’t new—but it’s long been overlooked. “The tech is really old. I mean really old,” Rossini said. “The seeds of this technology were actually at the very beginning of the industrial revolution.” Drawing on the work of 19th-century mechanical engineers like Sadi Carnot and later insights from John von Neumann, the theoretical underpinnings of reversible computing stretch back decades. A pivotal 1961 paper formally connected reversibility to energy efficiency in computing. But progress stalled—until now. “Nothing really happened until a team of MIT students built the first chip in the 1990s,” Rossini noted. “But they were trying to build a CPU, which is a world of pain. There’s a reason why I don’t think there’s been a startup trying to build CPUs for a very, very long time.” AI, the

Pennsylvania’s Homer City Energy Campus: A Brownfield Transformed for Data Center Innovation

The redevelopment of the Homer City Generating Station in Pennsylvania represents an important transformation from a decommissioned coal-fired power plant to a state-of-the-art natural gas-powered data center campus, showing the creative reuse of a large brownfield site and the creation of what can be a significant location in power generation and the digital future. The redevelopment will address the growing energy demands of artificial intelligence and high-performance computing technologies, while also contributing to Pennsylvania’s digital advancement, in an area not known as a hotbed of technical prowess. Brownfield Development Established in 1969, the original generating station was a 2-gigawatt coal-fired power plant located near Homer City, Indiana County, Pennsylvania. The site was formerly the largest coal-burning power plant in the state, and known for its 1,217-foot chimney, the tallest in the United States. In April 2023, the owners announced its closure due to competition from cheaper natural gas and the rising costs of environmental compliance. The plant was officially decommissioned on July 1, 2023, and its demolition, including the iconic chimney, was completed by March 22, 2025. The redevelopment project, led by Homer City Redevelopment (HCR) in partnership with Kiewit Power Constructors Co., plans to transform the 3,200-acre site into the Homer City Energy Campus, via construction of a 4.5-gigawatt natural gas-fired power plant, making it the largest of its kind in the United States. Gas Turbines This plant will utilize seven high-efficiency, hydrogen-enabled 7HA.02 gas turbines supplied by GE Vernova, with deliveries expected to begin in 2026. The GE Vernova gas turbine has been seeing significant interest in the power generation market as new power plants have been moving to the planning stage. The GE Vernova 7HA.02 is a high-efficiency, hydrogen-enabled gas turbine designed for advanced power generation applications. As part of GE Vernova’s HA product line, it

Dell data center modernization gear targets AI, HPC workloads

The update starts with new PowerEdge R470, R570, R670 and R770 servers featuring Intel Xeon 6 with P-cores processors in single- and dual-socket configurations designed to handle high-performance computing, virtualization, analytics and artificial intelligence inferencing. Dell said they save up to half of the energy costs of previous server generations while supporting up to 50% more cores per processors and 67% better performance. With the R770, up to 80% of space can be saved and a 42U rack. They feature the Dell Modular Hardware System architecture, which is based on Open Compute Project standards. The controller system also received a significant update, with improvements to Dell OpenManage and Integrated Dell Remote Access Controller providing real-time monitoring, while the Dell PowerEdge RAID Controller for PCIe Gen 5 hardware reduces write latency up to 33-fold.

Intel sells off majority stake in its FPGA business

Altera will continue offering field-programmable gate array (FPGA) products across a wide range of use cases, including automotive, communications, data centers, embedded systems, industrial, and aerospace. “People were a bit surprised at Intel’s sale of the majority stake in Altera, but they shouldn’t have been. Lip-Bu indicated that shoring up Intel’s balance sheet was important,” said Jim McGregor, chief analyst with Tirias Research. The Altera has been in the works for a while and is a relic of past mistakes by Intel to try to acquire its way into AI, whether it was through FPGAs or other accelerators like Habana or Nervana, note Anshel Sag, principal analyst with Moor Insight and Research. “Ultimately, the 50% haircut on the valuation of Altera is unfortunate, but again is a demonstration of Intel’s past mistakes. I do believe that finishing the process of spinning it out does give Intel back some capital and narrows the company’s focus,” he said. So where did it go wrong? It wasn’t with FPGAs because AMD is making a good run of it with its Xilinx acquisition. The fault, analysts say, lies with Intel, which has a terrible track record when it comes to acquisitions. “Altera could have been a great asset to Intel, just as Xilinx has become a valuable asset to AMD. However, like most of its acquisitions, Intel did not manage Altera well,” said McGregor.

Microsoft will invest $80B in AI data centers in fiscal 2025

And Microsoft isn’t the only one that is ramping up its investments into AI-enabled data centers. Rival cloud service providers are all investing in either upgrading or opening new data centers to capture a larger chunk of business from developers and users of large language models (LLMs). In a report published in October 2024, Bloomberg Intelligence estimated that demand for generative AI would push Microsoft, AWS, Google, Oracle, Meta, and Apple would between them devote $200 billion to capex in 2025, up from $110 billion in 2023. Microsoft is one of the biggest spenders, followed closely by Google and AWS, Bloomberg Intelligence said. Its estimate of Microsoft’s capital spending on AI, at $62.4 billion for calendar 2025, is lower than Smith’s claim that the company will invest $80 billion in the fiscal year to June 30, 2025. Both figures, though, are way higher than Microsoft’s 2020 capital expenditure of “just” $17.6 billion. The majority of the increased spending is tied to cloud services and the expansion of AI infrastructure needed to provide compute capacity for OpenAI workloads. Separately, last October Amazon CEO Andy Jassy said his company planned total capex spend of $75 billion in 2024 and even more in 2025, with much of it going to AWS, its cloud computing division.

John Deere unveils more autonomous farm machines to address skill labor shortage

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Self-driving tractors might be the path to self-driving cars. John Deere has revealed a new line of autonomous machines and tech across agriculture, construction and commercial landscaping. The Moline, Illinois-based John Deere has been in business for 187 years, yet it’s been a regular as a non-tech company showing off technology at the big tech trade show in Las Vegas and is back at CES 2025 with more autonomous tractors and other vehicles. This is not something we usually cover, but John Deere has a lot of data that is interesting in the big picture of tech. The message from the company is that there aren’t enough skilled farm laborers to do the work that its customers need. It’s been a challenge for most of the last two decades, said Jahmy Hindman, CTO at John Deere, in a briefing. Much of the tech will come this fall and after that. He noted that the average farmer in the U.S. is over 58 and works 12 to 18 hours a day to grow food for us. And he said the American Farm Bureau Federation estimates there are roughly 2.4 million farm jobs that need to be filled annually; and the agricultural work force continues to shrink. (This is my hint to the anti-immigration crowd). John Deere’s autonomous 9RX Tractor. Farmers can oversee it using an app. While each of these industries experiences their own set of challenges, a commonality across all is skilled labor availability. In construction, about 80% percent of contractors struggle to find skilled labor. And in commercial landscaping, 86% of landscaping business owners can’t find labor to fill open positions, he said. “They have to figure out how to do

2025 playbook for enterprise AI success, from agents to evals

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More 2025 is poised to be a pivotal year for enterprise AI. The past year has seen rapid innovation, and this year will see the same. This has made it more critical than ever to revisit your AI strategy to stay competitive and create value for your customers. From scaling AI agents to optimizing costs, here are the five critical areas enterprises should prioritize for their AI strategy this year. 1. Agents: the next generation of automation AI agents are no longer theoretical. In 2025, they’re indispensable tools for enterprises looking to streamline operations and enhance customer interactions. Unlike traditional software, agents powered by large language models (LLMs) can make nuanced decisions, navigate complex multi-step tasks, and integrate seamlessly with tools and APIs. At the start of 2024, agents were not ready for prime time, making frustrating mistakes like hallucinating URLs. They started getting better as frontier large language models themselves improved. “Let me put it this way,” said Sam Witteveen, cofounder of Red Dragon, a company that develops agents for companies, and that recently reviewed the 48 agents it built last year. “Interestingly, the ones that we built at the start of the year, a lot of those worked way better at the end of the year just because the models got better.” Witteveen shared this in the video podcast we filmed to discuss these five big trends in detail. Models are getting better and hallucinating less, and they’re also being trained to do agentic tasks. Another feature that the model providers are researching is a way to use the LLM as a judge, and as models get cheaper (something we’ll cover below), companies can use three or more models to

OpenAI’s red teaming innovations define new essentials for security leaders in the AI era

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More OpenAI has taken a more aggressive approach to red teaming than its AI competitors, demonstrating its security teams’ advanced capabilities in two areas: multi-step reinforcement and external red teaming. OpenAI recently released two papers that set a new competitive standard for improving the quality, reliability and safety of AI models in these two techniques and more. The first paper, “OpenAI’s Approach to External Red Teaming for AI Models and Systems,” reports that specialized teams outside the company have proven effective in uncovering vulnerabilities that might otherwise have made it into a released model because in-house testing techniques may have missed them. In the second paper, “Diverse and Effective Red Teaming with Auto-Generated Rewards and Multi-Step Reinforcement Learning,” OpenAI introduces an automated framework that relies on iterative reinforcement learning to generate a broad spectrum of novel, wide-ranging attacks. Going all-in on red teaming pays practical, competitive dividends It’s encouraging to see competitive intensity in red teaming growing among AI companies. When Anthropic released its AI red team guidelines in June of last year, it joined AI providers including Google, Microsoft, Nvidia, OpenAI, and even the U.S.’s National Institute of Standards and Technology (NIST), which all had released red teaming frameworks. Investing heavily in red teaming yields tangible benefits for security leaders in any organization. OpenAI’s paper on external red teaming provides a detailed analysis of how the company strives to create specialized external teams that include cybersecurity and subject matter experts. The goal is to see if knowledgeable external teams can defeat models’ security perimeters and find gaps in their security, biases and controls that prompt-based testing couldn’t find. What makes OpenAI’s recent papers noteworthy is how well they define using human-in-the-middle