Get ONMINE

Stay Ahead, Stay ONMINE

Mastering Prompt Engineering with Functional Testing: A Systematic Guide to Reliable LLM Outputs

Creating efficient prompts for large language models often starts as a simple task… but it doesn’t always stay that way. Initially, following basic best practices seems sufficient: adopt the persona of a specialist, write clear instructions, require a specific response format, and include a few relevant examples. But as requirements multiply, contradictions emerge, and even minor modifications can introduce unexpected failures. What was working perfectly in one prompt version suddenly breaks in another. If you have ever felt trapped in an endless loop of trial and error, adjusting one rule only to see another one fail, you’re not alone! The reality is that traditional prompt optimisation is clearly missing a structured, more scientific approach that will help to ensure reliability. That’s where functional testing for prompt engineering comes in! This approach, inspired by methodologies of experimental science, leverages automated input-output testing with multiple iterations and algorithmic scoring to turn prompt engineering into a measurable, data-driven process.  No more guesswork. No more tedious manual validation. Just precise and repeatable results that allow you to fine-tune prompts efficiently and confidently. In this article, we will explore a systematic approach for mastering prompt engineering, which ensures your Llm outputs will be efficient and reliable even for the most complex AI tasks. Balancing precision and consistency in prompt optimisation Adding a large set of rules to a prompt can introduce partial contradictions between rules and lead to unexpected behaviors. This is especially true when following a pattern of starting with a general rule and following it with multiple exceptions or specific contradictory use cases. Adding specific rules and exceptions can cause conflict with the primary instruction and, potentially, with each other. What might seem like a minor modification can unexpectedly impact other aspects of a prompt. This is not only true when adding a new rule but also when adding more detail to an existing rule, like changing the order of the set of instructions or even simply rewording it. These minor modifications can unintentionally change the way the model interprets and prioritizes the set of instructions. The more details you add to a prompt, the greater the risk of unintended side effects. By trying to give too many details to every aspect of your task, you increase as well the risk of getting unexpected or deformed results. It is, therefore, essential to find the right balance between clarity and a high level of specification to maximise the relevance and consistency of the response. At a certain point, fixing one requirement can break two others, creating the frustrating feeling of taking one step forward and two steps backward in the optimization process. Testing each change manually becomes quickly overwhelming. This is especially true when one needs to optimize prompts that must follow numerous competing specifications in a complex AI task. The process cannot simply be about modifying the prompt for one requirement after the other, hoping the previous instruction remains unaffected. It also can’t be a system of selecting examples and checking them by hand. A better process with a more scientific approach should focus on ensuring repeatability and reliability in prompt optimization. From laboratory to AI: Why testing LLM responses requires multiple iterations Science teaches us to use replicates to ensure reproducibility and build confidence in an experiment’s results. I have been working in academic research in chemistry and biology for more than a decade. In those fields, experimental results can be influenced by a multitude of factors that can lead to significant variability. To ensure the reliability and reproducibility of experimental results, scientists mostly employ a method known as triplicates. This approach involves conducting the same experiment three times under identical conditions, allowing the experimental variations to be of minor importance in the result. Statistical analysis (standard mean and deviation) conducted on the results, mostly in biology, allows the author of an experiment to determine the consistency of the results and strengthens confidence in the findings. Just like in biology and chemistry, this approach can be used with LLMs to achieve reliable responses. With LLMs, the generation of responses is non-deterministic, meaning that the same input can lead to different outputs due to the probabilistic nature of the models. This variability is challenging when evaluating the reliability and consistency of LLM outputs. In the same way that biological/chemical experiments require triplicates to ensure reproducibility, testing LLMs should need multiple iterations to measure reproducibility. A single test by use case is, therefore, not sufficient because it does not represent the inherent variability in LLM responses. At least five iterations per use case allow for a better assessment. By analyzing the consistency of the responses across these iterations, one can better evaluate the reliability of the model and identify any potential issues or variation. It ensures that the output of the model is correctly controlled. Multiply this across 10 to 15 different prompt requirements, and one can easily understand how, without a structured testing approach, we end up spending time in trial-and-error testing with no efficient way to assess quality. A systematic approach: Functional testing for prompt optimization To address these challenges, a structured evaluation methodology can be used to ease and accelerate the testing process and enhance the reliability of LLM outputs. This approach has several key components: Data fixtures: The approach’s core center is the data fixtures, which are composed of predefined input-output pairs specifically created for prompt testing. These fixtures serve as controlled scenarios that represent the various requirements and edge cases the LLM must handle. By using a diverse set of fixtures, the performance of the prompt can be evaluated efficiently across different conditions. Automated test validation: This approach automates the validation of the requirements on a set of data fixtures by comparison between the expected outputs defined in the fixtures and the LLM response. This automated comparison ensures consistency and reduces the potential for human error or bias in the evaluation process. It allows for quick identification of discrepancies, enabling fine and efficient prompt adjustments. Multiple iterations: To assess the inherent variability of the LLM responses, this method runs multiple iterations for each test case. This iterative approach mimics the triplicate method used in biological/chemical experiments, providing a more robust dataset for analysis. By observing the consistency of responses across iterations, we can better assess the stability and reliability of the prompt. Algorithmic scoring: The results of each test case are scored algorithmically, reducing the need for long and laborious « human » evaluation. This scoring system is designed to be objective and quantitative, providing clear metrics for assessing the performance of the prompt. And by focusing on measurable outcomes, we can make data-driven decisions to optimize the prompt effectively.      Step 1: Defining test data fixtures Selecting or creating compatible test data fixtures is the most challenging step of our systematic approach because it requires careful thought. A fixture is not only any input-output pair; it must be crafted meticulously to evaluate the most accurate as possible performance of the LLM for a specific requirement. This process requires: 1. A deep understanding of the task and the behavior of the model to make sure the selected examples effectively test the expected output while minimizing ambiguity or bias. 2. Foresight into how the evaluation will be conducted algorithmically during the test. The quality of a fixture, therefore, depends not only on the good representativeness of the example but also on ensuring it can be efficiently tested algorithmically. A fixture consists of:     • Input example: This is the data that will be given to the LLM for processing. It should represent a typical or edge-case scenario that the LLM is expected to handle. The input should be designed to cover a wide range of possible variations that the LLM might have to deal with in production.     • Expected output: This is the expected result that the LLM should produce with the provided input example. It is used for comparison with the actual LLM response output during validation. Step 2: Running automated tests Once the test data fixtures are defined, the next step involves the execution of automated tests to systematically evaluate the performance of the LLM response on the selected use cases. As previously stated, this process makes sure that the prompt is thoroughly tested against various scenarios, providing a reliable evaluation of its efficiency. Execution process     1. Multiple iterations: For each test use case, the same input is provided to the LLM multiple times. A simple for loop in nb_iter with nb_iter = 5 and voila!     2. Response comparison: After each iteration, the LLM response is compared to the expected output of the fixture. This comparison checks whether the LLM has correctly processed the input according to the specified requirements.     3. Scoring mechanism: Each comparison results in a score:         ◦ Pass (1): The response matches the expected output, indicating that the LLM has correctly handled the input.         ◦ Fail (0): The response does not match the expected output, signaling a discrepancy that needs to be fixed.     4. Final score calculation: The scores from all iterations are aggregated to calculate the overall final score. This score represents the proportion of successful responses out of the total number of iterations. A high score, of course, indicates high prompt performance and reliability. Example: Removing author signatures from an article Let’s consider a simple scenario where an AI task is to remove author signatures from an article. To efficiently test this functionality, we need a set of fixtures that represent the various signature styles.  A dataset for this example could be: Example InputExpected OutputA long articleJean LeblancThe long articleA long articleP. W. HartigThe long articleA long articleMCZThe long article Validation process: Signature removal check: The validation function checks if the signature is absent from the rewritten text. This is easily done programmatically by searching for the signature needle in the haystack output text. Test failure criteria: If the signature is still in the output, the test fails. This indicates that the LLM did not correctly remove the signature and that further adjustments to the prompt are required. If it is not, the test is passed.  The test evaluation provides a final score that allows a data-driven assessment of the prompt efficiency. If it scores perfectly, there is no need for further optimization. However, in most cases, you will not get a perfect score because either the consistency of the LLM response to a case is low (for example, 3 out of 5 iterations scored positive) or there are edge cases that the model struggles with (0 out of 5 iterations).  The feedback clearly indicates that there is still room for further improvements and it guides you to reexamine your prompt for ambiguous phrasing, conflicting rules, or edge cases. By continuously monitoring your score alongside your prompt modifications, you can incrementally reduce side effects, achieve greater efficiency and consistency, and approach an optimal and reliable output.  A perfect score is, however, not always achievable with the selected model. Changing the model might just fix the situation. If it doesn’t, you know the limitations of your system and can take this fact into account in your workflow. With luck, this situation might just be solved in the near future with a simple model update.  Benefits of this method  Reliability of the result: Running five to ten iterations provides reliable statistics on the performance of the prompt. A single test run may succeed once but not twice, and consistent success for multiple iterations indicates a robust and well-optimized prompt. Efficiency of the process: Unlike traditional scientific experiments that may take weeks or months to replicate, automated testing of LLMs can be carried out quickly. By setting a high number of iterations and waiting for a few minutes, we can obtain a high-quality, reproducible evaluation of the prompt efficiency. Data-driven optimization: The score obtained from these tests provides a data-driven assessment of the prompt’s ability to meet requirements, allowing targeted improvements. Side-by-side evaluation: Structured testing allows for an easy assessment of prompt versions. By comparing the test results, one can identify the most effective set of parameters for the instructions (phrasing, order of instructions) to achieve the desired results. Quick iterative improvement: The ability to quickly test and iterate prompts is a real advantage to carefully construct the prompt ensuring that the previously validated requirements remain as the prompt increases in complexity and length. By adopting this automated testing approach, we can systematically evaluate and enhance prompt performance, ensuring consistent and reliable outputs with the desired requirements. This method saves time and provides a robust analytical tool for continuous prompt optimization. Systematic prompt testing: Beyond prompt optimization Implementing a systematic prompt testing approach offers more advantages than just the initial prompt optimization. This methodology is valuable for other aspects of AI tasks:     1. Model comparison:         ◦ Provider evaluation: This approach allows the efficient comparison of different LLM providers, such as ChatGPT, Claude, Gemini, Mistral, etc., on the same tasks. It becomes easy to evaluate which model performs the best for their specific needs.         ◦ Model version: State-of-the-art model versions are not always necessary when a prompt is well-optimized, even for complex AI tasks. A lightweight, faster version can provide the same results with a faster response. This approach allows a side-by-side comparison of the different versions of a model, such as Gemini 1.5 flash vs. 1.5 pro vs. 2.0 flash or ChatGPT 3.5 vs. 4o mini vs. 4o, and allows the data-driven selection of the model version.     2. Version upgrades:         ◦ Compatibility verification: When a new model version is released, systematic prompt testing helps validate if the upgrade maintains or improves the prompt performance. This is crucial for ensuring that updates do not unintentionally break the functionality.         ◦ Seamless Transitions: By identifying key requirements and testing them, this method can facilitate better transitions to new model versions, allowing fast adjustment when necessary in order to maintain high-quality outputs.     3. Cost optimization:         ◦ Performance-to-cost ratio: Systematic prompt testing helps in choosing the best cost-effective model based on the performance-to-cost ratio. We can efficiently identify the most efficient option between performance and operational costs to get the best return on LLM costs. Overcoming the challenges The biggest challenge of this approach is the preparation of the set of test data fixtures, but the effort invested in this process will pay off significantly as time passes. Well-prepared fixtures save considerable debugging time and enhance model efficiency and reliability by providing a robust foundation for evaluating the LLM response. The initial investment is quickly returned by improved efficiency and effectiveness in LLM development and deployment. Quick pros and cons Key advantages: Continuous improvement: The ability to add more requirements over time while ensuring existing functionality stays intact is a significant advantage. This allows for the evolution of the AI task in response to new requirements, ensuring that the system remains up-to-date and efficient. Better maintenance: This approach enables the easy validation of prompt performance with LLM updates. This is crucial for maintaining high standards of quality and reliability, as updates can sometimes introduce unintended changes in behavior. More flexibility: With a set of quality control tests, switching LLM providers becomes more straightforward. This flexibility allows us to adapt to changes in the market or technological advancements, ensuring we can always use the best tool for the job. Cost optimization: Data-driven evaluations enable better decisions on performance-to-cost ratio. By understanding the performance gains of different models, we can choose the most cost-effective solution that meets the needs. Time savings: Systematic evaluations provide quick feedback, reducing the need for manual testing. This efficiency allows to quickly iterate on prompt improvement and optimization, accelerating the development process. Challenges Initial time investment: Creating test fixtures and evaluation functions can require a significant investment of time.  Defining measurable validation criteria: Not all AI tasks have clear pass/fail conditions. Defining measurable criteria for validation can sometimes be challenging, especially for tasks that involve subjective or nuanced outputs. This requires careful consideration and may involve a difficult selection of the evaluation metrics. Cost associated with multiple tests: Multiple test use cases associated with 5 to 10 iterations can generate a high number of LLM requests for a single test automation. But if the cost of a single LLM call is neglectable, as it is in most cases for text input/output calls, the overall cost of a test remains minimal.   Conclusion: When should you implement this approach? Implementing this systematic testing approach is, of course, not always necessary, especially for simple tasks. However, for complex AI workflows in which precision and reliability are critical, this approach becomes highly valuable by offering a systematic way to assess and optimize prompt performance, preventing endless cycles of trial and error. By incorporating functional testing principles into Prompt Engineering, we transform a traditionally subjective and fragile process into one that is measurable, scalable, and robust. Not only does it enhance the reliability of LLM outputs, it helps achieve continuous improvement and efficient resource allocation. The decision to implement systematic prompt Testing should be based on the complexity of your project. For scenarios demanding high precision and consistency, investing the time to set up this methodology can significantly improve outcomes and speed up the development processes. However, for simpler tasks, a more classical, lightweight approach may be sufficient. The key is to balance the need for rigor with practical considerations, ensuring that your testing strategy aligns with your goals and constraints. Thanks for reading!

Creating efficient prompts for large language models often starts as a simple task… but it doesn’t always stay that way. Initially, following basic best practices seems sufficient: adopt the persona of a specialist, write clear instructions, require a specific response format, and include a few relevant examples. But as requirements multiply, contradictions emerge, and even minor modifications can introduce unexpected failures. What was working perfectly in one prompt version suddenly breaks in another.

If you have ever felt trapped in an endless loop of trial and error, adjusting one rule only to see another one fail, you’re not alone! The reality is that traditional prompt optimisation is clearly missing a structured, more scientific approach that will help to ensure reliability.

That’s where functional testing for prompt engineering comes in! This approach, inspired by methodologies of experimental science, leverages automated input-output testing with multiple iterations and algorithmic scoring to turn prompt engineering into a measurable, data-driven process. 

No more guesswork. No more tedious manual validation. Just precise and repeatable results that allow you to fine-tune prompts efficiently and confidently.

In this article, we will explore a systematic approach for mastering prompt engineering, which ensures your Llm outputs will be efficient and reliable even for the most complex AI tasks.

Balancing precision and consistency in prompt optimisation

Adding a large set of rules to a prompt can introduce partial contradictions between rules and lead to unexpected behaviors. This is especially true when following a pattern of starting with a general rule and following it with multiple exceptions or specific contradictory use cases. Adding specific rules and exceptions can cause conflict with the primary instruction and, potentially, with each other.

What might seem like a minor modification can unexpectedly impact other aspects of a prompt. This is not only true when adding a new rule but also when adding more detail to an existing rule, like changing the order of the set of instructions or even simply rewording it. These minor modifications can unintentionally change the way the model interprets and prioritizes the set of instructions.

The more details you add to a prompt, the greater the risk of unintended side effects. By trying to give too many details to every aspect of your task, you increase as well the risk of getting unexpected or deformed results. It is, therefore, essential to find the right balance between clarity and a high level of specification to maximise the relevance and consistency of the response. At a certain point, fixing one requirement can break two others, creating the frustrating feeling of taking one step forward and two steps backward in the optimization process.

Testing each change manually becomes quickly overwhelming. This is especially true when one needs to optimize prompts that must follow numerous competing specifications in a complex AI task. The process cannot simply be about modifying the prompt for one requirement after the other, hoping the previous instruction remains unaffected. It also can’t be a system of selecting examples and checking them by hand. A better process with a more scientific approach should focus on ensuring repeatability and reliability in prompt optimization.

From laboratory to AI: Why testing LLM responses requires multiple iterations

Science teaches us to use replicates to ensure reproducibility and build confidence in an experiment’s results. I have been working in academic research in chemistry and biology for more than a decade. In those fields, experimental results can be influenced by a multitude of factors that can lead to significant variability. To ensure the reliability and reproducibility of experimental results, scientists mostly employ a method known as triplicates. This approach involves conducting the same experiment three times under identical conditions, allowing the experimental variations to be of minor importance in the result. Statistical analysis (standard mean and deviation) conducted on the results, mostly in biology, allows the author of an experiment to determine the consistency of the results and strengthens confidence in the findings.

Just like in biology and chemistry, this approach can be used with LLMs to achieve reliable responses. With LLMs, the generation of responses is non-deterministic, meaning that the same input can lead to different outputs due to the probabilistic nature of the models. This variability is challenging when evaluating the reliability and consistency of LLM outputs.

In the same way that biological/chemical experiments require triplicates to ensure reproducibility, testing LLMs should need multiple iterations to measure reproducibility. A single test by use case is, therefore, not sufficient because it does not represent the inherent variability in LLM responses. At least five iterations per use case allow for a better assessment. By analyzing the consistency of the responses across these iterations, one can better evaluate the reliability of the model and identify any potential issues or variation. It ensures that the output of the model is correctly controlled.

Multiply this across 10 to 15 different prompt requirements, and one can easily understand how, without a structured testing approach, we end up spending time in trial-and-error testing with no efficient way to assess quality.

A systematic approach: Functional testing for prompt optimization

To address these challenges, a structured evaluation methodology can be used to ease and accelerate the testing process and enhance the reliability of LLM outputs. This approach has several key components:

  • Data fixtures: The approach’s core center is the data fixtures, which are composed of predefined input-output pairs specifically created for prompt testing. These fixtures serve as controlled scenarios that represent the various requirements and edge cases the LLM must handle. By using a diverse set of fixtures, the performance of the prompt can be evaluated efficiently across different conditions.
  • Automated test validation: This approach automates the validation of the requirements on a set of data fixtures by comparison between the expected outputs defined in the fixtures and the LLM response. This automated comparison ensures consistency and reduces the potential for human error or bias in the evaluation process. It allows for quick identification of discrepancies, enabling fine and efficient prompt adjustments.
  • Multiple iterations: To assess the inherent variability of the LLM responses, this method runs multiple iterations for each test case. This iterative approach mimics the triplicate method used in biological/chemical experiments, providing a more robust dataset for analysis. By observing the consistency of responses across iterations, we can better assess the stability and reliability of the prompt.
  • Algorithmic scoring: The results of each test case are scored algorithmically, reducing the need for long and laborious « human » evaluation. This scoring system is designed to be objective and quantitative, providing clear metrics for assessing the performance of the prompt. And by focusing on measurable outcomes, we can make data-driven decisions to optimize the prompt effectively.     

Step 1: Defining test data fixtures

Selecting or creating compatible test data fixtures is the most challenging step of our systematic approach because it requires careful thought. A fixture is not only any input-output pair; it must be crafted meticulously to evaluate the most accurate as possible performance of the LLM for a specific requirement. This process requires:

1. A deep understanding of the task and the behavior of the model to make sure the selected examples effectively test the expected output while minimizing ambiguity or bias.

2. Foresight into how the evaluation will be conducted algorithmically during the test.

The quality of a fixture, therefore, depends not only on the good representativeness of the example but also on ensuring it can be efficiently tested algorithmically.

A fixture consists of:

    • Input example: This is the data that will be given to the LLM for processing. It should represent a typical or edge-case scenario that the LLM is expected to handle. The input should be designed to cover a wide range of possible variations that the LLM might have to deal with in production.

    • Expected output: This is the expected result that the LLM should produce with the provided input example. It is used for comparison with the actual LLM response output during validation.

Step 2: Running automated tests

Once the test data fixtures are defined, the next step involves the execution of automated tests to systematically evaluate the performance of the LLM response on the selected use cases. As previously stated, this process makes sure that the prompt is thoroughly tested against various scenarios, providing a reliable evaluation of its efficiency.

Execution process

    1. Multiple iterations: For each test use case, the same input is provided to the LLM multiple times. A simple for loop in nb_iter with nb_iter = 5 and voila!

    2. Response comparison: After each iteration, the LLM response is compared to the expected output of the fixture. This comparison checks whether the LLM has correctly processed the input according to the specified requirements.

    3. Scoring mechanism: Each comparison results in a score:

        ◦ Pass (1): The response matches the expected output, indicating that the LLM has correctly handled the input.

        ◦ Fail (0): The response does not match the expected output, signaling a discrepancy that needs to be fixed.

    4. Final score calculation: The scores from all iterations are aggregated to calculate the overall final score. This score represents the proportion of successful responses out of the total number of iterations. A high score, of course, indicates high prompt performance and reliability.

Example: Removing author signatures from an article

Let’s consider a simple scenario where an AI task is to remove author signatures from an article. To efficiently test this functionality, we need a set of fixtures that represent the various signature styles. 

A dataset for this example could be:

Example Input Expected Output
A long article
Jean Leblanc		 The long article
A long article
P. W. Hartig		 The long article
A long article
MCZ		 The long article

Validation process:

  • Signature removal check: The validation function checks if the signature is absent from the rewritten text. This is easily done programmatically by searching for the signature needle in the haystack output text.
  • Test failure criteria: If the signature is still in the output, the test fails. This indicates that the LLM did not correctly remove the signature and that further adjustments to the prompt are required. If it is not, the test is passed. 

The test evaluation provides a final score that allows a data-driven assessment of the prompt efficiency. If it scores perfectly, there is no need for further optimization. However, in most cases, you will not get a perfect score because either the consistency of the LLM response to a case is low (for example, 3 out of 5 iterations scored positive) or there are edge cases that the model struggles with (0 out of 5 iterations). 

The feedback clearly indicates that there is still room for further improvements and it guides you to reexamine your prompt for ambiguous phrasing, conflicting rules, or edge cases. By continuously monitoring your score alongside your prompt modifications, you can incrementally reduce side effects, achieve greater efficiency and consistency, and approach an optimal and reliable output. 

A perfect score is, however, not always achievable with the selected model. Changing the model might just fix the situation. If it doesn’t, you know the limitations of your system and can take this fact into account in your workflow. With luck, this situation might just be solved in the near future with a simple model update. 

Benefits of this method 

  • Reliability of the result: Running five to ten iterations provides reliable statistics on the performance of the prompt. A single test run may succeed once but not twice, and consistent success for multiple iterations indicates a robust and well-optimized prompt.
  • Efficiency of the process: Unlike traditional scientific experiments that may take weeks or months to replicate, automated testing of LLMs can be carried out quickly. By setting a high number of iterations and waiting for a few minutes, we can obtain a high-quality, reproducible evaluation of the prompt efficiency.
  • Data-driven optimization: The score obtained from these tests provides a data-driven assessment of the prompt’s ability to meet requirements, allowing targeted improvements.
  • Side-by-side evaluation: Structured testing allows for an easy assessment of prompt versions. By comparing the test results, one can identify the most effective set of parameters for the instructions (phrasing, order of instructions) to achieve the desired results.
  • Quick iterative improvement: The ability to quickly test and iterate prompts is a real advantage to carefully construct the prompt ensuring that the previously validated requirements remain as the prompt increases in complexity and length.

By adopting this automated testing approach, we can systematically evaluate and enhance prompt performance, ensuring consistent and reliable outputs with the desired requirements. This method saves time and provides a robust analytical tool for continuous prompt optimization.

Systematic prompt testing: Beyond prompt optimization

Implementing a systematic prompt testing approach offers more advantages than just the initial prompt optimization. This methodology is valuable for other aspects of AI tasks:

    1. Model comparison:

        ◦ Provider evaluation: This approach allows the efficient comparison of different LLM providers, such as ChatGPT, Claude, Gemini, Mistral, etc., on the same tasks. It becomes easy to evaluate which model performs the best for their specific needs.

        ◦ Model version: State-of-the-art model versions are not always necessary when a prompt is well-optimized, even for complex AI tasks. A lightweight, faster version can provide the same results with a faster response. This approach allows a side-by-side comparison of the different versions of a model, such as Gemini 1.5 flash vs. 1.5 pro vs. 2.0 flash or ChatGPT 3.5 vs. 4o mini vs. 4o, and allows the data-driven selection of the model version.

    2. Version upgrades:

        ◦ Compatibility verification: When a new model version is released, systematic prompt testing helps validate if the upgrade maintains or improves the prompt performance. This is crucial for ensuring that updates do not unintentionally break the functionality.

        ◦ Seamless Transitions: By identifying key requirements and testing them, this method can facilitate better transitions to new model versions, allowing fast adjustment when necessary in order to maintain high-quality outputs.

    3. Cost optimization:

        ◦ Performance-to-cost ratio: Systematic prompt testing helps in choosing the best cost-effective model based on the performance-to-cost ratio. We can efficiently identify the most efficient option between performance and operational costs to get the best return on LLM costs.

Overcoming the challenges

The biggest challenge of this approach is the preparation of the set of test data fixtures, but the effort invested in this process will pay off significantly as time passes. Well-prepared fixtures save considerable debugging time and enhance model efficiency and reliability by providing a robust foundation for evaluating the LLM response. The initial investment is quickly returned by improved efficiency and effectiveness in LLM development and deployment.

Quick pros and cons

Key advantages:

  • Continuous improvement: The ability to add more requirements over time while ensuring existing functionality stays intact is a significant advantage. This allows for the evolution of the AI task in response to new requirements, ensuring that the system remains up-to-date and efficient.
  • Better maintenance: This approach enables the easy validation of prompt performance with LLM updates. This is crucial for maintaining high standards of quality and reliability, as updates can sometimes introduce unintended changes in behavior.
  • More flexibility: With a set of quality control tests, switching LLM providers becomes more straightforward. This flexibility allows us to adapt to changes in the market or technological advancements, ensuring we can always use the best tool for the job.
  • Cost optimization: Data-driven evaluations enable better decisions on performance-to-cost ratio. By understanding the performance gains of different models, we can choose the most cost-effective solution that meets the needs.
  • Time savings: Systematic evaluations provide quick feedback, reducing the need for manual testing. This efficiency allows to quickly iterate on prompt improvement and optimization, accelerating the development process.

Challenges

  • Initial time investment: Creating test fixtures and evaluation functions can require a significant investment of time. 
  • Defining measurable validation criteria: Not all AI tasks have clear pass/fail conditions. Defining measurable criteria for validation can sometimes be challenging, especially for tasks that involve subjective or nuanced outputs. This requires careful consideration and may involve a difficult selection of the evaluation metrics.
  • Cost associated with multiple tests: Multiple test use cases associated with 5 to 10 iterations can generate a high number of LLM requests for a single test automation. But if the cost of a single LLM call is neglectable, as it is in most cases for text input/output calls, the overall cost of a test remains minimal.  

Conclusion: When should you implement this approach?

Implementing this systematic testing approach is, of course, not always necessary, especially for simple tasks. However, for complex AI workflows in which precision and reliability are critical, this approach becomes highly valuable by offering a systematic way to assess and optimize prompt performance, preventing endless cycles of trial and error.

By incorporating functional testing principles into Prompt Engineering, we transform a traditionally subjective and fragile process into one that is measurable, scalable, and robust. Not only does it enhance the reliability of LLM outputs, it helps achieve continuous improvement and efficient resource allocation.

The decision to implement systematic prompt Testing should be based on the complexity of your project. For scenarios demanding high precision and consistency, investing the time to set up this methodology can significantly improve outcomes and speed up the development processes. However, for simpler tasks, a more classical, lightweight approach may be sufficient. The key is to balance the need for rigor with practical considerations, ensuring that your testing strategy aligns with your goals and constraints.

Thanks for reading!

Shape
Shape
Stay Ahead

Explore More Insights

More Articles

Stay ahead with more perspectives on cutting-edge power, infrastructure, energy,  bitcoin and AI solutions. Explore these articles to uncover strategies and insights shaping the future of industries.

Shape

Linux 7.0 debuts with some big changes for networking

The problem is that the existing ECN is too blunt. It’s a situation that AccECN looks to fix. “ECN was originally specified for TCP in such a way that only one feedback signal can be transmitted per Round-Trip Time,” the IETF draft specification for AccECN states. For basic congestion control

Read More »

U.S. Department of Energy Announces $14 Million for Enhanced Geothermal Systems Demonstration Project in Pennsylvania

WASHINGTON—The U.S. Department of Energy’s Hydrocarbons and Geothermal Energy Office (HGEO) today announced a $14 million project to support field tests for enhanced geothermal systems (EGS). EGS demonstration projects explore the greater potential for geothermal technology to provide reliable, cost-effective electricity using the earth’s abundant heat resources, supporting the Trump Administration’s commitments to advance energy addition and reduce energy costs for American families and businesses. Led by the Pennsylvania Department of Environmental Protection, the project will leverage the significant thermal resources in the Appalachian Utica Shale to assess the efficacy and scalability of EGS in the eastern United States.  “The Department of Energy’s investments in enhanced geothermal systems represent a key advancement in our national energy strategy as we explore innovative ways to reach and use geothermal resources beyond what is currently possible,” said Kyle Haustveit, Assistant Secretary of the Hydrocarbons and Geothermal Energy Office. “As the first enhanced geothermal systems demonstration site located in the eastern United States, this project offers an important opportunity to assess the ability of such systems to deliver reliable, affordable geothermal electricity to Americans nationwide.” Using geothermal resources for electricity production requires fluid to flow among hot rocks in the subsurface and then be drawn to the surface in the form of steam or hot water. While underground heat exists everywhere, many locations lack adequate water or conditions that facilitate fluid flow necessary to recover that heat energy. In those cases, EGS can be used to create a human-made underground reservoir to tap that heat for energy. Demonstration projects are vital to help expand knowledge and data about EGS reservoirs and how they function, and to understand EGS in a variety of geographic locations, geologic formations, and subsurface conditions. Successful demonstrations will help spur further growth of geothermal energy. The Pennsylvania EGS project activities will include converting a horizontal shale gas

Read More »

Wright, Zeldin, and Burgum Break Ground on NESE Pipeline in New York City to Deliver Reliable, Affordable Natural Gas to the Northeast

NEW YORK—U.S. Secretary of Energy Chris Wright, U.S. Environmental Protection Agency Administrator Lee Zeldin, and U.S. Secretary of the Interior Doug Burgum today participated in a groundbreaking ceremony for the Northeast Supply Enhancement (NESE) Pipeline. This pipeline, of Williams Companies, will transport natural gas from Pennsylvania into New York City and Long Island, providing affordable and reliable energy for millions of Americans while meeting the growing energy demands of the region. President Trump and his National Energy Dominance Council worked across party lines to secure the necessary permits for this project from the states of New York and New Jersey last fall. NESE is an expansion of Williams’ Transco pipeline system across Pennsylvania, New Jersey, and New York that will add 400,000 dekatherms per day of capacity. This is enough energy to serve the equivalent of 2.3 million homes. NESE remains on track to be in service by the fourth quarter of 2027. “For decades, poor political choices obstructed the building of energy infrastructure, leading to higher energy costs for millions of Americans. President Trump promised to lower energy costs and to get America building again—that is exactly what the groundbreaking of the NESE pipeline will accomplish,” said Energy Secretary Chris Wright. “This project is a win-win: natural gas is a reliable, low-cost, clean burning option for New Yorkers to heat and power their homes and businesses. President Trump, Secretary Burgum, Administrator Zeldin and I will continue fighting to build more energy infrastructure so that all Americans have access to affordable, reliable, and secure American energy.” “Breaking ground on the NESE pipeline marks a massive milestone for millions of New Yorkers seeking access to reliable, affordable natural gas. Delivering natural gas from Pennsylvania to New York City and Long Island will lower costs while helping to meet the growing energy

Read More »

BW Energy granted 25-year extension of license offshore Gabon

BW Energy Gabon has received approval from the Ministry of Oil and Gas of the Gabonese Republic to extend the Dussafu Marin production license offshore Gabon, West Africa. The license period has been extended to 2053 from 2028, inclusive of three 5-year option periods from 2038 onwards. The prior contract was until 2038 inclusive of two 5-year option periods from 2028 onwards. The extra time “provides long-term visibility for production, investments, and reserve development” of the operator’s “core producing asset,” the company said in a release Apr. 7. Ongoing license projects include MaBoMo Phase 2, with planned first oil in second-half 2026, and the Bourdon development following its discovery last year. The timeline also “strengthens the foundation for future infrastructure‑led growth opportunities across the adjacent Niosi and Guduma licenses, both operated by BW Energy,” the company continued. The Dussafu Marin permit is a development and exploitation license with multiple discoveries and prospects lying within a proven oil and gas play fairway within Southern Gabon basin. To the northwest of the block is the Etame-Ebouri Trend, a collection of fields producing from the pre-salt Gamba and Dentale sandstones, and to the north are Lucina and M’Bya fields which produce from the syn-rift Lucina sandstones beneath the Gamba. Oil fields within the Dussafu Permit include Moubenga, Walt Whitman, Ruche, Ruche North East, Tortue, Hibiscus, and Hibiscus North. BW Energy Gabon is operator at Dussafu (73.50%) with partners Panoro Energy ASA (17.5%) and Gabon Oil Co. (9%). Dussafu.

Read More »

Santos plans development of North Slope’s Quokka Unit

Santos Ltd. has started development planning in the Quokka Unit on Alaska’s North Slope after further delineating the Nanushuk reservoir. The Quokka-1 appraisal well spudded on Jan. 1, 2026, about 6 six miles from the Mitquq-1 discovery well drilled in 2020. It was drilled to 4,787 ft TD and encountered a high-quality reservoir with about 143 ft of net oil pay in the Nanushuk formation, demonstrating an average porosity of 19%. Following a single stage fracture stimulation, the well achieved a flow rate of 2,190 bo/d. Reservoir sands correlated between the two discoveries, coupled with fluid analyses, confirm the presence of high‑quality, light‑gravity oil, supporting strong well performance and improved pricing relative to Pikka oil. Together with additional geological data, these results underpin the potential for a two‑drill‑site development with production capacity comparable to Pikka phase 1, the company said.  Rate and resource potential for the two-drill-site development is being evaluated. Resource estimation is ongoing and appraisal results will be evaluated as part of the FY26 contingent resource assessment. In FY25, Santos reported 2C contingent resources of 177 MMboe for the Quokka Unit. Based on these results, Santos has started development planning, including the initiation of key permitting activities. Santos is operator of the Quokka Unit (51%) with partner Repsol (49%).

Read More »

Fluor, Axens secure contracts for US grassroots refinery project

Fluor Corp. and Axens Group have been awarded key contracts for America First Refining’s (AFR) proposed grassroots refinery at the Port of Brownsville, Tex., advancing development of what would be the first new US refinery to be built in more than 50 years. Fluor will execute front-end engineering and design (FEED) for the project, while Axens will serve as technology licensor of core refining process technologies to be used at the site, the service providers said in separate Apr. 7 releases. The AFR refinery is designed to process more than 60 million bbl/year—or about 164,400 b/d—of US light shale crude into transportation fuels, including gasoline, diesel, and jet fuel. Contract details Without disclosing a specific value of its contract, Fluor said the scope of its FEED study will cover early-stage engineering and design required to define project execution, cost, and schedule based on a complex that will incorporate commercially proven technologies to improve efficiency and emissions performance while processing domestic shale crude. As technology licensor, Axens said it will deliver process technologies for key refining units at the site, including those for: Naphtha, diesel hydrotreating. Continuous catalytic reforming. Isomerization. Alongside supporting improved fuel-quality specifications, the unspecified technologies to be supplied for the refinery will also help to reduce overall energy consumption at the site. Axens—which confirmed its involvement since 2017 in working with AFR on early-stage development of the project—said this latest licensing agreement will also cover engineering support, equipment, catalysts, and services across the refinery’s process configuration. Project background, commercial framework Upon first announcing the project in March 2026, AFR said the proposed development came alongside an already signed 20-year offtake agreement with a global integrated oil company covering 1.2 billion bbl of US light shale crude, as well as capital investment to support construction. As part of the

Read More »

EIA: US crude inventories up 3.1 million bbl

US crude oil inventories for the week ended Apr. 3, excluding the Strategic Petroleum Reserve, increased by 3.1 million bbl from the previous week, according to data from the US Energy Information Administration (EIA). At 464.7 million bbl, US crude oil inventories are about 2% above the 5-year average for this time of year, the EIA report indicated. EIA said total motor gasoline inventories decreased by 1.6 million bbl from last week and are about 3% above the 5-year average for this time of year. Finished gasoline inventories increased while blending components inventories decreased last week. Distillate fuel inventories decreased by 3.1 million bbl last week and are about 5% below the 5-year average for this time of year. Propane-propylene inventories increased by 600,000 bbl from last week and are 71% above the 5-year average for this time of year, EIA said. US crude oil refinery inputs averaged 16.3 million b/d for the week ended Apr. 3, which was 129,000 b/d less than the previous week’s average. Refineries operated at 92% of capacity. Gasoline production decreased, averaging 9.4 million b/d. Distillate fuel production increased, averaging 5.0 million b/d. US crude oil imports averaged 6.3 million b/d, down 130,000 b/d from the previous week. Over the last 4 weeks, crude oil imports averaged about 6.6 million b/d, 9.1% more than the same 4-week period last year. Total motor gasoline imports averaged 571,000 b/d. Distillate fuel imports averaged 152,000 b/d.

Read More »

Cisco just made two moves to own the AI infrastructure stack

In a world of autonomous agents, identity and access become the de facto safety rails. Astrix is designed to inventory these non-human identities, map their permissions, detect toxic combinations, and remediate overprivileged access before it becomes an exploit or a data leak. That capability integrates directly with Cisco’s broader zero-trust and identity-centric security strategy, in which the network enforces policy based on who or what the entity is, not on which subnet it resides in. How this strengthens Cisco’s secure networking story Cisco has positioned itself as the vendor that can deliver “AI-ready, secure networks” spanning campus, data center, cloud, and edge. Galileo and Astrix extend that narrative from infrastructure into AI behavior and identity governance: The network becomes the high‑performance, policy‑enforcing substrate for AI traffic and data. Splunk plus Galileo becomes the observability plane for AI agents, linking AI incidents to network and application signals. Security plus Astrix becomes the identity and permission-control layer that constrains what AI agents can actually do within the environment. This is the core of Cisco’s emerging “Secure AI” posture: not just using AI to improve security but securing AI itself as it is embedded across every workflow, API, and device. For customers, that means AI initiatives can be brought under the same operational and compliance disciplines already used for networks and apps, rather than existing as unmanaged risk islands. Why this matters to Cisco customers Most large Cisco accounts are exactly the enterprises now experimenting with AI agents in contact centers, IT operations, and business workflows. They face three practical problems: They cannot see what agents are doing end‑to‑end, or measure quality beyond offline benchmarks. They lack a coherent model for managing the identities, secrets, and permissions those agents depend on. Their security and networking teams are often disconnected from AI projects happening in lines of business.

Read More »

From Buildings to Token Factories: Compu Dynamics CEO Steve Altizer On Why AI Is Rewriting the Data Center Design Playbook

Not Falling Short—Just Not Optimized Altizer drew a clear distinction. Traditional data centers can run AI workloads, but they weren’t built for them. “We’re not falling short much, we’re just not optimizing.” The gap shows up most clearly in density. Legacy facilities were designed for roughly 300 to 400 watts per square foot. AI pushes that to 2,000 to 4,000 watts per square foot—changing not just rack design, but the logic of the entire facility. For Altizer, AI-ready infrastructure starts with fundamentals: access to water for heat rejection, significantly higher power density, and in some cases specific redundancy topologies favored by chip makers. It also requires liquid cooling loops extended to the rack and, critically, flexibility in the white space. That last point is the hardest to reconcile with traditional design. “The GPUs change… your power requirements change… your liquid cooling requirements change. The data center needs to change with it.” Buildings are static. AI is not. Rethinking Modular: From Containers to Systems “Modular” has been part of the data center vocabulary for years, but Altizer argues most of the industry is still thinking about it the wrong way. The old model centered on ISO containers. The emerging model focuses on modularizing the white space itself. “We’re not building buildings—we’re building assemblies of equipment.” Compu Dynamics is pushing toward factory-built IT modules that can be delivered and assembled on-site. A standard 5 MW block consists of 10 modules, stacked into a two-story configuration and designed for transport by trailer across the U.S. From there, scale becomes repeatable. Blocks can be placed adjacent or connected to create larger deployments, moving from 5 MW to 10 MW and beyond. The point is not just scalability; it’s repeatability and speed. Altizer ties this directly to a broader shift in how data centers are

Read More »

Data centers are moving inland, away from some traditional locations

The future is even less clear the further you go out. The vast majority of data centers planned for launch between 2028 and 2032 have yet to break ground and only a sliver are under construction. Those delays, it seems, appear to be twofold: first, the well-documented component shortage. Not just memory and storage, but batteries, electrical transformers, and circuit breakers. They all make up less than 10% of the cost to construct one data center, but as Andrew Likens, energy and infrastructure lead at AI data center provider Crusoe’s told Bloomberg, it’s impossible to build new data centers without them. “If one piece of your supply chain is delayed, then your whole project can’t deliver,” Likens said. “It is a pretty wild puzzle at the moment.” Second problem is the growing rebellion against data centers, both by citizens and governments alike. The latest pushback comes from the Seminole nation of Native Americans, who have banned data centers on their tribal lands. Of the data centers that are coming online in the next few months, the top states reflect what Synergy has been saying about data center migration to the interior of the country. Texas is leading the way, with 22.5 GW coming online, followed by New Mexico at 8.3 GW and Pennsylvania, which is making a major push for data centers to come to the state, at 7.1 GW.

Read More »

Hillwood, PowerHouse Advance $20B Joliet Data Campus as Midwest AI Buildout Accelerates

The approval of the Joliet Technology Center signals that the Chicago region is being pulled into the Midwest’s next phase of AI infrastructure development, one that has so far been led by Ohio and defined by scale, power demand, and rising public scrutiny. It also underscores a growing reality: local governments are beginning to understand exactly what that shift entails. On March 19, 2026, the Joliet City Council voted 8–1 to approve the conditional annexation of roughly 795 acres for the proposed Joliet Technology Center, a $20 billion data center campus backed by Hillwood and PowerHouse Data Centers. The site, near Rowell and Bernhard Roads on Joliet’s east side, is planned as a 24-building, multi-phase development that would rank among the most consequential digital infrastructure projects ever approved in Illinois. Joliet is now a clear case study in how the Midwest’s data center market is evolving: massive land assemblies, utility-scale power requirements, front-loaded community concessions, increasingly organized local opposition, and regulators working to ensure that the costs of AI infrastructure are not shifted onto ratepayers. A Project Too Large to Call Routine The Joliet Technology Center is a campus-scale industrial platform built for the AI era. Plans call for 24 two-story buildings of roughly 144,500 square feet each, with total development estimated at approximately 6.9 million square feet and up to 1.8 GW of eventual capacity. That places the project firmly in the emerging “AI factory” category, e.g. far-removed from the incremental, metro-edge data center expansions that defined earlier growth cycles. The distinction is critical. AI-scale campuses operate on a different economic and technical model. Fiber access and metro proximity are no longer enough. These developments require large, contiguous power blocks, land to support phased substation and utility infrastructure, and a political framework capable of absorbing what is effectively heavy

Read More »

AI is a Positive Catalyst for Grid Growth

Data centers, particularly those optimized for artificial intelligence workloads, are frequently characterized in public discourse as a disruptive threat to grid stability and ratepayer affordability. But behind-the-narrative as we are, the AI‑driven data center growth is simply illuminating pre‑existing systemic weaknesses in electric infrastructure that have accumulated over more than a decade of underinvestment in transmission, substations, and interconnection capacity. Over the same period, many utilities operated under planning assumptions shaped by slow demand growth and regulatory frameworks that incentivized incremental upgrades rather than large, anticipatory capital programs. As a result, the emergence of gigawatt‑scale computing campuses appears to be a sudden shock to a system that, in reality, was already misaligned with long‑term decarbonization, electrification, and digitalization objectives. Utilities have been asked to do more with aging grids, slow permitting, and chronically constrained capital, and now AI and cloud are finally putting real urgency — and real investment — behind modernizing that backbone. In that sense, large‑scale compute is not the problem; it is the catalyst that makes it impossible to ignore the problem any longer. We are at a moment when data centers, and especially AI data centers, are being blamed for exposing weaknesses that were already there, when in reality they are giving society a chance to fix a power system that has been underbuilt for more than a decade. Utilities have been asked to do more with aging grids, slow permitting, and limited investment, and now AI and cloud are finally putting real urgency — and real capital — behind modernizing that backbone. In that sense, data centers aren’t the problem; they are the catalyst that makes it impossible to ignore the problem any longer. AI Demand Provided a Long‑Overdue Stress Test The nature of AI workloads intensified this dynamic. High‑performance computing clusters concentrate substantial power

Read More »

From Land Grab to Structured Scale: Kirkland & Ellis Explains How Capital, Power, and Deal Complexity Are Defining the AI Data Center Boom

The AI data center market is no longer defined by speed alone. For much of the past three years, capital moved aggressively into digital infrastructure, chasing land, power, and platform scale as generative AI workloads began to reshape demand curves. But as Melissa Kalka, M&A and private equity partner, and Kimberly McGrath, real estate partner at Kirkland & Ellis, explain on the latest episode of the Data Center Frontier Show, the industry is now entering a more complex and more consequential phase. The land grab is over. Execution has begun. Capital remains abundant, but it is no longer forgiving. From Capital Rush to Capital Discipline As noted by Kalka and McGrath, the period from roughly 2022 through 2025 marked a rapid acceleration in AI infrastructure investment. Take-private deals involving CyrusOne, QTS, and Switch signaled a structural shift, while hyperscale demand scaled from tens of megawatts to hundreds, and now toward gigawatt-class campuses. But the current phase is not defined by a pullback in capital. Instead, it reflects an expansion of investment pathways and a corresponding increase in scrutiny. “There’s actually more deal flow now,” Kalka notes, pointing to the growing range of entry points across the capital stack, including development vehicles, yield-oriented structures, and private credit. With more capital chasing larger and more complex opportunities, investors are evaluating not just platforms, but the full lifecycle of assets from early-stage development through stabilization and long-term hold. That shift has pulled capital earlier into the process, where risk is higher and less defined. Power availability, permitting, and execution timelines are now central to underwriting decisions. What Defines a “Bankable” Platform In this environment, the definition of a bankable data center platform has tightened. Execution history remains foundational. Investors are looking for consistent delivery, operational reliability, and clean contractual performance. But those factors alone

Read More »

Microsoft will invest $80B in AI data centers in fiscal 2025

And Microsoft isn’t the only one that is ramping up its investments into AI-enabled data centers. Rival cloud service providers are all investing in either upgrading or opening new data centers to capture a larger chunk of business from developers and users of large language models (LLMs).  In a report published in October 2024, Bloomberg Intelligence estimated that demand for generative AI would push Microsoft, AWS, Google, Oracle, Meta, and Apple would between them devote $200 billion to capex in 2025, up from $110 billion in 2023. Microsoft is one of the biggest spenders, followed closely by Google and AWS, Bloomberg Intelligence said. Its estimate of Microsoft’s capital spending on AI, at $62.4 billion for calendar 2025, is lower than Smith’s claim that the company will invest $80 billion in the fiscal year to June 30, 2025. Both figures, though, are way higher than Microsoft’s 2020 capital expenditure of “just” $17.6 billion. The majority of the increased spending is tied to cloud services and the expansion of AI infrastructure needed to provide compute capacity for OpenAI workloads. Separately, last October Amazon CEO Andy Jassy said his company planned total capex spend of $75 billion in 2024 and even more in 2025, with much of it going to AWS, its cloud computing division.

Read More »

John Deere unveils more autonomous farm machines to address skill labor shortage

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Self-driving tractors might be the path to self-driving cars. John Deere has revealed a new line of autonomous machines and tech across agriculture, construction and commercial landscaping. The Moline, Illinois-based John Deere has been in business for 187 years, yet it’s been a regular as a non-tech company showing off technology at the big tech trade show in Las Vegas and is back at CES 2025 with more autonomous tractors and other vehicles. This is not something we usually cover, but John Deere has a lot of data that is interesting in the big picture of tech. The message from the company is that there aren’t enough skilled farm laborers to do the work that its customers need. It’s been a challenge for most of the last two decades, said Jahmy Hindman, CTO at John Deere, in a briefing. Much of the tech will come this fall and after that. He noted that the average farmer in the U.S. is over 58 and works 12 to 18 hours a day to grow food for us. And he said the American Farm Bureau Federation estimates there are roughly 2.4 million farm jobs that need to be filled annually; and the agricultural work force continues to shrink. (This is my hint to the anti-immigration crowd). John Deere’s autonomous 9RX Tractor. Farmers can oversee it using an app. While each of these industries experiences their own set of challenges, a commonality across all is skilled labor availability. In construction, about 80% percent of contractors struggle to find skilled labor. And in commercial landscaping, 86% of landscaping business owners can’t find labor to fill open positions, he said. “They have to figure out how to do

Read More »

2025 playbook for enterprise AI success, from agents to evals

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More 2025 is poised to be a pivotal year for enterprise AI. The past year has seen rapid innovation, and this year will see the same. This has made it more critical than ever to revisit your AI strategy to stay competitive and create value for your customers. From scaling AI agents to optimizing costs, here are the five critical areas enterprises should prioritize for their AI strategy this year. 1. Agents: the next generation of automation AI agents are no longer theoretical. In 2025, they’re indispensable tools for enterprises looking to streamline operations and enhance customer interactions. Unlike traditional software, agents powered by large language models (LLMs) can make nuanced decisions, navigate complex multi-step tasks, and integrate seamlessly with tools and APIs. At the start of 2024, agents were not ready for prime time, making frustrating mistakes like hallucinating URLs. They started getting better as frontier large language models themselves improved. “Let me put it this way,” said Sam Witteveen, cofounder of Red Dragon, a company that develops agents for companies, and that recently reviewed the 48 agents it built last year. “Interestingly, the ones that we built at the start of the year, a lot of those worked way better at the end of the year just because the models got better.” Witteveen shared this in the video podcast we filmed to discuss these five big trends in detail. Models are getting better and hallucinating less, and they’re also being trained to do agentic tasks. Another feature that the model providers are researching is a way to use the LLM as a judge, and as models get cheaper (something we’ll cover below), companies can use three or more models to

Read More »

OpenAI’s red teaming innovations define new essentials for security leaders in the AI era

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More OpenAI has taken a more aggressive approach to red teaming than its AI competitors, demonstrating its security teams’ advanced capabilities in two areas: multi-step reinforcement and external red teaming. OpenAI recently released two papers that set a new competitive standard for improving the quality, reliability and safety of AI models in these two techniques and more. The first paper, “OpenAI’s Approach to External Red Teaming for AI Models and Systems,” reports that specialized teams outside the company have proven effective in uncovering vulnerabilities that might otherwise have made it into a released model because in-house testing techniques may have missed them. In the second paper, “Diverse and Effective Red Teaming with Auto-Generated Rewards and Multi-Step Reinforcement Learning,” OpenAI introduces an automated framework that relies on iterative reinforcement learning to generate a broad spectrum of novel, wide-ranging attacks. Going all-in on red teaming pays practical, competitive dividends It’s encouraging to see competitive intensity in red teaming growing among AI companies. When Anthropic released its AI red team guidelines in June of last year, it joined AI providers including Google, Microsoft, Nvidia, OpenAI, and even the U.S.’s National Institute of Standards and Technology (NIST), which all had released red teaming frameworks. Investing heavily in red teaming yields tangible benefits for security leaders in any organization. OpenAI’s paper on external red teaming provides a detailed analysis of how the company strives to create specialized external teams that include cybersecurity and subject matter experts. The goal is to see if knowledgeable external teams can defeat models’ security perimeters and find gaps in their security, biases and controls that prompt-based testing couldn’t find. What makes OpenAI’s recent papers noteworthy is how well they define using human-in-the-middle

Read More »
More Articles