Microsoft.Extensions.AI.Evaluation - API Usage Examples - Instructions

September 3, 2025 · View on GitHub

All examples are included in Examples.sln and are structured as unit tests.

If you plan to run the examples against an LLM endpoint hosted on Azure Open AI, you can read the quick version of the instructions below to get started quickly.
Read the long version of the instructions if you plan to use an LLM endpoint hosted on Azure AI Inference, Ollama or Open AI instead. The long version also contains some additional details and clarifications that you may find useful.
[Optional] To run the examples that demonstrate how to use the evaluators included as part of the Microsoft.Extensions.AI.Evaluation.Safety NuGet package (such as ProtectedMaterialEvaluator, IndirectAttackEvaluator, ViolenceEvaluator, etc.), see Running content safety evaluation examples. These examples are skipped by default and are only enabled after the corresponding environment variables that configure the Azure AI Foundry Evaluation service have been set.
[Optional] To run the examples that demonstrate how to use the Azure storage providers to store evaluation results and cached LLM responses, see Running Azure storage examples. These examples are skipped by default and are only enabled after the corresponding environment variables that configure Azure storage have been set.
Then read on to learn how to generate reports using the aieval dotnet tool.

Note: The examples included in this solution have been primarily tested against the GPT-4o model. The prompts present within the examples directly, as well as the prompts present within the evaluators included as part of the Microsoft.Extensions.AI.Evaluation.Quality NuGet package (such as CoherenceEvaluator, RelevanceEvaluator, etc.) perform well against GPT-4o. However, they may not perform as well against other models. So, the evaluations performed in the examples may produce poor results against some other models.

That said, it can still be an interesting exercise to try out the examples against other models to understand how different models perform (i.e., how quickly or slowly are they able to perform the evaluations, how accurately can they score the coherence, relevance, fluency, etc. of the supplied responses, and so on).

Quick version

Set the following environment variables. The most convenient option may be to set these environment variables globally for your user / machine.

SET EVAL_SAMPLE_AZURE_OPENAI_ENDPOINT=https://<your-endpoint>.openai.azure.com/
SET EVAL_SAMPLE_AZURE_OPENAI_MODEL=<model-deployment-name>
SET EVAL_SAMPLE_STORAGE_ROOT_PATH=<full path to a directory under which the examples can store evaluation data>

Note that the example code uses DefaultAzureCredential to authenticate with Azure Open AI. See Authentication during local development for more information on how to authenticate with DefaultAzureCredential during local development. For example, you can use one of the methods described here to sign in to Azure with your own Azure credentials using Visual Studio or the command line.
[Optional] Open Examples.sln in Visual Studio or Visual Studio Code.
Run dotnet test from under src\microsoft-extensions-ai-evaluation\api to build and run all the unit tests. Alternatively, if you have the solution open in Visual Studio or Visual Studio Code, build and run all the unit tests using the IDE's test runner.
Inspect the evaluation reports generated by the last couple of unit tests in the Reporting API Examples project to understand how the different concepts demonstrated in the examples are surfaced in these reports. The reports should be automatically opened in your default browser after the tests complete running.

Long version

The following setup steps are required to set up the LLM connection / endpoint that will be used for all examples (i.e., for both the Evaluation API Examples, as well as the Reporting API Examples).

Select your LLM provider: The examples are authored to run against Azure Open AI by default and have been tested against GPT-4o. You can easily switch to use Azure AI Inference, Ollama, or Open AI by changing one line of code within TestSetup.GetChatConfiguration().
Configure environment variables that define the connection parameters for your LLM endpoint: Open EnvironmentVariables.cs to figure out the set of environment variables that you need to set depending on the LLM provider selected above. For example, if the model is hosted on Open AI, you will need to set the following environment variables to specify the API key, and model name respectively: EVAL_SAMPLE_OPENAI_API_KEY and EVAL_SAMPLE_OPENAI_MODEL.

Note: If you plan to run the example unit tests using Visual Studio or Visual Studio Code, the most convenient option may be to set the above environment variables globally for your user / machine. This will ensure that the environment variables are always available to your IDE's test runner (and to all test runner child processes) regardless of how the IDE was launched.

The following step is required to run the Reporting API Examples.

Set the EVAL_SAMPLE_STORAGE_ROOT_PATH environment variable: This variable should point to a dedicated directory on your machine under which all the cached LLM responses, evaluation result data, and generated reports for the Reporting API Examples will be saved. Again, if you plan to run the examples using Visual Studio or Visual Studio Code, the most convenient option may be to set this environment variable globally for your user / machine.
[Optional] Open Examples.sln in Visual Studio or Visual Studio Code.
Run dotnet test from under src\microsoft-extensions-ai-evaluation\api to build and run all the unit tests. Alternatively, if you have the solution open in Visual Studio or Visual Studio Code, build and run all the unit tests using the IDE's test runner.
The last few unit tests in the Reporting API Examples project demonstrate how to programmatically generate reports containing the results for all evaluations performed in previously executed tests. The reports should be automatically opened in your default browser after the tests complete running. Inspect each report to understand how the different concepts demonstrated in the examples are surfaced in the report.

Note: The report generation tests above will produce a report that includes results from all included examples only if you execute all unit tests in the project as part of a single execution. In other words, the report generation examples above won't quite work if you execute the tests one at a time. This is because the code in the report generation examples only includes results for the the latest (current) execution in the generated report. When the examples are executed one at a time, each result is considered part of a different (previous) execution, and the results from these older executions will not be included in the generated report.

Running content safety evaluation examples

Follow the below steps to run the included examples that demonstrate how to use the evaluators available as part of the Microsoft.Extensions.AI.Evaluation.Safety NuGet package to evaluate responses for presence of harmful or protected content. These evaluators leverage the Azure AI Foundry Evaluation service and require some special setup. The corresponding examples are skipped by default and can be enabled by setting the following environment variables.

Firstly, you need an Azure subscription.
Within this subscription, create an Azure AI Foundry project.
Finally, copy the endpoint URL for the above Azure AI Foundry project and set the following environment variable. This will configure the examples to use the Azure AI Foundry Evaluation service from the above Azure AI Foundry project.

set EVAL_SAMPLE_AZURE_AI_PROJECT_ENDPOINT=<The endpoint URL for the above Azure AI Foundry project>

Note: If you are using an older hub-based Azure AI Foundry project, set the following environment variables instead.

set EVAL_SAMPLE_AZURE_SUBSCRIPTION_ID=<The ID of the Azure subscription containing your hub-based Azure AI Foundry project>
set EVAL_SAMPLE_AZURE_RESOURCE_GROUP=<The name of the Azure resource group containing your hub-based Azure AI Foundry project>
set EVAL_SAMPLE_AZURE_AI_PROJECT=<The name of the your hub-based Azure AI Foundry project>

Running Azure storage examples

Follow the below steps to run the included examples that demonstrate how to use the Azure storage providers available as part of the Microsoft.Extensions.AI.Evaluation.Reporting.Azure NuGet package to store evaluation results and cached LLM responses. These examples are skipped by default and can be enabled by setting the following environment variables.

Firstly, you need an Azure subscription.
Within this subscription, create a resource group.
Then create an Azure storage account within the above resource group. Make sure to select the 'hierarchical namespace' option when creating the storage account. This is required to enable the use of the Azure Data Lake Storage Gen2 APIs that the storage providers used in the above examples require.
Finally, create a storage container within the above storage account.
Once you have created the above artifacts, set the following environment variables to configure the examples to use the storage container created above.

set EVAL_SAMPLE_AZURE_STORAGE_ACCOUNT_ENDPOINT=<The endpoint URL of the above Azure storage account>
set EVAL_SAMPLE_AZURE_STORAGE_CONTAINER=<The name of the above Azure storage container>

Generating reports using the `aieval` dotnet tool

You can also use the command line based aieval dotnet tool that ships as part of the Microsoft.Extensions.AI.Evaluation.Console NuGet package to generate and view reports. In fact, the recommended way to generate reports as part of your CI builds is to run this tool in your CI/CD pipeline.

To generate a report using the aieval dotnet tool,

First run all the included examples as part of a single test run as described above (either from the command line by running dotnet test, or from within the IDE).
Then execute the following command from under src/microsoft-extensions-ai-evaluation/api. This will make the tool available for use on the command line under this directory.
```
dotnet tool restore
```
Note: It is a good idea to re-run the above command any time you pull down newer versions of this sample.
Then run the installed aieval tool using the following command after replacing <EVAL_SAMPLE_STORAGE_ROOT_PATH> with the path to the storage root directory that you specified above. This will also open the generated report.html below in your default browser.
```
dotnet aieval report -p <EVAL_SAMPLE_STORAGE_ROOT_PATH> -o <EVAL_SAMPLE_STORAGE_ROOT_PATH>\report.html --open
```
Note: The aieval tool also supports some options for cleaning up cached responses and stored evaluation results. Run dotnet aieval --help for more information.

Installing the `aieval` dotnet tool in your own repository

For production usages outside the current sample, we recommend that you install the aieval tool under your repository. To do this,

Run the following command from the directory in your repository where you wish to install the tool. Before running, remember to replace <VERSION> to match the version of the Microsoft.Extensions.AI.Evaluation NuGet package referenced in your repository.
```
dotnet tool install Microsoft.Extensions.AI.Evaluation.Console --version <VERSION> --create-manifest-if-needed
```
Commit the resulting .config/dotnet-tools.json file to your repository's source control. Other users will then be able to restore the tool by running the following command from the same directory.
```
dotnet tool restore
```
Then as you update the versions of Microsoft.Extensions.AI.Evaluation and related NuGet packages in your repository, you can also update the version of the aieval tool in the committed .config/dotnet-tools.json file to keep the installed tool and the referenced libraries in sync.