Benchmarking of Instance Segmentation Model

To use this service and achieve your goals, you can use the user_config.yaml or directly update the benchmarking_config.yaml file and use it. This file provides an example of how to configure the benchmarking service to meet your specific needs.

Alternatively, you can follow the tutorial below, which shows how to benchmark your pre-trained instance segmentation model using our benchmarking service.

1.1 Set the model and the operation mode

operation_mode should be set to benchmarking and the benchmarking section should be filled as in the following example:

general:
  project_name: coco_instance_seg
model:
  model_type: yolov8n_seg
  model_path: https://github.com/stm32-hotspot/ultralytics/raw/refs/heads/main/examples/YOLOv8-STEdgeAI/stedgeai_models/segmentation/yolov8n_256_quant_pc_ii_seg_coco-st.tflite

operation_mode: benchmarking

1.2 Set benchmarking tools and parameters

The STEdgeAI Developer Cloud allows you to benchmark your model and estimate its footprints and inference time for different STM32 target devices. To use this feature, set the on_cloud attribute to True. Alternatively, you can use STEdgeAI to benchmark your model and estimate its footprints for STM32 target devices locally. To do this, make sure to add the path to the stedgeai executable under the path_to_stedgeai attribute and set the on_cloud attribute to False.

The optimization defines the optimization used to generate the C model, options: "balanced", "time", "ram".

The board attribute is used to provide the name of the STM32 board to benchmark the model on. The available boards are 'STM32N6570-DK', 'STM32H747I-DISCO', 'STM32H7B3I-DK', 'STM32F469I-DISCO', 'B-U585I-IOT02A', 'STM32L4R9I-DISCO', 'NUCLEO-H743ZI2', 'STM32H735G-DK', 'STM32F769I-DISCO', 'NUCLEO-G474RE', 'NUCLEO-F401RE', and 'STM32F746G-DISCO'.

tools:
  stedgeai:
    optimization: balanced
    on_cloud: True
    path_to_stedgeai: C:/ST/STEdgeAI/<x.y>/Utilities/windows/stedgeai.exe
  path_to_cubeIDE: C:/ST/STM32CubeIDE_<*.*.*>/STM32CubeIDE/stm32cubeide.exe

benchmarking:
   board:   STM32MP257F-EV1    # Name of the STM32 board to benchmark the model on

1.3 Hydra and MLflow settings

The mlflow and hydra sections must always be present in the YAML configuration file. The hydra section can be used to specify the name of the directory where experiment directories are saved and/or the pattern used to name experiment directories. With the YAML code below, every time you run the Model Zoo, an experiment directory is created that contains all the directories and files created during the run. The names of experiment directories are all unique as they are based on the date and time of the run.

hydra:
  run:
    dir: ./tf/src/experiments_outputs/${now:%Y_%m_%d_%H_%M_%S}

The mlflow section is used to specify the location and name of the directory where MLflow files are saved, as shown below:

mlflow:
  uri: ./tf/src/experiments_outputs/mlruns

If you chose to modify the user_config.yaml you can evaluate the model by running the following command from the UC folder:

python stm32ai_main.py

If you chose to update the benchmarking_config.yaml and use it then run the following command from the UC folder:

python stm32ai_main.py --config-path ./config_file_examples/ --config-name benchmarking_config.yaml

Note that you can provide YAML attributes as arguments in the command, as shown below:

python stm32ai_main.py operation_mode='benchmarking'

To view the detailed benchmarking results, you can access the log file stm32ai_main.log located in the directory experiments_outputs/<date-and-time>. Additionally, you can navigate to the experiments_outputs directory and use the MLflow Webapp to view the metrics saved for each trial or launch. To access the MLflow Webapp, run the following command:

mlflow ui

This will open a browser window where you can view the metrics and results of your benchmarking trials.