Evaluation of Semantic Segmentation Model

To use this service and achieve your goals, you can use the user_config.yaml or directly update the evaluation_config.yaml file and use it. This file provides an example of how to configure the evaluation service to meet your specific needs.

Alternatively, you can follow the tutorial below, which shows how to evaluate your pre-trained semantic segmentation model using our evaluation service.

1.1 Set the model and the operation mode

As mentioned previously, all the sections of the YAML file must be set in accordance with this evaluation_config.yaml. In particular, operation_mode should be set to evaluation and the evaluation section should be filled as in the following example:

model:
   model_type: deeplab
   model_path: <path-to-a-Keras-or-TFlite-model-or-ONNX-file> 
operation_mode: evaluation

The path to the model and its type are provided in the model section.

evaluation:
   target: host # host, stedgeai_host, stedgeai_n6

In the 'evaluation' section, if users are using a quantized TFLITE or ONNX model, they can decide to do the inferences with the classic python interpreters (host -> by default), with the C code generated by stedgeai on the PC (stedgeai_host), or with the C code generated by stedgeai on the N6 board directly (stedgeai_n6) using the target attribute.

1.2 Prepare the dataset

Information about the dataset you want to use for evaluation is provided in the dataset section of the configuration file, as shown in the YAML code below.

dataset:
   dataset_name: person_coco_2017_pascal_voc_2012  
   class_names: ["background", "person"]
   test_path: ./datasets/person_COCO2017_VOC2012/JPEGImages
   test_masks_path: ./datasets/person_COCO2017_VOC2012/SegmentationClassAug
   test_files_path: ./datasets/person_COCO2017_VOC2012/ImageSets/Segmentation/val.txt
   check_image_files: False   # Enable/disable image file checking.

The state machine below describes the rules to follow when handling dataset paths for the evaluation.



When working with a dataset for the first time, we suggest setting the check_image_files attribute to True. This will enable the system to load each image file and identify any corrupt, unsupported, or non-image files. The path to any problematic files will be reported.

The class_names attribute specifies the classes in the dataset. This information must be provided in the YAML file. If the class_names attribute is absent, the classes_name_file argument can be used as an alternative, pointing to a text file containing the class names.

The service first seeks a test set specification defined by test_path, test_masks_path, and test_files_path parameters. If no test set is specified, the data loader checks if a validation set is specified either by the parameters validation_path, validation_masks_path, validation_files_path or alternatively by training_path, training_masks_path, validation_files_path. In the latter, it has to be verified that no images in the validation_files_path list of names belong to the set of images used for training, otherwise, there will be an over-estimation of accuracy and IoU.

To finish, in case there is no validation set or test set properly specified, the available data under the training_path directory is randomly split into two to create a training set and a validation set. By default, 80% of the data is used for training and the remaining 20% is used for the validation set in the evaluation service. If you want to use a different split ratio, you need to specify the percentage to be used for the validation set in the validation_split parameter (to ensure consistency in the training and evaluation process, you must specify the same validation_split parameter value in both the training and evaluation services), as shown in the YAML example below:

dataset:
   dataset_name: person_coco_2017_pascal_voc_2012
   class_names: ["background","person"]
   training_path: ./datasets/person_COCO2017_VOC2012/JPEGImages
   training_masks_path: ./datasets/person_COCO2017_VOC2012/SegmentationClassAug
   training_files_path: ./datasets/person_COCO2017_VOC2012/ImageSets/Segmentation/trainaug.txt  
   validation_split: 0.20
   test_path:

1.3 Apply preprocessing

The images from the dataset need to be preprocessed before they are presented to the network for evaluation. This includes rescaling and resizing. In particular, they need to be rescaled exactly as they were at the training step. This is illustrated in the YAML code below:

preprocessing:
   rescaling: { scale: 1/127.5, offset: -1 }
   resizing: 
     aspect_ratio: "fit"
     interpolation: bilinear
   color_mode: rgb

In this example, the pixels of the input images read in the dataset are in the interval [0, 255], that is UINT8. If you set scale to 1./255 and offset to 0, they will be rescaled to the interval [0.0, 1.0]. If you set scale to 1/127.5 and offset to -1, they will be rescaled to the interval [-1.0, 1.0].

The resizing attribute specifies the image resizing methods you want to use:

• The value of interpolation must be one of {"bilinear", "nearest", "bicubic", "area", "lanczos3", "lanczos5", "gaussian", "mitchellcubic"}.
• The value of aspect_ratio must be either "fit" or "crop". If you set it to "fit", the resized images will be distorted if their original aspect ratio is not the same as in the resizing size. If you set it to "crop", images will be cropped as necessary to preserve the aspect ratio.

The color_mode attribute must be one of "grayscale", "rgb" or "rgba".

1.4 Hydra and MLflow settings

The mlflow and hydra sections must always be present in the YAML configuration file. The hydra section can be used to specify the name of the directory where experiment directories are saved and/or the pattern used to name experiment directories. With the YAML code below, every time you run the Model Zoo, an experiment directory is created that contains all the directories and files created during the run. The names of experiment directories are all unique as they are based on the date and time of the run.

hydra:
   run:
      dir: ./tf/src/experiments_outputs/${now:%Y_%m_%d_%H_%M_%S}

The mlflow section is used to specify the location and name of the directory where MLflow files are saved, as shown below:

mlflow:
   uri: ./tf/src/experiments_outputs/mlruns

If you chose to modify the user_config.yaml you can evaluate the model by running the following command from the UC folder:

python stm32ai_main.py 

If you chose to update the evaluation_config.yaml and use it then run the following command from the UC folder:

python stm32ai_main.py --config-path ./config_file_examples/ --config-name evaluation_config.yaml

In case you want to evaluate the accuracy of the quantized model then benchmark it, you can either launch the evaluation operation mode followed by the benchmark service that describes in detail how to proceed or you can use chained services like launching chain_eqeb example with the command below:

python stm32ai_main.py --config-path ./config_file_examples/ --config-name chain_eqeb_config.yaml

There are no specific plots available for semantic segmentation evaluation. The service only reports the accuracy and the average IoU. Both metrics include the contribution of the "background" class.

The accuracy estimates how well pixels are classified by comparing network output with ground truth mask. It is the ratio of well classified pixels (whatever the class) over the number of pixels within the image. This calculation is done on each image and then averaged over the full evaluation set.

IoU (Intersection over Union) is a popular metric for segmentation that estimates how much network prediction and ground truth mask overlap. In accordance with its name, it is the ratio between prediction and ground truth intersection (in pixels) over prediction and ground truth union (in pixels). The IoU we report is already averaged on all classes and all images of the evaluation set.

Both accuracy and IoU will be displayed on your screen and then saved in the log file stm32ai_main.log under tf/src/experiments_outputs/.