Vision Transformers apply the transformer architecture (Consisting of Multi Headed Attention (MHA) and Feed Forward Network (FFN) blocks) to a wide variety of vision tasks such as classification, object detection, semantic segmentation, etc. This document explains how vision transformers are currently supported with TIDL

Support for Vision Transformer Operators

The below table covers the current status of transformer operators:

Limitations

• TIDL currently supports vision transformers via ONNX models only
• TIDL has validated vision transformers from timm exported to ONNX in the current release
• Support for the following models has been added,
  • DEIT, SWIN, DETR, Segformer (can be found in TI's ModelZoo)
• ONNX-RT Optimization Level must be set to ORT_DISABLE_ALL while compiling models offloaded to C7x for vision transformers

TIDL Layer Mapping of Transformer Operators

Layernorm

• The following sequence of ONNX operators are converted to a layernorm layer in TIDL
• The γ (Multiplication factor) and β (Addition Factor) are expressed outside TIDL's layernorm block as eltwise layers

GELU

• The following sequence of ONNX operators which represent the GELU activation are mapped to TIDL's Batchnorm layer
• Individual operator Erf is not supported in isolation and is only supported if it is a part of GELU pattern
• GELU can be identified by Batchnorm's activation parameters

Patch Merging

• Patch merging is expressed as 8x strided slices and a concat layer.
• Patch merging is only supported when channels (i.e. depth) is in the lowest dimension

DeiT Transformer Example

• DeiT model can be generated from timm using the following steps,
• pip install timm onnx onnxsim
• import timm
• import torch
• deit = timm.create_model('deit_tiny_patch16_224', pretrained=True)
• deit.eval()
• x = torch.randn(1, 3, 224, 224)
• deit(x).shape
• torch.onnx.export(deit,x, "deit_tiny.onnx",export_params=True,opset_version=14,do_constant_folding=True,input_names=['input'],output_names=['output'])
• !onnxsim deit_tiny.onnx deit_tiny_1.onnx
• The above commands can be run in a Google colab notebook, they have been validated using the same.
• Sample model config for the model has been added to model_configs.py, place the model in 'model_base_path' and you can run the model using the standard steps.