LightSeq Examples

Table of Contents

• Cpp Examples
• Python Examples
  • Train the models
  • Export and infer the models
• Deploy using Tritonbackend

Cpp Examples

We provide multiple cpp examples of LightSeq inference.

First you should use the training examples in the following to train a model, and then export it to protobuf or HDF5 format.

Then use the cpp examples to infer the models:

1. Uncomment the add_subdirectory(examples/inference/cpp) in the CMakeLists.txt.
2. Build the LightSeq. Refer to build.md for more details.
3. Switch to build/temp.linux-xxx/examples/inference/cpp, and then run sudo make to compile the cpp example.
4. Run the cpp examples by ./xxx_example MODEL_PATH.

Python Examples

We provide a series of Python examples to show how to use LightSeq to do model training and inference.

Train the models

Currently, LightSeq supports training from Fairseq, Hugging Face, DeepSpeed and from scratch. For more training details, please refer to the respective README.

Export and infer the models

First export the models training by Fairseq, Hugging Face or LightSeq to protobuf or HDF5 format. Then test the results and speeds using the testing scripts.

Refer to here for more details.

Deploy using Tritonbackend

Refer to here for more details.