Exporting Models for Use with MMS

A key feature of MMS is the ability to export all model artifacts into a single model archive file. It is a separate command line interface (CLI) that can take model checkpoints and package them into a .model file that can then be redistributed and served by anyone using MMS. It takes in the following model artifacts: network definitions in the form of a JSON file, the trained network weight in the form of a parameters file, the description of the models' inputs and outputs in the form of a signature file, a service file describing how to handle inputs and outputs, and other optional assets that may be required to serve the model. The CLI creates a .model file that MMS's server CLI uses to serve the models.

Technical Details of a Model Archive

To export a model in MMS, you will need:

1. A model-symbol.json file, which describes the neural network,

2. A much larger model-0000.params file containing the parameters and their weights

   • For the purpose of a quick example, we'll pretend that you've already saved a checkpoint which is numbered by those digits in the middle of model-0000.params filename.

3. For MMS to understand your model, you must provide a signature.json file, which describes the model's inputs and outputs.

4. Most models will require the inputs to go through some pre-processing, and your application will likely benefit from post-processing of the inference results. These functions go into custom-service.py.

5. You also have the option of providing assets that assist with the inference process. These can be labels for the inference outputs, key/value vocabulary pairs used in an LSTM model, and so forth.

This gives you the first two assets by providing those files for you to download, or that you've acquired the trained models from a model zoo. We'll also provide the latter two files that you would create on your own based on the model you're trying to serve. Don't worry if that sounds ominous; creating those last two files is easy. More details on this can be found in later the Required Assets section.

The files in the model-example.model file are human-readable in a text editor, with the exception of the .params file: this file is binary, and is usually quite large.

Example Model File Exploration

In the quick start export example on the main README, we provide a zip file of all of the artifacts you need to run your first export. However, when you start using MMS and downloading and sharing models you will be using .model files, so here we'll start with one of those.

Download and extract a model file:

• SqueezeNet v1.1 - 5 MB
• Or choose a model file from the model zoo

Note: A .model file is a zip file under the hood, so if you have trouble extracting it, change the extension to .zip first and then extract it. It might be worth assigning your favorite unzip program to the .model filetype. You also might be able to just use unzip from the terminal:

unzip squeezenet_v1.model

Once the model archive has been extracted you can review the following files:

• Manifest (json file) - a description of the files in the archive that is generated by mxnet-model-export.

  • Example: MANIFEST.json - it describes each of the artifacts, the deep learning engine to use, and metadata about versions, and more.

• Model Definition (json file) - contains the description of the layers and overall structure of the neural network.

  • Example: squeezenet_v1.1-symbol.json - the name, or prefix, here is "squeezenet_v1.1".

• Model Parameters and Weights (binary params file) - contains the parameters and the weights.

  • Example: squeezenet_v1.1-0000.params - again, the prefix is "squeezenet_v1.1".

• Model Signature (json file) - defines the inputs and outputs that MMS is expecting to hand-off to the API.

  • Example: signature.json - in this case for squeezenet_v1, it expects images of 224x224 pixels and will output a tensor of 1,000 probabilities.

• Custom Service (py file) - customizes the inference request handling for both pre-processing and post-processing.

  • Example: custom-service.py - in this case, it is a copy of the mxnet_vision_service.py which does standard image pre-processing to match the input required and limits the output results to the top 5 instead of the full 1,000.

• assets (text files) - auxiliary files that support model inference such as vocabularies, labels, etc. Will vary depending on the model.

  • Example: synset.txt - an optional list of labels (one per line) specific to a image recognition model, in this case based on the ImageNet dataset.
  • Example: vocab_dict.txt - an optional list of word/index pairs specific to an LSTM model, in this case based on the PenTreeBank dataset.

MMS Export Command Line Interface

Now let's cover the details on using the CLI tool: mxnet-model-export.

Example usage with the squeezenet_v1.1 model you may have downloaded or exported in the main README's examples:

mxnet-model-export --model-name squeezenet_v1.1 --model-path models/squeezenet_v1.1

Arguments

$ mxnet-model-export -h
usage: mxnet-model-export [-h] --model-name MODEL_NAME --model-path MODEL_PATH
                          [--service-file-path SERVICE_FILE_PATH]

MXNet Model Export

optional arguments:
  -h, --help            show this help message and exit
  --model-name MODEL_NAME
                        Exported model name. Exported file will be named as
                        model-name.model and saved in current working
                        directory.
  --model-path MODEL_PATH
                        Path to the folder containing model related files.
                        Signature file is required.
  --service-file-path SERVICE_FILE_PATH
                        Service file path to handle custom MMS inference
                        logic. if not provided, this tool will package
                        MXNetBaseService if input in signature.json is
                        application/json or MXNetVisionService if input is
                        image/jpeg

Required Arguments

1. model-name: required, prefix of exported model archive file.
2. model-path: required, directory which contains files to be packed into exported archive.

Export Example

Given the files downloaded in the exploration section above, you can use the mxnet-model-export CLI to generate a .model file that can be used with MMS.

To try this out, open your terminal and go to the folder you just extracted. You may have already taken a look at the MANIFEST.json file, but it isn't necessary at this step, so you may delete it. A new one will be injected into the model file when you run the export.

The export tool is going to look for the following at a minimum:

• symbol file (name-symbol.json) - in our example it will be: squeezenet_v1.1-symbol.json
• params file (name-checkpoint#.params) - in our example it will be: squeezenet_v1.1-0000.params
• signature file (signature.json)
• labels file (synset.txt)

Now, we could export just with these artifacts an would get a .model file. It's going to look at the signature file and see that you're using "input_type": "image/jpeg", and assume that you want the default vision service, so it will include mxnet_vision_service.py for you. It will also generate the manifest. Let's try it out.

We're going to tell it our model's name is squeezenet_v1.1 with the --model-name argument. The name works like a prefix, so it will assume that you've named the symbol file and the params file according to the pattern name-symbol.json and name-0000.params. The "0000" can be another checkpoint if that's what you have. Then we're giving it the --model-path to the model's assets, which are in the current working directory, so we'll use . for the path.

mxnet-model-export --model-name squeezenet_v1.1 --model-path .

This will output squeezenet_v1.1.model in the current working directory. Try serving it with:

mxnet-model-server --models squeezenet=squeezenet_v1.1.model

Export Example with Specified Custom Service

Let's try the export again, but this time we will also pass the --service argument. So that it is very clear what is happening with the export process, we'll copy the service file to a different name and specify it when we call the export tool.

cp mxnet_vision_service.py my_awesome_service.py
mxnet-model-export --model-name squeezenet_v1.1 --model-path . --service my_awesome_service.py

Once the model file is exported, unzip it and take a look. Your custom service should be in there. The other service file is in there too! This means you can pack up an entire application, and specify the custom service entry point with the --service argument. You can also verify the manifest.json and see that it setup the model based on the provided files and your service specification. Here's an example extract from the manifest:

"Model": {
    "Description": "squeezenet",
    "Service": my_awesome_service.py",
    "Symbol": "squeezenet_v1.1-symbol.json",
    "Parameters": "squeezenet_v1.1-0000.params",
    "Signature": "signature.json",
    "Model-Name": "squeezenet_v1.1",
    "Model-Format": "MXNet-Symbolic"

Exporting with Labels and Other Assets

You may have noticed that we required the synset.txt file in this example, but didn't mention it as part of the process. It's not even in the manifest. This is because it is required by the service file or the upstream classes that the service file is extending. In our example here it uses those labels in the post-processing step to provide human-readable inference results.

If you're curious you can look in the service file and note the line that "sort of" mentions it by require a labels file, and then if you look upstream at the mxnet_model_service.py you will see it specifically mentioned:

archive_synset = os.path.join(model_dir, 'synset.txt')

Of course, if you write your own custom service, you can handle labels in another way. You may want to take look at the examples too. One is for an LSTM which uses a vocabulary labels file, or the SSD example which uses a much shorter synset.txt than our SqueezeNet examples for the short list of objects it is intended to identify.

Note: You may get an error about a missing synset if you use a custom service that is `expecting one and you didn't provide one in the folder with the other artifacts. Each sysnet correlates to the model, so make sure you have one in the directory with your other artifacts when you try to export your model.

Export Example with Customizations

To give you an idea of how you might download another's model, modify it, then serve it, let's try out a simple use case. The example we have been using will serve the SqueezeNet model, and upon inference requests it will return the top 5 results. Let's change the custom service so that it returns 10 results instead.

Open the my_awesome_service.py file in your text editor.

Find the function for _postprocess and the line that says the following:

return [ndarray.top_probability(d, self.labels, top=5) for d in data]

Change the top=5 to top=10, then save the file.

Run the export process again:

mxnet-model-export --model-name squeezenet_v1.1 --model-path . --service my_awesome_service.py

Run the server on the updated model:

mxnet-model-server --models squeezenet=squeezenet_v1.1.model

Then in a different terminal window, upload an image file the API, and see your results.

curl -X POST http://127.0.0.1:8080/squeezenet/predict -F "data=@kitten.jpg"

Instead of the top 5 results, you will now get the top 10!

This is just one example of customization. There are many variations, but here are a couple of ideas to get your creative juices flowing:

• You might decide that you want to take a model, grab the params as a checkpoint and retrain it using additional training images. This is often called fine tuning a model. A fine tuning tutorial using MXNet with Gluon can be found in The Straight Dope's computer vision section.

• You also might decide that you want to change the labels - maybe by simplifying the results without having to retrain the entire model. A result like "class": "n02123045 tabby, tabby cat", could just be cat. You would go through the synset.txt file, make your edits and then export the model.

Artifact Details

Model Archive Overview

Model archives have the following artifacts:

<Model Name>-symbol.json
<Model Name>-<Epoch>.params
signature.json
<Service File>.py
MANIFEST.json
{asset-x.txt, asset-y.txt, ...}

Model Definition

<Model Name>-symbol.json

This is the model's definition in JSON format. You can name it whatever you want, using a consistent prefix. The pattern expected is my-awesome-network-symbol.json or mnist-symbol.json so that when you use mxnet-model-export you're passing in the prefix and it'll look for prefix-symbol.json. You can generate this file in a variety of ways, but the easiest for MXNet is to use the .export feature or the mms.export_model method described later.

Model Parameters and Weights

<Model Name>-<Epoch>.params

This is the model's hyper-parameters and weights. It will be created when you use MXNet's .export feature or the mms.export_model method described later.

Signature

signature.json

1. input: Contains MXNet model input names and input shapes. It is a list contains { data_name : name, data_shape : shape } maps. Client side inputs should have the same order with the input order defined here.
2. input_type: Defines the MIME content type for client side inputs. Currently all inputs must have the same content type. Only two MIME types are currently supported: "image/jpeg" and "application/json".
3. output: Similar to input, it contains MXNet model output names and output shapes.
4. output_type: Similar to input_type. Currently all outputs must have the same content type. Only two MIME types are currently supported: "image/jpeg" and "application/json".

Using the squeezenet_v1.1 example, you can view the signature.json file in the folder that was extracted once you dowloaded and served the model for the first time. The input is an image with 3 color channels and size 224 by 224. The output is named 'softmax' with length 1000 (one for every class that the model can recognize).

{
 "inputs": [
   {
     "data_name": "data",
     "data_shape": [0, 3, 224, 224]
   }
 ],
 "input_type": "image/jpeg",
 "outputs": [
   {
     "data_name": "softmax",
     "data_shape": [0, 1000]
   }
 ],
 "output_type": "application/json"
}

The data_shape is a list of integers. It should contain batch size as the first dimension as in NCHW. Also, 0 is a placeholder for MXNet shape and means any value is valid. Batch size should be set as 0.

Note that the signature output isn't necessarily the final output that is returned to the user. This is directed by your model service class, which defaults to the MXNet Model Service. In this signature.json example your output_type is json and a shape of 1000 results, but API's response is actually limited to the top 5 results via the vision service. In the object detection example, it is using a signature.json that has "data_shape": [1, 6132, 6] and has a custom service to modify the output to the API in such a way as to identify the objects AND their locations, e.g. [(person, 555, 175, 581, 242), (dog, 306, 446, 468, 530)].

Service

<Service File>.py

This is a stubbed out version of the class extension you would use to override the SingleNodeService as seen in mxnet_model_service.py. You may instead want to override the MXNetBaseService as seen in mxnet_vision_service.py

class MXNetBaseService(SingleNodeService):
  def __init__(self, path, synset=None, ctx=mx.cpu()):

  def _inference(self, data):

  def _preprocess(self, data):

  def _postprocess(self, data, method='predict'):

Further details and specifications are found on the custom service page.

Labels (synset.txt)

synset.txt

This optional text file is for classification labels. Simply put, if it were for MNIST, it would be 0 through 9 where each number is on its own line. For a more complex example take a look at the synset for Imagenet-11k.

If synset.txt is included in exported archive file and each line represents a category, MXNetBaseModel will load this file and create labels attribute automatically. If this file is named differently or has a different format, you need to override __init__ method and manually load it.

Using Your Own Trained Models and Checkpoints

While all of these features are super exciting you've probably been asking yourself, so how do I create these fabulous MMS model files for my own trained models? We'll provide some MXNet code examples for just this task.

There are two main routes for this: 1) export a checkpoint or use the new .export function, or 2) using a MMS Python class to export your model directly.

The Python method to export model is to use export_serving function while completing training:

   import mxnet as mx
   from mms.export_model import export_serving

   mod = mx.mod.Module(...)
   # Training process
   ...

   # Export model
   signature = { "input_type": "image/jpeg", "output_type": "application/json" }
   export_serving(mod, 'resnet-18', signature, aux_files=['synset.txt'])

In MXNet with version higher than 0.12.0, you can export a Gluon model directly, as long as your model is Hybrid:

from mxnet import gluon
net = gluon.nn.HybridSequential() # this mode will allow you to export the model

with net.name_scope():
    net.add(gluon.nn.Dense(128, activation="relu")) # an example first layer
    # Add the rest of your network architecture here

net.hybridize() # hybridize your network so that it can be exported

# Then train your network before moving on to exporting

signature = {
                "input_type": "application/json",
                "inputs" : [
                    {
                        "data_name": "data",
                        "data_shape": [1, 100]
                    }
                ],
                "outputs" : [
                    {
                        "data_name": "softmax",
                        "data_shape": [1, 128]
                    }
                ],
                "output_type": "application/json"
            }

export_serving(net, 'gluon_model', signature, aux_files=['synset.txt'])

Note: be careful with versions. If you export a v0.12 model and try to run it with MMS running v0.11 of MXNet, the server will probably throw errors and you won't be able to use the model.