apis.md
January 4, 2023 · View on GitHub
API of federated learning
We provide APIs to access job scheduler and model manager components to manage federated jobs and models, mainly are some CRUD operations. All the interfaces comply with the HTTP protocol.
Table of Contents
Job Management
APIs used to manage federated jobs, the objects definition as follows:
Objects
Job
| name | type | property | description |
|---|---|---|---|
| id | string | required | The id of federated job, is unique in namespace |
| description | string | optional | The description of federated job |
| create_time | string | - | Create time of this federated job (generated by server) |
| model_path | string | conditional | The init model path in the storage Note: model_path、model_id、checkpoint_id are mutually exclusive. You only need to fill in one. If all fields are filled in, then the priority will be: checkpoint_id > model_id > model_path |
| model_id | string | conditional | The init model id, you can get the model id through model manager. Note: model_path、model_id、checkpoint_id are mutually exclusive. You only need to fill in one. If all fields are filled in, then the priority will be: checkpoint_id > model_id > model_path |
| checkpoint_id | string | conditional | The checkpoint_id, you can get the checkpoint_id through model manager. Note: model_path、model_id、checkpoint_id are mutually exclusive. You only need to fill in one. If all fields are filled in, then the priority will be: checkpoint_id > model_id > model_path |
| status | Status | - | The current state of federated job, set by the server |
| runtime | string | required | The runtime of model used in the federated job. Currently supporting “pytorch”, “keras” |
| clients | string | optional | The ip:port address of clients to join into the federated training. seperated by ',' , for example : “192.168.0.1:8080,192.168.0.2:8080” |
| resource | Resource | optional | The required client resource to running the federated job |
| port | int | optional | The port of server service, used to report the job state |
| hyper_parameters | HyperParameters | optional | The hyper parameters of federated job. If not configured, will be filled with default training parameters |
| task_entry | string | conditional | The training enterypoint in the client , should correspond to the scripts name already on the client at startup Note: This is mutually exclusive with scripts configuration |
| scripts | ScriptsConfig | conditional | The training scripts to broadcast to all the clients. This configuration is suitable for the scene that clients has no scripts in local. Note: This is mutually exclusive with task_entry configuration |
| ssl | string | optional | The ssl store path, used for secure transfer for grpc |
| parameters | Parameters | optional | The parameters that need to be passed to all training clients, need to be customized by the user |
| extenders | Extenders | optional | The extender interface configuration, through this interface, the custom extended interface scripts and functions can be injected into the federation process |
| secure_algorithm | SecureAlgorithm | optional | Federal job security algorithm configuration. If configured, the job will protect the intermediate data of the job under the corresponding secure computing method. |
| compression | Compression | optional | Federal job compression algorithm configuration. If configured, client will compress updated weights before sending to coordinator. |
| checkpoints | CheckpointsList | optional | The checkpoints generated during the federated job training process, are convenient for users to select the checkpoint for next running corresponding to the checkpoint_id |
| datasets | string | optional | The dataset to use for the job, when there are multiple datasets, seperated by ',', for example: "dataset1, dataset2" |
| output | string | required | The output path of the checkpoint and metrics after job finished |
| random_client | bool | optional | Whether to randomly choose from suitable clients Default is false |
| untrained_first | bool | optional | When selecting clients for the job, whether those not participating in training is preferred Default is false |
| redundancy | float | optional | Redundancy when selecting devices. For example, 1.5 means selecting at most 1.5 times the number of cleints in each round of configuration; the value ranges from 1.0 to 2.0. Default 1.0 means that there is no redundancy |
| optimizer | Optimizer | optional | Optimizer cofiguration, currently for non iid datasets, you can use fedprox, scaffold two optimizers |
The definition of objects involved above as follows:
Status
| name | type | property | description |
|---|---|---|---|
| state | string | - | The status of the federation job, which has the following status: QUEUED，STARTING，RUNNING，FINISHED，FAILED，DELETING |
| reason | string | - | If the federated job is in the FAILED state, this field will give the exception reason |
| progress | int | - | The progress of the federation job, a value from 0 to 100 |
HyperParameters
| name | type | property | description |
|---|---|---|---|
| max_round_num | int | optional | The maximum number of running rounds for federated jobs, default is 10 |
| client_num | int | optional | The number of clients required to run federated jobs, default is 1 |
| threshold_client_num | int | optional | The minimum number of clients that the federated job runs successfully in each round, default is 1 |
| evaluate_interval | int | optional | Federated job evaluation interval, how many rounds to run evaluation once, default is 2 |
| save_interval | int | optional | Federated job model save interval, how many rounds to save the checkpoint, default is 5 |
| round_timeout | int | optional | The timeout for each round of federated jobs waiting for client results, unit is seconds, default is 3600 |
| learning_rate | float | optional | Learning rate for federated jobs, default is 1.0 |
ScriptsConfig
| name | type | property | description |
|---|---|---|---|
| path | string | required | The relative path where the script is located |
| config_file | string | required | The configuration file required for federated jobs to run tasks on the client side |
Resource
| name | type | property | description |
|---|---|---|---|
| worker_num | int | optional | How many workers are needed for parallel training when the federated job runs on the client, the default is 1 |
| cpu | float | optional | The number of CPU resources required for each worker participating in training on the client, the default value is 1.0 |
| gpu | int | optional | The number of GPU resources required for each worker participating in training on the client, the default value is 0 |
| memory | int | optional | The number of memory resources required for each worker participating in the training on the client, unit is MB, the default is 1000MB |
Parameters
This parameter is passed to the user training or evaluation scripts, and will be provided as a command line argument. This paramters is dict with key-value.
For example, if you provide this config as follows, then the commad will be with the form:
python3 srcipt.py –key value1 -k value2 -v
| name | type | property | description |
|---|---|---|---|
| –key | value1 | optional | paramaeter 1 |
| -k | value2 | optional | parameter 2 |
| -v | None | optional | single parameter 3 |
CheckpointsList
| name | type | property | description |
|---|---|---|---|
| id | CheckPoint | required | Details of the checkpoint corresponding to the id |
CheckPoint
| name | type | property | description |
|---|---|---|---|
| accuracy | float | required | The accuracy of this checkpoint after joint evaluation on the federated dataset |
| path | string | required | The relative path of the checkpoint file in the namespace working directory |
Extenders
| name | type | property | description |
|---|---|---|---|
| script_path | string | optional | The path to the script file that implements the extension interface. (If this field is empty, other fields are ignored) |
| broadcast | string | optional | The function name of the broadcast interface in the script. This function should return two parameters , which are custom file and custom parameters, both in the form of dict. And these files and params will be sent to each client |
| aggregate | string | optional | The function name of the aggregate interface in the script. This function's input is the updates uploaded by the client, which can be customized for aggregation calculation |
| finish | string | optional | The function name of the finish interface in the script. This function is executed after the current federated round. Users can customize the final aggregated results, such as saving or converting to other format |
SecureAlgorithm
| name | type | property | algorithm | description |
|---|---|---|---|---|
| type | string | required | -- | Type of security algorithm, currently supported: "DP", "SSA" DP is Differential Privacy SSA is Secure Aggregate |
| noise_multiplier | float | required | DP | Add the Gaussian noise, which is the variance of the Gaussian distribution |
| adding_same_noise | bool | optional | DP | When adding noise to the model, whether to add the same noise to all weights |
| threshold | int | optional | SSA | The minimum threshold for the number of clients participating in secret sharing, the minimum value is 2, the maximum value is threshold_num defined in HyperParameters |
| mode | string | optional | SSA | onemask or doublemask mode in ssa。onemask mode is more suitable for cross-slio scenarios and does not support client disconnection; doublemask is more suitable for cross-device scenarios, supports client disconnection, and is more secure |
Compression
| name | type | property | algorithm | description |
|---|---|---|---|---|
| type | string | required | -- | Type of compression algorithm, currently supported: "quantization", "subsampling", "selective_masking". |
| quantization_bits | int | optional | quantization | A integer specifying the quantization bitwidth |
| sampling_rate | float | optional | subsampling | Specify the sampling ratio, how much data needs to be sampled from the original data, which can be simply understood as the compression ratio |
| top_k_ratio | float | optional | selective_masking | Specify the sampling ratio, how much data will be selected from the original data, the selected data is top K largest absolute difference of the original data. This parameter can be simply understood as the compression ratio |
Optimizer
| name | type | property | description |
|---|---|---|---|
| name | string | required | The name of the optimizer, currently supported: "fedprox", "scaffold" |
| params | dict | optional | Optimizer parameters (see the table of below) |
| name | type | property | algorithm | description |
|---|---|---|---|---|
| mu | float | optional | fedprox | Correction term coefficient, to correct the difference between the local model and the server model, the recommended range is 0~1, 0 means no correction; default is 0.6 |
| learning_rate | float | optional | -- | Local training learning rate |
| batch_size | int | optional | scaffold | The batch size used when training the local model |
| sample_num | int | optional | scaffold | The number of samples used in this round when training the local model |
Loss
| name | type | property | description |
|---|---|---|---|
| name | string | required | The name of the loss, currently supported: "feddc" |
JobList
| name | type | property | description |
|---|---|---|---|
| jobs | list | - | List of federated jobs, each element of list is Job |
HttpResponse
| name | type | property | description |
|---|---|---|---|
| state | string | - | Status: "SUCCESSFUL", "FAILED" |
| reason | string | - | If status is "FAILED", giving the error reason |
Operations
the operations is defined as follows
create job
Description: create a federated job.
HTTP Request:
POST /api/v1/namespaces/{namespace}/jobs
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the belonging namespace of this job |
| body | Job | yes | the details of job configuration |
Response:
| code | body | description |
|---|---|---|
| 201 | HttpResponse | Successfully created federated job |
| 403 | HttpResponse | Failed to create a job, such as the job already exists, the namespace does not exist, the job parameter verification fails, etc. |
| 500 | HttpResponse | Internal server error |
update job
Description: Update a federated job.
HTTP Request:
PUT /api/v1/namespaces/{namespace}/jobs/{job_id}
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the belonging namespace of this job |
| job_id(in path) | string | yes | the unique id of job |
| body | Job | yes | the details of job configuration |
Response:
| code | body | description |
|---|---|---|
| 200 | HttpResponse | Successfully modified federated job |
| 403 | HttpResponse | Modifying the job failed, such as the job does not exist, the namespace does not exist, the job parameter verification fails, etc. |
| 500 | HttpResponse | Internal server error |
delete job
Description: Delete a federated job
HTTP Request:
DELETE /api/v1/namespaces/{namespace}/jobs/{job_id}
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the belonging namespace of this job |
| job_id(in path) | string | yes | the unique id of job |
Response:
| code | body | description |
|---|---|---|
| 200 | HttpResponse | The federated job was deleted successfully, or the job does not exist |
| 500 | HttpResponse | Internal server error |
start job
Description: Start a federated job to running.
HTTP Request:
PUT /api/v1/namespaces/{namespace}/jobs/{job_id}:start
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the belonging namespace of this job |
| job_id(in path) | string | yes | the unique id of job |
Response:
| code | body | description |
|---|---|---|
| 200 | HttpResponse | Federated job started successfully |
| 403 | HttpResponse | Failed to start the job, such as the job does not exist, the namespace does not exist, etc. |
| 500 | HttpResponse | Internal server error |
stop job
Description: Stop a federated job.
HTTP Request:
PUT /api/v1/namespaces/{namespace}/jobs/{job_id}:stop
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the belonging namespace of this job |
| job_id(in path) | string | yes | the unique id of job |
Response:
| code | body | description |
|---|---|---|
| 200 | HttpResponse | Successfully stopped federated job |
| 403 | HttpResponse | Failed to stop the job, such as the job does not exist, the namespace does not exist, etc. |
| 500 | HttpResponse | Internal server error |
get job
Description: Get the detailed info of a job.
HTTP Request:
GET /api/v1/namespaces/{namespace}/jobs/{job_id}
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the belonging namespace of this job |
| job_id(in path) | string | yes | the unique id of job |
Response:
| code | body | description |
|---|---|---|
| 200 | Job | The federated job is successfully obtained and the job information is returned |
| 404 | HttpResponse | Failed to get job, such as job does not exist, namespace does not exist, etc. |
| 500 | HttpResponse | Internal server error |
get jobs
Description: Get all the jobs info of one namespace.
HTTP Request:
GET /api/v1/namespaces/{namespace}/jobs
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the namespace name |
Response:
| code | body | description |
|---|---|---|
| 200 | JobList | Get all the federated jobs successfully, return job information list |
| 500 | HttpResponse | Internal server error |
health check
Description: Health check is used to judge whether the job scheduler componment is working normally.
HTTP Request:
GET /api/v1/health
Parameters :
None
Response:
| code | body | description |
|---|---|---|
| 200 | 'OK' | job scheduler is workinng normally |
Model Management
APIs to access the model manager, which is used to manage the model meteadta in user-space in the federated learning system, including the initial model uploaded by the user, the checkpoint generated during the training process, and the final trained model, etc.
Note: Model management only manages the meta information of the model. The upload and download of the model data need to access API according to your backend data storage scheme. If your backend supports s3 standard object storage, then we strongly recommend that you use the 'cli' tool we provide. The 'cli' tool provide a complete model management solution, including the data downlond and upload.
However, if you use other backend storage solutions, you should wrap a script yourself to provide full model management functionality, especially the data upload and download. At the sametime, access our model managers API to maintain your model data metadata.
objects definition as follows:
Objects
Model
| name | type | property | description |
|---|---|---|---|
| name | string | required | The name of created model |
| version | string | required | The version of created model |
| runtime | string | required | The runtime of created model |
| description | string | optional | The description of created model |
| namespace | string | optional | The namespace of created model Default is 'default 'nampspace |
| progress | int | optional | The current progress of model data. When created model, the model should be uploaded to storege server, and this field means the progress, you can use the update opertion to update the progress. And when the progress is 100, means the model is ready for use. |
| state | string | optional | The state of model, support 'unready', 'ready', 'error', 'deleting'. Note: only ready model can be used for federated job |
| model_path | string | conditional | The path of model in the storage, use this field means the created model is already in storage server, only need to copy to user path. Note: the model_path is mutually exclusive with file_name, file_name means user will upload the model to storage server |
| file_name | string | conditional | The file name of model, if the model still in user local disk, then should upload the model to the storage server. |
ModelDetail
| name | type | property | desrcription |
|---|---|---|---|
| name | string | required | The name of model |
| runtime | string | required | The runtime of model |
| namespace | string | required | The namespace of model |
| id | string | required | The unique id of model |
| state | string | required | The current state of model, support 'unready', 'ready', 'error', 'deleting'. |
| version_info | VersionInfo | required | The version info of this model |
| storage_info | tuple | required | The storage info of this model, the tuple is format with (namespace, path), the path is releative to the namespace directory |
| model_path | string | optional | The origin model path when created |
| file_name | string | optional | The file name of model when created |
| error_msg | string | optional | The error reason if model state is error |
VersionInfo
| name | type | property | description |
|---|---|---|---|
| version | string | required | The version name of the model |
| time | string | required | The create time of this version |
| description | string | optional | The description of this version |
ModelList
| name | type | property | description |
|---|---|---|---|
| models | list | - | The list of model, each element is ModelDetail |
HttpResponse
| name | type | property | description |
|---|---|---|---|
| state | string | - | Status: "SUCCESSFUL", "FAILED" |
| reason | string | - | If status is "FAILED", giving the error reason |
Operations
the operations is defined as follows
create model
Description: Create a model
HTTP Request:
POST /api/v1/namespaces/{namespace}/models
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the belonging namespace of this model |
| body | Model | yes | the configuration of model |
Response:
| code | body | description |
|---|---|---|
| 201 | ModelDetail | Model created successfully |
| 400 | HttpResponse | Failed to create model, such as configuration error, data type is not equal |
| 404 | HttpResponse | Failed to create model, namespace does not exist |
| 500,503 | HttpResponse | Internal server error |
get model
Description: Get a model
Note: A model may have many versions. If version is not specified, then will return all the versions of model, otherwise will return the specified version of model.
HTTP Request:
GET /api/v1/namespaces/{namespace}/models/{model_name}?version=
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the belonging namespace of this model |
| model_name(in path) | string | yes | the name of this model |
| version(in query) | string | no | the version name of this model |
Response:
| code | body | description |
|---|---|---|
| 200 | ModelList | Model obtained successfully |
| 404 | HttpResponse | model or namespace not exist |
| 500,503 | HttpResponse | Internal server error |
get models
Description: Get all models in one namespace
HTTP Request:
GET /api/v1/namespaces/{namespace}/models
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the namespace of models |
Response:
| code | body | description |
|---|---|---|
| 200 | ModelList | Successfully fetched all models in one namesapce |
| 404 | HttpResponse | namespace not exist |
| 500,503 | HttpResponse | Internal server error |
get model by id
Description: Get a model through its model-id
Note: After model created success, each model(version) will be given a unique model id, and you can access the model through this id.
HTTP Request:
GET /api/v1/models?model_id=
Parameters :
| name | type | required | description |
|---|---|---|---|
| model_id(in query) | string | yes | the namespace of models |
Response:
| code | body | description |
|---|---|---|
| 200 | ModelDetail | succcess, return the model detail in body |
| 404 | HttpResponse | namespace not exist |
| 500,503 | HttpResponse | service internal error or unavailable |
delete model
Description: Delete a model
Note: A model may have many versions. If version is not specified, then will delete all the versions of model, otherwise will only detele the specified version of model.
HTTP Request:
DELETE /api/v1/namespaces/{namespace}/models/{model_name}?version=
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the belonging namespace of this model |
| model_name(in path) | string | yes | the name of this model |
| version(in query) | string | no | the specific version name of this model |
Response:
| code | body | description |
|---|---|---|
| 200 | None | delete model succcess |
| 404 | HttpResponse | model not exist |
| 500,503 | HttpResponse | service internal error or unavailable |
delete model by id
Description: Delete a model through its model-id
Note: After model created success, each model(version) will be given a unique model id, and you can access the model through this id.
HTTP Request:
DELETE /api/v1/models?model_id=
Parameters :
| name | type | required | description |
|---|---|---|---|
| model_id(in query) | string | yes | the unique id of model |
Response:
| code | body | description |
|---|---|---|
| 200 | None | delete succcess |
| 404 | HttpResponse | namespace not exist |
| 500,503 | HttpResponse | service internal error or unavailable |
update model
Description: Update a model
Note: A model may have many versions. When you update a model, you must set the model version, otherwise the operation will failed.
HTTP Request:
PUT /api/v1/namespaces/{namespace}/models/{model_name}?version=
Parameters :
| name | type | required | description |
|---|---|---|---|
| namespace(in path) | string | yes | the belonging namespace of this model |
| model_name(in path) | string | yes | the name of this model |
| version(in query) | string | yes | the specific version name of this model |
Response:
| code | body | description |
|---|---|---|
| 200 | ModelDetail | update model succcess, return the model detail |
| 404 | HttpResponse | model not exist |
| 500,503 | HttpResponse | service internal error or unavailable |
update model by id
Description: Update a model through its model-id
Note: After model created success, each model(version) will be given a unique model id, and you can access the model through this id.
HTTP Request:
PUT /api/v1/models?model_id=
Parameters :
| name | type | required | description |
|---|---|---|---|
| model_id(in query) | string | yes | the unique id of model |
Response:
| code | body | description |
|---|---|---|
| 200 | ModelDetail | update succcess |
| 404 | HttpResponse | namespace not exist |
| 500,503 | HttpResponse | service internal error or unavailable |
health check
Description: Health check is used to judge whether the model manager is working normally.
HTTP Request:
GET /api/v1/health
Parameters :
None
Response:
| code | body | description |
|---|---|---|
| 200 | 'OK' | model manager is workinng normally |