Extracting Fine-Tuned LoRA Models from DFWBP and Deploying with NIM

August 19, 2025 · View on GitHub

Overview

This guide explains how to extract fine-tuned LoRA (Low-Rank Adaptation) models from the Data Flywheel Blueprint (DFWBP) datastore and deploy them with standalone NVIDIA NIM for inference outside the DFWBP environment.

After completing fine-tuning with DFWBP, you can extract the resulting LoRA adapters using the HuggingFace CLI and deploy them in production environments, edge deployments, or integrate them with other systems.

Prerequisites

Before extracting LoRA models, ensure you have:

  • ✅ Completed LoRA Fine-tuning: Trained a LoRA model using DFWBP
  • ✅ HuggingFace CLI: Installed huggingface-cli tool
  • ✅ Access Credentials: Valid HF_TOKEN with datastore access
  • ✅ Datastore Endpoint: External URL for your DFWBP datastore

1. Install Required Tools

Install the HuggingFace CLI if not already available:

pip install huggingface_hub[cli]

Verify installation:

huggingface-cli --help

2. Configure Environment

Set up your environment variables for datastore access:

# Set your HuggingFace token (same token used by DFWBP)
export HF_TOKEN="your_huggingface_token"

# Set the external datastore endpoint 
# Replace with your actual DFWBP datastore URL
export HF_ENDPOINT="https://your-datastore-domain.com/v1/hf"

Note: The HF_ENDPOINT variable is deployment-specific and should point to your DFWBP datastore's external URL. Contact your DFWBP administrator for the correct endpoint URL for your deployment.

3. Identify Your Fine-Tuned Model

DFWBP stores fine-tuned models using a predictable naming convention. However, you should always get the exact model name and revision from the job details API.

Model Name Pattern: customized-{original-model-name}

Examples:

  • Original: meta/llama-3.2-1b-instruct → Fine-tuned: customized-meta-llama-3.2-1b-instruct
  • Original: meta/llama-3.1-8b-instruct → Fine-tuned: customized-meta-llama-3.1-8b-instruct

Full Model Identifier Format: The actual model identifier in DFWBP includes the namespace and revision:

dfwbp/customized-meta-llama-3.2-1b-instruct@cust-DbcC6k3UH3iDhfnhats9ZP

Where:

  • dfwbp is the namespace
  • customized-meta-llama-3.2-1b-instruct is the model name
  • cust-DbcC6k3UH3iDhfnhats9ZP is the auto-generated revision

Namespace: DFWBP stores models in the dfwbp namespace by default (configurable via nmp_namespace in DFWBP configuration). The namespace is included in the customized_model field returned by the job details API.

Source: src/tasks/tasks.py:790

output_model_name = f"customized-{target_llm_model}".replace("/", "-")

4. Get the Model Revision from Job Details

Before downloading, you need to get the exact model revision from your DFWBP job. The revision is generated automatically during fine-tuning and is required for downloading.

Get Job Details

Call the job details API to retrieve your customized model information:

# Replace {job_id} with your actual DFWBP job ID
JOB_ID="your_job_id_here"
curl -X GET "http://your-dfwbp-api-url/jobs/$JOB_ID"

Extract Namespace, Model Name and Revision

Look for the customized_model field in the response. It will have this format:

dfwbp/customized-meta-llama-3.2-1b-instruct@cust-DbcC6k3UH3iDhfnhats9ZP

Extract the namespace, model name, and revision:

# Example response parsing (using jq)
CUSTOMIZED_MODEL=$(curl -s "http://your-dfwbp-api-url/jobs/$JOB_ID" | jq -r '.nims[0].customizations[0].customized_model')

# Split at @ symbol to get model part and revision
MODEL_PART=$(echo "$CUSTOMIZED_MODEL" | cut -d'@' -f1)
REVISION=$(echo "$CUSTOMIZED_MODEL" | cut -d'@' -f2)

# Split model part at / to get namespace and model name
NAMESPACE=$(echo "$MODEL_PART" | cut -d'/' -f1)
MODEL_NAME=$(echo "$MODEL_PART" | cut -d'/' -f2)

echo "Full Model: $CUSTOMIZED_MODEL"
echo "Namespace: $NAMESPACE"
echo "Model Name: $MODEL_NAME"
echo "Revision: $REVISION"

5. Download the LoRA Model

Use the extracted model information to download your fine-tuned model:

# Set local directory
LOCAL_DIR="./extracted_models/$MODEL_NAME"

# Create local directory
mkdir -p "$LOCAL_DIR"

# Download the model using the extracted namespace and revision
huggingface-cli download \
  --force-download \
  --repo-type model \
  $NAMESPACE/$MODEL_NAME \
  --revision $REVISION \
  --local-dir "$LOCAL_DIR"

6. Verify Downloaded Files

Check that the LoRA adapter files downloaded as expected:

ls -la "$LOCAL_DIR"

You should see files like:

  • adapter_config.json - LoRA adapter configuration
  • adapter_model.safetensors - LoRA adapter weights
  • Other model artifacts and metadata

Note: The specific file structure follows standard HuggingFace LoRA adapter formats as generated by the underlying NeMo training process.

7. Deploy with Standalone NIM

Important: The following NIM deployment examples are based on standard NVIDIA NIM patterns. You may need to adjust environment variables, mount paths, or configuration based on your specific NIM version and deployment requirements.

Option A: Using NVIDIA NIM Container

Deploy the LoRA model with a standalone NIM container:

# Pull the appropriate NIM container for your base model
docker pull nvcr.io/nim/meta/llama-3.2-1b-instruct:latest

# Run NIM with your LoRA adapter
docker run -d \
  --name nim-with-lora \
  --gpus all \
  -p 8000:8000 \
  -v "$LOCAL_DIR:/opt/nim/loras/custom-adapter" \
  nvcr.io/nim/meta/llama-3.2-1b-instruct:latest

Option B: Using Kubernetes Deployment

Create a Kubernetes deployment with your LoRA model:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nim-lora-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nim-lora
  template:
    metadata:
      labels:
        app: nim-lora
    spec:
      containers:
      - name: nim
        image: nvcr.io/nim/meta/llama-3.2-1b-instruct:latest
        ports:
        - containerPort: 8000
        volumeMounts:
        - name: lora-volume
          mountPath: /opt/nim/loras/custom-adapter
      volumes:
      - name: lora-volume
        persistentVolumeClaim:
          claimName: lora-model-pvc

8. Test Your Deployment

Verify that your NIM deployment is working with the LoRA adapter:

# Test inference with the customized model
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.2-1b-instruct",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "max_tokens": 100
  }'

Troubleshooting

Common Issues

  1. Authentication Errors

    • Verify your HF_TOKEN is correct and has proper permissions
    • Ensure the token matches the one used in your DFWBP deployment

  2. Model Not Found

    • Ensure you're using the exact customized_model value from the job details API response
    • Verify the model training completed successfully in DFWBP
    • Check that you've correctly parsed the namespace, model name, and revision from the API response

  3. Download Failures

    • Verify the datastore endpoint URL is correct and accessible
    • Check network connectivity to the DFWBP datastore
    • Ensure the model revision is correct (get it from the job details API)

  4. NIM Deployment Issues

    • Verify the base model in NIM matches the original model used for fine-tuning
    • Check that LoRA adapter files are properly mounted
    • Ensure adequate GPU resources are available
    • Consult NVIDIA NIM documentation for model-specific deployment requirements

Example: Complete Extraction Workflow

Here's a complete example for extracting a fine-tuned Llama 3.2 1B model:

# 1. Set environment variables
export HF_TOKEN="hf_your_token_here"
export HF_ENDPOINT="https://datastore.int.aire.nvidia.com/v1/hf"

# 2. Get job details and extract model information
JOB_ID="your_job_id_here"
CUSTOMIZED_MODEL=$(curl -s "http://your-dfwbp-api-url/jobs/$JOB_ID" | jq -r '.nims[0].customizations[0].customized_model')

# Extract namespace, model name and revision
MODEL_PART=$(echo "$CUSTOMIZED_MODEL" | cut -d'@' -f1)
REVISION=$(echo "$CUSTOMIZED_MODEL" | cut -d'@' -f2)
NAMESPACE=$(echo "$MODEL_PART" | cut -d'/' -f1)
MODEL_NAME=$(echo "$MODEL_PART" | cut -d'/' -f2)

echo "Full Model: $CUSTOMIZED_MODEL"
echo "Namespace: $NAMESPACE"
echo "Model Name: $MODEL_NAME"
echo "Revision: $REVISION"

# 3. Download the customized model with extracted information
huggingface-cli download \
  --force-download \
  --repo-type model \
  $NAMESPACE/$MODEL_NAME \
  --revision $REVISION \
  --local-dir ./extracted_models/$MODEL_NAME

# 4. Verify download
ls -la ./extracted_models/$MODEL_NAME/

# 5. Deploy with NIM (example)
docker run -d \
  --name nim-custom-llama \
  --gpus all \
  -p 8000:8000 \
  -v "./extracted_models/$MODEL_NAME:/opt/nim/loras/custom-adapter" \
  nvcr.io/nim/meta/llama-3.2-1b-instruct:latest

# 6. Test deployment
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta/llama-3.2-1b-instruct", "messages": [{"role": "user", "content": "Test message"}], "max_tokens": 50}'