• Optimized translation. Automatic model conversion from PyTorch or ONNX for efficient on-device deployment with TensorFlow Lite or Qualcomm AI Engine Direct . The resulting model is optimized for deployment on CPU, GPU, or NPU.

• Qualcomm® AI Hub Models. A collection of 100+ optimized AI models covering vision, speech, audio, and text applications, deployable on devices powered bySnapdragon® and Qualcomm platforms within minutes. Fully open sourced recipes available on GitHub and Hugging Face.

• Real devices.Easily run models on hosted Qualcomm platform devices provisioned within minutes. Run performance profiling and on-device inference with a few lines of code.

Installation

The Qualcomm AI Hub library for optimization, profiling, and validation can be installed via PyPI. We recommend using Miniconda to manage your python versions and environments. To install, run the following command in your terminal. We recommend Python > 3.8 and < 3.10.

pip3 install qai-hub

Sign in to Qualcomm AI Hub with your Qualcomm® ID. After signing in, navigate to [your Qualcomm ID] -> Settings -> API Token. This should provide an API token that you can use to configure your client.

qai-hub configure --api_token API_TOKEN

One configured, you can check that your API token is installed correctly by fetching a list of available devices with the following command

qai-hub list-devices

Run your first PyTorch model on a hosted device

Once you have set up your Qualcomm AI Hub environment, the next step is to optimize, validate, and deploy a PyTorch model on a cloud hosted device. This example requires some extra dependencies which can be installed using the following:

pip3 install "qai-hub[torch]"

Now, you can request an automated performance analysis of the MobileNet v2 network. This example compiles and profiles the model on a real hosted device:

import qai_hub as hub
import torch
from torchvision.models import mobilenet_v2

# Using pre-trained MobileNet
torch_model = mobilenet_v2(pretrained=True)
torch_model.eval()

# Trace model (for on-device deployment)
input_shape = (1, 3, 224, 224)
example_input = torch.rand(input_shape)
torch_model = torch.jit.trace(torch_model, example_input)

# Choose a device
devices = hub.get_devices()
for device_num in range(1, len(devices)+1):
    print(f"{ device_num }. { devices[device_num-1].name }")

device_choice = 0
while device_choice < 1 or device_choice > len(devices):
    try:
        device_choice = int(input("Choose a device: "))
    except: pass
device = devices[device_choice-1]

# Optimize model for the chosen device
compile_job = hub.submit_compile_job(
    model=torch_model,
    name="MyMobileNetModel",
    device=device,
    input_specs=dict(image=input_shape),
)

# Run the model on a hosted device
profile_job = hub.submit_profile_job(
  model=compile_job.get_target_model(),
  device=device,
)

This will submit a compilation job and then a profiling job, printing the URLs for both jobs. See the documentation for more details.

Easily select from Qualcomm AI Hub Models

Qualcomm AI Hub Models are a collection of state-of-the-art machine learning models optimized for performance and ready to deploy on Qualcomm platform devices. You can explore models optimized for on-device deployment for vision, speech, text, and generative AI applications. All models come with open-source recipes on GitHub and Hugging Face.

Here is a simple example of real-time selfie segmentation that can be easily exported and deployed on-device. First, we install the Qualcomm AI Hub Models python package.

pip3 install "qai-hub-models[ffnet_40s]"

Run a local PyTorch based CLI demo to validate the model off-device with some sample input.

python -m qai_hub_models.models.ffnet_40s.demo

Once verified off-device, you can then run this model on a hosted Qualcomm platform device within minutes.

python -m qai_hub_models.models.ffnet_40s.export

The above script optimizes the model for on-device execution, profiles the model on a cloud hosted Qualcomm platform devices, runs the model on-device, and compares accuracy between a local CPU based PyTorch run and the on-device run. Finally, you can run a full demo with the inference running on-device.

python -m qai_hub_models.models.ffnet_40s.demo --on-device

What else can you do with Qualcomm AI Hub?

Qualcomm AI Hub can be used to streamline most common workflows for on-device deployment. Refer to the documentation page for more details.

• Deploying PyTorch models with TensorFlow Lite
• Deploying PyTorch models with Qualcomm® AI Engine Direct
• Deploying ONNX models to multiple runtimes
• Profiling models on-device
• Running inference on your data on-device
• Deploying AIMET quantized models to multiple runtimes