Qualcomm® AI HubAI Hub

Qwen2.5-7B-Instruct

State‑of‑the‑art large language model useful on a variety of language understanding and generation tasks.

The Qwen2.5‑7B‑Instruct is a state‑of‑the‑art multilingual language model with 7 billion parameters, excelling in language understanding, generation, coding, and mathematics. AI Hub provides with four QNN context binaries (shared weights) that can be deployed on Snapdragon 8 Elite with Genie SDK.

View README FileView Genie SDK Tutorial
Model RepositoryHugging FaceResearch Paper

Technical Details

Input sequence length for Prompt Processor:128
Context length:4096
Number of parameters:7B
Precision:w4a16 + w8a16 (few layers)
Num of key-value heads:4
Information about the model parts:Prompt Processor and Token Generator are split into 6 parts each. Each corresponding Prompt Processor and Token Generator part share weights.
Prompt processor model size:4.61 GB
Prompt processor input (part1):128 tokens
Prompt processor output (part1):Embeddings output
Prompt processor input (other parts):128 tokens + KVCache initialized with pad token
Prompt processor output (other parts):128 output tokens + KVCache for token generator
Token generator model size:4.61 GB
Token generator input (part1):128 tokens
Token generator output (part1):Embeddings output
Token generator input (other parts):1 input token + past KVCache
Token generator output (other parts):1 output token + KVCache for next iteration
Use:Initiate conversation with prompt-processor and then token generator for subsequent iterations.
Minimum QNN SDK version required:2.27.7
Supported languages:Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more.
TTFT:Time To First Token is the time it takes to generate the first response token. This is expressed as a range because it varies based on the length of the prompt. The lower bound is for a short prompt (up to 128 tokens, i.e., one iteration of the prompt processor) and the upper bound is for a prompt using the full context length (4096 tokens).
Response Rate:Rate of response generation after the first response token.

Applicable Scenarios

  • Dialogue
  • Content Generation
  • Customer Support

Supported Form Factors

  • Phone
  • Tablet

Licenses

Source Model:APACHE-2.0
Deployable Model:AI-HUB-MODELS-LICENSE
Terms of Use:Qualcomm® Generative AI usage and limitations

Tags

  • llm
  • generative-ai

Supported Devices

  • Snapdragon X Elite CRD

Supported Chipsets

  • Snapdragon® X Elite

Related Models

See all models
View details for the Qwen2-7B-Instruct model.

Qwen2-7B-Instruct

State-of-the-art large language model useful on a variety of language understanding and generation tasks.

Looking for more? See models created by industry leaders.

Discover Model Makers