Qualcomm® AI HubAI Hub

IBM-Granite-v3.1-8B-Instruct

State‑of‑the‑art large language model useful on a variety of code understanding and generation tasks.

Granite‑3.1‑8B‑Instruct is a 8B parameter long‑context instruct model finetuned from Granite‑3.1‑8B‑Base using a combination of open source instruction datasets with permissive license and internally collected synthetic datasets tailored for solving long context problems. This model is developed using a diverse set of techniques with a structured chat format, including supervised finetuning, model alignment using reinforcement learning, and model merging.

Not supported

This model is currently not supported on any IoT chipset.

To see performance metrics for this model on other chipsets, click the button below.

View for other chipsets

Technical Details

Context length:4096
Information about the model parts:Prompt Processor and Token Generator are split into 5 parts each. Each corresponding Prompt Processor and Token Generator part share weights.
Input sequence length for Prompt Processor:128
Minimum QNN SDK version required:2.3
Num of key-value heads:8
Precision:w4a16 + w8a16 (few layers)
Prompt processor input (other parts):128 tokens + KVCache initialized with pad token
Prompt processor input (part1):128 tokens
Prompt processor output (other parts):128 output tokens + KVCache for token generator
Prompt processor output (part1):Embeddings output
Response Rate:Rate of response generation after the first response token.
Supported natural languages:English
Supported programming languages:The Granite code foundation models support 116 programming languages including Python, Javascript, Java, C++, Go, and Rust.
Token generator input (other parts):1 input token + past KVCache
Token generator input (part1):1 token
Token generator output (other parts):1 output token + KVCache for next iteration
Token generator output (part1):Embeddings output
Use:Initiate conversation with prompt-processor and then token generator for subsequent iterations.

Applicable Scenarios

  • Coding
  • Coding assist

License

Tags

  • llm
  • generative-ai
  • model-providers

Supported IoT Devices

  • Dragonwing IQ-X7181

Supported IoT Chipsets

  • Qualcomm® Dragonwing™ IQ-X7181
  • Qualcomm® Dragonwing™ Q-8750

Models from IBM watsonx

See all model makers

Looking for more? See pre‑optimized models for all solutions.

Browse All