Llama-v2-7B-Chat

State‑of‑the‑art large language model useful on a variety of language understanding and generation tasks.

Llama 2 is a family of LLMs. The "Chat" at the end indicates that the model is optimized for chatbot‑like dialogue. The model is quantized to w4a16(4‑bit weights and 16‑bit activations) and part of the model is quantized to w8a16(8‑bit weights and 16‑bit activations) making it suitable for on‑device deployment. For Prompt and output length specified below, the time to first token is Llama‑PromptProcessor‑Quantized's latency and average time per addition token is Llama‑TokenGenerator‑KVCache‑Quantized's latency.