TII

Falcon 7B Instruct

旧版

124.0K下载量1.0K点赞Apr 2023发布日期8K tokens上下文Apache 2.0许可证40 基础质量

Falcon 7B Instruct (7B parameters) requires approximately 5.9 GB of VRAM with Q4_K_M quantization. For the best balance of quality and speed, we recommend hardware with at least 7 GB of VRAM.

快速开始

— 复制粘贴即可本地运行

Copy-paste commands to run Falcon 7B Instruct on your machine.

Run

lms load falcon-7b-instruct && lms server start

Quick specs

Parameters7B

Architecturedense

Context8K tokens

Modalitytext

Min RAM2.7 GB

Rec. RAM4.3 GB (Q4_K_M)

LicenseApache 2.0

FamilyFalcon

✓ Chat✓ Reasoning

About this model

Falcon-7B-Instruct is a 7B parameters causal decoder-only model built by TII based on Falcon-7B and finetuned on a mixture of chat/instruct datasets. It is made available under the Apache 2.0 license.

•You are looking for a ready-to-use chat/instruct model based on Falcon-7B
•Falcon-7B is a strong base model, outperforming comparable open-source models: (e.g., MPT-7B, StableLM, RedPajama etc.), thanks to being trained...
•It features an architecture optimized for inference: , with FlashAttention (Dao et al., 2022) and multiquery (Shazeer et al., 2019)

Falcon 7B Instruct 的最佳选择

运行此模型

Falcon 7B Instruct on RTX 3070 Ti 8GB Falcon 7B Instruct on RTX 3070 8GB Falcon 7B Instruct on RTX 3060 Ti 8GB

量化选项

各量化级别的 VRAM 估算

No hardware detected — fit column shows raw VRAM estimates

Quant	Bits	VRAM	Quality	Fit
Q2_K	2	2.7 GB	Low	—
Q3_K_S	3	3.4 GB	Low	—
NVFP4	4	3.9 GB	Medium	—
Q4_K_M	4	4.3 GB	Medium	—
Q5_K_M	5	5.0 GB	High	—
Q6_K	6	5.7 GB	High	—
Q8_0	8	7.5 GB	Very High	—
F16	16	14.3 GB	Maximum	—

Quality benchmarks

Falcon 7B Instruct benchmark scores

Benchmark verified

Reasoning

MMLU-Pro1.7%

GPQA Diamond—

MATH-5001.2%

ARC Challenge45.8%

General

Chatbot Arena—

IFEval19.7%

Source: community · 2023-05-25

硬件兼容性

全部硬件的适配估算

打开计算器

Computing compatibility...

内存详细分析

Reference: RTX 2060 6GB

Weights4.3 GB

KV Cache0.1 GB

Runtime0.9 GB

Headroom0.6 GB

常见问题

FAQ — Falcon 7B Instruct

另请参阅

量化指南评分方法打开计算器