快速开始
本文相关脚本都可以在开源代码仓库 https://github.com/MooreThreads/tutorial_on_musa/tree/master/vllm 找到，如需帮助可联系 MT 技术人员。
模型权重文件准备
模型权重文件准备因用户需要不同，此处 不作赘述。如使用开源模型常见平台包括Hugging Face, 魔搭/ModelScope.cn。
环境变量设置
提示
推荐提前设置好对应的 TP(Tensor Parallelism)大小与模型权重路径变量。
本文以Qwen2.5-72B-Instruct为例。所有支持的模型以及配置注意事项可以在功能与模型支持-模型支持列表查询到。以下环境变量命令根据实际需要更改。
export model_name="Qwen2.5-72B-Instruct"
export tp_size=8
export model_dir="/data/mtt/models/${model_name}/"
export converted_model_dir="/data/mtt/models_convert/${model_name}-${tp_size}-converted/"
旧版本镜像(KuaE 版本<=1.2.0)中可能找不到 vllm module，可通过设置环境变量解决：
# 以上文中启动的容器为例，将vllm_mtt代码路径设置为python环境变量
export PYTHONPATH=/home/workspace/vllm_mtt
模型权重转换
在做模型推理之前, 需要将 pytorch 格式的权重文件转换成 MT-Transformer 所支持的格式
python -m mttransformer.convert_weight \
--in_file ${model_dir} \
--saved_dir ${converted_model_dir} \
--tensor-para-size ${tp_size}
其中：
-
--tensor-para-sizeor-tp：张量并行数，可简单理解为参与运行的 GPU 数 -
--in_file：从 huggingface 上下载的模型 pytorch 格式权重文件的路径 -
--saved_dir：MTTransformer 转换之后的权重文件路径 -
--model_type: 该参数可省略, 默认自动匹配模型类别。如果匹配与预期不符，可以手动传入：目前支持llama,mistral,chatglm2,baichuan,qwen,qwen2,yayi(可以通过功能与模型支持-模型支持与配置快速找到对应模型配置).
也可以执行 convert_weight.sh 脚本进行转换。如果使用脚本，需注意重设converted_model_dir变量。
# Usage: ./convert_weight.sh <original_model_path> <tp_size>
./convert_weight.sh ${model_dir} ${tp_size}
运行离线推理示例程序
执行程序:
python generate_chat.py -ckpt ${converted_model_dir}
generate_chat.py
import vllm
from vllm import LLM, SamplingParams
import configparser
import pathlib
import argparse
import os
def main():
os.environ["TOKENIZERS_PARALLELISM"] = "true"
parser = argparse.ArgumentParser(description="Generate answer from LLM")
parser.add_argument(
"-ckpt",
"--checkpoint-path",
required=True,
help="Path to the model or checkpoint who must be the one converted by MT-Transformer",
)
args = parser.parse_args()
config_path = pathlib.Path(args.checkpoint_path + "/config.ini")
if config_path.exists():
cfg = configparser.ConfigParser()
cfg.read(config_path)
config_name = "config"
model_type = cfg.get(config_name, "model_name")
tp_size = int(cfg.get(config_name, "tensor_para_size"))
else:
raise ValueError(f"The model path must be the one converted by MT-Transformer")
print(f">>>>>>>>>>>>>>>> model type from the config is {model_type}")
print(f">>>>>>>>>>>>>>>> tensor parallel size is {tp_size}")
llm = LLM(
model=args.checkpoint_path,
trust_remote_code=True,
gpu_memory_utilization=0.9,
tensor_parallel_size=tp_size,
device="musa",
block_size=64,
max_num_seqs=128,
max_model_len=2048,
max_num_batched_tokens=2048,
)
prompts = [
"什么是牛顿第三定律？",
"如何保持良好的睡眠？",
"我想去北京，有哪些景点值得去？",
]
tokenizer = llm.get_tokenizer()
if model_type == "yayi":
os.environ["VLLM_NO_USAGE_STATS"] = "1"
# Try to use the apply_chat_template from the Transformers Library.
# However, due to the absence of certain model templates,
# this might lead to incorrect responses. It’s essential to verify
# that the selected template aligns with the model’s architecture and
# expected input format to ensure accurate outputs. If a template
# mismatch occurs, it could result in inconsistencies or unexpected
# behavior in the generated answers. Therefore, consider reviewing or
# customizing the template when necessary to better match the specific
# model in use.For more details, refer to the official Transformers
# documentation: https://huggingface.co/docs/transformers/main/chat_templating
prompts_tmp = []
for prompt in prompts:
messages = [{"role": "user", "content": prompt}]
text = llm.get_tokenizer().apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
prompts_tmp.append(text)
print(f">>>>>>>>>>>> prompts is {prompts_tmp}")
sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=1024)
outputs = llm.generate(prompts_tmp, sampling_params)
print(f"\n")
for idx, output in enumerate(outputs):
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt {idx}: {prompt}, \nGenerated text: {generated_text}\n\n")
if __name__ == "__main__":
main()
LLM与SamplingParams的参数选择可见功能与模型支持-vLLM 参数支持。
