部署OpenAI兼容API服务并进行流式聊天

启动vllm大模型服务

python -m vllm.entrypoints.openai.api_server \
    --served-model-name DeepSeek-R1-7B --model DeepSeek-R1-Distill-Qwen-7B-GPTQ-Int4-MTT \
    --gpu-memory-utilization 0.4 --max-model-len 2048 --device musa \
    --port 8000 --tensor-parallel-size 1 -pp 1 --block-size 64 --max-num-seqs 1 \
    --trust-remote-code --disable-log-stats --disable-log-requests --swap-space=0

测试大模型接口

获取模型列表

curl http://localhost:8000/v1/models

# 输出
{
    "object": "list",
    "data": [
        {
            "id": "DeepSeek-R1-7B",
            "object": "model",
            "created": 1732267605,
            "owned_by": "vllm",
            "root": "DeepSeek-R1-Distill-Qwen-7B-GPTQ-Int4-MTT",
            "parent": null,
            "permission": [
                {
                    "id": "modelperm-c163862182504fbda3f2591a1a6670e7",
                    "object": "model_permission",
                    "created": 1732267605,
                    "allow_create_engine": false,
                    "allow_sampling": true,
                    "allow_logprobs": true,
                    "allow_search_indices": false,
                    "allow_view": true,
                    "allow_fine_tuning": false,
                    "organization": "*",
                    "group": null,
                    "is_blocking": false
                }
            ]
        }
    ]
}

调用模型接口

注意需要修改对应的model名称

# 中文问答
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-R1-7B",
    "temperature": 0.7,
    "top_p": 0.8,
    "top_k": 20,
    "repetition_penalty":1.05,
    "messages": [{"role": "user", "content": "介绍一下北京"}]
  }'

# 输出
{
    "id": "cmpl-4e44a394c2ce4627a73a35fcc919d8db",
    "object": "chat.completion",
    "created": 1732268219,
    "model": "DeepSeek-R1-7B",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "北京，中国的首都，位于中国北部的平原上，是全国的政治、文化和交通中心。北京的地势平坦，主要由山地和低平原构成。它周围被长城环绕，是中国著名的文化名城之一。\n\n北京的历史悠久，可追溯到公元前2300年前。自秦汉时期起，北京就是中国的重要政治中心。在隋唐至清朝，北京曾被称为“大都”或“燕京”，是当时中国最大的城市之一。直到明朝，北京一直被称为“北京”。\n\n北京不仅有着悠久的历史和浓厚的文化底蕴，同时也是中国的工业和商业中心之一。北京拥有丰富的自然景观和文化遗迹，如故宫、长城、颐和园、天坛等，都是中国著名的旅游景点。\n\n北京也是国际交往的重要城市，也是中国乃至亚洲的政治、经济、文化中心之一。北京拥有众多的高校和科研机构，是中国的人才和科研中心之一。北京还是中国重要的交通枢纽之一，拥有多个国际机场和高速铁路网络。北京还拥有丰富的教育资源和科研机构，吸引了大量的人才和科研力量聚集。北京是中国乃至亚洲的科技、教育和文化中心之一，也是中国现代化和国际化的重要象征。"
            },
            "logprobs": null,
            "finish_reason": "stop",
            "stop_reason": null
        }
    ],
    "usage": {
        "prompt_tokens": 31,
        "total_tokens": 276,
        "completion_tokens": 245
    }
}

# 英文问答
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-R1-7B",
    "temperature": 0.7,
    "top_p": 0.8,
    "top_k": 20,
    "repetition_penalty":1.05,
    "messages": [{"role": "user", "content": "Hello! What is your name?"}]
  }'

# 输出
{
    "id": "cmpl-1c39630a5d204549af9d9055a01f10b1",
    "object": "chat.completion",
    "created": 1732268198,
    "model": "DeepSeek-R1-7B",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Hello! I'm Qwen, created by Alibaba Cloud. I'm here to help you with any questions you might have. Is there anything specific you'd like to know or discuss?"
            },
            "logprobs": null,
            "finish_reason": "stop",
            "stop_reason": null
        }
    ],
    "usage": {
        "prompt_tokens": 36,
        "total_tokens": 74,
        "completion_tokens": 38
    }
}

python调用

流式输出

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    # defaults to os.environ.get("OPENAI_API_KEY")
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

chat_completion = client.chat.completions.create(
    messages=[{
        "role": "system",
        "content": "You are a helpful assistant."
    }, {
        "role": "user",
        "content": "北京有哪些名胜古迹？"
    }],
    model=model,
    temperature=0.7,
    top_p=0.8,
    extra_body={
        'top_k':20,
        'repetition_penalty':1.05, # 惩罚重复,vllm默认没有加载需要添加，参考：https://modelscope.cn/models/Qwen/Qwen2.5-7B-Instruct/file/view/master?fileName=generation_config.json&status=1#L9
    },
    max_tokens=512,
    stream=True,  # 启用流式输出
)

# 处理流式响应
print("Chat response (streaming):")
for chunk in chat_completion:
    if chunk.choices:
        delta = chunk.choices[0].delta
        content = delta.content
        if content:
            print(content, end='', flush=True)
print("\n - Chat response (end) -\n")

非流式输出

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    # defaults to os.environ.get("OPENAI_API_KEY")
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

chat_completion = client.chat.completions.create(
    messages=[{
        "role": "system",
        "content": "You are a helpful assistant."
    }, {
        "role": "user",
        "content": "北京有哪些名胜古迹？"
    }],
    model=model,
    temperature=0.7,
    top_p=0.8,
    extra_body={
        'top_k':20,
        'repetition_penalty':1.05,
    },
    max_tokens=512,
    stream=False,
)

print("Chat completion results:")
print(chat_completion)

提示

服务器测部署参考链接。