Qwen

/

Qwen3-235B-A22B-Thinking-2507-FP8

like 76

Follow

Qwen 63.8k

Hi !
After uses:

docker run -d --name vllm-qwen-235b \
  --gpus '"device=1,2,3,4"' \
  --ipc=host \
  -p 8001:8001 \
  -v /srv/models:/models \
  vllm/vllm-openai:latest \
  --model /models/qwen3-235b-fp8 \
  --port 8001 \
  --tensor-parallel-size 4 \
  --max-model-len 65536 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --enable-chunked-prefill

I tried:

root@srv-ia-020:/home/fontesc# curl -s http://10.120.191.20:8001/v1/chat/completions   -H "Content-Type: application/json"   -H "Authorization: Bearer dummy"   -d '{
    "model": "/models/qwen3-235b-fp8",
    "messages": [{"role":"user","content":"how many is 1+1?"}]
  }'

but the answer is:

{"id":"chatcmpl-a552f115b7aa5d68","object":"chat.completion","created":1766171761,"model":"/models/qwen3-235b-fp8","choices":[{"index":0,"message":{"role":"assistant","content":"Okay, the user asked "how many is 1+1?" Hmm, this seems like a very basic math question. Maybe they're a young child just learning addition, or perhaps someone testing if I'm paying attention. \n\nI should keep it simple and clear. The answer is obviously 2, but let me think if there's any trick here. In standard arithmetic? Definitely 2. In binary? Still 10 which is two. In weird modular systems? Nah, overcomplicating it. \n\nUser probably wants the straightforward answer. I'll say "2" but add a friendly explanation since they said "how many" - shows they might be thinking concretely. Should I mention it's the foundation of math? Yeah, a little educational touch won't hurt. \n\n...Wait, is this a meme reference? Like the "1+1=3" propaganda thing? No, the question seems genuine. Better assume good faith. \n\nSmiley face at the end. Kids like those. If it's an adult testing me, still harmless. Done.\n\n\nThe answer to 1 + 1 is 2. \n\nThis is the most fundamental operation in arithmetic: \n- 1 (one unit) + 1 (another unit) = 2 (two units). \n\nIt’s the foundation of counting, addition, and all higher mathematics! 😊 \nLet me know if you'd like to explore more math concepts!","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null,"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":18,"total_tokens":323,"completion_tokens":305,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}root@srv-ia-020:/home/fontesc#

as you can see the first token is not <think> is directly Okay

What wrong am I doing?