CohereLabs

/

c4ai-command-r-plus

like 1.82k

Follow

Cohere Labs 3.58k

good

Sorry for unrelated question, but is it possible to use command-r-+ with hf inference api with pro subscription although it is 103B?

Okay I got this to work after I read the docs properly and did more research.

Out of the box it's just a model. What I'm after is a chat-like template. Hugging Face offers something called TGI (Text Generation Inference): https://huggingface.co/docs/text-generation-inference/en/index

To use it, you need to specify it in a request like so using OpenAI's chat completion API: https://huggingface.co/blog/llama3#how-to-prompt-llama-3

from openai import OpenAI

# initialize the client but point it to TGI
client = OpenAI(
    base_url="/static-proxy?url=https%3A%2F%2Fapi-inference.huggingface.co%2Fmodels%2FCohereForAI%2Fc4ai-command-r-plus" + "/v1/",  # replace with your endpoint url
    api_key="<HF_API_TOKEN>",  # replace with your token
)
chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "user", "content": "Why is open-source software important?"},
    ],
    stream=True,
    max_tokens=500
)

# iterate and print stream
for message in chat_completion:
    print(message.choices[0].delta.content, end="")

This works.

Or, to use this exact example with postman.

Post Endpoint: /static-proxy?url=https%3A%2F%2Fapi-inference.huggingface.co%2Fmodels%2FCohereForAI%2Fc4ai-command-r-plus%2Fv1%2Fchat%2Fcompletions%3C%2Fa%3E%3C%2Fp%3E

JSON body:

{
  "model": "tgi",
  "messages": [
    {"role": "user", "content": "Who is Elon Musk?"}
  ],
  "stream": false,
  "max_tokens": 500
}

{
    "id": "",
    "object": "text_completion",
    "created": 1714021803,
    "model": "text-generation-inference/commandrplus-medusa",
    "system_fingerprint": "1.4.5-native",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Elon Musk is a business magnate, investor, and philanthropist. He is the founder, CEO, and Chief Engineer of SpaceX; angel investor, CEO and Product Architect of Tesla, Inc.; owner and CEO of Twitter, Inc.; founder of the Boring Company; co-founder of Neuralink; and president of the philanthropic Musk Foundation. With an estimated net worth of around $191 billion, Musk is the second-wealthiest person in the world, behind Bernard Arnault as of August 3, 2023."
            },
            "logprobs": null,
            "finish_reason": "eos_token"
        }
    ],
    "usage": {
        "prompt_tokens": 11,
        "completion_tokens": 109,
        "total_tokens": 120
    }
}

Closing it as it seems resolved!