Overview

Use these examples to learn best practices for using the inference API.

Prompt formatting

The following code gives an example of automatically formatting a prompt during inference to match the expected format for the given model. Note, not all models include a prompt_format; make sure you validate that a prompt format is present in your code.

python

import together

together.api_key = "xxxxx"

model = "garage-bAInd/Platypus2-70B-instruct"

stop_words = list(together.Models.info(model)['config']['stop']) # =
['</s>', '###']
prompt_format = str(together.Models.info(model)['config']['prompt_format'])
# = '### Instruction:\n{prompt}\n### Response:\n'

prompt = "hello"
formatted_prompt = prompt_format.format(prompt=prompt)

for token in together.Complete.create_streaming(prompt=formatted_prompt,
model='upstage/SOLAR-0-70b-16bit', stop=stop_words):
    print(token, end="", flush=True)
print("\n")

Grammar correction

Try RedPajama-INCITE-Chat-3B to correct sentences to standard English.

1. For example, begin the prompt by asking to correct the grammar of a sentence:

Prompt 1

Correct this to standard English: I no Sandwich want.

Prompt 2

Correct this to standard English: If I’m stressed out about something, I tend to have problem to fall asleep.

2. Sample response

Response 1

I don’t want a sandwich.

Response 2

Correct this to standard English: If I’m stressed out about something, I tend to have a hard time falling asleep

3. Send the prompt to the API with any appropriate parameters. The code below shows an example in Python using the requests package.

python

import requests

url = "https://api.together.xyz/inference"

payload = {
    "model": "togethercomputer/togethercomputer/RedPajama-INCITE-7B-Chat",
    "prompt": "Correct this to standard English:\nI no Sandwich want.",
    "max_tokens": 256,
    "stop": ".",
    "temperature": 0.1,
    "top_p": 0.7,
    "top_k": 50,
    "repetition_penalty": 1
}
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "Authorization": "Bearer <YOUR_API_KEY>",
  	"User-Agent": "<YOUR_APP_NAME>",
}    

response = requests.post(url, json=payload, headers=headers)

print(response.text)

Further reading

Prompt engineering is a relatively new discipline for developing and optimizing prompts to efficiently use language models (LMs) for a wide variety of applications and research topics. The Prompt Engineering Guide is a great introduction to the subject.

Updated 22 days ago

---