And use it in the following codes.
import os import openai client = openai.OpenAI( base_url="https://mixtral-8x7b.lepton.run/api/v1/", api_key=os.environ.get('LEPTON_API_TOKEN') ) completion = client.chat.completions.create( model="mixtral-8x7b", messages=[ {"role": "user", "content": "say hello"}, ], max_tokens=128, stream=True, ) for chunk in completion: if not chunk.choices: continue content = chunk.choices[0].delta.content if content: print(content, end="")
The rate limit for the Model APIs is 10 requests per minute across all models under Basic Plan. For the pricing plan, you may check out pricing page, If you need a higher rate limit with SLA or dedicated deployment, please contact us.