Vercel AI SDK

Notes on how vercel ai sdk works

Starting from scratch

What is it?

OS Library for easily stream API responses from AI models

  • Has built-in LLM adapters for OpenAI, LangChain, and Hugging Face
import { OpenAIStream, StreamingTextResponse } from 'ai'
import { Configuration, OpenAIApi } from 'openai-edge'

// Create an OpenAI API client (that's edge friendly!)
const config = new Configuration({
  apiKey: process.env.OPENAI_API_KEY
})
const openai = new OpenAIApi(config)

// IMPORTANT! Set the runtime to edge
export const runtime = 'edge'

export async function POST(req: Request) {
  // Extract the `messages` from the body of the request
  const { messages } = await req.json()

  // Ask OpenAI for a streaming chat completion given the prompt
  const response = await openai.createChatCompletion({
    model: 'gpt-3.5-turbo',
    stream: true,
    messages
  })
  // Convert the response into a friendly text-stream
  const stream = OpenAIStream(response)
  // Respond with the stream
  return new StreamingTextResponse(stream)
}
  • Includes UI hooks for data fetching as well as rendering streaming responses -> Real-time data representation -> useChat or useCompletion

Generative UI

Using RSC for streaming the component response. It expects LLMs to support OpenAI SDK and assitant or function calling. Calling createStreameableUI and createStreamableValue we can use langchain and llama index

SDK Overview

####Core functions

  • generateText -> Generates text and tool calls -> Ideal for non interactive use cases (ex: automation of text)
    • Accepts tool calling by passing it under the tools key when calling the function
    • Tool choice cna be influenced by calling toolChoice and letting the model decide
    • It accepts other params for LLM like temperature, topP, presencePenalty, maxRetries, ..
  • streamText -> Stream text and tool calls -> Ideal for interactive use cases (ex: chatbots). Can also be used with Generative UI
    • Contains helper functions to integrate with AI SDK UI easier check
  • generateObject -> Typed, structured Zod-schema-matching object to force the language to return structured data -> A zod schema must be passed
  • streamObject -> Typed, structured Zod-schema-matching object to force the language to stream structured data

Providers

  • Out of the box: OpenAI, Anthropic, Google Generator AI, Google Vertex, Mistral
  • OpenAI compatible APIs: Grok, Perplexity, Fireworks
  • OS: LlamaCpp, Ollama

-> Check here for capabilities

Embeddings

Accept embeddings as input to the LLM. It can be used to improve the quality of the generated text.

Links

SDK