Calling an LLM API, continued
Mini Project 3: API-Powered App (repo URL) Due Wednesday, 11/04 Checkpoint 5 Due Sunday, 11/08 Mini Project 3: API-Powered App Due Sunday, 11/08
Agenda Slides Open fullscreen β
Open slides ↗Post-Class Notes
TL;DR#
Today we used ai-chat.html to look at what sits underneath an AI product: how your page reaches a model through your own function, what comes back, and what a model knows and does not. Instructions, your own material and memory each change what it answers, and each costs tokens.
Before next class#
- Checkpoint 5 is due Sunday 11/08. In your
oim3690repo:post-message.htmlandfetch-fun.htmlfrom 10/29ai-chat.htmlin the root folder andask.jsinside a folder namedapi- a link in your
oim3690README toai-chat.htmlon your vercel.app link, such ashttps://<project>.vercel.app/ai-chat.html, where the page answers a question logs/wk09.mdwith your three ideas for Mini Project 3
- Mini Project 3 is due Sunday 11/08, deployed and working. If it uses a key, deploy it on Vercel. The showcase and Walkthrough 3 will be in class on Tuesday 11/17, together with Mini Project 4's.
Next Tuesday, 11/10, we will finish the last slides of Calling an LLM API, you will write down your own idea for a browser extension, and we will see how an extension works. Mini Project 4 starts then.
How your page reaches the model#
- Your page calls your own function at
/api/askand sends it the prompt. - Your function,
api/ask.js, runs on Vercel's server. It adds the key from Vercel's settings and sends a POST to OpenRouter, the same way you posted to the Message Board, with the key in place of your token. - The reply comes back as JSON, with much more in it than your page shows: the model's reasoning, the message, and
usage, which includes what the answer cost.
The page and the function are separate so that the key stays on the server. To test it, open the vercel.app link. Live Server cannot run a function, and the key is not on your computer.
What a model knows#
- The same question can get a different answer. A model writes one token at a time and picks each with some randomness, which is why a coffee shop name changed every time.
- It does not know today's date or anything about this course. It is the bare model: nothing from training, and no tool to look it up. It may refuse, or state a wrong date as if it were sure.
- Your own material in the prompt lets it answer about that material. Pasting the syllabus brought back the right room. It can still misread what you send, so check its answer against the source. A real product usually searches your material first and sends only the passages that match, which is called retrieval augmented generation (RAG).
Instructions cost tokens on every prompt#
The instructions in ask.js turn a general model into one with a job: a JavaScript tutor, a recipe assistant, a study tool. They are sent with every prompt, so they add input tokens each time; in class, "hi" cost five input tokens on one page and 90 on a page with instructions. Anyone can read them in Network, because the reply repeats them, so never put anything private in them.
A chat that remembers#
The API keeps nothing between calls. A chat with memory sends every earlier message along with the new one, so each turn sends more and costs more. This is also why a long chat with your agent gets slower, and why a new task deserves a new chat.
From a chatbot to an agent#
The model did not know the date. A small JavaScript function can tell it. Give the model a function it can ask your code to run, let it use the result, and repeat until the job is done: an AI that calls tools in a loop is an agent. We will build one later in the term.
When the reply has no answer#
In class I said that raising max_output_tokens would fix a reply with no answer. It may not: a model can spend any limit on thinking before it writes the answer, and ask.js has no setting for input. Find out what happened first:
- Open Network, send a question, and click the
askrequest. - In Response, read
status,incomplete_detailsand the items inoutput. Look for an item whosetypeismessage. - Fix it from what you found, with your agent: paste the reply and ask why there is no message. The slide When the Reply Has No Answer in Calling an LLM API shows the same case.
An answer longer than the limit still stops at the limit, and status stays incomplete.
The course key#
The course key now has a shared limit of $10 for the whole class. Use it as much as you like for this course. Never call your function in a loop, because when the limit runs out every page that uses the key stops at once.
Giving an agent its own accounts#
After the lightning talk, one way to limit what an agent can see came up: give it accounts of its own. A separate email that receives only the messages it needs, such as flight confirmations forwarded from your main inbox, lets a travel agent read those and nothing else. Some people give agents a separate computer for the same reason. Whether an agent is you or a second person working for you decides how much of your own context you hand it.