Home / Guides / Make Your Website Agent-Ready

Make Your Website Agent-Ready

For twenty years, the main robot reading websites was the search engine. Now AI agents read them too. When someone asks an AI "who is Jane Doe," the agent visits your site, summarizes what it finds, and answers for you. This guide shows three small files you can add to your GitHub Pages site so it answers correctly.

None of this is required for the course. It is a strong addition to your final project, and it is the kind of thing almost no beginner site has.

One rule before you start: only add what you can explain. If a checkpoint reviewer asks "what does this file do," you should have an answer.

🤖 What "Agent-Ready" Means

When an AI agent visits your site, it wants plain facts, not a guess from your layout. An agent-ready site hands it those facts directly:

  • Stable URLs. Once a page is published, keep its address. People link to it, search engines index it, and agents cache it. Renaming about-me.html after you share it breaks all of those at once.
  • Structured facts. A small block of machine-readable data that says who you are.
  • A map. A short file that points an agent at your most important pages.

The three sections below are the practical version of that.

🗂️ 1. Structured Data with JSON-LD

JSON-LD is a block of data you put in your page <head> that describes the page in a format machines read. It uses a shared vocabulary from schema.org. Google uses it for richer search results, and AI agents read it to get facts without guessing from your HTML.

For a personal site, describe yourself as a Person:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Person",
  "name": "Jane Doe",
  "jobTitle": "Babson Student",
  "url": "https://janedoe.github.io",
  "sameAs": [
    "https://www.linkedin.com/in/janedoe",
    "https://github.com/janedoe"
  ]
}
</script>

Put it inside <head>. It does not show up on the page; it is there for machines. The sameAs list links your other profiles so an agent knows they are the same person.

Ask AI to help, but read the result: "Add a JSON-LD Person block to my index.html using my name, my role, my site URL, and my LinkedIn and GitHub links." Then check every value is actually yours.

📋 2. An llms.txt Index

llms.txt is a plain Markdown file at the root of your site (https://yourname.github.io/llms.txt) that points an AI agent at your most useful pages. Think of it as a short table of contents written for a robot. It is an emerging convention, not an official standard yet, but it is simple and cheap to add.

Create a file named llms.txt next to your index.html:

# Jane Doe

> Babson student building web projects. This site is my portfolio.

## Pages

- [About me](https://janedoe.github.io/about-me.html): background, interests, and contact
- [Projects](https://janedoe.github.io/projects.html): things I have built this semester

Keep it short and link to real pages. The first line is your name, the quote is a one-line summary, and the list points to your key pages.

🚦 3. A robots.txt File

robots.txt is a plain text file at your site root (https://yourname.github.io/robots.txt) that tells robots which parts of your site they may read. It has been around for decades for search engines, and the same file now controls AI crawlers.

A simple version that welcomes everyone and points to a sitemap:

User-agent: *
Allow: /

Sitemap: https://janedoe.github.io/sitemap.xml

User-agent: * means "all robots," and Allow: / means "you may read everything." If you ever want to keep a specific AI crawler out, you name it instead of *. For a public portfolio, allowing everyone is usually what you want, because you are trying to be found.

✅ Quick Checklist

Add these to your final project and you have a site most beginners never build:

  • A JSON-LD Person block in your index.html <head>
  • An llms.txt at your site root linking your key pages
  • A robots.txt at your site root
  • Every URL still works after you publish (do not rename published files)

📚 Going Deeper

This guide covers the starter set. The full, sourced reference for everything a modern website should do, including security headers, performance, and more advanced agent protocols like MCP, lives at The Website Specification. It is a good place to look when your project is ready for the next level.

Last updated: Sunday, 6/7/2026