How should you approach technical SEO for AI / LLM visibility? — Watch now.
AI is a hot topic in search. While many SEOs are understandably nervous about how it is changing the SERPs and impacting organic traffic, we also know that we need to be adapting our SEO techniques to ensure we remain visible to AI crawlers and can appear in AI Overviews and LLM results.
Join technical SEO experts Sophie Brannon (StudioHawk) and Richard Barrett (Lumar) for this deep dive into how to update your technical SEO strategies for our new AI search era.
In this session, we cover:
- The latest developments in AI search — AI Overviews, ChatGPT, Gemini, Perplexity, and more.
- How generative AI is changing search behavior
- Crawlability tips for AI bots
- Blocking AI bots — and accidental blocking issues
- JavaScript rendering issues for LLMs
- Semantic HTML
- Structured data and LLMs.txt
- Multimodal search
- and more!
Watch the full webinar (including Q&A session + audience poll results!) on-demand now, above, or check back here next week for the full written recap.
Meet the Webinar Speakers
- Sophie Brannon – Co-Founder & Director, StudioHawk US
- Richard Barrett (host) – Head of SEO Professional Services at Lumar
The latest developments in AI search — AI Mode, AI Overviews, and more
For Brannon there are 3 key things to think about in regard to today’s AI search landscape:
- LLMs are getting better, with new versions deployed regularly.
- Google’s “AI Mode” officially launched in the US in May 2025.
- Google’s AI Overviews are now rolled out to 200+ countries in more than 40 languages.
She provides a useful side-by-side analysis of AI Mode results compared to classic Google SERPs for the search term: “burger spots.”
Here’s how the search results appear in AI Mode:
Compare the AI Mode results to these results for the same search query in the classic Google SERP:
On the traditional Google SERPs, we see the business results and the map, as well as the classic blue link ranked results for Tripadvisor, etc.
The AI Mode results are more content-led, with an AI-generated list of the best burger restaurants in the user’s location.
Brannon also notes that the links in the AI list go through to the Google Business overview pages rather than the websites for these brands.
She also looks at another example query in AI Mode: “Who sells the cheapest white sneakers?”
“It’s definitely really interesting to see,” Brannon says, “There’s not any links to websites on this version, and these snapshots of AI mode. Which is super scary for a lot of SEOs out there.”
This not the case for all AI Mode searches, though.
Another key phrase search, “buy Marc Jacobs bags,” does involve plenty of the traditional-style website links in AI Mode. But clearly, things are moving and changing quickly in AI search.
Of course, AI is increasingly a feature of the classic Google SERPs, too, with AI Overviews appearing on many search results:
We are frequently presented with Google results where a lot of the SERP real estate is given over to an AI-generated block of content, aka Google’s “AI Overviews”.
Brannon sees this as something of an evolution of featured snippets. Google is trying to drive a zero-click answer to user queries here. Often, these AI Overviews appear for simple informational searches.
Semantic, vector-powered AI search models vs. traditional search
Traditional search models, for many years, was all about matching exact, or similar, keywords. It relied heavily on PageRank, backlinks, TF-IDF keyword-matching, etc. AI search, on the other hand, is more dependent on vector search models:
- Vector embeddings are used to represent the meaning of — and relationships between — words, sentences, entire documents, or entities.
- These vector embeddings enable semantic search, allowing search engines (and LLMs) to better understand what you’re asking, not just how you’re phrasing it.
According to Brannon, what this shift to semantically informed, vector-based models means for SEO is:
- Content relevance is now multi-dimensional: Your content should be semantically relevant to users’ queries and address topics thoroughly and contextually, not just repeat keywords. (aka, stop keyword-stuffing!)
- Indexing is more meaning-based: Search engines today look at semantic similarity between queries and content, not just keyword overlap.
- Content chunking and retrieval: Google can retrieve relevant passages (not just full pages), using vectors to locate the most meaningful chunks.
AI Search Crawlers
Our presenters also highlighted how AI search crawlers work differently than traditional search engine crawlers:
- Deeper content analysis: AI crawlers often use machine learning and natural language processing (NLP) to understand the semantic meaning and context of web content, going beyond simple keyword matching.
- Crawl frequency and intensity: Some reports suggest that AI crawlers can be more aggressive in their crawling patterns, requesting large batches of pages, which can strain website resources.
- Selective crawling: Some AI crawlers might be more selective about the content they prioritize based on specific training or retrieval needs. I.e. they may focus on high-quality informative text.
“When an LLM is used to crawl a website, it generally will fetch the raw HTML content,” Brannon says. “It won’t go through the full browser rendering process like Googlebot does.”
There are also a number of things LLM crawlers can’t or don’t do. For example, AI crawlers can’t:
- Execute JavaScript to dynamically render content.
- Apply CSS for visual styling.
- Render the document object model (DOM) to a visual output.
This means that if your site is heavily reliant on JavaScript, LLMs will likely have a hard time crawling it.
Brannon’s suggestion, here, is simple: For the AI era of search, consider keeping your websites more HTML-based rather than JavaScript-based.
Server-side rendering, which delivers fully rendered HTML directly in the initial server response, makes your content as accessible as it can be if your site uses JavaScript.
LLMs & JavaScript Rendering
Implications for Technical SEO in the Age of AI Crawlers:
- Client-side rendering can be problematic: If your website relies heavily on JavaScript to render the main content, AI-powered crawlers that only fetch raw HTML might not see that content. This could hinder their ability to understand the page’s purpose and extract information effectively.
- Server-side rendering (SSR) or pre-rendering becomes more important: These techniques deliver the fully rendered HTML content directly in the initial server response, making it accessible to crawlers that don’t execute JavaScript.
- Semantic HTML is crucial: Clear and well-structured HTML helps LLMs understand the content even without full rendering.
When it comes to the HTML itself, semantic, clear, and well-structured HTML is key to ensuring your site is sufficiently crawled. Brannon urges SEOs to speak to their web developers about using HTML5 to implement semantic HTML moving forward.
Blocking AI bots — on purpose, or by accident
There may be reasons that you want to block AI bots from crawling certain areas of your sites – particularly if, for example, your site includes sensitive information such as medical or financial data.
SEOs need to be mindful that achieving this is a little different in the era of AI search compared to the traditional era. LLMs don’t always respect robots.txt files.
Brannon suggests SEOs and web developers consider a server-level or reverse proxy if they want to block AI bots.
- Things like Cloudflare or other firewall-type software may be able to block different bots.
- “Google-Extended” was introduced as a standalone product token to block Bard and Vertex AI (Google’s machine learning platform) in 2023.
- There was also talk about introducing a new meta tag specifically for LLMs, although this hasn’t formally been launched yet.
Remember that your page still must be indexed by Google to show up in its AI Mode. (John Mueller and Barry Schwartz have recently confirmed this.)
Brannon also points to research from Patrick Stox, which found that the ‘nosnippet’ tag may also block content from appearing in AI Mode.
There is also LLMS.txt, a recently proposed standard similar to robots.txt, which is meant to simplify website content into a markdown file for easier AI crawling, according to an article by Chris Green. This could help bypass some of the JavaScript rendering issues mentioned before. However, Green found that adoption of LLMs.txt is extremely low; he only found it present on 15 out of 1 million sites he crawled. He also notes that no major AI companies have officially supported LLMs.txt yet.
Structured data and LLMs
Brannon describes structured data quite simply as an extra bit of code you can use that provides additional context for Google. This might include added context about who you are, what you’re selling (and at what price), what reviews your product has, etc. Historically, SEOs have implemented structured data to gain rich results on Google.
There is, however, conflicting information available about whether or not structured data can impact how AI models understand content — it may provide more contextual information to AI systems in a roundabout way, for example, if some of the underlying systems LLMs rely on incorporate structured data into the AI training data.
Structured data may also impact your brand’s appearance in knowledge graphs, which LLMs and AI SERP features may also tap.
Brannon’s conclusion on structured data is: Use it in the best way that you can.
“The fundamentals are still going to matter,” she says. “You still want to show up in Google.”
Multimodal search
Multimodal search refers to search methods that allow users to query using multiple input types—such as images, voice, or video search—instead of just using text keywords. In AI, multimodal search combines different types of data (like text and images) to improve how systems understand and retrieve relevant information across formats.
Last year, Google announced “Project Astra,” a voice-operated AI assistant that is being developed as a multimodal answer to ChatGPT.”
Shortly before Google’s Project Astra announcement, OpenAI launched GPT-4o with a tagline that stated: “We’re announcing GPT-4o, our new flagship model that can reason across audio, vision, and text in real-time.”
This means SEOs need to think comprehensively about optimizing all of these elements.
“So it’s not just your content,” Brannon says. “It’s: your images are being well optimized. Your video is well optimized. Making sure if you’re based in a specific location and your customer base is in a location, you’re really owning that location.”
“It’s about making sure that for every asset you own – you are really making sure that it’s doing its job.”
Summary: Tech SEO fundamentals still matter for AI search
In the era of AI search, technical SEO fundamentals still matter, says Brannon. This means making sure you’re optimizing for:
- Responsiveness and user experience: Speed, mobile-friendliness, and responsiveness are important as AI focuses on accessibility and user experience across devices. Making sure your forms are easy to fill in for users and bots is important. Make sure your checkout process is accessible.
- Site architecture and linking: A clear site architecture can help both users and AI models navigate and understand your content’s hierarchy and relationships.
- Client vs. Server Side Rendering: Server-side rendering is becoming even more critical, but many SEOs have been pushing dev teams toward this for a while because even traditional Google search systems also struggle with JavaScript at times.
- Optimize all site assets: It’s important that all of your site’s assets, including images and video, as well as text content, are well optimized.
Lumar tools to help you with AI search
Many of the reports and features in Lumar’s website optimization platform will help your efforts in AI search. These include our analytics for:
- Crawlability: Identifying if AI bots can access your pages.
- Renderability: Identifying discrepancies between raw and rendered HTML.
- Indexability & Visibility: Revealing which pages are ignored by search engines or missing from SERPs.
- Structured Data: Exposing missing or invalid markup and highlighting high-value structured content — key for improving AI comprehension.
- Bot Behavior & Crawl Budget: Showing how often AI bots hit your pages and which are ignored.
Want to learn more about how Lumar can help with AI search and SEO? Book your personalized Lumar platform demo here.
Don’t miss the next Lumar webinar!
Sign up for our newsletter below to be alerted about upcoming webinars, or give us a follow on LinkedIn or Twitter/X to stay up-to-date with all the latest news in SEO and digital optimization.
Want even more SEO insights on-demand? Browse Lumar’s full library of SEO and website optimization webinars here.
