「docs crawler」の検索結果

Crawl websites (news, blogs, papers, GitHub, product docs, RSS feeds) into a fixed-schema JSONL file, then create a dataset and a searchable application in Viking AI Search. Supports one-time crawl and scheduled recurring crawl with automatic incremental sync.

日本語の概要は準備中です。原文の説明を表示しています。

Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs site, scrape documentation, scrape knowledge base, scrape blog, build RAG corpus, build vector database from website, knowledge base for chatbot, GPT knowledge files, llms.txt, sitemap crawl, BFS crawl, scrape with depth or page limit, include exclude URL globs, remove boilerplate, strip navigation header footer, website to markdown, website to text, multi-page extraction, bulk page scraping, clean markdown from URL, docs site to markdown corpus, site to clean corpus. Also applies to building RAG pipelines, indexing a customer site, syncing docs into a vector store, generating training corpora from any docs hub, or expanding a single start URL into a clean corpus of every reachable in-scope page.

日本語の概要は準備中です。原文の説明を表示しています。

Uses htmltest to crawl generated documentation or static site output, detect broken internal and external links, and return a link-focused validation report before deploy. This is a narrow docs QA skill for agents validating already-built HTML, not a generic site generator or crawler listing.

日本語の概要は準備中です。原文の説明を表示しています。

Generative Engine Optimization plus agent readiness. Use this skill whenever the user asks about GEO, AEO, LLMO, optimizing content for AI search engines (ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini, Copilot), getting cited by LLMs, brand mentions in AI responses, llms.txt, robots.txt for AI crawlers, Schema.org JSON-LD, FAST framework, agent readiness, MCP server discovery, Web Bot Auth, OAuth for agents, API Catalog, agentic commerce (x402, ACP, UCP), Markdown content negotiation, or auditing a site or page for AI visibility. Also trigger when the user says "AI SEO", asks how to "rank in ChatGPT", "show up in Perplexity", "appear in AI Overviews", or wants a site to "speak to AI agents". Trigger even when the request is implicit, such as "review this article so AI search picks it up" or "make our docs agent-friendly".

日本語の概要は準備中です。原文の説明を表示しています。