「web crawler」の検索結果

Complete knowledge domain for Firecrawl v2 API - web scraping and crawling that converts websites into LLM-ready markdown or structured data. Use when: scraping websites, crawling entire sites, extracting web content, converting HTML to markdown, building web scrapers, handling dynamic JavaScript content, bypassing anti-bot protection, extracting structured data from web pages, or when encountering "content not loading", "JavaScript rendering issues", or "blocked by bot detection". Keywords: firecrawl, firecrawl api, web scraping, web crawler, scrape website, crawl website, extract content, html to markdown, site crawler, content extraction, web automation, firecrawl-py, firecrawl-js, llm ready data, structured data extraction, bot bypass, javascript rendering, scraping api, crawling api, map urls, batch scraping

日本語の概要は準備中です。原文の説明を表示しています。

Crawl websites (news, blogs, papers, GitHub, product docs, RSS feeds) into a fixed-schema JSONL file, then create a dataset and a searchable application in Viking AI Search. Supports one-time crawl and scheduled recurring crawl with automatic incremental sync.

日本語の概要は準備中です。原文の説明を表示しています。

Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs site, scrape documentation, scrape knowledge base, scrape blog, build RAG corpus, build vector database from website, knowledge base for chatbot, GPT knowledge files, llms.txt, sitemap crawl, BFS crawl, scrape with depth or page limit, include exclude URL globs, remove boilerplate, strip navigation header footer, website to markdown, website to text, multi-page extraction, bulk page scraping, clean markdown from URL, docs site to markdown corpus, site to clean corpus. Also applies to building RAG pipelines, indexing a customer site, syncing docs into a vector store, generating training corpora from any docs hub, or expanding a single start URL into a clean corpus of every reachable in-scope page.

日本語の概要は準備中です。原文の説明を表示しています。

Search the public web and crawl URLs via the Unique AI Platform's `unique-cli web-search` command, with automatic per-turn citation tracking so cited web facts render as clickable external reference chips and `<sup>N</sup>` footnotes on the Unique platform. ALWAYS use this skill when the user asks to search the public web, research a topic online, look up news/articles/regulations, fetch the contents of one or more URLs, or perform any "two-phase" web research (search → review snippets → crawl selected URLs for full content). Routes through the Unique AI Platform's `/web-search-api` endpoints, so the engine (Google, Brave, Tavily, Jina, Firecrawl, …) and crawler are resolved server-side from the current environment — you never need to manage engine API keys yourself. Use this instead of fabricating answers from training data whenever fresh / source-cited information is needed. Do NOT use this skill for the internal knowledge base (`unique-cli search`) or for arbitrary tool execution (`unique-cli mcp`).

日本語の概要は準備中です。原文の説明を表示しています。

Build reliable web scrapers and crawlers with Crawlee — Apify's open-source framework for structured web scraping. Use when someone asks to "scrape a website", "build a crawler", "Crawlee", "web scraping at scale", "scrape JavaScript-rendered pages", "crawl with Playwright/Puppeteer", or "extract data from websites reliably". Covers HTTP crawling, browser crawling, request queues, proxy rotation, and data export.

日本語の概要は準備中です。原文の説明を表示しています。

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations.

日本語の概要は準備中です。原文の説明を表示しています。

"Implement a web crawler pipeline covering URL discovery, fetching, parsing, and storage. Use this skill when the user needs to build a site crawler, audit website structure, or collect web data systematically — even if they say 'scrape a website', 'crawl all pages', or 'site audit spider'.".

日本語の概要は準備中です。原文の説明を表示しています。

Browsertrix Crawler is a browser-based crawling system from Webrecorder for high-fidelity site capture, QA replay analysis, and configurable crawl behavior. It runs complex crawls in a single Docker container and uses Puppeteer plus Chrome DevTools Protocol under the hood.

日本語の概要は準備中です。原文の説明を表示しています。

Build a complete web scraping Actor with Crawlee and deploy to Apify. Use when you need end-to-end web scraping on Apify: defining an input schema, building a router-based Crawlee crawler, extracting structured data, storing results in a dataset, testing locally, and deploying the Actor to the platform. Trigger with "apify scrape website", "build apify actor", "crawlee scraper", "apify main workflow".

日本語の概要は準備中です。原文の説明を表示しています。

You are an expert in Crawl4AI, the open-source web crawler built for AI applications. You help developers extract clean, structured data from websites for LLM training, RAG pipelines, and content analysis — with automatic markdown conversion, JavaScript rendering, CSS-based extraction, LLM-powered structured extraction, and session management for multi-page crawling.

日本語の概要は準備中です。原文の説明を表示しています。

Deploys intelligent web scraping actors on the Apify platform using the Apify SDK with RequestQueue and Dataset APIs. Handles dynamic content via Apify CheerioCrawler and PlaywrightCrawler with automatic scaling.

日本語の概要は準備中です。原文の説明を表示しています。

Comprehensive SEO analysis for any website or business type. Performs full site audits, single-page deep analysis, technical SEO checks (crawlability, indexability, Core Web Vitals with INP), schema markup detection/validation/generation, content quality assessment (E-E-A-T framework per Dec 2025 update extending to all competitive queries), image optimization, sitemap analysis, and Generative Engine Optimization (GEO) for AI Overviews, ChatGPT, and Perplexity citations. Analyzes AI crawler accessibility (GPTBot, ClaudeBot, PerplexityBot), llms.txt compliance, brand mention signals, and passage-level citability. Industry detection for SaaS, e-commerce, local business, publishers, agencies. Triggers on: "SEO", "audit", "schema", "Core Web Vitals", "sitemap", "E-E-A-T", "AI Overviews", "GEO", "technical SEO", "content quality", "page speed", "structured data".

日本語の概要は準備中です。原文の説明を表示しています。

Comprehensive SEO analysis for any website or business type. Orchestrates 52 sub-skills (52 active): full site audits, technical SEO (CWV/INP), schema, E-E-A-T content quality, images, sitemaps, GEO for AI Overviews, llms.txt generation, AI crawler audit, plus live GSC and Ahrefs MCP data. GSC: overview, drops, opportunities, brand-vs-nonbrand, cannibalization, compare, content-decay, index-issues, new-keywords. Ahrefs: overview, backlinks, keywords, competitors, content-gap, broken-links, new-links, anchor-analysis, dr-history, top-pages. Cross-MCP: serp, content-brief, brand-radar, site-audit-pro, report. Local: log-analysis, ai-content-check, internal-links, local, migration-check, obsidian-kb. AI Readability: llms-txt, robots-ai. Performance: lighthouse-audit, core-web-vitals, page-experience. Quality & AI Search: rich-results, ai-optimization, eeat-audit, helpful-content. Triggers on: "SEO", "audit", "schema", "Core Web Vitals", "sitemap", "E-E-A-T", "AI Overviews", "GEO", "technical SEO", "content quality", "structured data", "GSC", "Ahrefs", "backlinks", "keywords", "search console", "domain rating", "local SEO", "migration", "lighthouse", "pagespeed", "LCP", "INP", "CLS", "page experience", "interstitials", "mobile friendly", "rich results", "rich snippets", "schema eligibility", "AI optimization", "AI search readiness", "EEAT", "trust signals", "helpful content", "people-first", "content quality gate".

日本語の概要は準備中です。原文の説明を表示しています。

Audits and fixes both halves of being found — classic search and AI answers. Search side: rankings, site structure, page speed, titles and descriptions, and content that answers what customers search for. AI side: whether assistant crawlers can reach the site, pages render without JavaScript, an llms.txt file, schema.org data, and presence in the directories assistants read. For online stores it also runs a bulk catalog cleanup, rewriting product titles, descriptions, alt text, and SEO fields in approved batches. Works on any public website with no connector. Use this whenever the owner talks about being found online, including "nobody finds us on Google," "how do I show up in ChatGPT," "make my website AI friendly," "SEO audit," "GEO," "AEO," "AI search visibility," "my competitor ranks above me," "fix my product listings," or "I asked an AI about my industry and we weren't in the answer."

日本語の概要は準備中です。原文の説明を表示しています。

Audit whether a website can be found, crawled, and cited by AI answer engines such as ChatGPT Search, Perplexity, Google AI Overviews, and Microsoft Copilot. Use when someone asks why their brand is missing from AI answers, whether AI crawlers can read their site, how to get cited by ChatGPT or Perplexity, or asks for a GEO or AEO (generative / answer engine optimization) review. Produces a citation baseline across buyer-intent prompts, a crawler-access check, a citability review of named pages, and a ranked fix list. Not for keyword rank tracking, paid search, or pages behind a login.

日本語の概要は準備中です。原文の説明を表示しています。

WeChat Mini Program development skill for building, debugging, previewing, testing, publishing, and optimizing mini program projects (小程序开发、调试、预览、发布). Covers project structure and config (`project.config.json`, `appid`, `miniprogramRoot`, `tabBar`, routing/navigation, icon assets), WeChat Developer Tools Nightly workflows (`wechatide` CLI, WeChat IDE Skills/MCP), `miniprogram-ci` preview/upload, console/network debugging, message push (消息推送) and customer-service auto-reply (客服消息), mini program SEO / search indexing (小程序搜索优化、页面收录、搜索推广、mpcrawler), and CloudBase integration (`wx.cloud`, 腾讯云开发, 云开发) when explicitly used. Use when users create, develop, modify, debug, preview, deploy, publish, or promote WeChat Mini Programs. NOT for Web frontend (use web-development), pure backend services (use cloudrun-development / cloud-functions), or UI-design-only tasks (use ui-design).

日本語の概要は準備中です。原文の説明を表示しています。

Deploys custom Apify Actors via the Apify API v2 for large-scale web crawling using CrawleeJS. Leverages Apify dataset storage, RequestQueue, and proxy configuration for distributed scraping at scale.

日本語の概要は準備中です。原文の説明を表示しています。

Apify SDK is the official JavaScript SDK for building Actors, crawlers, and data extraction workflows on Apify. It gives agents a structured way to run scraping jobs, store outputs, manage inputs, and combine crawler logic with browser automation when needed.

日本語の概要は準備中です。原文の説明を表示しています。

Analyze credential-free public website monitoring evidence from server access logs, HTTP fetches, robots/llms/sitemap checks, and exported summaries while protecting raw IP and private query data. Use this skill when the user asks whether crawlers or AI bots reached a site, whether public SEO/LLM discovery files respond, or how to separate observed facts from unknown Search Console, rank, analytics, or citation claims.

日本語の概要は準備中です。原文の説明を表示しています。

Technical SEO audit across 8 categories: crawlability, indexability, security, URL structure, mobile, Core Web Vitals, structured data, JS rendering. Schema deep validation → seo-schema. AI crawlers → seo-geo-technical. Use when user says "technical SEO", "crawl issues", "robots.txt", "Core Web Vitals".

日本語の概要は準備中です。原文の説明を表示しています。

Optimize content for AI Overviews (formerly SGE), ChatGPT web search, Perplexity, and other AI-powered search experiences. Generative Engine Optimization (GEO) analysis including brand mention signals, AI crawler accessibility, llms.txt compliance, passage-level citability scoring, and platform-specific optimization. Use when user says "AI Overviews", "SGE", "GEO", "AI search", "LLM optimization", "Perplexity", "AI citations", "ChatGPT search", or "AI visibility".

日本語の概要は準備中です。原文の説明を表示しています。

GEO-first SEO analysis tool. Optimizes websites for AI-powered search engines (ChatGPT, Claude, Perplexity, Gemini, Google AI Overviews) while maintaining traditional SEO foundations. Performs full GEO audits, citability scoring, AI crawler analysis, llms.txt generation, brand mention scanning, platform-specific optimization, schema markup, technical SEO, content quality (E-E-A-T), and client-ready GEO report generation. Use when user says "geo", "seo", "audit", "AI search", "AI visibility", "optimize", "citability", "llms.txt", "schema", "brand mentions", "GEO report", or any URL for analysis.

日本語の概要は準備中です。原文の説明を表示しています。

Technical SEO audit with GEO-specific checks — crawlability, indexability, security, performance, SSR, and AI crawler access

日本語の概要は準備中です。原文の説明を表示しています。

Audit websites for Google AdSense application readiness and ad-serving compliance. Use when checking whether a site is likely to satisfy AdSense eligibility, site ownership, content quality, navigation, crawler access, ads.txt, privacy disclosure, Google Publisher Policies, AdSense Program policies, or when the user asks if a site can apply for AdSense, pass AdSense review, show ads, or fix AdSense rejection/site-not-ready issues.

日本語の概要は準備中です。原文の説明を表示しています。