「scraping」の検索結果

Browser automation powers web testing, scraping, and AI agent interactions. The difference between a flaky script and a reliable system comes down to understanding selectors, waiting strategies, and anti-detection patterns. This skill covers Playwright (recommended) and Puppeteer, with patterns for testing, scraping, and agentic browser control. Key insight: Playwright won the framework war. Unless you need Puppeteer's stealth ecosystem or are Chrome-only, Playwright is the better choice in 2025. Critical distinction: Testing automation (predictable apps you control) vs scraping/agent automation (unpredictable sites that fight back). Different problems, different solutions. Use when "playwright, puppeteer, browser automation, headless, web scraping, e2e test, end-to-end, selenium, chromium, browser test, page.click, locator, playwright, puppeteer, browser, automation, testing, scraping, headless, e2e, selenium, web-testing" mentioned.

日本語の概要は準備中です。原文の説明を表示しています。

Complete knowledge domain for Firecrawl v2 API - web scraping and crawling that converts websites into LLM-ready markdown or structured data. Use when: scraping websites, crawling entire sites, extracting web content, converting HTML to markdown, building web scrapers, handling dynamic JavaScript content, bypassing anti-bot protection, extracting structured data from web pages, or when encountering "content not loading", "JavaScript rendering issues", or "blocked by bot detection". Keywords: firecrawl, firecrawl api, web scraping, web crawler, scrape website, crawl website, extract content, html to markdown, site crawler, content extraction, web automation, firecrawl-py, firecrawl-js, llm ready data, structured data extraction, bot bypass, javascript rendering, scraping api, crawling api, map urls, batch scraping

日本語の概要は準備中です。原文の説明を表示しています。

Scrapes tweets from X (Twitter) by search query, user handle, or direct URL — returns full tweet data including text, author info, engagement metrics, media, and hashtags. Use when user mentions X, Twitter, tweet scraping, scrape tweets, get tweets, fetch tweets, twitter data, x.com data, tweet collector, tweet search, twitter search, social media scraping, twitter handle scraping, scrape user tweets, from:username, twitter keyword search, hashtag search, twitter timeline, twitter list scraping, collect twitter posts, export tweets, tweet analysis, twitter monitoring, search twitter, get twitter data, twitter api alternative, tweet dump.

日本語の概要は準備中です。原文の説明を表示しています。

Escalate blocked scraping campaigns with provider-neutral proxy rotation — decide between datacenter, residential, and mobile pools, integrate rotation with scrapling, configure session stickiness for stateful flows, monitor cost and health, and stay inside legal and ethical boundaries. Use as the next step after `headless-web-scraping` client-side stealth (StealthyFetcher, rate limiting, robots.txt) is insufficient and traffic is legitimate.

日本語の概要は準備中です。原文の説明を表示しています。

4-tier progressive web scraping that auto-escalates WebFetch to curl to Interceptor to Bright Data proxy for bot detection and CAPTCHAs, with single-URL and multi-page crawl modes, output as markdown. USE WHEN Bright Data, scrape URL, web scraping, bot detection, crawl site, CAPTCHA, can't access, site blocking, extract page content, scrape whole site, spider domain, convert URL to markdown, getting blocked. NOT FOR simple public content (use WebFetch directly), social platform scraping with named actors (use Apify), or real-Chrome bot bypass with logged-in sessions and zero CDP fingerprint (use Interceptor).

日本語の概要は準備中です。原文の説明を表示しています。

Web scraping with anti-bot bypass, content extraction, undocumented APIs and poison pill detection. Use when extracting content from websites, handling paywalls, implementing scraping cascades or processing social media. Covers requests, trafilatura, Playwright with stealth mode, yt-dlp and instaloader patterns.

日本語の概要は準備中です。原文の説明を表示しています。

Production-grade architecture patterns for Apify-powered applications. Use when designing scraping infrastructure, building multi-Actor pipelines, or integrating Apify into a larger system architecture. Trigger with "apify architecture", "apify best practices", "apify project structure", "scraping architecture", "apify system design".

日本語の概要は準備中です。原文の説明を表示しています。

Build a complete web scraping Actor with Crawlee and deploy to Apify. Use when you need end-to-end web scraping on Apify: defining an input schema, building a router-based Crawlee crawler, extracting structured data, storing results in a dataset, testing locally, and deploying the Actor to the platform. Trigger with "apify scrape website", "build apify actor", "crawlee scraper", "apify main workflow".

日本語の概要は準備中です。原文の説明を表示しています。

Deploy Apify Actors and integrate scraping into external applications. Use when deploying an Actor to the platform, integrating Actor results into a web app (Next.js, Express), wiring webhooks, or scheduling scraping pipelines. Trigger with "deploy apify actor", "apify Vercel integration", "apify production deploy", "integrate apify results", "apify API endpoint".

日本語の概要は準備中です。原文の説明を表示しています。

Assesses lawful basis for AI training data processing per EDPB April 2025 report on LLMs and general-purpose AI. Covers legitimate interest balancing tests, consent challenges for ML training, public dataset assessment, and web scraping lawfulness. Keywords: AI training data, lawful basis, EDPB LLM, legitimate interest, consent, web scraping.

日本語の概要は準備中です。原文の説明を表示しています。

Build reliable web scrapers and crawlers with Crawlee — Apify's open-source framework for structured web scraping. Use when someone asks to "scrape a website", "build a crawler", "Crawlee", "web scraping at scale", "scrape JavaScript-rendered pages", "crawl with Playwright/Puppeteer", or "extract data from websites reliably". Covers HTTP crawling, browser crawling, request queues, proxy rotation, and data export.

日本語の概要は準備中です。原文の説明を表示しています。

Bright Data Web MCP gives MCP-compatible agents live web search, scraping, and optional browser-automation access through Bright Data’s web-access platform. It is built for agents that need current information, anti-bot resilience, and structured extraction without maintaining their own scraping stack.

日本語の概要は準備中です。原文の説明を表示しています。

Stealth headless browser with CDP server for anti-fingerprint browsing, parallel scraping, and low-memory CDP automation. This skill should be used when stealth browsing, anti-detection scraping, headless CDP automation with fingerprint protection, or tracker-free page fetching is needed. Triggers on 'stealth browse', 'anti-fingerprint', 'obscura', 'stealth CDP', 'undetectable scraping', 'headless stealth', 'tracker-free browsing'.

日本語の概要は準備中です。原文の説明を表示しています。

CLI tool for AI-powered web scraping, data extraction, search, and crawling via ScrapeGraph AI. Use when the user needs to scrape websites, extract structured data from URLs, convert pages to markdown, crawl multi-page sites, search the web for information, automate browser interactions (login, click, fill forms), get raw HTML, discover sitemaps, or generate JSON schemas. Triggers on tasks involving: (1) extracting data from websites, (2) web scraping or crawling, (3) converting webpages to markdown, (4) AI-powered web search with extraction, (5) browser automation, (6) generating output schemas for scraping. The CLI is just-scrape (npm package just-scrape).

日本語の概要は準備中です。原文の説明を表示しています。

Integrate Firecrawl into application code whenever a product, agent, or workflow needs web data inside the app — web search, live search results, page scraping, structured extraction, or browser interaction. Use when building any feature that needs data from the web in code, even if the user does not mention Firecrawl explicitly and only describes wanting web data, website content, search, scraping, or interaction in an application. Trigger for Firecrawl requests, "fire girl" shorthand, and generic app-level web-data needs that should map to `/scrape`, `/search`, or `/interact`. Do not use this skill for one-off terminal-only web tasks during the current session; use `firecrawl/cli` for those.

日本語の概要は準備中です。原文の説明を表示しています。

Collects every tweet in an X (Twitter) conversation thread given a conversation id (root tweet id) — the focal tweet plus all replies, sub-replies, and quote chains — and returns normalized per-tweet data with text, author, engagement counts, media, hashtags, mentions, in_reply_to mapping, and cursor for pagination. Use when user mentions Twitter conversation, X conversation thread, conversation_id, thread scraper, scrape Twitter replies, all replies to a tweet, replies under a tweet, sub-replies, nested replies, thread harvester, get replies of a tweet, scrape comments on Twitter, scrape comments on X, full thread extraction, conversation export, conversation tree, reply chain, thread dump, X tweet thread, twitter thread scrape, comment scraping twitter, comment scraping x, focal tweet plus context, root tweet plus replies. Also applies to sentiment analysis on a single viral tweet, controversy mapping, harvesting community Q&A threads, capturing AMA threads, recovering long-running discussions, and any paginated bulk reply collection driven by a conversation id.

日本語の概要は準備中です。原文の説明を表示しています。

Extract complete product information from any e-commerce product page. Returns name, price, currency, brand, images, description, SKU/ASIN/EAN/UPC/GTIN/MPN identifiers, stock availability, rating, review count, variants, and seller. Works on Shopify, Amazon, WooCommerce, eBay, Walmart, Etsy, AliExpress, Alibaba, Target, Best Buy, Rakuten, Magento, BigCommerce, PrestaShop, and any public e-commerce site. Accepts product URL, keyword, or product identifier (SKU/ASIN/EAN/UPC). Use when: scrape product page, get product details, extract price and availability, product info extraction, check product data, product detail scraping, get product from URL, keyword product search, ASIN lookup, EAN search, UPC lookup, price check, product research, compare products, monitor product price, get product images, product brand and description, ecommerce data extraction, product catalog scraping, product page scraper, get item price, fetch product info.

日本語の概要は準備中です。原文の説明を表示しています。

Implement Apify webhooks for Actor run notifications and event-driven pipelines. Use when setting up run completion alerts, building event-driven scraping pipelines, or configuring ad-hoc webhooks for individual runs. Trigger with "apify webhook", "apify events", "actor run notification", "apify run succeeded webhook", "apify ad-hoc webhook".

日本語の概要は準備中です。原文の説明を表示しています。

Web data extraction using 55+ Apify Actors for AI-driven scraping. Supports Instagram, Facebook, TikTok, YouTube, Google, and more. Auto-selects best Actor for the task. Structured output in JSON/CSV with rate limiting and ethical scraping guidelines.

日本語の概要は準備中です。原文の説明を表示しています。

Compatibility alias for the original ScrapeCreators social scraping skill. Use scrapecreators-api for current endpoint guidance and the task-specific Brand Growth research skills for finished analysis.

日本語の概要は準備中です。原文の説明を表示しています。

Web scraping, crawling, and AI-powered answer extraction at scale

日本語の概要は準備中です。原文の説明を表示しています。

Web scraping with structured data extraction - define your output schema

日本語の概要は準備中です。原文の説明を表示しています。

AI-powered web scraping - extract data using natural language prompts

日本語の概要は準備中です。原文の説明を表示しています。

This skill should be used when the user asks to "build web automation scripts", "check browser automation for detection", "generate web scraping code", "create form filling automation", or "build anti-detection browser scripts".

日本語の概要は準備中です。原文の説明を表示しています。