30 件 ・ 関連度順
概要と使いどころ
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
日本語の概要は準備中です。原文の説明を表示しています。
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
日本語の概要は準備中です。原文の説明を表示しています。
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
日本語の概要は準備中です。原文の説明を表示しています。
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
日本語の概要は準備中です。原文の説明を表示しています。
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
日本語の概要は準備中です。原文の説明を表示しています。
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
日本語の概要は準備中です。原文の説明を表示しています。
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
日本語の概要は準備中です。原文の説明を表示しています。
文章をAIが扱う小さな単位に分け、大量のテキスト処理や独自の語彙学習を支援するスキル。分割結果と元の文章の位置対応、入力の長さ調整も扱います。
PCI DSS v4.0 payment card data security compliance, assessment, and implementation. Use for PCI DSS scoping, cardholder data environment (CDE) security, SAQ and ROC preparation, QSA engagement, tokenization, and merchant compliance.
日本語の概要は準備中です。原文の説明を表示しています。
Assessment of pseudonymization techniques and re-identification risk. Covers tokenization, hashing, encryption-based pseudonymization, and hybrid approaches. Includes re-identification risk scoring using the motivated intruder test, quantitative metrics (marketer, journalist, prosecutor models), and linkage attack resilience evaluation. References ENISA 2019 pseudonymization report.
日本語の概要は準備中です。原文の説明を表示しています。
ベンガル語の検索やAI向けに、文字表現の統一、単語分割、数字変換、結合文字を壊さない表示処理を整理し、学習データの整備や保存にも役立つスキル。
Supports Gtars for local genomic interval models and set algebra, overlaps and counts, consensus and coverage, tokenization, fragment processing, and refget/BEDbase planning across Python, Rust, and the CLI.
日本語の概要は準備中です。原文の説明を表示しています。
Add or verify a model's chat prompt rendering and tokenization in rust/sglang-processor so it matches SGLang's Python serving path exactly (same prompt text, same token ids). Use when adding a model to sglang-processor, porting a model's rendering from sgl-router, bumping Dynamo's renderer, or debugging a processor-vs-Python prompt mismatch.
日本語の概要は準備中です。原文の説明を表示しています。
High-performance toolkit for genomic interval analysis in Rust with Python bindings. Use when working with genomic regions, BED files, coverage tracks, overlap detection, tokenization for ML models, or fragment analysis in computational genomics and machine learning applications.
日本語の概要は準備中です。原文の説明を表示しています。
A high-performance Rust toolkit (with Python bindings and a CLI) for genomic interval analysis; use it when you need fast overlap queries, coverage track generation, genomic tokenization for ML, reference sequence verification, or fragment processing.
日本語の概要は準備中です。原文の説明を表示しています。
Audit applications and infrastructure handling payment card data against PCI DSS v4.0. Heavy emphasis on scope determination (the single most-leveraged variable) plus the engineering-relevant requirements — Req 3 (storage of CHD), Req 4 (transmission), Req 6 (secure SDLC), Req 7-8 (access), Req 10 (logging), Req 11 (testing), Req 12 (program). Use when the user mentions 'PCI,' 'PCI DSS,' 'PCI DSS 4.0,' 'payment card,' 'cardholder data,' 'CHD,' 'PAN,' 'PCI scope,' 'PCI compliance,' 'SAQ,' 'AoC,' 'attestation of compliance,' 'tokenization,' 'P2PE,' 'network segmentation for PCI,' or audits any system that stores, processes, or transmits payment card data.
日本語の概要は準備中です。原文の説明を表示しています。
High-performance toolkit for genomic interval analysis in Rust with Python bindings. Use when working with genomic regions, BED files, coverage tracks, overlap detection, tokenization for ML models, or fragment analysis in computational genomics and machine learning applications.
日本語の概要は準備中です。原文の説明を表示しています。
Rust-backed Python library for fast genomic token arithmetic and BED processing. High-performance BED I/O, interval set ops (intersect, merge, complement, subtract), region tokenization against a universe, universe construction. Use for preprocessing large BED collections and ML token vocabularies.
日本語の概要は準備中です。原文の説明を表示しています。
HuggingFace Transformers with biomedical LMs (BioBERT, PubMedBERT, BioGPT, BioMedLM) for scientific NLP: NER (genes, diseases, chemicals), relation extraction, QA, text classification, abstract summarization. Covers loading, biomedical tokenization, inference pipelines, fine-tuning. Alternatives: spaCy en_core_sci_lg (rule-based NER), Stanza (biomedical models), NLTK.
日本語の概要は準備中です。原文の説明を表示しています。
Applies the mental models and frameworks of Andrej Karpathy (deep learning, former Director of AI at Tesla, founding member of OpenAI, Eureka Labs). Use this skill whenever you are helping the user build neural networks from scratch, debug deep learning pipelines, evaluate AI agent workflows, design LLM apps, or navigate the transition to Software 3.0 (vibe coding). It is highly relevant for pedagogy (untangling complex knowledge), assessing AI capabilities vs. limitations (jagged intelligence, tokenization limits), and architectural decisions (end-to-end optimization vs. complex pipelines). Reach for this whenever discussing LLM training, autonomous systems, or AI-assisted coding.
日本語の概要は準備中です。原文の説明を表示しています。
Applies the mental models and frameworks of Andrej Karpathy (deep learning, former Director of AI at Tesla, founding member of OpenAI, Eureka Labs). Use this skill whenever you are helping the user build neural networks from scratch, debug deep learning pipelines, evaluate AI agent workflows, design LLM apps, or navigate the transition to Software 3.0 (vibe coding). It is highly relevant for pedagogy (untangling complex knowledge), assessing AI capabilities vs. limitations (jagged intelligence, tokenization limits), and architectural decisions (end-to-end optimization vs. complex pipelines). Reach for this whenever discussing LLM training, autonomous systems, or AI-assisted coding.
日本語の概要は準備中です。原文の説明を表示しています。
Use Neo4j GenAI Plugin ai.text.* functions and procedures for in-Cypher embedding generation, text completion, structured output, chat, tokenization, and batch ingestion. Covers ai.text.embed(), ai.text.embedBatch(), ai.text.completion(), ai.text.structuredCompletion(), ai.text.aggregateCompletion(), ai.text.chat(), ai.text.tokenCount(), ai.text.chunkByTokenLimit(), and provider configuration for OpenAI, Azure OpenAI, VertexAI, and Amazon Bedrock. Requires CYPHER 25. Replaces deprecated genai.vector.encode(). Use when writing pure-Cypher GraphRAG, embedding nodes in-graph, generating structured maps from prompts, or calling LLMs inside Cypher queries. Does NOT handle neo4j-graphrag Python library pipelines — use neo4j-graphrag-skill. Does NOT handle vector index creation/search — use neo4j-vector-index-skill.
日本語の概要は準備中です。原文の説明を表示しています。
Fintech engineering expertise — payment processing (Stripe, Plaid), PCI DSS compliance, financial data modeling (double-entry bookkeeping), fraud detection patterns, bank-grade security (encryption, tokenization), open banking APIs, cryptocurrency/blockchain integration, regulatory compliance (KYC/AML), and idempotent financial transaction design. Use for payment systems, banking apps, trading platforms, and fintech infrastructure.
日本語の概要は準備中です。原文の説明を表示しています。
Tokenizador agnóstico de linguagem que trata texto como Unicode bruto. Suporta algoritmos BPE e Unigram. Rápido (50k sentenças/seg), leve (6MB de memória), vocabulário determinístico. Usado por T5, ALBERT, XLNet, mBART. Treina em texto bruto sem pré-tokenização. Use quando você precisar de suporte multilíngue, linguagens CJK ou tokenização reproduzível.
日本語の概要は準備中です。原文の説明を表示しています。