count-dataset-tokens

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Count Dataset Tokens

数据集Token计数

Overview

概述

This skill provides a systematic approach for accurately counting tokens in datasets. It emphasizes thorough data exploration, proper interpretation of task requirements, and verification of results to avoid common mistakes like incomplete field coverage or misinterpreting terminology.

本方法提供了一套准确计算数据集中token数量的系统化流程。它强调全面的数据探索、对任务需求的正确解读以及结果验证，以此避免字段覆盖不全或术语解读错误等常见问题。

When to Use This Skill

适用场景

Counting tokens in HuggingFace datasets or similar data sources
Tasks involving tokenization of text fields
Filtering datasets by domain, category, or other metadata
Working with datasets that have multiple text fields that may contribute to token counts
Any task requiring accurate quantification of textual content in structured datasets

对HuggingFace数据集或类似数据源进行token计数
涉及文本字段token化的任务
按领域、类别或其他元数据过滤数据集
处理包含多个可能计入token统计的文本字段的数据集
任何需要对结构化数据集中的文本内容进行准确定量的任务

Critical Pre-Implementation Steps

关键预实施步骤

1. Clarify Terminology Before Proceeding

1. 先明确术语含义再开展工作

When a task uses specific terms (e.g., "deepseek tokens", "science domain"), verify exactly what content this refers to:

Examine the README/documentation thoroughly - Documentation often contains critical definitions
List all available fields in the dataset schema before making assumptions
Identify all fields that could potentially be relevant to the token count
Do not assume field names tell the complete story - A field like
```
deepseek_reasoning
```
may not be the only field relevant for counting "deepseek tokens"

当任务中使用特定术语（如“deepseek tokens”、“science domain”）时，需确认这些术语具体指代的内容：

仔细查看README/文档 - 文档中通常包含关键定义
列出数据集schema中的所有可用字段，避免主观假设
识别所有可能与token计数相关的字段
不要仅通过字段名称判断全部含义 - 例如
```
deepseek_reasoning
```
字段可能并非“deepseek tokens”计数的唯一相关字段

2. Explore Dataset Structure Thoroughly

2. 全面探索数据集结构

Before writing any counting logic:

1. Load a sample of the dataset
2. Print ALL column names and their types
3. Examine multiple sample entries in full detail
4. Identify relationships between fields
5. Check for nested structures or JSON fields
6. Look for metadata columns that might indicate which fields to include

在编写任何计数逻辑之前：

1. 加载数据集样本
2. 打印所有列名及其类型
3. 详细检查多个样本条目
4. 识别字段间的关联关系
5. 检查是否存在嵌套结构或JSON字段
6. 查找可能指示需包含字段的元数据列

3. Understand Domain/Category Mappings

3. 理解领域/类别映射关系

When filtering by categories like "science":

List all unique values in the domain/category column
Determine if the target category is an explicit value OR a grouping of related values
Example: "science" might mean
```
biology + chemistry + physics
```
rather than a literal "science" value
Document your interpretation and verify it aligns with the task intent

当按“science”等类别过滤数据时：

列出领域/类别列中的所有唯一值
确定目标类别是明确的单个值还是相关值的组合
示例：“science”可能指
```
biology + chemistry + physics
```
，而非字面意义上的“science”值
记录你的解读，并确认其与任务意图一致

Implementation Workflow

实施流程

Step 1: Data Discovery

步骤1：数据探索

1. Load the dataset (or a representative sample)
2. Enumerate all columns/fields
3. For each text field, examine:
   - Field name and description
   - Sample content from multiple entries
   - Whether it should be included in token counts
4. Check for a metadata or schema subset that documents field purposes

1. 加载数据集（或具有代表性的样本）
2. 枚举所有列/字段
3. 对每个文本字段，检查：
   - 字段名称和描述
   - 多个条目中的样本内容
   - 是否应纳入token计数
4. 查找是否有记录字段用途的元数据或schema子集

Step 2: Define Scope Explicitly

步骤2：明确定义范围

Before counting, explicitly document:

Which fields will be tokenized
Which filter criteria will be applied (e.g., domain == "biology")
The tokenizer to be used and why
Any fields being excluded and the reasoning

在开始计数前，需明确记录：

哪些字段将被token化
应用的过滤条件（如domain == "biology"）
使用的tokenizer及其原因
排除的字段及理由

Step 3: Implement with Verification Points

步骤3：带验证点的实施

1. Load the appropriate tokenizer
2. Apply filters to select relevant entries
3. For each entry, tokenize ALL relevant fields
4. Sum token counts with running totals
5. Print progress checkpoints (e.g., every 1000 entries)
6. Track statistics: entry count, empty fields, errors

1. 加载合适的tokenizer
2. 应用过滤条件选择相关条目
3. 对每个条目，tokenize所有相关字段
4. 累加token计数并记录运行总计
5. 打印进度检查点（如每处理1000条记录）
6. 跟踪统计信息：条目数量、空字段、错误情况

Step 4: Validate Results

步骤4：验证结果

Before reporting final numbers:

Spot-check individual entries - Manually verify token counts for 3-5 random samples
Sanity check totals - Does the average tokens per entry seem reasonable?
Cross-reference with metadata - If the dataset provides expected statistics, compare
Verify filter results - Confirm the filtered count matches expected entries

在报告最终数值前：

抽查单个条目 - 手动验证3-5个随机样本的token计数
合理性检查总计 - 每条记录的平均token数是否合理？
与元数据交叉核对 - 如果数据集提供了预期统计数据，进行对比
验证过滤结果 - 确认过滤后的条目数量符合预期

Common Pitfalls to Avoid

需避免的常见误区

Pitfall 1: Incomplete Field Identification

误区1：字段识别不完整

Mistake: Assuming a single field (e.g.,

deepseek_reasoning

) contains all relevant content

Solution:

Examine the full schema before deciding which fields to include
Consider whether multiple fields contribute to the "complete" content
Check if there are fields like
```
prompt
```
,
```
response
```
,
```
full_text
```
, or
```
conversation
```
that should be included

错误做法：假设单个字段（如

deepseek_reasoning

）包含所有相关内容

解决方案：

在决定纳入哪些字段前，先检查完整的schema
考虑是否有多个字段共同构成“完整”内容
检查是否存在
```
prompt
```
、
```
response
```
、
```
full_text
```
或
```
conversation
```
等需纳入的字段

Pitfall 2: Ambiguous Terminology

误区2：术语解读模糊

Mistake: Interpreting "deepseek tokens" as "tokens in the deepseek_reasoning field only"

Solution:

Research what the terminology means in the dataset's context
Read any available documentation or README files completely
When uncertain, consider multiple interpretations and document your choice

错误做法：将“deepseek tokens”解读为“仅

deepseek_reasoning

字段中的tokens”

解决方案：

研究该术语在数据集语境中的含义
完整阅读所有可用文档或README文件
若存在不确定性，考虑多种解读并记录你的选择

Pitfall 3: Assuming Category Names Are Literal

误区3：假设类别名称为字面含义

Mistake: Looking for

domain == "science"

when science is actually a group of domains

Solution:

Always enumerate unique values in category/domain fields first
Understand the taxonomy before applying filters
Common groupings: science (biology, chemistry, physics), stem (science + math), humanities

错误做法：当“science”实际是多个领域的组合时，仍按

domain == "science"

进行过滤

解决方案：

先枚举类别/领域列中的所有唯一值
在应用过滤前理解其分类体系
常见组合：science（biology、chemistry、physics）、stem（science + math）、人文科学

Pitfall 4: Not Validating Intermediate Results

误区4：未验证中间结果

Mistake: Running a complete count without checking partial results

Solution:

Process in batches with intermediate output
Verify token counts for sample entries manually
Compare against any available reference statistics

错误做法：直接运行完整计数而不检查部分结果

解决方案：

分批次处理并输出中间结果
手动验证样本条目的token计数
与任何可用的参考统计数据进行对比

Verification Checklist

验证 checklist

Example Exploration Code

探索代码示例

When starting a dataset token counting task, use exploratory code like:

python

undefined

启动数据集token计数任务时，可使用如下探索性代码：

python

undefined

Initial exploration

from datasets import load_dataset

Load a small sample first

ds = load_dataset("dataset_name", split="train[:100]")

Print all column names

print("Columns:", ds.column_names)

Examine a single entry in full

print("\nSample entry:") for key, value in ds[0].items(): print(f" {key}: {type(value).name} = {str(value)[:200]}...")

Check for domain/category distributions if filtering

if 'domain' in ds.column_names: from collections import Counter domains = Counter(ds['domain']) print("\nDomain distribution:", domains)

undefined

if 'domain' in ds.column_names: from collections import Counter domains = Counter(ds['domain']) print("\nDomain distribution:", domains)

undefined

Key Principles

核心原则

Explore before implementing - Understand the full data structure first
Clarify ambiguity explicitly - Don't assume; document interpretations
Verify incrementally - Check results at multiple stages
Consider all relevant fields - Token counts often span multiple columns
Read documentation thoroughly - READMEs contain critical context

先探索再实施 - 先全面理解数据结构
明确消除歧义 - 不要主观假设，记录你的解读
逐步验证 - 在多个阶段检查结果
考虑所有相关字段 - Token计数通常涉及多个列
仔细阅读文档 - README包含关键上下文