scraping-tweets-by-keyword

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Scraping Tweets by Keyword

按关键词抓取推文

Raw tweet collection for any keyword, hashtag, or boolean search query. No assumed use case — returns the full tweet dataset for downstream analysis.

针对任意关键词、话题标签或布尔搜索查询收集原始推文。不预设使用场景——返回完整的推文数据集供下游分析使用。

Prerequisites

前提条件

```
APIFY_TOKEN
```
environment variable set
Optional: Apify MCP server installed

设置
```
APIFY_TOKEN
```
环境变量
可选：安装Apify MCP服务器

Inputs

输入参数

Parameter	Type	Required	Default	Notes
`searchTerms`	array	✅	`[]`	Twitter advanced search queries (e.g. `["#AI lang:en", "from:NASA"]` )
`sort`	string	Optional	`Top`	Sort order: `Latest` , `Top` , or `Latest+Top`
`tweetLanguage`	string	Optional	—	ISO 639-1 language code (e.g. `en` )
`maxItems`	number	Optional	Unlimited	Maximum tweets to return
`onlyVerifiedUsers`	boolean	Optional	`false`	Only tweets from verified users
`onlyTwitterBlue`	boolean	Optional	`false`	Only Twitter Blue subscribers
`onlyImage`	boolean	Optional	`false`	Only tweets with images
`onlyVideo`	boolean	Optional	`false`	Only tweets with videos
`onlyQuote`	boolean	Optional	`false`	Only quote tweets
`author`	string	Optional	—	Filter to a specific author handle
`inReplyTo`	string	Optional	—	Tweets replying to a specific handle
`mentioning`	string	Optional	—	Tweets mentioning a specific handle
`geotaggedNear`	string	Optional	—	Tweets near a location
`withinRadius`	string	Optional	—	Radius around geotaggedNear
`geocode`	string	Optional	—	Lat/lng + radius string
`placeObjectId`	string	Optional	—	Tweets tagged with a place
`minimumRetweets`	number	Optional	—	Minimum retweet count
`minimumFavorites`	number	Optional	—	Minimum like count
`minimumReplies`	number	Optional	—	Minimum reply count
`start`	string	Optional	—	Tweets after this date (YYYY-MM-DD)
`end`	string	Optional	—	Tweets before this date (YYYY-MM-DD)
`includeSearchTerms`	boolean	Optional	`false`	Add the matched search term to each tweet
`customMapFunction`	string	Optional	—	JavaScript function to transform each output object

参数	类型	是否必填	默认值	说明
`searchTerms`	array	✅	`[]`	Twitter高级搜索查询（例如 `["#AI lang:en", "from:NASA"]` ）
`sort`	string	可选	`Top`	排序方式： `Latest` （最新）、 `Top` （热门）或 `Latest+Top` （最新+热门）
`tweetLanguage`	string	可选	—	ISO 639-1语言代码（例如 `en` ）
`maxItems`	number	可选	无限制	返回的最大推文数量
`onlyVerifiedUsers`	boolean	可选	`false`	仅抓取认证用户的推文
`onlyTwitterBlue`	boolean	可选	`false`	仅抓取Twitter Blue订阅用户的推文
`onlyImage`	boolean	可选	`false`	仅抓取包含图片的推文
`onlyVideo`	boolean	可选	`false`	仅抓取包含视频的推文
`onlyQuote`	boolean	可选	`false`	仅抓取引用推文
`author`	string	可选	—	过滤特定作者的推文（需填写账号handle）
`inReplyTo`	string	可选	—	抓取回复特定账号的推文
`mentioning`	string	可选	—	抓取提及特定账号的推文
`geotaggedNear`	string	可选	—	抓取特定地点附近的推文
`withinRadius`	string	可选	—	`geotaggedNear` 参数对应的范围半径
`geocode`	string	可选	—	纬度/经度 + 半径字符串
`placeObjectId`	string	可选	—	抓取标记特定地点的推文
`minimumRetweets`	number	可选	—	最低转发数限制
`minimumFavorites`	number	可选	—	最低点赞数限制
`minimumReplies`	number	可选	—	最低回复数限制
`start`	string	可选	—	抓取该日期之后的推文（格式：YYYY-MM-DD）
`end`	string	可选	—	抓取该日期之前的推文（格式：YYYY-MM-DD）
`includeSearchTerms`	boolean	可选	`false`	在每条推文中添加匹配的搜索词
`customMapFunction`	string	可选	—	用于转换每个输出对象的JavaScript函数

Workflow

工作流程

Progress:
- [ ] Step 1: Build search query string
- [ ] Step 2: Run tweet-scraper
- [ ] Step 3: Poll for SUCCEEDED
- [ ] Step 4: Fetch and deliver dataset

进度:
- [ ] 步骤1：构建搜索查询字符串
- [ ] 步骤2：运行推文爬虫
- [ ] 步骤3：轮询直到状态为SUCCEEDED
- [ ] 步骤4：获取并交付数据集

Step 1: Build Search Query

步骤1：构建搜索查询

Hashtag search →
```
#keyword
```
Exact phrase →
```
"exact phrase"
```
Boolean →
```
word1 AND word2 -exclude
```
From account →
```
from:username
```
Mention →
```
@username
```

话题标签搜索 →
```
#keyword
```
精确短语 →
```
"exact phrase"
```
布尔查询 →
```
word1 AND word2 -exclude
```
指定账号 →
```
from:username
```
提及账号 →
```
@username
```

Step 2: Run the Actor

步骤2：运行Actor

Recommended — run_actor.js (handles waiting, output, and file saving automatically):

bash

undefined

推荐方式 — run_actor.js（自动处理等待、输出和文件保存）：

bash

undefined

Quick answer (prints table to chat)

快速查看结果（在聊天窗口打印表格）

node scripts/run_actor.js
--actor "apidojo~tweet-scraper"
--input '{"param": "value"}'

Save as CSV

保存为CSV格式

node scripts/run_actor.js
--actor "apidojo~tweet-scraper"
--input '{"param": "value"}'
--output YYYY-MM-DD_results.csv --format csv

Save as JSON

保存为JSON格式

node scripts/run_actor.js
--actor "apidojo~tweet-scraper"
--input '{"param": "value"}'
--output YYYY-MM-DD_results.json --format json

> `APIFY_TOKEN` must be set in environment or `.env` file.

**If Apify MCP is available:**

Tool: apify:run-actor Actor: "apidojo~tweet-scraper" Input: { "searchTerms": ["<query>"], "maxItems": 200, "since": "<YYYY-MM-DD>", "lang": "<lang_code>" }


**REST API fallback:**
```bash
curl -X POST \
  "https://api.apify.com/v2/acts/apidojo~tweet-scraper/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"searchTerms": ["<query>"], "maxItems": 200}'

Save

id

RUN_ID

. Poll until

status = SUCCEEDED

bash

curl "https://api.apify.com/v2/actor-runs/$RUN_ID?token=$APIFY_TOKEN" | grep '"status"'

Fetch results:

bash

curl "https://api.apify.com/v2/actor-runs/$RUN_ID/dataset/items?token=$APIFY_TOKEN&format=json"

node scripts/run_actor.js
--actor "apidojo~tweet-scraper"
--input '{"param": "value"}'
--output YYYY-MM-DD_results.json --format json

> 必须在环境变量或`.env`文件中设置`APIFY_TOKEN`。

**如果Apify MCP可用：**

工具: apify:run-actor Actor: "apidojo~tweet-scraper" 输入: { "searchTerms": ["<query>"], "maxItems": 200, "since": "<YYYY-MM-DD>", "lang": "<lang_code>" }


**REST API备选方案：**
```bash
curl -X POST \
  "https://api.apify.com/v2/acts/apidojo~tweet-scraper/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"searchTerms": ["<query>"], "maxItems": 200}'

将返回的

id

保存为

RUN_ID

，轮询直到

status = SUCCEEDED

：

bash

curl "https://api.apify.com/v2/actor-runs/$RUN_ID?token=$APIFY_TOKEN" | grep '"status"'

获取结果：

bash

curl "https://api.apify.com/v2/actor-runs/$RUN_ID/dataset/items?token=$APIFY_TOKEN&format=json"

Step 3: Handle Edge Cases

步骤3：处理边缘情况

0 results: Query may be too narrow, misspelled, or language-filtered. Broaden term, remove language filter, extend date range.
< 20 results: Try removing
```
since
```
/
```
until
```
constraints. Some low-volume terms have sparse data.
Duplicate tweet IDs: Deduplicate by
```
id
```
field before delivering.
Suspended/deleted accounts: Tweets from suspended accounts return with empty author fields — flag these rows.

0条结果：查询范围可能过窄、拼写错误或语言过滤过严。建议扩大关键词范围、移除语言过滤、延长日期范围。
结果少于20条：尝试移除
```
since
```
/
```
until
```
约束。部分低热度关键词的数据较为稀疏。
重复推文ID：交付前通过
```
id
```
字段去重。
账号被封禁/删除：来自被封禁账号的推文会返回空作者字段——标记这些行。

Output Format

输出格式

undefined

undefined

Tweet Dataset: "<query>"

推文数据集: "<query>"

Total collected: N | Date range: SINCE – UNTIL | Language: LANG

Tweet ID	Author	Text (truncated)	Likes	Retweets	Replies	Timestamp
...	...	...	...	...	...	...

Full dataset: N rows × 15 fields Available fields: id, text, author_id, author_username, likeCount, retweetCount, replyCount, quoteCount, lang, createdAt, tweetUrl, media, isRetweet, isQuote, source

undefined

总收集数量: N | 日期范围: SINCE – UNTIL | 语言: LANG

推文ID	作者	文本（截断）	点赞数	转发数	回复数	时间戳
...	...	...	...	...	...	...

完整数据集: N行 × 15字段可用字段: id, text, author_id, author_username, likeCount, retweetCount, replyCount, quoteCount, lang, createdAt, tweetUrl, media, isRetweet, isQuote, source

undefined

Troubleshooting

故障排除

Empty results for a valid hashtag: Twitter API indexing lag — try again after 15 minutes. Rate limit error: Reduce

maxItems

to 100 and retry. Timeout on large requests: Set

maxItems: 500

max per run; chain multiple runs with date ranges for larger datasets.

有效话题标签返回空结果：Twitter API索引延迟——15分钟后重试。 速率限制错误：将

maxItems

降至100后重试。 大请求超时：每次运行设置

maxItems: 500

为上限；通过日期范围拆分任务，链式执行多次运行以获取更大数据集。