AutoCLI Web Scraping Skill

Skill by ara.so — Devtools Skills collection.

AutoCLI is a blazing fast, memory-safe command-line tool written in Rust that fetches information from 55+ websites with a single command. It covers Twitter/X, Reddit, YouTube, HackerNews, Bilibili, Zhihu, Xiaohongshu, and more, with support for browser session reuse, AI-powered adapter generation, and multi-format output.

Key Features:

55 sites, 333 commands built-in
Browser session reuse (no token management needed)
AI-powered adapter generation via autocli.ai
Declarative YAML pipeline for custom adapters
Single 4.7MB binary, zero runtime dependencies
Up to 12x faster and 10x less memory than Node.js alternatives

Installation

One-line Install (macOS / Linux)

bash

curl -fsSL https://raw.githubusercontent.com/nashsu/autocli/main/scripts/install.sh | sh

Manual Installation

Download the appropriate binary from GitHub Releases:

macOS (Apple Silicon):
```
autocli-aarch64-apple-darwin.tar.gz
```
macOS (Intel):
```
autocli-x86_64-apple-darwin.tar.gz
```

Linux (x86_64):

autocli-x86_64-unknown-linux-musl.tar.gz

Windows (x64):
```
autocli-x86_64-pc-windows-msvc.zip
```

Extract and place in your PATH:

bash

tar -xzf autocli-*.tar.gz
sudo mv autocli /usr/local/bin/

Chrome Extension (Required for Browser Commands)

Download
```
autocli-chrome-extension.zip
```
from releases
Extract to any directory
Open
```
chrome://extensions
```
Enable "Developer mode"
Click "Load unpacked" and select the extracted folder

Public API commands (hackernews, devto, etc.) work without the extension.

Basic Usage

Discovery Commands

bash

# List all available commands
autocli --help

# List commands for specific site
autocli twitter --help
autocli bilibili --help

# Run diagnostics
autocli doctor

# List all sites and their command counts
autocli list

Public API Commands (No Browser Required)

bash

# Hacker News top stories
autocli hackernews top --limit 10

# Hacker News search
autocli hackernews search "rust" --limit 5

# Dev.to top articles
autocli devto top --limit 10

# Lobsters hot stories
autocli lobsters hot --limit 10

# Stack Overflow hot questions
autocli stackoverflow hot --limit 10

# arXiv paper search
autocli arxiv search "machine learning" --limit 5

# Wikipedia search
autocli wikipedia search "rust programming"

# Linux.do forum hot topics
autocli linux-do hot --limit 10

Browser-Based Commands (Requires Extension + Login)

bash

# Twitter trending topics
autocli twitter trending

# Twitter search
autocli twitter search "rust lang" --limit 10

# Twitter timeline
autocli twitter timeline --limit 20

# Reddit frontpage
autocli reddit frontpage --limit 10

# Reddit subreddit posts
autocli reddit subreddit rust --limit 15

# Bilibili hot videos
autocli bilibili hot --limit 20

# Bilibili search
autocli bilibili search "rust programming" --limit 10

# Xiaohongshu search
autocli xiaohongshu search "travel" --limit 10

# Zhihu hot topics
autocli zhihu hot --limit 10

# YouTube search
autocli youtube search "rust tutorial" --limit 5

# Weibo hot topics
autocli weibo hot --limit 10

Output Formats

AutoCLI supports multiple output formats for easy integration with other tools:

bash

# Table format (default)
autocli hackernews top --limit 5

# JSON output
autocli hackernews top --limit 5 --format json

# YAML output
autocli hackernews top --limit 5 --format yaml

# CSV output
autocli hackernews top --limit 5 --format csv

# Markdown output
autocli hackernews top --limit 5 --format markdown

JSON Processing with jq

bash

# Extract specific fields
autocli hackernews top --limit 5 --format json | jq '.[].title'

# Filter by score
autocli hackernews top --limit 20 --format json | jq '.[] | select(.score > 100)'

# Count results
autocli reddit frontpage --limit 50 --format json | jq 'length'

AI-Powered Features

AutoCLI integrates with autocli.ai for AI-powered adapter generation and sharing.

Authentication

bash

# Authenticate with autocli.ai
autocli auth

This opens your browser to get an API token and saves it to

~/.autocli/config.json

Generate Adapters with AI

Use the Chrome extension to visually select data elements:

Navigate to any website in Chrome
Click the AutoCLI extension icon
Use the selector tool to pick data elements
Click "Generate" to let AI create the adapter
The adapter is saved locally and synced to autocli.ai

Search Community Adapters

bash

# Search by URL
autocli search https://news.ycombinator.com

# Search by domain
autocli search producthunt.com

Searches autocli.ai for community-shared adapters. Select one to download and use immediately.

Environment Variables

bash

# Override API server (default: https://www.autocli.ai)
export AUTOCLI_API_BASE=https://custom-server.com

Advanced Usage

Download Media

bash

# Download YouTube video (requires yt-dlp)
autocli youtube download "https://youtube.com/watch?v=..."

# Download Bilibili video
autocli bilibili download "https://www.bilibili.com/video/..."

# Download Xiaohongshu content
autocli xiaohongshu download "https://www.xiaohongshu.com/..."

Social Media Interactions

bash

# Twitter operations
autocli twitter post "Hello from AutoCLI!"
autocli twitter reply TWEET_ID "Great post!"
autocli twitter like TWEET_ID
autocli twitter bookmark TWEET_ID
autocli twitter follow USERNAME
autocli twitter bookmarks --limit 10

# Reddit operations
autocli reddit upvote POST_ID
autocli reddit comment POST_ID "Interesting discussion"
autocli reddit subscribe SUBREDDIT_NAME
autocli reddit saved --limit 10

# Xiaohongshu publishing
autocli xiaohongshu publish --title "My Post" --content "Content here" --images "image1.jpg,image2.jpg"

Job Search (BOSS Zhipin)

bash

# Search jobs
autocli boss search "Rust developer" --city "Beijing"

# Greet recruiter
autocli boss greet JOB_ID "Hello, I'm interested in this position"

# Batch greet
autocli boss batchgreet JOB_ID1,JOB_ID2,JOB_ID3 "Hello, I'm interested"

# View chat list
autocli boss chatlist

# View messages
autocli boss chatmsg BOSS_ID

Shell Completion

Generate shell completions for better autocomplete experience:

bash

# Bash
autocli completion bash >> ~/.bashrc

# Zsh
autocli completion zsh >> ~/.zshrc

# Fish
autocli completion fish > ~/.config/fish/completions/autocli.fish

# PowerShell
autocli completion powershell >> $PROFILE

Configuration

AutoCLI stores configuration in

~/.autocli/config.json

json

{
  "api_token": "your-autocli-ai-token",
  "browser_ws_endpoint": "ws://localhost:9222",
  "default_format": "table",
  "adapters_dir": "~/.autocli/adapters"
}

Creating Custom Adapters

AutoCLI uses declarative YAML pipelines for custom adapters. Adapters are stored in

~/.autocli/adapters/

Example Adapter Structure

yaml

name: example-site
description: Example site adapter
version: 1.0.0
author: Your Name
commands:
  - name: hot
    description: Get hot posts
    mode: public  # or 'browser'
    pipeline:
      - step: fetch
        url: https://api.example.com/hot
        method: GET
      - step: extract
        selector: $.data[*]
        fields:
          - name: title
            path: $.title
          - name: url
            path: $.url
          - name: score
            path: $.score
      - step: output
        format: table

Adapter Modes

public: Uses public APIs, no authentication needed
browser: Requires Chrome extension and browser session

Pipeline Steps

fetch: HTTP request to target URL
extract: Extract data using JSON path or CSS selectors
transform: Modify/filter extracted data
output: Format and display results

Integration with AI Agents

Register in Agent Configuration

Add to

.cursorrules

AGENT.md

bash

# Discover all available AutoCLI commands
autocli list

# Use specific commands as needed
autocli hackernews top --limit 10 --format json
autocli twitter search "topic" --format json

Register Local CLI Tools

bash

# Register local CLI tools for AI agent access
autocli register gh      # GitHub CLI
autocli register docker  # Docker CLI
autocli register kubectl # Kubernetes CLI

Common Patterns

Pipeline Data from Multiple Sites

bash

# Get tech news from multiple sources
autocli hackernews top --limit 5 --format json > hn.json
autocli lobsters hot --limit 5 --format json > lobsters.json
autocli devto top --limit 5 --format json > devto.json
jq -s 'add' hn.json lobsters.json devto.json > combined.json

Monitor Topics Across Platforms

bash

#!/bin/bash
TOPIC="rust"

echo "=== Hacker News ==="
autocli hackernews search "$TOPIC" --limit 3

echo "=== Reddit ==="
autocli reddit search "$TOPIC" --limit 3

echo "=== Twitter ==="
autocli twitter search "$TOPIC" --limit 3

echo "=== Dev.to ==="
autocli devto tag "$TOPIC" --limit 3

Archive Bookmarks

bash

# Export Twitter bookmarks
autocli twitter bookmarks --limit 100 --format json > bookmarks_$(date +%Y%m%d).json

# Export Reddit saved posts
autocli reddit saved --limit 100 --format json > reddit_saved_$(date +%Y%m%d).json

Troubleshooting

Browser Connection Issues

bash

# Run diagnostics
autocli doctor

# Check Chrome extension is loaded and active
# Verify browser is running on ws://localhost:9222

# Restart Chrome with remote debugging:
# macOS:
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222

# Linux:
google-chrome --remote-debugging-port=9222

# Windows:
"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222

Authentication Errors

bash

# Re-authenticate
autocli auth

# Verify token in config
cat ~/.autocli/config.json | jq '.api_token'

# Clear and re-auth
rm ~/.autocli/config.json
autocli auth

Command Not Found

bash

# Verify installation
which autocli

# Check available commands
autocli --help

# Update to latest version
curl -fsSL https://raw.githubusercontent.com/nashsu/autocli/main/scripts/install.sh | sh

Rate Limiting

Some sites may rate-limit requests. Use

--limit

to reduce request size:

bash

# Reduce limit to avoid rate limiting
autocli twitter search "topic" --limit 5

# Add delays between requests (if supported)
autocli twitter timeline --limit 10 --delay 1000

Extension Not Connecting

Ensure Chrome extension is enabled
Check extension has permissions for target sites
Verify autocli daemon is running
Restart browser and extension

Performance Tips

Use
```
--format json
```
for processing with other tools
Limit results with
```
--limit
```
for faster responses
Public API commands are faster than browser commands
Browser commands reuse logged-in sessions (no token refresh)
Use
```
--quiet
```
flag to suppress progress output in scripts

Resources

GitHub: https://github.com/nashsu/autocli
AutoCLI.ai: https://autocli.ai (adapter marketplace)
Documentation: Check
```
autocli --help
```
and subcommand help
Issues: https://github.com/nashsu/autocli/issues

autocli-web-scraping

NPX Install

Tags

SKILL.md Content

AutoCLI Web Scraping Skill

Installation

One-line Install (macOS / Linux)

Manual Installation

Chrome Extension (Required for Browser Commands)

Basic Usage

Discovery Commands

Public API Commands (No Browser Required)

Browser-Based Commands (Requires Extension + Login)

Output Formats

JSON Processing with jq

AI-Powered Features

Authentication

Generate Adapters with AI

Search Community Adapters

Environment Variables

Advanced Usage

Download Media

Social Media Interactions

Job Search (BOSS Zhipin)

Shell Completion

Configuration

Creating Custom Adapters

Example Adapter Structure

Adapter Modes

Pipeline Steps

Integration with AI Agents

Register in Agent Configuration

Register Local CLI Tools

Common Patterns

Pipeline Data from Multiple Sites

Monitor Topics Across Platforms

Archive Bookmarks

Troubleshooting

Browser Connection Issues

Authentication Errors

Command Not Found

Rate Limiting

Extension Not Connecting

Performance Tips

Resources