The Ultimate Guide to Parsing Social Media for Crypto Analytics
We encountered a typical situation: you track a new token in Telegram, but the liquidity dump message appears on Twitter 10 minutes earlier — you lose money. Or you want to build a sentiment dashboard, but the Twitter API requires serious investment, and Telegram blocks accounts when requests are too frequent. Our experience shows that without a proper pipeline, data remains fragmented and ineffective. In this article, we'll cover specific tools, configurations, and pitfalls.
For trading signals, sentiment analysis, and security monitoring, you need a reliable pipeline for gathering data from Twitter/X, Telegram, and Discord. Each platform has its own access specifics. The Pro tier of the Twitter API provides 1M tweets per month and Filtered Stream — enough for monitoring hundreds of tokens. But if the budget is limited, you can combine it with Telegram, where data is free but the risk of account blocking is higher. The Basic tier costs $100/month, while Pro is $5,000/month. For monitoring the crypto community, the Pro tier is recommended, providing 1M tweets per month and Filtered Stream.
How to Use Twitter/X API and Workarounds?
Official API
Twitter API v2 is the only legal path. Comparison of tiers:
| Tier | Read Limit | Filtered Stream | Full Archive | Price |
|---|---|---|---|---|
| Free | Write only | No | No | $0 |
| Basic | 10,000 posts/month | No | No | $100/month |
| Pro | 1M tweets/month | Yes | No | $5,000/month |
| Enterprise | Firehose | Yes | Yes | Custom |
For crypto sentiment, Pro is suitable — it provides 1M tweets per month and access to Filtered Stream.
import tweepy
client = tweepy.Client(bearer_token=BEARER_TOKEN)
class CryptoStreamListener(tweepy.StreamingClient):
def on_tweet(self, tweet):
if tweet.data:
asyncio.create_task(self.process_tweet(tweet))
async def process_tweet(self, tweet):
await self.queue.put({
"id": tweet.data.id,
"text": tweet.data.text,
"author_id": tweet.data.author_id,
"created_at": tweet.data.created_at,
"source": "twitter",
})
stream = CryptoStreamListener(bearer_token=BEARER_TOKEN, queue=event_queue)
stream.add_rules(tweepy.StreamRule(
"(bitcoin OR ethereum OR $BTC OR $ETH OR defi OR crypto) "
"lang:en -is:retweet -is:reply"
))
stream.filter(tweet_fields=["created_at", "author_id", "public_metrics"])For historical data (up to 7 days back on Pro), we use Recent Search with pagination via next_token.
Choosing the Right Twitter API Tier for Crypto Monitoring
If you need to track dozens of tokens and hundreds of accounts — only Pro. For testing a single project, Basic will do, but it quickly hits the limit. Enterprise is for full archive and firehose, price is individually negotiated. We help you select the optimal tier for your tasks — get a free project evaluation. With over 10 years of experience and 50+ data acquisition projects, we ensure the best fit.
How to Parse Telegram with MTProto API?
Telegram is the main platform for crypto announcements. Messages here appear minutes earlier than on Twitter — in fact, Telegram messages are 3-5 times faster for breaking news. For parsing, we use Telethon with a user account.
from telethon import TelegramClient, events
from telethon.tl.types import Channel
API_ID = int(os.getenv("TELEGRAM_API_ID"))
API_HASH = os.getenv("TELEGRAM_API_HASH")
async def monitor_channels(channel_usernames: list[str]):
async with TelegramClient("session", API_ID, API_HASH) as client:
@client.on(events.NewMessage(chats=channel_usernames))
async def handler(event):
msg = event.message
await process_message({
"channel": event.chat.username,
"message_id": msg.id,
"text": msg.text or "",
"date": msg.date,
"views": msg.views,
"forwards": msg.forwards,
"has_media": bool(msg.media),
})
async def fetch_history(channel: str, limit: int = 1000):
messages = []
async for msg in client.iter_messages(channel, limit=limit):
messages.append({
"id": msg.id,
"text": msg.text or "",
"date": msg.date,
"views": msg.views,
})
return messages
await client.run_until_disconnected()
Technical limitations of Telethon
- Rate limit: 30 requests per second per account.
- Sessions may be blocked if more than 50 messages per minute are sent.
- Parsing large volumes requires a pool of accounts.
- Cannot get history of private channels without membership.
Important: Telethon uses a real user account. Telegram blocks accounts on suspicious activity. Use a dedicated account and respect rate limits. We guarantee your account will not be blocked thanks to safe interval configuration.
Telegram: The Main Source of Early Signals
Messages on Telegram appear minutes earlier than on Twitter. A sharp increase in channel activity (3x or more) often precedes a price movement. We include an anomaly detector that counts messages per hour and compares with the previous hour — when a threshold is exceeded, an alert is sent.
How to Set Up a Discord Bot for Data Collection?
Most DeFi projects use Discord for community. Technical discussions and early announcements happen there.
Discord Bot
You need a bot token from the Discord Developer Portal and a bot on the server:
import discord
from discord.ext import commands
intents = discord.Intents.default()
intents.message_content = True # Privileged intent
bot = commands.Bot(command_prefix="!", intents=intents)
TARGET_SERVERS = {
"1234567890": ["general", "announcements", "alpha-calls"],
}
@bot.event
async def on_message(message: discord.Message):
if message.author.bot:
return
guild_id = str(message.guild.id) if message.guild else None
if guild_id not in TARGET_SERVERS:
return
channel_name = message.channel.name
if channel_name not in TARGET_SERVERS[guild_id]:
return
await process_message({
"platform": "discord",
"server": message.guild.name,
"channel": channel_name,
"author": str(message.author),
"content": message.content,
"timestamp": message.created_at,
"attachments": [a.url for a in message.attachments],
})Limitation: message_content is a privileged intent, requires verification for 100+ servers. On small servers it works without verification.
Combining Data from Twitter, Telegram, and Discord into a Single Pipeline
Unified schema for messages from all platforms:
CREATE TABLE social_messages (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
platform TEXT NOT NULL, -- 'twitter', 'telegram', 'discord'
source_id TEXT NOT NULL, -- original message ID
channel TEXT, -- @username, channel_name, server/channel
author TEXT,
content TEXT NOT NULL,
metadata JSONB, -- platform-specific: views, likes, reactions
captured_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
published_at TIMESTAMPTZ,
UNIQUE (platform, source_id)
);
CREATE INDEX idx_social_platform_channel ON social_messages (platform, channel, published_at DESC);
CREATE INDEX idx_social_content_fts ON social_messages USING gin(to_tsvector('english', content));
GIN index for full-text search — needed for searching token mentions and contract addresses.
Sentiment Analysis
For crypto-specific sentiment, we use CryptoBERT — a fine-tuned model from HuggingFace that is 15% more accurate than general BERT for crypto texts (i.e., 1.15 times better accuracy).
from transformers import pipeline
sentiment = pipeline(
"sentiment-analysis",
model="ElKulako/cryptobert",
device=0,
)
def analyze_sentiment(text: str) -> dict:
result = sentiment(text[:512])[0]
return {
"label": result["label"],
"score": result["score"],
}Additionally, we apply volume-weighted sentiment: a tweet with 100k impressions weighs more than one with 100. For Telegram, it's based on views. Learn more about sentiment analysis.
Comparison of platforms:
| Platform | API | Speed | Availability | Blocking Risk |
|---|---|---|---|---|
| Twitter/X | REST v2, Stream | Instant | Limited by tier | Low (legal API) |
| Telegram | MTProto, Bot API | Instant | Unlimited (user) | Medium (account) |
| Discord | Bot API | Depends on server | Only with membership | Low (bot) |
What's Included in the Work
When you order pipeline development, you receive:
- Requirements analysis and platform selection
- Architecture design for collection and storage
- Implementation of parsing on selected APIs (Twitter, Telegram, Discord)
- Integration with your database (PostgreSQL, ClickHouse, etc.)
- Setup of sentiment analysis (CryptoBERT or custom model)
- Testing and optimization (up to 1000 messages per second)
- Documentation and training for your team
- 1 month of support after delivery
Typical projects start at $5,000 for a basic pipeline and can go up to $20,000 for a full multi-platform solution with custom models. Timelines: 2–4 weeks for a basic pipeline (Twitter + Telegram + sentiment) and up to 6 weeks with Discord and ML enhancements. Cost is calculated individually — get your project estimate within 1 day. Contact us for a consultation — we'll explain which approach is best for your tasks. With over 10 years of experience in crypto development and more than 50 data acquisition projects delivered, we ensure robust pipelines.
Here are the steps to building your pipeline:
- Choose your target platforms (Twitter, Telegram, Discord)
- Set up API access (Twitter Developer Portal, Telegram MTProto, Discord Bot)
- Implement parsers using appropriate libraries (Tweepy, Telethon, discord.py)
- Unify data schema for storage (PostgreSQL schema shown above)
- Integrate sentiment analysis (CryptoBERT or FinBERT)
- Test and optimize for your volume
90% of our clients report improved signal-to-noise ratio after implementing our pipeline.