Realtime Recommendation Ingestion System
Production-grade multi-source realtime intelligence ingestion platform that transforms noisy human recommendations into execution-ready structured trade state.
Multi-source convergence architecture
Human Moderation & Verification
Approve · Edit · Reject · Fino Pick — human-supervised intelligence, not schema-only checks.
Operational Tools & Monitoring
Queues, replay, and diff views for live operations.
Recovery & Reprocessing
Idempotent windows for backfills and connector fixes.
Downstream Systems
Execution engines, collections sync, and client surfaces.
Human Moderation & Verification
Approve · Edit · Reject · Fino Pick — human-supervised intelligence, not schema-only checks.
Operational Tools & Monitoring
Queues, replay, and diff views for live operations.
Recovery & Reprocessing
Idempotent windows for backfills and connector fixes.
Downstream Systems
Execution engines, collections sync, and client surfaces.
Ingestion
Source deep dive
- Telethon listener
- Realtime message capture
- Celery-backed queues
- Dedup + burst filtering
- High-volume noise gates
- S3 ingestion
- Scheduled cron processing
- Table extraction pipeline
- OCR normalization
- Livestream monitoring
- OpenCV frame capture
- 4 fps OCR sampling
- Duplicate suppression
- Realtime stream constraints
- API polling
- Keyword + handle tracking
- Advisor channel watchlists
- Provider ingestion
- Entity extraction
- Recommendation / sentiment parse
- Web intelligence augmentation
- Bounded discovery pulls
- Rate-aware search workflows
Processing core
AI extraction, schema normalization & live follow-ups
AI
AI extraction layer
Classification gates spend; structured extraction runs only when a recommendation is likely.
Step 1
Classification
“Does this message contain a trade recommendation?”
Filters Telegram-scale noise before any heavy extraction.
- Binary / low-token classifier paths
- Queue-aware batching under burst load
- Confidence thresholds for ambiguous phrasing
Step 2
Structured extraction
“Map noisy advisor language into candidate fields.”
- Instrument & option-chain disambiguation
- Target / SL / entry recovery without inventing fields
- Quality payload for moderation queue
Latency optimized
Short classifier prompts; extraction only on signal.
Cost optimized
Two-stage workflow gates expensive model calls.
Hallucination safeguards
Thresholds, checks, and moderation handoff.
Advisor pattern library
Few-shots and per-advisor parsing templates.
Schema
Message transformation flow
Raw human text → model JSON → normalized trade object ready for review.
Raw message
NIFTY OPTION INTRADAY BUY NSEPUT NIFTY 12MAY2026 23600 @ 45.2 SL 15 TGT 85,100
AI extraction (JSON)
{
"ticker": "NIFTY23600PE",
"category": "Index Options",
"position": "Buy",
"entry": 45.2,
"stoploss": 15,
"target": [85, 100],
"period": "Intraday"
}Normalized trade object
- ticker: NIFTY2360012MAY26PE
- parent_ticker: NIFTY
- instrument_type: OPTIDX
- category: Index Options
- position: Buy
- entry: 45.2
- stoploss: 15
- target: [85,100]
- period: Intraday
- advisor: KSL Research
- source: Telegram
- status: Pending Review
Continuity
Follow-up tracking & user notifications
The system continuously tracked follow-up updates from advisors across multiple sources — evolving trade intelligence, not one-shot captures.
Follow-up Detection Engine
Correlates advisor deltas against open positions and pending trades.
Operational Update Flow
Realtime propagation from applied trade state to clients.
Trade update applied
Execution engine sync
User notification (if enabled)
Engineering
Challenges, priorities & pipeline outcomes
Key engineering challenges
Managing high-volume Telegram streams with mostly irrelevant traffic.
Normalizing inconsistent advisor formats into one execution-ready schema.
Option instrument parsing and parent-ticker resolution under ambiguity.
Hallucination management and confidence-aware extraction.
Latency vs cost tradeoffs at realtime capture scale.
Engineering priorities
Accuracy
- Execution realism
- Extraction correctness
- Moderation workflows
- Trade state integrity
Latency
- Realtime ingestion
- Low-delay execution visibility
- Fast downstream propagation
- Timely user notifications
Cost
- AI optimization
- Selective processing
- Bounded infrastructure
- Cost-aware scaling
The system optimized for accuracy and latency during early scale because execution realism mattered more than infrastructure cost.
Pipeline outcomes
- Reduced ingestion latency
- Realtime capture
- Standardized schema
- Scalable ingestion
- Operational moderation
- Downstream execution readiness
core
Technology stack
