Realtime Recommendation Ingestion System
Sept 2024 — PresentConvergent multi-source ingestion with AI orchestration and human-in-the-loop verification before execution-ready trade state.
Saras— CTO, Co-FounderBuilt realtime and batch ingestion paths across Telegram, PDF reports, YouTube livestreams, Twitter/X, news APIs, and Perplexity-augmented discovery.
Each connector applied source-specific reliability patterns — deduplication, rate limits, OCR and frame pipelines, and queue-backed processing — before converging on a shared LLM classification and extraction stack.
The platform minimized irrelevant API spend via a two-stage model workflow, normalized inconsistent advisor language into a single trade schema, and gated publication through operational moderation for downstream execution readiness.
System Overview
The platform continuously ingested trading recommendations from highly inconsistent multi-platform sources including Telegram, PDFs, YouTube livestreams, Twitter/X, news feeds, and Perplexity search workflows.
Each source used custom ingestion logic, AI-assisted extraction, normalization pipelines, and operational verification workflows before converging into a unified execution-ready trade schema.
The architecture was specifically designed to:
- handle noisy human communication
- minimize ingestion latency
- optimize AI processing cost
- support realtime recommendation capture
- normalize inconsistent advisor formats
- maintain operational verification workflows
Source preview
Key engineering challenges
Managing extremely high-volume Telegram streams with mostly irrelevant messages.
Converting inconsistent advisor communication styles into structured trade state.
Balancing latency, cost, and extraction accuracy at scale.
Handling OCR and livestream ingestion for realtime YouTube recommendations.
Maintaining operational verification workflows for hallucination prevention.
Tools & Technologies
Python
MongoDB
Redis
OpenAI
Celery
Telethon
OpenCV
AWS S3
AWS EC2