Async Video Polling Engine - Background Job Handler for AI Video Generation Workflow Solution

Workflow overview

Why this workflow matters

Improves internal consulting operations and productivity. Relevant for managed services and support workflows.

🔄 AI Video Polling Engine - Long-Running Job Handler for Veo, Sora & Seedance The async backbone that makes AI video generation production-ready ⚡🎬 🎥 See It In Action 🔗 Full Demo: youtu.be/OI_oJ_2F1O0 ⚠️ Must Read First This is a companion workflow for the main AI Shorts Generator: 🔗 Main Workflow: AI Shorts Reactor This workflow handles the "waiting game" so your main bot stays fast and responsive. Think of it as the backstage crew that handles the heavy lifting while your main workflow performs on stage. 🤔 The Problem This Solves Without This Workflow: User sends message ↓ Bot calls AI API ↓ ⏳ Bot waits 2-5 minutes... (BLOCKED) ↓ ❌ Timeout errors ❌ Execution limits exceeded ❌ Users think bot is broken ❌ Can't handle multiple requests With This Workflow: User sends message ↓ Bot calls AI API ↓ ✅ Bot responds instantly: "Video generating..." ↓ 🔄 This webhook polls in background ↓ ⚡ Main bot handles other users ↓ ✅ Video ready → Auto-sends to user Result: Your bot feels instant, scales infinitely, and never times out 🚀 🔁 What This Workflow Does This is a dedicated polling webhook that acts as the async job handler for AI video generation. It's the invisible worker that: 1️⃣ Receives the Job POST /webhook/poll-video { "sessionId": "user_123", "taskId": "veo_abc456", "model": "veo3", "attempt": 1 } 2️⃣ Responds Instantly 200 OK - "Polling started" (Main workflow never waits!) 3️⃣ Polls in Background Wait 60s → Check status → Repeat ⏱️ Waits 1 minute between checks (API-friendly) 🔄 Polls up to 15 times (~15 minutes max) 🎯 Supports Veo, Sora, and Seedance APIs 4️⃣ Detects Completion Handles multiple API response formats: // Veo format { status: "completed", videoUrl: "https://..." } // Market format (Sora/Seedance) { job: { status: "success", result: { url: "..." } } } // Legacy format { data: { video_url: "..." } } (No matter how the API responds, this workflow figures it out) 5️⃣ Delivers the Video Once ready: 📥 Downloads video from AI provider ☁️ Uploads to your S3 storage 💾 Restores user session from Redis 📱 Sends Telegram preview with buttons 🔄 Enables video extension (Veo only) 📊 Logs metadata for analytics ⚙️ Technical Architecture The Flow: Main Workflow Polling Webhook │ │ ├──[Trigger AI Job]──────────┤ │ "Task ID: abc123" │ │ │ ├──[Return Instantly] │ │ "Generating..." │ │ │ ├──[Handle New User] │ │ ├──[Wait 60s] │ │ │ ├──[Check Status] │ │ "Processing..." │ │ │ ├──[Wait 60s] │ │ │ ├──[Check Status] │ │ "Completed!" │ │ │ ├──[Download Video] │ │ │ ├──[Upload to S3] │ │ │ └──[Send to User] │ │ └──────────────────────────────────┘ "Your video is ready!" 🚀 Key Features ⚡ Non-Blocking Architecture Main workflow never waits Handle unlimited concurrent jobs Each user gets instant responses 🔄 Intelligent Polling Respects API rate limits (60s intervals) Auto-retries on transient failures Graceful timeout handling (15 attempts max) 🎯 Multi-Provider Support Handles different API formats: Veo** - record-info endpoint Sora** - Market job status Seedance** - Market job status 🛡️ Robust Error Handling ✅ Missing video URL → Retry with fallback parsers ✅ API timeout → Continue polling ✅ Invalid response → Parse alternative formats ✅ Max attempts reached → Notify user gracefully 💾 Session Management Stores state in Redis Restores full context when video is ready Supports video extension workflows Maintains user preferences 📊 Production Features Detailed logging at each step Metadata tracking (generation time, model used, etc.) S3 storage integration Telegram notifications Analytics-ready data structure 🧩 Integration Points Works Seamlessly With: | Use Case | How It Helps | |----------|--------------| | 🤖 Telegram Bots | Keeps bot responsive during 2-5 min video generation | | 📺 YouTube Automation | Polls video, then triggers auto-publish | | 🎬 Multi-Video Pipelines | Handles 10+ videos simultaneously | | 🏢 Content Agencies | Production-grade reliability for clients | | 🧪 A/B Testing | Generate multiple variations without blocking | Required Components: ✅ Main workflow that triggers video generation ✅ Redis for session storage ✅ S3-compatible storage for videos ✅ KIE.ai API credentials ✅ Telegram Bot (for notifications) 📋 How to Use Step 1: Set Up Main Workflow Import and configure the AI Shorts Reactor Step 2: Import This Webhook Add this workflow to your AlekSystem instance Step 3: Configure Credentials KIE.ai API key Redis connection S3 storage credentials Telegram bot token Step 4: Link Workflows In your main workflow, call this webhook: // After triggering AI video generation const response = await httpRequest({ method: 'POST', url: 'YOUR_WEBHOOK_URL/poll-video', body: { sessionId: sessionId, taskId: taskId, model: 'veo3', attempt: 1 } }); Step 5: Activate & Test Activate this polling webhook Trigger a video generation from main workflow Watch it poll in background and deliver results 🎯 Real-World Example Scenario: User generates 3 videos simultaneously Without This Workflow: User A: "Generate video" → Bot: ⏳ Processing... (BLOCKED 5 min) User B: "Generate video" → Bot: ❌ Timeout (main workflow still processing User A) User C: "Generate video" → Bot: ❌ Never receives request With This Workflow: User A: "Generate video" → Bot: ✅ "Generating! Check back in 3 min" → Polling webhook handles in background User B: "Generate video" → Bot: ✅ "Generating! Check back in 3 min" → Second polling instance starts User C: "Generate video" → Bot: ✅ "Generating! Check back in 3 min" → Third polling instance starts 3 minutes later--- User A: 📹 "Your video is ready!" [Preview] [Publish] User B: 📹 "Your video is ready!" [Preview] [Publish] User C: 📹 "Your video is ready!" [Preview] [Publish] All three users served simultaneously with zero blocking! 🚀 🔧 Customization Options Adjust Polling Frequency // Default: 60 seconds // For faster testing (use credits faster): const waitTime = 30; // seconds // For more API-friendly (slower updates): const waitTime = 90; // seconds Change Timeout Limits // Default: 15 attempts (15 minutes) const maxAttempts = 20; // Increase for longer videos Add More Providers Extend to support other AI video APIs: switch(model) { case 'veo3': // Existing Veo logic case 'runway': // Add Runway ML polling case 'pika': // Add Pika Labs polling } Custom Notifications Replace Telegram with: Discord webhooks Slack messages Email notifications SMS via Twilio Push notifications 📊 Monitoring & Analytics What Gets Logged: { "sessionId": "user_123", "taskId": "veo_abc456", "model": "veo3", "status": "completed", "attempts": 7, "totalTime": "6m 32s", "videoUrl": "s3://bucket/videos/abc456.mp4", "metadata": { "duration": 5.2, "resolution": "1080x1920", "fileSize": "4.7MB" } } Track Key Metrics: ⏱️ Average generation time per model 🔄 Polling attempts before completion ❌ Failure rate by provider 💰 Cost per video (API usage) 📈 Concurrent job capacity 🚨 Troubleshooting "Video never completes" ✅ Check KIE.ai API status ✅ Verify task ID is valid ✅ Increase maxAttempts if needed ✅ Check API response format hasn't changed "Polling stops after 1 attempt" ✅ Ensure webhook URL is correct ✅ Check AlekSystem execution limits ✅ Verify Redis connection is stable "Video downloads but doesn't send" ✅ Check Telegram bot credentials ✅ Verify S3 upload succeeded ✅ Ensure Redis session exists ✅ Check Telegram chat ID is valid "Multiple videos get mixed up" ✅ Confirm sessionId is unique per user ✅ Check Redis key collisions ✅ Verify taskId is properly passed 🏗️ Architecture Benefits Why Separate This Logic? | Aspect | Monolithic Workflow | Separated Webhook | |--------|--------------------|--------------------| | ⚡ Response Time | 2-5 minutes | <1 second | | 🔄 Concurrency | 1 job at a time | Unlimited | | 💰 Execution Costs | High (long-running) | Low (short bursts) | | 🐛 Debugging | Hard (mixed concerns) | Easy (isolated logic) | | 📈 Scalability | Poor | Excellent | | 🔧 Maintenance | Complex | Simple | 🛠️ Requirements Services Needed: ✅ AlekSystem Instance (cloud or self-hosted) ✅ KIE.ai API (Veo, Sora, Seedance access) ✅ Redis (session storage) ✅ S3-compatible Storage (videos) ✅ Telegram Bot (optional, for notifications) Skills Required: Basic AlekSystem knowledge Understanding of webhooks Redis basics (key-value storage) S3 upload concepts Setup Time: ~15 minutes Technical Level: Intermediate 🏷️ Tags webhook polling async-jobs long-running-tasks ai-video veo sora seedance production-ready redis s3 telegram youtube-automation content-pipeline scalability microservices AlekSystem-webhook job-queue background-worker 💡 Best Practices Do's ✅ Keep polling interval at 60s minimum (respect API limits) Always handle timeout scenarios Log generation metadata for analytics Use unique session IDs per user Clean up Redis after job completion Don'ts ❌ Don't poll faster than 30s (risk API bans) Don't store videos in Redis (use S3) Don't skip error handling Don't use this for real-time updates (<10s) Don't forget to activate the webhook 🌟 Success Stories After Implementing This Webhook: | Metric | Before | After | |--------|--------|-------| | ⚡ Bot response time | 2-5 min | <1 sec | | 🎬 Concurrent videos | 1 | 50+ | | ❌ Timeout errors | 30% | 0% | | 😊 User satisfaction | 6/10 | 9.5/10 | | 💰 Execution costs | $50/mo | $12/mo | 🔗 Related Workflows 🎬 Main: AI Shorts Reactor - The full video generation bot 📤 YouTube Auto-Publisher - Publish completed videos 🎨 Video Style Presets - Custom prompt templates 📊 Analytics Dashboard - Track all generations 📜 License MIT License - Free to use, modify, and distribute! ⚡ Make your AI video workflows production-ready. Let the webhook handle the waiting. ⚡ Created by Joe Venner | Built with ❤️ and AlekSystem | Part of the AI Shorts Reactor ecosystem

Best fit

Need another direction?