Open-HeyGen (Experimental Sandbox)

A lightweight experimental setup for generating talking avatars using open-source AI models and public Hugging Face Gradio spaces.

This project is an experimental lab / proof-of-concept exploring:

• Voice Synthesis: TTS generation using public SWivid/F5-TTS Hugging Face Space.
• Avatar Animation: Driving face photos from generated audio using LivePortrait / SadTalker Gradio clients.
• Lip-Sync RL: A prototype PyTorch Group Relative Policy Optimization (GRPO) training loop for frame alignment experiments.

---

🧪 Quickstart

1. Installation

# Clone repository
git clone https://github.com/Durgaprasad-Developer/HeyGen-clone.git
cd HeyGen-clone

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install dependencies
pip install fastapi uvicorn pydantic gradio_client torch pytest

2. Run Pipeline Experiment

Ensure your sample inputs exist under sample_inputs/ (my_face.jpg and my_voice.wav), then execute:

python run_live_pipeline.py

This runs an end-to-end generation test and outputs the final result to mock_storage/output_videos/live_generation.mp4.

---

📁 Project Layout

Heygen-clone/
├── api-gateway/          # FastAPI service for job ingestion & SQLite queueing
├── workers/
│   ├── audio_worker/     # F5-TTS Gradio client worker
│   └── video_worker/     # LivePortrait / SadTalker worker
│       └── alignment/    # Experimental PyTorch GRPO RL loop & rewards
├── frontend/             # Demo landing page (index.html)
├── sample_inputs/        # Sample face photo & reference audio WAV
├── mock_storage/         # Output audio chunks & generated MP4 videos
├── tests/                # Verification test suite (Phases 1-4)
└── run_live_pipeline.py  # Master pipeline execution script

---

🔬 Core Components

1. API Gateway (api-gateway/): FastAPI endpoints (POST /api/v1/generate, GET /api/v1/jobs/{job_id}) for job submission and status tracking.
2. Audio Worker (workers/audio_worker/): Fetches TTS audio via Gradio Client SDK from F5-TTS Hugging Face spaces.
3. Video Worker (workers/video_worker/): Animates the reference portrait with the synthesized WAV file via LivePortrait / SadTalker Gradio spaces.
4. GRPO Alignment Loop (workers/video_worker/alignment/): PyTorch experiment exploring Group Relative Policy Optimization using custom lip-sync MSE and temporal smoothness reward heuristics.

---

🧪 Tests

Run the phase verification tests:

pytest tests/ -v

---

📜 License

MIT License. Feel free to experiment, modify, and build upon this setup!