This project is a complete pipeline for creating a Turkish QA dataset from raw review data and training AI models. We collected multiple large-scale datasets, cleaned and categorized them, generated question-answer pairs, and trained AI models for testing and evaluation. The pipeline handles the full lifecycle from raw data to ready-to-use AI models.
Highlights:
- Collect and aggregate large-scale review datasets 📂
- Clean and normalize text data ✨
- Categorize reviews by topic and sentiment (positive 👍, negative 👎, neutral 😐)
- Generate QA pairs automatically (questions & answers in Turkish) 📝
- Train and test AI models with pre-trained or custom architectures 🧠
1️⃣ Clean raw data: python scripts/clean_data.py
2️⃣ Generate QA dataset: python scripts/generate_qa.py
3️⃣ Train and test AI model: python scripts/train_model.py
Notes:
- QA pairs are in Turkish 🇹🇷
- Large raw datasets and trained models are excluded from the repository
- Use branches for experiments or new features 🌱
- The raw datasets used for QA generation are not included in the repository due to size and licensing.
- You can download them locally using the provided script.
- First, make sure you have the required packages installed (
pip install datasets pandas), then runpython src/download_all_datasets.pyfrom the project root. - This will automatically download all datasets from Hugging Face and save them as CSV files in
data/raw/(turkish_reviews1.csv,turkish_reviews2.csv,turkish_reviews3.csv,turkish_reviews4.csv,turkish_reviews5.csv,turkish_reviews6.csv). - Each CSV contains the full dataset in a single split, meaning train/test splits are not separated, and all data is stored together for further processing.