SEO Toolkit - Comprehensive Guide

Overview

ContentSwift's SEO Toolkit provides professional-grade SEO analysis and web scraping capabilities using essential Python libraries trusted by SEO professionals worldwide.

Essential Python Libraries Used

Core Libraries

requests - HTTP requests, headers, status codes analysis
BeautifulSoup4 (bs4) - HTML/XML parsing and data extraction
pandas - Data manipulation and analysis
scikit-learn - Machine learning for keyword clustering
spaCy - Advanced Natural Language Processing
NLTK - Text processing and n-gram analysis
sumy - Automatic text summarization
re - Regular expressions for pattern matching

Additional SEO Libraries

textstat - Readability analysis (Flesch, Gunning Fog, etc.)
validators - URL and data validation
advertools - SEO and SEM analysis tools
selenium - Browser automation for JavaScript-heavy sites
scrapy - Web scraping framework

Features

1. 🔍 On-Page SEO Analysis

Comprehensive analysis of web pages including:

Meta Tags Analysis
- Title tag (length, optimization)
- Meta description (length, optimization)
- Meta keywords
- Robots directives
- Canonical URLs
- Open Graph tags
- Twitter Cards
Heading Structure
- H1-H6 extraction
- Hierarchy validation
- Multiple H1 detection
- Length analysis
Structured Data
- JSON-LD extraction
- Schema.org markup
- Microdata detection
Image SEO
- Alt text analysis
- Missing alt tags detection
- Alt text coverage percentage
- Image optimization metrics
Link Analysis
- Internal vs external links
- Nofollow detection
- Link text analysis
- Broken link detection
Content Analysis
- Word count
- Readability scores (Flesch, Gunning Fog, SMOG, etc.)
- Sentence structure
- Reading level estimation
Technical SEO
- Response time
- HTTPS detection
- Mobile viewport
- Character encoding
- Redirect chains

Endpoint: POST /seo/analyze-page

Example Request:

{
  "url": "https://example.com/page"
}

Example Response:

{
  "status": "success",
  "data": {
    "url": "https://example.com/page",
    "status_code": 200,
    "meta_tags": {
      "title": {
        "content": "Example Page Title",
        "length": 45,
        "optimal": true
      },
      "description": {
        "content": "This is an example meta description...",
        "length": 150,
        "optimal": true
      },
      "canonical": "https://example.com/page",
      "open_graph": {
        "og:title": "Example Page",
        "og:description": "...",
        "og:image": "..."
      }
    },
    "headings": {
      "h1_count": 1,
      "h1_optimal": true,
      "total_headings": 25,
      "hierarchy_issues": []
    },
    "content_stats": {
      "word_count": 1500,
      "readability": {
        "flesch_reading_ease": 65.5,
        "gunning_fog": 10.2
      },
      "readability_grade": "Standard (8th-9th grade)"
    },
    "links": {
      "total_links": 45,
      "internal_links": 30,
      "external_links": 15,
      "nofollow_count": 5
    }
  }
}

2. 📊 Keyword Density Analysis

Analyze keyword density with SEO best practice recommendations.

Endpoint: POST /seo/keyword-density

Example Request:

{
  "text": "Your content text here...",
  "keywords": ["content marketing", "SEO", "digital marketing"]
}

Example Response:

{
  "status": "success",
  "data": {
    "total_words": 1000,
    "keyword_densities": {
      "content marketing": {
        "count": 15,
        "density_percent": 1.5,
        "optimal": true,
        "recommendation": "Optimal density range."
      },
      "SEO": {
        "count": 8,
        "density_percent": 0.8,
        "optimal": false,
        "recommendation": "Low density. Could be increased slightly."
      }
    }
  }
}

Keyword Density Guidelines:

0%: Keyword not found
< 0.5%: Very low - consider increasing
0.5-1.0%: Low - could be increased
1.0-3.0%: Optimal range ✅
3.0-5.0%: High - risk of keyword stuffing
> 5.0%: Very high - keyword stuffing ⚠️

3. 🎯 TF-IDF Keyword Extraction

Extract important keywords using TF-IDF (Term Frequency-Inverse Document Frequency) analysis.

Endpoint: POST /seo/extract-keywords

Example Request:

{
  "documents": [
    "Content marketing is essential for SEO...",
    "Digital marketing strategies include SEO...",
    "SEO optimization requires keyword research..."
  ],
  "max_features": 50
}

Example Response:

{
  "status": "success",
  "data": [
    {"keyword": "SEO", "tfidf_score": 0.85},
    {"keyword": "content marketing", "tfidf_score": 0.72},
    {"keyword": "digital marketing", "tfidf_score": 0.68}
  ]
}

4. 🔗 Keyword Clustering

Group keywords by semantic similarity using K-means clustering.

Endpoint: POST /seo/cluster-keywords

Example Request:

{
  "keywords": [
    "content marketing",
    "content strategy",
    "SEO optimization",
    "SEO ranking",
    "social media marketing",
    "social media strategy"
  ],
  "n_clusters": 3
}

Example Response:

{
  "status": "success",
  "data": {
    "clusters": [
      {
        "id": 0,
        "keywords": ["content marketing", "content strategy"],
        "size": 2
      },
      {
        "id": 1,
        "keywords": ["SEO optimization", "SEO ranking"],
        "size": 2
      },
      {
        "id": 2,
        "keywords": ["social media marketing", "social media strategy"],
        "size": 2
      }
    ],
    "n_clusters": 3
  }
}

5. ✍️ Meta Description Generator

Auto-generate SEO-friendly meta descriptions using advanced text summarization.

Endpoint: POST /seo/generate-meta-description

Example Request:

{
  "text": "Your long article content here...",
  "max_length": 155
}

Example Response:

{
  "status": "success",
  "data": {
    "meta_description": "Learn about content marketing strategies that drive results. Discover proven techniques for SEO optimization and audience engagement.",
    "length": 145,
    "optimal": true
  }
}

6. 📝 Content Summarization

Summarize long-form content for various purposes.

Endpoint: POST /seo/summarize-content

Example Request:

{
  "text": "Your long content...",
  "sentence_count": 5
}

7. 🤖 Robots.txt Analysis

Analyze robots.txt files for SEO insights.

Endpoint: GET /seo/robots-txt/{domain}

Example: GET /seo/robots-txt/example.com

Example Response:

{
  "status": "success",
  "data": {
    "exists": true,
    "url": "https://example.com/robots.txt",
    "sitemaps": [
      "https://example.com/sitemap.xml",
      "https://example.com/sitemap-news.xml"
    ],
    "user_agents": ["*", "Googlebot", "Bingbot"],
    "disallow_rules": ["/admin/", "/private/"],
    "size_bytes": 450
  }
}

8. 🗺️ Sitemap Analysis

Analyze XML sitemaps for structure and optimization.

Endpoint: POST /seo/analyze-sitemap

Example Request:

{
  "sitemap_url": "https://example.com/sitemap.xml"
}

Example Response:

{
  "status": "success",
  "data": {
    "url": "https://example.com/sitemap.xml",
    "is_index": false,
    "url_count": 1250,
    "sitemap_count": 0,
    "sample_urls": [
      {
        "loc": "https://example.com/page1",
        "lastmod": "2024-12-01",
        "changefreq": "weekly",
        "priority": "0.8"
      }
    ],
    "size_bytes": 125000
  }
}

9. 🥊 Competitor Keyword Analysis

Analyze and compare keywords across competitor pages.

Endpoint: POST /seo/competitor-analysis

Example Request:

{
  "competitor_urls": [
    "https://competitor1.com/page",
    "https://competitor2.com/page",
    "https://competitor3.com/page"
  ]
}

Example Response:

{
  "status": "success",
  "data": {
    "competitor_urls": {
      "https://competitor1.com/page": {
        "word_count": 1500,
        "title": "Competitor 1 Page Title"
      }
    },
    "common_keywords": [
      {"keyword": "content marketing", "tfidf_score": 0.85},
      {"keyword": "SEO strategy", "tfidf_score": 0.78}
    ],
    "total_competitors_analyzed": 3
  }
}

10. 📈 N-gram Extraction

Extract bi-grams, tri-grams, and n-grams for keyword research.

Endpoint: POST /seo/extract-ngrams

Example Request:

{
  "text": "Content marketing is a strategic approach...",
  "n": 2,
  "top_n": 20
}

Example Response:

{
  "status": "success",
  "data": {
    "ngram_type": "2-gram",
    "ngrams": [
      {"phrase": "content marketing", "frequency": 15},
      {"phrase": "strategic approach", "frequency": 8},
      {"phrase": "marketing strategy", "frequency": 7}
    ]
  }
}

11. 📦 Batch URL Analysis

Analyze multiple URLs in a single request.

Endpoint: POST /seo/batch-analyze

Example Request:

{
  "urls": [
    "https://example.com/page1",
    "https://example.com/page2"
  ],
  "analyses": ["meta", "technical", "content"]
}

Installation & Setup

1. Install Dependencies

cd backend-crt/src
pip install -r requirements.txt

2. Download NLTK Data

import nltk
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('averaged_perceptron_tagger')

3. Download Spacy Model

python -m spacy download en_core_web_sm

4. Configure Environment

No additional API keys needed for basic SEO analysis. All analysis is performed locally.

Use Cases

For SEO Professionals

Content Optimization
- Analyze keyword density
- Generate meta descriptions
- Check readability scores
- Optimize heading structure
Technical SEO Audits
- Meta tag validation
- Robots.txt analysis
- Sitemap verification
- Link structure analysis
Competitor Research
- Keyword gap analysis
- Content strategy insights
- TF-IDF comparison

For Content Creators

Content Planning
- Keyword clustering for topic groups
- N-gram analysis for trending phrases
- Readability optimization
Content Optimization
- Auto-generate meta descriptions
- Keyword density optimization
- Content summarization

For Web Scraping

Data Extraction
- Structured data extraction
- Meta tag scraping
- Link harvesting
- Content analysis
SEO Data Collection
- Batch URL analysis
- Competitor monitoring
- SERP feature detection

Best Practices

Keyword Optimization

Target Density: Aim for 1-3% keyword density
Natural Language: Avoid keyword stuffing
Semantic Variations: Use related terms and synonyms
Long-tail Keywords: Include specific, multi-word phrases

Meta Tag Optimization

Title Tags: 30-60 characters
Meta Descriptions: 120-160 characters
Unique Tags: Every page should have unique meta tags
Include Keywords: Naturally incorporate target keywords

Content Quality

Readability: Aim for 8th-10th grade reading level
Word Count: Minimum 300 words, ideally 1000+ for pillar content
Heading Hierarchy: One H1, logical H2-H6 structure
Internal Linking: 2-5 internal links per page

Technical SEO

HTTPS: Always use secure connections
Mobile Optimization: Include viewport meta tag
Page Speed: Monitor response times
Structured Data: Implement Schema.org markup

API Response Codes

200 - Success
400 - Bad Request (missing parameters)
404 - URL not found
500 - Server error
503 - Service unavailable

Performance Considerations

Response Times

Page Analysis: 2-5 seconds per URL
Keyword Analysis: < 1 second
Batch Analysis: 2-5 seconds per URL
Sitemap Analysis: 3-10 seconds (depends on size)

Rate Limiting

When scraping external websites:

Implement delays between requests
Respect robots.txt
Use appropriate User-Agent headers
Consider using proxy rotation for large-scale scraping

Memory Usage

Basic Analysis: ~100MB
Batch Analysis: ~200-500MB
Large Content: ~1GB for extensive analysis

Troubleshooting

Common Issues

1. NLTK Data Not Found

import nltk
nltk.download('all')

2. Spacy Model Not Found

python -m spacy download en_core_web_sm

3. Connection Timeout

Increase timeout in requests
Check firewall settings
Verify URL is accessible

4. Memory Errors

Reduce batch size
Process URLs sequentially
Increase Docker memory allocation

Advanced Examples

Complete SEO Audit Workflow

# 1. Analyze page SEO
page_analysis = await seo_analyze_page({"url": "https://example.com"})

# 2. Extract keywords from content
keywords = await seo_extract_keywords({
    "documents": [page_content],
    "max_features": 50
})

# 3. Analyze keyword density
density = await seo_keyword_density({
    "text": page_content,
    "keywords": ["target keyword 1", "target keyword 2"]
})

# 4. Generate optimized meta description
meta_desc = await seo_generate_meta_description({
    "text": page_content,
    "max_length": 155
})

# 5. Analyze competitors
competitors = await seo_competitor_analysis({
    "competitor_urls": ["https://comp1.com", "https://comp2.com"]
})

FilesExpand file tree

SEO_TOOLKIT_GUIDE.md

Latest commit

History

SEO_TOOLKIT_GUIDE.md

File metadata and controls

SEO Toolkit - Comprehensive Guide

Overview

Essential Python Libraries Used

Core Libraries

Additional SEO Libraries

Features

1. 🔍 On-Page SEO Analysis

2. 📊 Keyword Density Analysis

3. 🎯 TF-IDF Keyword Extraction

4. 🔗 Keyword Clustering

5. ✍️ Meta Description Generator

6. 📝 Content Summarization

7. 🤖 Robots.txt Analysis

8. 🗺️ Sitemap Analysis

9. 🥊 Competitor Keyword Analysis

10. 📈 N-gram Extraction

11. 📦 Batch URL Analysis

Installation & Setup

1. Install Dependencies

2. Download NLTK Data

3. Download Spacy Model

4. Configure Environment

Use Cases

For SEO Professionals

For Content Creators

For Web Scraping

Best Practices

Keyword Optimization

Meta Tag Optimization

Content Quality

Technical SEO

API Response Codes

Performance Considerations

Response Times

Rate Limiting

Memory Usage

Troubleshooting

Common Issues

Advanced Examples

Complete SEO Audit Workflow

Roadmap

License

Support & Resources

Documentation

Learning Resources

Python SEO Libraries