Course material file upload, content indexing and topic extraction

.gitignore

@@ -15,6 +15,7 @@ laravel/database/seeders/UserSeeder.php
 laravel/resources/views/emails/
 laravel/.phpunit.cache
 .phpunit.result.cache
+laravel/.phpunit.cache
 
 laravel/tests/Browser/Screenshots
 
@@ -37,6 +38,7 @@ experiments/llm-evaluation/venv
 python/services/logs/
 python/services/*/env/
 python/services/lo_mapping_service/app/services/batch_inputs/**
+python/services/lo_mapping_service/deploy/*.zip
 **.env
 python/services/lo_mapping_service/deploy/*.zip
 python/services/lo_mapping_service/env

docs/FastAPITextExtractionService.md

@@ -0,0 +1,156 @@
+# FastAPI Text Extraction Service
+
+## Overview
+
+The Text Extraction Service is a FastAPI application that extracts per-page text, and file-wide topics from PDF course materials. 
+
+Text extraction is performed using a PDF parser, or OCR if required (using Tesseract or Textract). Topic extraction uses a combination of font properties and BERTopic scores to identify suggested topics.
+
+## Primary Responsibility
+
+This service is responsible for:
+
+- accepting text-topic extraction and topic refresh requests
+- extracting text content from PDF files, page by page, and using OCR if requested
+- extracting potential topics from document text by
+  - using BERTopic to extract keywords/phrases from the text
+  - using Font properties (or estimated font properties with OCR) and the material type to identify potential topics
+  - finding existing course topics in course text
+
+## API
+
+### `GET /health`
+
+Returns a simple response:
+
+```json
+{
+  "status": "ok"
+}
+```
+
+### `POST /extract`
+
+The main endpoint. Accepts an [`ExtractRequest`](../python/services/text_extraction_service/app/schemas.py) with:
+
+- `file` (string, required): base64-encoded PDF content
+- `ocr_enabled` (boolean, default `false`): whether to apply OCR on low-text pages
+- `extraction_engine` (string, default `"tesseract"`): `"tesseract"` or `"textract"`
+- `ocr_threshold` (integer, default `0`): page text length below which OCR is triggered
+- `material_type` (string, optional): `"slides"`, `"article"`, or `null` for default handling
+- `existing_topics` (list of strings, default `[]`): course topics to match against extracted text
+
+Note (TODO): Having both `ocr_enabled` and `extraction_engine` is slightly redundant - we could have a third `extraction_engine` option be `"text-only"` or similar to cover both. However, this set up allows us to easily remove textract if we need to, as that was discussed to help simplify the set up but later retained as the service was modularized further. A potential improvement is having an `int` field with `0` correspond to text-only, and `1, 2, ...` correspond to OCR engines.
+
+Example request:
+
+```json
+{
+  "file": "JVBERi0xLjQK...",
+  "ocr_enabled": true,
+  "extraction_engine": "tesseract",
+  "ocr_threshold": 50,
+  "material_type": "slides",
+  "existing_topics": ["Climate Change", "Forest Ecology"]
+}
+```
+
+Returns an [`ExtractResponse`](../python/services/text_extraction_service/app/schemas.py) with:
+
+- `pages`: list of `{ page_number, content }` for each page with extracted text
+- `page_count`: total number of pages in the PDF
+- `topics`: list of `{ topic, score, source }` extracted from the document
+
+### `POST /refresh-topics`
+
+Re-extracts topics from existing page content without re-running text extraction. Accepts a [`RefreshTopicsRequest`](../python/services/text_extraction_service/app/schemas.py) with:
+
+- `pages` (list, required): list of `{ page_number, content }` objects from a previous extraction
+- `material_type` (string, optional): `"slides"`, `"article"`, or `null`
+- `existing_topics` (list of strings, default `[]`): course topics to match against
+
+Example request:
+
+```json
+{
+  "pages": [
+    { "page_number": 1, "content": "Introduction to Forest Ecology" },
+    { "page_number": 2, "content": "Climate Change Impacts" }
+  ],
+  "material_type": "article",
+  "existing_topics": ["Climate Change", "Forest Ecology"]
+}
+```
+
+Returns the same `ExtractResponse` format as `/extract`.
+
+## Text Extraction
+
+The extraction pipeline in [`document_extractor.py`](../python/services/text_extraction_service/app/services/text_readers/document_extractor.py) works as follows:
+
+1. PDF is opened with PyMuPDF
+2. For each page:
+   - If OCR is enabled and the page has little text (below `ocr_threshold`), the page is processed with Tesseract or Textract (see **OCR Engines** below).
+   - Otherwise, text is extracted directly from the text layer with font size and font weight.d
+3. Each page returns a list of lines with `{ text, size, bold }` metadata. The `size` and `bold` may be `None` depending on the extraction engine.
+
+### OCR Engines
+
+- **Tesseract**: Renders page to image at 300 DPI, groups words into lines, estimates font size from word-box height. Estimating whether the font is bold or not is possible but quite complex, so isn't implemented for now.
+- **AWS Textract**: Single-page PDF extraction uses a synchronous call. Multi-page PDFs are uploaded to S3, processed asynchronously, and the service polls for completion. Textract returns line text only, no font metadata. However, Textract is much faster than Tesseract for large PDFs.
+
+## Topic Extraction Pipeline
+
+Topic extraction is handled by [`type_specific_handlers.py`](../python/services/text_extraction_service/app/services/topic_extraction/type_specific_handlers.py) with material-type-specific handlers. Currently, there is a specific handler for Slides, and one for Articles, as well as a default handler.
+
+### BERTopic Extraction
+
+The BERTopic extractor in [`bertopic_extractor.py`](../python/services/text_extraction_service/app/services/topic_extraction/bertopic_extractor.py):
+
+1. Lemmatizes text with spaCy to collapse plurals
+2. Splits text into overlapping windows
+3. Fits BERTopic with the `all-mpnet-base-v2` embedding model.
+4. Extracts top keywords from each cluster
+5. Returns up to `TOPICS_COUNT` (currently 100) topics, then sorts by relevance. This limit should ideally never be reached, and is only there to prevent the database exploding with topics. We can increase the limit any time.
+
+## Runtime Dependencies
+
+Outside of what is installed via `pip`, you will need two dependencies:
+
+**Tesseract**: Local OCR engine. Install  `tesseract-ocr` via package manager or from the [official Tesseract page](https://tesseract-ocr.github.io/tessdoc/Installation.html) and ensure `tesseract` is on your `PATH`. English language data (`eng`) must be included.
+
+Verify Tesseract is reachable and has English data:
+
+```
+tesseract --version
+tesseract --list-langs # eng must appear in output
+```
+
+**spaCy English model**: The English NLP model used by the BERTopic extraction logic for Text lematization. After running `pip install -r requirements.txt`, run
+```
+python -m spacy download en_core_web_sm
+```
+Note that `_sm` stands for 'small', and is the lightest model. Heavier models like `en_core_web_lg` may yield better results, but also use more resources and space.
+
+## Key Environment Variables
+
+The environment variables are only used for Textract OCR.
+If using Tesseract, none are needed.
+
+- `AWS_ACCESS_KEY_ID`: AWS access key for Textract (optional)
+- `AWS_SECRET_ACCESS_KEY`: AWS secret key for Textract (optional)
+- `AWS_REGION`: AWS region for Textract (default: `ca-central-1`)
+- `AWS_S3_BUCKET`: S3 bucket for async Textract jobs (default: `text-extraction-temp`)
+
+## Error Handling
+
+Both `/extract` and `/refresh-topics` return HTTP 500 with the error message when processing fails.
+
+## Test Coverage
+The `tests/` folder for this service contains tests for the various file handlers and postprocessor:
+- **Default Material Handling** with topic matching
+- **Slides Handling** with font-based extraction
+- **Article Handling** with font-based extraction and soft assertion for BERTopic extraction
+- **Postprocessing** with various patterns and edge cases
+
+AWS Textract OCR isn't covered in the tests, as it is not available in LocalStack and costs credits to use.

docs/HighLevelArchitecture.md

@@ -2,11 +2,12 @@
 
 ## Overview
 
-This application is split into a primary Laravel web application and two Python services:
+This application is split into a primary Laravel web application and a set of Python services:
 
 - `laravel/`: the main user interface of the Curriculum Mapping Tool with courses, programs, syllabi, persistence, permissions, and reporting
 - `python/services/syllabi_service/`: a FastAPI service that parses uploaded syllabus files and extracts structured course data
 - `python/services/lo_mapping_service/`: a FastAPI service that prepares and manages learning-outcome mapping via LLM (Qwen/Qwen3-8B) and post-processes the results
+- `python/services/text_extraction_service/`: a FastAPI service that extracts text from PDF course materials (with Tesseract OCR if it's a scanned PDF) and also extracts likely topics based on keyword extraction and font properties
 
 ![Architecture Diagram](./images/ArchitectureDiagram.png)
 

docs/Setup.md

@@ -32,7 +32,7 @@ npm install
 cp .env.example .env
 ```
 
-The committed `.env.example` is the template every developer starts from. Its relevant database section is:
+Use `.env.example` as a template for `.env`, and modify values as required.
 
 ```
 DB_CONNECTION=pgsql
@@ -76,13 +76,13 @@ If you see `psql: command not found`, PostgreSQL is not installed; install it fi
 
 #### Create the `root` role
 
-`initdb` creates the `postgres` superuser but not `root`. Since `.env.example` ships with `DB_USERNAME=root`, create that role once:
+Create the root role:
 
 ```
 psql -U postgres -h 127.0.0.1 -c "CREATE ROLE root LOGIN PASSWORD '';"
 ```
 
-The empty password matches `DB_PASSWORD=` in the template. // TODO Check this
+The above command has a blank password. If setting a password, also set it in `.env`, under `DB_PASSWORD=`
 
 #### Create the application database
 
@@ -97,6 +97,28 @@ psql -U postgres -h 127.0.0.1 -c "CREATE DATABASE laravel OWNER root;"
 php artisan storage:link
 ```
 
+### Optional: PHP upload limits (required for Course Material File uploads)
+
+The Course Material File extraction feature lets users upload PDF course materials. PHP's default limits are small, so set these larger values in your active `php.ini`:
+
+```ini
+upload_max_filesize = 50M
+post_max_size = 60M
+memory_limit = 256M
+```
+
+Your active `php.ini` is the file listed as **"Loaded Configuration File"** when you run
+
+```
+php --ini
+```
+
+### Optional: PDF Thumbnails for Materials Search
+
+Rendering thumbnails requires `pdftoppm` from `poppler-utils`. Install via package manager or from [Poppler releases page](https://poppler.freedesktop.org/) and ensure `pdftoppm` is on your `PATH`.
+
+Verify with `pdftoppm -v`.
+
 ### Run migrations and seeders
 
 ```
@@ -133,6 +155,11 @@ Start the queue worker in one terminal:
 php artisan queue:work
 ```
 
+Note: If you need OCR, add this flag
+``` 
+php artisan queue:work --memory=2048
+```
+
 Run the application in another:
 
 ```
@@ -227,6 +254,7 @@ python -m venv env
 ``` 
 pip install -r requirements.txt
 ```
+Also check the service's documentation under `docs/` for additional specific setup steps.
 
 ### Run the Service
 

laravel/.env.example

@@ -18,6 +18,7 @@ DB_PASSWORD=
 BROADCAST_DRIVER=log
 CACHE_DRIVER=file
 QUEUE_CONNECTION=sync
+# QUEUE_CONNECTION=database # Use database if you're using OCR
 SESSION_DRIVER=file
 SESSION_LIFETIME=120
 
@@ -48,3 +49,8 @@ VITE_PUSHER_APP_KEY="${PUSHER_APP_KEY}"
 VITE_PUSHER_APP_CLUSTER="${PUSHER_APP_CLUSTER}"
 
 PYTHON_API_URL=http://
+
+# Required on Windows only
+# Required for Tesseract OCR and search PDF previews
+# Should look like path\to\pdftoppm.exe
+PDFTOPPM_PATH=

laravel/app/Exceptions/Handler.php

@@ -3,6 +3,7 @@
 namespace App\Exceptions;
 
 use Illuminate\Foundation\Exceptions\Handler as ExceptionHandler;
+use Illuminate\Http\Exceptions\PostTooLargeException;
 use Throwable;
 
 class Handler extends ExceptionHandler
@@ -26,5 +27,20 @@ public function register(): void
         $this->reportable(function (Throwable $e) {
             //
         });
+
+        $this->renderable(function (PostTooLargeException $e, $request) {
+            $limit = ini_get('post_max_size');
+            $referer = $request->headers->get('referer');
+
+            if ($referer) {
+                $separator = str_contains($referer, '?') ? '&' : '?';
+                return redirect($referer . $separator . 'upload_error=too_large&limit=' . urlencode($limit));
+            }
+
+            return response(
+                "Upload too large. The current server limit is {$limit}B. Please go back and try a smaller file.",
+                413
+            );
+        });
     }
 }