Skip to content

Course material file upload, content indexing and topic extraction - #19

Open
Apps247 wants to merge 86 commits into
mainfrom
aprameya/course-material-coverage-analysis
Open

Course material file upload, content indexing and topic extraction#19
Apps247 wants to merge 86 commits into
mainfrom
aprameya/course-material-coverage-analysis

Conversation

@Apps247

@Apps247 Apps247 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

This PR primarily adds a new microservice for text and topic extraction from PDFs associated with course materials. It also allows full text search of these course material PDFs.

Additions

  • Text and topic extraction microservice
    • supports OCR for scanned PDFs
    • has special topic suggestion handling for Slides & Articles using BERTopic and font properties
  • Frontend UI for uploading course material PDFs
  • DB text indexing for uploaded PDFs using tsvectors.
  • Temporary Search UI with text highlighting and thumbnails (see note)
  • Frontend & DB changes to support associating one or more files to
  • Frontend & DB changes to support adding/removing topics, requesting topic suggestions, and reviewing these suggestions
  • Python tests for various handlers and postprocessing patterns
  • Laravel tests for Suggested Topic review functionality
  • End to End Test (currently not working, see last note)

Notes:

  • The search UI is marked (Temp), and can be accessed via a course page or program page. The logic behind the search is the same as that of Course search functionality + saved filter presets #18, so it should be easy to integrate it into the main search functionality once that is merged.
  • The NLP models used are currently pulled (and cached) from HuggingFace by their Python libraries during runtime. We may want to have local copies of these models that our code can use.
  • This PR also refactors some of the composer test set up to share common fixtures across tests
  • The E2E test currently fails, because of a known bug with Pest for file uploads. Someone made a fix and it should be merged in to the new version soon. Please ignore TopicReviewTest.php until then.

@FabianLCH FabianLCH changed the title Aprameya/course material coverage analysis Course material file upload, content indexing and topic extraction Jul 27, 2026
@FabianLCH
FabianLCH self-requested a review July 27, 2026 16:51
@FabianLCH FabianLCH added the enhancement New feature or request label Jul 27, 2026
@FabianLCH FabianLCH linked an issue Jul 27, 2026 that may be closed by this pull request

@FabianLCH FabianLCH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't had a chance to set things up locally and manually try out the changes yet, but I wanted to at least provide a review based off the code since we're short on time. No major changes needed for now, but please have a look at my comments!

Comment thread docs/FastAPITextExtractionService.md Outdated
Comment thread docs/FastAPITextExtractionService.md Outdated
Comment thread laravel/app/Http/Controllers/CourseMaterialFileController.php Outdated
Comment thread laravel/app/Http/Controllers/CourseMaterialFileController.php Outdated
Comment thread laravel/app/Http/Controllers/CourseMaterialFileController.php
Comment thread python/services/text_extraction_service/tests/test_handlers.py Outdated
Comment thread python/services/text_extraction_service/tests/test_handlers.py Outdated
@Apps247
Apps247 force-pushed the aprameya/course-material-coverage-analysis branch from a78b5ba to 5a4b834 Compare July 31, 2026 17:21
@Apps247

Apps247 commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author
  • Completed requested changes
  • Minor UI improvement
  • Updated Database.md schema for both the Text Extraction service and the LO Mapping service

Note: the Pest/Playwright issue with file uploads still stops the E2E test from working. Here is the Playwright PR to track, if necessary pestphp/pest-plugin-browser#237.

Thanks for the feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Course learning material upload for topic coverage analysis

2 participants