University Course Catalog Mining Pipeline
An end-to-end Python pipeline for discovering, collecting, processing, and validating historical university course catalogs. The project turns inconsistent archival PDFs into structured course records that researchers can use for downstream analysis.
Project Summary
- Status
- In Progress
- Timeline
- 2026 - Present
- Visual proof
- PDF -> OCR -> Dataset
Placeholder visual
Featured case-study visual
Placeholder visual: scanned catalog page beside extracted structured table.
TODO: Replace this block with a real screenshot, chart, map, or diagram.
Problem
Historical course catalogs contain valuable academic program data, but the records are scattered across archives, stored as inconsistent PDFs, and often require OCR before analysis is possible.
Approach
Build a reproducible data pipeline that discovers catalog sources, extracts text with OCR/PDF tooling, parses course information, validates extracted fields, and exports structured datasets for research.
Architecture
System flow and processing stages.
Stage 1
Online archives and institutional sources
Stage 2
Catalog discovery and URL collection
Stage 3
PDF download and file organization
Stage 4
OCR and text extraction
Stage 5
Course parser and metadata extractor
Stage 6
Validation layer for missing fields, duplicates, and parsing quality
Stage 7
Structured CSV/SQL research dataset
Data Sources
- Historical university course catalogs
- Online institutional archives
- Scanned PDFs and text-based PDFs
- Extracted catalog metadata
Methods
- Web scraping
- OCR
- PDF processing
- Information extraction
- Data validation
- ETL
Technologies
- Python
- SQL
- pandas
- OCR tooling
- PDF processing
- Git
- Technical documentation
Evidence and Screenshots
Visual assets to replace placeholders.
Placeholder visual
OCR before/after catalog page
Create this screenshot and replace this placeholder in the project assets.
TODO: Replace this block with a real screenshot, chart, map, or diagram.
Placeholder visual
Structured data table of extracted course records
Create this screenshot and replace this placeholder in the project assets.
TODO: Replace this block with a real screenshot, chart, map, or diagram.
Placeholder visual
Validation dashboard showing missing fields and parsing issues
Create this screenshot and replace this placeholder in the project assets.
TODO: Replace this block with a real screenshot, chart, map, or diagram.
Results
- Created a reusable workflow for transforming archival catalogs into structured research data.
- Organized extracted course information for downstream historical analysis.
- Documented edge cases including OCR noise, inconsistent formatting, and missing catalog records.
Metrics and Evaluation Needed
- Catalogs processed
- Courses extracted
- Validation error breakdown
- OCR confidence distribution
Challenges
- Catalog formatting varies significantly across institutions and years.
- OCR introduces errors in department names, course titles, and descriptions.
- Historical records can be missing, duplicated, or inconsistently labeled.
Lessons Learned
- Validation is as important as extraction when building research datasets.
- Real-world archival data requires flexible parsing strategies instead of one brittle rule set.
- A useful research pipeline needs provenance, repeatability, and clear documentation.
Future Work
- Add OCR confidence scoring and extraction quality dashboards.
- Build a searchable interface for catalog records.
- Add LLM-assisted validation for ambiguous course descriptions.
Interactive Demo Ideas
- Pipeline stepper: PDF -> OCR Text -> Parsed Records -> Validation
- Dataset preview with institution/year filters
- Validation report explorer
What This Demonstrates