Course shopper was a project I tinkered with while in school. The idea was to scrape the classes from my school’s awful courses website: classes.uoregon.edu and little known course evaluations reports behind our student logins on duckweb.uoregon.edu to make a better course searching experience. Never got around to the front end and api but did do the scraper.
Each part of the scraper would be ran with Django custom commands which used the settings file to determine which function to use to update the course offerings and evals. The idea being if someone from another school wanted to do the same thing all they’d have to do is write a parser and plug in their functions in the settings.
For the class info: For each subject code and term I loaded the corresponding url for all the classes of that subject in that term, example page and parsed those search results (code) and for each search result parses the details (code).
For the evals: The eval reports were behind a login on duckweb.uoregon.edu (which I no longer have credentials for so no example pages) and in a jframe so I ended up extending the selenium web driver with some convenience functions (code) and using that to mimic a user to scrape the evals (code)