I'm not sure that I understand what you are trying to to with your layout based on the enrollment table. A single record in enrollment matches up a single student to a single course when teh _fk fields are given values that match to existing records in the other two tables so It's not clear to me what you mean by: Right now if I type in a students ID their record fills up with the classes they've taken
That sounds like something you are doing on a layout based on Students, not enrollment.
Seems like you need a global search field where you can enter information about a student--perhaps a name or externally generated Student ID--not the __pkStudentID value and then a script can search Students for a record for that student--either listing all matching student records or, if none are found, it can display a message offering to create a new student record with the search data you entered.
I want to be able to enter in information into enrollment (ie a student ID and a course ID), and have my database create student records for the students that I enter in or edit their existing record (and add the course that I just enrolled them in). That way I don't have to go to student records and enter in each eid on a record to see the classes they've taken.
But your student ID does not exist until the record is created so you can't enter a value that does not yet exist. (I know it's tempting to use a different value external to FileMaker as the student ID, but this is not the best way to go about this.)
I'd use a portal to the join table to record new enrollment data for new and existing students. You can add a search tool that allows you to find existing students and when no such student is found, the script can offer to create a new record for that student.