For 3 fields, you could create 3 calculated fields, one to find the oldest person, one to find the youngest, one to find the middle person, then sort those three new fields in the order you want.
Within each record (families enrolled) I have multiple fields listing the names and DOB of students within that family.
Sounds like you should have a table of related records for this. Then you will be sorting records instead of fields within a record and this becomes much easier. Consider setting up a portal on your layout for listing all the students from the same family. Then the portal can be sorted by DOB and reports listing students, familiy info and birthdates can be produced from a layout based on the portal's table.
This sounds like a good option. I will look into the possibility of adding a portal and revamping the DB.