You don't have your relationship set up correctly. For this relationship, you need only one pair of match fields. You don't link 4 pairs of match fields for this type of relationship.
If contact info::Serial Number is an auto-entered serial number that uniquely identifies the contact--not the key, then the serial number fields are the only match fields needed in the relationship:
Contact Info::Serial Number = Key Inventory::Serial Number
The building, room and key tag fields can be removed from Contact Info. They need only be defined in Key Inventory.
But given how keys for a large building are managed, you may need more tables and relationships than you have currently. Usually, there's a set of master key copies--each that opens a specific lock or group of locks--maybe even one that opens all doors in the physical plant. Then each person is issued a copy of one or more of those keys.
Since you don't want to have to keep entering the same building, room number info each time you issue a key, it would make sense to set up a table of the master copies with all that info stored in that table or in a table linked to it so that you need only select a key from that table to issue it and the relationships can then serve to list the relevant information about that key.
So by all means, set up a portal to Key Inventory on your Contact Information layout as a way to test what you want to set up thus far, but be prepared to go much further than this before you are done. Your final relationships might look like this:
This enables you to issue multiple keys to the same contact and copies of the same master key to multiple contacts. The key can open multiple doors and you do not have to do more than select a key once you have the data recorded a single time in Keys and Rooms.
I think I've got this portal thing down but was curious if I can have it auto populate the building and room numbers from another table instead of entering each one separately
Here's the other image of the portal in use