Drag to select the portal and the fields. Then hold down the shift key and click the portal without clicking a field to remove it from the selection.
Or just hold down the shift key and click one field at a time to select the fields without selecting the portal.
Shift clicking adds an object to the selected group if it is not already selected. Shift clicking a selected object de-selects it.