TimeStampIn and TimeStampOut must be fields defined as being type Timestamp. The result type for your calculation can be either number or time depending on what you want to do with the result.
But "null" is a string (Text value). It's not a valid value that can possibly be entered into a field of type TimeStamp. Thus, your calculation always returns the result computed for the False result of your If function: TimeStampOut - TimeStampIn. This difference will be the elapsed time in seconds. If you choose a Time result type, that's fine, you can use data formatting and get a results such as 3:23--indicating an elapsed time of 3 hours 23 minutes. But more commonly, you need a number result in hours. To get a result in hours, the difference should be divided by the number of seconds in an hour: 3600s/hr.
I would guesss that you have used "null" because you want an empty result when TimeStampOUt is null. If that's the case, you need this calculation:
If( Not IsEmpty ( TimestampOut ) ; ( TimestampOut-TimestampIn ) / 3600 )