Fields that require calculations or linking between tables and files will generally process quite a bit slower than just raw data. The number of fields returned will also impact performance a bit as you've seen. Checked that you have the key you called reference1 indexed.
Just curious, how many rows do you have that your app is even allocating 400MB? And is this a custom app you're writing that interface with the ODBC API? It gotta really really do something funky to allocate that much memory if you're executing something returning 1 row.