It is not very clear what you are doing here.I can understand the relationship you have set up but not how you are using it.
When you say "When I did this in FM8.5" - did what? You are apparently "filtering data (without a portal)" does that mean that you are using a Go to Related Records using the relationship? If so, it is still confusing when "in FM11, I get everything from the main table with only values from the second table...". So you get all records from the main table? And some records from the second table? How can this be?
All I can suggest at this stage is that you cannot traverse a relationship from a number field to a global field. The field on the right hand end of the relationship has to be indexable. So any GTRR from the main table (with the boolean value) to the second table (with the gTrue) will fail.
As I said, I don't think you have clearly explained how you are using the relationship.
In filemaker, relationships do not affect what records are in a foundset except in one specific script step: Go To Related Records or if you specify criteria in a related record's field while setting up a find.
Thus, I am at a bit of a loss as to how you ever got:
"only ... records where both tables had a 1 in the joined field"
without performing either a find or a GTRR--both of which can be overridden with a show all records command.
If your main table has gTrue and it "always contains the number 1" then the whole issue of outer and inner joins is meaningless here as you have just defined all records in your Main table to have the same value since gTrue is a global field.
This has worked this way since filemaker 3--the first relational version of filemaker.
I'd think you were describing a portal but you said you using this technique without a portal....
Hence my confusion at your post...