AnsweredAssumed Answered

Tip for faster list intersections

Question asked by calexmac on May 22, 2015


Tip for faster list intersections


A tip for any developers looking for fast set intersections...

If you have two values lists (sets) in variables $vA and $vB and want to find their intersection, you might be tempted to use a custom function (which limits the list size to the recursion limit) or use the built-in function FilterValues...

    Set Variable [ $vX; Value:FilterValues ( $vA ; $vB ) ]

If your values lists are large, you may find that FilterValues is a bit slow.
For my example, $vA has 28,714 values, $vB has 21,621 values and their intersection has 18,425 values, and the above FilterValues step on my system (3 Ghz) takes 29 seconds (with app nap off).  This is tiresome when doing a lot of set intersections.

However, you can achieve a set intersection using the functions from the Base Elements donation-ware plugin as per below:

    Set Variable [ $vC; Value:$vA & ¶ & $vB ]
    Set Variable [ $vC1; Value:BE_Values_FilterOut ( $vA ; $vB ) ]
    Set Variable [ $vC2; Value:BE_Values_FilterOut ( $vB ; $vA ) ]
    Set Variable [ $vC3; Value:$vC1 & ¶ & $vC2 ]
    Set Variable [ $vF; Value:BE_Values_FilterOut ( $vC ; $vC3 ) ]
    Set Variable [ $vX; Value:BE_Values_Unique ( $vF ) ]

Using the same sets as above, on the same environment, these 6 steps are executed in 0.5 seconds.
Thats 1.7% of the duration for the built-in function, despite executing 6 steps.

Enjoy the speed!