aknudsen

Save time with GetField and Get(LayoutTableName)

Discussion created by aknudsen on May 23, 2018
Latest reply on May 23, 2018 by philmodjunk

I am writing a tool for processing and analyzing firewall logs and I have lots and lots of logs. The largest kit passed 50 million lines of log some time back. Without the MBS plugin and some of the functions I'm using, this would not have been possible to do in FM.

 

As I've been working on this, new requirements have come in, new ways of capturing logs is being done, and more people want reports. This all have an impact on what I'm doing so I'm in the process of writing the next version.

 

The goal is to automate the analysis as much as possible , but there will always be some hands-on work involved. Those familiar with Layer 7 network traffic know there is a lot going on. Layer 7 is the application layer and instead of the traditional port and protocol we are identifying traffic by application such web-browsing, ssl, Skype etc. That's all well and good when the traffic goes one way, but there is also return traffic (response) which complicate things quite a bit.

 

So for that reason, when I collect data I separate it into 6 different tables. During the import I have several filters such as rejecting tier source or destination IP addresses, application and source or destination ports etc. I preload that into hash tables and the largest hash table is about 105k items that is being checked for each individual line I import. I have a total of 8 hash tables. Yeah, it processes a lot of data, but I have to say that the MBS hash functions and QuickList functions are great for this.

 

But back to what this post is all about. As mentioned, I working on the next version and the goal is to make a lot more efficient in the manual analysis process. Iv'e already mentioned that I store traffic into 6 different tables base on the traffic type.

 

The 6 tables have the same structure and the 4 functions is the same for each of the tables.

 

Screen Shot 2018-05-23 at 16.22.33.png

 

So one script with 4 script parameters sounded just what I needed, and it was, but what about the other 5 tables? Do I copy the 4 parameter script 5 times so I have 6 scripts to deal with. Nah, don't want to do that.

 

I'm sure that what I did next, many of you have already done, but I've not and this is my post What comes next is something I picked up pretty early. The Data Viewer and the Let() function are your two best friends when you're trying to solve a problem.

 

First I put the field name and got the result I wanted, but that doesn't help me much in reducing the number of scripts

Screen Shot 2018-05-23 at 15.41.42.png

Next, what if I use the Get(LayoutTableName) and the :: and fieldname. Yep, gives me the filename, but you can't pass that to a variable, I needed to get the value.

Screen Shot 2018-05-23 at 15.42.18.png

Ah, the GetField function, yeah that will do it.

Screen Shot 2018-05-23 at 15.45.34.png

And then once you get your answer, you try to clean it up and trim it down, and the end result becomes this.

Screen Shot 2018-05-23 at 15.48.21.png

Perfect and exactly what I wanted. Now I can take this and pass it as a variable and suddenly I have one script that covers 6 layouts with 4 functions on each.

 

Screen Shot 2018-05-23 at 16.21.14.png

Hopefully this will be of value to some and as they say "and the moral of the story is". Learn the Let() function, it is invaluable when trying to solve problems. Also, sit back and think through your solution and you might save yourself a lot of work.

 

Cheers!!

Outcomes