amallo

A problem with Secure Storage

Discussion created by amallo on Jun 6, 2016
Latest reply on Jun 15, 2016 by amallo

My client has 200GB of documents linked to tasks in FM via external secure storage.

We moved from SuperContainer to this option and it served us well.

We turned on secure because we didn’t want others to look at the files.

Then we decided for WebDirect speed we needed to move it to a 2 server deployment in a data centre.

We wanted to archive most of the data off, to save disk space, and only take the last 12 months data to the data centre.

Here is the problem. You can’t.

You have to convert it to open folders based on folders that relate to year or month containers etc. eg YYYYMM

So we started down this path.

It started ok until we kept running out of disk space. We couldn’t work out why.

What seems to happen is FM takes the secure files and writes them into the fmp12 file and then writes them out. Nothing is deleted in this process until the new 200GB open folder data is confirmed to be complete.

Meanwhile the FM file keeps growing in size. It looks like it is wanting to grow to 200GB.

I had the server crash 4 times. After each time we expanded the disk. Looking back on the files I can see the FM file went from 5mb to 20GB then 80GB and then 125GB.

(Note: make sure if you do this process that you turn off any server backups on this file.)

The process never finished. I think the client now has access to about 50% of their files. This is a real problem.

When it crashes the file never gets the required command to flush the temp files inside the fmp12 file.

This is now a big issue for future backups.

 

A file recover on the 125GB file crashed the machine again.

 

I’m going to restore from a pre-bloated file and try again with a TB drive to see how it goes ok without crashing. Then I'll clone the current bloated file and reimport the data and move the hopefully open storage created files to the appropriate places.

Instead of cloning and importing, I have asked FMI if there is a temporary table or field that is holding this bloated data and can it be deleted with the help of an SQL plugin.

 

So the take home is, if you ever need to archive off older secure data then it can’t be done unless you have the space and time to convert to a date based Open Storage. Often the reason to archive is lack of disk space which could lead to the above outcome. So be careful turning it on.

A date (YYYY or YYYYMM) or calculation folder naming option above the secure folder option looks like it is needed.

I will likely be only using Open Storage in the future.

Outcomes