8 Replies Latest reply on Jul 20, 2017 4:35 AM by Jonathan Jeffery

    Downloading large files from FM Cloud

    Jonathan Jeffery

      I have a solution hosted on FMcloud, and I've run into a problem.

       

      The solution comprises a couple of small interface files, and a large data file (possibly 10Gb, although the console doesn't share that information)

       

      I now need to get hold of a copy of the data file for some off-line work. However, the download fails repeatedly as the console times out (even if I do my best to keep the session live).

       

      This is a known issue (see Downloading large database files from FileMaker Cloud may be interrupted if the FileMaker Cloud Admin Console times out … ), which I suppose that FileMaker is working on. I've also raised it as a support ticket.

       

      That said, in the mean time, I'm stuck with a hosted, backed-up, database that I cannot get a copy of.

       

      I was wondering about using SFTP to connect to FM cloud instance's volumes and downloading the file directly -- has anyone had any success with this?

       

      I'm also wondering about making a volume out of a back-up snapshot, and then attaching it to a small Linux instance (so that I know the required username for my SFTP connection, which I don't for Orbitera's FMcloud image).

       

      However, I'd need to take ownership of the volume, and that's a bit beyond my linux/shh/AWS knowledge.

       

      Can anyone help?

       

      Regards,

       

      J.

        • 1. Re: Downloading large files from FM Cloud
          user6596

          The workaround is to use "scp" to securely transfer files between the instance and your machine.

          - Open port 22 on ec2 instance from AWS consol

          -  close the file you want to copy - (from FMCloud console)

          -  scp -i path/YourKeyPair.pem centos@host:/opt/FileMaker/FileMaker\\\Server/Data/Database/Data.fmp12 .

          -  ssh -i path/YourKeyPair.pem centos@host and delete the zip files in /FileMakerData/tmp/DownloadTemp_FMS/

          - reopen the file from FMC console

          - close port 22 from AWS console

           

          Hope this helps!

          • 2. Re: Downloading large files from FM Cloud
            Jonathan Jeffery

            You star!

             

            I couldn't seem to get the scp syntax correct, but the user 'centos' was the missing piece of the puzzle for me to get sftp to work with FileZilla -- shamefacedly, I prefer a GUI...

             

            Steps are:

            1. As suggested above, close the file via the FM cloud console
            2. Open port 22 on the server via the AWS console (security group)
            3. On FileZilla, go to Settings ->  SFTP, and click 'add a key file...'

                     Select the private key file that you generated when setting up your AWS instance (you

                     have hopefully kept this somewhere safe, for just such an eventuality!). On older

                     versions of FileZilla may convert the file format, but my version accepted the PEM file

                     without commenting.

               4. Close settings

               5. On FileZilla, make a new site;

                   •  Host = your AWS instance DNS name

                   •  Protocol = SFTP

                   •  Logon type = Normal

                   •  User = centos

               6. Now click 'connect'

                     You should see the root of your FMcloud AWS instance.

                     Files are under /FileMakerData/Data/Databases

                     Drag and drop to download.

               7. It's best security practice to close port 22 on the server again.

             

            Caveats

            • this is a workaround for an known issue with FMcloud, and that it is far better to download a backup copy via the Admin Console if at all possible
            • the files should be shut down gracefully before downloading
            • this process is not supported or recommended by FileMaker Inc., and is done at your own risk.
            • 3. Re: Downloading large files from FM Cloud
              amsc

              I've taken a similar approach, but used a second nano instance mounted to a FM backup snapshot.  With this method you are not messing with the main FM cloud instance.  See here for Soliant's write up on this approach:

               

              FileMaker Cloud & Backup Restore - Soliant Consulting

              1 of 1 people found this helpful
              • 4. Re: Downloading large files from FM Cloud
                Jonathan Jeffery

                Hi,

                 

                Thanks for the heads-up!

                 

                I tried this approach (not realising that someone had already blogged about it) but I didn't seem to have ownership (or permissions) of the mounted volume -- it didn't appear in /dev/ except under /drives/uuid and when I tried to access it, there was a permissions denied error.

                 

                As I was in a hurry to get a copy of the file, I took the simplest approach. I may re-try this method, following the blog post, to see where I went wrong.

                 

                I suppose that you could, in theory, mount a preserved backup in the FMcloud console, and do much the same thing for the FM cloud instance.

                 

                J.

                • 5. Re: Downloading large files from FM Cloud
                  amsc

                  In theory, yes. But by accessing a backup snapshot and utilizing a secondary instance, you are completely outside of the FM cloud instance, and there is no need to interrupt service to users.  The "issue" I ran into the first time I set it up was that I inadvertently chose the wrong AWS "server farm", so I couldn't connect the backup to the new nano instance. There may be a way around that if you know AWS well enough. I simply deleted the nano instance and remade it. 

                  • 6. Re: Downloading large files from FM Cloud
                    Jonathan Jeffery

                    > But by accessing a backup snapshot and utilizing a secondary instance,

                    > you are completely outside of the FM cloud instance, and there is no

                    > need to interrupt service to users.

                    Very true.

                     

                    I had no trouble creating a volume from a snapshot, and attaching it to my free-tier linux instance (using the AWS console), but then when I tried to access it via FileZilla it was not visible, although I could navigate the instance's main volume without difficulty. 

                    • 7. Re: Downloading large files from FM Cloud
                      Jonathan Jeffery

                      File permissions, or file-system problems?

                      • 8. Re: Downloading large files from FM Cloud
                        Jonathan Jeffery

                        or my inadequate understanding of linux file systems