Today I had to figure out how to transfer 65GB of data (including dozens of files in separate subfolders) from a linux server (where it was stored after being pulled from a FilesAnywhere account using a recursive wget command, “wget -r -l 0 ftp://user:pass@server.com/”) to another storage account at egnyte.com.
I first tried tarring up all the files and folders and transferring that as one file, but I couldn’t get the support people at egnyte to untar it for me. So I began creating a perl script to do this task. After awhile of reading about how to make multidimensional arrays in perl for the file structure (technically this is not possible, but interacting with arrays of references in perl is very similar to multidimensional arrays in php) I decided to look for a simpler solution.
What I found was a CPAN module that allows for recursive gets and puts, (rget and rput, respectively). The details of the module are described at http://search.cpan.org/~jdlee/Net-FTP-Recursive-2.04/Recursive.pm. So after a bunch of trial and error I came up with the following script, which you can download here or copy from below.
All you need to do (once you have perl and the Net::FTP::Recursive cpan module installed-see links below for help with that) is put your ftp credentials into the file, put the perl script in the directory where you have files and folders to transfer (I actually put the perl file in the folder above where I wanted to transfer from so I wouldn’t transfer the perl file too), then execute the file by typing ‘perl putFTPDir.pl “ftp.host.com”‘ (or in my case I typed perl ../putFTPDir.pl “ftp.host.com”).
#!/usr/bin/perl
# Filename: putFTPDir.pl - this script will recursively transfer files in a directory tree to an ftp server
# Inputs:
# $host is an ip address of the ftp server to connect to
# Configurable parameters:
# $user is the ftp username
# $password is the ftp password
# $remoteDirectory to transfer files and folders of files to (defaults to top level directory)
# Example usages:
# perl putFTPDir.pl "1.1.1.1"
# perl putFTPDir.pl "ftp.example.com"
use strict;
use Net::FTP::Recursive;
use FileHandle;
my $numParams = scalar(@ARGV);
if ($numParams<1) {
print "Usage:\n";
print 'perl putFTPDir.pl "ftphost"'."\n";
print "ftphost is an ip address or domain name of the ftp server to connect to\n";
}
my $host=$ARGV[0];
my $user='ftpuser'; #ftp username, use single quotes so special characters are not interpreted
my $password='ftppassword'; #ftp password, use single quotes so special characters are not interpreted
my $remoteBaseDir='/path/to/sub/directory'; #ftp remote base directory where to put all the files under
my $ftp = Net::FTP::Recursive->new("$host", Debug => 1); #use Debug=>0 if you dont want to see what is happening or detailed error messages
$ftp->login($user,"$password");
if (length($remoteBaseDir)>0) {
$ftp->cwd("$remoteBaseDir");
}
$ftp->rput();
print "Done transferring files\n";
$ftp->quit;
exit;
In order to install the Net::FTP::Recursive CPAN module you can follow the instructions at http://www.cpan.org/modules/INSTALL.html. Alternatively, if you don’t even have CPAN yet you can start with http://www.thegeekstuff.com/2008/09/how-to-install-perl-modules-manually-and-using-cpan-command/.