FileTransfer

Automating a Secure File Transfer between Servers

After marcexport creates an output file, we need to move it to the OAI server. There are a few different ways in which this can be accomplished. We like using Secure Copy (SCP) because it is secure, comes with OpenSSH, and can be automated. While the output file could be manually copied via SCP, you run the risk of someone forgetting and having your XC repository become out of date. Automating the process eliminates that risk.

Automating SCP requires setting up a public/private key authentication. Please read up on OpenSSH public key authentication to make sure you understand the process and security ramifications. You can find more information here.

Both your Voyager server and the OAI server need to have OpenSSL and OpenSSH installed and have sshd (the SSH daemon) running. You may already have both of these installed. To check, enter the following command:

ssh –Version

You should see this response:

OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007

The version is not really important, but this might be a good time to update them if they are really old.

Since the output file is owned by Voyager, we have Voyager SCP the file over. We also have created a Voyager user on the OAI server. While you do not have to do it this way, you will have permission issues to deal with if you decide to use different users. Do not set up root to root trust between servers.

To enable public/private key authentication, find the sshd_config file and open it with your preferred text editor. Make the following changes on both servers:

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile     .ssh/authorized_keys

On your Voyager server, as the Voyager user, generate the key pair. The keys should reside in the .ssh directory in Voyager’s home directory (/export/home/voyager in Unix and /home/voyager in Linux). Also, leave the passphrase blank, otherwise you will be prompted to enter the passphrase whenever you try to use this, thus defeating the purpose of automation.

ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/export/home/voyager/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
<long string of hexadecimal numbers> voyager@servername

The id_rsa key is the private key. Take care to protect it, as anyone who has possession of your private key can gain access to servers with a matching public key. Give it read/write permissions for Voyager only. Remove all permissions for group and other.

-rw-------   1 voyager  endeavor    1675 Apr 23 12:03 id_rsa 

On the OAI server, create a .ssh directory in Voyager’s home directory. Enter that directory and create a file called “authorized_keys” (without the quotes). Copy the contents of id_rsa.pub from your Voyager server. It will be one line that looks like this:

ssh-rsa <very long string of number & letter> voyager@servername

Set the permissions of the .ssh directory to 700 and 600 to authorized_keys as follows:

chmod 700 .ssh
chmod 600 authorized_keys

And that is it. To test it, try to SCP a file from your Voyager server:

touch testfile
scp tesfile voyager@oaiserver:/export/home/voyager/

If it is working properly, the file will get copied over without requesting you to enter a password. Note that for the very first time you try this, you may be asked to accept the RSA key. This will only be asked one time. Once you accept it, then you will not have to do so again. If that happens, try it again. The file should get copied over without any user prompting.

At this point, you have the the ability to copy over the MARC record file to Voyager’s home directory on the OAI server. In our XcExport script, however, we SCP the file to a directory where the OAI server expects it. If you want to do the same it is just a matter of setting the proper directory ownership. We put the Voyager user in the same group as the owner of the directory to which we want to SCP the file. There are other ways to accomplish this as well.

The last step of XcExport moves the deleted/replaced records to the OAI server. The deleted/replaced records are actually generated by another script called del_rep_archive as described in the following section.