Simple ftp get to ftp delivery
This is a very simple copy flow. Files are simply got from a number of internal servers, and dropped into a folder for further ftp’ing on to the external site of a company called ‘DELIVERY’.
It is suitable for a small and relatively stable number of local sites where we need to fetch the data regularly and deliver it as soon as we have got it.
If you have a larger number of similar sites, and would like more complicated scheduling a copyflow using iptimer to trigger the ftp gets might be more appropriate. See [[IptimerFtp]] for a means of running this type of copy flow.
In the /fip/tables/sys/SYSTEM file
get1 local ipftp -G 600 -z MYFIRSTGET -Z -o 2delivery -q get2 local ipftp -G 600 -z MYSECONDGET -Z -o 2delivery -q get3 local ipftp -G 600 -z MYTHIRDGET -Z -o 2delivery -q delivery local ipftp -i 2delivery -d 2delivery_done -z 2delivery -Z
For the getting:
get1 local ipftp -G 600 -z MYFIRSTGET -Z -o 2delivery -q
means run an ftp get every 10 mins (600 secs), using the parameter file /fip/tables/ftp/MYFIRSTGET and deliver the files to /fip/spool/2delivery. Be quiet about (-q) occasions when you ftp to the site and do not find any files there.
In the parameter file /fip/tables/ftp/MYFIRSTGET you then specify the server, logon and password, andy special instructions (like a cd before the file) and any particular logging requirements. You may also wish to add some header information at this point.
; ---------------------------------- ; which system should do the getting ? ; see sys/DEST_REDUN check-primary-server-for-getfiles:pdfgetter ; ---------------------------------- ; where are we getting it from ? ; Remote server - Name/IP Address remhost:126.96.36.199 ; Remote server - Logon logon:logon ; Remote server -Password password:password ; Filename on remote server newname:\EN ; Commands to execute at the beginning of the session ; Note fipdelays to slow things up for NT ftpbefore:fipdelay ; exit if the directory doesn't exist ftpbefore: fipon-error:abort ftpbefore:cwd /Place/To/PickFiles/Upfrom ; Files to get ... get-and-zapall:*.pdf ; Commands to execute before sending each file - especially important ; if fetching from a windows server ftpbeffile:fipdelay ; destination - this isn't used as the files are put straight ; into the outbound q, but I have forced it anyway ; as otherwise the default is woops which looks distracting in the logs dest:pdfpurchaser ; Add a little fip header information so that we can improve the logging ... get-extra-fiphdr:\nQ1:MYFIRSTGET\nQ2:\W0\nQ3:\WN\n ; Log file name ; NB - may need to change this to reflect the source of the files logfile:/fip/log/ftp/MYFIRSTGET.\$D\$M\$E\$Y ; *** End ***
This will therefore get any files on the remote sever in /Place/To/PickFiles/Upfrom tand delivery them into /fip/spool/2delivery where they would site except for the fact that:
2delivery local ipftp -i 2delivery -d 2delivery_done -z 2delivery -Z
is monitoring the 2 delivery queue. This process will ftp the files according to a parameter file /fip/tables/ftp/2delivery, dump the original copies in /fip/spool/2delivery_done. Files are delivered into a directory on the remote site called ‘special’.
The parameter file /fip/tables/ftp/2delivery contains:
; '2delivery' - ; ; Delivery of pdfs send to the delivery ftp server ; These pdfs are fetched from various internal sites and ; then sent on to 'delivery' ; NB there is a primary and a secondary server ; so a pseudohostname is used, which is referenced to the actual servers ; in /fip/tables/sys/DEST_REDUN ; Tim/CC 7th February 2013 ; Remote server - Name/IP Address remhost:188.8.131.52 ; Remote server - Logon logon:delivery_Logon ; Remote server -Password password:secret ; Filename on remote server newname:\SN ; Commands to execute at the beginning of the session ; Note fipdelays to slow things up for NT ftpbefore:fipdelay ftpbefore:mkd special ftpbefore:fipdelay ftpbefore:cwd special ftpbefore:fipdelay ; in case there is a file with the same name ; Commands to execute before sending each file ftpbeffile:dele \SN ftpbeffile:fipdelay ftpbeffile:fipdelay ; Log file name ; NB - may need to change this to reflect the source of the files logfile:/fip/log/ftp/DELIVERY.\$D\$M\$E\$Y ; Minimal logging - logs success and failure only - ; comment out if you want wordy session debugging ;;minimum-log: ; *** End ***
Other things needed:
1) Purging of 2delivery_done.
This would normally be done with the nightly maintenance and can be done by adding a line to /fip/local/zapfiplog.
# delete ‘delivery’ files older than 4 days ..
if [ -d /fip/spool/2delivery_done ]; then /fip/bin/ipdelque -q /fip/spool/2delivery_done -m 4 -i 1 fi
2) Entry in /fip/tables/sys/DEST_REDUN to determinw which of a pair of machines is the primary ‘getter’.
This enables you to have these system file entries set up and running on two machines, wna only the primary machine actually do any getting. The secondary machine will remain passive until the primary machine is down.
; 'DEST_REDUN' - Destination redundancy ; ; Primary Secondary ; Name system system ; ---- ------- --------- remotesvr jj-dist1 jj-dist2 pdfgetter jj-dist2 jj-dist1