ampswire

ampswire

This periodically attachs to, checks for and grabs new articles in groups on a
remote AMPS server using WNDP

The whole docuement is then left in (outputfolderstub)(fileextension). The
default folder stub is 'xamps_', so :
    txt files will be left in /fip/spool/xamps_txt
    xml files will be left in /fip/spool/xamps_xml
    jpg files will be left in /fip/spool/xamps_jpg
    htm files will be left in /fip/spool/xamps_htm

The parameter file, normally tables/wire/AMPS, is read for the the names of all
the groups to scan for.
    ; comment
    logon: (logon on the remote server)     default: none
    password: (password on the remote server)   default: none

    class: Class of data                default: ALL
        see the amps doc for a full description
        includes    XMLNews-story
                ANPA
                IPTC
                HTML
                MarketData
                ALL
                KNRIDDER-NITF
                PRNewswire-Raw

    wndp-version: Version of WNDP           default: 3
    xmlgen-version: Version of XML          default: 1

    include-filter: (service name)          default: ALL
    exclude-filter: (service name)          default: none
        There can be up to 1000 filters specified.

    force-output: yes/no                default: no
        force all output to the output folder
        default is use the output folder name as a stub and append the file 'ext'
            eg if the file is 'txt' and output:xamps_
                output folder is xamps_txt
        in this case NO extension is added !

    process-additions:ignore/process/file       default: process
        AMPS does not usually send duplicate data files but sends a '.add' file which
denotes the original one sent plus new metadata.
        ignore  - just ignore all references
        process - find the original and send that with the new metadata
        file    - treat '.add' messages as just other file
        Note that the 'process' option uses /fip/data/amps and this should be purged
nightly for anything over 4 or 5 days (check with the AMPS setup how far back
it might go to)

    extra-fiphdr-ext:(ext)  (FipSeq to add)     default: none
        add this extra fiphdr information to each file with this file extention.
        extra-fiphdr-ext:TXT    #SR:ALLPA
        extra-fiphdr-ext:XML    #DY:RIPOUTTXT.XML
    default-ext:(ext)               default: none
        if the incoming file has an extention which is NOT mapped by an existing
extra-fiphdr-ext, use this one.
        It also will leave it in the xamps_(ext) folder rather than making a new
folder.

    extra-fiphdr-class:(ext)    (FipSeq to add)     default: none
        add this extra fiphdr information to each file with this file extention.
        extra-fiphdr-class:JPG  #AB:JPEG

    round-robin: (number)              default: none
    round-robin-fiphdr: (2 letter FipHdr field) default: none
        Round-Robin the output files and add the RR number to the fipHdr.
        Both parameters are required - the Number is the MAXimum.
        eg  round-robin:9
            round-robin-fiphdr:RR
        and a suitable output folder might be   /fip/spool/xchg\RR

Commons extensions are
    TXT text - anpa or iptc
    XML xml variants; prnews
    JPG pix jpegs
    HTM html files

Optional
    chrset: (Source character set ie SC header field)   default: ascii
    response-timeout: (timeout in seconds wanting for the remote to reply)
                                default: 60 secs
    connection-timeout: (timeout in seconds wanting to connect to the remote)
                                default: 120 secs
    extra-fiphdr: (more FipHdr information to add)      default: none
    current-balance-group: name of a balance group (in tables/sys/BALANCE) to
distribute the current totals file when changed (see doc on 'ipbalan')
        This is often used where a second system could be used as a redundant server
if the main system fails.

    log-line: (FipSeq)
        Custom log line for files in (ie depends on -l 20)
        eg  log-line:\S2 \SF (\S1) \SA -> \SN

        Fiphdr fields for each incoming file are :
            SN - filename
            SU - from the '-n' input switch or AMPS
            SC - 'ascii' or chrset parameter
            SA - route from the request
            ST - class from the request
            SE - file extention
            S1 - size from the request
            S2 - data/time from the request
    timing-stats:yes/no
        Show timings statistics     default: no

Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).
    They are normally specified :
        fixed:QZ    1234543
        partial:QT  ST,3,2,U,<,>
        combie:QY   ep|na,(0000000)a
        option:QE   ep,11,7,s
        repeat:QK   XK,-,3
    or  repeat:QP   PK,,4,#X
        style:QS    XN,%.03d
        replace:QN  NN  abc=DEF def=GHI
        newdate:QT  hours+3 "\ZD"

Input switches are :
Mandatory :
    -s : Hostname where the AMPS is running.    default: none
Optional :
    -A : Processing ADD - additions             default: process
            ignore - just delete them
            process - copy the original data and add the new routing, send as a new file
            file - just send like a new file
    -d : done folder                    default: none
        This holds a copy of all incoming data files from every source
        The structure is
            (done folder) / (date)_(logon) eg 20110921_fip / (filename as written to the
output folder)
        It can be purged with an entry in maintenance (zapfiplog)
        eg if '-d raw.amx' and we want the last 30 days data
            /fip/bin/ipdelque -q/fip/spool/raw.amx -i1 -a30
    -D : display the conversation with the remote server    default: no
        used for debugging troublesome connections
        do NOT add in the SYSTEM file
    -F : Force the output folder to the '-o' and do NOT add the file type
extension
    -I : id of this instance                default: ignored
        Where there are several copies of 'ampswire' running with the same -n
    -l : log level                default: 10 for connect/disc messages too
        -l 0 errors only
        -l 10 plus connect/disconnect
        -l 20 plus each file
    -n : name of the service        def: AMPS or name of -z param file
    -o : Next fip queue for incoming files    default: spool/xamps_ZZZ
            the file type is added as the last 3 letters of the output folder
    -p : port number on the remote host     default: 39030
    -V : major version              default: xmlgen
        -V wndp for wndp
        -V xmlgen for xmlgen
    -z : parameter file             default: wire/AMPS
        this file MUST exist - ut can be a single comented line.
    -v : display version number and exit.

-- Copy Flow

For text files which are ANPA or IPTC, spool the output folder with ipdskwir :
eg a snippet from the SYSTEM file
    ; ampswire using default parameter file wire/AMPS and default folders -
xamps_txt
    amps    local   ampswire -s localhost
    ; txt files - process and route as normal
    atxt    local   ipdskwir -i xamps_txt -n AZMEDIA
    ; jpgs - move to pix system directly
    apix    local   ipftp -i xamps_jpg -n AZPIX2EDIT

-- Maintenance
The zapfiplog entry if using the process-additions:process option (which is the
default) :
    If the -n name is different to AMPS, then change the 2nd 'amps' in the 2
strings

if [ -d /fip/data/amps/amps ]; then
    /fip/bin/ipdelque -q /fip/data/amps/amps -A 2 -i 0
fi

Version Control
;001b17 29mar05 original version
    ;a3-5 added process-addition
    ;b1-8 better XML ;9-10 added round-robin ;11 added -d ;12 bugette in -d
    ;b13-14 added extra-fiphdr-class and better ext ;15-16 -I wireId
    ;b17 6jan16 added default-ext

(copyright) 2017 and previous years FingerPost Ltd.