ipremsvr

ipremsvr

This program handles the connection with the outside world.

There are three basic modes which it can work :
    1. Permanent connection for a single client
    2. Start-stop connections for a group of clients
        This uses the -g input switch to hold a list of clients.
    3. Local copy for a group of clients using other transmissions
    such as email/SMTP and FTP which are serviced by 'ipsmtp' and 'ipftp'
        This uses the -LOCAL input switch for each client.

It copes with connections which are
    either client or server initiated
        ie Active or Passive - also called Push or Pull
    either broadcast or FipServer protocol
        ie fipagent or not - handshaking or serial-over-ip
    either permanent or start-stop connections

It adds the format and modifies the chrset (or converts the file type for
non-text files) according to the client's requirements.

'ipremsvr' first reads a parameter file in either
    tables/remote/groups/GROUPNAME if the '-g' switch is on
or  tables/remote/clients/CLIENTNAME.

This will point to a format file too which is in tables/remote/formats
if a text-format has been specified , that is also read...

The GROUP file (if used) has the following syntax:
    ; comment
    client: (clientname)
        There can be up to 10 clients specified. Each one will have
        a client file.
    connection-started-by:client or server
        Who makes the initial call - client connecting to the server
        or vice versa. default is server
        (if not LOCAL client)
    port: local port if connection started by client.
        (if not LOCAL client)

Up to 10 clients may be specified - ALL using the same 'connection-started-by'.
The 'connection-type' is 'start-stop' unless forced to be LOCAL by the '-L'
input switch

The CLIENT file has the following syntax:
    ; comment
    delivery-method:bdcast or server or local
        The two options are 'bdcast' for one-way traffic (default) and
        'server' for fip-agent type traffic.
        If using the -L for Local, this is ignored.
        If using the 'fipremote' program at the remote site and the
        delivery-method is  'broadcast', make sure that the 'end-of-message' in
        remote's 'REMOTE.FIP' parameter file is matched in the format for
        'ipremsvr'.

    connection-started-by:client or server
        Who makes the initial call - client connecting to the server
        or vice versa. Default is server
        If using the -L for Local, this is ignored.

    connection-type: permanent or start-stop
        The connection is dropped for Start/Stop clients immediately
        after all valid data has been sent and acknowledged.
        This is used for low-data clients and allows other clients
        to be serviced almost at the same time.
        If using the -L for Local, this is ignored.

    local-folder: for LOCAL clients, this determines the local folder to stuff
data files.
        The default is /fip/spool/remotes/(clientname forced lowercase)
    output-filename : Define the filename (for both local and remote
        clients. The default is (SERVICENAME) (SEQUENCE NO)
        eg  output-filename:\$d\R9\R7
    local-extra-fiphdr: (fipSeq) Extra metadata to add to the FipHdr for LOCAL
client files.
        default: none
    add-local-fiphdr: (yes/no)  Add/do not add the FipHdr to the LOCAL client
files.
        default: add

    host: hostname of the remote
    port: port of the remote

    password: Password the client uses to logon
    no-password:        There is no Password handshake
    no-logon-required:  There is no Password handshake
        (These two keywords have the same effect)

    service: name of a service that this client will receive
        There can be as many 'service' lines as required.
        Each line specifies one and only one service.
        The name of the service MUST be exactly the same as specified
        in the main SERVICES file as used by 'ipremdata'.
        Eg :
            service:europe
            service:biz
            service:commodities
            service:funnies
    ignoreforservice: Ignore certain files in a particular service
        eg  ignoreforservice:epd QA=pp
            if the FipHdr field QA is 'pp' then ignore that file on service EPD

    http-proxy-host: Hostname of Proxy HTTP server. default: direct connection
    http-proxy-port: Port Number of Proxy server.   default: direct connection
    http-proxy-logon: This is the logon and password to get thru the firewall
        if required. The format is (logon) (colon) (password) and is
        converted to base 64.
        http-proxy-logon:Y2hyaXMuaHVnaGpvbmVzOnBhbnRoZXIK=

        To generate (use 'type' on Winnt and just 'echo' for other Unix) :
            echo -n "logon:password" | sffb64 -i
        eg  echo -n "chris:sleekpanther" | sffb64 -i
        gives   Y2hyaXM6c2xlZWtwYW50aGVy
            http-proxy-logon:Y2hyaXM6c2xlZWtwYW50aGVy=

    protocol-header-file:
    protocol-trailer-file:
        Used where the protocol, like HTTP, requires extra information
        before and/or after the data.

    text-format: (name of a format file in remote/formats)

    encrypt-size: (number)
    encrypt-string: (FipSeq string of encrypt-size)
        These state a length and string to map to the data.
        They MUST be exactly the same as the Remote system.

    allow-ip-address: 192.3.22.1
    reject-ip-address: (IP addresses/mask)
        Allow or reject connections from named systems.
        up to 100 addresses may be specified.
        if one or more 'allow's are specified, then the
        client MUST connect from one of these addresses.

    if start-stop,
        reconnect-every: Number of seconds to wait
        do-not-connect-on-days:sat,sun
        do-not-connect-before-time:09:21
        do-not-connect-after-time:19:30

    check-message-timeout:600
        Number of seconds of quiet before a Check Message is sent
        default is 0 for No check messages.
    check-message-file:
        Name of a file in tables/remote which contains a Data file
        with a suitable FipHdr to use for a Check Message
        The default is CHECK_MESSAGE
    response-timeout:60
        Timeout for 'server' type connections waiting for an
        acknowledgement for the data.

    speed: (number)
        Pace the data out at this speed (in bps).
        Use this where a serial line with no flow control is on
        the circuit.    eg  speed:9600
        default: not specified.

    allow-remote-repeats: yes/no
        Allow the remote user to specify a sequence number to restart transmissions
on.
        default is no
    fiphdr-client-id: (2 chr FipHdr field)
        FipHdr field in the incoming data which holds the client id which will be
used for resends.
        default the absolute sequence id.

    dump-data:
        archive the data sent in a dump file in /fip/dump/DOY_CLIENT

    do-not-send-duplicates:
        If a single file is in more than one service, then it is normally
        a copy is normally sent for EACH service.
        Use this flag to send only one copy.
        There is an item in the log which says that rg second and subsequents are NOT
sent.
    send-all-old-data:
        Normally all files are sent to a client.
        However if the client disconnects at the end of a day (ie before
        midnight) and does not reconnect for hours afterwards, they
        will normally ONLY get files from midnight.
        Use this option to force all of the previous day too.
    send-no-old-data:
        Send no back data previous to the logon time (see above)
    create-ZQ:
        Use this to create a fake output sequence number in the FipHdr
        field ZQ. If there is a number to this parameter it will be used
        as the amount to increment - default is 2.
        eg default create ZQ:3-4, the next ZQ:5-6
        if create-ZQ:12, file will have ZQ:22-33 then ZQ:34-45 etc.
        if create-ZQ:1,  file will have ZQ:2-2 then ZQ:3-3 etc.
    track-high-priority:yes/no
        If 'ipremdata' is also set to track these, this will send the
        HiPris before any other traffic.
        This option is NOT for Groups - only single Clients.
        This automatically flags 'do-not-send-duplicates:'
    log-duplicates:yes/no

    create-md5-signature:
    check-md5-signature:
        Create the MD5 signature of the incoming file and send to the FipRemote
running in Server mode
        The file is logged if in error at the remote end.
        default: none
    output-md5: (FipSeq)
        Output this string with the MD5 signature as FipHdr RM AFTER the data has
been sent
        Use this for straight braodcast feeds
        eg - top and tail with an ETX and EOT
            output-md5:\003\RM\004
        default: none
    encrypt-script:
        External script to encrypt or compress the data.
        There is one and only one parameter to this script which is
        the name of the file to encrypt and in which the new contents
        will be stuffed.
        encrypt-script:/fip/local/remote_pgp

    log-message: (FipSeq)
        logging information for fipremote   default: nonw

The text-format file has the Syntax of :
    ; comment
    before: Text to add at the top of the data file.
    after:  Text to add at the bottom of the data file.
    filebefore: File to add at the top of the data file.
    fileafter:  File to add at the bottom of the data file.
    zaptags: If the source data is in XML (or HTML or any other tagged
        format), zap all the tags.
        eg  tag:P   \r\n\t
    tag: Replace a start tag with some string   (eg <p>)
    endtag: Replace an END tag with some string (eg </p>)
    alonetag: Replace an Alone tag with some string (eg <br/>)
        Syntax: endtag:(tag)    (replacement data in FipSeq)
        There can be a 'tag' for each that needs replacing.
        Do not use these with wrap - pretty meaningless if you do anyway !
        It is case insensitive.
        tag:P   <text>
        endtag:P </text>
        alonetag:br <padline/>\n
    force-binary: If the text starts with this string, assume it is binary
        and make NO changes to end-of-line of characher maps
        force-binary:<p>
    chr:(chr)   (replacement data in FipSeq)
        Syntax: chr:(chr)   (replacement data in FipSeq)
        There can be a 'chr' for each character that needs replacing.
        chr:\006    +Bell+
        chr:\037    <TD><!-- \037 -->
    hdrchr:(chr)    (replacement data in FipSeq)
        As for 'chr' but for FipHdr fields only.
        Use this to map XML entities for example
            hdrchr:&    &amp;
            hdrchr:< &lt;
            hdrchr:> &gt;
            hdrchr:"   &apos;
    txchr:(chr) (replacement data in FipSeq)
        As for 'chr' but for 'before', 'after'. 'filebefore','fileafter'
        Use this for any '#' in the FipHdr fields for example
    paragraph-marker: End of line sequence in FipSeq
or  eoln: End of line sequence in FipSeq
        default is to leave the end-of-line as in the data
    ignore-eoln: Test for eoln - if therem, do NOT add the eoln
        eg  eoln:</p>\n<p>
            ignore-eoln:</p>
        In this case if there is a single preceeding <p>, no changes to the eoln will
be made
        If not, all end of lines (usually NL or CR NL) are replaced with '</p> NL
<p>'
        This works on a file by file basis so files with/without <p> can be
interlaced in the output
        default - none
    wordwrap: Wrap the text at a number of characters per line.
        This will wrap on the nearest space to the end of the line.
        default is to not change the data at all.
        eg  Wrap the text at 72 characters.
            wordwrap:72
    ww-eoln: For word-wrapped lines, End-of-line sequence in FipSeq.
        default is CR NL ie '\r\n'
        eg  ww-eoln:\f
    add-wordwrap-for-service: ONLY wordwrap for this service. There can
        be several lines each specifying a single service.
    remove-wordwrap-for-service: Remove any existing wordwrap for files for
        a service.  There can be several 'remove-wordwrap-for-service'
        lines, one per service.
        Eg : if files for service 'biz' are wrapped at 70 chrs and
        you want to have NO wordrap :
            remove-wordwrap-for-service:biz
        This is done BEFORE applying any wordwrap , so if you want to
        put a wordwrap on 40 chrs:
            remove-wordwrap-for-service:biz
            wordrap:40
    convert-to-utf8:
        Convert the incoming data to the UTF8 variant of Unicode.
    max-data-size-for-service:(servce name) (no of chrs)
        To create an abstract wire of the first n chrs for a particular service
        Headlines only, set this to 0 (zero)
        eg  max-data-size-for-service:biz   0
        default is -1 for ALL data/text
    repeat-service-name: Service code for Repeated items.  default is 'RPT'
        To use the original, use a blank 'repeat-service-name:'

    service-name-alias: Original Service code and replacement
        eg service-name-alias:XX2   SPT
        There can be up to 1000 aliases
        The aliased name does not have to be unique

Input Parameters :
Mandatory
Either  -g : name of a Groups file in tables/remote/groups  default: none
or  -n : name of a single Client file in remote/clients default: none
Optional
    -N : replacename name for the client            default: -n
        This allows a 2 clients to use the same parameter file
        but have different idents.
        Resending files from Fip will go to both clients
        Care MUST be taken to have UNIQUE client names on a single server
    -s : hostname as override to parameter file     default: name
        for client-started feeds
        eg -s + to listen on all ports
    -P : port as override of 'port:' in  parameter file default: port
        for client-started feeds
    -Q : do NOT send Seqnos (ie remote is fipremote < 2e12)  default: yes
    -l : do NOT log each item               default: do
    -LOCAL : only process this client (or Group) locally    default: network
    -T : do NOT log the trace of each transaction       default: do
        Normally a one line log of each file sent is stored in
        log/remote_trace with a name of 'date_client'.
        Use this flag to stop this trace.
    -V : version level                  default: 3
        (use for old remdata)
    -v : Print the version number and exit

FipHdr fields POSSIBLY used in the incoming data and used by ipremdata/svr :
    R0- clientname for traffic for a particular client only
        This is used for resends and system messages
        R0:MegaCorp
    R6- Mime Type of file
        R6:TEXT - used for flagging to use the TEXT format
        Any other value/string is user-defined.
    RX- Full path and filename on this device

FipHdr fields added by ipremdata are :
    R1- date CCYYMMDD
        R1:20000630
    R2- hour
        R2:09
    R3- Overall System item number
        R3:06132
    R4- Uno
        R4:200006300906132
    R5- SystemId - not used by RemSvr

FipHdr fields added by ipremsvr are :
    R7- Service item number
    R8- Client item number
    R9- Service name, usually a service code
        R9:COM

--------------------------------------------------------------
Version Control
;2z17   15dec03 cleanups
    ;d-f 27feb04 cleanup Groups
    ;g-h 29jul04 added 'track-high-priority:'
    ;i-k 15aug04 added check-md5-signature and encrypt-script
    ;l 07jan05 bugette over chg of day
    ;m-n 26jan06 added allow-remote-repeats:yes/no (n dump md5)
    ;o 24sep06 added tag, endtag, zaptags and alone tags,
    ;p-q 04nov06 added -P port as override to parameter file
    ;r 06nov06 added send-no-old-data: and -N
    ;s-u 22nov06 added ignore-eoln
    ;v-w5 11dec06 added -k remkey
    ;x-y6 8may07 added service-name-alias and StuffFile bugette (y4 no-pwd for
fipremote)
    ;y7-10 for txchr: (10 for space in service-..alias)
    ;z1-9 17aug08 added take-size/fiphdr/top/tail (2 bugfix for output buffer) (5
new services buglette)
        ;10-11 27apr10 added local-fiphdr and local-extra-fiphdr
        ;12 added SEQNO and tracking for super bad links
        ;13-17 added log-message and -Q for no SEQNO
;001z   22feb01 no-password/no-logon-required added for fipagent too
    ;a 20mar01 added proxies
    ;b 27mar01 added wordwrap properly
    ;c/d/e 20apr01 bugette in non-encrypted feeds
    ;f 27jun01 WINNT
    ;g 27nov01 closeup if bad logon.
    ;h 10dec01 added do-not-send-duplicates.
    ;i/j 08jan02 added hdrchr
    ;k/l 09mar02 added remove-wordwrap-for-service
    ;m 11apr02 added convert-to-utf8/base64
    ;n 26apr02 added add-wordwrap-for-service
    ;o 02may02 ignore the Missing msg if Pos=0
    ;p/q 09may02 added send-all-old-data:
    ;r/s 12jun02 finally got the svce bug !!
    ;t 28sep02 added max-data-size-for-service:
    ;u 27mar03 added ignoreforservice:
    ;v-y 23may03 added createZQ
    ;z 05dec03 added repeat-service-name

(copyright) 2024 and previous years FingerPost Ltd.