ippost

ippost

IPPOST is used to send a single file to a group of destinations or a
combination of groups and clients. Its main use is for syndication, library,
telex, fax and mailbox. It compares FIP header fields to the GROUPS parameter
file in tables/sys.

IPPOST scans the 'post' queue and for each file it checks the source,
destination (DU) and sub-destination (DA hdr field) and decides which
destinations it is to go.

Slightly different rules are used for differing types of traffic.
- All syndication and library clients MUST be specified in the GROUPS file.
- All mailbox clients MUST be specified in the mail/LOGON file.
- All ftp, internet-mail, dialup, telex and fax destinations are allowed to
pass through.
    Fax clients get sent to POSTFAX, telex to POSTTLX, ftp to POSTFTP and dialups
to POSTDIAL - all of which must be in USERS

Note that client names are case INsensitive so it matters not at all whether
the name is Upper or lower or both.

The groups file has the following syntax :
    ; comment
    group: (name)   (list of clients in this group)
        eg :    group:syn   profile mead    dtimes ql
    client: (name)  type: (type of client)  number: (telex of fax number)
or  client: (name)  type:other  dest: (destination)
        eg: client:gorby    type:telex  number:064-77878
        eg: client:gorby    type:other  dest:russia

WHERE Client types are :
    mail    - mailbox, files loaded into spool/mail/logon
        where logon is in tables/mail/LOGON.
    syn - syndication, file moved to spool/syn/clientname
    lib - library download, file moved to spool/lib/in
        for further processing by iplibfilter or iplib
    pc  - pc, file is translated and moved to
        spool/pc/clientname with a pc style filename
    bdcast  - news agency broadcast, file moved to spool/clientname
    dialup  - send file using dialup    (probably 'ipdial')
    ftp - send file using ftp       (probably 'ipftp')
    fax - send file using fax
    telex   - send file using telex
    alertmsg or atexmsg - msg for ipbox logon (or the Atex j11)
    email
    unixmail - mail for unix sendmail, file send to spool/2go
        for wheel to distribute using DU TOUNIXMAIL, DA client
    other   - send to another destination for this client.
        The 'dest' subparameter MUST exist for this client.
        If not the default dest is OTHER.
    ignore - ignore all files for this client !!

    Note that it is just the first letter of the client type that is checked and
is unique (or first two for FTP).

Various subparameters are possible for each client line (except for ALERTMSG) :
    Note that spaces are used to separate the subparams, so embed with double
quotes (or use '\s'
        if you need to enter them in a string.
    type:   as above. This is the only mandatory subparameter.
    number: for EMAIL   this is the email address of the subscriber.
        for TELEX   this is the telex number of the subscriber.
        for FAX     this is the telephone number of the subscriber.
        for DIAL    this is the telephone number of the subscriber.
        eg  client:fippo    type:unixmail   number:dot@fingerpost.co.uk
    dest:   for other   this is the DU or destination (in tables/sys/USERS)
    fiphdr: by default the FipHdr is stripped off the output file.
        Use this to retain it.  Generally this is used to retain
        valuable FipHdr information for 'bdcast', 'other' types.
        There is no data with this subparam.
    extra:  Add more FipHdr fields to the outbound FipHdr.
        Any FipSeq can be added. Remember to add 'fiphdr:' too to carry
        over any old fipHdr fields, otherwise they will be ignored.
        eg  client:drongo type:bdcast extra:\nQD:PostTime=\$h:\$n\n
    before: Add data at the top of the output file.
        eg  client:helene type:telex before:"Telex fpr The Mekons\n"
    after:  Add data at the end of the output file.
    xchg:   Run 'ipxchg' against the output file before sending/storing it.
        eg  client:daswong  type:bdcast xchg:cr2nl
        where there is an xchg parameter file called tables/xchg/CR2NL
        For FTP, Fax and Unixmail this xchg is NOT run by 'ippost' but
        is flagged so that 'ipxchg' is run in the usual Fip Copy Flow.
    convnl: Convert all CR or NL to CR NL to create a Dos/Windows
        readable file.  This is automatically triggered for 'type:pc'
    outfmt: The Output format file - FipHdr field DF or (for bdcast) DB.
    queue:  Place all files in this queue under spool/syn. This overrides
        the default which is :
            for dial    spool/2go
            for ftp     spool/2go
            for fax     spool/2go
            for unixmail    spool/2go
            for syn     spool/syn/(name of client)
            for bdcast  spool/(name of client)
        This allows several clients or sub-clients to share a transmission.

-- CSV files instead of the default-style GROUPS
    ; ........
    ; define the Clients file and how it is structured ...
    ; (all lines NOT starting 'E|' are ignored)
    ; E|0000|goa_cit    |Guardian|Business||UK_GUARDIAN|||GDN|CIT
    ; E|0000|indy_eye|Ieye|All|jpress_eye+eye_news|UK_ASSOC|||/ind/news
    csv-client-file:/fip/tables/setup/ALL_ADDRESSES.LOOKUP
        filename of the CSV
    ; Sep is a Pipe
    csv-client-sep:|
        field separator
    csv-client-key:3
        which field is the client key
    csv-group-key:6
        (optional) which field is the duplicate-this-file-to-these clients
        Use this to split/override clients
    csv-default-type:remote

    ; ........
    ; Groups files are individual files in tables/channels with the same name as
the Group
    ; Incoming files can specify Group codes which will be expanded
    ; Each Group file contains only the client codes (which must be specified in
the 'csv-client-file' above.)
    ; folder where the group files are held in /fip/tables
    csv-grp2cli-folder:channels

    ; Syntax of the Group files
    ; the client key - which is all we want is the 2nd field
    ; H|Code|Status|Publication|Description|Extra|Comment
    ; E|es_nws  |live|EveStd|News
    ; (all lines NOT starting 'E|' are ignored)
    csv-grp2cli-client-key:2

(THe difference between CSV-Group files (normally in channels)s and the
'csv-group-key' in the csv-client-file' :
    channel/(group) file    - these are clients chosen by the Sender.
    group-key       - these are used by sysAdmin to override the choice, temporarily or
permanent

Any errors such as 'No Such Routing', are sent back to the sender if there is
both a source header field (SU) and a sub-source or author field (SA). In any
case they are logged.

Text is NOT translated except for 'pc' clients where NL-> CR NL unless FBIN.

For Alert messages, (ie if DU:ALERTMSG or ATEXMSG) then DA holds the logon for
the message. IPPOST checks the tables/mail/ALERTMSG (if it exists) or the
ATXMSG file (if it exists) and sends to a DU of MSGFORALERT or MSGFOR ATEX
respectively.
    If not then the tables/mail/LOGON file is checked - if there, it is added to
the spool/mail/LOGON; otherwise it is sent back if there is an SA else it is
junked.

The destination for Msgs to Atex is MSGFORATX_NETWORK (uppercase) which should
be in the destination file (tables/sys/USERS) and uses output format (DF) of
ATXMSG which should be in tables/out, tables/news or tables/gateway.

For Fax, telex, ftp, unixmail (ie internet-mail) and dialup - If a specific
'queue' subkeyword is present for that client, the files are stuffed in that
folder. Otherwise all files are moved to spool/2go for 'ipwheel' to distribute
using a destination of :
    email   - TOUNIXMAIL
    unixmail- TOUNIXMAIL
    w4  - POSTW4
    ftp - POSTFTP
    dialup  - POSTDIAL
    fax - POSTFAX
    telex   - POSTTLX

All incoming syndication and library files are saved in saves/DU - ie saves/lib
for library. (NOTE version change - this is now valid ONLY if the '-S' input
switch is specified).

Input Parameters are (all optional) :
    -q : input queue            default: spool/post
    -i : input queue            default: spool/post
    -d : done queue for incoming files  default: none
        Normally files are deleted after processing.
    -f : extra fiphdr for all files     default: none
    -g : name of parameter file     default: tables/sys/GROUPS
    -G : FipHdr field containing the name of the GROUPS file
                        default: none - always use GROUPS
    -t : scan time for the input queue  default: 5 secs
    -l : do NOT return logs to sender   default: log all errors
    -L : log all client entries for each file   default: no
    -m : report errors as log type "ox"   default: report as a failure "x"
    -M : do NOT report if the client does NOT exist default: do
    -O : (2 letter FipHdr field)        default: split
        do NOT split into separate output files
        output is a single file with this FipHdr containing all the expanded codes
    -X : FipHdr containing prefix for Group names   default: none
        if -X NS and the NS:nell, then groups can be called 'nell/(groupname)'
    -z : name of parameter file     default: tables/sys/GROUPS
        (same as -g)
    -Z : name of the optional POST file default: sys/POST
    -v : version number and exit
others :
    -1 : single filename to run and stop    default: spooled
    -9 : do NOT run speedy on a speedy Fip. default: do

An optional Parameter file - tables/sys/POST can be used for:
    ; comment
    okmsg:\r\nLooks good to me\r\n
    dudmsg:\rnAbsolute rubbish, Thanks\r\n
    client-not-found:(destination)
    fiphdr-not-found: (2 letter FipHdr field)
    log-copy:(FipHdr DU(s))
    log-raw-copy:(FipHdr DU(s))
    balance-group:(Balance Group)
    balance-all-group:(Balance Group)
    balance-this-type: (POST type of file)
    round-robin-dial: (number)
    round-robin-ftp: (number)
    round-robin-unixmail: (number)
    round-robin-fax: (number)
    round-robin-telex: (number)
    add-unique-id: (optional FipHdr field)
    global-extra-fiphdr:    (FipHdr to be added to all files with 'fiphdr')
    newSN: (FipSeq for the SN)
    newDA: (FipSeq for the DA)
    chkexists: (Full Path/file or queuename)
    chkexists-timeout: (time between trying/logging if path is not there)
    xchg-program: Replacement path/file for ipxchg      default:/fip/bin/ipxchg
    xchg-queue: Replacement output folder for xchg      default: xchg
    wheel-queue: Replacement output folder for wheel/2go    default: 2go
    syn-queue: Replacement output folder for syn        default: syn
    extra-email-fiphdr: (FipHdr)
        Send a copy of the incoming file to the email addresses in this FipHdr (if it
is non-blank)
    check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
    log-line: (FipSeq) log line
    default-xchg: (name of xchg)
        A default if none has been specified for a client   default: none at all
    done-queue (done queue for input data - as per -d input switch) default: file
is deleted
        eg done-queue:/fip/data/raw.data/\$e\$y\$i\$d_post

In more detail...
 - change the 2 messages  and to define any client types :
    ; comment
    okmsg:\r\nLooks good to me\r\n
    dudmsg:\rnAbsolute rubbish, Thanks\r\n

 - any entry in the FipHdr DA field which is NOT specified in the GROUPS file
(or LOGON etc for mail types) is normally ignored/deleted. To send them to
another destination for further processing, use
    client-not-found:(destination)
where destination is the usual entry in the sys/USERS file.
Plus    fiphdr-not-found: (2 letter FipHdr field)
        This adds a FipHdr with either a '+' if the client is valid and '-' if not.

 -  make a copy of all files going to valid clients and send to this
destination
    log-copy:(FipHdr DU(s))
eg  log-copy:w4xpost
Use this for W4 or another logging system.
    log-raw-copy: (ditto) is similar but is the IMPUT file (ie raw copy) rather
than the OUTPUT file which may have been through an xchg (UNIX only)

 -  make a mirror copy of all files going to valid clients using balance
    balance-group:(Balance Group)
eg  balance-group:balanceData
Normally only files NOT being passed to 'ipwheel' or 'ipxchg are balanced as
they too will balance their files.

To balance EVERYTHING, use balance-all-group:

Alternatively use 'balance-this-type' to balance ONLY files from that type.
ie  balance-this-type:syn

Alternatively use 'balance-on-queue-only' to balance ONLY files which are for a
client with a 'queue' subparameter.
ie  balance-on-queue-only:

 - Use several outputs for this type of delivery.
    round-robin-dial: (number)
    round-robin-ftp: (number)
    round-robin-unixmail: (number)
    round-robin-fax: (number)
    round-robin-telex: (number)
        Eg  round-robin-ftp:3
        So the first file for ftp is sent to POSTFTP1
            the second  to POSTFTP2
            the third   to POSTFTP3
            the fourth  to POSTFTP1 etc

    add-unique-id: (optional FipHdr field)
        This adds a new FipHdr field containing a unique ID
        which is made from the IPaddress, time and sequence number.
        It is quite long !

    global-extra-fiphdr:    (FipHdr to be added to all files with 'fiphdr')
        Use this to add more FipHdr fields to an existing hdr.
        It works only for those clients WITH the 'fiphdr:' specified.
        It is is addition to the 'extra:' keyword specified on each line.

    newSN: (FipSeq)
        The FipHdr field SN (source name) is used to track the file thru the system.
If there is traffic which does NOT have an SN, use this to add one. eg
            ;  test if there IS an SN
            combie:Q1   SN
            ; add the SN if there is NOT one - use the original if there is.
            newSN:\Q1\SN#XX:\$o\WO-\WR-\WD-\WE
        In this example, if there IS an SN wth something in it, that is used.
        Otherwise, use the contents of WO, WR, WD and WE with dashes between.

    newDA: (FipSeq)
        The FipHdr field DA (sub-destinations) contains a list of all the groups
and/or clients to send to.
        If there are files which do NOT have a DA, use this to add one.
        eg
            ;  test if there IS an DA
            combie:QA   DA|AA,default
            newDA:\QA
        Here we say use the QA FipHdr field which will be either the DA if there, or
the AA FipHdr field if not, or the string 'default' if neither have data.

    check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
        Use this when an ipspool on 2 systems is accessing the same remote folder -
in a redundant way.
        that is used whether the current host should be getting the files or not.
    ie in the parameter file is
            check-primary-server:postsvr
        and in the DEST_REDUN is
            ; psuedohost    primary secondary
            postsvr fip1    fip2
        and in the SYSTEM file for both fip1 AND fip2 there is  the same 'ippost'
line with the same param
eter file
        If fip1 is up,  'ippost' on fip1 will always get while on fip2 it will just
check/loop.
        If fip1 is down 'ippost' on fip2 will start processing the files.

- CHKEXISTS is a complete pathname of either a standing file or a queue. This
is used to check that NFS drives are actually mounted BEFORE writing new files.
NOTE THAT PROCESSING BLOCKS FOR THIS FILE AND WILL BE RETRIED AT LEAST ONCE A
MINUTE.
eg : chkexists:/data1/nfs/helios/.Desktop
WinNT/2k remote drives (also called UNC) may be specified as :
(note the double \\ for each \
 and the 'ignore-escapes to ignore the dots in the IP address or hostname being
escaped):
The use of fwd slashes '/' rather than '\' make the string much easier to read
!!
Use 'chkexists' to make sure the drive is available first
    chkexists://195.13.83.99/d$/.DSstore

- log-line: (FipSeq) log line
    A couple of temporary fipHdr fields are added
        E1 - index number of the client
        E2 - name of the client
        E3 - type of method - FTP, EMAIL etc
    default is "client.\E2 : \SN \DU \DA (\E1.\E3)"

The incoming file may optionally have a FipHdr POST_LOGCOPY: (FipSeq) to
override the 'logcopy' parameter. The fipHdr can have NO data to disable just
for that one file.

The incoming file may optionally have a FipHdr POST_CLIENT_ONLY: which
signifies that the sub-destination (DA:) must be a client only - all groups are
ignored.

Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).

    They are normally specified :
        fixed:QZ    1234543
        partial:QT  ST,3,2,U,<,>
        combie:QY   ep|na,(0000000)a
        option:QE   ep,11,7,s
        repeat:QK   XK,-,3
    or  repeat:QP   PK,,4,#X
        style:QS    XN,%.03d
        replace:QZ  XK  a=b c=d ee=f

The current maximum number of clients in a group or a single transmission is
500.

Version Control
;55x38  30mar00 added Sort of Groups and the Groups ptr table
    ;a 28nov00 added POST_DIAL as a type
    ;b 01dec00 added add-unique-id:
    ;c/d 03feb01 added global-extra-fiphdr and newSN
    ;e 08feb01 added round-robin-telex
    ;f 15may01 bugette with case of first chr for Sort of clients
    ;g 17aug01 added 'newDA' as FipSeq for DA
    ;h 18mar02 added balance-this-type/DGtype
    ;i 22mar02 added POST_RESEND_QUEUE as FipHdr
    ;j 23mar02 cleanup of alerts
    ;k-m 17dec03 WINNT cleanup
    ;n 09feb04 bugette - allow 40 chrs for SN if none in FipHdr
    ;o 03nov04 make names up to 100 chrs from 20.
    ;p 11oct05 added FipRemote and Email
    ;q 17jan06 added chkexists...
    ;r-u 17aug08 added xchg/wheel/syn-queue and xchg-program
    ;v 14sep06 added validClientFipHdr + or -
    ;w 06nov06 added extra-email-fiphdr
    ;x1-2 28feb07 added FipHdr POST_LOGCOPY: ;x1 strparse logcopy ;x2 log-raw-copy
        ;x3 pramfile as -Z ;x4-7 added -L ;8 POSTW4 and reorder fiphdr
        ;10 5jan09 added POST_CLIENT_ONLY:
        ;11 28jun10 allow queue:/...
        ;12 Win2k missing clients better
        ;13-15 22jun11 added check-primary-server ;15 added log-line:
        ;16 added speedy
        ;17 30apr14 added file-trace
        ;18-19 4apr15 added -G -X -O
        ;20-25 29may15 added CSV files and default-xchg and -H no fiphdr on DUDmsg
        ;26-27 sort - no path bugette
        ;28-29 15mar17 cleanups ; 30 buglette if Group and Client had the same name
        ;31-33 9oct17 allow Personal List up to 64k !
        ;34 donque strparse - eg done-queue:/fip/data/raw.data/\$e\$y\$i\$d_post
        ;35-36 30apr20 major rework of in and out
        ;37 where is sort (diff rh v ubuntu)
        ;38 1jan21 buglette in old style GROUPS

(copyright) 2024 and previous years FingerPost Ltd.