ipbdcast (Sat Oct 25 2014 01:31:01)


	Variable-headered broadcast text-out program

IPBDCAST is used to send text down a leased line. It uses a Format table in
tables/bdcast to define the message format for that file.

In order to present a standard feed to clients, different types of messages
from different sources use different format files which reflect the different
information held in the header fields.

Features include :
	- Check message generation or suppression.
	- Conversion to non-ascii character sets on the fly.
	- Creation of log copies of transmitted files
	- Exact dumps of transmitted data.
	- Writer flag for Writer PCs to review traffic.
	- High Priority BreakIn and Interrupt.
	- Ignore messages which are older than a 'sell-by' date/time.
	- Send messages either in Unix queue order or priority/time.
	- Wordwrap at defined line lengths so that the Editorial system does not need
to HNJ the messages before transmission.
	- Split into takes where the Receiver cannot accept large files
		or split on a string in the data
	- Send/Post to an HTTP server like Apache, Domino, NetScape/iPlanet or IIS
	- Failover to a second port/server
	OR Failover to another technique such as dialout or ftp

IPBDCAST reads the file to send, searches for the DB FIP header field which
says what format file to use (if no DB, the default file is used). It then
checks  whether the format file is already in core - normally 10 tables are
held. If no chk_msg, queue, format and archive are specified then they are set
to the same as 'name'.

It can also be used to send SOAP or other http POST - see later for notes on

Essentially the parameter file is used to describe the strings to be output for
the :
	- PRE HEADER	ie optional strings to start Satellite selectors etc
	- HEADER	ie strings to be sent BEFORE any DATA
	- TRAILER	ie strings to be sent AFTER all the DATA

Example of a parameter file
	; setup keywords- ascii like chrset, max 80 columns and insert CR NL

	; send the preheader
	; send the header
	; the text is sent now
	; send the trailer

The syntax of the Format file is :
	; comment line
	(keyword:) (subparameters) (eoln)

In more detail :

	pre:(FipSeq)	preheader string to be output
	hdr:(FipSeq)	header string to be output BEFORE any data
	tlr:(FipSeq)	trailer string to be output AFTER any data

These can be preset files of  FipSeq which are specified as
	http-header-file: (Filename)
	pre-file: (Filename)
	hdr-file: (Filename)
	tlr-file: (Filename)

The Http Header is used only for transferring data to a Httpd server like
Apache, Domino or IIS, It is output BEFORE any of the other files/templates.
The only diference between this and any other is that each line is ended with
one and only one CR NL and the header is terminated by a blank line even if you
do not specify one.

The PRE, HDR and TLR keywords plus default fields can contain any FipSeq
including :
	- printable ascii text (not NL, CR or FF as they end the string)
	- unix escape chrs : r, n, s, \, t etc
		such as r is a carriage return
			n is a line feed or new line
	- numbers : 01, 377
	- FIP header fields : SN, DQ
		See the Chapter on FipHdr fields for more information
		Note that Source Header fields (SH) can be accessed by X? ie XP for
priority, XC for category.
	- system variables : all the normal FipSeq ones plus
		Field   Remarks		 Example
		$A	J11 style NAME;dd/mm	JAMES;03/12
		$S	3 figure sequence number 123
		$Z	4 figure sequence number 0037
		$Y	year  (99)		97
		$E	century  (19 or 20)	19
		$I	month (mm)		11
		$M	month (Mmm)		Mar
		$D	day   (99)		23
		$H	hour  (99)		10
		$N	min   (99)		54
		$B	sec   (99)		45
		$J	julian day of the year   111
		$R	random alpha numeric character between a-z, 0-9
		$X	strip trailing spaces from this point to the last non-space
		$O	if Optional field failed, strip to this point
		$C	Total No of Chrs
		$W	Total No of Words
		$T	1st 64 chrs of text
		$U	1st line of text - up to 64 chrs
		$1	1st line of text
		$9	9th line of text

Note the normal FipSeq fields are available (see the System Admin Guide)
	fixed:QT	2233344
	partial:QU	ST,3,2,u
	combie:QX	ep|na,(000000)a
	option:QO	ep,11,7
	repeat:Q5	XK,+,1
	style:Q6	HJ,%.03d
	replace:Q7	XC	abc=def	spo=dog
	unique:Q8	XC,=
The old style notation for these fields are also allowed :
		Fixed text fields 	ie %QT:2233344
		Partial hdr fields	ie %QTST,3,2,u
		Combination fields	ie %QT|ep|na,(000000)a
		Optional fields		ie %QT?ep,11,7
	or	Repeating fields	ie %Q5*xk,+
	Note that the punctuation following the header field defines the type of
		eg For partial fields

Other (optional) keywords are :

	chrset:	ascii, baudot or none		default: none
	eol:	string for end of line		default: none
	spceol:	string for end of line on non CRLF default:none
or	wordwrap:	max line length or 0 if none	default: 0
	The above 4 keywords are used to automatically wordwrap text so it
	does not need to be hnj-ed by the editorial system. In all 3/4 cases
	the parameter can be in FipSeq.
		; a para is FormFeed, an inserted wordwrapped line is CRNL and
		; the actual line length is whatever is in the AW FipHdr field or 80

	log-line: Replace line for the item log if sent successfully in FipSeq
		eg	log-line:DB SU SN ER
	log: destination of a log copy of file as it was sent. default: none
		ie log:sgwcpy
		where sgwcpy is a valid destination (DU) in tables sys/USERS.
	fiphdr-for-log:(FipSeq) Additional information to add to the FipHdr of the
'log' file. This should be in FipHdr format and be in
		FipSeq. It can be used to pass FipHdr fields in the outgoing file into the
log file.
			eg	fiphdr-for-log:	DF:statusnSS:SSn
			default: nothing added
	size:	min, max or fixed length of hdr fields. default: size of data
		ie size: ep 1,5 , xk 24
		ep field is min 1, max 5; xk is fixed at 24
		(Note that no FIP header field starts with a number).
	delay:	The delay-for-one-second character		default: none
		If specified, when the chr is encountered in the pre:, hdr:,
		or tlr: the program waits a second before continuing
		This an be a printable character (like punctuation), Unix escape
		or an octal number AS LONG AS it is not needed for any field.
	inter-file-delay: Delay in milliseconds between each file - default is 0 ms
	pkt-delay: Delay in milliseconds after each 1000 chrs sent - default is 0 ms
	wrdlen:	Length of an average word for the FipSeq variable $W
		This defaults to 6. To estimate the number of lines in the
		file, this can be set to say 48 or 49 if the line length is 64.
		This MUST be greater than 4.
	stopww:	A single chr used to flag that the following text should NOT be
		word wrapped. For example a table embedded in text.
	startww: A single chr used to flag that the following text should be
		word wrapped.
	wrap-header: wrap the header too
		if the parameter is '0' then the same as text
		if a positive value, wrap at this.
	take-splitter-string:  Split the outgoing file into takes at this point in the
		default is Do NOT split.
	split-delay: Delay in milliseconds between each take of a split - default is 0
	max-take-size:  Split the outgoing file into takes with this maximum.
		default is Do NOT split.
	take-start:	Text (in FipSeq) to add at the begining of a take
		but NOT the first file.	 default: none
	take-end:	Text (in FipSeq) to add at the end of a take
		but NOT the end of file.	default: none
		Note that FipHdr fields are filled in
			D1 holds the take number.
			D2 holds the last take number
			D3 holds the next take number
			D4 holds 'YES' to mean this is part of a multi-take
				or NO if the filesize is smaller so no splitting is needed.
			D9 holds 'YES' for the first take overall.
		D1 may be changed to another FipHdr field using 'newD1:'
			eg	newD1:PD
		This also changes D2, D3 and D4 to follow sequentially so in
			this case these would be PE,PF and PG
	extra-fiphdr-for-takes: (fiphdr)
or	take-extra-fiphdr: (fiphdr)
		This is for extra FipHdr information in addition to the D1-D4 above.
	major-take-start: (FipSeq)
	major-take-end; (FipSeq)
	major-take-every: (number)
		Use these for a second level of takes where every X takes results in
		a slightly different start and end.
			D5 holds the major take number.
			D6 holds the last major take number
			D7 holds the next take major number
			D8 holds 'YES' to mean this is a major part of a multi-take
				or NO if it is one of the intermediate ones

	send-no-data: send NO data from the file at all.
		Probably this would be used ONLY to send a header/trailer.

	newD1:FipHdr field for the Take Number
		D1 may be changed to another FipHdr field using 'newD1:'
			eg	newD1:D6
	fiphdr-for-hostname: (2 letters)
		Two character FipHdr field that will contain the Hostname
		so it can be used in the Hdr/Tlr of the message.
			eg	fiphdr-for-hostname:JH
			and the hostname is 'wobbly.wibble.com'
			will give	JH:wobbly.wibble.com
	fiphdr-for-ip-address: (2 letters)
		Two character FipHdr field that will contain the IP address
		so it can be used in the Hdr/Tlr of the message
	fiphdr-for-service: (2 letters)
		Two character FipHdr field that will contain the service
		(ie the -n input switch) so it can be used in the Hdr/Tlr of the message
	fiphdr-for-uuid: (2 letters)
		Two character FipHdr field that will contain Fip generated UUID (RFC4412
		There can be multiple fipHdrs specified separated by commas/spaces
		each one will be given a difference UUID - probably for a different use - for
each file.
		eg fiphdr-for-uuid:F1,F2,F3

		When sending to an HTTP server like Apache, use this to make sure the file is
Posted correctly.
	http-ack-fiphdr: (2letter)
		FipHdr to save the http response - eg 201 File Created
		default : none
	ack-wait-timeout: (no of seconds)
		Number of seconds to wait for an ACK from the remote server
		default is 20 seconds.
	http-error-folder: (Path)
		When sending to an HTTP server like Apache, this folder
		will hold any files which have NOT been posted correctly.
		(ie response code from 300->599)
		Otherwise the files are retried - possibly forever !
		Files are ONLY put in this folder if the return code is a valid one.
		- so files which timeout (if the server is down for example) are retried.
	http-keep-alive: yes/no
		Keep the HTTP connection alive or not. 	default for http is YES
		so the connection is dropped only when there are no more files
		to send.
	save-ack-response: (FipDU)
		Where the HTTP server may be sending a long xml response which
		will need to be converted/processed/displayed somewhere.

	locale: use a different 'locale' ( look at the man pages for locale)
		Most computers are set to US English and never changed.
		Use this parameter to customise any date/time or Chr translations
		The parameter MUST be a valid locale on your system!
		To find out what valid locale exist, type 'locale -a' on unix.
		Eg	; Set for for Brazil, portugese
	balance-seqno:	Send the Sequence number to this Balance Group (see
	balance-delete:	Delete the Data on remote systems once sent
			using this Balance Group (see 'ipbalance')
	no-archive: Do NOT archive the incoming files in /fip/log/data
			default: archive
	connect-timeout: Max time in seconds for the connection
			The call is aborted if it takes longer than this.
			If the remote is across the computer room, this time can be
			reduced to 5 or 10 secs; if over the internet, 120 secs is the norm.
			default: 30 for 30 seconds
	connect-wait: Wait in secs after the connecting before sending any data.
default: none
	disconnect-wait: Wait in secs before dropping the connection	default: 3 secs
			Use this when the '-D' flag is set on those platforms like RS6000
			which have nasty socket-closes.	For HTTP connections this is set to 0.
	logeachfile:(dest) Send a Success/failed msg to this destination for each
			There is no default.
			This log file is just a FipHdr with the following extra fields :
				DR-File Sent OK	 DR:ok or DR:error
				DG-Will Retry later	DG:retrying or DG:stopped
				DT-Some message text	DT:No connection
				DO-Number of last attempt	DO:5
				HT-Date and Time	HT:25 Nov 98 15:35:25
			default: no log created.
			The DR and DG messages can be changed by using :
			eg: log-dr-ok:File VN sent to Remote Host ($h:$n)
	msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
			msg. This should be in FipHdr format and be in FipSeq. It can be
			used to pass FipHdr fields in the outgoing file into the log file.
			eg	msgeachfile:	DF:logdialnSS:SSn
			default: nothing added
			Hostname and port of a second remote connection to be used if the main link
is down.
			The default port is the same as the existing port.
			This option is only applicable for connections to a remote socket and not
for x25, passive port, tty etc.
			'failover-host' is a single entry and will always be used.
			'failover-other-host' is where there are two redundant systems and
			we want the OTHER system that is NOT specified as the '-s'
			input parameter to be the failover.
			If NOT using failover-other-host, there can be up to 4 hosts and ports -
failover-host3 etc.
	failover-check-time: (seconds) Time in seconds before trying the primary
			if we have failed over to the secondary.
			default is	failover-check-time:300 for after 5 minutes
		Use this to create a fake output sequence number in the FipHdr
		field ZQ. If there is a number to this parameter it will be used
		as the amount to increment - default is 2.
		eg default create ZQ:3-4, the next ZQ:5-6
		if create-ZQ:12, file will have ZQ:22-33 then ZQ:34-45 etc.
		if create-ZQ:1,  file will have ZQ:2-2 then ZQ:3-3 etc.
	grab-first-file: Always process the first file in the queue first.
		This is for WinNT/2000 which sorts alphabetically/timewise.
	hdr-hash: (FipSeq for a single chr) This chr is replaced with a '#'
		eg hdr-hash:200
	data-must-end-with: (FipSeq string)
		If the data part of a take/file does NOT end with this, it is added.
	max-dollar-t: size of data for $t	default is 68 chrs
		This can be up to 2048 chrs
	ignore-chkmsg-log: yes/no
		If YES, do not log Check Messages		default: log
	bw-binary-hdr: (yes/no)
		Send 4 digit filesize before hdr		default: no

or	use-external-file:
		if there is an SX FipHdr field with a path to the data file, use that rather
than the data in the input file.

	ssl-method: (1,2,3,23,999)
		Version number to use for TLS/SSL		default: 999 for current default (2 or 3)
	ssl-password: (password)
	ssl-passwd: (password)				  default: none
		Optional password if the handshake requires a shared secret
	ssl-cert: (name of a PEM certificate file)		default: none
	ssl-root-cert: (name of a root PEM certificate file)	defaunt: none
		Optional certificates - held in tables/ssl

-------- SOAP and HTTP/POST notes ---------

In this case the data MUST be 100% unchanged by ipbdcast - any header and
trailers should have already been applied by a previous program sucg as ipxchg
or ipedsys.

And there MUST be no parameter which change the data - so no ww:, wws: etc

And it is rare for check_messages to be needed.

System entry would be something like

ipbdcast -s demo.stonewing.net -p 80 -q 2ww -D -m -n stonewing -d

in the Parameter file, some of the following parameters are recommended -

For tuning/debugging, use save-ack-reponse to check what is being sent back

; Save what is sent and is replied and send it to DU:saveack

; Wait for an HTTP acknowledgement

; default is 60

; Do not keep the HTTP connection alive

; Send a MIME header at the top of each file

; HTTP error directory

A simple HTTP Push template would be (for the http-header-file:) -

Host: financial_deformer.bampot.ukr
Content-type: text/xmlr
Content-length: $cr
Accept: * / *r
Accept-Language: enr
User-Agent: FippoWizzo 1.1r

- note the r at the end of each line and the extra blank line at the end
- note $c for the Content-length which will take the exact number of bytes of
the data part of the file to send
- replace the Host name with the name of the remote host (and port numberif not
- replace the Content-type with the proper mime type if not xml	text/plain,
text/html application/msword
- replace the path after the POST with the proper path

POST /ad_hoc/xxx_push HTTP/1.0r

For SOAP you can logon, download a response (in XML), extract information from
the response and use that ads more FipHdr fields for filling templates

-------- BAUDOT notes --------

If chrset:BAUDOT, all strings including octal or unix is translated as per
standard tables. NOTE that LET and NUM can be sent by sending { (octal 123 -
LET) or } (octal 125 - NUM). Nuls are normally stripped unless expressly sent
with '~' (octal 176). Chrs NOT In the Baudot chrset are sent as NULs.

-------- Input Parameters are :

Mandatory :
	-n : name of service					default: none
Either	-s : Terminal server (spider name) or TTY device name	default: none
and	-p : port number if spider				default: none
or	-P : Passive port number 				default: none

Optional :
	-a : archive file					default: name
	-c : Time check message file in tables/bdcast
		This MUST be in the correct chrset for the service.
		Note that the format should cater for time/date in the trailer.
		This is NOT required if '-m' no check messages is set.
	-d : take exact copy of outgoing text in ~/dump/SERDES	default: no
		used for debugging and testing.
	-D : disconnect from the port after each transmission   default: no
		This will connect to the remote host/port ONLY when a file
		is ready for sending and will disconnect after all files
		in the input folder have been sent.
		This allows more than one bdcast (on one or more systems)
		to send to the same port.
	-E : If the program cannot send the file in this number of seconds
		Move the file to spool/2dial for 'ipdial' to attempt.	def: no
	-f : default parameter (or format) file			default: same as '-n'
		in tables/bdcast
	-h : set High priority					default: 1
		Any file with a higher priority is set to the default (-R)
		Used for the Priority Send option only.
	-H : Fip Hdr field for Priority 			default: PR
		Only the first numeric part of the field is used.
		ie in PR:abc146-153, only 146 is significant
		Used for the Priority Send option only.
	-i : interval in secs between chk_msgs		default: 600 secs/10 min
	-I : force chk_msgs out 				default: no
		Use this to always transmit chk msgs every 10 mins or so.
		This will automatically adjust to though it is starting on the hour
		ie '-I -i 600' for 10 mins, chk msgs will be sent at
			0, 10, 20, 30, 40, 50 mins past the hour.
		An extra chkmsg will also be sent out immediately the program is restarted.
	-k : Delay all files by this number of seconds before sending	def: no delay
	-l : do NOT log items					default: log
	-m : no chk_msgs pls				default: chk msgs pls
	-M : extra dump id number				default: none
		 when using multiple bdcasts
	-o : send files in priority/time order	default: send in unix order
		Files are normally sent in the order they are in the
		unix queue. This switch is for low speed lines where
		we want to send in priority/time order.
	-O : done queue in spool for raw files			default: none
	-q : queue to scan in spool				default: same as '-n'
	-Q : keep quiet and do NOT message if the terminal	default: log
		server port is busy
	-r : set Lowest priority.				default: 9
		If the Priority is less than this, the default is used.
		Used for the Priority Send option only.
	-R : set the default priority for files with no		default: 3
		priority or priority out of range.
		Used for the Priority Send option only.
	-S : Pause before continuing to scroll.			default: no
		Time in seconds to wait before checking if any other
		file with higher priority is waiting. eg -S 5 for 5 secs
		Used for the Priority Send option only.
	-t : scan interval of the queue				default: 1 sec
	-T : make a log trace of each transaction		default: do not
		This creates a one line log of each file sent is stored in
		log/remote_trace with a name of 'date_service'.
	-u : default delay in milliseconds between takes.	default: 0 for no delay
		This can be overridden by the 'split-delay' parameter in the param file.
	-U : Break the Dump file (-d) into Hourly files.	default: no - daily
	-w : this is a writer double queue where a second queue
		(spool/pc/'name') is used for access by Writer	default: no
	-x : client specific for SGW : at end of each file/group of files
		 send 1024 spaces to flush their buffer		default: no
	-X : If remote device is a Host or Terminal Server,	default: 60 secs
		and it goes offline, attempt reconnect and message every -X seconds.
	-W : Wrap the data in Format header and trailer at this
		line length - same as 'wrap-header'		default: no
	-z : allow high-priority breakin			default: no
	-4 : make $W Word Count 4 digits with leading zeros	default: no
		no longer used - pls use 'style'
	-5 : make $W Word Count 5 digits with leading zeros	default: no
		no longer used - pls use 'style'

	-v : print version no and exit

This program uses 2 environment variables :
	FIP_USA_DATE y/n for dates
	FIP_BDCAST_WRITER for Writer style Pounds/Hashs

Note that in this version 'chrset' does not refer to a file in tables/chrset
but to the basic Ascii or Baudot.

Version Control
;059m29	07oct04 minors..
	;b-d 21jan05 added -M, bugette with speedy
	;e-h 28jul06 added remote_trace style stats to send msg as fiphdr H4
	;i-l 05apr07 better soap control
	;m4 18jan08 added soap-meta-script ;2 bugette when connection dropped during
		;4 added interDelay/pktDelay
		;5-6 added use-ssl and connect-wait
		;7-14 note_balance_action added and -B default is binary and wireId for
seqno; pktdelay
		;15 added use-sx
		;16-22 minor mod to failover - altport defaults to the main port if NOT
			 added alt234 and ignore-chkmsg-log ;22 bugette with long filenames
		;23 26feb13 added bw-binary-hdr:yes ;24 wireId in timings ;25-26 added
httpAckFH and content-length tracking
		;27 30apr14 added file_trace
		;28-29 28may14 bug-if speedy, no chkmsg!
;058z	18apr02 added major-takes*
	;a 23apr02 added wordwrap as well as ww and made it FipSeq
	;b/c 28may02 added grab-first-file: and SerDes is now FipHdr SF:
	;d 19jun02 added chgchr to text as well
	;e/f 17oct02 added split-text-string
	;g 19dec02 added 'data-must-end-with:n' and buglette with 1st splitter string
	;h/i 01jan03 MACOSX
	;j-n 12mar03 added split-delay/bugette in splitter/hdr-hash in splitters
	;o 17sep03 added -U : hourlyDmpFiles = yes
	;p 28oct03 added timing-stats:
	;q-w 08mar04 better handling of http error q and connect-timeout
	;x-y 12aug04 added save-http-response:
	;z 23sep04 speedy
;057z	26apr00 added -k for delaying ALL traffic x seconds
	;a 07jul00 cleanup
	;b/c 29nov00 improved logeachmsg
	;d 02mar01 cleanup after network error better
	;e/f/g/h 13mar01 added Takes plus tweaked Check Messages
	;j 04sep01 added newD1 - FipHdr D1 is ONLY output if there is a take-end/start
	;k/m 10sep01 added fiphdr-for-hostname/ip-address and log-line
	;o 21sep01 added failover-* and http-header
	;p/q/r 24sep01 added take-extra-fiphdr and added the date to the dump file
	;s 12nov01 if fixed interval chkmsgs, start with a chkmsg
	;t 08jan02 added failover-check-time
	;u/v 07mar02 added send-no-data
	;w 18mar02 added remote-trace
	;x/y 03apr02 create-ZQ added
	;z 16apr02 added (optionally) D2, D3 for previous/last
		plus D4 is filled in at first go if filesize > than takesize

(copyright) 2014 and previous years FingerPost Ltd.