ipremsvr (Sat Oct 25 2014 01:31:01)

ipremsvr

This program handles the connection with the outside world.

There are three basic modes which it can work :
	1. Permanent connection for a single client
	2. Start-stop connections for a group of clients
		This uses the -g input switch to hold a list of clients.
	3. Local copy for a group of clients using other transmissions
	such as email/SMTP and FTP which are serviced by 'ipsmtp' and 'ipftp'
		This uses the -LOCAL input switch for each client.

It copes with connections which are
	either client or server initiated
		ie Active or Passive - also called Push or Pull
	either broadcast or FipServer protocol
		ie fipagent or not - handshaking or serial-over-ip
	either permanent or start-stop connections

It adds the format and modifies the chrset (or converts the file type for
non-text files) according to the client's requirements.

'ipremsvr' first reads a parameter file in either 
	tables/remote/groups/GROUPNAME if the '-g' switch is on
or	tables/remote/clients/CLIENTNAME.

This will point to a format file too which is in tables/remote/formats
if a text-format has been specified , that is also read...

The GROUP file (if used) has the following syntax:
	; comment
	client: (clientname)
		There can be up to 10 clients specified. Each one will have
		a client file.
	connection-started-by:client or server
		Who makes the initial call - client connecting to the server
		or vice versa. default is server
		(if not LOCAL client)
	port: local port if connection started by client.
		(if not LOCAL client)

Up to 10 clients may be specified - ALL using the same 'connection-started-by'.
The 'connection-type' is 'start-stop' unless forced to be LOCAL by the '-L'
input switch

The CLIENT file has the following syntax:
	; comment
	delivery-method:bdcast or server or local
		The two options are 'bdcast' for one-way traffic (default) and
		'server' for fip-agent type traffic.
		If using the -L for Local, this is ignored.
		If using the 'fipremote' program at the remote site and the
		delivery-method is  'broadcast', make sure that the 'end-of-message' in
		remote's 'REMOTE.FIP' parameter file is matched in the format for
		'ipremsvr'.

	connection-started-by:client or server
		Who makes the initial call - client connecting to the server
		or vice versa. Default is server
		If using the -L for Local, this is ignored.

	connection-type: permanent or start-stop
		The connection is dropped for Start/Stop clients immediately
		after all valid data has been sent and acknowledged.
		This is used for low-data clients and allows other clients
		to be serviced almost at the same time.
		If using the -L for Local, this is ignored.

	local-folder: for LOCAL clients, this determines the local folder to stuff
data files.
		The default is /fip/spool/remotes/(clientname forced lowercase)
	output-filename : Define the filename (for both local and remote
		clients. The default is (SERVICENAME) (SEQUENCE NO)
		eg	output-filename:$dR9R7
	local-extra-fiphdr: (fipSeq) Extra metadata to add to the FipHdr for LOCAL
client files.
		default: none
	add-local-fiphdr: (yes/no)	Add/do not add the FipHdr to the LOCAL client
files.
		default: add

	host: hostname of the remote
	port: port of the remote

	password: Password the client uses to logon
	no-password:		There is no Password handshake
	no-logon-required:	There is no Password handshake
		(These two keywords have the same effect)

	service: name of a service that this client will receive
		There can be as many 'service' lines as required.
		Each line specifies one and only one service.
		The name of the service MUST be exactly the same as specified
		in the main SERVICES file as used by 'ipremdata'.
		Eg :
			service:europe
			service:biz
			service:commodities
			service:funnies
	ignoreforservice: Ignore certain files in a particular service
		eg	ignoreforservice:epd QA=pp
			if the FipHdr field QA is 'pp' then ignore that file on service EPD

	http-proxy-host: Hostname of Proxy HTTP server. default: direct connection
	http-proxy-port: Port Number of Proxy server.   default: direct connection
	http-proxy-logon: This is the logon and password to get thru the firewall
		if required. The format is (logon) (colon) (password) and is
		converted to base 64.
		http-proxy-logon:Y2hyaXMuaHVnaGpvbmVzOnBhbnRoZXIK=

		To generate (use 'type' on Winnt and just 'echo' for other Unix) :
			echo -n "logon:password" | sffb64 -i
		eg	echo -n "chris:sleekpanther" | sffb64 -i
		gives   Y2hyaXM6c2xlZWtwYW50aGVy
			http-proxy-logon:Y2hyaXM6c2xlZWtwYW50aGVy=


	protocol-header-file:
	protocol-trailer-file:
		Used where the protocol, like HTTP, requires extra information
		before and/or after the data.

	text-format: (name of a format file in remote/formats)

	encrypt-size: (number)
	encrypt-string: (FipSeq string of encrypt-size)
		These state a length and string to map to the data.
		They MUST be exactly the same as the Remote system. 

	allow-ip-address: 192.3.22.1
	reject-ip-address: (IP addresses/mask)
		Allow or reject connections from named systems.
		up to 100 addresses may be specified.
		if one or more 'allow's are specified, then the
		client MUST connect from one of these addresses.

	if start-stop, 
		reconnect-every: Number of seconds to wait
		do-not-connect-on-days:sat,sun
		do-not-connect-before-time:09:21
		do-not-connect-after-time:19:30

	check-message-timeout:600
		Number of seconds of quiet before a Check Message is sent
		default is 0 for No check messages.
	check-message-file:
		Name of a file in tables/remote which contains a Data file
		with a suitable FipHdr to use for a Check Message
		The default is CHECK_MESSAGE
	response-timeout:60
		Timeout for 'server' type connections waiting for an
		acknowledgement for the data.

	speed: (number)
		Pace the data out at this speed (in bps).
		Use this where a serial line with no flow control is on 
		the circuit.	eg	speed:9600
		default: not specified.

	allow-remote-repeats: yes/no
		Allow the remote user to specify a sequence number to restart transmissions
on.
		default is no
	fiphdr-client-id: (2 chr FipHdr field)
		FipHdr field in the incoming data which holds the client id which will be
used for resends.
		default the absolute sequence id.

	dump-data:
		archive the data sent in a dump file in /fip/dump/DOY_CLIENT

	do-not-send-duplicates:
		If a single file is in more than one service, then it is normally
		a copy is normally sent for EACH service.
		Use this flag to send only one copy.
		There is an item in the log which says that rg second and subsequents are NOT
sent.
	send-all-old-data:
		Normally all files are sent to a client.
		However if the client disconnects at the end of a day (ie before
		midnight) and does not reconnect for hours afterwards, they
		will normally ONLY get files from midnight.
		Use this option to force all of the previous day too.
	send-no-old-data:
		Send no back data previous to the logon time (see above)
	create-ZQ:
		Use this to create a fake output sequence number in the FipHdr
		field ZQ. If there is a number to this parameter it will be used
		as the amount to increment - default is 2.
		eg default create ZQ:3-4, the next ZQ:5-6
		if create-ZQ:12, file will have ZQ:22-33 then ZQ:34-45 etc.
		if create-ZQ:1,  file will have ZQ:2-2 then ZQ:3-3 etc.
	track-high-priority:yes/no
		If 'ipremdata' is also set to track these, this will send the
		HiPris before any other traffic.
		This option is NOT for Groups - only single Clients.
		This automatically flags 'do-not-send-duplicates:' 
	log-duplicates:yes/no

	create-md5-signature:
	check-md5-signature:
		Create the MD5 signature of the incoming file and send to the FipRemote
running in Server mode
		The file is logged if in error at the remote end.
		default: none
	output-md5: (FipSeq)
		Output this string with the MD5 signature as FipHdr RM AFTER the data has
been sent
		Use this for straight braodcast feeds
		eg - top and tail with an ETX and EOT
			output-md5:03RM04
		default: none
	encrypt-script:
		External script to encrypt or compress the data.
		There is one and only one parameter to this script which is
		the name of the file to encrypt and in which the new contents
		will be stuffed.
		encrypt-script:/fip/local/remote_pgp

The text-format file has the Syntax of :
	; comment
	before: Text to add at the top of the data file.
	after:  Text to add at the bottom of the data file.
	filebefore: File to add at the top of the data file.
	fileafter:  File to add at the bottom of the data file.
	zaptags: If the source data is in XML (or HTML or any other tagged
		format), zap all the tags.
		eg	tag:P	rnt
	tag: Replace a start tag with some string	(eg <p>)
	endtag: Replace an END tag with some string	(eg </p>)
	alonetag: Replace an Alone tag with some string	(eg <br/>)
		Syntax:	endtag:(tag)	(replacement data in FipSeq)
		There can be a 'tag' for each that needs replacing.
		Do not use these with wrap - pretty meaningless if you do anyway !
		It is case insensitive.
		tag:P	<text>
		endtag:P </text>
		alonetag:br	<padline/>n
	force-binary: If the text starts with this string, assume it is binary
		and make NO changes to end-of-line of characher maps
		force-binary:<p>
	chr:(chr)	(replacement data in FipSeq)
		Syntax: chr:(chr)	(replacement data in FipSeq)
		There can be a 'chr' for each character that needs replacing.
		chr:06	+Bell+
		chr:37	<TD><!-- 37 -->
	hdrchr:(chr)	(replacement data in FipSeq)
		As for 'chr' but for FipHdr fields only.
		Use this to map XML entities for example
			hdrchr:&	&amp;
			hdrchr:<	&lt;
			hdrchr:>	&gt;
			hdrchr:"	&apos;
	txchr:(chr)	(replacement data in FipSeq)
		As for 'chr' but for 'before', 'after'. 'filebefore','fileafter'
		Use this for any '#' in the FipHdr fields for example
	paragraph-marker: End of line sequence in FipSeq
or	eoln: End of line sequence in FipSeq
		default is to leave the end-of-line as in the data
	ignore-eoln: Test for eoln - if therem, do NOT add the eoln
		eg	eoln:</p>n<p>
			ignore-eoln:</p>
		In this case if there is a single preceeding <p>, no changes to the eoln will
be made
		If not, all end of lines (usually NL or CR NL) are replaced with '</p> NL
<p>'
		This works on a file by file basis so files with/without <p> can be
interlaced in the output
		default - none
	wordwrap: Wrap the text at a number of characters per line.
		This will wrap on the nearest space to the end of the line.
		default is to not change the data at all.
		eg	Wrap the text at 72 characters.
			wordwrap:72
	ww-eoln: For word-wrapped lines, End-of-line sequence in FipSeq.
		default is CR NL ie 'rn'
		eg	ww-eoln:f
	add-wordwrap-for-service: ONLY wordwrap for this service. There can
		be several lines each specifying a single service.
	remove-wordwrap-for-service: Remove any existing wordwrap for files for
		a service.  There can be several 'remove-wordwrap-for-service'
		lines, one per service.
		Eg : if files for service 'biz' are wrapped at 70 chrs and
		you want to have NO wordrap :
			remove-wordwrap-for-service:biz
		This is done BEFORE applying any wordwrap , so if you want to
		put a wordwrap on 40 chrs: 
			remove-wordwrap-for-service:biz
			wordrap:40
	convert-to-utf8:
		Convert the incoming data to the UTF8 variant of Unicode.
	max-data-size-for-service:(servce name)	(no of chrs)
		To create an abstract wire of the first n chrs for a particular service
		Headlines only, set this to 0 (zero)
		eg	max-data-size-for-service:biz	0
		default is -1 for ALL data/text
	repeat-service-name: Service code for Repeated items.  default is 'RPT'
		To use the original, use a blank 'repeat-service-name:'

	service-name-alias: Original Service code and replacement
		eg service-name-alias:XX2	SPT
		There can be up to 1000 aliases
		The aliased name does not have to be unique

Input Parameters :
Mandatory
Either	-g : name of a Groups file in tables/remote/groups	default: none
or	-n : name of a single Client file in remote/clients	default: none
Optional
	-N : replacename name for the client			default: -n
		This allows a 2 clients to use the same parameter file
		but have different idents.
		Resending files from Fip will go to both clients
		Care MUST be taken to have UNIQUE client names on a single server
	-s : hostname as override to parameter file		default: name
		for client-started feeds
		eg -s + to listen on all ports
	-P : port as override of 'port:' in  parameter file	default: port
		for client-started feeds
	-l : do NOT log each item 				default: do
	-LOCAL : only process this client (or Group) locally	default: network
	-T : do NOT log the trace of each transaction		default: do
		Normally a one line log of each file sent is stored in
		log/remote_trace with a name of 'date_client'.
		Use this flag to stop this trace.
	-V : version level					default: 3
		(use for old remdata)
	-v : Print the version number and exit

FipHdr fields POSSIBLY used in the incoming data and used by ipremdata/svr :
	R0- clientname for traffic for a particular client only
		This is used for resends and system messages
		R0:MegaCorp
	R6- Mime Type of file
		R6:TEXT - used for flagging to use the TEXT format
		Any other value/string is user-defined.
	RX- Full path and filename on this device

FipHdr fields added by ipremdata are :
	R1- date CCYYMMDD
		R1:20000630
	R2- hour
		R2:09
	R3- Overall System item number
		R3:06132
	R4- Uno
		R4:200006300906132
	R5- SystemId - not used by RemSvr

FipHdr fields added by ipremsvr are :
	R7- Service item number
	R8- Client item number
	R9- Service name, usually a service code
		R9:COM


--------------------------------------------------------------
Version Control
;002z10	15dec03 cleanups
	;d-f 27feb04 cleanup Groups
	;g-h 29jul04 added 'track-high-priority:'
	;i-k 15aug04 added check-md5-signature and encrypt-script
	;l 07jan05 bugette over chg of day
	;m-n 26jan06 added allow-remote-repeats:yes/no (n dump md5)
	;o 24sep06 added tag, endtag, zaptags and alone tags, 
	;p-q 04nov06 added -P port as override to parameter file
	;r 06nov06 added send-no-old-data: and -N
	;s-u 22nov06 added ignore-eoln
	;v-w5 11dec06 added -k remkey
	;x-y6 8may07 added service-name-alias and StuffFile bugette (y4 no-pwd for
fipremote)
	;y7-10 for txchr: (10 for space in service-..alias)
	;z1-9 17aug08 added take-size/fiphdr/top/tail (2 bugfix for output buffer) (5
new services buglette)
		;10 27apr10 added local-fiphdr and local-extra-fiphdr
;001z	22feb01 no-password/no-logon-required added for fipagent too
	;a 20mar01 added proxies
	;b 27mar01 added wordwrap properly
	;c/d/e 20apr01 bugette in non-encrypted feeds
	;f 27jun01 WINNT
	;g 27nov01 closeup if bad logon.
	;h 10dec01 added do-not-send-duplicates.
	;i/j 08jan02 added hdrchr
	;k/l 09mar02 added remove-wordwrap-for-service
	;m 11apr02 added convert-to-utf8/base64
	;n 26apr02 added add-wordwrap-for-service
	;o 02may02 ignore the Missing msg if Pos=0
	;p/q 09may02 added send-all-old-data:
	;r/s 12jun02 finally got the svce bug !!
	;t 28sep02 added max-data-size-for-service:
	;u 27mar03 added ignoreforservice:
	;v-y 23may03 added createZQ
	;z 05dec03 added repeat-service-name

(copyright) 2014 and previous years FingerPost Ltd.