ampswire (Sat Oct 25 2014 01:31:01)

ampswire

This periodically attachs to, checks for and grabs new articles in groups on a
remote AMPS server using WNDP

The whole docuement is then left in (outputfolderstub)(fileextension). The
default folder stub is 'xamps_', so :
	txt files will be left in /fip/spool/xamps_txt
	xml files will be left in /fip/spool/xamps_xml
	jpg files will be left in /fip/spool/xamps_jpg
	htm files will be left in /fip/spool/xamps_htm


The parameter file, normally tables/wire/AMPS, is read for the the names of all
the groups to scan for.
	; comment
	logon: (logon on the remote server)		default: none
	password: (password on the remote server)	default: none

	class: Class of data				default: ALL
		see the amps doc for a full description
		includes	XMLNews-story
				ANPA
				IPTC
				HTML
				MarketData
				ALL
				KNRIDDER-NITF
				PRNewswire-Raw

	wndp-version: Version of WNDP			default: 3
	xmlgen-version: Version of XML			default: 1

	include-filter: (service name)			default: ALL
	exclude-filter: (service name)			default: none
		There can be up to 1000 filters specified.

	force-output: yes/no				default: no
		force all output to the output folder
		default is use the output folder name as a stub and append the file 'ext'
			eg if the file is 'txt' and output:xamps_
				output folder is xamps_txt
		in this case NO extension is added !

	process-additions:ignore/process/file		default: process
		AMPS does not usually send duplicate data files but sends a '.add' file which
denotes the original one sent plus new metadata.
		ignore	- just ignore all references
		process	- find the original and send that with the new metadata
		file	- treat '.add' messages as just other file
		Note that the 'process' option uses /fip/data/amps and this should be purged
nightly for anything over 4 or 5 days (check with the AMPS setup how far back
it might go to)

	extra-fiphdr-ext:(ext)	(FipSeq to add)		default: none
		add this extra fiphdr information to each file with this file extention.
		extra-fiphdr-ext:TXT	#SR:ALLPA
		extra-fiphdr-ext:XML	#DY:RIPOUTTXT.XML

	extra-fiphdr-class:(ext)	(FipSeq to add)		default: none
		add this extra fiphdr information to each file with this file extention.
		extra-fiphdr-class:JPG	#AB:JPEG

	round-robin: (number)			   default: none
	round-robin-fiphdr: (2 letter FipHdr field)	default: none
		Round-Robin the output files and add the RR number to the fipHdr.
		Both parameters are required - the Number is the MAXimum.
		eg	round-robin:9
			round-robin-fiphdr:RR
		and a suitable output folder might be   /fip/spool/xchgRR

Commons extensions are
	TXT	text - anpa or iptc
	XML	xml variants; prnews
	JPG	pix jpegs
	HTM	html files

Optional
	chrset:	(Source character set ie SC header field)	default: ascii
	response-timeout: (timeout in seconds wanting for the remote to reply)
								default: 60 secs
	connection-timeout: (timeout in seconds wanting to connect to the remote)
								default: 120 secs
	extra-fiphdr: (more FipHdr information to add)		default: none
	current-balance-group: name of a balance group (in tables/sys/BALANCE) to
distribute the current totals file when changed (see doc on 'ipbalan')
		This is often used where a second system could be used as a redundant server
if the main system fails.

	log-line: (FipSeq)
		Custom log line for files in (ie depends on -l 20)
		eg	log-line:S2 SF (S1) SA -> SN

		Fiphdr fields for each incoming file are :
			SN - filename
			SU - from the '-n' input switch or AMPS
			SC - 'ascii' or chrset parameter
			SA - route from the request
			ST - class from the request
			SE - file extention
			S1 - size from the request
			S2 - data/time from the request
	timing-stats:yes/no
		Show timings statistics		default: no

Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).
	They are normally specified :
		fixed:QZ	1234543
		partial:QT	ST,3,2,U,<,>
		combie:QY	ep|na,(0000000)a
		option:QE	ep,11,7,s
		repeat:QK	XK,-,3
	or	repeat:QP	PK,,4,#X
		style:QS	XN,%.03d
		replace:QN	NN	abc=DEF def=GHI
		newdate:QT	hours+3	"ZD"

Input switches are :
Mandatory :
	-s : Hostname where the AMPS is running.	default: none
Optional :
	-A : Processing ADD - additions				default: process
			ignore - just delete them
			process - copy the original data and add the new routing, send as a new file
			file - just send like a new file
	-d : done folder					default: none
		This holds a copy of all incoming data files from every source
		The structure is
			(done folder) / (date)_(logon) eg 20110921_fip / (filename as written to the
output folder)
		It can be purged with an entry in maintenance (zapfiplog)
		eg if '-d raw.amx' and we want the last 30 days data
			/fip/bin/ipdelque -q/fip/spool/raw.amx -i1 -a30
	-D : display the conversation with the remote server	default: no
		used for debugging troublesome connections
		do NOT add in the SYSTEM file
	-F : Force the output folder to the '-o' and do NOT add the file type
extension
	-I : id of this instance				default: ignored
		Where there are several copies of 'ampswire' running with the same -n
	-l : log level				  default: 10 for connect/disc messages too
		-l 0 errors only
		-l 10 plus connect/disconnect
		-l 20 plus each file
	-n : name of the service		def: AMPS or name of -z param file
	-o : Next fip queue for incoming files	  default: spool/xamps_ZZZ
			the file type is added as the last 3 letters of the output folder
	-p : port number on the remote host		default: 39030
	-V : major version				default: xmlgen
		-V wndp for wndp
		-V xmlgen for xmlgen
	-z : parameter file				default: wire/AMPS
		this file MUST exist - ut can be a single comented line.
	-v : display version number and exit.

-- Copy Flow

For text files which are ANPA or IPTC, spool the output folder with ipdskwir :
eg a snippet from the SYSTEM file
	; ampswire using default parameter file wire/AMPS and default folders -
xamps_txt
	amps	local   ampswire -s localhost
	; txt files - process and route as normal
	atxt	local   ipdskwir -i xamps_txt -n AZMEDIA
	; jpgs - move to pix system directly
	apix	local   ipftp -i xamps_jpg -n AZPIX2EDIT

-- Maintenance
The zapfiplog entry if using the process-additions:process option (which is the
default) :
	If the -n name is different to AMPS, then change the 2nd 'amps' in the 2
strings

if [ -d /fip/data/amps/amps ]; then
	/fip/bin/ipdelque -q /fip/data/amps/amps -A 2 -i 0
fi

Version Control
;001b15	29mar05 original version
	;a3-5 added process-addition
	;b8 better XML ;9-10 added round-robin ;11 added -d ;12 bugette in -d
		;13-14 added extra-fiphdr-class and better ext ;15 -I wireId

(copyright) 2014 and previous years FingerPost Ltd.