Taming p0f by chunk processing STDIN

P0f is a useful, but slightly outdated, tool for passive operating system fingerprinting. Unfortunately it generates fingerprint information for every identified packet matching the BPF filter provided by the user. There is no built in option to aggregate the data so that it is framed per host within a given time interval or set of contiguous records. This means that when p0f is attached to the SPAN port or tap of a high volume pipe the log files that it generates grow very quickly as they are full of repeated information.

The following little script wraps the p0f process and removes the redundant information from each 1000 record chunk of output:

#! /bin/sh

exec 3>&-
exec 2>&-
exec 1>&-
cd /

nohup p0f -i eth2 -u p0f -N -U -q -p -t -l 'src net 143.210.0.0/16' | \
sed -n -e 's/^<\([A-Za-z0-9: ]*\)> \([0-9.]\{7,15\}\):[0-9]\{1,5\} - \
\(.*\)/\2 \3/p' | gawk 'ORS=NR%1000?"\n":"\000"' | xargs -0 -i bash -c \
'date +"*** %c ***"; echo "$0" | sort | uniq' {} >> /srv/p0f/os.log &

p0f aside, the interesting part boils down to this useful Unix shell programming paradigm:

$INPUT_CMD | gawk "ORS=NR%$BLOCK_LINES?'\n':'\000'" | xargs -0 -i $PROCESS_CMD {}

It splits the streamed output of $INPUT_CMD down into chunks of $BLOCK_LINES lines which are immediately independently processed by $PROCESS_CMD. It chunks the data by replacing the ordinary line separator on every $BLOCK_LINES line into an ASCII 0 character which xargs -0 uses as the argument separator.