Wednesday, June 25, 2008

Upgrading from PostgreSQL 7.4 to 8.1 in Debian Etch

Debian allows both versions 7.4 and 8.1 of PostgreSQL to coexist on the same system which somewhat simplifies the upgrade procedure.

Assuming that the PostgreSQL 7.4 packages are currently installed with a fairly standard configuration, we start by installing PostgresSQL 8.1.

apt-get install postgresql-8.1 postgresql-client-8.1

The newly installed system will be started on a port 5433 whilst the old system continues to run on standard port 5432. For the time being it is best to leave both instances running on these ports which will allow you to migrate the data via a simple pipe.

Amend the configuration files for the new instance by hand, based on the existing configuration and restart the instance.

/etc/init.d/postgresql-8.1 restart

Populate the new instance with the existing databases:

sudo su - postgres
/usr/lib/postgresql/7.4/bin/pg_dumpall -p 5432 | /usr/lib/postgresql/8.1/bin/psql -p 5433

Stop the old database instance and use the new client.

/etc/init.d/postgresql-7.4 stop

Switch the new instance to using standard port 5432.

vi /etc/postgresql/8.1/main/postgresql.conf [set port = 5432]
/etc/init.d/postgresql-8.1 restart

Use the new psql client and your own applications to ensure that the new installation is operating correctly and that the data import was successful.

sudo -u postgres /usr/lib/postgresql/8.1/bin/psql -p 5432

If all is well then remove the old package, maybe backing up the old data.

tar cvzf ~/oldpgdata.tgz /var/lib/postgresql/7.4/data
apt-get purge postgresql-7.4 postgresql-client-7.4

Sunday, April 13, 2008

Catching up with Barcode Writer in Pure PostScript

It has been a while since I last wrote about Barcode Writer in Pure PostScript. The project has been far from dormant in recent months so here is a chance to catch up with what's been keeping me busy in my “spare” time. (And even very busy at other times!)

It's hard to recall from memory all of the improvements that have been made to the project over the last year but thankfully the commit logs do not forget! There have been the usual bug fixes, some code optimisations and new miscellaneous features, but the main highlight has to be the inclusion of support for 2D barcodes which went in to the mix as follows:

MaxiCode (June to July 2007)

MaxiCode is an irregular matrix symbologies whose symbols consist of a hexagonal grid of dots around a bullseye finder pattern. The sequencing of these dot positions does not follow any regular pattern and so unfortunately the mapping matrix must be hard-coded into the software. MaxiCode also has various different "modes" of operation, some of which impose a strict format on the initial part of the data which makes the input encoding quite complicated.

PDF417 (Boxing Day to New Year's Day)

Technically speaking, you might refer to PDF417 as a "stacked-linear" symbology, however BWIPP renders it using a grid of tall, rectangular cells. The worst thing about this symbology is that it requires a set of lookup tables that contain the "cluster sets" - three groups of 930 numbers used to convert from codewords to bar/space widths. The sequencing of the numbers within these sets appears to be quite random (if you know otherwise then please let me know) and so they must be hard-coded into the software which leads to a lot of uninteresting code – ouch!

Data Matrix (early- to mid-January)

The is a matrix symbology that can be rendered using a grid of squares through which the data zig-zags in eight-module, L-shaped clusters. Whilst the ordering of the modules within the grid is reasonably complicated, it can nevertheless be determined algorithmically for only a small amount of computational cost and requires only some minor tweaking to fix up the corner cases for matrices that do not contain some multiple of eight modules. So overall this symbology can be coded very nicely. We can generate both the standard square symbols types as well as the optional rectangular symbols.

Aztec Code (early- to mid-February)

This is a matrix symbology that can be rendered using a grid of squares with the data wrapping clockwise in two-module wide layers around a square finder pattern in the centre of the symbol. Whilst there are a few different types of symbols it is possible to fold the implementation for each of these into a single relatively sophisticated but direct algorithm that does containing excessive branching. So again, this symbology can be coded quite cleanly.

QR Code (February to late-March)

This is a matrix symbology that can be rendered using a grid of squares with the data vertically meandering in two-modules columns from right to left. With respect to implementation this symbology is quite hideous with its one saving grace being that is does not require the inclusion of hard-coded lookup tables for module placement. Firstly, in certain symbols the final data codeword is defined to be four bits wide rather than the usual eight which results in an awkward bit shift having to be applied to the trailing codewords in order to avoid propagating the exceptional processing required for these specials cases throughout the remainder of symbol generation process. Secondly, the "drunken walk" algorithm for placing the modules within the symbol (whilst avoiding the pre-defined static feature placeholders) has an unexplained inconsistency in the way that you perform the hop over the vertical timing pattern. Thirdly, the format and version information functions are unnecessarily complicated, however since their domain is very small it is possible to use a small set of pre-calculated lookup tables for these in order to avoiding using a significant amount of complex code. But finally, the worst aspect of this symbology is the optional, but recommended, process of apply eight distinct mask patterns to the candidate symbol in turn and then to evaluate these in order to select the one that would produce an output symbol with the fewest undesirable properties. To perform the evaluation algorithm as given by the specification turns out to be significantly more operationally expensive that the entire remainder of the symbol generation process! So for the time being we always select one particular mask.

So, we presently support all major 2D barcode formats, but with one major caveat - the user (or application developer) that is working with BWIPP has to do some preparative work to process the barcode data into the particular intermediate format required by each encoder for which they require the corresponding specification. This is a small task compared to the sometimes sophisticated numeric manipulation involved in the remainder of the symbol generation process. However it does involve extensive string manipulation which is a task for which PostScript is definitely not well suited whilst purpose-built application development languages (such as Perl and C++) have much better support for this task either natively or through libraries.

So the next major set of challenges on the BWIPP roadmap is to integrate the high-level encoding routine for each 2D symbology that convert from a user-supplied ASCII string to the intermediate format that is required by the encoders at present. The result will be that the novice user can simply enter the data that they require to place into a barcode, with only the minimal restrictions as necessarily imposed by each symbology, and our code will create the most optimal encoding that produces the best symbol for the given data, thereby making the system much easier to use for the uninitiated user.

Lastly, but by no means least, an extremely useful component in the implementation of support for 2D barcode generation has been the extensive testing performed by Jean-François Barbeau. He has helped detect and fix a number of bugs, some obvious, and some much more subtle so that we can place much greater confidence in the correctness of the output – so a big thank you on behalf of the PostScript barcoding community!

Saturday, January 12, 2008

Toying with Barcodes - a talk from 24C3

Interesting and entertaining talk by FX of Phenoelit from 24C3 about hacking real-world systems that (mis)use barcodes. Doesn't mention BWIPP though :-P

Friday, November 30, 2007

Taming p0f by chunk processing STDIN

P0f is a useful, but slightly outdated, tool for passive operating system fingerprinting. Unfortunately it generates fingerprint information for every identified packet matching the BPF filter provided by the user. There is no built in option to aggregate the data so that it is framed per host within a given time interval or set of contiguous records. This means that when p0f is attached to the SPAN port or tap of a high volume pipe the log files that it generates grow very quickly as they are full of repeated information.

The following little script wraps the p0f process and removes the redundant information from each 1000 record chunk of output:


#! /bin/sh

exec 3>&-
exec 2>&-
exec 1>&-
cd /

nohup p0f -i eth2 -u p0f -N -U -q -p -t -l 'src net 143.210.0.0/16' | \
sed -n -e 's/^<\([A-Za-z0-9: ]*\)> \([0-9.]\{7,15\}\):[0-9]\{1,5\} - \
\(.*\)/\2 \3/p' | gawk 'ORS=NR%1000?"\n":"\000"' | xargs -0 -i bash -c \
'date +"*** %c ***"; echo "$0" | sort | uniq' {} >> /srv/p0f/os.log &


p0f aside, the interesting part boils down to this useful Unix shell programming paradigm:

$INPUT_CMD | gawk "ORS=NR%$BLOCK_LINES?'\n':'\000'" | xargs -0 -i $PROCESS_CMD {}


It splits the streamed output of $INPUT_CMD down into chunks of $BLOCK_LINES lines which are immediately independently processed by $PROCESS_CMD. It chunks the data by replacing the ordinary line separator on every $BLOCK_LINES line into an ASCII 0 character which xargs -0 uses as the argument separator.

Friday, November 09, 2007

Apache accesslog to syslog

Apache allows its error logs to be written to the local syslog, however it does not natively support the directing of access logs to the syslog. How frustrating!

It does however allow access logs to be written to a pipe and I have seen a number of home-brew scripts that essentially redirect the Apache access log data from STDIN to syslog.

I've yet to see anything quite as simple as the following directive that I cooked up today:

CustomLog "|/usr/bin/logger -t apache -i -p local6.notice" combined

It pipes the access log data to the BSD logger(1) utility that is installed by default on almost any Unix system. No need for any more of those STDIN wrapper scripts!