Benchmarks

Various benchmarks have been done to evaluate how good was the compression level of the PCIF algorithm in comparison to the most widespread formats. Since benchmarks are always a delicate argument, multiple image sets have been used for the tests:

  1. The waterloo color image set, a popular set images of various types of images
  2. The kodak image set, a set of photographical images. Unfortunately, it turns out to be sub optimal for benchamrking, as it originated from a lossy compression.
  3. Another image set proposed by the author, with the goal to investigate properties of the compression algorithms on a wide type of test images
  4. Another image set proposed in www.imagecompression.info containing bigger images than those of the other sets, useful to see how the algorithm compression ratios scale with file dimension

Resulting file sizes have been compared with three popular lossless image formats: the Portable Network Graphics (PNG), the Jpeg2000 (JP2) format and the JPEG-LS format (to the LOCO algorithm, to be exact). The first is a standard defined by W3C and is actually the most popular lossless image format used on the web together with GIF, that supports only 256 colors and is so not adapt for comparison. The JP2 format has been designed by the jpeg commitee to be the successor of the jpeg standard. Its algorithm is thinked first of all for lossy compression, but it offers also a highly effective lossless coding. The LOCO algorithm developed by HP labs (www.hpl.hp.com/loco/) is the core of JPEG-LS, another very efficient lossless image format proposed by the Jpeg commitee. The settings used for the creation of the files are described here.

The global results for the various image sets are in the following table. The total produced filesize for every algorithm is compared with the one generated by the PCIF program; the BMP column represents the uncompressed filesize.

Update: the BCIF algorithm, evolution of PCIF, has also been included in the benchamrks. It has a better compression ratio and a strongly inscreased speed. Take a look at the BCIF homepage.

Image set \ size (% of pcif) BCIF PCIF JPEG-LS JPEG 2000 PNG BMP
Waterloo image set 84.1% 100% 105.7% 151.7% 106.6% 322.0%
Kodak image set 96.8% 100% 97.5% 95.6% 130.8% 240.6%
Own image set 90.3% 100% 97.0% 105.8% 126.9% 346.6%
ICI image set 95.6% 100% 100.1% 99.9% 117.5% 225.9%
Average 91.7% 100% 100.1% 113.2% 120.4% 283.9%

Results of the tests vary from image to image and are discussed in the sections of the various sets. Globally, the pcif algorithm obtained the best compression ratio among the tested algorithms, with results that are very close to the LOCO (JPEG-LS) algorithm. Shortly, the PCIF algorithm gives better results than PNG (in average from 25% to 40% smaller files) except than for very regular images. Further, the PCIF files results slightly bigger (usually <5%) than the JP2 files for photographical images, but are significatively smaller (usually more than 15%) for images that are computer-generated or have high edges. Results in comparison with JPEG-LS are generally quite similar, but as before PCIF seems to be more stable: for many images LOCO is slightly better, but for others PCIF compresses noticeably more.

Since the pcif algorithm is actually implemented in Java and not in a machine-compiled language, time benchmarking should not be done. Anyway to show how the execution time is acceptable even on a non compiled language, we have a table that shows the average compression and decompression rate of the algorithm. Tests have been done with an old machine with an Athlon 2600 processor and a Java Virtual Machine version 1.4.2. The results have been mesured on the 'own image set'; the other ones give anyway very similar results.

  • Compression speed: 500 KB per second
  • Compression speed in fast mode: 626 KB per second
  • Decompression speed: 1565 KB per second
Average speed for image elaboration on an Athlon 2600 processor
Image size Compression time Fast compression time Decompression time
512 x 512 1.5 seconds 1.2 seconds 0.5 seconds
800 x 600 1.8 seconds 1.4 seconds 0.6 seconds
1024 x 768 2.8 seconds 2.2 seconds 0.9 seconds
1280 x 1024 7.7 seconds 6.1 seconds 2.5 seconds

The tables show that even with a computer several years old the decompression of an average image can be often done in the order of a second. If the image is embedded in a web page and processed with the pcf applet, once the Java Virtual Machine is loaded (when the first pcif image is found) then the decompression can be done while the image is being downloaded, causing the user to do not perceive any wait.

The evolution of the PCIF algorithm is now available ! It has a greater compression ratio, it is much faster and the implementation is available in both Java bytecode and native executables. Take a look at the new BCIF algorithm.