Benchmarks: the Waterloo image set

The Waterloo color image set is a commonly used set of images for various benchmarking purposes, containing also the famous image lenna, used for various tests and demostrations about image manipulation. The original images can be found at links.uwaterloo.ca. They can also be downloaded in the PNG format by clicking on the thumbnails below.

The images in the Waterloo image set have many different characteristics from one to another allowing an analysis to seek out which kind of images are suitable for a given algorithm or another. In the set we have three high-quality photographical images (monarch, sail and tulips), two photographical images of lower quality obtained through a scanner (lena and peppers) and three computer-generated 'artistic' images (clegg, frymire and serrano). Image size goes from 512x512 of lena and peppers to 1118x1105 of frymire.

In respect to the other two used image sets, this one gives the major different results for the various images. The PNG files result to be from 43% smaller to 40% bigger than the PCIF files. This is because the algorithm used in PNG tends to be very good where there are big areas of uniform colors, but results not as good for the other cases.

JP2 files result to be from 0% to 9% smaller than PCIF files for photographical images, but are also from 139% to 157% bigger for the artistical images. This depends on the fact that JP2 is not suitable to compress high-edged images as these ones. The JPEG-LS algorithm has similar performances, but with much less loss of performances in the artistical images.

The Waterloo image set with its extremably variable images gives several usefull informations about the performances of various compression algorithms. Anyway having a very low number of images the very peculiar 'artistical' ones have a very high weight on the global result. It would be also usefull to have some computer-generated images that resemble photographical images as an intermediate sample between purely photographic and purely artistical images. This kind of data is nor in this set nor in the kodak image set, and this is one of the reasons why another image set has been taken into account for benchmarking.

In the table, the BMP column represents the size of the uncompressed image. The settings used for the creation of the files are described here.

Benchmarks should always be read with care, as results may be different from image to image depending on a great number of factors. Other benchmarks used to test the PCIF format have been done on the kodak image set, on another set of images gathered by the author and on a fourth image set of large images proposed on www.imagecompression.info. Finally, if you are interested in the PCIF algorithm, you are encouraged to test its performances yourself by downloading the program in the download section.

Update: the BCIF algorithm, evolution of the PCIF algorithm, has also been included in the benchamrks. It has a better compression ratio and a strongly inscreased speed. Take a look at the BCIF homepage.

Filename \ size (KB) BMP BCIF PCIF JPEG-LS JPEG2000 PNG
clegg21003785716461369490
frymire36214136417961560361
lena768414431442434500
monarch1152450468448431621
peppers768331343328327441
sail1152541554541511777
serrano1463152242286623147
tulips1152500527507477691
Total1217831793782399757364031

Below there are the thumbnails of the 8 Waterloo true color images, linking to the original ones in the PNG format.

Clegg Frymire Lena Monarch Peppers Serrano Sail Tulips
The evolution of the PCIF algorithm is now available ! It has a greater compression ratio, it is much faster and the implementation is available in both Java bytecode and native executables. Take a look at the new BCIF algorithm.