Forum: Suggestions
Topic: ISZ format revision 2
started by: Iwo
Posted by Iwo on Apr. 15 2008,07:05
Greetings,

"Chunk data may be compressed by ZLIB or BZIP2 method.". ZLIB has 32 or 64kb dictionary size, BZIP2 has 100,200,300,400,500,600,700,800,900kb dictionary sizes. Deflate is quite fast and well known, but it is a weak algorithm in general.
BZIP (bzw) is better, but it has one great disadvantage - decompression speed is equal to compression speed - SLOW.

So, how about releasing ISZ2 format with LZMA:
< http://www.7-zip.org/sdk.html >
and larger dictionary size. WINRAR and WINACE use LZMA with 4mb dictionary size and special multimedia algorithms. Also 7-ZIP uses special converters to compress EXE files better.
Dictionary size problems - larger size - better compression ratio, slower compression, more memory needed for compression, smaller size - a contrario.

Dictionary size - mem needed for compr. / mem needed decompr:
64kb - 38mb/3mb (megabytes)
2mb - 60mb/4mb
4mb - 83mb/6mb
8mb - 129mb/10mb
16mb - 221mb/18mb
32mb - 405mb/34mb
64mb - 709mb/66mb

"Memory (RAM) usage for LZMA compressing is about 11 times more than dictionary size. Memory usage for LZMA decompressing is close to value of dictionary size."

Also about CBC (for encryption) - it is old algorithm. Better is LRW and currently the best one is XTS:
< http://www.truecrypt.org/docs/modes-of-operation.php >

"XTS mode is in fact XEX mode [12], which was designed by Phillip Rogaway in 2003, with a minor modification (XEX mode uses a single key for two different purposes, whereas XTS mode uses two independent keys). XTS mode was approved as the IEEE 1619 standard for cryptographic protection of data on block-oriented storage devices in December 2007."

Also optional recovery blocks will be nice (like in WinRAR). Look there:
< http://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction >
And there: < http://www.ice-graphics.com/ICEECC/IndexE.html >

So, ISZ could be much better. Have fun for coders ;)

Peter

Posted by xoben on Apr. 15 2008,09:45
Thank you Iwo for your nice suggestions.
Posted by Iwo on Apr. 15 2008,13:40
I hope, that my suggestions will be useful. It is very seldom to see any thanks from authors or forum crew (I post many suggestions for many products). I also have something to add:

1. LZMA decompression speed is very good - ~36-38mb/sec on Core Duo 2GHz (8mb-2mb dictionary size).
2. About features - providing better compression spares more bits for error recovery, so these suggestions are connected in some way. So, EZB could add a nice option of "Target image size" like in ICE ECC target ECC file size. Also option to calculate needed % of recovery data for MORE >1 images to fit ALL these images on given media will be VERY nice. Currently I make ISZ files and then add ECC file and then burn files on DVD. So, Ultraiso could optimize and compress image(s), then add recovery data to fit on for example 700mb CD or single or double layer DVD. It will be an ideal solution for making backups - backup will be:
a. More useful than backup made by compressing ISO file with winrar.
b. More fault-tolerant than normal CD/DVD or normal image file.
c. More protected (in case of security) than backups made in other formats (see also below).

And also, please read my suggestions for different product - (some of) these suggestions will apply also for UltraISO:
< http://forums.truecrypt.org/viewtopic.php?t=10588 >
Posted by Iwo on Apr. 15 2008,14:03
Please be advised, that "word size" parameter gives best results for Deflate is it's at its maximum size (258), so you could optimize your current compression algorithm. BUT! For LZMA best results are achieved for WordSize/FastBytes = 160 or 192, not 273 (LZMAs maximum)!

Examples (PSI IM, 280 files, 34.3mb folder):
psi.zip for FB=128 (Deflate32) - 14167148 bytes
psi.zip for FB=258 (Deflate32) - 14165981 bytes
psi.7z for FB=160 (LZMA/8mb) - 10021377 bytes
psi.7z for FB=273 (LZMA/8mb) - 10026389 bytes
psi.7z for FB=160 (LZMA/64kb) - 11065683 bytes
psi.7z for FB=160 (LZMA/48mb) - 9873343 bytes

So LZMA easily outperforms Deflate! Also, I think that compression speed was comparable (engine of 7-zip is now very optimized).
Posted by xoben on Apr. 16 2008,00:50
Thank you Iwo. Please understand that not all suggestions will be realized in future versions.
Posted by Iwo on Apr. 16 2008,09:48

(xoben @ Apr. 16 2008,05:50)
QUOTE
Thank you Iwo. Please understand that not all suggestions will be realized in future versions.

I understand ;) It is quite common, that authors have different vision of their program than users.

Not all suggestions are equally important:

XTS supports is theoretically better than CBC, but I don't know any attack on CBC.

Certainly the most important suggestion is about LZMA and dictionary size. I also think that BZIP support should be abandoned (It's SLOW and weaker than LZMA).
Posted by Iwo on Jun. 13 2009,17:46
Since LZMA is more popular - I saw Linux and sourceforge packages in ".lzma" - do you consider extending ISZ by additional algorithm ? I tested many images and NTFS compression beats ISZ in some cases.

"LZMA SDK is placed in the public domain." So you can do anything with it.
Posted by dma_k on Apr. 06 2010,18:19
I vote for this feature as well.