Здравствуйте,I am the author of Pack, and I am happy to see you all interested in Pack. I am sorry for writing in English; my understanding of Russian is limited to that Здравствуйте.
I try to answer most of the comments here, but if you have any further questions or comments, let me know or email me (you can find it in the linked post).
Warm or Cold:
On the linked post (https://pack.ac/note/pack), there is a line noting the test condition: "All corresponding official programs were used in an out-of-the-box configuration at the time of writing in a warm state."
Please note that for ZIP, the official tool is WinZip and I tried the test many times to find the best warm time and remove any file system and disk interference.
As noted in the post, you should test it for yourself. The numbers were not as "Look, others are bad". They are great; my point was, "Look what we can do if we update our design and code"
Random access (extracting one):
You can do that with Pack:
`pack -i ./test.pack --include=/a/file.txt`
or a couple files and folders at once:
`pack -i ./test.pack --include=/a/file.txt --include=/a/folder/`
Use `--list` to get a list of all files:
`pack -i ./test.pack --list`
Such random access using `--include` is very fast. As an example, if I want to extract just a .c file from the whole codebase of Linux, it can be done (on my machine) in 30 ms, compared to near 500 ms for WinRAR or 2500 ms for tar.gz. And it will just worsen when you count encryption. For now, Pack encryption is not public, but when it is, you can access a file in a locked Pack file in a matter of milliseconds rather than seconds.
Is it a compressed database?:
No. It is a container for files and raw data. Projects like sqlite-zstd are for any database for your projects. They are not made for files.
sqlar:
Pack is a new format. sqlar inspired me to create Pack as a new improved solution.
Here is the latest sqlar result on Linux source code on the same test machine in warm state:
sqlar: 268 MB, 30.5 s
Pack: 194 MB, 1.3 s
SQLite security:
It is one, if not the most secure, library out there. It is very hard to crack it, and it will not allow running any harmful code on a machine. It is used in almost anything with a computer, partially because of its security and reliability.
File system:
Noted projects like dwarfs are file systems.
It is read-only; Pack is not. Update and delete are not just public yet, as I wanted people to get the taste first.
It is clearly focused on archiving, rather than Pack wanting to be a container option for people who want to pack some files/data and store or send them with no privacy dangers.
Big files instead of many small files:
Reading many files (81K in this test) is way slower than reading just one big file. For bigger files, Pack is much faster. Here is a link to some results from a kind contributor: https://forum.lazarus.freepascal.org/index.php/topic,66281.msg507177.html#msg507177
(Too long to post here)
tar.zst:
As requested, here are some numbers on tar.zst of Linux source code (the test subject in the note):
tar.zst: 196 MB, 5420 ms (using out-of-the-box config and -T0 to let it use all the cores. Without it, it would be, 7570 ms. And it is done in two steps: first creating tar and then compression.)
Pack: 194 MB, 1300 ms Slightly smaller size, and more than 4X faster. (Again, it is on my machine; you need to try it for yourself.)
Honestly, ZSTD is great. Tar is slowing it down (because of its old design and being one thread).
Pack does all the steps (read, check, compress, and write) together, and this weaving helped achieve this speed and random access.
Rate of compression:
7-Zip and other similar tools are focused on compression rate. Pack is on another chart, which is why I proposed CompressedSpeed [2]. The speed of getting to compression needs to be accounted for. You can store anything on an atom if you try hard enough, but hard work takes time. Deduplication step may get added, but in Hard Press [3].
RAR compression speed is great too. Many times, it can do better than Pack. My argument is that does it worth the wait? After all, if you can use Pack for those cases too,. Hard Press [3] creates a more compressed pack for cases of pack once, unpack many.
[1] CompressedSpeed = (InputSize / OutputSize) * (InputSize / Speed). Materialized compression speed.
[2] You can choose --press=hard to ask for better compression. Even with Hard Press, Pack does not try to eat your hardware just to get a little more; it goes the optimized way I described.
User experience:
Pack value comes from user experience, and speed is being one. I was not following the best speed or compression; I wanted an instantaneous feeling for most files. I wanted a better API, an easier CLI, improved OS integration (soon), and more safety and reliability.
On a final note, I suggest trying Pack for yourself and on your machine. I will be glad to hear more about your experience.
Thank you
