Why haven't there been any significant advancements in compression in like 30 years? New video codecs just find ways to make compression artifacts less perceptible but nobody is actually improving compression anymore. And lossless compression hasn't gotten better since the 90s. Is it just mathematically impossible for compression to get better or is there some technological breakthrough waiting to happen?
>>109568780
i think the compression of any file is limited by its entropy, which is a measure formalized in information theory, and since this is a property of the file itself there's a hard limit on how good your algorithm can be. but i could be wrong since i didn't research it as much as i would have liked. if anyone knows i'd be fine with being lectured desu
https://en.wikipedia.org/wiki/Kolmogorov_complexity#Compression
>Is it just mathematically impossible for compression to get better
Past a certain point, yes. The magic lays in determining that point.
>>109568780
Just check the quality of the programmers over time.
What we need to do is start recording video so that the bits are all very early sequences found within pi. Then it will all compress really well.
>>109568843
Then drop math and use something else. Simple as.
NOTHING is impossible.
for many kinds of compression we kinda just did most of the obvious stuff really quickly and most practical data compression uses require a balance between speed and ratio, so spending disproportionately more effort on tiny ratio bumps just doesn't make sense.
it's why modern algorithms like zstandard got popular quickly despite not even compressing as much as older formats, what makes zstandard so popular is because it has /good-enough/ ratios, meaning about as well as gzip, while being /way/ faster than gzip.
it's like do you use zpaq and make the file 10% smaller but take 1000x more time to do it? probably not.
there is a theoretical maximum any file can be compressed to as well, so it's not a case of "not knowing what you don't know", either. actually reaching that maximum with any reasonable amount of time and complexity for any file is difficult. getting close enough faster is what most algorithms aim for since they have the most practical use
>>109569114
should add i'm just talking about general purpose lossless compression here (like zip/7z/xz/zstd/gz/etc).
domain-specific lossless compression (like say flac) is a bit different, and lossy compression (h.264/av1/vorbis/mp3/etc) opens up many, many more options since those are as much subjective as they are mathematics.
>>109568780
ANS powering your zstd compression format was invented in 2014, 12 years ago.
>>109569114
this and files have gotten bigger, what was tolerable compressing a 1GB file may not be tolerable compressing a 100GB file even with cpus getting faster
>>109568997
Unless you find a better way to store the files than 1s and 0s you will not be able to fit more actual data into the same space, especially if the video isnt repetitive and its background changes a lot.
>>109568780
Because of low hanging fruits. You can't just keep compressing shit harder. It's like asking why we haven't found any bigger animals than the blue whale.
>inb4 seething
>等着看有人急眼
LLM technology is literally compression. We just haven't used it widely as a compression algorithm yet.
>>109570329
dictionaries in compression has been a thing for ages. zstd supports dictionaries if you have data which benefits from it
（如果你觉得这篇文章有启发，可以点击这里付费）
