How to train a small Transformer model for extreme data compression? A practical guide

Data compression using Transformer models opens up new possibilities but requires a precise approach. In this article, I will show how to fine-tune a small model for compression tasks, discuss technical limitations, and compare the results with classic algorithms.

Why does AI-based data compression still spark controversy?

The topic of data compression using artificial intelligence has resurfaced thanks to a thread on Hacker News, where a user described an experiment involving the reduction of 100 MB of data to just 7 MB. While the result sounds impressive, it is worth examining the methods behind such results and whether they are reproducible in practice.

The key question is: can Transformer-based models actually compete with classic compression algorithms? The answer is not straightforward. On one hand, AI excels with high-redundancy data (e.g., natural language, images), but on the other, it fails with random bit sequences, where methods like Zstandard or LZMA still dominate.

Transformer-based compression methods: what works already?

Transformers, originally designed for natural language processing, have also found applications in data compression. They are most commonly used in the form of autoencoders, where:

• The Encoder reduces data to a smaller representation (e.g., latent vectors).
• The Decoder reconstructs the original data from this representation.

Examples of documented applications:

• deepzip (2018) – a combination of LSTM and Transformers for text compression, achieving better results than ZIP for certain datasets.
• compressai (Google) – a library for image and video compression using deep learning, including Vision Transformer (ViT) variants.
• Perceiver IO (deepmind) – a universal model that can act as an autoencoder for various types of data.

Unfortunately, most of these solutions are lossy compression – meaning that some information is irreversibly lost. For binary data (e.g., executables), this is unacceptable, but for text or images, it may be permissible.

Limitations and challenges

Despite promising results, AI-based compression has several significant limitations:

• High computational requirements – training the model requires a GPU, and compression/decompression is slower than with ZIP or Zstandard.
• Model specialization – a model trained on text will not work for image compression.
• Lossiness – for many applications (e.g., data archiving), irreversible information loss is unacceptable.

How to train a small Transformer model for compression?

If you want to train a model for data compression yourself, here are the key stages of the process:

1. Data preprocessing

Data must be properly prepared before being fed into the model:

• Tokenization (for text) – e.g., using BytePairEncoding or WordPiece.
• Normalization (for images) – scaling pixels to the [0, 1] range.
• Chunking – e.g., 512-token chunks for text.

2. Architecture selection

Transformer autoencoders are most commonly used:

• Encoder: Reduces data to a smaller representation (e.g., 7 MB for 100 MB of data).
• Decoder: Reconstructs data from this representation.

Example hyperparameters:

• Model size: 1M–10M parameters (for a small model).
• Context length: 512 tokens.
• Loss function: MSE (for continuous data) or Cross-Entropy (for discrete data).

3. Model training

The training process requires:

• Optimizer: Usually adamw with a learning rate of 1e-4 to 1e-5.
• Batch size: Limited by GPU memory (e.g., 32–128).
• Epochs: Depends on the dataset (e.g., 10–100).

Example code (pseudocode) in pytorch:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
tokenizer = AutoTokenizer.from_pretrained("t5-small")

inputs = tokenizer("Tekst do kompresji...", return_tensors="pt", padding=True, truncation=True)
outputs = model(**inputs, labels=inputs["input_ids"])  # Autoenkoder: wejście = wyjście
loss = outputs.loss
loss.backward()

4. Model optimization

To reduce model size and speed up inference, you can apply:

• Knowledge Distillation – training a small model under the supervision of a larger one.
• Pruning – removing insignificant weights.
• Quantization – reducing weight precision (e.g., float32 → int8).

Extreme compression: is 100 MB → 7 MB a realistic result?

In the Hacker News thread, a user described reducing 100 MB of data to 7 MB, which is a 14x size reduction. Is this possible?

In academic literature, there are no confirmed cases of such extreme compression using Transformers alone. Most likely, the author of the thread:

• Used a hybrid approach – combining AI with classic methods (e.g., entropy coding).
• Compressed highly redundant data (e.g., server logs with repeating entries).
• Applied preprocessing (e.g., tokenization, normalization) before compression.

It is also worth noting that 7 MB might be the encoded representation (e.g., model weights + compressed data), not a direct result of compression.

Comparison with classic compression methods

How do AI models perform compared to classic algorithms?

*Depends on data and model.

Classic methods are faster and lossless, but AI can achieve a better compression ratio for high-redundancy data. In practice, the best results come from combining both approaches.

Potential applications of extreme compression

Where can AI-based compression find practical use?

• Data archiving – storing server logs, databases.
• Data transfer – reducing cloud transfer costs (e.g., AWS S3).
• IoT systems – memory-constrained devices (e.g., sensors).
• Blockchain – reducing blockchain size (e.g., Bitcoin, Ethereum).
• Computer games – asset compression (textures, 3D models).

Unfortunately, there are also legal and ethical limitations:

• Lossy compression may violate data regulations (e.g., GDPR for medical data).
• Information loss can lead to incorrect decisions in AI systems.

Summary: Is it worth investing in AI-based compression?

Data compression using Transformers is a promising but still experimental field. Although AI models can achieve better results than classic algorithms for certain types of data, their implementation comes with challenges:

• High computational requirements.
• Lossiness of compression.
• Model specialization.

However, if you work with redundant data (e.g., text, images) and have access to a GPU, it is worth testing this approach. In other cases, classic methods (ZIP, Zstandard) remain the best choice.

If you are interested in AI model optimization, check out our article on Fugu Ultra from Sakana AI and running large models locally on NVIDIA Jetson.

Sources

• https://news.ycombinator.com/item?id=48644463
• https://github.com/InterDigitalInc/CompressAI
• https://github.com/facebookresearch/DeepZip
• https://huggingface.co/docs/transformers/model_doc/t5
• https://github.com/tensorflow/compression
• https://arxiv.org/abs/1805.09902
• https://arxiv.org/abs/2102.05030
• https://arxiv.org/abs/2006.09965
• https://arxiv.org/abs/2107.14795
• https://www.cs.brandeis.edu/~dcc/
• https://github.com/google-research/google-research/tree/master/neural_compression
• https://www.nvidia.com/en-us/research/ai-playground/