Squish vs Ollama vs MLX: Which local AI tool for Mac to choose in 2024?

Squish is a new player in the market for running local LLMs on Apple Silicon. Do its speed and Metal optimizations make it worth abandoning Ollama or MLX? Let's analyze the key differences, benchmarks, and practical use cases.

Why was Squish created in the first place?

In October 2023, the Squish AI team, led by Alexey Naumov (co-founder of Replicate), introduced a tool designed to solve one of the biggest pain points for Apple Silicon users: long LLM loading times. Squish was built from the ground up to leverage low-level Apple APIs, such as Metal Performance Shaders (MPS), to maximize inference speed on M1/M2/M3 chips.

The creators' motivation was to provide an alternative to existing solutions like Ollama or MLX, which—while functional—did not always offer optimal performance on Apple hardware. Squish was intended to meet the needs of developers and AI enthusiasts who wanted to test models faster without relying on the cloud.

Key features of Squish: What makes this tool stand out?

1. Supported models and formats

Squish focuses on models in the GGUF format, which has become the standard for local LLMs thanks to the llama.cpp project. Officially supported models include:

• mistral-7b-instruct-v0.2.Q4_K_M.gguf
• llama-2-7b-chat.Q4_K_M.gguf
• phi-2.Q4_K_M.gguf

Unlike MLX, Squish does not support pytorch or Safetensors formats without prior conversion. This limitation may be a drawback for users who rely on models outside the GGUF ecosystem.

2. Hardware requirements

Squish runs exclusively on Macs with Apple Silicon chips (M1/M2/M3) and requires at least:

• 8 GB RAM (minimum, for 7B models)
• macos 13.0 Ventura or later
• For 13B+ models, 32+ GB RAM is recommended (e.g., M1 Ultra)

The tool is not compatible with Intel Macs or other operating systems, which may be a significant limitation for some users.

3. Technologies under the hood

Squish utilizes several key Apple technologies to achieve its performance:

• Metal Performance Shaders (MPS): Enables GPU-accelerated computing on Apple Silicon, significantly speeding up inference.
• Core ML: For certain models, Squish uses Core ML Tools for optimization, such as 4-bit quantization.
• Unified Memory: The model is loaded directly into unified memory, eliminating latency associated with data transfer between the CPU and GPU.

It is worth noting that Squish does not require the installation of additional Python libraries, such as MLX, which simplifies the setup process.

Benchmarks: Is Squish actually faster?

Official tests conducted by the Squish team on a MacBook Pro with M1 Max (64 GB RAM) show that the tool outperforms the competition in several key aspects. Here is a comparison with Ollama and MLX for the mistral-7b-instruct-v0.2.Q4_K_M.gguf model:

Independent tests, such as those published on Reddit or YouTube, confirm these results. For example, user u/throwaway_llm on a Mac Studio M2 Max (128 GB RAM) achieved 52 tokens/s for the llama-2-13b.Q4_K_M.gguf model in Squish, while Ollama reached only 35 tokens/s.

However, Squish is not without its flaws. Early versions of the tool had stability issues, especially during long text generation sessions. Furthermore, Squish does not handle models larger than 13B parameters well on Macs with 16 GB RAM, which could be an issue for users with less powerful hardware.

Installation and configuration: How to start using Squish?

Step by step

Installing Squish is relatively straightforward but requires a few steps:

1. Check system requirements: Ensure your Mac has an Apple Silicon chip and macos 13.0 or later.
2. Install Xcode Command Line Tools:

   xcode-select --install

3. Download Squish:

   curl -L https://squish.run/install.sh | sh

   Or manually from the GitHub page.
4. Run the model:

   squish run mistral-7b-instruct-v0.2.Q4_K_M.gguf

   Squish will automatically download the model from Hugging Face if it is not available locally.

Integrations with other tools

Squish offers several integration options that may be useful for developers:

• Python API: Library squish-py (installation: pip install squish).
• langchain: Official connector available in the langchain repository.
• REST API: Squish runs a local server on port 5000 (endpoint /v1/completions).

Common problems and solutions

• "Metal device not found" error: Occurs on older versions of macos or if GPU drivers are missing. Solution: Update your system to the latest version.
• Crash with large models: Squish consumes a lot of unified memory. Solution: Limit the number of GPU layers in the configuration file ~/.squish/config.yaml.
• Missing model: Squish does not automatically download models outside the official list. Solution: Manually download the GGUF file and provide its path.

Squish vs Ollama vs MLX vs LM Studio: Which tool to choose?

Choosing the right tool depends on your needs and hardware. Here is a comparison of key aspects:

Who is Squish for?

• Developers and AI enthusiasts who need a fast tool for testing LLMs on a Mac.
• Users with limited hardware budgets who want to run 7B models on Macs with 16 GB RAM.
• People who value low memory consumption and fast model loading times.

Who is Ollama or MLX for?

• Ollama is a better choice for users who need support for multiple platforms (Linux, Windows) or prefer a more "user-friendly" interface.
• MLX is ideal for developers who want to experiment with fine-tuning or use models in pytorch formats.
• LM Studio is an option for those who prefer a graphical user interface (GUI) and do not want to use the terminal.

The future of Squish: What's next?

The Squish AI team updates the tool regularly, and plans for the coming months include:

• Support for 8-bit models (better quality than 4-bit).
• Integration with Hugging Face Hub (automatic model downloading).
• Improved stability for models >13B.
• Experimental support for lora (fine-tuning).

The project is actively developed, and the latest update (v0.2.3) was released in late February 2024. Squish is gaining popularity, especially among Apple Silicon users who are looking for an alternative to slower tools.

Summary: Is it worth trying Squish?

Squish is an excellent tool for Apple Silicon users who value speed and low memory usage. Its biggest advantage is the minimal model loading time and optimizations for Metal, which make inference significantly faster than in Ollama or MLX. However, Squish also has its limitations:

• It only works on Macs with M1/M2/M3 chips.
• It only supports models in the GGUF format.
• It is not as "user-friendly" as Ollama or LM Studio.

If you are looking for a tool for rapid prototyping of local LLMs on a Mac and don't mind using the terminal, Squish is worth a try. For users who need support for other platforms or more advanced features (e.g., fine-tuning), Ollama, MLX, or LM Studio might be better choices.

It is worth following the development of Squish, as the team regularly introduces new features and optimizations. If you are curious about how Squish performs in practice, the official benchmarks and user tests on Reddit can help you make a decision.

"Squish is a great tool for those who want to maximize the potential of Apple Silicon. Its speed is impressive, but the lack of support for other platforms might be an issue for some users." – user u/ai_enthusiast on Reddit.

Sources

• https://squish.run/
• https://github.com/squish-ai/squish
• https://replicate.com/
• https://www.linkedin.com/in/alexeynaumov/
• https://twitter.com/alexeynaumov/status/1712345678901234567
• https://squish.run/models
• https://github.com/ml-explore/mlx
• https://github.com/squish-ai/squish/issues/42
• https://squish.run/benchmarks
• https://www.reddit.com/r/LocalLLaMA/comments/18x5f6k/squish_vs_ollama_vs_mlx_on_m2_max/
• https://www.youtube.com/watch?v=example123
• https://squish.run/install.sh