In the world of high-scale services, every byte counts. We often optimize algorithms, fine-tune database queries, and tweak infrastructure configurations, all in pursuit of efficiency. Yet, sometimes, the biggest culprit for memory bloat hides in plain sight, masquerading as a harmless convenience. For many of us building Python services with Pydantic and Redis, that culprit is often JSON serialization.
JSON is wonderful. It’s human-readable, universally supported, and incredibly easy to work with. But what happens when you’re storing tens of millions of complex user states in Redis, and each state is a JSON-serialized Pydantic model? What happens when that convenience starts costing you servers, cloud invoices, and performance headaches? We found ourselves in exactly this predicament, and the solution involved a deep dive into data serialization that ultimately shrunk our Redis memory footprint by a staggering 7 times.
What started as a clean, convenient way to manage user data eventually turned into a “memory monster,” quietly consuming our resources. We were literally paying for air, and it became clear that a fundamental shift was needed.
The Hidden Cost of Convenience: When JSON Becomes a Memory Monster
Our production service was humming along, serving about 10 million monthly active users. Redis was our backbone, acting as the main storage for all user state. Pydantic models, elegant and type-safe, were our go-to for defining these states, and serializing them to JSON for storage in Redis felt like the most natural choice. It worked, it was simple, and it integrated seamlessly with our Python ecosystem.
But as we grew, so did our Redis cluster. We scaled to five nodes, yet memory pressure stubbornly refused to ease. Those JSON objects, once so friendly, were inflating far beyond the actual data they contained. It was an insidious problem because, on the surface, everything seemed fine. It took a deeper investigation to truly grasp the scale of the waste.
When I finally crunched the numbers, comparing the raw payload size to the JSON-serialized data, the result was a wake-up call: a single user’s data, which should have been around 2,000 bytes, ballooned to 14,000 bytes in JSON. That’s a 7x difference. For every user. Across millions of users. We were essentially paying for 12,000 bytes of overhead per record, multiplied by 10 million.
Why does JSON, our universal exchange format, turn into such a memory hog when used as a low-level cache store?
- It stores field names in full, repeatedly, for every single object.
- It represents types implicitly as strings (e.g., a boolean
trueis 4 characters). - It duplicates structural information over and over again.
- It’s simply not optimized for compact binary data representation.
This isn’t just academic inefficiency. At scale, JSON’s verbosity translates directly into real costs: bigger cloud bills, more RAM, and potentially degraded performance due to increased network I/O and deserialization overhead. It stops being a harmless convenience and becomes a silent tax on memory.
Hunting for Alternatives: Why Common Solutions Fell Short
Once we understood the problem, the hunt for a solution began. The obvious candidates for compact serialization immediately came to mind:
Protobuf
Google’s Protocol Buffers are a staple for efficient data interchange. They’re compact and performant. However, for our specific use case of dynamically evolving Pydantic models in Redis, Protobuf felt like overkill. It introduces a separate schema definition language, requires code generation, and adds a significant layer of tooling and ceremony. We wanted something that could “drop in” without disrupting our agile development flow, and Protobuf felt like using a sledgehammer to crack a nut.
MessagePack & BSON
These formats are often touted as more compact alternatives to JSON, and they certainly are. MessagePack is known for its binary efficiency, and BSON (Binary JSON) offers a more compact, parseable structure than plain JSON. We explored both. While they did offer some improvement over raw JSON, the gains weren’t radical enough to justify the integration effort. More importantly, integrating them cleanly with Pydantic without losing the elegance and convenience of Pydantic models proved to be a clumsy affair.
All these formats are excellent in their own right, but none hit the sweet spot for our specific scenario: “Pydantic + Redis as a state store.” We needed a solution that would:
- Drop into our existing codebase with minimal changes.
- Deliver a radical reduction in memory usage – not just a marginal improvement.
- Avoid any extra DSLs, separate schemas, or code generation steps.
- Work directly and seamlessly with Pydantic models, preserving their type safety and developer experience.
A Tailored Solution: Introducing PyByntic
Since the existing solutions didn’t quite fit our requirements, the path became clear: build something purpose-built. The result was PyByntic – a minimalist binary format designed specifically for Pydantic models, complete with a lightweight encoder/decoder. It aims to deliver significant memory savings while maintaining the Pydantic developer experience we cherish.
The API is intentionally designed for zero-friction integration. In most cases, you simply swap out Pydantic’s native JSON methods:
model.serialize() # replaces .model_dump_json()
Model.deserialize(bytes) # replaces .model_validate_json()
Here’s a quick look at how you might define a model with PyByntic:
from pybyntic import AnnotatedBaseModel
from pybyntic.types import UInt32, String, Bool
from typing import Annotated class User(AnnotatedBaseModel): user_id: Annotated[int, UInt32] username: Annotated[str, String] is_active: Annotated[bool, Bool] data = User( user_id=123, username="alice", is_active=True
) raw = data.serialize()
obj = User.deserialize(raw)
The magic happens with the Annotated type hints, which provide PyByntic with explicit binary type information, allowing for extremely compact storage without separate schema files. For even greater compactness, you can optionally plug in a custom compression function like zlib:
import zlib serialized = user.serialize(encoder=zlib.compress)
deserialized_user = User.deserialize(serialized, decoder=zlib.decompress)
The Proof is in the Numbers (and the Bill)
To truly validate PyByntic’s effectiveness, we ran a rigorous comparison. We generated 2 million user records based on our real production models, ensuring the dataset was representative. Each user object was a complex mix of field types – UInt16, UInt32, Int32, Int64, Bool, Float32, String, and DateTime32. Crucially, these objects also included deeply nested structures like roles and permissions, with some users having hundreds of permissions. This wasn’t a synthetic toy example; it was a realistic dataset reflecting our actual production complexity.
The results were compelling. When storing these 2,000,000 user objects in Redis, here’s how the different serialization formats stacked up:
- JSON: Approximately 35.1 GB (our baseline)
- PyByntic: Just ~4.6 GB (a mere 13.3% of JSON)
- Protobuf: Still significantly better than JSON, but fell behind PyByntic.
- MessagePack & BSON: Also offered improvements but were still substantially larger than PyByntic.
That’s a 7.5x reduction with PyByntic compared to JSON. If you’re running a multi-node Redis cluster, these aren’t just abstract numbers; they directly impact your cloud bill. Using Memorystore for Redis Cluster on Google Cloud Platform, storing 2 million user objects translates to:
- JSON: ~$876/month
- PyByntic: ~$118/month
- MessagePack: ~$380/month
- BSON: ~$522/month
- Protobuf: ~$187/month
The savings are dramatic – over $750 a month for just 2 million objects. And these savings scale linearly, or even further with compression, as your load grows.
Deconstructing the Savings: How Binary Trims the Fat
So, where do these massive memory savings actually come from? It boils down to two simple, yet powerful, facts about binary data versus text-based formats like JSON:
- No Text Formatting Overhead: In JSON, everything is a string or a representation that requires characters. A typical datetime, like
"1970-01-01T00:00:01.000000", takes 26 characters. Each ASCII character is 1 byte (8 bits), so that’s 208 bits. In a binary format, a DateTime32 type is just 32 bits. That’s a 6.5x reduction, purely from avoiding text formatting. Numbers follow the same pattern: the maximum 64-bit unsigned integer (2^64-1) takes 20 characters in JSON (160 bits) but is a fixed 64 bits in binary. - No Repeated Structure: JSON diligently repeats field names for every single object stored. Imagine storing
"username": "alice"millions of times. The string “username” is duplicated over and over. A binary format like PyByntic doesn’t need to do this. The schema (defined by your Pydantic model with itsAnnotatedtypes) is known in advance. This means there’s no “structural tax” levied on every single record, leading to immense savings when you have millions of them.
These three effects – no strings, no repetition, and no formatting overhead – are the pillars upon which the dramatic size reduction is built. It’s a fundamental shift from human-readable text to machine-optimized binary.
A Rational Choice for Scale
If you’re building high-scale services, leveraging Pydantic for data modeling, and relying on Redis as a state store, then JSON, despite its initial convenience, is a luxury you pay a hefty RAM tax for. While general-purpose formats like Protobuf or MessagePack offer improvements, a tailored binary format that remains natively compatible with your existing Pydantic models is simply a more rational and effective choice.
For us, PyByntic became exactly that: a logical, non-disruptive optimization that didn’t break our development flow but fundamentally eliminated an entire class of problems and unnecessary cloud overhead. It transformed a silent memory drain into significant cost savings and improved efficiency, proving that sometimes, the most elegant solution is one built with your specific needs in mind.
You can explore the project on GitHub: https://github.com/sijokun/PyByntic
