Uber Shrinks Its Go Translation Cache by Keeping Only Hot Values in Memory
Uber redesigned the Go-based translation cache behind its services, storing keys and frequently used values in memory and the rest on disk. The reported memory footprint dropped sharply while cache hit rates stayed above 99.9%.
TL;DR
- Uber reworked its Go translation cache so that all keys and only frequently used values stay in memory, while the remaining values live in disk-backed files.
- In one reported workload, memory use fell from about 2.7 GB to 750 MB, and the cache hit rate stayed above 99.9%.
- Replacing a heavy concurrent map with a plain map and reducing object count also cut garbage collection overhead.
The problem: memory profiling showed heavy garbage collection
Uber found the issue by profiling memory across its Go services. Garbage collection (GC), the runtime process that finds and frees unused objects, consumed more than 20% of CPU in some services. Translations made up roughly 70% of allocated objects. The cost was not only the data itself but the number of small objects that GC had to track.
The first fix targeted the data structure. Uber replaced Go's sync.Map, a concurrency-safe map, with a regular map[string]string protected by an RWMutex, a read-write lock that allows many concurrent readers. The workload was mostly reads, with few updates and almost no deletions, so a plain map with a lock fit well. The old implementation created several objects and pointers for each entry, and the new value-based map reduced allocations. Uber reports that CPU utilization improved by more than 30% in the affected service.
Object count was not the whole story
Fewer objects did not solve the memory footprint. Translation datasets could hold two to three million keys, but any single service usually read only a small subset of them.
Refreshes made things worse. The service built a replacement cache in the background and then swapped it atomically with the live one. For a short time both copies sat in memory, and some services ran out of memory during refreshes.
Uber considered two alternatives: an external cache and generating a subset of translations for each service. Both would add operational or maintenance work, so the team kept the cache in-process and changed how it stores data.
The hybrid design
The new design keeps the existing Get() and Put() API. All keys stay in memory for fast lookups, while translation values are written to disk-backed files. The cache is split into 256 shards, and each shard has its own read-write mutex, in-memory entries and disk file. Keys are reduced to six-byte hashes to lower metadata overhead, with a fallback to the original key format when hashes collide.
In-memory entries are managed by a time-based global least recently used (LRU) policy, which evicts the values that have gone unused the longest. During a refresh, the system uses recent access history to preload translations that were read in the previous 15 minutes. This avoids a cold cache right after the swap, when every lookup would otherwise go to disk.
The reported numbers are as follows. One translation dataset that previously used about 800 MB in memory needed around 40 MB for metadata after the redesign. With a limit of 50,000 in-memory entries, actual memory use typically ranged from 80 MB to 120 MB. In another workload, about 30,000 of more than four million entries were held in memory, and the cache hit rate remained above 99.9%. The p99 latency for cache misses served from disk stayed below 100 microseconds, and the p99 miss latency across the design was reported as below one millisecond.
The sources do not explain how the headline reduction of more than 90% in initial memory footprint relates to the 2.7 GB to 750 MB example, so readers should treat them as separate measurements.
Why it matters
For backend teams, the main lesson is about cost at scale. Uber's engineers say that "at Uber's scale, even small inefficiencies in memory management can add up." Their advice is to minimize object references, since each pointer adds GC work, and to measure how often cached values are actually accessed.
If only a small share of cached data is used at any time, a hybrid layout can keep the hot part in memory and move the rest to disk without changing the calling code. Teams that run large in-process caches, especially ones that refresh by building a second copy, should check whether refreshes can double their peak memory. Preloading recently used keys is a practical way to avoid a cold cache after a swap.
Sources
This article was written by Blozhik from the reporting below. Read the originals for full detail.
InUber Cuts Translation Cache Memory by up to 90% While Keeping P99 Latency Below 1 MSInfoQ