Skip to content

Embedded Go Databases I Like

Go's ability to produce static binaries without external libc dependencies makes it an excellent choice for building portable, self-contained applications. When it comes to embedded databases that support ACID transactions, I've had good experiences with two particular options: BBolt and SQLite.

BBolt - The Simple Key-Value Store

BBolt is a fork of Bolt DB, now maintained at etcd-io/bbolt. While it might not be the first database that comes to mind, BBolt powers critical infrastructure that runs the internet:

BBolt excels at read-heavy workloads with sequential writes. Its API surface is minimal, making it straightforward to understand and integrate.

The Good

Simplicity is its strength. BBolt's API is refreshingly simple - you work with buckets (namespaces) and key-value pairs. There's no query language to learn, no complex schema migrations to manage.

Pure Go implementation means no CGO, no external dependencies, and truly portable binaries. This aligns perfectly with Go's philosophy of simplicity and portability.

LLM-friendly codebase. Modern AI assistants like Claude Sonnet understand BBolt's patterns well, making it easier to get help when needed.

ACID transactions provide the consistency guarantees you need for critical data operations.

The Limitations

Low-level operations require manual implementation. BBolt provides the storage layer, but you'll need to implement your own indexing via structuring the buckets in a way that is optimised for access but creates a lot of data redundancy, query patterns, and data access layers.

Key-value semantics can feel limiting when coming from relational databases. Designing bucket structures for complex relationships requires a mental shift from traditional RDBMS thinking.

Single-process access model. BBolt uses file locking, which means only one process can access the database at a time. This makes it suitable for single-daemon applications but limits multi-process architectures.

SQLite with modernc.org/sqlite

SQLite traditionally requires CGO due to its C implementation. However, modernc.org/sqlite provides a pure Go implementation by partially implementing libc in Go. This gives you SQLite's full power while maintaining Go's portability promise.

After initial skepticism, I've put this library through extensive testing and found it performs admirably under various workloads. Most importantly, it supports WAL (Write-Ahead Logging) mode, enabling concurrent reads and limited concurrent writes.

The Good

Full SQL support with all the features you expect from SQLite - triggers, views, CTEs, and more.

WAL mode enables concurrency. Multiple readers can access the database simultaneously from multiple threads and processes, and readers don't block writers, and a writer doesn't block readers.

Battle-tested SQLite semantics means extensive documentation, tooling, and community knowledge.

Production-ready with Litestream. The Litestream project enables real-time replication of your SQLite database to cloud storage (S3, GCS) with minimal cost, making SQLite a viable production database choice. See my article on serverless SQLite on K8s using Litestream for a practical implementation.

Concurrency Gotchas

While modernc.org/sqlite handles concurrent reads well with WAL mode, concurrent writes require careful consideration.

Each sql.DB instance can have sequential writes. However, concurrent writes from multiple goroutines using the same sql.DB instance will quickly trigger database is locked (5) (SQLITE_BUSY) errors. This behavior is well-documented in the SQLite FAQ and has been discussed extensively in the Go community (see mattn/go-sqlite3#274).

The solution is straightforward - limit the connection pool to a single connection:

db.SetMaxOpenConns(1)

This ensures serialized write access within a process while maintaining read concurrency. Alternatively, you could implement a write mutex, but the connection limit approach is less error-prone.

Making the Choice

BBolt shines when you need a simple, embedded storage engine for a single-process application with straightforward data access patterns. Its use in infrastructure tools demonstrates its reliability for specific use cases.

SQLite excels when you need relational data modeling, complex queries, or multi-process access patterns. The pure Go implementation removes the traditional CGO barrier while maintaining SQLite's extensive capabilities.

Both databases offer ACID guarantees and pure Go implementations, making them excellent choices for different scenarios. The decision ultimately depends on whether you need the simplicity of key-value storage or the expressiveness of SQL.

Due to the versatility of SQLite and its ability to handle complex queries and relationships, plus the availability of high-level tooling, I've been using it more frequently in recent projects, such as kodelet.