SQLite + Zstd = Instant Document Loading on Mobile

The Problem

Documentation must reach the device. Three paths exist.

Fetch from a server. Every page waits on a network round trip. Offline access demands a caching layer. Users on planes, trains, and conference wifi get nothing.

Bundle in the app. Ship 100,000 HTML files as assets. Expo and Metro allow this. First launch kills it. The JavaScript bundle ships compressed; when the app starts, the OS unpacks everything into the data directory. At launch, all docsets combined run ~300MB. Compressed bundle plus expanded files balloons storage to nearly a gigabyte. The user watches a frozen splash screen while the phone grinds through documentation they may never open. Updates require a new app release.

Download individual files. Users fetch what they need. The app ships small. Content updates without App Store review. But file system overhead piles up. Open a page: read from disk, parse the path, locate the file, load it into memory. Multiply by thousands of pages. Cold starts drag. Navigation stutters.

We wanted downloads without the file system tax. Instant loads, offline access, content updates decoupled from app releases.

The answer: SQLite with zstd-compressed blobs. One query, one native decompress call, instant content.

The Schema

Each downloaded docset becomes rows in a unified SQLite database. The mobile app maintains one master database (docnative.db) containing all documentation.

export const content = sqliteTable('content', {
  id: integer('id').primaryKey({ autoIncrement: true }),
  docroot: text('docroot').notNull(),
  slug: text('slug').notNull(),
  pageName: text('page_name').notNull(),
  chunkedHtmlZst: blob('chunked_html_zst').notNull(),
  rawTextFts: text('raw_text_fts').notNull(),
});

Two fields matter:

chunkedHtmlZst: The page content as a zstd-compressed JSON array. Each array element is an HTML chunk (a heading, paragraph, code block, or table). Compression reduces storage by 70-80%.

rawTextFts: Plain text extracted from the HTML, stored uncompressed. This powers full-text search via SQLite's FTS5 engine. Search needs to be fast; decompressing every page to search it would be slow.

The tradeoff is intentional. We store content twice: compressed for rendering, uncompressed for searching. The database is slightly larger, but search queries never touch the compression layer.

Why Zstd

Gzip is ubiquitous. Brotli compresses tighter. We chose zstd.

Zstandard landed in 2016, designed by Yann Collet at Facebook. The algorithm treats decompression speed as a first-class constraint. Gzip decompresses around 300 MB/s. Zstd exceeds 1,500 MB/s. On a phone, where the main thread blocks during decompression, that multiplier determines whether navigation feels instant or sticky.

Brotli achieves better ratios at equivalent settings. It was designed for static web assets: compress once on the server, decompress once in the browser. We decompress on every page load. Raw throughput matters more than squeezing bytes.

Native zstd libraries exist for both platforms. The react-native-zstd package wraps C on iOS, Kotlin on Android. Decompression executes at native speed.

Level 5

Zstd offers 22 compression levels. We use level 5.

Level 5 yields 60-70% size reduction on documentation HTML. Higher levels gain another 5-10% but decompress slower. Shaving kilobytes costs milliseconds. Mobile users feel milliseconds.

Both our documentation cleaning pipelines support zstd natively. Python 3.14 added compression.zstd to the standard library. Bun ships Bun.zstdCompressSync. No external dependencies, identical output across toolchains.

The Build Pipeline

Documentation goes through four transformations before reaching the database.

HTML to Chunks

The cleaning pipeline produces sanitized HTML. The chunker splits this into block-level elements.

interface HtmlChunk {
  tag: string;      // h1, p, pre, table
  html: string;     // Raw HTML for this block
  ids: string[];    // Anchor IDs for navigation
}

Each chunk represents one renderable block: a heading, a paragraph, a code block, a table. The tag field enables height estimation during virtualized rendering. The ids array enables scroll-to-anchor navigation.

Wrapper elements (section, article, main) are flattened. Their children become separate chunks. Divs containing only block children are flattened too. Divs containing inline content become single chunks.

Chunks to JSON

The chunk array serializes to JSON. A typical page produces 20-100 chunks, resulting in 50-150KB of JSON.

JSON to Zstd

Native Bun compression:

export function compressSync(input: string): Uint8Array {
  const data = new TextEncoder().encode(input);
  return Bun.zstdCompressSync(data, { level: 5 });
}

The compressed blob is 15-40KB for most pages.

Blob to SQLite

The blob inserts directly into the chunked_html_zst column. SQLite handles binary data natively. No base64 encoding required.

Bun ships with bun:sqlite built in. Our build pipeline writes to SQLite without external dependencies.

Runtime Loading

When a user navigates to a documentation page, the loading sequence is:

1. Check in-memory cache
2. Check in-flight request deduplication
3. Query database
4. Decompress
5. Parse JSON
6. Validate and cache
7. Render

Cache Check

An LRU cache holds the 50 most recently accessed pages. Cache hits skip everything else.

private chunksCache = new Map<string, HtmlChunk[]>();
private maxCacheSize = 50;

const cached = this.chunksCache.get(cacheKey);
if (cached) {
  return cached;
}

50 documents covers the typical reading session. Users rarely access more than 30 different pages before closing the app.

Request Deduplication

Concurrent requests for the same page should not query the database twice. We track in-flight requests and return the pending promise.

private inFlightRequests = new Map<string, Promise<HtmlChunk[]>>();

const inFlight = this.inFlightRequests.get(cacheKey);
if (inFlight) {
  return inFlight;
}

This handles the case where a component mounts, requests content, unmounts due to rapid navigation, and remounts before the first request completes.

Database Query

A single indexed query:

SELECT chunked_html_zst FROM content
WHERE docroot = ? AND slug = ?

The docroot + slug combination is unique. The query returns one row.

Native Decompression

The react-native-zstd module wraps native C on iOS and Kotlin on Android. Pass in the compressed bytes, get back the decompressed string. The work happens outside the JavaScript runtime.

Parse and Validate

The decompressed data is a JSON string. We parse it and validate each chunk with Zod.

const rawChunks = JSON.parse(chunksJson);
const chunks: HtmlChunk[] = [];

for (const chunk of rawChunks) {
  try {
    chunks.push(htmlChunkSchema.parse(chunk));
  } catch {
    log.warn('Skipping invalid chunk');
  }
}

Invalid chunks are skipped, not fatal. A corrupted chunk should not prevent the rest of the page from rendering.

Cache and Return

The validated chunks go into the LRU cache. If the cache exceeds 50 entries, the oldest entry is evicted.

We also update a last_accessed timestamp in the database. This is fire-and-forget; we do not wait for the write to complete before returning chunks.

Download and Merge

Docsets download as .db.gz files from Cloudflare R2. Each file contains the docset's content in a SQLite database with temporary table names (dl_content, dl_meta, dl_files).

Background Decompression

Gzip decompression is CPU-intensive. We run it on a background thread using React Native's worklet system.

export async function decompressOnBackgroundThread(
  compressedData: Uint8Array
): Promise<Uint8Array> {
  'worklet';
  return pako.ungzip(compressedData);
}

The UI thread stays responsive. Users can navigate while downloads decompress.

Merge Transaction

After decompression, we merge the docset into the master database. This happens inside a transaction.

await masterDb.execAsync('BEGIN TRANSACTION');

// Delete old version if updating
await masterDb.runAsync(
  'DELETE FROM sources WHERE name = ?',
  [slug]
);

// Insert metadata
await masterDb.runAsync(
  `INSERT INTO sources (name, version, tree_json, timestamp)
   VALUES (?, ?, ?, ?)`,
  [slug, version, treeJson, timestamp]
);

// Insert all content rows
for (const row of contentRows) {
  await masterDb.runAsync(
    `INSERT INTO content (...)  VALUES (...)`,
    [...]
  );
}

await masterDb.execAsync('COMMIT');

The transaction ensures atomic updates. If the merge fails partway through, nothing changes. Users never see a half-merged docset.

A merge lock serializes concurrent downloads. Two docsets downloading simultaneously merge one at a time. This prevents transaction conflicts.

FTS Rebuild

After all downloads complete, we rebuild the FTS5 index.

INSERT INTO content_fts(content_fts) VALUES('rebuild');

FTS5's shadow table design means the rebuild reads from the content table directly. No data duplication required on our side.

Multi-Level Caching

Four cache layers, from fastest to slowest:

Level 1: In-Memory Chunks

50-document LRU cache. Sub-millisecond access. Cleared when the app terminates.

Level 2: Scroll Position

Stores the scroll offset for each visited page. When you navigate back, the page restores your position.

class ScrollCache {
  private maxSize = 50;
  private targetSize = 40;  // Evict to 40 for headroom

  save(slug: string, path: string, position: number): void {
    // Batch eviction every 10 saves
    this.savesSinceLastEviction++;
    if (this.savesSinceLastEviction >= 10) {
      this.evictOldEntriesIfNeeded();
    }
  }
}

Persisted to MMKV (fast native key-value storage). Survives app restarts.

Level 3: Downloaded Docsets List

The list of installed docsets is queried once and cached. Adding or removing a docset invalidates the cache.

Level 4: SQLite + FTS5

The database itself. Indexed queries return in single-digit milliseconds. FTS5 searches return ranked results across all docsets.

Performance Numbers

Measured on iPhone 14 Pro:

For large pages (500KB HTML):

The 200ms upper bound is rare. Most documentation pages are under 100KB.

Compression ratios:

MDN JavaScript documentation: 45MB uncompressed, 12MB in the database.

Error Handling

Corrupted data should not crash the app. Zod validates every stage: the raw database row, the decompressed output, each parsed chunk. Failures surface as typed errors, not runtime exceptions.

private handleFetchError(error: unknown, docroot: string, path: string): never {
  const message = error instanceof Error ? error.message : String(error);

  const isDecompressionError =
    message.includes('zstd') ||
    message.includes('decompress');

  if (isDecompressionError) {
    throw new DocDataCorruptedError(docroot, path, 'decompression failed');
  }

  const isJsonError =
    message.includes('JSON') ||
    message.includes('Unexpected');

  if (isJsonError) {
    throw new DocDataCorruptedError(docroot, path, 'invalid JSON');
  }

  throw error;
}

DocDataCorruptedError triggers a UI that offers to re-download the docset. Users can recover without losing other documentation.

Why This Architecture

The alternatives were worse.

Individual files: Each page as a separate .html file. Directory traversal overhead. No search without building a separate index. No atomic updates.

Server-side rendering: Every page load hits the network. Offline access requires a full cache implementation. Latency on every navigation.

In-memory everything: Load all documentation into RAM at startup. Works for small docsets, fails when MDN alone is 45MB uncompressed.

SQLite without compression: Viable, but storage doubles. Users with multiple docsets hit device limits faster.

SQLite with zstd compression threads the needle. Fast queries, small storage, offline by default, atomic updates. The database is the cache.

The Result

DocNative ships with docsets including MDN, Python, React, TypeScript, Go, and Rust. Each docset downloads as a compressed database, merges into the master store, and loads in under 50ms. Search works across all docsets simultaneously. Everything works on airplane mode.

The architecture is simple: one database, one decompression library, one query per page. No distributed caching, no CDN configuration, no offline-first framework. SQLite and zstd do the work.