Handling Large JSON Files: What Breaks and When

The export lands as a single 40 MB .json file. The editor’s syntax highlighter gives up, and pasting it into a web formatter turns the tab white for several seconds. Someone suggests streaming, someone else says the recursion will blow the stack, and both pieces of advice are aimed at problems you do not have yet.

The useful question is not “is this file large” but “which limit am I about to hit”. There are only about four, and they arrive in a fixed order.

The size ladder

Size	What happens
1 MB	Nothing. Native parse is a few milliseconds, the object tree is tens of MB. Every tool works, including the ones written badly.
10 MB	Native `JSON.parse` takes about 158 ms. A tokeniser written in JavaScript, like this site’s, takes about 780 ms to format the same input. Building a navigable node tree for it costs about 294 MB of heap. Everything still works, but a synchronous parse is now long enough to look like a crash.
100 MB	Extrapolate the same numbers: several seconds of parse in a JS tokeniser, and a tree measured in gigabytes. This is where browser tabs start dying from memory rather than time. Server side it is still routine.
500 MB+	V8 caps a single string at 536,870,888 characters, about 512 MB of ASCII. In Chrome and Node the file cannot be read into a string at all, so no tool built on that step can touch it regardless of how it is written.

Those first three rows are measured on this site’s own tooling; your parser will differ. The fourth is not a performance number, it is a hard ceiling in the engine.

The 512 MB wall

Every browser JSON tool follows the same path: read the file into a string, hand the string to a parser. That first step is where a very large file dies. The engine’s maximum string length is a fixed constant, and FileReader.readAsText or Response.text() on anything past it throws before your code runs.

The constant is per engine. V8 stops at 536,870,888 characters (require('buffer').constants.MAX_STRING_LENGTH on 64-bit Node), SpiderMonkey at 1,073,741,822, and JavaScriptCore at 2,147,483,647. Firefox and Safari therefore survive files that Chrome refuses, but all three have a ceiling, so a tool that has to work everywhere is designed against V8’s.

A chunked parser over a ReadableStream pushes slightly further, because it never materialises the document as one string. That solves the string cap and not the next problem, which is that the parsed result has to fit in memory too. Anything past a couple of hundred MB is not a browser problem: move it to a shell, a language runtime, or a database.

Why the parsed tree is so much bigger than the file

10 MB of text becoming about 294 MB of heap surprises people, but the arithmetic is not mysterious. Consider {"id":1,"ok":true}, which is 18 bytes on disk. In memory it is:

An object with a header and a pointer to its shape or property map.
One pointer-width slot per property before the values, four bytes on a 64-bit build with pointer compression and eight without.
Each string value carrying its own header, length field and character data, and each string key doing the same unless the engine interned it.
Values that are not small integers stored as separate heap cells with their own headers, reached through another pointer.

The overhead is per node, not per byte, so the ratio gets worse as the data gets more structured: 10 MB of one long string is cheap, 10 MB of eighty thousand small objects with eight keys each is not. A plain JSON.parse result is lighter than a viewer tree carrying per node metadata, but it is still a multiple of the source. Budget an order of magnitude, then measure your own shape.

The recursion myth, corrected

The standard warning is that deeply nested JSON blows the stack when you parse it. In a browser, on a current engine, that is no longer true. V8 replaced the recursive JSON parser with an iterative one in v7.6, and it will read a million levels of nesting without complaint. Measured on Node v24.15.0 with V8 13.6.233.17, parsing was fine at depths that used to be fatal.

The failure moved to the other side. JSON.stringify still recurses and throws a RangeError a few thousand levels down, on the same build somewhere near 4,800:

const deepText = '{"a":'.repeat(1000000) + '1' + '}'.repeat(1000000);
const deep = JSON.parse(deepText); // fine, one million levels

JSON.stringify(deep);              // RangeError: Maximum call stack size exceeded

That number is not a constant. It moves with the stack size the runtime started with and with whatever else is on the stack when the call happens, so it is not a figure to design against.

So a service can accept a hostile payload, parse it without complaint, store it, and then fall over when it tries to log or re-emit the same value. Depth limits still belong on the input boundary, even though the input side is the part that survives.

Other runtimes are less forgiving in both directions. CPython’s json module recurses on decode as well as encode, so deeply nested input raises RecursionError on the way in. How deep you get before that depends on the build: the C scanner in CPython 3.14 gave out near 14,000 levels here, well short of what V8 accepts. If you cross language boundaries, the depth your service tolerates is the depth of its strictest hop.

Streaming, with the parts people get wrong

Streaming means never holding the whole document. Every mainstream language has a pull parser for this.

Python’s ijson yields values matching a prefix path. The prefix records.item means “each element of the array at the top level key records”, and item is the literal token for an array element, not a placeholder for a field name. That is the detail people get wrong on first use:

import ijson

total = 0
with open("events.json", "rb") as f:            # binary mode, not text
    for record in ijson.items(f, "records.item"):
        if record["status"] == "failed":
            total += 1

print(total)

ijson picks the fastest backend available at import time, and a C backend is far faster than the pure Python fallback. Check which one you got before concluding that streaming is slow.

Go’s encoding/json does the same thing with Decoder, and the trap is different. Calling Decode once on a top level array decodes the entire array into one slice, which is exactly what you were trying to avoid. You have to consume the opening bracket as a token first, then decode element by element:

f, err := os.Open("events.json")
if err != nil { log.Fatal(err) }
defer f.Close()

dec := json.NewDecoder(f)
if _, err := dec.Token(); err != nil { log.Fatal(err) } // reads the '['

for dec.More() {
    var r Record
    if err := dec.Decode(&r); err != nil { log.Fatal(err) }
    process(r)
}

In Node, stream-json (with Pick to select a subtree and StreamArray to emit elements) or the older JSONStream do the equivalent, and Jackson’s JsonParser gives you the same token loop on the JVM. All of them buy a constant memory profile by giving up anything that needs the whole document at once.

The format was the problem

Streaming a giant JSON array is work you are doing because the file should never have been one array. NDJSON, one complete JSON value per line, removes the whole category of problem: you read a line, parse a line, drop it, and memory is bounded by your largest single record. It splits with split, greps like text, appends without rewriting, and survives a truncated write with the loss of one record instead of the file.

If you are stuck with an array today and want lines tomorrow, NDJSON to JSON converts in both directions, and the JSON viewer will open either.

Why 780 ms is a broken tab

A synchronous parse holds the main thread. Nothing paints and no click registers. Past about 100 ms an interaction stops feeling instant, and past a second the page reads as frozen and the user reaches for the reload. Reload restarts the parse.

The fix is not a faster parser, it is moving the work off the thread that renders. This site’s beautifier parses and formats in a Web Worker, so the tab keeps painting and the progress state is real rather than a lie posted just before a blocking call. That structural choice matters more than any micro optimisation in the tokeniser.

When the answer is not a tool

Some things are worth doing before you reach for any of the above.

Filter first with jq, so the thing you load is small:

jq -c '.records[] | select(.status == "failed")' events.json > failed.ndjson

Note that plain jq reads the whole document into memory. For files larger than RAM, jq --stream is the mode that does not, at the cost of a much stranger event based syntax.

For a slice you just want to look at, the filter tool does the same selection in the browser, and the minifier strips formatting whitespace, which on a pretty printed export is a real fraction of the bytes.

And sometimes the honest answer is that this is not a file problem. If you are grepping a 2 GB export repeatedly, writing it into SQLite or DuckDB once and querying it there makes every later question cheap. A file you keep re-parsing has already told you it wants to be a table.