Skip to content
jsonbeautifiers
English

NDJSON and JSON Lines: one record per line

A giant JSON array cannot be streamed, appended to, or partially recovered. One JSON value per line fixes all three.

Every claim here is either measured or sourced. Where it is neither, it says so.

You download an export, it has a .json extension, and the first two lines look like this:

{"ts":"2026-09-01T10:00:00Z","level":"info","msg":"started"}
{"ts":"2026-09-01T10:00:01Z","level":"warn","msg":"retry 1"}

Every line is valid JSON. The file is not. There is no opening bracket, no commas, and no closing bracket, so any parser you point at the whole thing will fail on the second record. This is NDJSON, and you did not choose it; whatever produced the file did, because the alternative does not work at the size the file was going to reach.

The format, precisely

  • One complete JSON value per line. Usually an object, but a bare number or string is legal.
  • UTF-8, with no byte order mark.
  • Records separated by a line feed. Most readers tolerate CRLF, and you should not emit it.
  • Blank lines are ignored, so a trailing newline at the end of the file is fine and is the convention.
  • No enclosing array. No commas between records.
  • Extensions .ndjson and .jsonl, though plenty of files in the wild are named .json or .log.

The reason the format holds together at all is a property of JSON strings: a JSON string cannot contain a literal newline. Control characters below U+0020 must be escaped, so a correctly serialised record can never contain a raw \n. That is what makes “split on newline” a safe tokeniser rather than a guess.

NDJSON or JSON Lines?

They are the same thing. Two small specifications were written separately, and they agree on everything that decides whether a file parses: one JSON value per line, UTF-8, newline separated. The JSON Lines side prefers the .jsonl extension, the NDJSON side prefers .ndjson, and every reader accepts both. Nothing in either document changes a line of your code. If someone asks you which one you are producing, the honest answer is “both”.

Three things a single array cannot do

It streams. A JSON array is one value, so a conventional parser has to hold the whole thing before it hands you anything. Building a document tree costs multiples of the input size in heap: on this site’s own parser, a 10 MB document costs roughly 294 MB. In a browser you hit a harder wall first, because a JavaScript engine caps a single string at 536,870,888 characters, about 512 MB. Nothing above that can even be read into memory as text, let alone parsed. NDJSON has no such ceiling, because you never hold more than one record. See handling large JSON files for the parser-level version of this.

It appends. Adding a record to a JSON array means seeking back over the closing bracket, writing a comma, writing the record, writing the bracket again. Two writers doing that concurrently produce garbage. Appending to NDJSON is a single write at the end of the file with nothing to read first, which is exactly why every log shipper on earth is built on it.

It survives corruption. Truncate a JSON array anywhere and you lose the entire document: Unexpected end of JSON input, no records recovered. Truncate NDJSON and you lose the last line. A bad record costs one record, and a reader that catches per line keeps going.

Where you have already met it

Docker’s default json-file log driver, which writes one JSON object per line per container. Elasticsearch’s _bulk API, which uses a variant of it: an action line, then a document line, and it insists on a trailing newline. BigQuery load jobs, where the source format is literally named NEWLINE_DELIMITED_JSON. ClickHouse’s JSONEachRow. Anything jq -c writes. Streaming APIs, including LLM completions, are usually adjacent rather than identical: server-sent events carry one JSON value per data: line but add their own framing, so an SSE stream is not an NDJSON file even though the payloads are.

What it looks like when you get it wrong

Parse the two-record example above as one document and the messages are specific enough to identify immediately.

Node (V8):

Unexpected non-whitespace character after JSON at position 61 (line 2 column 1)

Python:

Extra data: line 2 column 1 (char 61)

Both mean the same thing: a complete JSON value was parsed successfully and then the input kept going. If you are chasing that one, Extra data in Python covers the variants.

The reverse mistake is just as common. Feed a pretty-printed document to an NDJSON reader and it tries to parse the first line, {, on its own:

# Node:   Expected property name or '}' in JSON at position 1 (line 1 column 2)
# Python: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

A column-2 error on line 1 of every line-oriented read is the signature of a document that was indented before it was written.

Reading and writing it

Python. Iterate the file handle; do not read it into memory.

import json

with open("events.ndjson", encoding="utf-8") as f:
    for n, line in enumerate(f, 1):
        line = line.strip()
        if not line:
            continue
        try:
            record = json.loads(line)
        except json.JSONDecodeError as e:
            print(f"line {n}: {e}")

Writing is where people break the format, and it is one argument:

with open("out.ndjson", "w", encoding="utf-8") as f:
    for record in records:
        f.write(json.dumps(record, separators=(",", ":"), ensure_ascii=False) + "\n")

separators=(",", ":") removes the spaces json.dumps adds by default. Never pass indent=, which emits newlines inside the record and destroys the file. ensure_ascii=False is optional and keeps non-ASCII characters as themselves instead of \uXXXX escapes; the default is True, which is valid but larger.

Node. readline handles the buffer boundaries, and crlfDelay: Infinity stops a \r\n split across two chunks from being read as two line breaks.

import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";

const rl = createInterface({
  input: createReadStream("events.ndjson", "utf8"),
  crlfDelay: Infinity,
});

let n = 0;
for await (const line of rl) {
  n++;
  if (!line.trim()) continue;
  try {
    handle(JSON.parse(line));
  } catch (e) {
    console.error(`line ${n}: ${e.message}`);
  }
}

Go is already correct by default: json.NewEncoder(w).Encode(v) writes a compact record and appends a newline. Call SetEscapeHTML(false) if you do not want <, > and & turned into escapes.

jq reads a stream of whitespace-separated values natively. -c emits one compact value per line, -s slurps the stream into a single array.

jq -c '.[]' big-array.json > events.ndjson   # array to NDJSON
jq -s '.'   events.ndjson  > big-array.json  # NDJSON to array
jq -c 'select(.level == "warn")' events.ndjson

pandas takes lines=True on both sides, and chunksize turns the read into an iterator of frames so you never materialise the file.

import pandas as pd

df = pd.read_json("events.ndjson", lines=True)
df.to_json("out.ndjson", orient="records", lines=True)

for chunk in pd.read_json("events.ndjson", lines=True, chunksize=50_000):
    ...

The rules people break

The only hard rule is that a record occupies exactly one line, which means each record must be minified. If you are producing NDJSON from a formatter, minify each record rather than the file. End the file with a newline: readers skip blank lines, some consumers require the terminator, and cat a.ndjson b.ndjson only works if both files have one.

One underrated benefit of the line discipline: the file is now text your existing tools understand. wc -l counts records, grep filters them, sort and diff work, split shards the file without a parser. A pretty-printed array gives you none of that, which is why comparing two exports usually means loading both into a structural diff.

When not to use it

Anything a browser consumes in one piece. fetch(...).then(r => r.json()) cannot read NDJSON, and neither can a <script type="application/json"> block. Anything that has to be a single valid document: config files, API response bodies, a payload you validate against a schema, a file you hand to a viewer to explore. NDJSON is a transport and storage format for record streams, not a document format.

When you need to cross that line, convert rather than hand-edit. The NDJSON to JSON tool goes both directions in the browser and, when a record fails, tells you which line number it was on instead of failing the whole file.