JSON, YAML & Data Architecture: Schema Validation, Typing, Streaming & Serialization Best Practices

Despite being the universal language of web APIs, standard JSON has critical architectural limitations that cause silent bugs in production: ```javascript // 1. BigInt Truncation & Serialization Error: const bigNumber = 9007199254740993n; // Exceeds Number.MAX_SAFE_INTEGER JSON.stringify({ id: bigNumber }); // ❌ Uncaught TypeError: Do not know how to serialize a BigInt! // Remedy: Convert BigInt to string or implement BigInt.prototype.toJSON. // 2. Silent Field Dropping: const payload = { active: true, notes: undefined, // ❌ Silently stripped out by JSON.stringify! fn: () => {}, // ❌ Silently stripped out! sym: Symbol('secret') // ❌ Silently stripped out! }; console.log(JSON.stringify(payload)); // '{"active":true}' // 3. Date Object Degrades to String: const session = { loggedInAt: new Date() }; const restored = JSON.parse(JSON.stringify(session)); console.log(typeof restored.loggedInAt); // "string" (NOT Date!) ``` **IEEE 754 Floating Point Precision Trap:** Standard JavaScript numbers are 64-bit floats. Any 64-bit integer (like Snowflake IDs or database `BIGINT` primary keys $> 2^{53}-1$) sent as raw numbers in JSON will be rounded and corrupted silently in transit. Always serialize 64-bit IDs as JSON strings.

const safeStringify = (obj) => JSON.stringify(obj, (key, value) =>
  typeof value === 'bigint' ? value.toString() : value
);

YAML (YAML Ain't Markup Language) is widely used for Kubernetes, Docker Compose, and CI/CD pipelines because of its human readability. However, it contains severe syntactic traps: **The Infamous "Norway Problem" (YAML 1.1 Boolean Trap):** In YAML 1.1 (used by many parsers and GitHub Actions runners), the two-letter country code for Norway (`NO`) is parsed as boolean `false`! ```yaml # ❌ VULNERABLE: Parsed as ["CA", "US", false] instead of string "NO"! countries: - CA - US - NO # ✅ FIXED: Always quote strings that could match boolean keywords (y, n, yes, no, on, off) countries: - "CA" - "US" - "NO" ``` **DRY Configurations with YAML Anchors (`&`) and Aliases (`*`):** Eliminate repetitive configuration blocks using native YAML inheritance: ```yaml defaults: &default_settings timeout: 30 retry: 3 logging: "json" production_service: <<: *default_settings host: "api.production.internal" staging_service: <<: *default_settings host: "api.staging.internal" retry: 1 # Overrides default retry setting ```

TypeScript types only exist at compile time; they evaporate completely when compiled to JavaScript. Untrusted JSON payloads from network APIs must be verified at runtime. Adopt the **"Parse, don't validate"** philosophy using Zod to infer compile-time types directly from runtime schemas: ```typescript import { z } from 'zod'; export const UserSchema = z.object({ id: z.string().uuid(), email: z.string().email(), role: z.enum(['admin', 'member', 'guest']).default('member'), age: z.number().int().min(18).optional(), metadata: z.record(z.string()).default({}) }); // Infer TypeScript type automatically without duplicating declarations: export type User = z.infer; export function processApiPayload(rawJson: unknown): User { // Throws informative ZodError with path if schema is violated: return UserSchema.parse(rawJson); } ```

Standard `JSON.parse()` requires loading the entire file into V8 heap memory at once. Parsing a 2 GB JSON array will crash Node.js with `JavaScript heap out of memory`. **JSON Lines (`.jsonl` or NDJSON - Newline Delimited JSON):** Store one valid, self-contained JSON object per line separated by `\n`. This enables constant $O(1)$ memory streaming line by line:

import fs from 'fs';
import readline from 'readline';

async function processBigData(filePath) {
  const fileStream = fs.createReadStream(filePath);
  const rl = readline.createInterface({ input: fileStream, crlfDelay: Infinity });

  for await (const line of rl) {
    if (!line.trim()) continue;
    const record = JSON.parse(line);
    // Process individual record without filling heap memory:
    await handleRecord(record);
  }
}

Choosing the optimal serialization format based on engineering tradeoffs:

Rapid CLI one-liners for filtering, mapping, and transforming complex JSON documents using `jq`:

curl -s https://api.github.com/repos/facebook/react/issues | jq ".[ ] | {id: .id, title: .title, user: .user.login}"

cat telemetry.json | jq ".events[].category" | sort | uniq -c | sort -nr

cat users.json | jq -r ".[] | [.id, .name, .email] | @csv"