{"id":147,"date":"2026-11-22T16:40:00","date_gmt":"2026-11-22T16:40:00","guid":{"rendered":"https:\/\/www.masters-tool.com\/blog\/?p=147"},"modified":"2026-11-11T03:39:07","modified_gmt":"2026-11-11T03:39:07","slug":"remove-utf8-bom-before-json-or-yaml-parse","status":"publish","type":"post","link":"https:\/\/www.masters-tool.com\/blog\/remove-utf8-bom-before-json-or-yaml-parse\/","title":{"rendered":"Remove UTF-8 BOM Before JSON or YAML Parse"},"content":{"rendered":"<p><strong>TL;DR \/ Summary:<\/strong> A UTF-8 byte order mark at the start of a config file is invisible in many editors yet fatal to strict JSON and YAML parsers. Strip the BOM before parse, fix the exporter that added it, and add a CI check so Notepad-saved secrets stop breaking deploys.<\/p>\n<h2>What the BOM actually is<\/h2>\n<p>UTF-8 does not need a byte order mark, but some Windows tools still prepend the bytes <code>EF BB BF<\/code>. On screen the file looks identical. At parse time the first key becomes <code>\\ufeffapiVersion<\/code> or the JSON tokenizer rejects the document as invalid unexpectedly near line 1.<\/p>\n<p>YAML libraries vary: some ignore the BOM, others treat it as content. Relying on \u201cit worked on my machine\u201d hides the hazard until a stricter runner appears.<\/p>\n<h2>Symptoms that point to a BOM<\/h2>\n<p>Classic clues include parsers complaining about unexpected character at position 0, Kubernetes rejecting a manifest that validates in a web linter after a re-paste, and diff tools showing no visible change while hashes differ by three bytes.<\/p>\n<p>Hex-dump the first eight bytes. If you see <code>ef bb bf 7b<\/code> before a JSON object brace, you found the culprit. Removing those three bytes restores a clean document.<\/p>\n<h2>Strip locally, then fix the source<\/h2>\n<p>Emergency fix: open the file in a tool that can rewrite UTF-8 without signature, or strip the prefix in a small script before parse. For a one-off paste into a ticket, run the text through a BOM-aware cleaner on <a href=\"https:\/\/www.masters-tool.com\/\">Masters Tool<\/a>, then re-copy the result.<\/p>\n<p>Permanent fix: stop the exporter from writing a BOM. In .NET, prefer encodings that omit the preamble. In editors, set \u201cUTF-8\u201d not \u201cUTF-8 with BOM\u201d as the default for repo file types.<\/p>\n<h3>CI should refuse signed UTF-8<\/h3>\n<p>Add a cheap check that fails when any <code>*.json<\/code>, <code>*.yml<\/code>, or <code>*.yaml<\/code> in config paths starts with <code>EF BB BF<\/code>. Failing early is kinder than debugging a production ConfigMap that \u201clooks fine.\u201d<\/p>\n<h2>JSON vs YAML vs dotenv edges<\/h2>\n<p>JSON is unforgiving: a BOM before <code>{<\/code> is not whitespace in older parsers. YAML 1.1\/1.2 libraries differ on whether a BOM is allowed. Dotenv loaders may include the BOM in the first key name, producing lookups that miss <code>DATABASE_URL<\/code> because the real key is <code>\\ufeffDATABASE_URL<\/code>.<\/p>\n<p>Treat each format explicitly in your strip helper. A shared \u201ctrim weird whitespace\u201d function that only removes ASCII spaces will not catch a BOM.<\/p>\n<h2>Editors and copy-paste pipelines<\/h2>\n<p>Notepad historically wrote BOM-marked UTF-8. Spreadsheet exports and some email clients introduce the mark when \u201chelpful.\u201d Any pipeline that round-trips config through Office tools needs a sanitize step before the file re-enters git.<\/p>\n<p>When reviewing a PR that only \u201ctouches whitespace,\u201d inspect the binary prefix. Three invisible bytes are a real change.<\/p>\n<h2>Library defaults worth pinning<\/h2>\n<p>Document which HTTP clients or SDK serializers emit BOM-marked bodies. If a partner API sends BOM-prefixed JSON, strip on ingest at the edge rather than sprinkling try\/except around every parse call.<\/p>\n<p>Unit-test your strip function with fixtures that include BOM and fixtures that do not. Both must parse to identical structures.<\/p>\n<p><strong>Operational playbook for the next outage.<\/strong><\/p>\n<p>When a config suddenly fails after an innocent edit, hex-dump first, ask about editor settings second, blame the schema third. That order saves hours. Keep a known-good canonical copy in the runbook and a one-liner to compare prefixes.<\/p>\n<p>After recovery, file a ticket to encode the CI guard. Outages that only teach one person are scheduled to recur.<\/p>\n<p><strong>Git attributes and checkout on Windows.<\/strong><\/p>\n<p>Working trees on Windows are more likely to reintroduce BOMs when someone saves with the wrong encoding. Add a note to CONTRIBUTING that config files must be UTF-8 without signature, and consider a pre-commit hook that rejects BOM prefixes. Hooks catch the problem before the PR, which is cheaper than a failed deploy.<\/p>\n<p>If a colleague must use an editor that defaults to BOM, give them a project-level settings snippet that overrides the default for this repository only.<\/p>\n<p><strong>HTTP APIs that echo BOM-prefixed bodies.<\/strong><\/p>\n<p>Partners who export \u201cJSON\u201d from Excel-related pipelines sometimes POST bodies that start with a BOM. Strip at your API edge and log a metric when you do, so you can show the partner evidence. Silently accepting forever trains them never to fix the exporter.<\/p>\n<p>Return a clear 400 with a message naming the BOM when the client is under your control; be more tolerant at the edge when a critical vendor cannot change quickly\u2014but still meter the tolerance.<\/p>\n<p><strong>Container images and baked-in config.<\/strong><\/p>\n<p>Config copied into Docker images during build can freeze a BOM forever if the build context file was saved incorrectly. Rebuilds will keep failing until the source file in git is fixed\u2014editing the running container is a temporary mirage. Add the BOM check to the image build job as well as to the app repo so baked configs cannot bypass developer hooks.<\/p>\n<p>When debugging, <code>od -An -tx1<\/code> or a PowerShell format-hex on the first bytes inside the container confirms whether the image layer still carries the mark.<\/p>\n<p><strong>Clipboard paths from chat apps.<\/strong><\/p>\n<p>Paste from some chat and ticket systems prepends invisible characters beyond BOM, including zero-width spaces. Your stripper should target the UTF-8 BOM specifically and optionally a small allowlist of other known junk, without aggressively deleting legitimate leading whitespace in YAML block scalars. Test fixtures should include a BOM case and a zero-width-space case as separate expectations.<\/p>\n<p>Document for support staff: \u201cre-type the first key\u201d is a valid emergency workaround when paste is contaminated.<\/p>\n<h2>Treat the BOM as noise, then parse with confidence<\/h2>\n<p>Invisible prefixes are still bytes. Strip UTF-8 BOMs before JSON or YAML parse, ban them in CI, and correct exporters at the source. A quick sanitize pass\u2014whether in your pipeline or via <a href=\"https:\/\/www.masters-tool.com\/\">online text utilities<\/a>\u2014turns mysterious position-zero errors into a boring, solved class of incident.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>TL;DR \/ Summary: A UTF-8 byte order mark at the start of a config file is invisible in many editors yet fatal to strict JSON and YAML parsers. Strip the BOM before parse, fix the exporter that added it, and add a CI check so Notepad-saved secrets stop breaking deploys. What the BOM actually is [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":43,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-147","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-troubleshooting","entry","has-media"],"_links":{"self":[{"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/posts\/147","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/comments?post=147"}],"version-history":[{"count":1,"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/posts\/147\/revisions"}],"predecessor-version":[{"id":148,"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/posts\/147\/revisions\/148"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/media\/43"}],"wp:attachment":[{"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/media?parent=147"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/categories?post=147"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.masters-tool.com\/blog\/wp-json\/wp\/v2\/tags?post=147"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}