Lines and tokens

Use token output to render code yourself, inspect syntax captures, or send highlighting results to another part of your application. A token contains source text and optional styling information, not HTML or final colors.

For ready-to-display output, use the HTML renderer instead.

Highlight without rendering

import { highlight } from "@comfit/node";

const lines = highlight("const answer = 42;\n", "javascript");
for (const line of lines) {
  for (const token of line.tokens) {
    console.log(line.number, token.text, token.highlight, token.rainbow);
  }
}

const lines = highlighter.highlight("const answer = 42;\n", "javascript");
for (const line of lines) {
  for (const token of line.tokens) {
    console.log(line.number, token.text, token.highlight, token.rainbow);
  }
}

let mut highlighter = comfit::Highlighter::new()?;
let highlighted = highlighter.highlight(
    "const answer = 42;\n",
    "javascript",
    &comfit::HighlightConfig::default(),
)?;
let lines = &highlighted.lines;
for line in lines {
    for token in &line.tokens {
        println!("{} {:?} {:?} {:?}", line.number, token.text, token.highlight, token.rainbow);
    }
}

import comfit

lines = comfit.highlight("const answer = 42;\n", "javascript")
for line in lines:
    for token in line.tokens:
        print(line.number, token.text, token.highlight, token.rainbow)

Read the result

Each line contains:

Each token contains:

Absent captures and depths are null in JavaScript, None in Python, and None in Rust's optional fields. A token boundary is not necessarily a syntax-node boundary: captures can overlap and be split across lines.

An empty source produces one empty line. A trailing newline produces a final empty line. To reconstruct the source, use the lines from the highlighting example:

const source = lines
  .map((line) => line.tokens.map((token) => token.text).join(""))
  .join("\n");

let source = lines
    .iter()
    .map(|line| line.tokens.iter().map(|token| token.text).collect::<String>())
    .collect::<Vec<_>>()
    .join("\n");

source = "\n".join(
    "".join(token.text for token in line.tokens) for line in lines
)

Use captures as style names

Captures describe syntax roles rather than final colors. A theme maps them to styles. Dotted names allow more specific styling, such as function and function.call.

Tokens with no capture still contain source text and must be rendered. A rainbow depth is independent of its final color. The renderer cycles through a palette by depth.

Render tokens safely

In React, render token.text as a text child. In plain DOM code, assign it to textContent. Do not concatenate unescaped token text into HTML.

If you do not need custom markup, use Comfit's HTML renderer, which already escapes source text.

Keep results after edits

Node.js, Python, and browser token arrays own their text. Rust's one-shot result borrows the source and highlighter. A document session returns an owned snapshot that remains usable after later edits.