Coder Social home page Coder Social logo

starry-night's Introduction

Close up of The Starry Night by Vincent van Gogh (1889)
with examples of starry-night over it


starry-night

Build Coverage Downloads Size

Syntax highlighting, like what GitHub uses to highlight code, but free and open source and JavaScript!

Contents

What is this?

This package is an open source version of GitHub’s closed-source PrettyLights project (more on that later). It supports 600+ grammars and its extremely high quality. It uses TextMate grammars which are also used in popular editors (SublimeText, Atom, VS Code, &c). They’re heavy but high quality.

When should I use this?

starry-night is a high quality highlighter (when your readers or authors are programmers, you want this!) that can support tons of grammars (from new things like MDX to much more!) which approaches how GitHub renders code.

It has a WASM dependency, and rather big grammars, which means that starry-night might be too heavy particularly in browsers, in which case lowlight or refractor might be more suitable.

This project is similar to the excellent shiki, and it uses the same underlying dependencies, but starry-night is meant to match GitHub in that it produces classes and works with the CSS it ships, making it easier to add dark mode and other themes with CSS compared to inline styles.

Finally, this package produces objects (an AST), which makes it useful when you want to perform syntax highlighting in a place where serialized HTML wouldn’t work or wouldn’t work well. For example, when you want to show code in a CLI by rendering to ANSI sequences, when you’re using virtual DOM frameworks (such as React or Preact) so that diffing can be performant, or when you’re working with hast or rehype.

Bundled, minified, and gzipped, starry-night and the WASM binary are 185 kB. There are two lists of grammars you can use: common (±35 languages, good for your own site) adds 250 kB and all (~600 languages, useful if you are making a site like GitHub) is 1.6 MB. You can also manually choose which grammars to include (or add to common): a language is typically between 3 and 5 kB. To illustrate, Astro costs 2.1 kB and TSX costs 25.4 kB.

What is PrettyLights?

PrettyLights is the syntax highlighter that GitHub uses to turn this:

```markdown
# Hello, world!
```

…into this:

<span class="pl-mh"><span class="pl-mh">#</span><span class="pl-mh"> </span>Hello, world!</span>

…which is what starry-night does too (some small differences in markup, but essentially the same)!

PrettyLights is responsible for taking the flag markdown, looking it up in languages.yml from github-linguist to figure out that that means markdown, taking a corresponding grammar (in this case atom/language-gfm), doing some GPL magic in C, and turning it into spans with classes.

GitHub is using PrettyLights since December 2014, when it replaced Pygments. They wanted to open source it, but were unable due to licensing issues. Recently (Feb 2019?), GitHub has slowly started to move towards TreeLights, which is based on TreeSitter, and also closed source. If TreeLights includes a language (currently: C, C#, CSS, CodeQL, EJS, Elixir, ERB, Gleam, Go, HTML, Java, JS, Nix, PHP, Python, RegEx, Ruby, Rust, TLA, TS), that’ll be used, for everything else PrettyLights is used.

starry-night does what PrettyLights does, not what TreeLights does. I’m hopeful that that will be open sourced in the future and we can mimic both.


Install

This package is ESM only. In Node.js (version 16+), install with npm:

npm install @wooorm/starry-night

In Deno with esm.sh:

import {common, createStarryNight} from 'https://esm.sh/@wooorm/starry-night@3'

In browsers with esm.sh:

<script type="module">
  import {common, createStarryNight} from 'https://esm.sh/@wooorm/starry-night@3?bundle'
</script>

To get the CSS in browsers, do (see CSS for more info):

<!-- This supports light and dark mode automatically. -->
<link rel="stylesheet" href="https://esm.sh/@wooorm/starry-night@3/style/both">

Use

import {common, createStarryNight} from '@wooorm/starry-night'

const starryNight = await createStarryNight(common)

const scope = starryNight.flagToScope('markdown')
const tree = starryNight.highlight('# hi', scope)

console.log(tree)

Yields:

{
  type: 'root',
  children: [
    {
      type: 'element',
      tagName: 'span',
      properties: {className: ['pl-mh']},
      children: [
        {type: 'text', value: '# '},
        {
          type: 'element',
          tagName: 'span',
          properties: {className: ['pl-en']},
          children: [{type: 'text', value: 'hi'}]
        }
      ]
    }
  ]
}

API

This package exports the identifiers all, common, and createStarryNight from the main module. There is no default export.

It also includes grammars directly in its export map. Do not use the lang/ folder or the .js extension. For CSS files, do use style/ but don’t use .css:

import mdx from '@wooorm/starry-night/source.mdx' // Grammar.
import tritanopiaDark from '@wooorm/starry-night/style/tritanopia-dark' // CSS.

all

List of all grammars (Array<Grammar>)

common

List of ±35 common grammars (Array<Grammar>)

createStarryNight(grammars[, options])

Create a StarryNight that can highlight things with the given grammars. This is async to allow async loading and registering, which is currently only used for WASM.

Parameters
Returns

Promise that resolves to an instance which highlights with the bound grammars (Promise<StarryNight>).

starryNight.flagToScope(flag)

Get the grammar scope (such as text.md) associated with a grammar name (such as markdown) or grammar extension (such as .mdwn).

This function uses the first word (when splitting on spaces and tabs) that is used after the opening of a fenced code block:

```js
console.log(1)
```

To match GitHub, this also accepts entire paths:

```path/to/example.js
console.log(1)
```

👉 Note: languages can use the same extensions. For example, .h is reused by many languages. In those cases, you will get one scope back, but it might not be the most popular language associated with an extension.

Parameters
  • flag (string) — grammar name (such as 'markdown'), grammar extension (such as '.mdwn'), or entire file path ending in extension
Returns

Grammar scope, such as 'text.md' (string or undefined).

Example
import {common, createStarryNight} from '@wooorm/starry-night'

const starryNight = await createStarryNight(common)

console.log(starryNight.flagToScope('pandoc')) // `'text.md'`
console.log(starryNight.flagToScope('workbook')) // `'text.md'`
console.log(starryNight.flagToScope('.workbook')) // `'text.md'`
console.log(starryNight.flagToScope('path/to/example.js')) // `'source.js'`
console.log(starryNight.flagToScope('whatever')) // `undefined`

starryNight.highlight(value, scope)

Highlight programming code.

Parameters
  • value (string) — code to highlight
  • scope (string) — registered grammar scope to highlight as (such as 'text.md')
Returns

Node representing highlighted code (Root).

Example
import {createStarryNight} from '@wooorm/starry-night'
import sourceCss from '@wooorm/starry-night/source.css'

const starryNight = await createStarryNight([sourceCss])

console.log(starryNight.highlight('em { color: red }', 'source.css'))

Yields:

{
  type: 'root',
  children: [
    {type: 'element', tagName: 'span', properties: [Object], children: [Array]},
    {type: 'text', value: ' { '},
    {type: 'element', tagName: 'span', properties: [Object], children: [Array]},
    {type: 'text', value: ': '},
    {type: 'element', tagName: 'span', properties: [Object], children: [Array]},
    {type: 'text', value: ' }'}
  ]
}

starryNight.missingScopes()

List scopes that are needed by the registered grammars but that are missing.

To illustrate, the text.xml.svg grammar needs the text.xml grammar. When you register text.xml.svg without text.xml, it will be listed here.

Returns

List of grammar scopes, such as 'text.md' (Array<string>).

Example
import {createStarryNight} from '@wooorm/starry-night'
import textXml from '@wooorm/starry-night/text.xml'
import textXmlSvg from '@wooorm/starry-night/text.xml.svg'

const svg = await createStarryNight([textXmlSvg])
console.log(svg.missingScopes()) //=> ['text.xml']

const svgAndXml = await createStarryNight([textXmlSvg, textXml])
console.log(svgAndXml.missingScopes()) //=> []

starryNight.register(grammars)

Add more grammars.

Parameters
Returns

Promise resolving to nothing (Promise<undefined>).

Example
import {createStarryNight} from '@wooorm/starry-night'
import sourceCss from '@wooorm/starry-night/source.css'
import textMd from '@wooorm/starry-night/text.md'
import {toHtml} from 'hast-util-to-html'

const markdown = '```css\nem { color: red }\n```'

const starryNight = await createStarryNight([textMd])

console.log(toHtml(starryNight.highlight(markdown, 'text.md')))

await starryNight.register([sourceCss])

console.log(toHtml(starryNight.highlight(markdown, 'text.md')))

Yields:

<span class="pl-s">```</span><span class="pl-en">css</span>
<span class="pl-c1">em { color: red }</span>
<span class="pl-s">```</span>
<span class="pl-s">```</span><span class="pl-en">css</span>
<span class="pl-ent">em</span> { <span class="pl-c1">color</span>: <span class="pl-c1">red</span> }
<span class="pl-s">```</span>

starryNight.scopes()

List all registered scopes.

Returns

List of grammar scopes, such as 'text.md' (Array<string>).

Example
import {common, createStarryNight} from '@wooorm/starry-night'

const starryNight = await createStarryNight(common)

console.log(starryNight.scopes())

Yields:

[
  'source.c',
  'source.c++',
  // …
  'text.xml',
  'text.xml.svg'
]

GetOnigurumaUrl

Function to get a URL to the oniguruma WASM (TypeScript type).

👉 Note: this must currently result in a version 2 URL of onig.wasm from vscode-oniguruma.

⚠️ Danger: when you use this functionality, your project might break at any time (when reinstalling dependencies), except when you make sure that the WASM binary you load manually is what our internally used vscode-oniguruma dependency expects. To solve this, you could for example use an npm script called dependencies (which runs everytime node_modules is changed) which copies vscode-oniguruma/release/onig.wasm to the place you want to host it.

Returns

URL object to a WASM binary (Promise<URL> or URL).

Example
import {common, createStarryNight} from '@wooorm/starry-night'

const starryNight = await createStarryNight(common, {
  getOnigurumaUrlFetch() {
    return new URL('/onig.wasm', window.location.href);
  }
})

Grammar

TextMate grammar with some extra info (TypeScript type).

Fields
  • dependencies (Array<string>, optional, example: ['source.tsx']) — list of scopes that are needed for this grammar to work
  • extensions (Array<string>, example: ['.mdx']) — list of extensions
  • extensionsWithDot (Array<string>, optional, example: ['.php']) — list of extensions that only match if used w/ a dot
  • injections (Record<string, Rule>, optional) — TextMate injections
  • names (Array<string>, example: ['mdx']) — list of names
  • patterns (Array<Rule>) — TextMate patterns
  • repository (Record<string, Rule>, optional) — TextMate repository
  • scopeName (string, example: 'source.mdx') — scope

Options

Configuration (TypeScript type).

Fields
  • getOnigurumaUrlFetch (GetOnigurumaUrl, optional) — get a URL to the oniguruma WASM, typically used in browsers
  • getOnigurumaUrlFs (GetOnigurumaUrl, optional) — get a URL to the oniguruma WASM, typically used in Node.js

Examples

Example: serializing hast as html

hast trees as returned by starry-night can be serialized with hast-util-to-html:

import {common, createStarryNight} from '@wooorm/starry-night'
import {toHtml} from 'hast-util-to-html'

const starryNight = await createStarryNight(common)

const tree = starryNight.highlight('"use strict";', 'source.js')

console.log(toHtml(tree))

Yields:

<span class="pl-s"><span class="pl-pds">"</span>use strict<span class="pl-pds">"</span></span>;

Example: using starry-night on the client

You don’t have to do preprocess things on a server. Particularly, when you are not using Node.js or so. Or, when you have a lot of often changing content (likely markdown), such as on a page of comments.

In those cases, you can run starry-night in the browser. Here is an example. It also uses hast-util-to-dom, which is a light way to turn the AST into DOM nodes.

Say we have this example.js on our browser (no bundling needed!):

import {
  common,
  createStarryNight
} from 'https://esm.sh/@wooorm/starry-night@3?bundle'
import {toDom} from 'https://esm.sh/hast-util-to-dom@4?bundle'

const starryNight = await createStarryNight(common)
const prefix = 'language-'

const nodes = Array.from(document.body.querySelectorAll('code'))

for (const node of nodes) {
  const className = Array.from(node.classList).find(function (d) {
    return d.startsWith(prefix)
  })
  if (!className) continue
  const scope = starryNight.flagToScope(className.slice(prefix.length))
  if (!scope) continue
  const tree = starryNight.highlight(node.textContent, scope)
  node.replaceChildren(toDom(tree, {fragment: true}))
}

…and then, if we would have an index.html for our document:

<!doctype html>
<meta charset=utf8>
<title>Hello</title>
<link rel=stylesheet href=https://esm.sh/@wooorm/starry-night@3/style/both>
<body>
<h1>Hello</h1>
<p>…world!</p>
<pre><code class=language-js>console.log('it works!')
</code></pre>
<script type=module src=./example.js></script>
</body>

Opening that page in a browser, we’d see the <code> being swapped with:

<code class="language-js"><span class="pl-en">console</span>.<span class="pl-c1">log</span>(<span class="pl-s"><span class="pl-pds">'</span>it works!<span class="pl-pds">'</span></span>)
</code>

Example: turning hast into react nodes

hast trees as returned by starry-night can be turned into preact, react, solid, svelte, vue, etc., with hast-util-to-jsx-runtime:

import {common, createStarryNight} from '@wooorm/starry-night'
import {toJsxRuntime} from 'hast-util-to-jsx-runtime'
import {Fragment, jsx, jsxs} from 'react/jsx-runtime'

const starryNight = await createStarryNight(common)

const tree = starryNight.highlight('"use strict";', 'source.js')
const reactNode = toJsxRuntime(tree, {Fragment, jsx, jsxs})

console.log(reactNode)

Yields:

{
  '$$typeof': Symbol(react.element),
  type: Symbol(react.fragment),
  key: null,
  ref: null,
  props: { children: [ [Object], ';' ] },
  _owner: null,
  _store: {}
}

Example: adding line numbers

GitHub itself does not add line numbers to the code they highlight. You can do that, by transforming the AST. Here’s an example of a utility that wraps each line into a span with a class and a data attribute with its line number. That way, you can style the lines as you please. Or you can generate different elements for each line, of course.

Say we have our utility as hast-util-starry-night-gutter.js:

/**
 * @typedef {import('hast').Element} Element
 * @typedef {import('hast').ElementContent} ElementContent
 * @typedef {import('hast').Root} Root
 * @typedef {import('hast').RootContent} RootContent
 */

/**
 * @param {Root} tree
 *   Tree.
 * @returns {undefined}
 *   Nothing.
 */
export function starryNightGutter(tree) {
  /** @type {Array<RootContent>} */
  const replacement = []
  const search = /\r?\n|\r/g
  let index = -1
  let start = 0
  let startTextRemainder = ''
  let lineNumber = 0

  while (++index < tree.children.length) {
    const child = tree.children[index]

    if (child.type === 'text') {
      let textStart = 0
      let match = search.exec(child.value)

      while (match) {
        // Nodes in this line.
        const line = /** @type {Array<ElementContent>} */ (
          tree.children.slice(start, index)
        )

        // Prepend text from a partial matched earlier text.
        if (startTextRemainder) {
          line.unshift({type: 'text', value: startTextRemainder})
          startTextRemainder = ''
        }

        // Append text from this text.
        if (match.index > textStart) {
          line.push({
            type: 'text',
            value: child.value.slice(textStart, match.index)
          })
        }

        // Add a line, and the eol.
        lineNumber += 1
        replacement.push(createLine(line, lineNumber), {
          type: 'text',
          value: match[0]
        })

        start = index + 1
        textStart = match.index + match[0].length
        match = search.exec(child.value)
      }

      // If we matched, make sure to not drop the text after the last line ending.
      if (start === index + 1) {
        startTextRemainder = child.value.slice(textStart)
      }
    }
  }

  const line = /** @type {Array<ElementContent>} */ (tree.children.slice(start))
  // Prepend text from a partial matched earlier text.
  if (startTextRemainder) {
    line.unshift({type: 'text', value: startTextRemainder})
    startTextRemainder = ''
  }

  if (line.length > 0) {
    lineNumber += 1
    replacement.push(createLine(line, lineNumber))
  }

  // Replace children with new array.
  tree.children = replacement
}

/**
 * @param {Array<ElementContent>} children
 * @param {number} line
 * @returns {Element}
 */
function createLine(children, line) {
  return {
    type: 'element',
    tagName: 'span',
    properties: {className: 'line', dataLineNumber: line},
    children
  }
}

…and a module example.js:

import {common, createStarryNight} from '@wooorm/starry-night'
import {toHtml} from 'hast-util-to-html'
import {starryNightGutter} from './hast-util-starry-night-gutter.js'

const starryNight = await createStarryNight(common)

const tree = starryNight.highlight(
  '# Some heading\n\n```js\nalert(1)\n```\n***',
  'text.md'
)

starryNightGutter(tree)

console.log(toHtml(tree))

Now running node example.js yields:

<span class="line" data-line-number="1"><span class="pl-mh"># <span class="pl-en">Some heading</span></span></span>
<span class="line" data-line-number="2"></span>
<span class="line" data-line-number="3"><span class="pl-s">```</span><span class="pl-en">js</span></span>
<span class="line" data-line-number="4"><span class="pl-en">alert</span>(<span class="pl-c1">1</span>)</span>
<span class="line" data-line-number="5"><span class="pl-s">```</span></span>
<span class="line" data-line-number="6"><span class="pl-ms">***</span></span>

Example: integrate with unified, remark, and rehype

This example shows how to combine starry-night with unified: using remark to parse the markdown and transforming it to HTML with rehype. If we have a markdown file example.md:

# Hello

…world!

```js
console.log('it works!')
```

…and a plugin rehype-starry-night.js:

/**
 * @typedef {import('@wooorm/starry-night').Grammar} Grammar
 * @typedef {import('hast').ElementContent} ElementContent
 * @typedef {import('hast').Root} Root
 */

/**
 * @typedef Options
 *   Configuration (optional)
 * @property {Array<Grammar> | null | undefined} [grammars]
 *   Grammars to support (default: `common`).
 */

import {common, createStarryNight} from '@wooorm/starry-night'
import {toString} from 'hast-util-to-string'
import {visit} from 'unist-util-visit'

/**
 * Highlight code with `starry-night`.
 *
 * @param {Options | null | undefined} [options]
 *   Configuration (optional).
 * @returns
 *   Transform.
 */
export default function rehypeStarryNight(options) {
  const settings = options || {}
  const grammars = settings.grammars || common
  const starryNightPromise = createStarryNight(grammars)
  const prefix = 'language-'

  /**
   * Transform.
   *
   * @param {Root} tree
   *   Tree.
   * @returns {Promise<undefined>}
   *   Nothing.
   */
  return async function (tree) {
    const starryNight = await starryNightPromise

    visit(tree, 'element', function (node, index, parent) {
      if (!parent || index === undefined || node.tagName !== 'pre') {
        return
      }

      const head = node.children[0]

      if (!head || head.type !== 'element' || head.tagName !== 'code') {
        return
      }

      const classes = head.properties.className

      if (!Array.isArray(classes)) return

      const language = classes.find(function (d) {
        return typeof d === 'string' && d.startsWith(prefix)
      })

      if (typeof language !== 'string') return

      const scope = starryNight.flagToScope(language.slice(prefix.length))

      // Maybe warn?
      if (!scope) return

      const fragment = starryNight.highlight(toString(head), scope)
      const children = /** @type {Array<ElementContent>} */ (fragment.children)

      parent.children.splice(index, 1, {
        type: 'element',
        tagName: 'div',
        properties: {
          className: [
            'highlight',
            'highlight-' + scope.replace(/^source\./, '').replace(/\./g, '-')
          ]
        },
        children: [{type: 'element', tagName: 'pre', properties: {}, children}]
      })
    })
  }
}

…and finally a module example.js:

import fs from 'node:fs/promises'
import rehypeStringify from 'rehype-stringify'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import {unified} from 'unified'
import rehypeStarryNight from './rehype-starry-night.js'

const file = await unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeStarryNight)
  .use(rehypeStringify)
  .process(await fs.readFile('example.md'))

console.log(String(file))

Now running node example.js yields:

<h1>Hello</h1>
<p>…world!</p>
<div class="highlight highlight-js"><pre><span class="pl-en">console</span>.<span class="pl-c1">log</span>(<span class="pl-s"><span class="pl-pds">'</span>it works!<span class="pl-pds">'</span></span>)
</pre></div>

Example: integrating with markdown-it

This example shows how to combine starry-night with markdown-it. If we have a markdown file example.md:

# Hello

…world!

```js
console.log('it works!')
```

…and a module example.js:

/**
 * @typedef {import('hast').ElementContent} ElementContent
 */

import fs from 'node:fs/promises'
import {common, createStarryNight} from '@wooorm/starry-night'
import {toHtml} from 'hast-util-to-html'
import markdownIt from 'markdown-it'

const file = await fs.readFile('example.md')
const starryNight = await createStarryNight(common)

const markdownItInstance = markdownIt({
  highlight(value, lang) {
    const scope = starryNight.flagToScope(lang)

    return toHtml({
      type: 'element',
      tagName: 'pre',
      properties: {
        className: scope
          ? [
              'highlight',
              'highlight-' + scope.replace(/^source\./, '').replace(/\./g, '-')
            ]
          : undefined
      },
      children: scope
        ? /** @type {Array<ElementContent>} */ (
            starryNight.highlight(value, scope).children
          )
        : [{type: 'text', value}]
    })
  }
})

const html = markdownItInstance.render(String(file))

console.log(html)

Now running node example.js yields:

<h1>Hello</h1>
<p>…world!</p>
<pre class="highlight highlight-js"><span class="pl-en">console</span>.<span class="pl-c1">log</span>(<span class="pl-s"><span class="pl-pds">'</span>it works!<span class="pl-pds">'</span></span>)
</pre>

Syntax tree

The generated hast starts with a root node, that represents the fragment. It contains up to three levels of <span> elements, each with a single class. All these levels can contain text nodes with the actual code. Interestingly, TextMate grammars work per line, so all line endings are in the root directly, meaning that creating a gutter to display line numbers can be generated rather naïvely by only looking through the root node.

CSS

starry-night does not inject CSS for the syntax highlighted code (because well, starry-night doesn’t have to be turned into HTML and might not run in a browser!). If you are in a browser, you can use the packaged themes, or get creative with CSS! 💅

All themes accept CSS variables (custom properties). With the theme core.css, you have to define your own properties. All other themes define the colors on :root. Themes either have a dark or light suffix, or none, in which case they automatically switch colors based on a @media (prefers-color-scheme: dark). All themes are tiny (under 1 kB). The shipped themes are as follows:

name Includes light scheme Includes dark scheme
@wooorm/starry-night/style/core
@wooorm/starry-night/style/light
@wooorm/starry-night/style/dark
@wooorm/starry-night/style/both
@wooorm/starry-night/style/colorblind-light
@wooorm/starry-night/style/colorblind-dark
@wooorm/starry-night/style/colorblind
@wooorm/starry-night/style/dimmed-dark
@wooorm/starry-night/style/dimmed
@wooorm/starry-night/style/high-contrast-light
@wooorm/starry-night/style/high-contrast-dark
@wooorm/starry-night/style/high-contrast
@wooorm/starry-night/style/tritanopia-light
@wooorm/starry-night/style/tritanopia-dark
@wooorm/starry-night/style/tritanopia

Languages

Checked grammars are included in common. Everything (that’s needed) is available through all. You can add more grammars as you please.

Each grammar has several associated names and extensions. See source files for which are known and use flagToScope to turn them into scopes.

Some grammars need other grammars to work. You are responsible for loading those, use missingScopes to find which dependencies are needed.

All licenses are permissive and made available in notice. Changes should go to upstream repos and languages.yml in github-linguist.

Types

This package is fully typed with TypeScript. It exports the additional types GetOnigurumaUrl, Grammar, and Options.

Compatibility

This project is compatible with maintained versions of Node.js.

When we cut a new major release, we drop support for unmaintained versions of Node. This means we try to keep the current release line, wooorm@starry-night@^3, compatible with Node.js 16.

You can pass your own TextMate grammars, provided that they work with vscode-textmate, and that they have the added fields extensions, names, and scopeName (see types for the definitions and the grammars in lang/ for examples).

Security

This package is safe.

Related

  • lowlight — similar but based on highlight.js
  • refractor — similar but based on Prism

Contribute

Yes please! See How to Contribute to Open Source.

License

The grammars included in this package are covered by their repositories’ respective licenses, which are permissive (apache-2.0, mit, etc), and made available in notice.

All other files MIT © Titus Wormer


starry-night's People

Contributors

janosh avatar sebastinez avatar wooorm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

starry-night's Issues

esm.sh cannot find `source.css.js`

If I open https://esm.sh/@wooorm/[email protected]/ I see a bunch of imports like:

/* esm.sh - @wooorm/[email protected] */
import "/v132/@wooorm/[email protected]/es2022/source.c.js";
import "/v132/[email protected]/es2022/vscode-oniguruma.mjs";
import "/v132/@wooorm/[email protected]/es2022/source.c++.js";
import "/v132/[email protected]/es2022/vscode-textmate.mjs";
import "/v132/@wooorm/[email protected]/es2022/source.c.platform.js";
import "/v132/@wooorm/[email protected]/es2022/source.cs.js";
import "/v132/@wooorm/[email protected]/es2022/source.css.js";
import "/v132/@wooorm/[email protected]/es2022/source.css.less.js";
import "/v132/@wooorm/[email protected]/es2022/source.css.scss.js";

Opening the first one: https://esm.sh/v132/@wooorm/[email protected]/es2022/source.c.js works fine.

However, if I try to open source.css.js by going to https://esm.sh/v132/@wooorm/[email protected]/es2022/source.css.js I get "File Not Found".

This appears to be the root cause of the following build error I get when building with deno:

Command failed with exit code 1: deno run --allow-all --no-config --import-map=/opt/buildhome/node-deps/node_modules/@netlify/edge-bundler/deno/vendor/import_map.json --quiet /opt/buildhome/node-deps/node_modules/@netlify/edge-bundler/deno/bundle.ts {basePath:/opt/build/repo,destPath:/tmp/edge-650becb62c763a0008a91779/023c2e8e-d4e8-4758-aa47-08d49427fcba.eszip,externals:[],functions:[{name:markdown-preview,path:/opt/build/repo/netlify/edge-functions/markdown-preview.js}],importMapData:{\imports\:{\unist-util-visit-parents\:\https://esm.sh/[email protected]\,\unist-util-visit\:\https://esm.sh/[email protected]\,\unified\:\https://esm.sh/[email protected]\,\remark-smartypants\:\https://esm.sh/[email protected]\,\remark-rehype\:\https://esm.sh/[email protected]\,\remark-parse\:\https://esm.sh/[email protected]\,\remark-breaks\:\https://esm.sh/[email protected]\,\rehype-stringify\:\https://esm.sh/[email protected]\,\rehype-sanitize\:\https://esm.sh/[email protected]\,\rehype-external-links\:\https://esm.sh/[email protected]\,\normalize-url\:\https://esm.sh/[email protected]\,\hast-util-to-string\:\https://esm.sh/[email protected]\,\email-regex-safe\:\https://esm.sh/[email protected]\,\@wooorm/starry-night\:\https://esm.sh/@wooorm/[email protected]\,\@netlify/edge-functions\:\https://edge.netlify.com/v1/index.ts\,\netlify:edge\:\https://edge.netlify.com/v1/index.ts?v=legacy\},\scopes\:{}}} (https://ntl.fyi/exit-code-1)
error: Uncaught (in promise) Error: Module not found https://esm.sh/v132/@wooorm/[email protected]/denonext/source.css.js.
      const ret = new Error(getStringFromWasm0(arg0, arg1));
                  ^
    at __wbg_new_15d3966e9981a196 (file:///opt/buildhome/node-deps/node_modules/@netlify/edge-bundler/deno/vendor/deno.land/x/[email protected]/eszip_wasm.generated.js:417:19)
    at <anonymous> (file:///opt/buildhome/node-deps/node_modules/@netlify/edge-bundler/deno/vendor/deno.land/x/[email protected]/eszip_wasm_bg.wasm:1:93412)
    at <anonymous> (file:///opt/buildhome/node-deps/node_modules/@netlify/edge-bundler/deno/vendor/deno.land/x/[email protected]/eszip_wasm_bg.wasm:1:1499594)
    at <anonymous> (file:///opt/buildhome/node-deps/node_modules/@netlify/edge-bundler/deno/vendor/deno.land/x/[email protected]/eszip_wasm_bg.wasm:1:1938165)
    at __wbg_adapter_40 (file:///opt/buildhome/node-deps/node_modules/@netlify/edge-bundler/deno/vendor/deno.land/x/[email protected]/eszip_wasm.generated.js:231:6)
    at real (file:///opt/buildhome/node-deps/node_modules/@netlify/edge-bundler/deno/vendor/deno.land/x/[email protected]/eszip_wasm.generated.js:215:14)
    at eventLoopTick (ext:core/01_core.js:182:11)

Did source.css.js somehow fail to get included?

[feature] allow wrapping lines in spans

it would be great if individual lines could be wrapped in <span>s, to make adding line highlights and line numbers easier (although i'm not sure if this is intentionally not done to stay close to what prettylights does?)

`flagToGrammar()`?

It'd be useful to be able to grab the grammar object without copying and pasting the flagToScope algorithm. This is useful to get metadata about the grammar, such as to display the language in a code block.

Any interest in adding this method?

Is this package still private on npm?

Running the install instructions:

npm install starry-night

Results in a 404 error from npm.

Have you published the package yet? Or is the package still private?

Missing some punctuation highlight

Hey Titus! Thank you so much for Starry Night. It's exactly what I've been searching for.

I'm currently working on a theme that uses punctuation in a slightly different way than the usual text. However, I'm missing some classes for punctuation in the final HTML.

export { Root } from 'hast'

This is how GitHub parses the code:

<span class="pl-k">export</span>
<span class="pl-kos">{</span>
<span class="pl-smi">Root</span>
<span class="pl-kos">}</span>
<span class="pl-k">from</span>
<span class="pl-s">'hast'</span>

This is how Starry Night parses the code:

<span class="pl-k">export</span>
{
<span class="pl-smi">Root</span>
}
<span class="pl-k">from</span>
<span class="pl-s">
  <span class="pl-pds">'</span>
  hast
  <span class="pl-pds">'</span>
</span>

At first glance, this appears to be a list of rules that haven't yet been implemented:

punctuation.definition.block.js
punctuation.terminator.statement.js
punctuation.definition.parameters.begin.js
punctuation.definition.parameters.end.js
punctuation.accessor.js
punctuation.separator.comma.js
meta.brace.square.js

Did I miss something?

Rewrite in Typescript

I see that you have everything very nicely typed out with JSDoc, but I still wanted to ask if you would be interested in a migration to TypeScript?
Or do you prefer the flexibility of JSDoc over TypeScript?

Rehype/Remark Plugin?

This looks great! I'd love to use it with a static site generator. Do you have any plans for a Remark/Rehype plugin wrapping this or would that be fair game for the community?

Is it possible to expand the list of scopes->classes?

The systemverilog language has a lot of scope options (https://github.com/wooorm/starry-night/blob/main/lang/source.systemverilog.js)

It appears a lot of them are getting flattened when classes are applied. Things like storage types (storage-type-systemverilog) and keywords for marking modules/classes/functions all get grouped under the keyword class. I would be interested in overriding the conversion somehow so I can make more granular classes and have more control over how the code is highlighted.

Is there an official way to do this? If not, am not opposed to forking the repository and locally modifying what I need if it is relatively straightforward to do.

Cannot import grammar from default export

There is no export at the root of starry-night:

import mdx from '@wooorm/starry-night/source.mdx'

show this error:

✘ [ERROR] TS2307: Cannot find module '@wooorm/starry-night/source.json' or its corresponding type declarations. [plugin angular-compiler]

    src/app/components/code/code.service.ts:3:24:
      3 │ import jsonGrammar from '@wooorm/starry-night/source.json'

but in /lang/ are all grammars exported. Unfortunately when using lang I get this error:

✘ [ERROR] Could not resolve "@wooorm/starry-night/lang/source.json"

    src/app/components/code/code.service.ts:3:24:
      3 │ import jsonGrammar from '@wooorm/starry-night/lang/source.json';
        ╵                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  The module "./lang/lang/source.json.js" was not found on the file system:

    node_modules/@wooorm/starry-night/package.json:37:11:
      37 │     "./*": "./lang/*.js",

Use a more narrowing scope

Hello, thanks for the great project!

One question: can we use a more specific scope than only the language?

In my case, I need to highlight only a code snippet for a TS declaration (e.g., string, or JSX.Element).

Then if I use highlight(source, 'source.ts'), I won't have the right result.

I would like something as highlight(source, ['source.ts', 'meta.type.declaration.tsx']), but it isn't supported yet.

Can we have a workaround for it? Or am I missing something?

Thanks,

Highlight differences with GitHub.com

I found out that in JavaScript and TypeScript the name of a const is assigned a span with a pl-c1 class which doesn't match with what GitHub does in their highlighting, where they mark it as a pl-s1.

Current starry-night with source.js.js grammar

Bildschirm­foto 2022-12-13 um 07 27 28

Current GitHub rendering of the same file

Bildschirm­foto 2022-12-13 um 07 27 51

It seems that the name of the pattern is constant.other at least in Javascript.
I'm still trying to get into the codebase but am I right that the mapping between grammar pattern names and css classes are done here? https://github.com/wooorm/starry-night/blob/main/lib/theme.js

In that case I would propose to add, to /lib/theme.js

  "constant.other": "pl-s1"

Also there is some different behaviour in method calls e.g. string.slice() slice is also assigned a pl-c1 instead of pl-en.

The `pl-c1` class is applied too greedy.

I'm seeing that the pl-c1 class is applied in a lot of places, e.g. const variable names, methods calls, numbers, some primitives like undefined, null or this.

I would classify this as a bug, since there is no way to uncolor a const variable name, like the majority of IDEs or code hosting sites do.

const likeThisExample = undefined;

Is there any solution to this? Maybe a change in the grammar, or a change to the lib/theme.js or something else?

References
Bildschirm­foto 2022-12-15 um 11 55 33
Bildschirm­foto 2022-12-15 um 11 55 27
Bildschirm­foto 2022-12-15 um 11 55 18

How to make it work with multiline text?

This is my code for test:

import {toHtml} from 'hast-util-to-html'
import {createStarryNight, common} from '@wooorm/starry-night'
import {readFileSync, writeFileSync} from "fs"

const starryNight = await createStarryNight(common)

const tree = starryNight.highlight(readFileSync("index.js").toString(), 'source.js')

const html = toHtml(tree)

const out = `
<title>StarryNight Playground</title>
<link rel="stylesheet" href="https://esm.sh/@wooorm/starry-night@1/style/both.css">
${html}`

writeFileSync('index.html', out.trim())

console.log('done highlighting index.js')

And the output index.html looks like this:
image

All the newlines and tabs are disappeared, so what should I do to make it work?

Thank you!

vue color parsing error

<template>
  <v-button
      v-for="(item, index) in color"
      :color="item"
      :text="usePascalCase(item) || 'Default'"
      to="/"
  />
</template>

<script lang='ts' setup>
import { ref } from 'vue'
import { usePascalCase } from '#hooks'
const color = ref([
  '',
  'primary',
  'success',
  'warning',
  'danger',
  'info',
  'secondary'
])
</script>

image

import { Fragment, jsx, jsxs } from 'react/jsx-runtime' (all imports are undefined in Next.js app)?

How do I get this working (found this straight in the starry-night README):

import { Fragment, jsx, jsxs } from 'react/jsx-runtime'

For this line:

const reactNode = toJsxRuntime(tree, { Fragment, jsx, jsxs })

Getting this error for each of the imports:

Module '"react/jsx-runtime"' has no exported member 'Fragment'.
Screenshot 2024-03-02 at 8 05 07 PM

This is in a Next.js app, do I need to add something to my webpack config or next.config.js?

I am using React 18.2.0 on Next.js 14.1.2-canary.1 with these:

+ @wooorm/starry-night 3.2.0
+ hast-util-to-jsx-runtime 2.3.0

My full code is currently this:

import { common, createStarryNight } from '@wooorm/starry-night'
import { toJsxRuntime } from 'hast-util-to-jsx-runtime'
import { useEffect, useState } from 'react'
import { Fragment, jsx, jsxs } from 'react/jsx-runtime'

let starryNight: any

export default function Code({
  scope = 'source.ts',
  children,
}: {
  scope: string
  children: string
}) {
  const [sn, setStarryNight] = useState<any>(starryNight)
  const [code, setCode] = useState<React.ReactNode>(children)

  useEffect(() => {
    createStarryNight(common).then(sn => {
      starryNight = sn
      setStarryNight(sn)
    })
  }, [])

  useEffect(() => {
    if (!sn) {
      return
    }

    const tree = sn.highlight(children, scope)
    const reactNode = toJsxRuntime(tree, { Fragment, jsx, jsxs })
    setCode(reactNode)
  }, [sn, scope, children])

  // <pre>
  //   <code className="ts language-ts"></code>
  return code
}

update to vscode-textmate 7.0.3 will cause error

in vscode-textmate 7.0.1, starry-night work well
but update to vscode-textmate 7.0.3, it will cause an error

because it change the type of private variable, now currentRegistry._syncRegistry._grammars is a map object

so this line

const grammar = currentRegistry._syncRegistry._grammars[scope]
will cause an error, it will return undefined

solution:

-const grammar = currentRegistry._syncRegistry._grammars[scope]
+const grammar = currentRegistry._syncRegistry._grammars.get(scope)

but it is not compatible with 7.0.1

Themes: selector used for regexp is also used for string

.pl-sr .pl-cce is used as a selector for both the "string" and "regexp" rules:

.pl-s,
.pl-pds,
.pl-s .pl-pse .pl-s1,
.pl-sr,
.pl-sr .pl-cce,
.pl-sr .pl-sre,
.pl-sr .pl-sra {
color: var(--color-prettylights-syntax-string);
}

starry-night/style/both.css

Lines 128 to 131 in 1b85c76

.pl-sr .pl-cce {
font-weight: bold;
color: var(--color-prettylights-syntax-string-regexp);
}

Because the regexp rule occurs after the string rule, it wins. But I think the selector should be deleted from the string rule to reduce reliance on happenstance like this 🌞.

This is a problem in every theme I've looked at, though I've not looked at them all.

`onig.wasm` missing from Next.js standalone build

I use Next.js for https://tigyog.app, using its standalone build feature. When installing starry-night, all worked well, until I got errors in production saying that onig.wasm was missing. This is due to Next.js's dependency tracing not detecting this dynamic dependency. I was able to work around this ugly issue by adding this to pages/index.ts:

export const config = {
  unstable_includeFiles: ['node_modules/vscode-oniguruma/release/onig.wasm'],
};

I don't know whether to consider this a bug in starry-night or in Next.js, but I expect it will be a common and mysterious error when using starry-night with file tracing, so I'm reporting it here for you. I've also reported the problem here: vercel/next.js#43973

Support locally hosted wasm injection

Hey @wooorm,

Thank you very much for this library, looks and feels really nice. 👍
Wdyt about allowing to host the Oniguruma WASM bin locally, instead of fetching it from the esm.sh CDN.

const wasmBin = await getOniguruma()

I'm thinking either, it could be passed as part of an optional options object into createStarryNight(grammars, options)
Or a new function, e.g. configure that takes a options object, could also be used to pass in grammars 🤔

If you agree, I will gladly create a PR

Can rehypeStarryNight be used in react-markdown?

/**
 * @typedef {import('@wooorm/starry-night').Grammar} Grammar
 * @typedef {import('hast').ElementContent} ElementContent
 * @typedef {import('hast').Root} Root
 */

/**
 * @typedef Options
 *   Configuration (optional)
 * @property {Array<Grammar> | null | undefined} [grammars]
 *   Grammars to support (default: `common`).
 */

import {common, createStarryNight} from '@wooorm/starry-night'
import {toString} from 'hast-util-to-string'
import {visit} from 'unist-util-visit'

/**
 * Highlight code with `starry-night`.
 *
 * @param {Options | null | undefined} [options]
 *   Configuration (optional).
 * @returns
 *   Transform.
 */
export default function rehypeStarryNight(options) {
  const settings = options || {}
  const grammars = settings.grammars || common
  const starryNightPromise = createStarryNight(grammars)
  const prefix = 'language-'

  /**
   * Transform.
   *
   * @param {Root} tree
   *   Tree.
   * @returns {Promise<undefined>}
   *   Nothing.
   */
  return async function (tree) {
    const starryNight = await starryNightPromise

    visit(tree, 'element', function (node, index, parent) {
      if (!parent || index === undefined || node.tagName !== 'pre') {
        return
      }

      const head = node.children[0]

      if (!head || head.type !== 'element' || head.tagName !== 'code') {
        return
      }

      const classes = head.properties.className

      if (!Array.isArray(classes)) return

      const language = classes.find(function (d) {
        return typeof d === 'string' && d.startsWith(prefix)
      })

      if (typeof language !== 'string') return

      const scope = starryNight.flagToScope(language.slice(prefix.length))

      // Maybe warn?
      if (!scope) return

      const fragment = starryNight.highlight(toString(head), scope)
      const children = /** @type {Array<ElementContent>} */ (fragment.children)

      parent.children.splice(index, 1, {
        type: 'element',
        tagName: 'div',
        properties: {
          className: [
            'highlight',
            'highlight-' + scope.replace(/^source\./, '').replace(/\./g, '-')
          ]
        },
        children: [{type: 'element', tagName: 'pre', properties: {}, children}]
      })
    })
  }
}

The code above is copied from the current project. Can this code be used in react-markdown?

<ReactMarkdown
        remarkPlugins={[remarkGfm]}
        rehypePlugins={[rehypeStarryNight]}
        skipHtml={true}
        components={{
          pre: Pre,
        }}
      >
        {markdown}
      </ReactMarkdown>

Why do I encounter this error after using it: runSync finished async. Use run instead

What are the supported common languages?

This might be a dumb question but how do I find the list of 'common' languages supported by pretty-lights. I see the #languages section in the readme, and the link to languages.yml , but I can't figure out which of the listed languages/grammars are included in common and supported by FlagToScope?

thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.