dprint / dprint-swc-ext

Functionality to make swc easier to work with.

dprint-swc-ext

Extensions for swc used in dprint-plugin-typescript and Deno.

What does this do?

1. Adds a SourcePos and SourceRange type to compensate for swc having BytePos(0) as a magical value.
2. Adds many helper methods.

With the view cargo feature enabled:

1. Creates a wrapper AST around swc's AST that stores the node parents.
   • This is similar to a "red tree", but it creates it for every node. It's very fast to create these.
   • Most of this code is code generated.
2. Adds a Node enum type to allow referencing any kind of node.

Helpers

All (SourceRanged trait):

• .start(&self) -> SourcePos
• .end(&self) -> SourcePos
• .text_fast(&self, root_node: &dyn SourceTextProvider) -> &'a str -- Doesn't require going up the tree to the root node
• .start_line_fast(&self, root_node: &dyn SourceTextInfoProvider) -> usize
• .end_line_fast(&self, root_node: &dyn SourceTextInfoProvider) -> usize
• .start_column_fast(&self, root_node: &dyn SourceTextInfoProvider) -> usize
• .end_column_fast(&self, root_node: &dyn SourceTextInfoProvider) -> usize
• .width_fast(&self, root_node: &dyn SourceTextInfoProvider) -> usize
• .tokens_fast(&self, root_node: &dyn RootNode) -> &'a [TokenAndSpan]
• .leading_comments_fast(&self, root_node: &dyn RootNode) -> CommentsIterator<'a>
• .trailing_comments_fast(&self, root_node: &dyn RootNode) -> CommentsIterator<'a>
• .previous_token_fast(&self, root_node: &dyn RootNode) -> Option<&TokenAndSpan>
• .next_token_fast(&self, root_node: &dyn RootNode) -> Option<&TokenAndSpan>
• .previous_tokens_fast(&self, root_node: &dyn RootNode) -> &'a [TokenAndSpan]
• .next_tokens_fast(&self, root_node: &dyn RootNode) -> &'a [TokenAndSpan]

Node/Enum Node/Nodes (view cargo feature only):

• .module(&self) -> &'a Module - Gets the root node if the view was created from a Module. Otherwise panics.
• .script(&self) -> &'a Script - Gets the root node if the view was created from a Script. Otherwise panics.
• .program(&self) -> Program<'a> - Gets the root node whether it be a Module or a Script.
• .parent(&self) -> Option<Node<'a>>
• .children(&self) -> Vec<Node<'a>>
• .child_index(&self) -> usize
• .ancestors(&self) -> AncestorsIterator<'a>
• .previous_sibling(&self) -> Option<Node<'a>>
• .next_sibling(&self) -> Option<Node<'a>>
• .previous_siblings(&self) -> Vec<Node<'a>>
• .next_siblings(&self) -> Vec<Node<'a>>
• .text(&self) -> &str - Slightly slower than .text_fast(module) because it requires going up the tree to get the root node
• .start_line(&self) -> usize
• .end_line(&self) -> usize
• .start_column(&self) -> usize
• .end_column(&self) -> usize
• .width(&self) -> usize
• .tokens(&self) -> &[TokenAndSpan] - All the descendant tokens within the span of the node.
• .children_with_tokens(&self) -> Vec<NodeOrToken<'a>> - Gets the children with the tokens found between the children
• .children_with_tokens_fast(&self, root_node: &dyn RootNode) -> Vec<NodeOrToken<'a>>
• .leading_comments(&self) -> CommentsIterator<'a>
• .trailing_comments(&self) -> CommentsIterator<'a>
• .kind(&self) -> NodeKind - Gets the "node kind" enum variant associated with the node (ex. NodeKind::ClassDecl).
• .previous_token(&self) -> Option<&TokenAndSpan>
• .next_token(&self) -> Option<&TokenAndSpan>
• .previous_tokens(&self) -> &'a [TokenAndSpan]
• .next_tokens(&self) -> &'a [TokenAndSpan]

Node/Enum Node (view cargo feature only):

• .to::<NodeType>(&self) -> Option<&NodeType>
• .expect::<NodeType>(&self) -> &NodeType
• .is::<NodeType>(&self) -> bool

TokenAndSpan:

• .token_index(&self, root_node: &dyn RootNode) -> usize - Gets the token index of the specified module.

Root Node (Program/Module/Script):

• token_at_index(&self, index: &usize) - Option<&TokenAndSpan>

View Construction Functions

• with_ast_view - Creates a view from an swc Program (either Module or Script)
• with_ast_view_for_module - Creates a view from an swc Module
• with_ast_view_for_script - Creates a view from an swc Script

TODO

• Right now this only works if analyzing one file at a time. It would be good to improve the API to accept a large collection of source files (should be easy).
• Unit tests

view - Example

Given the following parsed input code:

class MyClass { prop: string; myMethod() {}}

Code can be written like so:

// setup swc (parse an AST and optionally get the comments and tokens)
let text_info = SourceTextInfo::new(...);
let program: swc_ecmascript::ast::Program = ...;
let comments: swc_common::comments::SingleThreadedComments = ...;
let tokens: Vec<TokenAndSpan> = ...;

// setup for creating a view
let program_info = ProgramInfo {
  program: &program,
  text_info: Some(&text_info),
  // optionally provide the comments for comment related methods
  comments: Some(&comments)
  // optionally provide the tokens for token related methods
  tokens: Some(&tokens),
};

// now create and use the view
dprint_swc_ecma_ast_view::with_ast_view(program_info, |program| {
  let class = program.children()[0].expect::<ClassDecl>().class;
  println!("{:?}", class.text());

  for child in class.children() {
    println!("---------");
    println!("Child: {:?}", child.text());
    println!("Parent: {:?}", child.parent().unwrap().text());
    if let Some(prev_sibling) = child.prev_sibling() {
      println!("Previous sibling: {:?}", prev_sibling.text());
    }
    if let Some(next_sibling) = child.next_sibling() {
      println!("Next sibling: {:?}", next_sibling.text());
    }
  }
});

Outputs:

"class MyClass { prop: string; myMethod() {}}"
---------
Child: "prop: string;"
Parent: "class MyClass { prop: string; myMethod() {}}"
Next sibling: "myMethod() {}"
---------
Child: "myMethod() {}"
Parent: "class MyClass { prop: string; myMethod() {}}"
Previous sibling: "prop: string;"

dprint-swc-ext's Issues

Ability to release from GitHub action

As title.

Example does not work

I tried following the example program below, but got stuck with passing the comments into a ProgramInfo struct.

// setup swc (parse an AST and optionally get the comments and tokens)
let text_info = SourceTextInfo::new(...);
let program: swc_ecmascript::ast::Program = ...;
let comments: swc_common::comments::SingleThreadedComments = ...;
let tokens: Vec<TokenAndSpan> = ...;

// setup for creating a view
let program_info = ProgramInfo {
  program: &program,
  text_info: Some(&text_info),
  // optionally provide the comments for comment related methods
  comments: Some(&comments)
  // optionally provide the tokens for token related methods
  tokens: Some(&tokens),
};

// now create and use the view
dprint_swc_ecma_ast_view::with_ast_view(program_info, |program| {
  let class = program.children()[0].expect::<ClassDecl>().class;
  println!("{:?}", class.text());

  for child in class.children() {
    println!("---------");
    println!("Child: {:?}", child.text());
    println!("Parent: {:?}", child.parent().unwrap().text());
    if let Some(prev_sibling) = child.prev_sibling() {
      println!("Previous sibling: {:?}", prev_sibling.text());
    }
    if let Some(next_sibling) = child.next_sibling() {
      println!("Next sibling: {:?}", next_sibling.text());
    }
  }
});

Here is my current code:

let text = read_to_string(file).expect("Failed to read file");
let text_info = SourceTextInfo::new(text.into());
let parsed_source = parse_module(ParseParams {
    specifier: "file:///my_file.js".to_string(),
    media_type: JavaScript,
    text_info,
    capture_tokens: true,
    maybe_syntax: None,
    scope_analysis: false,
}).expect("Should parse");

let comments = parsed_source.comments();
// let program: swc_ecmascript::ast::Program = Program::Module(parsed_source.module().to_owned());
let tokens = parsed_source.tokens();

let program_ref: dprint_swc_ext::view::ProgramRef = dprint_swc_ext::view::ProgramRef::Module(parsed_source.module());

let new_comments = comments.as_single_threaded();
let program_info = ProgramInfo {
    program: program_ref,
    text_info: Some(&text_info),
    comments: Some(&new_comments), // <-- here
    tokens: Some(&tokens)
};

It throws an error "Mismatched Types"

error[E0308]: mismatched types
  --> src/parsers/javascript/parser.rs:39:28
   |
39 |             comments: Some(&new_comments),
   |                            ^^^^^^^^^^^^^ expected struct `dprint_swc_ext::view::Comments`, found `&SingleThreadedComments`

I tried messing around with .into(), .to_owned(), but I can't seem to figure out how to convert a single threaded comment into a "Comments" object. Any help is welcome, I appreciate the work you do on this repo.

The format is compatible with Espree

swc-project/swc#246

The format is compatible with Espree? Can I use it as custom eslint parser?

Feature Timeline

Do you have a timeline for when this feature will be implemented?

Right now this only works if analyzing one file at a time. It would be good to improve the API to accept a large collection of source files (should be easy).

Generate script should pull swc version from cargo.toml file

Right now the generate.ps1 script has a hardcoded version. This should instead pull the version from cargo.toml in the rs-lib file (maybe via cargo show).

Remove usage of `swc_common::SourceFile` and `SingleThreadedComments`

We should get it to work without these as they are not Sync and can't be stored in deno's LSP.

Use source of swc_ecma_ast from cargo instead of submodule for code generation

As title.

CI should ensure generated code in repo is up to date

Will prevent publishing versions without the latest code generation.

Note: I want the code generation output to be in the repo in order to always code review it.

Refactor the way of constructing graph to avoid undefined behavior

At the moment, the generated code has lots of codes that might lead to undefined behavior: https://github.com/dprint/dprint-swc-ecma-ast-view/blob/dfc9347015603495579eec592b27ca301abb093e/rs-lib/src/generated.rs#L7097-L7111

Here MaybeUninit::uninit().assume_init() is called, but Rust doesn't allow us to invoke assume_init() on values that are not properly initialized in memory. It looks like the code works fine for now, but there's no guarantee that it will keep working.

So I'd like to propose replacing these codes to avoid undefined behavior. I think this refactoring is not so easy or simple but it's worth it.

Here are my thoughts on how to refactor:

• make all fields of Node structs (such as BindingIdent) private and create getter methods for them
• change type of fields that currently use MaybeUninit::uninit().assume_init() (such as id of BindingIdent) to Cell<Option<..>>. They become None only when being constructed. Once the AST view has been completely created, they should be Some. Therefore, in the getter methods, we can safely call .unwrap()
• rewrite the construction of the view

As a whole, BindingIdent could be refactored into the following:

// Make all fields private
pub struct BindingIdent<'a> {
  parent: Node<'a>,
  inner: &'a swc_ast::BindingIdent,
  id: Cell<Option<&'a Ident<'a>>>, // wrap in Cell
  type_ann: Cell<Option<&'a TsTypeAnn<'a>>>, // wrap in Cell
}

// Instead, expose getters for all fields
impl<'a> BindingIdent<'a> {
  pub fn parent(&self) -> Node<'a> {
    self.parent
  }

  pub fn inner(&self) -> &'a swc_ast::BindingIdent {
    self.inner
  }

  pub fn id(&self) -> &'a Ident<'a> {
    self.id.get().unwrap() // when the library users call it, this `unwrap` is totally safe
  }

  pub fn type_ann(&self) -> Option<&'a TsTypeAnn<'a>> {
    self.type_ann.get()
  }
}

fn get_view_for_binding_ident<'a>(inner: &'a swc_ast::BindingIdent, parent: Node<'a>, bump: &'a Bump) -> &'a BindingIdent<'a> {
  let node = bump.alloc(BindingIdent {
    inner,
    parent,
    id: Cell::new(None), // wrap in Cell
    type_ann: Cell::new(None), // wrap in Cell
  });
  let parent: Node<'a> = (&*node).into();
  node.id.set(Some(get_view_for_ident(&inner.id, parent.clone(), bump)));
  node.type_ann.set(match &inner.type_ann {
    Some(value) => Some(get_view_for_ts_type_ann(value, parent, bump)),
    None => None,
  });
  node
}

As a side note: In the above example, type_ann is not required to wrap in Cell as it is not related to MaybeUninit. But it's preferable because the assignment node.type_ann = ... requires mutable reference to node while parent holds immutable reference to node. The Rust compiler is supposed to warn it, but somehow it compiles (probably due to std::mem::transmute in From trait impl.) For immutable reference and mutable reference not to exist at the same time, Cell is helpful.

Let me know what you think about this refactoring :)

Serialization - Implement `Serializer` rather than cloning into `SerializableX`

As title. This would probably be a performance improvement and build on #13.

Add unit tests

It would be useful to add unit tests for the methods this crate adds.

TypeScript generated code

Building on #13, we should generate TypeScript types and code for deserializing a source file to these types.

1. Generate TS classes of each node that extends from a base Node class (should be possible to do node instanceof ClassDecl)
2. Generate code for getting a node's children.
3. Generate code that will take a deserialized JS object, set the prototype of each object to the appropriate type, set the node's parent (in the case of source file, undefined), then traverse all the child nodes doing the same.

cc @magurotuna, I wouldn't mind working on this, but if you would like to do this work let me know! For me it might take a few weeks.

Is is possible to let SourceFileInfo have swc's `Program` rather than `Module`?

I've been re-implementing a few of deno_lint's rules using this crate.
deno_lint uses swc_ecmascript::ast::Program as an entry point of linting, therefore it would be nice if this crate also took Program instead of Module. That is, I'd be happy if SourceFileInfo was altered to:

pub struct SourceFileInfo<'a> {
  pub module: &'a swc_ecmascript::ast::Program, // altered!
  pub source_file: Option<&'a swc_common::SourceFile>,
  pub tokens: Option<&'a Vec<TokenAndSpan>>,
  pub comments: Option<&'a SingleThreadedComments>,
}

and the signature of with_ast_view to:

pub fn with_ast_view<'a, T>(
    source_file_info: SourceFileInfo<'_>, 
    with_view: impl FnOnce(&'a Program<'a>) -> T // altered!
) -> T

From my taking a look at the code of this crate and how the underlying code generation work, it seems like the above alternation is achievable (although it looks a bit complicated to me :-|)