jhvst / scie

基于正则表达式的通用语言转换器。An interpreter for grammar files as defined by TextMate with Rust

Scie

Scie is a research about how to build simple code identify engine for different languages.

goal: build a better code figure engine for code refactoring.

• scie-cli. cli part of Scie.
• scie-bingen. generate languages bin-data.
• scie-detector. detector for different frameworks & languages.
• scie-grammar. A library that helps tokenize text using Text Mate grammars.
• scie-infra. common infrastructure support, like fs
• scie-onig. Wrapper Rust Oniguruma FFI api based on rust-onig
• scie-model. common model of VSCode models & Miao Model.
• scie-scanner. Rewrite of VS Code Oniguruma

Usage

Scie CLI

scie-cli 0.1
Phodal HUANG<[email protected]>

USAGE:
    scie-cli [FLAGS] [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -v, --verbose
    -V, --version    Prints version information

OPTIONS:
    -c, --config <config>    [default: default.conf]
    -p, --path <path>        [default: .]

DevSetup

1. git clone

git clone https://github.com/phodal/scie/

2. run

cargo run scie

install just

curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to DEST

run tests

just tests

Guideline

major issues:

• performance
  • rule in Grammar seed in rule_container.rs
  • UTF 8 to UTF 16 in UtfString
  • normal issue
• unstable
  • Random test failure on OnigScanner.
  • GC issues on OnigScanner.
    • GC issue seems resolved with Jemalloc.
    • Signal 6 (SIGABRT) = SIGABRT is commonly used by libc and other libraries to abort the program in case of critical errors. For example, glibc sends an SIGABRT in case of a detected double-free or other heap corruptions.
    • maybe UTF8 encoding issue

Todo:

• replace with fancy-regex for pure Rust impl
• process todo
  • back references
  • multiple languages
• rewrite VSCode-textmate with Rust
  • language for testing
  • support others language
• benchmark
  • fast than VSCode version
• multiple languages one project support
• analyser
  • line counts
  • keywords map

Documents

refs

• Oniguruma Regular Expressions
• https://github.com/atom/node-oniguruma
• Sloc Cloc and Code - What happened on the way to faster Cloc
• ripgrep for todo?

License

scie-grammar based on vscode-textmate with MIT LICENSE see in scie-grammar/src/scanner/LICENSE

onigvs based on rust-onig

@ 2020 A Phodal Huang's Idea. This code is distributed under the MPL license. See LICENSE in this directory.