harpocrates / language-rust Goto Github PK
View Code? Open in Web Editor NEWParser and pretty-printer for the Rust language
Home Page: https://hackage.haskell.org/package/language-rust
License: BSD 3-Clause "New" or "Revised" License
Parser and pretty-printer for the Rust language
Home Page: https://hackage.haskell.org/package/language-rust
License: BSD 3-Clause "New" or "Revised" License
It may be the case that identifiers or names should be interned, or use Text
, but in any case the current situation should change.
The rust AST has special forms for inline and global assembly. It might be worth adding explicit support for these. This is not urgent since both inline and global assembly can already be represented via regular macros.
It would nonetheless be nice to not have to muck around with token trees in order to generate/read these special macros.
The main challenge is in parsing.
Range expressions are a queer operator: it looks like precedence on the left is not the same as precedence on the right. It also has a prefix and postfix form. Right now, it is broken.
rustc-tests
The project already has a dependency on pretty
for pretty-printing. In Text.PrettyPrint.Annotated.HughesPJ
, there is
data Span a = Span
{ spanStart :: !Int
, spanLength :: !Int
, spanAnnotation :: a
} deriving (Show,Eq)
Which might be a good candidate as the output tag type for parsing. Some thoughts on this:
One uniform Span
data type for both pretty-printing and parsing is slick, especially a reused one.
I'm not sure I see any use for spanAnnotation
with parsing - maybe a means of plugging in more helpful information about position. For the latter, one would probably define
data Pos = Pos { row :: !Int, col :: !Int } deriving (Eq, Ord)
type ParseSpan = Span Pos
then parsing functions would produce things of type Expr ParseSpan
, Ty ParseSpan
, etc.
All changes would be in the Parser
module, just returning this type.
There is currently an AST representation of asm!
macros (which can be appropriately pretty-printed). It may be worth parsing asm expression macros directly into this representation. Currently, asm!
macros are treated just like any other macro.
Currently, language-rust
imposes upper version bounds of <0.6
on transformers
:
language-rust/language-rust.cabal
Line 89 in 9d509c4
This prevents it from building with transformers-0.6.*
. As this is the version of transformers
that is bundled with GHC 9.6, this makes it difficult to build language-rust
with GHC 9.6.
This appears to be a more modern (yay) more experimental (nay) iteration on Wadler's wl-pprint
. I like it because it lets me print blocks nicely, by giving me access to the underlying constructors of Doc a
(but I must ensure not to violate any invariants).
{-# LANGUAGE OverloadedStrings #-}
import Text.PrettyPrint.Annotated.WL
-- | Asserts a 'Doc a' cannot render on multiple lines.
oneLine :: Doc a -> Bool
oneLine (FlatAlt d _) = oneLine d
oneLine (Cat a b) = oneLine a && oneLine b
oneLine (Union a b) = oneLine a && oneLine b
oneLine (Annotate _ d) = oneLine d
oneLine Line = False
oneLine _ = True
-- | Make a curly-brace delimited block. When possible, permit fitting everything on one line
block :: Doc a -> Doc a
block b | oneLine b = hsep ["{", b, "}"] `Union` vsep [ "{", indent 2 b, "}" ]
| otherwise = vsep [ "{", indent 2 b, "}" ]
This should let me print curly-brace delimited macros nicely - if they are short and clearly meant to go on one line, I fit them on one line. Otherwise, I play it safe and use the multiline approach.
Pretty printing is nearly complete. The following things still need to be addressed:
Pretty
and Pretty.Internal
modulesShort of any drastic new enhancements, this issue will be the tracking point for everything pretty-printing related.
Add test suites for performance and allocations, probably using criterion
and weigh
.
Consider adding benchmarks to check memory-use and speed of the pretty-printer.
Lex and parse multiline string literals.
Attributes are currently sometimes not parsed, not printed, and not tested. The following needs to happen:
sample-sources
specifically for this as part of rustc-tests
(since attributes are constantly evolving).Diff
to support attributesrustc-tests
to also check that pretty-print and re-parse is a no-op; this will check that attributes are properly pretty-printed.This has been merged into Rust as of rust-lang/rust#35954.
See this PR . This changes token trees a fair bit.
The difficulty here is figuring out how to do this in a backwards compatible fashion.
Now that parsing is almost finished, it is worth writing down a list of things not done:
CrateConfig
, then mods)MacStmt
and MacStmtStyle
)MacroDef
, KleeneOp
, Nonterminals
)asm!
expressionsSpan
(don't just shove in mempty
whenever withSpan
doesn't immediately work)Some things are missing before this can be properly made a package
stack.yaml
(and make sure everything is Stackage happy)Some things that would be nice to support
CPP
hack)The following sorts of things should parse:
trait Foo {
type Bar<'a>;
type Bar<'a, 'b>;
type Bar<'a, 'b,>;
type Bar<'a, 'b, T>;
type Bar<'a, 'b, T, U>;
type Bar<'a, 'b, T, U,>;
type Bar<'a, 'b, T: Debug, U,>;
type Bar<'a, 'b, T: Debug, U,>: Debug;
type Bar<'a, 'b, T: Debug, U,>: Deref<Target = T> + Into<U>;
type Bar<'a, 'b, T: Debug, U,> where T: Deref<Target = U>, U: Into<T>;
type Bar<'a, 'b, T: Debug, U,>: Deref<Target = T> + Into<U>
where T: Deref<Target = U>, U: Into<T>;
}
Right now, due to a happy bug, the expression grammar is difficult to refactor (moving productions around changes the semantics!!). Once this is resolved, several things need to happy:
union
, but that are not union
s (e.g. union.tag + 1
- union
is a "weak" keyword")Looking at language-c
's Parser.y, I think their approach to keeping track of spans is better than what I am currently doing (using Span
as a Monad
). It should also be
On that note, if they use snoc
instead of an extra production rule then reverse, it might be worth doing what they are doing (and cutting down on the number of generic production rules whose types have to be inferred). Maybe just have star :: Parser [a]
, plus :: Parser (NonEmpty a)
, and optional :: Parser (Maybe a)
...
Every generic has a where clause, but it seems rather than generics and where clauses just come together (not that they should be one within the other). Perhaps we should move the where clause out of the Generic a
data type.
The following test case (taken from rustc
's testsuite) doesn't parse:
// build-pass (FIXME(62277): could be check-pass?)
#![allow(bare_trait_objects)]
type A = Box<(Fn(u8) -> u8) + 'static + Send + Sync>; // OK (but see #39318)
fn main() {}
Besides being already controversial (check out the issue referenced) and one hell of an edge case, bare trait objects are on their way out.
I see you've updated the library for ghc-8.8. Would you mind doing a new release to Hackage?
Rust supports two syntaxes for mod
- one for declaring inline a module and another for referencing the contents of another file. We don't currently try to go follow those files (since that requires IO
and parsing is otherwise pure).
IO
based parse that chases down mod ...;
items to expand themmod ...;
forms to make the difference tests passThis line in the lexer:
language-rust/src/Language/Rust/Parser/Lexer.x
Line 1048 in 9d509c4
Implements the "not immediately followed" part in the Rust lexical syntax spec: https://doc.rust-lang.org/reference/tokens.html#floating-point-literals
The code uses _a-zA-Z
as the beginning of an identifier, but according to Rust spec any XID_Start
is a valid identifier first character, so that part of the lexer should be updated with _
and @xid_start
. (not sure what the right syntax for this is in alex)
(I don't have Haskell toolchain installed otherwise I would submit a PR)
Another day, another new construct which has a "weak" keyword catch
. Tracking issue.
Since catch
is a "weak" keyword there are problems around using it as the scrutinee of if/while/etc. just like structs.
None of this is actually in any way decided. Relevant internals thread.
Someone already added support for this upstream in Happy. When the next version of Happy is released, we should
%errorhandlertype explist
and modify the %error
to show the tokensThe following should parse, but it doesn't currently:
pub unsafe extern "C" fn test(_: i32, ap: ...) { }
pub unsafe extern "C" fn test_valist_forward(n: u64, mut ap: ...) -> f64 {
rust_valist_interesting_average(n, ap.as_va_list())
}
It is expected that the following will be valid macro definitions:
macro foo($a: ident) => {
return $a + 1;
}
pub macro foo { ... }
To make this more GitHub and contribution friendly, add
Besides the obvious quasi quotation, I want to capture variables with SubstNt
tokens in Exp
and bind variables with MatchNt
in Pat
. The following interaction should work:
>>> import Language.Rust.Quote
>>> :set -XQuasiQuotes
>>> let one = [lit| 1i32 |]
>>> [expr| |x: i32| -> $ret:ty $body:expr |] = [expr| |x: i32| -> i32 { x + $one } |]
ret :: Ty Span
body :: Expr Span
>>> import Language.Rust.Pretty
>>> pretty ret
i32
>>> pretty body
{ x + 1 }
Here is my current approach. It is terrible. I can't believe I have been reduced to this, but I've been thinking non-stop about this for 2 weeks now and have come up with nothing better.
Q (String -> Maybe Type)
function (instead of String -> Q Type
) and use that to make the swapping function. Alternately, lex twice and extract all of the interesting tokens, look them up, and pass them in as a function in the second lexing pass.NonTerminal
s containing error <variable-name>
- but of the right type!dataToPatQ
and dataToExpQ
, whenever something is of the right type, check if it is defined. If so, do nothing, otherwise, use the error message (ewwwwww!) to lookup variable or make an identifier pattern.Maybe there is. Here are some unorganized thoughts on this topic:
P
monad over an arbitrary monad (in particular Q
and Identity
, allowing me to swap tokens in this monad) - this is already done in another branch, although I have misgivings about performance. I had to disable the monomorphism restriction to get the generated files to compile, and I have a feeling that polymorphic lexer/parser will take a hit in performance.The "good"TM solution is probably an extensible AST with type families, but given that the AST is already ~1kloc, adding a type family for every constructor may get... a bit too big. Plus this also looks ugly to the end user.
Also, the solution described initially isn't so bad. It does not change the reliability of regular parsing - it only risks failing in the case of quasi quotes (which will be marked experimental and non-portable, and will only cause compile-time headaches).
Here are some approaches to testing correctness of parsing and pretty printing
-Z
option for outputting a JSON of the AST, parse that JSON into our AST and diff them against the AST we get from regular parsing.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.