haskell / alex Goto Github PK
View Code? Open in Web Editor NEWA lexical analyser generator for Haskell
Home Page: https://hackage.haskell.org/package/alex
License: BSD 3-Clause "New" or "Revised" License
A lexical analyser generator for Haskell
Home Page: https://hackage.haskell.org/package/alex
License: BSD 3-Clause "New" or "Revised" License
The failure message is:
templates/GenericTemplate.hs:77:4:
Illegal bang-pattern (use -XBangPatterns):
! i
Tried with ghc 7.6.1 / alex 3.0.2.
Citing from http://hydra.cryp.to/build/1389306/nixlog/1/raw:
Building alex-3.1.5...
Preprocessing executable 'alex' for alex-3.1.5...
Setup: The program 'happy' is required but it could not be found
Apparently a build-tools: happy
attribute is missing from the Cabal file.
Alex-3.0.2 is failing with "Prelude.Enum.Char.pred: bad argument" when I have a [\x00-\xff] or [^'\"\0\n] range in my *.x file:
@inComment = ([^*] | $white)+ | ([*]+ ([\x00-\xff] # [/]))
@Instr = @hexEscape | @octEscape | @charescape | [^'\"\0\n]
The above worked in alex-2.3.1. I can work around this:
@inComment = ([^*] | $white)+ | ([*]+ [^\/])
@Instr = @hexEscape | @octEscape | @charescape | [^'\"\n] | [\0]
I would guess your character range code uses pred on the null Char and dies.
in a number of application domains, I need to deal with handling streaming inputs in an incremental fashion, and having a streaming lexer / tokenization layer helps immensely with writing the layers on top.
If adding such capabilities to Alex are viable, i'd be very interested in trying to help add them. (rather than having to reinvent a lot of the tooling that alex provides)
would this be a feature you'd be open to having added? @simonmar ?
Maybe this has already been solved but not yet found its way to hackage:
When compiling modules created by Alex 3.1.4 with GHC 7.10.2 the following warning comes up:
templates/wrappers.hs:7:1: Warning:
The import of ‘Control.Applicative’ is redundant
except perhaps to import instances from ‘Control.Applicative’
To import instances alone, use: import Control.Applicative()
I'm using Alex 3.1.4 - the newest I could find on hackage:
$ alex --version
Alex version 3.1.4, (c) 2003 Chris Dornan and Simon Marlow
I think this is one of the topics covered in https://ghc.haskell.org/trac/ghc/wiki/Migration/7.10
thx, nikomi
I recently upgraded ghc and reinstalled yesod-dsl.
I got the following error:
[ 2 of 27] Compiling Lexer ( dist/build/yesod-dsl/yesod-dsl-tmp/Lexer.hs, dist/build/yesod-dsl/yesod-dsl-tmp/Lexer.o )
dist/build/yesod-dsl/yesod-dsl-tmp/Lexer.hs:565:34:
Couldn't match expected type ‘Bool’ with actual type ‘Int#’
In the first argument of ‘(&&)’, namely ‘(offset >=# 0#)’
In the expression: (offset >=# 0#) && (check ==# ord_c)
In the expression:
if (offset >=# 0#) && (check ==# ord_c) then
alexIndexInt16OffAddr alex_table offset
else
alexIndexInt16OffAddr alex_deflt s
dist/build/yesod-dsl/yesod-dsl-tmp/Lexer.hs:565:53:
Couldn't match expected type ‘Bool’ with actual type ‘Int#’
In the second argument of ‘(&&)’, namely ‘(check ==# ord_c)’
In the expression: (offset >=# 0#) && (check ==# ord_c)
In the expression:
if (offset >=# 0#) && (check ==# ord_c) then
alexIndexInt16OffAddr alex_table offset
else
alexIndexInt16OffAddr alex_deflt s
This error happens because ghc 7.6.3 used ghc-prim 0.3.0.0 which defined >=# as Int# -> Int# -> Bool
ghc 7.8.2 used ghc-prim 0.3.1.0 which defined >=# as Int# -> Int# -> Int#
Relevant lines:
https://github.com/simonmar/alex/blob/master/src/Scan.hs#L349
https://github.com/simonmar/alex/blob/master/templates/GenericTemplate.hs#L170
I've noticed that examples/Makefile
and templates/Makefile
no longer work:
naesten@hydrogen:~/hacking/haskell/alex% make -C examples
make: Entering directory `/home/naesten/hacking/haskell/alex/examples'
Makefile:2: ../mk/boilerplate.mk: No such file or directory
Makefile:31: ../mk/target.mk: No such file or directory
make: *** No rule to make target `../mk/target.mk'. Stop.
make: Leaving directory `/home/naesten/hacking/haskell/alex/examples'
naesten@hydrogen:~/hacking/haskell/alex% make -C templates
make: Entering directory `/home/naesten/hacking/haskell/alex/templates'
Makefile:2: ../mk/boilerplate.mk: No such file or directory
Makefile:67: ../mk/target.mk: No such file or directory
make: *** No rule to make target `../mk/target.mk'. Stop.
make: Leaving directory `/home/naesten/hacking/haskell/alex/templates'
You should probably just drop templates/Makefile
, since it seems to have been replaced by some code in Setup.lhs
anyway, but examples/Makefile
should probably be fixed, for lack of any other obvious way to build the examples...
It seems like one must handle AlexLastAcc like this with alex <3.2
AlexLastAcc k input len -> ... AlexToken input len k ...
and like this with alex >= 3.2
AlexLastAcc k input len -> ... AlexToken input len (alex_actions ! k) ...
Is there a CPP define for alex version so I can at least put this line under #ifdef?
Or maybe is there another way to write it so it works with both versions like a typeclass with instances for Int and a type of alex_action?
Problem is that stackage LTS provides 3.1.7 and stackage nightly provides 3.2.1 and I'd like very much for yi to compile with both.
Can we please have an option for case-insensitive matching?
197317b added the strict-bytestring
wrapper, but the User Guide doesn't mention it.
In most wrappers, AlexInput
is a tuple, whereas in strict-bytestring
(and also in basic-bytestring
, since #51) it is a record:
data AlexInput = AlexInput { alexChar :: {-# UNPACK #-} !Char,
alexStr :: !ByteString.ByteString,
alexBytePos :: {-# UNPACK #-} !Int64}
When compiling Alex lexers with GHC 7.8.2 the following message is emitted:
‘Alex’ is an instance of Monad but not Applicative - this will become an
error in GHC 7.10, under the Applicative-Monad Proposal.
Could you please make 'Alex' an instance of Applicative to remove this?
thx, nik
The hackage 'cabal' / stackage build looks bad/ broken with respect to 'AlexWrapper-monad-bytestring'
These files don't contain the 'instance Applicative'
~/.stack/snapshots/x86_64-linux/lts-3.14/7.10.2/share/x86_64-linux-ghc-7.10.2/alex-3.1.4/AlexWrapper-monad-bytestring
~/.cabal/share/x86_64-linux-ghc-7.8.4/alex-3.1.4/AlexWrapper-monad-bytestring
I tried a local build of alex with the latest source code from github and it does contain the Applicative instance. So I am not sure where the issue might be.
For me, the unicode test cases always fail
$ make -k -C tests clean all
rm -f *.n.hs *.g.hs *.info *.hi *.o *.bin *.exe
../dist/build/alex/alex --template=.. unicode.x -o unicode.n.hs
unicode.x:19:6: lexical error at '\915'
make: *** [unicode.n.hs] Error 1
../dist/build/alex/alex --template=.. -g unicode.x -o unicode.g.hs
unicode.x:19:6: lexical error at '\915'
make: *** [unicode.g.hs] Error 1
This is using the versions of the pre-processed .x/.hs files from the repo, and the same effect if I regenerate them with the latest alex.
The line it's complaining about is:
[AΓ] { character }
that second char there, 'Γ'.
I think I've hit a bug with using right_ctx regular expression. The following code works in alex 2.x, but fails with 3.0.2:
{
module AlexBug where
}
%wrapper "basic"
@opstr = [^:]+
$preop = \,
tokens :-
$preop^ @opstr / \: { TokenOperation }
{
data Token = TokenOperation String
deriving (Read, Show, Eq, Ord)
}
Error message:
$ cabal-dev install
Resolving dependencies...
Configuring alexbug-0.1.0.0...
Building alexbug-0.1.0.0...
Preprocessing executable 'alexbug' for alexbug-0.1.0.0...
[1 of 2] Compiling AlexBug ( dist/build/alexbug/alexbug-tmp/AlexBug.hs, dist/build/alexbug/alexbug-tmp/AlexBug.o )
templates/wrappers.hs:428:119:
Couldn't match expected type `AlexInput'
with actual type `(t0, t1)'
Expected type: t2 -> AlexInput -> Int -> AlexInput -> Bool
Actual type: t2 -> (t0, t1) -> t3 -> t4 -> t5
In the return type of a call of `alexPrevCharMatches'
In the first argument of `alexAndPred', namely
`alexPrevCharMatches (\ c -> c >= ',' && c <= ',' || False)'
Failed to install alexbug-0.1.0.0
cabal: Error: some packages failed to install:
alexbug-0.1.0.0 failed during the building phase. The exception was:
ExitFailure 1
Am I doing something wrong? If so, what should my alex definition look like? Thanks!
Given this alex file, Main.x:
{
{-# OPTIONS -w #-}
module Main where
}
%wrapper "basic"
tokens :-
$white+ ;
λ { const TokenLambda }
{
-- The token type:
data Token = TokenLambda
deriving Show
main :: IO ()
main = do
i <- readFile "input"
print (alexScanTokens i)
}
And a single unicode character in the the input
file:
$ cat input
λ
[dagit@mango:~/Documents/Repos/testing/alex-unicode]
$ alex Main.x
[dagit@mango:~/Documents/Repos/testing/alex-unicode]
$ ghc --make Main.hs
[1 of 1] Compiling Main ( Main.hs, Main.o )
Linking Main.exe ...
[dagit@mango:~/Documents/Repos/testing/alex-unicode]
$ ./Main.exe
Main.exe: lexical error
I tried setting the encoding of the file to utf8 before reading it but that resulted in a lexical error looking for EOF.
This happens on Windows 7 64 bit, ghc 7.6.1 (32bit build), alex 3.0.2. I tried the above on OSX 10.8.2 using ghc 7.6.2 and alex 3.0.2 and it worked as expected, so I believe this to be a windows specific ghc bug.
Setup: The program happy is required but it could not be found
Please declare happy
as a required build tool.
0043130
added explicit type signature for alex_accept
:
--- a/src/Output.hs
+++ b/src/Output.hs
@@ -63,43 +66,89 @@ outputDFA target _ _ typesig dfa
- outputAccept
- = case typesig of
- Nothing ->
- -- No type signature: we don't know what the type of the actions is.
- -- str accept_nm . str " :: Array Int (Accept Code)\n"
- str accept_nm . str " = listArray (0::Int," . shows n_states
- . str ") [" . interleave_shows (char ',') (map outputAccs accept)
+ outputAccept =
+ str accept_nm . str " :: Array Int (AlexAcc ())\n"
+ . str accept_nm . str " = listArray (0::Int," . shows n_states . str ") ["
+ . interleave_shows (char ',') (snd (mapAccumR outputAccs 0 accept))
+ . str "]\n"
and GHC relies it to be user-defined:
compiler/parser/Lexer.x:2658:21: error:
• Couldn't match type ‘Word64’ with ‘()’
Expected type: ()
Actual type: ExtsBitmap
• In the first argument of ‘alexScanUser’, namely ‘exts’
In the expression: alexScanUser exts inp sc
In a stmt of a 'do' block:
case alexScanUser exts inp sc of {
AlexEOF
-> do { let ...;
setLastToken span 0;
.... }
AlexError (AI loc2 buf)
-> reportLexError loc1 loc2 buf "lexical error"
AlexSkip inp2 _
-> do { setInput inp2;
lexToken }
AlexToken inp2@(AI end buf2) _ t
-> do { setInput inp2;
let ...;
.... } }
compiler/stage1/build/Lexer.hs:116:15: error:
• Couldn't match type ‘Word64’ with ‘()’
Expected type: Array Int (AlexAcc ())
Actual type: Array Int (AlexAcc ExtsBitmap)
• In the expression:
listArray
(0 :: Int, 207)
[AlexAccNone, AlexAcc 167, AlexAcc 166, AlexAcc 165, ....]
In an equation for ‘alex_accept’:
alex_accept
= listArray
(0 :: Int, 207) [AlexAccNone, AlexAcc 167, AlexAcc 166, ....]
alex-3.2.0 generates the following alex_accept:
alex_accept :: Array Int (AlexAcc ())
alex_accept = listArray ...
Removing explicit type signature makes lexer build fine.
Adding @emc2 for visibility.
It would be nice if Alex supported lazy/non-greedy regex operators such as *?
and +?
so I would not have to reimplement them myself.
Alex assumes that tab size is 8
, as an example, gcc takes tabs as 1 in character size.
Using tab size 8
also makes the errors like:
program:2:9
To give addresses that are not real.
I propose we make the size of a tab to be 1
, it's just changing this line.
While building for Stackage Nightly:
../dist/build/alex/alex --template=.. -g tokens_bytestring_unicode.x -o tokens_bytestring_unicode.g.hs
ghc -Wall -fno-warn-missing-signatures -fno-warn-name-shadowing -fno-warn-unused-imports -fno-warn-tabs -Werror -package array -package bytestring tokens_bytestring_unicode.g.hs -o tokens_bytestring_unicode.g.bin
[1 of 1] Compiling Main ( tokens_bytestring_unicode.g.hs, tokens_bytestring_unicode.g.o )
Linking tokens_bytestring_unicode.g.bin ...
./tokens_bytestring_unicode.g.bin
make: *** No rule to make target 'basic_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'basic_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'basic_typeclass_bytestring.n.run', needed by 'all'.
make: *** No rule to make target 'basic_typeclass_bytestring.g.run', needed by 'all'.
make: *** No rule to make target 'strict_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'strict_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'posn_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'posn_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'posn_typeclass_bytestring.n.run', needed by 'all'.
make: *** No rule to make target 'posn_typeclass_bytestring.g.run', needed by 'all'.
make: *** No rule to make target 'gscan_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'gscan_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'monad_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'monad_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'monad_typeclass_bytestring.n.run', needed by 'all'.
make: *** No rule to make target 'monad_typeclass_bytestring.g.run', needed by 'all'.
make: *** No rule to make target 'monadUserState_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'monadUserState_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'monadUserState_typeclass_bytestring.n.run', needed by 'all'.
make: *** No rule to make target 'monadUserState_typeclass_bytestring.g.run', needed by 'all'.
make: *** No rule to make target 'default_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'default_typeclass.g.run', needed by 'all'.
make: Target 'all' not remade because of errors.
make: Leaving directory '/tmp/stackage-build8/alex-3.2.0/tests'
Cabal still doesn't support giving custom Alex options in configuration file (see issue: haskell/cabal#1223), and some programs need to specify their own --template directory, or use predicates that are as of recenlty incompatible with default invocation (see: yi-editor/yi#435.)
Thus it would be nice to have %option pragma that could add any command-line options. (Like GHC does with {-# OPTIONS_GHC #-}.)
The following pretty clearly describes the issue with the generated code on it's own. I'll let it speak for itself. Not a major issue, but the warning is annoying and I can't suppress it even with {-# OPTIONS_GHC -w #-}.
dist/build/stgma/stgma-tmp/Scan.hs:315:17:
Warning: Pattern bindings containing unlifted types should use an outermost bang pattern:
((I# (ord_c))) = fromIntegral c
In the expression:
let
(base) = alexIndexInt32OffAddr alex_base s
((I# (ord_c))) = fromIntegral c
(offset) = (base +# ord_c)
....
in
case new_s of {
-1# -> (new_acc, input)
_ -> alex_scan_tkn
user
orig_input
(if c < 128 || c >= 192 then (len +# 1#) else len)
new_input
new_s
new_acc }
In a case alternative:
Just (c, new_input)
-> let
(base) = alexIndexInt32OffAddr alex_base s
((I# (ord_c))) = fromIntegral c
....
in
case new_s of {
-1# -> (new_acc, input)
_ -> alex_scan_tkn
user
orig_input
(if c < 128 || c >= 192 then (len +# 1#) else len)
new_input
new_s
new_acc }
In the second argument of `seq', namely
`case alexGetByte input of {
Nothing -> (new_acc, input)
Just (c, new_input)
-> let
(base) = ...
....
in
case new_s of {
-1# -> ...
_ -> alex_scan_tkn
user
orig_input
(if c < 128 || c >= 192 then (len +# 1#) else len)
new_input
new_s
new_acc } }'
main :-
<0> {
. { c () }
}
is fine but
main :-
<0> {
}
Gives a somewhat uninformative error: alex: (Array.!): undefined array element
According to this:
http://www.haskell.org/ghc/docs/latest/html/users_guide/options-phases.html
You can ship a script:
#!/bin/sh
alex "$2" -o="$3"
And then people can use a file with a pragma:
{-# OPTIONS_GHC -pgmF alex-ghc -F #-}
without having fun with indirect .x files and resulting .hs files
drift already uses it, and derive possibly will:
https://code.google.com/p/ndmitchell/issues/detail?id=513&colspec=ID%20Status%20Proj%20Summary%20Priority%20Stars
Using %wrapper "monad-bytestring"
(or monadUserState-bytestring
) results in a compiler error:
templates/wrappers.hs:347:9:
Couldn't match type ‘Int’ with ‘Int64’
Expected type: AlexInput → Int64 → Alex Token
Actual type: (AlexPosn, Char, ByteString.ByteString, [Char])
→ Int → Alex Token
In a stmt of a 'do' block: action (ignorePendingBytes inp) len
In the expression:
do { alexSetInput inp';
action (ignorePendingBytes inp) len }
In a case alternative:
AlexToken inp'@(_, _, _, n') _ action
-> do { alexSetInput inp';
action (ignorePendingBytes inp) len }
where
len = n' - n
This seems to be caused by the definitions
data AlexReturn a
= AlexEOF
| AlexError !AlexInput
| AlexSkip !AlexInput !Int
| AlexToken !AlexInput !Int a
and
alexMonadScan = do
inp@(_,_,str,n) <- alexGetInput
sc <- alexGetStartCode
case alexScan inp sc of
AlexEOF -> alexEOF
AlexError ((AlexPn _ line column),_,_,_) -> alexError $ "lexical error at line " ++ (show line) ++ ", column " ++ (show column)
AlexSkip inp' len -> do
alexSetInput inp'
alexMonadScan
AlexToken inp'@(_,_,_,n') _ action -> do
alexSetInput inp'
action (ignorePendingBytes inp) len
where
len = n'-n
together with ByteString using Int64 for lengths.
The basic-bytestring
wrapper does not work correctly with left contexts when provided with characters which are encoded as multiple bytes in UTF-8.
The following program produces True,False
while I expect it to produce True,True
.
{
{-# LANGUAGE OverloadedStrings #-}
module Main where
import qualified Data.ByteString.Lazy.Char8 as B
}
%wrapper "basic-bytestring"
tokens :-
a^b {const True}
a {const True}
∃^∀ {const True}
∃ {const True}
. {const False}
{
main::IO ()
main = do
print . and . alexScanTokens $ "ab"
print . and . alexScanTokens $ "∃∀"
}
I think this is due to alexGetByte
for this wrapper remembering the last byte rather than the last character.
Since converting input bytes to characters puts unnecessary costs on the users of this wrapper maybe we should just not implement left contexts in this case?
theres thousands of newlines in that attached file!
also some of the source / line attributions seem likely to cause nondeterminstic builds under some conditions
eg
{-# LINE 1 "/var/folders/py/wgp_hj9d2rl3cx48yym_ynj00000gn/T/ghc64947_0/ghc_2.h" #-}
cc @hvr
The motivation is that e.g. alex-options: --latin1
can't be set in .cabal
files, but if we had a way to set alex
options directly in .x
-files, we could workaround more easily...
Currently alex cabal file defines pretty simple test suite (https://github.com/simonmar/alex/blob/master/alex.cabal#L125-L128) that just calls out to make
in order to build sample test programs with freshly built executable (NB executable must be built by cabal
, stack
won't work) and check that they're producing expected output. There's also a dependency on QuickCheck
(https://github.com/simonmar/alex/blob/master/alex.cabal#L99) along with definitions of some properties and generators, but it seems they're not used anywhere and thus are not checked regularly nor automatically.
So, my question is: would it be reasonable to introduce a some testing framework to organize alex tests, both quickcheck properties and sample inputs? It seems that tasty
could fit the job, are there any objections to adding it as a dependency?
The template files assume UTF-8 in the alex_scan_tkn
function, even when the --latin1 option is given.
In templates/GenericTemplate.hs:
alex_scan_tkn user orig_input (if c < 0x80 || c >= 0xC0 then PLUS(len,ILIT(1)) else len)
This leads to an incorrect length being given to the lexer actions for tokens containing bytes between 0x80 and 0xC0
I cannot seem to lex the Null character in a string. I would like to have a lexer that can read the string
"ab\NULcd"
However, no matter what I tried, when running the lexer I get Lexical error.
Here is my definition
$escchars = [abfnrtv\"'&]
@escape = \ ($escchars | \NUL)
@gap = \ $white+
@string = $printable # ["] | " " | @escape | @gap
I would like to do
" string* " {\s -> String (strip s)}
(where strip removes the beginning and trailing ")
I have tried all kind of things for doing the null character from just NUL to \0 to 0, ... to copying Marlow's example definition of the ASCII characters, and nothing works. I always get a lexical error when trying to lex a file with a string with the NUL character in it.
Please help. Thank you!!
It'd be super nice if Alex exposed some of the types it uses so that any functions working with them don't have to be included directly into the lexers.
For example, in Yi wi have to #include
https://github.com/yi-editor/yi/blob/master/yi/src/library/Yi/Lexer/common.hsinc in every lexer we want to use the functions in: if we try to move them to a regular Haskell module, we don't have the types in scope when necessary.
It seems the new primops are only used when bootstrapping GHC-7.7 from GHC-7.6, not when building alex with GHC-7.7. Here are the errors that I get:
[20 of 22] Compiling Scan ( dist/build/alex/alex-tmp/Scan.hs, dist/build/alex/alex-tmp/Scan.o )
dist/build/alex/alex-tmp/Scan.hs:349:29:
Couldn't match expected type ‛Bool’ with actual type ‛Int#’
In the first argument of ‛(&&)’, namely ‛(offset >=# 0#)’
In the expression: (offset >=# 0#) && (check ==# ord_c)
In the expression:
if (offset >=# 0#) && (check ==# ord_c) then
alexIndexInt16OffAddr alex_table offset
else
alexIndexInt16OffAddr alex_deflt s
dist/build/alex/alex-tmp/Scan.hs:349:48:
Couldn't match expected type ‛Bool’ with actual type ‛Int#’
In the second argument of ‛(&&)’, namely ‛(check ==# ord_c)’
In the expression: (offset >=# 0#) && (check ==# ord_c)
In the expression:
if (offset >=# 0#) && (check ==# ord_c) then
alexIndexInt16OffAddr alex_table offset
else
alexIndexInt16OffAddr alex_deflt s
src/Scan.x:152:2:
No instance for (Num a0) arising from a use of ‛go’
The type variable ‛a0’ is ambiguous
Note: there are several potential instances:
instance Num Double -- Defined in ‛GHC.Float’
instance Num Float -- Defined in ‛GHC.Float’
instance Integral a => Num (GHC.Real.Ratio a)
-- Defined in ‛GHC.Real’
...plus 37 others
In a stmt of a 'do' block: go inp 1 ""
In the expression:
do { inp <- getInput;
go inp 1 "" }
In an equation for ‛code’:
code (p, _, inp) len
= do { inp <- getInput;
go inp 1 "" }
where
go inp 0 cs
= do { setInput inp;
.... }
go inp n cs = do { ... }
go_char inp n (c1 : c2 : cs)
| isAlphaNum c2 = go inp n (c1 : c2 : cs)
go_char inp n cs = go_str inp n cs '\''
go_str inp n cs end = do { ... }
err inp
= do { setInput inp;
.... }
ζ: cabal install
Resolving dependencies...
[1 of 1] Compiling Main ( dist/setup/setup.hs, dist/setup/Main.o )
dist/setup/setup.hs:3:2: Parse error in pattern: begin {code}
Failed to install alex-3.1.0
cabal: Error: some packages failed to install:
alex-3.1.0 failed during the configure step. The exception was:
ExitFailure 1
It turns out that dist/setup/setup.hs
is written in Literate Haskell style:
#!/usr/bin/runhaskell
\begin{code}
module Main where
import Distribution.Verbosity
{- snip -}
]
\end{code}
See CHANGELOG on Hackage.
Some packages provide the generated files to ease installation. If a user wants to open these in an editor or run tooling over them, this presents challenges.
e.g. http://hackage.haskell.org/package/Agda-2.4.2.2/docs/src/Agda-Syntax-Parser-Lexer.html#lexer
Consider splitting the generated code over multiple lines for
alex_base
alex_table
alex_check
alex_dflt
alex_accept
http://hackage.haskell.org/cgi-bin/hackage-scripts/package/alex refers here for the latest source code, but it appears to me that the code on Github is not up-to-date: alex.cabal
on Github claims to be version 3.0.3, but Hackage has version 3.0.5 already.
Everything compiles fine and works as far a I can tell, but I didn't have the 20+ lines of warnings before ghc 7.10. Just a heads up.
[3 of 5] Compiling Token ( dist/build/verifier/verifier-tmp/Token.hs, dist/build/verifier/verifier-tmp/Token.o )
dist/build/verifier/verifier-tmp/Token.hs:391:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:392:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:393:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:397:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:398:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:402:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:404:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:408:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:410:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:414:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:423:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:437:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:439:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:440:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:441:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:443:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:444:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:445:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:446:5: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:448:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:451:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:452:1: Warning:
Tab character
dist/build/verifier/verifier-tmp/Token.hs:453:1: Warning:
Tab character
At the moment cabal does not enforce build-tools clauses.
This cause a problem for language-javascript, which is a deep dependency in the yesod family, and it often fails to install due to a missing alex (and happy), as per
yesodweb/yesod#747
How about creating a tiny library that is part of the alex package, perhaps with a single function to return a version number, so that the dependencies can be managed in the normal way.
cc @snoyberg
It looks like the zero-or-more matching functionality in Alex 3.1.4 has an error. In the following minimal example, alex emits the following error: alex: Prelude.head: empty list
.
In the event that I change line 7 from @whitespaces = $whitespace*
to @whitespaces = $whitespace+
, the error goes away.
{
module Data.CSS.Lex where
import Data.ByteString.Lazy (ByteString)
}
%wrapper "posn-bytestring"
$whitespace = [\ \n\t]
@whitespaces = $whitespace*
state:-
<0> @whitespaces { mkL Whitespace }
{
mkL :: CSSToken -> AlexPosn -> ByteString -> CSSLexeme
mkL x _ _ = x
data CSSToken = Whitespace
type CSSLexeme = CSSToken
}
Several new features and fixes have been added since 3.1.7
In particular, PR 68 adds support for using type classes in lexers, and PR 90 fixes warnings in generated code. This is arguably enough for subminor version bump (to 3.2.0).
Releasing these features would allow them to be used in cabal-based projects without having to manually install from github.
The example I'm seeing in Stackage is:
Were there any changes to alex that would explain this?
I would like to build my software using stack with the --pedantic flag.
When using the LTS 7.14, the wrappers.hs file yields unfortunately:
C:..\templates\wrappers.hs:67:25: warning: [-Wunused-matches]
Defined but not used: `ps'
C:..\templates\wrappers.hs:70:20: warning: [-Wunused-matches]
Defined but not used: `p'
C:..\templates\wrappers.hs:70:24: warning: [-Wunused-matches]
Defined but not used: `bs'
C:..\templates\wrappers.hs:70:27: warning: [-Wunused-matches]
Defined but not used: `s'
C:..\templates\wrappers.hs:74:14: warning: [-Wunused-matches]
Defined but not used: `p'
C:..\templates\wrappers.hs:74:16: warning: [-Wunused-matches]
Defined but not used: `c'
C:..\templates\wrappers.hs:157:22: warning: [-Wunused-matches]
Defined but not used: `c'
C:..\templates\wrappers.hs:447:25: warning: [-Wname-shadowing]
This binding for `str' shadows the existing binding
bound at templates\wrappers.hs:446:16
C:..\templates\wrappers.hs:451:32: warning: [-Wunused-matches]
Defined but not used: `len'
It seems that the latest code at https://github.com/simonmar/alex/blob/master/templates/wrappers.hs
addresses the unused-matches resulting in "Defined but not used" warnings.
However, the name-shadowing is still present!
Could the alex code be made such that pedantic builds succeed?
Thanks in advance!
As alex-2.x.x worked off of characters, interesting encodings were supported automatically, by nature of Char being able to describe them. As libraries like text provide a way of decoding from multiple input formats, and interpreting them as a buffer of Chars, alex was more or less able to handle those formats.
With the change to expecting bytes as input in alex-3.x.x, using libraries like text means first decoding the original format, then re-encoding each character as a list of bytes that can then be given, one at a time, to alex. Is this the best way to use a library like text with alex, or is there a better way that isn't covered in the documentation?
As stated in #40 . I'm including the two commands I ran. The interesting part is:
Configuring alex-3.1.4...
Building alex-3.1.4...
Preprocessing executable 'alex' for alex-3.1.4...
setup: The program 'happy' is required but it could not be found
Failed to install alex-3.1.4
The whole output is:
$ cabal sandbox init
Writing a default package environment file to
Users/user/Projects/alex/cabal.sandbox.config
Creating a new sandbox at Users/user/Projects/alex/.cabal-sandbox
$ cabal install
Warning: The package list for 'hackage.haskell.org' is 17 days old.
Run 'cabal update' to get the latest list of available packages.
Resolving dependencies...
Configuring primitive-0.5.4.0...
Building primitive-0.5.4.0...
Preprocessing library primitive-0.5.4.0...
[ 1 of 10] Compiling Data.Primitive.Internal.Compat ( Data/Primitive/Internal/Compat.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/Internal/Compat.o )
[ 2 of 10] Compiling Data.Primitive.MachDeps ( Data/Primitive/MachDeps.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/MachDeps.o )
[ 3 of 10] Compiling Data.Primitive.Internal.Operations ( Data/Primitive/Internal/Operations.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/Internal/Operations.o )
[ 4 of 10] Compiling Control.Monad.Primitive ( Control/Monad/Primitive.hs, dist/dist-sandbox-ff7bbd35/build/Control/Monad/Primitive.o )
[ 5 of 10] Compiling Data.Primitive.Types ( Data/Primitive/Types.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/Types.o )
[ 6 of 10] Compiling Data.Primitive.Array ( Data/Primitive/Array.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/Array.o )
Data/Primitive/Array.hs:32:1: Warning:
The import of ‘Control.Monad.ST’ is redundant
except perhaps to import instances from ‘Control.Monad.ST’
To import instances alone, use: import Control.Monad.ST()
[ 7 of 10] Compiling Data.Primitive.ByteArray ( Data/Primitive/ByteArray.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/ByteArray.o )
[ 8 of 10] Compiling Data.Primitive.Addr ( Data/Primitive/Addr.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/Addr.o )
[ 9 of 10] Compiling Data.Primitive ( Data/Primitive.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive.o )
[10 of 10] Compiling Data.Primitive.MutVar ( Data/Primitive/MutVar.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/MutVar.o )
In-place registering primitive-0.5.4.0...
Creating package registration file:
/var/folders/yn/lxnhmck55w70_1cwlp4tq6h00000gn/T/pkgConf-primitive-0.5.421332.0
Installing library in
Users/user/Projects/alex/.cabal-sandbox/lib/x86_64-osx-ghc-7.8.3/primitive-0.5.4.0
Registering primitive-0.5.4.0...
Installed primitive-0.5.4.0
Configuring random-1.1...
Building random-1.1...
Preprocessing library random-1.1...
[1 of 1] Compiling System.Random ( System/Random.hs, dist/dist-sandbox-ff7bbd35/build/System/Random.o )
In-place registering random-1.1...
Creating package registration file:
/var/folders/yn/lxnhmck55w70_1cwlp4tq6h00000gn/T/pkgConf-random-121332.1
Installing library in
Users/user/Projects/alex/.cabal-sandbox/lib/x86_64-osx-ghc-7.8.3/random-1.1
Registering random-1.1...
Installed random-1.1
Configuring tf-random-0.5...
Building tf-random-0.5...
Preprocessing library tf-random-0.5...
[1 of 4] Compiling System.Random.TF.Gen ( src/System/Random/TF/Gen.hs, dist/dist-sandbox-ff7bbd35/build/System/Random/TF/Gen.o )
[2 of 4] Compiling System.Random.TF.Init ( src/System/Random/TF/Init.hs, dist/dist-sandbox-ff7bbd35/build/System/Random/TF/Init.o )
src/System/Random/TF/Init.hs:94:5: Warning:
In the use of ‘bitSize’ (imported from Data.Bits):
Deprecated: "Use 'bitSizeMaybe' or 'finiteBitSize' instead"
[3 of 4] Compiling System.Random.TF.Instances ( src/System/Random/TF/Instances.hs, dist/dist-sandbox-ff7bbd35/build/System/Random/TF/Instances.o )
[4 of 4] Compiling System.Random.TF ( src/System/Random/TF.hs, dist/dist-sandbox-ff7bbd35/build/System/Random/TF.o )
In-place registering tf-random-0.5...
Creating package registration file:
/var/folders/yn/lxnhmck55w70_1cwlp4tq6h00000gn/T/pkgConf-tf-random-021332.5
Installing library in
Users/user/Projects/alex/.cabal-sandbox/lib/x86_64-osx-ghc-7.8.3/tf-random-0.5
Registering tf-random-0.5...
Installed tf-random-0.5
Configuring QuickCheck-2.7.6...
Building QuickCheck-2.7.6...
Preprocessing library QuickCheck-2.7.6...
[ 1 of 15] Compiling Test.QuickCheck.Random ( Test/QuickCheck/Random.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Random.o )
[ 2 of 15] Compiling Test.QuickCheck.Exception ( Test/QuickCheck/Exception.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Exception.o )
[ 3 of 15] Compiling Test.QuickCheck.Text ( Test/QuickCheck/Text.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Text.o )
[ 4 of 15] Compiling Test.QuickCheck.State ( Test/QuickCheck/State.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/State.o )
[ 5 of 15] Compiling Test.QuickCheck.Gen ( Test/QuickCheck/Gen.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Gen.o )
[ 6 of 15] Compiling Test.QuickCheck.Gen.Unsafe ( Test/QuickCheck/Gen/Unsafe.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Gen/Unsafe.o )
[ 7 of 15] Compiling Test.QuickCheck.Arbitrary ( Test/QuickCheck/Arbitrary.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Arbitrary.o )
[ 8 of 15] Compiling Test.QuickCheck.Modifiers ( Test/QuickCheck/Modifiers.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Modifiers.o )
[ 9 of 15] Compiling Test.QuickCheck.Poly ( Test/QuickCheck/Poly.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Poly.o )
[10 of 15] Compiling Test.QuickCheck.Function ( Test/QuickCheck/Function.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Function.o )
[11 of 15] Compiling Test.QuickCheck.Property ( Test/QuickCheck/Property.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Property.o )
[12 of 15] Compiling Test.QuickCheck.Test ( Test/QuickCheck/Test.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Test.o )
[13 of 15] Compiling Test.QuickCheck.All ( Test/QuickCheck/All.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/All.o )
[14 of 15] Compiling Test.QuickCheck.Monadic ( Test/QuickCheck/Monadic.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Monadic.o )
[15 of 15] Compiling Test.QuickCheck ( Test/QuickCheck.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck.o )
In-place registering QuickCheck-2.7.6...
Creating package registration file:
/var/folders/yn/lxnhmck55w70_1cwlp4tq6h00000gn/T/pkgConf-QuickCheck-2.721332.6
Installing library in
Users/user/Projects/alex/.cabal-sandbox/lib/x86_64-osx-ghc-7.8.3/QuickCheck-2.7.6
Registering QuickCheck-2.7.6...
Installed QuickCheck-2.7.6
[1 of 1] Compiling Main ( dist/dist-sandbox-ff7bbd35/setup/setup.hs, dist/dist-sandbox-ff7bbd35/setup/Main.o )
Linking ./dist/dist-sandbox-ff7bbd35/setup/setup ...
Configuring alex-3.1.4...
Building alex-3.1.4...
Preprocessing executable 'alex' for alex-3.1.4...
setup: The program 'happy' is required but it could not be found
Failed to install alex-3.1.4
cabal: Error: some packages failed to install:
alex-3.1.4 failed during the building phase. The exception was:
ExitFailure 1
I don't know how tricky this may be, but as it stands, having alex as a whole require QuickCheck makes things somewhat tricky for the new minimal platform, and we have to hack around it in silly ways. If it isn't a ridiculous hassle, it would be nice for QuickCheck to be only required when building the tests?
It seems to me that the implementation of the basic-bytestring
wrapper will in most cases destroy the benefits of using a lazy bytestring.
The following line in wrappers.hs:
where len = ByteString.length str - ByteString.length str'
will necessarily read in all of str
and str'
into memory, thus causing the lexer to use memory equal to the size of the input remaining when the first non-skipped rule is applied.
I am not sure how to fix this. The problem being that the length argument of AlexToken
needs to be the number of characters when the stream is something like String
or Text
and the number of bytes for something like Bytestring
. Maybe use another argument and let the consumer decide?
This Stackoverflow question shows a lexer that uses memory proportionate to input size when using basic-bytestring
while the corresponding lexer using the basic
wrapper uses constant memory.
the upload of alex-3.1.5 http://hackage.haskell.org/package/alex
broke our travis builds https://travis-ci.org/spechub/Hets/builds
Failed to install gtk2hs-buildtools-0.13.0.4
Last 10 lines of the build log (
/home/travis/.cabal/logs/gtk2hs-buildtools-0.13.0.4.log ):
[23 of 41] Compiling CAttrs ( c2hs/c/CAttrs.hs,
dist/build/gtk2hsC2hs/gtk2hsC2hs-tmp/CAttrs.o )
[24 of 41] Compiling CBuiltin ( c2hs/c/CBuiltin.hs,
dist/build/gtk2hsC2hs/gtk2hsC2hs-tmp/CBuiltin.o )
[25 of 41] Compiling CParserMonad ( c2hs/c/CParserMonad.hs,
dist/build/gtk2hsC2hs/gtk2hsC2hs-tmp/CParserMonad.o )
[26 of 41] Compiling CLexer (
dist/build/gtk2hsC2hs/gtk2hsC2hs-tmp/CLexer.hs,
dist/build/gtk2hsC2hs/gtk2hsC2hs-tmp/CLexer.o )
c2hs/c/CLexer.x:400:49:
Not in scope: ‘ord’
Perhaps you meant one of these:
‘odd’ (imported from Prelude), ‘or’ (imported from Prelude),
‘ord#’ (imported from GHC.Exts)
cabal: Error: some packages failed to install:
gtk2hs-buildtools-0.13.0.4 failed during the building phase. The exception
was:
ExitFailure 1
Unfortunately, we still rely on the glade package that does not work with ghc-7.10
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.