Coder Social home page Coder Social logo

haskell / alex Goto Github PK

View Code? Open in Web Editor NEW
292.0 292.0 82.0 4.58 MB

A lexical analyser generator for Haskell

Home Page: https://hackage.haskell.org/package/alex

License: BSD 3-Clause "New" or "Revised" License

Haskell 66.07% Shell 0.65% Logos 27.12% C 0.03% Makefile 2.35% Yacc 3.13% RPC 0.66%
lexer-generator

alex's People

Contributors

airmover avatar alexbiehl avatar andreasabel avatar chamini2 avatar damhiya avatar davidlazar avatar dcoutts avatar donsbot avatar emc2 avatar erantapaa avatar ericson2314 avatar erikd avatar ezyang avatar hsyl20 avatar hvr avatar igfoo avatar ivanperez-keera avatar josephcsible avatar jtdaugherty avatar kazu-yamamoto avatar m-schmidt avatar ndmitchell avatar nkpart avatar nochiel avatar samb avatar sergv avatar simonmar avatar thomie avatar tlawson avatar trofi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

alex's Issues

Fails to build on ppc64/Linux

The failure message is:

templates/GenericTemplate.hs:77:4:
    Illegal bang-pattern (use -XBangPatterns):
    ! i

Tried with ghc 7.6.1 / alex 3.0.2.

Null character causes crash

Alex-3.0.2 is failing with "Prelude.Enum.Char.pred: bad argument" when I have a [\x00-\xff] or [^'\"\0\n] range in my *.x file:

@inComment = ([^*] | $white)+ | ([*]+ ([\x00-\xff] # [/]))
@Instr = @hexEscape | @octEscape | @charescape | [^'\"\0\n]

The above worked in alex-2.3.1. I can work around this:

@inComment = ([^*] | $white)+ | ([*]+ [^\/])
@Instr = @hexEscape | @octEscape | @charescape | [^'\"\n] | [\0]

I would guess your character range code uses pred on the null Char and dies.

feature request: support incremental/streaming lexing

in a number of application domains, I need to deal with handling streaming inputs in an incremental fashion, and having a streaming lexer / tokenization layer helps immensely with writing the layers on top.

If adding such capabilities to Alex are viable, i'd be very interested in trying to help add them. (rather than having to reinvent a lot of the tooling that alex provides)

would this be a feature you'd be open to having added? @simonmar ?

The import of ‘Control.Applicative’ is redundant - when using GHC 7.10

Maybe this has already been solved but not yet found its way to hackage:

When compiling modules created by Alex 3.1.4 with GHC 7.10.2 the following warning comes up:

templates/wrappers.hs:7:1: Warning:
    The import of ‘Control.Applicative’ is redundant
      except perhaps to import instances from ‘Control.Applicative’
    To import instances alone, use: import Control.Applicative()

I'm using Alex 3.1.4 - the newest I could find on hackage:

$ alex --version
Alex version 3.1.4, (c) 2003 Chris Dornan and Simon Marlow

I think this is one of the topics covered in https://ghc.haskell.org/trac/ghc/wiki/Migration/7.10

thx, nikomi

Broken parsers with ghc 7.8.2

I recently upgraded ghc and reinstalled yesod-dsl.

I got the following error:

[ 2 of 27] Compiling Lexer ( dist/build/yesod-dsl/yesod-dsl-tmp/Lexer.hs, dist/build/yesod-dsl/yesod-dsl-tmp/Lexer.o )

dist/build/yesod-dsl/yesod-dsl-tmp/Lexer.hs:565:34:
Couldn't match expected type ‘Bool’ with actual type ‘Int#’
In the first argument of ‘(&&)’, namely ‘(offset >=# 0#)’
In the expression: (offset >=# 0#) && (check ==# ord_c)
In the expression:
if (offset >=# 0#) && (check ==# ord_c) then
alexIndexInt16OffAddr alex_table offset
else
alexIndexInt16OffAddr alex_deflt s

dist/build/yesod-dsl/yesod-dsl-tmp/Lexer.hs:565:53:
Couldn't match expected type ‘Bool’ with actual type ‘Int#’
In the second argument of ‘(&&)’, namely ‘(check ==# ord_c)’
In the expression: (offset >=# 0#) && (check ==# ord_c)
In the expression:
if (offset >=# 0#) && (check ==# ord_c) then
alexIndexInt16OffAddr alex_table offset
else
alexIndexInt16OffAddr alex_deflt s

This error happens because ghc 7.6.3 used ghc-prim 0.3.0.0 which defined >=# as Int# -> Int# -> Bool
ghc 7.8.2 used ghc-prim 0.3.1.0 which defined >=# as Int# -> Int# -> Int#

Relevant lines:
https://github.com/simonmar/alex/blob/master/src/Scan.hs#L349
https://github.com/simonmar/alex/blob/master/templates/GenericTemplate.hs#L170

Broken Makefiles

I've noticed that examples/Makefile and templates/Makefile no longer work:

naesten@hydrogen:~/hacking/haskell/alex% make -C examples
make: Entering directory `/home/naesten/hacking/haskell/alex/examples'
Makefile:2: ../mk/boilerplate.mk: No such file or directory
Makefile:31: ../mk/target.mk: No such file or directory
make: *** No rule to make target `../mk/target.mk'.  Stop.
make: Leaving directory `/home/naesten/hacking/haskell/alex/examples'
naesten@hydrogen:~/hacking/haskell/alex% make -C templates
make: Entering directory `/home/naesten/hacking/haskell/alex/templates'
Makefile:2: ../mk/boilerplate.mk: No such file or directory
Makefile:67: ../mk/target.mk: No such file or directory
make: *** No rule to make target `../mk/target.mk'.  Stop.
make: Leaving directory `/home/naesten/hacking/haskell/alex/templates'

You should probably just drop templates/Makefile, since it seems to have been replaced by some code in Setup.lhs anyway, but examples/Makefile should probably be fixed, for lack of any other obvious way to build the examples...

Supporting alex <3.2 and >= 3.2 in the same file

It seems like one must handle AlexLastAcc like this with alex <3.2

  AlexLastAcc k input len -> ... AlexToken input len k ...

and like this with alex >= 3.2

  AlexLastAcc k input len -> ... AlexToken input len (alex_actions ! k) ...

Is there a CPP define for alex version so I can at least put this line under #ifdef?

Or maybe is there another way to write it so it works with both versions like a typeclass with instances for Int and a type of alex_action?

Problem is that stackage LTS provides 3.1.7 and stackage nightly provides 3.2.1 and I'd like very much for yi to compile with both.

strict-bytestring wrapper is undocumented

197317b added the strict-bytestring wrapper, but the User Guide doesn't mention it.

In most wrappers, AlexInput is a tuple, whereas in strict-bytestring (and also in basic-bytestring, since #51) it is a record:

data AlexInput = AlexInput { alexChar :: {-# UNPACK #-} !Char,
                             alexStr ::  !ByteString.ByteString,
                             alexBytePos :: {-# UNPACK #-} !Int64}

Make 'Alex' an instance of Applicative

When compiling Alex lexers with GHC 7.8.2 the following message is emitted:

‘Alex’ is an instance of Monad but not Applicative - this will become an
error in GHC 7.10, under the Applicative-Monad Proposal.

Could you please make 'Alex' an instance of Applicative to remove this?

thx, nik

Cabal/ Stack build creates bad AlexWrapper-monad-bytestring

The hackage 'cabal' / stackage build looks bad/ broken with respect to 'AlexWrapper-monad-bytestring'
These files don't contain the 'instance Applicative'

~/.stack/snapshots/x86_64-linux/lts-3.14/7.10.2/share/x86_64-linux-ghc-7.10.2/alex-3.1.4/AlexWrapper-monad-bytestring
~/.cabal/share/x86_64-linux-ghc-7.8.4/alex-3.1.4/AlexWrapper-monad-bytestring

I tried a local build of alex with the latest source code from github and it does contain the Applicative instance. So I am not sure where the issue might be.

Unicode test cases fail

For me, the unicode test cases always fail

$ make -k -C tests clean all
rm -f *.n.hs *.g.hs *.info *.hi *.o *.bin *.exe
../dist/build/alex/alex --template=.. unicode.x -o unicode.n.hs
unicode.x:19:6: lexical error at '\915'
make: *** [unicode.n.hs] Error 1
../dist/build/alex/alex --template=.. -g unicode.x -o unicode.g.hs
unicode.x:19:6: lexical error at '\915'
make: *** [unicode.g.hs] Error 1

This is using the versions of the pre-processed .x/.hs files from the repo, and the same effect if I regenerate them with the latest alex.

The line it's complaining about is:

   [AΓ]        { character }

that second char there, 'Γ'.

Invalid generated code when using a right_ctx

I think I've hit a bug with using right_ctx regular expression. The following code works in alex 2.x, but fails with 3.0.2:

{
module AlexBug where
}
%wrapper "basic"

@opstr = [^:]+
$preop = \,

tokens :-
  $preop^ @opstr / \: { TokenOperation }

{
data Token = TokenOperation String
  deriving (Read, Show, Eq, Ord)
}

Error message:

$ cabal-dev install
Resolving dependencies...
Configuring alexbug-0.1.0.0...
Building alexbug-0.1.0.0...
Preprocessing executable 'alexbug' for alexbug-0.1.0.0...
[1 of 2] Compiling AlexBug          ( dist/build/alexbug/alexbug-tmp/AlexBug.hs, dist/build/alexbug/alexbug-tmp/AlexBug.o )

templates/wrappers.hs:428:119:
    Couldn't match expected type `AlexInput'
                with actual type `(t0, t1)'
    Expected type: t2 -> AlexInput -> Int -> AlexInput -> Bool
      Actual type: t2 -> (t0, t1) -> t3 -> t4 -> t5
    In the return type of a call of `alexPrevCharMatches'
    In the first argument of `alexAndPred', namely
      `alexPrevCharMatches (\ c -> c >= ',' && c <= ',' || False)'
Failed to install alexbug-0.1.0.0
cabal: Error: some packages failed to install:
alexbug-0.1.0.0 failed during the building phase. The exception was:
ExitFailure 1

Am I doing something wrong? If so, what should my alex definition look like? Thanks!

Unicode appears to be broken on windows String wrappers

Given this alex file, Main.x:

{
{-# OPTIONS -w  #-}
module Main where
}

%wrapper "basic"

tokens :-
  $white+                               ;
  λ                                     { const TokenLambda  }

{
-- The token type:
data Token = TokenLambda
  deriving Show

main :: IO ()
main = do
  i <- readFile "input"
  print (alexScanTokens i)

}

And a single unicode character in the the input file:

$ cat input
λ
[dagit@mango:~/Documents/Repos/testing/alex-unicode]
$ alex Main.x
[dagit@mango:~/Documents/Repos/testing/alex-unicode]
$ ghc --make Main.hs
[1 of 1] Compiling Main             ( Main.hs, Main.o )
Linking Main.exe ...
[dagit@mango:~/Documents/Repos/testing/alex-unicode]
$ ./Main.exe
Main.exe: lexical error

I tried setting the encoding of the file to utf8 before reading it but that resulted in a lexical error looking for EOF.

This happens on Windows 7 64 bit, ghc 7.6.1 (32bit build), alex 3.0.2. I tried the above on OSX 10.8.2 using ghc 7.6.2 and alex 3.0.2 and it worked as expected, so I believe this to be a windows specific ghc bug.

alex-3.2.0 fails to generate correct parser for ghc-HEAD

0043130
added explicit type signature for alex_accept:

--- a/src/Output.hs
+++ b/src/Output.hs
@@ -63,43 +66,89 @@ outputDFA target _ _ typesig dfa
-    outputAccept
-        = case typesig of
-          Nothing ->
-              -- No type signature: we don't know what the type of the actions is.
-              -- str accept_nm . str " :: Array Int (Accept Code)\n"
-              str accept_nm . str " = listArray (0::Int," . shows n_states
-            . str ") [" . interleave_shows (char ',') (map outputAccs accept)
+    outputAccept =
+        str accept_nm . str " :: Array Int (AlexAcc ())\n"
+      . str accept_nm . str " = listArray (0::Int," . shows n_states . str ") ["
+      . interleave_shows (char ',') (snd (mapAccumR outputAccs 0 accept))
+      . str "]\n"

and GHC relies it to be user-defined:

compiler/parser/Lexer.x:2658:21: error:
    • Couldn't match type ‘Word64’ with ‘()’
      Expected type: ()
        Actual type: ExtsBitmap
    • In the first argument of ‘alexScanUser’, namely ‘exts’
      In the expression: alexScanUser exts inp sc
      In a stmt of a 'do' block:
        case alexScanUser exts inp sc of {
          AlexEOF
            -> do { let ...;
                    setLastToken span 0;
                    .... }
          AlexError (AI loc2 buf)
            -> reportLexError loc1 loc2 buf "lexical error"
          AlexSkip inp2 _
            -> do { setInput inp2;
                    lexToken }
          AlexToken inp2@(AI end buf2) _ t
            -> do { setInput inp2;
                    let ...;
                    .... } }

compiler/stage1/build/Lexer.hs:116:15: error:
    • Couldn't match type ‘Word64’ with ‘()’
      Expected type: Array Int (AlexAcc ())
        Actual type: Array Int (AlexAcc ExtsBitmap)
    • In the expression:
        listArray
          (0 :: Int, 207)
          [AlexAccNone, AlexAcc 167, AlexAcc 166, AlexAcc 165, ....]
      In an equation for ‘alex_accept’:
          alex_accept
            = listArray
                (0 :: Int, 207) [AlexAccNone, AlexAcc 167, AlexAcc 166, ....]

alex-3.2.0 generates the following alex_accept:

alex_accept :: Array Int (AlexAcc ())
alex_accept = listArray ...

Removing explicit type signature makes lexer build fine.

Adding @emc2 for visibility.

Alex assumes tab to be size 8

Alex assumes that tab size is 8, as an example, gcc takes tabs as 1 in character size.

Using tab size 8 also makes the errors like:

program:2:9

To give addresses that are not real.

I propose we make the size of a tab to be 1, it's just changing this line.

alex-3.2.0: tests missing targets

While building for Stackage Nightly:

../dist/build/alex/alex --template=.. -g tokens_bytestring_unicode.x -o tokens_bytestring_unicode.g.hs
ghc -Wall -fno-warn-missing-signatures -fno-warn-name-shadowing -fno-warn-unused-imports -fno-warn-tabs -Werror -package array -package bytestring  tokens_bytestring_unicode.g.hs -o tokens_bytestring_unicode.g.bin
[1 of 1] Compiling Main             ( tokens_bytestring_unicode.g.hs, tokens_bytestring_unicode.g.o )
Linking tokens_bytestring_unicode.g.bin ...
./tokens_bytestring_unicode.g.bin
make: *** No rule to make target 'basic_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'basic_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'basic_typeclass_bytestring.n.run', needed by 'all'.
make: *** No rule to make target 'basic_typeclass_bytestring.g.run', needed by 'all'.
make: *** No rule to make target 'strict_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'strict_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'posn_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'posn_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'posn_typeclass_bytestring.n.run', needed by 'all'.
make: *** No rule to make target 'posn_typeclass_bytestring.g.run', needed by 'all'.
make: *** No rule to make target 'gscan_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'gscan_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'monad_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'monad_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'monad_typeclass_bytestring.n.run', needed by 'all'.
make: *** No rule to make target 'monad_typeclass_bytestring.g.run', needed by 'all'.
make: *** No rule to make target 'monadUserState_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'monadUserState_typeclass.g.run', needed by 'all'.
make: *** No rule to make target 'monadUserState_typeclass_bytestring.n.run', needed by 'all'.
make: *** No rule to make target 'monadUserState_typeclass_bytestring.g.run', needed by 'all'.
make: *** No rule to make target 'default_typeclass.n.run', needed by 'all'.
make: *** No rule to make target 'default_typeclass.g.run', needed by 'all'.
make: Target 'all' not remade because of errors.
make: Leaving directory '/tmp/stackage-build8/alex-3.2.0/tests'

Allow giving options through input pragmas...

Cabal still doesn't support giving custom Alex options in configuration file (see issue: haskell/cabal#1223), and some programs need to specify their own --template directory, or use predicates that are as of recenlty incompatible with default invocation (see: yi-editor/yi#435.)

Thus it would be nice to have %option pragma that could add any command-line options. (Like GHC does with {-# OPTIONS_GHC #-}.)

Annoying Warning Message Due to Missing !

The following pretty clearly describes the issue with the generated code on it's own. I'll let it speak for itself. Not a major issue, but the warning is annoying and I can't suppress it even with {-# OPTIONS_GHC -w #-}.

dist/build/stgma/stgma-tmp/Scan.hs:315:17:
    Warning: Pattern bindings containing unlifted types should use an outermost bang pattern:
               ((I# (ord_c))) = fromIntegral c
    In the expression:
      let
        (base) = alexIndexInt32OffAddr alex_base s
        ((I# (ord_c))) = fromIntegral c
        (offset) = (base +# ord_c)
        ....
      in
        case new_s of {
          -1# -> (new_acc, input)
          _ -> alex_scan_tkn
                 user
                 orig_input
                 (if c < 128 || c >= 192 then (len +# 1#) else len)
                 new_input
                 new_s
                 new_acc }
    In a case alternative:
        Just (c, new_input)
          -> let
               (base) = alexIndexInt32OffAddr alex_base s
               ((I# (ord_c))) = fromIntegral c
               ....
             in
               case new_s of {
                 -1# -> (new_acc, input)
                 _ -> alex_scan_tkn
                        user
                        orig_input
                        (if c < 128 || c >= 192 then (len +# 1#) else len)
                        new_input
                        new_s
                        new_acc }
    In the second argument of `seq', namely
      `case alexGetByte input of {
         Nothing -> (new_acc, input)
         Just (c, new_input)
           -> let
                (base) = ...
                ....
              in
                case new_s of {
                  -1# -> ...
                  _ -> alex_scan_tkn
                         user
                         orig_input
                         (if c < 128 || c >= 192 then (len +# 1#) else len)
                         new_input
                         new_s
                         new_acc } }'

Int/Int64 issue with ByteString wrappers

Using %wrapper "monad-bytestring" (or monadUserState-bytestring) results in a compiler error:

templates/wrappers.hs:347:9:
    Couldn't match type ‘Int’ with ‘Int64’
    Expected type: AlexInput → Int64 → Alex Token
      Actual type: (AlexPosn, Char, ByteString.ByteString, [Char])
                   → Int → Alex Token
    In a stmt of a 'do' block: action (ignorePendingBytes inp) len
    In the expression:
      do { alexSetInput inp';
           action (ignorePendingBytes inp) len }
    In a case alternative:
        AlexToken inp'@(_, _, _, n') _ action
          -> do { alexSetInput inp';
                  action (ignorePendingBytes inp) len }
          where
              len = n' - n

This seems to be caused by the definitions

data AlexReturn a
  = AlexEOF
  | AlexError  !AlexInput
  | AlexSkip   !AlexInput !Int
  | AlexToken  !AlexInput !Int a

and

alexMonadScan = do
  inp@(_,_,str,n) <- alexGetInput
  sc <- alexGetStartCode
  case alexScan inp sc of
    AlexEOF -> alexEOF
    AlexError ((AlexPn _ line column),_,_,_) -> alexError $ "lexical error at line " ++ (show line) ++ ", column " ++ (show column)
    AlexSkip  inp' len -> do
        alexSetInput inp'
        alexMonadScan
    AlexToken inp'@(_,_,_,n') _ action -> do
        alexSetInput inp'
        action (ignorePendingBytes inp) len
      where
        len = n'-n

together with ByteString using Int64 for lengths.

basic-bytestring wrapper does not work with left contexts

The basic-bytestring wrapper does not work correctly with left contexts when provided with characters which are encoded as multiple bytes in UTF-8.

The following program produces True,False while I expect it to produce True,True.

{
{-# LANGUAGE OverloadedStrings #-}
module Main where
import qualified Data.ByteString.Lazy.Char8 as B
}

%wrapper "basic-bytestring"

tokens :-
  a^b    {const True}
  a      {const True}
  ∃^∀    {const True}
  ∃      {const True}
  .      {const False}

{
main::IO ()
main = do
  print . and . alexScanTokens $ "ab"
  print . and . alexScanTokens $ "∃∀"
}

I think this is due to alexGetByte for this wrapper remembering the last byte rather than the last character.

Since converting input bytes to characters puts unnecessary costs on the users of this wrapper maybe we should just not implement left contexts in this case?

Tests improvement

Currently alex cabal file defines pretty simple test suite (https://github.com/simonmar/alex/blob/master/alex.cabal#L125-L128) that just calls out to make in order to build sample test programs with freshly built executable (NB executable must be built by cabal, stack won't work) and check that they're producing expected output. There's also a dependency on QuickCheck (https://github.com/simonmar/alex/blob/master/alex.cabal#L99) along with definitions of some properties and generators, but it seems they're not used anywhere and thus are not checked regularly nor automatically.

So, my question is: would it be reasonable to introduce a some testing framework to organize alex tests, both quickcheck properties and sample inputs? It seems that tasty could fit the job, are there any objections to adding it as a dependency?

token length incorrect using --latin1 option

The template files assume UTF-8 in the alex_scan_tkn function, even when the --latin1 option is given.

In templates/GenericTemplate.hs:

alex_scan_tkn user orig_input (if c < 0x80 || c >= 0xC0 then PLUS(len,ILIT(1)) else len)

This leads to an incorrect length being given to the lexer actions for tokens containing bytes between 0x80 and 0xC0

Problem with NUL character.

I cannot seem to lex the Null character in a string. I would like to have a lexer that can read the string
"ab\NULcd"
However, no matter what I tried, when running the lexer I get Lexical error.

Here is my definition

$escchars = [abfnrtv\"'&]
@escape = \ ($escchars | \NUL)
@gap = \ $white+
@string = $printable # ["] | " " | @escape | @gap

I would like to do

" string* " {\s -> String (strip s)}

(where strip removes the beginning and trailing ")

I have tried all kind of things for doing the null character from just NUL to \0 to 0, ... to copying Marlow's example definition of the ASCII characters, and nothing works. I always get a lexical error when trying to lex a file with a string with the NUL character in it.

Please help. Thank you!!

Building Alex fails on GHC 7.7 due to wrong primps

It seems the new primops are only used when bootstrapping GHC-7.7 from GHC-7.6, not when building alex with GHC-7.7. Here are the errors that I get:

[20 of 22] Compiling Scan             ( dist/build/alex/alex-tmp/Scan.hs, dist/build/alex/alex-tmp/Scan.o )

dist/build/alex/alex-tmp/Scan.hs:349:29:
    Couldn't match expected type ‛Bool’ with actual type ‛Int#’
    In the first argument of ‛(&&)’, namely ‛(offset >=# 0#)’
    In the expression: (offset >=# 0#) && (check ==# ord_c)
    In the expression:
      if (offset >=# 0#) && (check ==# ord_c) then
          alexIndexInt16OffAddr alex_table offset
      else
          alexIndexInt16OffAddr alex_deflt s

dist/build/alex/alex-tmp/Scan.hs:349:48:
    Couldn't match expected type ‛Bool’ with actual type ‛Int#’
    In the second argument of ‛(&&)’, namely ‛(check ==# ord_c)’
    In the expression: (offset >=# 0#) && (check ==# ord_c)
    In the expression:
      if (offset >=# 0#) && (check ==# ord_c) then
          alexIndexInt16OffAddr alex_table offset
      else
          alexIndexInt16OffAddr alex_deflt s

src/Scan.x:152:2:
    No instance for (Num a0) arising from a use of ‛go’
    The type variable ‛a0’ is ambiguous
    Note: there are several potential instances:
      instance Num Double -- Defined in ‛GHC.Float’
      instance Num Float -- Defined in ‛GHC.Float’
      instance Integral a => Num (GHC.Real.Ratio a)
        -- Defined in ‛GHC.Real’
      ...plus 37 others
    In a stmt of a 'do' block: go inp 1 ""
    In the expression:
      do { inp <- getInput;
           go inp 1 "" }
    In an equation for ‛code’:
        code (p, _, inp) len
          = do { inp <- getInput;
                 go inp 1 "" }
          where
              go inp 0 cs
                = do { setInput inp;
                       .... }
              go inp n cs = do { ... }
              go_char inp n (c1 : c2 : cs)
                | isAlphaNum c2 = go inp n (c1 : c2 : cs)
              go_char inp n cs = go_str inp n cs '\''
              go_str inp n cs end = do { ... }
              err inp
                = do { setInput inp;
                       .... }

Building Alex fails with Cabal 1.19.1: it's Literate Haskell?

ζ: cabal install    
Resolving dependencies...
[1 of 1] Compiling Main             ( dist/setup/setup.hs, dist/setup/Main.o )

dist/setup/setup.hs:3:2: Parse error in pattern: begin {code}
Failed to install alex-3.1.0
cabal: Error: some packages failed to install:
alex-3.1.0 failed during the configure step. The exception was:
ExitFailure 1

It turns out that dist/setup/setup.hs is written in Literate Haskell style:

#!/usr/bin/runhaskell

\begin{code}
module Main where

import Distribution.Verbosity
{- snip -}
     ]

\end{code}

Strange Alex warnings, ghc 7.11, cabal 1.22.0.1

Everything compiles fine and works as far a I can tell, but I didn't have the 20+ lines of warnings before ghc 7.10. Just a heads up.

[3 of 5] Compiling Token ( dist/build/verifier/verifier-tmp/Token.hs, dist/build/verifier/verifier-tmp/Token.o )

dist/build/verifier/verifier-tmp/Token.hs:391:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:392:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:393:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:397:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:398:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:402:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:404:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:408:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:410:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:414:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:423:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:437:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:439:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:440:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:441:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:443:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:444:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:445:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:446:5: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:448:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:451:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:452:1: Warning:
Tab character

dist/build/verifier/verifier-tmp/Token.hs:453:1: Warning:
Tab character

Create library to manage install dependencies

At the moment cabal does not enforce build-tools clauses.

This cause a problem for language-javascript, which is a deep dependency in the yesod family, and it often fails to install due to a missing alex (and happy), as per
yesodweb/yesod#747

How about creating a tiny library that is part of the alex package, perhaps with a single function to return a version number, so that the dependencies can be managed in the normal way.

cc @snoyberg

Prelude.head empty list exception for zero or more matches.

It looks like the zero-or-more matching functionality in Alex 3.1.4 has an error. In the following minimal example, alex emits the following error: alex: Prelude.head: empty list.

In the event that I change line 7 from @whitespaces = $whitespace* to @whitespaces = $whitespace+, the error goes away.

{
module Data.CSS.Lex where
import Data.ByteString.Lazy (ByteString)
}

%wrapper "posn-bytestring"

$whitespace = [\ \n\t]

@whitespaces = $whitespace*

state:-

<0> @whitespaces { mkL Whitespace }

{
mkL :: CSSToken -> AlexPosn -> ByteString -> CSSLexeme
mkL x _ _ = x

data CSSToken = Whitespace

type CSSLexeme = CSSToken
}

Release for new features

Several new features and fixes have been added since 3.1.7

In particular, PR 68 adds support for using type classes in lexers, and PR 90 fixes warnings in generated code. This is arguably enough for subminor version bump (to 3.2.0).

Releasing these features would allow them to be used in cabal-based projects without having to manually install from github.

Pedantic stack build fails due to code of wrappers.hs

I would like to build my software using stack with the --pedantic flag.
When using the LTS 7.14, the wrappers.hs file yields unfortunately:

C:..\templates\wrappers.hs:67:25: warning: [-Wunused-matches]
Defined but not used: `ps'

C:..\templates\wrappers.hs:70:20: warning: [-Wunused-matches]
Defined but not used: `p'

C:..\templates\wrappers.hs:70:24: warning: [-Wunused-matches]
Defined but not used: `bs'

C:..\templates\wrappers.hs:70:27: warning: [-Wunused-matches]
Defined but not used: `s'

C:..\templates\wrappers.hs:74:14: warning: [-Wunused-matches]
Defined but not used: `p'

C:..\templates\wrappers.hs:74:16: warning: [-Wunused-matches]
Defined but not used: `c'

C:..\templates\wrappers.hs:157:22: warning: [-Wunused-matches]
Defined but not used: `c'

C:..\templates\wrappers.hs:447:25: warning: [-Wname-shadowing]
This binding for `str' shadows the existing binding
bound at templates\wrappers.hs:446:16

C:..\templates\wrappers.hs:451:32: warning: [-Wunused-matches]
Defined but not used: `len'

It seems that the latest code at https://github.com/simonmar/alex/blob/master/templates/wrappers.hs
addresses the unused-matches resulting in "Defined but not used" warnings.
However, the name-shadowing is still present!

Could the alex code be made such that pedantic builds succeed?

Thanks in advance!

UTF-8 and text

As alex-2.x.x worked off of characters, interesting encodings were supported automatically, by nature of Char being able to describe them. As libraries like text provide a way of decoding from multiple input formats, and interpreting them as a buffer of Chars, alex was more or less able to handle those formats.

With the change to expecting bytes as input in alex-3.x.x, using libraries like text means first decoding the original format, then re-encoding each character as a list of bytes that can then be given, one at a time, to alex. Is this the best way to use a library like text with alex, or is there a better way that isn't covered in the documentation?

alex doesn't bootstrap correctly when built in a sandbox

As stated in #40 . I'm including the two commands I ran. The interesting part is:

Configuring alex-3.1.4...
Building alex-3.1.4...
Preprocessing executable 'alex' for alex-3.1.4...
setup: The program 'happy' is required but it could not be found
Failed to install alex-3.1.4

The whole output is:

$ cabal sandbox init
Writing a default package environment file to
Users/user/Projects/alex/cabal.sandbox.config
Creating a new sandbox at Users/user/Projects/alex/.cabal-sandbox
$ cabal install
Warning: The package list for 'hackage.haskell.org' is 17 days old.
Run 'cabal update' to get the latest list of available packages.
Resolving dependencies...
Configuring primitive-0.5.4.0...
Building primitive-0.5.4.0...
Preprocessing library primitive-0.5.4.0...
[ 1 of 10] Compiling Data.Primitive.Internal.Compat ( Data/Primitive/Internal/Compat.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/Internal/Compat.o )
[ 2 of 10] Compiling Data.Primitive.MachDeps ( Data/Primitive/MachDeps.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/MachDeps.o )
[ 3 of 10] Compiling Data.Primitive.Internal.Operations ( Data/Primitive/Internal/Operations.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/Internal/Operations.o )
[ 4 of 10] Compiling Control.Monad.Primitive ( Control/Monad/Primitive.hs, dist/dist-sandbox-ff7bbd35/build/Control/Monad/Primitive.o )
[ 5 of 10] Compiling Data.Primitive.Types ( Data/Primitive/Types.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/Types.o )
[ 6 of 10] Compiling Data.Primitive.Array ( Data/Primitive/Array.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/Array.o )

Data/Primitive/Array.hs:32:1: Warning:
The import of ‘Control.Monad.ST’ is redundant
  except perhaps to import instances from ‘Control.Monad.ST’
To import instances alone, use: import Control.Monad.ST()
[ 7 of 10] Compiling Data.Primitive.ByteArray ( Data/Primitive/ByteArray.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/ByteArray.o )
[ 8 of 10] Compiling Data.Primitive.Addr ( Data/Primitive/Addr.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/Addr.o )
[ 9 of 10] Compiling Data.Primitive   ( Data/Primitive.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive.o )
[10 of 10] Compiling Data.Primitive.MutVar ( Data/Primitive/MutVar.hs, dist/dist-sandbox-ff7bbd35/build/Data/Primitive/MutVar.o )
In-place registering primitive-0.5.4.0...
Creating package registration file:
/var/folders/yn/lxnhmck55w70_1cwlp4tq6h00000gn/T/pkgConf-primitive-0.5.421332.0
Installing library in
Users/user/Projects/alex/.cabal-sandbox/lib/x86_64-osx-ghc-7.8.3/primitive-0.5.4.0
Registering primitive-0.5.4.0...
Installed primitive-0.5.4.0
Configuring random-1.1...
Building random-1.1...
Preprocessing library random-1.1...
[1 of 1] Compiling System.Random    ( System/Random.hs, dist/dist-sandbox-ff7bbd35/build/System/Random.o )
In-place registering random-1.1...
Creating package registration file:
/var/folders/yn/lxnhmck55w70_1cwlp4tq6h00000gn/T/pkgConf-random-121332.1
Installing library in
Users/user/Projects/alex/.cabal-sandbox/lib/x86_64-osx-ghc-7.8.3/random-1.1
Registering random-1.1...
Installed random-1.1
Configuring tf-random-0.5...
Building tf-random-0.5...
Preprocessing library tf-random-0.5...
[1 of 4] Compiling System.Random.TF.Gen ( src/System/Random/TF/Gen.hs, dist/dist-sandbox-ff7bbd35/build/System/Random/TF/Gen.o )
[2 of 4] Compiling System.Random.TF.Init ( src/System/Random/TF/Init.hs, dist/dist-sandbox-ff7bbd35/build/System/Random/TF/Init.o )

src/System/Random/TF/Init.hs:94:5: Warning:
In the use of ‘bitSize’ (imported from Data.Bits):
Deprecated: "Use 'bitSizeMaybe' or 'finiteBitSize' instead"
[3 of 4] Compiling System.Random.TF.Instances ( src/System/Random/TF/Instances.hs, dist/dist-sandbox-ff7bbd35/build/System/Random/TF/Instances.o )
[4 of 4] Compiling System.Random.TF ( src/System/Random/TF.hs, dist/dist-sandbox-ff7bbd35/build/System/Random/TF.o )
In-place registering tf-random-0.5...
Creating package registration file:
/var/folders/yn/lxnhmck55w70_1cwlp4tq6h00000gn/T/pkgConf-tf-random-021332.5
Installing library in
Users/user/Projects/alex/.cabal-sandbox/lib/x86_64-osx-ghc-7.8.3/tf-random-0.5
Registering tf-random-0.5...
Installed tf-random-0.5
Configuring QuickCheck-2.7.6...
Building QuickCheck-2.7.6...
Preprocessing library QuickCheck-2.7.6...
[ 1 of 15] Compiling Test.QuickCheck.Random ( Test/QuickCheck/Random.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Random.o )
[ 2 of 15] Compiling Test.QuickCheck.Exception ( Test/QuickCheck/Exception.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Exception.o )
[ 3 of 15] Compiling Test.QuickCheck.Text ( Test/QuickCheck/Text.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Text.o )
[ 4 of 15] Compiling Test.QuickCheck.State ( Test/QuickCheck/State.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/State.o )
[ 5 of 15] Compiling Test.QuickCheck.Gen ( Test/QuickCheck/Gen.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Gen.o )
[ 6 of 15] Compiling Test.QuickCheck.Gen.Unsafe ( Test/QuickCheck/Gen/Unsafe.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Gen/Unsafe.o )
[ 7 of 15] Compiling Test.QuickCheck.Arbitrary ( Test/QuickCheck/Arbitrary.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Arbitrary.o )
[ 8 of 15] Compiling Test.QuickCheck.Modifiers ( Test/QuickCheck/Modifiers.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Modifiers.o )
[ 9 of 15] Compiling Test.QuickCheck.Poly ( Test/QuickCheck/Poly.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Poly.o )
[10 of 15] Compiling Test.QuickCheck.Function ( Test/QuickCheck/Function.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Function.o )
[11 of 15] Compiling Test.QuickCheck.Property ( Test/QuickCheck/Property.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Property.o )
[12 of 15] Compiling Test.QuickCheck.Test ( Test/QuickCheck/Test.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Test.o )
[13 of 15] Compiling Test.QuickCheck.All ( Test/QuickCheck/All.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/All.o )
[14 of 15] Compiling Test.QuickCheck.Monadic ( Test/QuickCheck/Monadic.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck/Monadic.o )
[15 of 15] Compiling Test.QuickCheck  ( Test/QuickCheck.hs, dist/dist-sandbox-ff7bbd35/build/Test/QuickCheck.o )
In-place registering QuickCheck-2.7.6...
Creating package registration file:
/var/folders/yn/lxnhmck55w70_1cwlp4tq6h00000gn/T/pkgConf-QuickCheck-2.721332.6
Installing library in
Users/user/Projects/alex/.cabal-sandbox/lib/x86_64-osx-ghc-7.8.3/QuickCheck-2.7.6
Registering QuickCheck-2.7.6...
Installed QuickCheck-2.7.6
[1 of 1] Compiling Main             ( dist/dist-sandbox-ff7bbd35/setup/setup.hs, dist/dist-sandbox-ff7bbd35/setup/Main.o )
Linking ./dist/dist-sandbox-ff7bbd35/setup/setup ...
Configuring alex-3.1.4...
Building alex-3.1.4...
Preprocessing executable 'alex' for alex-3.1.4...
setup: The program 'happy' is required but it could not be found
Failed to install alex-3.1.4
cabal: Error: some packages failed to install:
alex-3.1.4 failed during the building phase. The exception was:
ExitFailure 1

Factor out QuickCheck dep into only being required for the test suite?

I don't know how tricky this may be, but as it stands, having alex as a whole require QuickCheck makes things somewhat tricky for the new minimal platform, and we have to hack around it in silly ways. If it isn't a ridiculous hassle, it would be nice for QuickCheck to be only required when building the tests?

basic-bytestring wrapper destroys laziness

It seems to me that the implementation of the basic-bytestring wrapper will in most cases destroy the benefits of using a lazy bytestring.

The following line in wrappers.hs:

where len = ByteString.length str - ByteString.length str'

will necessarily read in all of str and str' into memory, thus causing the lexer to use memory equal to the size of the input remaining when the first non-skipped rule is applied.

I am not sure how to fix this. The problem being that the length argument of AlexToken needs to be the number of characters when the stream is something like String or Text and the number of bytes for something like Bytestring. Maybe use another argument and let the consumer decide?

This Stackoverflow question shows a lexer that uses memory proportionate to input size when using basic-bytestring while the corresponding lexer using the basic wrapper uses constant memory.

alex-3.1.5 breaks gtk2hs-buildtools

the upload of alex-3.1.5 http://hackage.haskell.org/package/alex
broke our travis builds https://travis-ci.org/spechub/Hets/builds

Failed to install gtk2hs-buildtools-0.13.0.4
Last 10 lines of the build log (
/home/travis/.cabal/logs/gtk2hs-buildtools-0.13.0.4.log ):
[23 of 41] Compiling CAttrs           ( c2hs/c/CAttrs.hs,
dist/build/gtk2hsC2hs/gtk2hsC2hs-tmp/CAttrs.o )
[24 of 41] Compiling CBuiltin         ( c2hs/c/CBuiltin.hs,
dist/build/gtk2hsC2hs/gtk2hsC2hs-tmp/CBuiltin.o )
[25 of 41] Compiling CParserMonad     ( c2hs/c/CParserMonad.hs,
dist/build/gtk2hsC2hs/gtk2hsC2hs-tmp/CParserMonad.o )
[26 of 41] Compiling CLexer           (
dist/build/gtk2hsC2hs/gtk2hsC2hs-tmp/CLexer.hs,
dist/build/gtk2hsC2hs/gtk2hsC2hs-tmp/CLexer.o )
c2hs/c/CLexer.x:400:49:
    Not in scope: ‘ord’
    Perhaps you meant one of these:
      ‘odd’ (imported from Prelude), ‘or’ (imported from Prelude),
      ‘ord#’ (imported from GHC.Exts)
cabal: Error: some packages failed to install:
gtk2hs-buildtools-0.13.0.4 failed during the building phase. The exception
was:
ExitFailure 1

Unfortunately, we still rely on the glade package that does not work with ghc-7.10

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.