xach / buildapp Goto Github PK

Buildapp-generated binaries should accept runtime arguments that limit memory footprint, just like a normal sbcl session does.

It is documented at https://ccl.clozure.com/manual/chapter16.1.html and is comparable to --dynamic-space-size in SBCL I think. Following up on problems to build pgloader with CCL on dockerhub and travis environments I got the following piece of advice:

rme: The second thing might be to try running ccl with "--heap-reserve 150g". Perhaps the container environment doesn't like ccl's default reservation of 512 gigabytes of address space.

It looks like the following patch would allow buildapp to support --heap-reserve?

modified   buildapp.lisp
@@ -400,7 +400,8 @@ ARGV. See *USAGE* for details."
     (quit))
   (let* ((dumper (command-line-dumper (rest argv)))
          (*package* (find-package :buildapp))
-         #+sbcl (dynamic-space-size (dynamic-space-size dumper)))
+         #+sbcl (dynamic-space-size (dynamic-space-size dumper))
+         #+ccl  (heap-reserve (heap-reserve dumper)))
     (with-tempfile (stream ("dumper.lisp" file))
       (write-dumpfile dumper stream)
       (force-output stream)
@@ -415,6 +416,11 @@ ARGV. See *USAGE* for details."
                                       (list "--dynamic-space-size"
                                             (princ-to-string
                                              dynamic-space-size)))
+                                    #+ccl
+                                    (when heap-reserve
+                                      (list "--heap-reserve"
+                                            (princ-to-string
+                                             heap-reserve)))
                                     #+sbcl "--noinform"
                                     #+ccl  "--quiet"
                                     #+sbcl "--disable-debugger"
modified   command-line.lisp
@@ -160,6 +160,8 @@
              (push entry (dispatched-entries plan))))
           (:dynamic-space-size
            (setf (dynamic-space-size plan) (parse-integer value)))
+          (:heap-reserve
+           (setf (heap-reserve plan) (parse-integer value)))
           (t
            (error 'unknown-argument :flag argument)))))))
 
modified   dumper.lisp
@@ -86,6 +86,10 @@
    (dynamic-space-size
     :initarg :dynamic-space-size
     :accessor dynamic-space-size
+    :initform nil)
+   (heap-reserve
+    :initarg :heap-reserve
+    :accessor heap-reserve
     :initform nil)))
 
 (defgeneric needs-asdf-p (dumper)

When I make buildapp with an absolute pathname for SBCL, like this:

make DESTDIR=/home/rpg/.local/bin/ LISP=/opt/sbcl/current/bin/sbcl

I get the wrong command line arguments to SBCL (--no-init instead of --no-userinit and --no-sysinit):

/opt/sbcl/current/bin/sbcl --quiet --no-init \
  --eval "(require 'asdf)" \
  --eval "(push \"$(pwd)/\" asdf:*central-registry*)" \
  --eval "(require 'buildapp)" \
          --eval "(buildapp::build-buildapp)" \
          --eval "#+sbcl (exit) #+ccl (quit)"

When I use simply sbcl I do not get this problem:

$ make DESTDIR=/home/rpg/.local/bin/ LISP=sbcl
sbcl --noinform --no-userinit --no-sysinit --disable-debugger \
  --eval "(require 'asdf)" \
  --eval "(push \"$(pwd)/\" asdf:*central-registry*)" \
  --eval "(require 'buildapp)" \
          --eval "(buildapp::build-buildapp)" \
          --eval "#+sbcl (exit) #+ccl (quit)"

This is due to a bug in the Makefile:

ifeq ($(LISP),sbcl)
FLAGS=--noinform --no-userinit --no-sysinit --disable-debugger
else
FLAGS=--quiet --no-init
endif

The check for sbcl should be a substring or basename check, not an absolute check.

I will try to get you an MR for this; I don't have time at the moment, but am filing this issue so I don't forget.

Easy to reproduce:

$ echo "(" > bug.lisp
$ buildapp --output bug --load bug.lisp

The subprocess gets stuck in a debugger that isn't attached to the terminal.

When there is no default entry in a set of dispatched-entries, buildapp errors out without building.

I believe that it's going wrong at this line:

Here's an example of the error:

BUILDAPP(3): (dispatched-entry-form (dispatched-entries *dumper*))
rpg@rpg-mbp182: ~/lisp/shop/shop3/buildapp $

To replicate, do rlwrap sbcl --eval '(ql)' --eval '(ql:quickload "buildapp")' --eval '(in-package :buildapp)' (cl-user::ql is my function that load setup) and then

(defparameter *args* (list "--logfile" "/tmp/buildapp-shop-app.log" "--output" "shop-app" "--dispatched-entry" "shop/shop-app::main" "--dispatched-entry" "ess-shop/shop-app::ess-main" "--load" "/Users/rpg/lisp/asdf/build/asdf.lisp" "--asdf-path" "/Users/rpg/lisp/shop/shop3/buildapp/../" "--asdf-tree" "/Users/rpg/lisp/shop/shop3/buildapp/../../jenkins/ext/" "--eval" "(declaim (optimize speed space safety))" "--load-system" "shop3" "--load" "/Users/rpg/lisp/shop/shop3/buildapp/shop-app-entrypoints.lisp" "--dumpfile-copy" "/tmp/dumpfile-copy.lisp"))
(defparameter *dumper* (command-line-dumper *args*))
(dispatched-entry-form (dispatched-entries *dumper*))

If you add --dispatched-entry "/shop-app::main" to the arguments, all is well.

I was having trouble with an application I built with this command line:

buildapp --dynamic-space-size 100000 --output strike-force-one --load game-player.lisp --entry 'cl-user::main' --dumpfile-copy 'strike-force-one-dumpfile.lisp' --logfile 'build-game-player.log'

When I ran the resulting application, it ran out of memory quickly. But when I ran sbcl with --dynamic-space-size 100000 and passed it a script that did the equivalent, it did not run out of memory.

Tracking it down, I found that run-program for SBCL, invoked in main was not getting the --dynamic-space-size 100000 argument:

(RUN-PROGRAM "sbcl"
             '("--noinform" "--disable-debugger" "--no-userinit" "--no-sysinit"
               "--load"
               "/home/rpg/projects/scepter/shop-mcts/strike-force-one/dumper-2SKVI5f7.lisp"))

Further debugging shows that buildapp does not see the --dynamic-space-size argument:

Constructing a dumper from (--output strike-force-one --load game-player.lisp
                            --entry cl-user::main --dumpfile-copy
                            strike-force-one-dumpfile.lisp --logfile
                            build-game-player.log)
After processing the command line arguments, we have the following dumper object:
#<BUILDAPP::DUMPER strike-force-one cl-user::main>
  [standard-object]

Slots with :INSTANCE allocation:
  PACKAGE                        = #:DUMPER124
  ACTIONS                        = ((:LOAD "game-player.lisp"))
  ENTRY                          = cl-user::main
  DISPATCHED-ENTRIES             = NIL
  ASDF-DIRECTIVES                = NIL
  LOAD-PATHS                     = NIL
  SBCL                           = "sbcl"
  CCL                            = "ccl"
  OUTPUT                         = "strike-force-one"
  LOGFILE                        = "build-game-player.log"
  DUMPFILE-COPY                  = "strike-force-one-dumpfile.lisp"
  CORE-ONLY                      = NIL
  COMPRESS-CORE                  = NIL
  DYNAMIC-SPACE-SIZE             = NIL

The "constructing a dumper" message above formats the arguments that are passed to command-line-dumper.

I conjecture that SBCL is stripping out this argument before the buildapp executable can see it.

I will attempt to verify this.

(buildapp:build-buildapp) fails with
Error #<BUILDAPP::SILENT-EXIT-ERROR #x302002BECB5D>
[Condition of type BUILDAPP::SILENT-EXIT-ERROR]

Restarts:
0: [RETRY] Retry SLIME REPL evaluation request.
1: [*ABORT] Return to SLIME's top level.
2: [ABORT-BREAK] Reset this thread
3: [ABORT] Kill this thread

Backtrace:
0: ((:INTERNAL BUILDAPP::MAIN) #<BASIC-FILE-CHARACTER-OUTPUT-STREAM ("dumper-3SdqQGqO.lisp"/4 UTF-8) #x302002BF220D> #P"dumper-3SdqQGqO.lisp")
1: (BUILDAPP::CALL-WITH-TEMPORARY-OPEN-FILE "dumper.lisp" #<COMPILED-LEXICAL-CLOSURE (:INTERNAL BUILDAPP::MAIN) #x302002BF2ACF>)
2: (BUILDAPP::MAIN ("sbcl" "--asdf-path" "/Users/joshuakordani/quicklisp/dists/quicklisp/software/buildapp-1.5.4/" "--load-system" "buildapp" "--entry" ...))
3: (CCL::CALL-CHECK-REGS BUILDAPP:BUILD-BUILDAPP)
4: (CCL::CHEAP-EVAL (BUILDAPP:BUILD-BUILDAPP))
5: (SWANK::EVAL-REGION "(buildapp:build-buildapp)\n")
6: ((:INTERNAL SWANK-REPL::REPL-EVAL))
7: (SWANK-REPL::TRACK-PACKAGE #<COMPILED-LEXICAL-CLOSURE (:INTERNAL SWANK-REPL::REPL-EVAL) #x302002A706BF>)
8: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation request." #<COMPILED-LEXICAL-CLOSURE (:INTERNAL SWANK-REPL::REPL-EVAL) #x302002A7073F>)
9: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<COMPILED-LEXICAL-CLOSURE (:INTERNAL SWANK-REPL::REPL-EVAL) #x302002A7077F>)
10: (SWANK-REPL::REPL-EVAL "(buildapp:build-buildapp)\n")
11: (CCL::CALL-CHECK-REGS SWANK-REPL:LISTENER-EVAL "(buildapp:build-buildapp)\n")
12: (CCL::CHEAP-EVAL (SWANK-REPL:LISTENER-EVAL "(buildapp:build-buildapp)\n"))
13: (SWANK:EVAL-FOR-EMACS (SWANK-REPL:LISTENER-EVAL "(buildapp:build-buildapp)\n") "COMMON-LISP-USER" 144)
14: (SWANK::PROCESS-REQUESTS NIL)
15: ((:INTERNAL SWANK::HANDLE-REQUESTS))
16: ((:INTERNAL SWANK::HANDLE-REQUESTS))
17: (SWANK/BACKEND:CALL-WITH-DEBUGGER-HOOK #<Compiled-function SWANK:SWANK-DEBUGGER-HOOK #x3020008CA55F> #<COMPILED-LEXICAL-CLOSURE (:INTERNAL SWANK::HANDLE-REQUESTS) #x302000A88B7F>)
18: (SWANK::CALL-WITH-BINDINGS ((STANDARD-OUTPUT . #<SWANK/GRAY::SLIME-OUTPUT-STREAM #x302000A87B1D>) (STANDARD-INPUT . #<SWANK/GRAY::SLIME-INPUT-STREAM #x302000A87E8D>) ..))) #<COMPILED-LEXICAL-CLOSU..
19: (SWANK::HANDLE-REQUESTS #<MULTITHREADED-CONNECTION #x30200098D27D> NIL)
20: (CCL::RUN-PROCESS-INITIAL-FORM #<PROCESS repl-thread(14) [Active] #x302000A76D6D> (#<COMPILED-LEXICAL-CLOSURE (:INTERNAL CCL::%PROCESS-RUN-FUNCTION) #x302000A76AFF>))
21: ((:INTERNAL (CCL::%PROCESS-PRESET-INTERNAL (PROCESS))) #<PROCESS repl-thread(14) [Active] #x302000A76D6D> (#<COMPILED-LEXICAL-CLOSURE (:INTERNAL CCL::%PROCESS-RUN-FUNCTION) #x302000A76AFF>))
22: ((:INTERNAL CCL::THREAD-MAKE-STARTUP-FUNCTION))

See the following debug output, it almost looks like the working directory isn't being set in sbcl.

$ sbcl --version
SBCL 1.3.15

$ make clean
rm -f buildapp *~ *.fasl *.lx32fsl

$make
sbcl --noinform --no-userinit --no-sysinit --disable-debugger \
  --eval "(require 'asdf)" \
  --eval "(push \"$(pwd)/\" asdf:*central-registry*)" \
  --eval "(require 'buildapp)" \
          --eval "(buildapp::build-buildapp)" \
          --eval "#+sbcl (exit) #+ccl (quit)"
fatal error encountered in SBCL pid 51157(tid 0x7ffff7fd5fc0):
can't find core file at NIL/sbcl.core


make: *** [Makefile:11: buildapp] Error 1

I tested building buildapp on CCL and it failed with "Permission denied: '/buildapp'". Complete output here: http://paste.lisp.org/display/138728

To me it seems that NATIVE-NAMESTRING binds default-pathname-defaults to #P"/".

I may have broken something in my branch (see #39). Now when I compile dispatched-entry-form, I am getting this compilation note:

; file: /Users/rpg/lisp/buildapp/dumper.lisp
; in: DEFUN DISPATCHED-ENTRY-FORM
;     (MACROEXPAND-1 (BUILDAPP::QUIT 1))
; 
; note: The first argument never returns a value.

;     (LIST
;      `(FORMAT *ERROR-OUTPUT* "Unknown dispatch name '~A', quitting~%"
;               BUILDAPP::BINARY-NAME)
;      (MACROEXPAND-1 (BUILDAPP::QUIT 1)))
; 
; note: deleting unreachable code

Any idea what's causing this? Is it a problem? I'm not even sure what "The first argument" refers to:

• First argument to macroexpand-1: (BUILDAPP::QUIT 1)?
• First argument to list: `(FORMAT *ERROR-OUTPUT* "Unknown dispatch name '~A', quitting~%" BUILDAPP::BINARY-NAME)?

Or is it just that we're missing a quote in front of the argument to macroexpand-1?

It'd be nice if buildapp could run some (declaim (optimize)) depending on the build.

Thoughts?

On CCL1.10 on Windows XP 32b, (buildapp:build-buildapp) fails. run-program is called as follows:
(run-program "ccl" '(...))

If run-program is called like below, it works.
(run-program "cmd" '("/c" "ccl" ...))

Take the version from the system file and display it when given a --version option.

There seems to be a --sbcl option to set which program is invoked for sbcl, but no corresponding --ccl option to set the ccl binary. It seems hardcoded to "ccl".

This is a problem if you really want to use a different binary to e.g. produce 64 vs 32 bit executables.

osx 10.10 ccl64 build from trunk, make line was make DESTDIR="path/to/buildapp" LISP="path/to/dx86cl64" install

note hardcoded use of sbcl at frame 14.

(20E7C6A0) : 0 (>-2 NIL 0) 2293
(20E7C6D0) : 1 (<=-2 NIL 0) 29
(20E7C6F0) : 2 (FUNCALL #'#<#<CCL::STANDARD-KERNEL-METHOD PRINT-OBJECT (STANDARD-OBJECT T)>> #<SILENT-EXIT-ERROR #x302000D5F3FD> #<STRING-OUTPUT-STREAM #x302000D5EC5D>) 69
(20E7C710) : 3 (%CALL-NEXT-METHOD ((#) #<CCL::STANDARD-KERNEL-METHOD PRINT-OBJECT #> #<SILENT-EXIT-ERROR #x302000D5F3FD> #<STRING-OUTPUT-STREAM #x302000D5EC5D>)) 869
(20E7C790) : 4 (FUNCALL #'#<(:INTERNAL (PRINT-OBJECT (CONDITION T)))>) 221
(20E7C7B8) : 5 (%CALL-NEXT-METHOD ((#) #<CCL::STANDARD-KERNEL-METHOD PRINT-OBJECT #> #<SILENT-EXIT-ERROR #x302000D5F3FD> #<STRING-OUTPUT-STREAM #x302000D5EC5D>)) 869
(20E7C838) : 6 (FUNCALL #'#<(:INTERNAL (PRINT-OBJECT :AROUND (T T)))>) 221
(20E7C860) : 7 (%%CNM-WITH-ARGS-COMBINED-METHOD-DCODE #<SIMPLE-VECTOR 5> 69007642 NIL) 1029
(20E7C8E8) : 8 (FUNCALL #'#<#<CCL::STANDARD-KERNEL-METHOD CCL::REPORT-CONDITION (CONDITION T)>> #<SILENT-EXIT-ERROR #x302000D5F3FD> #<STRING-OUTPUT-STREAM #x302000D5EC5D>) 237
(20E7C928) : 9 (%BREAK-MESSAGE "Error" #<SILENT-EXIT-ERROR #x302000D5F3FD> 69007728 #>) 1101
(20E7CAF0) : 10 (BREAK-LOOP-HANDLE-ERROR #<SILENT-EXIT-ERROR #x302000D5F3FD> 69007728) 701
(20E7CB58) : 11 (%ERROR #<SILENT-EXIT-ERROR #x302000D5F3FD> NIL 69007728) 333
(20E7CB80) : 12 (FUNCALL #'#<(:INTERNAL MAIN)> #<BASIC-FILE-CHARACTER-OUTPUT-STREAM ("dumper-f0dIhgSY.lisp"/4 UTF-8) #x302000D76E5D> #P"dumper-f0dIhgSY.lisp") 597
(20E7CBB0) : 13 (CALL-WITH-TEMPORARY-OPEN-FILE "dumper.lisp" #<CCL:COMPILED-LEXICAL-CLOSURE (:INTERNAL MAIN) #x302000D777AF>) 781
(20E7CC48) : 14 (MAIN ("sbcl" "--asdf-path" "/Users/joshuakordani/Source/buildapp/" "--load-system" "buildapp" ...)) 461
(20E7CC88) : 15 (CALL-CHECK-REGS BUILD-BUILDAPP) 221
(20E7CCC0) : 16 (CHEAP-EVAL (BUILD-BUILDAPP)) 101
(20E7CCF8) : 17 (FUNCALL #'#<(:INTERNAL CCL::EVAL-STRING CCL::STARTUP-CCL)> "(buildapp::build-buildapp)") 485
(20E7CD40) : 18 (STARTUP-CCL NIL) 1541
(20E7CDA8) : 19 (FUNCALL #'#<(:INTERNAL (CCL:TOPLEVEL-FUNCTION (CCL::LISP-DEVELOPMENT-SYSTEM T)))>) 61
(20E7CDC8) : 20 (FUNCALL #'#<(:INTERNAL CCL::MAKE-MCL-LISTENER-PROCESS)>) 661
(20E7CE60) : 21 (RUN-PROCESS-INITIAL-FORM #<TTY-LISTENER listener(1) [Active] #x3020004476CD> (#<CCL:COMPILED-LEXICAL-CLOSURE # #x3020004471FF>)) 669
(20E7CEF0) : 22 (FUNCALL #'#<(:INTERNAL (CCL::%PROCESS-PRESET-INTERNAL (CCL:PROCESS)))> #<TTY-LISTENER listener(1) [Active] #x3020004476CD> (#<CCL:COMPILED-LEXICAL-CLOSURE # #x3020004471FF>)) 573
(20E7CF98) : 23 (FUNCALL #'#<(:INTERNAL CCL::THREAD-MAKE-STARTUP-FUNCTION)>) 277

After performing a write test, delete the write test file. Otherwise it can look like a successful build.

Hi, I'm trying to build a skeleton Caveman app with the following:

(caveman.skeleton:generate #p"~/Lisp/test")

buildapp --output test-build \
--asdf-tree ~/quicklisp/dists/quicklisp/software/ \
--asdf-path ~/Lisp/test/ 
--load-system test \
--entry test:start

but get the following error:

;; loading system ...
;; ...

Fatal SIMPLE-PACKAGE-ERROR:
  The name "QUICKLISP-CLIENT" does not designate any package.

I can't seem to find where this comes from (neither asdf-path/tree include a mention of grep -nr quicklisp-client .)

Additional flag for verbose compilation log output?

Running make gives the following:

make
sbcl --noinform --no-userinit --no-sysinit --disable-debugger \
  --eval "(require 'asdf)" \
  --eval "(push \"$(pwd)/\" asdf:*central-registry*)" \
  --eval "(require 'buildapp)" \
          --eval "(buildapp::build-buildapp)" \
          --eval "#+sbcl (exit) #+ccl (quit)"
Unhandled SB-INT:EXTENSION-FAILURE in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                                {10013D0613}>:
  Don't know how to REQUIRE BUILDAPP.

Explicitly loading the ASDF in Makefile fixes it: --eval "(progn (load \"buildapp.asd\") (asdf:load-system :buildapp))".

When I do (ql:quickload "buildapp") I get the following error:

The value #P"buildapp.exe" is not of the expected type SEQUENCE.
  [Condition of type TYPE-ERROR]

It looks like it is caused by this form in buildapp.lisp
(defparameter *output-type-pathname* ....)

I am using CCL "Version 1.10-r16196 (WindowsX8632)" on Windows XP 32b.

Buildapp's dumper script currently sets sb-ext:*invoke-debugger-hook* to NIL as one of its final steps. The intent is that user code should initialize the value at application startup. However, SBCL can fail and enter the debugger before any of the init hooks or custom toplevel functions are called.

One strategy to avoid this problem is to wrap all loading operations in a form that binds *invoke-dynamic-hook*, and checks to see whether it's modified as a result of the load, and then saves that modified value at the end of the dumper script.

Per tcr, it would be nice to have buildapp store fasls in a given directory to be blown away at will, separate from the user global ~/.cache directory.

Allow the use of a manifest file, where each system file is listed, one per line, with preferred versions of a library listed first.

This should go hand-in-hand with a command in Quicklisp to emit a manifest file.

As far as I can tell, sbcl supports 1-9 as a compression level if built with :SB-CORE-COMPRESSION feature, but buildapp does not allow for setting the compression level.

Could the --compress-core flag accept an (optional) compression level?