Coder Social home page Coder Social logo

packr's Introduction

NOTICE: Please consider migrating your projects to embed which is native file embedding feature of Go, or github.com/markbates/pkger. It has an idiomatic API, minimal dependencies, a stronger test suite (tested directly against the std lib counterparts), transparent tooling, and more.

https://blog.gobuffalo.io/introducing-pkger-static-file-embedding-in-go-1ce76dc79c65

Packr (v2)

GoDoc Actions Status

Packr is a simple solution for bundling static assets inside of Go binaries. Most importantly it does it in a way that is friendly to developers while they are developing.

At this moment, supported go versions are:

  • 1.16.x
  • 1.17.x

even though it may (or may not) working well with older versions.

Intro Video

To get an idea of the what and why of Packr, please enjoy this short video: https://vimeo.com/219863271.

Library Installation

Go 1.16 and above

$ go install github.com/gobuffalo/packr/[email protected]

or

$ go install github.com/gobuffalo/packr/v2@latest

Go 1.15 and below

$ go get -u github.com/gobuffalo/packr/...

Binary Installation

Go 1.16 and above

$ go install github.com/gobuffalo/packr/v2/[email protected]

or

$ go install github.com/gobuffalo/packr/v2/packr2@latest

Go 1.15 and below

$ go get -u github.com/gobuffalo/packr/packr2

New File Format FAQs

In version v2.0.0 the file format changed and is not backward compatible with the packr-v1.x library.

Can packr-v1.x read the new format?

No, it can not. Because of the way the new file format works porting it to packr-v1.x would be difficult. PRs are welcome though. :)

Can packr-v2.x read packr-v1.x files?

Yes it can, but that ability will eventually be phased out. Because of that we recommend moving to the new format.

Can packr-v2.x generate packr-v1.x files?

Yes it can, but that ability will eventually be phased out. Because of that we recommend moving to the new format.

The --legacy command is available on all commands that generate -packr.go files.

$ packr2 --legacy

Usage

In Code

The first step in using Packr is to create a new box. A box represents a folder on disk. Once you have a box you can get string or []byte representations of the file.

// set up a new box by giving it a name and an optional (relative) path to a folder on disk:
box := packr.New("My Box", "./templates")

// Get the string representation of a file, or an error if it doesn't exist:
html, err := box.FindString("index.html")

// Get the []byte representation of a file, or an error if it doesn't exist:
html, err := box.Find("index.html")

What is a Box?

A box represents a folder, and any sub-folders, on disk that you want to have access to in your binary. When compiling a binary using the packr2 CLI the contents of the folder will be converted into Go files that can be compiled inside of a "standard" go binary. Inside of the compiled binary the files will be read from memory. When working locally the files will be read directly off of disk. This is a seamless switch that doesn't require any special attention on your part.

Example

Assume the follow directory structure:

├── main.go
└── templates
    ├── admin
    │   └── index.html
    └── index.html

The following program will read the ./templates/admin/index.html file and print it out.

package main

import (
  "fmt"

  "github.com/gobuffalo/packr/v2"
)

func main() {
  box := packr.New("myBox", "./templates")

  s, err := box.FindString("admin/index.html")
  if err != nil {
    log.Fatal(err)
  }
  fmt.Println(s)
}

Development Made Easy

In order to get static files into a Go binary, those files must first be converted to Go code. To do that, Packr, ships with a few tools to help build binaries. See below.

During development, however, it is painful to have to keep running a tool to compile those files.

Packr uses the following resolution rules when looking for a file:

  1. Look for the file in-memory (inside a Go binary)
  2. Look for the file on disk (during development)

Because Packr knows how to fall through to the file system, developers don't need to worry about constantly compiling their static files into a binary. They can work unimpeded.

Packr takes file resolution a step further. When declaring a new box you use a relative path, ./templates. When Packr receives this call it calculates out the absolute path to that directory. By doing this it means you can be guaranteed that Packr can find your files correctly, even if you're not running in the directory that the box was created in. This helps with the problem of testing, where Go changes the pwd for each package, making relative paths difficult to work with. This is not a problem when using Packr.


Usage with HTTP

A box implements the http.FileSystem interface, meaning it can be used to serve static files.

package main

import (
	"net/http"

	"github.com/gobuffalo/packr/v2"
)

func main() {
	box := packr.New("someBoxName", "./templates")

	http.Handle("/", http.FileServer(box))
	http.ListenAndServe(":3000", nil)
}

Building a Binary

Before you build your Go binary, run the packr2 command first. It will look for all the boxes in your code and then generate .go files that pack the static files into bytes that can be bundled into the Go binary.

$ packr2

Then run your go build command like normal.

NOTE: It is not recommended to check-in these generated -packr.go files. They can be large, and can easily become out of date if not careful. It is recommended that you always run packr2 clean after running the packr2 tool.

Cleaning Up

When you're done it is recommended that you run the packr2 clean command. This will remove all of the generated files that Packr created for you.

$ packr2 clean

Why do you want to do this? Packr first looks to the information stored in these generated files, if the information isn't there it looks to disk. This makes it easy to work with in development.


Debugging

The packr2 command passes all arguments down to the underlying go command, this includes the -v flag to print out go build information. Packr looks for the -v flag, and will turn on its own verbose logging. This is very useful for trying to understand what the packr command is doing when it is run.


FAQ

Compilation Errors with Go Templates

Q: I have a program with Go template files, those files are named foo.go and look like the following:

// Copyright {{.Year}} {{.Author}}. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package {{.Project}}

When I run packr2 I get errors like:

expected 'IDENT', found '{'

A: Packr works by searching your .go files for github.com/gobuffalo/packr/v2#New or github.com/gobuffalo/packr/v2#NewBox calls. Because those files aren't "proper" Go files, Packr can't parse them to find the box declarations. To fix this you need to tell Packr to ignore those files when searching for boxes. A couple solutions to this problem are:

  • Name the files something else. The .tmpl extension is the idiomatic way of naming these types of files.
  • Rename the folder containing these files to start with an _, for example _templates. Packr, like Go, will ignore folders starting with the _ character when searching for boxes.

Dynamic Box Paths

Q: I need to set the path of a box using a variable, but packr.New("foo", myVar) doesn't work correctly.

A: Packr attempts to "automagically" set it's resolution directory when using github.com/gobuffalo/packr/v2#New, however, for dynamic paths you need to set it manually:

box := packr.New("foo", "|")
box.ResolutionDir = myVar

I don't want to pack files, but still use the Packr interface.

Q: I want to write code that using the Packr tools, but doesn't actually pack the files into my binary. How can I do that?

A: Using github.com/gobuffalo/packr/v2#Folder gives you back a *packr.Box that can be used as normal, but is excluded by the Packr tool when compiling.

Packr Finds No Boxes

Q: I run packr2 -v but it doesn't find my boxes:

DEBU[2019-03-18T18:48:52+01:00] *parser.Parser#NewFromRoots found prospects=0
DEBU[2019-03-18T18:48:52+01:00] found 0 boxes

A: Packr works by parsing .go files to find github.com/gobuffalo/packr/v2#Box and github.com/gobuffalo/packr/v2#NewBox declarations. If there aren't any .go in the folder that packr2 is run in it can not find those declarations. To fix this problem run the packr2 command in the directory containing your .go files.

Box Interfaces

Q: I want to be able to easily test my applications by passing in mock boxes. How do I do that?

A: Packr boxes and files conform to the interfaces found at github.com/gobuffalo/packd. Change your application to use those interfaces instead of the concrete Packr types.

// using concrete type
func myFunc(box *packr.Box) {}

// using interfaces
func myFunc(box packd.Box) {}

packr's People

Contributors

bastiankoetsier avatar bitspill avatar caarlos0 avatar claushellsing avatar dinedal avatar domenipavec avatar ghatwala avatar hdm avatar ialidzhikov avatar jasonish avatar johanbrandhorst avatar markbates avatar meowsbits avatar metalmatze avatar misha-plus avatar paganotoni avatar pamburus avatar rickyshrestha avatar sagikazarmark avatar sebholstein avatar selfup avatar sio4 avatar spenczar avatar stanislas-m avatar stevexuereb avatar techknowlogick avatar tfeng avatar webrat avatar williamhgough avatar zbindenren avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

packr's Issues

Support for prefix of embedded files

I am trying to move from go-bindata since its unsupported, that project has a flag in the CLI that allows you to add a prefix to all the files that are embedded so you can do something like this:

router.Handler("GET", "/static/*filepath", fileServer)

Right now this works:

staticBox := packr.NewBox("../dist")
fileServer := http.FileServer(staticBox)
router.Handler("GET", "/*filepath", fileServer)

But I have to serve from / and I have other routes there.

The following doesn't work because it will look for a file under ../dist/static/app.js the file that exists is ../dist/app.js.

staticBox := packr.NewBox("../dist")
fileServer := http.FileServer(staticBox)
router.Handler("GET", "/static/*filepath", fileServer)

What I want is something like:

staticBox := packr.NewBox("../dist", "static")
fileServer := http.FileServer(staticBox)
router.Handler("GET", "/static/*filepath", fileServer)

Maybe this is something already exists that I could not find or maybe you can point me how to write a small function that changes the routes of the embedded files inside a box, not a go expert :)

panic: runtime error: invalid memory address or nil pointer dereference

(GO noob here)
I'm trying to follow some instructions to build a go project. (I'm running Ubuntu 17.10 , go 1.10)
I set my GOPATH to a new workspace directory.
Then, I installed packr using the instructions found in this project:

go get -u github.com/gobuffalo/packr/...

And when I try to run packr I get a segmentation fault:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x6df339]

goroutine 633 [running]:
github.com/gobuffalo/packr/builder.(*visitor).eval(0xc4206cb810, 0x81a600, 0xc4204ea090, 0x0, 0x0)
  /.../gorkspace/src/github.com/gobuffalo/packr/builder/visitor.go:88 +0x69
github.com/gobuffalo/packr/builder.(*visitor).Visit(0xc4206cb810, 0x81a600, 0xc4204ea090, 0x818540, 0xc4206cb810)
  /.../gorkspace/src/github.com/gobuffalo/packr/builder/visitor.go:69 +0x4c
go/ast.Walk(0x818540, 0xc4206cb810, 0x81a600, 0xc4204ea090)
  /usr/local/go/src/go/ast/walk.go:52 +0x66
go/ast.walkDeclList(0x818540, 0xc4206cb810, 0xc420478600, 0xb, 0x10)
  /usr/local/go/src/go/ast/walk.go:38 +0x81
go/ast.Walk(0x818540, 0xc4206cb810, 0x81a580, 0xc420607200)
  /usr/local/go/src/go/ast/walk.go:353 +0x2650
github.com/gobuffalo/packr/builder.(*visitor).Run(0xc4206cb810, 0x9f86e8, 0x5a)
  /.../gorkspace/src/github.com/gobuffalo/packr/builder/visitor.go:42 +0x1e0
github.com/gobuffalo/packr/builder.(*Builder).process(0xc4200aa460, 0xc42040d0e0, 0x5a, 0xc420034448, 0xc4203eaf90)
  /.../gorkspace/src/github.com/gobuffalo/packr/builder/builder.go:90 +0x175
github.com/gobuffalo/packr/builder.(*Builder).Run.func1.1(0xc400000008, 0x7ed968)
  /.../gorkspace/src/github.com/gobuffalo/packr/builder/builder.go:48 +0x3c
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc420034440, 0xc4204021c0)
  /.../gorkspace/src/golang.org/x/sync/errgroup/errgroup.go:58 +0x57
created by golang.org/x/sync/errgroup.(*Group).Go
  /.../gorkspace/src/golang.org/x/sync/errgroup/errgroup.go:55 +0x66

Question: `packr.Box` Safe for concurrent access?

I've tested concurrent box access with race detection and load testing, and the answer to this question seems to be yes.

Could you add a line in the docs regarding a box's safety for concurrent access? Or best practices for using boxes? Are they extremely cheap to make, or should they be passed around via pointers? I'm relatively new as a go dev, so this sort of info is very helpful. Thank you!

Remove prints from builder

I have incorporated packr into my build system, however, the builder prints messages to stdout.

It would be great if there was a way to stop making the builder print to stdout.

go fmt generated code

What do you think of running go fmt on the generated code? We exclude linting on generated code but our go fmt check currently does not exclude generated code.

packr tool cannot find packr.NewBox

package main

import (
	"github.com/gobuffalo/packr"
)

type S struct{}

func (S) f(packr.Box) {}

func main() {
	s := S{}
	s.f(packr.NewBox("test"))
}

Packr is not generating main-packr.go because it cannot find packr.NewBox.

Temporary solution: assign value of function packr.NewBox to variable and use that variable in s.F.

Cross compilation to windows

Hi,

box := packr.NewBox("./static") html, err = box.MustString("index.html")
When compiling everything looks ok with the verbose setting and index.html being bundled, but when I move the exe from my C9 enviroment on AWS Linux to Windows. I get the following error:

ERROR: FindFirstFile c:\users\jxxxx\go\src\home\ec2-user\enviroment\ngram\static\index.html: The system cannot find the path specified.

Note the path used is not my current directory, it seems to be a mashup of three different paths. I recognize the first part as home on the windows computer, the second path is maybe the home of go on linux, and the third path is the absolute path on C9 AWS Linux enviroment where I compiled the .exe file.

GOOS=windows GOARCH=amd64 packr -v build -o explore.exe explore.go

Any ideas ?

Issue with packr in Docker

Hey 👋

I'm having some trouble with getting it to run in my Docker container, I read through other issues that were dealing with Docker but these didn't help me in this regard.

Building and running locally:

This works fine and everything is as expected, I'm printing List() and Path here in my / Handler to show the problem. The Box is filled in this case.

feedbridge|feature/ui⚡ ⇒ packr build
feedbridge|feature/ui⚡ ⇒ ./feedbridge
ts=2018-08-22T20:13:26.12318744Z caller=main.go:106 msg="feedbridge listening on http://localhost:8080"
[index.tmpl]
./ui/templates
[styles.css tachyons.min.css icon.png]
./ui/assets
^C

Building Multi-Stage Docker image and run packr build -v:

I'm building an image and use packr build -v to show the list of files. It also looks like the Box is packed correctly, at least the files seem to be found by packr. Once I ran it with my docker-compose file the Boxes seem to be empty though.

I'm looking at this for a while now and can't figure it out. Any hints?

Thanks!

feedbridge|feature/ui⚡ ⇒ docker build -t feedbridge-test .
Sending build context to Docker daemon  145.1MB
Step 1/10 : FROM golang:1.10-alpine as builder
 ---> 34d3217973fd
Step 2/10 : WORKDIR $GOPATH/src/github.com/dewey/feedbridge
 ---> Using cache
 ---> 0ccba0adcbf3
Step 3/10 : ADD ./ $GOPATH/src/github.com/dewey/feedbridge
 ---> e0183c6f9ce9
Step 4/10 : RUN apk update &&     apk upgrade &&     apk add git
 ---> Running in de32cef4cc23
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
v3.8.0-88-g3e3519a996 [http://dl-cdn.alpinelinux.org/alpine/v3.8/main]
v3.8.0-88-g3e3519a996 [http://dl-cdn.alpinelinux.org/alpine/v3.8/community]
OK: 9542 distinct packages available
Upgrading critical system libraries and apk-tools:
(1/1) Upgrading apk-tools (2.10.0-r0 -> 2.10.0-r3)
Executing busybox-1.28.4-r0.trigger
Continuing the upgrade transaction with new apk-tools:
(1/2) Upgrading busybox (1.28.4-r0 -> 1.28.4-r1)
Executing busybox-1.28.4-r1.post-upgrade
(2/2) Upgrading ssl_client (1.28.4-r0 -> 1.28.4-r1)
Executing busybox-1.28.4-r1.trigger
OK: 5 MiB in 14 packages
(1/6) Installing nghttp2-libs (1.32.0-r0)
(2/6) Installing libssh2 (1.8.0-r3)
(3/6) Installing libcurl (7.61.0-r0)
(4/6) Installing expat (2.2.5-r0)
(5/6) Installing pcre2 (10.31-r0)
(6/6) Installing git (2.18.0-r0)
Executing busybox-1.28.4-r1.trigger
OK: 19 MiB in 20 packages
Removing intermediate container de32cef4cc23
 ---> e7c1d3652d58
Step 5/10 : RUN go get -u github.com/gobuffalo/packr/... &&     packr build -v -o /feedbridge ./main.go
 ---> Running in f984f5dd48ea
building box ./ui/assets
packing file icon.png
packed file icon.png
packing file styles.css
packed file styles.css
packing file tachyons.min.css
packed file tachyons.min.css
built box ./ui/assets with ["icon.png" "styles.css" "tachyons.min.css"]
building box ./ui/templates
packing file index.tmpl
packed file index.tmpl
built box ./ui/templates with ["index.tmpl"]
github.com/dewey/feedbridge/vendor/github.com/caarlos0/env
github.com/dewey/feedbridge/vendor/github.com/gorilla/feeds
github.com/dewey/feedbridge/vendor/github.com/patrickmn/go-cache
github.com/dewey/feedbridge/plugin
github.com/dewey/feedbridge/vendor/github.com/go-chi/chi
github.com/dewey/feedbridge/store
github.com/dewey/feedbridge/vendor/github.com/go-chi/cors
github.com/dewey/feedbridge/vendor/github.com/go-logfmt/logfmt
github.com/dewey/feedbridge/vendor/github.com/go-stack/stack
github.com/dewey/feedbridge/vendor/golang.org/x/net/html/atom
github.com/dewey/feedbridge/vendor/github.com/go-kit/kit/log
github.com/dewey/feedbridge/vendor/golang.org/x/net/html
github.com/dewey/feedbridge/vendor/github.com/go-kit/kit/log/level
github.com/dewey/feedbridge/api
github.com/dewey/feedbridge/vendor/github.com/beorn7/perks/quantile
github.com/dewey/feedbridge/vendor/github.com/golang/protobuf/proto
github.com/dewey/feedbridge/vendor/github.com/andybalholm/cascadia
github.com/dewey/feedbridge/vendor/github.com/PuerkitoBio/goquery
github.com/dewey/feedbridge/scrape
github.com/dewey/feedbridge/plugins/scmp
github.com/dewey/feedbridge/vendor/github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg
github.com/dewey/feedbridge/vendor/github.com/prometheus/common/model
github.com/dewey/feedbridge/vendor/github.com/prometheus/procfs/internal/util
github.com/dewey/feedbridge/vendor/github.com/prometheus/procfs/nfs
github.com/dewey/feedbridge/vendor/github.com/prometheus/client_model/go
github.com/dewey/feedbridge/vendor/github.com/matttproud/golang_protobuf_extensions/pbutil
github.com/dewey/feedbridge/vendor/github.com/prometheus/procfs/xfs
github.com/dewey/feedbridge/vendor/github.com/prometheus/common/expfmt
github.com/dewey/feedbridge/vendor/github.com/prometheus/procfs
github.com/dewey/feedbridge/vendor/github.com/pkg/errors
github.com/dewey/feedbridge/vendor/github.com/prometheus/client_golang/prometheus
github.com/dewey/feedbridge/vendor/github.com/gobuffalo/packr
github.com/dewey/feedbridge/runner
github.com/dewey/feedbridge/vendor/github.com/prometheus/client_golang/prometheus/promhttp
command-line-arguments
Removing intermediate container f984f5dd48ea
 ---> 34867dbf820b
Step 6/10 : FROM alpine:latest
 ---> 11cd0b38bc3c
Step 7/10 : RUN apk --no-cache add ca-certificates
 ---> Using cache
 ---> a5d3e1451438
Step 8/10 : WORKDIR /root/
 ---> Using cache
 ---> 5953534e3d63
Step 9/10 : COPY --from=builder /feedbridge /feedbridge
 ---> c0290144e21f
Step 10/10 : CMD ["/feedbridge"]
 ---> Running in a43801c8ad61
Removing intermediate container a43801c8ad61
 ---> cb3b6e99852e
Successfully built cb3b6e99852e
Successfully tagged feedbridge-test:latest
feedbridge|feature/ui⚡ ⇒ docker tag feedbridge-test:latest feedbridge-test:staging
feedbridge|feature/ui⚡ ⇒ docker-compose -f docker-compose.yml up
Recreating feedbridge_app_1 ... done
Attaching to feedbridge_app_1
app_1  | ts=2018-08-22T20:17:45.6340962Z caller=main.go:106 msg="feedbridge listening on http://localhost:8080"
app_1  | []
app_1  | ./ui/templates
app_1  | []
app_1  | ./ui/assets
Killing feedbridge_app_1    ... done

Update:

I just tried to look into the both containers to see how the binary in there is like.

The intermediate container used to build the binary with packr build seems to work correctly, the binary has the assets in there.

feedbridge|feature/ui⚡ ⇒ docker run -p 127.0.0.1:8080:8080 -it 3940cc50cf37 /bin/ash
/go/src/github.com/dewey/feedbridge # cd /
/ # ./feedbridge
ts=2018-08-22T21:22:18.5301751Z caller=main.go:106 msg="feedbridge listening on http://localhost:8080"
[index.tmpl]
./ui/templates
[icon.png styles.css tachyons.min.css]
./ui/assets
[index.tmpl]
./ui/templates
[icon.png styles.css tachyons.min.css]
./ui/assets
^C

But now comes the weird part, once I enter the second container, the one where it should just copy over the binary from the first container it doesn't seem to have them? That seems very odd. Almost like it didn't actually bake them into the binary in the first container and is somehow using them from disk?

feedbridge|feature/ui⚡ ⇒ docker run -p 127.0.0.1:8080:8080 -it cb3b6e99852e /bin/ash
~ # cd /
/ # ls
bin         etc         home        media       proc        run         srv         tmp         var
dev         feedbridge  lib         mnt         root        sbin        sys         usr
/ # ./feedbridge
ts=2018-08-22T21:24:04.7353837Z caller=main.go:106 msg="feedbridge listening on http://localhost:8080"
[]
./ui/templates
[]
./ui/assets
[]
./ui/templates
[]
./ui/assets
^C

Verbose not verbose enough

When set to verbose mode and packr finds itself but doesn't know how to interpret the path it doesn't say anything. This is not nice DX as you don't know what's wrong. You just stare at the blank screen.

IMO even in non verbose mode at the end should be report found X boxes crated Y, run with -v to find out more....

Packr builds & "internal" directories

Hi All,

i've just had the hardest time resolving my own issue trying to use packr + internal directories in my go project.

The issue was compounded by the fact that there isn't any error messaging lending itself to the issue. In fact, the only method i was able to use to resolve this was trail and error with the -v flag.

I have a directory structure where i'm using go's built-in feature of using an "internal" directory to keep internal app code private & local to the app. Inside this directory, i created a static file folder and put a few json files in there.

However, when i built the app with 'packr build', it did not include any static files within the "internal" directory. I'm not sure if this is a limitation due to golang internals, but, it did cause me some pain for 24 hrs.

Either way, error messaging or fixing the problem would be greatly appreciated.

conflict/override problem when same box name & path inside different module

├── main.go
└── module_a
     └── model
└── module_b
     └──model

both module_a & module_b have some code like following .

modelBox := packr.NewBox("./model")
modelBytes := modelBox.MustBytes("model.pb")

It works in development mode . (go build )
But when using packr build , modelBytes become same in different module ..
Because the box name & path is identical , the generated code in one module will override another ..

Windows file path

In windows box.go find don't handle correctly file path.

Example :

Used template name - "auth/login.html"
Packr stored name - "auth/login.html"

In find func( name = "auth/login.html" )
filepath.ToSlash - "auth/login.html" - Ok
filepath.Clean
Using os specific path separator , returns wrong name "auth\login.html"
and exit with file not found

Build errors on Alpine Linux

I'm trying to get packr working on alpine linux in a docker container.

After I wget it from the releases page, I put it in my PATH.

When running packer build -v app ., I get this error:

/go/bin/packr: line 1: S��8���: not found
: not foundkr: line 1: ���
/go/bin/packr: line 2: z�L���Ap�M: not found
/go/bin/packr: line 2: syntax error: unexpected word (expecting ")")

In alpine, before I run packr I'm running apk add --update --no-cache alpine-sdk linux-headers wget... am I missing any other packages? Should I not be building on alpine?

go generate?

Looks like a nice replacement for go-bindata! But it seems to not allow you to use it with go:generate, unless that wasn't the intent. It'd be awesome to just drop // go:generate packr in there.

go-bindata had a few weird defaults so you always have to specify flags, I usually end up doing this if checked into source:

//go:generate go-bindata -ignore bindata.go -modtime 0 -pkg NAME .

or I guess just:

//go:generate go-bindata -ignore bindata.go -pkg NAME .

if it's not checked into GIT, but I think the CWD and name could be inferred easily and would be a nicer experience

cheers!

how to test?

hi,

im interested to help debug and solve issues here, but how do i run test?

FileServer doesn't serve embeded index.html from /

package main

import (
	"net/http"

	"github.com/gobuffalo/packr"
)

func main() {
	http.ListenAndServe("localhost:1234", http.FileServer(packr.NewBox("public")))
}

Index.html not served when is embedded. It returns error '404 page not found'.

Steps:

  1. Run packr build
  2. Delete folder public
  3. Run builded exe
  4. Open in browser http://localhost:1234
  5. 404 page not found

os: Windows 7 x64

Panic when given a relative non-existing dir

Hi!

This is a very minor issue but I still thought I'd report it : when given a relative non-existing path, packr panics :

packr install ./dirDoesNotExists
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x56eeec]

goroutine 1 [running]:
github.com/gobuffalo/packr/builder.(*Builder).Run.func1(0x7ffd60ceec18, 0x12, 0x0, 0x0, 0x72cae0, 0xc4200108d0, 0x20, 0x5f5b60)
        /home/hectorj/go/src/github.com/gobuffalo/packr/builder/builder.go:38 +0xcc
path/filepath.Walk(0x7ffd60ceec18, 0x12, 0xc42000cc80, 0x411168, 0x10)
        /home/hectorj/.local/share/umake/go/go-lang/src/path/filepath/path.go:401 +0x76
github.com/gobuffalo/packr/builder.(*Builder).Run(0xc4200761e0, 0xc42001e3c0, 0xc4200108a0)
        /home/hectorj/go/src/github.com/gobuffalo/packr/builder/builder.go:32 +0x9f
github.com/gobuffalo/packr/packr/cmd.glob..func3(0x746880, 0xc42004c500, 0x1, 0x1, 0x0, 0x0)
        /home/hectorj/go/src/github.com/gobuffalo/packr/packr/cmd/install.go:33 +0x236
github.com/spf13/cobra.(*Command).execute(0x746880, 0xc42004c500, 0x1, 0x1, 0x746880, 0xc42004c500)
        /home/hectorj/go/src/github.com/spf13/cobra/command.go:746 +0x475
github.com/spf13/cobra.(*Command).ExecuteC(0x746ac0, 0xc4200260b8, 0x0, 0x15)
        /home/hectorj/go/src/github.com/spf13/cobra/command.go:831 +0x30e
github.com/spf13/cobra.(*Command).Execute(0x746ac0, 0x0, 0x0)
        /home/hectorj/go/src/github.com/spf13/cobra/command.go:784 +0x2b
github.com/gobuffalo/packr/packr/cmd.Execute()
        /home/hectorj/go/src/github.com/gobuffalo/packr/packr/cmd/root.go:32 +0x2d
main.main()
        /home/hectorj/go/src/github.com/gobuffalo/packr/packr/main.go:20 +0x20

The trace starts here :

if !info.IsDir() {

Working with Package directories

I have a project structure like this:
├── main.go
└── package
│ └── mypackage.go
└── templates
├── admin
│ └── index.html
└── index.html

When I run go run or go build from main.so directory, everything works as expected. the mypackage.go contains a line like box := packr.NewBox("./templates/")
However, when i run packr build in the same directoy I get the following error.

Error: could not find folder for box: /package/templates

How can I get this working?

Thank you

Must.... should panic

I stumble on a link to this package on reddit. The api took me a bit per surprise because I thought that he function starting by Must panic instead of returning an error. At least it is the case across the std lib:

Thank you for sharing this lib

Cross compilation with xgo and packr

I have a database driver, which I have to use xgo for cross comiling to windows. I also would like to use packr.

But I can't guess how to get them working together. Is it possible at all?

Thanks for the support.

packr.File.Read() always returns empty

While in Walk func, reading content using File.Read() returns empty string.

Code -

box.Walk(func(p string, f packr.File) error {
    var b []byte
    err := f.Read(b)
    // ...
}

b remains always empty. Though content can be read using box.MustByte(...) but then why we have Read function.

How to use packr with ParseGlob

How to use packr with ParseGlob? Right now I'm trying to use Packr with html/template ParseGlob by just accessing the Path, but it isn't working...how would this be done? I don't want to parse each string manually..would that be necessary?

templates := packr.NewBox("./templates")
tpl = template.Must(template.New("").ParseGlob(templates.Path + "/*.gohtml"))

Accepted / rejected extensions

Because of how this package works with unrolled/render AssetNames and Asset functions, the only way I was able to get packr and render to work together was to create a Box from a parent directory, and access the assets from inside. This causes the Box to contain several .go files that don't need to be included.

I'm fairly new to Go, so I could also be just implementing them incorrectly.

Is there an appetite for supporting accepted or rejected extensions when creating a Box? That way one could either exclude .go files or only include .html or .tmpl files recursively.

I'm guessing it would necessitate an additional API function (NewBoxWithOptions or something) to support, since the existing NewBox function only takes a path - any thoughts?

Packr embed silently fails when project dir is a symlink

Gitlab CI does not support custom checkout directories (like in $GOPATH), which meant my application directory was a symlink. Which seemed to work just fine, but the binary lacked the embedded files. For example:

$ packr -i opencl
$ ls opencl
opencl-packr.go
# all good
$ go build
$ ./myapp
# ok
$ rm -rf opencl
# let's try this again wiithout the files on disk
$ ./myapp
panic: file does not exist

Real build log here: https://gitlab.com/blockforge/blockforge/-/jobs/50043767 (relevant portion at the bottom)

As a workaround, I've symlinked the parent directory instead, and it's working now. Not sure if there's anything that can be done about this.

Having invalid (but unused) go files anywhere in the project path fails packr build

Trying to use the packr build command in a project that has invalid (but unused) go files fails the command.

I'm using JetBrains File Templates to create some boilerplate code in my project. Those involve creating and storing invalid go files within my project structure (inside the .idea project directory).

Now, when I execute packr build in the project root I get the following output:

$ packr build
Error: /Users/fletcher/go/src/project/.idea/fileTemplates/file_template_1.go:1:1: illegal character U+0023 '#'
Usage:
  packr build [flags]

Flags:
  -h, --help   help for build

Note: go build works just fine

resolved using development path after every deploy

hi,

my packr is using commit 65f7ad7 (which is latest) as shown in my glide.lock

ive amended box.go and add in debug on method find, and found that,

when i open http://example.com/
calling is refering to my development path
calling: /Users/james/www/gowork/src/ossdc/actions
path: ../public/assets
name: manifest.json
p: /Users/james/www/gowork/src/ossdc/public/assets/manifest.json
error: Unable to open /Users/james/www/gowork/src/ossdc/public/assets/manifest.json open

build using buffalo build -e

weird part is, when i tried to open another uri like
http://example.com/admin/
and it works!

then when i revisit http://example.com/

Now my root also working..

what may be the issue?
and also i noticed that, when i open /admin,
i do not get the debug message.

i suspect its already been loaded somewhere

func (b Box) find(name string) (File, error) {
	name = strings.TrimPrefix(name, "/")
	name = filepath.ToSlash(name)
	if _, ok := data[b.Path]; ok {
		if bb, ok := data[b.Path][name]; ok {
			bb = b.decompress(bb)
			return newVirtualFile(name, bb), nil
		}
		if filepath.Ext(name) != "" {
			return nil, errors.Errorf("could not find virtual file: %s", name)
		}
		return newVirtualDir(name), nil
	}
	p := filepath.Join(b.callingDir, b.Path, name)
	fmt.Println("find: calling", b.callingDir, "path", b.Path, "name", name, "final", p)
	f, err := os.Open(p)
	if err != nil {
		fmt.Println("Unable to open", p, err.Error())
		return nil, err
	}
	return physicalFile{f}, nil
}

Removing “envy” dependency?

Depending on envy to get the GOPATH introduces a long chain of dependencies that could be avoided. Something like:

func GoPath() string {
	p := strings.Split(os.Getenv("GOPATH"), string(os.PathListSeparator))
	if len(p) == 0 || p[0] == "" {
		return build.Default.GOPATH
	}
	return p[0]
}

Would it be possible to switch to something like this? Using packr in a project that vendors and commits its dependencies adds a lot of unnecessary files.

Packr multi-box support

Hello!

I have a case where I have to wrap packr into an array of Box in order to preserve semantic. So, I wonder if it is a good idea to provide a type alias which is an array of box.

type Boxes []Box

This type will implement all methods that a box implements. An example of the usage may be:

package main

import (
	"net/http"

	"github.com/gobuffalo/packr"
)

func main() {
  boxes := packr.NewBoxes("./public", "./dist")

  http.Handle("/", http.FileServer(boxes))
  http.ListenAndServe(":3000", nil)
}

What do you thinks about this feature ?

Use `github.com/karrick/godirwalk` instead of `filepath.Walk`

I came across this library today, it started as a part of dep and became it's own library. I really liked the way it created abstraction for the path between Windows and Unix style paths (from the Readme):

In other words, even on Windows, filepath.Walk will invoke the
callback with some/path/to/foo.txt, requiring well written clients
to perform pathname normalization for every file prior to working with
the specified file. In truth, many clients developed on unix and not
tested on Windows neglect this subtlety, and will result in software
bugs when running on Windows. This library would invoke the callback
function with some\path\to\foo.txt for the same file when running on
Windows, eliminating the need to normalize the pathname by the client,
and lessen the likelyhood that a client will work on unix but not on
Windows.

If it can avoid us problems between the paths, it could be nice.

making a box for a directory above this one doesn't work

making a box of a directory that is not a subdirectory of this directory doesn't work:
e.g.

box := packr.NewBox("../public")

See https://github.com/natefinch/markbates/blob/master/test/test.go#L13

And by doesn't work, I mean that if you remove the folder after generating the packr.go file for it, the built executable fails to find those files. You can test it by running packr install for github.com/natefinch/markbates ... if you run markbates then, it'll open up a webpage saying "hi mark!" if you then rename the public directory to something else, running markbates will open up a 404 page

Always packed as JSON

I thought that if I run command as packr -v -z that there will be gzipped content, however the content is still some sort of base 64 packed into json container. What am I missing?

edit: It seems that the content is compressed, but then again inflated for 30% as its base 64 encoded again.

Box still looks at development path when binary deployed to server.

I've followed the example in the README file and I'm not able to run my binary on a different machine, packr always looks for the path on my development machine.

Relevant project structure:

|-model/database.go
|-sql/subDir/file.sql
|-main.go

In model/database.go

getQuery("subDir/file.sql")

func getQuery(filename) {
box := packr.NewBox("../sql")
query := box.String(filename)
return query
}

I build with:

packr -v
go build

or

packr build -v

I see expected output:

packing file subDir/file.sql
packed file subDir/file.sql
built box ../sql with ["subDir/file.sql"]

and run the binary:

./app-binary

On my server, the files are not found, I just get a blank string from box.String(filename). If I use box.MustString(filename) and inspect the error I can see it is looking at the path on my development machine (which doesn't exist on the server). If I call box.Has(filename) it returns false.

I can see the -packr.go file after running packr and my files exist there:

func init() {
packr.PackJSONBytes("../sql", "subDir/file.sql", ""SU5TRVJblah"")
}

Am I missing something?

How to build the packr binary?

I have packr working with my go program running directly from the filesystem. It's cool. Want to package those files into the binary and I need the packr command to do that but I can't figure out how to build it.

It would be great to add these instructions to the README.

Packr silently fails when project built by Jenkins

We have a very hard to solve issue that involves packr, when run on our local workstations (Fedora and MacOS) the packr build command generates a correct executable containing all the files in the packr box.

However, when built on Jenkins (using a Alpine image), the resulting executable contains no files.

Its very perplexing issue, Jenkins just clones the same repository that we are using, running the same commands, but the result ends up differently and we cannot see any error messages in the log from packr.

Any suggestion where we can start looking ?

packr not detecting files even though it is in the binary

Packr Test

This repository was created to test out an error I got when using packr.

Usage

go get -d github.com/yadunut/packr-test/...

or

git clone https://github.com/yadunut/packr-test

No Issues

cd packr-test
packr build cmd/server/main.go
./main

Expected

Init Function
2 files in ./templates
ListFiles function
Files in box:
index.tmpl
register.tmpl

Actual

Init Function
2 files in ./templates
ListFiles function
Files in box:
index.tmpl
register.tmpl

Issues

Changed the path to the repository

cd ..
mv packr-test packr-tests
cd packr-tests
./main
cd ..
mv packr-tests packr-test
cd packr-test

Expected

Init Function
2 files in ./templates
ListFiles function
Files in box:
index.tmpl
register.tmpl

Actual

Init Function
0 files in ./templates
ListFiles function
Files in box:
index.tmpl
register.tmpl

What Happened?

The init function doesn't detect files in the binary, but detects files when they're in the folders.

Outside of the init function, there are no issues.

confusion over how packr finds folders for boxes

Strange title I know, but...

I'm writing a small web framework which is going to be very opinionated. It mandates that templates must be in a folder called 'views' and that static files are in a folder called 'static'. I would like to be able to use packr to package the files for delivery.

I wanted to create the packr.Box within the framework but it seems that by doing that packr looks in the folder of the framework, rather than the folder of the main application, for the 'boxed' files. If I create the packr.Box within the main app and pass that to the framework all works as expected.

Am I going slightly mad? or is what I describe above actually happening?

Proposal: Embedding files as raw binary data

Background

For one of my own projects, I ran into the problem that the memory overhead for embedding some of the files involved was unacceptably high over the potential lifetime of the program. I still wanted self contained binaries though, so I did some digging and ran into this on stackoverflow, and as it turns out none of the major operating systems care if you dump crap at the end of an executable. This, combined with the discovery of the osext library for finding the path to the current executable prompted the development of the following scheme for embedding files, which I have working as far as I need for my own purposes at this point.

Relevant code can be found here for the writer and here for the reader

Currently, installing transcodebot and calling

transcodebot watch [binary_standin] [file_to_append]

Will pack $file_to_append onto $binary_standin and then unpack it to stdout.

High level description

  1. Compile the binary normally
  2. Write each file to embed to the end of the binary through a gzip.writer, noting the start and end point on the file.
  3. Write the gathered start and end points, along with their names (paths) out to a json-encoded metadata block at the end of binary
  4. Write the start position of said json data out as a magic number for the last 8 bytes of the file.

The self-extraction process looks like this, then:

  1. Find the path to the source of the current process
  2. Open the file
  3. Read the last 8 bytes
  4. Open a json.Decoder at the position written to the last 8 bytes
  5. Decode the json into a struct in memory
  6. Close the file

Then, when the process wants to access some of the data in the file, a file reader is opened on it that is simply started at $start_ptr and is limited to reading $size bytes

Visualization:

0                        400                       500              7000
+------------------------+-------------------------+----------------+---------------------+--------------+
| Executable Binary Data | Embeded File ./assets/A | Embeded Data B | Json metadata block | Magic Number |
+------------------------+-------------------------+----------------+---------------------+--------------+

In this case the json would look something like this:

{
  "Version": "0.1",
  "Data":
  {
    "./assets/A": { "start_ptr": 400, "size": 100 },
    "B":          { "start_ptr": 500, "size": 6500 }
  }
}

And the magic number would be 7000

Comparison with current system

Disadvantages:

  • Weird-looking executables
  • Relies on runtime time code to find static assets
  • May not work on a new operating system with a new binary format that go targets in the future

Advantages:

  • No additional files generated at compile time
  • Data storage format is self-describing, so it's possible for other programs to pull out the files/data stored in the binary
  • Process of appending data is self contained and doesn't need to interact with source code or the compiler
  • Smaller file sizes as there is no need to convert the binary data into a stringable format
  • Everything is stream based, so massive files (e.g. videos, executables) can be embedded at no memory cost for the compile process or compiled program
  • Can expose an api on uncompressed data that looks like an os.File, walks like an os.File, and quacks like an os.File
  • Fixes all of the problems with embedded static files mentioned on this page
  • Can fall back on compile-in method if need be

Known work to be done

I'll probably be doing these anyways:

  • Move the code into it's own repo with better examples
  • Clean up the names in the public api

Trivial and would be happy to add:

  • Uncompressed data api

Things I would need help with

  • Gluing packr high level api onto this
  • Writing new tests for packr where appropriate

Thanks for reading!

Calling `packr build/install` with a `*.go` files causes boxes not to be packed

Hey so this is a longshot..

I am using packr to compile static assets to a single go binary.

It's a webserver. I can move the exe around on my machine and it works.

However when someone else on a different windows machine tries it, it says 404 not found :thinking_face:

Haven't tried on Mac yet, but I guess I would run into the same issues 😂

Here is my repo: https://github.com/selfup/hmrcmd

Any help would be awesome! 🙏

Publish binaries for releases

Hey!
I really enjoy packr and use it on over 10+ projects now. During my CI/CD I sometimes need to go get packr, which is rather annoying because it takes a minute. It would be super dope, if you could provide some pre-build binaries (linux-amd64 for me). That way I could just curl them in my CI/CD and that's it.
Thanks!

List of the box is empty if we call it again ...

When this code is called during init() phase, then files are collected and listed

var (
	templatesF  []string
	box 		     = packr.NewBox(templateDir)
)

func init() {
	// set up a new box by giving it a (relative) path to a folder on disk:
	templatesF := box.List()
	log.Infof("Create a box of the template files",templatesF)
}

but if another function of the same package call it again, then the result is empty ....

func test() {
	log.Infof("List of files :",templatesF)
}

Cobra command called and result

sb generate        
INFO[0000] Create a box of the template files%!(EXTRA []string=[pom.xml simple/RestApplication.java simple/service/Greeting.java simple/service/GreetingEndpoint.java])                     
INFO[0000] List of files :%!(EXTRA []string=[])  

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.