Problem parsing .env file ? about herokuish

[pierredavidbelanger]

Notice I didn't say "official"...

Oh, indeed.

Well... I guess, for a clean fix, one need to parse the .env file as key=value entries instead of plain sourceing it.

Unfortunately, I am not very good with bash nor regepx ;)

I will think about this.

Thank you for your time :)

[michaelshobbs]

@pierredavidbelanger v0.3.20 has a basic (emphasis on basic) env file parser that will work with spaces. please give it shot.

[josegonzalez]

Ah that is for tests, hence why it's in the Dockerfile.dev.

@michaelshobbs @progrium thoughts on including that in the base image, or replacing the usage in tests for something is in the base image?

[pierredavidbelanger]

I just saw #195

I do have pushed this issue too hard after all :)

I changed my mind on this.

I realized that herokuish is not (required to be) Heroku, and as so, Dokku take for granted that .env file is shell source-able.

I vote you revert 465bbe1, instead of trying to code a key=value parser (that supports complex shell syntax and variable interpolation) in shell :)

And, for my original issue (being able to build the official unmodified java-getting-started), the workaround will be to simply rm the .env file; indeed, on the Heroku build servers, the .env file is not used, because it should not be under source control anyways.

Does it make sens ?

[josegonzalez]

Spaces aren't valid values in unquoted shell, so yes, you will have to quote the value in order to have it be assigned.

[pierredavidbelanger]

Spaces aren't valid values in unquoted shell

I know that. And I will surely not make this mistake in my own application.

But it seems Heroku will not treat .env file like a shell.

You understand that I am using the official Heroku unmodified java-getting-started app ?

And since:

The goal is to be the definitive, well maintained and heavily tested Heroku emulation utility shared by all.

I just assumed the official Heroku java-getting-started app would work unmodified.
This is why I reported this issue.

[josegonzalez]

If you use the heroku cli to set the above environment variable, that is the following command:

heroku config:set ENERGY="20 GeV"

Without quoting, that would result in an error since GeV would be parsed as a key without a specified value (actually it might set it to empty string, but its certainly not what you want).

Via the web ui, you are explicitly quoting as there isn't a way to specify multiple key=value pairs in the web form on a single form input.

For their api, same deal, except it's using json dictionaries.

Their backing store is not a .env file, but rather a database of those credentials. Thus they have it in key => value format, where you can think of that value as being quoted (otherwise it wouldn't be in the same field in their database.

So yes, I do think this works as a pretty good test of the heroku parsing scheme, given that our .env parsing is done via shell, where the value must be quoted in order to be used.

[pierredavidbelanger]

During this tutorial https://devcenter.heroku.com/articles/getting-started-with-java
when I git push heroku master, it does not fail with an GeV: command not found.

I guess, what I expect is that I can build the official example app without error.

Why parsing the .env file at all during build time ?

[michaelshobbs]

@pierredavidbelanger I get your point. Do you have a solution in mind or better yet a PR to make this work?

[pierredavidbelanger]

@michaelshobbs Thank you.

Also, I am sorry, I may be pushing this issue too hard ;)

I am new with this project, I may be wrong, but I think that removing this if will help.

Though, given the comment, it seems to be there for a reason ?

If I understand correctly, .env file is there to help devs simulate Heroku ENV when running locally (just like foreman do, for example). Heroku does not care for this file at build or run time.

[michaelshobbs]

Unfortunately the reasoning for herokuish's support of .env is not so intentional. Additionally, removing that line will cause all apps that do not have a .env file to fail to build. That block is read "if the file $app_path/.env exists, then source it."

The problem is that bash has very strict parsing rules and doesn't have the additional constructs of a language like ruby; like what foreman is written in.

[pierredavidbelanger]

When I say, removing the if, I mean, completely .. like, not sourcing .env at all ..

Or, if you really want to continue to source it (let say, maybe because actual users may depends on this), maybe you can try to source it, is a way compatible with foreman.

[michaelshobbs]

Ah! Understood. So that is there because some buildpacks still use it.

Please feel free to contribute to the project and make this work for you and the rest of the folks out there.

[pierredavidbelanger]

I really wonder which official buildpack uses this file.

Even Heroku recommend to .gitignore this file.

[michaelshobbs]

Notice I didn't say "official"...

[pierredavidbelanger]

@michaelshobbs That was fast! Your fix is working for the heroku/java-getting-started application.

I just tried this:

$ docker build -t herokuish .
$ git clone https://github.com/heroku/java-getting-started.git
$ docker run --rm -it -e TRACE=true -v `pwd`/java-getting-started:/tmp/app herokuish test

It worked. Here is the relevant parts of the logs:

...

+ [[ -f /app/.env ]]
++ cat /app/.env
++ _envfile-parse
++ declare 'desc=Parse input into shell export commands'
++ local key
++ local value
+++ cat
++ read line
++ [[ ENERGY=20 GeV =~ ^#.* ]]
++ key=ENERGY
++ key=ENERGY
++ value='20 GeV'
++ case "$value" in
++ value='"20 GeV"'
++ echo 'export ENERGY="20 GeV"'
++ read line
++ [[ -n '' ]]
+ eval export 'ENERGY="20' 'GeV"'
++ export 'ENERGY=20 GeV'
++ ENERGY='20 GeV'

...

++ curl --fail --retry 10 --retry-delay 2 -v -s localhost:5678/
* Hostname was NOT found in DNS cache
*   Trying ::1...
* Connected to localhost (::1) port 5678 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5678
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
* Server Jetty(9.0.z-SNAPSHOT) is not blacklisted
< Server: Jetty(9.0.z-SNAPSHOT)
< 

...

[pierredavidbelanger]

Also, not related to the original issue:

I spotted this, near the end of the output:

::: CHECKING APP :::
+ local output
++ xargs
++ html2text -ascii
/tmp/bashenv.409484805: line 122: html2text: command not found

I guess html2text is missing from the image.

[josegonzalez]

@pierredavidbelanger yeah you can add that via apt-get in your own custom image. Is that available on heroku? If so, they haven't pushed out their latest changes to their oss project.

[pierredavidbelanger]

I am not using a custom image. I am docker building straight from the root of your freshly cloned project.

Try this :

$ git clone https://github.com/gliderlabs/herokuish
$ cd herokuish
$ docker build -t herokuish .
$ git clone https://github.com/heroku/java-getting-started.git
$ docker run --rm -it -e TRACE=true -v `pwd`/java-getting-started:/tmp/app herokuish herokuish test  | grep html2text
++ html2text -ascii
/tmp/bashenv.788662760: line 122: html2text: command not found

This tool seems not to be used used by the official java buildpack.

The html2text: command not found error, come from your herokuish test command here

I guess you expect it to be present on the heroku/cedar:14 image ?

[josegonzalez]

Right, I'm saying that this package is not on the official heroku setup either:

$ heroku run bash
Running bash on ⬢ my-fancy-app... up, run.6417 (Hobby)
~ $ which html2text
~ $ html2text
bash: html2text: command not found

If you want that binary, you'll need to make a new Dockerfile that uses our image and calls apt-get install on whatever package contains html2text.

[joshmanders]

But herokuish relies on it,

herokuish/include/herokuish.bash

Line 109 in 3d79244

output="$(curl --fail --retry 10 --retry-delay 2 -v -s localhost:${PORT}$path | html2text -ascii | xargs)"

So shouldn't that technically mean it should be installed in herokuish by default?

[pierredavidbelanger]

I have no use for this package myself :)

I guess we are lost on translation here, I am sorry for that, my primary language is not english.

I just wanted to report to you, that your herokuish test command is throwing some command not found error. And this is due to the fact that heroku/cedar:14 does not contains this package, nor your default image.

I see, though, that this one does though.

[michaelshobbs]

@josegonzalez I'm fine with including that in the base image. That's the "easiest" fix.

[josegonzalez]

What else does it pull in? If it's just that binary, lets do it, otherwise I'm 👎 .

[michaelshobbs]

It has no new dependencies

Package: html2text
Versions:
1.3.2a-18 (/var/lib/apt/lists/httpredir.debian.org_debian_dists_jessie_main_binary-amd64_Packages.gz) (/var/lib/dpkg/status)
 Description Language:
                 File: /var/lib/apt/lists/httpredir.debian.org_debian_dists_jessie_main_binary-amd64_Packages.gz
                  MD5: a57fb9ed5921cd11bfc59269f601f3c0


Reverse Depends:
  urlwatch,html2text
  dhelp,html2text
Dependencies:
1.3.2a-18 - libc6 (2 2.14) libgcc1 (2 1:4.1.1) libstdc++6 (2 4.9) curl (16 (null)) wget (0 (null))
Provides:
1.3.2a-18 -
Reverse Provides:

[josegonzalez]

Alright, ship it (but add a caveat to our readme saying we include it for tests).

[michaelshobbs]

Actually I think I only added that as part of the emberjs buildpack addition. I'll drop that too since we're not shipping with that buildpack anymore. c797745