Config file about goxc

[laher]

Good idea, I was thinking of a similar idea but with goxc.json.
I'd prefer .json to .ini but I'm open to discussion on it

[laher]

just to give more info, chrome plugins use manifest.json: http://developer.chrome.com/extensions/manifest.html
and node.js uses package.json: http://package.json.nodejitsu.com/
I was thinking of a format similar to these...

Also, golang's json support seems to be really good: http://golang.org/pkg/encoding/json/

[matrixik]

Well, whatever you prefer is OK.
Don't you think that ini files would be simpler and shorter?

{
    "destination": "./out/",
    "zip": false,
    "verbose": true,
    "platforms": {
        "windows": "386",
        "linux": "amd64"
    }
}

destination = ./out/
zip = false
verbose = true
[platforms]
windows = 386
linux = amd64

As a side note I made some updates to Goconf package: http://godoc.org/bitbucket.org/gosimple/conf
It can now read slices from option:

windows = 386, amd64
c, _ := conf.ReadFile("<conf_file>")
c.StringList("", "windows") // return ["386", "amd64"]

Full json example:
http://play.golang.org/p/j0PE_S5JSF
Full conf example:
http://play.golang.org/p/Bc9Fgu2B39

But probably json files would be better for bigger configs.

Choose what suit you the most.

Best regards,
Dobrosław Żybort

[laher]

Thanks for the examples, that'll give me a head-start tonight (it's morning time here, I'm heading to a kids party)

I think you hit the nail on the head with 'bigger configs'. I think this could grow quite rapidly - keep an eye on the todos because it's mostly still in my head. I'll definitely take a look at your gconf though

Anyway thanks heaps for your contributions, I'm keen to hear/see more if you have the inclination. Also do let me know what you're building with goxc

Cheers, have a good day

[matrixik]

Maybe you can make issue for every todo you plan (and mark them as enhancement or sth), that way will be easier to follow them if someone is interested in some specific functionality.

Maybe you can find some useful ideas here:

At some point I made bat script for my Python app to pack and create singe exe from it:

:: Batch file for creating exe files on Windows from py files with py2exe
:: Based on: http://www.py2exe.org/index.cgi/WinBatch

@echo off

:: App setings
set app_name=Link Engine Checker


:: Compilation settings
set bin_folder_name=bin
set output_folder_name=Link Engine Checker
set output_folder_path=%bin_folder_name%\%output_folder_name%
set archive_name=Link_Engine_Checker
set app_script=link_engine_checker.py
set app_destination_base=link_engine_checker
for /f "tokens=3" %%i in ('findstr /R /C:"__version__ = " %app_script%') do set app_version=%%i
set app_version=%app_version:~1,-1%

:: Compress=1 - Use CompressFiles
:: Compress=0 - Don't CompressFiles
set Compress=1

:: Check if compilation tools are accessible
:: if not exist python goto:PythonNotFound
:: if not exist upx goto:UPXNotFound

:: Remove old compilation
if exist "%output_folder_path%\" rd "%output_folder_path%" /s /q

::Write the Py2EXE-Setup File
call :MakeSetupFile > "setup_exe.py"


::Compile the Python-Script
python "setup_exe.py" py2exe -q
if not "%errorlevel%"=="0" (
        echo Py2EXE Error!
        pause
        goto:eof
)


:: Delete the Py2EXE-Setup File
del "setup_exe.py"


if "%Compress%"=="1" call:CompressFiles

:: Create archive with exe
cd ..
7z a "%archive_name%-%app_version%".7z "%output_folder_name%"

echo.
echo.
if "%Compress%"=="0" sleep 2
if exist build rd build /s /q
echo Done, files in "%bin_folder_name%" folder
echo.
pause
goto:eof


:CompressFiles
        cd %output_folder_path%\
        upx --best --all-methods *.*
goto:eof


:MakeSetupFile
        echo.
        echo # ======================================================== #
        echo # File automagically generated by GUI2Exe version 0.5.3
        echo # Copyright: (c) 2007-2012 Andrea Gavana
        echo # ======================================================== #
        echo.
        echo # Let's start with some default (for me) imports...
        echo.
        echo from distutils.core import setup
        echo from py2exe.build_exe import py2exe
        echo.
        echo import glob
        echo import os
        echo import shutil
        echo import zlib
        echo.
        echo # Remove the build folder
        echo shutil.rmtree("build", ignore_errors=True)
        echo.
        echo.
        echo class Target(object):
        echo     """ A simple class that holds information on our executable file. """
        echo     def __init__(self, **kw):
        echo         """ Default class constructor. Update as you need. """
        echo         self.__dict__.update(kw)
        echo.
        echo.
        echo def find_data_files(source, target, patterns):
        echo     """Locates the specified data-files and returns the matches
        echo     in a data_files compatible format.
        echo.
        echo     source is the root of the source data tree.
        echo         Use '' or '.' for current directory.
        echo     target is the root of the target data tree.
        echo         Use '' or '.' for the distribution directory.
        echo     patterns is a sequence of glob-patterns for the
        echo         files you want to copy.
        echo     """
        echo     if glob.has_magic(source) or glob.has_magic(target):
        echo         raise ValueError("Magic not allowed in src, target")
        echo     ret = {}
        echo     for pattern in patterns:
        echo         pattern = os.path.join(source, pattern)
        echo         for filename in glob.glob(pattern):
        echo             if os.path.isfile(filename):
        echo                 targetpath = os.path.join(target, os.path.relpath(filename, source))
        echo                 path = os.path.dirname(targetpath)
        echo                 ret.setdefault(path, []).append(filename)
        echo     return sorted(ret.items())
        echo.
        echo # Ok, let's explain why I am doing that.
        echo # Often, data_files, excludes and dll_excludes (but also resources)
        echo # can be very long list of things, and this will clutter too much
        echo # the setup call at the end of this file. So, I put all the big lists
        echo # here and I wrap them using the textwrap module.
        echo.
        echo data_files = find_data_files('engines', 'engines', ['*'])
        echo.
        echo includes = []
        echo excludes = ['_gtkagg', '_tkagg', 'bsddb', 'curses', 'email', 'pywin.debugger',
        echo             'pywin.debugger.dbgcon', 'pywin.dialogs', 'tcl',
        echo             'Tkconstants', 'Tkinter']
        echo packages = []
        echo dll_excludes = ['libgdk-win32-2.0-0.dll', 'libgobject-2.0-0.dll', 'tcl84.dll',
        echo                 'tk84.dll', 'w9xpopen.exe']
        echo icon_resources = []
        echo bitmap_resources = []
        echo other_resources = []
        echo.
        echo.
        echo # This is a place where the user custom code may go. You can do almost
        echo # whatever you want, even modify the data_files, includes and friends
        echo # here as long as they have the same variable name that the setup call
        echo # below is expecting.
        echo.
        echo # No custom code added
        echo.
        echo.
        echo # Ok, now we are going to build our target class.
        echo # I chose this building strategy as it works perfectly for me :-D
        echo.
        echo GUI2Exe_Target_1 = Target(
        echo     # what to build
        echo     script = "%app_script%",
        echo     icon_resources = icon_resources,
        echo     bitmap_resources = bitmap_resources,
        echo     other_resources = other_resources,
        echo     dest_base = "%app_destination_base%",    
        echo     version = "%app_version%",
        echo     company_name = "Dobroslaw Zybort",
        echo     copyright = "Copyright 2012 Dobroslaw Zybort",
        echo     name = "%app_name%",
        echo     description = 'Command line tool to analyse links for used engine (like WordPress, phpBB and more)',
        echo     license = 'GPLv3',
        echo     author = 'Dobroslaw Zybort', 
        echo     author_email = '[email protected]',
        echo     download_url = 'https://bitbucket.org/matrixik/link-engine-checker/downloads',
        echo     url = 'https://bitbucket.org/matrixik/link-engine-checker', 
        echo     )
        echo.
        echo # That's serious now: we have all (or almost all) the options py2exe
        echo # supports. I put them all even if some of them are usually defaulted
        echo # and not used. Some of them I didn't even know about.
        echo.
        echo setup(
        echo.
        echo     # No UPX or Inno Setup
        echo.
        echo     data_files = data_files,
        echo.
        echo     options = {"py2exe": {"compressed": 2, 
        echo                           "optimize": 2,
        echo                           "includes": includes,
        echo                           "excludes": excludes,
        echo                           "packages": packages,
        echo                           "dll_excludes": dll_excludes,
        echo                           "bundle_files": 1,
        echo                           "dist_dir": "%output_folder_path%",
        echo                           "xref": False,
        echo                           "skip_archive": False,
        echo                           "ascii": False,
        echo                           "custom_boot_script": '',
        echo                          }
        echo               },
        echo.
        echo     zipfile = None,
        echo     console = [GUI2Exe_Target_1],
        echo     windows = [],
        echo     service = [],
        echo     com_server = [],
        echo     ctypes_com_server = []
        echo     )
        echo.
        echo # This is a place where any post-compile code may go.
        echo # You can add as much code as you want, which can be used, for example,
        echo # to clean up your folders or to do some particular post-compilation
        echo # actions.
        echo.
        echo # Remove the build folder
        echo shutil.rmtree("build", ignore_errors=True)
        echo.
        echo.
        echo # And we are done. That's a setup script :-D
        echo.
goto:eof


:FileNotFound
        echo.
        echo Error, File not found:
        echo [%1]
        echo.
        echo Check Path in %~nx0???
        echo.
        pause
goto:eof


:PythonNotFound
        echo.
        echo Error, Python not found (must be available in system PATH)
        echo.
        pause
goto:eof


:UPXNotFound
        echo.
        echo Error, UPX not found (must be available in system PATH)
        echo.
        pause
goto:eof

py2exe is nice because it can add resources to output file like icons, custom resources, file informations (like full app name, version, company etc.)

My script create 7z archives with name <app_name>-<version_nr>.7z and also pack exe files with UPX. It extract version number from py file.

but

as you can see it's way over complicated :/ Basically I made a script that creating additional install script. Pre- and post-processing scripts are OK if in simple version we can use simple config, without unnecessary magic.

Maybe you can force some standard for version name that will be easy to extract from file. Python devs like to use:

__title__ = 'Link Engine Checker'
__version__ = '1.0.1'
__author__ = 'Dobroslaw Zybort'
__email__ = '[email protected]'
__url__ = 'https://bitbucket.org/matrixik/link-engine-checker'
__license__ = 'GPLv3'
__date__ = '2012-10-18'
__copyright__ = 'Copyright 2012 Dobroslaw Zybort'

For file naming I also would prefer <app_name>_<version_nr>_<target_platform>.<ext>

Best regards,
Dobrosław Żybort

[laher]

OK, thanks. I've just been thinking this through aswell, and I agree it would be preferable to pull as much as possible from constants in the source code. Python progrmmers tend to be pragmatic so good to copy them :)
Not sure how best to parse Go source at this stage though.
Also, the app naming you suggest makes more sense.

Regarding 'platforms', I don't know if you're aware of the +build compiler flags which 'go build' honours.
See here: go build and scroll down to the 'Build Constraints' section. It has its own syntax, which maybe ought to be used by goxc.

Here's some json I was playing with, just to see what could be possible in future. Not sure if it's overkill yet but I feel like it's important decision so I'll give it some more thought...
Ofcourse it should be possible to define just the keys which are important to you, and perhaps the 'metadata' section is unnecessary. Just ideas ..

{
  name: "goxc",
  version: "0.1.4",
  manifest_version: "1.0-goxc",
  platforms : "windows,386 linux !darwin", // <- using +build flags .. 
  //or alternatively platforms: { windows : [ "386", "amd64" } }
  goxc : { //maybe a separate section for goxc.
        destination: "../goxc-pages/dl",
        artifact_type: "zip",
        verbose: true,
        resources: [ "README*", "LICENSE*", "INSTALL*"],
        scripts: {
            pre_build: "process-sources.sh",
            post_build: "process-resources.sh"
        },
        downloads_list: {
             page: 'downloads.md',
        }
   },
  dependencies : [
        // Could become necessary when creating a standalone exe. Maybe parsing file is better.
   ],
  metadata: { // separate section here?
    author: "Am Laher <[email protected]>",
    description: "A cross-compilation and build utility for Go",
    url: "http://www.laher.net.nz/goxc",
    contributors: [
     { name: "laher", url: "https://github.com/laher" },
     { name: "dchest", url: "https://github.com/dchest" },
     { name: "matrixik", url: "https://bitbucket.org/matrixik" }
    ],
    repository: {
     type: "git",
     url: "https://github.com/laher/goxc.git"
    },
    license : "Apache 2.0"
  },
}```

[laher]

Ah. Go has a 'parser' package and it looks very easy. Forget the json manifest for now, I'll look into using constants instead.

Perhaps PACKAGE_VERSION, PACKAGE_TITLE, etc would be idiomatic for Go.
I think the underscores are a pythonic thing.

[laher]

OK so, my last thought before sleep is that we could derive as much as possible from constants declared in the source, and then use another 'local override' Go file for any working-copy configuration. e.g. if your currently only interested in building unzipped for linux/386.
This special Go file could use the "// +build ignore" flag to stop it being built into the artifacts, or possibly end with .goxc for clarity.

.. Food for thought anyway

[matrixik]

+1 for build constraints, we should follow Go way as much as possible.
platforms : "windows,386,amd64 linux,amd64",

+1 for go/parser but only for some info, not all. Like title, version, author/authors/organization, url, rest should stay in separate config file, no .go file.

And best would be if you can make topic on go-nuts, show everything what you think will work nice, and ask more people what they want, how variables should be named etc.

Best regards,
Dobrosław Żybort

[laher]

Good call, I think the time to contact golang-nuts is arriving. I wanted to catch the low-hanging fruit before posting a golang-nuts topic, and I think it's almost ready.

I've added a couple of things since yesterday:

1. a 'sanity check' to see if GOROOT is set and make.bash/bat exists. If not it advises the user to install go from source,
2. the -include="README_,LICENCE_,INSTALL*" option is now implemented for zips (so far)

I'll pose the config question tonight, but I'll flesh out the go/parse & config ideas in some _test.go test cases first.

[laher]

OK, the PKG_VERSION constant is used now in v0.1.6.

[matrixik]

One more request about config file:
some kind of global config file where I can put platforms I want to be built by goxc -t (useful when I update Go code every few days from hg, user folder or same folder as where goxc executable is located like $GOPATH/bin)

[laher]

Hi Dobroslaw,

I've done a json-based config. You'll see it's not as concise as your ini format, but it leaves it very open to extension.

It seems to be working well, and just now I added a 'write config' option (-wc) so goxc can write your options straight to file for you.

1. The config file, by default, is .goxc.json.
   You can specify a different 'config name' using -c=configname

So, for your goxc -t usecase you could write 2 configs and re-run each of them as follows:

1. write config file toolchain.json before building toolchain

    goxc -t -wc -c=toolchain -os=windows,linux 
   

2. writes to the default config file .goxc.json, before building & zipping your app:

    go -wc -os=linux -arch=amd64 
   

Now, to re-build the toolchain you just run

goxc -c=toolchain

To rebuild your app with a new version number:

 goxc -pv=0.1.1

2. The versioning works differently now. Firstly, I'm using a gobuild flag (-ldflags) for generating the version during build, and applying it to a var called VERSION in the main package. This was suggested on golang-nuts (by minux).
   Secondly, I have also made variables available for branchname, prereleaseinfo, and build name, as in semantic versioning. So you might have a full version number of 0.0.1-mybranch.alpha+b32
   Thirdly, the PKG_VERSION constant is no longer used.

3. Note that there's also a 'config override' feature.
   You can put in a configname.local.json, and this will override configname.json. This would be where you'd normally store your branchname, prerelease info and anything you don't want to affect other users. Typically you'd put *.local.json in your .gitignore file.
   You can override any variable in this file. Please see my examples as linked in the readme.

I'm pretty happy with it. It's not perfect, but I think it'll give lots of flexibility moving forwards.
Please see what you think & let me know if it's not clear or not working for you.
Thanks

[matrixik]

1. Thank you for that but it's not exactly what I would like to have.
   I would like to put toolchain.json file in $GOPATH/bin (where goxc is stored) and then whenever I want to re-build the toolchain I can simply run goxc -t from any folder.
   goxc then will check if file toolchain.json (or goxc.toolchain.json or any other, must be hard-coded in goxc) exists in $GOPATH/bin and will get built setting from it.

2. and 3) Thank you for all.

Best regards,
Dobrosław Żybort

[laher]

Aaaah, OK I didn't realise. Makes sense now.
The toolchain is independent of the project so why not.

I think it'll be easier to use $HOME/.goxc.toolchain.json (or probably $HOME/.goxc.json, so it can support other preferences in future).
I think the user folder is more conventional, plus it's not normally possible to know where a running program is located. See this SO question. I guess goxc will normally be run from $GOPATH/bin (having split path separator ':' and checked each part), but it's not guaranteed.

[matrixik]

Thank you for everything.

[matrixik]

OT:
plus it's not normally possible to know where a running program is located.
It is: http://godoc.org/bitbucket.org/kardianos/osext
I also added it to stackoverflow.

[laher]

Fascinating project. It's cool how simple the implementation is for procfs. Less so for sysctl.
Makes me wonder how quick it would be to port things like http://code.google.com/p/psutil/ to Go.
I know there's https://github.com/surma/gobox but that doesn't do things like lsof (and it's proudly pure go anyway)

[laher]

OK, done.

For latest version 0.3.0, goxc -t looks for config in your $HOME folder.

(To use your current folder, you can still use goxc -t .)

Note that you can write the config using goxc -t -os=windows -wc or similar

... My next priority is an option to run 'go test' on the package before cross-compiling - failing on error (and maybe 'go vet', 'go fmt', 'go install'). It'll probably be a couple of weeks though ..

[laher]

ok, I'm closing this issue.

If you're interested please get latest version. It's got some major changes to the workflow (including the 'go test' I mentioned above).
You should find it's more flexible now moving forward, and contributions should be easier now as there's an obvious way to create new 'tasks' and per-task settings.

[matrixik]

Thank you again for everything.

Best regards,
Dobrosław Żybort