Separate the vagrant modules into their own repos about puphpet

[frastel]

Yes. Thanks for starting discussion about that. I did not had the heart to start this topic :) I see a real problem there, especially when perhaps puPHPet users want to help improve the original puppet modules and this could not be done when the modules are integrated in vagrant-puppet-lamp.

We could use vagrant-puppet-lamp for the definition of the puPHPet project so. However this repo should not be added the generated file archive.

I think the simpliest start would be to copy all the needed puppet modules from local vagrant-puppet-lamp to the temporary folder (refactoring nr.1). The source paths of this copy calls will then be replaced by the correct puppet module fork installed with composer.
This means we could internally refactor things before we start to create and install the real puppet modules.

[frastel]

Just have seen the pro tip of the day: https://help.github.com/articles/splitting-a-subpath-out-into-a-new-repository/
Doing this for the first step would help to manage all modules in their own (compat) repo. Afterwards we could check the history of each new repo and could analyze if we could create real forks from the original puppet modules.

[rogeriopradoj]

I just followed the pro tip and made this repo from modules/php:

• https://github.com/rogeriopradoj/puphpet-php (which I find out it is the https://github.com/example42/puppet-php with some vanilla, but without the original git history, right?)

In my opinion the Organization should host real forks from original, in order to maintain history and maybe giving back the improvements to that repos some day.

[jtreminio]

I've decided to go another route: Composer! wooo!

We now have a Composer installer 1: https://github.com/puphpet/puphpet-module-installer

puphpet/vagrant-puppet-lamp no longer includes modules in repo. Instead it's all handled through composer.json: https://github.com/puphpet/vagrant-puppet-lamp/blob/master/composer.json

All they need is a semi-special composer.json file: https://github.com/puphpet/puppet-phpmyadmin/blob/master/composer.json

Next up: Getting puphpet.com to use this new style.

[staabm]

looks promising!! 👏

[frastel]

Going this way you still will not have the possibility to add original repos and have to use unmergable forks in your composer.json as the repos have to be tagges as puphpet module for the installer.
We have two mainly different approaches for the system:

1. creating an optimized environment for PuPHPet itself
2. adding even every supported puppet module to PuPHPet so that it could compile the package for the user

The latter could already be done in theory . The only thing missing here is that all puppet modules (that means the original ones from vagrant-puppet-lamp) are added to composer lile it is already done with phpmyadmin, pgsql and symfony. PuPHPet may already handle every available puppet module and is able to add it to the user's archive.

The first point does not need the additional puppet modules (phpmyadmin, nginx...). Here you only need the lamp environment where PuPHPet runs on.
This evening I will provide a prototype using composer and the original puppet modules (install puppet modules in the normal way and create a post-install script which copies the modules from vendor dir to src/puppet/modules).
I strongly recommand not to continue your suggested way.

[jtreminio]

I felt it was a bit bulky after the second or third repo, but at that point I just wanted to get them off of the main repo and into separate places.

I agree it's not ideal - and since we have full control over what goes on in the VM I am not opposed to just moving the folders around after installation.

[jtreminio]

So I just realized something ... if someone doesn't have PHP installed, or composer, they may not be able to instantiate the vagrant-puppet-lamp repo ... Maybe submodules are the best way to go about this.

[scoutman57]

I was just catching up on the thread and was about to suggest that same
thing!

On Fri, Jun 21, 2013 at 11:31 AM, Juan Treminio [email protected]:

So I just realized something ... if someone doesn't have PHP installed, or
composer, they may not be able to instantiate the vagrant-puppet-lamp repo
... Maybe submodules are the best way to go about this.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/56#issuecomment-19826284
.

[frastel]

Why would you use git submodules in a repository which should be installable by composer? More infos here: composer/composer#1600

I do not understand why you still stay with the usage of vagrant-puppet-lamp. An image for PuPHPet could also be created in a clean way. PuPHPet itself should drop the dependency to this repo and only target real puppet modules.

Yes the composer.json will be bulky but if you want you could create a fresh repo where all this bulky requirements are defined and hide it in this way from PuPHPet.

[frastel]

Here is my prototype of the puppet modules installation via Composer:

https://github.com/frastel/puppet-composer-installer

This will still result in bulky composer.json as every puppet module has to be added to the repository list however you may use the original puppet modules without any custom fork and special composer.json in each repository.
The puppet modules are all downloaded in the vendor dir as usual but the post installer takes them and copies all to the one needed puppet module dir.
Clone it and try it out with

composer install

The user needs PHP and composer installed on his machine for installing it this way. Yes thats true, but this installation would only be needed during development of the PuPHPet environment. PuPHPet itself does not need this workflow. PuPHPet needs only the required packages and may copy and compile the needed puppet modules during archive generation.

[jtreminio]

@frastel that is much easier to maintain than the mess I was doing. Thanks. I'll whip something up. I agree with getting rid of vagrant-puppet-lamp since it has served its purpose: letting me get puphpet up and running originally!

now that puphpet can do everything, there's no need for it anymore.

[frastel]

I extrated the composer installer to an own repository https://github.com/frastel/puphpeteer-installer and marked the linked installer project as demo https://github.com/frastel/puphpeteer-installer-demo. Maybe someone could re-use this idea too (I don't think so but anyway :) )

[jtreminio]

@frastel ok to move this into puphpet org and place into a vendors directory?

./vendors/frastel/puphpeteer, for example?

[frastel]

I thought of this possibility for PuPHPet: create a "puphpeteer" repo in the PuPHPet org where we require all the needed puppet modules in composer.json. Nothing more is done there. The messy and long repository list is then hidden from PuPHPet itself.
PuPHPet itself requires puphpeteer and the puphpeteer installer. When we run Composer on PuPHPet all required puppet modules from pupheteer are automatically installed. The installer puts them into own modules dir (but this latter step is not really needed, as PuPHPet knows the locations in the vendor path too). PuPHPet now has access to all required puppet modules and could select the needed ones for the user's file archive.
Puphpeteer and puphpeteer installer is without any further logic. If somebody needs it she could re-use it. If not, no problem too :)

[jtreminio]

That sounds like a plan to me!

I am testing out the full integration of your puphpeteer-installer on the main repo right now. It'll be easy to move the completed changes to another repo.

[jtreminio]

@frastel So I'm coming across an issue here.

Your puphpeteer idea is a good one, and makes maintenance much easier than what I've got set up.

However, many of the repos we're using, particularly example42 modules, don't use tags, or branches, or anything really. We can still pull those, but per Composer we'll need to delete the repo from local before we can update to latest version and our composer.lock file won't be stable.

Also, we won't be able to vet any changes made to those repos. Anything could be added without us knowing that could break installations. For example, example42 modules require https://github.com/example42/puppi. What if they begin requiring something down the line?

What if a module starts going off in a direction we do not want?

It seems depending on outside repos for our modules may introduce some headaches down the road.

What do you think of sticking with the way I've set up using the https://github.com/puphpet/puphpet-module-installer . It'll require us to occasionally go in and update the modules if we want anything new - or not. We could keep them frozen at the commit we cloned, which is a known, working state. No need for any new features unless we specifically need them.

[frastel]

I was aware of this problem. The only thing missing in the repos are composer.json files. We have to possibilities here: Fork the project, add composer.json and add the requirements on those forks. Or add according PRs. The additional composer.json do not break something and I think the repository owners are open for this additions.

I prefer the latter one. When we would point to our own forks it would be too tempting to modify them to our needs and then we have again the same problem we xurrently have with vagrant-puppet-lamp.

The puppet-module-installer would be a nice idea besides the fact that the composer.json file would be un-mergable.
The repository owners would not allow them because they are too PuPHPet specific. We need a way to require the repositories in the native supported way and that would mean with no-specific PuPHPet stuff included.

Tomorrow I will test librarian-puppet. I knew the native puppet module installer which only works with modules hosted on puppet forge and which does not support git (http://docs.puppetlabs.com/puppet/2.7/reference/modules_installing.html). However with librarian we could access the git repositories too and fetch the versions we need.
In this case it would mean that during PuPHPet development and on the production host librarian-puppet has to be installed. The puppet modules wouldn't be located in the vendor dir then.

As we PHP developer I would love to install everything with composer though. But you are completely right, we have to find a way where we are able to pick up the module versions we want.

[frastel]

@jtreminio the schema claims (http://getcomposer.org/doc/04-schema.md#package-links) that it is possible to pick revisions by their sha values, e.g dev-master#somehash. If that will work for repositories without any composer.json file we are done. Although we have to manually update the hashes when really no release or tag is available for the required project.

[frastel]

Ok tested a little bit and it seems to work:

• without composer.json
• with composer.json

Repositories with composer.json have to be referenced as "dev-master#somehash" and repositories without them have to define the hash in the reference property in the package definition.

        {
            "type": "package",
            "package": {
                "name": "frastel/puppet-phpmyadmin",
                "version": "master",
                "source": {
                    "url": "git://github.com/frastel/puppet-phpmyadmin.git",
                    "type": "git",
                    "reference": "dbfcbf6"
                }
            }
        },

This should be enough or?

[jtreminio]

@Seldaek was pretty against this idea: composer/composer#1583

That said, if it works, I am definitely open to using it.

[frastel]

I think this will be no problem for us as we have to require every package on our own and do not use nested packages.
Besides Jordi prefers referencing packages by tags or branches. I think there is nothing to say against this referencing. He even suggested it on stackoverflow this way.

[jtreminio]

[frastel]

Just compared the php module with the original puppet-php repository. Maybe we should discuss in short about the differences before we are moving code:

• augeas.pp: is this needed?
• ini.pp + composer.pp: should be in an extra puppet module, maybe in puppet-puphpet?
• devel.pp + pear.pp: could be added via php::module

Should we move all additions of the php modules to puppet-puphpet? Resources would be called with Class {'puphpet::composer': ...} then. Not every change is worth a PR to the original repository and we should split up the original code and our custom "hacks" as much as possible. I would prefer going this way first before poluting a clean puppet-php fork.

Is there any "hack" needed for defining the PuPHPet development environment? Checked the manifest but I do not think so. If hacks are needed for defining the environment, we should maybe put this code to a seperate repository (puppet-puphpetdev for example) which is not used for PuPHPet itself.

[jtreminio]

Using a puphpet module is a good idea.

[jtreminio]

We should take this opportunity to finally get rid of puphpet/puppet-vagrant-lamp and do it right

[jtreminio]

With commit 9f777cc , puphpet/vagrant-puppet-lamp is no longer needed by PuPHPet.

@frastel I will close this ticket and open a new one to separate out our custom additions to the php module into their own modules.