This is the introductory post about the repositories in the GitHub Cookbooks account. It (belatedly) addresses the topic “Centralized cookbook-library repos vs distributed cookbook repos”, discussed in this thread.

I also describe unpleasant side effects of using Bundler 1.0.7 that are being worked on.
I’m in the process of trying to make Bundler accomodate Chef’s use case, hence the Alpha tag on this post. The reason why I’m describing something that is clearly incomplete is in response to Indirect’s question about the pain points of using Bundler (as of <= 1.0.7, and possibly higher versions).

The Github Cookbooks account repo’s are trial repo’s and should not yet be forked since they WILL be deleted and replaced, or abandoned if feedback (hint) and experience dictates.

The objective is to carve individiual Chef cookbooks from the monolithic cookbook-library repositories. The two cookbook-library repositories are those of Opscode and 37Signals. Doing this requires adopting some dependency management mechanism, since one reason for these cookbook ‘silos’ was as a quasi-dependency-management solution (see the mail list discussion).

Now, some details.

Chef cookbooks on Github:

Each Cookbooks repo is a pure Git repo and initializing the repo using Niev’s git-flow does not restrict you to using Niev’s git-flow. An initial description of what this workflow might look like in the context of Chef wil be in another post if interest in these repos warrants this. Anyway, it is nothing radically different from what is described here. Each repository name has a cc- prefix to disambiguate it from the actual application name, the Opscode/37 Signals repos, and to help when searching. There are currently no wiki or gh-pages added, however it might be useful to add a common howto page to each repo describing any best-practice workflows.
Please comment below or track-back to your blog-post about your suggestions/thoughts.

Bundler:

Bundler is just one way of gathering Chef cookbooks into the cookbooks folder of the the local chef-repo – from there they are uploaded to the Chef server via knife. Bundler’s Gemfile allows you to set out the cookbooks you wish to gather to make up your application stack, and eventually specify different groups of that stack in one simple file. The rational for choosing Bundler was: Grabbing Chef cookbooks for an application stack should be as fine-grained and simple as:

To get the above to work required adding the install-path feature to my Bundler fork, and shaving a herd of Yaks I bumped into. If you’d like this feature added to Bundler, please find and vote on, or open a Bundler issue.

Now to the possibly contentious part.

Each cookbook’s metadata.json file is parsed in the gemspec file, cc.gemspec. This gemspec has generic contents, so it has a generic name to convey that point. Nonetheless, gem build cc.gemspec will generate a gem file with correct name – in case you have dreamed up some way of doing interesting things with Chef cookbook gems…. The cc.gemspec is parsed by Bundler. Consequently, the version requirements in the cc.gemspec are checked by Bundler inspecting the cc.gemspec in each repo.

Potential benefits:

• Cookbook development can be more atomic, and each cookbook now has a dedicated issues forum.
• Cookbook changes/branches/tags/forks are more visible and cookbook commit history has been preserved
• A single repo houses both Opscode (live and qa branches) and 37 Signals (37s branch) cookbooks. Cookbook styles (37 Signals c.f. Opscode) may converge more easily (merging commits between branches)
• The extracted repositories can be bundle installed into a local Chef repo using my Chef branch of Bundler
• Repositories use Nvie’s Git workflow branching ideas in setting up live and qa branches. Comments welcome on branch names production, development and testing
• Each cookbooks dependency/version requirements are in the cc.gemspec for Bundler to resolve
• Additional control/flexibility is obtained via Git forks, refs, tags and branches. Cookbook deployment can be made from your fork’s latest master commit down to the level of a reference hash. You are no longer restricted to defining dependency by version numbers in the metadata file. This is more useful in dev and testing phases.
• Additional flexibility will be obtained from Bundlers groups, elaborated here. Specifically, when Bundler supports different versions (Git hashes) between different groups.
• Different Cookbook ‘styles’ (see here) coexist as a different branch, e.g. the 37s branches for 37 Signals cookbooks.
• Additional opportunities for people to participate as maintainers/collaborators of cookbooks they have a special interest in.
• Tagged cookbooks are, via GitHub download, easily accessible outside of Git

Potential costs

• More knowledgeable collaborators required to be active to review changes
• Cookbook styles (37 signals cf Opscode) may fragment further, see here for some discussion.
• Roles have to be explicit about which cookbook they are using. This involves more typing since Bundler delivers the cookbook decorated with a hash (I am working to eliminate this).

Example: The base role from Opscode rails-quick-start:

Using Opscode’s rails-quick-start repo the role description is:

Which with Bundler hash decoration would have to become this

Furthermore this would have to be edited everytime you updated your cookbooks with Bundler… ouch.

The Yeast

In the thread “Centralized cookbook-library repos vs distributed cookbook repos” Adam Jacob commented:

our job is to provide you guys the primitives you need to be able to do things the way you want to do them… it’s just me trying to suss-out where the primitives are.

This is a fantastic attitude, and I think that this usecase indicates that the Chef primitives here are the version and dependency data, parsed from metadata.json to the cc.gemspec. Hopefully they can stay.
Additional primitives are

1. the Git forks/branches/references you pull from, but that is an additional layer of flexibility outside of Chef’s scope, but within Bundler’s scope.
2. Chef environments c.f. Bundler groups. When (if?) Bundler allows different versions between groups we will have the ability to bundle install different Cookbook versions into different Chef environments – assuming Chef > 0.9 accomodates this (see this thread for a description of how this might work). Until then, the notion of a group of Cookbooks is not fully exploited. It is not yet clear to me if Chef’s upcoming environments feature will accomodate Bundler mapping (installing) its groups to Chef environments. A the time of writing the Chef documents suggest an environment’s cookbooks are specified in a file.
   Hopefully this will be answered in the positive in a Beta post on this topic.

Adam also summarises, correctly in my view, that there are three things teased out in that thread. I’ve inserted how I see Bundler+Git fitting in to the three:

• “How do you discover cookbooks you want to use?”
  • cc- is the naming convention in Google/Github searches for cookbooks people have contributed.
  • You feed Bundler a Gemfile you’ve been given describing the application stack you want installed and configured.
  • Any Git server becomes one (not the) publishing/distribution platform for cookbooks.
  • Github downloads provide non-git access (most useful when version tags are added to each repository. The following cookbook would be extracted with folder name cookbooks-cc-activemq-6f442c7

• “How do you track them over time, and potentially make site-specific changes, and track those over time?”

  • A classical Git repo per cookbook, either upstream or, equivalently, someone’s public|private fork, allows you to track site specific changes over time. Hopefully, just until they get accepted back into the upstream repo.
  • Cookbook tracking is explicitly controlled in the Gemfile, and can be at the level of Git forks, branches, tags, or tree hashes.

• “How do you track and resolve the dependencies one cookbook has on another?”

  • According to a cookbooks metadata and the Git fork/branch/tag/hash you pull from, as parsed from the cc.gemspec and Gemfile, and resolved by Bundler.

Example: Grabbing the rails-quick-start cookbooks

The environement is Ruby 1.9.2 via RVM. And assumes you’ve followed Opscode’s getting started.

I’ll demonstrate grabbing the Ruby-on-Rails quick start cookbooks, without the Amazon EC2 details. See here for full detail on the EC2 instructions.

Acquire Cookbooks

What follows is one way to describe application stacks. The Gemfile created is an alternative to
setting up cookbooks in a dedicated repository like: rails-quick-start.

Gemfile construction steps

• Add one cookbook (:branch => ‘live’) listed in the rails-quick-start’s cookbooks folder.
• Running Bundler’s install during construction of Gemfile:

• Adding missing dependencies (indented) or, if none, adding the next cookbook.
  • Not done here, but you could change the :branch to a :ref once the application stack is confirmed as working.
• Running Bundler’s install to point to the finished cookbook repo:

The finished Gemfile, rails-quick-start.gemfile, looks like this:

Download to your cookbooks folder

Kitchen sinks

At the moment Opscode and 37 Signals have single repo’s with multiple cookboooks. You could replicate their current respositories by creating a superproject with each of the repositories at Cookbooks as a submodule. In fact, you could have one repo with: live, qa, and 37s branches. Each branch corresponding to the branches in the individual repos.
As they say in the Math’s books ‘We leave this as an exercise for the reader’.

WIP Caveats

The current repository structure can be used, with some pain in the case of Bundler.
When downloaded via bundler, changes to recipe names in role files would be required, so would only be useful in production settings – where you want to pin a cookbook down.

Note 1:

Currently some work is still needed on Bundler for it to be used painlessly. – The hash decoration of the folder names needs to be eliminated, or accept an alias – bundle update and bundle check should accept --gemfile and --install-path – bundle could accept a --with counterpart to --without

Note 2:

There is a bug in how Bundler handles Gem specification files in Git repositories. Consequently, you’ll see this error once dependencies have been resolved for a gem:

Source does not contain any versions of 'cc- (>= 0, runtime)'

In which case add the next cookbook in step (3). This has the side effect of not generating the lock file which, when the bug is fixed, the lock file will be a very handy way of finding the hash – in case you want to pin your cookbooks that precisely.

Note 3:

Another limiation of Bundler 1.0 is that you cannot have different versions in different groups. This has been mentioned on the maillist, and may change, but is not explicitly listed as an upcoming feature in the issues tracker.