As a follow-up to an earlier post about ranking Homebrew formulae, this post describes creating a web page to show formulae by rank, with links to more information. The intent of this project, dubbed Ferment, is to enable discovery of new and interesting software that may not otherwise be noticed. The source code is on GitHub and (so far) looks like this in a browser:

Ferment running in a Vagrant box

In a nutshell, we analyze Homebrew’s GitHub repo and extract metadata from its catalog of software (a.k.a. formulae). The “first pass” takes a long time (several minutes) but subsequent updates are faster because the metadata is cached in a MongoDB database. To encapsulate the dependencies, we use Vagrant and define a Vagrantfile.

Running Ferment with Vagrant

Install VirtualBox, Vagrant and a Vagrant plugin called hostmanager. Then, simply launch the VM.

vagrant up

Upon vagrant up, Vagrant reads the Vagrantfile and creates a Ubuntu 14.04 LTS server, using the hostmanager plugin to assign a custom host name name (e.g. ferment.example.com).

Finally, Vagrantfile points to a provision script, which installs extra packages and performs configuration. This includes adding an hourly cron job to keep rankings up-to-date. In the provision script (and hourly via cron), the application calls gulp, which refers to a JavaScript build system called Gulp.

The Gulp build script

The heart of the application’s activity occurs in gulpfile.js, copied below for reference.

var gulp = require('gulp');
var react = require('gulp-react');
var git = require('gulp-git');
var fs = require('fs');
var shell = require('gulp-shell')

gulp.task('brew', function () {
  if (fs.existsSync('homebrew')) {
    git.pull('origin', 'master', { cwd: 'homebrew' }, function (err) {
      if (err) throw err;
    });
  } else {
    git.clone('https://github.com/Homebrew/homebrew', function (err) {
      if (err) throw err;
    });
  }
});

gulp.task('collect', shell.task([
  './bin/collect homebrew'
]));

gulp.task('info', shell.task([
  './bin/info homebrew'
]));

gulp.task('rank', shell.task([
  './bin/rank'
]));

gulp.task('dump', shell.task([
  './bin/dump > dist/data.json'
]));

gulp.task('copy', function () {
  return gulp.src('src/index.html')
    .pipe(gulp.dest('dist'))
})

gulp.task('react', function () {
  return gulp.src('src/index.js')
    .pipe(react())
    .pipe(gulp.dest('dist'));
})
 
gulp.task('default', [ 'brew', 'collect', 'info', 'rank', 'dump', 'copy', 'react' ], function () {
});

The default task executes all required tasks in sequence. Once complete, navigate to ferment.example.com to see the result.

Next steps

This iteration of the Ferment server carries some unnecessary baggage (e.g. Ruby) and the frontend is just my experimental trial of React. To be continued.