Whilst integrating issues into Moodle it's a multiple times per day occurrence to run phpunit tests on the whole Moodle codebase. Each day we add more and more tests and so the time taken to run these tests increases gradually. I'm not one for sword fighting on wheely chairs, so whilst waiting for phpunit tests to run I'll generally move onto another task, then frequently i'd forget completely that the phpunit run was going on in the background and not know what the phpunit output was showing me, need to run the tests again and fall into the same trap!

In order to resolve this situation I was inspired by Brett Terpstra's post about using the ruby gem terminal-notifier to send messages to Mountain Lion's notification center and decided to use this to remind me of phpunit executions.

After installing terminal-notifier and using Bret's finished.rb script I added a bash function for phpunit:

function phpunit() {
    vendor/bin/phpunit $@
    if [[ $? == 0 ]]; then
        finished "phpunit OK"
    else
        finished "phpunit FAILED"
    fi
}

In order that I don't miss a phpunit run I've configured the notifications to be 'Alert style notifications' (configured in OS X Preferences > Notifications > terminal-notifier) which works great, the only remaining issue is there does not seem to be a way to dismiss alerts using keyboard shortcuts.

I have just released a Moodle plugin for Skydrive in moodle.org/plugins/. Please try it out, enjoy and help improve the documentation!

Some developer background..

In Moodle 2.3, I converted the Google repository/portfolio plugins to authenticate using OAuth 2.0 as this was the recommended way forward from Google. Much of this work was simplified as I had already created a skydrive plugin for 2.2 using OAuth 2.0 (there were no good php OAuth 2.0 libraries at the time). Both Google and Microsoft now authenticate for their apis using OAuth 2.0 and so in lib/oauthlib.php we now have an OAuth 2.0 client class which is used for both Microsoft and Google. This plugin was updated to work with the new class in Moodle 2.3. Hopefully all the rest of the repository plugin providers will convert to OAuth 2.0 soon. Hurray for open standards :)

Thanks

Thanks to Universidad Teconológica de Chile (INACAP) who funded this plugin to be created by LUNS and to LUNS for releasing this work open source, allowing me to continue to work on this for Moodle 2.3.

The integration team

The integration team is staffed by a group of developers at Moodle HQ. We are a geographically disparate team, spread across the world in order to prevent natural disasters and political unrest from affecting the Moodle weekly release cycle. ((That might not be the reason.. ;-) )) Aparup and I are located in Perth, Australia; Eloy in Lardero, Spain and Sam in Nelson, New Zealand.

Together we have 20 years of Moodle development experience and this collective experience is an important part of how we work together as a team - knowing the idiosyncrasies of Moodle - how parts of the system have come to being and understanding how changes can potentially affect different areas which might otherwise be missed. I should say that since i'm the newest member of this team (and still learning the ropes) I hope that my comments are accurate!

The integration review

The integration team is responsible for 'pulling' code from all developers around the world, reviewing and bringing it together (integrating it!) to make up the Moodle core release. This process is the same for all developers no matter whether they work for Moodle HQ, a University, Moodle Partner or none of the above.

The purpose of the integration review could be misconceived simply as as a final technical code review before code gets into core. That is certainly a large part of it - we are striving to continually increase the quality of Moodle releases through technical reviews, increased testing and other processes. But the integration review is also about examining the impact of each change from our users point of view - ensuring that we don't loose focus of Moodle as supporting education! Other aspects of the integration review include ensuring that discussion has taken place with interested parties before a change lands and facilitating that if it hasn't as well as providing guidance and feedback to contributors - helping developers learn why we might want changes or if there is something that could be useful for next time.

Anyway enough preamble, the title promised mechanics so..

Down to mechanics

At the start of each week we being a new integration cycle and on average there are 30-40 issues waiting for integration into Moodle core. The process begins with an integrator moving issues waiting for integration into the 'current integration'. Issues marked as in the 'current integration' are those which we will work on for this cycle. We make this distinction simply to avoid the bucket of issues we are working to integrate for the weekly release from constantly refilling. The integrators work through the queue of issues waiting for integration one by one - in this case I cheated a bit and allocated myself MDL-32039 early, as I thought it'd be a good opportunity to do a follow up post to Tim.

Reviewing the History

The first step is to mark an issue as 'in integration review' in the tracker workflow (pictured) and starting to read back the comments in the issue - learning about the history of this change. In this case the change is fairly obvious and straight forward so there is not too much to consider. But sometimes there might be debate about a particular approach or how to solve an issue and comments from peer reviewers or watchers of the issue. Or indeed there might be no discussion at all when someone is proposing do something particularly radical which is equally important to note. As a brief side note - I personally encourage all developers to 'vocalise' even their internal though process through bug comments and commit messages. Its an often forgoten about communication method but comments on a bug like this can help reviewers now understand the decisions without too much back and forth. And perhaps more importantly months and years down the line it can be used by other developers to understand why a change was introduced even if you are no longer working on Moodle or if you've forgotten.

Reviewing the code

At the point of integration review, we review changes 'in situation' to see how the fit with other integrated issues. So the first step is to pull the change into the master integration branch.

danp@marge ~/git/integration> git checkout master
Switched to branch 'master'
danp@marge ~/git/integration> git fetch origin
danp@marge ~/git/integration> git reset --hard origin/master 
HEAD is now at abd2899 MDL-31768 - it is not possible to add a picture to the thanks page in feedback

Some notes about this first step. On git.moodle.org we have two repositories, the production moodle.git (which almost everyone should be using) is the result of what the integration process produces. We also have integration.git which is our 'staging' repository where we pull changes, integrate and test - this is where integration work happens before deploying to moodle.git. integration.git is intentionally separate and volatile, this is where we iron out the kinks before pushing to the production moodle.git. I have a separate integration checkout which I am using here for reviewing changes and pushing to the integration repository. To explain what I did - I first checked out the master branch of the integration repository and brutally reset this to the current state of the integration repository to accept new changes. In most git workflows this is not an advisable strategy but the way I use my integration repository I always want it to be completely clean and blow away any changes (in fact I have a script which does this on all branches) so I am happy to do this. Now to pull Tim's changes:

danp@marge ~/git/integration> git pull git://github.com/timhunt/moodle.git MDL-32039
From git://github.com/timhunt/moodle
 * branch            MDL-32039  -> FETCH_HEAD
Merge made by the 'recursive' strategy.
 .../lang/en/tool_qeupgradehelper.php               |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

I now have Tim's changes in the master integration branch and it merged successfully without issue. This was straight forward - which is not surprising as its a fairly self contained issue and nobody else has been working on that language file this week. But simple merges are not always the case. We ask developers to rebase their changes against the latest weekly release to reduce the frequency of conflicts, but sometimes the integrator will have two changes integrated during the same week and there will be conflicts. Common examples of merge conflicts are multiple changes to the major version number in one week. This is where integrators need to examine the differences and either resolve the conflict manually or delay an issue as it might be affected by another issue which is conflicting. Since that was straight forward I can move straight onto reviewing the change:

danp@marge ~/git/integration> git diff origin/master..master --stat
 .../lang/en/tool_qeupgradehelper.php               |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
danp@marge ~/git/integration> git diff origin/master..master

First I look at a quick summary of lines changed - this is useful to find if there are any unexpected places where lines are changed. Then I look at a diff in unified format - which when reviewing open source code you get very used to. But since this is a language change I actually decide to review it in a more readable format by using the --word-diff option of git diff.

danp@marge ~/git/integration> git diff origin/master..master --word-diff

In --word-diff format we are much more easily able to see this change and it looks good. In other circumstances the --ignore-space-change option can also be useful to find just the changes that have happened ignoring space changes such as indentations (when introducing a conditional and so forth). We will also look around the files in full form and history of file changes around the area in question. I tend to use the command line and vim, others will use IDEs such as netbeans and eclipse.

Commence pre-flight checks

At this stage I have reviewed the change, tested it and understood the impact and i'm confident in the code change that i'm ready to integrate the change. However, we have another tool up our sleeves for checking changes before integrating - our loyal servant Jenkins. Jenkins is a continuous integration server which we are in the early stages of integrating into the Moodle workflow thanks to some great work by Eloy.

I am able to run a pre-check on Tims branch before integrating. This pre-check will run the various code checkers against Tim's change to spot problems and then produce a report on this. In the future its planned that this will happen without an integrators involvement automatically when an issue is submitted for integration. Luckily (and as expected) the checks ran successfully and no issues were found so I am now ready to push to integration.

Rinse and repeat

Since the change in question affects multiple branches and not just master I need to repeat the review process for all other branches required (2.1 and 2.2 in this case). In the interests of keeping you awake, i've spared you this process. :)

Pushing.. done [1]

The time has come to push to the integration repository:

danp@marge ~/git/integration> git push integration-push master MOODLE_22_STABLE MOODLE_21_STABLE
Compressing objects: 100% (24/24), done.
Writing objects: 100% (26/26), 3.14 KiB, done
   21cadef..7d7331b  MOODLE_21_STABLE -> MOODLE_21_STABLE
   18ca35b..41b4bb0  MOODLE_22_STABLE -> MOODLE_22_STABLE
   abd2899..6461e14  master -> master

Done? Well not quite.. at this point the previously mentioned continuous integration server notices the change in the integration repository which has been pushed and starts running a series of tests on the changed branches. There are too many checks to go into here and they continually being evolved, but for example we will run all unit tests and ensure 'the build is not broken' as well as conduct fresh installs and upgrades and compare that upgraded data is exactly matching newly installed data. This automated testing stage can prevent serious and difficult to spot regressions before they make their way into production and indeed in last weeks integration cycle it caught two such bugs.

Mark as Integrated

Once the change has been pushed, we will then mark the change as integrated in the tracker and the worflow will change to mark the issue as ready for testing.

Unresolved

Since this blog post has grown to much longer than I expected, I will stop my description of the integration process as this point. I hope some of you made it to the end :)

[1] The title is a little in-joke as we integrators have a simple semaphore system to help avoid unnecessary conflicts by first calling out 'pushing' (on a chat channel) when starting the process of pushing branches and releasing 'our lock' completing with another shout of 'done'.

After seven years working at LUNS and nine years living in Lancaster, I've decided to have a change of scenery and i'm going to be leaving many great friends and colleagues behind to move to Moodle HQ in Perth, Australia. Needless to say, this was not an easy decision to make. (Although the weather like Lancaster demonstrated today could convince me otherwise!)

My involvement with Moodle started with LUNS and it has been a great journey. We were newcomers to the Moodle community with a small project to move a few keen high-schools onto supported hosting platform in 2005. My first post to moodle.org was asking the theme forums for advice on how to do multiple themes on a single codebase and the first patch I remember getting accepted was to help us solve that problem six months later: MDL-6784. Although the git history suggests my first fix was actually a horrible JS change which surprises me more than anyone!)) Today LUNS is one of the largest non-HQ code contributors to recent Moodle releases and have provided hundreds of thousands of people with access to Moodle through our various clients - primary school students[1], university lecturers[2], high school teachers, company employees and charity workers. I've thoroughly enjoyed that journey and the great people who have made it what it is.

I'm really looking forward to working for Moodle HQ! Despite being so close to Moodle as a developer, I continue to be amazed and motivated by how many people I can touch by working on Moodle. Friends from my days at school and university use Moodle as part of their job on a daily basis, my youngest sister is currently studying and supported with Moodle (unfortunately only at stage 1 of Martin's pedagogical stages) and I meet more and more moodlers ever day! I'm hoping that my time working as external developer working on the other side of the world to Perth will be a great new perspective to add to the team.

I'll be working for LUNS for a while yet, but it feels like a good opporunity to thank everyone i've worked with over the years at LUNS, clients and community members who accepted my patches. :) I'm touched by the kind post about my leaving. I expect to continue working with you if more indirectly at Moodle HQ. :)

Happy Moodling!

[1] Making their initial steps to learning how to use a computer [2] No comment

Eager to try this I hacked together a Moodle plugin for the siri proxy which would do lookups to the Moodle tracker. So, welcome Siri - the newest aid to the Moodle 2.2 QA testing effort :-)

(The plugin I wrote is available on github. It is my first ruby script and its not particularly elegant as it was just for fun :-) )