Automatic Promotion of stable code

We are working to achieve automatic promotion of "good" revisions to main. First of all, what do we mean with "good"? Right now we measure this as a set of builds and tests that are run successfully on a specific code revision of pi. When this happens then all the code up to that revision is selected to be promoted to the stable repository main.

Our goals:

• Main only includes stable code.
• Code is promoted from the development repositories to main as fast as possible. And as fast as the code quality permits it.

Current Model
We have pi, main and Continuois Integration engine

• pi: The pre-integration repository. Developers push directly here.
• main: The stable repository. A manual merge is done before the release time.
• builds: Integrated with Hudson tool, that executes about 12 builds/test on main and pi.

Process
This is how pi and main fit in our Continuous Integration engine:

• Developers commit locally and push their changes to pi.
• There are around 12 builds and tests that are polling the SCM for changes to the pi repository If there are changes the builds are run. If they fail developers are notified in the IRC channel #openbravo and in the development mailing list.

The main repository comes to action on the release time:

• When the release start date comes, we make sure that all the tests in pi run successfully.
• Once everything is green and good, we merge pi into main.
• QA starts the manual test process, testing those parts that are not automated yet.
• If major issues are found, we transplant changesets from pi into main.

Issues/Drawbacks of this model

• Main does not always contain stable code. Whenever we do a transplant there's a potential risk that so we need to run all the tests in main again. So main can only be considered reliable on release time. And we want the tip of the repository to always be trustworthy.
• Experience has shown that pi tends to be unstable. And this annoys developers and the release engineering team (us).
• Depending on the number of the commits and changesets pushed to pi since the merge to main was done, if a transplant is required there is a potential risk. Because pi is more advanced in features and fixes. And we don't want to freeze pi.
  

Proposed Model
We use Mercurial as our SCM, so its distributed nature will be an invaluable help solving this problem. We are already running bunch of builds and tests. This is great. We now wanted to automatically mark those revisions as "good" or "tested".

Need for integration stage
Doing integration in main is a bad idea, doing it in pi is even worse. We already have pre-integration and the final main repository. So having a integration repository is a natural choice that fits in this model.

Now we have pi, main, int and builds
How would this work then with a integration repository? (let's call it "int")

• Developers work on pi. A set of tests are run on pi to detect silly problems.
• int pulls from pi from time to time, and triggers a set of builds and tests.
• If all the builds and tests are run successful, we can consider the tip of int as "good". So int pushes all the changesets to main.
• Repeat the process forever.

Example
Step 1: Create first/top job that does

* Pulls the last successfull changesets from pi into int.
* Clone int locally in the system (int-1)

Step 2: Incremental build for PostgreSQL (erp_devel_int-inc-pgsql).
* This job polls from int-1. If there are changes it runs the job.
* If the job is successful, it pushes the changesets to a new repository, int-2.

Step3: Incremental build for Oracle (erp_devel_int-inc-oracle).
* This job polls from int-2. If there are changes it runs the job.
* If the job is successful, it pushes the changesets to a new repository, int-3.
(. . .)
Step 11: Smoke test on Oracle (erp_devel_int-oracle-smoke-test).
* This job polls from int-10. If there are changes it runs the job.

Step 12: Promote pi to main (erp_promote_pi_to_main)
* If the job is successful, it pushes the changesets from int-11 to main

What do we achieve with this model?

• Logical order: if an incremental job fails a full build will not be triggered. Because the full job is polling for changes from int-3, but as the job failed no push to int-3 has happened.
• The revision tested by the last job has been tested in all the jobs. So we have the guarantee that it has passed all the tests. And we can push it to main.
• The short-time jobs do not have to wait for the long time jobs. Not all the revisions tested by job 1 are tested by job 2. But the opposite is always true. All the jobs tested by 2 have been tested by 1.
• The model does not depend on a specific Continuous Integration software.

Continuous Integration-Hudson-Architecture-Install

Continuous Integration (CI)

The term "continuous integration" refers to a process that builds and tests code on a frequent basis.

• Continuous integration servers constantly monitor source code repositories
• As soon as new changes/commits are detected, they initiate a new build cycle.
• Build cycle actually involves code compilation and, in addition, may involve various tests and code analysis.
• If the process encounters errors, it notify the build master

Openbravo leading professional web-based Open Source ERP! decided to use Continuous Integration, a software development practice to make its team integrate their work often. I explored on few CI tools like Cruise Control (CC), Buildbot and Hudson. I liked Hudson for reason likely

• Installation and Configuration (friendly web GUI with extensive on-the-fly error checks , in-line help.)
• Extensibility through plugin
• Permanent links (gives you clean readable URLs for most of its pages)

Hudson Architecture
Hudson a continuous integration tool, which schedules the job process as cron job,

• It is primarily a set of Java classes likely where Hudson is the root Object model, it has 'project class' and 'build class' and some 'interfaces' to perform some part of build like scm
• Hudson classes are bound to Staple (Stapler is a library that "staples" your application objects to URLs, making it easier to write web applications. The core idea of Stapler is to automatically assign URLs for your objects, creating an intuitive URL hierarchy.)
• To render a html pages hudson uses Jelly (is a tool for turning XML into executable code. So Jelly is a Java and XML based scripting and processing engine)
• It uses file system to store, data directories are stored in HUDSON_HOME as plain text like the console-output (few as java properties file format, Majority of them uses Xstream, eg : project configuration, or various records of the build)
• Hudson can be installed either by running the hudson.jar file(ex. java -jar hudson.jar) or just by deploying it in a servlet container.
  

How I deployed

I deployed Hudon tomcat on port 8080 and redirected to port 80 (default) using mod_jk and apache on a gentoo operating system

• deploy the war file in tomcat directory
• Add "-D JK" to /etc/conf.d/apache2, in the APACHE2_OPTS line.
• Add "JkMount /* ajp13", at the end, before to /etc/apache2/modules.d/88_mod_jk.conf

check Openbravo's Continuous Integration system here