Business Need

OpenStack is a platform to build private and public cloud, it includes so many projects to do this, each project has its function. For example, Neutron is to provide network as a service, Nova is to provide compute as a service, etc. There is a large OpenStack community including developers and users to support it like developing new features. So one a big question is that how to verify source code from developers to make sure the source code is already in production. Although there is no any QA (Quality Assurance) and Tester.

Solution Approach

In order to answer this question, OpenStack community has launched CI (Continuous Integration). Each source code pushed will be verified carefully by CI server via so many test-cases.

The workflow of OpenStack contributing as below:

Image 1.1: Workflow contributing OpenStack

OpenStack uses GIT model. A developer will clone source code on master branch from GitHub (like Nova, Neutron, Keystone), then they have to check out a new branch with a name related to bugs or their feature. After coding the new feature then they must have unit tests or functional tests, even API test which depends on the scope of feature. After passing all old and new tests on local, they can push new source code to SCM (Sourcecode management). At that time, OpenStack CI will run all test cases in the project and mask +1 if passing and -1 if not.

So the requirement of each developer when uploading new feature or bugs is to write full unit tests and know how to set up a CI test for the feature if necessary. It will be reviewed by other developers and core-reviewer, they can discuss on the patch set to get the best solution. After getting an agreement then two core-reviewers will mask +2 and workflow +1, then CI server will run gate tests. If passing all gate tests then the source will be merged to master branch.

Infrastructure Design

Image 1.3: Zuul architecture
OpenStack uses Gerrit as SCM and Zuul as CI server. When a developer pushes a patch set to Gerrit then it is like this picture:
OpenStack uses Zuul as CI server and ansible playbook to control virtual machine.
  • Zuul: This component is the heart of Zuul, it listens to events from Gerrit and requests to other component run necessary jobs when having a patch set pushed.
  • Zuul merger: to merge patch set to GitHub. When passing all test cases then Zuul merger will start merging the patch set to GitHub.
  • Zuul launcher: to run ansible playbook and report the result to Zuul. After having VM information then Zuul will send it to Zuul launcher to run ansible playbook and send the result to Zull after finishing.
  • Nodepool: to manager Cloud backend. It is a middle layer to get requirements from Zuul and interact with OpenStack API to create VM for testing.
  • Cloud 1, 2, 3: This is private and public cloud, which are sponsored by Cloud providers. Especially, the private and public cloud is built from OpenStack with the last release. For example, the clouds are built on OpenStack Pike or Ocata.
When developers submit a patch set to Gerrit, then a notification will be sent to Zuul, then Zuul will request Cloud to provide a VM via Nodepool. Nodepool will send the information of the VM to Zuul, Zuul will request to Zuul Launcher with the information to run ansible playbook. After that, all test cases will be run on the VM to make sure source code is good. When core-reviewers accept this patch set then Zuul merger will start to merge this patch to GitHub repo. That’s all steps of a patch set before merging.
Currently, Zuul is running with some statistics:
  • 1500 repositories: There are more 1500 repositories in [OpenStack GitHub](, it means the repositories will be used CI system.
  • 2000 jobs per hour: CI server will create and delete 2000 VMs (Virtual Machine) per hour, that is an impressive number.
  • 13 regions of 5 public and 2 private OpenStack cloud: CI server backend is supported by the private and public clouds on 13 regions.
Conclusion: OpenStack is a good solution to build private and public cloud. Moreover, it is also a very good solution for backend in CICD. CI/CD server can interact with OpenStack system via API to provide VM for running test cases and deploy source code. After finishing, the VM will be deleted automatically to saving resources.

Leave a comment

Your email address will not be published. Required fields are marked *