This post was originally published by Ory Henn on Wix engineering blog, who is part of our continuous integration / continuous deployment team. Since I write about Wix’s continuous delivery I think this post might interest you.
There Are So Many CI/CD Tools, Why Write One In-house?
About 3 years ago we first set foot on the long and winding road to working in a full CI/CD mode. Since then we have made some impressive strides, finding the way to better development and deployment while handling several years’ worth of legacy code.
An important lesson learned was that CI/CD is a complex process, both in terms of conceptual changes required from developers and in terms of integrating quite a few tools and sub-processes into a coherent and cohesive system. It quickly became clear that with the growing size of the development team, a central point of entry into the system was needed. This service should provide a combined dashboard and action center for CI/CD, to prevent day to day development from becoming dependent on many scattered tools and requirements.
To this end, we started building Wix Lifecycle more than two years ago. This post describes Lifecycle in broad terms, giving an overview of what it can do (and a little about what it is going to be able to do). Following posts will describe interesting series and other fun stuff the system does.
Keep in mind that our processes are nowhere near full CI/CD yet, and that Wix Lifecycle has to allow a great deal of backward compatibility for processes that are remnants of earlier days, and for teams that move towards the end goal at a different pace.
So What Does It Do?
A full CI/CD process is a complicated creature. Not only does it require of many developers a significant change in their mindset and working practices, it also relies on many tools which are crucial for the process. Many of these tools are used in ‘classic’ development processes as well (everybody needs a VCS), but a main tenet of CI/CD is to make the process as automatic as possible (ideally completely).
In our experience this meant we had to write a single system, integrating all our tools and services in one central dashboard / action center. In my opinion, any company which embraces CI/CD in today’s tooling ecosystem will need to craft a similar system to suit its own needs, handling quirks and idiosyncrasies peculiar to its own build, deployment and monitoring process.
In our case, Lifecycle handles (either directly or through links to other services) all tasks associated with building, deploying, monitoring and – if necessary – rolling back our artifacts. In our case, this means that Lifecycle integrates with:
VCS – in our case both a local Gitorious installation and github projects.
Build servers – 2 instances of JetBrains TeamCity
Maven – through our own release plugin
Issue Tracker – Jira
Deployment services – handled by Chef through in-house services
Monitoring – NewRelic monitoring and our own AppInfo (method-level monitoring on individual artifacts) and other internal services.
As you will see below, Lifecycle has action buttons that allow users to control the deployment cycle of their artifacts. A user can release a build – provided it is green in CI – which sets the internal versions of dependencies, and allows it to pass to manual QA (where required). Released builds can be deployed to a ‘testbed’ production server, GA-ed (deployed to all relevant servers), and rolled back if necessary. In the same fashion, ‘Hotfix’ branches can be created in a project (as I mentioned, we are not fully CI/CD, so this happens) and artifacts can be released and deployed from those branches as well.
In addition, Lifecycle provides many discrete small services and reports which are easily generated once such a system is in place. E.g. a ‘where is my jar’ API that allows a developer to find out which configurations in TeamCity are responsible for the jar on which she just discovered a dependency, or an ‘Idle Builds’ report which shows projects that were not deployed for a specified time interval.
And How Does It Look?
On the left we have a list of TeamCity projects (the dark grey lines) and child build configurations (lighter grey lines). Each user adds the projects and builds relevant to her, to see a concise status of the relevant parts of the system.
The right side of the screen shows system events – current and recent deployments, summaries of pertinent NewRelic reports, system events from the Lifecycle application itself, and current running builds.
The middle section shows details for a selected project. The header has the build name, owner, and links to the Gitorious repository, CI and Release TeamCity servers, NewRelic monitoring for this build’s artifact, and a couple of internal monitoring services.
Below the header are two sets of buttons – Action buttons in dark blue, and information buttons in lighter blue. The pic below shows the central info panel when each of the buttons is selected. Naturally, artifact names link to the relevant Artifactory location. Server names, on the other hand, link to our internal server monitoring (AppInfo) screen on each server for the specific deployable associated with the build configuration.
How Does It Work, Really?
Lifecycle is written in Java, with the frontend developed in AngularJS. In most cases, it integrates with other tools through REST APIs. When needed, and possible, we extend those APIs for our own purposes. For example, you can see our extension of the TeamCity REST API at https://github.com/wix/teamcity-rest-api-extension. As mentioned above, we have our own Maven release plugin as well.
Wherever possible, we make use of existing features provided by external tools. Artifactory, Maven and TeamCity are pretty well integrated. We make extensive use of TeamCity API (originally a plugin, now part of TeamCity), as well as the tcWebHooks plugin (at http://netwolfuk.wordpress.com/teamcity-plugins/). Similarly, while we have our own Maven plugin for releasing projects, we prefer using existing plugins wherever possible.
To touch on a point made early in this post, while we believe that every company above a certain size will need to create its own CI/CD system (at least for the near future), the complexity of such a system makes relying on external tools all the more important.
What’s Next for Lifecycle?
Many more things are in store for Lifecycle, and more requirements become apparent as the development team grows.
The main things on the table in the short term are a better users/roles/permissions mechanism, a Build/project management system that will let users create and manage TC builds/projects; Git repos and more from the Lifecycle app itself; and some UI improvements.