6/24/2013

Continuous Delivery - Part 7 - Cultural Change

Filed under: — By Aviran Mordo @ 12:49 pm

Previous chapter: Backward and forward compatibility

In order for continuous delivery to work the organization has to go a cultural change and switch to Dev-Centric Culture.

Continuous delivery gives a lot of responsibility in the hand of the developer and as such the developer need to have a sense of ownership. At Wix we say that it is the developer’s responsibility to bring a product / feature to production, and he is responsible from the feature inception to the actual delivery of the product and maintaining it on the production environment.

In order to do that several things have to happen.

Know the business :
The developer has to know the business and be connected to the product he is developing. By understanding the product the developers makes better decisions and better products. Developers are pretty smart people and they don’t need to have product specs in their face (nobody actually reads them). Our developers work together with the product mangers to determine what the product for the feature should be. Remember while the actual product may have several features bundled together for it to be valuable for the customer, we develop per feature and deploy it as soon as it is ready. This way both the product manager and the developer get to test and experience each feature on production (it is only exposed to them via Feature toggle) and determine if it is good enough, and may cause the direction of the product to change not according to plan, and actually develop a different next feature than the planned one.

Take ownership
Developers are taking ownership on the whole process, which means that they are the ones that need to eventually deploy to production. This statement actually changes the roles of several departments. What the company needs to do is to remove every obstacle in the developers way to deploy quickly on his own to production.

The operations department will no longer be doing the deployments. What they will do from now on is to create the automated tooling that will allow the developers to deploy on his own.
Operations together with dev will create the tooling to make the production environment visible and easy to understand to developers. Developers should not start a shell console (ssh) in order to view and know what is going on with the servers. We created web views for monitored metrics of both system and application metrics and exceptions.

Wix deployment dashboard
Wix deployment dashboard

BI and monitoring need to no longer create an on-demand reporting. They now need to enable real time KPI monitoring available and trigger alerts on any suspicious change especially post deployment.

QA department will stop doing manual QA, and start writing end to end automated tests. At Wix we initially had QA helping the server developers writing IT tests, however after about a year where we had a server QA we decided that we are proficient enough and well covered in automated tests that we do not need QA for the server side. We still use QA on the client side development (i.e UI), they do both automated and manual testing.

Team leads need to give a lot of freedom to the developers and help remove obstacles from their path to deploy.

Product We trained our product managers to think lean and feature centric and mandated that every feature is A/B tested. Developers and product managers sit in the same room and discuss the product during the whole development cycle. Today product managers open and close tests and feature toggles dozens of times a day and no longer assume they know what the users want, everything is tested.

Created a CI team that will develop the CI tooling and automate the staging environment creation. Today we can create staging environment with any version and install what ever services we choose with just a click of a button.

The most important change is actually going to Test Driven Development. No feature is considered production ready unless it is fully tested . Product managers, developers and management understand the importance of automated testing and we take that under consideration in our time estimates. We invest a lot in testing and always improving our testing framework.

Next chapter: Deploying To Production


 

2 Responses to “Continuous Delivery - Part 7 - Cultural Change”

  1. Deryl Spielman Says:

    I feel as if test driven development and the extra work behind setting up and testing the code for toggling doubles the amount of time it takes to complete a product or feature. Was there a culture shock surrounding the pressure to provide new features faster yet agreeing to slow down development time or was everyone in agreement from the beginning? What I’ve found is a difficulty in convincing non developers such as product owners and directors that automating testing up front and even providing thorough test plans should consume the time of an already limited resource such as a small dev team.

  2. Aviran Mordo Says:

    When we began this process we (the R&D) explained the whole methodology to the product owners. Product can not determine how long a new feature would take to develop. Dev teams should put in their time estimate also testing and should not even suggest or think about how long it will take to develop without tests. Development cycle includes tests, never do a double estimates of how long it will take you to develop with tests and how long without. As time passes product will understand the benefits and also get used to the new estimates.

    The same way dev can not tell product owners how to work, product should not tell dev how to do their job.

    Also there is one important thing dev and product should understand. Development usually takes less time and cost less money than debugging, deployment and shipping poor quality code to production and then spending a lot of time fixing bug that were found on production.

    Time spent on testing pays it self big time. Although it takes more time to develop it actually reduces the time to deploy to production because you have less QA (we don’t have any QA on the server side), your code quality is much better, you don’t have to worry about regression (because you are already covered with tests) and also refactoring and maintaining the code is easier, it eliminates the known phrase “if it works don’t touch it”. If you have good tests you should not be afraid to change any code.

    If you want to save time save it by eliminating documentation

Leave a Reply

You must have Javascript enabled in order to submit comments.

All fields are optional (except comment).
Some comments may be held for moderation (depends on spam filter) and not show up immediately.
Links will automatically get rel="nofollow" attribute to deter spammers.

Powered by WordPress