We have been doing Continuous Delivery and DevOps at Wix for several years now and as part of that, I have been training my team at the methodology of DevOps and changing the company's culture. Last week I was trying to explain to some colleagues how we work at Wix and about our dev culture. While doing it I realized that we are doing much more than just DevOps. What we actually do is something I call “developer centric culture” or in short "Dev Centric" culture.
What is Dev Centric culture?
Dev Centric culture is about putting the developer in the center and create a support circle around him to enable the developer to create the product you need and operate it successfully.
Before I'll go into details about Dev Centric culture and why is it good let's look on the problems we have, what we are trying to solve and why it is more than just DevOps.
DevOps is about breaking the walls between developers and operations but after few years of doing DevOps and continuous delivery we realized that DevOps is not enough. The wall between the developers and the operation is just one wall out of many. If you really want to be agile (and we do) we need to break down ALL the walls.
Let's take a look at a traditional development pipeline.
Product that "tells" engineering what they need to do.
Software architect designs the system and "tells" the developer how to build it.
Developers write the code (and tests in case the work in TDD)
QA checks the developer's product and cover it with more test
Operations deploy the artifact to production
So what we have here is a series of walls. A wall between the product and engineering, a wall between engineering and QA and of course a wall between engineering and operation. (Sometimes there is even a wall between architecture team and developers)
When a product manager hands over a spec to a developer, the developers usually don't really read the whole spec, they misinterpret the product manager's intention, they lack the product manager's visions and they are also disconnected from the client's needs.
When you have a team of architects doing all the design and tell developers what to do, you again have a wall because developers might not understand the system's architecture and will not build what the architect intended.
When developers hand over their product to QA you create another wall because now QA need to guess how the system should work and there is also a shift in responsibility where developers don't feel ownership and responsibility to the quality of their product.
Then comes the last phase where Operations get a hold of the product and they are responsible to deploy and monitor a product they know nothing about.
So when something bad happens there is no one who is actually responsible to the product and you start with a blame game. If the product is not what the customer expected then product manager blames the developers that they did not develop the right thing. On the other hand developers blame the product managers for not defining the product clearly. If there is a bug (got forbid) on production then developer blame QA for not finding the bug, and QA blame the developers for writing a buggy software in the first place. If there are stability issues on production then everyone are pointing the figure to operations who do not maintain the product properly; on the other hand operation blame the developers for writing unstable software.
So as you can see there are many walls in the process that we need to break in order to have better quality.
So let's see how we can break these walls, how "Dev Centric" culture helps us with that.
Now if you think about a very small startup the developer is the key and is doing everything from talking to customer, define the product design the system, write the code, deploy and maintain it in production. This is also the phase where the startup is the most agile and progress very fast. Now if the startup succeeds and grow, more people join the company and slowly building walls around the different departments that eventually hurt productivity.
Now let's see in more details what walls do we have that we need to break down and how "Dev Centric" culture can help breaking it down and improve the quality and agility of the whole organization.
Wall #1: Product || Engineering The Problem: When getting a requirement to design a new feature or system, the product requirement is defined by the product manager. The product manager sits with the clients, understands their needs and interpret the needs to system requirements. The problem with that is that the product manager does not write the actual code and design the system. The one who is doing that is the developers. If developers get a design document, many times they don't understand the actual customer need, they miss-interpret the product manager thoughts (and between us, they don't usually read the whole product requirement document). This causes a waste of time because the developers might develop something completely wrong that don't meet the customer needs. If the product manager catch it on time the developers can fix the application before it gets to the customer, but it just wasted their time because they could have built is good in the first place.
In Dev Centric: Instead of product manager tell the developer what to do, the developer sit down with the product manager and TOGETHER they define the product vision. This way the developer gets to understand the product and the customer's needs. By understanding the product, the developer will create a much better system that fits the client's requirements. If we'll add to that continuous delivery and "lean startup" concepts, the developer is a very good resource to help the product managers to define the MVP (Minimal Viable Product) and making sure that no feature is being develop that is not needed.
Wall #2: Architect || Developers The problem: In many organizations once the product manager finished with the product definition, the "ball" is coming to the architect's court. The architect designs the system and hands the design to the developers to develop the application. In this scenario the developers don't completely understands the architecture and may not produce what the architect intended. Another side effect for this is that we treat the developer as a coder and not as a software engineer, thus limiting their ability to cope with the complexity of designing a good product. This also create a big workload on the architect who now has to supervise the engineering work and make sure they follow the design.
In Dev Centric: Instead of handing the designs to the developers, we need to invest in the developers growth and to teach them how to design a good system. Architecture should be done by the developers themselves while the architect should become a consultant and a supervisor. There is a challenge here for the architects because in many cases there is not one correct architecture and the developer might choose a different architecture than the architect would have chosen. The architect should be open to accept different approach as long as it is a good one even if it is different from what he thinks. Architects should mentor the developers in how to design a system, teach them how to approach a problem, how to analyze it and guide the developers to think about the architecture concerns the developers are not aware of or missed. By doing this you invest in your team and grow their knowledge, quality and abilities, thus creating a better products and better developers. Do not be afraid let the developers make mistakes. As long as they learn from the mistakes you will end up with better engineers. If architect will feed the developers with the answers all the time the developers will not learn to cope with complex problems on their own and will come to the architect on every problem, thus creating a much bigger workload on the architects.
Wall #3: Developers || QA The problem: When developers finish their work and hand it over to QA they are being put in a state of mind that they hand over the responsibility of ensuring the quality of their work to someone else. Instead of owning the product that they have just wrote it is up to somebody else to find their bugs. This case causes developers to not invest too much in the quality of their code and not feel responsible for their own work quality. Now when a bug is found on production the developers believe that it is the QA fault for not fining the bugs.
In Dev Centric: The solution for this problem is really simple. You need to make your developers write tests and be responsible for their own work. The only way to do that is to start doing Test Driven Development. Developers who work in TDD methodology write better code, their designs is much better because it has to be testable, with the right abstractions and separation of concerns. Now for developers who are not used to write test this can be a challenge, but from my experience, once they do that long enough the developers understand the benefit of TDD and never want to go back to not writing tests.
Developers should write both unit tests and integration tests. For back-end development this is usually enough, no QA is necessary unless in very rare cases where changes are very deep and might affect other system. At Wix the back-end group works without any QA. Now you might think that if developers do their own testing why we need QA for. Well you still need QA but QA should change their role. Instead of verifying the developer's work quality they should complement the developer's tests by writing automated system tests (end-to end) and on some cases do manual QA for UI components during development until an automated test is created for them.
Wall #4: Developer || Operations
The problem: In order to deploy the application the developers or QA hand over the artifact to operations and the operation people install or deploy the application. The problem is that the operation people know nothing about the application, what it does and how it should behave. Now we expect them to install it and also maintain the applications that they are not familiar with.
In Dev Centric: The developer should be responsible for their own applications in production. The developer should install and maintain the applications with the support of the DevOps engineers. DevOps engineers should create the tools to help the developer install and maintain their applications on production. While developers should have ssh access to production, ops should create the tools so that developers will not have to ssh to production. Make the production environment visible in a developer friendly way. Via monitoring system, easy access to logs (log aggregation). Developers should see how their system behaves in production at all times. This creates a better quality product because now the developers are also responsible to their work on production. Every time there is a problem on production the developer who is the owner of the rough process should be alerted and handle it on production with the help of the operation engineers.
Wall #5 Developer || BI
The Problem: We see BI as a business concern and generate reports to marketing and management while the developers are not exposed to most of them. This creates a lack of knowledge by the developers of what is going on with his system and how it is used by the customers. If the developer does not know how his system is used he will not be able to make improvements on the parts that really matter.
In Dev Centric: Expose the BI reports to the developers. Make the BI tools also accessible to developers so they can add also technical data and take advantage of the BI analytic tools to understand better how the system is used. By doing that the developer can have a sense of priorities and the knowledge of how the system is used in real life. Once developers understand that, they can make improvements and optimizations to the most heavily used parts of the system and by doing that help customer satisfaction.
Maintaining a Dev Centric culture is not easy. You give a lot of responsibility in the hands of the developers. You actually give all the responsibility to the developers. But if you think about it this is exactly what you should do. The developers are your company assembly line, they are the ones who actually create the product. They have a direct impact on the company's success and bottom line. By understanding that and entrusting the developers with the power to change and take critical decisions it is a win-win situation.
Remember, developers CAN DO the work of a product manager, architect, QA and Ops, in fact this is exactly how a small startup succeeds, because developers actually do all of it.
What do you think is it time to go back to the future by breaking all the walls?