Investing in Engineering Culture Attracts Engineers

Filed under: — Aviran Mordo

I have been a software engineer for most of my life but until I joined Wix I had not worked in a place that put engineering as a core value. In Wix I had the opportunity to change that and one of my main goals was to create a great engineering team and a company with great engineering culture.
One of the reasons I took upon myself this task was selfish, I wanted to work at a place that I will be happy to come to work every day, to be challenged and a place that software engineers can be proud of their profession.

Another task that I had ever since I joined Wix, was to grow the engineering teams and bring on more developers. This was in perfect alignment to my own agenda since I believe that in order to attract the best software engineers we had to create a great engineering culture, promote it, share our knowledge with the community and by doing so to be more attractive and competitive to attract talented engineers.

Operating in a highly competitive area with many startups and deep pockets companies like Google, Facebook and Microsoft the options for good software engineers are endless. The best way to attract talent is not by offering extra high salary since we would always lose to larger and more established companies like Google or Microsoft which can always outbid anything we can offer. Having a great engineering culture, giving back to the community and taking care of our employees is a great way to compete and attract professionals.

In any case this has been my theory and now for the second year I can say it is working pretty good for us.


Looking at Google trends I have compared search keywords for people who are looking for a job at the leading companies here in Israel. Comparing “wix jobs” to “google jobs” and “facebook jobs” (and others) resulted in Wix being the #1 in this search term. Of course this is not a scientific data but the assumption is that software engineers are more likely to search these terms in English, while other non technical employees will probably search in Hebrew.

Investing in people, engineering and culture pays back !


Sustainable Software Deliverability with Timelines

Filed under: — Aviran Mordo

In my previous post “Kill the Deadlines” I rant about how (fake) deadlines are demotivating, reduce quality and burn development teams. But if you think about why people use deadlines, is to deliver a software project in a timely manner. While I’m not a proponent of using fake deadlines as a way to push developers, in a product development process it is extremely important to have a time frame where you want to launch the product.

Agile software development has good tools and methodologies that can help you ship products fast, as long as you do it right and not follow the process blindly, but you understand the tools in your arsenal and pick the ones that work for you and adjust them to your needs and culture.

While it is extremely hard (if not impossible) to accurately predict how long a software project will last, setting a timeframe for a product launch is necessary to keep the team in focus and decide on priorities.

Release content
A project consists of many features: some take a long time to implement, and some take a short time to implement. While a product manager envisions a whole set of features, implementing all of them before the release will take a very long time and will probably never happen, or the product will never launch as there are always more features you would like to add, or more polishing to the product you would like to do.

Set a timeline
In order to keep everyone focused and be able to launch a product in a timely manner we need to set a timeline when we want to launch the product. The timeline consists of estimations of how long each feature should take. Based on these estimations we can build a release content list with all the features we would like to launch in this version. When we set a timeline we force ourselves to think about what are the features we would like to get into this version. The list of features in a version should be determined by the business value they provide. For instance, we can have one big feature with a lot of value or maybe two small features what each one has less value, but releasing two of these at the same time have a bigger value to the product that the one big feature.

Timeline should have checkpoints or milestones where you can evaluate your progress. In the axis of Time/Quality/Content I tend to keep quality as the constant (you should always strive to produce high quality software) and now you will have to play with time and content. Given a timeline, these milestones are good points to understand if you are going to be able to ship with your initial list of features in time, or you would have to take a good look and see what you can leave out for the next version. You will be surprised how “critical” features can be “less critical” and be cut down when you are forced to deal with a decision of extending the timeline or cut features, which helps you ship the product faster.

Synchronizing the company
Timelines help you synchronizing different departments inside the company. Since releasing a product requires also involvements from other department in the company, such as marketing, support, content writing, QA, etc’, having a public timeline that is visible and communicated to the whole company, helps synchronizing all the departments and help them plan accordingly to be ready for the release. You can communicate the timelines via periodic email updates, public boards, posters on the walls, publicly displayed on monitors in the office or any other method that will keep the timeline transparent and accessible to all.

So what are timelines?
Timeline is a rough time frame that you would like to ship a product. Timeline is NOT a deadline and is flexible to changes and delays (unfortunately most of the times it will be postponed, but you should try to keep it to a minimum).
Depending on the amount of the work timelines should be in a resolution of a week, month or a quarter.

Sometimes due to business constraints the timeline becomes a hard deadline. This is not an arbitrary date that you have set, but it is a direct result of a business constraints. For instance, a special feature that needs to be released before a holiday or a regulatory hard limit on something you need to change or add to your system. In this instance the real reason needs to be clearly communicated to the team.

When a timeline is delayed it should be as part of a discussion to understand what is the business impact of the delay as you may come to a conclusion that maybe instead of delaying the timeline you would make the hard choice of reducing the scope (content) of this version and keep the time unchanged.

So if a timeline is a rough estimate, when is the release date?
The release date, AKA deadline should be set towards the end of the timeline at the point you understand all the unknowns and you feel comfortable that you will be ready to launch (and so all the other dependencies). Setting the deadline late in the process will make the deadline based on REALITY and not a fake one. Yes, you will probably need to push the teams to get to this deadline but it will be something that they will be able to relate, understand and not sacrifice quality and release a bad product.

Continuous improvement
Agile software development is talking about retrospective as a tool to improve development process, in many organizations the retrospective is being done after a sprint and unfortunately geared towards improving the estimations which is causing the side effect of developers taking buffers just to be on the safe side. This should NOT be the point of retrospectives.
Retrospectives should be treated as a tool for continuous improvement of development velocity and not as a tool to improve estimations. In order to improve velocity, you should have a continuous feedback about what would have help you to deliver software faster.
Here are some examples of issues developers should point that would have made them finish their tasks faster: less context switches, faster build time, too many production alerts, I was waiting for QA, provisioning staging machines is taking a lots of time, I had too many meetings, etc’.
As opposed to I estimated a task to take 3 days and I realized that the initial database schema I chose was wrong and I had to redo it which caused delays, or I thought this was an easy task and realized I had to first do a big refactoring before I could implement it.
Once you get these feedbacks you will start to identify patterns and identify what are the bottlenecks in your development process and you could then tackle these bottlenecks and improve your development velocity. (Just between us, as a side effect of this process you will also get better estimations but that is accidental and not the real purpose of this)

In order to release a product efficiently, you can use agile software delivery practices, set a rough timeline and checkpoints along the way to see if you are on schedule. In case you are late you should re-evaluate the version content you may want to cut or switch some features.
Communicate and make the timeline visible to all the people in the company so everyone can be in sync, and when you feel that you can have a release date that you can actually make and it is based on real progress, only then set the actual date, and lastly have a process in place for continuous improvement of your development velocity.


Scaling Engineering by Hacking Conway’s Law

Filed under: — Aviran Mordo

So Wix.com has a very unique company structure. While we are over 1200 people strong, we keep behaving like 400 people company. One of our goals is to have and keep the startup feeling and velocity. In order to do that Wix has evolved the company structure form functional teams, to Gangs and Guilds and now the latest incarnation of the company’s structure to Guilds and Companies.

I recently gave a talk at DevoxxUK about the evolution of the company structure and how we managed to scale our engineering teams in a fast growing company while still keeping a top of the line engineering group without losing quality on the way.

Watch the video from the conference (Download slides)


Why I don’t like to hire team leads

Filed under: — Aviran Mordo

Every company has its own culture that it wants to preserve. As a company grows it becomes harder to preserve its culture as you as a manager need (and should) give up control to team leads and to people you manage.

Good company has a set of values, best practices and culture. When you hire a new person to the company it takes a while until he/she learns and assimilates himself with the company’s culture.

When you put someone in a managerial position where they need to lead other people, there is a mini culture that is being created for each team. If you did your job right this mini culture is more or less aligned with the overall culture of the company.

For a fast growing company building new teams is necessary. The team lead plays a major role in building the team and setting its mini-culture. Hiring a person for a team-lead position is a huge gamble, since he comes with a different set of values, methodologies and culture from the previous jobs they had.

Also, the fact that a person was a team-lead on a different company does not necessary means he has what it takes to be a team lead at another company with a different culture and methodologies. This is especially true for people who want to change positions and did not previously led a team, but are looking to be one. Since you don’t know them you don’t know if they are the kind of people you want to lead your teams.

So what to do in order to make your new team lead successful at his job? The thing I like to do with people who may be hired as team leads is to tell them straight that we don’t promise a team lead position, but they will need to start working as an engineer with a team lead potential, learn the culture, methodologies and best practices for about 6 months where we will have the chance to evaluate if that person is good enough to be a team lead. After a period of time when the need to a new team lead arises we will consider them to be one in case they fit the position.

I also like this method because it promotes people from within the organization and allows people to grow inside the company and not having to leave the company in order to get a promotion. Also by the time a person becomes a team lead he already had a chance to gain the respect of his peers and be more accepted as a team lead.

What do you think?


Building a Guild

Filed under: — Aviran Mordo

A lot of people heard about Spotify company structure of Guilds and tribes. We at Wix.com have a similar structure that has evolved over time and influenced by their ideas, however we have our own interpretation of the structure and the role of the Guild.

In this article I will try to walk down memory lane and describe my experience in building the first Guild (back-end JVM Guild) in Wix, which is now the role model of all the other Guilds at the company, and how it has evolved from the time I joined Wix when we had one back-end team of 4 developers to a about 100 back-end engineers in the back-end guild.

The Guild model did not start right away, when you are a relatively small startup all you have is teams, and this is exactly what we had. We had one server team (4 engineers) that was basically responsible for all the back-end development at Wix. As there was a demand for more back-end engineers the team grew very slowly. As with a small startup the recruitment process was very picky and we were only looking for the best engineers. At the course of a year I have only recruited 4 senior engineers. While this is very slow at this stage of the company it was very important to pick only the best engineers you can find, as these are the core engineering team that will help to build and shape the Guild and the engineering culture at the company in the future.

At this point where we had around 10 engineers we were pretty much functional teams, where everybody knew almost everything and I could move people from project to project according to the company’s priorities.

As we continue to grow (doubling the number of people every year) we saw that we are very good in focusing our efforts in some areas where are that point the company decided to invest, but were neglecting other existing products that had to compete on shared engineering resources but without any priority.

At this point we realized that we need dedicated engineers for each product group (at least for the big ones). We still didn’t have a name for that but I had essentially assigned some developers to be dedicated on some products while the other still remained shared resources.

As Wix continued its growth we had different groups of people who worked on different projects and were less engage with each other. So what we started to do is to formalize our engineering culture. While we always had a strong ownership and DevOps culture we started more and more being involved in knowledge sharing activities in order to keep our engineering teams on the cutting edge and learn from each team’s experience.

At this point we started to have discussions about how to structure the company. We looked around and found the Spotify paper. We realized that while we don’t have a name for our current structure it resembled to what Spotify had. So we adopted some of the naming and agreed that we should be working at a form of Guilds that are defined by a profession; and Gangs, which are the product teams.
Initially we only had the engineering Gangs who were dedicated to a product with all the other as shared resources across products.

This was the point where the role of the Guild had started to form.

The Guild is the one who is responsible for the person’s profession thus the Guild has the following responsibilities:

Recruitment (Hiring and firing)
Assignment to product teams according to the company’s priorities.
Setting the professional guidelines.
Set the engineers compensation (salary, bonuses etc’).
Create an engineering brand for the company.
Be responsible for the professional development / career of the engineers.

As Wix continued to grow we started to have more and more projects and product teams. What we realized then is that while having dedicated engineering teams (Gangs) is not enough because there was a bottleneck on the other shared resources. Also we had multiple products that had a common domain. What we wanted to do is to give as much independence to each product domain / vertical.

So once more we had to evolve and created what we call now a “Company”. A company is like a startup within Wix, it has all the resources it needs (developers, product manages, analysts, UX engineers, marketing etc’) in order to progress fast and create the best product they can do regardless if, the other products at Wix.

At this point the Guild also had to take on more responsibilities. While we want the “Companies” to progress as fast as they can, the “Companies” also has to keep alignment with Wix as a whole. Another issue is that we expect these “Companies” to create products that compete in the free market with other startups and big companies, but with limited resources.

The Guild now needs to play a big role in enabling the success of the “Companies” within Wix. If each “Company” had to develop everything on their own, for instance the frameworks, deployment, taking care of the infrastructure, monitoring etc’ they would not stand a chance to compete with whole companies that are doing the same product with more resources. So the Guild now took another responsibility in taking care of all the infrastructure, deployment pipeline, and core services that all the “Companies” share. For instance is we see a service that is needed in more than two companies (for example mailing service), we develop it in the Guild (which has its own core services teams) and all the other “Companies” can use this service, thus focusing only on the product itself and not having to worry about the infrastructure.

In order to keep alignment with the other “Companies”, and make it easier for engineers to move between “Companies”, share knowledge and best practices, all the “Companies” share the same infrastructure and methodologies. This is a tradeoff between freedom and velocity. You loose some freedoms but gain a lot of velocity as many of the things you need for your service are already there for you.

Now a “Company” may decide (in coordination with the Guilds) that using the existing infrastructure is the wrong solution for the product they own, and they want to develop on a different stack. They can do that, however they will need to take full responsibility over the whole application lifecycle, deployment, monitoring and integration with the other Wix echo-system. This is a lot of work and usually time to market will be very long, having to develop all the infrastructure on their own, so almost every “Company” opt-in to use the current infrastructure, although we have several cases where it was the right decision to develop some products on a different stack .

So if I was to describe the line of responsibility between a “Company” and a Guild is that the “Company” decides what to do, and the Guild say how to do it.

So now that we have “Companies” and Guilds, the Guild needs to assume more responsibilities in addition to the above:

Align between “Companies”. The Guilds are horizontal while “Companies” are verticals.
Support the engineers working in the “Companies”
Review and guidance
Develop shared infrastructure
Improving development velocity
Temporary help “Companies” in need with additional resources from the Guild.

Guild masters:
Guild masters are senior engineers that part of their responsibilities is to support engineers in different “Companies”. Guild masters conduct reviews, training, mentoring and also since they are horizontal and working with many companies they identify common issues, duplication of code between companies, understand the development bottlenecks and are trying to solve them. Also because of that they also pollinate “Companies” by bringing best practices and lessons learned from other “Companies”

Guild activities:
In order for the Guild to be able to take on these responsibilities it needs developer’s time so at Wix 20% of the engineering time is dedicated to the Guild activities.

Every Thursday we have a Guild day in which the Guild is conducting training activities and Guild tasks. All the engineers from all the “Companies” are assembled at one place for the Guild day.

Here is the back-end guild day schedule:
10:00-11:00 – Guild retrospective in which we discuss engineering dilemmas and lesson learned from across “Companies”.
11:00-11:15 – Break
11:15-11:30 – Project spotlight – where someone is presenting a new project that is being worked on, some lesson learned and challenges they have faced
11:30-13:00 (usually not the whole 1.5 hours is needed) – Tech talk, which if it does not contain any sensitive information is also open to the public at a meetup.
13:00–EOD – Lunch and Guild tasks. (The guild tasks are called “Games of Gangs”, but on “Games of Gangs” we’ll discuss on another post).


Kill The Deadlines

Filed under: — Aviran Mordo

I have been building software for over 20 years and participated in many projects. Usually when you come to write a new feature or starting a new project one of the first thing your manager asks you is a time estimate (if you are lucky) and then will set a deadline for the project completion.

Once a deadline is set everybody is start working to meet this date. While setting a deadline helps management plan and have visibility about what is coming from the development pipeline, the act of setting a deadline, especially an aggressive one is a destructive act that in most cases hearts the company in the long run.

Development work consists of much more than writing and testing code, it also consists of research and design. The problem is that while (experienced) developer can estimate the coding effort there is no real way to estimate the research phase, problems you may encounter and how long the design is going to take (if you want to make a good design). How can you really estimate how long it will take you to learn something that you don’t know?

If deadlines are aggressive, meeting them usually means that developers will start cutting corners. Do a bad design just because you don’t have the time to the right one. If developers are pressed in time they may stick to the bad design choice just because they don’t have time to switch the design to a better one after they realize their initial design has flaws.

Other things developers do in order to meet the deadline is to cut down on testing, while doing that hurts the quality of their work. While cutting down on automated testing may let you believe the work is progressing at a higher rate, however you will usually find the problems in production and spend much more time stabilizing the system after it is deployed. You might think you met the deadline shipping the product, but the quality of the product is low and you are going to pay for that in maintenance and reputation.

In addition to all that working to meet deadlines create stress on the developers which is not a healthy environment to be in for a long time, if they move from one deadline to another.

Now don’t get me wrong, by not setting a deadline you are not giving a free hand to project to stretch beyond reason. Developers should be aware that their performance is measured but they should also know that they can take the time to produce a high quality product by not cutting corners. In most cases a project can be delayed by few days or weeks without any real consequences to the company, and developers should know that if they have the time they need to produce a good product.

In the exception where you do have a deadline which you cannot postpone the delivery, you should take into consideration that there will quality issues and design flaws. After the delivery you should give developers time to complete the missing tests, do necessary refactoring and bring the product to the desired quality.

See Next post: Sustainable Software Deliverability with Timelines


Dev Centric - Trust And Collaboration

Filed under: — Aviran Mordo

After my first post about Dev Centric culture I got many questions on the topic which I will try to explain in the next few posts.
At first glance Dev Centric sounds like the only one that matters is the developer and everyone else are just simple helpers. This cannot be farther from the truth.

To understand Dev Centric let’s take a step back and describe how a company grows but before that we need to understand how a software company ship a product. Like every manufacturing plant a software company also have a pipe line that a product need to go through in order for it to be shipped to the customers.

Here is a standard software pipe line.

Product definition -> Design -> Develop -> Build -> QA -> Deploy

When a company is small you have only a handful of developers which are the pipeline, and do all the work. They design, code, deploy, maintain, do QA and also define the product. However while good developer can do all these tasks it comes with a price, they do not focus on what they are best of, which is writing code.

So in order to make the product better a company hires a product manager, which is probably going to do a better job at designing a product than the developer. Product manager has better skills and specializes in product definition phase of the pipe line. Now while the company is small PM work closely with the developers with a lot of collaboration.

Same goes with QA, operations and architects which each one can probably do a better job than the developer on their own field of experties. While the company is small they all work together. However when a company grows then walls starting to show as every group want to control their aspect of the pipe line, which in turns causes mistrust, and slows down the whole “production line” as you define more structured process and work flows.

Dev Centric culture tries to make the production line as fast as possible. Now if we look at the pipeline the one person who we cannot be without is the developer. The developer IS the production line, he is the one that manufacture the product. However since we also want the best quality product we cannot give up PM, QA and Ops, they are very important part of the manufacturing floor.

So since the developer is the one that ships the product we want to make the process as fast and as efficient as we can without losing quality. In order for the developer to create the best product he needs to understand it.
The best way to understand a product is to help define it. This is where Dev Centric comes into the picture. Developers should work together with PM and help define the product. Not buy getting a thousand pages spec but to sit in the same room with the PM and discuss the product, define it while writing the code. This way the code that the developer writes is the right one and there are no misunderstandings between what the PM intent and what the developer understood.

Same goes with QA. Developers should work closely with QA so they understand each other. QA understands the product the same way the developers do. The developers should continuously release code to QA during the development process to shorten the cycles. The best way is to work Test Driven Development (TDD) where developers are also writing the automated tests for their code and QA backs up the developers with more comprehensive end to end tests which also serve as acceptance tests. Another important role for QA is to review the developer’s test cases and point out if there are un-tested use cases.
So the same goes with Ops and architect like mentioned in my previous post.

Dev centric basically clears the path for the developer to deliver faster and better product by breaking the walls in the manufacturing floor having everyone working in collaboration and focusing on what is important (delivering the best product) by creating trust between people with different agendas and by doing that create a productive environment.


Building Engineering Culture Based On Quality To Drive Innovation

Filed under: — Aviran Mordo

When I joined Wix in 2010 my job was to rebuild the back-end engineering team. The previous team that took the company to where it was back then was scattered to other projects and except for one developer from the original team I had a completely new team.

A couple of months after I arrived and started to figure things out with my new team we decided to move to continuous delivery methodology. Since we faced many challenges in both moving to continuous delivery and the need to re-write the whole back-end system, we needed very good software engineers to build a new framework and to be the first ones to lead the company’s Dev-Centric culture.

We wanted to create a culture based on quality in terms of software engineering and people responsibilities. Since every person in a growing company has a profound effect on the company’s culture, it sets the tone for the recruitment process. Ever since I got to Wix I have never stopped recruiting engineers, however recruiting is a big challenge. I was looking for exceptional software engineers. The standards for passing the interview process is very high and very few actually succeeded, but that is a price I’m willing to pay in order to build an ‘A team’.



Continuous Delivery - Part 7 - Cultural Change

Filed under: — Aviran Mordo

Previous chapter: Backward and forward compatibility

In order for continuous delivery to work the organization has to go a cultural change and switch to Dev-Centric Culture.

Continuous delivery gives a lot of responsibility in the hand of the developer and as such the developer need to have a sense of ownership. At Wix we say that it is the developer’s responsibility to bring a product / feature to production, and he is responsible from the feature inception to the actual delivery of the product and maintaining it on the production environment.

In order to do that several things have to happen.

Know the business :
The developer has to know the business and be connected to the product he is developing. By understanding the product the developers makes better decisions and better products. Developers are pretty smart people and they don’t need to have product specs in their face (nobody actually reads them). Our developers work together with the product mangers to determine what the product for the feature should be. Remember while the actual product may have several features bundled together for it to be valuable for the customer, we develop per feature and deploy it as soon as it is ready. This way both the product manager and the developer get to test and experience each feature on production (it is only exposed to them via Feature toggle) and determine if it is good enough, and may cause the direction of the product to change not according to plan, and actually develop a different next feature than the planned one.

Take ownership
Developers are taking ownership on the whole process, which means that they are the ones that need to eventually deploy to production. This statement actually changes the roles of several departments. What the company needs to do is to remove every obstacle in the developers way to deploy quickly on his own to production.

The operations department will no longer be doing the deployments. What they will do from now on is to create the automated tooling that will allow the developers to deploy on his own.
Operations together with dev will create the tooling to make the production environment visible and easy to understand to developers. Developers should not start a shell console (ssh) in order to view and know what is going on with the servers. We created web views for monitored metrics of both system and application metrics and exceptions.

Powered by WordPress