Previous Chapter: Startup - Self Test

One very important mind set developers will have to adopt and practice is backward and forward compatibility.

Most production system do not consist on just one server, but a cluster of servers. When deploying new piece of code, you do not deploy it to all the servers at once because part of Continuous deployment strategy is zero downtime during deployment. If you deploy to all the servers at once and the deployment require a server restart then all the servers will restart at the same time causing a downtime.

Now think of the following scenario. You write a new code that requires a new field in a DTO and it is written to a database. Now if you deploy your servers gradually you will have a period of time that some servers will have the new code that uses the new field and some will not. The servers that have the new code will send the new field in the DTO and the servers that not yet deployed will not have the new field and will not recognize it.

One more important concept is to avoid deployment dependencies where you have to deploy one set of services before you deploy the other set. If we’ll use our previous example this will even make things worse. Let’s say you work with SOA architecture and you have now clients that send the new field and some clients that do not. Or you deploy the clients that now send the new field but you have not yet deployed the servers that can read them and might break. You might say, well I will not do that and I will first deploy the server that can read the new field and only after that I’ll deploy the client that sends it. However in Continuous deployment as easily as you can deploy new code you can also rollback you code. So even if you deploy first the server and then the client you might now roll back the server without rolling back the client, thus creating again the situation where clients send unknown fields to the server.
(more…)

Previous Chapter: A/B Testing

So far we discussed Feature Toggle and A/B testing. These two methods enable safe guards that your code does not harm your system. Feature toggles enable to gradually use new features and gradually expose it to users, while monitoring that the system behaves as expected. A/B testing on the other hand let you test how your users react to new features. There is one more crucial test you as a developer should write that will protect your system from bad deployments (and also be a key to gradual deployment implementation).

Self-test sometimes called startup test or post deployment test is a test where your system checks that it can actually work properly. A working program does not only consist of the code that the developer write. In order for an application to work it needs configuration values, external resources and dependencies such as databases and external services it depends on to work properly.

When an application loads and starts up its first operation should be Self-test. Your application should refuse to process any operation if the self-test did not pass successfully.
(more…)

Previous chapter: Continuous Delivery - Part 3 - Feature Toggles

From Wikipedia: In web development and marketing, as well as in more traditional forms of advertising, A/B testing or split testing is an experimental approach to web design (especially user experience design), which aims to identify changes to web pages that increase or maximize an outcome of interest (e.g., click-through rate for a banner advertisement). As the name implies, two versions (A and B) are compared, which are identical except for one variation that might impact a user’s behavior. Version A might be the currently used version, while Version B is modified in some respect. For instance, on an e-commerce website the purchase funnel is typically a good candidate for A/B testing, as even marginal improvements in drop-off rates can represent a significant gain in sales. Significant improvements can be seen through testing elements like copy text, layouts, images and colors.

Although it sounds similar to feature toggles, there is a conceptual difference between A/B testing and feature toggles. With A/B test you measure an outcome for of a completed feature or flow, which hopefully does not have bugs. A/B testing is a mechanism to expose a finished feature to your users and test their reaction to it. While with feature toggle you would like to test that the code behaves properly, as expected and without bugs. In many cases feature toggles are used on the back-end where the users don’t not really experience changes in flow, while A/B tests are used on the front-end that exposes the new flow or UI to users.

Consistent user experience.
One important point to notice in A/B testing is consistent user experience. For instance you cannot display a new menu option one time, not show the same option a second time the user returns to your site or if the user refreshes the browser. So depending on the strategy you’re A/B test works to determine if a user is in group A or in group B , it should be consistent. If a user comes back to your application they should always “fall” to the same test group.
(more…)

Previous chapter:The Road To Continuous Delivery - Part 2 - Visibility

One of the key elements in Continuous Delivery is the fact that you stop working with feature branches in your VCS repository; everybody works on the MASTER branch. During our transition to Continuous Deployment we switched from SVN to Git, which handles code merges much better, and has some other advantages over SVN; however SVN and basically every other VCS will work just fine.

For people who are just getting to know this methodology it sounds a bit crazy because they think developers cannot check-in their code until it’s completed and all the tests pass. But this is definitely not the case. Working in Continuous Deployment we tell developers to check-in their code as often as possible, at least once a day. So how can this work? Developers cannot finish their task in one day? Well there are few strategies to support this mode of development.

Feature toggles
Telling your developers they must check-in their code at least once a day will get you the reaction of something like “But my code is not finished yet, I cannot check it in”. The way to overcome this “problem” is with feature toggles.

Feature Toggle is a technique in software development that attempts to provide an alternative to maintaining multiple source code branches, called feature branches.
Continuous release and continuous deployment enables you to have quick feedback about your coding. This requires you to integrate your changes as early as possible. Feature branches introduce a by-pass to this process. Feature toggles brings you back to the track, but the execution paths of your feature is still “dead” and “untested”, if a toggle is “off”. But the effort is low to enable the new execution paths just by setting a toggle to “on”.

So what is really a feature toggle?
Feature toggle is basically an “if” statement in your code that is part of your standard code flow. If the toggle is “on” (the “if” statement == true) then the code is executed, and if the toggle is off then the code is not executed.
Every new feature you add to your system has to be wrapped in a feature toggle. This way developers can check-in unfinished code, as long as it compiles, that will never get executed until you change the toggle to “on”. If you design your code correctly you will see that in most cases you will only have ONE spot in your code for a specific feature toggle “if” statement.
(more…)

Previous chapter: The road to continuous delivery - Part 1

Production visibility

A key point for a successful continuous delivery is to make the production matrix available to the developers. At the heart of continuous delivery methodology is to empower the developer and make the developers responsible for deployment and successful operations of the production environment. In order for the developers to do that you should make all the information about the applications running in production easily available.

Although we give our developers root (sudo) access for the production servers, we do not want our developers to look at the logs in order to understand how the application behaves in production and to solve problems when they occur. Instead we developed a framework that every application at Wix is built on, which takes care of this concern.

Every application built with our framework automatically exposes a web dashboard that shows the application state and statistics. The dashboard shows the following (partial list):
• Server configuration
• All the RPC endpoints
• Resource Pools statistics
• Self test status (will be explained in future post)
• The list of artifacts (dependencies) and their version deployed with this version
• Feature toggles and their values
• Recent log entries (can be filtered by severity)
• A/B tests
• And most importantly we collect statistics about methods (timings, exceptions, number of calls and historical graphs).

We use code instrumentation to automatically expose statistics on every controller and service end-point. Also developers can annotate methods they feel is important to monitor. For every method we can see the historical performance data, exception counters and also the last 10 exceptions for each method.

We have 2 categories for exceptions: Business exceptions and System exceptions.
Business exception is everything that has to do with application business logic. You will always have these kinds of exceptions like validation exceptions. The important thing to monitor on this kind of exception is to watch for sudden increase of these exceptions, especially after deployment.

The other type of exception is System exception. System exception is something like: “Cannot get JDBC connection”, or “HTTP connection timeout”. A perfect system should have zero System exceptions.
(more…)

The following series of posts are coming from my experience as the back-end development manager at Wix.com. I will try to tell the story of Wix and how we see and practice continuous delivery, hoping it will help you make the switch too.

So you decided that your development process is too slow and thinking to go to continuous delivery methodology instead of the “not so agile” Scrum. I assume you did some research, talked to a couple of companies and attended some lectures about the subject and want to practice continuous deployment too, but many companies asking me how to start and what to do?
In this series of articles I will try to describe some strategies to make the switch to Continuous delivery (CD).

Continuous Delivery is the last step in a long process. If you are just starting you should not expect that you can do this within a few weeks or even within few months. It might take you almost a year to actually make several deployments a day.
One important thing to know, it takes full commitment from the management. Real CD is going to change the whole development methodologies and affect everyone in the R&D.

Phase 1 – Test Driven Development
In order to do a successful CD you need to change the development methodology to be Test Driven Development. There are many books and online resources about how to do TDD. I will not write about it here but I will share our experience and the things we did in order to do TDD. One of the best books I recommend is “Growing Object Oriented Guided by tests”

A key concept of CD is that everything should be tested automatically. Like most companies we had a manual QA department which was one of the reasons the release process is taking so long. With every new version of the product regression tests takes longer.

Usually when you’ll suggest moving to TDD and eventually to CI/CD the QA department will start having concerns that they are going to be expandable and be fired, but we did not do such thing. What we did is that we sent our entire QA department to learn Java. Up to that point our QA personnel were not developers and did not know how to write code. Our initial thought was that the QA department is going to write tests, but not Unit tests, they are going to write Integration and End to End Tests.

Since we had a lot of legacy code that was not tested at all, the best way to test it, is by integration tests because IT is similar to what manual QA is doing, testing the system from outside. We needed the man power to help the developers so training the QA personal was a good choice.

Now as for the development department, we started to teach the developers how to write tests. Of course the first tests we wrote were pretty bad ones but as time passes, like any skill, knowing how to write good test is also a skill, so it improves in time.
In order to succeed in moving to CD it is critical to get support from the management because before you see results there is a lot of investments to be done and development velocity is going to sink even further as you start training and building the infrastructure to support CD.
(more…)

Whether you work in a high-tech company, law enforcement, health care or any other job that require you to be on call or available for emergencies you probably familiar with the dilemma of going to sleep with your smartphone turned on next to your bed.

In the middle of the night you get email alerts, annoying push notifications from games and apps installed on your phone or irritating text messages that are most likely not important enough for you to wake up at 2:00am, and can wait until the morning for you to read.

On the other hand you cannot turn off your phone because you do want to be available for emergency calls from your job (or your kids) and would like the phone to ring in order to wake you up.

I use two apps that in one hand keep my phone silent at times I want and on the other hand if someone important calls the phone “wakes up” and in turn wake me up from my beauty sleep.

The first one is called Silent Time. Silent Time automatically silences your phone based on your weekly schedule. You can also add exceptions to let important callers through and use the Quick Quiet widget to silent it for whatever else might come up like meetings and movies.

The second app is “My VIP Calls“.
“My VIP Calls” let you decide who can reach you even when you put your phone to silent/vibrate mode.
You can mark any number in your contact list as VIP (add to white list).
You can also check “Unknown” option to let unknown numbers to contact you.
Useful if you are expecting important call from unknown number.
Works with SMS messages too!
You can set sound volume and app will use this settings for ringing.

There are other alternatives apps that do similar things, but I use these two and very happy with them.

On Saturday, at midnight Greenwich Mean Time, as June turned into July, the Earth’s official time keepers held their clocks back by a single second in order to keep them in sync with the planet’s daily rotation, and due to many production problems we were faceing here at Wix.com with our Java systems, and also according to reports from across the web, some of the net’s fundamental software platforms — including the Linux operating system and the Java application platform — were unable to cope with the extra second.

Many computing systems use what’s called the Network Time Protocol, or NTP, to keep themselves in sync with the world’s atomic clocks, and when an extra second is added, some just don’t know how to handle it.

We had problems with Hadoop, Tomcat, Jetty, Flum, Tyrent, Tokyo and other systems. One major symptoms we saw was system crushing and CPU overload.

Fortunately we found a solution which is simply to run the following linux command:

date; sudo date `date +"%m%d%H%M%C%y.%S"`; date;

This should solve your problem and you don’t have to restart your java applications

At the JavaOne convention Oracle announced a new NoSQL database system.

The NoSQL database is a key-value store. Later today Oracle will formally announce this new product

Adobe Systems Incorporated today announced Adobe Creative Suite 5.5 Production Premium, the complete software solution for video and post-production that helps deliver content to virtually any screen.

Better performance, workflow improvements, creative innovations, and powerful new audio editing capabilities build upon the huge customer momentum Production Premium is experiencing with broadcasters, filmmakers and video professionals worldwide.

New versions receiving major updates include Adobe Premiere® Pro CS5.5, Adobe After Effects® CS5.5, Adobe Flash® Professional CS5.5, Adobe Flash Catalyst® CS5.5, Adobe Story, Adobe Media Encoder CS5.5, and Adobe Device Central CS5.5. New to Creative Suite 5.5 is Adobe Audition® CS5.5, bringing its audio-for-video multitrack editing environment to both Mac OS and Windows® for the first time.

Microsoft has released its Internet Explorer 9 (IE9) web browser, saying that it “unleashes a more beautiful web”.

Microsoft has been publicly testing IE9 beta releases for some time and revealed that over 40 million people had downloaded the betas. The Vole has been aggressively promoting a number of IE9 features, with the highlight being HTML5 hardware acceleration.

This has led Microsoft to come out with unsubstantiated hyperbole such as “IE9 harnesses the power of the graphics processing unit (GPU), unlocking 90 per cent of the PC’s power that went previously untapped by web browsers.” Admittedly GPU acceleration for both HTML5 and Adobe’s Flash Player will help performance, but it should also improve battery life on laptops.

Google has released version 10 of its browser. The update brings hundreds of bug fixes as well as many features that have been available on the Chrome beta and dev channels to users interested in using Chrome’s latest builds. Chrome 10 also addresses 23 security vulnerabilities in the WebKit-based browser (easily more than Google has ever fixed before): 15 rated as High, three rated as Medium, and five rated as Low.

The new Chrome version is significantly faster in some respects (a 66 percent improvement in JavaScript performance on Google’s own V8 benchmark suite) and adds features such a new settings interface with a search box that shows you the settings you’re looking for as you type, the ability to synchronize passwords across your computers and encrypt them with your own secret passphrase for extra security, as well as the options to sync bookmarks, extensions, preferences, themes, and more.