Safe Database Migration Pattern Without Downtime

Filed under: — Aviran Mordo

I’ve been doing a continuous delivery talk for a while now and during my talk I describe a pattern of how to safely migrating one database to another database without downtime. Since many people contacted me and asked for more details about it, I will describe it here in more details as promised.

You can use this pattern to migrate between two different databases, for instance between MySql and MongoDB or between two schemas in the same database.

The idea of this pattern is to do a lazy database migration using feature toggles to control the behaviour of your application and progressing through the phases of the migration.

Let’s assume two databases you want to migrate from the “old” database to the “new” database.

Step 1
Build and deploy the “new” database schema onto production.
In this phase your system stays the same, nothing changes other than the fact that you have deployed a new database which you can start using when ready.

Step 2
Add a new DAO to your app that writes to the “new” database.
You may need to refactor your application to have a single (or very few) point(s) in which you access the database.
At the points you access the database or DAO you add a multi-state feature toggle that will control the flow of writing to the database.
The first state of this feature toggle is “use old database”. In this state your code ignores the “new” database and simply uses the “old” one as always.

Step 3
Start writing to the “new” database but use the “old” one as primary.
We are now getting into the distributed transaction world because you can never be 100% sure that writing to 2 databases can succeed of fail at the same time.
When your code performs a write operation it first writes to the “old” database and if it succeeds it writes to the “new” database as well. Notice that in this step the “old” database is in a consistent state while the “new” database can potentially be inconsistent since the writes to it can fail while the “old” database write succeeded.

It is important to let this step run for a while (several days or even weeks) before moving to the next step. This will give you the confidence that the write path of your new code works as expected and that the “new” database is configured correctly with all the replications in place.

At any time you decide that something is not working you can simply change the feature toggle back to the previous state and stop writing to the “new” database. You can make modification to the new schema or even drop it if you need as all the data is still in the “old” database and in a consistent state.

Safe database migration pattern

Step 4
Enable the read path. Change the feature toggle to enable reading from both databases.
In this step the it is important to remember that “old” database is the consistent one and should still be treated as the authoritative data.

Since there are many read patterns I’ll describe just a couple here but you can adjust it to your own use case.

In case you have immutable data and you know the record id you first read from the “new” database and in case you did not find the record you need to fall back to the “old” database and look for the record there. Only if both databases don’t have the record you return a “not found” to the client. Otherwise if the record is found you return the result preferring the “new” database.

If your data is mutable you’ll need to perform the read operation from both databases and prefer the “new” one only if the timestamp is equal to the record in the “old” database. Remember in this phase only the “old” database is considered consistent.

If you don’t know the record id and need to fetch unknown number of records you basically need to query both databases and merge the results coming from both DBs.

Whatever your read pattern is, remember that in this case the consistent database is the “old” one, but in this phase you need to read and use the “new” database read path as much as you can, in order to test your application and your new DAO on a real production environment. In this phase you may find out that you are missing some indices or need more read replicas.

Let this phase run for a while before moving to the next phase. Like in the previous phase you can always turn the feature toggle back to the previous states without a fear of data loss.

Another thing to note that since you are reading data from two schemas you will probably need to maintain backward and forward compatibility for the two data sets.

Step 5
Making the “new” database the primary one. Change the feature toggle to first write to the new database (you still read from both but now prefer the new DB).
This is a very important step. In this step you already run the write and read path of your code for a while now and when you feel comfortable you now switch roles and making the “new” database the consistent one and the “old” as a not consistent.
Instead of first writing to the “old” database first you now write to the “new” database first and do a “best effort” writing to the old database.
This phase also requires you to change the read priority. Up until now we considered the “old” database as having the authoritative data but now you would prefer the data in the “new” database (of course you need to consider the record timestamp).

This is also the point where you should try as much as you can to avoid switching back the feature toggle to the previous state as you’ll need to run a manual migration script to compare the two databases as writes to the “old” one may not have succeeded (remember distributed transaction). I call this “the point of no return“.

Step 6
Stop writing to the “old” database (read from both).
Change the feature toggle again to now stop writing to the “old” database having only a write path with the “new” database. Since the “new” database still does not have all the records you will still need to read from the “old” database as well as from the new and merge the data coming from both.

This is an important step as it basically transforms the “old” database to a “read-only” database.

If you feel comfortable enough you can do step 5 and 6 in one go.

Step 7
Eagerly migrate data from the “old” database to the “new” one.
Now that the “old” database is in a “read-only” mode it is very easy to write a migration script to migrate all the records from the “old” database that are not present in the “new” database.

Step 8
Delete the “old” DAO.
This is the last step where all the data is migrated to the “new” database you can now safely remove the old DAO from your code and leave only the new DAO that uses the new database. You now of course stop reading from the “old” DB and remove the data merging code that handle merging data from both DAOs.

This is it you are done and safely migrated the data between two databases without downtime.


Side note:
At Wix we usually run steps 3 and 4 for at least 2 weeks each and sometimes even a month before moving on to the next step. Examples for issues we had encounter during these steps were:
On the write path we were holding large objects in memory which caused GC storms during peak traffic.
Replications were not configured/working properly.
Missing proper monitoring.

On the read path we had issues like missing index.
Inefficient data model that caused poor performance which let us to rethink our data model for better read performance.


Tip: How to remove / disable metro UI windows 8

Filed under: — Aviran Mordo

For those of you who installed Windows 8 and don’t like the new Metro UI there is a simple way to revert back to the old familiar look of Windows 7. Here is how you do it:

Open the Regedit by typing regedit or by pressing the Windows Key + R combination and type regedit and click OK.
Navigate to HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer
Find the key RPenabled and change the value from 1 to 0.


‘Leap Second’ Bug Wreaks Havoc Across Java Production Systems

Filed under: — Aviran Mordo

On Saturday, at midnight Greenwich Mean Time, as June turned into July, the Earth’s official time keepers held their clocks back by a single second in order to keep them in sync with the planet’s daily rotation, and due to many production problems we were faceing here at Wix.com with our Java systems, and also according to reports from across the web, some of the net’s fundamental software platforms — including the Linux operating system and the Java application platform — were unable to cope with the extra second.

Many computing systems use what’s called the Network Time Protocol, or NTP, to keep themselves in sync with the world’s atomic clocks, and when an extra second is added, some just don’t know how to handle it.

We had problems with Hadoop, Tomcat, Jetty, Flum, Tyrent, Tokyo and other systems. One major symptoms we saw was system crushing and CPU overload.

Fortunately we found a solution which is simply to run the following linux command:

date; sudo date `date +"%m%d%H%M%C%y.%S"`; date;

This should solve your problem and you don’t have to restart your java applications


Make Add-ons Work In Firefox 4

Filed under: — Aviran Mordo

Mozilla is soon to release a new version of the popular browser, Firefox 4, many add-ons will stop working. A big hurdle for people is switching to the new browser is that some plugins are not compatible with the new browser version. Luckily there are few workarounds to make old plugins work with Firefox 4

The first one is an extension called Add-on Compatibility Reporter After installing the Add-on Compatibility Reporter, your incompatible extensions will become enabled for you to test whether they still work with the version of Firefox or Thunderbird that you’re using. If you notice that one of your add-ons doesn’t seem to be working the same way it did in previous versions of the application, just open the Add-ons Manager and click Compatibility next to that add-on to send a report to Mozilla.

Another method is to simply add a new configuration to Firefox. Here is how its done:

In the address bar type: about:config
Right click on the setting page and select: New->Boolean
Type: extensions.checkCompatibility.4.0 and set the value to false


Graphic Designer’s Journey: Freelance to Freedom (Infographic)

Filed under: — Aviran Mordo

Website Builder

[Via: Free Website Builder]


Make Extensions Work With Firefox 3.6

Filed under: — Aviran

As Mozilla released a new version of the popular open source browser, Firefox 3.6 many add-ons will stop working. Previous workarounds to enable extensions may not work. However there are two new workarounds that will make your favorite plugins work with the new version of Firefox.

The first one is an extension called Add-on Compatibility Reporter After installing the Add-on Compatibility Reporter, your incompatible extensions will become enabled for you to test whether they still work with the version of Firefox or Thunderbird that you’re using. If you notice that one of your add-ons doesn’t seem to be working the same way it did in previous versions of the application, just open the Add-ons Manager and click Compatibility next to that add-on to send a report to Mozilla.

Another method is to simply add a new configuration to Firefox. Here is how its done:

In the address bar type: about:config
Right click on the setting page and select: New->Boolean
Type: extensions.checkCompatibility.3.6 and set the value to false


Understanding Windows 7’s ‘GodMode’

Filed under: — Aviran

Windows enthusiasts are excited over the discovery of a hidden “GodMode” feature that lets users access all of the operating system’s control panels from within a single folder.

Enabling God Mode is very simple all you need to do is to create a new folder and then rename the folder to the following:


Once that is done, the folder’s icon will change to resemble a control panel and will contain dozens of control options.


How to Make Extensions Work with the Latest Firefox

Filed under: — Aviran Mordo

When you upgrade you version of Firefox some extension might stop working, however most of the times they will work fine with the new version of Firefox. There is an easy way to make Firefox stop checking for compatibility.

1. Open Firefox and type about:config in the address bar. Then click the button promising to be careful.

2. Right-click anywhere on the screen, choose: New > Boolean and name this boolean:


Press OK. Then set it to false and press OK again.

3. Now right-click again anywhere choose New > Boolean and make the name of this one:


and set the value of that one to false.


Tip: Schedule Restart After Hours

Filed under: — Aviran Mordo

When you run a server or even you own desktop you probably do not want to restart your machine during work hours, this especially concern servers which should run 24/7. However sometimes when you install a new program or after you run Windows Update you need to restart the machine in order to complete the installation. Fortunately you can schedule the time your machine restart by using Windows own task scheduler.

Here is how you do it:

  • Go to Start->All Programs->Accessories->System Tools->Scheduled Tasks
  • Double-click Add Scheduled Task
  • Click Next and on the next screen click Browse
  • Under %SystemRoot%\System32 folder, select the program called Shutdown.exe
  • In the next screen choose the name and select One time only
  • Click Next and fill in the time you want this shutdown (and restart) task to run
  • Click Next and fill in your user name and password
  • Click Nextand CHECK the box named: “Open advanced properties for this task when I click Finish”
  • Click Finish - Now the advance screen will appear
  • This is very important step: In the Run box you need to add parameters to the command line in order for the system to restart. After shutdown.exe add the parameters -r -t 00 so you’ll end up with something like this in the Run box: C:\WINDOWS\System32\Shutdown.exe -r -t 00
  • Click OK and you are done. (You may need to enter your user name and password again)


Tip: Command Prompt Here In Vista And Win 7

Filed under: — Aviran Mordo

If you are a power user you probably know Windows XP power toy called “Open Command Prompt Here” which adds a context menu item to open a command prompt at the selected folder in the windows explorer.

In Windows Vista and also in Windows 7, you can simply hold the Shift key down while selecting the context menu (right click on a folder) to see this option. If the current working directory is a network location, it will automatically map a drive letter for you.


Change Screen Resolution For Ubuntu Running Under VirtualBox

Filed under: — Aviran Mordo

I’m experimenting with VirtualBox, and having Ubuntu 8.04 running on it I could not find an easy way to increase the screen resolution higher than 800X600.

Note: Although this guide was originally written for Ubuntu 8.04 it also works on Ubuntu 8.10, Ubuntu 9.04, Ubuntu 9.10

Luckily I found this great tip that shows how to configure the screen resolution in Ubunto 8.04 running as a virtual machine under VirtualBox.

Start up your Ubuntu 8.04 guest. In VirtualBox, click Devices->Install Guest Additions and your virtual Ubuntu should recognize a new CD. Open a terminal and run this command:
sudo /media/cdrom/VBoxLinuxAdditions.run

The installer should prompt you that it’s time to restart the virtual machine once it’s done. Just logging out and back in should work.

This is where things get sticky. For Ubuntu 8.04 guests, the VirtualBox video driver will not be enabled because of the minimalistic xorg.conf file.

Open your guest machine’s xorg.conf file:
gksu gedit /etc/X11/xorg.conf

Find the device section, which should look like this:

Section "Device"
  Identifier "Configured Video Device"

And replace it with this to enable the VirtualBox driver:

Section "Device"
   Identifier   "Configured Video Device"
   Driver    "vboxvideo"

Changing screen resolutions in Ubuntu 8.04 is done with RandR, which is not supported by VirtualBox. So you’ll also need to set a resolution manually. Find the screen section, which should look like this:

Section "Screen"
   Identifier    "Default Screen"
  Monitor    "Configured Monitor"
  Device    "Configured Video Device"

And replace it with this, but change 1024×768 to your preferred resolution:

Section "Screen"
   Identifier    "Default Screen"
   Device    "VirtualBox graphics card"
   Monitor    "Generic Monitor"
   DefaultDepth    24
   SubSection "Display"
     Depth    24
     Modes      "1024x768"

Save, exit the editor, and log out. You should now have a large virtual machine window, as well as mouse pointer integration.


How To Make Active Virus Shield Obtain Updates

Filed under: — Aviran Mordo

Active Virus Shield is an AOL rebranded Kaspersky anti virus. Recently AOL stopped offering the anti virus and starting September people who installed AVS can not obtain signature updates since AOL do not post them on their servers.

Since AVS is basically a Kaspersky anti virus you can make a simple hack that will make AVS to update the signature files from Kaspersky servers instead of the AOL server, thus keeping your anti virus program updated with the latest virus signatures.

All you need to do is update your HOSTS file. Follow these steps:

  • Open your file explorer and navigate to: C:\WINDOWS\system32\drivers\etc
  • In the etc folder you’ll find a file named hosts. Edit this file in Notepad
  • Add the following line to the bottom of the hosts file: download.newaol.com
  • Save the hosts file and close Notepad
  • Right click your Active Virus Shield tray icon and select Update in the menu

That is it. Now AVS will obtain the updates from Kaspersky server

Powered by WordPress