Component: Data migration - Preparation guide

Data migration can be a daunting task that requires careful planning and execution. This article outlines the necessary steps to ensure a successful data migration.

Step 1: Set up the production environment

A typical data migration environment for Casewhere requires the following servers:

• Source database: To restore the legacy database.
• Casewhere database: To host the Casewhere MongoDB databases.
• Casewhere web: To deploy the Casewhere instance. This server must be able to access the other servers.

The size of these servers depends on your database size and the available timing window. To ensure Casewhere has sufficient power to migrate data, you can consider temporarily scaling up the environment and downsizing it when the job is done. A typical setup for PROD should be:

• Source database: CPU 4 cores, RAM 16 Gbs
• Casewhere database: CPU 4-8 cores, RAM 16-32 Gbs
• Casewhere web: CPU 4-8 cores, RAM 16-32 Gbs

You also need to adjust the number of background workers in Casewhere.Worker.Api/web/config accordingly. For example:

• dataTrigger:Workers: 2
• eventTrigger:Workers: 2
• cwjobs:Workers: Start with 5 and then adjust later if needed.

Some other settings in Casewhere.Worker.Api/web/config should also be adjusted, turned off, or disabled in order to ensure Casewhere runs smoothly during the migration:

• events:EnableHotAndCold: false
  • Remember to transfer all events from Hot to Cold before turning it off.
• perfLog:enabled: false
• serilog:minimum-level: Debug

Checklist:

• [ ] All 3 servers are up and running.
• [ ] You reviewed and considered the need for a temporary scaleup.
• [ ] You adjusted the background workers in Casewhere.Worker.Api/web/config.
• [ ] You disabled the features not optimized for data migration, e.g., Hot & Cold.
• [ ] You documented all the changes so that you can later revert the system configuration to the original state

Step 2: Deploy the business solution and back up databases

How to deploy the solution is outside the scope of this document. When you finish the final smoke tests and everything is up and running, remember to back up the Casewhere databases and ensure the backup is stored as long as needed. This backup will be required when restoring the system to a working state if something is horribly wrong in one of the following steps.

Checklist:

• [ ] The solution is up and running.
• [ ] The system state (data & configuration) is in good condition for a backup.
• [ ] You took a backup and stored it in a safe place.

Step 3: Deploy data migration processes

At this step, you should already have the data migration processes configured and tested. The safest way to deploy the processes to PROD is to package them from one of your TEST environments. After deployed, you can double check:

• Create a temporary worker site and link all the data migration pages to it

  • Use the default theme to keep things simple.
  • Localization: Show seconds in DateTime formats.
  • Security: Ensure only relevant team members are allowed to access. Keep it simple because the site is temporary.

• Check if all the data migration processes and jobs are active and in the correct order.

• Check if the plugin can connect to the Source Database.

• Decide if you should enable the job setting "Stop if the test failed".

• Ensure the "Process watcher" scheduler is running. This is to ensure the data migration can auto-resume if the server goes down for some reason.

• If you need to modify the Source Database (adding indexes, creating views, etc.), make sure those modifications are done before running data migration.

You must also review and list the factors or elements that might interrupt your data migration process. Some of the items worth considering:

• Any workflow trigger that might trigger unwanted actions, e.g., sending emails, creating payment transactions, schedulers, etc.
• Put the worker sites under maintenance to prevent users from making unwanted changes during the migration.

Checklist:

• [ ] You created the temporary worker site for data migration.
• [ ] You deployed the data migration processes.
• [ ] You double-checked the data migration processes, jobs, and other settings.
• [ ] You verified the connection to the Source Database.
• [ ] You verified that the Source Database is updated with items needed for the data migration, such as indexes and views.
• [ ] You disabled all elements that might interrupt the data migration.

Step 4: Backup databases before running data migration

Backing up the databases at this step is critical to ensure that if something goes wrong when running migration, we can quickly reset the environment and run again. Make sure you keep two backup revisions (this one and the one at step 2) separately.

Checklist:

• [ ] The system state (data & configuration) is in good condition for a backup.
• [ ] You took a backup and stored it in a safe place separated from the other backup.

Step 5: Run migration

Run the processes in the expected order.

Monitor the progress constantly.

Monitor the resource consumption on 3 servers. You can consider adjusting the "Max concurrent batches" to best utilize the available resources.

If there is an error that you can't just simply fix and resume the process, you can restore the whole database from the backups taken in the previous steps.

Step 6: Verify data

You can do the following:

• Review the "Migration result" report.

• Browse the data lists to review the data. Pay more attention to the lists with the largest amount of data.

• Run smoke tests to ensure the system still works as expected.

Checklist:

• [ ] You reviewed and compared the data in source and destination databases.
• [ ] You reviewed the data in major data collections.
• [ ] You prepared a set of test cases to smoke-test the system after data migration.
• [ ] You smoke-tested the system after data migration.

Step 7: Restore the original settings

After the data migration finishes, you should restore the configurations to the previous state, which is documented in step 1.