We Have a Database Migration to Do

Let’s set up this scenario: you have a current system that you are using from 3rd party. Now you decide that you want to take ownership of your own data and be able to query your data to build reports on top of it.

Yes, you can periodically download a bunch of CSV files import into a spreadsheet and give you the result.

What’s the problem?

First of all, this method is very fragile, as it would require human interaction to download the files and input them into a spreadsheet

Note: Assuming that there is no API or plug-in to connect to your system, for example, a connection from HubSpot> Zapier > Google Sheets.

Second, if you delegate this task to someone or a department, only they will know how it works; if someone leaves the company, only “John” knew how to keep that report updated.

Then you decide: let’s build our own system, BUT we need to bring all the historical data.

Building the internal system is the easy part

• Everything works
• Your test data is working
• The system runs very smoothly

Now the real test is getting the historical data into the system

(oh here we go)

Getting the data

You contact your provider to send you a full data dump from their system because you want to migrate.

• They send you the data catalogue
• Select what data types you want
• They send it to you.

So far so good

Wait, we designed the database and the table for clients requires first name and last name, but the data they provided only has full name.

Yes, that’s the kind of issue you will face when doing a migration for something you didn’t own initially, without knowing the full data definition and making assumptions when designing a database.

From here, you will need to build microprocesses to metigate this kind of scenarios.

Ready to migrate

Finally, when you can align your database design with the data provided. You are ready to go

• Create migration files
• Rollback plan
• Downtime expected
• Communications
• etc

It’s time to import into the production database

MIGRATION COMPLETED!

Job done?

Nope

The critical phase

Now you’re in the critical phase; you will need to run close monitoring on production to see that all the features you developed display the correct data expected from the data you imported.

Perhaps you will need to fix some other data, or maybe it’s only bug fixes you need to do on the backend to handle historical data.

Doing a database migration is a process that need to be though very careful for each steps, never easy job to do and take a lot of effort from the team to achieve it.

If you’re reading this and you’re doing, or will be doing, a migration, the advice I can recommend is to make sure you have a good plan of migration before even start - mapping all the possible scenarios makes it 10x easier to act during the migration phase.

Not all the cases the migrations needs to be done all at once, assess your use case, plan and then act on it.