Data Migration Briefs & Planning

Must have process and guidelines for planning and briefing data migrations

Data migrations are incredibly complex with massive risk to the client if they are not done properly.

Before embarking on a Data migration the person managing the work must have at least 4 hours free to do the initial planning and design and 2-3 consecutive days available to manage through the implementation and testing.

Data Migration Brief

A Data Brief should be created from the "{3CODE} | {Project} | Data Brief | working" Template within the client project or presales folder.

This brief will aim to confirm:

  • Original Platform
  • New Platform
  • What needs to be Migrated:
    • Contacts
    • Companies
    • Deals
    • Tickets
    • Activities - including what type
    • Attachments
    • Object Associations - multiple or single object associations
    • Anything else, e.g. custom objects
  • The approximate number of records of each of the above
  • Migration timeline:
    • When Access will be provided to the original platform
    • When the team will stop using the original platform
    • When the team will start using the new platform

Data Migration Planning & Design

Where possible existing migration tools should be used.

Existing Migration Tool Exists

  1. Send the Brief to the supplier and get a quote back
  2. Ensure that all migration items from the brief are included 
  3. If anything is not available in the migration stop and organise a session with Commercial and the Client to discuss a way forward
  4. Do a sample migration
  5. Internally validate the sample records migrated against the original platform records - ensure that all items from the data brief are included in the migration, ensure that all data in the new platform reflects the original platform data
  6. Have the client approve the sample migration - note they must check off each item from the data brief for the sample records

Migration Tool does not Exist

  1. Work with a developer to review the APIs of the Original and New platforms to ensure that the data specified within the brief can be extracted and imported
  2. If anything is not available in the migration stop and organise a session with Commercial and the Client to discuss a way forward
  3. Get access to the original platform including access to the API
  4. Work with a developer to test the data received from the API to ensure that the data specified within the brief can be extracted and imported - this should be tested for each piece of data specified in the brief
  5. If anything is not available in the migration stop and organise a session with Commercial and the Client to discuss a way forward
  6. Organise a half-day session to do a detailed review of the current platform
  7. In the session review each object page specified in the Data brief and note what data needs to be migrated
  8. Identify 5x sample records for each object
  9. Create a test plan for testing the data migration, this should include what objects and data needs to be verified and should reflect the original brief
  10. Work with a developer to test migrate the 5 sample records 
  11. Once the data is migrated, execute the test plan - verify the data migrated for each object ensuring that data in the original platform records matches the data in the new platform
  12. If not, work with the Developer to troubleshoot and fix
  13. If anything can not be fixed stop and organise a session with Commercial and the Client to discuss a way forward
  14. If the sample test data is verified then provide to the client to review and approve the sample records, note that the client should have this sample data approved by at least 3x CRM users who are intimate with the current data
  15. On approval execute a test full migration of data test with the client and resolve any issues - note testing of this data should include record counts of the original and new platforms for migrated records by record type - if there is a mismatch this is a failed test and it will need to be rerun
  16. Once a full migration has successfully executed coordinate the cut over time and date and schedule the developer for the final migration - dependent on the client preferences this may need be done on a weekend to minimise CRM access downtime for the client team
  17. Ensure that the client has sent internal comms, has completed training and reconfirm cut over date
  18. On cutover, if possible stop access to the original platform
  19. Coordinate with the developer to start the migration
  20. Once the data is migrated, execute the test plan - verify the data migrated for each object ensuring that data in the original platform records matches the data in the new platform and that all data has been migrated - note testing of this data should include record counts of the original and new platforms for migrated records by record type - if there is a mismatch this is a failed test and it will need to be rerun
  21. If there are any issues notify the client and work with the Developer to resolve
  22. If the issues are going to take more than an hour to resolve, speak with the Client to discuss how they would like to progress
  23. Once all issues are resolved and the test plan is successfully passed notify the client and get them to do a final verification, note this should include at least 3x CRM users who are intimate with the current data
  24. If there are any issues work with the Developer to resolve
  25. If the issues are going to take more than an hour to resolve, speak with the Client to discuss how they would like to progress
  26. Once all issues are resolved send an email to all parties confirming the completion of the migration and organise for the retirement of the original platform (recommend 1-2 weeks from a successful migration just in case new issues are identified)

Note during the final migration If there are significant issues which would cause CRM access to be down longer than acceptable the client may need to revert back to the original CRM and the migration may need to be stopped and rescheduled. So it is critical that you are in constant communication with the client around any issues that arise and estimated resolution times.

If a migration fails a deep dive of the issues and retest of migration must be done prior to the next migration.

Best Practice Steps for Data Integrity, Verification and Validation

1.   Test at each step of importation that data is correctly transformed when necessary. If                    feasible, duplicate the dataset to ensure data integrity while wrangling before the final                    migration is made.

2.   Data verification can be achieved by creating "block lines". Highlight lines of rows                          systematically throughout the dataset, and validate that the lines match up after each column        transformation during wrangling.

3.   Sampling should be strenuous throughout the wrangling process. Choosing appropriate test        cases (such as testing the start and end values of the dataset, boundary values of category          metrics, or closely related entries) will greatly aid in the wrangling process.

4.   Determine what the "unique key" of your dataset is, when importing. Many tools such as              HubSpot will require a unique key when matching up one dataset to another, so it is                      important that careful consideration of what the unique key for each of your rows are, when          using your specific migration tools. Unique keys are an efficient way to do data validation              through importations.