File Migration Reconciliation & Reporting Challenges
The biggest challenge at the end of any migration is addressing file migration reconciliation. This entails identifying and handling content that has not yet successfully migrated and determining when the migration can be considered complete. The data reconciliation and validation process must be rock solid to prove to the migration team, project sponsors, and business users that the migration was confidently successful.
Defining File Migration Reconciliation Success
What does migration success look like?
It’s imperative that one defines migration expectations before the migration ever begins. Many organizations believe that all business data is 100% mission-critical and that only a perfect migration is acceptable. Unfortunately, this is often an unrealistic expectation. While perfect migrations do exist, they are the exception and not the rule.
Disparate Storage Platforms
There are many factors that can affect the migration success rate. Platform compatibility between source and destination alone might make it impossible to achieve 100% migration success. For example, larger files in the source system might be too big for the destination system to handle. Or file paths that are very long in the source system might be too long for the destination system. These differences in technical paradigms from one platform to another can cause content to be incompatible for multiple reasons.
What items can’t be migrated?
A common reason items may not be migrated is because they may be “corrupt” in the source system. Zero-byte files often find their way into systems for instance. Failed uploads or imports are commonly the culprits. Perhaps there’s a written record, but there is no binary data to go with it. These items, for example, would not be to be migrated. On the bright side, it can be easy to find corrupt items as one can usually trace the error in the migration back to an exception that also happens when a user tries to manually open the file on the source system.
Sufficed to say, systems are not perfect. Thus, migrations often are not perfect. However, we believe that they can be “perfected”. In the same sense that everything can be accounted for, a migration can be brought to a state of completeness. But this is not an effortless process, particularly when data continues to change. The key to defining success is to run a pilot migration to identify where the land mines are going to be. Then you can create a plan to either have end-users manually remediate content that cannot be migrated or to leave behind content that is either corrupted or otherwise unimportant.
Migration Status Detail Can Be Confusing
Every migration solution worth its own weight includes some form of status regarding the success or failure of each processed item. While migration teams do learn to understand these details, they can sometimes be confusing to end-users. And, not to mention, they are critical during the remediation process. Consider the following example of a migration status:
As a migration engineer, the status amounts above show that the migration process did everything possible to execute a flawless migration. It shows that 883 items migrated perfectly while 6 items needed to be automatically remediated in some way to get them to migrate successfully. It additionally showed 72 completely ignored items.
However, as an end-user, this looks confusing. What exactly does “revised” mean anyway? Did those items migrate successfully – or not? And what about the ignored items? All end-users know is that they have a certain number of documents in the home drive in the old system, and a certain number of files in the user drive in the new system. But when those numbers do not add up, they can get very confused!
File Migration Validation Reporting Makes Things Easy
It’s important that every migration includes validation reporting. It should be an accounting of content on the source system as well as content on the destination system. The goal is to identify all items in both systems and account for them, even if they are not all migrated. If done properly, there should be a bottom line comparison that factors in things like ignored items or items that already existed in the destination system in order to come up with matching numbers that prove the migration was successful.
DryvIQ includes a Validation Reporting area for every job that does exactly this. Here is an example of validation reporting from a DryvIQ job:
Continuing from the example above, the end-user might notice that they have 961 items in their source Box account. But after the migration, they only have 889 items in their destination OneDrive for Business account. This often causes confusion and doubt that the migration process is incorrect, and cannot be trusted.
They naturally contact the migration team who provided the user with this report, showing that the filtering rule ignored 72 items. It’s possible that they were simply video files or other unnecessary file types, for instance. Then the user remembers that they uploaded their vacation movies to their home drive years ago and it all makes sense now. The important concept in the image above is that the accounted for numbers in the example above actually match up. When we subtract the number of ignored items from the identified items in the source, the 889 item count matches the number of items in the destination!
Compliance and User Confidence
Ultimately, any successful migration will accomplish replicating content from one endpoint to another. But that is only half the battle. You must be able to also prove this success through validation reporting to address any file migration reconciliation issues. Validation reports provide the evidence needed to prove to auditors that the migration was successful while also proving to end-users that their data has been kept safe. Then, and only then, can the migration be considered “done.”
Ready to begin your migration project?
Before you do, setting expectations is critical. Read our 7 Best Practices on File Migration Without Getting Burned to make sure you’re setting yourself up for success.
Or have you already planned on what to expect? Contact us today.
