Component: Data migration - Working with files

Assuming you are migrating product photos from the well-known database AdventureWorks2019. The article will show how the component handles files when migrating data.

Step 1: Setting up the data class for storing files

Configure a data class CwDmSampleProductPhoto for storing data migrated from the table [AdventureWorks2019].[Production].[ProductPhoto]. The files will be stored in ThumbNailPhoto and LargePhoto, which have the type Document. Don't forget to add the field CwDmLegacyId required for tracking data.

Step 2: Configure a copy-data job

Configure a copy-data job as described below:

We will implement the Transform rule and the Test rule in the following steps.

Step 3: Implement the data-transform rule

Add a new Casewhere rule CwDm SO Migrate Product Photo:

We will use IDocumentApi to store the files. Remember to check null before proceeding.

#predicate
var documentApi = @apiFactory.Get<IDocumentApi>();
var item = @parameters["SourceItem"];

DocumentInfo thumbnailDocInfo = null;
if (item["ThumbNailPhoto"] != null)
{
	thumbnailDocInfo = documentApi.Create(item["ThumbnailPhotoFileName"]);
	documentApi.Store(thumbnailDocInfo, Convert.FromBase64String(item["ThumbNailPhoto"].ToString()));
}

DocumentInfo largeDocInfo = null;
if (item["LargePhoto"] != null)
{
	largeDocInfo = documentApi.Create(item["LargePhotoFileName"]);
	documentApi.Store(largeDocInfo, Convert.FromBase64String(item["LargePhoto"].ToString()));
}

var data = new 
{
	ProductPhotoID = item["ProductPhotoID"],
	ThumbnailPhotoFileName = item["ThumbnailPhotoFileName"],
	ThumbNailPhoto = thumbnailDocInfo,
	LargePhotoFileName = item["LargePhotoFileName"],
	LargePhoto = largeDocInfo,
	CwDmLegacyId = item["ProductPhotoID"]
};

// You can either return a DynamicDataObject or a list of DynamicDataObjects
var migratedItem = new DynamicDataObject(new {
	DataClassName = "CwDmSampleProductPhoto",
	Data = data,
	LegacyId = item["ProductPhotoID"]
});
return new List<DynamicDataObject>(){ migratedItem };

Step 4: Implement the data-verification rule

Add a new Casewhere rule CwDm SO Test Product Photo:

Remember to log errors that fail the test.

#predicate
var item = @parameters["SourceItem"];

var dataApi = @apiFactory.Get<IDataApi>();

var filter = FilterBuilder.Create().Eq("CwDmLegacyId", item["ProductPhotoID"]).Build();
var migratedItem = (dataApi.Search(DataObjectApiQuery
    .For("CwDmSampleProductPhoto")
    .Paging(0, 1)
    .FilterBy(filter)).Data as IList<DynamicDataObject>).FirstOrDefault();
 
if(migratedItem == null) {
    Log.Error("CwDmSampleProductPhoto not found for {ProductPhotoID}", item["ProductPhotoID"]);
    return false;
}

if(migratedItem.Get<DocumentInfo>("ThumbNailPhoto") == null && item["ThumbNailPhoto"] != null)
{
    Log.Error("ThumbnailPhoto is missing");
    return false;
}

if(migratedItem.Get<string?>("ThumbnailPhotoFileName") != item["ThumbnailPhotoFileName"])
{
    Log.Error("ThumbnailPhotoFileName is not matched. Expected={Expected}. Actual={Actual}", item["ThumbnailPhotoFileName"], migratedItem.Get<string?>("ThumbnailPhotoFileName"));
    return false;
}

if(migratedItem.Get<string?>("LargePhotoFileName") != item["LargePhotoFileName"])
{
    Log.Error("LargePhotoFileName is not matched. Expected={Expected}. Actual={Actual}", item["LargePhotoFileName"], migratedItem.Get<string?>("LargePhotoFileName"));
    return false;
}

if(migratedItem.Get<DocumentInfo>("LargePhoto") == null && item["LargePhoto"] != null)
{
    Log.Error("LargePhoto is missing");
    return false;
}

return true;

Step 5: Run and verify

Open the job Migrate [Production].[ProductPhoto] and click Start. If everything is configured correctly, you will see batches are generated and run sequentially.