Content Deployment - Best Practices

In the past I have released several blogs about the various problems that can occur with Content Deployment. As it is often hard to find the right resource I have now decided to compile the different known issues into one large article:

  

Avoiding common problems

Problem 1: Mixing deployments with and without retaining object identity

First of all: if possible you should avoid this! Importing with different settings for this property into the same database can lead to serious problems during future deployment and such databases will become hard to maintain.

Be aware that this also means that you should not use STSADM -o import against a database that should be used as the destination of a content deployment job!

STSADM -o import will not retain the object identity while content deployment does.

So why is there a problem when mixing imports with different RetainObjectIdentity settings?

The reason is that with RetainObjectIdentity enabled the imported object will have to be created with the same name and the same guid at the same location in the destination database as it was in the source database. If the item already exists it will be updated. If not it will be created.

Problems occur if there is an item with the same name but a different Guid in the destination database. This can happen if someone has authored on the destination server and created items with the same name but or if he imported content from the source server WITHOUT RetainObjectIdentity setting set to true.

In case that items with the same name but different GUID are allowed for the affected item you will end up with two items with the same name on the destination server. This will be the case (e.g.) for usual ListItems.

In case that items with the same name but different GUID are not allowed for the affected item the import will run into an exception similar to the one below and stop:

Failed to create the 'Pages' library. OriginalException: There can only be one instance of this list type in a web.

=> In order to avoid this problem you have to guarantee that content added to the destination database will not have any name/guid conflicts with the source database - even if new content is added to the source database in the future!
 

Problem 2: Running multiple imports without retaining object identity for updates of the same content

When doing an export and import without retaining the object identity on the destination server you can end up with duplicate items in lists as each import tries to create the same list item again with a different GUID. The import is not able to decide whether you would like to overwrite an existing item with the same name or if you would like to have multiple list items with the same name. Without retaining the object identity you will end up with multiple list items with potentially the same content. To force overwrite of list items you have to retain the object identity.

That means you cannot use STSADM -o export/import as a replacement for content deployment! If you need to do deploy content to a remote destination server without connectivity you need to write a custom tool that has retain object identity enabled rather than using STSADM -o import based on the code samples provided in Part 3 of this article series.

=> STSADM -o export and import should only be used if the content being imported does not already exist in the destination database and if the database will not be used as the destination database for content deployment (see Problem 1 above).
 

Problem 3: delete an item from the source site that belonged to the site definition and recreate it

This is a different flavor of the problem discussed as Problem 1. 

During provisioning of a site items defined in the site definition template are added to the site. Problems can occur when changes are made to the provisioned items. Especially if the provisioned items are deleted and replaced with items with the same name. That approach will work well on a single server installation. But it will cause problems when using content deployment.

The reason is that during content deployment the site will be provisioned on the destination server using the site definition template. And this will also cause all items defined in this template to be created. When content deployment now tries to import the updated or replaced items there will be a conflict. You will end up with an exception similar to the one in Problem 1.

=> You should never modify or delete one of the items created through the site definition in your site. If the site definition does not suite your needs you should create a custom site definition that fits to your needs and use this instead to avoid the need to customize some of the provisioned items.
 

Problem 4: deploy from destination back to source

This is something that theoretically can be done but only if the source hasn't changed since it was last deployed to the destination. Otherwise the same issues as in Problem 1 can occur.

Also be aware that it will not be an incremental deployment - means you cannot just deploy the changes since you deployed from source to destination. The reason is that the timestamp information about what to deploy is stored with the deployment job. As this information only exists on the source system the first deployment from destination back to source will deploy everything! So the result would be the same as deploying into an empty site collection on the soruce system. And actually deploying into an empty site collection would be better to avoid problems in case that changes have been done on the source system.
 

Problem 5: deploy partial content without exporting the parent items

When deploying with retaining the object identity (as you can do with content deployment in the central admin) it is not possible to reparent items. Deployment with retaining the object identity requires that the identity of the object is the same on the destination server and the identity is defined by Id, Name and by the Url.

So the parent of each deployed object has to exist on the destination server in order to successfully import the package on the destination server. We have seen that customers are trying to export a specific subtree of the site without exporting the parents. E.g. only a specific variation label without the variation root.

If the parent of the exported objects does not exist on the importing site then the item cannot be imported and the deployment will fail.

=> Ensure that all parents of all content being exported exists on the destination server.
=> Or create a custom export tool that does not retain object identity and changes the parent during import as discussed in Part 3 of my content deployment and migration API article series but be aware about the limitations discussed as Problem 2.
 

Problem 6: deploy partial content with references outside the selected scope

This is similar to Problem 5 except that we assume that the parent objects of the selected object exists in the destination database. In this situation you might assume that no problems should occur. That is not correct. When exporting items in a subtree per default all referenced objects (like images or documents) will be exported as well. Even if these objects are outside the selected scope. In this situation the export package will contain objects which might not have a parent in the destination database. 

If the parent of this image or document does not exist on the importing site then the item cannot be imported and the deployment will fail.

=> Ensure that authors do not use resources from other parts of the site collection which is not being exported.
=> Or create a custom export tool that uses the ExcludeDependencies to exclude objects outside the selected export scope. See Part 2 of my content deployment and migration API article series for more details.
  

Problem 7: mixing full and incremental deployment

A common error I often see is that customers are mixing incremental and full deployment when deploying a site collection. E.g. use incremental deployment every day and full deployment once a week - just to be sure (maybe because they do not trust incremental?)

The problem is that incremental and full deployment do not deploy the same type of content. So the data on source and destination might no longer be the same.

Why can this happen? The reason is that only incremental deployment exports deleted items which is required to ensure that the items also get deleted on the target server.

Consider the following scenario:

1. Site A has been created
2. Site A gets deployed to the target server
3. Site A is deleted
4. a full deployment is done
5. further actions
6. an incremental deployment is done

With step 4 the deleted site will not be exported. The incremental deployment in step 6 will only deploy changes done between step 4 and 6. That means that after step 6 site A still exists on the target server!

You might not even notice this if you are not actively monitoring if the content on source and on target is in sync. Only if content with the same name gets recreated later you will notice it as you will get errors similar to the following:

• "The Web site address "/A" is already in use." or
• "The specified name is already in use. A document cannot have the same name as another document or folder in this document library or folder."

That affects all type of objects (e.g. sites, pages, lists).

To ensure that the target and the source site collection are in sync it is mandatory to use full deployment only once: when doing the initial deployment from source to target. Afterwards you should only use incremental deployment and not add additional full deployment steps. That also means you should avoid situations where content deployment fails to do an incremental due to the fact that the timespan between incremental deployments exceeds the configured retention of the change log (see here for detail).

On the other hand that also means: whenever you see a need to do a full deployment you should do this full deployment into an empty site collection rather than the earlier created site collection that already received content from earlier deployments.
 

Problem 8: the cab file size exceeds the configured maximum content length configured in IIS 7

Some background infos: Content Deployment in MOSS 2007 first exports all content to the file system as XML and binary files and afterwards packages these files into cab files which then get uploaded through http to the target MOSS server where they then get extracted and imported.

The preconfigured maximum size of the cab files generated by content deployment is 10 MB as discussed in this article. IIS 7 on the other hand has a preconfigured upload limit of 29 MB as discussed in KB article 925083.

So with these two limits (maximum size of a cab file = 10 MB and maximum upload size in IIS 7 = 29 MB) we would not expect any problems.

The problem here is that MOSS does not split single exported files into multiple CAB files. So in case that the MOSS site contains single documents which cannot be compressed to less then 10 MB then the CAB file size can become bigger than 10 MB.

This will become critical as soon as the CAB file size exceeds 29 MB.

As soon as content deployment trys to upload a CAB file with more than 29 MB in size the content deployment job will fail and you will find the following entries in the application event log of the exporting server:

Event ID: 5323 
Source: Content Deployment
Description: Failed to transfer files to destination server for content deployment job 'test 1'.  Exception was 'System.Net.WebException: The remote server returned an error: (404) Not Found.

Event ID: 4958
Source: Content Deployment
Description: Publishing: Content deployment job failed. Error: 'System.Net.WebException: The remote server returned an error: (404) Not Found.

Event ID: 6398
Source: Windows Sharepoint Services
Description: The Excecute method of job definition Microsoft.Sharepoint.Publishing.Administration.ContentDeploymentJobDefination threw an exception. The remote server returned an error (404) Not Found.

In addition you will find similar entries in the ULS log on the source server:

OWSTIMER.EXE (0x0778)                            0x079C CMS                                      Content Deployment                            0             Unexpected      ContentDeploymentJob.DoServerToServer: Remote connection failed while uploading files for source-Job 'Test' with exception 'The remote server returned an error: (404) Not Found.' 

To see if the problem really is the upload size you need to check the IIS log on the target server to see if the response is indeed a 404.13

2008-09-01 08:55:08 10.10.10.2 POST /_admin/Content+Deployment/DeploymentUpload.aspx filename=%22ExportedFiles13.cab%22&remoteJobId=%11456fa7ed-ddcdedcdd-9aae-a1adsf5re1db%22 1976 - 10.10.10.3 - 404 13 0 62

To solve this issue you need to modify the web.config of the Central Admin site and add an entry similar to the following:

<system.webServer>
    <security>
        <requestFiltering>
            <requestLimits maxAllowedContentLength="52428800"/>
         </requestFiltering>
    </security>
</system.webServer>                

52428800 = 50 MB in this sample. You might need to adjust this to your specific needs.

Problem 9: Features used in the site collection are missing on the exporting server

A common problem we see with content deployment and with STSADM -o export is the error message below:

[4/11/2008 9:25:01 AM]: FatalError: Failed to compare two elements in the array.
   at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.Sort[TValue](T[] keys, TValue[] values, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Collections.Generic.ArraySortHelper`1.Sort(T[] items, Int32 index, Int32 length, IComparer`1 comparer)
   at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
...
*** Inner exception:
Object reference not set to an instance of an object.
   at Microsoft.SharePoint.SPFeature.EnsureProperties()
   at Microsoft.SharePoint.SPFeature.get_TimeActivated()
   at Microsoft.SharePoint.Deployment.WebSerializer.ExportFeatureComparer.System.Collections. Generic.IComparer<Microsoft.SharePoint.Deployment.ExportObject>.Compare(ExportObject exportObject1, ExportObject exportObject2)
   at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)

The usual reason for this problem is that some sites in the site collection have features assigned where the feature is not installed in the server farm. The main problem here is that you cannot easily identify which features are missing as the missing features are not reported in the error message.

To overcome this problem I have written a tool which allows to identify all features used in a site collection which are missing on the server: WssAnalyzeFeatures.

To resolve the problem you then have to install the identified missing features on the exporting server. In case that this is not possible you have to remove the features from the site colleciton or the affected sites. Usually this can be done using STSADM -o deactivatefeature but sometimes fails if the feaure definition is not installed on the server. In this case you can use WssRemoveFeatureFromSite.

Problem 10: SQL Deadlocks during STSADM import operations

In the last couple of weeks I have seen several cases where STSADM import operations failed with random exceptions. With other words: performing the same import into an empty site collection multiple times the import operation failed at different points during the import. Checking the ULS logs showed errors like the following:

10/20/2008 12:47:26.59 STSADM.EXE (0x78BC)                         0x4FF4 Windows SharePoint Services            Database                        6f8g      Unexpected       Unexpected query execution failure, error code 1205. Additional error information from SQL Server is included below. "Transaction (Process ID 123) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction." Query text (if available): "..."         

Such a behavior is usually an indication that asynchronous actions interact with the import operation and caused a deadlock in SQL server.

Another interesting tidbit is that this only affects STSADM import but not content deployment. 

Isolating these issues is not very easy as SQL asap kills one of the deadlocking queries and the child process (in our case STSADM.EXE and potentially a second process) continue to run till they finally fail due to the fact that the SQL query did not succeed.

In a test environment it is possible to isolate the issue by attaching a debugger to the SQL server and setting a breakpoint right before the deadlock victim is killed. That causes the deadlock to persist and allows to take memory dumps of STSADM and the other involved processes.

In the cases I have worked on the problem was always caused by a custom event receiver fireing when importing the items. That also explains why only STSADM -o import is affected but not Content Deployment: with STSADM -o import the After event handlers will fire while Content Deployment suppresses After events through the import settings SPImportSettings.SuppressAfterEvents.

Unfortunatelly STSADM does not provide an option to suppress the after events. So there are two possible way to resolve the issue:

1. Disable the event receivers in the features on the target machine when performing the import
2. Create a custom import application which uses the content deployment and migration API as discussed in Part 3 of my Deep Dive into the content deployment and migration API series and sets the SuppressAfterEvents property to true.
    

Problem 11: Insufficient disk space on the target server

Another issue which I have seen a couple of times in the past when using STSADM -o  export or content deployment is a failure during the export phase with the following error message:

Failed to create package file
Unknown compression type in a cabinet folder

In all cases I have seen this problem was caused by insufficient disk space. Please monitor the disk space during the export/compression phase and ensure that sufficient disk space is available to perform this operation.

Problem 12: Incremental deployment fails with "The changeToken refers to a time before the start of the current change log."

I have seens this problem a couple of times in the past: sometimes when running incremental content deployment the deployment job fails with the following error message

The changeToken refers to a time before the start of the current change log.

To make a long story short: to resolve the problem you should do a full deployment into an empty database. It is not recommended to run full deployment into the previously used database as the full deployment will not perform delete operations of items that have been deleted in the source database but still exist in the destination database. It will also not reliably work if items have been moved on the source database to other places after the last successful incremental deployment has been used.

Why does this problem occur? This can happen mainly for three different reasons:

A) The timespan since the last incremental deployment job is too long

MOSS stores the change token of the last successful incremental deployment inside the properties of the incremental content deployment job. When a new incremental deployment is run it compares the change token in theses settings with the entries in the change log.

Per default the change log is configured to keep changes for 15 days. If the timespan between two incremental deployment job exceeds this timespan then the change log does not contain entries from before the change token and incremental deployment will not start to prevent deploying only parts of the required content.

A solution for this would be to increase the timespan the change log should be preserved. This can done using the following steps:

Central Administration - Application Management - Web Application General Settings - (choose the desired website) - Change Log

B) The source database has been overwritten with a backup

When a database is restored through STSADM -o restore or using SQL backup and STSADM -o addcontentdb the change log is cleared.

Incremental deployment will not work in this case and you have to do a full deployment to synchronize the content with the destination database.

C) No changes have happend on the source server for a long time

As mentioned in A) the change log will be used to determine the items that need to be deployed. Per default changes are preserved for 15 days in the change log. So if no changes have been done for 15 days the change log will become empty.

Two possible solutions exist: 

1. Increase the timespan the change log should be preserved. This can done using the following steps:
   Central Administration - Application Management - Web Application General Settings - (choose the desired website) - Change Log
2. Ensure that at least one item is modified within the configured timespan

D) The source database has been merged with a different site collection

After merging two content databases using stsadm -o mergecontentdbs the EventCache table will be empty. This is similar to problem B) listed above.

Incremental deployment will not work in this case and you have to do a full deployment to synchronize the content with the destination database.

  
Problem 13: Timeouts during content deployment

During the import the exporting server that is hosting the content deployment job definition will poll the destination server in specific intervals to get the status of the import operation. The import will go through different phases like extracting and then performing the import. This timeout actually defines the maximum time that is allowed between changes to the status report. During the actual import the report changes quite often. But there are some phases - e.g. the decompression - which can take some time. If this time exceeds this remote timeout value then a timeout is reported back to the exporting machine. You might have seen this status in the content deployment jobs. Such a timeout does not mean that the import has failed! It just means that no status update did arrive for the configured time. The default here is 600 seconds which means 10 minutes.

To see if the deployment has succeeded or not you need to hover over the job that reported the timeout and choose "Check Status" in the drop down box. This option will only show up if the Job is in the status "Timed out". The Check Status option allows to see if the job succeeded or failed.

In case that you have a large content deployment running with several GB of content it can happen that the decompression phase takes longer than 600 seconds - and then you will the a timeout. Adjusting this parameter allows you to avoid such timeout messages.

Here is some sample code which allows you to configure the RemoteTimeout to one hour. It would be required to run this on the exporting farm if required:

using System;
using Microsoft.SharePoint.Publishing.Administration;  

namespace StefanG.Tools
{
    class AdjustContentDeploymentDeploymentSettings 
    {
        static void Main(string[] args)
        {
            ContentDeploymentConfiguration config = ContentDeploymentConfiguration.GetInstance();
            config.RemoteTimeout = 3600;
            config.Update();
        }
    }
}  
   

Problem 14: Columns with same name defined on sub site and root site

Consider the following scenario: you create a new column named "MyCustomColumn" in a sub site of a site collection. Later you create a column with the same name in the root site of the site collection. If you now check again on the sub site you will notice that there are two columns listed with the same name.

Such duplicate column names cannot be handled by content deployment import. You will get an error error message similar to this:

A duplicate name "MyCustomField" was found. 
   at Microsoft.SharePoint.SPFieldCollection.AddFieldToWeb(String strXml, Boolean checkDisplayName) 
   at Microsoft.SharePoint.SPFieldCollection.AddFieldAsXmlInternal(String schemaXml, Boolean addToDefaultView, SPAddFieldOptions op) 
   at Microsoft.SharePoint.Deployment.FieldTemplateSerializer.CreateField(SPWeb web, SerializationInfoHelper infoHelper) 
   at Microsoft.SharePoint.Deployment.FieldTemplateSerializer.SetObjectData(Object obj, SerializationInfo info, StreamingContext context, ISurrogateSelector selector) 
   at Microsoft.SharePoint.Deployment.XmlFormatter.ParseObject(Type objectType, Boolean isChildObject) 
   at Microsoft.SharePoint.Deployment.XmlFormatter.DeserializeObject(Type objectType, Boolean isChildObject, DeploymentObject envelope) 
   at Microsoft.SharePoint.Deployment.XmlFormatter.Deserialize(Stream serializationStream) 
   at Microsoft.SharePoint.Deployment.ObjectSerializer.Deserialize(Stream serializationStream) 
   at Microsoft.SharePoint.Deployment.ImportObjectManager.ProcessObject(XmlReader xmlReader) 
   at Microsoft.SharePoint.Deployment.SPImport.DeserializeObjects() 
   at Microsoft.SharePoint.Deployment.SPImport.Run()

To resolve this issue you need to ensure that the column names are unique - either by deleting and recreating the column with a new name in the sub site or the root site. Having such a configuration is not a good idea anyway as it will be hard to identify which of the columns is the one you would like to pick when creating a new content type or when adding to a list.
     

Problem 15: Conflicting content on source and target

Content deployment expects that items it is trying to import do not exist on the target or are older revisions of the items it is currently deploying. In case it finds an item with the same URL as an item that gets imported it verifies the GUID of the item to verify that it is indeed the same item as the one being imported.

If the GUID is the same import will create a new revision (if versioning is enabled) or replace the item in the target site.

In case that the GUID is different you will get an error similar to the following:

• "The Web site address "/A" is already in use." or
• "The specified name is already in use. A document cannot have the same name as another document or folder in this document library or folder."

To avoid the problem it is required to ensure that no content is added on the target through (e.g.) authoring activities that exists on the source. Best would be to avoid changes on the target server at all.

  
Problem 16: Maximum upload size on target web application smaller than files to be deployed 

Often the web application on the target is created with the default values - which means a maximum upload size of 50 MB. In case that the source site collection contains files which are bigger than the upload limit the deployment will fail with an error message similar to the following:

The form submission cannot be processed because it exceeded the maximum length allowed by the Web administrator. Please resubmit the form with less data.

Content Deployment will still succeed! So be aware that you need to review the content deployment logs even if the status of the job is Succeeded.

To avoid this problem ensure that the maximum upload size on the target is at least as big as the larges item in the source site collection.
 

Problem 17: Document in source library is marked as virus infected

I have seen a couple of cases on this. Often the document is not really virus infected but something went wrong during with the virus scanning during the upload. A common reason is that while the document was uploaded the signature files were in the process of being updated. This can cause the uploaded files to be incorrectly marked as virus infected.

Independent if the file is really infected or not: Items being marked as virus infected cannot be exported!

Unfortunately the error message that occurs when items which are marked as virus infected are exported is not self-explaining:

Exception from HRESULT: 0x80041050
at Microsoft.SharePoint.Library.SPRequest.GetFileAsByteArray(String bstrUrl, String bstrWebRelativeUrl, Boolean bHonorLevel, Byte iLevel, OpenBinaryFlags grfob)
at Microsoft.SharePoint.Deployment.FileSerializer.SaveFile(SerializationInfo info, ExportObjectManager objectManager, ExportDataFileManager fileManager, SPExportSettings settings, SPWeb parentWeb, Boolean isGhosted, String setupPath, String setupPathUser, Byte setupPathVersion, String webRelativeFileUrl, Int32 size, Byte level)

The error code 0x80041050 means that the item is virus infected.

To resolve the problem either run another full scan to remove the incorrectly virus infection flags or delete the infected items.

  

Requirements for a successful content deployment

Be aware that this article cannot provide a complete list of all requirements. I will update it from time to time if required. Today I will list all the requirements I have seen in the past which - when missed - can cause content deployment jobs to fail.
 

1) The servers configured as export and import server need to host an instance of the Central Administration website

When configuring the "Content Deployment Settings" for your farm you have the chance to select different servers in your farm to work as export and import servers for the content deployment. This allows to offload this task to a dedictated server to reduce the load on your web frontend servers.

The comments of this configuration options look as follows (here shown for the Import server):

Import Server
Specify the server you want to receive incoming content deployment jobs. This server must have enough available disk space to store the incoming jobs, and it must be running an administration Web application for the farm.

So this requirement is actually listed but a little bit hidden in the second half of a sentence.

In case you configure a server that does not host the Central Admin website then you will not get a warning here!

Impact: The effect you will see is that the content deployment export or import phase will not start.

How to resolve:
1) Provision the Central Administration website on the desired server
2) Change the configuration of the export and/or import server
 

2) Ensure that sufficient disk space is available

This sounds like a simple prerequesit but it isn't. Content deployment uses different places to store the extracted and compressed files. The compressed files are stored in the location you can configure on the "Content Deployment Settings" page in the "Temporary Files" row.

But before export creates the compressed files it first exports everything into a temporary directory. And this directory is placed inside the directory which is configured in the TMP environment variable of the user the Windows SharePoint Services Timer service (OWSTIMER.EXE) runs on (usually refered to as farm credentials). Per default this variable has the following value: "%USERPROFILE%\Local Settings\Temp" which is usually on your system drive.

Impact: So per default MOSS content deployment requires the disk space for the uncompressed exported files on your system drive!

How to resolve: The easiest way to resolve this is to logon to the machine with the farm credentials and adjust the TMP variable to point to a different location. Afterwards you would need to restart the OWSTIMER service.
 

3) Use an empty site collection as the destination of your content deployment job

As already discussed in Part 5 content deployment will fail if the destination database contains conflicting content. To avoid this it is required that the initial deployment is done into an empty site collection.

Be aware that the only way to create an empty site collection is to use the following STSADM command:

STSADM.EXE -o createsite -url <url-to-site-collection> -ownerlogin domain\user -owneremail <email-address>

Using the "Blank Site" template will NOT create an empty site collection! It will actually create a site collection with content. You can see the difference if you create a site collection using both methods and then inspect the content of the created sites using SharePoint designer.

I personally recommed to always use the STSADM command with the syntax above to ensure that you really have an empty site collection as destination.

Impact: If the site collection has been created using a different method or already contains data the content deployment job will fail.

How to resolve: Deploy into an empty site collection
  

4) Install all required features for your site on the destination server

If your site requires custom features ensure that the features are installed on the destination server before running content deployment.

Impact: If the are missing the import phase will fail.

How to identify this: I have written the tool WssAnalyzeFeatures which allows you to identify such problems.

How to resolve: Copy the features to the destination server and install them using STSADM -o installfeature. 

5) Do not activate custom features for your site collection on the destination server manually

You should not activate custom features on the destination server if this activation creates content in the destination database as this can cause conflicts as outlined in "Problem 1" of Part 5 of this article series. Instead you should run content deployment and let the import process activate the features on your destination server as this will ensure that all items get created using the same ids as on the source server which is otherwise not guaranteed.

Impact: If the features have been activated and the content deployment import can fail with similar error messages in as "Problem 1" of Part 5.

How to resolve: Deactivate the feature in the destination site and ensure that all items created by the feature are removed. Alternatively do a full deployment into an empty site collection instead.
 

6) Do not expect that incremental deployment will deactivate features in the destination server site collection

The content deployment and migration API was not designed to deactivate features on the destination server. If a feature needs to be deactivated on the destination server you need to manually perform this deactivation.

7) Ensure that all feature definitions of features activated on the site collection exist on the source server

This is actually a high call generated for Microsoft Support Services: in the development a feature becomes obsolete and is removed or replaced with a different version with a different Guid but on some sites or site collection the old feature is still activated.

Impact: The affected sites can no longer be exported. You will get the errors listed in this article.

How to identify this: I have written the tool WssAnalyzeFeatures which allows you to identify such problems.

How to resolve: Either copy the missing feature files to the required location or uninstall the feature using STSADM -o deactivatefeature/uninstallfeature. In case that STSADM -o deactivatefeature fails to deactivate the feature you can use my tool WssRemoveFeatureFromSite.
 

8) Configure the retention period of the change log to be long enough for incremental deployment

See here for details.
 

9) Ensure that content deployment jobs do not run in parallel

The current implementation of the content deployment and migration API does not allow parallel execution. There are plans to change this behavior in the near future but as is you need to ensure that only one deployment is running at a time.

So if you have multiple deployment paths and jobs for the same site collection/content database you need to ensure to schedule them in a way that they don't overlap.

But this is not the only place to look at! Sometimes it is nearly impossible to prevent parallel execution. Just think that the content deployment and migration API is not only used to deploy content between different webfarms. The same API is used in the copy/move implementation inside site manager and in the variation feature.

With other words: you can experience problems with content deployment if an author copies a page at the same time or creates a new page in the source variation label which is then replicated to the destination. And also vice versa a copy operation can fail because a quick deploy job was running at the same time.

Impact: parallel execution of deployment jobs can lead to failing content deployment.

How to resolve: you need to restart the failed deployment job

  
10) Ensure that the patch level on source and destination farm is identical

Content deployment is only supported if the WSS and MOSS patch level is identical on source and destination. Some hotfixes have changed the schema of the export packages slightly which can cause deployments between different patch levels to fail.

Impact: content deployment and STSADM -o export/import can fail if the patch level is not identical

How to resolve: ensure that both farms are on the same WSS and MOSS patch level

  
11) Required Language Packs need to be available

Language Packs used in the source site collection have to be installed on the target farm as well. 

Impact: if a required language is missing content deployment will fail

How to resolve: install the required language packs on the target farm 

12) Avoid authoring activities on the target farm

Authoring activities on the target can cause conflicting content between source and target. E.g. if an author creates a page named "A" in site "B" and the same is done on the source server you will have conflicting content.

Impact: If conflicting content exists on the target server the import process will fail with an exception

How to resolve: Delete the conflicting item on the target server
 

13) Avoid authoring activities on the source farm while content deployment is running

Authoring activities on the source farm can modify collections while content deploymeng is trying to export them. E.g. if a list is being exported and in the middle of the export a new list item is added to the list content dpeloyment can fail to export the list. So if possible you should ensure that content deployment jobs are only run during a time when no authoring activities are expected.

Impact: Content Deployment Export can fail if authoring activities happen in parallel.

How to resolve: Rerun the content deployment job. You can also configure an automatic retry to cover such problems. See the Retry options mentioned in the following article: Pimp my Content Deployment Job