DocumentTaskEngine Class

public class DocumentTaskEngine

Inheritance

Object    DocumentTaskEngine

See methods RunTask and RunTaskBlocking, these methods provide highly parallel and very fast processing of batches of documents and processing of "large" archives or mail stores as a single task. Large archives and mail stores can also be broken into multiple separate tasks, see IsPartitioned.

When processing hundreds to thousands of documents as a single task, these documents should not add up to more than 4-5 gigabytes in combined file size, or else the outputted document data archive (.dda) could become too large to read into memory (see DocumentDataArchiveReader). If processing > 10 gigabytes of documents, break the documents into 2 to 4 gigabytes sized DocumentTaskSettings tasks and for "large" archive and mail stores ("large" being a subjective term) create separate tasks for these 'large' archives and mail stores using SingleArchive or SingleMailStore processing type, respectively.

For archives, be aware of the expansion size before deciding how to process. Archives can have very high compression ratios, for example, a 500 MB sized archive could expand into 50 GB worth of files. It is wise to test archives for true expansion size before expanding/extracting.

Breaking large document processings sets into separate tasks aides in distribution across multiple desktops or VMs and also aides in re-queuing of any failed task(s).

Note: DocumentTaskEngine can handle long file paths (>255 characters in length) for input documents.

Reference