The detection of duplicates in document image databases

Document imaging technology has developed to the point where it is not uncommon for organizations to scan large numbers of documents into databases with little or no index information. This may be done for archival purposes with an index as simple as a case number, or with the ultimate goal of automatically extracting index information for content-based queries. Maintaining the integrity of such a database is difficult, however, especially in a distributed environment where copies of the same documents may be scanned at different times.In this paper we present a novel approach to detecting duplicate documents in very large databases using only features extracted from the image. The method is based on a robust `signature' extracted from each document image which is used to index into a table of previously processed documents. The system is able to deal with differences between scanned document instances such resolution, skew and image quality. The approach has a number of advantages over OCR or other recognition-based methods including speed and robustness to imaging distortions.
To justify the approach and demonstrate its scalability, we have developed a simulator which allows us to change parameters of the system and examine performance while processing millions of document signatures. A complete system has been implemented and tested on a collection of technical articles and memos.