Iterative record linkage for cleaning and integration

TitleIterative record linkage for cleaning and integration
Publication TypeConference Papers
Year of Publication2004
AuthorsBhattacharya I, Getoor L
Conference NameProceedings of the 9th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery
Date Published2004///
PublisherACM
Conference LocationNew York, NY, USA
ISBN Number1-58113-908-X
Keywordsclustering, deduplication, distance measure, record linkage
Abstract

Record linkage, the problem of determining when two records refer to the same entity, has applications for both data cleaning (deduplication) and for integrating data from multiple sources. Traditional approaches use a similarity measure that compares tuples' attribute values; tuples with similarity scores above a certain threshold are declared to be matches. While this method can perform quite well in many domains, particularly domains where there is not a large amount of noise in the data, in some domains looking only at tuple values is not enough. By also examining the context of the tuple, i.e. the other tuples to which it is linked, we can come up with a more accurate linkage decision. But this additional accuracy comes at a price. In order to correctly find all duplicates, we may need to make multiple passes over the data; as linkages are discovered, they may in turn allow us to discover additional linkages. We present results that illustrate the power and feasibility of making use of join information when comparing records.

URLhttp://doi.acm.org/10.1145/1008694.1008697
DOI10.1145/1008694.1008697