How to Handle Data Duplication in Airtable Part 2

This is the second part of our updated deduplication series. While we focused on scripting and automations to remove duplicates in part 1, this article will focus on the Dedupe extension. If you have not read the first part, you can read by Clicking here.

What is the Dedupe extension?

The Dedupe extension is easy to use and can be a lifesaver for anyone who works with Airtable. It is an extension available on the Airtable marketplace that streamlines the removal of duplicates. We usually recommend using Dedupe if you expect to have less than 100 duplicates. Otherwise,

Here's a quick rundown of how it works:

1. Find Duplicates: You can use the Dedupe extension to find duplicates in a specific Airtable table or view. The extension can find duplicates based on exact, similar, or fuzzy matching. This means that you can find duplicates even if the data in the records is not exactly the same.
2. Merge Duplicates: Once you've identified the duplicates, you can use the Dedupe extension to merge them. This will create a single record that contains all of the unique data from the duplicate records. You can choose which fields to merge from the duplicate records, so you can be sure that you're not losing any important information.

If you are already an expert in using Airtable, here you can find the link to the documentation of the extension. If you are not, below you can find a guide on how to install the extension and how to use it.

Installing Dedupe Extension

Step 1: logging in to Airtable and navigate to Extensions

To install the Airtable Dedupe extension, begin by logging into your Airtable account. Once you've successfully signed in, navigate to the 'Extensions' tab at the top right of the screen. This will open the extension sidebar with an option to add extensions. After you clicked “add extension”, the marketplace will open.

Step 2: installing the Dedupe Extension

In the search bar at the top of the marketplace, type 'Dedupe' and press Enter. The first option should be Airtable’s official “Dedupe” extension. (see screenshot below).

Click the 'Add' button to initiate the installation process. The Dedupe extension will then be listed among your installed extensions, ready for use.

Using the Dedupe Extension

Step 1: Setting up the deduplication process

To begin using the Dedupe functionality, you need to select the table and the view where it should be applied on. If you have a lot of data, it might be useful to pre filter the data using a view before you use the Dedupe extension. Given that it still has a manual component to it, pre-filtering your data might save you a lot of time during the deduplication process.

In addition to the table and the view, you need to select the fields which the extension should check for duplicates using the toggles on the left hand side. You can also combine multiple fields and decide for each of them whether the match should be exact, similar or fuzzy.

Here you have a brief explanation on how the different types of matches compare to each other:

• Exact match: The values are the same, even considering capitalization and punctuation.
• Similar match: The values are the same, but may have different capitalization, punctuation, accents, whitespace, or word order.
• Fuzzy match: The values are similar, even with typos or transposed characters. This type of matching is prone to false positives.

Step 2: Merging the duplicate records

After you set up the extension, it will automatically find all the duplicates. Nevertheless, you will need to manually handle all the duplicates it found. In our fictional example, customers filled out a form multiple times because they were not sure whether the website worked.

In the screenshot below you can see that the extension prompts you to choose which fields you would like to keep in a combined record and deletes the remaining records replacing it by a combined record. This then needs to be repeated for all the duplicates the Extension found.

If you have thousands of records, it might make sense to set up a script that automatically combines and merges records instead of doing it manually. In the first part of this article series, we explained the basics of setting up a script. Nevertheless, if you are not familiar with Java Script, it might be more time and cost effective to talk to an accredited Airtable developer. Feel free to reach out for a complimentary consultation to discover the solution space for your problem.

Additional tips for using Dedupe

• Test on a Small Dataset: Before running the Dedupe extension on your entire dataset, try it on a smaller sample to ensure it works as expected and to fine-tune your settings.
• Backup Your Data: It's always a good practice to create a backup of your Airtable base before making significant changes.
• Iterative Process: Dedupe might be an iterative process. You might need to run it multiple times with different settings to achieve the desired results.

If you face any issues and you need a professional's help, Feel free to contact us for consulting by clicking here