User Tools

Site Tools


transformations:deduplicate

DEDUPLICATE ROWS

Category: Transform / Filters


Description

This action removes all duplicate rows in the entire table. Deduplication can be performed based on all columns, or only on specific columns.


Use cases

Use this action to remove records that may have been duplicated in the source dataset or during previous actions, and keep unique records only.


Action settings

Setting Description
Apply toSelect whether to base deduplication on all columns in the dataset, or only selected columns.
Options: All columns or Selected columns (and select the columns to use from the list).


Remarks

When deduplicating based on specific columns:

  • The uniqueness of values in non-selected columns will be ignored.
  • Duplicate rows are removed from the dataset from the bottom, up.


Examples

Example #1

Find and remove duplicate rows based on the values in all columns.

Before (source table)

River Length (km) Continent
Nile 6650 Africa
Amazon 6400 South America
Nile 6650 Africa

After (result table)

River Length (km) Continent
Nile 6650 Africa
Amazon 6400 South America

Action parameters

Apply to: All columns


Example #2

Remove duplicate rows based on First and Last Names.

Before (source table)

FirstName LastName MiddleInit
Leah Boswell A
José Silva D
Gino Basso E
Leah Boswell F
Gino Basso R
Santa Alegio G

After (result table)

FirstName LastName MiddleInit
Leah Boswell A
José Silva D
Gino Basso E
Santa Alegio G

Action parameters

Apply to: Selected columns (FirstName, LastName)


Community examples


See also

transformations/deduplicate.txt · Last modified: 2025/02/02 22:16 by craigt

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki