How to Remove Duplicates from an Array

Duplicate values in an array can cause bugs, inflate memory usage, and produce misleading results in data processing. Whether you’re working with user input, API responses, or internal data structures, cleaning up an array is a common task for JavaScript developers.

Why Remove Duplicates?

Removing duplicates improves:

• Performance: Smaller arrays mean faster iteration and lower memory consumption.
• Data Integrity: Guarantees that each item is unique, which is essential for sets, IDs, or option lists.
• Readability: Cleaner data makes debugging and maintenance easier.

Popular Techniques for De‑duplicating Arrays

1. Using Set (ES6)

The Set object stores unique values of any type. Converting a Set back to an array gives you a deduplicated list in just one line.

// Original array with duplicates
const numbers = [1, 2, 3, 2, 4, 1, 5];

// Remove duplicates using Set
const uniqueNumbers = [...new Set(numbers)];

console.log(uniqueNumbers); // [1, 2, 3, 4, 5]

2. Array.filter() + indexOf()

This classic method keeps the first occurrence of each element and filters out the rest.

const fruits = ['apple', 'banana', 'apple', 'orange', 'banana'];

const uniqueFruits = fruits.filter((item, index) => {
    return fruits.indexOf(item) === index;
});

console.log(uniqueFruits); // ['apple', 'banana', 'orange']

3. Array.reduce() with an Accumulator

Using reduce gives you full control over how duplicates are identified, which is handy for complex objects.

const objects = [
    { id: 1, name: 'John' },
    { id: 2, name: 'Jane' },
    { id: 1, name: 'John' }
];

const uniqueObjects = objects.reduce((acc, cur) => {
    if (!acc.some(item => item.id === cur.id)) {
        acc.push(cur);
    }
    return acc;
}, []);

console.log(uniqueObjects);
// [{ id: 1, name: 'John' }, { id: 2, name: 'Jane' }]

4. Traditional for Loop with a Helper Object

For environments that don’t support ES6, a simple loop combined with an object lookup works well.

var items = ['a', 'b', 'a', 'c', 'b'];
var seen = {};
var result = [];

for (var i = 0; i < items.length; i++) {
    if (!seen[items[i]]) {
        seen[items[i]] = true;
        result.push(items[i]);
    }
}

console.log(result); // ['a', 'b', 'c']

5. Using lodash or underscore Libraries

If you already use a utility library, _.uniq (lodash) or _.unique (underscore) provide a concise solution.

// Using lodash
const _ = require('lodash');

const mixed = [1, 2, 2, 3, 4, 4, 5];
const unique = _.uniq(mixed);

console.log(unique); // [1, 2, 3, 4, 5]

Choosing the Right Method

Method	Best For	Complexity	Browser Support
Set	Simple primitive arrays	O(n)	IE 11+ (with polyfill)
filter + indexOf	Small to medium arrays, readability	O(n²) worst‑case	All browsers
reduce	Arrays of objects or custom equality	O(n)	All browsers
Traditional for loop	Legacy environments	O(n)	All browsers
lodash _.uniq	Projects already using lodash	O(n)	All browsers

Performance Tips

• Avoid nested loops when possible; they quickly become O(n²).
• When dealing with large data sets, Set or a for loop with a hash map is usually fastest.
• For arrays of objects, consider creating a Map keyed by a unique property (e.g., id) to achieve O(n) deduplication.

Full Example: A Reusable Utility Function

Below is a small utility that automatically selects the most efficient strategy based on the input type.

/**
 * Remove duplicates from an array.
 * Supports primitives and objects (by a key).
 *
 * @param {Array} arr - The array to deduplicate.
 * @param {String} [key] - Optional property name for object comparison.
 * @returns {Array} A new array with unique values.
 */
function dedupe(arr, key) {
    if (!Array.isArray(arr)) throw new TypeError('Expected an array');

    // Primitive values – use Set (fastest)
    if (!key) {
        return [...new Set(arr)];
    }

    // Objects – use a Map keyed by the provided property
    const map = new Map();
    for (const item of arr) {
        if (item && typeof item === 'object' && key in item) {
            map.set(item[key], item);
        }
    }
    return Array.from(map.values());
}

// Example usage
const users = [
    { id: 1, name: 'Alice' },
    { id: 2, name: 'Bob' },
    { id: 1, name: 'Alice' }
];

console.log(dedupe(users, 'id'));
// [{ id: 1, name: 'Alice' }, { id: 2, name: 'Bob' }]

Conclusion

Removing duplicates from an array is a fundamental skill for any JavaScript developer. Whether you prefer the elegance of Set, the flexibility of reduce, or the familiarity of a classic for loop, the right technique depends on your data type, project requirements, and browser support.

Apply the examples above, test performance on your actual data set, and you’ll keep your arrays clean, efficient, and ready for the next challenge.