How to Filter List in Python the Right Way to Get More Out of Your Data

In Python, one of the most popular ways to store sequential or ordered data is through the use of lists. A list in Python is an ordered, mutable, built-in data structure that is used to store a collection of data. Items stored in a list are indexed starting from zero, and users can change the content of a list after it has been created.

The fact that items in a list are indexed allows the storage of duplicate items in a list. Lists in Python can also contain elements of different data types. Items in a list are separated using commas and enclosed within square brackets.

Here are examples of lists in Python:

# A list containing one data type - String
colors = ['Red', 'Orange', 'Yellow', 'Green', 'Blue', 'Indigo', 'Violet']

# A list containing multiple data types
movies = ['Transformers', 2012, 'Avengers', 300]

# A list with duplicate values
users = ['John', 'Mary', 'Elizabeth', 'John']

Lists are a very versatile data structure, and you can perform lots of operations on the data stored in lists. One common and useful operation performed on lists is filtering the data stored in the list.

Why Filtering Lists is Important

Filtering a list means extracting specific subsets of data that meet a given criteria. For instance, we might only be interested in even numbers in a list containing the numbers 1 through 10. To extract such data from the list, all we need to do is filter the list to get numbers that are exactly divisible by two.

Filtering is particularly useful in data manipulation and data analysis as it allows removing unwanted items from lists, creating new lists containing items that meet given criteria, and extracting data that meets certain conditions.

Some real-world applications of filtering lists include:

• Data validation – When working with a list, you might want to only work with data that meets a given criteria. For instance, in a list of users, you might be interested in users with both a first and last name. Filtering can allow you to verify and validate the data in the list before processing it. This way, you avoid working with data that don’t meet your requirements.
• Data Cleaning – Since lists can store a variety of data types, it is not rare to find that lists contain unwanted data that you’re not interested in. For instance, in a list containing names, you might not be interested in numerical values in the list. Through filtering, you can remove the unwanted data.
• Data analysis – Filtering data is particularly useful in data analysis as it allows you to focus on specific subsets of data, which allows you to easily identify patterns and trends. For instance, in a list of students, you can filter out male and female students and use the result in data analysis. You can even filter according to age, location, and grade. This allows you to draw more useful conclusions from specific subsets of data in a list

Filtering lists is a very useful function as it gives you control over the data stored in a list allowing you to only work with the data that you’re interested in.

Targeting specific subsets of data stored in a list is very useful as it streamlines data processing and data analysis making the processes better, faster, and more accurate.

Using filter() function

The filter() function is a built-in Python function that can be used to iterate through an iterable such as a list, tuples, set, or dictionary and extract items in the iterable that satisfy a given condition.

An Iterable in Python is an object which can be looped through its elements one by one. Iterating through an iterable returns the items in the iterable one at a time. 

The syntax for the filter function is as follows:

filter(function, iterable)

function – a Python function that contains the filtering condition

iterable – the iterable that is going to be filtered. In this case, we’ll be using a list.

The filter() function takes the passed function and applies it to each item in the passed iterable, testing the filtering condition against the item. If the item satisfies the condition, that is, it returns the boolean true, which will be selected by the filter() function. In case it does not satisfy the condition, the item is not selected.

The filter() function returns an iterable with items that passed the filtering condition. You can create a new list containing items that passed the filtering condition by using the list() function.

To see the filter() function in action, consider the list below, which is being filtered to select numbers that are less than 50:

#A list of numbers
numbers = [79, 15, 92, 53, 46, 24, 81, 77, 37, 61]

# function containing the filtering condition
def is_even(num):
   if num < 50:
      return True
   else:
      return False

# the function list() is used to create a list from the iterable
# returned by the filter() function
filtered_numbers = list(filter(is_even, numbers))
print(filtered_numbers)

The above code prints the following result:

[15, 46, 24, 37]

Using a for loop

In Python, a for loop is a control flow statement used to iterate over sequential data structures such as a list, tuples, strings,/ and arrays. A for loop repeatedly executes a block of code for each item in a sequence.

The general syntax for a for loop is as follows:

for item in iterable:
    # Block of code to be executed for each item in the iterabele

item – a variable representing the current item being processed in an iteration of the loop

iterable – the sequence over which the for loop iterates. In this case, a list

To filter a list using a for loop, we’ll need to pass in our filtering condition in the section for the block of code to be executed. This way, each item will be evaluated to see whether it satisfies a given condition.

When using a for loop to filter a list, you also need to create an empty list where you’ll append values that pass your filtering condition.

To see this in action, let us filter a list of numbers to get numbers less than 50 using a for loop:

numbers = [79, 15, 92, 53, 46, 24, 81, 77, 37, 61]

filtered_numbers = []

for num in numbers:
    if num < 50:
        # append() used to add a number that passes the condition
        # into filtered_numbers.
        filtered_numbers.append(num)

print(filtered_numbers)

The above code prints the following result:

[15, 46, 24, 37]

Using another list

You can filter a list using another by checking whether an item in a list you want to filter appears in another list. For instance, consider the following two lists

letters = ['a', 'h', 'q', 'd', 's', 'x', 'g', 'j', 'e', 'o', 'k', 'f', 'c', 'b', 'n']

vowels = ['a', 'e', 'i', 'o', 'u']

In the list called letters above, we can determine which items in the list are not vowels by checking whether the item appears in the vowels list or not. If an item does not appear in the vowels list, then it is not a vowel.

This way, we can get all characters in letters that are not vowels. To do this, execute the following code:

letters = ['a', 'h', 'q', 'd', 's', 'x', 'g', 'j', 'e', 'o', 'k', 'f', 'c', 'b', 'n']
vowels = ['a', 'e', 'i', 'o', 'u']

not_vowel = []

for letter in letters:
    if letter not in vowels:
        not_vowel.append(letter)
print(not_vowel)

The above code prints the following output, which contains characters in letters that are not vowels.

['h', 'q', 'd', 's', 'x', 'g', 'j', 'k', 'f', 'c', 'b', 'n']

Using List Comprehension

In Python, list comprehension offers a shorter, more concise, and cleaner syntax for creating a new list from an existing list. List comprehension has the ability to reduce the multiple lines of code used to create a new list from another list using a for loop into a single line.

This is because when using list comprehension, you define and add items to the new list all in one line.

The syntax for list comprehension is as follows:

new_list = [expression for element in iterable if condition]

new_list – a new list containing elements added by the list comprehension

expression – an operation that will be applied  to each item of the iterable

item – variable name representing the currently active item in the iterable

iterable – an iterable from which items will be selected from.

if condition – an optional part where a condition can be added to filter items to only add those that satisfy a given condition into the new list that is being created.

To see list comprehension in action and how much it can simplify the process of filtering and creating a new list, we’ll use the letters and vowels list to filter out items in the letters list that are not in the vowels list. To do this, execute the following code:

letters = ['a', 'h', 'q', 'd', 's', 'x', 'g', 'j', 'e', 'o', 'k', 'f', 'c', 'b', 'n']
vowels = ['a', 'e', 'i', 'o', 'u']

# list comprehension
not_vowel = [letter for letter in letters if letter not in vowels]
print(not_vowel)

The output of the code above is shown below:

['h', 'q', 'd', 's', 'x', 'g', 'j', 'k', 'f', 'c', 'b', 'n']

The output is similar to the previous example, which used a for loop to filter a list using another list. However, it uses fewer lines of code to achieve the same result.

Conclusion

When working with data in Python, a common operation that you are bound to perform is filtering data from an iterable such as a list. This is a very useful step in data analysis and data processing as it allows you to only work with data that is relevant to the operation or processing being done.

A very popular sequential data structure in Python is the list. In the event that you find yourself working with a list and you need to filter it, consider using any of the methods highlighted in the article.

You may also read about how to use lambda functions in Python with examples.