Pandas is a fundamental library in Python for data manipulation and analysis, which provides rich functionalities and methods that enable data professionals to process and analyze large datasets. There are some advanced functions in Pandas that you should know for Data Analysis. If you want to know about such functions, this article is for you. In this article, I’ll take you through some advanced Pandas functions for Data Analysis and how to use them.
Advanced Pandas Functions for Data Analysis
Below are some advanced Pandas functions for Data Analysis, explained with examples, that will help you understand how they work and when to use them.
GroupBy
GroupBy allows you to group together rows that share the same value in one or more columns and then perform an operation on each group. It can be anything from aggregation (e.g., sum, mean) to transformation (e.g., standardizing data within a group) or filtration (e.g., removing data that does not meet a certain condition).
Here’s an example of using GroupBy in Pandas:
import pandas as pd
# sample dataframe
df = pd.DataFrame({
'A': ['Delhi', 'Mumbai', 'Delhi', 'Mumbai'],
'B': [1, 2, 3, 4],
'C': [2.0, 5., 8., 1.]
})
# grouping by column 'A' and summing up the other columns
grouped = df.groupby('A').sum()
print(grouped)B C
A
Delhi 4 10.0
Mumbai 6 6.0
Use GroupBy for segmenting the dataset into groups and applying calculations for each group separately, such as summarizing sales data by region or calculating average scores by student groups.
Pivot Tables
A pivot table is a data summarization tool frequently used in data processing. Pandas pivot_table can automatically sort, count, and total the data stored in one DataFrame or Series and is particularly useful for quickly summarizing data and highlighting important aspects.
Here’s an example of using pivot tables in Pandas:
import numpy as np pivot = pd.pivot_table(df, values='C', index=['A'], aggfunc=np.sum) print(pivot)
C
A
Delhi 10.0
Mumbai 6.0
Use pivot_table when you need to create a spreadsheet-style pivot table as a DataFrame. It’s particularly useful for summarizing and analyzing data to see comparisons, patterns, and trends in data.
Multi-Indexing
Pandas support multi-level indexing, or hierarchical indexing, which allows you to store and manipulate data with an arbitrary number of dimensions in lower dimensional data structures like Series (1D) and DataFrame (2D).
Here’s an example of using multi-indexing in Pandas:
# creating a dataframe with multi-index
index = pd.MultiIndex.from_tuples([('Delhi', 'one'), ('Delhi', 'two'),
('Mumbai', 'one'), ('Mumbai', 'two')],
names=['first', 'second'])
df_multi = pd.DataFrame(np.random.randn(4, 2), index=index, columns=['A', 'B'])
print(df_multi)A B
first second
Delhi one -0.471666 1.164131
two 0.194125 -0.470828
Mumbai one -0.487325 -1.401571
two 0.250847 -0.573393
Use multi-indexing when you need to work with data that can be categorized by two or more keys per data point. It is especially useful for grouping data into categories and subcategories.
Merging, Joining, and Concatenating
These functions are used to combine different DataFrames together. Merge is for combining data on columns or indices, Join is for combining data on a key column or an index, and Concat is for appending dataframes one below the other or side by side.
Use merge when you need database-style joining of columns, use join for combining data on a key index or column, and use concat for stacking DataFrames vertically or horizontally.
You can learn about merging, joining, and concatenation with implementation from here.
Data Transformation with apply() and map()
apply() allows you to apply a function along an axis of the DataFrame or on a Series. map() is a Series method used to substitute each value in a Series with another value.
Below are examples of using these functions in Pandas:
# using apply()
df['B'] = df['B'].apply(lambda x: x * 2)
# using map() for a Series
df['A'] = df['A'].map({'Delhi': 'New Delhi', 'Mumbai': 'Bombay'})Use apply() for applying a function across rows/columns of a DataFrame or on a Series, and use map() for element-wise transformations and substitutions in a Series.
Query Function
The query() function in Pandas allows you to filter a DataFrame using a query expression passed as a string. This method offers a more readable syntax for filtering data compared to traditional boolean indexing, especially for complex conditions. It can be particularly useful when working with large DataFrames and performing dynamic data filtering operations.
Here’s an example of using the query function in Pandas:
import pandas as pd
# sample dataframe
df = pd.DataFrame({
'A': range(1, 6),
'B': range(10, 60, 10),
'C': ['Delhi', 'Mumbai', 'Delhi', 'Mumbai', 'Delhi']
})
# using query() to filter rows
filtered_df = df.query('(A > 2) & (C == "Delhi")')
print(filtered_df)A B C
2 3 30 Delhi
4 5 50 Delhi
Use the query() function when you need to perform filtering operations on a DataFrame and prefer a concise, readable syntax. It’s especially useful in situations where the filter conditions are complex or when you’re dynamically constructing query strings based on user input or program logic.
Time Series Functionality
Pandas provide powerful time series functionality to manipulate and analyze time series data. It includes capabilities for date range generation, frequency conversion, moving window statistics, date shifting, and lagging. These functions are essential for working with financial, economic, environmental, and many other types of data that are indexed by time.
Use Pandas time series functionality when you are working with data indexed by time, and you need to perform operations such as:
- Resampling for different time frequencies (e.g., converting minute data into 5-minute data).
- Filling missing dates and times in a time series.
- Shifting or lagging values for time series analysis.
- Calculating rolling or moving window statistics (e.g., moving average).
- Working with time zones.
You can learn using time series functions in Pandas from here.
Summary
So these were some advanced Pandas functions for Data Analysis you should know. Practice and exploration of these functions with real datasets are essential for mastering Pandas for data analysis. You can find many datasets to use these functions here.
I hope you liked this article on advanced Pandas functions for Data Analysis. Feel free to ask valuable questions in the comments section below. You can follow me on Instagram for many more resources.
