library(dplyr)
library(palmerpenguins)
penguins <- palmerpenguins::penguinsTidy Data Manipulation: dplyr vs polars
R
Python
Manipulation
A comparison of R’s dplyr and Python’s polars data manipulation packages
There are a myriad of options to perform essential data manipulation tasks in R and Python (see, for instance, my other posts on dplyr vs ibis and dplyr vs pandas). However, if we want to do tidy data science in R, there is a clear forerunner: dplyr. In the world of Python, polars is a relatively new kid on the block that shares a lot of semantic with dplyr. In this blog post, I illustrate their syntactic similarities and highlight differences between these two packages that emerge for a few key tasks.
Before we dive into the comparison, a short introduction to the packages: the dplyr package in R allows users to refer to columns without quotation marks due to its implementation of non-standard evaluation (NSE). NSE is a programming technique used in R that allows functions to capture the expressions passed to them as arguments, rather than just the values of those arguments. The primary goal of NSE in the context of dplyr is to create a more user-friendly and intuitive syntax. This makes data manipulation tasks more straightforward and aligns with the general philosophy of the tidyverse to make data science faster, easier, and more fun.1
polars is also designed for data manipulation and heavily optimized for performance, but there are significant differences in their approach, especially in how they handle column referencing and expression evaluation. Python generally relies on standard evaluation, meaning expressions are evaluated to their values before being passed to a function. In polars, column references typically need to be explicitly stated, often using quoted names or through methods attached to data frame objects.
Loading packages and data
We start by loading the main packages of interest and the popular palmerpenguins package that exists for both R and Python. We then use the penguins data frame as the data to compare all functions and methods below. Note that we also limit the print output of polars data frames to 10 rows to prevent this post being flooded by excessively long tables.
import polars as pl
from palmerpenguins import load_penguins
pl.Config(tbl_rows = 10)<polars.config.Config object at 0x168797f70>
penguins = load_penguins().pipe(pl.from_pandas)Work with rows
Filter rows
Filtering rows works very similarly for both packages, they even have the same function names: dplyr::filter() and polars.filter(). To select columns in polars, you need the polars.col() selector.
penguins |>
filter(species == "Adelie" &
island %in% c("Biscoe", "Dream"))# A tibble: 100 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Biscoe 37.8 18.3 174 3400
2 Adelie Biscoe 37.7 18.7 180 3600
3 Adelie Biscoe 35.9 19.2 189 3800
4 Adelie Biscoe 38.2 18.1 185 3950
5 Adelie Biscoe 38.8 17.2 180 3800
6 Adelie Biscoe 35.3 18.9 187 3800
7 Adelie Biscoe 40.6 18.6 183 3550
8 Adelie Biscoe 40.5 17.9 187 3200
9 Adelie Biscoe 37.9 18.6 172 3150
10 Adelie Biscoe 40.5 18.9 180 3950
# ℹ 90 more rows
# ℹ 2 more variables: sex <fct>, year <int>(penguins
.filter(
(pl.col("species") == "Adelie") &
(pl.col("island").is_in(["Biscoe", "Dream"])))
)
shape: (100, 8)
| species | island | bill_length_mm | bill_depth_mm | flipper_length_mm | body_mass_g | sex | year |
|---|---|---|---|---|---|---|---|
| str | str | f64 | f64 | f64 | f64 | str | i64 |
| "Adelie" | "Biscoe" | 37.8 | 18.3 | 174.0 | 3400.0 | "female" | 2007 |
| "Adelie" | "Biscoe" | 37.7 | 18.7 | 180.0 | 3600.0 | "male" | 2007 |
| "Adelie" | "Biscoe" | 35.9 | 19.2 | 189.0 | 3800.0 | "female" | 2007 |
| "Adelie" | "Biscoe" | 38.2 | 18.1 | 185.0 | 3950.0 | "male" | 2007 |
| "Adelie" | "Biscoe" | 38.8 | 17.2 | 180.0 | 3800.0 | "male" | 2007 |
| … | … | … | … | … | … | … | … |
| "Adelie" | "Dream" | 36.6 | 18.4 | 184.0 | 3475.0 | "female" | 2009 |
| "Adelie" | "Dream" | 36.0 | 17.8 | 195.0 | 3450.0 | "female" | 2009 |
| "Adelie" | "Dream" | 37.8 | 18.1 | 193.0 | 3750.0 | "male" | 2009 |
| "Adelie" | "Dream" | 36.0 | 17.1 | 187.0 | 3700.0 | "female" | 2009 |
| "Adelie" | "Dream" | 41.5 | 18.5 | 201.0 | 4000.0 | "male" | 2009 |
Slice rows
dplyr::slice() takes integers with row numbers as inputs, so you can use ranges and arbitrary vectors of integers. polars.slice() only takes the start index and the length of the slice as inputs. For instance, to the the same result of slicing rows 10 to 20, the code looks as follows (note that indexing starts at 0 in Python, while it starts at 1 in R):
penguins |>
slice(10:20)# A tibble: 11 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 42 20.2 190 4250
2 Adelie Torgersen 37.8 17.1 186 3300
3 Adelie Torgersen 37.8 17.3 180 3700
4 Adelie Torgersen 41.1 17.6 182 3200
5 Adelie Torgersen 38.6 21.2 191 3800
6 Adelie Torgersen 34.6 21.1 198 4400
7 Adelie Torgersen 36.6 17.8 185 3700
8 Adelie Torgersen 38.7 19 195 3450
9 Adelie Torgersen 42.5 20.7 197 4500
10 Adelie Torgersen 34.4 18.4 184 3325
11 Adelie Torgersen 46 21.5 194 4200
# ℹ 2 more variables: sex <fct>, year <int>(penguins
.slice(9, 11)
)
shape: (11, 8)
| species | island | bill_length_mm | bill_depth_mm | flipper_length_mm | body_mass_g | sex | year |
|---|---|---|---|---|---|---|---|
| str | str | f64 | f64 | f64 | f64 | str | i64 |
| "Adelie" | "Torgersen" | 42.0 | 20.2 | 190.0 | 4250.0 | null | 2007 |
| "Adelie" | "Torgersen" | 37.8 | 17.1 | 186.0 | 3300.0 | null | 2007 |
| "Adelie" | "Torgersen" | 37.8 | 17.3 | 180.0 | 3700.0 | null | 2007 |
| "Adelie" | "Torgersen" | 41.1 | 17.6 | 182.0 | 3200.0 | "female" | 2007 |
| "Adelie" | "Torgersen" | 38.6 | 21.2 | 191.0 | 3800.0 | "male" | 2007 |
| … | … | … | … | … | … | … | … |
| "Adelie" | "Torgersen" | 36.6 | 17.8 | 185.0 | 3700.0 | "female" | 2007 |
| "Adelie" | "Torgersen" | 38.7 | 19.0 | 195.0 | 3450.0 | "female" | 2007 |
| "Adelie" | "Torgersen" | 42.5 | 20.7 | 197.0 | 4500.0 | "male" | 2007 |
| "Adelie" | "Torgersen" | 34.4 | 18.4 | 184.0 | 3325.0 | "female" | 2007 |
| "Adelie" | "Torgersen" | 46.0 | 21.5 | 194.0 | 4200.0 | "male" | 2007 |
Arrange rows
To orders the rows of a data frame by the values of selected columns, we have dplyr::arrange() and polars.sort(). Note that dplyr::arrange() arranges rows in an an ascending order and puts NA values last. polars.sort(), on the other hand, arranges rows in an ascending order and starts with null as default. Note that there are options to control these defaults.
penguins |>
arrange(island, desc(bill_length_mm))# A tibble: 344 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Gentoo Biscoe 59.6 17 230 6050
2 Gentoo Biscoe 55.9 17 228 5600
3 Gentoo Biscoe 55.1 16 230 5850
4 Gentoo Biscoe 54.3 15.7 231 5650
5 Gentoo Biscoe 53.4 15.8 219 5500
6 Gentoo Biscoe 52.5 15.6 221 5450
7 Gentoo Biscoe 52.2 17.1 228 5400
8 Gentoo Biscoe 52.1 17 230 5550
9 Gentoo Biscoe 51.5 16.3 230 5500
10 Gentoo Biscoe 51.3 14.2 218 5300
# ℹ 334 more rows
# ℹ 2 more variables: sex <fct>, year <int>(penguins
.sort(["island", "bill_length_mm"],
descending=[False, True], nulls_last=True)
)
shape: (344, 8)
| species | island | bill_length_mm | bill_depth_mm | flipper_length_mm | body_mass_g | sex | year |
|---|---|---|---|---|---|---|---|
| str | str | f64 | f64 | f64 | f64 | str | i64 |
| "Gentoo" | "Biscoe" | 59.6 | 17.0 | 230.0 | 6050.0 | "male" | 2007 |
| "Gentoo" | "Biscoe" | 55.9 | 17.0 | 228.0 | 5600.0 | "male" | 2009 |
| "Gentoo" | "Biscoe" | 55.1 | 16.0 | 230.0 | 5850.0 | "male" | 2009 |
| "Gentoo" | "Biscoe" | 54.3 | 15.7 | 231.0 | 5650.0 | "male" | 2008 |
| "Gentoo" | "Biscoe" | 53.4 | 15.8 | 219.0 | 5500.0 | "male" | 2009 |
| … | … | … | … | … | … | … | … |
| "Adelie" | "Torgersen" | 34.6 | 17.2 | 189.0 | 3200.0 | "female" | 2008 |
| "Adelie" | "Torgersen" | 34.4 | 18.4 | 184.0 | 3325.0 | "female" | 2007 |
| "Adelie" | "Torgersen" | 34.1 | 18.1 | 193.0 | 3475.0 | null | 2007 |
| "Adelie" | "Torgersen" | 33.5 | 19.0 | 190.0 | 3600.0 | "female" | 2008 |
| "Adelie" | "Torgersen" | null | null | null | null | null | 2007 |
Work with columns
Select columns
Selecting a subset of columns works essentially the same for both and dplyr::select() and polars.select() even have the same name. Note that you don’t have to use polars.col() but can just pass strings in the polars.select() method.
penguins |>
select(bill_length_mm, sex)# A tibble: 344 × 2
bill_length_mm sex
<dbl> <fct>
1 39.1 male
2 39.5 female
3 40.3 female
4 NA <NA>
5 36.7 female
6 39.3 male
7 38.9 female
8 39.2 male
9 34.1 <NA>
10 42 <NA>
# ℹ 334 more rows(penguins
.select(pl.col("bill_length_mm"), pl.col("sex"))
)
shape: (344, 2)
| bill_length_mm | sex |
|---|---|
| f64 | str |
| 39.1 | "male" |
| 39.5 | "female" |
| 40.3 | "female" |
| null | null |
| 36.7 | "female" |
| … | … |
| 55.8 | "male" |
| 43.5 | "female" |
| 49.6 | "male" |
| 50.8 | "male" |
| 50.2 | "female" |
Rename columns
Renaming columns also works very similarly with the major difference that polars.rename() takes a dictionary with mappings of old to new names as input, while dplyr::rename() takes variable names via the usual NSE.
penguins |>
rename(bill_length = bill_length_mm,
bill_depth = bill_depth_mm)# A tibble: 344 × 8
species island bill_length bill_depth flipper_length_mm body_mass_g sex
<fct> <fct> <dbl> <dbl> <int> <int> <fct>
1 Adelie Torgersen 39.1 18.7 181 3750 male
2 Adelie Torgersen 39.5 17.4 186 3800 female
3 Adelie Torgersen 40.3 18 195 3250 female
4 Adelie Torgersen NA NA NA NA <NA>
5 Adelie Torgersen 36.7 19.3 193 3450 female
6 Adelie Torgersen 39.3 20.6 190 3650 male
7 Adelie Torgersen 38.9 17.8 181 3625 female
8 Adelie Torgersen 39.2 19.6 195 4675 male
9 Adelie Torgersen 34.1 18.1 193 3475 <NA>
10 Adelie Torgersen 42 20.2 190 4250 <NA>
# ℹ 334 more rows
# ℹ 1 more variable: year <int>(penguins
.rename({"bill_length_mm": "bill_length",
"bill_depth_mm" : "bill_depth"})
)
shape: (344, 8)
| species | island | bill_length | bill_depth | flipper_length_mm | body_mass_g | sex | year |
|---|---|---|---|---|---|---|---|
| str | str | f64 | f64 | f64 | f64 | str | i64 |
| "Adelie" | "Torgersen" | 39.1 | 18.7 | 181.0 | 3750.0 | "male" | 2007 |
| "Adelie" | "Torgersen" | 39.5 | 17.4 | 186.0 | 3800.0 | "female" | 2007 |
| "Adelie" | "Torgersen" | 40.3 | 18.0 | 195.0 | 3250.0 | "female" | 2007 |
| "Adelie" | "Torgersen" | null | null | null | null | null | 2007 |
| "Adelie" | "Torgersen" | 36.7 | 19.3 | 193.0 | 3450.0 | "female" | 2007 |
| … | … | … | … | … | … | … | … |
| "Chinstrap" | "Dream" | 55.8 | 19.8 | 207.0 | 4000.0 | "male" | 2009 |
| "Chinstrap" | "Dream" | 43.5 | 18.1 | 202.0 | 3400.0 | "female" | 2009 |
| "Chinstrap" | "Dream" | 49.6 | 18.2 | 193.0 | 3775.0 | "male" | 2009 |
| "Chinstrap" | "Dream" | 50.8 | 19.0 | 210.0 | 4100.0 | "male" | 2009 |
| "Chinstrap" | "Dream" | 50.2 | 18.7 | 198.0 | 3775.0 | "female" | 2009 |
Mutate columns
Transforming existing columns or creating new ones is an essential part of data analysis. dplyr::mutate() and polars.with_columns() are the work horses for these tasks. Both approaches have a very similar syntax. Note that you have to split up variable assignments if you want to refer to a newly created variable in polars, while you can refer to the new variables in the same mutate block in dplyr.
penguins |>
mutate(ones = 1,
bill_length = bill_length_mm / 10,
bill_length_squared = bill_length^2) |>
select(ones, bill_length_mm, bill_length, bill_length_squared)# A tibble: 344 × 4
ones bill_length_mm bill_length bill_length_squared
<dbl> <dbl> <dbl> <dbl>
1 1 39.1 3.91 15.3
2 1 39.5 3.95 15.6
3 1 40.3 4.03 16.2
4 1 NA NA NA
5 1 36.7 3.67 13.5
6 1 39.3 3.93 15.4
7 1 38.9 3.89 15.1
8 1 39.2 3.92 15.4
9 1 34.1 3.41 11.6
10 1 42 4.2 17.6
# ℹ 334 more rows(penguins
.with_columns(ones = pl.lit(1),
bill_length = pl.col("bill_length_mm") / 10)
.with_columns(bill_length_squared = pl.col("bill_length") ** 2)
.select(pl.col("ones"), pl.col("bill_length_mm"),
pl.col("bill_length"), pl.col("bill_length_squared"))
)
shape: (344, 4)
| ones | bill_length_mm | bill_length | bill_length_squared |
|---|---|---|---|
| i32 | f64 | f64 | f64 |
| 1 | 39.1 | 3.91 | 15.2881 |
| 1 | 39.5 | 3.95 | 15.6025 |
| 1 | 40.3 | 4.03 | 16.2409 |
| 1 | null | null | null |
| 1 | 36.7 | 3.67 | 13.4689 |
| … | … | … | … |
| 1 | 55.8 | 5.58 | 31.1364 |
| 1 | 43.5 | 4.35 | 18.9225 |
| 1 | 49.6 | 4.96 | 24.6016 |
| 1 | 50.8 | 5.08 | 25.8064 |
| 1 | 50.2 | 5.02 | 25.2004 |
Relocate columns
dplyr::relocate() provides options to change the positions of columns in a data frame, using the same syntax as dplyr::select(). In addition, there are the options .after and .before to provide users with additional shortcuts.
The recommended way to relocate columns in polars is to use the polars.select() method, but there are no options as in dplyr::relocate(). In fact, the safest way to consistently get the correct order of columns is to explicitly specify them.
penguins |>
relocate(c(species, bill_length_mm), .before = sex)# A tibble: 344 × 8
island bill_depth_mm flipper_length_mm body_mass_g species bill_length_mm
<fct> <dbl> <int> <int> <fct> <dbl>
1 Torgersen 18.7 181 3750 Adelie 39.1
2 Torgersen 17.4 186 3800 Adelie 39.5
3 Torgersen 18 195 3250 Adelie 40.3
4 Torgersen NA NA NA Adelie NA
5 Torgersen 19.3 193 3450 Adelie 36.7
6 Torgersen 20.6 190 3650 Adelie 39.3
7 Torgersen 17.8 181 3625 Adelie 38.9
8 Torgersen 19.6 195 4675 Adelie 39.2
9 Torgersen 18.1 193 3475 Adelie 34.1
10 Torgersen 20.2 190 4250 Adelie 42
# ℹ 334 more rows
# ℹ 2 more variables: sex <fct>, year <int>(penguins
.select(pl.col("island"), pl.col("bill_depth_mm"),
pl.col("flipper_length_mm"), pl.col("body_mass_g"),
pl.col("species"), pl.col("bill_length_mm"), pl.col("sex"))
)
shape: (344, 7)
| island | bill_depth_mm | flipper_length_mm | body_mass_g | species | bill_length_mm | sex |
|---|---|---|---|---|---|---|
| str | f64 | f64 | f64 | str | f64 | str |
| "Torgersen" | 18.7 | 181.0 | 3750.0 | "Adelie" | 39.1 | "male" |
| "Torgersen" | 17.4 | 186.0 | 3800.0 | "Adelie" | 39.5 | "female" |
| "Torgersen" | 18.0 | 195.0 | 3250.0 | "Adelie" | 40.3 | "female" |
| "Torgersen" | null | null | null | "Adelie" | null | null |
| "Torgersen" | 19.3 | 193.0 | 3450.0 | "Adelie" | 36.7 | "female" |
| … | … | … | … | … | … | … |
| "Dream" | 19.8 | 207.0 | 4000.0 | "Chinstrap" | 55.8 | "male" |
| "Dream" | 18.1 | 202.0 | 3400.0 | "Chinstrap" | 43.5 | "female" |
| "Dream" | 18.2 | 193.0 | 3775.0 | "Chinstrap" | 49.6 | "male" |
| "Dream" | 19.0 | 210.0 | 4100.0 | "Chinstrap" | 50.8 | "male" |
| "Dream" | 18.7 | 198.0 | 3775.0 | "Chinstrap" | 50.2 | "female" |
Work with groups of rows
Simple summaries by group
Let’s suppose we want to compute summaries by groups such as means or medians. Both packages are very similar again: on the R side you have dplyr::group_by() and dplyr::summarize(), while on the Python side you have polars.group_by() and polars.agg().
Note that dplyr::group_by() also automatically arranges the results by the group, so the reproduce the results of dplyr, we need to add polars.sort() to the chain.
penguins |>
group_by(island) |>
summarize(bill_depth_mean = mean(bill_depth_mm, na.rm = TRUE))# A tibble: 3 × 2
island bill_depth_mean
<fct> <dbl>
1 Biscoe 15.9
2 Dream 18.3
3 Torgersen 18.4(penguins
.group_by("island")
.agg(bill_depth_mean = pl.mean("bill_depth_mm"))
.sort("island")
)
shape: (3, 2)
| island | bill_depth_mean |
|---|---|
| str | f64 |
| "Biscoe" | 15.87485 |
| "Dream" | 18.344355 |
| "Torgersen" | 18.429412 |
More complicated summaries by group
Typically, you want to create multiple different summaries by groups. dplyr provides a lot of flexibility to create new variables on the fly, while polars seems to be a bit more restrictive. For instance, to compute the share of female penguins by group, it makes more sense to create an ìs_female indicator column using polars because polars.mean() does not accept expressions as inputs.
penguins |>
group_by(island) |>
summarize(count = n(),
bill_depth_mean = mean(bill_depth_mm, na.rm = TRUE),
flipper_length_median = median(flipper_length_mm, na.rm = TRUE),
body_mass_sd = sd(body_mass_g, na.rm = TRUE),
share_female = mean(sex == "female", na.rm = TRUE))# A tibble: 3 × 6
island count bill_depth_mean flipper_length_median body_mass_sd share_female
<fct> <int> <dbl> <dbl> <dbl> <dbl>
1 Biscoe 168 15.9 214 783. 0.491
2 Dream 124 18.3 193 417. 0.496
3 Torgers… 52 18.4 191 445. 0.511(penguins
.with_columns(is_female = pl.when(pl.col("sex") == "female").then(1)
.when(pl.col("sex").is_null()).then(None)
.otherwise(0))
.group_by("island")
.agg(
count = pl.count(),
bill_depth_mean = pl.mean("bill_depth_mm"),
flipper_length_median = pl.median("flipper_length_mm"),
body_mass_sd = pl.std("body_mass_g"),
share_female = pl.mean("is_female")
)
.sort("island")
)
shape: (3, 6)
| island | count | bill_depth_mean | flipper_length_median | body_mass_sd | share_female |
|---|---|---|---|---|---|
| str | u32 | f64 | f64 | f64 | f64 |
| "Biscoe" | 168 | 15.87485 | 214.0 | 782.855743 | 0.490798 |
| "Dream" | 124 | 18.344355 | 193.0 | 416.644112 | 0.495935 |
| "Torgersen" | 52 | 18.429412 | 191.0 | 445.10794 | 0.510638 |
Conclusion
This post highlights syntactic similarities and differences across R’s dplyr and Python’s polars packages. One key point emerges: dplyr heavily relies on NSE to enable a syntax that refrains from using strings and column selectors, something that is not possible in Python. I want to close this post by emphasizing that both languages and packages have their own merits and I won’t strictly recommend one over the other - maybe in another post 😄
Footnotes
See the unifying principles of the tidyverse: https://design.tidyverse.org/unifying.html.↩︎