daniel/blog
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Re-organize files and folders.

Browse Source
This commit is contained in:
daniel
2024-08-28 16:31:20 +02:00
parent b5118b2835
commit 7092b10849
10 changed files with 7 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
content/posts/2024/duckplyr-performance/benchmarking1.svg Normal file
View File

@ -0,0 +1,74 @@
<?xml version='1.0' encoding='UTF-8' ?>
<svg xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink' class='svglite' width='576.00pt' height='360.00pt' viewBox='0 0 576.00 360.00'>
<defs>
<style type='text/css'><![CDATA[
.svglite line, .svglite polyline, .svglite polygon, .svglite path, .svglite rect, .svglite circle {
fill: none;
stroke: #000000;
stroke-linecap: round;
stroke-linejoin: round;
stroke-miterlimit: 10.00;
}
.svglite text {
white-space: pre;
}
]]></style>
</defs>
<rect width='100%' height='100%' style='stroke: none; fill: #292E32;'/>
<defs>
<clipPath id='cpMC4wMHw1NzYuMDB8MC4wMHwzNjAuMDA='>
<rect x='0.00' y='0.00' width='576.00' height='360.00' />
</clipPath>
</defs>
<g clip-path='url(#cpMC4wMHw1NzYuMDB8MC4wMHwzNjAuMDA=)'>
<rect x='0.000000000000064' y='0.00' width='576.00' height='360.00' style='stroke-width: 1.07; stroke: #FFFFFF; fill: #292E32;' />
</g>
<defs>
<clipPath id='cpNTkuMTB8NDIxLjE5fDQ2LjQxfDMwMi45OQ=='>
<rect x='59.10' y='46.41' width='362.09' height='256.58' />
</clipPath>
</defs>
<g clip-path='url(#cpNTkuMTB8NDIxLjE5fDQ2LjQxfDMwMi45OQ==)'>
<polyline points='59.10,302.99 421.19,302.99 ' style='stroke-width: 1.07; stroke: #3B3E48; stroke-linecap: butt;' />
<polyline points='59.10,238.06 421.19,238.06 ' style='stroke-width: 1.07; stroke: #3B3E48; stroke-linecap: butt;' />
<polyline points='59.10,173.14 421.19,173.14 ' style='stroke-width: 1.07; stroke: #3B3E48; stroke-linecap: butt;' />
<polyline points='59.10,108.22 421.19,108.22 ' style='stroke-width: 1.07; stroke: #3B3E48; stroke-linecap: butt;' />
<circle cx='151.61' cy='98.97' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='177.19' cy='91.03' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='174.46' cy='99.26' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='296.36' cy='101.41' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='320.05' cy='105.05' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='353.79' cy='91.38' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='325.12,89.89 331.35,100.67 318.89,100.67 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='327.97,92.20 334.20,102.98 321.75,102.98 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='327.74,96.20 333.97,106.98 321.52,106.98 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='167.19,87.42 173.41,98.20 160.96,98.20 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='190.42,81.99 196.65,92.77 184.20,92.77 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='165.99,82.14 172.22,92.92 159.77,92.92 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
</g>
<g clip-path='url(#cpMC4wMHw1NzYuMDB8MC4wMHwzNjAuMDA=)'>
<polyline points='59.10,302.99 59.10,46.41 ' style='stroke-width: 2.13; stroke: #3B3E48; stroke-linecap: butt;' />
<text x='54.17' y='306.91' text-anchor='end' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='6.29px' lengthAdjust='spacingAndGlyphs'>0</text>
<text x='54.17' y='241.99' text-anchor='end' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='12.59px' lengthAdjust='spacingAndGlyphs'>10</text>
<text x='54.17' y='177.07' text-anchor='end' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='12.59px' lengthAdjust='spacingAndGlyphs'>20</text>
<text x='54.17' y='112.14' text-anchor='end' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='12.59px' lengthAdjust='spacingAndGlyphs'>30</text>
<polyline points='56.36,302.99 59.10,302.99 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='56.36,238.06 59.10,238.06 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='56.36,173.14 59.10,173.14 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='56.36,108.22 59.10,108.22 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='59.10,302.99 421.19,302.99 ' style='stroke-width: 2.13; stroke: #3B3E48; stroke-linecap: butt;' />
<polyline points='157.85,305.73 157.85,302.99 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='322.44,305.73 322.44,302.99 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<text x='157.85' y='315.77' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='26.52px' lengthAdjust='spacingAndGlyphs'>dplyr</text>
<text x='322.44' y='315.77' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='44.47px' lengthAdjust='spacingAndGlyphs'>duckplyr</text>
<text x='240.15' y='329.01' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='36.24px' lengthAdjust='spacingAndGlyphs'>Library</text>
<text transform='translate(36.20,174.70) rotate(-90)' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='40.16px' lengthAdjust='spacingAndGlyphs'>Time (s)</text>
<rect x='432.15' y='143.95' width='115.50' height='61.50' style='stroke-width: 1.07; stroke: none; fill: #3B3E48;' />
<text x='437.63' y='158.61' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='104.54px' lengthAdjust='spacingAndGlyphs'>Laptop power mode</text>
<circle cx='446.27' cy='174.05' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='446.27,184.14 452.50,194.92 440.04,194.92 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<text x='460.39' y='177.19' style='font-size: 8.80px;fill: #FFFFFF; font-family: "Arial";' textLength='37.56px' lengthAdjust='spacingAndGlyphs'>balanced</text>
<text x='460.39' y='194.47' style='font-size: 8.80px;fill: #FFFFFF; font-family: "Arial";' textLength='53.76px' lengthAdjust='spacingAndGlyphs'>performance</text>
<text x='59.10' y='37.76' style='font-size: 13.20px;fill: #FFFFFF; font-family: "Arial";' textLength='224.18px' lengthAdjust='spacingAndGlyphs'>Time elapsed with dplyr vs. duckplyr</text>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 6.5 KiB

87
content/posts/2024/duckplyr-performance/benchmarking2.svg Normal file
View File

@ -0,0 +1,87 @@
<?xml version='1.0' encoding='UTF-8' ?>
<svg xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink' class='svglite' width='576.00pt' height='360.00pt' viewBox='0 0 576.00 360.00'>
<defs>
<style type='text/css'><![CDATA[
.svglite line, .svglite polyline, .svglite polygon, .svglite path, .svglite rect, .svglite circle {
fill: none;
stroke: #000000;
stroke-linecap: round;
stroke-linejoin: round;
stroke-miterlimit: 10.00;
}
.svglite text {
white-space: pre;
}
]]></style>
</defs>
<rect width='100%' height='100%' style='stroke: none; fill: #292E32;'/>
<defs>
<clipPath id='cpMC4wMHw1NzYuMDB8MC4wMHwzNjAuMDA='>
<rect x='0.00' y='0.00' width='576.00' height='360.00' />
</clipPath>
</defs>
<g clip-path='url(#cpMC4wMHw1NzYuMDB8MC4wMHwzNjAuMDA=)'>
<rect x='0.000000000000064' y='0.00' width='576.00' height='360.00' style='stroke-width: 1.07; stroke: #FFFFFF; fill: #292E32;' />
</g>
<defs>
<clipPath id='cpNTkuMTB8NDIxLjE5fDQ2LjQxfDI3OS4yMw=='>
<rect x='59.10' y='46.41' width='362.09' height='232.82' />
</clipPath>
</defs>
<g clip-path='url(#cpNTkuMTB8NDIxLjE5fDQ2LjQxfDI3OS4yMw==)'>
<polyline points='59.10,279.23 421.19,279.23 ' style='stroke-width: 1.07; stroke: #3B3E48; stroke-linecap: butt;' />
<polyline points='59.10,225.86 421.19,225.86 ' style='stroke-width: 1.07; stroke: #3B3E48; stroke-linecap: butt;' />
<polyline points='59.10,172.48 421.19,172.48 ' style='stroke-width: 1.07; stroke: #3B3E48; stroke-linecap: butt;' />
<polyline points='59.10,119.11 421.19,119.11 ' style='stroke-width: 1.07; stroke: #3B3E48; stroke-linecap: butt;' />
<polyline points='59.10,65.74 421.19,65.74 ' style='stroke-width: 1.07; stroke: #3B3E48; stroke-linecap: butt;' />
<circle cx='127.08' cy='111.51' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='142.09' cy='105.00' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='132.50' cy='111.77' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='246.17' cy='113.52' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='223.06' cy='116.55' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='260.64' cy='105.27' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='249.90,102.73 256.13,113.51 243.67,113.51 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='243.36,104.64 249.58,115.42 237.13,115.42 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='218.81,108.00 225.03,118.79 212.58,118.79 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='119.28,100.75 125.50,111.54 113.05,111.54 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='114.27,96.29 120.49,107.07 108.04,107.07 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='131.88,96.40 138.10,107.19 125.65,107.19 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='335.41' cy='90.10' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='351.34' cy='85.38' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<circle cx='336.95' cy='85.21' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='344.58,92.55 350.81,103.33 338.36,103.33 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='349.73,94.20 355.96,104.98 343.51,104.98 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='344.11,90.98 350.33,101.77 337.88,101.77 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
</g>
<g clip-path='url(#cpMC4wMHw1NzYuMDB8MC4wMHwzNjAuMDA=)'>
<polyline points='59.10,279.23 59.10,46.41 ' style='stroke-width: 2.13; stroke: #3B3E48; stroke-linecap: butt;' />
<text x='54.17' y='283.15' text-anchor='end' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='6.29px' lengthAdjust='spacingAndGlyphs'>0</text>
<text x='54.17' y='229.78' text-anchor='end' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='12.59px' lengthAdjust='spacingAndGlyphs'>10</text>
<text x='54.17' y='176.41' text-anchor='end' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='12.59px' lengthAdjust='spacingAndGlyphs'>20</text>
<text x='54.17' y='123.04' text-anchor='end' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='12.59px' lengthAdjust='spacingAndGlyphs'>30</text>
<text x='54.17' y='69.67' text-anchor='end' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='12.59px' lengthAdjust='spacingAndGlyphs'>40</text>
<polyline points='56.36,279.23 59.10,279.23 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='56.36,225.86 59.10,225.86 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='56.36,172.48 59.10,172.48 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='56.36,119.11 59.10,119.11 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='56.36,65.74 59.10,65.74 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='59.10,279.23 421.19,279.23 ' style='stroke-width: 2.13; stroke: #3B3E48; stroke-linecap: butt;' />
<polyline points='126.99,281.97 126.99,279.23 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='240.15,281.97 240.15,279.23 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<polyline points='353.30,281.97 353.30,279.23 ' style='stroke-width: 1.07; stroke: #333333; stroke-linecap: butt;' />
<text x='126.99' y='292.01' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='26.52px' lengthAdjust='spacingAndGlyphs'>dplyr</text>
<text x='240.15' y='292.01' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='44.47px' lengthAdjust='spacingAndGlyphs'>duckplyr</text>
<text x='353.30' y='292.01' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='44.47px' lengthAdjust='spacingAndGlyphs'>duckplyr</text>
<text x='353.30' y='303.89' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='22.26px' lengthAdjust='spacingAndGlyphs'>with</text>
<text x='353.30' y='315.77' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='101.25px' lengthAdjust='spacingAndGlyphs'>`as_duckplyr_tibble`</text>
<text x='240.15' y='329.01' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='36.24px' lengthAdjust='spacingAndGlyphs'>Library</text>
<text transform='translate(36.20,162.82) rotate(-90)' text-anchor='middle' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='40.16px' lengthAdjust='spacingAndGlyphs'>Time (s)</text>
<rect x='432.15' y='132.07' width='115.50' height='61.50' style='stroke-width: 1.07; stroke: none; fill: #3B3E48;' />
<text x='437.63' y='146.73' style='font-size: 11.00px;fill: #FFFFFF; font-family: "Arial";' textLength='104.54px' lengthAdjust='spacingAndGlyphs'>Laptop power mode</text>
<circle cx='446.27' cy='162.17' r='4.62' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<polygon points='446.27,172.26 452.50,183.04 440.04,183.04 ' style='stroke-width: 0.71; stroke: none; fill: #FFFFFF; fill-opacity: 0.80;' />
<text x='460.39' y='165.31' style='font-size: 8.80px;fill: #FFFFFF; font-family: "Arial";' textLength='37.56px' lengthAdjust='spacingAndGlyphs'>balanced</text>
<text x='460.39' y='182.59' style='font-size: 8.80px;fill: #FFFFFF; font-family: "Arial";' textLength='53.76px' lengthAdjust='spacingAndGlyphs'>performance</text>
<text x='59.10' y='37.76' style='font-size: 13.20px;fill: #FFFFFF; font-family: "Arial";' textLength='224.18px' lengthAdjust='spacingAndGlyphs'>Time elapsed with dplyr vs. duckplyr</text>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 8.4 KiB

127
content/posts/2024/duckplyr-performance/index.md Normal file
View File

@ -0,0 +1,127 @@
---
title: Performance experiments with duckplyr in R
description: I recently became aware of the 'duckplyr' library for R. Here are the results of my experimenting with it and benchmarking it against `dplyr`.
date: 2024-08-26T19:00:00+0200
draft: false
# ShowLastmod: true
toc: false
scrolltotop: true
tags:
- R
- statistics
---
I recently became aware of the [duckplyr][] library for R, which takes the place
of tidyverse's [dplyr][] library, but uses the [DuckDB] database under the hood.
Without really knowing anything about how dplyr works and if the use of DuckDB
would improve my workflow at all, I decided to perform an experiment. I am
currently analyzing two datasets, one with ~80k records and ~70 variables and
one with ~60k records and ~100 variables. Both datasets are wrangled with
[Tidyverse][]-foo in multiple ways and finally combined. The wrangling of the
data involves things like `rowwise()` and `c_across()`, which I know from
experience is quite an 'expensive' operation.
In order to get the execution times of my code, I did this repeatedly:
1. Restart R (by pressing <kbd>CTRL</kbd> <kbd>SHIFT</kbd> <kbd>F10</kbd>).
2. Run
```r
system.time(rmarkdown::render("my_file.Rmd"))
```
3. Record the user time and the system time elapsed.
4. Repeat twice.
I did this with both the "balanced power mode" and the "performance mode" on my
[laptop][]. During execution of the code, I left the laptop alone in order not
to interfere with the timing.
This is the result of my benchmarking:
{{< figure src="benchmarking1.svg" >}}
The times are user times. I left out the system times, which are in the range of
2-3 seconds.
Not really mind-boggling, right? It occurred to me that I rather double-check
that `duckplyr` was really being used. Indeed, this was _not_ the case:
```r
> class(clinical_data)
[1] "tbl_df" "tbl" "data.frame"
```
`clinical_data` was missing the `duckplyr_df' class. How come?
I import the raw data from Excel files (don't ask...) into tibbles, and
evidently, this prevents `duckplyr` from seeing the data frames. So I piped the
data frames through `as_duckplyr_tibble()` explicitly, and this got me the right
classes:
```r
> class(clinical_data)
[1] "duckplyr_df" "tbl_df" "tbl" "data.frame"
```
However, this did not really speed up the execution either.
{{< figure src="benchmarking2.svg" >}}
I looked around my RMarkdown chunks and their outputs, but I did not find any
warning that `duckplyr` had to fall back to `dplyr`'s methods. This could have
explained the absence of a noticeable difference.
Here are the average times (in seconds) for the benchmarking runs.
```r
> runs_table
# A tibble: 6 × 4
# Groups: library, power_mode [6]
library power_mode mean sd
<chr> <chr> <dbl> <dbl>
1 dplyr balanced 31.8 0.722
2 dplyr performance 32.6 0.477
3 duckplyr balanced 31.4 1.10
4 duckplyr performance 31.3 0.495
5 duckplyr with `as_duckplyr_tibble` balanced 36.0 0.517
6 duckplyr with `as_duckplyr_tibble` performance 33.6 0.303
```
So at least for my (!!!) use case, the use of `duckplyr` instead of `dplyr` did
not make any practical difference, and I can also leave my laptop's performance
mode alone. When it comes to optimizing performance, you can't just buy a
solution off the shelf, you always have to try and find the best solution for
your specific problem.
Your mileage will vary, of course. The people who develop `duckplyr` are
brilliant, and the fact that it does not work for me tells more about me and my
work than it does about `duckplyr`.
## The duckplyr demo dataset
As a case in point, the [duckplyr demo repository][duckplyr-demo] contains a
taxi data set. The ZIP file alone is a ~1.7 GB download. Deflated, the files
take up 2.4 GB. With about 21 million records (24 variables), this dataset
is _considerably_ larger than mine.
Here are the results from running `dplyr/run_all_queries.R` and
`duckplyr/run_all_queries.R` on my Thinkpad P14s (performance mode in F40 KDE):
| Library | q01 | q02 | q03 | q04 |
|----------|------:|------:|------:|-------:|
| dplyr | 3.4 s | 3.9 s | 9.1 s | 14.3 s |
| duckplyr | 4.3 s | 4.4 s | 9.4 s | 14.8 s |
I should add that execution times vary with each run, but the big picture stays
the same.
Maybe I'm missing the point and it's not about execution times, after all.
`¯\_(ツ)_/`
[dplyr]: https:/dplyr.tidyverse.org
[duckdb]: https://duckdb.org
[duckplyr]: https://duckplyr.tidyverse.org
[duckplyr-demo]: https://github.com/Tmonster/duckplyr_demo
[laptop]: {{< relref "P14s" >}}
[tidyverse]: https://tidyverse.org

24
content/posts/2024/new-blog/index.md Normal file
View File

@ -0,0 +1,24 @@
---
title: "Welcome to the new blog"
date: 2024-08-01T00:00:00+02:00
draft: false
tags:
- meta
---
Welcome to my new blog.
This blog continues where my old one at
<https://www.xltoolbox.net/blog/index.html> left off.
I thought about moving the old blog posts over to this new blog, but those posts
are really ancient and not relevant any more. I leave them at the old place for
archival purposes.
I have implemented a commenting system based on [Discourse][]. Please feel free
to comment on my posts, ask questions etc. You'll either have to create a
standalone account, or you can log in with your Google or Github account.
See the discussion forum at <https://discuss.bovender.de>.
[discourse]: https://discourse.org

267
content/posts/2024/p14s/index.md Normal file
View File

@ -0,0 +1,267 @@
---
title: "Experience with running Fedora Linux on a Thinkpad P14s Gen 5"
description: "Everything works out of the box with KDE Plasma, but the keyboard and the battery leave something to be desired."
date: 2024-08-05T21:00:00+02:00
ShowLastmod: false
draft: false
toc: true
scrolltotop: true
tags:
- Fedora
- F40
- KDE
- Linux
- NVIDIA
- Thinkpad
---
I have recently acquired a Lenovo Thinkpad P14s Gen 5. As of the time of writing
-- summer of '24 -- these machines are brand new, and so is their architecture:
Intel Core Ultra (Meteor Lake) with a dedicated NVIDIA GPU. Here I report my
experience with running Linux with the [KDE Plasma][] desktop on this machine.
My previous daily driver was a T480s (see my old blog post over at [xltoolbox.net][]
for a report). I also administer two different X1 Carbons (5th generation and
9th generation), however, I don't use these very often myself.
Here are the specs (the 1 TB SSD has been swapped for a Samsung 990 PRO with 2 TB).
```plain
daniel@seppel9 ~> fastfetch
.',;::::;,'. daniel@seppel9
.';:cccccccccccc:;,. --------------
.;cccccccccccccccccccccc;. OS: Fedora Linux 40 (KDE Plasma) x86_64
.:cccccccccccccccccccccccccc:. Host: 21G3S00A00 (ThinkPad P14s Gen 5)
.;ccccccccccccc;.:dddl:.;ccccccc;. Kernel: Linux 6.9.11-200.fc40.x86_64
.:ccccccccccccc;OWMKOOXMWd;ccccccc:. Uptime: 1 hour, 47 mins
.:ccccccccccccc;KMMc;cc;xMMc;ccccccc:. Packages: 4338 (rpm)
,cccccccccccccc;MMM.;cc;;WW:;cccccccc, Shell: fish 3.7.0
:cccccccccccccc;MMM.;cccccccccccccccc: Display (LEN8AB1): 3072x1920 @ 120 Hz (as 2048x1280) in 14″ [Built-in]
:ccccccc;oxOOOo;MMM000k.;cccccccccccc: DE: KDE Plasma 6.1.3
cccccc;0MMKxdd:;MMMkddc.;cccccccccccc; WM: KWin (Wayland)
ccccc;XMO';cccc;MMM.;cccccccccccccccc' WM Theme: Breeze
ccccc;MMo;ccccc;MMW.;ccccccccccccccc; Theme: Breeze (Dark) [Qt], Breeze [GTK3]
ccccc;0MNc.ccc.xMMd;ccccccccccccccc; Icons: breeze-dark [Qt], breeze-dark [GTK3/4]
cccccc;dNMWXXXWM0:;cccccccccccccc:, Font: IBM Plex Sans (10pt) [Qt], IBM Plex Sans (10pt) [GTK3/4]
cccccccc;.:odl:.;cccccccccccccc:,. Cursor: breeze (24px)
ccccccccccccccccccccccccccccc:'. Terminal: konsole 24.5.2
:ccccccccccccccccccccccc:;,.. CPU: Intel(R) Core(TM) Ultra 7 155H (22) @ 4,80 GHz
':cccccccccccccccc::;,. GPU 1: Intel rc Graphics @ 2,25 GHz [Integrated]
GPU 2: NVIDIA RTX 500 Ada Generation Laptop GPU
Memory: 9,13 GiB / 62,29 GiB (15%)
Swap: 302,50 MiB / 104,00 GiB (0%)
Disk (/): 18,77 GiB / 146,59 GiB (13%) - ext4
Disk (/home): 524,56 GiB / 1,55 TiB (33%) - ext4
Local IP (wlp0s20f3): 192.168.3.114/24
Battery: 93% [AC Connected]
Locale: en_US.UTF-8
```
## Difficulties and finally success with installing Linux
At first, I was totally unable to install [KDE neon][], my preferred Linux distribution,
on this system. I freaked out. (I don't care if it's officially a 'distribution' or not.)
The live CD just would not boot. It did work in safe graphics mode, but when I installed
the system onto the laptop, I was once again unable to boot into Plasma.
(I already posted in the [KDE forum][] about this).
At some point it occurred to me that I may need to try a distribution, so I tried [Fedora][]
-- and it worked!! In order to boot the live CD, I also had to choose basic graphics mode
just like with KDE neon (which is Ubuntu under the hood). Once I had the live desktop up
and running, I could enable the NVIDIA repositories, so that the NVIDIA drivers were
included during system installation. Luckily, the Fedora Project provides a [KDE spin][]
so that I could continue to use my favorite desktop environment.
## Hardware
Except for the initial difficulties getting this system with a dedicated NVIDIA GPU
to run Linux, everything else just works out of the box. Nice! :-)
The **trackpoint** works as usual. This is really the main reason why I keep buying
Thinkpads. A laptop without trackpoint is a no-go for me. In fact, I did consider buying
a MacBook Pro, but the absence of a trackpoint and the uncertainty whether I could do
my beloved Linux foo with MacOS made me stick with a Thinkpad.
The **fingerprint scanner** is integrated into the power button. While not essential, it's
a nice gimmick and KDE Plasma provides a pretty UI to set it up. Currently, logging
into the system after booting is not (yet) supported, but unlocking (e.g., after resume
from sleep) works very well.
The 3k **screen** is really good, it's very bright and super crisp. At 100%, the
user interface is quite tiny. Luckily, with Wayland, fractional scaling is now
easy to accomplish and reliable. With a scale factor of 1.2 (120%), I am very happy
with my P14s' display. Interestingly though, the corners are rounded, even though
the laptop frame has sharp corners. I tend to think: If you have enough time wondering
about something like this, it's probably time to take a break and do something else
for a while.
Unfortunately, the **keyboard** is the worst that I have ever had in any Thinkpad.
It feels like typing on jelly. Luckily, for most of the heavy-duty work, I connect
the laptop to a Thunderbolt dock with a proper mechanical keyboard attached.
I wonder why Lenovo does not equip this machine with the same keyboard as in the
other models -- given the price tag of this machine, the extra bucks surely cannot
make such a big difference?
**Camera**, mike, and speakers just work. No problem running Zoom. I have not yet
tried out Teams.
## Form factor and weight
The P14s is somewhat bigger than the old T480s and certainly somewhat heavier.
Before deciding on the purchase, I carefully thought about my use cases and decided
that I can live with an extra 200-300 grams that I mostly only carry onto and
off from the commute train, as well as on my bicycle's rack. Therefore, I made a
decision for the extra performance (with dedicated GPU, which the alternative
models, i.e. X1 Carbons or T14s's do not offer).)
Lenovo [says][p14s-weight] the P14s starts at 1.6 kg, which compares to a MacBook Pro's
nominal weight. On our kitchen scale, my P14s weighs 1.74 kg.
In real life, the laptop's body is somewhat bigger than the actual casing, because
it has a very long "foot" underneath it. This ensures proper ventilation, because
the two fans blow out the warm air underneath the laptop. While this "foot" makes
the P14s rather thick, it turns out that this also provides a good grip when pulling
it ouf of a bag or carrying it around.
I have yet to come to terms with the new camera casing that all new Thinkpads
seem to have nowadays.
{{< figure src="p14s-camera-casing.jpg"
alt="A photograph of the camera casing of a Thinkpad P14s Gen 5"
caption="The camera casing. It disrupts the smooth surface of the Thinkpad.">}}
{{< figure src="p14s-foot.jpg"
alt="A photograph of the 'foot' underneath a Thinkpad P14s Gen 5"
caption="The 'foot' underneath the P14s makes it thicker, but also provides a good grip on the machine.">}}
## Performance
Compared to the old T480s, the performance gain of the Intel Core Ultra and the
dedicated NVIDIA GPU is very noticeable. Whether I compile R packages or analyze
clinical data with ~80k subjects in [RStudio][], develop RAW images in [Darktable][],
or work with Office in a VirtualBox-hosted Windows machine, everything is very
snappy. For instance, a typical RAW image is now processed in under 1 second, as
opposed to ~10 seconds on the T480s.
The machine wakes from sleep (suspend to RAM) extremely quickly, I guess this is due to
[novel suspend mechanisms][s3]? On the other hand, boot times are rather slow
(34-40s on AC power, T480s on battery 31 s!!).
## Fan
The fans spin up quickly, but the noise is not unpleasant. Maybe I can tweak the
fan control in order to have them spin up a little more lazily. But when I had both
my old laptop and the new one side by side, I realized that the T480s' fan noise
drowns the P14s' fan noise. Notably, the P14s was running on AC power, while the
T480s ran on battery. In fact, even the T480s with built-in graphics had its fan
running most of the time when I was working with it (academic work, researching,
writing, data analysis etc.).
## Energy consumption
The P14s' battery has much more capacity than the T480s' one, 75 Wh vs. 42 Wh.
However, with my normal work load, the battery drains rather quickly. The
following screenshot shows the charge level over time as I sat outside,
editing a manuscript with Word running in a virtual machine (VirtualBox
with Windows 11 guest) and doing some internet research in a browser. The
screen brightness was set to 100%. After some 3.5 hours, it's time for a
break...
{{< figure src="p14s-battery-drain.png">}}
## Making the switch from Ubuntu to Fedora
When I downloaded the Fedora [KDE spin][], I was admittedly a bit anxious.
I had been working with Ubuntu and KDE neon, which is based on Ubuntu,
for many years, and I wondered if Fedora would be very different.
It's not.
The three things that are really different from my perspective:
- To install something on the command line, use `dnf` rather than `apt`:
```bash
sudo dnf install good-stuff
```
- Removable disks (USB drives) are mounted on `/run/media/<USER>` rather
than `/media/<USER>`.
- More up-to-date kernel and software
## Conclusion
All in all, I am quite happy with the P14s. In the past couple of weeks, it has
already proven to be a reliable work horse, and this is what I need. The
keyboard really should be better, but hey, life is not perfect, is it?
-----
## Update 2024-08-14
Two important things to add:
1. After a recent system update, **suspend (sleep) stopped working**. First I
suspected the NVIDIA driver to be responsible, it had been upgrade to version
560. However, when I downgraded the driver, the problem persisted. Finally I
found a [thread][] on the Fedora Discussion forum which suggested to disable
ethernet in BIOS. Tada! That was it. Hopefully, with a future kernel or
firmware upgrade, I will be able to use ethernet again, as I prefer to have
cable-bound internet at home. The current linux kernel is 6.10, upgraded from
6.9 just recently.
2. Here's another [interesting report][techtipsy] from someone running **Linux on a P14S**,
albeit Gen 4 (mine is Gen 5) and with AMD and integrated graphics (I have
Intel with dedicated NVIDIA GPU). The author (Herman) also notes the battery
drain (even with AMD and iGPU) and the build quality which is somewhat below
the T series. Herman has also posted about [controlling fan speed with a simple
script][fancontrol], and I think I am going to tinker with it in the future,
because the following works on my P14s Gen 5 with Intel CPU and NVIDIA GPU:
```bash
# This is just a proof of concept, spin up the fan and spin it down again
# Run as root
echo level 7 > /proc/acpi/ibm/fan; sleep 5; echo level 1 > /proc/acpi/ibm/fan
```
## Update 2024-08-22
Fedora installs only free software by default, which means that [Kdenlive][] and
other multimedia software will have trouble using H.264 and H.265 video codecs.
A free version of [ffmpeg][] is installed by default. This is what I did to
replace `ffmpeg-free` with the non-free `ffmpeg` on F40:
1. Enable the non-free [RPM Fusion][] repositories as written on the [RPM Fusion
homepage][]:
```bash
sudo dnf install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf config-manager --enable fedora-cisco-openh264
```
2. Remove `ffmpeg-free` in favor of `ffmpeg`:
```bash
sudo dnf swap ffmpeg-free ffmpeg --allowerasing
```
After doing that and restarting Kdenlive, I was able to render my project using
the H.264 and H.265 codecs as I used to be able to do with Ubuntu-based systems.
[Darktable]: https://darktable.org
[fancontrol]: https://ounapuu.ee/posts/2022/09/26/minimum-viable-fan-control-script/
[Fedora]: https://fedoraproject.org/
[ffmpeg]: https://ffmpeg.org
[KDE forum]: https://discuss.kde.org/t/unable-to-start-kde-neon-plasma-on-core-ultra-nvidia-rtx-500-ada/18578
[KDE neon]: https://neon.kde.org
[KDE plasma]: https://kde.org/plasma-desktop/
[KDE spin]: https://fedoraproject.org/spins/
[Kdenlive]: https://kdenlive.org
[p14s-weight]: https://www.lenovo.com/us/en/p/laptops/thinkpad/thinkpadp/thinkpad-p14s-gen-5-(14-inch-intel)-mobile-workstation/21g2002cus#tech_specs
[rpm fusion]: https://rpmfusion.org
[rpm fusion homepage]: https://rpmfusion.org/Configuration
[rstudio]: https://posit.co/products/open-source/rstudio
[s3]: https://en.wikipedia.org/wiki/ACPI#S3
[techtipsy]: https://ounapuu.ee/posts/2024/04/12/lenovo-p14s-gen4/
[thread]: https://discussion.fedoraproject.org/t/suspend-failure-and-degraded-performance-after-failed-suspend-on-kernel-6-10-3/128299
[xltoolbox.net]: https://www.xltoolbox.net/blog/2018/08/exit-thinkpad-t430s-enter-thinkpad-t480s.html

BIN
content/posts/2024/p14s/p14s-battery-drain.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 177 KiB

BIN
content/posts/2024/p14s/p14s-camera-casing.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 954 KiB

BIN
content/posts/2024/p14s/p14s-foot.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.8 MiB

31
content/posts/2024/plasma-5-tiling-manager/index.md Normal file
View File

@ -0,0 +1,31 @@
---
title: "Windows tiling in KDE Plasma 5"
date: 2024-08-08T18:04:20+0200
draft: false
toc: false
# ShowLastmod: true
images:
tags:
- Linux
- KDE
- Plasma
- productivity
---
KDE Plasma 5 introduced a (more or less) advanced window tiling manager. Most of
the time I use `META` plus one or two of the arrow keys to move windows quickly
to one half or one quarter of the screen. Plasma's advanced tiling manager
enables us to use layouts that are not defined by half or quarter of the screen
real estate.
I keep forgetting how to use this, so this is my note to self:
- Define layout: `META+T`
- Move window to predefined location: hold `SHIFT` while dragging the title bar
of a window with the mouse, or press `META`, klick anywhere into the window
and start dragging, _then_ while still holding `META`, additionally press
`SHIFT` to make the window snap to one of the predefined layout.
Link: <https://kde.org/announcements/plasma/5/5.27.0/>
I wish there were keyboard shortcuts to move windows into one of the custom tile
layouts as well: <https://discuss.kde.org/t/keyboard-shortcuts-for-new-tiling-feature/1843>