Avoid libc buffered IO #294

stewartsmith · 2024-02-09T17:23:24Z

the FILE related IO functions in libc do buffering inside the C library, and are generally less performant than using the file descriptor based open()/read()/write() functions.

The FILE based approach somewhat limits the maximum throughput of librepo, as well as increases CPU usage. In my benchmarks of a reposync of the Amazon Linux 2023 x86-64 repositories, this move to file descriptor based IO saves about 1 second of user time, and .5 seconds of system time, for a wall clock time benefit of a few seconds (102s vs 99s).

stewartsmith · 2024-02-09T17:28:37Z

For reference, my benchmarking has been done on a m5n.16xlarge EC2 instance to the in-region S3 buckets as well as to the CDN repositories. That instance type has 256GB memory, a 75Gbit network connection, and is a 64 core Cascade Lake system. The root volume is a 256GB gp3 EBS volume with 500MB/sec of IO and 3000 IOPs.

The background of this is that a lot of EC2 instances don't live that long (relatively speaking), and never install RPMs except on launch - so all the time-to-install RPMs is time spent scaling up a system that could be better served by running the customer workload.

ppisar · 2024-02-12T09:26:48Z

librepo/downloader.c

@@ -619,7 +619,7 @@ lr_writecb(char *ptr, size_t size, size_t nmemb, void *userdata)
    if (range_start <= 0 && range_end <= 0) {
        // Write everything curl give to you
        target->writecb_recieved += all;
-        return fwrite(ptr, size, nmemb, target->f);
+        return write(target->fd, ptr, all);


Replacing fwrite() with write() is not so simple:

write() expects size_t in its last argument, while you pass gint64. That won't match on 32-bit memory model, e.g. on 32-bit x86 platform.

write() returns a signed ssize_t, while fwrite() unsigned size_t. Your return statement does match lr_writecb() prototype.

write() can return before writing all data with errno==EINTR and you need to call write() again.

Please fix these issues at all places you replaced fwrite() with write().

fixed.

Included some code for keeping to reasonable IO sizes and just avoiding any of these issues with signed/unsigned.

ppisar · 2024-02-12T09:30:13Z

librepo/downloader.c

@@ -684,7 +684,7 @@ lr_writecb(char *ptr, size_t size, size_t nmemb, void *userdata)
    }

    assert(nmemb > 0);
-    cur_written = fwrite(ptr, size, nmemb, target->f);
+    cur_written = write(target->fd, ptr, size * nmemb);


What if size*nmemb overflows size_t? fwrite() handled it for you. Now you need to implement it check yourself.

Updated to check for overflow and fall back to a well known size.

ppisar · 2024-02-12T09:34:12Z

librepo/downloader.c

@@ -1033,9 +1033,9 @@ remove_librepo_xattr(LrDownloadTarget * target)
 gboolean
 lr_zck_clear_header(LrTarget *target, GError **err)
 {
-    assert(target && target->f && target->target && target->target->path);
+    assert(target && target->fd && target->target && target->target->path);


fd is an int. fd==0 is a valid descriptor. You should compare it to -1, or at least to < 0 as in invalid descriptor.

ppisar · 2024-02-12T09:34:53Z

librepo/downloader.c

@@ -1295,7 +1295,7 @@ static gboolean
 check_zck(LrTarget *target, GError **err)
 {
    assert(!err || *err == NULL);
-    assert(target && target->f && target->target);
+    assert(target && target->fd && target->target);


Again. fd == 0 is a valid value. Compare to -1.

ppisar · 2024-02-12T09:35:45Z

librepo/downloader.c

-            fclose(target->f);
-            target->f = NULL;
+            close(target->fd);
+            target->fd = 0;


ppisar · 2024-02-12T09:38:35Z

librepo/downloader.c

@@ -1573,8 +1563,7 @@ prepare_next_transfer(LrDownload *dd, gboolean *candidatefound, GError **err)

        if (target->original_offset == -1) {
            // Determine offset
-            fseek(target->f, 0L, SEEK_END);
-            gint64 determined_offset = ftell(target->f);
+            gint64 determined_offset = lseek(target->fd, 0L, SEEK_END);


lseek() returns off_t. Not gint64. Those won't match on 32-bit platforms without large file support enabled.

fixed (in upcoming updated patch...)

there's this snipped below from the original code:

gint64 used_offset = target->original_offset; g_debug("%s: Used offset for download resume: %"G_GINT64_FORMAT, __func__, used_offset); c_rc = curl_easy_setopt(h, CURLOPT_RESUME_FROM_LARGE, (curl_off_t) used_offset);

So there might be some places in existing code that doesn't match everywhere. Possible that this should all just move to off_t ?

ppisar · 2024-02-12T09:39:05Z

librepo/downloader.c

-    if (target->f != NULL) {
-        fclose(target->f);
-        target->f = NULL;
+    if (target->fd != 0) {


Compare to -1 and set to -1.

ppisar · 2024-02-12T09:40:22Z

librepo/downloader.c

-        fclose(target->f);
-        target->f = NULL;
+        close(target->fd);
+        target->fd = 0;


ppisar · 2024-02-12T09:40:37Z

librepo/downloader.c

-            fclose(target->f);
-            target->f = NULL;
+            close(target->fd);
+            target->fd = 0;


ppisar · 2024-02-12T09:40:49Z

librepo/downloader.c

@@ -2803,7 +2791,7 @@ lr_download(GSList *targets,
    for (GSList *elem = dd.targets; elem; elem = g_slist_next(elem)) {
        LrTarget *target = elem->data;
        assert(target->curl_handle == NULL);
-        assert(target->f == NULL);
+        assert(target->fd == 0);


Compare against -1.

ppisar · 2024-02-12T09:44:25Z

In general, I like your patch, but please address the different semantics I pointed in-line.

stewartsmith · 2024-02-12T15:32:25Z

Thanks for the eyes on it. I can clean up the review comments, and spin a rev2.

the FILE related IO functions in libc do buffering inside the C library, and are generally less performant than using the file descriptor based open()/read()/write() functions. The FILE based approach somewhat limits the maximum throughput of librepo, as well as increases CPU usage. In my benchmarks of a reposync of the Amazon Linux 2023 x86-64 repositories, this move to file descriptor based IO saves about 1 second of user time, and .5 seconds of system time, for a wall clock time benefit of a few seconds (102s vs 99s).

stewartsmith · 2024-06-05T18:06:49Z

I think I've managed to address the comments and ensure correct behavior in (hopefully) all error conditions.

Conan-Kudo

Overall, it looks fine. One small change, though.

Conan-Kudo · 2024-08-24T13:44:15Z

librepo/downloader.c

+     * We write up to 32MB at a time, mainly to ensure that the below loops
+     * never bitrot, and it's regularly tested with real world RPMs.
+     */
+#define WRITECB_CHUNK_MAX 32*1024*1024


Please indent to match the line starting point.

This was referenced Feb 9, 2024

Increase checksum buffer to 128kb, improving download performance. #295

Open

use mmap for checksum of larger RPMs #296

Open

Don't fsync() in checksum #297

Open

ppisar reviewed Feb 12, 2024

View reviewed changes

ppisar self-assigned this Feb 12, 2024

ppisar added the Triaged Someone on the DNF 5 team has read the issue and determined the next steps to take label Feb 12, 2024

stewartsmith force-pushed the no-libc-buffered-io branch from a38226c to ebd4be0 Compare June 5, 2024 00:47

Conan-Kudo requested changes Aug 24, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid libc buffered IO #294

Avoid libc buffered IO #294

stewartsmith commented Feb 9, 2024

stewartsmith commented Feb 9, 2024 •

edited

Loading

ppisar Feb 12, 2024

stewartsmith Jun 5, 2024

ppisar Feb 12, 2024

stewartsmith Jun 5, 2024

ppisar Feb 12, 2024

stewartsmith Jun 5, 2024

ppisar Feb 12, 2024

stewartsmith Jun 5, 2024

ppisar Feb 12, 2024

stewartsmith Jun 5, 2024

ppisar Feb 12, 2024

stewartsmith Jun 4, 2024 •

edited

Loading

ppisar Feb 12, 2024

stewartsmith Jun 5, 2024

ppisar Feb 12, 2024

stewartsmith Jun 5, 2024

ppisar Feb 12, 2024

stewartsmith Jun 5, 2024

ppisar Feb 12, 2024

stewartsmith Jun 5, 2024

ppisar commented Feb 12, 2024

stewartsmith commented Feb 12, 2024

stewartsmith commented Jun 5, 2024

Conan-Kudo left a comment

Conan-Kudo Aug 24, 2024

Avoid libc buffered IO #294

Are you sure you want to change the base?

Avoid libc buffered IO #294

Conversation

stewartsmith commented Feb 9, 2024

stewartsmith commented Feb 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stewartsmith Jun 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ppisar commented Feb 12, 2024

stewartsmith commented Feb 12, 2024

stewartsmith commented Jun 5, 2024

Conan-Kudo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stewartsmith commented Feb 9, 2024 •

edited

Loading

stewartsmith Jun 4, 2024 •

edited

Loading