Rapid in-place overwrite results in excessive disk writing & laggy apps

[DeHackEd]

This is something I originally saw with an OpenLDAP host, but a guy in IRC saw it with his bitcoin client. What they have in common is a BDB backend database being populated with initial data from a replication source.

Generally speaking it seems that if you write X megabytes of data to disk, a transaction commit can occur. The catch is this can happen even if Y unique megabytes of data actually change and Y < X by a substantial margin.

This is my test program. CHUNKS needs to be tuned to the local system depending on memory availability. If I set CHUNKS really high (say, 2 gigabytes on my modest 6Gig workstation) then I can see that the first commit happens at about 130 megabytes and the program hangs there. If I set it to 80 megabytes as shown, then in theory it should never find itself grinding on the disk since I'm well under the memory limit, but it happens anyway. The program first blocks at about 1.5 iterations which is about the same amount of data submitted via [p]write but that's still only 80 megabytes of unique disk data.

#include "stdio.h"
#define _XOPEN_SOURCE 500
#include "unistd.h"
#include "fcntl.h

#include "malloc.h"

// 1 megabyte for sake of example, read from /dev/urandom so it's not compressible
#define BLOCKSIZE 1048576

// Select a size much smaller than the amount of
// RAM you have, but still big enough to be significant
// (eg: 80 megabytes for a blocksize of 1M)
#define CHUNKS 80

// App should run for a while, but not forever
#define ITERATIONS 128

// Whether you use write or pwrite doesn't seem to make a difference, but it's what BDB uses.
#define PWRITE

int main() {
  int fd;
  int len;
  int i, j;

  char *data = malloc(BLOCKSIZE);
  if (!data)
    return 1;

  fd = open("/dev/urandom", O_RDONLY); // uncompressable content
  if (fd < 0)
    return 2;
  len = read(fd, data, BLOCKSIZE);
  if (len != BLOCKSIZE)
    return 3;
  close(fd);

  fd = open("testfile", O_RDWR | O_CREAT, 0644);
  if (fd < 0)
    return 4;

  for (i=0; i < ITERATIONS; i++) {
    for (j=0; j < CHUNKS ; j++) {
#ifdef PWRITE
      pwrite(fd, data, BLOCKSIZE, BLOCKSIZE * j);
#else
      write(fd, data, BLOCKSIZE);
#endif
      printf("Chunk %d/%d\n", j, CHUNKS);
    }
    printf("Lap %d/%d\n", i, ITERATIONS);
    lseek(fd, 0, 0); // back to start (if using regular write, harmless otherwise);
  }

  close(fd);
  return 0;

}

In my BDB example I watched disk usage by gkrellm. Once the database had reached a certain size replication speed ground to a halt.

[DeHackEd]

Apparently if you put the databases on an ext4 filesystem on a zvol this doesn't happen. Turning the target of my test app on a zvol does eliminate the issue and the value of CHUNKS has to be big enough that it would overflow my write buffer size anyway.

[DeHackEd]

I think I'm just going to close this one as old and I haven't seen indications of it being an issue.