Skip to content

Commit

Permalink
config/libvirt/cache: Decompress when URI has ".gz" suffix
Browse files Browse the repository at this point in the history
Passing the compressed images to libvirt was giving me "no bootable
device" errors with my:

  $ rpm -q libvirt qemu-kvm-rhev
  libvirt-3.9.0-14.el7_5.7.x86_64
  qemu-kvm-rhev-2.9.0-16.el7_4.13.x86_64

Ideally, the HTTP headers would tell us that the image was compressed
(via Content-Type [1] or Content-Encoding [2]), but
aos-ostree.rhev-ci-vms.eng.rdu2.redhat.com isn't configured to do that
at the moment:

  $ curl -I http://aos-ostree.rhev-ci-vms.eng.rdu2.redhat.com/rhcos/images/cloud/latest/rhcos-qemu.qcow2.gz
  HTTP/1.1 200 OK
  Server: nginx/1.8.0
  Date: Sat, 22 Sep 2018 04:59:53 GMT
  Content-Type: application/octet-stream
  Content-Length: 684751099
  Last-Modified: Fri, 21 Sep 2018 13:56:07 GMT
  Connection: keep-alive
  ETag: "5ba4f877-28d078fb"
  Accept-Ranges: bytes

  $ curl -I http://aos-ostree.rhev-ci-vms.eng.rdu2.redhat.com/rhcos/images/cloud/latest/rhcos-qemu.qcow2
  HTTP/1.1 404 Not Found
  Server: nginx/1.8.0
  Date: Sat, 22 Sep 2018 04:59:56 GMT
  Content-Type: text/html
  Content-Length: 168
  Connection: keep-alive

I've opened [3] about getting Content-Encoding support via gzip_static
[4], but in the meantime, just assume that anything with a ".gz"
suffix is gzipped.

Unzipping is also somewhat expensive (although not as expensive as
network requests).  It would be nice for callers to be able to
configure whether we cache the compressed images (less disk
consumption) or the uncompressed images (less CPU load when launching
clusters).  For now, we're just caching the network response.

[1]: https://tools.ietf.org/html/rfc7231#section-3.1.1.5
[2]: https://tools.ietf.org/html/rfc7231#section-3.1.2.2
[3]: https://projects.engineering.redhat.com/browse/COREOS-593
[4]: http://nginx.org/en/docs/http/ngx_http_gzip_static_module.html
  • Loading branch information
wking committed Sep 22, 2018
1 parent 8993664 commit 7abe94d
Showing 1 changed file with 12 additions and 1 deletion.
13 changes: 12 additions & 1 deletion pkg/types/config/libvirt/cache.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package libvirt

import (
"compress/gzip"
"fmt"
"io"
"io/ioutil"
Expand Down Expand Up @@ -51,6 +52,16 @@ func (libvirt *Libvirt) UseCachedImage() (err error) {
}
defer resp.Body.Close()

var reader io.Reader
if strings.HasSuffix(libvirt.Image, ".gz") {
reader, err = gzip.NewReader(resp.Body)
if err != nil {
return err
}
} else {
reader = resp.Body
}

// FIXME: diskcache's diskv backend doesn't expose direct file access.
// We can write our own cache implementation to get around this,
// but for now just dump this into /tmp without cleanup.
Expand All @@ -60,7 +71,7 @@ func (libvirt *Libvirt) UseCachedImage() (err error) {
}
defer file.Close()

_, err = io.Copy(file, resp.Body)
_, err = io.Copy(file, reader)
if err != nil {
return err
}
Expand Down

0 comments on commit 7abe94d

Please sign in to comment.