Merge branch 'core3-storage3'

doc/index.rst

@@ -16,6 +16,7 @@ manpages and API documentation. For primary user documentation, see
    qubes
    qubes-vm/index
    qubes-events
+   qubes-storage
    qubes-exc
    qubes-ext
    qubes-log

doc/qubes-storage.rst

@@ -0,0 +1,145 @@
+:py:mod:`qubes.storage` -- Qubes data storage
+=============================================
+
+Qubes provide extensible API for domains data storage. Each domain have
+multiple storage volumes, for different purposes. Each volume is provided by
+some storage pool. Qubes support different storage pool drivers, and it's
+possible to register additional 3rd-party drivers.
+
+Domain's storage volumes:
+
+ - `root` - this is where operating system is installed. The volume is
+   available read-write to :py:class:`~qubes.vm.templatevm.TemplateVM` and
+   :py:class:`~qubes.vm.standalonevm.StandaloneVM`, and read-only to others
+   (:py:class:`~qubes.vm.appvm.AppVM` and :py:class:`~qubes.vm.dispvm.DispVM`).
+ - `private` - this is where domain's data live. The volume is available
+   read-write to all domain classes (including :py:class:`~qubes.vm.dispvm.DispVM`,
+   but data written there is discarded on domain shutdown).
+ - `volatile` - this is used for any data that do not to persist. This include
+   swap, copy-on-write layer for `root` volume etc.
+ - `kernel` - domain boot files - operating system kernel, initial ramdisk,
+   kernel modules etc. This volume is provided read-only and should be provided by
+   a storage pool respecting :py:attr:`qubes.vm.qubesvm.QubesVM.kernel` property.
+
+Storage pool concept
+--------------------
+
+Storage pool is responsible for managing its volumes. Qubes have defined
+storage pool driver API, allowing to put domains storage in various places. By
+default two drivers are provided: :py:class:`qubes.storage.file.FilePool`
+(named `file`) and :py:class:`qubes.storage.lvm.ThinPool` (named `lvm_thin`).
+But the API allow to implement variety of other drivers (like additionally
+encrypted storage, external disk, drivers using special features of some
+filesystems like btrfs, etc).
+
+Most of storage API focus on storage volumes. Each volume have at least those
+properties:
+ - :py:attr:`~qubes.storage.Volume.rw` - should the volume be available
+   read-only or read-write to the domain
+ - :py:attr:`~qubes.storage.Volume.snap_on_start` - should the domain start
+   with its own state of the volume, or rather a snapshot of its template volume
+   (pointed by a :py:attr:`~qubes.storage.Volume.source` property). This can be
+   set to `True` only if a domain do have `template` property (AppVM and DispVM).
+   If the domain's template is running already, the snapshot should be made out of
+   the template's before its startup.
+ - :py:attr:`~qubes.storage.Volume.save_on_stop` - should the volume state be
+   saved or discarded on domain
+   stop. In either case, while the domain is running, volume's current state
+   should not be committed immediately. This is to allow creating snapshots of the
+   volume's state from before domain start (see
+   :py:attr:`~qubes.storage.Volume.snap_on_start`).
+ - :py:attr:`~qubes.storage.Volume.revisions_to_keep` - number of volume
+   revisions to keep. If greater than zero, at each domain stop (and if
+   :py:attr:`~qubes.storage.Volume.save_on_stop` is `True`) new revision is saved
+   and old ones exceeding :py:attr:`~qubes.storage.Volume.revisions_to_keep` limit
+   are removed.
+ - :py:attr:`~qubes.storage.Volume.source` - source volume for
+   :py:attr:`~qubes.storage.Volume.snap_on_start` volumes
+ - :py:attr:`~qubes.storage.Volume.vid` - pool specific volume identifier, must
+   be unique inside given pool
+ - :py:attr:`~qubes.storage.Volume.pool` - storage pool object owning this volume
+ - :py:attr:`~qubes.storage.Volume.name` - name of the volume inside owning
+   domain (like `root`, or `private`)
+ - :py:attr:`~qubes.storage.Volume.size` - size of the volume, in bytes
+
+Storage pool driver may define additional properties.
+
+Storage pool driver API
+-----------------------
+
+Storage pool driver need to implement two classes:
+ - pool class - inheriting from :py:class:`qubes.storage.Pool`
+ - volume class - inheriting from :py:class:`qubes.storage.Volume`
+
+Pool class should be registered with `qubes.storage` entry_point, under the
+name of storage pool driver. Volume class instances should be returned by
+:py:meth:`qubes.storage.Pool.init_volume` method of pool class instance.
+
+Methods required to be implemented by the pool class:
+ - :py:meth:`~qubes.storage.Pool.init_volume` - return instance of appropriate
+   volume class; this method should not alter any persistent disk state, it is
+   used to instantiate both existing volumes and create new ones
+ - :py:meth:`~qubes.storage.Pool.setup` - setup new storage pool
+ - :py:meth:`~qubes.storage.Pool.destroy` - destroy storage pool
+
+Methods and properties required to be implemented by the volume class:
+ - :py:meth:`~qubes.storage.Volume.create` - create volume on disk
+ - :py:meth:`~qubes.storage.Volume.remove` - remove volume from disk
+ - :py:meth:`~qubes.storage.Volume.start` - prepare the volume for domain start;
+   this include making a snapshot if
+   :py:attr:`~qubes.storage.Volume.snap_on_start` is `True`
+ - :py:meth:`~qubes.storage.Volume.stop` - cleanup after domain shutdown; this
+   include committing changes to the volume if
+   :py:attr:`~qubes.storage.Volume.save_on_stop` is `True`
+ - :py:meth:`~qubes.storage.Volume.export` - return a path to be read to extract
+   volume data; for complex formats, this can be a pipe (connected to some
+   data-extracting process)
+ - :py:meth:`~qubes.storage.Volume.import_data` - return a path the data should
+   be written to, to import volume data; for complex formats, this can be pipe
+   (connected to some data-importing process)
+ - :py:meth:`~qubes.storage.Volume.import_data_end` - finish data import
+   operation (cleanup temporary files etc); this methods is called always after
+   :py:meth:`~qubes.storage.Volume.import_data` regardless if operation was
+   successful or not
+ - :py:meth:`~qubes.storage.Volume.import_volume` - import data from another volume
+ - :py:meth:`~qubes.storage.Volume.resize` - resize volume
+ - :py:meth:`~qubes.storage.Volume.revert` - revert volume state to a given revision
+ - :py:attr:`~qubes.storage.Volume.revisions` - collection of volume revisions (to use
+   with :py:meth:`qubes.storage.Volume.revert`)
+ - :py:meth:`~qubes.storage.Volume.is_dirty` - is volume properly committed
+   after domain shutdown? Applies only to volumes with
+   :py:attr:`~qubes.storage.Volume.save_on_stop` set to `True`
+ - :py:meth:`~qubes.storage.Volume.is_outdated` - have the source volume started
+   since domain startup? applies only to volumes with
+   :py:attr:`~qubes.storage.Volume.snap_on_start` set to `True`
+ - :py:attr:`~qubes.storage.Volume.config` - volume configuration, this should
+   be enough to later reinstantiate the same volume object
+ - :py:meth:`~qubes.storage.Volume.block_device` - return
+   :py:class:`qubes.storage.BlockDevice` instance required to configure volume in
+   libvirt
+
+Some storage pool drivers can provide limited functionality only - for example
+support only `volatile` volumes (those with
+:py:attr:`~qubes.storage.Volume.snap_on_start` is `False`,
+:py:attr:`~qubes.storage.Volume.save_on_stop` is `False`, and
+:py:attr:`~qubes.storage.Volume.rw` is `True`). In that case, it should raise
+:py:exc:`NotImplementedError` in :py:meth:`qubes.storage.Pool.init_volume` when
+trying to instantiate unsupported volume.
+
+Note that pool driver should be prepared to recover from power loss before
+stopping a domain - so, if volume have
+:py:attr:`~qubes.storage.Volume.save_on_stop` is `True`, and
+:py:meth:`qubes.storage.Volume.stop` wasn't called, next
+:py:meth:`~qubes.storage.Volume.start` should pick up previous (not committed)
+state.
+
+See specific methods documentation for details.
+
+Module contents
+---------------
+
+.. automodule:: qubes.storage
+   :members:
+   :show-inheritance:
+
+.. vim: ts=3 sw=3 et

qubes/api/admin.py

@@ -275,7 +275,7 @@ def vm_volume_info(self):
         volume = self.dest.volumes[self.arg]
         # properties defined in API
         volume_properties = [
-            'pool', 'vid', 'size', 'usage', 'rw', 'internal', 'source',
+            'pool', 'vid', 'size', 'usage', 'rw', 'source',
             'save_on_stop', 'snap_on_start']
         return ''.join('{}={}\n'.format(key, getattr(volume, key)) for key in
             volume_properties)

qubes/app.py

@@ -28,6 +28,7 @@
 import logging
 import os
 import random
+import subprocess
 import sys
 import tempfile
 import time
@@ -548,6 +549,46 @@ def get_new_unused_dispid(self):
             'https://xkcd.com/221/',
             'http://dilbert.com/strip/2001-10-25')[random.randint(0, 1)])
 
+def _default_pool(app):
+    ''' Default storage pool.
+
+    1. If there is one named 'default', use it.
+    2. Check if root fs is on LVM thin - use that
+    3. Look for file-based pool pointing /var/lib/qubes
+    4. Fail
+    '''
+    if 'default' in app.pools:
+        return app.pools['default']
+    else:
+        rootfs = os.stat('/')
+        root_major = (rootfs.st_dev & 0xff00) >> 8
+        root_minor = rootfs.st_dev & 0xff
+        for pool in app.pools.values():
+            if pool.config.get('driver', None) != 'lvm_thin':
+                continue
+            thin_pool = pool.config['thin_pool']
+            thin_volumes = subprocess.check_output(
+                ['lvs', '--select', 'pool_lv=' + thin_pool,
+                '-o', 'lv_kernel_major,lv_kernel_minor', '--noheadings'])
+            if any((str(root_major), str(root_minor)) == thin_vol.split()
+                    for thin_vol in thin_volumes.splitlines()):
+                return pool
+        # not a thin volume? look for file pools
+        for pool in app.pools.values():
+            if pool.config.get('driver', None) != 'file':
+                continue
+            if pool.config['dir_path'] == '/var/lib/qubes':
+                return pool
+        raise AttributeError('Cannot determine default storage pool')
+
+def _setter_pool(app, prop, value):
+    if isinstance(value, qubes.storage.Pool):
+        return value
+    try:
+        return app.pools[value]
+    except KeyError:
+        raise qubes.exc.QubesPropertyValueError(app, prop, value,
+            'No such storage pool')
 
 class Qubes(qubes.PropertyHolder):
     '''Main Qubes application
@@ -629,6 +670,27 @@ class Qubes(qubes.PropertyHolder):
     default_dispvm = qubes.VMProperty('default_dispvm', load_stage=3,
         doc='Default DispVM base for service calls')
 
+    default_pool = qubes.property('default_pool', load_stage=3,
+        default=_default_pool,
+        doc='Default storage pool')
+
+    default_pool_private = qubes.property('default_pool_private', load_stage=3,
+        default=lambda app: app.default_pool,
+        doc='Default storage pool for private volumes')
+
+    default_pool_root = qubes.property('default_pool_root', load_stage=3,
+        default=lambda app: app.default_pool,
+        doc='Default storage pool for root volumes')
+
+    default_pool_volatile = qubes.property('default_pool_volatile',
+        load_stage=3,
+        default=lambda app: app.default_pool,
+        doc='Default storage pool for volatile volumes')
+
+    default_pool_kernel = qubes.property('default_pool_kernel', load_stage=3,
+        default=lambda app: app.default_pool,
+        doc='Default storage pool for kernel volumes')
+
     # TODO #1637 #892
     check_updates_vm = qubes.property('check_updates_vm',
         type=bool, setter=qubes.property.bool,
@@ -662,7 +724,7 @@ def __init__(self, store=None, load=True, offline_mode=None, lock=False,
         else:
             self._store = os.environ.get('QUBES_XML_PATH',
                 os.path.join(
-                    qubes.config.system_path['qubes_base_dir'],
+                    qubes.config.qubes_base_dir,
                     qubes.config.system_path['qubes_store_filename']))
 
         super(Qubes, self).__init__(xml=None, **kwargs)
@@ -907,6 +969,8 @@ def load_initial_values(self):
         for name, config in qubes.config.defaults['pool_configs'].items():
             self.pools[name] = self._get_pool(**config)
 
+        self.default_pool_kernel = 'linux-kernel'
+
         self.domains.add(
             qubes.vm.adminvm.AdminVM(self, None, label='black'))
 

qubes/config.py

@@ -36,8 +36,6 @@
     'qrexec_client_path': '/usr/lib/qubes/qrexec-client',
     'qubesdb_daemon_path': '/usr/sbin/qubesdb-daemon',
 
-    'qubes_base_dir': qubes_base_dir,
-
     # Relative to qubes_base_dir
     'qubes_appvms_dir': 'appvms',
     'qubes_templates_dir': 'vm-templates',