Dump raw guest VM memory feature

[desecnd]

Dump raw memory

Whilist researching the feature set of Drakvuf Sandbox at Poznan Supercomputing and Networking Center we identified that it lacked ability to postprocess raw guest memory with tools of our choice for further analysis. In order to do so, we've implemented optional functionality of fully-dumping RAM memory from guest VM. We aimed our solution for compatibility with Volatility3 framework.

Dumping process

We are using libvmi tool - vmi-dump-memory - which is currently shipped with drakvuf engine. The dump is performed twice:

• memory_dump_pre_sample.raw - after restoring the VM, before injecting sample.
• memory_dump_post_sample.raw - after finished analysis with drakvuf engine.

Both images are then stored under outdir in order to be uploaded to minio server. The whole process is wrapped in one function _memory_dump located in drakrun/main.py.

Volatility3 compatibility

We have encountered problems with RAM dumps made with xl dump-memory command line tool. Resulting memory dumps were stored in a custom Xen ELF format, which is not recognized by Volatility3 framework. vmi-dump-memory dumps memory in raw format, which works well with Volatility3.

Configuration

We've added raw_memory_dump boolean option, into drakrun/config.dist.ini, which is disabled by default. When disabled the code does not interfere into the analyzing process.

Development

This feature was successfully tested with guidelines specified in Sanbox Development

[chivay]

Hi thanks for the PR! Code-wise it looks good, however I have some questions/ideas I'd like to discuss:

1. Pre-sample memory dump looks redundant. It would be basically the same image as the snapshot file we're using to restore the VM. Uploading it over and over to MinIO seems wasteful. I think it would be better to use it instead of generating new ones each time.
2. Compression - from our experience, full memory dumps compress quite well (for example the 3G snapshot file has less than 500M after gzip). What do you think about adding this feature? Does it fit your use case?

[desecnd]

Hi, thanks for sharing your thoughts! I will try to address your questions : )

---

1. Idea of pre-sample at first was to have two RAM images in similar formats for easy comparison, but I agree, it is totally redundant to upload the same 4G file with every analysis. However, we'd like to have access to both pre-sample and post-sample dump with MinIO, but as post-sample would be analysis-dependent (stored in the drakrun bucket under uid of analysis), pre-sample could be stored in some "general" location.

We've came up with two theoretical ideas how we could deal with it. We'd be glad if you could address them or maybe put forward a better solution:

• Make the pre-sample dump during draksetup process. Just as snapshot.sav is saved, vmi-dump would be performed and uploaded to MinIO. This solution would require moving the code to draksetup, or/and making it visible from both parts. I'm also concerned about using minIO in draksetup, as most minIO config is done with drakrun and drakcore, but I will leave it to you to judge.

• When drakrun is starting, check MinIO for presence of pre-sample dump, and perform dump if file was not found. This would however require usage of MinIO outside of Karton interface (i think?) which is not preferred. There also comes the case of running draksetup postinstall and modyfing the snapshot which would result in different pre-sample image. Maybe embedding some kind of hash of the snapshot in the name of the file or config would help, but it seems a "hacky" solution.

What do you think?

---

2. We definitely agree, compression fits our use case and we will be adding it in next the commits. Currently we are testing which compression will be best in terms of time/space ratio. At this point We've reached best results with gzip as you mentioned before, and probably will stick with it : )

[chivay]

• Make the pre-sample dump during draksetup process. Just as snapshot.sav is saved, vmi-dump would be performed and uploaded to MinIO. This solution would require moving the code to draksetup, or/and making it visible from both parts. I'm also concerned about using minIO in draksetup, as most minIO config is done with drakrun and drakcore, but I will leave it to you to judge.

We already have a (though maybe a bit under documented) feature - snapshot exporting - which I think could be adapted to enable this. You can run draksetup snapshot export to see the usage. Basically, it stores the entire "state" in a MinIO bucket under a given name. I think we could extend it to upload not only Xen checkpoint file but also a raw memory image.

You can find the function here:

drakvuf-sandbox/drakrun/drakrun/draksetup.py

Line 1003 in 258a78a

def snapshot_export(name, bucket, full, force):

The downside of this approach is that currently we have no link between what's currently used by drakrun and the name of exported snapshot. But after fixing, I think that it should be usable. What do you think of this approach? 🤔

1. We definitely agree, compression fits our use case and we will be adding it in next the commits. Currently we are testing which compression will be best in terms of time/space ratio. At this point We've reached best results with gzip as you mentioned before, and probably will stick with it : )

That's cool, exporter also uses gzip so maybe we could reuse some parts of this code.

[desecnd]

We already have a (though maybe a bit under documented) feature - snapshot exporting - which I think could be adapted to enable this. You can run draksetup snapshot export to see the usage. Basically, it stores the entire "state" in a MinIO bucket under a given name. I think we could extend it to upload not only Xen checkpoint file but also a raw memory image.

Great, having such option would be cool, and we're definitely up for it : )

The downside of this approach is that currently we have no link between what's currently used by drakrun and the name of exported snapshot. But after fixing, I think that it should be usable. What do you think of this approach? 🤔

At this point we were thinking about having a dedicated bucket on MinIO for exported "pre-sample" (clean) memory dumps. "post-sample" images would be stored as it is now - in the analysis folder in the drakrun bucket. To resolve problems with version of used snapshot (matching corresponding pre-sample and post-sample dumps) I think the cleanest solution would be to include snapshot hash into the raw image names e.g. <snap-hash>_post_sample.raw_memdump and <snap-hash>_pre_sample.raw_memdump.

That's cool, exporter also uses gzip so maybe we could reuse some parts of this code.

Actually, during the last compression tests, we've found a solution that may suit our needs well. MinIO offers its own transparent compression. We've seen that it compresses our raw dumps to similar but slightly bigger size than gzip does:

• 1.6G - MinIO compression
• 1.4G - gzip -1
• 1.3G - gzip -9

But it comes first in terms of speed and usability. In our test process of uploading raw memory image to MinIO was as fast as running gzip for only compressing the file:

• ~1.5m - uploading file to a MinIO
• ~1.5m - gzip -1
• >5m - gzip -9

MinIO compression can be applied to any files, based on extension, not only the raw memory images. In our opinion it seems to be a convenient solution, taking into account both transparency and speed. It's easily configured, detailed guide can be found here: https://docs.min.io/docs/minio-compression-guide.html

We are willing to implement this solution and would be glad to hear what you think : )

[chivay]

Yes, I'm aware that MinIO has a feature of transparent compression. However it comes with a set of some drawbacks

1. it has to be enabled manually
2. it won't work for other S3-like storage providers
   So I think it would be more flexible to do the compression on our side.

Storing post analysis dumps together with the rest of analysis artifacts sounds good 👍

I think the cleanest solution would be to include snapshot hash into the raw image names e.g. _post_sample.raw_memdump and _pre_sample.raw_memdump.

Sounds OK, although I think we'd prefer enhancing the metadata.json file with a new key containing snapshot hash ;)

We are willing to implement this solution and would be glad to hear what you think : )

Great! If you have any questions feel free to contact us!

[desecnd]

At this point we've added 2 features:

• snapshot.sav sha256 hash was added to metadata.json file

  • hash is calculated in process function with other metadata information
  • helping function _file_sha256 was added

• dump mechanism was embedded with gzip compression

  • vmi-dump-memory is invoked with python tempfile
  • dumps are compressed with gzip subprocess and stored in outdir
  • pre-sample dumps were deleted in order to implement them with export clear-state image function

Currently we're working on exporting functionality. We'll be glad to hear what you think about commited changes : )

[desecnd]

draksetup memdump export

I've finished demo version of the exporter code,

• default bucket is presample-memdumps
• naming convention of saved dumps is <hash>_pre_sample.raw_memdump.gz

# draksetup memdump export --instance=1 --bucket="presample-memdumps"

Following exceptions has been tested:

• bucket does not exists
• file with that hash already exists in bucket
• vm with specified instance_id is already running

Whole execution chain (export presample-memdump -> download from minio -> decompress -> analyze using volatility3) was tested and seems to work properly.

One issue I've stumbled upon is get_minio_client in draksetup.py:

drakvuf-sandbox/drakrun/drakrun/draksetup.py

Lines 997 to 1004 in c4d6db3

	def get_minio_client(config):
	minio_cfg = config["minio"]
	return Minio(
	endpoint=minio_cfg["address"],
	access_key=minio_cfg["access_key"],
	secret_key=minio_cfg["secret_key"],
	secure=minio_cfg.getboolean("secure", fallback=True),
	)

Reading access parameters from default file results in empty strings and access denied, which might be misleading given config description and behavior of drakrun (reading credentials from minio.env):

drakvuf-sandbox/drakrun/drakrun/config.dist.ini

Lines 14 to 23 in c4d6db3

	; MinIO access credentials
	;
	; NOTE:
	; if this is empty, credentials will be read from /etc/drakcore/minio.env
	;
	; you only need to fill this out if you are:
	; * using external MinIO server
	; * using multi-node setup (drakcore is not on the same machine as drakrun)
	access_key=
	secret_key=

I wasn't sure if it is intended solution, so I thought I should bring this up.

I would be happy to hear your thoughts : )

[chivay]

Some small nitpicks, otherwise looks good :)

[chivay]

One issue I've stumbled upon is get_minio_client in draksetup.py:

drakvuf-sandbox/drakrun/drakrun/draksetup.py

Lines 997 to 1004 in c4d6db3

	def get_minio_client(config):
	minio_cfg = config["minio"]
	return Minio(
	endpoint=minio_cfg["address"],
	access_key=minio_cfg["access_key"],
	secret_key=minio_cfg["secret_key"],
	secure=minio_cfg.getboolean("secure", fallback=True),
	)

Reading access parameters from default file results in empty strings and access denied, which might be misleading given config description and behavior of drakrun (reading credentials from minio.env):

Yup, it's a bug. I've created an issue to track this - #632 .

[chivay]

Thanks!

[icedevml]

@pavveu7 Hello, many thanks for your contribution and interest in the project! 🚀 l

Please note that this approach assumes that at the end of the analysis, the VM is still alive and DRAKVUF engine has exited cleanly. This might not be true under some circumstances:

• the malware is leading VM to crash
• there is a bug in DRAKVUF engine that causes VM crash at the end of analysis

In such circumstances, the VM will be destroyed and you will not be able to make any memory dump. This behavior is controlled by on_poweroff/on_reboot/on_crash keys of xl.cfg:

https://github.com/CERT-Polska/drakvuf-sandbox/blob/master/drakrun/drakrun/scripts/cfg.template#L16-L18

Instead of destroy action, you can use preserve/pause here (I don't remember which one). Then, even if something goes really wrong, the VM resources will remain available for you to analyse, even if the VM crashed at the end.

I wonder how DRAKVUF engine would behave in combination with such mechanism, but it should be fine though (probably some minimal adjustments would be required).

This is a completely optional remark, but I think it might be useful to you.

[desecnd]

@icedevml hi, thanks for the remark! I really appreciate it : )

This is a case that should be definitely taken into account. I will investigate it further, many thanks for this note! : )