#
# Gdev - Managing GPUs as First-Class Computing Resources
# 
# README
#
# Copyright (C) Shinpei Kato
# University of California, Santa Cruz
# Systems Research Lab.
#
# All Rights Reserved.
#
#
# This project is in collaboration with PathScale Inc.
# Many interesting solutions for GPU programming can be found at:
# http://www.pathscale.com/
#

Gdev is a runtime-unified operating system module that manages GPUs 
as first-class computing resources.
Currently it supports only NVIDIA's Fermi GPUs, but the concept of
Gdev is also applicable to generic "compute devices".
Gdev coordinates with a DRM-based GPU device driver (pscnv/nouveau)
in the operating system, providing APIs for application programs.
Gdev API is a low-level primitive that allows programmers to control
the details of GPU resource parameters, while Gdev also supports a
high-level API, such as CUDA.
Gdev is available for GPGPU and graphics applications. It is self-
contained for GPGPU, though graphics applications require additional
packages, such as OpenGL, LIBDRM, and DDX.

Gdev is open-source. We believe that this open-source implementation
facilitates further research and development of GPU technology.

The following instruction will tell you how to install Gdev. Some of 
the installation stages may require you to install additional tools 
and packages.
$(TOPDIR) will represent your top working directory, henceforth.

1. Download

cd $(TOPDIR)
git clone git://0x04.net/envytools.git
git clone git://github.com/shinpei0208/gdev.git

2. envytools

envytools is a rich set of open-source tools to compile or decompile
NVIDIA GPU program code, firmware code, macro code, and so on. It is
also used to generate C header files with GPU command definitions.
In addition, envytools document the NVIDIA GPU architecture details,
while are not disclosed to the public so far. If you are interested 
in GPU system software development, this is what you should read!
Please follow the instruction below to install envytools.

cd $(TOPDIR)/envytools
mkdir build
cd build
cmake .. # may require some packages on your distro
make
sudo make install # will install tools to /usr/local/{bin,lib}

3. Device Driver

Recent versions of Gdev disgregate from the device driver. You need
to install some GPU device driver underlying Gdev. Currently there
are two open-source drivers available in Linux: pscnv and nouveau.
The following shows how to install pscnv.
NOTE1: pscnv is PathScale's open-source driver, but we have applied
several patches, making it available for Gdev.
NOTE2: pscnv is available with Linux 2.6.33 or later. It may also 
have issues with some Fermi chipsets - nvc0 is best tested.
See http://nouveau.freedesktop.org/wiki/CodeNames for chipset IDs.

3.1. When you use pscnv

cd $(TOPDIR)/gdev/mod/pscnv
./configure
make
sudo make modules_install
sudo shutdown -r now # will reboot your machine

3.2. When you use nouveau

cd $(TOPDIR)/gdev/mod/nouveau
make NOUVEAUROOTDIR=. # if make files, try Step 3.3.
sudo make install NOUVEAUROOTDIR=.
cd drivers/gpu/drm/nouveau
sudo sh gdev.sh
sudo shutdown -r now # will reboot your machine

If you are not sure if Nouveau is loaded successfully, do:
modprobe -r nouveau; modprobe nouveau modeset=1 noaccel=0

3.3. When you use very latest kernels

cd $(TOPDIR)
git clone --depth 1 git://anongit.freedesktop.org/nouveau/linux-2.6
cd linux-2.6
git remote add nouveau git://anongit.freedesktop.org/nouveau/linux-2.6
git remote update
git checkout -b nouveau-master nouveau/master
patch -p0 < $(TOPDIR)/gdev/mod/patches/gdev-nouveau.patch 
make menuconfig
make 
sudo make modules_install install
sudo shutdown -r now # will reboot your machine

4. Gdev Module

Gdev is double-edge, i.e., it provides a runtime-unified operating
system approach as well as a typical user-space runtime approach.
The following installation is required only for the former.

cd $(TOPDIR)/gdev/mod
mkdir build
cd build
../configure
make
sudo insmod gdev.ko
sudo sh gdev.sh

5. Gdev Library

Gdev's user-space library provides Gdev API. This API can be used
by either user programs directly or another high-level API library.
For instance, third party's CUDA library can use Gdev API.
If you have taken Step 4, i.e., chosen the runtime-unified operating
system approach, this library is just a set of wrapper functions
that call Gdev module's functions via ioctl.

cd $(TOPDIR)/gdev/lib
mkdir build
cd build
../configure # if you skipped Step 4, you must specify --target=user
make
sudo make install
export LD_LIBRARY_PATH="/usr/local/gdev/lib64:$LD_LIBRARY_PATH"
export PATH="/usr/local/gdev/bin:$PATH"

6. CUDA Driver API

Gdev currently supports a limited set of CUDA Driver API. We plan to 
support a full set of CUDA Driver API in future work. If you need
CUDA Runtime API, you should use some compiler framework, such as
Ocelot, which can translate CUDA Drier API to Runtime API.

cd $(TOPDIR)/gdev/cuda
mkdir build
cd build
../configure
make
sudo make install

Gdev also supports CUDA in the operating system. You are required to
install "kcuda" module to use this functionality.

cd $(TOPDIR)/gdev/cuda
mkdir kbuild
cd kbuild
../configure --target=kcuda
make
sudo make install

7. CUDA Driver API test (user-space programs)

cd $(TOPDIR)/test/cuda/user/madd
make
./user_test 256 # a[256] x b[256] = c[256]

8. CUDA Driver API test (OS-space programs)

cd $(TOPDIR)/test/cuda/kernel/memcpy
make
sudo insmod ./kernel_test.ko size=10000 # copy 0x10000 size

NOTE: Please be careful when doing this test as it runs a program
in module_init(). If you run a very long program as it is, you may
crash your system. If you want to run a very long program, you must
provide a proper module implementation, e.g., using kernel threads.

9. Reclock

NVIDIA's graphics cards are set very low clocks by default. To get
performance, you need to reclock your card at the maximum level.
How? Be root first, and then echo 3 to the following file:

echo 3 > /sys/class/drm/card0/device/performance_level

You can downclock your card by echoing 0 to the same file, i.e.,

echo 0 > /sys/class/drm/card0/device/performance_level

There are middleground levels 1 and 2, too. Note that Reclocking
is not completely supported by the open-source solution yet.
There are still some performance levels missing, and hence you may
not get as high performance as the blob. If you really need the same
level of performance as the blob, you can run some long-running
CUDA program with the blob, and do kexec -f your kernel before the
program is finished. Then the clock remains at the maximum level.