"faces" dataset: how to create hdf5 from images?

[FeepingCreature]

Given a set of pictures, it's unclear how to create a correctly formatted dataset. I understand if the pictures used cannot be made available, but could you please share whatever script or method you used to create the hdf5 dataset used as an input?

Thanks.

[udibr]

I've created a command line tool for converting gzipped tar of jpeg images to fuel's hdf5 file.
This is how to use it. Install my fork of fuel https://github.com/udibr/fuel instead of the official fuel.

git clone https://github.com/udibr/fuel.git
cd fuel
python setup.py install

next create a directory (lets call it inputdir) and this directory you should have
a file called train.tar.gz which contains all your images in jpg format.
All the images should be already in the same size. for example 64x64x3.

also create a directory in which you want the hdf5 file to be created (lets call it outputdir)

next run fuel converter to transform this directory to hdf5

fuel-convert jpgtgz -d inputdir -o outputdir

If everything goes well you should find the file jpg.hdf5 in outputdir

there is one problem with the order in which the color, width and height indexes are ordered.
I modified the transform function at https://github.com/Newmu/dcgan_code/blob/master/faces/train_uncond_dcgan.py#L28
to look like:

def transform(X):
    # X = [center_crop(x, npx) for x in X]  # only works for (H,W,3)
    assert X[0].shape == (npx,npx,3) or X[0].shape == (3,npx,npx)
    if X[0].shape == (npx,npx,3):
        X = X.transpose(0, 3, 1, 2)
    return floatX(X/127.5 - 1.)

there are more options.
for more details read the code at https://github.com/udibr/fuel/blob/master/fuel/converters/jpgtgz.py

one big problem is that for now the hdf5 file is about 8 times bigger than the train.tar.gz file.
I am working on a better solution.

[soumith]

@udibr if you want to take a look at my solution: https://github.com/soumith/dcgan.torch
I created a generic multi-threaded data loader for a folder of images. You give the data loader info on loadSize and cropSize and it takes care of the rest. I've used it for a variety of datasets including imagenet, and it's nice and easy to use. It mostly leverages gnu utils like find, cut, wc etc.

[dribnet]

Hi @FeepingCreature - if you'd like I can release code as a sharable fuel plugin to do this for the celebA dataset. I've done this for LFW, but that dataset format is different because it exports face images as pairs to be compared. I can also offer a merge request to dcgan_code that will cause the celebA hdf5 file to be downloaded and used automatically via kerosene if there is general consensus that this would be useful.