Extend ImageGenerator to multilabel case

[tholor]

Problem:
So far the method flow_from_directory of ImageDataGenerator requires an image to belong to one class (the name of the folder it's stored in). For a multilabel problem (e.g. images containing multiple objects) this convenient approach does not work. The alternatives are also not really feasible: e.g. reading the images into an array and using the flow function results in heavy memory loads and a slow read-in process.

Solution:
I have extended the DirectoryIterator to the multilabel case (one image belongs to multiple classes). Each image is still stored in one subfolder, but can be associated with multiple classes. A dictionary containing the fileurl and the associated labels (numpy array) can be passed to the DirectoryIterator (e.g. {'folder/some_image.jpg': np.array([ 0., 1., 0., 1.]),...})

Alternative approaches were briefly discussed on stackoverflow: http://stackoverflow.com/questions/2003505/how-to-delete-a-git-branch-both-locally-and-remotely

[fchollet]

I think this may be a useful feature, but the proposed API is too ad-hoc.

• A class, when specified by the user, should be an integer, not a binary Numpy array
• It should be possible to yield either integer labels or categorical binary labels

Additionally the fact that the current iterator relies on directory structure to infer classes may make it fundamentally incompatible with multi labeling. Maybe what we need is a different generator, with a more appropriate API for this use case, not an extension of the current generator.

[Dref360]

Please provide some unit tests.

[tholor]

A class, when specified by the user, should be an integer, not a binary Numpy array
It should be possible to yield either integer labels or categorical binary labels

Makes sense. I will adjust it.

I am not sure though, if it will be helpful to create a separate generator without the directory logic. In my use case, I started from a binary problem and extended it to a multi label problem by manually labeling ~ 20% of the images. It was still very convenient to store the images in separate folders, since it allowed:

• to infer the single class for 80% of the images
• to easily check some exemplary pictures of a class in the file system

But if you think it's more helpful to separate both approaches, I will change the code accordingly.

[fchollet]

If the class of an image is tied to a directory, since each image can only belong to one directory it makes sense to stick to one class per image.

If we want a mapping from one image to a set of classes, we may be better off using something like TFRecords... Store each image as JPEG data + class annotations, in a same file.

[fchollet]

Overall, I think we should come up with a different design to support multi-label problems. Relying on directories as a labeling mechanism will fundamentally only work neatly for single-label problems.

[tholor]

I will look into TFRecords. I like the idea of not relying on the directory structure. For our use case it is important though that also other applications have still straightforward access to the images in a standard format (png / jpgeg ...). Let's see if this is feasible with something like TFRecords.

[tholor]

As I saw there are already ongoing efforts to integrate TFRecords as an input format:

• ImageDataGenerator.flow() with tf tensor of images/labels, ex: TFRecord #5356: Discussing options to use TFrecords in ImageDataGenerator.flow()
• [WIP] initial TensorFlow TFRecord integration with Keras keras-contrib#27: Proposal for feeding tensors from TFrecords using adjusted functions of fit() and compile()
• TFRecord integration with Keras API tensorflow/tensorflow#8787: Extending Estimator API in tensorflow to supply TFRecords to Keras
  I would propose to wait for the successful integrations of TFRecords and then generalize it to multi-label problems.

[ahundt]

@tholor I created some hypothetical example code that solves this in #6891 (comment). Also related to the TFRecords comments: #6538

[tholor]

Nice! Do I understand you right here? You suggest to:

1. read in TFRecords containing multilabels with datainput = keras.RecordInput("*.tfrecord")
2. transform it to the DataSet currently discussed in Fix the ordering bugs when using pickle_safe=True #6891 and split it into labels and images with

dataset = keras.Dataset(datainput, ordering=['shuffle', 'parallel'])(datainput)
labels = dataset.get_feature('labels')
images = dataset.get_feature('images')

3. feed the labels as outputs parameter into keras.Model

Especially 3) is not 100% clear to me.

[ahundt]

So that linked comment was just a hypothetical design, it does not exist. Since you only need multi class labels I think you can directly try actual code in #6928 for TFRecords, tfrecords_mnist.py is an example.

That shows how to create a TFRecord and read it (assuming you've merged the pull request), then train on mnist & test. You'll need to figure out the way to save your label values as a vector but it should be quite similar to what's there now. If you don't feel like looking around just copy + modify the lines that save mnist images so they save the labels and set the dimensions.

Keep in mind it is super fresh code & needs review. If you give it a try I'd appreciate a review/testing/fixes!

[myurasov]

I quite like the PR – often you're given a bunch of pics + CSV with filename vs tags – those you can feed directly. TFRecord is another thing to learn...