Sort out models, data, and tools

[shelhamer]

1. Re-arrange dirs for cleanliness.
2. Move learned models and data to caffe-mug repo dropbox for now (and a suitable server later).
3. Provide scripts to download models and data as needed.
4. Ignore models and data in the main repo by default. This makes local experimentation convenient.
5. Keep model definitions around.

Orchestrating updates between commits in caffe and uploaded models and data has overhead, but is worth the separation of concerns and keeping the repo lean.

[sergeyk]

Decided to have a separate repository for data, model definition files, and example models, with scripts in this repo to download them as needed. The separate repo will be stable against master, not dev. @shelhamer will do this.

[mavenlin]

@sergeyk that will be nice, currently the repo is big and slow to download because of the synsets. Can these be removed from the git history also?

[shelhamer]

@mavenlin the synsets are on my list for this reorganization. Filtering them from the history is necessary to save space, and a simple command, but it breaks history.

We'll consider such house cleaning when it comes time to release Caffe 1.0, but we're not going to rewrite history on master so casually now.

[shelhamer]

@sergeyk I added you as a collaborator on my fork so that we can jointly take care of the documentation updates trigged by this PR. All you have to do is push to my fork's data-aux branch.

[shelhamer]

Instead of scripts to pull models and data, a mug submodule for data and models was considered but this lacks choice: with a submodule it's all or nothing.

[shelhamer]

This isn't going to work with github's (not unreasonable) file size and traffic limitations. git's a drag with large binary files too, so perhaps it's best.

The alternative is self-hosting from campus or ICSI.

[sergeyk]

Let's host as many models, sample data, and model def files as possible in
a github repo. For really large models, we can upload them to a publicly
accessible ICSI place, and note the version (just the date, probably) in
the filename.

On Tue, Feb 25, 2014 at 12:59 AM, Evan Shelhamer
[email protected]:

This isn't going to work with github's (not unreasonable) file size and
traffic limitations. git's a drag with large binary files too, so perhaps
it's best.

The alternative is self-hosting from campus or ICSI.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/142#issuecomment-35987363
.

[shelhamer]

Oh, sorry I wasn't clear. This isn't going to work at all. Not even the Caffe reference imagenet model fits on its own as there's a filesize cap of 100mb.

My fallback plan is ICSI hosting and versioning the fetch urls of the scripts in master. We can have a simple script to publish the models, defs, and data into a dir based on the timestamp.

[sergeyk]

I know that the reference imagenet model won't fit. I still think that prototxt files and small sample data should be hosted on github -- everything that fits under 100mb, which is gonig to be basically everything except imagenet models.

[shelhamer]

Resolution: keep model definitions in the repo, drop included data, and add scripts to fetch learned models and data as needed. Auxiliary data and model weights will live on dropbox for the moment, and will find their permanent home on a Berkeley server after March 7. Our group will be bringing a demo server online after that date which can hold the data.

[shelhamer]

This feels ready to me, modulo fixing the docs changes this triggers. @sergeyk, how about we update this once #155 is in?

[kloudkl]

It seemed not a high priority to have a demo (#78). May I ask what demo will the server host?

[shelhamer]

Any suggestions on the dir structure or names are welcome–this is the time to arrange everything neatly.

@kloudkl re: demo, there will in fact be a Caffe demo along the lines of the DeCAF demo, and along with it other demos of our research group's projects. @Yangqing was not against a demo so much as spending too much time engineering a simple illustration of the framework and not focusing on the research hacking.

[sguada]

What about this dir structure?

• data/ //Contains one folder per dataset. Each folder can have a script to get the data
  • mnist/
  • cifar10/
  • ilsvrc_2012/
  • ...
• tools/ //Contains the main caffe tools for training, testing and finetunning, net_speed, ...
  • train_net.cpp
  • test_net.cpp
  • fine_tuninng.cpp
  • ...
• models/ //Contains the different protxt defining the models
• docs/ //Contains the documentation
• examples/ //Could contain some samples of uses of caffe, but should mix everything

[shelhamer]

That looks right, but I'm torn about how examples fit in. Packing example code, model, and data together makes the example clear, but reuse weird. I'll package purely example files up together, but keep data on its own.

[shelhamer]

Ok everyone, feast your eyes and let me know. Speak now or forever hold your peace.

@Yangqing @jeffdonahue @sergeyk @sguada @longjon

[jeffdonahue]

Looks great to me! Thanks for the reorganization work @shelhamer.

[sguada]

It looks good but I cannot compile it due the hdf5 dependency introduced in #147
So not sure if it will work or not.

There are some small error in the get_data.sh
./get_ilsvrc_aux.sh: 9: ./get_ilsvrc_aux.sh: Bad substitution
./get_mnist.sh: 4: ./get_mnist.sh: Bad substitution
./get_cifar10.sh: 4: ./get_cifar10.sh: Bad substitution

[shelhamer]

Perhaps we should shortlist this for master like Sergey's #157 since this changes a lot and everything should be merged into the new arrangement and not the old.

dev and feature branches will need to be rebased, so we should incorporate this, Jeff's #163 and Eric's #152 into master then rebase dev all at once.

[sguada]

That will solve the problems for now. We still need to figure out a way
either make hdf5 optional or to explain how to install it properly.

Sergio

2014-02-25 19:09 GMT-08:00 Evan Shelhamer [email protected]:

Perhaps we should merge this to master instead of dev like Sergey's #157https://github.com/BVLC/caffe/pull/157since a lot depends on this change. Then it would work cleanly for whoever
doesn't have hdf5 for now too.

dev and feature branches would need to be rebased, so we should
incorporate this, Jeff's #163 https://github.com/BVLC/caffe/pull/163and Eric's
#152 #152 into master then rebase
dev all at once.

Reply to this email directly or view it on GitHubhttps://github.com//pull/142#issuecomment-36086428
.

[sergeyk]

I'm for merging this one and #163 into master directly, but not #152, as it does not currently exist and is an involved code change.

[sergeyk]

Looks good to me, we'll fix potential mistakes once we merge.

[shelhamer]

@sguada there's no script get_data.sh? I tested the fetch scripts on osx and ubuntu.

@sergeyk. Agreed on all counts. I'll merge soon.

[sguada]

@shelhamer I meant all the get_ilsvrc_aux.sh, get_mnist.sh
But maybe it is just me.

[sguada]

@shelhamer beside that it is great, ready to merge

[shelhamer]

@sguada you might have some kind of weird shell. Check which sh perhaps?