Where could I get the database and testInfant_list.txt?

[John1231983]

Thank you for sharing your code.

I will download it but I would like to ask you something about database and files txt.
In your code, you used training data (raw+ground truth). In your paper, you said it done by manual segmentation from IBEAT. Is it possible to publish the database? Or how to register it?
If all ways are not possible, could you create it by HDF5Data and share with us? I think HDF5Data will be fine.
Finally, I saw some txt file such as testInfant_list.txt. I think it contains the path of dataset. Could you also provide them for us

Thank you so much

[ginobilinie]

Hi, thank you for your interest.

I donot own the dataset, thus I cannot publish the database now. I'll try to upload some hdf5 after miccai ddl.

Yes, the testInfant_list.txt is the generated text file which contains the path to the hdf5.

[John1231983]

Thanks. I think I will use the IBSR dataset at https://www.nitrc.org/frs/?group_id=48
It contains 20 peoples in each IBSR_* folder. For example the IBSR_06 folder, we have IBSR_06_ana_strip.nii.gz for raw image and IBSR_06_seg_ana.nii.gz is for manual segmentation.
I read your README. It is very long and redundancy for me. I just want to run some step as follows:

• Step 1: Convert the nii files in these folder to HDF5 dataset. Assume that, I divided two group testing and training groups. Each groups contains image and label folders. I think the output maybe like your_train_data.hdf5 and your_test_data.hdf5
• Step 2: After that, I will run the training process. I think you made a good script that is trainCaffe.py. I think what I have to do is that changes the path from line 12 to line 22. Is it right?

For step 1, what sould I do to generate the hdf5 files( or which files should I run)? Assume that my input is
├── Testing
│   ├── Label Images
│   │   ├── IBSR_06_segTRI_ana.nii.gz
│   │   └── IBSR_08_segTRI_ana.nii.gz
│   └── Raw Images
│   ├── IBSR_06_ana_strip.nii.gz
│   └── IBSR_08_ana_strip.nii.gz
└── Training
├── Label Images
│   ├── IBSR_01_segTRI_ana.nii.gz
│   ├── IBSR_02_segTRI_ana.nii.gz
│   ├── IBSR_03_segTRI_ana.nii.gz
│   ├── IBSR_04_segTRI_ana.nii.gz
│   ├── IBSR_05_segTRI_ana.nii.gz
│   └── IBSR_07_segTRI_ana.nii.gz
└── Raw Images
├── IBSR_01_ana_strip.nii.gz
├── IBSR_02_ana_strip.nii.gz
├── IBSR_03_ana_strip.nii.gz
├── IBSR_04_ana_strip.nii.gz
├── IBSR_05_ana_strip.nii.gz
└── IBSR_07_ana_strip.nii.gz

[ginobilinie]

Yes. What you said is right. One more thing, you have to pay attention to the size of the patch, the size of data patch should be same size with the size of label patch.

you can use https://github.com/ginobilinie/infantSeg/blob/master/readMedImg4CaffeCropNie4SingleS.py
to compute the patches. You can change the size by dFA (patch size for data patch) and dSeg (patch size for label patch).

I hope it helps.

[John1231983]

Thanks for your information. Now, I copy all nii files (raw+label) into same folder Raw_Label_Images.

IBSR_01_ana_strip.nii.gz   IBSR_03_ana_strip.nii.gz   IBSR_05_ana_strip.nii.gz
IBSR_01_segTRI_ana.nii.gz  IBSR_03_segTRI_ana.nii.gz  IBSR_05_segTRI_ana.nii.gz

I also modify the readMedImg4CaffeCropNie4SingleS.py code as

    path='./Raw_Label_Images/'
    saveto='./Raw_Label_Patch_Images/'
   
    ids=[01,03,05]
    for id in ids:
        datafilename='IBSR_%02d_ana_strip.nii.gz'%id #provide a sample name of your filename of data here
        datafn=os.path.join(path,datafilename)
        labelfilename='IBSR_%02d_segTRI_ana.nii.gz'%id  # provide a sample name of your filename of ground truth here
        labelfn=os.path.join(path,labelfilename)

I got a error about index as

./Raw_Label_Images/IBSR_01_ana_strip.nii.gz ./Raw_Label_Images/IBSR_01_segTRI_ana.nii.gz

trainFA shape,  (10000, 1, 32, 32, 32)
matFA shape is  (256, 128, 256)
matFAOut shape is  (264, 136, 264)
Traceback (most recent call last):
  File "readMedImg4CaffeCropNie4SingleS.py", line 129, in <module>
    main()
  File "readMedImg4CaffeCropNie4SingleS.py", line 125, in main
    cubicCnt=cropCubic(mrimg,labelimg,fileID,dFA,step,rate)
  File "readMedImg4CaffeCropNie4SingleS.py", line 88, in cropCubic
    trainFA[cubicCnt,0,:,:,:]=volFA #32*32*32
IndexError: index 10000 is out of bounds for axis 0 with size 10000

How should I fix it? In additions, I have three questions:

1. Why your patches of input data dFA and patches of label data dSeg are not same size?
2. As I understood your idea, you will extract the patch of 3D raw image to 32x32x32 (because FCN provides a small receptive field). However, I think we can directly use all 3D raw image and extract its patch in traning processing as VNet done, instead of these patches. In additions, Could you provide train_test prototxt for single modality, instead of two single modalities T1, T2 as your published infant_train_test prototxt?
3. Why did you use cropCubic ? I think crop a rectangular is more simple and less code than cropCubic. Does cropCubic haveany benefit?

[ginobilinie]

For this error, it is because the patch you extracted is more than 10000, I assume the patch number for a subject is less than 10000:
Change: estNum=20000 or some value you expect.

1. you can change dSeg=32. For segmentation, it should better be the same size
2. Yes, you can do like that. I can provide single modality prototxt: https://github.com/ginobilinie/infantSeg/blob/master/infant_train_test_singleModality.prototxt
3. because I need 3D patches instead of 2D ones. If you use 2D, you can crop just a rectangular.

[John1231983]

Grateful to your help. I can run readMedImg4CaffeCropNie4SingleS.p to crop patches from medical image data for both training and testing data. I obtained testMS_list.txt and
./testMS_06.h5 ./testMS_08.h5
And trainMS_list.txt
./trainMS_01.h5 ./trainMS_02.h5 ...
When I ran, I got some errors (some of them are fixed). Based on my fixed error, I can give you some comments:
1. Because in your readMedImg4CaffeCropNieSingleS.py uses f['dataMR32']=trainFA, f['dataSeg32']=trainSeg. Hence, in your prototxt, it has to change dataT1-> dataMR32, dataSeg->dataSeg32 (or dataSeg24 if you use original code)
2. Which caffe version did you use? I tried with three versions of caffe: BVLC, U-Net, V-Net and DL_BigNeuron.

This is error when using 3D-caffe of V-Net (did not fix)

I0215 13:35:22.168412   778 hdf5_data_layer.cpp:94] Number of HDF5 files: 6
F0215 13:35:46.432451   778 io.cpp:253] Check failed: class_ == H5T_FLOAT (0 vs. 1) Expected float or double data

This is error when I used DL_BigNeuron (built without USE_CUDNN, because I am using cuda 8.0 and cudnn 5.1). Hence, I changed engine: CUDNN->engine: CAFFE in pool2 and pool3. However, I still got the error

F0215 14:50:09.357628 16397 pooling_layer.cpp:61] Check failed: num_kernel_dims == 1 || num_kernel_dims == num_spatial_axes_ kernel_size must be specified once, or once per spatial dimension (kernel_size specified 0 times; 3 spatial dims);

Finally, this is error when I use BVLC and U-Net

F0215 13:22:25.698622 27551 blob.hpp:140] Check failed: num_axes() <= 4 (5 vs. 4) Cannot use legacy accessors on Blobs with > 4 axes.
*** Check failure stack trace: ***

For above error, we can remove line 140, 141 in include/caffe/blob.hpp but I don't think it is official way. Someone suggest to use gaussian filter, instead of xavier in here. I used removing line 140,141 for debug. This is result of 1000iters using fixed blob.hpp

I0215 15:07:57.757104 23039 sgd_solver.cpp:106] Iteration 900, lr = 0.001
I0215 15:08:19.705832 23039 solver.cpp:337] Iteration 1000, Testing net (#0)
I0215 15:08:23.910619 23039 solver.cpp:404]     Test net output #0: loss = 0.670852 (* 1 = 0.670852 loss)
I0215 15:08:23.957226 23039 solver.cpp:228] Iteration 1000, loss = 0.430156
I0215 15:08:23.957267 23039 solver.cpp:244]     Train net output #0: loss = 0.430156 (* 1 = 0.430156 loss)
I0215 15:08:23.957293 23039 sgd_solver.cpp:106] Iteration 1000, lr = 0.001

Hence, I think the caffe version is very importance. Could you please provide it in README. It must be support 3D operations.
3. After 1000 iters, I got train_iter_1000.caffemodel. Now, I will run evaluation code to measure the DSC value. I changed your evalCaffeModel4ImgNie.py as here, in which the infant_deploy_singleModality.prototxt is modified such as (adding softmax, dataMR32 and xavier->gaussian). However, I got the error

blobs ['dataMR32', 'conv1a', 'conv1b', 'conv1c', 'pool1', 'conv2a', 'conv2b', 'pool2', 'conv3a', 'pool3', 'deconv4', 'deconv5', 'deconv6', 'pred']
params ['conv1a', 'conv1b', 'conv1c', 'conv2a', 'conv2b', 'conv3a', 'deconv4', 'deconv5', 'deconv6']
matFA shape is  (256, 128, 256)
matFAOut shape is  (256, 128, 256)
Traceback (most recent call last):
  File "evalCaffeModel4ImgNie.py", line 154, in <module>
    main()
  File "evalCaffeModel4ImgNie.py", line 146, in main
    matOut=cropCubic(mrimg,labelimg,fileID,dSeg,step,rate)
  File "evalCaffeModel4ImgNie.py", line 100, in cropCubic
    matOut[i:i+d[0],j:j+d[1],k:k+d[2]]=matOut[i:i+d[0],j:j+d[1],k:k+d[2]]+temppremat;
ValueError: operands could not be broadcast together with shapes (32,32,32) (32,8,8,8) 

The error is mismatch size between matOut and temppremat, where temppremat = mynet.blobs['conv3a'].data[0] as your code. I guess it must be softmax, instead of conv3a. I did not fix the error
Sorry about long comment.

[ginobilinie]

Thank you so much. You really provide a lot of details which I didn't pay much attention to. I'll update the readme soon.

1. You're right. I indeed need to change the input for the prototxt which makes it corresponds to the generated ones.
   2.The caffe version need to support 3D (I assume it that all medical image analysis researchers should know it, my bad), actually if you use cudnn support, I think the official bvlc version is okay.
2. I think I write the softmax stuff in the evalxxx.py, you have to uncomment it, the conv3a is also one solution. We can switch to softmax easily by uncommenting it.

Thanks again.

[John1231983]

you are welcome. in additions, the top of sofmax must be softmax, instead of prob as your deploy.prototxt. right?
and, i am using current bvlc caffe. it looks that does not support convolution with initial weight as xavier, i need to change it to gaussian filter, even i built with cudnn. howe about your caffe version? I am also running the eval file to generate full mri file but it takes very long time. i have been waiting for 3hours, but it does not finished.....

[ginobilinie]

Of course, the deploy should add a softmax layer if you need to combine them. As my previous uploaded one is not only for segmentation, but also for regression, so my uploaded deploy prototxt looks like it.

Actually, I think bvlc version support xavier.

For the long time issue, you can adjust the step size, (it is 1,1,1 by default, you can make it to 8, 8,8)

[John1231983]

Thanks, the current BVLC does not support xavier with 3D input data. I used UNet for that task. This is my current result after 10000 iterations. It looks wrong. I am not sure what is happen. Even size of image are expected as 256x128x256 but the output does not show the label result. I shown here two person ids 06 and 08 with ground-truth image (second and fourth). First and third image are your result

[ginobilinie]

It seems that the output dimension is not consistent with the ground truth. I guess it's a problem about input dimension order. Can you adjust the input dimension order? In my experience, even though the result is not good, the dimension should be matched.

[John1231983]

I am using mipav to show the result. It looks the inconsistent between Ground truth and output. But I checked the output dimension. Its dimension is same with ground truth (still 256x128x256). I check the value of output. It is not label value, it is float value. I think it is reason

[ginobilinie]

I see. You should convert the round the value to integer. you can use np.rint(x)

[John1231983]

Thanks, I added the code in the function cropCubic of evalCaffeModels4MedImg.py

    matOut = matOut / used
    matOut =np.rint(matOut)
    return matOut

Although the output is label output but the result is still very bad. This is configuration what I did

1. Caffe U-Net (because current caffe BVLC does not work for 3D image)
2. infant_train_test_singleModality.prototxt for training, infant_deploy_singleModality.prototxt and evalCaffeModels4MedImg.py for making output, readMedImg4CaffeCropNie4SingleS.py for convering nii to hdf5.

Any wrong in my procedure?

[ginobilinie]

What's your test loss for the last iteration? the result is worse than my worst case.

[John1231983]

Sorry for waiting. I tried to run again. This is my log. I saw that the .cafemodel size just 1.1MB (very small)

I0219 15:23:52.390311  8591 solver.cpp:337] Iteration 5000, Testing net (#0)
I0219 15:23:56.332168  8591 solver.cpp:404]     Test net output #0: loss = 0.562929 (* 1 = 0.562929 loss)
I0219 15:23:56.373838  8591 solver.cpp:228] Iteration 5000, loss = 0.307311
I0219 15:23:56.373850  8591 solver.cpp:244]     Train net output #0: loss = 0.307311 (* 1 = 0.307311 loss)
I0219 15:23:56.373855  8591 sgd_solver.cpp:106] Iteration 5000, lr = 0.001
....
I0219 15:43:40.078007  8591 solver.cpp:228] Iteration 9900, loss = 0.322395
I0219 15:43:40.078148  8591 solver.cpp:244]     Train net output #0: loss = 0.322395 (* 1 = 0.322395 loss)
I0219 15:43:40.078156  8591 sgd_solver.cpp:106] Iteration 9900, lr = 0.001
I0219 15:44:01.501957  8591 solver.cpp:454] Snapshotting to binary proto file snapshot/train_iter_10000.caffemodel
I0219 15:44:01.678994  8591 sgd_solver.cpp:273] Snapshotting solver state to binary proto file snapshot/train_iter_10000.solverstate
I0219 15:44:01.727710  8591 solver.cpp:317] Iteration 10000, loss = 0.36826
I0219 15:44:01.727732  8591 solver.cpp:337] Iteration 10000, Testing net (#0)
I0219 15:44:05.720078  8591 solver.cpp:404]     Test net output #0: loss = 0.405399 (* 1 = 0.405399 loss)

[ginobilinie]

I suggest you to train more iterations. And your loss should better smaller than 0.1

[John1231983]

Thank you for your support. I will run more iteration and let you know the result.
I want to ask you something about the order of matrix in hdf5. As I know, we cannot directly feed the whole volumetric data into network due to limited memory space. We will randomly extract n_sample the patches and save it to hdf5 as follows:

                    Original data                   Extracted data
Raw:           256x128x256                 n_samplex1x32x32x32
Label:         256x128x256                 n_samplex1x32x32x32

where 1 is gray channel. Do you think the above order of matrix is match with CAFFE? Do we need to remove channel dimension in label volume from n_samplex1x32x32x32 to n_samplex32x32x32?

[ginobilinie]

if you use 3D format, it is in this manner: n_samplex1x32x32x32

[John1231983]

Hello, after test your code. I achieved the result (output, ground-truth and training loss) as follows

I cannot achieve the expected result, although I trained with 100.000 iterations. After running many version of caffe, I guess some reasons

1. The current caffe does not support ND convolution and pooling. We must choose special PR of caffe (3D U-Net supports only ND msra filter, the PR #3983 supports ND xavier). I guess your code is using PR #3983. Can you clarify which caffe version did you used?
2. I did not found the ND Deconvolution in caffe (just 2D). Do we need to use 3D Deconvolution in the prototxt?
3. In your evalCaffeModels4MedImg.py that I am using to create above output and label file. Lines 95-99, you used normalization for the input image and feed the normalized image to the network. However, in your readMedImg4CaffeCropNie4SingleS.py (for generating hdf5 from nii) did not use normalization. It makes inconsistency. Please check it and let me know which is the redundancy?

Thank you so much

[ginobilinie]

For 1,2, you'd better use ND convolution, PR#3983 is okay, I think the code from U-net is also fine, actually, the ND convolution is supported by cudnn instead of such codes, so you have to install the cudnn.

For 3, you have to adjust the code yourself, actually, I use normalization when generating hdf5 and evaluation. So you have to normalize the data in readMedImg4CaffeCropNie4SingleS.py by yourself.

[ginobilinie]

@John1231983 I read your result, it is bad, do you really use 32x32x32 as input? and do you use 3x3x3 filters and 4x4x4 deconv filters? Even the result is assumed to be much better very simple training.

[ginobilinie]

@John1231983 you donot necessarily follow everything as I published in this github. Msra initialization is actually better than xavier, you can use that.

[ginobilinie]

@John1231983 Can you please give me an email, I can send you codes and demos for training and testing.

[John1231983]

My email is [email protected]
Thank you for your sharing and your help
Edit: If the attached file is bigger than 25MB, please give me the link that I can download