rgt-hint differential: multiprocessing error

[daria-dc]

Hi,

I am trying to run a differential footprinting analysis for different comparisons. For one particular comparison, the pipeline runs successfully. For all the other comparisons, however, I get the following error message:

The output (if any) follows:

Traceback (most recent call last):
File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/bin/rgt-hint", line 10, in
sys.exit(main())
File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/site-packages/rgt/HINT/Main.py", line 90, in main
args.func(args)
File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/site-packages/rgt/HINT/DifferentialAnalysis.py", line 311, in diff_analysis_run
pool.map(line_plot, plots_list)
File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/multiprocessing/pool.py", line 253, in map
return self.map_async(func, iterable, chunksize).get()
File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/multiprocessing/pool.py", line 572, in get
raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: 'HTTPError()'. Reason: 'TypeError("can't pickle cStringIO.StringO objects",)'

It is like the pipeline was interrupted somewhere in the middle, because the factor.txt file is there and also there are some files in the Lineplots folder.
I generated all of my input files the same way.
Did you ever encounter this error message or do you have any idea what might be causing it? Any help is greatly appreciated!

Thanks,
Daria

[lzj1769]

Hi @daria-dc

The reason for this error is to generate a motif logo, we send a request to weblogo3 and download the picture. If too many requests are sent at the same time, there is a chance to get the communication problem.

In our develop branch, we have solved this problem by using self-contained method to get the picture, therefore no internet connection is needed. We are going to release it soon. If you want to use it now, please install RGT use develop branch as follows:

git clone [email protected]:CostaLab/reg-gen.git
cd reg-gen
git checkout develop
pip install ./ --user

Be sure you have uninstalled the old RGT and let me know if you have any questions.

Thanks,
Li

[daria-dc]

Hi Li,

thanks for the quick response.
I installed the develop branch, but now I get the following error message:

The output (if any) follows:


This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was *originally* set to 'agg' by the following code:
  File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/bin/rgt-hint", line 6, in <module>
    from rgt.HINT.Main import main
  File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/site-packages/rgt/HINT/Main.py", line 10, in <module>
    from rgt.HINT.Plotting import plotting_args, plotting_run
  File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/site-packages/rgt/HINT/Plotting.py", line 13, in <module>
    import matplotlib.pyplot as plt
  File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/site-packages/matplotlib/pyplot.py", line 71, in <module>
    from matplotlib.backends import pylab_setup
  File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/site-packages/matplotlib/backends/__init__.py", line 17, in <module>
    line for line in traceback.format_stack()



This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was *originally* set to 'agg' by the following code:
  File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/bin/rgt-hint", line 6, in <module>
    from rgt.HINT.Main import main
  File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/site-packages/rgt/HINT/Main.py", line 10, in <module>
    from rgt.HINT.Plotting import plotting_args, plotting_run
  File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/site-packages/rgt/HINT/Plotting.py", line 13, in <module>
    import matplotlib.pyplot as plt
  File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/site-packages/matplotlib/pyplot.py", line 71, in <module>
    from matplotlib.backends import pylab_setup
  File "/icgc/dkfzlsdf/analysis/OE0285_projects/ctc-epigenome/doncevic/miniconda3/envs/rgtEnv/lib/python2.7/site-packages/matplotlib/backends/__init__.py", line 17, in <module>
    line for line in traceback.format_stack()

and also, the arguments are not recognized anymore...
Do you have any idea how to solve this?

Thanks,

Daria

[lzj1769]

Hi Daria,

The above is not an error message, just warning, so no worries.
The reason is to get rid of weblogo3, I used another package, called logomaker. In this package, matplotlib with default backend is called to generate the picture. However, to be able to run RGT on some devices that do not have a screen to show plots, such as HPC, I have to switch the backend to 'agg', which leads to the above warning message.

Regarding arguments, rgt-hint differential now supports comparing more than two conditions, so I changed the arguments to reflect this, you can set it as follows:

before:
rgt-hint differential --organism=mm9 --bc --nc 30 --mpbs-file1=./match/cDC1_mpbs.bed --mpbs-file2=./match/pDC_mpbs.bed --reads-file1=cDC1.bam --reads-file2=pDC.bam --condition1=cDC1 --condition2=pDC --output-location=cDC1_pDC

now:
rgt-hint differential --organism=mm9 --bc --nc 30 --mpbs-files=./match/cDC1_mpbs.bed,./match/pDC_mpbs.bed --reads-files=cDC1.bam,pDC.bam --conditions=cDC1,pDC --output-location=cDC1_pDC

Best,
Li

[daria-dc]

Hi Li,

thank you very much for your help. I was able to successfully run the program.
I just have one more question: For the lineplots, I get this message

signal generation is done!
generating line plot for each motif...

This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.

What does it mean, are the results still reliable?

Thanks,

Daria

[lzj1769]

What does it mean, are the results still reliable?

Just another warning for making plots, so the results are still reliable.