[gt-users] gt python with (multi)processing

[Brent Pedersen]

On Wed, Jan 20, 2010 at 7:07 AM, Sascha Steinbiss
<steinbiss at zbh.uni-hamburg.de> wrote:
> On 01/16/2010 04:43 PM, Brent Pedersen wrote:
>>>> #import multiprocessing as processing
>>>> import processing
>>>> import gt
>>>> p = processing.Pool(4)
>>>>
>>>> f = gt.FeatureIndexMemory()
>>>> f.add_gff3file('./testdata/encode_known_genes_Mar07.gff3')
>>>> f.add_gff3file('./testdata/encode_known_genes_Mar07.gff3')
>
> Sorry to come back to this, but I cannot see why this script uses
> multithreading. It looks very sequential as there are no tasks
> distributed to the workers in the pool.

hi, yes it is just a dummy script to demonstrate the problem. that's the
minimum require to cause problems on my machine.
your script below works for me as well, but then if i add simply:
print f.get_seqids()

it segfaults again. regardless, i'll wait as the new threadsafe stuff
progresses.
also, i notice in recent commits, you use the @function.setter
decorator for the
range module. that's cool syntax i hadn't used, but it is only
available in >=python 2.6
i'm attaching a patch that works with (at least) 2.4 and 2.5
as well.
-brent


>
> To reproduce your problem, I tried the following script (using the
> parallelized map() function) with 4 workers and
> testdata/encode_known_genes_Mar07.gff3 and a much larger D. melanogaster
> annotation and got no segfault:
>
> $ cat mp_fi_test.py
> #!/usr/bin/env python
> import processing
> import gt
> import sys
> numthreads = 4
> f = gt.FeatureIndexMemory()
> p = processing.Pool(numthreads)
>
> print p.map(f.add_gff3file, [sys.argv[1] for i in range(0,numthreads)])
> print "done."
>
> $ ./mp_fi_test.py testdata/encode_known_genes_Mar07.gff3
> [None, None, None, None]
> done.
>
> This seems to work without problems repeatedly, even with a GenomeTools
> version from before the MT patches were added. Very strange.
>
> Sascha
>
> --
> Sascha Steinbiss
> Center for Bioinformatics
> University of Hamburg
> Bundesstr. 43
> 20146 Hamburg
> Germany
>
> Email:  steinbiss at zbh.uni-hamburg.de
> URL:    http://www.zbh.uni-hamburg.de/steinbiss
> Phone:  +49 (40) 42838 7322
> FAX:    +49 (40) 42838 7312
>
> _______________________________________________
> gt-users mailing list
> gt-users at genometools.org
> http://genometools.org/mailman/listinfo/gt-users
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gtrange_compat.diff
Type: text/x-patch
Size: 1005 bytes
Desc: not available
Url : http://genometools.org/pipermail/gt-users/attachments/20100120/69e28623/attachment.bin