[gt-users] gt python with (multi)processing

[Sascha Steinbiss]

On 01/20/2010 02:03 PM, Gordon Gremme wrote:
>> Any ideas or opinions? Could making the C library thread-safe be
>> worth the effort (and how much would that be)? And does anyone have
>> experience with that?
> I started making libgenometools multithreading-safe (MT-safe),
> thanks for pushing the issue ;-)

Thanks to you for your commitment! :)

> The following pointers should make it easy to make the FeatureIndex
> MT-safe. I haven't looked into the details, but it would probably be
> best to use the Mutexes (or Read/Write locks) on the interface level
> to avoid reimplementation for every FeatureIndex implementation.

Yes, that's what I did (in annotationsketch/feature_index.c).

> - If I didn't miss something, everything with a global state in 
> libgenometools should be MT-safe now (file allocator, symbol and 
> random number generator module). The memory allocator is MT-safe, if 
> the memory bookkeeping is not used. Making the memory bookkeeping 
> MT-safe is not possible without a major redesign which I consider 
> low-priority.

Agreed. Making the reference counting thread-safe is another major
issue, IMHO. What do you think?

> - Testing is even more important for multithreaded-code! I thought 
> twice the memory allocator with bookkeeping is now MT-safe before I 
> realized it is not possible with the current design. As a testing
> strategy, I added a -j development option to the gt binary which can
> then be used in the unit tests to start parallel threads to stress
> test the corresponding data structure in parallel (gt_multithread()
> starts -j many threads, see core/symbol.c for an example).

Very useful. Do you think we should add a multithreading test case to
every unit test when the class in question has been modified for
thread-safety?

> If possible, store the mutexes or read/write-locks in the data 
> structure itself and not globally, as in the file allocator module.

Sounds natural.

> - The valgrind tool helgrind is your friend (valgrind 
> --tool=helgrind)! Without it I would still wonder why the memory 
> allocator with bookkeeping isn't MT-safe. The valgrind tool DTD
> might be useful, too.

Cool. I will familiarize myself with that.

> @Sascha: Could you look into making the FeatureIndex MT-safe? It 
> shouldn't be much work at this point as the basic infrastructure is 
> already there. If anything of the above is unclear or questions
> arise, please ask!

Yes, I am working on that. Synchronizing the read/write accesses to the
FeatureIndex is quite simple, I will also test everything later and
report back when I am ready.

> Gordon

Sascha

-- 
Sascha Steinbiss
Center for Bioinformatics
University of Hamburg
Bundesstr. 43
20146 Hamburg
Germany

Email:  steinbiss at zbh.uni-hamburg.de
URL:    http://www.zbh.uni-hamburg.de/steinbiss
Phone:  +49 (40) 42838 7322
FAX:    +49 (40) 42838 7312