Python - "can't Pickle Thread.lock" Error When Creating A Thread Under A Multiprocess In Windows

I'm getting stuck on what I think is a basic multiprocess and threading issue. I've got a multiprocess set up, and within this a thread. However, when I set up the thread class wit

Solution 1:

It looks like there is no simple answer, and it appears to be a restriction of Windows (Win 7, python 3.6 in my case); on Windows it looks like you need to start the process before you can start the worker thread inside the owned object.

There appears to be no such restriction on Unix (CentOS 7, python 2.7.5).

As an experiment I modified your code as follows; this version checks the OS and starts either the process first, or the thread first:

import multiprocessing
import threading
import time
import os

classMyProcess(multiprocessing.Process):

    def__init__(self, **kwargs):
        super(MyProcess, self).__init__(**kwargs)
        self.dostuff = DoStuff(self)

    defrun(self):
        print("MyProcess.run()")
        print("MyProcess.ident = " + repr(self.ident))
        if os.name == 'nt':
            self.dostuff.start_thread()

classDoStuff(object):
    def__init__(self, owner, **kwargs):
        super(DoStuff, self).__init__(**kwargs)
        self.owner = owner
        if os.name != 'nt':
            self.start_thread()

    defstart_thread(self):
        print("DoStuff.start_thread()")
        self.my_thread_instance = MyThread(self)
        self.my_thread_instance.start()
        time.sleep(0.1)

classMyThread(threading.Thread):
    def__init__(self, owner):
        super(MyThread, self).__init__()
        self.owner = owner

    defrun(self):
        print("MyThread.run()")
        print("MyThread.ident = " + repr(self.ident))
        print("MyThread.owner.owner.ident = " + repr(self.owner.owner.ident))

if __name__ == '__main__':
    mp_target = MyProcess()       # Also pass the pipe to transfer data
    mp_target.daemon = True
    mp_target.start()
    time.sleep(0.1)

... and got the following on Windows, where the process starts first:

MyProcess.run()
MyProcess.ident = 14700
DoStuff.start_thread()
MyThread.run() 
MyThread.ident = 14220
MyThread.owner.owner.ident = 14700

... and the following on Linux, where the thread is started first:

DoStuff.start_thread()
MyThread.run()
MyThread.ident = 140316342347520
MyThread.owner.owner.ident = None
MyProcess.run()
MyProcess.ident = 4358

If it were my code I'd be tempted to always start the process first, then create the thread within that process; the following version works fine for me across both platforms:

import multiprocessing
import threading
import time

classMyProcess(multiprocessing.Process):

    def__init__(self, **kwargs):
        super(MyProcess, self).__init__(**kwargs)
        self.dostuff = DoStuff()

    defrun(self):
        print("MyProcess.run()")
        self.dostuff.start_thread()

classDoStuff(object):
    def__init__(self, **kwargs):
        super(DoStuff, self).__init__(**kwargs)

    defstart_thread(self):
        self.my_thread_instance = MyThread()
        self.my_thread_instance.start()
        time.sleep(0.1)

classMyThread(threading.Thread):
    def__init__(self):
        super(MyThread, self).__init__()

    defrun(self):
        print("MyThread.run()")

if __name__ == '__main__':
    mp_target = MyProcess()       # Also pass the pipe to transfer data
    mp_target.daemon = True
    mp_target.start()
    time.sleep(0.1)