14,808 questions
4
votes
2
answers
155
views
Python multiprocessing shared memory seems to hang or crash when interacting with small numpy arrays
I've been trying to parallelize some code that I wrote in python. The actual work is embarrassingly parallel, but I don't have much experience with multiprocessing in Python.
The actual code I'm ...
- 76
-6
votes
0
answers
77
views
how implement in a Python algorithm to avoid exceeding the server's rate limits? [closed]
I'm working on a tourism data extractor focused on the Foursquare platform.
I currently don't have any proxies or techniques to avoid being blocked for exceeding the server's request limit (50 ...
0
votes
0
answers
70
views
XCB error when using Python with Matplotlib and Multiprocessing
I ran into a problem when I'm trying to plot figures with matplotlib in separate processes using multiprocessing. I tried it on OpenBSD 7.7 with Python 3.12.11 and on Debian 12 with Python 3.11.2 and ...
- 1
-3
votes
0
answers
51
views
Why does 'multiprocessing.Pool' hang when I use it inside a Jupyter Notebook? [duplicate]
I'm testing Python's multiprocessing module within a Jupyter notebook. When I apply multiprocessing.Pool with a trivial function, the kernel is frozen and I don't have any output. The same code works ...
0
votes
1
answer
64
views
subprocess.run() eventually reaches python's global thread limit
I have a python script that pings another device every so often.
PING_SIZE = "1"
PING_RETRYS = "1"
while True:
try:
result = subprocess.run(
["ping&...
2
votes
2
answers
124
views
How to safely check a multiprocessing.Event in an asyncio loop without blocking?
I'm working on an asynchronous server using asyncio, but I need to monitor a multiprocessing.Event (used to signal termination from another process) inside my async event loop. Here's the simplified ...
2
votes
0
answers
108
views
What is the best/correct way to handle shared memory objects in a heterogeneous multi-CPU embedded microprocessor context [closed]
I'm working on several projects that are targeting bare-metal multi-core microprocessors, with hardware provisions for shared memory.
In other words, the device has several separate CPU cores (of ...
- 5,924
3
votes
1
answer
64
views
Accessing any object type from multiprocessing shared_memory?
Suppose I create a shared memory object:
from multiprocessing import shared_memory
shm_a = shared_memory.SharedMemory(create=True, size=1024)
buffer = shm_a.buf
and put a generic object of a generic ...
- 5,826
3
votes
4
answers
154
views
Where should I allocate the pool in Python multiprocessing?
I want to distribute a python app over several kernels and from the documentation I understand that a pool is the way to do this.
My problem can be reproduced by the following code:
#!/usr/bin/python3
...
- 91
1
vote
1
answer
58
views
Сonnection timeout for function does not seem to work in Python script
for some reason the connection timeout does not seem to work when connecting to a Firebird database. For example, if an incorrect IP address is specified, the script hangs and waits for a long time.
#!...
- 11
2
votes
2
answers
142
views
Setting global variables for python multiprocessing
I have a large array and an object I'd like to call multiple times with multiprocessing. Neither the data nor the object internals get modified.
This works:
import numpy as np
from multiprocessing ...
- 995
2
votes
1
answer
132
views
Starting multiprocessing.Process() in a Pytest test/Python creates a Windows fatal exception: access violation
On windows, running the below code with Python or pytest makes it print out Windows fatal exception: access violation (but the script will continue with no issue)
Reproduction:
import multiprocessing ...
- 694
0
votes
1
answer
80
views
Cannot pickle weakref error in the multiprocessing in a class in Python
In the project I am currently working on, I am parsing a json file and according to client list, I need to start processes in parallel. However there might be the case where a inside a client list, I ...
- 13
0
votes
0
answers
54
views
FastAPI becomes unresponsive while running partition_pdf from unstructured in a background process
I have a FastAPI application where one of the API endpoints needs to parse PDF files using unstructured.partition_pdf. Since the parsing is CPU-heavy, I want to run it in the background so that the ...
1
vote
0
answers
100
views
Windows Python 3.12 Multiprocessing Deadlock
I'm having trouble getting this example working. In serial this function works just fine, but when I attempt to run it in a multiprocessing.Pool it locks up and will not return a simple random integer....
- 51