I was wondering if anyone else in the community here has attempted to use multi-processing for spatial analyses. Namely I am trying to iterate through a series of rasters, create a multiprocessing job for each and run them through a number of geoprocessing steps within one def function. Something along the lines of

def deposition(RasterImage, OutFolderDir):
    arcpy.env.overwriteOutput = True  
    arcpy.env.workspace = OutFolderDir 
    DEM_Prj = DEM_Prj.tif
    
    try:
        arcpy.ProjectRaster_management(RasterImage, DEM_Prj....
        arcpy.FocalStatistics(DEM_prj....)
        ...

if __name__ == '__main__':  
    InputFolder = r'C:\test\somepath'  
    Output = r'C:\test\somepath2'  
    arcpy.env.workspace = InputFolder  
    arcpy.env.scratchWorkspace = r'C:\test.gdb'    
   
    fcs = arcpy.ListRasters('*')
    pool = multiprocessing.Pool(4)   
    jobs = []                 
    for fc in fcs:
        rIn = os.path.join(InputFolder,fc)
        rOut = os.path.join(Output,fc[:-4])
        jobs.append(pool.apply_async(deposition,(rIn, rOut)))    

Now the multiprocessing does run, usually for the first batch! However, I keep running into several different errors when attempting several datasets(more than 4 files - i.e. 4 core multiprocessing) including:

ERROR 010302: Unable to create the output raster: C:\somepath\sr6f8~1\FocalSt_srtm1
ERROR 010067: Error in executing grid expression.
Failed to execute (FocalStatistics).

and

ERROR 999999: Error executing function.
Failed to copy raster dataset
Failed to execute (ProjectRaster)

Notice in the first error the strange folder that is created (in the OutFolderDir location) associated with the focal statistics that nearly creates an exact replica of the final output.

My question is based off your experience is it impossible to create several step geoprocessing within one multiprocessing function? Or to I need to tile these steps into their individual geoprocessing steps?

Kind Regards,

Bjorn