-2

feature class fc1 (daily data) and feature class fc2 (geodatabase) both have unique_id field which will be used to map Col1 rows in fc1 to update corresponding Col2 rows (not all rows) in fc2. Below is the code I used;

with arcpy.da.SearchCursor(fc1, ["Unique_id"]) as cursor:
    for row in cursor:
        SQL_stat= "Unique_id= "+ str(row[0])
        with arcpy.da.SearchCursor(fc1, ["Col1"], SQL_stat) as cursor2: #SQL_stat works here
            for row2 in cursor2:
                UpdatedValue = row2[0]
                with arcpy.da.UpdateCursor(fc2, ["Col2"], SQL_stat) as cursor3: #not working here
                    for row3 in cursor3:
                        row3[0] = UpdatedValue
                        cursor3.updateRow(row3)

I get the error below:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
In  [26]:
Line 7:     with arcpy.da.UpdateCursor(fc2, ["Col2"], SQL_stat) as cursor3:

RuntimeError: An invalid SQL statement was used. [Table_in_fc2]
---------------------------------------------------------------------------
9
  • 3
    Triple-nesting cursors is extraordinarily bad practice, and completely unnecessary based on your code sample. Best practice would be to build a dictionary of { unique_id : col1 } pairs in one pass of the SearchCursor, then updateRow using a lookup from unique_id in an UpdateCursor pass. Commented Jul 21, 2021 at 20:33
  • Thank you for that advise, I'm new in scripting. I'll implement your advise. Commented Jul 22, 2021 at 13:09
  • The error indicates type incompatibility between the two tables. Commented Jul 22, 2021 at 13:57
  • Oh wow, I was thinking in that regards yesterday. So, I should check if both tables have same data types? Is that what it is? Commented Jul 22, 2021 at 14:52
  • If unique_id is varchar, a numeric comparison will fail in Python just like SQL (though casting is available in both). To complete this question, edit it to include the table definition (column names and types) for both fc1 and fc2. Commented Jul 22, 2021 at 15:31

1 Answer 1

3

I adapted your code to work with a string unique key and a set of file geodatabase tables with 3124 US counties, copying the contents of randomly generated rnd_val from one table to another, using various improvements...

First, I modularized your code, using variables for field names and str.format instead of string math:

import os
import datetime

print('Importing arcpy....')
import arcpy

fc1 = r'D:\GIS_SE\gisse.gdb\counties1'
fc2 = r'D:\GIS_SE\gisse.gdb\counties3'

fldUnique = 'fips'
fldSource = 'rnd_val'
fldDest   = 'rnd_val'

start = datetime.datetime.utcnow()
with arcpy.da.SearchCursor(fc1, [fldUnique]) as cursor:
    for row in cursor:
        where_string = "{:s} = '{:s}'".format(fldUnique,row[0])
        with arcpy.da.SearchCursor(fc1, [fldSource], where_string) as cursor2:
            for row2 in cursor2:
                updatedValue = row2[0]
                with arcpy.da.UpdateCursor(fc2, [fldDest], where_string) as cursor3:
                    for row3 in cursor3:
                        row3[0] = updatedValue
                        cursor3.updateRow(row3)
elapsed = (datetime.datetime.utcnow() - start).total_seconds()
print("{:>16s} : {:.3f}".format('Pass1',elapsed))

This executed in 201.639 seconds.

Then I eliminated the redundant second cursor:

start = datetime.datetime.utcnow()
with arcpy.da.SearchCursor(fc1, [fldUnique,fldSource]) as cursor:
    for row in cursor:
        where_string = "{:s} = '{:s}'".format(fldUnique,row[0])
        UpdatedValue = row[1]
        with arcpy.da.UpdateCursor(fc2, [fldDest], where_string) as cursor3:
            for row3 in cursor3:
                row3[0] = UpdatedValue
                cursor3.updateRow(row3)
elapsed = (datetime.datetime.utcnow() - start).total_seconds()
print("{:>16s} : {:.3f}".format('Pass2',elapsed))

This executed in 99.306 seconds.

Then I used a FGDB table with an index on the unique field:

start = datetime.datetime.utcnow()
fc2 = r'D:\GIS_SE\gisse.gdb\counties2'
with arcpy.da.SearchCursor(fc1, [fldUnique,fldSource]) as cursor:
    for row in cursor:
        where_string = "{:s} = '{:s}'".format(fldUnique,row[0])
        updatedValue = row[1]
        with arcpy.da.UpdateCursor(fc2, [fldDest], where_string) as cursor3:
            for row3 in cursor3:
                row3[0] = updatedValue
                cursor3.updateRow(row3)
elapsed = (datetime.datetime.utcnow() - start).total_seconds()
print("{:>16s} : {:.3f}".format('Pass3',elapsed))

This ran in 23.358 seconds.

Just for grins, I ran the triple-nested query on the indexed table:

start = datetime.datetime.utcnow()
fc1 = r'D:\GIS_SE\gisse.gdb\counties2'
fc2 = r'D:\GIS_SE\gisse.gdb\counties2'
with arcpy.da.SearchCursor(fc1, [fldUnique]) as cursor:
    for row in cursor:
        where_string = "{:s} = '{:s}'".format(fldUnique,row[0])
        with arcpy.da.SearchCursor(fc1, [fldSource], where_string) as cursor2:
            for row2 in cursor2:
                updatedValue = row2[0]
                with arcpy.da.UpdateCursor(fc2, [fldDest], where_string) as cursor3:
                    for row3 in cursor3:
                        row3[0] = updatedValue
                        cursor3.updateRow(row3)
elapsed = (datetime.datetime.utcnow() - start).total_seconds()
print("{:>16s} : {:.3f}".format('Pass4',elapsed))

This ran in 44.215 seconds (nearly twice as long as the two-cursor solution)

Finally, I used two discrete passes, caching the source values by unique key, then updating the unindexed target with a lookup:

start = datetime.datetime.utcnow()
fc1 = r'D:\GIS_SE\gisse.gdb\counties1'
fc2 = r'D:\GIS_SE\gisse.gdb\counties3'
lookup = {}
with arcpy.da.SearchCursor(fc1, [fldUnique,fldSource]) as cursor:
    for row in cursor:
        lookup[row[0]] = row[1]

with arcpy.da.UpdateCursor(fc2, [fldUnique,fldDest]) as cursor3:
    for row3 in cursor3:
        row3[1] = lookup[row3[0]]
        cursor3.updateRow(row3)
elapsed = (datetime.datetime.utcnow() - start).total_seconds()
print("{:>16s} : {:.3f}".format('Pass5',elapsed))

And this ran in 0.314 seconds.

Thus, I repeat my comment: Best practice would be to build a dictionary of { unique_id : col1 } pairs in one pass of the SearchCursor, then updateRow using a lookup from unique_id in an UpdateCursor pass.


Note that this code assumes 1:1 correspondence on the keys in both tables. If the target table has keys not present in the source, then you'd need to test for row3[0] in lookup and decide whether a value should be assigned, and what that value should be:

    for row3 in cursor3:
        row3[1] = lookup[row3[0]] if row3[0] in lookup else None
        cursor3.updateRow(row3)

or

    for row3 in cursor3:
        if (row3[0] in lookup):
            row3[1] = lookup[row3[0]]
            cursor3.updateRow(row3)

If there are keys in fc1 not present in fc2, then you can detect this with a destructive read then processing the resultant dictionary pairs:

    for row3 in cursor3:
        if (row3[0] in lookup):
            row3[1] = lookup[row3[0]]
            cursor3.updateRow(row3)
            del lookup[row3[0]]

for k in lookup:
    print("Key '{:s}' with value '{:d}' not present in fc2!".format(
             k,lookup[k]))
9
  • I do appreciate you for these lines of codes, it helped me to visually understand what you meant by building a dictionary of {unique_id:col1} pairs in one pass of the SearchCursor. One more request-- in the case where my unique_id from fc1 has some rows that are new entries (not present in the unique_id of the fc2), I need to pass the corresponding Col1 values into Col2 attributes simply by inserting them under Col2. I wrote a script that inserted all of my Col1 attributes (all 116 records from unique_id) under Col2. I only need the 5 unmatched rows/records to be inserted as new records Commented Jul 23, 2021 at 15:17
  • Pretty much, every matched record of unique_id should be updated/overwritten for Col1 into Col2, while unmatched records of unique_id should be inserted for Col1 under new records of Col2. Commented Jul 23, 2021 at 15:51
  • Tweaking the script to perform a destructive read from the dictionary, then using InsertCursor on the remaining rows is outside the scope of the original question, but no more than five additional lines of code (one to del and four to insert). Give it a try, first, and if you can't manage it, make a new question. Commented Jul 23, 2021 at 15:56
  • I meant this; with arcpy.da.SearchCursor(fc1, ["unique_id","Col1"]) as cursor: for row in cursor: where= "unique_id= "+ str(row[0]) with arcpy.da.UpdateCursor(fc2, ["Col2"], where) as cursor1: for row1 in cursor1: row1[0] = row[1] cursor1.updateRow(row1) Commented Jul 25, 2021 at 5:30
  • No, don't bother putting multi-line Python in comments -- it's illegible. Edit the question instead. Commented Jul 25, 2021 at 11:42

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.