Possible differences between list and iterator

Question

I have a weird problem with iterators, which I can't figure out. I have a complicated numerical routine returning a generator object (or after some changes to the code an islice). Afterwards I check, the results as I know that the results must have a negative imaginary part:

import numpy as np
threshold = 1e-8  # just check up to some numerical accuracy

results = result_generator(**inputs)
is_valid = [np.all(_result.imag < threshold) for _result in results]
print("Number of valid results: ", is_valid.count(True))

(Sorry for not giving an executable code, but I can't come up with a simple code at the moment.) The problem is now, that this returns one valid solution. If I change the code to

import numpy as np
threshold = 1e-8  # just check up to some numerical accuracy

results = list(result_generator(**inputs))
is_valid = [np.all(_result.imag < threshold) for _result in results]
print("Number of valid results: ", is_valid.count(True))

using a list instead of a generator, I get zero valid solution. I can however not wrap my head around what is different and thus have no idea how to debug the problem. If I go through the debugger and print out the result with the corresponding index the results are even different, the one of the generator is correct, the one of the list is wrong.

Here the numerical function:

def result_generator(z, iw, coeff, n_min, n_max):
    assert n_min >= 1
    assert n_min < n_max
    if n_min % 2:
        # index must be even
        n_min += 1

    id1 = np.ones_like(z, dtype=complex)
    A0, A1 = 0.*id1, coeff[0]*id1
    A2 = coeff[0] * id1
    B2 = 1. * id1

    multiplier = np.subtract.outer(z, iw[:-1])*coeff[1:]
    multiplier = np.moveaxis(multiplier, -1, 0).copy()

    def _iteration(multiplier_im):
        multiplier_im = multiplier_im/B2
        A2[:] = A1 + multiplier_im*A0
        B2[:] = 1. + multiplier_im

        A0[:] = A1
        A1[:] = A2 / B2
        return A1

    complete_iterations = (_iteration(multiplier_im) for multiplier_im in multiplier)
    return islice(complete_iterations, n_min, n_max, 2)

What's _result.imag? I don't see any image manipulation in your code. I recently discovered that some image manipulation libraries implement an iterator that can't be turned directly into a list, and instead you have to make a copy of each image (as such), so maybe that has something to do with it. An MCVE would help. — TigerhawkT3
– TigerhawkT3, Commented Jan 11, 2019 at 9:27
Is it possible that the np.all(some_list) returns a single value False? — Uvar
– Uvar, Commented Jan 11, 2019 at 9:32
@TigerhawkT3 sorry that the question wasn't very clear here. The elements _result in results are numpy arrays. imag corresponds to the imaginary part. — DerWeh
– DerWeh, Commented Jan 11, 2019 at 10:08
@Uvar np.all returns indeed a single value. It is used to check that all numbers in the numpy arrays _result have a positive imaginary part. — DerWeh
– DerWeh, Commented Jan 11, 2019 at 10:10

user2357112 · Accepted Answer · 2019-01-11 10:10:34Z

2

You're yielding the same array over and over instead of making new arrays. When you call list, you get a list of references to the same array, and that array is in its final state. When you don't call list, you examine the array in the state the generator yields it, each time it's yielded.

Stop reusing the same array over and over.

answered Jan 11, 2019 at 10:10

user2357112

286k32 gold badges489 silver badges569 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Iguananaut Over a year ago

Or do reuse the same array, as doing so may be more efficient, but then you have to take care never to eagerly evaluate all items in the iterator or else you'll have similar bugs.

user2357112 Over a year ago

@Iguananaut: They're making new arrays anyway and then spending extra effort to copy those arrays into the existing arrays, so they're actually losing time by trying this. There are ways to avoid the extra copy, but they don't look like this.

DerWeh Over a year ago

@user2357112 It is true, as we create already the temporary array A2 / B2 we should return this array and not A1 to make the code more robust. If you know other improvements, saving array allocations and thus time, I would be glad to hear them. This part is still my bottleneck.

Collectives™ on Stack Overflow

Possible differences between list and iterator

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related