7

I have an unsorted array of numbers.

I need to replace certain numbers (given in a list) with specific alternatives (also given in a corresponding list)

I wrote the following code (which seems to works): 

import numpy as np

numbers = np.arange(0,40)
np.random.shuffle(numbers)
problem_numbers = [33, 23, 15]  # table, night_stand, plant
alternative_numbers = [12, 14, 26]  # desk, dresser, flower_pot

for i in range(len(problem_numbers)):
    idx = numbers == problem_numbers[i]
    numbers[idx] = alternative_numbers[i]

However, this seems highly inefficient (this needs to be done several millions of times for much larger arrays).

I found this question which answers a similar problem however in my case the numbers are not sorted and they need to maintain their original location.

Note: numbers may contain multiple or no occurrences of elements in problem_numbers

Improve this question
5
  • why you code idx = numbers == problem_numbers[i]?Does problem_numbers need to be in the numbers range? Commented Aug 17, 2017 at 12:38
  • @itzikBenShabat Just another question: Do you want cascading replacements, for example if the problem numbers is [1, 3] and the alternatives are [3, 5] do you want a 1 in numbers to be 3 afterwards or 5? Your solution converts it to 5 while not-cascading replacements would replace it with 3. Commented Aug 17, 2017 at 14:17
  • No cacading is required. This is not a posible scenario with my data Commented Aug 17, 2017 at 14:19
  • 1
    @itzikBenShabat Can we assume unique numbers in problem_numbers? Commented Aug 17, 2017 at 15:02
  • @Divakar Yes. we can assume unique numbers in problem_numbers and alternative_numbers, but not in numbers Commented Aug 17, 2017 at 15:43

2 Answers 2

Reset to default
4

EDIT: I implemented a TensorFlow version of this in this answer (almost exactly the same, except replacements are a dict).

Here is a simple way to do it:

import numpy as np

numbers = np.arange(0,40)
np.random.shuffle(numbers)
problem_numbers = [33, 23, 15]  # table, night_stand, plant
alternative_numbers = [12, 14, 26]  # desk, dresser, flower_pot

# Replace values
problem_numbers = np.asarray(problem_numbers)
alternative_numbers = np.asarray(alternative_numbers)
n_min, n_max = numbers.min(), numbers.max()
replacer = np.arange(n_min, n_max + 1)
# Mask replacements out of range
mask = (problem_numbers >= n_min) & (problem_numbers <= n_max)
replacer[problem_numbers[mask] - n_min] = alternative_numbers[mask]
numbers = replacer[numbers - n_min]

This works well an should be efficient as long as the range of the values in numbers (the difference between the smallest and the biggest) is not huge (e.g you don't have something like 1, 7 and 10000000000).

Benchmarking

I've compared the code in the OP with the three (as of now) proposed solutions with this code:

import numpy as np

def method_itzik(numbers, problem_numbers, alternative_numbers):
    numbers = np.asarray(numbers)
    for i in range(len(problem_numbers)):
        idx = numbers == problem_numbers[i]
        numbers[idx] = alternative_numbers[i]
    return numbers

def method_mseifert(numbers, problem_numbers, alternative_numbers):
    numbers = np.asarray(numbers)
    replacer = dict(zip(problem_numbers, alternative_numbers))
    numbers_list = numbers.tolist()
    numbers = np.array(list(map(replacer.get, numbers_list, numbers_list)))
    return numbers

def method_divakar(numbers, problem_numbers, alternative_numbers):
    numbers = np.asarray(numbers)
    problem_numbers = np.asarray(problem_numbers)
    problem_numbers = np.asarray(alternative_numbers)
    # Pre-process problem_numbers and correspondingly alternative_numbers
    # such that repeats and no matches are taken care of
    sidx_pn = problem_numbers.argsort()
    pn = problem_numbers[sidx_pn]
    mask = np.concatenate(([True],pn[1:] != pn[:-1]))
    an = alternative_numbers[sidx_pn]

    minN, maxN = numbers.min(), numbers.max()
    mask &= (pn >= minN) & (pn <= maxN)

    pn = pn[mask]
    an = an[mask]

    # Pre-pocessing done. Now, we need to use pn and an in place of
    # problem_numbers and alternative_numbers repectively. Map, index and assign.
    sidx = numbers.argsort()
    idx = sidx[np.searchsorted(numbers, pn, sorter=sidx)]
    valid_mask = numbers[idx] == pn
    numbers[idx[valid_mask]] = an[valid_mask]

def method_jdehesa(numbers, problem_numbers, alternative_numbers):
    numbers = np.asarray(numbers)
    problem_numbers = np.asarray(problem_numbers)
    alternative_numbers = np.asarray(alternative_numbers)
    n_min, n_max = numbers.min(), numbers.max()
    replacer = np.arange(n_min, n_max + 1)
    # Mask replacements out of range
    mask = (problem_numbers >= n_min) & (problem_numbers <= n_max)
    replacer[problem_numbers[mask] - n_min] = alternative_numbers[mask]
    numbers = replacer[numbers - n_min]
    return numbers

The results:

import numpy as np

np.random.seed(100)

MAX_NUM = 100000
numbers = np.random.randint(0, MAX_NUM, size=100000)
problem_numbers = np.unique(np.random.randint(0, MAX_NUM, size=500))
alternative_numbers = np.random.randint(0, MAX_NUM, size=len(problem_numbers))

%timeit method_itzik(numbers, problem_numbers, alternative_numbers)
10 loops, best of 3: 63.3 ms per loop

# This method expects lists
problem_numbers_l = list(problem_numbers)
alternative_numbers_l = list(alternative_numbers)
%timeit method_mseifert(numbers, problem_numbers_l, alternative_numbers_l)
10 loops, best of 3: 20.5 ms per loop

%timeit method_divakar(numbers, problem_numbers, alternative_numbers)
100 loops, best of 3: 9.45 ms per loop

%timeit method_jdehesa(numbers, problem_numbers, alternative_numbers)
1000 loops, best of 3: 822 µs per loop
Improve this answer
Sign up to request clarification or add additional context in comments.

12 Comments

This assumes elements in numbers cover the entire sequence from its min to max.
@Divakar No, it does not require numbers to cover the whole range, the values in replacer that are not in numbers will just not be used (which is why I say that having huge "gaps" in numbers may turn this inefficient, memory-wise at least).
Here's what I tried to get a random array that doesn't cover all numbers within the range : numbers = np.unique(np.random.randint(0,100,(50))) ; numbers[10] = 23 ; numbers[20] = 15 ; numbers[30] = 33 and then shuffle. Value mis-match with this approach.
@Divakar Ohh right, there was an error in the code, thanks.
When you time functions that do some in-place operations you should be using a setup code that is executed before each timing run. That makes sure that subsequent runs aren't biased because the first run already modified the input.
|
1

In case not all problem_values are in numbers and they may even occur multiple times:

In that case I would just use a dict to keep the values to be replaced and use dict.get to translate problematic numbers:

replacer = dict(zip(problem_numbers, alternative_numbers))
numbers_list = numbers.tolist()
numbers = np.array(list(map(replacer.get, numbers_list, numbers_list)))

Even though it has to go "through Python" this is almost self-explaining and it's not much slower than a NumPy solution (probably).

In case every problem_value is actually present in the numbers array and only once:

If you have the numpy_indexed package you could simply use numpy_indexed.indices:

>>> import numpy_indexed as ni
>>> numbers[ni.indices(numbers, problem_numbers)] = alternative_numbers

That should be pretty efficient even for big arrays.

Improve this answer

3 Comments

There is an extra argument in map. Using map over numbers (you can pass it directly as array instead of converting it to list) is probably not better than iterating over problem_numbers, which is apparently smaller, as the OP does.
@jdehesa The constant factor in iterating over an array is much higher than iterating over a list (even if one has to use tolist) so if performance is a concern and python-iteration is required always convert to a list!
@jdehesa The extra argument is because dict.get takes two arguments.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.