GC overhead limit exceeded in MySQL query

Question

I have a big database consisting of 170k Items and each Item having 20-150 features. When I run the method, which retrieves this data into a map, consisting of item_id and list of features, I get Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded. The method runs around 30 minutes and throws exception. Here is the method:

public Map<Integer, List<ItemFeatureMapping>> getItemFeatures() {
    List<Item> allItems = getAllItems();
    Map<Integer, List<ItemFeatureMapping>> result = new HashMap<>();
    for (Item i : allItems) {
        List<ItemFeatureMapping> itemFeatures = new ArrayList<>();
        for (ItemFeatureMapping feature: i.getItemFeatures()) {
            itemFeatures.add(feature);
        }
        result.put(i.getId(), itemFeatures);
    }
    return result;
}

I looked in the manual:

The parallel collector will throw an OutOfMemoryError if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If necessary, this feature can be disabled by adding the option -XX:-UseGCOverheadLimit to the command line.

How can I sort this out? Or is it possible to optimize the method?

P.S.: I tried to do via command line, but getting an error that Command not found. However, I dont like this solution, since the method then may be running for a long time.

You can increase the heap size of the jvm by using -Xmx4g parameter. — dabaicai
– dabaicai, Commented Jun 1, 2017 at 14:42

Simulant · Accepted Answer · 2017-06-01 15:29:51Z

You could remove every processed item from your list allItems and item.getItemFeatures() after you added it to your target map/list. The memory used by this item can then be garbage collected and reused by the map/list you create.

In Addition you could initialize your collections with the expected size in the constructor. In that way you would not waste any memory by an allocated but never used place for entries in your collections.

If this does not fix your problem, you have to increase your heap size. Because otherwise the representation of all your features will not fit into your allocated memory:

public Map<Integer, List<ItemFeatureMapping>> getItemFeatures() {
    List<Item> allItems = getAllItems();
    Map<Integer, List<ItemFeatureMapping>> result = new HashMap<>(allItems.size(), 1f);
    Iterator<Item> iterator = allItems.iterator();
    while (iterator.hasNext()) {
        Item item = iterator.next();
        List<ItemFeatureMapping> itemFeatures = new ArrayList<>(item.getItemFeatures().size());

        Iterator<ItemFeatureMapping> iteratorFeature = item.getItemFeatures().iterator();
        while (iteratorFeature.hasNext()) {
            ItemFeatureMapping feature = iteratorFeature.next();
            itemFeatures.add(feature);
            iterator.remove();
        }
        result.put(item.getId(), itemFeatures);
        iterator.remove();
    }
    return result;
}

phoenixSid · Accepted Answer · 2017-06-01 14:52:27Z

0

Instead of creating new itemFeatures objects for every loop execution, you can create the list instance once outside the loop and then clearing the list using

itemFeatures.clear();

And, as @dabaicai mentioned, try giving it more jvm heap size.

Also, to help debug the issue faster, please try to include all possible information such as

Java version
heap size
jvm parameters

answered Jun 1, 2017 at 14:52

phoenixSid

4771 gold badge9 silver badges22 bronze badges

1 Comment

dabaicai Over a year ago

In your solution,the every key will hold a same value(the itemFeatures) ,I think it isn't correctly.

Collectives™ on Stack Overflow

GC overhead limit exceeded in MySQL query

2 Answers 2

Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Related