java.lang.OutOfMemoryError when reading from a large table

Question

I'm trying to pre generate reports from a very large table(300GB) in PostgreSQL. I do something like this:

rs = stmt.executeQuery("SELECT * FROM tbl");
System.out.println("select all finished");
while (rs.next()) {
    /* generate report and save it in report table */
    /* generated reports are not in memory, 
     * They are saved in a summary table in each iteration */
}

When I start the application it gives Exception in thread "main" java.lang.OutOfMemoryError: Java heap space. I tried to use stmt.setFetchSize(1000) But it doesn't solve the problem.

What is the solution? I'm using PostgreSQL 8.4.11 on Debian 6.0.5 and openJDK 6.

[UPDATE]

The printed stack trace shows that the OutOfMemoryError exception has been generated in rs = stmt.executeQuery("SELECT * FROM tbl"); line. Also System.out.println("select all finished"); never shows.

1. I'm running in autocommit mode.
2. stmt.getResultSetConcurrency() returns 1007.
3. stmt.getResultSetHoldability() return 2.
4. rs.getType() returns 1003.

Answer 1

The problem is probably that PostgreSQL only uses the fetchSize in a narrow set of circumstances. See: http://jdbc.postgresql.org/documentation/91/query.html#fetchsize-example

• The connection to the server must be using the V3 protocol. This is the default for (and is only supported by) server versions 7.4 and later.
• The Connection must not be in autocommit mode. The backend closes cursors at the end of transactions, so in autocommit mode the backend will have closed the cursor before anything can be fetched from it.
• The Statement must be created with a ResultSet type of ResultSet.TYPE_FORWARD_ONLY. This is the default, so no code will need to be rewritten to take advantage of this, but it also means that you cannot scroll backwards or otherwise jump around in the ResultSet.
• The query given must be a single statement, not multiple statements strung together with semicolons.

So if you are executing this in auto-commit, or with a resultset type other than TYPE_FORWARD_ONLY PostgreSQL will fetch all rows. Also looking at the sources of the PostgreSQL JDBC 9.0-801 driver it looks like using a holdable resultset will also make it fetch all rows.

Answer 2

I don't think what you have there would cause that kind of error. I believe the garbage collector comes along as your iterating through rs.next(), so you shouldn't have a memory problem. It probably has something to do with what you're trying to do with the result set. Without knowing exactly what you're doing though, I can only guess you're trying to store everything in an object in memory. So, if you were storing the values into a StringBuilder or something that would be a problem. I recommend writing the results to disk as you go, rather than trying to collect it all into an object in memory (again, I'm just guessing what you're doing because you didn't provide that kind of information). In the Java Helper Library, there's a resultSetToCSVFile(ResultSet rs, String destination) method which you might find helpful. Handling it this way prevents you from holding it all in memory but you can write the report you're trying to. By the way, to do this, you'll need to include the opencsv library. Or you could just call the method directly by including the Java Helper Library.

 /**
   * Prints the given ResultSet to a comma separated file (the destination)
   *
   * @param rs
   * @param destination
   * @throws SQLException
   * @throws FileNotFoundException
   */
  public static void resultSetToCSVFile(ResultSet rs, String destination) throws SQLException, FileNotFoundException, IOException {
    ResultSetMetaData metaData = rs.getMetaData();
    int columnCount = metaData.getColumnCount();
    String[] header = new String[columnCount];
    for (int i = 0; i < columnCount; i++) {
      header[i] = metaData.getColumnName(i + 1);
    }
    File file = new File(destination);
    IOHelper.checkDirectory(file);
    try (PrintWriter pw = new PrintWriter(file); CSVWriter writer = new CSVWriter(pw)) {
      writer.writeNext(header);
      while (rs.next()) {
        String[] row = new String[columnCount];
        for (int i = 0; i < columnCount; i++) {
          String string = rs.getString(i + 1);
          if (string == null) {
            string = "";
          }
          row[i] = string;
        }
        writer.writeNext(row);
      }
    }
  }