4

I want to read binary integers in R and convert them into raster grids. The files have the following charterers:

NCols= 4320
NRows= 2160
pixel-size: 1/12=0.833 degrees
upper-left-lat: 90.0-1/24
upper-left-lon: -180.0+1/24
lower-right-lat: -90.0+1/24
lower-right-lon: 180.0
nodata= -5000
scale-factor= 10000
datatype: 16-bit signed integer
byte-order: big endian

Here is what I do:

file <-"http://nasanex.s3.amazonaws.com/AVHRR/GIMMS/3G/1980s/geo81aug15a.n07-VI3g"
dat <- readBin(file,what="integer", size=4, signed = TRUE, n = NRows * NCols, endian = "big")
r <- raster(nrow=2160, ncol=4320)
r[] <- dat

But this doesn't seem to be right, I appreciate any suggestions. .

Improve this question
5
  • Not really sure what you're after here, can you be more descriptive? Commented Sep 24, 2015 at 19:16
  • I think the data has subsets. Here is what I found from the documents: Each NDVI data set is an INT16 file saved with ieee-big_endian it ranges from -10000->(10000->10004) with the flagW file added to the ndvi values as follows: ndvi3g = round(ndvi*10000) + flagW - 1; flagW ranges from 1->7 to retrieve the original ndvi and flagW values flagW = ndvi3g-floor(ndvi3g/10)*10 + 1; ndvi = floor(ndvi3g/10)/1000 Do you know how I can retrieve the ndvi values? Commented Sep 24, 2015 at 19:22
  • @DNM Do you have a special interest in the precise file format of these files and want to understand those details? Or do you just want to end up with a raster so you can display it? Commented Sep 24, 2015 at 19:39
  • Perhaps this will help you out. This question is really more suitable for the GIS stack, you may have more luck searching there. Good Luck! Commented Sep 24, 2015 at 19:42
  • @WhiteViking right now I'm just want to be able to properly make a raster grid out of them, then I want to re-project and analyze them. Commented Sep 24, 2015 at 19:50

2 Answers 2

Reset to default
4

I built greenbrown from source (based on the files staged on GitHub) and found that it took considerably long to process one single file. 

system.time(
  r1 <- ReadVI3g("http://nasanex.s3.amazonaws.com/AVHRR/GIMMS/3G/1980s/geo81aug15a.n07-VI3g")
)

#   user  system elapsed 
#  3.252   0.973 143.846

Therefore, I suggest to have a look at the gimms package which has been designed for this particular kind of data and, moreover, is available from CRAN. Note that in contrast to ReadVI3g, it does not offer automated quality control yet, but this feature is scheduled for the next version update. In the meantime, overlay from the raster package should be employed to discard low-quality values.

# install.packages("gimms")
library(gimms)

system.time({
  ## download file, see ?downloadGimms for further options
  f <- updateInventory()
  f <- downloadGimms(f[3], overwrite = TRUE) # download 3rd file in 'f', viz. geo81aug15a.n07-VI3g

  ## rasterize ndvi and flags
  ndvi <- rasterizeGimms(f)
  flag <- rasterizeGimms(f, flag = TRUE)

  ## perform quality control
  r2 <- overlay(ndvi, flag, fun = function(x, y) {
    x[y[] > 1] <- NA
    return(x)
  })
})

#   user  system elapsed 
#  4.538   3.894  26.781

The two resulting images are obviously identical

> unique(r1 - r2, na.rm = TRUE)
[1] 0

but as you can see, the gimms-based code performs much faster. Moreover, it offers parallel functionality (via doParallel) in case you would like to download and process multiple files at once.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks @fdetsch! This is very interesting!
2

You can read such files with the greenbrown R package.

Install it in R with

install.packages("greenbrown", repos="http://R-Forge.R-project.org")

If that fails because the package needs to be rebuilt by its authors, an alternative is to first download the sources directly from the repo, and then install them manually, as explained in the greenbrown installation instructions. In the latter case you may also have to manually install a couple of packages that greenbrown depends on first: install.packages on Kendall, bfast, strucchange.

After installation, reading the raster from a URL is as easy as:

library(greenbrown)
r <- ReadVI3g("http://nasanex.s3.amazonaws.com/AVHRR/GIMMS/3G/1980s/geo81aug15a.n07-VI3g")

The object returned by greenbrown::ReadVI3g is a RasterLayer. We can display it with

plot(r)

which gives

enter image description here

Improve this answer

4 Comments

it seems like an interesting package! but as you pointed the package fails to install.
@DNM It failed for me to, but I used svn to clone the repo on my machine locally, then installed the package "manually" following the installation instructions I linked to in the answer. (Straightforward, only takes a couple of minutes.)
thanks it works now! I built it under linux first then copied the package into windows environment. It seems that they created a function for these data sets, but I'm still interested to know how to read in the data manually. Thanks for your help!
If you like to understand the details of how to read the data manually, just have a look at the greenbrown package source code... The file greenbrown/pkg/greenbrown/R/ReadVI3g.R only implements ReadVI3g() It is well documented, nice and small and quite easy to understand. It uses readBin() too, just like you started.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.