Importing and processing data in R line by line

[gwilymh]

I am analyzing large datasets in R. To analyze data, my current practice is to import the entire dataset into the R workspace using the read.table() function. Rather than importing the entire dataset, however, I was wondering if it is possible to import, analyze and export each line of data individually so that the analysis would take up less computer memory.

Can this be done? And if so, how?

[dariober]

Originally Posted by gwilymh

I am analyzing large datasets in R. To analyze data, my current practice is to import the entire dataset into the R workspace using the read.table() function. Rather than importing the entire dataset, however, I was wondering if it is possible to import, analyze and export each line of data individually so that the analysis would take up less computer memory.

Can this be done? And if so, how?

Hi- R is designed to read all the datafile in one go. Reading line by line might be possible but is probably going to be horribly slow. However, instead of reading line by line you could read-in chunks of several lines in a loop like (pseudocode):

totlines<- 10000000 ## Number of lines in your big input. Get it from wc -l
skip<- 0
chunkLines= 10000 ## No. of lines to read in one go. Set to 1 to really read one line at a time.
while (skip < totlines){
    df<- read.table(myinput, skip= skip, nrows= chunkLines, stringsAsFactors= FALSE)
    skip<- skip + chunkLines
    [...do something with df...]
}

Essentially you use args skip and nrows to read chunks of lines. To speed-up read.table set stringsAsFactors to false.

A better alternative might be to use packages designed for dealing with data larger than memory, ff (http://cran.r-project.org/web/packages/ff/index.html) is one of them.

Hope this helps!

Dario

[gwilymh]

Thanks Dario, much appreciated.

[nexgengirl]

Check out the readLines function in R.