Use of glmer.nb to estimate mixed model effects on RNAseq data

[gtosser]

Hello,

This is my first post on this site but I have already read many interesting posts here and I wish to thank all the participants.

I have a RNAseq dataset generated with different factors:
- species: 2 levels (pig, cattle)
- treatment: 2 levels (control, treated)
- animal: 10 levels
I want to know what are the differentially expressed genes between control and treated conditions and if there is a species effect and also if there are species*gene interactions
I have normalized the data (11882 genes) with Deseq2 and extracted the data from Deseq2

>dds<-estimateSizeFactors(dds)
>dds<-estimateDispersions(dds)
>dds<-nbinomWaldTest(dds)
>normalizedCounts <- t( t(counts(dds)) / sizeFactors(dds) )
>dim(normalizedCounts)
# [1] 11882 40

After that I have constructed data1 where all the normalizedCounts are in column2 and the factors in columns 3,4,5. Column 1 is the gene name.
>dim(data1)
# [1] 475280 5
I want to know what are the differentially expressed genes when treatment is used and if there is a species effect and a gene*species interaction and so on...

>glmer.nbIA <- glmer.nb(counts~gene+treatment+species+gene*species+(1|animal)-1,data=data1, verbose=TRUE)

But, the model does not run:
>Error: cannot allocate vector of size 126.2 Gb

I thought glmer.nb was adapted to deal with RNAseq data but maybe I am wrong. I did not find any publication using it on RNAseq data with lists of differentially expressed genes.
Should the needed memory needed so huge?
Thanks a lot for your help
Gwenola Tosser-Klopp