Illuminate: processor for Illumina InterOP Run binaries (in Python)

[nthmost]

Greetings all,

I work at InVitae and we just publicly released a library called Illuminate.

https://bitbucket.org/invitae/illuminate/

The purpose of Illuminate is to emulate the stats you see when you load a run data folder within Illumina SAV, providing programmatic access to these metrics for whatever purposes you may have -- data storage, analysis, automated machine monitoring, and so on.

This is completely free, open source software (MIT License) written in Python with the intent to be used, tested, and improved upon by the bioinformatics community.

Features:

• Simple command-line tool you can use to quickly inspect a run.
• Built to be easily integrated into other code.
• Easily extensible even if you think you are "not much of a programmer".
• Results standardized to pandas DataFrame objects (so if you know how to work in R, you can probably get up to speed quickly with this)

Here's an example of the smallest python script you could get away with using this tool.

import illuminate
myDataset = illuminate.InteropDataset('path/to/rundata/')
print myDataset.meta
print myDataset.IndexMetrics()
print myDataset.TileMetrics()
print myDataset.QualityMetrics()

And here's an example of how you would use the command-line reporter to do the same thing:

python illuminate --meta --index --tile --quality /path/to/rundata

You can even have illuminate open up in an interactive iPython shell, where the dataset will be loaded up into an InteropDataset object for you:

python illuminate -i /path/to/rundata

Not all of the metrics objects are fully fleshed out yet, although all of the binary parsers are "feature complete" in that you can produce a data dictionary and a DataFrame from them.

I'm hoping that some of you fine folks can pipe up and let me know what might be useful to you -- or better, submit contributions, bug reports, and so on that will help Illuminate become as full-featured as it needs to be.

This library has been in our production pipeline for several months now, reporting on cluster density, quality, and yield so we can keep tabs on sequencing run quality in an automated fashion.

If you use it, or you have questions about it, please comment here and let me know!

Cheers,
Naomi

[lingfung.tang]

interesting, never thought of cracking the bin files.. thanks.

[nthmost]

Now available through pypi (pip)

I just pushed the latest stable version of Illuminate to the public pypi server, so you can install it via:

• pip install illuminate

Note that, due to weirdness in python packaging, you will probably have to install two of its dependencies manually:

• pip install numpy
• pip install pandas

More detailed instructions can be found on the pypi.python.org page for Illuminate.

Stable versions will always be found there; development versions will always come from the bitbucket repository (see original post in this thread).

Cheers,
Naomi