server for 'omics

[furor]

Hello all,

I know threads like these appear once in a while, but hardware changes constantly. I need to build/buy a server/workstation for the processing and analysis of quite large meta-omics and genomics datasets (e.g. 40+ HiSeq runs for one upcoming project). Our current server mainly lacks RAM and storage, and we want a dedicated server for these large datasets, for a limited number of people to work on (2-4).

I currently have a budget of €10-11k, incl. VAT. I know it won't be state of the art with this budget (Xean E7 ...), but will be a vast improvement.

One prerequisite is that it can be extended (CPU, RAM, storage).
Since the number of cores (24) is not the bottleneck with our current server, I reckon anywhere between 16 and 24 cores (physical + virtual) will suffice. Nevertheless, since some analyses or processing steps are per definition single-threaded, it should be powerful (fast) enough.

Sufficient memory, however, is a limiting factor. Our current server is limited to 64 GB. 128 (192) GB seems to be the bare minimum, but preferably in the 300-400 GB range, which is not that impressive to current standards I guess, but somewhat futureproof for our research group. Being able to extend this amount is thus necessary.

Sufficient local storage to not having to read/write to an external data server constantly.

I am not sure if having RAID configurations for OS/data disks is really necessary, since the budget needed for this might be more useful to increase the effective storage capacities ...

I have composed and built several desktops in the past, and this was always cheaper than buying a branded PC (HP, Dell, ...). In other words, for the same amount of money, you could have better components. Can I assume this holds true for a server, or are servers of e.g. Dell more competitively priced?

Anyway, my current configuration is:

CPU: Xeon E5-2640 v4, 10 cores (20 virtual), 2.4 GHz (3.6 turboboost), 25 MB L3 (€1129)
Motherboard: Intel Server Board S2600CW2R (€649)
OS SSD: Samsung 850 EVO pro 512 GB (€259)
Data HDD: WD gold 3*8 TB local storage + 3*8 TB permanent storage/back-up in separate NAS (€2514)
RAM: 8*32 GB (256 GB) (~ €3312)
CPU cooler (± €50)
tower case (± €200)
PSU 700 W (± €150)
UPS (± €200)
DVD/CD ROM drive (± €30)
(raid controller (± €50))

current total ~€10400

What do you think of the current set-up? With this motherboard, 1 additional processor can be added (= 20 threads), RAM and storage can be extended. Although I have not really looked into alternatives.

No definite brands/types of the 'smaller' components yet. Suggestions welcome
Am I forgetting something? (network controller, GPU?)
Other suggestions or alternatives are also welcome.
Does the case really matter? I came across much more expensive server cases.

Would a RAID 0 or 5 vastly improve speed? (Then I would need either 2 or 3 SSDs). If not, I'd rather use this money for an additional HDD or other component.

An alternative is to effectively look at e.g. Dell servers, but propositions by our IT guy did not impress.
Is Supermicro any good/reliable? Suggestions welcome.


Thanks for any input/heads-up, ...!

[furor]

Anybody?

Any input is highly appreciated!

Thanks

[avierstr]

If you have some more months to decide, maybe wait for AMD and their Naples server processors.

https://www.amd.com/en/events/naples-tech-day

32-cores, 2 threads per core = 64 threads in total.
Should become available in Q2 of 2017 (so pretty soon).
Their Ryzen processors for pc has 8 cores (16 threads and is cheaper than intel).

Maybe something to think about.

[GenoMax]

Just go for what you like/fits your budget. Systems administration (in today's world) is a task itself. If you would rather be doing science (than systems administration), leave that part to the pros.

Over the years I have realized that it is impossible for forum participants to know specific circumstances applicable, so it is hard to provide concrete recommendations. Always get the most RAM you can upfront. It would be hard to match/upgrade that later. If that means dropping a couple of notches on the process GHz, then so be it (you will wait for a half hour more on a job that may be running for half-a-day to complete).

[Brian Bushnell]

If you are interested in upgradability, you might consider 64 GB memory modules rather than 32 GB, depending on how many slots the board has. And I'll second avierstr's suggestion regarding Ryzen, if you want to wait around a month or two. With the current Ryzen chips you'll get similar performance to a 10-core Intel chip for a fraction of the cost:

http://techreport.com/review/31846/the-tech-report-system-guide-may-2017-edition/2

...but they don't have enough slots for memory for your situation. The server-class ones will, though.

[avierstr]

It seems that "Naples" was the codename for the new AMD server processor. Actual name is EPYC Processor.
EPYC based servers should be available in June according to this press release:

http://www.amd.com/en-us/press-releases/Pages/amd-raises-expectations-2017may16.aspx

[Markiyan]

For NAS-es, you can go with high end Qnap's or Thecuses:

I've got Thecus N10850, for 1.8K, popped in 32GB of RAM (4x8GB DDR3L 1600) instead of 4GB stock, and than created raid 6 on 9x6TB WD-red pro drives. You can also use 8TB HE filled drives, just DON'T USE SEAGATE SMR disks.


Thecus N10850

https://pricespy.co.uk/product.php?e=1242514

or
QNAP TVS-1282-I7-64G-450W

https://pricespy.co.uk/product.php?e=3919332

or
QNAP TS-1685-D1531-64GR-550W

https://pricespy.co.uk/product.php?p=4249633

with a suitable number of:

WD Red Pro WD8001FFWX 128MB 8TB

https://pricespy.co.uk/product.php?e=3752311

PS: If on a shoe string budget, You can source He drives from WD my book DUO 16TB (~450 quids for a pair), but test them first (make sure they are not SMR ones).

If using NAS as working storage, than use RAID10 instead of RAID6.
Use SSD caching feature if available (have a hybrid array),

Make sure to use large stripe size 1MB - 16MB (depending on a NAS config), and up the soft raid stripe cache size:

/etc/udev/rules.d/60-md-stripe-cache.rules:

#increase RAID5/6 stripe cache size
SUBSYSTEM=="block", KERNEL=="md*", ACTION=="change", TEST=="md/stripe_cache_size", ATTR{md/stripe_cache_size}="32768", ATTR{bdi/read_ahead_kb}="8192"

#and also up the read ahead for devs
SUBSYSTEM=="block", ATTR{queue/rotational}=="1", ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{bdi/read_ahead_kb}="4096"

#some usb speed tweaks
ACTION=="add", KERNEL=="sd[a-z]", SUBSYSTEM=="block", ENV{ID_BUS}=="usb", ATTR{max_sectors}="32678"

[furor]

Thanks for the input all!

64GB RAM is nice, but too expensive (at least, those I find).

The new AMD seems nice, but I don't know if my superiors would like to wait much longer.

It appears that I need a GPU. What could you recommend?
Is it better to have a dedicated swap (SSD) disk?

OS will be on 2 SSDs in RAID 0.
Local storage will be RAID5 (3 to 6 8 TB disks)

The main goal of this server/workstation is shotgun/amplicon (meta)genome processing, assembly and analysis, intensive data analysis, alignments, and (remote sensing) image processing.
So both parallel as single thread jobs, with likely lots of RAM needed.

I do think that this is an ok set-up for this budget?

[GenoMax]

It appears that I need a GPU. What could you recommend?
Is it better to have a dedicated swap (SSD) disk?

No you don't, but if you want it we won't tell Unless "image processing" refers to a real task you are planning to do.

SSD's have a lifetime of writes and using one for swap would not do it any favors. So I would say use a regular fast drive.