Author Topic: Extremely poort performance with LVM vs. RAW disc  (Read 6884 times)

0 Members and 1 Guest are viewing this topic.

Offline ggulik

  • Jr. Member
  • **
  • Posts: 1
  • Karma: 0
  • I've just joined!
    • View Profile
    • Awards
Extremely poort performance with LVM vs. RAW disc
« on: October 29, 2012, 09:00:48 pm »
I am working on setting up a large RAID setup for a variety of functions that aren't very performance critical.  I'm trying to use LVM on the filesystem to allow me to carve up the space for different functions with the flexibility to adjust things later if necessary.

However I'm finding the performance of the LVM based filesystem is many times slower than the raw partition.  I guess I expect some overhead but not 10x.

Specifics of my test configuration:
Dell PowerEdge 1950 with 12G RAM
Dell PERC 6/E RAID controller
Dell MD1000 Controller
8 x Seagate 2TB SATA drives in RAID-6 config (otherwise using default settings in OpenManage)
Operating System is CentOS 6.3

Code: [Select]
Total space available is 12TB and partitioned using parted as follows:
Disk /dev/sdb: 12.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size     File system  Name   Flags
 1      17.4kB  10.0TB  10000GB               data1  lvm
 2      10.0TB  12.0TB  1999GB   ext4         data3

I then created a 2TB ext4 filesystem mounted on /data2 using /dev/sdb2 physical partition and created a 2TB logical volume using the LVM partition and also formatted it with ext4 mounted on /data2

Then when I compare basic read/write performance the difference is shocking.  Using:
time sh -c "dd if=/dev/zero of=ddfile bs=8k count=2000000 && sync"
and
time dd =if=ddfile of=/dev/null bs=8k
Below are the average of several attempts

Raw partition:
Read:  48.3 secs
Write:  70.2 secs

LVM partition:
Read:  288.3 secs
Write: 701.6 secs

I'm stumped as to why it's so much slower

Offline Mad Penguin

  • Administrator
  • Hero Member
  • *****
  • Posts: 1420
  • Karma: 10018
  • Gender: Male
    • View Profile
    • Linux in the UK
    • Awards
Re: Extremely poort performance with LVM vs. RAW disc
« Reply #1 on: October 30, 2012, 03:56:12 am »
Hi, your problem is almost certainly your RAID setup.

In general hardware RAID should be avoided, Dell implementations especially. Configure up each drive as a RAID-0 or JBOD device (depending on your RAID BIOS) and then use Linux RAID10 or ZFS-Z2 on top. I would be expecting the setup you describe to be delivering ~ 350Mb/sec, whereas with a software RAID implementation you should see closer to 1000Mb/sec on that hardware.

There are many (subtle) reasons, and you can tune the performance up, bit it simply isn't worth is given the underlying overheard your hardware RAID is already imposing. (You can check to see if your RAID driver supports the merge_bvec_fn() function, I'm guessing not ..)

I would *strongly* recommend using ZFS on a Raid-Z2, which is effectlively RAID-6, but it will stripe-read across all devices, so you should see a full 900Mb/sec+ AND you can allocate block devices directly from ZFS pools without using LVM.

Eg;

Code: [Select]
zfs create -b 4096 -V 12G pool/images/block1 

Will create a block device;

Quote
/dev/pool/images/block1


This device is a *sparse* device, so it will only eat space as you need it, AND you can turn on implicit compression, AND you can turn on automatic deduplication. Makes LVM look a little old / sad .. :)

There is a Howto on getting ZFS going here ... http://linux.co.uk/pages/spof-1-storage-node/

 


SimplePortal 2.3.3 © 2008-2010, SimplePortal