Xen and LVM backed domu's

[Sam Fourman Jr.]

Hello List,

There does not seem to be much on Google, for how to use a NetBSD-5
kernel (Not 5.02) and use LVM to slop disk space to several xen domu's
does anyone have a quick example? I have been a BSD addict for years
and I never bothered to look up the Linux Documentation for LVM.

I assume the Linux Docs would be the same for NetBSD (syntax and such)
but what I am confused about is how to I get LVM enabled without
switching to current?

Thank you for your Help

Sam Fourman Jr.

[Adam Hamsik]

Post by Sam Fourman Jr.
Hello List,
There does not seem to be much on Google, for how to use a NetBSD-5
kernel (Not 5.02) and use LVM to slop disk space to several xen domu's
does anyone have a quick example? I have been a BSD addict for years
and I never bothered to look up the Linux Documentation for LVM.
I assume the Linux Docs would be the same for NetBSD (syntax and such)
but what I am confused about is how to I get LVM enabled without
switching to current?

Unfortunately you need current for LVM support. You can find nice documentation for LVM at [1].

[1] http://www.netbsd.org/docs/guide/en/chap-lvm.html

Regards

Adam.

[Hugo Silva]

Post by Adam Hamsik

Post by Sam Fourman Jr.
Hello List,
There does not seem to be much on Google, for how to use a NetBSD-5
kernel (Not 5.02) and use LVM to slop disk space to several xen domu's
does anyone have a quick example? I have been a BSD addict for years
and I never bothered to look up the Linux Documentation for LVM.
I assume the Linux Docs would be the same for NetBSD (syntax and such)
but what I am confused about is how to I get LVM enabled without
switching to current?

Unfortunately you need current for LVM support. You can find nice documentation for LVM at [1].
[1] http://www.netbsd.org/docs/guide/en/chap-lvm.html
Regards
Adam.

FYI,

I recently found out that HVM domUs running with lvm-backed storage
detect a 0MB hard disk - this is not a domU-OS specific problem, as qemu
itself reports 0MB
in the "bios" - thus making them unusable.

Not sure where the problem might be here, but I suspect qemu. Xen, as I
recall, (I don't have my NetBSD/Xen box nearby) correctly detects the
size of the logical volume.

[Sam Fourman Jr.]

FYI,
I recently found out that HVM domUs running with lvm-backed storage detect a
0MB hard disk - this is not a domU-OS specific problem, as qemu itself
reports 0MB
in the "bios" - thus making them unusable.
Not sure where the problem might be here, but I suspect qemu. Xen, as I
recall, (I don't have my NetBSD/Xen box nearby) correctly detects the size
of the logical volume.

What would you suggest is the best way to allocate expandable disk
space to several xen domu's

I Favor ZFS but ZFS will need some time on NetBSD before it is ready
to go, LVM seemed like a logical second choice,
but 0 bytes for HVM is kind of a show stopper, I have heard that file
backed solutions are bad disk i/o, so I am open to suggestions.


Sam Fourman Jr.

[haad]

I recently found out that HVM domUs running with lvm-backed storage detect a
0MB hard disk - this is not a domU-OS specific problem, as qemu itself
reports 0MB
in the "bios" - thus making them unusable.
Not sure where the problem might be here, but I suspect qemu. Xen, as I
recall, (I don't have my NetBSD/Xen box nearby) correctly detects the size
of the logical volume.

Hmm yes that can be easily problem if Xen qemu uses some different
disk ioctls which are not supported by dm driver.

I think that ideal would be change disk_ioctl[1] routine to work with
more that one IOCTL and listen for all other different disk size
ioctls calls. disk_ioctl can use dictionary to create every possible
structure for caller.

Please make a PR for this issue because otherwise it can be lost.
Because I'm little bit busy these
days with ZFS.

[1] http://nxr.netbsd.org/xref/src/sys/kern/subr_disk.c#487

[Christoph Egger]

Post by haad

Post by Hugo Silva
I recently found out that HVM domUs running with lvm-backed storage
detect a 0MB hard disk - this is not a domU-OS specific problem, as qemu
itself reports 0MB
in the "bios" - thus making them unusable.
Not sure where the problem might be here, but I suspect qemu. Xen, as I
recall, (I don't have my NetBSD/Xen box nearby) correctly detects the
size of the logical volume.

Hmm yes that can be easily problem if Xen qemu uses some different
disk ioctls which are not supported by dm driver.
I think that ideal would be change disk_ioctl[1] routine to work with
more that one IOCTL and listen for all other different disk size
ioctls calls. disk_ioctl can use dictionary to create every possible
structure for caller.

This problem is not specific to LVM. You can reproduce this when you tell Xen
to use a physical disk in general for the domU.

Post by haad
Please make a PR for this issue because otherwise it can be lost.
Because I'm little bit busy these
days with ZFS.
[1] http://nxr.netbsd.org/xref/src/sys/kern/subr_disk.c#487

[Sam Fourman Jr.]

Post by Christoph Egger

Post by haad

Post by Hugo Silva
I recently found out that HVM domUs running with lvm-backed storage
detect a 0MB hard disk - this is not a domU-OS specific problem, as qemu
itself reports 0MB
in the "bios" - thus making them unusable.
Not sure where the problem might be here, but I suspect qemu. Xen, as I
recall, (I don't have my NetBSD/Xen box nearby) correctly detects the
size of the logical volume.

Hmm yes that can be easily problem if Xen qemu uses some different
disk ioctls which are not supported by dm driver.
I think that ideal would be change disk_ioctl[1] routine to work with
more that one IOCTL and listen for all other different disk size
ioctls calls. disk_ioctl can use dictionary to create every possible
structure for caller.

This problem is not specific to LVM. You can reproduce this when you tell Xen
to use a physical disk in general for the domU.

is this a NetBSD specific problem? are there any proposed workarounds?

Sam Fourman Jr.
Fourman Networks

[Sam Fourman Jr.]

FYI,
I recently found out that HVM domUs running with lvm-backed storage detect a
0MB hard disk - this is not a domU-OS specific problem, as qemu itself
reports 0MB
in the "bios" - thus making them unusable.
Not sure where the problem might be here, but I suspect qemu. Xen, as I
recall, (I don't have my NetBSD/Xen box nearby) correctly detects the size
of the logical volume.

I am confused a bit, doesn't this fix these issues?

src/sys/arch/xen/xen/xbdback_xenbus.c v 1.31
Finaly fix problems with using WEDGE like devices as xen backend devices.

or is this issue specific to HVM's

Sam Fourman Jr.

[Brian]

Post by Sam Fourman Jr.
I am confused a bit, doesn't this fix these issues?
src/sys/arch/xen/xen/xbdback_xenbus.c v 1.31
Finaly fix problems with using WEDGE like devices as xen backend devices.
or is this issue specific to HVM's
Sam Fourman Jr.

No it does not, as this appears to be HVM specific. I noticed the same
problem Hugo reported recently and I have spent a small amount of time
trouble shooting this.

I usually use the character devices for my partition-backed domU's, like
/dev/rraid0j; however, with the dm driver (and phy:/dev/mapper/rv0-lv0)
I am getting an EBUSY error on VOP_OPEN in xbdback_xenbus.c. The qemu
BIOS then reports 0MB as Hugo described.

Any combination of /dev/mapper/v0-lv0 or /dev/mapper/rv0-lv0 produces
different errors from xbdback_xenbus.c while reporting 0MB in the qemu BIOS.

If I specify /dev/mapper/v0-saturn, I get this in my dmesg:
xbd backend: attach device v0-saturn (size 62914560) for domain 10

But qemu BIOS still reports 0MB for the drive.

If I specify /dev/mapper/rv0-saturn, I get this instead:
xbdback backend/vbd/11/832: unknown device 0xc202

With a disk entry like: 'phy:/dev/mapper/rv0-saturn,hdb,w'
Regardless of what I replace hdb with, the 0xc202 device number just
changes but still fails.

I have noticed that on one of my other HVM domU's which uses physical
drives (/dev/rwd2d) that an unknown device error from xbdback_xenbus.c
does not affect qemu's detection of the disk size.

When specifying the character device rv0-saturn, I do not get any
messages in my /var/log/xen/qemu-dm-saturn.log; however, when I specify
the block device v0-saturn, I get this:

qemu: could not open vbd '/local/domain/10/device/vbd/832/phantom_vbd'
or hard disk image '/dev/mapper/v0-saturn' (drv 'phy' format 'raw')

I will continue to work on figuring out why this is happening as I know
Adam is busy. I do not have any other HVM-capable machines here for the
time being so my progress is slow as the machine I was debugging with is
used for production.

Sincerely,
Brian Brombacher
PlanetUnix Networks

[Adam Hamsik]

Hi Brian,

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Sam Fourman Jr.
I am confused a bit, doesn't this fix these issues?
src/sys/arch/xen/xen/xbdback_xenbus.c v 1.31
Finaly fix problems with using WEDGE like devices as xen backend devices.
or is this issue specific to HVM's
Sam Fourman Jr.

No it does not, as this appears to be HVM specific. I noticed the same
problem Hugo reported recently and I have spent a small amount of time
trouble shooting this.
I usually use the character devices for my partition-backed domU's, like
/dev/rraid0j; however, with the dm driver (and phy:/dev/mapper/rv0-lv0)
I am getting an EBUSY error on VOP_OPEN in xbdback_xenbus.c. The qemu
BIOS then reports 0MB as Hugo described.

I think that we are fighting 2 problems here

1) EBUSY error
2) size 0 in qemu error

The 1) is probably not from dm driver because it do not return EBUSY from its
sources. Maybe this error comes from specfs_open but I'm not sure why.

The 2) is I think issue in qemu libraries I looked at source code and I have found one place which can be wrong. I will try to make some patch which can be tested.

Any combination of /dev/mapper/v0-lv0 or /dev/mapper/rv0-lv0 produces
different errors from xbdback_xenbus.c while reporting 0MB in the qemu BIOS.
xbd backend: attach device v0-saturn (size 62914560) for domain 10
But qemu BIOS still reports 0MB for the drive.
xbdback backend/vbd/11/832: unknown device 0xc202
With a disk entry like: 'phy:/dev/mapper/rv0-saturn,hdb,w'
Regardless of what I replace hdb with, the 0xc202 device number just
changes but still fails.
I have noticed that on one of my other HVM domU's which uses physical
drives (/dev/rwd2d) that an unknown device error from xbdback_xenbus.c
does not affect qemu's detection of the disk size.
When specifying the character device rv0-saturn, I do not get any
messages in my /var/log/xen/qemu-dm-saturn.log; however, when I specify
qemu: could not open vbd '/local/domain/10/device/vbd/832/phantom_vbd'
or hard disk image '/dev/mapper/v0-saturn' (drv 'phy' format 'raw')
I will continue to work on figuring out why this is happening as I know
Adam is busy. I do not have any other HVM-capable machines here for the
time being so my progress is slow as the machine I was debugging with is
used for production.
Sincerely,
Brian Brombacher
PlanetUnix Networks
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (OpenBSD)
iEYEARECAAYFAkuHMSsACgkQGTUZ+lOxJfDt/wCePkOUMHz4IElAQRd/Q/wsahOb
1/oAmgOO9gXsALV9gM4tkgLvRwme+Obg
=/R96
-----END PGP SIGNATURE-----

Regards

Adam.

[Adam Hamsik]

Hi folks,

I have patch witch fixes this issue for me it is at [1]. You need to apply this patch against xentools33 to make them working.I just type Ctr-C before pkgsrc starts build apply patch and continue in build.

To fix EBUSY issue you need to use raw block device in your config. Like example attached

# cat /usr/pkg/etc/xen/FreeBSD8
kernel = '/usr/pkg/lib/xen/boot/hvmloader'
builder = 'hvm'
memory = '384'
device_model='/usr/pkg/libexec/qemu-dm'

disk = [ 'phy:/dev/mapper/rvg00-testlv,ioemu:hda,w'] <- raw logical volume partition

I would like to thanks Sam Fourman who helped me a lot to find and fix this problem.

[1] www.netbsd.org/~haad/block-raw.diff



Regards

Adam.

[Hugo Silva]

Post by Adam Hamsik
Hi folks,
I have patch witch fixes this issue for me it is at [1]. You need to apply this patch against xentools33 to make them working.I just type Ctr-C before pkgsrc starts build apply patch and continue in build.
To fix EBUSY issue you need to use raw block device in your config. Like example attached
# cat /usr/pkg/etc/xen/FreeBSD8
kernel = '/usr/pkg/lib/xen/boot/hvmloader'
builder = 'hvm'
memory = '384'
device_model='/usr/pkg/libexec/qemu-dm'
disk = [ 'phy:/dev/mapper/rvg00-testlv,ioemu:hda,w'] <- raw logical volume partition
I would like to thanks Sam Fourman who helped me a lot to find and fix this problem.
[1] www.netbsd.org/~haad/block-raw.diff
Regards
Adam.

I'm giving this a go -- so far so good :-)

Thank you Adam.

BR,

Hugo

[Brian]

Post by Adam Hamsik
Hi folks,
I have patch witch fixes this issue for me it is at [1]. You need to apply this patch against xentools33 to make them working.I just type Ctr-C before pkgsrc starts build apply patch and continue in build.
To fix EBUSY issue you need to use raw block device in your config. Like example attached
# cat /usr/pkg/etc/xen/FreeBSD8
kernel = '/usr/pkg/lib/xen/boot/hvmloader'
builder = 'hvm'
memory = '384'
device_model='/usr/pkg/libexec/qemu-dm'
disk = [ 'phy:/dev/mapper/rvg00-testlv,ioemu:hda,w'] <- raw logical volume partition

I have applied this patch as well and it appears to be working great.
Thank you for your continued support.

Sincerely,
Brian Brombacher
PlanetUnix Networks