Time  Nick         Message
03:04 pdurbin      westmaas: well, oVirt/RHEV allows you to specify users who have the power to spin up VMs
03:04 pdurbin      or so the power point slides say, at least
03:05 pdurbin      or maybe it's coming in RHEV 3.1. i can't be bothered to check at the moment
03:05 pdurbin      what i do want to talk about is iSCSI with RHEV/oVirt
03:06 pdurbin      sjoeboo and I chatted with Dan (Dmitry) Yasny, Product Manager - Virtualization and Cloud at Red Hat: http://www.linkedin.com/in/dyasny
03:07 pdurbin      he was super nice and very helpful
03:07 pdurbin      explained a lot that would have taken a lot in a few minutes that would have taken a lot of reading
03:08 pdurbin      in a nutshell, RHEV/oVirt deals with a lot of the dirty details of iSCSI for you
03:08 pdurbin      basically, you present it a LUN and it can start using it for VM disk image storage
03:09 pdurbin      under the hood it's using LVM
03:09 pdurbin      and if you need more space, you just feed it another LUN
03:10 pdurbin      (a lot of this applies to fibre channel SANs too, not just iSCSI)
03:12 pdurbin      anyway, to put this in the context of agoddard's wonderful dissertation on iSCSI + libvirt ( http://lists.centos.org/pipermail/centos-virt/2012-June/002946.html ), this model that RHEV/oVirt uses is "5) large iSCSI LUN with LVM, with LVM operations managed by a single host"
03:12 pdurbin      except that you could add in more LUNs later if you need to
03:13 pdurbin      now, the "single host" in this case, that does all the LVM logical volume creation, deletion, etc. is called the Storage Pool Manager (SPM) in RHEV/oVirt: http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Virtualization/3.0/html/Technical_Reference_Guide/sect-Technical_Reference_Guide-Storage_Architecture-Role_The_Storage_Pool_Manager.html
03:14 pdurbin      in theory, i wouldn't need to worry about how all this works
03:15 pdurbin      i would just use some command line tool or language bindings to create VMs with disk images of a certain size
03:16 pdurbin      all in all, it sounds pretty slick
03:17 pdurbin      of course, the Storage Pool Manager (SPM) is going away in future versions of RHEV/oVirt
03:18 pdurbin      i think this is because it's a single point of failure?
03:19 pdurbin      anyway, i'm sure it'll be replaced with something that has the same role: to manage the LVM logical volumes for you... create, resize, and delete them for you
03:20 pdurbin      i'm starting to feel like we should give oVirt a try. it's supposed to be nicely packaged in fedora 17
03:21 pdurbin      this is not at all to say that we should stop looking at OpenStack
03:23 pdurbin      to repeat what i said around http://irclog.perlgeek.de/crimsonfu/2012-06-27#i_5763589 , i see RHEV/oVirt as a replacement for traditional VMware.  and OpenStack is a replacement for Amazon AWS
03:24 pdurbin      different use cases
03:24 pdurbin      i mean, i'm sure VMware is getting more "cloudy"
03:24 pdurbin      and maybe Amazon AWS is getting more "enterprisey"
03:25 pdurbin      but people typically use them for different things
03:25 pdurbin      and right now we're basically trying to build a platform that's like traditional VMware
03:26 pdurbin      i think we've made a good choice with libvirt/KVM
03:27 pdurbin      but if we want certain features in the future (automatic failover, a web gui, etc), we might want to look at RHEV/oVirt
03:29 pdurbin      now, i think we also want to offer AWS-style VMs. . . let people spin up their own.  have base images and all that
03:30 pdurbin      and OpenStack seems like a good choice for this. definitely worth trying
03:31 pdurbin      bascially, we can run both :) libvirt/KVM with or without RHEV/oVirt for enterprisey stuff. and OpenStack (or CloudStack??) for the AWS-style VMs
04:06 pdurbin      ok, i thought i was done, but one more thought. . .
04:07 pdurbin      what's wrong with what we have now? the new libvirt/kvm platform we built?
04:07 pdurbin      well, no automatic failover for one.  if a hypervisor goes down, the VMs that were on it go down too
04:08 pdurbin      and that would be unfortunate, but in practice it shouldn't be too hard to start them up on another hypervisor
04:08 pdurbin      what other problems?  well, we're using NFS for shared storage
04:08 pdurbin      "Important: This example uses NFS to share guest images with other KVM hosts. Although not practical for large installations, it is presented to demonstrate migration techniques only. Do not use this example for migrating or running more than a few guests." -- http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/shared-storage-nfs-migration.html
04:09 pdurbin      that quote continues with "iSCSI storage is a better choice for large deployments. Refer to Section 11.1.5, “iSCSI-based storage pools” for configuration details."
04:11 pdurbin      but to manage iSCSI you need some tools. again, agoddard lays this out very well at [CentOS-virt] Basic shared storage + KVM - http://lists.centos.org/pipermail/centos-virt/2012-June/002946.html
04:11 pdurbin      one such tool for managing iSCSI for libvirt is RHEV/oVirt
04:12 pdurbin      so that's what's driving my thinking here. . . the fact that i'm concerned that NFS won't scale for us with regard to VM disk image hosting
04:12 pdurbin      i guess time will tell
04:12 pdurbin      but i want to have some ideas in case we start seeing performance problems
04:13 pdurbin      i do love that NFS "just works"
04:13 pdurbin      it's so simple
04:31 pdurbin      ok, one more thought
04:32 pdurbin      i feel like i'm making iSCSI (plus some way to manage it, i.e. RHEV/oVirt) out to be a perfect solution
04:32 pdurbin      it should give you better performance than NFS
04:33 pdurbin      but what about fault tolerance?  what if a LUN suddenly goes away?  what happens to the VMs?
04:34 pdurbin      they get paused until the LUN is back
04:34 pdurbin      so the VMs are down, basically
04:35 pdurbin      because the LUN in part of a LVM volume group. and that LVM volume group doesn't like having physical volumes missing
04:35 pdurbin      is part
04:36 pdurbin      so you still have to worry about making those LUNs highly available somehow
04:37 pdurbin      now, RHEL 6.3 is supposed to have RAID built in to LVM, and I asked Dan Yasny if oVirt plans to use this
04:37 pdurbin      and he said they'd like to look into it, but haven't yet
04:37 pdurbin      that would probably solve a lot of problems
04:38 pdurbin      since LVM underlies so much of the strategy with LUNs. if LVM is set up for RAID it would be very nice
04:42 pdurbin      of course, there's still gluster to consider for VM image storage, but when I brought this up to Jeff Darcy, he called this use case "random I/O", which gluster is not ideal for
04:42 pdurbin      anyway, again, libvirt lets you define multiple storage pools, so we can play with some of these options...
04:43 pdurbin      but clearly, there are a lot of questions about vm image storage. i wasn't the only one asking questions at the talks today :)
13:20 agoddard     here's some other history on the matter - https://www.redhat.com/archives/libvirt-users/2011-February/msg00013.html
13:20 agoddard     (unanswered, I think I ask questions the wrong way or something :-/)
13:21 agoddard     I asked around a bunch of places ages ago
13:21 agoddard     Ganeti? check - http://groups.google.com/group/ganeti/msg/f2e7dc6c542c70bd
13:21 agoddard     Archipel? check - http://groups.google.com/group/archipelproject/browse_thread/thread/1c99f827fafe0875
13:22 agoddard     OpenStack last year? check - http://lists.openstack.org/pipermail/openstack-operators/2011-November/000411.html
13:22 agoddard     OpenNebula? yep - http://lists.opennebula.org/pipermail/users-opennebula.org/2010-December/003547.html
13:22 agoddard     ^ december 2010!! I need a new hobby
13:23 pdurbin_m    very interesting. thanks!
13:24 agoddard     re the above discussion (HA etc) you can do multipathing on the hosts for iSCSI HA, and then the LUNs can be built from RAIDs
13:24 agoddard     but if you do lose a LUN, your filesystems will go read-only etc..
13:25 agoddard     We had this happen recently, on an iSCSI backed LVM Xen cluster, we weren't multipathing and we lost the storage switch, when we replaced the switch, everything bounced, fsck'd and was happy
13:25 pdurbin_m    right so how do you fix that? with lvm when it supports raid within itself?
13:26 pdurbin_m    (to not worry about losing a LUN)
13:27 agoddard     multipathing would fix the switch issue, redundancy in the controllers etc + RAID on the SAN should solve the hardware issue.. but I suppose you could do some crazy madness where you RAID two LUNs from separate SANs? :-/
13:27 agoddard     I guess you could RAID two LUNs and then put LVM on top of it (though I imagine that's got issues of its own)
13:28 agoddard     "A plurality is not to be posited without necessity" :D
13:31 pdurbin_m    so, with rhel 6.3, just released, you can do raid within lvm. no need for traditional Linux software raid (mdadm)
13:31 agoddard     nice, so you can make the LVM across two devices and it manages the RAID within LVM?
13:32 pdurbin_m    yes
13:32 agoddard     nice
13:33 pdurbin_m    "The expanded RAID support in LVM is now fully supported in Red Hat Enterprise Linux 6.3. LVM is now capable of creating RAID 4/5/6 logical volumes and supports mirroring of these logical volumes. The MD (software RAID) modules provide the backend support for these new features."
13:33 pdurbin_m    https://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.3_Release_Notes/storage.html
13:34 pdurbin_m    hmm, so they're still using md...
13:34 agoddard     madness
13:35 pdurbin_m    I mean, wouldn't that be the way to do it? if it works?
13:37 pdurbin_m    maybe I'm not thinking straight
13:42 pdurbin_m    distracted by video by intel about how they built their fab in Arizona
13:43 pdurbin_m    brian stevens from red hat up now
13:45 pdurbin_m    (cto of red hat)
13:49 pdurbin_m    talking about Facebook's open compute initiative:  http://opencompute.org
13:50 pdurbin_m    rhel-certified open compute hardware
13:53 tychoish     heh, I'm friends with the guy who writes the LVM manual for redhat
13:56 sjoeboo      morning all
13:56 pdurbin_m    hi!
13:57 pdurbin_m    cool how gluster can replicate "hot data" (popular data) to more nodes, to spread out the load
13:58 pdurbin_m    i.e. a popular song on pandora
13:59 pdurbin_m    tychoish: cool
14:01 sjoeboo      yeah, very very
14:02 pdurbin_m    westmaas: he's talking openstack now
14:08 pdurbin_m    crimsonfubot: google open vswitch
14:08 crimsonfubot pdurbin_m: Open vSwitch: <http://openvswitch.org/>; Documentation | Open vSwitch: <http://openvswitch.org/support/>; Download | Open vSwitch: <http://openvswitch.org/download/>; FAQ | Open vSwitch: <http://openvswitch.org/faq/>; Features/Open vSwitch - FedoraProject: <http://fedoraproject.org/wiki/Features/Open_vSwitch>; Nicira Open vSwitch inside vSphere/ESX « ipSpace.net by @ioshints: (1 more message)
14:11 pdurbin_m    I'm still confused about what cloudforms is...
14:15 SEJeff       openvswitch is driving much of the "openflow" movement
14:17 pdurbin_m    CloudForms is part of the Red Hat IaaS stack comprimised of Red Hat Enterprise Linux, Enterprise Virtualization, JBoss Enterprise Middleware and Red Hat Storage stack.  CloudForms is based on the company’s open source DeltaCloud APIs but will incorporate full support for Red Hat’s OpenStack distribution.
14:17 pdurbin_m    -- http://m.zdnet.com/blog/open-source/red-hat-carefully-repositions-cloudforms-as-open-hybrid-cloud-management-platform/11165
14:17 sjoeboo      yeah
14:18 sjoeboo      in the "building an open hybrid cloud" talk yesterday (if i never here the word cloud again...)
14:18 sjoeboo      they showed how openshift was build
14:18 sjoeboo      about 30 redhat products all stacked on each others
14:18 sjoeboo      could forms was somewhere between rhev and openstack
14:18 sjoeboo      or maybe on top of openstack, but under openshift and another thing..
14:19 sjoeboo      phi, you rebase the pupept repo, or do a clean pull yet?
14:20 agoddard     pdurbin_m: sorry stepped out, but by "madness" I meant "sweeeeet"
14:20 sjoeboo      its still big as that big blob for ssh is in there (sigh), but, its MUCH smaller now
14:20 sjoeboo      oh, i'm building a fedora 17 x86_64 mirror for us now
14:21 agoddard     we have a cloudstack instance now with a /23 of public IPs, my plan is to keep that rockin for a while, but rebuild our infra so it's easy to move, either to a nicer network setup with cloudstack, or openstack
14:22 agoddard     and 'cause we have 2x DCs, about to potentially become 5x, then we might end up be best using what's already in prod at other DCs (cough cough openstack @ RC.. ) :D
14:25 sjoeboo      time to head to my rpm talk, see if i can't get a seat near power...
14:30 SEJeff       agoddard, *only* 5 dcs/
14:30 SEJeff       ?
14:30 SEJeff       We have 42 :/
14:31 agoddard     ha, nice
14:31 agoddard     but you don't have the center of the internet..
14:31 agoddard
14:31 agoddard     so I got y'all beat there ;)
14:32 SEJeff       touche! You win sir
14:32 * SEJeff     bows to agoddard's awesomeness
14:32 agoddard     lulz
14:32 agoddard     it would be awesome if we actually hosted something there
14:32 agoddard     it's probably a printer or something #tooManyIPsToKnowWhatToDoWith
14:32 SEJeff       "You have reached the middle of the internet. Welcome to the eye of the storm"
14:32 SEJeff       ha
14:32 agoddard     we should run a competition
14:33 SEJeff       agoddard, And you could at least give it a funny rdns entry
14:33 * agoddard   calls the network guys :)
14:33 SEJeff       win
14:35 SEJeff       Surprisingly good overview of what and how to setup anycast DNS: http://vincent.bernat.im/en/blog/2011-dns-anycast.html
14:35 SEJeff       For geo-locational based global load balancing. Thats how the googles and facebooks do it
14:37 agoddard     awwwwyeah :D
14:37 SEJeff       That is the guy who wrote lldpd too. If you don't run lldp on your 'nix servers... stop what you're doing and do it!
14:38 SEJeff       It is so nice to wonder what port a server is plugged into on a switch, just run: sudo lldpctl $interface and get the lldp info from the switch or in reverse, to login to a switch and see all of the hosts with their proper fqdn hostnames in the output of: show lldp nei
14:39 agoddard     holy shiz that's awesome
14:39 SEJeff       agoddard, Let me get you and example, hold one
14:39 * agoddard   makes a note to add it to a cookbook
14:40 pdurbin_m    agoddard: are you using deltacloud? to keep portability between cloud providers?
14:40 SEJeff       agoddard, From sudo lldpctl eth0: http://hastebin.com/lehirecuce.coffee
14:40 SEJeff       agoddard, https://github.com/vincentbernat/lldpd
14:41 agoddard     pdurbin_m: nope, planning on it all being awesome re-provisioning and fast data recovery
14:41 agoddard     so moving and DR are the same thing
14:41 agoddard     maybe a pipedream, but it'll be fun trying
14:42 pdurbin_m    basically, you're using chef for portability?
14:43 agoddard     ya
14:44 agoddard     "Your Prime Constraint should be the time it takes to restore your application data"
14:50 agoddard     (ftw)
14:51 agoddard     SEJeff: that's so awesome! I wish I knew about it on monday
14:51 agoddard     when I was googling how to get VLAN info off an interface
14:51 SEJeff       agoddard, :)
14:51 SEJeff       crimsonfu++
14:51 SEJeff       We help eachother
14:52 SEJeff       agoddard, Note that you'll need to make sure that lldp is enabled on the switches as well, but it is universally supported everywhere
14:52 agoddard     SEJeff: cool, I'm pretty sure it would be here, our network admin doesn't mess around.. he's off the chain
14:53 SEJeff       Yeah our guys are pretty nuts too
14:53 agoddard     42 DCs? I can imagine they would be :)
14:53 SEJeff       Yup and only 3 'nix admins
14:53 SEJeff       for a lot of servers :D
14:54 agoddard     we have 2x, going to be 1x later in the year :(
14:54 SEJeff       Easier to deal with though
14:54 agoddard     impossible w/out config management (obviously..) :)
14:55 SEJeff       For our env? Yes absolutely
14:55 agoddard     +1
14:55 SEJeff       We use puppet at work as we have ~6k LOC in our manifests
14:55 SEJeff       but I'm a comaintainer of the salt stack project, which also does config management ironically
14:55 agoddard     nice
14:56 SEJeff       salt's config management is actually quite good. We have a lot of users, but I'd like to see more unit tests and for it to bake in.
14:57 SEJeff       Speaking of... The great salt sprint is on the 30th. Would anyone have any interest in joining via online or from LA/Seattle/Salt Lake City?
14:57 SEJeff       Or and one in Portland I think
14:57 SEJeff       s/Or/Oh/
15:08 pdurbin_m    red hat is working on openstack rpms for rhel
15:11 pdurbin_m    oooo live openstack demo
15:11 SEJeff       Not surprising, that is one area that Ubuntu is spanking them handily
15:12 SEJeff       Ha
15:13 pdurbin_m    glance add ... nova image-list
15:50 sjoeboo      rpm talk/hand on was good...a bit super-entry level, but, helpful
15:51 SEJeff       sjoeboo, What conf are you gents at?
16:23 SEJeff       Any Arista switch users? Funny easter egg: http://blog.firedaemon.com/2012/06/28/arista-networks-switch-easter-egg/
16:53 sjoeboo      we're @ the redhat summit
17:02 SEJeff       totally jealous :)
17:03 SEJeff       Is anyone here in #crimsonfu near Santa Monica CA, Portland OR, or Salt Lake City UT?
17:37 agoddard     Cape Cod :(
17:37 agoddard     wait.. Cape Code :)
17:37 agoddard     wait
17:37 agoddard     Cape Cod :)
17:38 agoddard     there we go.. that was weird... :-/
17:38 agoddard     aaaanyhoo.. wondering how big y'all usually make your VM base images?
17:39 SEJeff       agoddard, You're in Cape Cod?
17:39 agoddard     ya
17:39 SEJeff       I'm friends with the couple that own this: http://www.sweetwaterforest.com
17:39 agoddard     crimsonfu folk always welcome to visit when the weather is nice :)
17:39 SEJeff       Big big fan
17:40 SEJeff       Cape Cod is beautiful
17:40 agoddard     that place looks cool. Brewster is where Thoughtbot run http://capeco.de
17:40 SEJeff       Ah nice
17:41 SEJeff       Beautiful area
17:41 agoddard     ya, we're pretty lucky over here
17:41 agoddard     where u @?
17:41 * SEJeff     is currently in Santa Monica CA
17:41 agoddard     nice!
17:41 SEJeff       About to move to Chicago... originally raised on a horse farm in Ky
17:41 agoddard     cool!
17:41 agoddard     I'm originally from.. Australia :D
17:59 SEJeff       Well g'day mate :)
19:15 agoddard     :D
19:16 SEJeff       I work with our Sydney, AU office quite a bit
19:17 SEJeff       And have a very good friend who married a kiwi
19:21 SEJeff       http://www.h-online.com/open/news/item/Perl-web-framework-Mojolicious-reaches-3-0-1628419.html
20:05 pdurbin_m    I just installed openstack on rhel :) in a lab
20:06 pdurbin_m    now another openstack talk by https://mobile.twitter.com/markmc_
21:44 pdurbin_m    had never heard of https://fedoraproject.org/wiki/SoftwareCollections
21:45 pdurbin_m    "The concept of Software Collections allows multiple versions of software to be installed at the same time without interfering in any negative way with the standard versions provided by the system."
21:45 SEJeff       Does it use alternatives?
21:45 * SEJeff     assumes it does
23:01 jimi_c       howdy :)
23:42 pdurbin_m    jimi_c: hi! you're the cobbler guy?
23:44 pdurbin_m    if so, you must meet sjoeboo. he'll be at the summit tomorrow. he keeps our cobbler patched and happy
23:45 pdurbin_m    and you already know SEjeff...