Time  Nick      Message
02:49 JoeJulian pdurbin: KVM images on Gluster: I don't back them up at all. I have generic images that are configured by puppet. If an image gets corrupted, I just delete and let puppet recreate.
08:12 jgtimmer  I don't know if this has been anounced here already, but the 14th quattor workshop is coming up
08:12 jgtimmer  http://indico.cern.ch/conferenceDisplay.py?confId=196712
13:22 pdurbin   bah! cp: writing `/sysroot/Temp/V2V/rhev-apt.exe': No space left on device at /usr/share/perl5/vendor_perl/Sys/VirtConvert/GuestfsHandle.pm line 203 at /usr/share/perl5/vendor_perl/Sys/VirtConvert/Converter/Windows.pm line 482
13:23 pdurbin   die windows die!
13:35 pdurbin   the C: drive has only 624 KB free
13:49 pdurbin   i don't know what i can safely delete. asking the VM owner to free up space. hopefully, that'll fix it
14:03 pdurbin   more space now. trying again. i'm sure you guys are fascinated by this
14:05 pdurbin   JoeJulian: no backups of your KVM disk images, huh? i've got to say... using rsnap to back up our disk images takes a lot of space. it's not like you can do incrementals, really. the disks (naturally) change every day, so a full copy is made every night
14:07 pdurbin   jgtimmer: i'd love to come to belgium :) #beer
14:08 pdurbin   westmaas: i don't know where you are but my boss is in texas: https://twitter.com/jamesdotcuff/status/258330636757118976
14:10 pdurbin   oh, for anyone googling in the future, i'm using virt-v2v to migrate this windows vm
14:10 pdurbin   the very last vm on our centos 5 kvm infrastructure
14:10 pdurbin   i think i mentioned already we're totally off of vmware now
14:10 pdurbin   so soon we'll be all kvm on centos 6
14:11 pdurbin   next up... need an api... ovirt/rhev/cloudstack/openstack ... something to make management of all this stuff easier. it's fun to write scripts and all but i want users to be able to provision their own vms
14:12 pdurbin   and i want a better backup strategy
14:14 pdurbin   oh, by the way we have 1708 hosts on puppet 3.0 now, thanks to sjoeboo
14:53 pdurbin   so! freeing up space on the C: drive worked! virt-v2v was able to migrate the VM. now all of VMs are running on our new infrastructure :)
15:34 pdurbin   "Six megawatts gets you an awful lot of equipment" -- http://photo.jcuff.net/post/33774497442/tacc
16:06 dotplus   pdurbin: "need an api"... what are you trying to achieve? what are your goals? I'm a couple of months away from looking at how to deal with automating provisioning for network/storage/VMs and I want something simmering in the back of my mind.
16:12 pdurbin   dotplus: well, when people want a vm... i build it for them (usually)
16:12 pdurbin   the process isn't bad. we wrote a bunch of scripts to automate it
16:12 pdurbin   adding the system to dns is the slowest part
16:12 pdurbin   otherwise, a script to add it to cobbler
16:13 pdurbin   then a script to build the vm with `koan` (kickstart over a network)
16:13 pdurbin   it would be nice to have some self service... for users to be able to spin up their own VMs
16:13 pdurbin   so there's that
16:13 pdurbin   plus there's the issue of vm storage. right now we store all of our disk images on nfs, hosted by our emc
16:14 pdurbin   backing up all those disk images is a challenge
16:14 pdurbin   i'm kind of wondering if we should explore using lvm instead... for vm storage... a logical volume per vm
16:15 pdurbin   but i would want something to orchestrate that... oVirt/RHEV or cloudstack or openstack or what have you
16:15 pdurbin   i also want better fault tolerance
16:15 pdurbin   right now if we lose a hypervisor, someone needs to go figure out which VMs were running on it (we store this as a fact in puppet)
16:16 pdurbin   and then `virsh define` each one on some other hypervisor. and then `virsh start`
16:16 pdurbin   something to automate all of that would be great
16:17 pdurbin   to be clear, we're using what i'd call "bare bones" kvm. we manage it ourselves with scripts and love
16:17 pdurbin   i think maybe we need ovirt
16:17 pdurbin   westmaas: or openstack if you think it can do all this for us
16:18 pdurbin   i actually just wrote up a summary of where we are... based on the goals i (voluntarily) wrote up when i started this project a year ago
16:18 pdurbin   i'm proud of how far we've come
16:18 pdurbin   the old system was barely in puppet
16:19 pdurbin   the old kvm system
16:19 pdurbin   i don't even want to talk about the old vmware system. i know so little about it
16:19 pdurbin   may it rest in peace
16:19 pdurbin   ... but there's still work to do... lots to improve
16:19 pdurbin   i'm just glad the new plaform is handling our...
16:20 * pdurbin checks
16:20 pdurbin   183 VMs
16:20 pdurbin   this is on 8 hypervisors. kinda inexpensive dell poweredge C6145 boxes
16:21 pdurbin   Processorcount: 48 (per box)
16:21 pdurbin   Memorytotal: 177.31 GB (per box)
16:22 pdurbin   oh, we also don't have a great process for building Windows VMs
16:22 pdurbin   those scripts i mentioned are for linux VMs
16:22 pdurbin   jimi_c: and koan/cobbler fu for Windows VMs?
16:23 * pdurbin reads through the email he just sent
16:24 pdurbin   oh, better monitoring of the whole system... knowing when hypervisors are reaching capacity
16:24 pdurbin   i'm still not sure the best way to measure capacity... how many VMs a hypervisor can host
16:24 pdurbin   docs. need more docs. of course
16:25 pdurbin   i think those are the main points
16:25 pdurbin   dotplus: you might be sorry you asked :)
16:45 jimi_c    windows support in cobbler is limited to images, but you should be able to do that today with koan/cobbler
16:53 dotplus   pdurbin: nonono, that's exactly the sort of thing I'm interested in. in fact, I'd be happy to hear more of your experiences. can you publish anything of either your results or your process? even your docs?
16:54 dotplus   I haven't used cobbler in years. In my brief contact with it, I seem to remember thinking that it wanted to control dns/dhcp more than I was willing to allow, but a) perhaps I was wrong b) perhaps it's changed some c) perhaps I'm more willing now.
16:55 jimi_c    dotplus: those are optional, you don't have to let it manage those
16:55 jimi_c    out of the box, manage_dns and manage_dhcp are disable
16:55 jimi_c    but it does manage tftp/xinetd out of the box
17:00 dotplus   I think I want to be able to specify a few things and have "it" create machines/VMs for me.1) grant it a/some network(s) 2) give it storage (or a storage provisioning process) 3) an installation server with kickstart files/yum repos
17:00 dotplus   is "it" cobbler? cobbler + one of the cloud mgmt suites? just one of the cloud suites?
17:00 jimi_c    just cobbler, koan helps you create vm's on some hypervisors, but it's not IaaS aware (yet)
17:01 jimi_c    as an interim solution, what most people do is just deploy base images that pxe boot
17:03 dotplus   sounds like I really need to revisit cobbler.
17:09 pdurbin   jimi_c: do you use cobbler to manage you dns? we edit dns by hand and back up our changes to a git repo
17:12 pdurbin   dotplus: have you already told us about how you do virtualization?
17:18 dotplus   pdurbin: not sure. at my current employment, it's KVM, virt-install/virsh, kickstart with %post, puppetize everything I care about. storage is iSCSI LUNs from a NAS box handed to guests with virt-install ... --disk vol=foo. It's for a small linux infrastructure (a few dozen VMs, most of which are for the infrastrcuture itself) mostly supporting a heavy-duty RT environment.
17:26 dotplus   not sure whether more automation (cobbler and/or IaaS) or more robustness ( replicated storage, multipath, multimaster db, haproxy in front of webservers/appservers, and/or failover to offsite "public" cloud or similar) should be top of the list of priorities now.
17:51 pdurbin   dotplus: so when you want to build a new VM, is there any automation around the creation of the iSCSI LUN? in our setup with disk images (qcow2 files) we just make a tweak in cobbler to say that the vm we're building should have 20 GB of local disk instead of 10
18:00 dotplus   no, not yet:(
18:18 jimi_c    pdurbin: no I don't use it for dns. I run my own servers, which are both remote from cobbler. I guess I could set them both up as slaves to the cobbler server, but to be honest in my home setup I just don't deploy that many systems that would be anything other than test boxes
18:18 jimi_c    at work, dns is MS AD, so we don't control that at all
18:53 westmaas  pdurbin: there are definitely tools to identify which VMs are on downhypervisors, and to support live migration
18:54 westmaas  extensions to the openstack api, and then extensinos to the client to support those server side extensions
18:55 westmaas  pdurbin: oh I thought your boss was at openstack summit
18:55 boegel    did I mention already that vacuum robots are cool? :)
18:55 westmaas  nevermind!
18:59 pdurbin   westmaas: he's speaking at CloudBeat: http://venturebeat.com/events/cloudbeat2012/speakers/
19:02 pdurbin   jimi_c: would you recommend using cobbler to manage dns? i kind of what *something* to know what IP addresses are free, etc.
19:06 pdurbin   oh, one last funny thing about virt-v2v with that windows 2003 vm... when i booted it up on the destination hypervisor, the clock had changed from 10:35am to 2:35pm. weird
19:07 pdurbin   "Work got busy so I slowed down on IRC, busier so I forgot to use twitter. Now work is much harder without IRC & Twitter. :-| #irony" -- https://twitter.com/anthonygoddard/status/258331816614834177
19:07 larsks    Previous host did not use UTC time in the BIOS, maybe?
19:08 agoddard  ^sadface :(
19:08 pdurbin   larsks: could be. no more virt-v2v! done migrating! well, until we switch to gluster for storage...
19:08 pdurbin   semiosis: gluster
19:08 pdurbin   JoeJulian: gluster
19:08 pdurbin   gluster!
19:11 agoddard  rewriting our old backup cookbook today.. "[2012/10/17 15:10:22][message] Backup for 'Backup of DNS zone files & config. (dns_config)' Completed Successfully in 00:00:00" w00t
19:11 agoddard  although.. we don't actually.. uhh.. need to backup DNS zone files.. they're autogenerated anyway :D
19:11 agoddard  this is me in therapy, trying to use IRC again during the day :_
19:11 agoddard  :)
19:12 pdurbin   KVM on gluster whiteboard - https://twitter.com/philipdurbin/status/258646361438822400
19:12 agoddard  *not actually in therapy. *not sure if that turns out to be unintentionally bad humor. *** shutting up now
19:13 agoddard  pdurbin: what are the md3000i's doing?
19:13 JoeJulian I just got back to my home office after getting a store back on DSL. Had them running through my phone since yesterday morning.
19:14 pdurbin   JoeJulian: fun
19:14 JoeJulian It kinda was, actually.
19:14 * pdurbin checks racktables to see how many md3000i's we have
19:15 pdurbin   i used to punch down phone wires
19:15 agoddard  http://twitter.com/anthonygoddard/status/258647251902140417
19:16 JoeJulian Setting my my laptop to be an ipsec router through my Evo 4G running CM7 (damn them for not building newer versions for this perfectly adequate phone).
19:19 pdurbin   agoddard: we have 10 md3000i's and 6 PowerEdge R610's
19:20 semiosis  pdurbin: gluster?
19:20 agoddard  pdurbin: nice, what the md3000's do?
19:20 pdurbin   we're thinking 2 gluster clusters. clusters? whatever you call a separate thing
19:20 agoddard  ah
19:20 agoddard  isn't the i for iSCSI?
19:20 pdurbin   agoddard: yes but we don't plan to use iSCSI
19:21 pdurbin   this is existing hardware. the stuff i just got the last VM off this morning
19:21 agoddard  pdurbin: ah!
19:21 pdurbin   the first gluster cluster is for primary VM storage. holds disk images of the running VMs
19:21 agoddard  pdurbin: we were thinking about maybe sending an md3220i up there btw..
19:23 pdurbin   it's powered by 2 R610's in data center A and 2 R610's in data center B. these are the servers running centos. the servers for this gluster cluster have md3000's as attached storage. two md3000i's in data center A and two in data center B
19:23 pdurbin   i hope that makes sense. half of the hardware is in each data center
19:24 pdurbin   total hardware used for that first gluster cluster: 4 servers and 4 disk arrays
19:24 jimi_c    pdurbin: of course I would :) lots of people do
19:24 jimi_c    but if you have a lot of systems that are outside of cobbler, then you might want to keep it limited to a sub-domain
19:25 jimi_c    i wouldn't try and use cobbler for general dns management
19:25 pdurbin   jimi_c: ok, just checking. thanks :)
19:25 agoddard  pdurbin: have you seen benchmarks yet for VM perf?
19:26 pdurbin   agoddard: you're always on about perf...
19:26 agoddard  lulz
19:26 pdurbin   i mean, i care about performance but for us VMs are non performant. we use hardware for performance
19:30 agoddard  pdurbin: hrm.. is that why one of our VMs just: CRITICAL - load average: 24.58, 19.02, 10.18
19:31 agoddard  0-O
19:31 agoddard  brb! :D
19:31 pdurbin   i wasn't sure i was even still connected to this channel
19:36 pdurbin   wee bit of a something-strange-going-on here...
20:21 pdurbin   https://twitter.com/jamesdotcuff/status/258656300311920640
20:26 semiosis  sounds bad
20:29 pdurbin   i've had better days
20:29 pdurbin   but! gluster gluster gluster. we'll get the VMs off this storage and on to gluster and it will be awesome
20:29 pdurbin   right?
20:29 semiosis  absolutely
20:33 pdurbin   good
20:33 semiosis  as long as your use-case is a good fit and your workload is well behaved ;)
20:34 pdurbin   i didn't even talk about the plan for the second gluster cluster. basically, take the rest of the hardware, but half in each data center. use it for backups. 1 R610 in each data center. 3 md3000i's in each data center
20:34 pdurbin   nothing 'round here is well behaved
20:39 pdurbin   anyway, all clear, i think