Time  Nick         Message
06:05 ventz        pdurbin: found issue
06:05 ventz        https://www.redhat.com/archives/libvirt-users/2012-August/msg00086.html
06:05 ventz        read about Ubuntu 12.04
06:05 ventz        you have the same isue
06:05 ventz        issue*
06:13 ventz        summary btw: you can take live snapshots, but you cannot restore live
12:00 pdurbin      ventz: on a related note... crimsonfubot is back!
12:00 pdurbin      crimsonfubot: lucky crimsonfubot
12:00 crimsonfubot pdurbin: http://wiki.greptilian.com/irc/bots/supybot/
12:01 pdurbin      sigh. but my wiki is still down :(
12:01 pdurbin      good old crimsonfubot
12:46 pdurbin      larsks: thanks for posting your resize doc. here's how my attempt with fdisk went wrong: VM stuck at grub prompt after resize/fdisk mishap - http://danceb.in/5nJYwg0d4hGUBYyl9DNYg
12:46 pdurbin      larsks: thanks also for the heads up about the salt security issue
14:16 pdurbin      getting happier and happier with my VM disk image resize doc
14:17 pdurbin      i even removed "here be dragons" from the top
14:18 pdurbin      now to update my "clone a vm" doc for el6 hypervisors
14:19 pdurbin      i tried to convince the user to have a freshly kickstarted VM instead. (latest ubuntu LTS even.) oh well
15:00 pdurbin      "What is the lightest weight, simplest, will mostly work shared storage solution that isn't NFS ?" -- http://irclog.perlgeek.de/gluster/2012-10-23#i_6090303
15:09 larsks       Argh, a link to a link.
15:10 larsks       He meant: https://twitter.com/kbsingh/status/260088365351841792
15:14 ventz        pdurbin: what is VMFS? :-D
15:15 ventz        (to your shared solution that isn't NFS) :-D
15:17 ventz        per Phil's request -- sharing this here
15:18 ventz        i found out that using KVM pre 1.2.0 has issues with qcow2 -- actually, Phil first discovered this
15:18 ventz        i doubted him since I hadn't seen it
15:18 ventz        after some research, it turns out that v 1.2.0 fixes all this
15:19 ventz        if you go here: http://wiki.qemu.org/ChangeLog/1.2 and look up 'qcow2' it mentions the fixes
15:19 ventz        Sadly, Ubuntu LTS 12.04 still ships out w/ 1.0
15:19 ventz        Ubuntu 12.10 ships out with 1.2
15:19 ventz        RHEL -- well it might be years before it sees 1.2 ;)
15:19 ventz        (kidding! - sort of)
15:20 pdurbin      no kidding :)
15:21 pdurbin      ventz: the news on our end is that sjoeboo figured out how to fix our qcow2 corruption faster than what i was doing
15:21 pdurbin      i was converting the corrupted qcow2 file to raw. and then back again
15:21 pdurbin      he simply tried doing a snapshot *through virsh* and that seemed to remove the corruption as well
15:22 pdurbin      (as reported by `qemu-img check`)
15:22 pdurbin      so our take away is to never never use the lower level qemu-img command to take snapshots. instead, always always use virsh or something that goes through the libvirt stack
15:23 ventz        pdurbin: interesting
15:23 ventz        i found out you can convert from qcow2 to qcow2 and it also fixes it
15:23 larsks       pdurbin: Did you ever find out exactly what was causing the corruption?
15:23 ventz        on v1.2.0 you can just use '-r' flag
15:23 ventz        larsks: it's a bug in KVM -- specifically qemu
15:23 pdurbin      oh, right. actually i think that's what i was doing too, now that you mention it
15:23 ventz        pre vs. 1.2.0
15:24 larsks       I'm just wondering why snapshotting via virsh -- which presumbly just talks to KVM on the backend -- solved the problem.
15:25 pdurbin      larsks: i would say the corruption is caused by trying to make snapshots of qcow2 files of running VMs using qemu-img rather than using virsh (thus going through libvirt). note that making snapshots through virsh pauses the VM! so we haven't rolled this out yet... i think we'll give users some time to opt out...
15:28 pdurbin      for a smallish VM (10 GB of local disk) it gets paused for a couple minutes during the snapshot
15:30 pdurbin      being on centos 6 we're using ancient versions: libvirt-0.9.10 and qemu-kvm-0.12.1.2
15:31 pdurbin      for posterity: a link to a link of previous discussion here: http://irclog.perlgeek.de/crimsonfu/2012-09-26#i_6020372
15:31 pdurbin      larsks: what's wrong with a link to a link? :)
15:31 larsks       So much clicking!
15:32 pdurbin      yes, but jdarcy replied :)
15:32 pdurbin      so it's worth it
15:32 pdurbin      conversation going on right now in #gluster :)
15:32 * pdurbin    fans flames
15:43 * semiosis   did some stoking of that fire last week
15:45 semiosis     but i wasnt so tactful :/
16:02 pdurbin      mmm, nagios bot. works with stock nagios i hear... https://github.com/cluenet/cluemon/blob/master/nagios-bin/nagiosbot.py
16:09 pdurbin      to send from nagios to irc: https://github.com/vvuksan/misc-stuff/blob/master/nagios/send_nagios_email.php
16:12 pdurbin      Vladimir Vuksan's blog » Blog Archive » My monitoring setup - http://blog.vuksan.com/2012/09/01/my-monitoring-setup/
16:13 pdurbin      vvuksan: welcome!
16:14 vvuksan      hello :-)
16:15 GitHub140    [crimsonfu.github.com] pdurbin pushed 1 new commit to master: https://github.com/crimsonfu/crimsonfu.github.com/commit/b91a911b2ca60237604fb40e47202d98ef1eb980
16:15 GitHub140    [crimsonfu.github.com/master] add the latest irc logs - Philip Durbin
17:53 pdurbin      /var/log/messages hates me: failed to bind to LDAP server ldaps://ldap/: Invalid credentials: Success
17:58 whorka       today in ambivalent return codes: http://www.hmdc.harvard.edu/ is throwing a 403 yet serving data anyway
18:04 larsks       pdurbin: ...that word you keep using...
18:21 pdurbin      heh
18:21 pdurbin      playing with http://logstash.net
18:21 pdurbin      sjoeboo just stood it up. seems very nice
18:21 pdurbin      severity_label:"Warning"
18:21 pdurbin      need to figure out how to get json out of this thing
18:24 semiosis     pdurbin: i'm all about the logstash... what do you need?
18:24 semiosis     and the #logstash channel is really good too
18:24 pdurbin      semiosis: can i query logstash and get json out of it?
18:24 pdurbin      i dunno... the last 20 lines of severity_label:"Warning" or something
18:24 ironcamel    pdurbin: http://git.greptilian.com/?p=data-liberation-shapado.git gives me a 404
18:24 semiosis     not exactly, normally you would have logstash put stuff into elasticsearch, which you can query through it's json/rest api
18:25 semiosis     pdurbin: also there's a web frontend for the data logstash stores in elasticsearch called kibana which will be the official web frontend for logstash going forward
18:25 pdurbin      ironcamel: ventz has some explain' to do ;)
18:27 pdurbin      semiosis: i seem something this right now: http://logstash.net/images/screenshot.png
18:29 semiosis     pdurbin: that is logstash-web, the original web frontend, which will be discontinued soon
18:29 semiosis     kibana is the future
18:29 semiosis     and present
18:30 semiosis     logstash-web is the past
18:30 semiosis     http://kibana.org/
18:31 pdurbin      ironcamel: here you go: https://github.com/pdurbin/data-liberation-shapado
18:31 pdurbin      semiosis: huh. discontinued. ok
18:32 semiosis     it's still around but it's not going to get much attention
18:32 semiosis     in any case, if you want to query & get json responses, hit the elasticsearch api
18:32 semiosis     if you're running the embedded es, that should be on port 9200
18:32 semiosis     http://www.elasticsearch.org/
18:33 pdurbin      semiosis: how do i hit the elasticsearch api? have a url for me i can tweak?
18:34 semiosis     1 sec
18:34 semiosis     you're running the embedded elasticsearch?
18:34 semiosis     that would be http://<hostname-of-your-logstash-indexer>:9200
18:34 pdurbin      semiosis: i have no idea :)
18:34 semiosis     pdurbin: pastie/gist your logstash config
18:35 semiosis     oh and we really should do this in #logstash, where others will be able to fill in the details idk about
18:35 * pdurbin    runs `cd puppet && git pull`
18:37 pdurbin      semiosis: this one: https://github.com/simonmcc/puppet-logstash
18:37 pdurbin      semiosis: is #logstash logged?
18:38 semiosis     pdurbin: eh, no i dont think so
18:38 semiosis     i did set up a supybot (logstashbot) tho, but it's not really set up to do much yet
18:39 semiosis     asking about logging now
18:40 pdurbin      semiosis: they can hang out here if they want :)
18:41 semiosis     i'm going to set up the ilbot thing
18:41 pdurbin      semiosis: host ilbot yourself or use the one hosted by moritz?
18:42 semiosis     http://irclog.gr/#browse/irc.freenode.net/logstash
18:42 semiosis     there's already one set up
18:42 semiosis     i would have used the aaS one
18:43 pdurbin      semiosis: i'm worried #gluster going to kill moritz's ilbot. so chatty you guys ;)
18:43 semiosis     haha
18:44 * pdurbin    needs to be careful what he wishes for
18:51 semiosis     pdurbin: in any case, looks like that puppet module runs embedded es by default, so on the "indexer" machine, you can connect via HTTP to port 9200 and make elasticsearch API requests
18:52 pdurbin      semiosis: ah, i'm hitting a different port for web. thanks
18:52 semiosis     yeah 9292 for web, 9200 for the api
18:53 pdurbin      {"tagline":"You Know, for Search","ok":true,"version":{"number":"0.19.8","snapshot_build":false},"status":200,"name":"Decay"}
18:53 pdurbin      from this: curl -s http://logs:9200
18:53 pdurbin      semiosis: thanks much!
18:54 semiosis     try again
18:55 semiosis     i got this: http://pastie.org/5104909
18:55 semiosis     you have a much newer version
19:00 pdurbin      nice
19:02 pdurbin      http://logs:9200/_status
19:03 pdurbin      http://logs:9200/_status?pretty=true
19:03 pdurbin      oooo, pretty
19:03 pdurbin      oh! we have kibana already
19:03 pdurbin      http://logs:5601
19:08 semiosis     w00t
19:08 pdurbin      http://logs:9200/logstash-2012.10.23/_search?q=@type:syslog&pretty=true
19:08 pdurbin      logstash is very very nice
19:09 semiosis     say that in #logstash :)
19:10 pdurbin      semiosis: done!
19:10 semiosis     nice!
19:13 pdurbin      love that pretty=true
19:13 pdurbin      saves me piping to python -m json.tool
19:13 semiosis     elasticsearch is really cool
19:14 pdurbin      ironcamel: though i should just use `p` for my slicing and dicing :)
19:15 pdurbin      crimsonfubot: lucky App::p
19:15 crimsonfubot pdurbin: http://search.cpan.org/perldoc?App%3A%3Ap
19:24 pdurbin      http://logs:9200/logstash-2012.10.23/_mapping?pretty=true
19:50 agoddard     log<3stash
19:50 agoddard     how many ES nodes are y'all running?
19:57 pdurbin      agoddard: whatever was stood up an hour ago
20:07 pdurbin      centos6bug.c:(.text+0x10e): warning: the `gets' function is dangerous and should not be used.
20:08 agoddard     pdurbin: :D
20:08 agoddard     you guys heading to http://monitorama.com next year?
20:08 agoddard     just down the road!
20:10 pdurbin      agoddard: huh. news to me. maybe!
20:10 agoddard     schweet. register!!
20:10 pdurbin      C coders may be interested in this. i can't reproduce this bug: this test on a CentOS 6 x64 machine often end up with incredibly many errors. But it doesn’t happen on CentOS 5 Dancebin! - http://danceb.in/Ylpw4Esd4hGr0I6yl9DNYg
20:13 pdurbin      agoddard: $100. cheap
20:13 pdurbin      John Allspaw. sold. http://monitorama.com/#speakers
20:13 pdurbin      Sean Porter
20:14 pdurbin      Jordan Sissel
20:14 pdurbin      all of our favorites
20:14 agoddard     ya
20:14 agoddard     it's an awesome lineup
20:14 agoddard     imma try to convince Sean to come to RC one day around the event
20:15 pdurbin      agoddard: cool. i've tweeted with him a bit
20:18 pdurbin      http://logs:9200/_cluster/state?pretty
20:18 pdurbin      agoddard: send us some more fun elasticsearch url's please
20:29 larsks       re: monitorama, "bring a light jacket".  Hah, those guys.
20:34 pdurbin      :)
20:36 semiosis     monitorama, you say?
20:36 pdurbin      semiosis: come on up! :)
20:37 semiosis     the price is reasonable
20:39 semiosis     i could think of a couple things to do at the hackathon
20:41 larsks       These $100 conferences seem like a good idea.  DevOpsDays is next week in NYC for the same price.
20:42 pdurbin      agoddard: check your email. subject "logs? I've got your logs right here"
21:01 ventz        pdurbin: so, i still need to clear your qcow from errors