Time Nick Message 01:56 semiosis pdurbin: irc logging with logstash FTW! http://demo.logstash.net/#eyJzZWFyY2giOiJAdHlwZTppcmMgQU5EIEBmaWVsZHMubmljazpcInNlbWlvc2lzXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjkwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzNTQxNTQwNDYyNjYsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0= 02:44 pdurbin_m "no events matched"? 02:44 pdurbin_m there is no better IRC logging bot than ilbot 02:45 pdurbin_m though I am dinking with the code a bit... 02:46 semiosis pdurbin_m: its been over 15m, set the timeframe back to 4h 02:50 JoeJulian Wow... I know where my mind is... "I am drinking with the code a bit..." "its beer over 15m..." 02:51 semiosis the problem with making dev, test, staging, and production exactly the same, is that when something works fine on dev & test, but not on staging & production, it's FREAKIN IMPOSSIBLE to figure out why 02:51 semiosis ...is where my mind is 02:54 JoeJulian Yeah, I know that one. It works from the command line but not from the init script is my favorite of those. 02:54 JoeJulian Had that twice in the last two weeks. :/ 02:56 semiosis or it works from the command line but not from puppet. the friendly folks of #crimsonfu helped me out of that jam last week :) 02:58 semiosis ok, album is over, it's 10 PM, time to go home 03:26 pdurbin_m ok that's pretty cool 04:56 pdurbin i actually have some data i care about on my home VM now... the mysql database that backs http://irclog.greptilian.com ... trying to think of a good way to back it up. i'm thinking about mysql dumps to http://git.greptilian.com and then i can just pull it down with the rest of my git stuff with my little script: https://github.com/pdurbin/scripts/blob/master/grgit.pl 04:57 pdurbin basically, when i sit down at my work computer, my laptop, my wife's iMac, i run `grgit.pl` and i does a `git pull` on all my repos. works great for my needs 05:00 pdurbin my model for that VM, by the way is that is should hold public data only. i can't think of any secrets i have on there as of this writing 08:34 simmel pdurbin: http://demo.logstash.net/#eyJzZWFyY2giOiJAdHlwZTppcmMgQU5EIEBmaWVsZHMubmljazpcInNlbWlvc2lzXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImFsbCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzNTQxNzgwMjc0MjYsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiI 08:34 simmel n0= 09:48 jgtimmer pdurbin: we have a logstash vm image if you're intrested in testing it 12:15 pdurbin simmel: sorry, what are you trying to show me? 12:15 pdurbin jgtimmer: thanks but i think we're done with logstash for now. here's where we were playing with it: http://irclog.perlgeek.de/crimsonfu/2012-10-23#i_6091084 12:16 pdurbin i think we're going to try to get into splunk instead 12:16 pdurbin hmm, no Pax in here. he gave us a nice demo the other week 12:20 simmel pdurbin: The link that you tried but you said "no events matched" 12:21 simmel pdurbin: Mine worked 12:22 simmel pdurbin: If you're gonna use it for work, Splunk is gonna be really expensive 12:46 jimi_c pdurbin: we were looking at using logstash, any reason you're moving away with it? Are you using elastic search too? 12:47 jimi_c i recently contributed a whole bunch of patches to kibana to add stuff like authentication/authorization 12:47 jimi_c they're in the process of getting that merged into the mainline i believe 12:56 jgtimmer we are seeing memory leaks in elastic search here thought, any updates on that? 12:58 jimi_c i only have it running on a single system for a POC, we haven't deployed it anywhere major yet 13:23 jgtimmer we have it running for over 4months now, having around 500M messages in it now. 13:25 jgtimmer we gave an anonomized subset of those to a few students for a machine learning project, they are going to do anomaly detection on it, hopefully this gives us some intresting results 13:29 pdurbin jimi_c: i'm looking over internal emails from sjoeboo regarding logstash 13:30 pdurbin one thing is that splunk is already installed and our data has been collected for years (by Pax's department)... we just haven't been making use of it... we're starting to get spun up now 13:33 * pdurbin is distracted... running `service openfire restart` 13:35 rackerhacker and now for something completely different: http://i.imgur.com/LYhSj.gif 13:36 pdurbin jimi_c: so i guess i would say... with the number of hosts we have (1705 according to http://software.rc.fas.harvard.edu/ganglia/ ) logstash needs some thought with regard to architecture... needs a number of boxes to run on... you can't just throw logstash on a VM and hope it all works 13:36 pdurbin rackerhacker: no fun allowed 13:36 pdurbin (but that was funny) 13:37 jimi_c pdurbin: i was actually planning on using the syslog portion, and using logstash just on a set of centralized systems for indexing only - more splunk-like 13:38 pdurbin jimi_c: notice how agoddard's first reaction was "how many ES [elastic search] nodes are y'all running?" -- http://irclog.perlgeek.de/crimsonfu/2012-10-23#i_6091541 13:38 rackerhacker pdurbin: i miss leslie for sure :/ 13:38 pdurbin rackerhacker: +1 :( 13:40 pdurbin rackerhacker: saw your tweet that you don't have as much time to listen to podcasts. i'm curious if you listen to any technical podcasts like i do 13:41 rackerhacker i can't stand the vocal podcasts :/ i'm more of a fan of the music ones 13:41 rackerhacker i'd rather read the technical things 13:42 pdurbin sadly, i find i don't have as much time as i'd like for reading technical things. podcasts are great for when i'm biking to work, doing dishes, folding laundry, etc. 13:43 pdurbin well, if anyone here has some favorite technical podcasts, please let us know 13:44 jgtimmer the linux action show used to be cool, but not that much anymore since lunduke left... 13:44 jgtimmer but some of the other jupiter broadcasting podcasts are good aswell 13:45 pdurbin jgtimmer: my grumpy tweet at lunduke: https://twitter.com/philipdurbin/status/180041357442826240 13:51 pdurbin linux action show is the only thing i've listened to from jupiter broadcasting. i dunno what else they have that's worth trying 13:52 jgtimmer tech snap 13:53 pdurbin crimsonfubot: lucky jupiter broadcasting tech snap 13:53 crimsonfubot pdurbin: http://www.jupiterbroadcasting.com/ 13:54 pdurbin jgtimmer: what's it about? 14:01 jgtimmer http://en.wikipedia.org/wiki/Jupiter_Broadcasting#TechSNAP 14:02 jgtimmer TechSNAP is a weekly show about Systems Network and Administration[16] and is hosted by Chris Fisher and Allan Jude (owner of scaleengine.com[17]). 14:02 jgtimmer TechSNAP takes recent stories of security flaws and general gaffs by web companies and dissects them to find the flaws in the network/web design. It also gives information about running and building your own servers and networks. 14:04 pdurbin huh. that sounds pretty interesting, actually 16:44 semiosis rackerhacker: congrats on the new position 17:03 rackerhacker semiosis: thanks 17:09 larsks Does anyone know if there is some sort of admin-level view of volumes in openstack? 17:23 pdurbin sjoeboo: interesting, the whole gluster's-own-nfs-server being faster than the native gluster client: http://irclog.perlgeek.de/gluster/2012-11-29#i_6197311 17:24 pdurbin heh. sjoeboo, don't call it "ugly left over iscsi kit"! maybe agoddard will buy it off us! http://irclog.perlgeek.de/gluster/2012-11-29#i_6197343 17:25 pdurbin agoddard: it's awesome. you'll love it. don't forget how much you love iscsi 17:27 JoeJulian pdurbin: It's not unexpected. The kernel nfs client implements in-kernel caching, so some things are faster at the expense of being stale. 17:28 semiosis adding "noac,sync" eliminates most of that caching afaik 17:33 pdurbin interesting 17:33 pdurbin it just seems odd to my that gluster implements its own nfs server. i guess it's fine 17:33 pdurbin odd to me* 17:34 GitHub87 [crimsonfu.github.com] hakamadare pushed 1 new commit to master: https://github.com/crimsonfu/crimsonfu.github.com/commit/301ea7f93cfb94291eaf7c4cc6c412744ffb7d94 17:34 GitHub87 crimsonfu.github.com/master 301ea7f Steve Huff: updating Lightning Exchange Provider link... 17:34 JoeJulian It saves quite a few context switches by doing it that way. 17:36 pdurbin JoeJulian: ok 17:36 pdurbin shuff: look at you all writing blog posts 17:38 shuff i discovered that the Exchange provider for Lightning was finally updated to work with modern Thunderbirds, and has a bunch of bugfixes! 17:38 shuff i have staved off having to switch to Outlook once again 17:38 pdurbin shuff: dude. g.harvard ;) 17:40 pdurbin i'm switching over. to the gmail thing, despite the problems i've noted here previously 17:42 shuff ehh 17:43 shuff if i can use it to schedule rooms and manage the on-call rotation, then i'm interested 17:44 pdurbin shuff: you and i have different needs (thank goodness) ;) 17:45 shuff true that :) 19:36 semiosis JoeJulian: my php session issue was varnish stripping out cookies :D 19:37 semiosis JoeJulian: varnish config was not in sync between test & staging 19:38 pdurbin i've heard good things about varnish 19:38 semiosis love it 20:22 miah i havent had a chance to use it yet 20:22 miah i've done lots of reading about it though 20:24 miah i have a chef cookbook hacked up from opscode too. but i need to do more testing with it. 21:28 pdurbin forcing nagios checks to run because i like seeing green 21:45 pdurbin https://twitter.com/search?q=power%20boston 21:46 semiosis wow! 21:48 larsks Huh. Wonder what that's going to do to my bus ride home... 21:48 semiosis does that mean the T stops "dead in its tracks?" 21:48 semiosis i'd hate to be stuck in a blacked out tunnel 21:53 larsks Apparently a red line train hit a deer? Not sure if that was the cause of all this or just a coincidence. 21:54 miah more trains need cow catchers 22:26 pdurbin managed to bike home. the drivers in boston are bad enough *with* working traffic lights 22:28 miah i once had a cab driver in boston that didn't know left from right