Time Nick Message 12:59 Pax morning 13:01 pdurbin morning! playing with salt states!! http://irclog.perlgeek.de/salt/2012-04-12#i_5432258 13:08 Pax hey cool! 13:08 pdurbin it's super easy 13:09 pdurbin and everything i need so far is in epel 13:10 pdurbin gonna see how far i can get translating this puppet manifest into salt states: http://git.greptilian.com/?p=grinfra.git;a=blob;f=puppet/modules/server1/manifests/init.pp;hb=HEAD 13:17 shuff oh hey pdurbin, check out http://taskwarrior.org <http://taskwarrior.org/> 13:17 shuff Todo.txt made significantly more awesome :) 13:18 pdurbin shuff: huh. ok 13:18 pdurbin Pax: here is are the salt states i've stubbed out so far: http://server1.greptilian.com/srv/salt/ 13:24 pdurbin here's how i installed a package (screen): http://server1.greptilian.com/scratch/salt-install-package.txt 13:24 pdurbin stupid vps.v2s.org. i'm not sure why it says that rather than server1.greptilian.com... 13:57 westmaas pdurbin: have you tried setting up openstack outside of devstack? 14:04 pdurbin westmaas: i'm working on it. being a centos shop, i'm a bit blocked on ubuntu 14:05 pdurbin last week i added the right ubuntu to cobbler. fiddled and fiddled and got a vm to install. now on to an installation on a physical server 14:05 westmaas meaning you are trying to do this on ubuntu even though you are a centos shop? 14:13 pdurbin yes. i thought i had to... "This is a tutorial style beginner's guide for OpenStack™ on Ubuntu 11.10" -- http://docs.openstack.org/diablo/openstack-compute/starter/content/index.html 14:13 pdurbin that's the guide i plan to follow 14:15 westmaas gotcha, yeah, to follow that guide you maybe do. 14:15 westmaas rpms are in EPEL for stable releases though 14:16 pdurbin is there a guide for centos? 14:16 westmaas I'd guess once you are through the installation/configuration pieces it doesn't matter 14:16 westmaas not yet 14:16 pdurbin please let me know when the guide for centos has been written :) 14:17 westmaas haha 14:17 westmaas k 14:40 SEJeff Hey pdurbin, thanks for introducing me to crimsonfu 14:41 SEJeff I'm a sysadmin, and I code. 14:41 pdurbin SEJeff: welcome! like #salt, we log this channel too 14:41 SEJeff Thats cool 14:41 SEJeff But I prefer the term: Systems Engineer for a sysadmin who codes 14:41 SEJeff DevOps is a buzzword 14:41 pdurbin yeah, i'm a little down on devops as a term 14:42 pdurbin i can't bring myself to call myself an engineer 14:42 * pdurbin tears up RHCE cert 14:43 SEJeff After spending almost 2.5 years building a vendor agnostic network config management system (glorified python + pexpect library + django + jQuery)... I think it is fair to call one's self an engineer 14:43 SEJeff Especially if you write more code than do day to day ops work 14:43 SEJeff But it is all semantics. 14:44 pdurbin fine, but architects build buildings 14:44 SEJeff I wouldn't dare call myself an architect :) 14:44 SEJeff More like a builder 14:44 pdurbin a maker 14:44 SEJeff Ever read Dune? 14:44 pdurbin i've heard of it 14:44 SEJeff sadness 14:45 SEJeff A maker is a giant sand worm that eats things walking along in the desert 14:46 pdurbin i just stick with sysadmin, usually 14:47 shuff SEJeff: welcome! the spice must flow 14:47 matt1337357 I've always more strongly identified with the House Harkonnen 14:48 SEJeff matt1337357, Hopefully not the Baron 14:49 SEJeff shuff, control the spice, control the universe 14:50 SEJeff pdurbin, So #salt is always looking for smart sysadmins, developers, or just plain users to help us shape the future of the project. Any suggestions so far? 14:52 pdurbin hmm, i struggled a tiny bit with the docs. i guess i should actually read them instead of just typing whatever is in a <code> tag ;) 14:53 pdurbin i think you salt guys are doing great 14:53 pdurbin we've using it often across ~1700 hosts 14:54 pdurbin there is a bug you could fix for me... 14:55 westmaas people in my office keep asking if the devs are gonna get things to the devops guys or the system engineers are going to get stuff over to the devops guys 14:55 pdurbin this one: Issue #891: Salt-master stops responding - [FIXED with libzmq 2.1.11] · saltstack/salt: https://github.com/saltstack/salt/issues/891 14:55 westmaas :( 14:55 pdurbin SEJeff: because we keep having to restart the salt master :( 14:57 SEJeff pdurbin, Thats shite 14:57 SEJeff How often does it happen? 14:57 pdurbin ya know... 14:58 pdurbin i always just restart the master every time i go to use it 14:58 SEJeff It is a bug in zeromq, but still, complete shite 14:58 SEJeff :( 14:58 pdurbin we've been putting off documenting how to use salt for the rest of our team until this is fixed 14:59 pdurbin we includes a guy who's not in the room at the moment, but usually is. here, i shared his g+ post about salt: https://plus.google.com/107770072576338242009/posts/YbCriPJCmqy 15:00 pdurbin 15 minutes vs. 6 seconds. salt ftw 15:00 SEJeff indeed 15:01 SEJeff Our production is 1800 servers 15:01 SEJeff Well a few more, but it varies. Around that 15:02 pdurbin is there an upstream zeromq bug? 15:02 SEJeff pdurbin, And you've upped the sysctls, right? 15:03 SEJeff Supposedly, upstream fixed it in zeromq 3.1 for good. 15:03 SEJeff keyword 15:06 pdurbin keyword? 15:06 SEJeff Keyword being supposedly 15:06 pdurbin ah. but no bug number 15:07 SEJeff pdurbin, I don't know the upstream bug. Tom spoke with the 0mq upstream about it. There was also a mailinglist post about it. Did you try upping the sysctls? 15:07 SEJeff http://readthedocs.org/docs/salt/en/v0.9.8/topics/troubleshooting/#salt-master-stops-responding 15:08 pdurbin hmm, didn't know about that doc. thanks. trying to summon the guy who would know if we upped the sysctls 15:09 pdurbin i'm just the idea guy. the one who listens to floss weekly :) 15:09 SEJeff Oh ha! 15:09 SEJeff Randall (Schwartz) works pretty close to here actually. I've went out a few times for drinks with him. Nice guy 15:10 pdurbin i tried to say hi to him at the sunset grill during a boston perlmongers meeting but he was very busy 15:11 SEJeff He lives in PDX and works near LAX 15:11 pdurbin he does seem very nice 15:11 SEJeff Super commuter 15:13 pdurbin argh. stupid bios stopped responding. westmaas: i'm getting ubuntu on that physical host 15:15 pdurbin shuff and whorka: i miss having real console access. this dell java console thing is terrible 15:16 shuff :( 15:16 shuff if it makes you feel any better 15:16 SEJeff pdurbin, What kind of servers? Dell R6xx R7xx and R8xx have the Drac and serial works like a champ 15:16 shuff imagine when all the systems are in Holyoke 15:17 pdurbin Dell PowerEdge C5125 15:18 pdurbin restarted firefox and i'm back in business 15:18 SEJeff Haven't played with the dell blade-ish stuff like that 15:19 pdurbin SEJeff: but you have real console access? i'm used to a wire going to a serial port 15:19 SEJeff pdurbin, Yeah via an ethernet cable plugged into an opengear serial concentrator 15:19 SEJeff Every server in our firm has a serial connection this way setup to 115200 baud 15:19 pdurbin right, for us it was avocent. for whorka it still is 15:20 pdurbin yeah, same same 15:20 SEJeff pdurbin, We used avocent's cyclades eons ago. They are 2x as expensive and not as easy to deal with compared to the opengear console servers 15:21 pdurbin good to know 15:30 SEJeff Also unlike the cyclades, opengear's software is actively developed and they add features. Just a better product. 15:33 pdurbin cool 15:34 pdurbin SEJeff: i don't see anything about sysctl in our salt::master puppet class. definitely worth a shot 15:34 SEJeff For sure 15:41 pdurbin how strange that "force pxe first" is enabled 15:43 pdurbin oh good, sjoeboo is here 15:43 sjoeboo hang on, client update, be right back 15:43 whorka SEJeff: thanks for the tip. I would love to find a better replacement for Avocent. 15:44 sjoeboo okay 15:46 sjoeboo whats up now? 15:46 pdurbin http://readthedocs.org/docs/salt/en/v0.9.8/topics/troubleshooting/#salt-master-stops-responding 15:47 SEJeff sjoeboo, Can you enable those sysctls and see if it fixes the "master stops responding" issues with salt? 15:49 sjoeboo [root@provisions ~]# rpm -qa |grep zeromq 15:49 sjoeboo zeromq-2.1.9-1.el5 15:49 sjoeboo so we're certainly behind 15:51 sjoeboo there done, time to wait and see 15:54 pdurbin great. thanks 15:57 pdurbin sjoeboo: SEJeff was asking earlier if we have any suggestions for salt: http://irclog.perlgeek.de/crimsonfu/2012-04-12#i_5432697 16:05 sjoeboo not yet, been too busy with all the other happenings to do much side from executer some things here and there 16:12 pdurbin yeah, like i said, we've been very impressed with salt's remote execution speed. i'm only just looking at salt's config mgmt features (salt states) 16:14 pdurbin whorka: how can you ever think of leaving cyclades/avocent/emersonnetworkpower/etc/etc? 16:18 SEJeff sentry makes pretty good managed power towers as well, but those aren't as important as a good console server 16:19 whorka agreed. OOBM has saved us when the network was unreliable. 16:20 whorka does opengear have SSH passthrough to serial? 16:23 SEJeff They have all of the features cyclades do at about 1/2 the price 16:23 SEJeff They also have this real crack rdp over serial thing for windows boxes 16:23 SEJeff if you are that unlucky 16:24 SEJeff I looked at it once, but we don't have windows servers. I think it creates a network interface over the serial link and then tunnels the rdp traffic. Something crack like that, but they are the only vendor doing it. 16:25 whorka that's pretty cool. we have a couple managed servers running Windows. their admins might like a more direct access path. 16:27 * pdurbin daydreams about not having windows servers 16:29 SEJeff Yup: http://www.opengear.com/News-PR060201.html CTRL f search in that page for RDP-over-serial 17:01 SEJeff sjoeboo, You know... Looking at https://plus.google.com/103415330307393824597/posts/VDBcjHJ4Hgd , you might be able to speed that up more. You could try bind mounting /var/cache/salt/jobs to /dev/shm or mounting it as tmpfs 17:01 SEJeff You should be able to do it faster 17:01 sjoeboo meh, thats far faster than a push run 17:02 sjoeboo plus the box i've been running this one servers a few other purposes, and is ram heavy, so this is fast enough for now 17:02 pdurbin SEJeff: you have to understand where we're comin from... :) 17:02 SEJeff heh 17:02 SEJeff pdsh is basically ssh, right? 17:02 pdurbin SEJeff: but in theory, how fast are you thinking? 3 seconds? 17:03 SEJeff Depends on the hardware and network ultimately 17:03 SEJeff But why not 3 instead of 6? 17:04 pdurbin because we're coming from 15 MINUTES :) 17:04 SEJeff Ha