Time  Nick         Message
01:05 pdurbin      i'm only a couple clicks into the weechat website and i'm already offended by a screenshot of a woman in bondage: http://www.weechat.org/screenshots/weechat_2010-02-22_caleb.png/ terrible first impression
01:07 pdurbin      i'm reminded of how i spent a little time on http://geekfeminism.wikia.com/wiki/FLOSS the other week. open source has a long way to go, sadly
01:12 pdurbin      i just linked back to here and http://irclog.perlgeek.de/crimsonfu/2012-09-25#i_6017406 in #weechat and let them know i got a bad first impression
02:49 pdurbin      you guys know who Dave Winer is, I hope. I finally tried out his "symphony of software" and wrote about it here: Philip Durbin - Google+ - As a fan of owning my own data (and data liberation) I've… - https://plus.google.com/107770072576338242009/posts/bFaQpQNfb6J
02:49 pdurbin      ah. exited wine. might be time for a shower ;)
12:09 boegel       hi y'all!
12:10 pdurbin      boegel: good morning :)
12:11 boegel       this is a room full of hard-core sysadmins? scary!
12:11 pdurbin      heh
12:11 boegel       Itkovian is my colleague @ HPC-UGent
12:11 * Itkovian   ducks
12:11 pdurbin      Itkovian: hi
12:11 Itkovian     yeah. hi too :-p
12:11 boegel       if EasyBuild is broken, blame him :P
12:12 pdurbin      so you guys want to talk about my tweet? https://twitter.com/philipdurbin/status/249898230073135104
12:12 boegel       pdurbin: sure!
12:12 pdurbin      " use "module load" on your HPC cluster? try building with easybuild. @fasrc plans to http://hpcugent.github.com/easybuild/ "
12:12 boegel       pdurbin: tell us how you picked up on EasyBuild, and how you like it so far
12:12 Itkovian     I'll just lurk for a while, got work to do :-)
12:12 pdurbin      me too :/
12:12 boegel       don't we all...
12:13 pdurbin      didn't one of you leave a comment on james cuff's blog?
12:13 boegel       pdurbin: ah, yes, jgtimmer did
12:13 boegel       pdurbin: he's not in today though
12:13 boegel       pdurbin: we're trying to promote EasyBuild where it seems relevant
12:13 pdurbin      anyone have a link?
12:13 boegel       pdurbin: we've only started to make it available in public since a couple of months, and are trying to get feedback mostly
12:14 boegel       pdurbin: the blog was down yesterday, I think
12:14 pdurbin      here it is. from Jens: http://blog dot jamesdotcuff dot net: scientific software as a service sprawl... - http://blog.jcuff.net/2012/07/scientific-software-as-service-sprawl.html?showComment=1348407332993#c6395090447973051400
12:15 pdurbin      i work for james. when he saw that comment, he forwarded it to us
12:15 pdurbin      i haven't really had a chance to look at easybuild
12:15 pdurbin      oh, did you see the recent blog post about modules from dell?
12:15 boegel       pdurbin: no
12:16 pdurbin      it's by Jeff Layton: Auditing Environment Modules - Home Base for HPC Professionals - http://hpc.admin-magazine.com/Articles/Auditing-Environment-Modules/
12:16 boegel       pdurbin: your tweet seemed to suggest you looked into it and really liked it, which we were happy about
12:16 boegel       pdurbin: we're working hard on EasyBuild v0.9, which is going to include quite a bit of changes compared to v0.8
12:16 pdurbin      boegel: sorry to give you that impression. really i only looked at the home page
12:17 boegel       pdurbin: so, I wanted to contact you to make sure you don't start coding like hell against v0.8, cause v0.9 will break that work
12:17 pdurbin      heh. ok
12:17 boegel       pdurbin: our v0.9 milestone is set for end of Sept, but it's going to have to shift a bit probably
12:17 boegel       pdurbin: does james cuff's blog work on your end? it doesn't here...
12:18 pdurbin      my involvement with our modules system so far has centered around exposing our list of modules as JSON and writing a little script to query it: https://github.com/fasrc/api/blob/master/modules
12:18 boegel       pdurbin: hpc admin magazine is a Dell thing?
12:18 boegel       pdurbin: because I did see that article, it's not the first time they've run that series on module
12:18 pdurbin      uh. i think Jeff Layton works for dell?
12:18 boegel       *modules
12:18 * boegel     doesn't know
12:19 boegel       pdurbin: LinkedIn says he does, yes
12:19 pdurbin      ok #notcrazy
12:19 boegel       pdurbin: so, anyway, are you guys planning to look into EasyBuild?
12:20 pdurbin      personally... i have a lot on my plate at the moment. but i think we should
12:20 pdurbin      other people on my team do the software building
12:20 boegel       how are you guys handling software builds now? any framework or somesuch you're using?
12:20 pdurbin      i'll certainly point them to this conversation
12:21 * pdurbin    checks docs
12:21 boegel       :)
12:21 boegel       pdurbin: are you, or the software build guys in your team, planning to attend SC'12?
12:21 pdurbin      i see a docs/cluster/modules/template.mdwn...
12:22 boegel       that's a template module? or?
12:22 boegel       ah, now, it's MarkDown probably
12:22 pdurbin      yeah, we use ikiwiki. (i love ikiwiki)
12:23 pdurbin      ha! "modulefile_template.very_simple is what I always use; modulefile_template.with_prereqs has some logic for auto-unloading conflicting modules, but I don't know how it works"
12:23 pdurbin      "There's a little helper named generate_setup.sh in the hpc/rc module to make this easier and more consistent. Just run it with the -m switch and the directory you're trying to setup, and it'll search for the variables that need to be set."
12:24 pdurbin      i'm not sure if this is helpful at all :)
12:24 boegel       pdurbin: ok, that's for creating modules
12:24 pdurbin      isn't that way easybuild is for? creating modules?
12:24 boegel       pdurbin: but how about building the software packages themselves?
12:24 Itkovian     pdurbin: creating the module files is a side effect, mostly :-)
12:24 boegel       pdurbin: it builds and installs software in a custom path, with a specified compiler toolchain, and then also creates module files for that software, yes
12:25 pdurbin      ok, that's what i thought
12:25 Itkovian     no building => no module files
12:25 boegel       pdurbin: we have a workshop paper on EasyBuild that we've submitted to the PyHPC workshop at SC'12, we can show you a preprint if that's helpful
12:26 pdurbin      if you can make it public you could post a link on james's blog
12:26 pdurbin      (if it's up) :P
12:26 boegel       pdurbin: we'll, we can't make it public yet, we're awaiting the acceptance decision for the workshop
12:27 boegel       pdurbin: but we can mail it for you (and your colleagues) to read
12:27 boegel       pdurbin: either way, we should get the acceptance decision Oct 1st
12:27 pdurbin      sure. if you mail it to rchelp@fas.harvard.edu it will land in our ticketing system (RT)
12:29 boegel       is that a good idea? I don't want to give the impression we're spamming the Harvard helpdesk :)
12:29 pdurbin      heh. no it's cool. i'll take the ticket
12:29 pdurbin      hey, have you guys looked at this? SoftwareCollections - FedoraProject - https://fedoraproject.org/wiki/SoftwareCollections
12:29 pdurbin      i keep meaning to add that as a comment on james's blog. the same post you commented on
12:30 pdurbin      "The concept of Software Collections allows multiple versions of software to be installed at the same time without interfering in any negative way with the standard versions provided by the system."
12:30 boegel       pdurbin: we briefly looked into it, well, the Red Hat counterpart at least
12:30 pdurbin      ok. i chatted with some red hat guys about it at their summit this summer
12:30 boegel       pdurbin: that's what EasyBuild does, but our focus in on HPC software
12:30 Itkovian     Mind that EasyBuild development has started three years ago, when there were no SoftwareCollections :-)
12:30 pdurbin      sure. and that's our focus as well
12:31 boegel       pdurbin: any of you guys planning to attend SC'12?
12:31 Itkovian     but yeah, they seem to have the same idea
12:31 pdurbin      well, how old is modules itself? 20 years? :)
12:31 pdurbin      i think james always goes to SC...
12:31 Itkovian     however, it would be hard for us to rely on RPMs, since we have custom/commercial software that meeds to be installed too
12:31 pdurbin      Itkovian: sure. us too. commercial software
12:32 Itkovian     so yeah .. you know how that works out ...
12:32 boegel       pdurbin: I'd love to chat with you guys at SC'12... Jens and I will be there the whole week
12:33 pdurbin      http://blog dot jamesdotcuff dot net: disruptive things spotted so far at #SC11 - http://blog.jcuff.net/2011/11/disruptive-things-spotted-so-far-at.html
12:34 pdurbin      boegel: i'll ask james to look for you :)
12:34 boegel       pdurbin: I can schedule an (informal) meeting with him, if he's OK with that
12:34 boegel       pdurbin: is he into beers? :)
12:35 pdurbin      he's english. of course he is
12:36 boegel       pdurbin: :D
12:36 boegel       pdurbin: mailed our paper to the rchelp@ address
12:36 boegel       pdurbin: maybe show it to James, and ask him if he's OK with meeting up at SC'12?
12:37 pdurbin      sounds like a plan
12:42 boegel       pdurbin: so, what's crimsonfu about? grouping together sysadmins who release their tools as open-source?
12:42 pdurbin      it's an experiment of mine. not a harvard thing
12:43 pdurbin      http://crimsonfu.github.com is my attempt to explain it :)
12:43 pdurbin      we talk about puppet, chef, kvm, etc. you name it. you're welcome to hang out here
12:45 boegel       pdurbin: we will :)
12:45 boegel       pdurbin: ever heard of Quattor?
12:45 pdurbin      crimsonfubot: google quattor
12:45 crimsonfubot pdurbin: quattor - fabric management for grids and clouds: <http://quattor.sourceforge.net/>; Quattor: <http://www.quattor.com.br/>; Quattor - Wikipedia, the free encyclopedia: <http://en.wikipedia.org/wiki/Quattor>; Quattor (company) - Wikipedia, the free encyclopedia: <http://en.wikipedia.org/wiki/Quattor_(company)>; Quattor, Quattor Petroquímica S.A., Company Profiles,: (1 more message)
12:45 pdurbin      nope. thanks
12:45 boegel       pdurbin: it's what we use here instead of Puppet to deploy all our systems
12:46 pdurbin      interesting
12:46 boegel       pdurbin: right now, about 500 spread out over 5 clusters
12:46 pdurbin      sjoeboo just arrived with his suitcase. he's off to puppet conf :)
12:46 boegel       pdurbin: :)
12:54 pdurbin      Science Collaboration Framework | MIND Informatics - http://www.mindinformatics.org/node/3
12:54 pdurbin      About | Science Collaboration Framework - http://sciencecollaboration.org
12:55 pdurbin      "The Science Collaboration Framework (SCF) is a software toolkit to establish web-based virtual team organizations for researchers in biomedicine"
12:55 pdurbin      "eXframe - a subproject of SCF - is a reusable framework for building genomics experiments repositories" http://sciencecollaboration.org/exframe
12:56 boegel       nice
13:19 boegel       pdurbin: how big is the HPC support team at Harvard, including sysadmins?
13:19 Itkovian     also, how big are the cluster(s)?
13:20 Itkovian     large variety of software?
13:21 pdurbin      we don't run the only cluster at harvard but http://rc.fas.harvard.edu/about-rc/research-computing-staff/
13:22 boegel       about 15 people, ncie
13:22 boegel       pdurbin: the systems are not university-wide?
13:22 pdurbin      it's complicated :)
13:22 boegel       pdurbin: it's only for the arts and sciences faculty?
13:22 boegel       Itkovian: http://software.rc.fas.harvard.edu/ganglia/ganglia2_master/
13:23 boegel       Itkovian: 20000 cores in total, about 1750 systems
13:23 Itkovian     nice
13:25 pdurbin      boegel: more on the way: http://en.wikipedia.org/wiki/Massachusetts_Green_High_Performance_Computing_Center
13:26 boegel       pdurbin: that sounds promising :)
13:33 pdurbin      :)
14:23 whorka       Harvard has Google apps now: http://g.harvard.edu/
14:23 Itkovian     Is everybody here from harvard?
14:24 boegel       Itkovian: well, we're not :)
14:31 Pax          Not everyone but a bunch :)
14:37 pdurbin      Itkovian: invite your friends :)
14:37 Itkovian     lol
14:37 Itkovian     except for boegel, I think only jgtimmer lurks on IRC
14:53 boegel       I'm starting to screw things up, so almost time to go home...
14:53 pdurbin      heh. see ya
14:53 Itkovian     I will not reply to that
14:55 * pdurbin    reads http://techtalk.daudfam.net/mysql/mysqlinnodb-unable-to-lock-issue
15:02 pdurbin      oh yeah, i forgot there's a http://dba.stackexchange.com
15:19 boegel       ttyl guys
15:19 boegel       and gals
15:19 pdurbin      do we even have any gals? we should recruit some
15:55 pdurbin      "got it started with innodb_force_recovery=5"
15:57 Pax          So I'm probably the last person to have caught this.. but triggers in cobbler are cool!
15:57 Pax          https://github.com/cobbler/cobbler/wiki/Triggers
16:06 pdurbin      jimi_c: ^^
16:08 pdurbin      Pax: we have /var/lib/cobbler/triggers/install/pre/clean_puppet.sh
16:30 pdurbin      whorka: nothing under "additional services" for me :( #googleapps
16:40 whorka       I had to set security questions and do a password change per http://g.harvard.edu/g-start.html
16:41 pdurbin      yeah, i did that
16:41 pdurbin      maybe i'm being punished for jumping the gun... for trying yesterday
16:41 whorka       nah, we had people signing up yesterday too
19:05 pdurbin      "heavy inserts"
19:08 pdurbin      boegel: for the record, i got your paper via ticket #29067
19:19 ventz        pdurbin: links reminder :)
19:23 pdurbin      ventz: oh, the qcow2 corruption?
19:23 pdurbin      ventz: here you go http://irclog.perlgeek.de/crimsonfu/2012-08-03#i_5871320
19:24 pdurbin      related tweet: https://twitter.com/philipdurbin/status/233280438884515840
19:25 pdurbin      which i sent to the author of this paper, whom i met at the red hat summit in july or whenever that was: The QCOW2 Image Format - http://people.gnome.org/~markmc/qcow-image-format.html
19:25 pdurbin      nice guy. i'm sure he's busy
19:31 ventz        pdurbin: did he ever respond to your tweet?
19:31 pdurbin      nope
19:32 pdurbin      in practice, we're not treating that corruption super seriously... the VMs seem fine. but i do want to clean it up. i have a half written nagios check
19:38 pdurbin      ventz: i'm glad you've never seen this. it gives me hope
19:48 ventz        what scares me, i have'nt seen it on the old version either
19:48 ventz        so now the question becomes did it screw up a file somewhere and i just never noticed
19:48 ventz        but you are saying it screws up the whole VM image
19:55 pdurbin      `qemu-img check` says so. but again the VM seems to work fine...
19:55 pdurbin      though snapshots are a mess. i'm sure they aren't reliable. if even accessible
19:59 ventz        hmm
20:02 pdurbin      ventz: but you're a fan of NFS for VM disk images?
20:02 pdurbin      we sometimes blame NFS for the qcow2 corruption. but we really don't know
20:02 pdurbin      and haven't looked into it deeply
20:03 pdurbin      i wonder what oVirt does. if it does the same qemu snapshotting under the hood
20:03 pdurbin      JoeJulian likes gluster for VM disk image storage. i think :)
20:11 ventz        pdurbin: i am, i like NFS in general and hate iscsi
20:11 ventz        i looked a lot into iscsi b/c of the performance mentions
20:11 ventz        and once i read up on it I gave up (well after i did my own tests and verified the performance mentioned in some white papers)
20:14 pdurbin      agoddard: you love iscsi. fight! fight!
20:15 Pax          why do you hate iscsi?
20:15 Pax          we've got lots of it, and haven't seen any performance issue.. what were you guys seeing?
20:19 pdurbin      Pax: i think this was in ventz's basement :)
20:19 pdurbin      ventz: no offense :)
20:22 pdurbin      i like NFS too. easy
20:22 pdurbin      but i worry about too many VMs on NFS
20:25 Pax          :)
20:26 pdurbin      don't make me link to red hat's doc again. the one that says don't use NFS for VM disk images in production
20:26 Pax          I feel like often we "over buy" thing like storage… so we spend more on FC or high end disk when lower end, cheaper solutions perform as good, or similarly
20:27 semiosis     i hope if you're doing vms over nfs you're at least using tcp,noac,sync
20:27 semiosis     but it still seems dangerous
20:27 agoddard     we've had great success with iSCSI backed LVMs backing our old xen cluster, but there's a lot of crap involved in making sure you don't corrupt Volume Groups in homegrown setups..
20:28 agoddard     I like the idea of orchestrations tools using SAN APIs to carve out LUNs for VMs, but then it's $$ gear.
20:28 Pax          agoddard: good, cheap or fast.. pick two :)
20:28 agoddard     more and more I'm a fan of (insert as raw as possible, fast, shared storage) for persistent volumes, and then local LVM devices for ephemeral storage
20:29 semiosis     pdurbin: have you heard of the work going on to make qemu talk directly to gluster (without going throuh a FUSE mount?)
20:29 agoddard     Pax: +1
20:29 agoddard     there's also ATAoE which seems pretty nice, but we haven't used it yet
20:30 agoddard     Ceph Rados block devices sound cool too :D
20:30 pdurbin      semiosis: somewhat. yes. the gluster thing
20:31 pdurbin      comptona: you can keep telling us about ceph :)
20:31 pdurbin      ata over ethernet, i guess? sounds like iscsi...
20:32 pdurbin      crimsonfubot: lucky ataoe
20:32 crimsonfubot pdurbin: http://en.wikipedia.org/wiki/ATA_over_Ethernet
20:37 comptona     pdurbin: is there anything in particular you'd like to know about Ceph?
20:40 pdurbin_m    is it awesome?
20:40 JoeJulian    Last I checked, and that was a long time ago in the area of clustered storage (and my ability to remember things) ceph had a central metadata server. Is that true today?
20:40 pdurbin_m    are you using it in production yet?
20:41 comptona     JoeJulian: yes, it has a central metadata server
20:41 comptona     and yes, I'm using it in production, but at a very small scale (four nodes)
20:41 JoeJulian    So how does it handle redundancy wrt metadata then?
20:42 comptona     oh, I misunderstood your question
20:43 comptona     so, the metadata service is provided by one daemon at a time
20:43 comptona     but you can run as many of them as you want, and they talk amongst each other
20:43 comptona     so if the current active one fails, the remainder hold an election and one of them becomes the new active service
20:44 comptona     I'm not 100% sure of the mechanism by which the metadata servers coordinate
20:44 JoeJulian    So it's active/passive[/passive...] Doesn't that create a bottleneck?
20:45 comptona     possibly? I imagine the metadata is very small relative to actual data
20:45 comptona     we don't actually use the metadata service ourselves
20:46 JoeJulian    Oh? I thought that was a necessity in order to address your data.
20:46 comptona     nope
20:47 comptona     so, ceph has several different access methodologies
20:47 comptona     you can use the librados library, which talks to the object store directly
20:47 comptona     you'd manage metadata yourself in that case
20:47 comptona     or you can use the kernel driver to create a block device
20:48 comptona     which you'd have to format with a standard filesystem, and then handle metadata that way
20:49 comptona     I think you only need the metadata service if you're using the S3-style object store or mounting a volume directly ("mount -t ceph x.x.x.x:/ /mnt/ceph")
20:49 comptona     since we're just providing volumes to openstack instances, we don't have any file-level access directly to ceph
20:49 JoeJulian    In the case of a block device, it's "striped and replicated across the entire storage cluster". How does it know which server to find any particular offset?
20:50 comptona     basically a super-fancy hashing algorithm
20:50 comptona     http://ceph.com/wiki/Custom_data_placement_with_CRUSH
20:52 comptona     I won't pretend I understand all of the details behind the placement group stuff
20:54 JoeJulian    Judging by the wiki, it's definately grown up a lot over the last 2 1/2 years. :D
20:55 comptona     I just learned about it for this project, but it definitely seems impressive
20:56 comptona     the performance is generally really good, too
20:58 comptona     the one thing I'm running into is that there's a pretty bad dropoff of small-block write speed from the raw performance level
20:58 comptona     everything else is about as fast as I could expect it to be