Little Dog Wins (but big dog was nice)

Great Video!

Strange SVN foo and the maximum hostname length

We had some tests running in AWS yesterday and the cluster we spun up was created by scripts.  The script sets some tags on the instances and sets the hostnames to match the tags.  Because it was a special test, we had specified some extra metadata into the tag string because other scripts can pull that for a variety of things (like creating groups in StackDriver).  Alas, that’s not what the story is about, it’s just how we arrived at the mess.  The important thing is that the hostname string was about 70 characters long.

The big problem came when trying to check out some code.  We were doing a canary test where some new code was being tested in a production context on production servers.  While our control systems are now deploying from git repos, our main detection engine is still pulling from an SVN repo.  This creates a pain point that we have to take a branch from github and actually move it to SVN, but that’s not a bad thing overall.  We’ll change that soon.  But that’s not what the story is about, it’s just how we arrived at the mess.  We needed to do an ‘svn up’ on these canary servers to get the test branch of code onto them.

And it failed.

Here is what we’d get from the ‘svn co <repo path>’ command:

svn: generic failure

Well, that’s informative.

We googled, we read, we tried everything.  Nothing working.  Lots of folks found that it was due to the host not knowing it’s name – a bug in another library required that the hostname match the hostname in /etc/hosts.  We tried that.  No joy.

Teeth gnashing.  Log review.  Who changed what?  When did this break?  Why does it work when our servers first spin up?  (lots of angst until we proved to ourselves that they were spinning up and getting the right stuff from SVN).

Turns out that it’s a hostname string length limit.  There is a limit, according to RFC-1035.  It’s 64 bytes.  Since our hostname was set longer, and that’s what we were trying to use, but the underlying system would only use the first 64 bytes, it was tripping on the same bug after all.

Make the hostname smaller than 64 bytes and all was well.

So there go.  You learn something new every day.




Measuring Job Performance – a nod to Submarines

I have been telling my daughter sea stories from my time on the USS Richard B Russell (SSN-687) - the ones I can at least.  She loves hearing about submarines.  She asked me the other day “Daddy, were you good at your job on the submarine?”  I said yes, I was.  “How did you know?” she asked.

Easy.  “Well honey, I was the battle stations Reactor Operator.  If we had to fight, I was the guy that they wanted to run the reactor.”  She thought that was cool.

“Was everyone good at their job?” she asked.

“Yes, honey.  Everybody had to be or we could all die.  That’s what it meant when you earned your dolphins.  It meant you were good enough to be trusted to do your job.”  She nodded.  She understands what dolphins mean.

Then she asked me a really good one.  “Well, are you good at your job now?”

“Yes, I think I am.  I have all the right experience and the CEO and the board hired me to do this particular job.”

“But how do you know?” she pressed.

And that, my friends, is the core of why I still think so much about my time in subs and why I still think so much about my former colleagues and our exploits.  Yes, there may be a little middle age retrospective going on, but from a bigger picture it’s been bubbling in my soul for a while now.   Will I ever get a chance to be in an organization as elite and superior as I was when I was in my 20′s?  How do you measure elite?

Don’t get me wrong:  I have worked with some great teams and built some great technology.  I have made great friends who I would work with again anytime – and in fact have pulled a LOT of those folks from over 15 years of Silicon Valley experience to join my current team.  But they are not a submarine crew.  And we lack the framework for measuring performance that we had in subs.  So I ask: what constitutes an elite development team?

How would a modern development team structure a process to get judged by your peers – not out of some book or exam where a certification company makes profits – but from working engineers and leaders who can measure your skills and give you the acknowledgement of your capabilities.   And even within that, how do you measure excellence?  On a submarine, when the COB/Bull Nuke (the senior-most middle managers on a sub) designates someone the battle stations watch they are saying that person was the best on board at that job at that time.   Anyone wearing dolphins belonged there, no doubt.  You knew what they were made of.   But among them some were on top of their game.   And the crew knew it.

To some extent that happens on small engineering teams.  You know who you can go to for a nasty networking problem, or who can re-write a chunk of C code in a hurry, or who can index a database to make it reply faster…  excellence makes itself known.  But on bigger teams at bigger companies, it’s often about who plays better politics, or plays golf with the right folks, or who can just look better.  Annual performance reviews are quite often a joke and don’t really tell you at all how you are doing unless you have a strong manager who believes in the process of continuous improvement and career planning.  And reviews are often mostly secret.  There’s not really anything like dolphins on someone’s shirt to tell you who is really capable.  If you are a hiring manager and looking at someone’s past it’s often about what school they went to, or whether they worked at companies whose Engineering teams have a good reputation.  There’s no objective way to measure someone.  On subs, a new guy shows up and he has dolphins on and you know quite a bit about him (and now that there are female officers, her).

And while a person may think they are good, they may not be.  It’s true that the younger you are the more likely you are to overestimate your skills.  You think you are at a certain level, but in reality you are below that – we are often blind to our own gaps.  On subs, the crew would make those gaps glaringly visible and then help you fill them, since those gaps could get everyone killed.  In the workforce today, it’s more about being friendly and to be fair, no one is going to die, so it’s go along to get along quite often.  The open source world is a meritocracy, of course, but as the OpenSSL heartbleed debacle has shown, contribution to open source alone is NOT indicative of elite status.  Daily standups are good – they provide the opportunity for folks to publicly share what they got done.  But it’s at a team level at best.

I don’t really know how to translate submarine dolphins to modern software development.  I’m still thinking about it.  So my daughters question still rings:  “how do you know you are good at your job?”  In today’s world of software development, that’s a really good question.


Too Long… and a favorite code library

My son Luca pointed out to me that it’s been way too long since I blogged.  Geesh, my kids are reading my blog.  Amazing.  But he is right.  I have a lot of things I should be writing about, but I’m not making it enough of a priority.  That changes now.

At work we’re doing some new projects in C.  Yep, writing close to the metal for maximum performance.  It’s actually really exciting stuff (not the least of which is that I get to work with Ed Okerson again).  Ed has chosen to use one of our favorite libraries – the UCL Common Code Library.  Back in the day (oh, say, 2005) it was a nearly forgotten hunk of code that didn’t seem to be getting much attention anymore.   In fact, if you go look at it’s web page you’d think ‘abandoned.’

Not true.  It’s actually part of the Debian repository as ‘libuclmmbase’ and it’s on github too.  I’ll probably be writing a bit more about using it in the coming weeks.

It’s fun to be doing serious C again.  Of course, you all may think I’m crazy.  Like a fox…

Thunderbolt Drive Test Results: 4x Faster than USB 3.0

I got a new drive and wanted something a bit faster so I got a Thunderbolt drive.  But, I wanted to really understand how fast it was so I tested it (credit).

For writes, I did this:

time dd if=/dev/zero bs=1024k of=tstfile count=1024

For reads, I did this:

dd if=tstfile bs=1024k of=/dev/null count=1024

The new Buffalo HD-PATU3 works with either USB 3.0 or Thunderbolt and is a 1 TB drive.  I got it for a tad under $200 at Fry’s.

My results:

Write                          Read

MBP internal SSD                    371.301Mbps           359.458 Mbps

Buffalo 1TB Thunderbolt        88.7247 Mbps         109.269 Mbps

Buffalo 1TB USB 3.0                20.1375 Mbps          27.254 Mbps

Passport 1TB USB 3.0              20.6062 Mbps        24.6428 Mbps



Thunderbolt is well worth it if speed matters for you.  It’s about 4 times faster than USB 3.0.  But nothing compares to that wonderful SSD drive inside!


No Sonos SoundBar for me…

Well, turns out that the new Samsung TV I bought last year does not have optical digital audio output.  Slam!  That’s the sound of the door as my dreams of getting a Sonos Soundbar ended.

Darn.  Time to go think about another solution.

What Managers Do

I follow Kent Beck on twitter.  He recently tweeted “When you work more hours you see the steps that get made but you don’t see the leaps that don’t get made.”  That resonated with me and I kept thinking about it – and it took a bit for me to realize why.  On one of my teams I got asked what exactly managers do.  They were heads down building product and they really didn’t understand.  Why did they need someone to tell them what to do, after all?  They were doing it – and basically thought that some boss person telling them what to do was just waste.  More than once I thought they would have preferred if I just dug in and coded with them full time.

But Kent hit a key point: leaps happen not from heads down coding, but from seeing a broader picture.  Now, don’t assume that I’m saying that managers make those leaps.  Sometimes yes, often no.  But good managers should be creating an environment where leaps can happen.  And that means making sure folks are not working too many hours – or at least that if those hours are needed that you are eyes wide open about the costs.  It’s not just about sustainable pace and not burning people out.  It’s not just about hitting a point where you are introducing more bugs than you are fixing.  It’s not just about making poor design decisions because you are too tired.  It’s all that, but it’s also that you get too into the weeds and you miss the leaps you could have made.

Coders code.  Teams pull together.  Managers make sure the environment is right for maximizing the value being created.  And often that means staying out of the weeds themselves.  And making sure their teams are not burning out.  And making sure their teams are staffed with the best talent they can get.  And making sure that product asks are clear and actionable.   And asking their teams to explain and defend their technology choices, designs, and implementation – and making sure that it all makes sense to everyone – including to the non-Engineers.

And when all that goes right you have the best opportunity for someone to have that leap of understanding that will make a difference.

That’s what managers do.


Sublime Text 2 and Arduino – Stino!

This morning I was coding some Arduino stuff for my ROV Eddie.  I was playing around with a basic motor control app that talks to a Raspberry Pi over USB serial.  I want the RPi to be the high order brains of the robot and let the Arduino be the real-time controller.

The first thing I found was that writing real code in the Arduino IDE sucks.  Badly.  It does not even honor the basic keybindings my fingers know from forever…  so it was frustrating to say the least.  I’ve become addicted to Sublime Text 2 – a great editor – and I’m a command line wonk.  So first I tried ino – and had to manually copy my boards.txt file to a place it could find, and yet it still cannot compile.   I suspect some kind of version mismatch.  So I banged off the other rail and tried playing around with installing all the tools natively and just using a raw makefile.  Eh.  Lots of dependencies.  So I floated to the middle and found Stino.

Stino was easy to install and it just worked.  It added another menu in Sublime Text that lets me do everything I could have done in the Arduino app – and I get a real editor with syntax highlighting and everything.  Wonderful!

So, now I can really get busy on Eddie’s code.  :)

Setback with Eddie – leak!

Water test of the pressure housing with the wire penetration failed – minor leakage is seen inside.  However, my ‘poor man’s gasket’ of using plumbers putty may be the culprit.  I could not sense any moisture coming in around the epoxy potting – but it did seem that the putty was wet.  I just bought a circle cutter (found in a sweet craft store) and I’m going to try to cut a real gasket instead.  If that still leaks I may have to go back to the drawing board.  Huge disappointment since I just got a basic motor control app running this morning.  I was hoping to test steerage with two motors later today.

Oh well.  If it was easy everyone would do it, right?


Underwater ROV: we’re naming it ‘Edmond’

I’ve been designing a simple underwater ROV (remotely operated vehicle – aka robot) for a while, and when I have time it gets some love and attention.  The pressure housing is done but untested at depth.  I hope it will be good to 100 feet, but 50 feet would be just fine for this design.  The Arduino and Raspberry Pi are ready to mount.  The Logitech C920 HD web cam is taken apart and mounted but I have to shorten the USB cable to get it to fit.  I should have some pictures up shortly.

But the kids and I have decided on a name.  All good ROVs have names – Jason and Argo come to mind.  So we chatted a bit and my wife Erin came up with the idea of calling this robot “Edmond” after her Grandfather.  Eddie passed away Thanksgiving of 2011.   Eddie was a tinkerer and a mechanical ‘hacker’ in his day and he would have totally loved this project.

So, the robot is called Edmond.  Somewhere Eddie is looking down and smiling.  Here’s to you Eddie!