Confession of a Chronic Futurist

I have a confession to make. I used to be a dreamer at work. Worse than that, I used to invest significantly into putting my dreams into motion.

What did that look like? It meant that when I was tasked with building a Model-2 presentation framework for the company I was working for, I bunkered down for over a month and came out with this massive framework that was going to be a solution to the whole world’s problems. It was basically most of what Struts did and some parts of what Spring does, before either of them existed. There were good things in there, and the company used the framework for many years, but I’m sure more than half the code I wrote never got used. I’d latched onto a good idea and run a marathon, but the team really only needed me to walk a mile.

Pragmatic Redemption

Then came the revelation of my software engineering life: agile. In 2006 I joined Tyro Payments, read Extreme Programming Explained, and realised instantly how my tendency to build for the future had resulted in large amounts of lean’s dirtiest word: waste. In particular, the type of waste known as “overwork” or “over-processing”.

My grand design thinking was assaulted by many of the tenets in the agile manifesto and principles:

Value working software over comprehensive documentation (GL: or software that’s not even used)

Value Responding to change over following a plan (GL: or responding to dreams)

satisfy the customer through early and continuous delivery of valuable software

and especially this principle:

Simplicity–the art of maximizing the amount of work not done–is essential

YAGNI 4 EVA

YAGNI – “You ain’t gonna need it” – became one of the chief guiding forces of the way I wrote software. But it didn’t stop at work. YAGNI started reaching its fingers into many of my life decisions.

'But it didn't stop at work. YAGNI started reaching its fingers into many of my life decisions.' Share on X

When I thought I wanted to take up cycling, but wasn’t sure if it would end up a long-term thing, I bought a $99 bike from K-mart and rode it until it fell apart before investing in a brand name bike. My wife and I often talk about making life decisions in an “agile” way: taking small steps towards a perceived long-term goal, but monitoring and reviewing along the way what the next best step is. For example, we’re thinking about making some  changes to the internals of our house over the next decade, and we’d love to be able to run this “project” that way.

Agile turned me from a chronic futurist into a raging pragmatist, which was quite a turnaround. I now often refer to myself as “a reformed perfectionist”, as I really do have an innate desire to do everything immaculately, but through learning how useful software can be built one highest-priority step at a time, I’ve come to realise that “good enough for today” is a great motto for most facets of life.

Enter: Dan

And then came Dan North. Dan was speaking at the YOW! conference in Sydney last week, and someone at Tyro who has a connection with him managed to swing an all-day exclusive workshop with Dan for Tyro’s engineering leadership team.

It’s no secret that Tyro has been growing its engineering teams massively over the last 3 years, or that we’re planning to keep doing it. So, as you’d expect, we spent a good part of the day with Dan discussing organisational structure and how we can meet the challenges of growing. Dan started describing how Spotify is scaling agile and, while we knew all about the tribes and guilds before and have even started copying some of their ideas, something new jumped out to me when Dan said:

“The reason they ended up like this is because they looked at how big they wanted to be in 3 years time, and they asked, ‘What structure are we going to need in place to support that?'”

That sentence hit me like a bolt out of the blue. Spotify is a modern poster-child of agile, and yet here they were asking “what do we need in 3 years’ time?” Have they never heard of YAGNI?! I pondered that over the next few days and came to realise a couple of important things about why agile methods work.

Putting Agile in Context

Firstly, YAGNI leaves you being chiefly reactionary in your planning and that means a lot of things happen “just in time”. While agile methods are great at responding to change, for “responding to change” to be a good thing you need to be able to respond in a reasonable amount of time. What if you can’t? What if you recognise a need to change, but the change will take a long time to effect? That’s how big corporations are getting disrupted the world over by technology right now. YAGNI is most beneficial in contexts where you can change quickly.

Secondly, the iterative feedback used in agile has a disadvantage in that it’s susceptible to getting caught in local maxima. By moving in small, reactionary steps, we hope to always move to a slightly better place, but if there is a much better place to be, it might be impossible to get there without a huge leap in thinking or activity. This means iterative processes are most useful where there are few local maxima (or the difference between multiple local maxima is small). Where there are multiple local maxima which differ greatly in payoff and are far enough apart, an iterative process may get you literally stuck in a rut.

Thirdly, agile methods have the most benefit when the end state is not well known. When you don’t know who your customers will be, or where they’ll be, or how many there will be, or what features they’ll like, or what price they’ll be willing to pay, or when your money or luck will run out, then agile methods provide a fantastic process for discovering that information and incorporating new knowledge into the direction of the product or project as it advances. However, if you have a very good idea of where the end goal is, trying to be agile might not give that much advantage. For example, if the goal of a project is to implement a well-known specification and get the implementation certified, there’s really only one iteration and one feedback event: “Did we get certified?” The benefit of feedback is not a dichotomy; it’s on a scale, so the more you know about the end goal, the less benefit you get from being iterative, because any feedback collected doesn’t provide new information.

Pulling all of that together, the perfect context for agile methods is where the end state is not well-known, there are few local maxima or they are relatively close in value, and you have the ability to change quickly. This describes a large majority of software projects, but it doesn’t describe everything in software, everything in business or everything life. Agile methods apply best in contexts with certain criteria; outside those contexts, they can do great harm. Yes, you can be too agile.

'Agile methods apply best in contexts with certain criteria; outside those contexts, they can do great harm.' Share on X

Spotify’s Non-Agile Change of their Agile Culture

Let’s look back at Spotify’s situation. A large part of their end state was known: they knew they wanted hundreds more software engineers, and they were very sure their existing process wouldn’t work with that many people. They knew it was culture change that they were embarking on, and so it would definitely be slow to change, or at least to get feedback. Finally, it can be difficult to measure and chart culture, and hence to figure out whether it has many local maxima or not, but the plan to stand back and take a look at the big picture rather than iterating towards incrementally better cultures seems like a wise one.

Which is Better? Pragmatist or Futurist?

What I’ve learned as I’ve pondered all this is that pragmatic thinking and futurist thinking both have their place. There are many problems – both in the software world and outside it – that benefit from an iterative, feedback-driven process. There’s also many problems where a better solution might be found by taking a futurist, long-term, big-picture, big-dreamer approach. And, as with most things, there’s a big spectrum in between.

This blog isn’t a call to abandon agile or pragmatism – far from it! While writing this, I wondered if the thesaurus listed “futurist” as an antonym to pragmatist. I didn’t find one that did. Instead, the antonyms I found for pragmatism (which I hence consider synonyms for being a futurist) were: theoretical, unrealistic, inefficient, idealistic, impractical, unprofessional, airy-fairy, starry-eyed. There’s a warning in these words: too much future thinking without enough short-term accomplishment and you can end up with your head in the clouds. Pragmatism still has its place; the world moves fast and you need to be moving with it. Perhaps a good combo to use is futurist thinking for strategic goal-setting and pragmatic thinking for tactical progress. How the mix works long-term is something I need to keep reflecting on with time.

What’s the takeaway?

Agile is great. Small iterative improvements with regular feedback are powerful. But whatever you’re using iterative processes for, make sure you temper them by sticking your head up every now and then and asking:

“Where are we going, and is this iterative process going to get us there, or do we need a big shift?”

Image credits:
‘Daydream‘ by Colton Witt
‘I’m not talking to you!‘ by Andrew Hurley