The Story So Far
About two weeks ago, I wrote a blog post suggesting that Scala can make Java developers more productive. One of the comments on that post was that I had no metrics and so was spreading “folklore”. I took the criticism constructively and decided to solicit a small survey of Scala developers to try and get some insight.
What I Wanted to Learn
One of the most important things I’ve read about Lean Startups – and it’s important because it can be applied to a gamut of contexts, not just startups – is that, while the execution of the Lean process is Build, Measure, Learn, you actually have to apply this backwards to get good results: first you decide what you want to learn, that will inform what you need to measure, and that will determine what you need to build.
I employed this advice with this survey, asking first: “What do I hope we learn from this”, and using that to inform the content. Here are the questions for which I wanted the survey to provide some insight:
- Do developers feel more productive working in Scala?
- Do teams deliver more value with Scala?
- Do feelings of productivity translate to teams delivering value?
- How is someone’s sense of productivity affected by:
- the length of time they’ve been working with Scala,
- the proportion of Scala they practice at home and at work,
- the proportion of Scala in their work codebase, and
- their depth of understanding of functional programming?
- Does depth of understanding of functional programming translate to teams delivering value?
The Method
Here is the method I used to collect data which I thought would help give some insight on these questions:
- I created a short online survey,
- that asked Scala developers with Java experience
- to give a subjective opinion on whether they feel more productive in Scala when compared to Java
- and whether they think their team produces more value than when using Java
- and to estimate the % of time they spend using Scala at home and at work and the % of Scala in their codebase
- and to rate their own expertise in Functional Programming
- I publicised the survey on Twitter and Google+
Criticism of The Method
It would be obvious to a 10 year old: this is not a highly-scientific method of collecting data. Every italicised phrase in the points above highlights a weakness of my technique: the participants are not randomly selected, are probably all biased to varying degrees, and have been asked to give subjective opinions and estimates for which they themselves probably have no data. In short, I’ve asked a skewed audience to share their gut feel. But, what else could I have done? I don’t think it would make much sense to ask people who don’t use Scala if it made them more productive. I don’t have the resources to undertake an accurate measure of people’s or teams’ productivity before and after taking up Scala. And I don’t have the social clout to get more people taking the survey than already took it. So, by my measure, what I’ve done is the best I could do given my situation.
If you think the method I’ve used is codswallop and makes the output totally useless, then there’s something productive you can do about that: stop reading now. Don’t bother reading the results and making yourself more upset. Please don’t try to “make things better” by adding a comment telling me how bad my method is; you’ll just expose yourself as someone who comments on blogs without reading them properly.
If you have some great ideas on how this survey could have been better – methods of collecting better data, questions that should have been asked, options that should have been available, channels that should have been tapped – there’s something productive you can do about that, too: create your own survey, that’s better than mine, and then tell publicise it better than I did. I won’t be offended. I’d be really interested in it and probably help you. But again, please don’t add a comment telling me my method could have been better if I’d just done X, Y and Z; I already know that it could have been better, and you’ll just expose yourself as someone who comments on blogs without reading them properly.
Results
Enough yabba yabba. I got 441 responses. Below I’ve charted the results of the survey, including some correlations between survey answers that I believe help answer the questions I started out with above. The images in the post are scaled down, so you’ll get a better view if you click on them to open the full version.
I’m not actually going to write about any conclusions from these. I think it would be far more interesting to hear what YOU think stands out, so if you see something you think is interesting, please add a comment! If you want to know what other people have seen in here that’s noteworthy, just keep scrolling.
Do developers feel more productive working in Scala?
Do teams deliver more value with Scala?
Do feelings of productivity translate to teams delivering value?
Does the length of time someone has been working with Scala affect their sense of productivity?
Does the proportion of new code that someone writes in Scala affect their sense of productivity?
Does the proportion of Scala in the codebase someone works with affect their sense of productivity?
Does a developer’s understanding of functional programming affect their sense of productivity?
Does understanding of functional programming translate to teams delivering value?
Data
As promised, here is the data if you have some interest in examining correlations other than those I’ve charted above.
ScalaProductivitySurvey_2013_Processed.xlsx
Image credit: Mykl Roventine
Want to learn more about Scala?
This is some very interesting data you have collected there, and while the sample is definitely biased as you say, the correlations still show some very interesting things. For example that people seem to grow more confident in Scala at home (there’s a slight shift in center of gravity going from zero years up) and not at work. This means that Scala is a technology which is partly picked up out of own interest and not due to workplace requirements; it would be interesting to see these measurements for other languages as well.
One thing I picked up on, you didn’t state exactly what is meant by “percentage of codebase” and “percentage of new code”. Is this strictly on a JavaScala axis, is it ScalaAnything but only for core system logic, or does it include build scripts and other such lint?
It would be interesting to run this again asking
* the infamous A/S/L triumvurate
* whether or not people have taken a formal Scala course
* …and Martin’s Coursera course
* education level
* Overall years experience as a programmer
* Top 3 other languages known/used
Thanks for your thoughts, Kevin.
The motivation behind these questions was a friend who works at Twitter and told me “I found 5% Scala to be a horrible experience. Finding 95% Scala to be great.” I wanted to test if that anecdote was supported widely, i.e. that using a little bit of Scala is worse than using none at all. A cursory glance at the relevant chart suggests the answer is no, but there is a strong theme running through the results (which Roland has already noted) that there are a lot of people doing Scala at home but not at work and still claiming increased productivity.
Your idea of delving deeper into what functional concepts people have and haven’t learned, and how they’ve learned them, would be quite interesting. Will you pursue it? 🙂
As a side note, it would be highly interesting to get an idea about what people made themselves rating their understanding of FP.
I.e.: What made them think they are beyond basic level (or not) and what they miss to enter high level (if they aren’t high or beyond).
But that is surely beyond this survey, that is purposefully based on subjective self estimation.
Great survey and sample about the perception of Scala vs. Java productivity.
Thanks, Stephan!
If you query a group of developers who have moved from language x to language y if they have been more productive with y, the results are to be expected .
What was the reasoning to have “with Java experience” in the criteria?
Hi Sean. Good question. If you look at the history that motivated the survey, it was an article where I encouraged Java developers to consider Scala like a library that would make them more productive. The challenge to produce some data came through that blog and that’s why I focused on Java.
Those graphs, Excel or something else?
I wrote two little Scala scripts to do the bubble charts. This one to create a Pivot table from a CSV:
https://gist.github.com/GrahamLea/5036541
and this one to turn the pivot tables into bubble charts:
https://gist.github.com/GrahamLea/5036561
I added the labels in using InkScape.
Nice survey. It suggests among others that understanding of Functional Programming as only moderate impact on the speed of delivering value.
That is also what I would have expected. Scala is far more than just OO+FP.
For a next survey, I’d suggest adding the following questions:
For each language in {Scala, Java}:
– do you like developing in this language
– to what extend would development in this language make a job offer more attractive to you
For each Scala language feature, library or tool in {
No semicolons
Functional style
Case classes
Singleton Objects
For comprehensions
Match expressions
Tuples
Definable operators
Call by value parameters
Traits with variables and values
Type classes
Macros
Scala Swing library
Normal collections library
Parallel collections library
Akka
SBT
Eclipse IDE Scala plugin
IntelliJ IDE Scala plugin
}:
– To what extend do you like it
– To what extend do you does it make you more productive
(both questions relative to what you were used to with Java)
Thanks, André. There’s a number of ways to interpret the FP results. One basic observation is just that there aren’t many people who feel they have a high understanding of it. But what you pointed out is nice: that there’s lots of weight among those with Intermediate understanding, and even Basic, that still feel they’re getting productivity gains out of it.
Your additional questions are interesting, but I think I’m done with surveys for the moment. Even this simple one with < 500 respondents absorbed quite a lot of my time. I think it would be great if Typesafe did some future research into the kind of things you've suggested to try and "read the pulse" of the community and feed that back to the community to help us know how to move forward and help others move forward.
It would be interesting to know what their business owners or customers felt about the productivity and value changes, if any.
For whatever reason I’m finding the text difficult to read — it is too bold, making text blurry particularly on the smaller fonts.
Thanks for the feedback, Daniel. On the histograms, the boldness is a rendering artefact of Libre Office printing diagonal labels. 🙁
On the bubble charts, the fonts should be more readable if you look at the full-res version by clicking on the images. I forgot to mention this in the post, so thanks!
So, having now gone through it all, I’d have liked if you had provided graphs for “Does the length of time someone has been working with Scala affect their sense of productivity?” and the next question with circles as a percentage of developers in that category.
Right now, because so many responders fall into 1-3 years experience, it is difficult to compare those columns with the ones around them.
I think you’re just saying it would have been better if the bubble chart labels were percentages instead of the # of samples? That would definitely be an improvement. Next time. 😉
If you’re dying to know the percentages, most of the questions have 441 respondents, so you can divide most sample numbers by 4.4 (or multiply by 2/9 if you’re a more rational kind of person).
Pingback: Scala Productivity. A Survey of the Community | JetBrains IntelliJ IDEA Blog