“I inherited a developer experience team and was supposed to be in charge of understanding our development lifecycle and its pain points. I had no idea how to gather that information. I started creating a form and then, after a couple of hours of ripping my hair out, I thought there must be a better way to do this.”
That’s how a Senior Engineering Manger at a SaaS company with around 200 engineers described their experience of creating surveys to me.
You might have considered running a developer experience survey but are wondering what to ask. Or perhaps you have already run surveys but aren’t sure if you’re asking the right questions. Either way, it can be a bit of a muddle. Lousy questions can create a reliability mess of inconsistent results or lead to validity snags by measuring something else than you think.
Good news! We’re publishing these 32 expert-created questions, originally designed for Swarmia’s developer surveys, for everyone to use. Scroll to the bottom of this post to copy our questions or keep reading to learn how we landed on this exact set and find out how to use them to run better surveys.
Building a solid survey framework
We wanted to create a framework to capture all corners influencing engineering effectiveness. To do so, we began with our three-part approach: business outcomes, developer productivity, and developer experience. We divided these into subcategories, the reasoning for which you can find at the end of this post, and sourced ideas from our engineers and leadership team.
We ended up with a list of more than a hundred items. To distill them, we looked for tangible indicators with the most predictive power. For example, getting “blocked by documentation gaps” is a specific event the respondents can easily recall, while it represents more broadly the overall availability and quality of documentation.
We also realize that software engineering expertise isn’t enough to create an accurate and actionable developer survey. Questionnaire design is a science of its own, and there are many pitfalls when creating even a simple survey. We recognized this early on and engaged with experts in psychometrics to help us get it right from the start.
With their advice, we designed simple statements to be used with a five-point Likert scale. Our methods may not excite anyone else than zealous surveyficionados, but are reliable and well-researched. Each statement underwent feedback cycles to ensure it adhered to best practices, such as referring to tangible events, measuring only one thing, and explicitly linking to context. You can read more about the methods here.
We tested the questionnaire in the real world, first in a survey of Swarmia engineers and then with our customers. The questions clearly differentiated between teams and helped surface frictions and frustrations, as well as sparked respondents to write constructive comments.
How and when to run surveys
We recommend running this type of questionnaire with your engineering teams every three months or so. You can have people respond with their names to facilitate transparent discussion or run a confidential survey to reduce the barrier to reporting problems. Remember to tell people in advance how you’ll use their responses.
To set the stage, ask the respondents to think about events in the last three months (or another specific timeframe), focus on their primary team, and answer based on the code and systems they work with. Request them to rate their level of agreement with each statement on a Likert scale, for example, with five steps. We also advocate collecting voluntary comments for each item to give more details and propose improvement ideas.
After closing the survey, you can compare the results among teams or other dimensions. Using the same questions in consecutive surveys allows you to analyze trends over time. Some issues are best addressed across the organization by the leadership, but teams often have the best grasp and drive to solve challenges on their own. Hence, we recommend openly sharing the results and being clear about the actions taken based on them.
The questions you should ask in your next developer survey
That’s it for the preaching. Without further ado, here’s our set of questions for you to use, along with our rationale on each category’s importance.
Direction
Empowered engineers who are aligned with the business direction make better decisions and are more motivated.
- My team is empowered to make decisions to reach business goals.
- My team has clear priorities.
- I can influence what my team works on.
Collaboration
Tight-knit teams can pool their efforts, making them more effective and resilient. Dependencies between teams add overhead and delays, although they can rarely be fully avoided.
- I always get help when I need it.
- My team’s new members become productive quickly.
- All the meetings I attend are useful.
- My team gets most of its work done without depending on other teams.
- My team collaborates effortlessly with other teams.
- Multiple people can work on each of my team’s features.
Culture
Teams with high psychological safety and regular feedback improve faster than others.
- My team frequently improves its ways of working.
- I frequently get high-quality feedback from my team.
- I feel safe expressing concerns to my team.
User needs
Software is only as valuable as the problems its users solve with it.
- My team systematically validates user needs.
- My team’s new features consistently get the user adoption we anticipate.
- My team extensively validates the usability of our solutions.
Managing work
Working in smaller increments at a time decreases risk and makes the feedback loop faster.
- My team releases code in the smallest practical increments.
- All deadlines in my team are realistic.
- My team tests our technical assumptions before implementing them.
- My team has enough time to experiment with new ideas.
Focus
Many complex software problems require long periods of intense focus.
- I have enough uninterrupted time for focus work.
- My team always has the chance to finish the work we start.
Quality
Software quality is a matter of pride for engineers and reliability to users. Fixing bugs and resolving incidents isn’t particularly fulfilling for most developers.
- Our automated tests catch issues reliably.
- Our code reviews adhere to high standards.
- Our technical debt is well under control.
- Our practices steer toward building secure solutions.
- The on-call load in my team is reasonable.
Development
The fluency of day-to-day development is a deciding factor in developer satisfaction and productivity.
- I never get blocked by documentation gaps.
- Our tech stack is well-suited for the problems we solve.
- It’s simple to make changes to our codebases.
Systems
Time spent waiting for machines is time out of building things and typically reduces the rate of delivering new value to customers.
- Deploying code is simple.
- We have the proper tools for debugging production issues.
- Our automated tests speed up the development feedback loop.
Over to you
Whether you use Swarmia to run your developer surveys or not, we hope you find this framework helpful when you ’re planning your next survey.
