Xebia’s Latest GitHub Copilot Survey

Our Latest GitHub Copilot Survey 

We at Xebia strive to stay current with the latest advancements in AI tools to build robust and durable solutions for our clients. Many of us employ a variety of Code Assistants to speed up our software output and optimize our products.  

Among the most popular code assistants we use is GitHub Copilot. This tool has seen incredible growth in the past two years, and as more developers use it, we’ve become curious about its effect on software development. This is why we conducted a survey among 56 developers in our network to gather the latest insights about the tool. Read on to discover the results!  

To Copilot or Not to Copilot  

The first question we asked developers was whether they use Copilot or not, and how often. An overwhelming majority of them (95%) use it at least some of the time. Around 42% of them use it constantly. The details of the distribution can be seen in the next two graphs for developers with under 10 years of experience (31% of the population) or more (69%).  

All these developers work with popular programming languages like Python, .NET, JavaScript, and C#. They also come from companies where cloud platforms are prevalent. Most of them work daily with Azure, AWS, and Google Cloud.  

Once we established how often developers work with Copilot, we wanted to know what type of projects they use it for. These were their answers: 

To understand this figure in detail, consider that the 0% line separates people who don’t use Copilot or have used it for less than 3 months for a given task, and people who have used it for longer than 3 months. As you can see, more than 50% of the developers have been using it for all sorts of projects for more than a quarter already, and many of them for more than a year. 

The Impact on the Job 

One of the main benefits that code assistants bring to developers is saving time. So, naturally, our next question was about how much time they save on different tasks. These were their answers. 

The most significant time savings came in basic code generation, code documentation, unit tests, and scriptwriting. On the other hand, code refactoring, helping in understanding existing code, and especially complex-problem code generation weren’t so affected by the use of Copilot. 

While this confirms the idea that code assistants are great at automating simple tasks but not so much at producing advanced code, there is more to it. We must consider three important factors in this survey: 

1. Subjective estimation based on experience: Programmers rely on their experience and subjective judgment to estimate the time saved by using Copilot. Many respondents reference their typical workflow and compare how long it would take to complete a task with and without Copilot.
2. Variable Impact Depending on Task and Skill Level: The perceived time savings vary with the type of task and the programmer’s familiarity with the language or tool. For simpler tasks like generating boilerplate code, Copilot can significantly speed up the process. However, for more complex tasks or when the generated code requires significant review or adjustments, the time savings may be less pronounced.
3. Balance Between Productivity and Quality: While Copilot helps save time by providing code suggestions and speeding up repetitive tasks, some developers emphasize the need to review the generated code for quality assurance. As shown in the next figure, both experienced and relatively newer developers agree that establishing the impact on code quality as positive or negative is “hard to say”. 

The comprehensive GitClear report, which analyses more than 150M lines of code, suggests that code quality might be declining in some respects. It is therefore key to keep on assessing code quality as developers integrate code assistants into their toolsets.   

The Emotional Impact 

The interviewed developers significantly agreed with statements like “Using GitHub Copilot makes me feel happy about my work” or “Using GitHub Copilot allows me to enter the ‘flow’ state quicker.”  

The most relevant question regarding job satisfaction was whether they agreed with the following statement: “Using GitHub Copilot allows me to focus more on satisfying and meaningful work.” The results are shown below.   

Indeed, there’s a consensus that Copilot allows developers to focus on more meaningful tasks. However, an interesting question arises when we compare this graph to the two previous visuals. Individually, developers seem very happy about the impact of Copilot on their work. But company-wise, will this affect the quality of the code? Only time will tell.  

In the meantime, if you want to dive deeper into the role of code assistants and Large Language Models in general, don’t forget to download our ebooks and whitepapers on GitHub Copilot, Code Assistants in Software Development, where we discuss in-depth the characteristics of Copilot and many other tools, and How to Prioritize LLMs Use Cases.