Survey shows AI coding tools require evolution of performance metrics

Survey shows AI coding tools require evolution of performance metrics

Amid the rapid advancements in generative AI, GitHub wanted to get a better understanding from developers about how new tools — and current workflows — are impacting the overall developer experience.

GitHub surveyed 500 U.S.-based developers at companies with more than 1,000 employees to get a better understanding from developers about how new tools — and current workflows — are impacting the overall developer experience.

The survey showed AI tools are already being widely adopted by developers, that these tools are having a measurable impact on individual productivity and team collaboration and that they will require engineering leaders to adjust how performance is measured.

One-third of developers report that their managers currently assess their performance based on the volume of code they produce (33%).

The other top reported metrics for performance are largely output-based: 40% are measured by code quality, 34% by speed of completion, 34% by production incidents and 33% by number of bugs resolved.

However, with the increasing use of AI tools in software development — which often contributes to code volume — engineering leaders will need to ask whether measuring code volume is still the best way to measure productivity and output.

According to GitHub’s research, developers say AI coding tools can help them meet existing performance standards with improved code quality, faster outputs and fewer production-level incidents.

They also believe that these metrics should be used to measure their performance beyond code quantity.

  • 35% of developers believe they should be measured on collaboration and communication
  • 34% of developers believe they should be measured by code quality.
  • 34% of developers believe they should be measured by number of bugs or issues resolved
  • 34% of developers believe they should be measured by time to complete a task
  • 32% of developers believe they should be measured by test coverage
  • 32% of developers believe they should be measured by velocity and quality of feature delivery

Inbal Shani, Chief Product Officer, GitHub, said: “Developers today do more than just write and ship code — they’re expected to navigate a number of tools, environments, and technologies, including the new frontier of generative artificial intelligence (AI) coding tools.”

“But the most important thing for developers isn’t story points or the speed of deployments. It’s the developer experience, which determines how efficiently and productively developers can exceed standards, enter a flow state, and drive impact.

“Ultimately, the way to innovate at scale is to empower developers by improving their productivity, increasing their satisfaction, and enabling them to do their best work — every day. After all, there can be no progress without developers who are empowered to drive impact.”

Click below to share this article

Browse our latest issue

Intelligent CIO North America

View Magazine Archive