Research: How GitHub Copilot Helps Improve Developer Productivity

Software developers know that an estimated 90% of all computer code has already been written for an application somewhere in the world, so manually re-entering it into a task is counterproductive. In addition to the artificial intelligence (AI) tools that we now have in 2022 already know where all that code is in the libraries and can basically fetch anything in a matter of microseconds.

Companies like startup tabnin and open source tool store GitHub are directly in line with this trend of development tools that stay one or more steps ahead of the code being written. They read the beginning of the code, then make intelligent assumptions — then offer suggestions — in real time about how to complete it.

GitHub’s Copilot Goes GA

A year ago, GitHub launched a technical preview of Second pilot, a new AI pair programmer that plugs into any source editor and provides coding suggestions. Despite there being a limited number of tester seats, those who started using GitHub copilot told GitHub that the tool immediately became an indispensable part of their daily workflows.

And why not? As soon as someone starts typing a certain line, the application recognizes it and gives suggestions on how to finish; it even closes the parentheses correctly and in order every time.

As of June 21, GitHub became Copilot generally available to all developers.

“The feedback we’ve heard and continue to hear confirms our core thesis: AI can help developers be more productive and happier while coding,” GitHub lead researcher Eddie Aftandilian said in a media advisory. “Still, we wanted to test our theory and see if GitHub Copilot itself actually leads to higher developer productivity rates.”

Productivity Survey: Do Copilot’s Suggestions Help?

To find out, GitHub’s research and engineering teams teamed up to combine qualitative research data from more than 2,000 US-based developers with anonymized data to determine whether developers think GitHub Copilot makes them more productive — and whether the data proves it. that they are, in even more productive when using GitHub Copilot.

The idea behind GitHub Copilot is to make developers happier and more productive by keeping them focused on what matters most: building great software, Aftandilian said.

But “productivity” is a word that carries a wide range of possible practical meanings. Ideally, developers want to save keyboard strokes, or do they really want to avoid searches on Google and StackOverflow? Should GitHub Copilot help them stay in the flow by providing them with highly accurate solutions for mechanical, calculator-like tasks? Or should it inspire them with speculative stubs that can help unblock them when they’re stuck?

The answer is all of the above, to be sure. But GitHub’s research offered more specific answers. The research focused on answering three questions:

• Do people feel that GitHub Copilot makes them more productive?
• Is that feeling reflected in objective usage measurements?
• Which usage measurement best reflects that feeling?

“We found that developers didn’t care too much about reworking the suggestion, as long as GitHub Copilot gave them a suitable starting point,” Aftandilian said. “And this makes sense: GitHub Copilot isn’t designed to build your own software — it’s designed to provide helpful suggestions that make it easier to stay in the flow. Copilot provides developers with the parts, but leaves it up to them to do it.” to assemble and design the end product.”

One keystroke, multiple options

Does Copilot offer multiple options depending on the keystrokes entered?

“Copilot can return suggestions with just one keystroke,” Aftandilian told VentureBeat. “However, the quality of suggestions usually improves with the amount of context available through both code and comments. Copilot often offers multiple suggestions via a ‘cycling’ menu, as well as the ‘Open Copilot’ side panel that shows 10 suggestions.”

Developers tell GitHub they like Copilot, Aftandilian said, because it helps them stay in the flow longer and focus on application architecture rather than the syntax of specific APIs and other methods, etc.

“We’ve heard feedback from developers that, even if Copilot isn’t quite right, users appreciate the suggestions as a helpful starting point,” he said. “For example, Copilot will often suggest the correct control structure (e.g. the structure of a loop), even if the details are not perfect. The user can keep the structure, but adjust the details, and they find that useful.”

You will find the latest research here in a published academic paper that GitHub teams presented last month at PLDI’s MAPS ’22 conference in San Diego.