GitHub Copilot is now public. Here’s what you need to know.

GitHub announced last week that it will release Copilot, its “AI pair programmer” tool, to the public. Copilot uses AI to provide a range of support functions, including autocomplete instructions, generating full functions, and transforming docstrings and descriptions into functional source code.

Copilot was introduced in 2021 as a technical preview. Now all developers can apply for Copilot, which installs IDEs as an extension in integrated development environments (IDEs) such as Visual Studio, US Code, Neovim and JetBrains.

At the time of the release of Copilot, there was a lot of excitement about its beautiful coding capabilities. But there were also concerns about how far its capabilities could be trusted and whether it would have a real impact on the productivity of developers. After a year and billions of lines of code, Copilot is finally ready to be in the hands of every developer.

Here’s what we know about Copilot’s effect on actual programming tasks, told by the creators and developers who used it in their daily work.

How much code is written with Copilot?

Behind Copilot is the transformer architecture, the kind of deep learning model used in large language models such as GPT-3 and LaMDA. Transformers are particularly good at processing sequential data such as text, software code, and protein sequences. Given an assignment, a transformer model can predict the following elements of the sequence, whether they are words or computer instructions. Copilot is built on OpenAI’s Codex, a transformer trained at tens of millions of code repositories. Once installed on your IDE, Copilot provides suggestions based on the existing code in your file as well as clues, such as the names of your features and classes and the comments in your code.

It is noteworthy that Copilot does not think and code like a programmer. However, since it is exposed to large amounts of source code, it can provide very good code suggestions, especially about standard and repetitive tasks that occur frequently in software writing.

According to GitHub, more than 1.2 million developers have used Copilot’s technical preview over the past 12 months. In files where Copilot is enabled, it accounts for almost 40% of the code in popular programming languages ​​such as Python.

Ryan J. Salva, VP of Product at GitHub, told VentureBeat that while it’s hard to say how much of the coding done with Copilot is real software development, as opposed to exploring the tool, the 40 % ratio appears to hold if developers use Copilot over several days.

Github Copilot reduces distractions

Developers often talk about the value of staying ‘in the flow’ and the positive impact [Copilot] on their productivity, creativity and overall happiness, ”said Salva.

According to Salva, in a survey distributed to 17,000 developers during Copilot’s technical preview, more than 75% of developers themselves reported that when they use Copilot, they “spend less mental effort on repetitive programming tasks”, “focus on more satisfactory work “and” stay “. in the stream. ”

“By reducing distractions and creating focus time, we not only get jobs done, we create better and less stressful days,” Salva said. “Anecdotally, we’ve heard stories of developers using Copilot to learn new coding languages, quickly generate boilerplate code for common tasks, write regular expressions, or simply recall syntax for an API without consulting documentation.”

Developers VentureBeat spoke to confirm some of these points. Abhishek Thakur, a machine learning engineer at Hugging Face, has been using Copilot since June 2021. He used it in the development of AutoTrain, a no-code tool for training modern machine learning models. He also uses it for machine learning competitions on Kaggle, making tutorials and participating in hackathons.

“When I code, I want the least distraction. That way, Copilot was a big help. It has reduced the time I may spend searching for solutions on the web and has it at my fingertips in my favorite IDE, ”said Thakur.

Many developers are looking for solutions to small problems on search engines and StackOverflow, a web forum where developers share snippets of code for specific tasks.

“After using Copilot, I rarely visit these sites and can rather focus on coding,” said Thakur.

“When in doubt, I try to write comments and let Copilot help me complete the code snippets,” Thakur said. “It may not always be perfect, but it gives a good idea of ​​how the code can be written. The rest is up to the developer to edit and reuse. The same goes for StackOverflow: it does not always have the answer, but it does have very good answers that may suit your use case, and you can change and reuse. ”

Taking on repetitive tasks with Copilot

Louis Castricato, a research intern at Hugging Face and formerly at EleutherAI, used Copilot for scientific computing, where features are often cumbersome and difficult to use. One of these features is PyTorch’s loneliness, which requires difficult parameters about the dimensions of tensors you want to calculate.

“Loneliness is very unintuitive for people who have never used it before, and it requires you to pay close attention to the shape your tensors take with each instruction within a call to your model,” Castricato said. “Copilot is especially strong at deriving the shape of a tensor and automatically writing loneliness operations, as well as writing comments that explain the choices he made in writing the loneliness expression.”

Snir Shechter, R&D team leader at Aporia, has been using Copilot for almost a year. “In developing our main product at Aporia, Copilot is helping me write the easy code,” he said. “Given good naming conventions, Copilot is able to complete the entire function / next block code. After that, I just need to review to see that everything is fine (and possibly add more specific logic). It is very good with the completion of generic / repetitive code and it finds out based on the context. ”

Copilot’s performance stands out especially for long and repetitive tasks. An example is the introduction of an HTTP server, which usually requires several lines of code and customizations, depending on the language in which it is written. In one study, GitHub required half of the participants to manually write the HTTP server code and the other half to complete the task with Copilot.

“Preliminary data indicate that developers not only tend to complete their task when using Copilot, but they also do so in about half the time,” Salva said.

Push developers to better document code

Copilot works better when programmers provide it with more detailed descriptions. Interestingly, its use has driven developers to better document their code.

“In the first few months of the technical preview, we saw Copilot change people’s behavior when writing code – by writing better, more detailed comments,” said Salva. “Not only does this improve Copilot’s suggestions, but it also makes it easier for others to read.”

Copilot has also become a great tool for documenting software code, a task that is often overlooked, especially when programmers are chasing deadlines.

Castricato uses Copilot to document his code, autocomplete docstrings, and type suggestions in Python. It improves the readability of the code and makes it easier for himself and other developers to manage the code later.

“Copilot has increased the amount of documentation I write for my code by at least 2x or 3x,” he said.

The boundaries of Copilot

“[Copilot] is often very weak with the implementation of whole algorithms, ”Castricato said. “For example, when I first got Copilot, I wanted to see if it could implement basic forms of dynamic programming without significant guidance. This failed miserably, and I realized very quickly that in order to use Copilot to its full potential, you need to explain in detail the steps that Copilot needs to take to implement a certain algorithm (through comments).

In addition to basic tasks, Copilot will need enough comments to function properly. And in some cases, it will require a fully structured code file to provide useful suggestions.

“In this respect, Copilot is very far from replacing even the most rudimentary software engineers,” Castricato said.

Salva acknowledged that Copilot is still a work in progress and is a new developer experience. The product team continues to learn lessons from how developers use it and adapts the AI ​​model that drives it.

“Copilot tries to understand your intent and to generate the best code it can, but the code it suggests may not always work or even make sense,” Salva said. “While we are working hard and seeing progress in Copilot that is generating better code, proposals need to be carefully tested, reviewed and reviewed, like any other code. We collect telemetry data to make the model better, with which we encourage users in the UI. ”

Is Copilot worth the price?

Currently, Copilot will be offered at $ 10 per month, or $ 100 per year, with a free 60-day trial, which appears to be a bargain for software developers. In addition, the tool will be free for students and maintainers of popular open source projects.

“I think it’s totally worth the price,” Thakur said. “As a machine learning engineer, I know there is a lot going on in building products like this, especially Copilot, which provides sub-millisecond latency proposals. Building an infrastructure that serves these types of models for free is not for ‘. a longer period in the real world. “

Thakur also noted that as the cost of AI infrastructure continues to decline, the price of Copilot may decrease in the future.

“But at this point, in my opinion, it’s totally worth the price, especially for someone like me who’s used Copilot almost every day since the preview,” he said.

Castricato said that Copilot saved him several hours a week, and sometimes even a day or two of problem solving a week.

“Copilot allows me to test many experiments quickly – often without spending a lot of time debugging. A set of experiments that would take me conventional days to implement would rather take me one day,” he said. As a professional tool it is well worth its price. It definitely makes me over $ 10 a month. I can easily see that any large company justifies a Copilot license for all their technical staff. It’s almost an insignificant expense. “

Salva believes this is just the beginning of AI-enhanced programming and sees Copilot as the next step in a long line of developer utilities.

“As we have seen with the compiler, higher-level programming languages ​​and open source itself, tool advancement has amplified the impact that developers have in our world,” he said. “At the same time, those same tools are no substitute for a developer’s experience, skill and creativity.”

With better tools, I added, the industry’s demand for developers has gradually increased. “We are optimistic that GitHub Copilot will have similar effects, which will complement the work of developers and empower them to write code more easily with greater focus and creativity,” he said.