The News, Explained
On November 5, GitHub announced two additions to the Copilot usage metrics API, which lets other software fetch report data. Daily repository reports now show how many code review comments developers addressed. User reports flag missing editor usage records, and organizations and enterprises can also see daily counts of people with those gaps. Source Source
A pull request asks a team to review a proposed code change and merge it into a project. GitHub evaluates Copilot review comments on the day that pull request merges. To get the addressed rate, divide the number of comments developers addressed by the number GitHub evaluated. In a hypothetical example, six out of ten comes to 60%. A developer can count toward that six by editing the code directly or even by replying that the fix will come in another pull request. If a comment lands on a different day from the merge, it counts on the merge day. Source Source
There is a separate count of merged pull requests where no person requested changes after Copilot’s first review. That count can include pull requests opened by bots. GitHub says it usually leaves comments on bot-authored pull requests out of the addressed-rate evaluation. The two measures use different populations, so comparing them as if they shared a denominator would be misleading. Source
The missing editor data flag means GitHub detected Copilot activity but received no usage events from the person’s editor that day. Whether an editor or extension meets the minimum version requirements is a separate signal. A person can appear in both daily counts, and a blank value means the data is unavailable. To see these reports, an organization or enterprise must enable the Copilot usage metrics policy and give the viewer the required permission. Source Source Source
OYOPICK’s Take
Teams trying to learn which problems Copilot helps fix should put the number of comments evaluated next to the addressed rate. Suppose all three addressed security comments were replies promising a later fix. The follow-up changes are where the team can see whether those fixes happened. Using the report to pick what to inspect, then checking the review record for the outcome, gives a team a better basis for deciding what to ask Copilot to review than aiming for a single number. Source Source