Estimating defect rate using GitHub API

#work 4 min.


How do you know if your team of software developers is improving?

What you really want to know about are the outcomes: revenue, customer satisfaction, retention - all that good stuff means clients are choosing your product over competitors. These are hard to pin down to specific departments, let alone individual contributors (ICs).

So instead, many resort to measuring the output: completed tasks, commits, or lines of code (LOC). These can be easily compared between ICs, but unfortunately, they can also be just as easily misinterpreted or manipulated.

If we closed 30 tasks in the previous sprint and 40 tasks in the current one, clearly we’re getting more productive, right? Maybe. Or maybe we just started shipping half-baked code before it’s ready, which resulted in more bug reports. Productive? Technically yes, you are producing more. Desirable? Not exactly.

“When a measure becomes a target, it ceases to be a good measure”. Goodhart’s law

This isn’t even malicious in most cases; it’s just people responding to subtle incentives.

A good tech lead must know the difference between the output and the outcome. The two share a relationship, but rarely a straightforward one. At best, output is the means to an end. At worst, it can be a distraction from actual business goals.

So what do you do instead?

For mature and mostly feature - complete products, I suggest measuring defects instead.

PMs and POs will usually ensure the team keeps shipping features and improvements. As a tech lead, it’s your responsibility to ensure the consistent quality, and absence of defects strikes me as a pretty indicator of that.

You can define defects in a way that fits your team best. It could be bug reports, exceptions, or in my case - hotfixes.

I went with hotfixes because these are the least ambiguous - by definition, it’s something to be fixed urgently. It’s not a perfect definition, as not every defect calls for a hotfix. Still, this should give you a better idea of how much time your team spends firefighting.


What you need to know about our git workflow for the following to make sense:

Pretty vanilla stuff, really.

With that in mind, if we divide the number of hotfix PRs by the number of feature PRs, we’d get a rate of how many defects we’re introducing with each released change (on average):



Set up and authenticate GitHub CLI if you haven’t already:

brew install gh
gh auth login

Navigate to your repository locally and run the following snippet. I’ll explain what it does below.

gh pr list --state merged --search "sort:updated-desc" --limit 1000 --json "mergedAt,baseRefName" -q 'group_by(.mergedAt[:7]) | .[1:] | map({ month : .[0].mergedAt[:7], total_prs: . | length, feature_prs: map(select(.baseRefName == "develop")) | length, hotfix_prs: map(select(.baseRefName == "main")) | length }) | map (. + { defect_rate: (.hotfix_prs / .feature_prs * 100 | round / 100) })'

What this does:

At this point, we pass the JSON to JQ using the –jq flag, which then:

Finally, you will get the output like this:

    "defect_rate": 0.46,
    "feature_prs": 128,
    "hotfix_prs": 59,
    "month": "2023-10",
    "total_prs": 188
    "defect_rate": 0.95,
    "feature_prs": 108,
    "hotfix_prs": 103,
    "month": "2023-11",
    "total_prs": 214
    "defect_rate": 0.38,
    "feature_prs": 138,
    "hotfix_prs": 52,
    "month": "2023-12",
    "total_prs": 191
    "defect_rate": 0.32,
    "feature_prs": 164,
    "hotfix_prs": 53,
    "month": "2024-01",
    "total_prs": 219
    "defect_rate": 0.36,
    "feature_prs": 113,
    "hotfix_prs": 41,
    "month": "2024-02",
    "total_prs": 155

If you want to go a step further and produce a more graphical representation, here’s how you draw a very basic chart using some more JQ:

gh pr list --state merged --search "sort:updated-desc" --limit 1000 --json "mergedAt,baseRefName" -q 'group_by(.mergedAt[:7]) | map({ month : .[0].mergedAt[:7], total_merges: . | length, feature_prs: map(select(.baseRefName == "develop")) | length, hotfix_prs: map(select(.baseRefName == "main")) | length }) | .[1:] | map (. + { defect_rate: (.hotfix_prs / .feature_prs * 100 | round / 100) }) | .[] | "\(.month):\t" + "\(.defect_rate)\t" + "*" * (.defect_rate * 100)'

Will produce:

2023-10:	0.46	**********************************************
2023-11:	0.95	***********************************************************************************************
2023-12:	0.38	**************************************
2024-01:	0.32	********************************
2024-02:	0.36	************************************

So now you know! What you do with all that data is a whole other blog post.

Do you have comments? Tweet X it at me.

Keep up with me

Consider subscribing. There's also an RSS feed if you're into that.

Wanna reach out? Tweet at me or drop me a line: golb [tod] sadat [ta] olleh.

Est. 2011