Measuring pull request standards

This guide is aimed at helping engineering teams implement working agreements and measure pull request (PR) standards.
We will implement working agreements using Port's scorecards and measure PR standards using aggregation properties.
By the end of this guide, you will be able to track teams' performance based on pull requests metrics as shown here:

Overview

Effective collaboration and clear expectations are crucial for high-performing engineering teams.
Working agreements establish shared processes and standards, enhancing teamwork.

Measuring Pull request (PR) standards is essential for assessing code quality, review processes, and team efficiency.

By integrating working agreements with measurable PR metrics, teams can monitor adherence to best practices and continuously improve workflows.

We will discuss how to implement working agreements and measure PR standards using Port.

Metrics

Metrics are essential for assessing how well teams adhere to their working agreements.
They enable teams to track compliance, identify bottlenecks, and drive continuous improvement.

For detailed insights into key metrics like deployment frequency, lead time for changes, and change failure rate,
please refer to our DORA Metrics guide.

Prerequisites

Complete the Port onboarding process.
Access to a GitHub repository that is connected to Port via the onboarding process.

Working agreements

The following working agreements and PR checks have been implemented in our demo environment:

PR Description Cannot be Empty: Ensures that every PR has a description.
PR Has Linked Issue: Verifies that each PR is linked to an issue.
PR Has No Unchecked Checkboxes: Checks that there are no unchecked items in the PR description.
PR Requires Reviewers: Confirms that at least one reviewer is assigned to the PR.
PR Is Linked to a Milestone: Ensures the PR is associated with a milestone.
PR Changed X Files or Less: Validates that the number of changed files is within acceptable limits.
PR Has Been Open for X Days: Monitors how long a PR has been open.
PR Batch Size Calculation: Calculates the batch size of the PR.

These checks are implemented using Port's scorecards.

Implementation

This section will guide you through implementing the working agreements and PR checks in your Port environment. Follow the steps below:

Add the properties to the pull request blueprint.
- Go to the Builder in your Port portal.
- Select the pull request blueprint.
- Click on the ... button in the top right corner, and choose "Edit JSON".
- Add the properties and mapping configurations as described below.
- Save the changes.
Expected JSON
This is the expected JSON definition after adding the properties.

Add the mapping configuration to the data source.
- Go to the Data Sources in your Port portal.
- Select the data source connected to your GitHub repository
- Add the mapping configurations as described below.
- Save the changes.
Add the scorecard definitions to the pull request blueprint.
- Go to the Builder in your Port portal.
- Select the pull request blueprint.
- Click on the Scorecard
- Click on the + New Scorecard button.
- Paste the scorecard definitions as described below.
- Save the changes.

PR Description Cannot be Empty

Scorecard Definition

Scorecard definition (click to expand)

{
  "identifier": "pr_descr_not_empty",
  "title": "PR Description Cannot be Empty",
  "description": "Ensures that the PR description is not empty.",
  "level": "Bronze",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "isNotEmpty",
        "property": "prDescription"
      }
    ]
  }
}

Property

Add the prDescription property to the pull request blueprint:

Add property (click to expand)

"prDescription": {
  "title": "PR Description",
  "type": "string"
}

Mapping Configuration

Map the PR's body field from your data source to the prDescription property:

Mapping config (click to expand)

  - kind: pull-request
    selector:
      query: 'true'
    port:
      entity:
        mappings:
          identifier: .head.repo.name + (.id|tostring)
          title: .title
          blueprint: '"githubPullRequest"'
          properties:
            #other properties
            prDescription: .body

PR Has Linked Issue

Scorecard Definition

Scorecard definition (click to expand)

{
  "identifier": "pr_has_issue_link",
  "title": "PR Has Linked Issue",
  "description": "Ensures that the PR is linked to an issue.",
  "level": "Silver",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "isNotEmpty",
        "property": "issueUrl"
      }
    ]
  }
}

Property

Add the issueUrl property to the pull request blueprint:

Add property (click to expand)

"issueUrl": {
  "title": "Issue URL",
  "type": "string",
  "format": "url"
}

Mapping Configuration

Map the issue URL from your data source to the issueurl property:

Mapping config (click to expand)

  - kind: pull-request
    selector:
      query: 'true'
    port:
      entity:
        mappings:
          identifier: .head.repo.name + (.id|tostring)
          title: .title
          blueprint: '"githubPullRequest"'
          properties:
            #other properties
            issueUrl: .issue_url

PR Has No Unchecked Checkboxes

Scorecard Definition

Scorecard definition (click to expand)

{
  "identifier": "pr_no_unchecked_chk",
  "title": "PR Has No Unchecked Checkboxes",
  "description": "Ensures that there are no unchecked checkboxes in the PR description.",
  "level": "Silver",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "doesNotContains",
        "property": "prDescription",
        "value": "- [ ]"
      }
    ]
  }
}

Property addition

If you haven't added the prDescription property and its relative mapping config, please refer to the PR Description Cannot be Empty section.

PR Requires Reviewers

Scorecard Definition

Scorecard definition (click to expand)

{
  "identifier": "pr_has_reviewers_req",
  "title": "PR Requires Reviewers",
  "description": "Ensures that the PR has at least one reviewer requested.",
  "level": "Bronze",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "isNotEmpty",
        "property": "reviewers"
      }
    ]
  }
}

Property

Add the reviewers property to the pull request blueprint if it doesn't exist:

Add property (click to expand)

"reviewers": {
  "title": "Reviewers",
  "type": "array"
}

Mapping Configuration

Map the list of reviewers from your data source to the reviewers property:

Mapping config (click to expand)

  - kind: pull-request
    selector:
      query: 'true'
    port:
      entity:
        mappings:
          identifier: .head.repo.name + (.id|tostring)
          title: .title
          blueprint: '"githubPullRequest"'
          properties:
            #other properties
            reviewers: '[.requested_reviewers[].login]'

PR Is Linked to a Milestone

Scorecard Definition

Scorecard definition (click to expand)

{
  "identifier": "pr_linked_milestone",
  "title": "PR Is Linked to a Milestone",
  "description": "Ensures that the PR is linked to a milestone.",
  "level": "Gold",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "isNotEmpty",
        "property": "milestone"
      }
    ]
  }
}

Property

Add property (click to expand)

Add the milestone property to the pull request blueprint:

"milestone": {
  "title": "Milestone",
  "type": "object"
}

Mapping Configuration

Mapping config (click to expand)

Map the milestone information from your data source to the milestone property.

  - kind: pull-request
    selector:
      query: 'true'
    port:
      entity:
        mappings:
          identifier: .head.repo.name + (.id|tostring)
          title: .title
          blueprint: '"githubPullRequest"'
          properties:
            #other properties
            milestone: .milestone

PR Changed X Files or Less

Scorecard Definition

This agreement has multiple levels based on the number of files changed.

Scorecard definition (click to expand)

Bronze level scorecard definition (click to expand)

{
  "identifier": "pr_file_limit_bronze",
  "title": "PR Can't Have More than 15 Changed Files",
  "description": "Ensures that the PR does not have more than 15 changed files.",
  "level": "Bronze",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "<=",
        "property": "changedFiles",
        "value": 15
      }
    ]
  }
}

Silver level scorecard definition (click to expand)

{
  "identifier": "pr_file_limit_silver",
  "title": "PR Can't Have More than 10 Changed Files",
  "description": "Ensures that the PR does not have more than 10 changed files.",
  "level": "Silver",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "<=",
        "property": "changedFiles",
        "value": 10
      }
    ]
  }
}

Gold level scorecard definition (click to expand)

{
  "identifier": "pr_file_limit_gold",
  "title": "PR Can't Have More than 5 Changed Files",
  "description": "Ensures that the PR does not have more than 5 changed files.",
  "level": "Gold",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "<=",
        "property": "changedFiles",
        "value": 5
      }
    ]
  }
}

value adjustment

You can adjust the value based on your team's requirements.

Property

Add the changedFiles property to the pull request blueprint:

Add property (click to expand)

    "changedFiles": {
      "title": "Changed Files",
      "type": "number"
    }

Mapping Configuration

Map the number of changed_files from the data source to the changedFiles property:

Mapping config (click to expand)

  - kind: pull-request
    selector:
      query: 'true'
    port:
      entity:
        mappings:
          identifier: .head.repo.name + (.id|tostring)
          title: .title
          blueprint: '"githubPullRequest"'
          properties:
            #other properties
            changedFiles: .changed_files

PR Has Been Open for X Days

Scorecard Definition

Scorecard definition (click to expand)

{
  "identifier": "pr_open_for_less_than_10_days",
  "title": "PR Has Been Open for Less Than 10 Days",
  "description": "Ensures that the PR has not been open for more than 10 days.",
  "level": "Silver",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "<=",
        "property": "days_old",
        "value": 10
      },
      {
        "operator": "=",
        "property": "status",
        "value": "open"
      }
    ]
  }
}

Property

Add a calculation property days_old to compute how many days the PR has been open:

Add property (click to expand)

"days_old": {
  "title": "Days Old",
  "icon": "DefaultProperty",
  "calculation": "(now / 86400) - (.properties.updatedAt | capture(\"(?<date>\\\\d{4}-\\\\d{2}-\\\\d{2})\") | .date | strptime(\"%Y-%m-%d\") | mktime / 86400) | floor",
  "type": "number"
}

Required properties

Ensure that createdAt and mergedAt properties are correctly mapped from your data source.

PR Batch Size Calculation

Scorecard Definition

This agreement has levels based on the batch size.

Scorecard definition (click to expand)

Bronze level scorecard definition (click to expand)

{
  "identifier": "pr_batch_size_bronze",
  "title": "PR Cannot Have Large or Gigantic Batch Size",
  "description": "Ensures that the PR does not have a Large or Gigantic batch size.",
  "level": "Bronze",
  "query": {
    "combinator": "or",
    "conditions": [
      {
        "operator": "!=",
        "property": "batchSize",
        "value": "Large"
      },
      {
        "operator": "!=",
        "property": "batchSize",
        "value": "Gigantic"
      }
    ]
  }
}

Silver level scorecard definition (click to expand)

{
  "identifier": "pr_batch_size_silver",
  "title": "PR Has Medium Batch Size",
  "description": "Ensures that the PR has a Medium batch size.",
  "level": "Silver",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "=",
        "property": "batchSize",
        "value": "Medium"
      }
    ]
  }
}

Gold level scorecard definition (click to expand)

{
  "identifier": "pr_batch_size_gold",
  "title": "PR Has Small Batch Size",
  "description": "Ensures that the PR has a Small batch size.",
  "level": "Gold",
  "query": {
    "combinator": "and",
    "conditions": [
      {
        "operator": "=",
        "property": "batchSize",
        "value": "Small"
      }
    ]
  }
}

Property

Add a property batchSize to categorize the PR's batch size:

Add property (click to expand)

"batchSize": {
  "title": "Batch Size",
  "type": "string",
  "enum": [
    "Small",
    "Medium",
    "Large",
    "Gigantic"
  ],
  "enumColors": {
    "Small": "lightGray",
    "Medium": "orange",
    "Large": "yellow",
    "Gigantic": "red"
  }
}

Mapping Configuration

Map the PR's additions, deletions, and changedFiles properties to the batchSize property:

Mapping config (click to expand)

    - kind: pull-request
      selector:
        query: 'true'
        port:
        entity:
            mappings:
            identifier: .head.repo.name + (.id|tostring)
            title: .title
            blueprint: '"githubPullRequest"'
            properties:
                #other properties
            batchSize: >
             if (.commits <= 3 and (.additions + .deletions) <= 50 and
             .changed_files <= 3) then "Small" elif (.commits <= 10 and
             (.additions + .deletions) <= 200 and .changed_files <= 7) then
             "Medium" elif (.commits <= 20 and (.additions + .deletions) <= 500
             and .changed_files <= 15) then "Large" else "Gigantic" end

Batch size calculation

You can adjust the thresholds based on your team's requirements.

Add total LOC changed property

Add the calculation property totalLocChanged on the pull request blueprint:

Calculation property (click to expand)

"totalLocChanged": {
  "title": "Total LOC Changed",
  "type": "number",
  "calculation": ".properties.additions + .properties.deletions"
}

Pull request metrics aggregation

To measure PR standards effectively, add aggregation properties on the service blueprint. This will allow us to capture important metrics such as:

PR Average Duration (Service Level)
PRs Opened (Service & Organization Level)
PRs Merged (Service & Organization Level)
Average Commits per PR (Service & Organization Level)
Average Lines of Code (LOC) Changed (Service & Organization Level)

Aggregation on higher hierarchy

To aggregate on an organization level, apply the same settings to a higher hierarchy.
If you want to see it on a team level, you need to make sure services are related to teams and add the same aggregations to team

Adding Aggregation to Blueprints

To add aggregation properties to your blueprints, follow these steps:

Go to the Builder in your Port portal.
Locate and select your blueprint.
Click the {...} button in the top right corner, and choose Edit JSON.
Insert the respective aggregation or calculation properties under the aggregationProperties or calculationProperties section in the blueprint's JSON schema.
Save your changes to apply the new aggregation configuration.

Average PR Duration
PRs Opened
PRs Merged
Average Commits per PR
Average LOC Changed

Add the following aggregation property to the service blueprint:

Aggregation property (click to expand)

"averagePrDuration": {
  "title": "Average PR Duration",
  "type": "number",
  "target": "githubPullRequest",
  "calculationSpec": {
    "func": "average",
    "averageOf": "week",
    "calculationBy": "property",
    "property": "days_old"
  },
  "query": {
    "combinator": "and",
    "rules": [
      {
        "property": "mergedAt",
        "operator": "isNotEmpty"
      }
    ]
  }
}

Add the following aggregation property to the service blueprint:

Aggregation property (click to expand)

"openPrs": {
  "title": "Open PRs",
  "type": "number",
  "target": "githubPullRequest",
  "calculationSpec": {
    "func": "count",
    "calculationBy": "entities"
  },
  "query": {
    "combinator": "and",
    "rules": [
      {
        "property": "status",
        "operator": "=",
        "value": "open"
      }
    ]
  }
}

Add the following aggregation property to the service blueprint:

Aggregation property (click to expand)

"mergedPrs": {
  "title": "Merged PRs",
  "type": "number",
  "target": "githubPullRequest",
  "calculationSpec": {
    "func": "count",
    "calculationBy": "entities"
  },
  "query": {
    "combinator": "and",
    "rules": [
      {
        "property": "status",
        "operator": "=",
        "value": "merged"
      }
    ]
  }
}

Add the following aggregation property to the service blueprint:

Aggregation property (click to expand)

"averageCommitsPerPr": {
  "title": "Average Commits per PR",
  "type": "number",
  "target": "githubPullRequest",
  "calculationSpec": {
    "averageOf": "week",
    "func": "average",
    "calculationBy": "property",
    "property": "commits",
    "measureTimeBy": "$createdAt"
  }
}

Add the following aggregation property to the service and organization blueprints:

Aggregation property (click to expand)

"averagePrLinesOfCode": {
  "title": "Average PR Lines of Code",
  "type": "number",
  "target": "githubPullRequest",
  "calculationSpec": {
    "func": "average",
    "averageOf": "week",
    "calculationBy": "property",
    "property": "totalLocChanged"
  }
}

By implementing these aggregation properties, you can effectively measure and monitor PR standards at both the service and organization levels.

Visualization

By leveraging Port's Dashboards, you can create custom dashboards to track the pr metrics and monitor your team's performance over time.

Dashboard setup

Go to your software catalog.
Click on the + New button in the left sidebar.
Select New dashboard.
Name the dashboard (PR Metrics), choose an icon if desired, and click Create.

This will create a new empty dashboard. Let's get ready-to-add widgets

Adding widgets

Average Pr Merged per month (click to expand)

Click + Widget and select Number Chart.
Title: Average Pr's Merged per month, (add the GitPullRequest icon).
Select Aggregate by property and choose Service as the Blueprint.
Choose Merged PRs as the Property.
Select average for the Function and choose month for Average of.
Select Created At for Measure time by.
Select custom as the Unit and input Pull Request merged per month as the Custom unit.
Click Save.

Total Pr's Merged (click to expand)

Click + Widget and select Number Chart.
Title: Total Pr's Merged , (add the GitPullRequest icon).
Select Aggregate by property and choose Service as the Blueprint.
Choose Merged PRs as the Property.
Select sum for the Function.
Click Save.

Mean Time to Merge (Days) (click to expand)

Click + Widget and select Number Chart.
Title: Mean Time to Merge (Days), (add the Merge icon).
Select Display single property and choose Service the Entity.
Choose Averge Time to Merge as the Property.
Click Save.

Total Weekly Pr commits (click to expand)

Click + Widget and select Number Chart.
Title: Total Weekly Pr commits, (add the GitPullRequest icon).
Select Aggregate by property and choose Service as the Blueprint.
Choose Averge Commits per PRs as the Property.
Select average for the Function and choose week for Average of.
Select Created At for Measure time by.
Select custom as the Unit and input Weekly as the Custom unit.
Click Save.

Service Scorecard Performance by Team (click to expand)

Click + Widget and select Table.
Title the widget Team Service Scorecard Performance.
Choose the Service blueprint.
Click Save to add the widget to the dashboard.
Click on the ... button in the top right corner of the table and select Customize table.
In the top right corner of the table, click on Manage Properties and add the following properties:
- Title: The name of each service.
- PR Metrics: PR Metrics scorecard aggregation on service.
- Owning Team: The team that owns the service.
Click on the Group by any Column on the top right conner and select Owning Team.
Click on the save icon in the top right corner of the widget to save the customized table.

Notification automation

Add this automation feature to notify a Slack channel when a scorecard value changes

Automation for sending Slack notifications on scorecard value change (click to expand)

{
  "identifier": "scorecardValueChanged",
  "title": "Notify Slack on Scorecard Value Change",
  "icon": "Slack",
  "description": "Sends a Slack message when the scorecard value changes.",
  "trigger": {
    "type": "automation",
    "event": {
      "type": "ENTITY_UPDATED",
      "blueprintIdentifier": "githubPullRequest"
    },
    "condition": {
      "type": "JQ",
      "expressions": [
        ".diff.after.scorecardsStats != .diff.before.scorecardsStats"
      ],
      "combinator": "and"
    }
  },
  "invocationMethod": {
    "type": "WEBHOOK",
    "url": "{{ .event.diff.after.properties.serviceSlackUrl }}",
    "agent": false,
    "synchronized": true,
    "body": {
      "channel": "{{ .event.diff.after.properties.serviceSlackChannel }}",
      "text": "*Scorecard value changed for PR <{{ .event.diff.after.properties.link }}|{{ .event.diff.after.title }}>*\n\n*Title:* {{ .event.diff.after.title }}\n\n*Old Scorecard Value:* {{ .event.diff.before.scorecardsStats }}\n\n*New Scorecard Value:* {{ .event.diff.after.scorecardsStats }}\n\n*Link:* <{{ .event.diff.after.properties.link }}|View PR>\n\n"
    }
  },
  "publish": true
}

Adding Automations

To add new automations, follow the steps outlined in the Automation Setup section of this guide.
and remember to set the serviceSlackUrl and serviceSlackChannel properties on the service blueprint.

Overview​

Prerequisites​

Working agreements​

Implementation​

PR Description Cannot be Empty​

Scorecard Definition

Property

Mapping Configuration

PR Has Linked Issue​

Scorecard Definition

Property

Mapping Configuration

PR Has No Unchecked Checkboxes​

Scorecard Definition

PR Requires Reviewers​

Scorecard Definition

Property

Mapping Configuration

PR Is Linked to a Milestone​

Scorecard Definition

Property

Mapping Configuration

PR Changed X Files or Less​

Scorecard Definition

Property

Mapping Configuration

PR Has Been Open for X Days​

Scorecard Definition

Property

PR Batch Size Calculation​

Scorecard Definition

Property

Mapping Configuration

Add total LOC changed property​

Pull request metrics aggregation​

Visualization​

Dashboard setup​

Adding widgets​

Notification automation​

Overview

Prerequisites

Working agreements

Implementation

PR Description Cannot be Empty

PR Has Linked Issue

PR Has No Unchecked Checkboxes

PR Requires Reviewers

PR Is Linked to a Milestone

PR Changed X Files or Less

PR Has Been Open for X Days

PR Batch Size Calculation

Add total LOC changed property

Pull request metrics aggregation

Visualization

Dashboard setup

Adding widgets

Notification automation