Skip to main content

Check out Port for yourself 

Datadog

Port's Datadog integration allows you to model Datadog resources in Port and ingest data into them.

Overview​

This integration allows you to:

  • Map and organize your desired Datadog resources and their metadata in Port (see supported resources below).
  • Watch for Datadog object changes (create/update/delete) in real-time, and automatically apply the changes to your software catalog.

Supported Resources​

Setup​

Choose one of the following installation methods:

Using this installation option means that the integration will be hosted by Port, with a customizable resync interval to ingest data into Port.

Live event support

Currently, live events are not supported for integrations hosted by Port.
Resyncs will be performed periodically (with a configurable interval), or manually triggered by you via Port's UI.

Therefore, real-time events (including GitOps) will not be ingested into Port immediately.
Support for live events is WIP and will be supported in the near future.

Self-hosted installation

Alternatively, you can install the integration using the Real-time (self-hosted) method to update Port in real time using webhooks.

Installation

To install, follow these steps:

  1. Go to the Data sources page of your portal.

  2. Click on the + Data source button in the top-right corner.

  3. Click on the relevant integration in the list.

  4. Under Select your installation method, choose Hosted by Port.

  5. Configure the integration settings and application settings as you wish (see below for details).

Application settings

Every integration hosted by Port has the following customizable application settings, which are configurable after installation:

  • Resync interval: The frequency at which Port will ingest data from the integration. There are various options available, ranging from every 1 hour to once a day.

  • Send raw data examples: A boolean toggle (enabled by default). If enabled, raw data examples will be sent from the integration to Port. These examples are used when testing your mapping configuration, they allow you to run your jq expressions against real data and see the results.

Integration settings

Every integration has its own tool-specific settings, under the Integration settings section.
Each of these settings has an ⓘ icon next to it, which you can hover over to see a description of the setting.

Port secrets

Some integration settings require sensitive pieces of data, such as tokens.
For these settings, Port secrets will be used, ensuring that your sensitive data is encrypted and secure.

When filling in such a setting, its value will be obscured (shown as ••••••••).
For each such setting, Port will automatically create a secret in your organization.

To see all secrets in your organization, follow these steps.

Port source IP addresses

When using this installation method, Port will make outbound calls to your 3rd-party applications from static IP addresses.
You may need to add these addresses to your allowlist, in order to allow Port to interact with the integrated service:

54.73.167.226  
63.33.143.237  
54.76.185.219

Configuration​

Port integrations use a YAML mapping block to ingest data from the third-party api into Port.

The mapping makes use of the JQ JSON processor to select, modify, concatenate, transform and perform other operations on existing fields and values from the integration API.

Examples​

To view and test the integration's mapping against examples of the third-party API responses, use the jq playground in your data sources page. Find the integration in the list of data sources and click on it to open the playground.

Examples of blueprints and the relevant integration configurations can be found on the datadog examples page

Let's Test It​

This section includes a sample response data from Datadog. In addition, it includes the entity created from the resync event based on the Ocean configuration provided in the previous section.

Payload​

Here is an example of the payload structure from Datadog:

Monitor response data
{
  "id":15173866,
  "org_id":1000147697,
  "type":"query alert",
  "name":"A change @webhook-PORT",
  "message":"A change has happened",
  "tags":[
    "app:webserver"
  ],
  "query":"change(avg(last_5m),last_1h):avg:datadog.agent.running{local} by {version,host} > 40",
  "options":{
    "thresholds":{
        "critical":40.0,
        "warning":30.0
    },
    "notify_audit":false,
    "include_tags":true,
    "new_group_delay":60,
    "notify_no_data":false,
    "timeout_h":0,
    "silenced":{
    }
  },
  "multi":true,
  "created_at":1706707941000,
  "created":"2024-01-31T13:32:21.270116+00:00",
  "modified":"2024-02-02T16:31:40.516062+00:00",
  "deleted":"None"[
    "REDACTED"
  ],
  "restricted_roles":"None"[
    "REDACTED"
  ],
  "priority":5,
  "overall_state_modified":"2024-03-08T20:52:46+00:00",
  "overall_state":"No Data",
  "creator":{
    "name":"John Doe",
    "email":"john.doe@gmail.com",
    "handle":"john.doe@gmail.com",
    "id":1001199545
  },
  "matching_downtimes":[
  ]
}
Service response data
{
   "type":"service-definition",
   "id":"04fbab48-a233-4592-8c53-d1bfe282e6c3",
   "attributes":{
      "meta":{
         "last-modified-time":"2024-05-29T10:31:06.833444245Z",
         "github-html-url":"",
         "ingestion-source":"api",
         "origin":"unknown",
         "origin-detail":"",
         "warnings":[
            {
               "keyword-location":"/properties/integrations/properties/opsgenie/properties/service-url/pattern",
               "instance-location":"/integrations/opsgenie/service-url",
               "message":"does not match pattern '^(https?://)?[a-zA-Z\\\\d_\\\\-.]+\\\\.opsgenie\\\\.com/service/([a-zA-Z\\\\d_\\\\-]+)/?$'"
            },
            {
               "keyword-location":"/properties/integrations/properties/pagerduty/properties/service-url/pattern",
               "instance-location":"/integrations/pagerduty/service-url",
               "message":"does not match pattern '^(https?://)?[a-zA-Z\\\\d_\\\\-.]+\\\\.pagerduty\\\\.com/service-directory/(P[a-zA-Z\\\\d_\\\\-]+)/?$'"
            }
         ],
         "ingested-schema-version":"v2.1"
      },
      "schema":{
         "schema-version":"v2.2",
         "dd-service":"inventory-management",
         "team":"Inventory Management Team",
         "application":"Inventory System",
         "tier":"Tier 1",
         "description":"Service for managing product inventory and stock levels.",
         "lifecycle":"production",
         "contacts":[
            {
               "name":"Inventory Team",
               "type":"email",
               "contact":"inventory-team@example.com"
            },
            {
               "name":"Warehouse Support",
               "type":"email",
               "contact":"warehouse-support@example.com"
            }
         ],
         "links":[
            {
               "name":"Repository",
               "type":"repo",
               "provider":"GitHub",
               "url":"https://github.com/example/inventory-service"
            },
            {
               "name":"Runbook",
               "type":"runbook",
               "provider":"Confluence",
               "url":"https://wiki.example.com/runbooks/inventory-service"
            }
         ],
         "tags":[
            "inventory",
            "stock"
         ],
         "integrations":{
            "pagerduty":{
               "service-url":"https://pagerduty.com/services/inventory"
            },
            "opsgenie":{
               "service-url":"https://opsgenie.com/services/inventory",
               "region":"US"
            }
         },
         "extensions":{
            "qui_6":{
               
            }
         }
      }
    }
  }
}
SLO response data
{
   "id":"b6869ae6189d59baa421feb8b437fe9e",
   "name":"Availability SLO for shopping-cart service",
   "tags":[
      "service:shopping-cart",
      "env:none"
   ],
   "monitor_tags":[
      
   ],
   "thresholds":[
      {
         "timeframe":"7d",
         "target":99.9,
         "target_display":"99.9"
      }
   ],
   "type":"monitor",
   "type_id":0,
   "description":"This SLO tracks the availability of the shopping-cart service. Availability is measured as the number of successful requests divided by the number of total requests for the service",
   "timeframe":"7d",
   "target_threshold":99.9,
   "monitor_ids":[
      15173866,
      15216083,
      15254771
   ],
   "creator":{
      "name":"John Doe",
      "handle":"john.doe@gmail.com",
      "email":"john.doe@gmail.com"
   },
   "created_at":1707215619,
   "modified_at":1707215619
}
SLO history response data
{
  "thresholds": {
    "7d": {
      "timeframe": "7d",
      "target": 99,
      "target_display": "99."
    }
  },
  "from_ts": 1719254776,
  "to_ts": 1719859576,
  "type": "monitor",
  "type_id": 0,
  "slo": {
    "id": "5ec82408e83c54b4b5b2574ee428a26c",
    "name": "Host {{host.name}} with IP {{host.ip}} is not having enough memory",
    "tags": [
      "p69hx03",
      "pages-laptop"
    ],
    "monitor_tags": [],
    "thresholds": [
      {
        "timeframe": "7d",
        "target": 99,
        "target_display": "99."
      }
    ],
    "type": "monitor",
    "type_id": 0,
    "description": "Testing SLOs from DataDog",
    "timeframe": "7d",
    "target_threshold": 99,
    "monitor_ids": [
      147793
    ],
    "creator": {
      "name": "John Doe",
      "handle": "janesmith@gmail.com",
      "email": "janesmith@gmail.com"
    },
    "created_at": 1683878238,
    "modified_at": 1684773765
  },
  "overall": {
    "name": "Host {{host.name}} with IP {{host.ip}} is not having enough memory",
    "preview": false,
    "monitor_type": "query alert",
    "monitor_modified": 1683815332,
    "errors": null,
    "span_precision": 2,
    "history": [
      [
        1714596313,
        1
      ]
    ],
    "uptime": 3,
    "sli_value": 10,
    "precision": {
      "custom": 2,
      "7d": 2
    },
    "corrections": [],
    "state": "breached"
  }
}
Service metric response data
Response Enrichment

The Datadog response is enriched with a variety of metadata fields, including:

  • __service: The name or identifier of the service generating the data.
  • __query_id: A unique identifier for the query that generated the data.
  • __query: The original query used to retrieve the data.
  • __env: The environment associated with the data (e.g., production, staging).

This enrichment significantly enhances the usability of the Datadog response by providing valuable context and facilitating easier analysis and troubleshooting.

{
  "status": "ok",
  "res_type": "time_series",
  "resp_version": 1,
  "query": "avg:system.mem.used{service:inventory-management,env:staging}",
  "from_date": 1723796537000,
  "to_date": 1723797137000,
  "series": [
    {
      "unit": [
        {
          "family": "bytes",
          "id": 2,
          "name": "byte",
          "short_name": "B",
          "plural": "bytes",
          "scale_factor": 1.0
        }
      ],
      "query_index": 0,
      "aggr": "avg",
      "metric": "system.mem.used",
      "tag_set": [],
      "expression": "avg:system.mem.used{env:staging,service:inventory-management}",
      "scope": "env:staging,service:inventory-management",
      "interval": 2,
      "length": 39,
      "start": 1723796546000,
      "end": 1723797117000,
      "pointlist": [
        [1723796546000.0, 528986112.0],
        [1723796562000.0, 531886080.0],
        [1723796576000.0, 528867328.0],
        [1723796592000.0, 522272768.0],
        [1723796606000.0, 533704704.0],
        [1723796846000.0, 533028864.0],
        [1723796862000.0, 527417344.0],
        [1723796876000.0, 531513344.0],
        [1723796892000.0, 533577728.0],
        [1723796906000.0, 533471232.0],
        [1723796922000.0, 528125952.0],
        [1723796936000.0, 530542592.0],
        [1723796952000.0, 530767872.0],
        [1723796966000.0, 526966784.0],
        [1723796982000.0, 528560128.0],
        [1723796996000.0, 530792448.0],
        [1723797012000.0, 527384576.0],
        [1723797026000.0, 529534976.0],
        [1723797042000.0, 521650176.0],
        [1723797056000.0, 531001344.0],
        [1723797072000.0, 525955072.0],
        [1723797086000.0, 529469440.0],
        [1723797102000.0, 532279296.0],
        [1723797116000.0, 526979072.0]
      ],
      "display_name": "system.mem.used",
      "attributes": {}
    }
  ],
  "values": [],
  "times": [],
  "message": "",
  "group_by": [],
  "__service": "inventory-management",
  "__query_id": "avg:system.mem.used/service:inventory-management/env:staging",
  "__query": "avg:system.mem.used",
  "__env": "staging"
}

Mapping Result​

The combination of the sample payload and the Ocean configuration generates the following Port entity:

Monitor entity in Port
{
  "identifier": "15173866",
  "title": "A change @webhook-PORT",
  "icon": "Datadog",
  "blueprint": "datadogMonitor",
  "team": [],
  "properties": {
    "tags": [
      "app:webserver"
    ],
    "overallState": "No Data",
    "priority": "5",
    "createdAt": "2024-01-31T13:32:21.270116+00:00",
    "updatedAt": "2024-02-02T16:31:40.516062+00:00",
    "createdBy": "john.doe@gmail.com",
    "monitorType": "query alert"
  },
  "relations": {},
  "createdAt": "2024-05-29T09:43:34.750Z",
  "createdBy": "<port-client-id>",
  "updatedAt": "2024-05-29T09:43:34.750Z",
  "updatedBy": "<port-client-id>"
}
Service entity in Port
{
  "identifier": "inventory-management",
  "title": "inventory-management",
  "icon": "Datadog",
  "blueprint": "datadogService",
  "team": [],
  "properties": {
    "owners": [
      "inventory-team@example.com",
      "warehouse-support@example.com"
    ],
    "links": [
      "https://github.com/example/inventory-service",
      "https://wiki.example.com/runbooks/inventory-service"
    ],
    "description": "Service for managing product inventory and stock levels.",
    "tags": [
      "inventory",
      "stock"
    ],
    "application": "Inventory System"
  },
  "relations": {},
  "createdAt": "2024-05-29T10:31:44.283Z",
  "createdBy": "<port-client-id>",
  "updatedAt": "2024-05-29T10:31:44.283Z",
  "updatedBy": "<port-client-id>"
}
SLO entity in Port
{
  "identifier": "b6869ae6189d59baa421feb8b437fe9e",
  "title": "Availability SLO for shopping-cart service",
  "icon": "Datadog",
  "blueprint": "datadogSlo",
  "team": [],
  "properties": {
    "description": "This SLO tracks the availability of the shopping-cart service. Availability is measured as the number of successful requests divided by the number of total requests for the service",
    "updatedAt": "2024-02-06T10:33:39Z",
    "createdBy": "ahosea15@gmail.com",
    "sloType": "monitor",
    "targetThreshold": "99.9",
    "tags": [
      "service:shopping-cart",
      "env:none"
    ],
    "createdAt": "2024-02-06T10:33:39Z"
  },
  "relations": {
    "monitors": [
      "15173866",
      "15216083",
      "15254771"
    ],
    "services": [
      "shopping-cart"
    ]
  },
  "createdAt": "2024-05-29T09:43:51.946Z",
  "createdBy": "<port-client-id>",
  "updatedAt": "2024-05-29T12:02:01.559Z",
  "updatedBy": "<port-client-id>"
}
SLO history entity in Port
{
  "identifier": "5ec82408e83c54b4b5b2574ee428a26c",
  "title": "Host {{host.name}} with IP {{host.ip}} is not having enough memory",
  "icon": "Datadog",
  "blueprint": "datadogSloHistory",
  "team": [],
  "properties": {
    "sampling_end_date": "2024-07-01T18:46:16Z",
    "sliValue": 10,
    "sampling_start_date": "2024-06-24T18:46:16Z"
  },
  "relations": {
    "slo": "5ec82408e83c54b4b5b2574ee428a26c"
  },
  "createdAt": "2024-07-01T09:43:51.946Z",
  "createdBy": "<port-client-id>",
  "updatedAt": "2024-07-01T12:02:01.559Z",
  "updatedBy": "<port-client-id>"
}
Service metric entity in Port
{
  "identifier": "avg:system.disk.used/service:inventory-management/env:prod",
  "title": "avg:system.disk.used{service:inventory-management,env:prod}",
  "icon": null,
  "blueprint": "datadogServiceMetric",
  "team": [],
  "properties": {
    "query": "avg:system.disk.used",
    "series": [],
    "res_type": "time_series",
    "from_date": "2024-08-16T07:32:00Z",
    "to_date": "2024-08-16T08:02:00Z",
    "env": "prod"
  },
  "relations": {
    "service": "inventory-management"
  },
  "createdAt": "2024-08-15T15:54:36.638Z",
  "createdBy": "<port-client-id>",
  "updatedAt": "2024-08-16T08:02:02.399Z",
  "updatedBy": "<port-client-id>"
}

Relevant Guides​

For relevant guides and examples, see the guides section.

Alternative installation via webhook​

While the Ocean integration described above is the recommended installation method, you may prefer to use a webhook to ingest alerts and monitor data from Datadog. If so, use the following instructions:

Note that when using this method, data will be ingested into Port only when the webhook is triggered.

Webhook installation (click to expand)

Port configuration

Create the following blueprint definitions:

Datadog microservice blueprint
{
  "identifier": "microservice",
  "title": "Microservice",
  "icon": "Service",
  "schema": {
    "properties": {
      "description": {
        "title": "Description",
        "type": "string"
      }
    },
    "required": []
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "relations": {}
}
Datadog alert/monitor blueprint
{
  "identifier": "datadogAlert",
  "description": "This blueprint represents a Datadog monitor/alert in our software catalog",
  "title": "Datadog Alert",
  "icon": "Datadog",
  "schema": {
    "properties": {
      "url": {
        "type": "string",
        "format": "url",
        "title": "Event URL"
      },
      "message": {
        "type": "string",
        "title": "Details"
      },
      "eventType": {
        "type": "string",
        "title": "Event Type"
      },
      "priority": {
        "type": "string",
        "title": "Metric Priority"
      },
      "creator": {
        "type": "string",
        "title": "Creator"
      },
      "alertMetric": {
        "type": "string",
        "title": "Alert Metric"
      },
      "alertType": {
        "type": "string",
        "title": "Alert Type",
        "enum": ["error", "warning", "success", "info"]
      },
      "tags": {
        "type": "array",
        "items": {
          "type": "string"
        },
        "title": "Tags"
      }
    },
    "required": []
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "relations": {
    "microservice": {
      "title": "Services",
      "target": "microservice",
      "required": false,
      "many": false
    }
  }
}

Create the following webhook configuration using Port UI:

Datadog webhook configuration

  1. Basic details tab - fill the following details:

    1. Title : Datadog Alert Mapper;
    2. Identifier : datadog_alert_mapper;
    3. Description : A webhook configuration for alerts/monitors events from Datadog;
    4. Icon : Datadog;

  2. Integration configuration tab - fill the following JQ mapping:

    [
      {
        "blueprint": "datadogAlert",
        "entity": {
          "identifier": ".body.alert_id | tostring",
          "title": ".body.title",
          "properties": {
            "url": ".body.event_url",
            "message": ".body.message",
            "eventType": ".body.event_type",
            "priority": ".body.priority",
            "creator": ".body.creator",
            "alertMetric": ".body.alert_metric",
            "alertType": ".body.alert_type",
            "tags": ".body.tags | split(\", \")"
          },
          "relations": {
            "microservice": ".body.service"
          }
        }
      }
    ]

  3. Click Save at the bottom of the page.

note

The webhook configuration's relation mapping will function properly only when the identifiers of the Port microservice entities match the names of the services or hosts in your Datadog.

Create a webhook in Datadog

  1. Log in to Datadog with your credentials.
  2. Click on Integrations at the left sidebar of the page.
  3. Search for Webhooks in the search box and select it.
  4. Go to the Configuration tab and follow the installation instructions.
  5. Click on New.
  6. Input the following details:

    1. Name - use a meaningful name such as Port_Webhook.

    2. URL - enter the value of the url key you received after creating the webhook configuration.

    3. Payload - When an alert is triggered on your monitors, this payload will be sent to the webhook URL. You can enter this JSON placeholder in the textbox:

      {
        "id": "$ID",
        "message": "$TEXT_ONLY_MSG",
        "priority": "$PRIORITY",
        "last_updated": "$LAST_UPDATED",
        "event_type": "$EVENT_TYPE",
        "event_url": "$LINK",
        "service": "$HOSTNAME",
        "creator": "$USER",
        "title": "$EVENT_TITLE",
        "date": "$DATE",
        "org_id": "$ORG_ID",
        "org_name": "$ORG_NAME",
        "alert_id": "$ALERT_ID",
        "alert_metric": "$ALERT_METRIC",
        "alert_status": "$ALERT_STATUS",
        "alert_title": "$ALERT_TITLE",
        "alert_type": "$ALERT_TYPE",
        "tags": "$TAGS"
      }

    4. Custom Headers - configure any custom HTTP header to be added to the webhook event. The format for the header should be in JSON.

  7. Click Save at the bottom of the page.
tip

To view the different payloads and structure of the events in Datadog webhooks, look here.

Done! Any problem detected on your Datadog instance will trigger a webhook event. Port will parse the events according to the mapping and update the catalog entities accordingly.

Let's Test It

This section includes a sample response data from Datadog. In addition, it includes the entity created from the resync event based on the Ocean configuration provided in the previous section.

Payload

Here is an example of the payload structure from Datadog:

Webhook response data (Click to expand)
{
  "id": "1234567890",
  "message": "This is a test message",
  "priority": "normal",
  "last_updated": "2022-01-01T00:00:00+00:00",
  "event_type": "triggered",
  "event_url": "https://app.datadoghq.com/event/jump_to?event_id=1234567890",
  "service": "my-service",
  "creator": "rudy",
  "title": "[Triggered] [Memory Alert]",
  "date": "1406662672000",
  "org_id": "123456",
  "org_name": "my-org",
  "alert_id": "1234567890",
  "alert_metric": "system.load.1",
  "alert_status": "system.load.1 over host:my-host was > 0 at least once during the last 1m",
  "alert_title": "[Triggered on {host:ip-012345}] Host is Down",
  "alert_type": "error",
  "tags": "monitor, name:myService, role:computing-node"
}

Mapping Result

The combination of the sample payload and the Ocean configuration generates the following Port entity:

Alert entity in Port (Click to expand)
{
  "identifier": "1234567890",
  "title": "[Triggered] [Memory Alert]",
  "blueprint": "datadogAlert",
  "team": [],
  "icon": "Datadog",
  "properties": {
    "url": "https://app.datadoghq.com/event/jump_to?event_id=1234567890",
    "message": "This is a test message",
    "eventType": "triggered",
    "priority": "normal",
    "creator": "rudy",
    "alertMetric": "system.load.1",
    "alertType": "error",
    "tags": "monitor, name:myService, role:computing-node"
  },
  "relations": {
    "microservice": "my-service"
  },
  "createdAt": "2024-2-6T09:30:57.924Z",
  "createdBy": "hBx3VFZjqgLPEoQLp7POx5XaoB0cgsxW",
  "updatedAt": "2024-2-6T11:49:20.881Z",
  "updatedBy": "hBx3VFZjqgLPEoQLp7POx5XaoB0cgsxW"
}

Ingest service level objectives (SLOs)

This guide will walk you through the steps to ingest Datadog SLOs into Port. By following these steps, you will be able to create a blueprint for a microservice entity in Port, representing a service in your Datadog account. Furthermore, you will establish a relation between this service and the datadogSLO blueprint, allowing the ingestion of all defined SLOs from your Datadog account.

The provided example demonstrates how to pull data from Datadog's REST API at scheduled intervals using GitLab Pipelines and report the data to Port.

Ingest service dependency from your APM

In this example, you will create a service blueprint that ingests all services and their related dependencies in your Datadog APM using REST API. You will then add some shell script to create new entities in Port every time GitLab CI is triggered by a schedule.

Ingest service catalog

In this example, you will create a datadogServiceCatalog blueprint that ingests all service catalogs from your Datadog account. You will then add some python script to make API calls to Datadog REST API and fetch data for your account.