Skip to main content

OpenCost

Our OpenCost integration allows you to import cost and cloudCost from your OpenCost instance into Port, according to your mapping and definition.

Common use cases​

  • Map your monitored Kubernetes resources and cost allocations in OpenCost.
  • Map your cost allocations from different cloud providers from OpenCost.

Prerequisites​

To install the integration, you need a Kubernetes cluster that the integration's container chart will be deployed to.

Please make sure that you have kubectl and helm installed on your machine, and that your kubectl CLI is connected to the Kubernetes cluster where you plan to install the integration.

Troubleshooting

If you are having trouble installing this integration, please refer to these troubleshooting steps.

Installation​

Choose one of the following installation methods:

Using this installation option means that the integration will be able to update Port in real time.

This table summarizes the available parameters for the installation. Set them as you wish in the script below, then copy it and run it in your terminal:

ParameterDescriptionRequired
port.clientIdYour port client id✅
port.clientSecretYour port client secret✅
port.baseUrlYour port base url, relevant only if not using the default port app❌
integration.identifierChange the identifier to describe your integration✅
integration.typeThe integration type✅
integration.eventListener.typeThe event listener type✅
integration.config.opencostHostThe Opencost server URL✅
scheduledResyncIntervalThe number of minutes between each resync❌
initializePortResourcesDefault true, When set to true the integration will create default blueprints and the port App config Mapping❌

To install the integration using Helm, run the following command:

helm repo add --force-update port-labs https://port-labs.github.io/helm-charts
helm upgrade --install my-opencost-integration port-labs/port-ocean \
	--set port.clientId="CLIENT_ID"  \
	--set port.clientSecret="CLIENT_SECRET"  \
	--set initializePortResources=true  \
	--set integration.identifier="my-opencost-integration"  \
	--set integration.type="opencost"  \
	--set integration.eventListener.type="POLLING"  \
	--set integration.config.opencostHost="https://myOpenCostInstance:9003"
Advanced integration configuration

For advanced configuration such as proxies or self-signed certificates, click here.

Ingesting OpenCost objects​

The OpenCost integration uses a YAML configuration to describe the process of loading data into the developer portal.

Here is an example snippet from the config which demonstrates the process for getting cost data from OpenCost:

createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: cost
    selector:
      query: "true"
      window: "month"
      aggregate: "pod"
      step: "window"
      resolution: "1m"
    port:
      entity:
        mappings:
          blueprint: '"openCostResourceAllocation"'
          identifier: .name
          title: .name
          properties:
            cluster: .properties.cluster
            namespace: .properties.namespace
            startDate: .start
            endDate: .end
            cpuCoreHours: .cpuCoreHours
            cpuCost: .cpuCost
            cpuEfficiency: .cpuEfficiency
            gpuHours: .gpuHours
            gpuCost: .gpuCost
            networkCost: .networkCost
            loadBalancerCost: .loadBalancerCost
            pvCost: .pvCost
            ramBytes: .ramBytes
            ramCost: .ramCost
            ramEfficiency: .ramEfficiency
            sharedCost: .sharedCost
            externalCost: .externalCost
            totalCost: .totalCost
            totalEfficiency: .totalEfficiency

The integration makes use of the JQ JSON processor to select, modify, concatenate, transform and perform other operations on existing fields and values from OpenCost's API events.

Configuration structure​

The integration configuration determines which resources will be queried from OpenCost, and which entities and properties will be created in Port.

Supported resources

The following resources can be used to map data from OpenCost, it is possible to reference any field that appears in the API responses linked below for the mapping configuration.

  • The root key of the integration configuration is the resources key:

    resources:
      - kind: cost
        selector:
        ...

  • The kind key is a specifier for an OpenCost object:

      resources:
        - kind: cost
          selector:
          ...

  • The selector and the query keys allow you to filter which objects of the specified kind will be ingested into your software catalog:

    resources:
      - kind: cost
        selector:
          query: "true" # JQ boolean expression. If evaluated to false - this object will be skipped.
          window: "month"
          aggregate: "pod"
          step: "window"
          resolution: "1m"
        port:
    • window - Duration of time over which to query. Accepts words like: today, week, month, yesterday, lastweek, lastmonth; durations like 30m, 12h, 7d; RFC3339 date pairs like 2021-01-02T15:04:05Z,2021-02-02T15:04:05Z; Unix timestamps like 1578002645,1580681045. If not specified, the default value is "today"
    • aggregate - Field by which to aggregate the results. Accepts: cluster, node, namespace, controllerKind, controller, service, pod, container, label:name, and annotation:name. Also accepts comma-separated lists for multi-aggregation, like namespace,label:app.
    • step - Duration of a single allocation set. If unspecified, this defaults to the window, so that you receive exactly one set for the entire window. If specified, such as 30m, 2h, 1d etc, it works chronologically backward, querying in durations of step until the full window is covered. Default is window.
    • resolution - Duration to use as resolution in Prometheus queries. Smaller values (i.e. higher resolutions) will provide better accuracy, but worse performance (i.e. slower query time, higher memory use). Larger values (i.e. lower resolutions) will perform better, but at the expense of lower accuracy for short-running workloads. Default is 1m.

  • The port, entity and the mappings keys are used to map the OpenCost object fields to Port entities. To create multiple mappings of the same kind, you can add another item in the resources array;

  resources:
    - kind: cost
      selector:
        query: "true"
      port:
        entity:
          mappings: # Mappings between one OpenCost object to a Port entity. Each value is a JQ query.
            identifier: .name
            title: .name
            blueprint: '"openCostResourceAllocation"'
            properties:
              cluster: .properties.cluster
              namespace: .properties.namespace
              startDate: .start
              endDate: .end
              cpuCoreHours: .cpuCoreHours
              cpuCost: .cpuCost
              cpuEfficiency: .cpuEfficiency
              gpuHours: .gpuHours
              gpuCost: .gpuCost
              networkCost: .networkCost
              loadBalancerCost: .loadBalancerCost
              pvCost: .pvCost
              ramBytes: .ramBytes
              ramCost: .ramCost
              ramEfficiency: .ramEfficiency
              sharedCost: .sharedCost
              externalCost: .externalCost
              totalCost: .totalCost
              totalEfficiency: .totalEfficiency
    - kind: cost # In this instance cost is mapped again with a different filter
      selector:
        query: '.name == "MyNodeName"'
      port:
        entity:
          mappings: ...
Blueprint key

Note the value of the blueprint key - if you want to use a hardcoded string, you need to encapsulate it in 2 sets of quotes, for example use a pair of single-quotes (') and then another pair of double-quotes (")

Ingest data into Port​

To ingest OpenCost objects using the integration configuration, you can follow the steps below:

  1. Go to the DevPortal Builder page.
  2. Select a blueprint you want to ingest using OpenCost.
  3. Choose the Ingest Data option from the menu.
  4. Select OpenCost under the Cloud cost providers category.
  5. Modify the configuration according to your needs.
  6. Click Resync.

Examples​

Examples of blueprints and the relevant integration configurations:

Cost​

Cost blueprint (click to expand)
{
  "identifier": "openCostResourceAllocation",
  "description": "This blueprint represents an OpenCost resource allocation in our software catalog",
  "title": "OpenCost Resource Allocation",
  "icon": "Cluster",
  "schema": {
    "properties": {
      "cluster": {
        "type": "string",
        "title": "Cluster"
      },
      "namespace": {
        "type": "string",
        "title": "Namespace"
      },
      "startDate": {
        "title": "Start Date",
        "type": "string",
        "format": "date-time"
      },
      "endDate": {
        "title": "End Date",
        "type": "string",
        "format": "date-time"
      },
      "cpuCoreHours": {
        "title": "CPU Core Hours",
        "type": "number"
      },
      "cpuCost": {
        "title": "CPU Cost",
        "type": "number"
      },
      "cpuEfficiency": {
        "title": "CPU Efficiency",
        "type": "number"
      },
      "gpuHours": {
        "title": "GPU Hours",
        "type": "number"
      },
      "gpuCost": {
        "title": "GPU Cost",
        "type": "number"
      },
      "networkCost": {
        "title": "Network Cost",
        "type": "number"
      },
      "loadBalancerCost": {
        "title": "Load Balancer Cost",
        "type": "number"
      },
      "pvCost": {
        "title": "PV Cost",
        "type": "number"
      },
      "ramBytes": {
        "title": "RAM Bytes",
        "type": "number"
      },
      "ramCost": {
        "title": "RAM Cost",
        "type": "number"
      },
      "ramEfficiency": {
        "title": "RAM Efficiency",
        "type": "number"
      },
      "sharedCost": {
        "title": "Shared Cost",
        "type": "number"
      },
      "externalCost": {
        "title": "External Cost",
        "type": "number"
      },
      "totalCost": {
        "title": "Total Cost",
        "type": "number"
      },
      "totalEfficiency": {
        "title": "Total Efficiency",
        "type": "number"
      }
    },
    "required": []
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "relations": {}
}
Integration configuration (click to expand)
createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: cost
    selector:
      query: "true"
      window: "month"
      aggregate: "pod"
      step: "window"
      resolution: "1m"
    port:
      entity:
        mappings:
          blueprint: '"openCostResourceAllocation"'
          identifier: .name
          title: .name
          properties:
            cluster: .properties.cluster
            namespace: .properties.namespace
            startDate: .start
            endDate: .end
            cpuCoreHours: .cpuCoreHours
            cpuCost: .cpuCost
            cpuEfficiency: .cpuEfficiency
            gpuHours: .gpuHours
            gpuCost: .gpuCost
            networkCost: .networkCost
            loadBalancerCost: .loadBalancerCost
            pvCost: .pvCost
            ramBytes: .ramBytes
            ramCost: .ramCost
            ramEfficiency: .ramEfficiency
            sharedCost: .sharedCost
            externalCost: .externalCost
            totalCost: .totalCost
            totalEfficiency: .totalEfficiency

CloudCost​

Cost blueprint (click to expand)
{
    "identifier": "openCostCloudcost",
    "description": "This blueprint represents cloud cost allocations from your OpenCost instance",
    "icon": "Opencost",
    "title": "OpenCost CloudCost",
    "schema": {
      "properties": {
        "startDate": {
          "title": "Start Date",
          "type": "string",
          "format": "date-time"
        },
        "endDate": {
          "title": "End Date",
          "type": "string",
          "format": "date-time"
        },
        "listCost": {
          "title": "List Cost",
          "type": "number"
        },
        "netCost": {
          "title": "Net Cost",
          "type": "number"
        },
        "amortizedNetCost": {
          "title": "Amortized Net Cost",
          "type": "number"
        },
        "invoicedCost": {
          "title": "Invoiced Cost",
          "type": "number"
        },
        "amortizedCost": {
          "title": "Amortized Cost",
          "type": "number"
        }
      },
      "required": []
    },
    "calculationProperties": {},
    "mirrorProperties": {},
    "relations": {}
  }
Integration Configuration (click to expand)
createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: cloudcost
    selector:
      query: "true"
      cloudcostAggregate: '"provider"'
      window: '"7d"'
      accumulate: '"all"'
    port:
      entity:
        mappings:
          blueprint: '"openCostCloudcost"'
          identifier: .properties.provider + "-" + .window.start + "-" + .window.end
          title: .properties.provider + "-" + .window.start + "-" + .window.end
          properties:
            startDate: .window.start
            endDate: .window.end
            listCost: .listCost.cost
            netCost: .netCost.cost
            amortizedNetCost: .amortizedNetCost.cost
            invoicedCost: .invoicedCost.cost
            amortizedCost: .amortizedCost.cost

Selector Parameters​

The following parameters under the selector key can be used to manipulate which data is ingested:

  • window - Duration of time over which to query. Accepts words like: today, week, month, yesterday, lastweek, lastmonth; durations like 30m, 12h, 7d; RFC3339 date pairs like 2021-01-02T15:04:05Z,2021-02-02T15:04:05Z; Unix timestamps like 1578002645,1580681045. If not specified, the default value is "today"
  • cloudcostAggregate - Field by which to aggregate the results. Accepts: invoiceEntityID, accountID, provider, providerID, category, and service. Also accepts comma-separated lists for multi-aggregation, like provider,service. If no value is provided, the entire list of items is returned.
  • accumulate - Step size of the accumulation. Accepts: all, hour, day, week, month, and quarter. If not specified, the default value is "day".

Ingesting specific providers (AWS, GCP, Azure)​

Perhaps you are not interested in ingesting cloud cost data for other providers and interested in only AWS. To ingest only AWS data, change the selector query key to look like so:

Integration Configuration (click to expand)
createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: cloudcost
    selector:
      // replace "AWS" with provider of choice. Others are "GCP", and "Azure".
      query: .properties.provider | startswith("AWS")
      ...
    port:
      entity:
        mappings:
          blueprint: '"openCostCloudcost"'
          identifier: .properties.provider + "-" + .window.start + "-" + .window.end
          title: .properties.provider + "-" + .window.start + "-" + .window.end
          ...

Let's Test It​

This section includes a sample response data from OpenCost. In addition, it includes the entity created from the resync event based on the Ocean configuration provided in the previous section.

Payload​

Here is an example of the payload structure from OpenCost aggregated on the namespace level:

Cost response data (click to expand)
{
  "name": "ingress-nginx",
  "properties": {
    "cluster": "cluster-one",
    "node": "minikube",
    "container": "controller",
    "controller": "ingress-nginx-controller",
    "controllerKind": "deployment",
    "namespace": "ingress-nginx",
    "pod": "ingress-nginx-controller-7799c6795f-29n7j",
    "services": [
      "ingress-nginx-controller-admission",
      "ingress-nginx-controller"
    ],
    "labels": {
      "app_kubernetes_io_component": "controller",
      "app_kubernetes_io_instance": "ingress-nginx",
      "app_kubernetes_io_name": "ingress-nginx",
      "gcp_auth_skip_secret": "true",
      "kubernetes_io_metadata_name": "ingress-nginx",
      "pod_template_hash": "7799c6795f"
    },
    "namespaceLabels": {
      "app_kubernetes_io_instance": "ingress-nginx",
      "app_kubernetes_io_name": "ingress-nginx",
      "kubernetes_io_metadata_name": "ingress-nginx"
    }
  },
  "window": {
    "start": "2023-10-30T00:00:00Z",
    "end": "2023-10-31T00:00:00Z"
  },
  "start": "2023-10-30T09:05:00Z",
  "end": "2023-10-30T11:50:00Z",
  "minutes": 165,
  "cpuCores": 0.1,
  "cpuCoreRequestAverage": 0.1,
  "cpuCoreUsageAverage": 0,
  "cpuCoreHours": 0.275,
  "cpuCost": 0.00869,
  "cpuCostAdjustment": 0,
  "cpuEfficiency": 0,
  "gpuCount": 0,
  "gpuHours": 0,
  "gpuCost": 0,
  "gpuCostAdjustment": 0,
  "networkTransferBytes": 0,
  "networkReceiveBytes": 0,
  "networkCost": 0,
  "networkCrossZoneCost": 0,
  "networkCrossRegionCost": 0,
  "networkInternetCost": 0,
  "networkCostAdjustment": 0,
  "loadBalancerCost": 0,
  "loadBalancerCostAdjustment": 0,
  "pvBytes": 0,
  "pvByteHours": 0,
  "pvCost": 0,
  "pvs": "None",
  "pvCostAdjustment": 0,
  "ramBytes": 94371840,
  "ramByteRequestAverage": 94371840,
  "ramByteUsageAverage": 0,
  "ramByteHours": 259522560,
  "ramCost": 0.00102,
  "ramCostAdjustment": 0,
  "ramEfficiency": 0,
  "externalCost": 0,
  "sharedCost": 0,
  "totalCost": 0.00972,
  "totalEfficiency": 0,
  "lbAllocations": "None"
}
CloudCost response data (click to expand)
{
    "properties": {
        "provider": "AWS",
        "invoiceEntityID": "123445954695",
    },
    "window": {
        "start": "2023-10-27T00:00:00Z",
        "end": "2023-10-28T00:00:00Z",
    },
    "listCost": {
        "cost": 1143.6968755312,
        "kubernetesPercent": 0.667336136012796,
    },
    "netCost": {
        "cost": 1143.6968755312,
        "kubernetesPercent": 0.667336136012796,
    },
    "amortizedNetCost": {
        "cost": 1143.6968755312,
        "kubernetesPercent": 0.667336136012796,
    },
    "invoicedCost": {
        "cost": 1143.6968755312,
        "kubernetesPercent": 0.667336136012796,
    },
    "amortizedCost": {
        "cost": 1143.6968755312,
        "kubernetesPercent": 0.667336136012796,
    },
}

Mapping Result​

The combination of the sample payload and the Ocean configuration generates the following Port entity:

Cost entity in Port (click to expand)
{
  "identifier": "ingress-nginx",
  "title": "ingress-nginx",
  "blueprint": "openCostResourceAllocation",
  "team": [],
  "properties": {
    "cluster": "cluster-one",
    "namespace": "ingress-nginx",
    "startDate": "2023-10-30T09:05:00.000Z",
    "endDate": "2023-10-30T11:50:00.000Z",
    "cpuCoreHours": 0.275,
    "cpuCost": 0.00869,
    "cpuEfficiency": 0,
    "gpuHours": 0,
    "gpuCost": 0,
    "networkCost": 0,
    "loadBalancerCost": 0,
    "pvCost": 0,
    "ramBytes": 94371840,
    "ramCost": 0.00102,
    "ramEfficiency": 0,
    "sharedCost": 0,
    "externalCost": 0,
    "totalCost": 0.00972,
    "totalEfficiency": 0
  },
  "relations": {},
  "createdAt": "2023-10-15T09:30:57.924Z",
  "createdBy": "hBx3VFZjqgLPEoQLp7POx5XaoB0cgsxW",
  "updatedAt": "2023-10-30T11:49:20.881Z",
  "updatedBy": "hBx3VFZjqgLPEoQLp7POx5XaoB0cgsxW"
}
CloudCost entity in Port (click to expand)
{
  "identifier": "AWS-2023-10-27T00:00:00Z-2023-10-28T00:00:00Z",
  "title": "AWS-2023-10-27T00:00:00Z-2023-10-28T00:00:00Z",
  "blueprint": "openCostResourceAllocation",
  "team": [],
  "properties": {
    "startDate": "2023-10-27T00:00:00Z",
    "endDate": "2023-10-28T00:00:00Z",
    "listCost": 1143.6968755312,
    "netCost": 1143.6968755312,
    "amortizedNetCost": 1143.6968755312,
    "invoicedCost": 1143.6968755312,
    "amortizedCost": 1143.6968755312
  },
  "relations": {},
  "createdAt": "2023-10-15T09:30:57.924Z",
  "createdBy": "hBx3VFZjqgLPEoQLp7POx5XaoB0cgsxW",
  "updatedAt": "2023-10-30T11:49:20.881Z",
  "updatedBy": "hBx3VFZjqgLPEoQLp7POx5XaoB0cgsxW"
}