Skip to main content

Kubecost

Our Kubecost integration allows you to import kubesystem and cloud cost allocations from your Kubecost instance into Port, according to your mapping and definition.

Common use cases​

  • Map your monitored Kubernetes resources and cloud cost allocations in Kubecost.

Prerequisites​

To install the integration, you need a Kubernetes cluster that the integration's container chart will be deployed to.

Please make sure that you have kubectl and helm installed on your machine, and that your kubectl CLI is connected to the Kubernetes cluster where you plan to install the integration.

Troubleshooting

If you are having trouble installing this integration, please refer to these troubleshooting steps.

Installation​

Choose one of the following installation methods:

Using this installation option means that the integration will be hosted by Port, with a customizable resync interval to ingest data into Port.

To install, follow the following steps:

  1. Go to the Data sources page of your portal.

  2. Click on the + Data source button in the top-right corner.

  3. Click on the relevant integration in the list.

  4. Under Select your installation method, choose Hosted by Port.

  5. Configure the integration settings and application settings as you wish (see below for details).

Application settings

Every integration hosted by Port has the following customizable application settings:

  • Resync interval: The frequency at which Port will ingest data from the integration. There are various options available, ranging from every 1 hour to once a day.

  • Send raw data examples: A boolean toggle (enabled by default). If enabled, raw data examples will be sent from the integration to Port. These examples are used when testing your mapping configuration, they allow you to run your jq expressions against real data and see the results.

Integration settings

Every integration has its own tool-specific settings, under the Integration settings section.
Each of these settings has an ⓘ icon next to it, which you can hover over to see a description of the setting.

Port secrets

Some integration settings require sensitive pieces of data, such as tokens.
For these settings, Port secrets will be used, ensuring that your sensitive data is encrypted and secure.

When changing such a setting, you will be prompted to choose an existing secret or create a new one:

Live event support

Currently, live events are not supported for integrations hosted by Port.
Resyncs will be performed periodically, based on the Resync interval you have configured, or manually triggered by you via Port's UI.

Therefore, real-time events such as GitOps pushes will not be ingested into Port immediately.
Live events are WIP and will be supported in the future.

Ingesting Kubecost objects​

The Kubecost integration uses a YAML configuration to describe the process of loading data into the developer portal.

Here is an example snippet from the config which demonstrates the process for getting cost allocation data from Kubecost:

createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: kubesystem
    selector:
      query: "true"
      window: "month"
    port:
      entity:
        mappings:
          blueprint: '"kubecostResourceAllocation"'
          identifier: .name
          title: .name
          properties:
            cluster: .properties.cluster
            namespace: .properties.namespace
            startDate: .start
            endDate: .end
            cpuCoreHours: .cpuCoreHours
            cpuCost: .cpuCost
            cpuEfficiency: .cpuEfficiency
            gpuHours: .gpuHours
            gpuCost: .gpuCost
            networkCost: .networkCost
            loadBalancerCost: .loadBalancerCost
            pvCost: .pvCost
            pvBytes: .pvBytes
            ramBytes: .ramBytes
            ramCost: .ramCost
            ramEfficiency: .ramEfficiency
            sharedCost: .sharedCost
            externalCost: .externalCost
            totalCost: .totalCost
            totalEfficiency: .totalEfficiency

The integration makes use of the JQ JSON processor to select, modify, concatenate, transform and perform other operations on existing fields and values from Kubecost's API events.

Configuration structure​

The integration configuration determines which resources will be queried from Kubecost, and which entities and properties will be created in Port.

Supported resources

The following resources can be used to map data from Kubecost, it is possible to reference any field that appears in the API responses linked below for the mapping configuration.

note

You will be able to see cloud cost data after you have successfully configured the Cloud Billing API on your Kubecost instance according to this documentation

  • The root key of the integration configuration is the resources key:

    resources:
      - kind: kubesystem
        selector:
        ...

  • The kind key is a specifier for an Kubecost object:

      resources:
        - kind: kubesystem
          selector:
          ...

  • The selector and the query keys allow you to filter which objects of the specified kind will be ingested into your software catalog:

    resources:
      - kind: kubesystem
        selector:
          query: "true" # JQ boolean expression. If evaluated to false - this object will be skipped.
          window: "month"
          aggregate: "pod"
          idle: true
        port:
    • window - Duration of time over which to query. Accepts: words like today, week, month, yesterday, lastweek, lastmonth; durations like 30m, 12h, 7d; RFC3339 date pairs like 2021-01-02T15:04:05Z,2021-02-02T15:04:05Z; Unix timestamps like 1578002645,1580681045.
    • aggregate - Field by which to aggregate the results. Accepts: cluster, node, namespace, controllerKind, controller, service, pod, container, label:name, and annotation:name. Also accepts comma-separated lists for multi-aggregation, like namespace,label:app.
    • step - Duration of a single allocation set. If unspecified, this defaults to the window, so that you receive exactly one set for the entire window. If specified, such as 30m, 2h, 1d etc, it works chronologically backward, querying in durations of step until the full window is covered. Default is window.
    • accumulate - If true, sum the entire range of sets into a single set. Default value is false.
    • idle - If true, include idle cost (i.e. the cost of the un-allocated assets) as its own allocation. Default is true.
    • external - If true, include external, or out-of-cluster costs in each allocation. Default is false.
    • filterClusters - Comma-separated list of clusters to match; e.g. cluster-one,cluster-two will return results from only those two clusters.
    • filterNodes - Comma-separated list of nodes to match; e.g. node-one,node-two will return results from only those two nodes.
    • filterNamespaces - Comma-separated list of namespaces to match; e.g. namespace-one,namespace-two will return results from only those two namespaces.
    • filterControllerKinds - Comma-separated list of controller kinds to match; e.g. deployment, job will return results with only those two controller kinds.
    • filterControllers - Comma-separated list of controllers to match; e.g. deployment-one,statefulset-two will return results from only those two controllers.
    • filterPods - Comma-separated list of pods to match; e.g. pod-one,pod-two will return results from only those two pods.
    • filterAnnotations - Comma-separated list of annotations to match; e.g. name:annotation-one,name:annotation-two will return results with either of those two annotation key-value-pairs.
    • filterControllerKinds - Comma-separated list of controller kinds to match; e.g. deployment, job will return results with only those two controller kinds.
    • filterLabels - Comma-separated list of annotations to match; e.g. app:cost-analyzer, app:prometheus will return results with either of those two label key-value-pairs.
    • filterServices - Comma-separated list of services to match; e.g. frontend-one,frontend-two will return results with either of those two services.
    • shareIdle - If true, idle cost is allocated proportionally across all non-idle allocations, per-resource. That is, idle CPU cost is shared with each non-idle allocation's CPU cost, according to the percentage of the total CPU cost represented. Default is false.
    • splitIdle - If true, and shareIdle == false, Idle Allocations are created on a per cluster or per node basis rather than being aggregated into a single idle allocation. Default is false.
    • idleByNode - If true, idle allocations are created on a per node basis. Which will result in different values when shared and more idle allocations when split. Default is false.
    • And any query parameter that could be found in the Kubecost allocation API and Kubecost Cloud API

  • The port, entity and the mappings keys are used to map the Kubecost object fields to Port entities. To create multiple mappings of the same kind, you can add another item in the resources array;

createMissingRelatedEntities: true
deleteDependentEntities: true
  resources:
    - kind: kubesystem
      selector:
        query: "true"
      port:
        entity:
          mappings: # Mappings between one Kubecost object to a Port entity. Each value is a JQ query.
            identifier: .name
            title: .name
            blueprint: '"KubecostResourceAllocation"'
            properties:
              cluster: .properties.cluster
              namespace: .properties.namespace
              startDate: .start
              endDate: .end
              cpuCoreHours: .cpuCoreHours
              cpuCost: .cpuCost
              cpuEfficiency: .cpuEfficiency
              gpuHours: .gpuHours
              gpuCost: .gpuCost
              networkCost: .networkCost
              loadBalancerCost: .loadBalancerCost
              pvCost: .pvCost
              ramBytes: .ramBytes
              ramCost: .ramCost
              ramEfficiency: .ramEfficiency
              sharedCost: .sharedCost
              externalCost: .externalCost
              totalCost: .totalCost
              totalEfficiency: .totalEfficiency
    - kind: kubesystem # In this instance cost is mapped again with a different filter
      selector:
        query: '.name == "MyNodeName"'
      port:
        entity:
          mappings: ...
Blueprint key

Note the value of the blueprint key - if you want to use a hardcoded string, you need to encapsulate it in 2 sets of quotes, for example use a pair of single-quotes (') and then another pair of double-quotes (")

Ingest data into Port​

To ingest Kubecost objects using the integration configuration, you can follow the steps below:

  1. Go to the DevPortal Builder page.
  2. Select a blueprint you want to ingest using Kubecost.
  3. Choose the Ingest Data option from the menu.
  4. Select Kubecost under the Cloud cost providers category.
  5. Modify the configuration according to your needs.
  6. Click Resync.

Examples​

Examples of blueprints and the relevant integration configurations:

Cost allocation​

Cost allocation blueprint
{
  "identifier": "kubecostResourceAllocation",
  "description": "This blueprint represents an Kubecost resource allocation in our software catalog",
  "title": "Kubecost Resource Allocation",
  "icon": "Cluster",
  "schema": {
    "properties": {
      "cluster": {
        "type": "string",
        "title": "Cluster"
      },
      "namespace": {
        "type": "string",
        "title": "Namespace"
      },
      "startDate": {
        "title": "Start Date",
        "type": "string",
        "format": "date-time"
      },
      "endDate": {
        "title": "End Date",
        "type": "string",
        "format": "date-time"
      },
      "cpuCoreHours": {
        "title": "CPU Core Hours",
        "type": "number"
      },
      "cpuCost": {
        "title": "CPU Cost",
        "type": "number"
      },
      "cpuEfficiency": {
        "title": "CPU Efficiency",
        "type": "number"
      },
      "gpuHours": {
        "title": "GPU Hours",
        "type": "number"
      },
      "gpuCost": {
        "title": "GPU Cost",
        "type": "number"
      },
      "networkCost": {
        "title": "Network Cost",
        "type": "number"
      },
      "loadBalancerCost": {
        "title": "Load Balancer Cost",
        "type": "number"
      },
      "pvCost": {
        "title": "PV Cost",
        "type": "number"
      },
      "pvBytes": {
        "title": "PV Bytes",
        "type": "number"
      },
      "ramBytes": {
        "title": "RAM Bytes",
        "type": "number"
      },
      "ramCost": {
        "title": "RAM Cost",
        "type": "number"
      },
      "ramEfficiency": {
        "title": "RAM Efficiency",
        "type": "number"
      },
      "sharedCost": {
        "title": "Shared Cost",
        "type": "number"
      },
      "externalCost": {
        "title": "External Cost",
        "type": "number"
      },
      "totalCost": {
        "title": "Total Cost",
        "type": "number"
      },
      "totalEfficiency": {
        "title": "Total Efficiency",
        "type": "number"
      }
    },
    "required": []
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "relations": {}
}
Integration configuration
createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: kubesystem
    selector:
      query: "true"
    port:
      entity:
        mappings:
          blueprint: '"kubecostResourceAllocation"'
          identifier: .name
          title: .name
          properties:
            cluster: .properties.cluster
            namespace: .properties.namespace
            startDate: .start
            endDate: .end
            cpuCoreHours: .cpuCoreHours
            cpuCost: .cpuCost
            cpuEfficiency: .cpuEfficiency
            gpuHours: .gpuHours
            gpuCost: .gpuCost
            networkCost: .networkCost
            loadBalancerCost: .loadBalancerCost
            pvCost: .pvCost
            pvBytes: .pvBytes
            ramBytes: .ramBytes
            ramCost: .ramCost
            ramEfficiency: .ramEfficiency
            sharedCost: .sharedCost
            externalCost: .externalCost
            totalCost: .totalCost
            totalEfficiency: .totalEfficiency

Cloud cost​

Cloud cost blueprint
{
  "identifier": "kubecostCloudAllocation",
  "description": "This blueprint represents an Kubecost cloud resource allocation in our software catalog",
  "title": "Kubecost Cloud Allocation",
  "icon": "Cluster",
  "schema": {
    "properties": {
      "provider": {
        "type": "string",
        "title": "Provider"
      },
      "accountID": {
        "type": "string",
        "title": "Account ID"
      },
      "invoiceEntityID": {
        "type": "string",
        "title": "Invoice Entity ID"
      },
      "startDate": {
        "title": "Start Date",
        "type": "string",
        "format": "date-time"
      },
      "endDate": {
        "title": "End Date",
        "type": "string",
        "format": "date-time"
      },
      "listCost": {
        "title": "List Cost Value",
        "type": "number"
      },
      "listCostPercent": {
        "title": "List Cost Percent",
        "type": "number"
      },
      "netCost": {
        "title": "Net Cost Value",
        "type": "number"
      },
      "netCostPercent": {
        "title": "Net Cost Percent",
        "type": "number"
      },
      "amortizedNetCost": {
        "title": "Amortized Net Cost",
        "type": "number"
      },
      "amortizedNetCostPercent": {
        "title": "Amortized Net Cost Percent",
        "type": "number"
      },
      "invoicedCost": {
        "title": "Invoice Cost",
        "type": "number"
      },
      "invoicedCostPercent": {
        "title": "Invoice Cost Percent",
        "type": "number"
      }
    },
    "required": []
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "relations": {}
}
Integration configuration
createMissingRelatedEntities: true
deleteDependentEntities: true
resources:
  - kind: cloud
    selector:
      query: "true"
    port:
      entity:
        mappings:
          blueprint: '"kubecostCloudAllocation"'
          identifier: .properties.provider + "/" + .properties.providerID + "/" + .properties.category + "/" + .properties.service | gsub("[^A-Za-z0-9@_.:\\\\/=-]"; "-")
          title: .properties.provider + "/" + .properties.service
          properties:
            provider: .properties.provider
            accountID: .properties.accountID
            invoiceEntityID: .properties.invoiceEntityID
            startDate: .window.start
            endDate: .window.end
            listCost: .listCost.cost
            listCostPercent: .listCost.kubernetesPercent
            netCost: .netCost.cost
            netCostPercent: .netCost.kubernetesPercent
            amortizedNetCost: .amortizedNetCost.cost
            amortizedNetCostPercent: .amortizedNetCost.kubernetesPercent
            invoicedCost: .invoicedCost.cost
            invoicedCostPercent: .invoicedCost.kubernetesPercent

Let's Test It​

This section includes a sample response data from Kubecost. In addition, it includes the entity created from the resync event based on the Ocean configuration provided in the previous section.

Payload​

Here is an example of the payload structure from Kubecost:

Cost response data
{
  "name": "argocd",
  "properties": {
    "cluster": "cluster-one",
    "node": "gke-my-regional-cluster-default-pool-e8093bfa-0bjg",
    "namespace": "argocd",
    "providerID": "gke-my-regional-cluster-default-pool-e8093bfa-0bjg",
    "namespaceLabels": {
      "kubernetes_io_metadata_name": "argocd"
    }
  },
  "window": {
    "start": "2023-10-30T00:00:00Z",
    "end": "2023-10-30T01:00:00Z"
  },
  "start": "2023-10-30T00:00:00Z",
  "end": "2023-10-30T01:00:00Z",
  "minutes": 60,
  "cpuCores": 0.00515,
  "cpuCoreRequestAverage": 0,
  "cpuCoreUsageAverage": 0.00514,
  "cpuCoreHours": 0.00515,
  "cpuCost": 0.00012,
  "cpuCostAdjustment": 0,
  "cpuEfficiency": 1,
  "gpuCount": 0,
  "gpuHours": 0,
  "gpuCost": 0,
  "gpuCostAdjustment": 0,
  "networkTransferBytes": 2100541.53,
  "networkReceiveBytes": 2077024.88318,
  "networkCost": 0,
  "networkCrossZoneCost": 0,
  "networkCrossRegionCost": 0,
  "networkInternetCost": 0,
  "networkCostAdjustment": 0,
  "loadBalancerCost": 0.02708,
  "loadBalancerCostAdjustment": 0,
  "pvBytes": 0,
  "pvByteHours": 0,
  "pvCost": 0,
  "pvs": "None",
  "pvCostAdjustment": 0,
  "ramBytes": 135396181.33333,
  "ramByteRequestAverage": 0,
  "ramByteUsageAverage": 135394433.70477,
  "ramByteHours": 135396181.33333,
  "ramCost": 0.00041,
  "ramCostAdjustment": 0,
  "ramEfficiency": 1,
  "externalCost": 0,
  "sharedCost": 0,
  "totalCost": 0.02761,
  "totalEfficiency": 1,
  "proportionalAssetResourceCosts": {},
  "lbAllocations": {
    "cluster-one/argocd/argocd-server": {
      "service": "argocd/argocd-server",
      "cost": 0.027083333333333334,
      "private": false,
      "ip": ""
    }
  },
  "sharedCostBreakdown": {}
}

Mapping Result​

The combination of the sample payload and the Ocean configuration generates the following Port entity:

Cost entity in Port
{
  "identifier": "argocd",
  "title": "argocd",
  "icon": null,
  "blueprint": "kubecostResourceAllocation",
  "team": [],
  "properties": {
    "cluster": "cluster-one",
    "namespace": "argocd",
    "startDate": "2023-10-30T04:00:00.000Z",
    "endDate": "2023-10-30T05:00:00.000Z",
    "cpuCoreHours": 0.0051,
    "cpuCost": 0.00012,
    "cpuEfficiency": 1,
    "gpuHours": 0,
    "gpuCost": 0,
    "networkCost": 0,
    "loadBalancerCost": 0.02708,
    "pvCost": 0,
    "pvBytes": 0,
    "ramBytes": 135396181.33333,
    "ramCost": 0.00041,
    "ramEfficiency": 1,
    "sharedCost": 0,
    "externalCost": 0,
    "totalCost": 0.02761,
    "totalEfficiency": 1
  },
  "relations": {},
  "createdAt": "2023-10-30T13:25:42.717Z",
  "createdBy": "hBx3VFZjqgLPEoQLp7POx5XaoB0cgsxW",
  "updatedAt": "2023-10-30T13:28:37.379Z",
  "updatedBy": "hBx3VFZjqgLPEoQLp7POx5XaoB0cgsxW"
}