Skip to main content

Manage Azure costs by using CCM on Harness Self-Managed Enterprise Edition

This topic walks you through the steps required to set up CCM for Azure in a self-managed platform.

Figure: Azure CCM Self-Managed Enterprise Edition architecture diagram

You need to perform the following tasks to set up CCM for Azure. For Step 1, 2 and 3 Sign in to your Azure Portal:

  1. Setup a new Application via App Registration.
  2. Create a new Client secret.
  3. Setup a new Storage Account and a new Storage Container.
  4. Deploy workloads via Helm charts.

Setup a new Application via App Registration​

  1. Give it Name like Harness CCM App.
  2. For Supported account types select: Accounts in any organizational directory (Any Microsoft Entra ID tenant - Multitenant).
  3. Click on Register. For more information, go to Quickstart: Register an App.

Create a new Client secret​

  1. Once you click register, the App Overview Page would be open.
  2. From the overview page, Copy and Save following from it: Copy & Save Directory (tenant) ID → tenantId Copy & Save Application (client) ID → clientId.
  1. Now click on Certificates & secrets in the left panel.
  2. Go to Client secrets (0) tab.
  3. Click on New client secret.
  4. Enter Description Harness CCM Client Secret.
  5. Select Expires from the drop down select 730 days (24 months)(or the maximum allowed time).
  6. Click Add Button.
  1. From the new Client secret, Copy & Save Value → clientSecret. For more information, go to Add credentials.
info

Make a note of the following:

  • tenantId

  • clientId

  • clientSecret

Setup a new Storage Account and a new Storage Container​

  1. Select a Subscription from the drop down.
  2. Select a Resource Group from the drop down.
  3. Enter Storage account name ccmbillingdatasmp. Save Storage account name → storageName
  4. Move to Advanced Tab.
  5. In Blob storage section, enable Allow cross-tenant replication
  6. Click on Review.
  1. Click on Create. For more information, go to Create a storage account.
  2. Once the storage account is created, Go to Containers in left panel.
  3. Click on + Container.
  4. Enter Name as billingdatacontainer. Save Name as → containerName
  5. Click on Create. For more information, go to Create a container.
  1. Go to Shared access signature in left panel.
  2. Check all Allowed resource types which are Service, Container and Object.
  3. Add 10 years to End in Start and expiry date/time.
  4. Click on Generate SAS and connection string.
  5. Save SAS token → sasToken starting from sv=, ignore ? in beginning. For more information, go to Create your SAS tokens.
info

Make a note of the following:

  • storageName

  • containerName

  • sasToken

Deploy workloads via Helm charts​

  1. Clone the chart repository.
git clone git@github.com:harness/helm-charts.git
cd main/src/harness
  1. Upgrade charts if you're already using Harness Self-managed Enterprise Edition services. Perform the following steps to update the override files:

    1. Retrieve the current override values provided during the installation or upgrade of the Helm charts.

      helm get values <chart-name> -n <namespace> > override.yaml

    2. After obtaining the override file, you can make necessary modifications as mentioned below.

Override file changes for a connected environment
global:
  ccm:
    enabled: true
  smtpCreateSecret:
    enabled: true
  license:
    ng: <SMP NG License with CCM>
  database:
    clickhouse:
      enabled: true

ccm:
  nextgen-ce:
    cloudProviderConfig:
      AZURE_APP_CLIENT_ID: "<clientId>"
  batch-processing:
    azureConfig:
      AZURE_SMP_ENABLED: true
      AZURE_SMP_REPORT_RETRIES: 5
      AZURE_SMP_HISTORY_TIME_DELTA: 1
      HARNESS_CE_AZURE_CONTAINER_NAME: "<containerName>"
      HARNESS_CE_AZURE_STORAGE_NAME: "<storageName>"
      HARNESS_CE_AZURE_IS_SYNC_JOB_DISABLED: "false"
info

Air-gapped environment for Azure in SMP is not supported as of now

  1. After making the necessary updates to the override file, you can proceed with the Helm chart upgrade.
helm upgrade <chart-name> <chart-directory> -n <namespace> -f override.yaml

Handling Kubernetes secrets​

When installing or upgrading the Helm charts, Kubernetes secrets with default values are created within the cluster. These generated secrets should be updated with the values mentioned above. Before updating the secrets, you need to convert the secret into base64 encoded format. For example, if your HARNESS_CE_AZURE_CLIENTID value is "clientId", it would be stored as Y2xpZW50SWQ== after encoding.

The following are the secrets specific to CCM services:

  • batch-processing
kubectl edit secret batch-processing -n <namespace>
HARNESS_CE_AZURE_CLIENTSECRET: "<clientSecret>"
HARNESS_CE_AZURE_SAS: "<sasToken>"
HARNESS_CE_AZURE_CLIENTID: "<clientId>"
HARNESS_CE_AZURE_TENANTID: "<tenantId>"
  • cloud-info-secret-mount [config-file]
kubectl edit secret cloud-info-secret-mount -n <namespace>
config-file: <Config file> [In "Config file" provided below: Replace <clientId>, <tenantId> and <clientSecret>]
  • nextgen-ce
kubectl edit secret nextgen-ce -n <namespace>
AZURE_APP_CLIENT_SECRET: "<clientSecret>"
Config file
environment = "production"
debug = false
shutdownTimeout = "5s"

[config.vault]
enabled = false
address = ""
token = ""
secretPath = ""

[log]
format = "json"
level = "info"

[metrics]
enabled = false
address = ":9090"

[jaeger]
enabled = false

# Configure either collectorEndpoint or agentEndpoint.
# When both are configured collectorEndpoint will take precedence and the exporter will report directly to the collector.
collectorEndpoint = "http://localhost:14268/api/traces?format=jaeger.thrift"
agentEndpoint = "localhost:6831"
# username = ""
# password = ""

[app]
address = ":8000"
basePath = "/"

[scrape]
enabled = true
interval = "24h"

[provider.amazon]
enabled = false

# See available regions in the documentation:
# https://aws.amazon.com/about-aws/global-infrastructure/regions_az
# region = "us-east-1"

# Static credentials
# accessKey = ""
# secretKey = ""

# Shared credentials
# sharedCredentialsFile = ""
# profile = ""

# IAM Role ARN to assume
# assumeRoleARN = ""

# http address of a Prometheus instance that has AWS spot price metrics via banzaicloud/spot-price-exporter.
# If empty, the cloudinfo app will use current spot prices queried directly from the AWS API.
prometheusAddress = ""

# advanced configuration: change the query used to query spot price info from Prometheus.
prometheusQuery = "avg_over_time(aws_spot_current_price{region=\"%s\", product_description=\"Linux/UNIX\"}[1w])"

# Amazon pricing API credentials (optional)
# Falls back to the primary credentials.
[provider.amazon.pricing]

# See available regions in the documentation:
# https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/using-pelong.html
# region = "us-east-1"

# Static credentials
# accessKey = ""
# secretKey = ""

# Shared credentials
# sharedCredentialsFile = ""
# profile = ""

# IAM Role ARN to assume
# assumeRoleARN = ""

[provider.google]
enabled = false

# base64 encoded credentials in json format (base64 encoded content of the credential file)
# credentials = ""

# credentialsFile = ""

# project = ""

[provider.alibaba]
enabled = false

# region = ""
# accessKey = ""
# secretKey = ""

[provider.oracle]
enabled = false

# tenancy = ""
# user = ""
# region = ""
# fingerprint = ""
# privateKey = ""
# privateKeyPassphrase = ""

# configFilePath = ""
# profile = ""

[provider.azure]
enabled = true

# subscriptionId = ""

# Client credentials
clientId = "<clientId>"
clientSecret = "<clientSecret>"
tenantId = "<tenantId>"

[provider.digitalocean]
enabled = false

[provider.vsphere]
enabled = false

# accessToken = ""

[management]
enabled = true
address = ":8001"

[serviceloader]
serviceConfigLocation = "./configs"
serviceConfigName = "services"
format = "yaml"

[store.redis]
enabled = false
host = "localhost"
port = 6379

[store.cassandra]
enabled = false
hosts = "localhost"
port = 9042
keyspace = "cloudinfo"
table = "products"

[store.gocache]
expiration = 0
cleanupInterval = 0

The following are some secrets from platform-service that you need to update:

  • smtp-secret - Required to support budget alerts email.
kubectl edit secret smtp-secret -n <namespace> 
SMTP_HOST: <SMTP_HOST>
SMTP_PASSWORD: <SMTP_PASSWORD>
SMTP_PORT: <SMTP_PORT>
SMTP_USE_SSL: <SMTP_USE_SSL>
SMTP_USERNAME: <SMTP_USERNAME>
info

To increase TimescaleDB to 100Gi, run: kubectl edit pvc wal-volume-harness-timescaledb-0 -n <namespace>. Features like Recommendations and Anomalies within CCM services use it.

Next steps​