DRAFT: [Chargeback] new integration

.github/CODEOWNERS

@@ -499,4 +499,5 @@
 /packages/cisco_meraki_metrics @elastic/obs-infraobs-integrations
 /packages/panw_metrics @elastic/obs-infraobs-integrations
 /packages/o365_metrics @elastic/obs-infraobs-integrations 
+/packages/chargeback @elastic/customer-architects
 /packages/aws_billing @elastic/obs-infraobs-integrations 

packages/chargeback/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: [email protected]

packages/chargeback/_dev/build/docs/README.md

@@ -0,0 +1,61 @@
+# Chargeback
+
+_Technical preview: This integration is being developed by Elastic's Customer Engineering team. Please report any issues to the Elastician who shared this integration with you._
+
+The Chargeback integration provides FinOps visibility into Elastic usage across tenants. By integrating data from the [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) and [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integrations, it enables the determination of value provided by each deployment, data stream, and tier accross the organisation. This allows Centre of Excellence (CoE) teams to accurately allocate costs back to the appropriate tenant.
+
+## What is FinOps?
+
+FinOps is an operational framework and cultural practice aimed at maximizing the business value of cloud usage. It facilitates timely, data-driven decision-making and promotes financial accountability through collaboration among engineering, finance, and business teams.
+
+## Purpose
+
+The Chargeback integration assists organisations in addressing a crucial question:
+
+> **"How is my organisation consuming the Elastic solution, and to which tenants can I allocate these costs?"**
+
+The integration provides a breakdown of Elastic Consumption Units (ECUs) per:
+
+- Deployment
+- Data tier
+- Data stream
+- Day
+
+Currently, Chargeback calculations consider only Elasticsearch data nodes. Contributions from other assets, like Kibana or ML nodes, are assumed to be shared proportionally among tenants. To incorporate indexing, querying, and storage in a weighted manner, a blended value is created using the following default weights (modifiable):
+- Indexing: `20` (applicable only to the hot tier)
+- Querying: `20`
+- Storage: `40`
+
+This default weighting means storage contributes most to the blended cost calculation, with indexing considered only on the hot tier. Adjust these weights based on your organisation's needs and best judgment.
+
+Chargeback is also present based on a configured rate and unit. These are used to display cost in the local currency, for instance `EUR`, with a rate of `0.85`.
+
+All configuration values can be updated, as follows:
+
+```
+POST chargeback_conf_lookup/_update/config
+{
+  "doc": {
+    "conf_ecu_rate": 0.85,
+    "conf_ecu_rate_unit": "EUR",
+    "conf_indexing_weight": 50,
+    "conf_query_weight": 20,
+    "conf_storage_weight": 40
+  }
+}
+```
+
+Chargeback data can be viewed in the `[Chargeback] Cost and Consumption breakdown` dashboard.
+
+![Cost and Consumption breakdown](../img/chargeback.png)
+
+## Requirements
+
+To use this integration, the following prerequisites must be met:
+
+- The monitoring cluster, where this integration is installed, must be on version 8.18.0+ due to its use of [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join).
+- The [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) integration (v1.4.1+) must be installed and running.
+- The [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integration (v1.16.0+) must be installed and collecting [usage data](https://www.elastic.co/docs/reference/integrations/elasticsearch/#indices-and-data-streams-usage-analysis) from all relevant deployments.
+- The Transform named `logs-elasticsearch.index_pivot-default-{VERSION}` must be running, which is an asset of the **Elasticsearch** integration.
+
+This integration must be installed on the **Monitoring cluster** where the above mentioned relevant usage and billing data is collected.

packages/chargeback/changelog.yml

@@ -0,0 +1,61 @@
+# newer versions go on top
+- version: 0.1.7
+  changes:
+    - description: "Swap the use of deployment_id or deployment name to a concatenation of both, to make it easier to identify the deployment in the dashboard."
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.1.6
+  changes:
+    - description: "Remove the use of usage alias, and stick to using `monitoring-indices` as usage sorce. ES Integration transform should be run regardless of wether the ES integration has been installed on an agent or not. This fix will increase performance, when relying on Stack Monitoring data. Also, use `metrics-ess_billing.billing-*` to be able to use not only the default namespace."
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.1.5
+  changes:
+    - description: "Fixing the control error in the dashboard by adding a data view."
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.1.4
+  changes:
+    - description: "Consistent naming of `datastream`. Add `| LIMIT 5000` to ESQL top query to cater for large organisations."
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.1.3
+  changes:
+    - description: "Made sure the colour palette is predictable by using the eui_amsterdam_color_blind palate. Add ECU rate to the dashboard."
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.1.2
+  changes:
+    - description: "Set transforms to not auto start to allow integration to be uploaded successfully."
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.1.1
+  changes:
+    - description: "Aligned fields returned to field names used in visualisation."
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.1.0
+  changes:
+    - description: "Adding ECU rate unit to the configuration lookup index."
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.0.4
+  changes:
+    - description: "Fix sorting on `Blended value: % ECU per data stream per day` visualisation."
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.0.3
+  changes:
+    - description: Added support to work with the new `chargeback-monitoring-read` alias, which is used to read from either the ouput of the Elasticsearch Integration, or from the Stack Monitoring indices.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.0.2
+  changes:
+    - description: Making use of config lookup values for ecu rate, indexing weight, querying weight, and storage weight.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14545 
+- version: 0.0.1
+  changes:
+    - description: Initial draft of the package
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14545 

packages/chargeback/data_stream/billing/elasticsearch/ingest_pipeline/default.yml

@@ -0,0 +1,11 @@
+---
+description: "Chargeback: Set composite_key from @timestamp and deployment_id."
+processors:
+  - script:
+      lang: painless
+      source: >
+        if (ctx['@timestamp'] != null) {
+            ctx.composite_key = ZonedDateTime.parse(ctx['@timestamp']).toLocalDate().toString() + '_' + ctx.deployment_id;
+        }
+      ignore_failure: true
+      tag: ess_billing

packages/chargeback/data_stream/billing/fields/base-fields.yml

@@ -0,0 +1,12 @@
+- name: data_stream.type
+  type: constant_keyword
+  description: Data stream type.
+- name: data_stream.dataset
+  type: constant_keyword
+  description: Data stream dataset.
+- name: data_stream.namespace
+  type: constant_keyword
+  description: Data stream namespace.
+- name: '@timestamp'
+  type: date
+  description: Event timestamp.

packages/chargeback/data_stream/billing/manifest.yml

@@ -0,0 +1,2 @@
+title: "Billing Metrics Aggregation"
+type: metrics

packages/chargeback/data_stream/usage/elasticsearch/ingest_pipeline/default.yml

@@ -0,0 +1,25 @@
+---
+description: "Chargeback: Set correlation keys and blended calculation weights."
+processors:
+  - script:
+      lang: painless
+      source: >
+        if (ctx.cluster_name != null) {
+          ctx.deployment_id = ctx.cluster_name;
+        } else if (ctx.elasticsearch?.cluster?.name != null) {
+          ctx.deployment_id = ctx.elasticsearch.cluster.name;
+        }
+
+        if (ctx['@timestamp'] != null && ctx.deployment_id != null) {
+          def date = ZonedDateTime.parse(ctx['@timestamp']).toLocalDate().toString();
+          ctx.composite_key = date + '_' + ctx.deployment_id;
+
+          if (ctx.tier != null) {
+            ctx.composite_tier_key = ctx.composite_key + '_' + ctx.tier.replace('/', '_');
+          }
+
+          if (ctx.datastream != null) {
+            ctx.composite_datastream_key = ctx.composite_key + '_' + ctx.datastream;
+          }
+        }
+      ignore_failure: true

packages/chargeback/data_stream/usage/fields/base-fields.yml

@@ -0,0 +1,12 @@
+- name: data_stream.type
+  type: constant_keyword
+  description: Data stream type.
+- name: data_stream.dataset
+  type: constant_keyword
+  description: Data stream dataset.
+- name: data_stream.namespace
+  type: constant_keyword
+  description: Data stream namespace.
+- name: '@timestamp'
+  type: date
+  description: Event timestamp.

packages/chargeback/data_stream/usage/manifest.yml

@@ -0,0 +1,2 @@
+title: Usage Metrics Aggregation
+type: metrics

packages/chargeback/docs/README.md

@@ -0,0 +1,61 @@
+# Chargeback
+
+_Technical preview: This integration is being developed by Elastic's Customer Engineering team. Please report any issues to the Elastician who shared this integration with you._
+
+The Chargeback integration provides FinOps visibility into Elastic usage across tenants. By integrating data from the [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) and [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integrations, it enables the determination of value provided by each deployment, data stream, and tier accross the organisation. This allows Centre of Excellence (CoE) teams to accurately allocate costs back to the appropriate tenant.
+
+## What is FinOps?
+
+FinOps is an operational framework and cultural practice aimed at maximizing the business value of cloud usage. It facilitates timely, data-driven decision-making and promotes financial accountability through collaboration among engineering, finance, and business teams.
+
+## Purpose
+
+The Chargeback integration assists organisations in addressing a crucial question:
+
+> **"How is my organisation consuming the Elastic solution, and to which tenants can I allocate these costs?"**
+
+The integration provides a breakdown of Elastic Consumption Units (ECUs) per:
+
+- Deployment
+- Data tier
+- Data stream
+- Day
+
+Currently, Chargeback calculations consider only Elasticsearch data nodes. Contributions from other assets, like Kibana or ML nodes, are assumed to be shared proportionally among tenants. To incorporate indexing, querying, and storage in a weighted manner, a blended value is created using the following default weights (modifiable):
+- Indexing: `20` (applicable only to the hot tier)
+- Querying: `20`
+- Storage: `40`
+
+This default weighting means storage contributes most to the blended cost calculation, with indexing considered only on the hot tier. Adjust these weights based on your organisation's needs and best judgment.
+
+Chargeback is also present based on a configured rate and unit. These are used to display cost in the local currency, for instance `EUR`, with a rate of `0.85`.
+
+All configuration values can be updated, as follows:
+
+```
+POST chargeback_conf_lookup/_update/config
+{
+  "doc": {
+    "conf_ecu_rate": 0.85,
+    "conf_ecu_rate_unit": "EUR",
+    "conf_indexing_weight": 50,
+    "conf_query_weight": 20,
+    "conf_storage_weight": 40
+  }
+}
+```
+
+Chargeback data can be viewed in the `[Chargeback] Cost and Consumption breakdown` dashboard.
+
+![Cost and Consumption breakdown](../img/chargeback.png)
+
+## Requirements
+
+To use this integration, the following prerequisites must be met:
+
+- The monitoring cluster, where this integration is installed, must be on version 8.18.0+ due to its use of [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join).
+- The [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) integration (v1.4.1+) must be installed and running.
+- The [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integration (v1.16.0+) must be installed and collecting [usage data](https://www.elastic.co/docs/reference/integrations/elasticsearch/#indices-and-data-streams-usage-analysis) from all relevant deployments.
+- The Transform named `logs-elasticsearch.index_pivot-default-{VERSION}` must be running, which is an asset of the **Elasticsearch** integration.
+
+This integration must be installed on the **Monitoring cluster** where the above mentioned relevant usage and billing data is collected.

packages/chargeback/elasticsearch/transform/billing_cluster_cost/fields/fields.yml

@@ -0,0 +1,15 @@
+- name: "@timestamp"
+  type: date
+  description: The timestamp representing the day of the billing data.
+- name: deployment_id
+  type: keyword
+  description: Unique ID of the deployment.
+- name: deployment_name
+  type: keyword
+  description: Human-readable name of the deployment.
+- name: total_ecu
+  type: double
+  description: Total ECU usage aggregated for the deployment per day.
+- name: composite_key
+  type: keyword
+  description: Composite key used for billing attribution (set by ingest pipeline) consisting of date and deployment ID.

packages/chargeback/elasticsearch/transform/billing_cluster_cost/manifest.yml

@@ -0,0 +1,8 @@
+start: true
+destination_index_template:
+  settings:
+    index:
+#      mode: "lookup"  
+      codec: best_compression
+  mappings:
+    dynamic: false

packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml

@@ -0,0 +1,42 @@
+description: Aggregates daily total ECU usage per deployment from billing metrics, using ingested timestamps with a 1-hour sync delay and running every 60 minutes.
+source:
+  index:
+    - metrics-ess_billing.billing-*
+  query:
+    range:
+      ess.billing.total_ecu:
+        gt: 0
+dest:
+  index: billing_cluster_cost_lookup
+  pipeline: metrics-chargeback.billing-0.1.7
+frequency: 60m
+sync:
+  time:
+    field: event.ingested
+    delay: 1h
+pivot:
+  group_by:
+    "@timestamp":
+      date_histogram:
+        field: "@timestamp"
+        calendar_interval: 1d
+    deployment_id:
+      terms:
+        field: ess.billing.deployment_id
+    deployment_name:
+      terms:
+        field: ess.billing.deployment_name
+  aggregations:
+    total_ecu:
+      sum:
+        field: ess.billing.total_ecu
+settings:
+  # This is required to prevent the transform from clobbering the Fleet-managed mappings.
+  deduce_mappings: false
+  unattended: true
+_meta:
+  managed: true
+  run_as_kibana_system: false
+  # Bump this version to delete, reinstall, and restart the transform during package.
+  # Version bump is needed if there is any code change in transform.
+  fleet_transform_version: 0.1.7

packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/fields/fields.yml

@@ -0,0 +1,24 @@
+- name: "@timestamp"
+  type: date
+  description: Daily timestamp representing when the metric was recorded.
+- name: cluster_name
+  type: keyword
+  description: Name of the Elasticsearch cluster.
+- name: datastream
+  type: keyword
+  description: The name of the data stream.
+- name: datastream_sum_indexing_time
+  type: long
+  description: Total indexing time in milliseconds for the data stream.
+- name: datastream_sum_query_time
+  type: long
+  description: Total query time in milliseconds for the data stream.
+- name: datastream_sum_store_size
+  type: long
+  description: Total primary shard store size in bytes for the data stream.
+- name: datastream_sum_data_set_store_size
+  type: long
+  description: Total dataset size in bytes for the data stream.
+- name: composite_key
+  type: keyword
+  description: Composite key used for billing attribution (set by ingest pipeline) consisting of date and deployment ID.

packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/manifest.yml

@@ -0,0 +1,8 @@
+start: false
+destination_index_template:
+  settings:
+    index:
+#     mode: "lookup"  
+      codec: best_compression
+  mappings:
+    dynamic: false