Skip to content

Instantly share code, notes, and snippets.

View seifgamal's full-sized avatar

Seif Abouseada seifgamal

  • Celonis
  • Munich, Germany
View GitHub Profile
@seifgamal
seifgamal / tagvalues-augmentation-tables.md
Created May 4, 2026 21:33
celostar: tagValues propagation for augmentation tables

tagValues propagation for augmentation tables — session summary

Initial prompt

We want to return tagValues for augmentation tables (tagged at the column-level) in GET_SCHEMA calls. tagValues (column-level) are stored in ColumnDeclaration which can be found in common/src/main/protos/augmentation_table_ddl.proto; these are already stored in the DB as part of the CreateRequest payload.

Goal:

  1. Write an integration test in ComputeApiLiveDataModelTest.java to make sure that when we create an augmentation table and call getLiveDataModel, we get the expected tagValues.
  2. Change what is needed in the mappers and make sure the tests added in the first step are passing.
@seifgamal
seifgamal / CPL-15401-design-plan-response.md
Created April 29, 2026 07:48
CPL-15401 — Reject queries referencing broken augmentation tables: initial prompt, design plan, and follow-up summary

Design: Reject queries that reference broken augmentation tables

1. Background

AugmentationValidator (AugmentationValidator.java) already wraps cloud.celonis.compute2.augmentation.validation.AugmentationTableValidator. Today its single consumer is validateLoad(DataModelLoad), called from DataModelLoadDAO after a successful load (DataModelLoadDAO.java:536) to surface warnings such as "Querying this augmentation table will not work until the data model schema inconsistency is fixed." Those warnings are stored on DataModelLoad.warningMessages but never block queries — once a load succeeds the broken augmentation tables remain visible in the cube's AUGMENTATION_TABLE set (AugmentationCompilers.java:18) and any query referencing them will be compiled and dispatched to the engine.

We want to add a new use of the same broken-tables computation: **reject queries that reference any b

CPL-15884: Routing Table Queue Burst Mitigation -- Design Plan

Current Architecture

The service uses a push-based mechanism via RabbitMQ to synchronize in-memory routing table caches across DMM instances. This is an optimization on top of a pull-based cron fallback (every 15s). When any instance changes the routing table (e.g., during a data model migration), it broadcasts a RoutingTableUpdateRequestMessage to all other instances via a direct exchange. Each instance has its own per-instance queue (data-model-manager.routing-table-update-req-<instanceId>), non-durable and auto-delete.

refreshRoutingTable() uses a watermark-based DB query (fetchRecentUpdates(latestUpdateTimestamp)) that fetches all changes since the last refresh. A single call after a burst captures everything. All subsequent calls for queued messages hit the DB only to find zero new rows.

Production Data (Datadog, last month)

@seifgamal
seifgamal / CPL-16200-support-generic-labeling-in-saola-2-chat-summary.md
Created April 10, 2026 14:58
Chat summary: CPL-16200 support generic labeling in saola 2

Chat Summary — CPL-16200: Support Generic Labeling in Saola 2


1. Remove ai_generated from protos and DataColumnTransport.TAG_AI_GENERATED

Prompt:

loading.proto and queries.proto should not have ai_generated, remove them and reserve the field number to make sure it will not be used in the future, see where these are referenced and delete. Also DataColumnTransport.TAG_AI_GENERATED should be removed.

Actions taken:

@seifgamal
seifgamal / CPL-16200-support-generic-labeling-in-saola-2.md
Created April 10, 2026 14:57
PR description: CPL-16200 support generic labeling in saola 2

Replace aiGenerated boolean with generic tagValues map

Summary

This PR replaces the specific aiGenerated boolean flag on augmentation columns with a generic tagValues: Map<String, String> mechanism. This allows arbitrary metadata to be attached to columns without requiring dedicated fields for each use case.

Motivation

The aiGenerated flag was a narrow, single-purpose field. As more column metadata needs arise, adding a new boolean for each would lead to API and schema bloat. A generic key-value tag map is more flexible and future-proof.

@seifgamal
seifgamal / pr-description.md
Created April 10, 2026 10:51
PR #2: Replace aiGenerated with tagValues on ColumnEntity — Persistence Changes

PR #2: Persistence Changes — Replace aiGenerated with tagValues on ColumnEntity

Summary

Replace the single-purpose aiGenerated: Boolean persistence field on ColumnEntity with a general-purpose Map<String, String> tagValues stored as JSONB, completing the persistence layer migration for column-level tagging.

This is PR #2 in a 3-part series:

  • PR #1: API contract changes (column + table level) — merged
  • PR #2: Persistence changes (column only) — this PR
  • PR #3: Protobuf cleanup — pending
@seifgamal
seifgamal / cpl-16151-tag-key-refactor.md
Last active April 2, 2026 10:35
CPL-16151 — Rename aiGenerated tag key to CELONIS_RESERVED_AI_GENERATED

CPL-16151 — Rename aiGenerated tag key to CELONIS_RESERVED_AI_GENERATED


Prompt 1

In DataModelMapper.java & AugmentationExtensionHandler.java, there are mentions of:

getTagValues().put("aiGenerated", String.valueOf(column.getAiGenerated()));