Add replace command with Calcite #4451

ahkcs · 2025-10-07T21:14:09Z

Description

Implement the replace command in PPL to replace text patterns in specified fields. This PR includes the grammar implementation and basic replacement functionality. This implementation reuses the existing replace core logic. Missing / new features for regex support will be added in a separate PR.

Original PR: #4248

Syntax

<source> | replace '<pattern>' WITH '<replacement>' IN field_list

pattern: The text pattern to search for (case-sensitive)
replacement: The text to replace matches with
field_list: Comma-separated list of fields to perform replacement in
The replace command modifies the specified fields in-place with the replaced text.

Semantics

Expected Behavior

Action: Modifies specified fields in-place with replaced text values
Scope: Operates only on the specified fields
Data Modification: Specified fields are updated with replacement text
Case Sensitivity: Text literal matching is case-sensitive
Pattern Type: Only supports literal string patterns (wildcards or regex support added later)

Implementation Approach

Modifies specified fields in-place using string replacement
Performs literal string replacement in the specified fields
Other fields remain unchanged
Validates the existence of fields and correctness of pattern / replacement parameters

Example Queries

-- Replace in a single field  
source=logs | replace 'error' WITH 'ERROR' IN message  

-- Replace in multiple fields  
source=logs | replace 'USA' WITH 'United States' IN country, state  

-- Combine with other commands  
source=logs  
  | where level = 'error'  
  | replace 'error' WITH 'ERROR' IN message  
  | sort @timestamp  

-- Replace and select specific fields  
source=logs | replace 'error' WITH 'ERROR' IN message | fields message, level

Output Schema

For each field specified in the IN clause, the field is modified in-place:

Specified fields: modified with replacement text
Other fields: remain unchanged

Example
Input:

message = "error occurred", level = "error"

After replace 'error' WITH 'ERROR' IN message, level →

message = "ERROR occurred"  
level = "ERROR"

Resolves

#3975

Check List

New functionality includes tests
New functionality has documentation
Javadoc added for new components
User manual / user documentation updated
PPL command checklist confirmed
Companion API changes PR created (if applicable)
Commits are signed per DCO (--signoff or -s)
Public documentation / issue or PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Swiddis · 2025-10-07T22:43:23Z

core/src/main/java/org/opensearch/sql/ast/tree/Replace.java

+@ToString
+@EqualsAndHashCode(callSuper = false)
+public class Replace extends UnresolvedPlan {
+  private final UnresolvedExpression pattern;


suggestion: We should annotate these with @NotNull.

Changed to @Nullable, received similar comment before: #3878 (comment)

Fwiw, I also prefer using optional in fields, despite the warning. I'm not sure why that warning exists. 🤷

But why mark it nullable if we validate that they can't be null?

We marked child as @nullable, which we didn't include in the validate logic

Swiddis · 2025-10-07T22:48:00Z

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

+    UnresolvedExpression pattern = internalVisitExpression(ctx.pattern);
+    UnresolvedExpression replacement = internalVisitExpression(ctx.replacement);
+
+    List<Field> fieldList =


suggestion: Should we have a global method for turning this into a LinkedHashSet?

This is currently checked in validate(), but I don't think this is the only command to rely on a fieldList. If we make Replace take a Set instead of a List, we simplify the validation logic there, and we can push the responsibility of deduplicating fieldLists to shared parsing code.

Updated to accept Set instead of List, removing duplicate validation logic

Swiddis · 2025-10-07T22:49:37Z

ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLReplaceTest.java

+    verifyPPLToSparkSQL(root, expectedSparkSql);
+  }
+
+  @Test(expected = Exception.class)


nit: Catch a more specific Exception

updated to be more specific

Signed-off-by: Kai Huang <[email protected]>

ykmr1224 · 2025-10-14T17:20:30Z

core/src/main/java/org/opensearch/sql/ast/tree/Replace.java

+  @Deprecated
+  public UnresolvedExpression getReplacement() {
+    if (replacePairs.isEmpty()) {
+      throw new IllegalStateException("No replacement pairs available");
+    }
+    return replacePairs.get(0).getReplacement();
+  }


Let's delete instead of deprecate.

ykmr1224 · 2025-10-14T17:20:48Z

core/src/main/java/org/opensearch/sql/ast/tree/Replace.java

+  @Deprecated
+  public UnresolvedExpression getPattern() {
+    if (replacePairs.isEmpty()) {
+      throw new IllegalStateException("No replacement pairs available");
+    }
+    return replacePairs.get(0).getPattern();
+  }


Let's delete instead of deprecate.

ykmr1224 · 2025-10-14T17:23:07Z

core/src/main/java/org/opensearch/sql/ast/tree/Replace.java

+  public Replace(
+      UnresolvedExpression pattern, UnresolvedExpression replacement, Set<Field> fieldList) {


Is this used?

Updated to remove unused parameter

dai-chen

Thanks for the changes!

Signed-off-by: Kai Huang <[email protected]>

ykmr1224 · 2025-10-16T00:04:51Z

core/src/main/java/org/opensearch/sql/ast/tree/ReplacePair.java

+@Getter
+@AllArgsConstructor
+@EqualsAndHashCode
+@ToString


optional: consider using @Value

opensearch-trigger-bot · 2025-10-16T18:21:15Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4451-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5677765e6d2f0203fc99014e5ebc4aa27424b57d
# Push it to GitHub
git push --set-upstream origin backport/backport-4451-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4451-to-2.19-dev.

* Add replace command with Calcite Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Co-authored-by: Manasvini B S <[email protected]> (cherry picked from commit 5677765) Signed-off-by: Kai Huang <[email protected]>

* Add replace command with Calcite (#4451) * Add replace command with Calcite Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Co-authored-by: Manasvini B S <[email protected]> (cherry picked from commit 5677765) Signed-off-by: Kai Huang <[email protected]> * fix compile Signed-off-by: Kai Huang <[email protected]> * backporting backslash handling from main and fix tests Signed-off-by: Kai Huang <[email protected]> * Fix tests Signed-off-by: Kai Huang <[email protected]> * fix tests Signed-off-by: Kai Huang <[email protected]> * compatability accross java versions Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Co-authored-by: Manasvini B S <[email protected]>

ahkcs force-pushed the feat/replace_cmd branch from 3621bc9 to b4fe742 Compare October 7, 2025 21:19

ahkcs marked this pull request as ready for review October 7, 2025 21:23

Swiddis reviewed Oct 7, 2025

View reviewed changes

Swiddis added the enhancement New feature or request label Oct 7, 2025

RyanL1997 previously approved these changes Oct 9, 2025

View reviewed changes

ahkcs dismissed RyanL1997’s stale review via 414a9f8 October 9, 2025 18:03

ahkcs force-pushed the feat/replace_cmd branch from 13fa309 to 414a9f8 Compare October 9, 2025 18:03

ahkcs added 12 commits October 15, 2025 13:00

fixes

a2df2fc

Signed-off-by: Kai Huang <[email protected]>

fix

af95e74

Signed-off-by: Kai Huang <[email protected]>

change new_ handling to in-place replacement

638ae34

Signed-off-by: Kai Huang <[email protected]>

update doctest

d099aed

Signed-off-by: Kai Huang <[email protected]>

update explain

72defe9

Signed-off-by: Kai Huang <[email protected]>

update test

9d1a2fc

Signed-off-by: Kai Huang <[email protected]>

fixes

b6fe452

Signed-off-by: Kai Huang <[email protected]>

doc update

d5beab7

Signed-off-by: Kai Huang <[email protected]>

update comma support

38eeb2f

Signed-off-by: Kai Huang <[email protected]>

remove validation logic since enforced by antlr

c0d7ffe

Signed-off-by: Kai Huang <[email protected]>

update doc

4b4217e

Signed-off-by: Kai Huang <[email protected]>

update validation logic

0ff7d97

Signed-off-by: Kai Huang <[email protected]>

ahkcs force-pushed the feat/replace_cmd branch from 8d93683 to 0ff7d97 Compare October 15, 2025 20:00

ykmr1224 reviewed Oct 15, 2025

View reviewed changes

dai-chen previously approved these changes Oct 15, 2025

View reviewed changes

update

e49d99f

Signed-off-by: Kai Huang <[email protected]>

ahkcs dismissed dai-chen’s stale review via e49d99f October 15, 2025 22:48

ahkcs requested review from Swiddis, dai-chen and ykmr1224 October 15, 2025 23:29

ykmr1224 approved these changes Oct 16, 2025

View reviewed changes

dai-chen approved these changes Oct 16, 2025

View reviewed changes

dai-chen added PPL Piped processing language backport 2.19-dev labels Oct 16, 2025

dai-chen merged commit 5677765 into opensearch-project:main Oct 16, 2025
37 checks passed

opensearch-trigger-bot bot added the backport-failed label Oct 16, 2025

LantaoJin added the backport-manually Filed a PR to backport manually. label Oct 23, 2025

		public Replace(
		UnresolvedExpression pattern, UnresolvedExpression replacement, Set<Field> fieldList) {

Uh oh!

Add replace command with Calcite #4451

Add replace command with Calcite #4451

Uh oh!

Conversation

ahkcs commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Syntax

Semantics

Expected Behavior

Implementation Approach

Example Queries

Output Schema

Resolves

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swiddis Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swiddis Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ahkcs commented Oct 7, 2025 •

edited

Loading

Swiddis Oct 7, 2025 •

edited

Loading

Swiddis Oct 7, 2025 •

edited

Loading