Fix template path validation including linked files #1009

teresaromero · 2025-10-31T10:32:31Z

What does this PR do?

Includes a fix at the integration packages template path validation, taking into account the existence of linked files.
A template can be shared accross data streams by using a shared file and at the agent/stream file, place a .link.

The validation previously implemented did not take into account this occurence, as the iterator was looking for specific file names. It has been spotted while running the integration tests with elastic-package, as the test case for with_links package was correctly mapped.

Why it was not detected at package-spec tests? The package representing a package with link files was not taking into account that streams are linked with the manifest policy template declaration. This PR modifies the package so the test is reliable and spots this occurance.

Why is it important?

When linked files are used to share data streams templates, the validation should be able to check if the file exists, and the final "included" file exists too.

The fs handler already has embedded the link files system, so when a file is .link it looks for the included directly.

package-spec/code/go/internal/linkedfiles/fs.go

Line 45 in 1bd98cf

l, err := NewLinkedFile(pathName)

Checklist

tested replacing package-spec at elastic-package and testing the error

replace github.com/elastic/package-spec/v3 v3.5.0 => ../package-spec

I have added test packages to test/packages that prove my change is effective.
I have added an entry in spec/changelog.yml.

Related issues

Related #703
Bug introduced at #1002

…package policy_template paths with linked files

teresaromero · 2025-10-31T10:41:04Z

Shall this also be check at input packages? 🤔 can the template be a link file at agent/input?

I can't seem to find in the spec the link allowance parameter for inputs. The spec does not have an "agent" folder

mrodm · 2025-10-31T16:13:09Z

Shall this also be check at input packages? 🤔 can the template be a link file at agent/input?

I can't seem to find in the spec the link allowance parameter for inputs. The spec does not have an "agent" folder

Input packages use the same definition for agent folder as the integration packages:

package-spec/spec/input/spec.yml

Lines 33 to 37 in b9e2c04

    
           - description: Folder containing agent-related definitions 
        
             type: folder 
        
             name: agent 
        
             required: true 
        
             $ref: "../integration/agent/spec.yml"

And in that file it is allowed link files:

package-spec/spec/integration/agent/spec.yml

Lines 10 to 15 in b9e2c04

    
           - description: Config template file for inputs defined in the policy_templates section of the top level manifest 
        
             type: file 
        
             sizeLimit: 2MB 
        
             pattern: '^.+\.yml\.hbs$' 
        
             required: true 
        
             allowLink: true

So, it should be also added for input packages.

mrodm · 2025-10-31T16:09:01Z

code/go/internal/validator/semantic/validate_integration_policy_template_path.go

+
+// findPathWithPattern looks for a file matching the templatePath in the given data stream directory
+// It checks for exact matches, files ending with the templatePath, or templatePath + ".link"
+func findPathWithPattern(fsys fspath.FS, dsDir, templatePath string) (string, error) {


Another option here is using directly the fsys that will iterate over the linked files already "rendered".

package-spec/code/go/pkg/validator/validator.go

Lines 53 to 62 in b9e2c04

func ValidateFromFS(location string, fsys fs.FS) error {

// If we are not explicitly using the linkedfiles.FS, we wrap fsys with

// a linkedfiles.BlockFS to block the use of linked files.

if _, ok := fsys.(*linkedfiles.FS); !ok {

fsys = linkedfiles.NewBlockFS(fsys)

}

pkg, err := packages.NewPackageFromFS(location, fsys)

if err != nil {

return err

}

So maybe it could be used fs.WalkDir and look for the template paths without checking explictely for *.link. Maybe something like this (it needs to be tested)?

func findPathWithPattern(fsys fspath.FS, dsDir, templatePath string) (string, error) { mainPath := path.Join(dsDir, "agent", "stream") foundFile := "" err := fs.WalkDir(fsys, mainPath, func(p string, d fs.DirEntry, walkErr error) error { if walkErr != nil { return walkErr } if d.IsDir() { return nil } base := filepath.Base(p) if base == templatePath || strings.HasSuffix(p, templatePath) { foundFile = base return fs.SkipAll // found a match, stop walking } return nil }) if err != nil { return "", err } if foundFile == "" { return "", errTemplateNotFound } return foundFile, "" }

As a advantage, there is no need to check *.link files.
As a disadvantage, it requires to iterate over all files in the directory.

WDYT @jsoriano @teresaromero ?

Just checked locally and my proposed solution here does not work as I expected.

It should be added also more conditions to that if in order to work successfully:

if base == templatePath || base == templatePath+".link" || strings.HasSuffix(match, templatePath) || strings.HasSuffix(match, templatePath+".link") { foundFile = match break }

So there is no advantage of using WalkDir here.
The current implementation based on fs.Glob could be kept 👍

mrodm · 2025-10-31T16:16:47Z

code/go/internal/validator/semantic/validate_integration_policy_template_path.go

 			}

-			_, err := fs.ReadFile(fsys, path.Join(dsDir, "agent", "stream", stream.TemplatePath))
+			foundFile, err := findPathWithPattern(fsys, dsDir, stream.TemplatePath)


It looks like that these changes should be added also into validateAgentInputTemplatePath:

package-spec/code/go/internal/validator/semantic/validate_input_policy_template_template_path.go

Lines 87 to 98 in b9e2c04

func validateAgentInputTemplatePath(fsys fspath.FS, tmplPath string) error {

templatePath := path.Join("agent", "input", tmplPath)

_, err := fs.Stat(fsys, templatePath)

if err != nil {

if errors.Is(err, os.ErrNotExist) {

return errTemplateNotFound

}

return fmt.Errorf("failed to stat template file %s: %w", fsys.Path(templatePath), err)

}

return nil

}

since that path also allow linked files:

package-spec/spec/integration/agent/spec.yml

Lines 4 to 15 in b9e2c04

- description: Folder containing input definitions

type: folder

name: input

required: true

additionalContents: false

contents:

- description: Config template file for inputs defined in the policy_templates section of the top level manifest

type: file

sizeLimit: 2MB

pattern: '^.+\.yml\.hbs$'

required: true

allowLink: true

I've updated the PR with this change. Made the finder function more broad with the dir param instead of data stream dir. Updated also test cases

mrodm · 2025-11-03T12:24:01Z

code/go/internal/validator/semantic/validate_integration_policy_template_path.go

+
+// findPathWithPattern looks for a file matching the templatePath in the given data stream directory
+// It checks for exact matches, files ending with the templatePath, or templatePath + ".link"
+func findPathWithPattern(fsys fspath.FS, dsDir, templatePath string) (string, error) {


Just checked locally and my proposed solution here does not work as I expected.

It should be added also more conditions to that if in order to work successfully:

if base == templatePath || base == templatePath+".link" || strings.HasSuffix(match, templatePath) || strings.HasSuffix(match, templatePath+".link") { foundFile = match break }

So there is no advantage of using WalkDir here.
The current implementation based on fs.Glob could be kept 👍

mrodm · 2025-11-03T12:32:47Z

code/go/internal/validator/semantic/validate_integration_policy_template_path.go

+		if strings.HasSuffix(match, templatePath) {
+			foundFile = match
+			break
+		}


It should be added a new if to validate whether or not it exists a template path with the suffix templatePath+".link" too.

In the with_links test package from elastic-package, for instance there could be a file named:

test/packages/other/with_links/data_stream/first/agent/stream/filestream.yml.hbs.link

And this implementation would fail as:

Error: checking package failed: linting package failed: found 1 validation error: 1. file "/home/user/Coding/work/elastic-package-tere/test/packages/other/with_links/manifest.yml" is invalid: policy template "sample" references input template_path: error validating input from streams "logfile": template file not found

You could create a new variable templatePathWithLink or similar to store the path adding the extension .link.

var foundFile string templatePathWithLink := fmt.Sprintf("%s.link", templatePath) for _, match := range matches { base := filepath.Base(match) if base == templatePath || base == templatePathWithLink { foundFile = match break } // fallback to check for suffix match, in case the path is prefixed if strings.HasSuffix(match, templatePath) { foundFile = match break } if strings.HasSuffix(match, templatePathWithLink) { foundFile = match break } }

code/go/internal/validator/semantic/validate_integration_policy_template_path.go

spec/changelog.yml

Co-authored-by: Mario Rodriguez Molins <[email protected]>

…nsion

…e related tests

jsoriano

LGTM, added a couple of small comments.

code/go/internal/validator/semantic/validate_integration_policy_template_path.go

jsoriano · 2025-11-04T09:58:46Z

code/go/internal/validator/semantic/validate_integration_policy_template_path.go

+// findPathWithPattern looks for a file matching the templatePath in the given directory (dir)
+// It checks for exact matches, files ending with the templatePath, or templatePath + ".link"
+func findPathWithPattern(fsys fspath.FS, dir, templatePath string) (string, error) {


Nit. It would be more correct to call this findPathWithSuffix, right?

Suggested change

// findPathWithPattern looks for a file matching the templatePath in the given directory (dir)

// It checks for exact matches, files ending with the templatePath, or templatePath + ".link"

func findPathWithPattern(fsys fspath.FS, dir, templatePath string) (string, error) {

// findPathWithSuffix looks for a file ending with the suffix in the given directory (dir)

// It checks for exact matches, files ending with the suffix, or suffix + ".link"

func findPathWithSuffix(fsys fspath.FS, dir, suffix string) (string, error) {

i got the point for renaming, but was not sure about the suffix suggestion. The function also matches exact without being a suffix... i've renamed it to findPathAtDirectory... as the files should be inside the dir param ... WDYT?

…es in validation and test files

mrodm

LGTM!
Just a minor comment related to the changelog entry description.

spec/changelog.yml

Co-authored-by: Mario Rodriguez Molins <[email protected]>

elasticmachine · 2025-11-04T12:02:02Z

💚 Build Succeeded

Buildkite Build
Commit: cf02b4b

History

💚 Build #1574 succeeded 9b1f855
💔 Build #1573 failed 3e3ae3f
💚 Build #1572 succeeded de4afbd
💚 Build #1571 succeeded a1623bb
💚 Build #1565 succeeded 6045457
💔 Build #1564 failed 1bd98cf

Fix template path including linked files

1bd98cf

teresaromero requested a review from a team as a code owner October 31, 2025 10:32

teresaromero added 2 commits October 31, 2025 11:36

Refactor TestFindPathWithPattern to remove unnecessary blank lines

4263be4

Add changelog entry for version 3.5.2: Fix validation of integration …

6045457

…package policy_template paths with linked files

teresaromero mentioned this pull request Oct 31, 2025

Chore(deps): bump github.com/elastic/package-spec/v3 from 3.5.0 to 3.5.1 elastic/elastic-package#3041

Closed

mrodm reviewed Oct 31, 2025

View reviewed changes

mrodm reviewed Nov 3, 2025

View reviewed changes

teresaromero and others added 4 commits November 4, 2025 09:31

Apply suggestion from @mrodm

3888151

Co-authored-by: Mario Rodriguez Molins <[email protected]>

Apply suggestion from @mrodm

a1623bb

Co-authored-by: Mario Rodriguez Molins <[email protected]>

Enhance findPathWithPattern to support matching files with .link exte…

43c92ce

…nsion

Refactor findPathWithPattern to use directory path directly and updat…

de4afbd

…e related tests

jsoriano reviewed Nov 4, 2025

View reviewed changes

teresaromero added 3 commits November 4, 2025 11:12

Remove unnecessary read file operation

d362bd4

Rename findPathWithPattern to findPathAtDirectory and update referenc…

3e3ae3f

…es in validation and test files

Use path instead of filepath

9b1f855

mrodm previously approved these changes Nov 4, 2025

View reviewed changes

spec/changelog.yml Outdated Show resolved Hide resolved

Update spec/changelog.yml

cf02b4b

Co-authored-by: Mario Rodriguez Molins <[email protected]>

teresaromero dismissed mrodm’s stale review via cf02b4b November 4, 2025 11:47

jsoriano approved these changes Nov 4, 2025

View reviewed changes

mrodm approved these changes Nov 4, 2025

View reviewed changes

teresaromero merged commit 2183978 into elastic:main Nov 4, 2025
3 checks passed

jsoriano mentioned this pull request Nov 4, 2025

[Change Proposal] Release package spec code independently to package spec versions #1011

Open

	func ValidateFromFS(location string, fsys fs.FS) error {
	// If we are not explicitly using the linkedfiles.FS, we wrap fsys with
	// a linkedfiles.BlockFS to block the use of linked files.
	if _, ok := fsys.(*linkedfiles.FS); !ok {
	fsys = linkedfiles.NewBlockFS(fsys)
	}
	pkg, err := packages.NewPackageFromFS(location, fsys)
	if err != nil {
	return err
	}

	func validateAgentInputTemplatePath(fsys fspath.FS, tmplPath string) error {
	templatePath := path.Join("agent", "input", tmplPath)
	_, err := fs.Stat(fsys, templatePath)
	if err != nil {
	if errors.Is(err, os.ErrNotExist) {
	return errTemplateNotFound
	}
	return fmt.Errorf("failed to stat template file %s: %w", fsys.Path(templatePath), err)
	}

	return nil
	}

	- description: Folder containing input definitions
	type: folder
	name: input
	required: true
	additionalContents: false
	contents:
	- description: Config template file for inputs defined in the policy_templates section of the top level manifest
	type: file
	sizeLimit: 2MB
	pattern: '^.+\.yml\.hbs$'
	required: true
	allowLink: true

Fix template path validation including linked files #1009

Fix template path validation including linked files #1009

Uh oh!

Conversation

teresaromero commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Related issues

Uh oh!

teresaromero commented Oct 31, 2025

Uh oh!

mrodm commented Oct 31, 2025

Uh oh!

mrodm Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teresaromero Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jsoriano Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

teresaromero Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Nov 4, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

teresaromero commented Oct 31, 2025 •

edited

Loading

mrodm Oct 31, 2025 •

edited

Loading