Skip to content

Commit 2ed3279

Browse files
committed
Support dynamic package construction
1 parent 98f2bc8 commit 2ed3279

File tree

5 files changed

+210
-37
lines changed

5 files changed

+210
-37
lines changed

README.md

Lines changed: 107 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -50,33 +50,6 @@ scripts:
5050
- ...
5151
```
5252

53-
## Package
54-
A package is an entry in a `BUILD.yaml` in the `packages` section. All packages share the following fields:
55-
```YAML
56-
# name is the component-wide unique name of this package
57-
name: must-not-contain-spaces
58-
# Package type must be one of: go, yarn, docker, generic
59-
type: generic
60-
# Sources list all sources of this package. Entries can be double-star globs and are relative to the component root.
61-
# Avoid listing sources outside the component folder.
62-
srcs:
63-
- "**/*.yaml"
64-
- "glob/**/path"
65-
# Deps list dependencies to other packages which must be built prior to building this package. How these dependencies are made
66-
# available during build depends on the package type.
67-
deps:
68-
- some/other:package
69-
# Argdeps makes build arguments version relevant. I.e. if the value of a build arg listed here changes, so does the package version.
70-
argdeps:
71-
- someBuildArg
72-
# Env is a list of key=value pair environment variables available during package build
73-
env:
74-
- CGO_ENABLED=0
75-
# Config configures the package build depending on the package type. See below for details
76-
config:
77-
...
78-
```
79-
8053
## Script
8154
Scripts are a great way to automate tasks during development time (think [`yarn scripts`](https://classic.yarnpkg.com/en/docs/package-json#toc-scripts)).
8255
Unlike packages they do not run in isolation by default, but have access to the original workspace.
@@ -113,13 +86,32 @@ script: |
11386
echo "build args work to: ${myBuildArg}"
11487
```
11588
116-
### Build arguments
117-
118-
In a package definition one can use _build arguments_. Build args have the form of `${argumentName}` and are string-replaced when the package is loaded.
119-
**It's advisable to use build args only within the `config` section of packages**. Constants and built-in build args do not even work outside of the config section.
120-
121-
Leeway supports built-in build arguments:
122-
- `__pkg_version` resolves to the leeway version hash of a component.
89+
## Package
90+
A package is an entry in a `BUILD.yaml` in the `packages` section. All packages share the following fields:
91+
```YAML
92+
# name is the component-wide unique name of this package
93+
name: must-not-contain-spaces
94+
# Package type must be one of: go, yarn, docker, generic
95+
type: generic
96+
# Sources list all sources of this package. Entries can be double-star globs and are relative to the component root.
97+
# Avoid listing sources outside the component folder.
98+
srcs:
99+
- "**/*.yaml"
100+
- "glob/**/path"
101+
# Deps list dependencies to other packages which must be built prior to building this package. How these dependencies are made
102+
# available during build depends on the package type.
103+
deps:
104+
- some/other:package
105+
# Argdeps makes build arguments version relevant. I.e. if the value of a build arg listed here changes, so does the package version.
106+
argdeps:
107+
- someBuildArg
108+
# Env is a list of key=value pair environment variables available during package build
109+
env:
110+
- CGO_ENABLED=0
111+
# Config configures the package build depending on the package type. See below for details
112+
config:
113+
...
114+
```
123115

124116
### Go packages
125117
```YAML
@@ -200,6 +192,87 @@ config:
200192
- ["sh", "-c", "ls *"]
201193
```
202194
195+
## Dynaimc package scripts
196+
Packages can be dynamically produced within a component using a dynamic package script named `BUILD.js`. This ECMAScript 5.1 file is executed using [Goja](https://github.com/dop251/goja) and produces a `packages` array which contains the package struct much like they'd exist within the `BUILD.yaml`. For example:
197+
198+
199+
<table>
200+
<tr>
201+
<td valign="top">
202+
203+
`BUILD.js` file
204+
205+
```JavaScript
206+
let packages = [];
207+
208+
let deps = [];
209+
for(let i = 0; i < 5; i++) {
210+
const name = "hello-"+i;
211+
deps.push(name);
212+
packages.push({
213+
name: name,
214+
type: "generic",
215+
config: {
216+
commands: [
217+
["echo", "hello from "+i]
218+
]
219+
}
220+
});
221+
}
222+
223+
packages.push({
224+
name: "all",
225+
type: "generic",
226+
deps: deps.map(d => ":" + d),
227+
})
228+
```
229+
</td>
230+
<td>
231+
232+
Equivalent `BUILD.yaml`
233+
```YAML
234+
pacakages:
235+
- name: all
236+
type: generic
237+
deps:
238+
- hello-1
239+
- hello-2
240+
- hello-3
241+
- hello-4
242+
- hello-5
243+
- name: hello-1
244+
type: generic
245+
config:
246+
commands:
247+
- ["echo", "hello from 1"]
248+
- name: hello-2
249+
type: generic
250+
config:
251+
commands:
252+
- ["echo", "hello from 2"]
253+
- name: hello-3
254+
type: generic
255+
config:
256+
commands:
257+
- ["echo", "hello from 3"]
258+
...
259+
```
260+
261+
</td>
262+
</tr>
263+
</table>
264+
265+
> **Note** that for a `BUILD.js` to become effective/be recodnized there needs to a (possibly empty) `BUILD.yaml` in the same directory.
266+
267+
## Build arguments
268+
269+
In a package definition one can use _build arguments_. Build args have the form of `${argumentName}` and are string-replaced when the package is loaded.
270+
**It's advisable to use build args only within the `config` section of packages**. Constants and built-in build args do not even work outside of the config section.
271+
272+
Leeway supports built-in build arguments:
273+
- `__pkg_version` resolves to the leeway version hash of a component.
274+
- `__git_commit` contains the current Git commit if the build is executed from within a Git working copy. If this variable is used and the build is not executed from within a Git working copy the variable resolution will fail.
275+
203276
## Package Variants
204277
Leeway supports build-time variance through "package variants". Those variants are defined on the workspace level and can modify the list of sources, environment variables and config of packages.
205278
For example consider a `WORKSPACE.YAML` with this variants section:

fixtures/pkgs/generic/BUILD.js

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
let packages = [];
2+
3+
let deps = [];
4+
for(let i = 0; i < 10; i++) {
5+
const name = "hello-"+i;
6+
deps.push(name);
7+
packages.push({
8+
name: name,
9+
type: "generic",
10+
config: {
11+
commands: [
12+
["echo", "hello from "+i]
13+
]
14+
}
15+
});
16+
}
17+
18+
packages.push({
19+
name: "all",
20+
type: "generic",
21+
deps: deps.map(d => ":" + d),
22+
})

go.mod

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,17 +27,21 @@ require (
2727
sigs.k8s.io/bom v0.1.0
2828
)
2929

30+
require github.com/dop251/goja v0.0.0-20220815083517-0c74f9139fd6
31+
3032
require (
3133
github.com/Microsoft/go-winio v0.4.16 // indirect
3234
github.com/ProtonMail/go-crypto v0.0.0-20210428141323-04723f9f07d7 // indirect
3335
github.com/acomagu/bufpipe v1.0.3 // indirect
3436
github.com/coreos/go-systemd/v22 v22.3.2 // indirect
3537
github.com/cyphar/filepath-securejoin v0.2.3 // indirect
38+
github.com/dlclark/regexp2 v1.7.0 // indirect
3639
github.com/emirpasic/gods v1.12.0 // indirect
3740
github.com/fatih/color v1.12.0 // indirect
3841
github.com/go-git/gcfg v1.5.0 // indirect
3942
github.com/go-git/go-billy/v5 v5.3.1 // indirect
4043
github.com/go-git/go-git/v5 v5.4.2 // indirect
44+
github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirect
4145
github.com/godbus/dbus/v5 v5.0.6 // indirect
4246
github.com/inconshreveable/mousetrap v1.0.0 // indirect
4347
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
@@ -59,7 +63,7 @@ require (
5963
golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b // indirect
6064
golang.org/x/net v0.0.0-20211111160137-58aab5ef257a // indirect
6165
golang.org/x/sys v0.0.0-20211116061358-0a5406a5449c // indirect
62-
golang.org/x/text v0.3.6 // indirect
66+
golang.org/x/text v0.3.7 // indirect
6367
golang.org/x/tools v0.1.7 // indirect
6468
gopkg.in/warnings.v0 v0.1.2 // indirect
6569
sigs.k8s.io/release-utils v0.3.0 // indirect

go.sum

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,15 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
9797
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
9898
github.com/disiqueira/gotree v1.0.0 h1:en5wk87n7/Jyk6gVME3cx3xN9KmUCstJ1IjHr4Se4To=
9999
github.com/disiqueira/gotree v1.0.0/go.mod h1:7CwL+VWsWAU95DovkdRZAtA7YbtHwGk+tLV/kNi8niU=
100+
github.com/dlclark/regexp2 v1.4.1-0.20201116162257-a2a8dda75c91/go.mod h1:2pZnwuY/m+8K6iRw6wQdMtk+rH5tNGR1i55kozfMjCc=
101+
github.com/dlclark/regexp2 v1.7.0 h1:7lJfhqlPssTb1WQx4yvTHN0uElPEv52sbaECrAQxjAo=
102+
github.com/dlclark/regexp2 v1.7.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
100103
github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
104+
github.com/dop251/goja v0.0.0-20211022113120-dc8c55024d06/go.mod h1:R9ET47fwRVRPZnOGvHxxhuZcbrMCuiqOz3Rlrh4KSnk=
105+
github.com/dop251/goja v0.0.0-20220815083517-0c74f9139fd6 h1:xHdUVG+c8SWJnct16Z3QJOVlaYo3OwoJyamo6kR6OL0=
106+
github.com/dop251/goja v0.0.0-20220815083517-0c74f9139fd6/go.mod h1:yRkwfj0CBpOGre+TwBsqPV0IH0Pk73e4PXJOeNDboGs=
107+
github.com/dop251/goja_nodejs v0.0.0-20210225215109-d91c329300e7/go.mod h1:hn7BA7c8pLvoGndExHudxTDKZ84Pyvv+90pbBjbTz0Y=
108+
github.com/dop251/goja_nodejs v0.0.0-20211022123610-8dd9abb0616d/go.mod h1:DngW8aVqWbuLRMHItjPUyqdj+HWPvnQe8V8y1nDpIbM=
101109
github.com/dsnet/compress v0.0.1/go.mod h1:Aw8dCMJ7RioblQeTqt88akK31OvO8Dhf5JflhBbQEHo=
102110
github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY=
103111
github.com/emirpasic/gods v1.12.0 h1:QAUIPSaCu4G+POclxeqb3F+WPpdKqFGlw36+yOzGlrg=
@@ -134,6 +142,8 @@ github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9
134142
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
135143
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
136144
github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas=
145+
github.com/go-sourcemap/sourcemap v2.1.3+incompatible h1:W1iEw64niKVGogNgBN3ePyLFfuisuzeidWPMPWmECqU=
146+
github.com/go-sourcemap/sourcemap v2.1.3+incompatible/go.mod h1:F8jJfvm2KbVjc5NqelyYJmf/v5J0dwNLS2mL4sNA1Jg=
137147
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
138148
github.com/godbus/dbus/v5 v5.0.6 h1:mkgN1ofwASrYnJ5W6U/BxG15eXXXjirgZc7CLqkcaro=
139149
github.com/godbus/dbus/v5 v5.0.6/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
@@ -256,8 +266,9 @@ github.com/klauspost/pgzip v1.2.4/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQ
256266
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
257267
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
258268
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
259-
github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
260269
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
270+
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
271+
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
261272
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
262273
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
263274
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -326,6 +337,8 @@ github.com/praetorian-inc/gokart v0.3.0/go.mod h1:P/ADPPKodL3GOPTpA9eeyAmtfQfu0/
326337
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
327338
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
328339
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
340+
github.com/rogpeppe/go-internal v1.6.1 h1:/FiVV8dS/e+YqF2JvO3yXRFbBLTIuSDkuC7aBOAvL+k=
341+
github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
329342
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
330343
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
331344
github.com/sclevine/spec v1.4.0/go.mod h1:LvpgJaFyvQzRvc1kaDs0bulYwzC70PbiYjC4QnFHkOM=
@@ -586,8 +599,9 @@ golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
586599
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
587600
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
588601
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
589-
golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M=
590602
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
603+
golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
604+
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
591605
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
592606
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
593607
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=

pkg/leeway/workspace.go

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ import (
1919
"sync"
2020
"time"
2121

22+
"github.com/dop251/goja"
2223
"github.com/imdario/mergo"
2324
"github.com/in-toto/in-toto-golang/in_toto"
2425
"github.com/minio/highwayhash"
@@ -656,6 +657,36 @@ func loadComponent(ctx context.Context, workspace *Workspace, path string, args
656657
err = nil
657658
}
658659

660+
builderFN := strings.TrimSuffix(path, ".yaml") + ".js"
661+
if _, err := os.Stat(builderFN); err == nil {
662+
addFC, err := runPackageBuilder(builderFN)
663+
if err != nil {
664+
return Component{}, err
665+
}
666+
for _, p := range addFC {
667+
fc, err := yaml.Marshal(p)
668+
if err != nil {
669+
return Component{}, err
670+
}
671+
log.WithField("fc", string(fc)).WithField("component", comp.Name).Debug("adding dynamic package")
672+
673+
var nd yaml.Node
674+
err = yaml.Unmarshal(fc, &nd)
675+
if err != nil {
676+
return Component{}, err
677+
}
678+
679+
var pkg Package
680+
err = yaml.Unmarshal(fc, &pkg)
681+
if err != nil {
682+
return Component{}, err
683+
}
684+
685+
comp.Packages = append(comp.Packages, &pkg)
686+
rawcomp.Packages = append(rawcomp.Packages, nd)
687+
}
688+
}
689+
659690
for i, pkg := range comp.Packages {
660691
pkg.C = &comp
661692
if pkg.Type == "typescript" {
@@ -864,3 +895,32 @@ func mergeEnv(pkg *Package, src []string) error {
864895
}
865896
return nil
866897
}
898+
899+
func runPackageBuilder(fn string) (fc []map[string]interface{}, err error) {
900+
defer func() {
901+
if err != nil {
902+
err = fmt.Errorf("failed to run package builder script at %s: %w", fn, err)
903+
}
904+
}()
905+
906+
prog, err := os.ReadFile(fn)
907+
if err != nil {
908+
return nil, err
909+
}
910+
911+
vm := goja.New()
912+
_, err = vm.RunString(string(prog))
913+
if err != nil {
914+
return nil, err
915+
}
916+
917+
var res []map[string]interface{}
918+
err = vm.ExportTo(vm.Get("packages"), &res)
919+
if err != nil {
920+
return nil, err
921+
}
922+
923+
log.WithField("res", res).WithField("fn", fn).Debug("ran package builder script")
924+
925+
return res, nil
926+
}

0 commit comments

Comments
 (0)