Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ test/lint/fix:

test/unit:
go test -race ./...
go test ./internal/generator/cli/confirm -run '' ./internal/generator/cli/confirm/confirm_race_off_test.go

test/cover: module=./...
test/cover:
Expand Down
8 changes: 6 additions & 2 deletions doc/en/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,13 +177,17 @@ Structure `output.params` for format `csv`:
- `datetime_format`: Date-time format. Default is `2006-01-02T15:04:05Z07:00`.
- `without_headers`: Flag indicating if CSV headers should be excluded from data files.
- `delimiter`: Single-character CSV delimiter. Default is `,`.
- `partition_files_limit`: Limit on the number of partition files, upon reaching which a prompt will appear asking whether to continue.
Ignored if the `--force` flag is specified. Default is `1000`.

Structure `output.params` for format `parquet`:

- `compression_codec`: Compression codec. Supported values: `UNCOMPRESSED`, `SNAPPY`, `GZIP`, `LZ4`, `ZSTD`.
Default is `UNCOMPRESSED`.
- `float_precision`: Floating-point number precision. Default is `2`.
- `datetime_format`: Date-time format. Supported values: `millis`, `micros`. Default is `millis`.
- `partition_files_limit`: Limit on the number of partition files, upon reaching which a prompt will appear asking whether to continue.
Ignored if the `--force` flag is specified. Default is `1000`.

Structure `output.params` for format `http`:

Expand Down Expand Up @@ -458,7 +462,7 @@ sdvg generate ./models.yml
### Ignoring conflicts

If you want to automatically remove conflicting files from the output directory
and continue generation without additional prompts, use the `-F` or `--force` flag:
and continue generation without additional prompts, use the `-f` or `--force` flag:

```shell
sdvg generate --force ./models.yml
Expand All @@ -469,7 +473,7 @@ sdvg generate --force ./models.yml
To continue generation from the last recorded row:

```shell
sdvg generate --continue-generation ./models.yml
sdvg generate --continue ./models.yml
```

> **Important**: To correctly continue generation, you must not change the generation configuration
Expand Down
8 changes: 6 additions & 2 deletions doc/ru/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,13 +183,17 @@ open_ai:
- `datetime_format`: Формат даты и времени. По умолчанию `2006-01-02T15:04:05Z07:00`.
- `without_headers`: Флаг, указывающий, исключать ли CSV заголовок из файлов с данными.
- `delimiter`: Односимвольный CSV разделитель. По умолчанию `,`.
- `partition_files_limit`: Ограничение количества файлов партиций, при достижении которого всплывет вопрос о продолжении.
Игнорируется при указании флага `--force`. По умолчанию `1000`

Структура `output.params` для формата `parquet`:

- `compression_codec`: Кодек сжатия. Поддерживаемые значения: `UNCOMPRESSED`, `SNAPPY`, `GZIP`, `LZ4`, `ZSTD`.
По умолчанию `UNCOMPRESSED`.
- `float_precision`: Точность чисел с плавающей запятой. По умолчанию `2`.
- `datetime_format`: Формат даты и времени. Поддерживаемые значения: `millis`, `micros`. По умолчанию `millis`.
- `partition_files_limit`: Ограничение количества файлов партиций, при достижении которого всплывет вопрос о продолжении.
Игнорируется при указании флага `--force`. По умолчанию `1000`

Структура `output.params` для формата `http`:

Expand Down Expand Up @@ -464,7 +468,7 @@ sdvg generate ./models.yml
### Игнорирование конфликтов

Если вы хотите автоматически удалить конфликтующие файлы в выходной директории
и продолжить генерацию без дополнительных сообщений, используйте флаг `-F` или `--force`:
и продолжить генерацию без дополнительных сообщений, используйте флаг `-f` или `--force`:

```shell
sdvg generate --force ./models.yml
Expand All @@ -475,7 +479,7 @@ sdvg generate --force ./models.yml
Для продолжения генерации с последней записанной строки:

```shell
sdvg generate --continue-generation ./models.yml
sdvg generate --continue ./models.yml
```

> **Важно**: для корректного продолжения генерации нельзя менять конфигурацию генерации и уже сгенерированные данные.
8 changes: 4 additions & 4 deletions internal/generator/cli/commands/consts.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ const (
ConfigPathDefaultValue = ""
ConfigPathUsage = "Location of config file"

ContinueGenerationFlag = "continue-generation"
ContinueGenerationShortFlag = "C"
ContinueGenerationFlag = "continue"
ContinueGenerationShortFlag = "c"
ContinueGenerationDefaultValue = false
ContinueGenerationUsage = "Continue generation from the last recorded row"

ForceGenerationFlag = "force"
ForceGenerationShortFlag = "F"
ForceGenerationShortFlag = "f"
ForceGenerationFlagDefaultValue = false
ForceGenerationUsage = "Force generation even if output file conflicts found"
ForceGenerationUsage = "Force generation even if output file conflicts found and partition files limit reached" //nolint:lll

TTYFlag = "tty"
TTYShortFlag = "t"
Expand Down
53 changes: 40 additions & 13 deletions internal/generator/cli/commands/generate/generate.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"github.com/spf13/cobra"
"github.com/spf13/pflag"
"github.com/tarantool/sdvg/internal/generator/cli/commands"
"github.com/tarantool/sdvg/internal/generator/cli/confirm"
"github.com/tarantool/sdvg/internal/generator/cli/options"
"github.com/tarantool/sdvg/internal/generator/cli/progress"
"github.com/tarantool/sdvg/internal/generator/cli/progress/bar"
Expand Down Expand Up @@ -124,7 +125,9 @@ func runGenerate(ctx context.Context, opts *generateOptions) error {
return err
}

out := general.NewOutput(generationCfg, opts.continueGeneration, opts.forceGeneration)
progressTrackerManager, confirm := initProgressTrackerManager(ctx, opts.renderer, opts.useTTY, opts.forceGeneration)

out := general.NewOutput(generationCfg, opts.continueGeneration, opts.forceGeneration, confirm)

taskID, err := opts.useCase.CreateTask(
ctx, usecase.TaskConfig{
Expand All @@ -143,12 +146,11 @@ func runGenerate(ctx context.Context, opts *generateOptions) error {
)

startProgressTracking(
ctx,
progressTrackerManager,
opts.useCase,
taskID,
&finished,
&wg,
opts.useTTY,
)

err = opts.useCase.WaitResult(taskID)
Expand All @@ -173,26 +175,51 @@ func runGenerate(ctx context.Context, opts *generateOptions) error {
return nil
}

// initProgressTrackerManager inits progress bar manager (progress.Tracker)
// and builds confirm.Confirm func based on useTTY and forceGeneration.
func initProgressTrackerManager(
ctx context.Context,
renderer render.Renderer,
useTTY bool,
forceGeneration bool,
) (progress.Tracker, confirm.Confirm) {
var (
progressTrackerManager progress.Tracker
confirmFunc confirm.Confirm
)

if useTTY {
progressTrackerManager = bar.NewProgressBarManager(ctx)

confirmFunc = confirm.BuildConfirmTTY(renderer, progressTrackerManager)
} else {
isUpdatePaused := &atomic.Bool{}

progressTrackerManager = log.NewProgressLogManager(ctx, isUpdatePaused)

confirmFunc = confirm.BuildConfirmNoTTY(renderer, progressTrackerManager, isUpdatePaused)
}

if forceGeneration {
confirmFunc = func(_ context.Context, _ string) (bool, error) {
return true, nil
}
}

return progressTrackerManager, confirmFunc
}

// startProgressTracking runs function to track progress of task
// by getting progress from usecase object and displaying it.
func startProgressTracking(
ctx context.Context,
progressTrackerManager progress.Tracker,
uc usecase.UseCase,
taskID string,
finished *atomic.Bool,
wg *sync.WaitGroup,
useTTY bool,
) {
const delay = 500 * time.Millisecond

var progressTrackerManager progress.Tracker

if useTTY {
progressTrackerManager = bar.NewProgressBarManager(ctx)
} else {
progressTrackerManager = log.NewProgressLogManager(ctx)
}

wg.Add(1)

go func() {
Expand Down
2 changes: 1 addition & 1 deletion internal/generator/cli/commands/generate/generate_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ func TestNewGenerateCommand(t *testing.T) {
cliOpts.SetOut(streams.NewOut(os.Stdout))

cmd := NewGenerateCommand(cliOpts)
cmd.SetArgs([]string{"-F"})
cmd.SetArgs([]string{"-f"})

err = cmd.Execute()

Expand Down
2 changes: 1 addition & 1 deletion internal/generator/cli/commands/serve/handlers.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ func handleGenerate(opts handlerOptions, c echo.Context) error {

generationConfig.OutputConfig.Dir = models.DefaultOutputDir

out := general.NewOutput(&generationConfig, false, true)
out := general.NewOutput(&generationConfig, false, true, nil)

taskID, err := opts.useCase.CreateTask(
c.Request().Context(), usecase.TaskConfig{
Expand Down
119 changes: 119 additions & 0 deletions internal/generator/cli/confirm/confirm.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
package confirm
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it would be beneficial to move confirm to the render package and generally implement it as a renderer structure method.


import (
"context"
"fmt"
"io"
"strings"
"sync/atomic"

"github.com/manifoldco/promptui"
"github.com/pkg/errors"
"github.com/tarantool/sdvg/internal/generator/cli/render"
"github.com/tarantool/sdvg/internal/generator/cli/utils"
)

var ErrPromptFailed = errors.New("prompt failed")

// Confirm asks user a yes/no question. Returns true for “yes”.
type Confirm func(ctx context.Context, question string) (bool, error)

//nolint:gocritic
func BuildConfirmTTY(in io.Reader, out io.Writer) func(ctx context.Context, question string) (bool, error) {
return func(ctx context.Context, question string) (bool, error) {
fmt.Fprintln(out)

cancelableIn := newCancelableReader(in)
defer cancelableIn.Close()

prompt := promptui.Prompt{
Label: question + " [y/N]: ",
Default: "y",
Stdin: cancelableIn,
Stdout: utils.DummyReadWriteCloser{Writer: out},
}
validate := func(s string) error {
if len(s) == 1 && strings.Contains("YyNn", s) || prompt.Default != "" && len(s) == 0 {
return nil
}

return errors.New("invalid input")
}
prompt.Validate = validate

var (
input string
err error
promptFinished = make(chan struct{})
)

go func() {
input, err = prompt.Run() // goroutine will block here until user input

promptFinished <- struct{}{}
}()

select {
case <-ctx.Done():
return false, ctx.Err()
case <-promptFinished:
}

if err != nil {
return false, errors.Wrap(ErrPromptFailed, err.Error())
}

return strings.Contains("Yy", input), nil
}
}

func BuildConfirmNoTTY(
in render.Renderer,
out io.Writer,
isUpdatePaused *atomic.Bool,
) func(ctx context.Context, question string) (bool, error) {
return func(ctx context.Context, question string) (bool, error) {
// here we pause ProgressLogManager to stop sending progress messages
isUpdatePaused.Store(true)
defer isUpdatePaused.Store(false)

for {
fmt.Fprintf(out, "%s [y/N]: ", question)

var (
input string
err error
inputReadFinished = make(chan struct{})
)

go func() {
input, err = in.ReadLine() // goroutine will block here until user input

inputReadFinished <- struct{}{}
}()

select {
case <-ctx.Done():
return false, ctx.Err()
case <-inputReadFinished:
}

if err != nil {
return false, err
}

if !in.IsTerminal() {
fmt.Fprintln(out, input)
}

switch strings.ToLower(strings.TrimSpace(input)) {
case "y", "yes":
return true, nil
case "", "n", "no":
return false, nil
default:
fmt.Fprintln(out, "Please enter y or n")
}
}
}
}
Loading