-
Notifications
You must be signed in to change notification settings - Fork 723
quickstart трансфер данных #23623
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
quickstart трансфер данных #23623
Changes from 4 commits
Commits
Show all changes
43 commits
Select commit
Hold shift + click to select a range
eacc223
quickstart трансфер данных
nshestakov 7ffd804
fix
nshestakov 695d95c
fix
nshestakov 098e660
fix
nshestakov 5d71110
fix
nshestakov aa68ea5
fix
nshestakov 581ad24
Merge branch 'main' into TR-doc-receipt
nshestakov 2ca1796
fix path to topic.md
nshestakov d15fde0
Пример с nginx
nshestakov 9815297
fix
nshestakov 0e854ce
fix
nshestakov 07b36cf
fixes
nshestakov d984a66
fix
nshestakov 957460d
fix
nshestakov 384311f
fix
nshestakov 9b01280
spelling
nshestakov 141a8c2
batching
nshestakov 3998685
added link
nshestakov 2fa6d3c
Update ydb/docs/ru/core/recipes/transfer/_includes/batching.md
nshestakov 66f6eeb
Update ydb/docs/ru/core/recipes/transfer/index.md
nshestakov b482bfc
Update ydb/docs/ru/core/recipes/transfer/transfer-nginx.md
nshestakov 835b849
Update ydb/docs/ru/core/recipes/transfer/transfer-nginx.md
nshestakov d453c54
Update ydb/docs/ru/core/recipes/transfer/transfer-nginx.md
nshestakov c805e4e
Update ydb/docs/ru/core/recipes/transfer/transfer-nginx.md
nshestakov 2715a43
Update ydb/docs/ru/core/recipes/transfer/transfer-nginx.md
nshestakov 1cecfc8
Update ydb/docs/ru/core/recipes/transfer/transfer-nginx.md
nshestakov e404212
Update ydb/docs/ru/core/recipes/transfer/transfer-quickstart.md
nshestakov 7435dc3
Update ydb/docs/ru/core/recipes/transfer/transfer-quickstart.md
nshestakov 3e326f3
Update ydb/docs/ru/core/reference/kafka-api/connect/connect-examples.md
nshestakov 873c443
renme files
nshestakov a471cf2
rename to line_lambda
nshestakov 5c228f4
fix
nshestakov 9ba018a
use table
nshestakov 52ad9d3
use table
nshestakov 1e3a8f3
more suggests
nshestakov 481fadb
fix
nshestakov c9c45ea
added Unescape
nshestakov b8bf8c4
more links
nshestakov 82f2792
fix
nshestakov 4d41679
Update ydb/docs/ru/core/recipes/transfer/quickstart.md
nshestakov 90a014e
rebase
orange13 ec32838
Update quickstart.md
orange13 579639d
Update connect-examples.md
orange13 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
# Рецепты работы с трансфером данных из топика в таблицу | ||
|
||
Этот раздел содержит рецепты работы с [трансфером](../../concepts/transfer.md) из топика в таблицу. | ||
|
||
Содержание: | ||
|
||
* [{#T}](transfer-quickstart.md) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
items: | ||
- name: Трансфер - быстрый старт | ||
href: transfer-quickstart.md |
101 changes: 101 additions & 0 deletions
101
ydb/docs/ru/core/recipes/transfer/transfer-quickstart.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
# Трансфер — быстрый старт | ||
|
||
Эта статья поможет быстро начать работу с [трансфером](../../concepts/transfer.md) в {{ ydb-short-name }} на простейшем модельном примере. | ||
|
||
В статье будут рассмотрены следующие шаги работы с трансфером: | ||
|
||
* [создание топика](#step1) из которого будет читать трансфер; | ||
* [создание таблицы](#step2) в которую будут записываться данные; | ||
* [создание трансфера](#step3); | ||
* [заполнение топика данными](#step4); | ||
* [проверка содержимого таблицы](#step5). | ||
|
||
## Шаг 1. Создание топика {#step1} | ||
|
||
Сначала нужно создать [топик](../../concepts/topic.md) в {{ ydb-short-name }}, из которого трансфер будет читать данные. Это можно сделать с помощью [SQL-запроса](../../yql/reference/syntax/create-topic.md): | ||
|
||
```yql | ||
CREATE TOPIC SourceTopic; | ||
``` | ||
|
||
Этот топик `SourceTopic` позволяет передавать любые неструктурированные данные. | ||
|
||
## Шаг 2. Создание таблицы {#step2} | ||
|
||
После создания топика следует добавить [таблицу](../../concepts/datamodel/table.md), в которую будут поставляться данные из топика `SourceTopic`. Это можно сделать с помощью [SQL-запроса](../../yql/reference/syntax/create_table/index.md): | ||
|
||
```yql | ||
CREATE TABLE TargetTable ( | ||
partition Uint32 NOT NULL, | ||
offset Uint64 NOT NULL, | ||
data String, | ||
PRIMARY KEY (partition, offset) | ||
); | ||
``` | ||
|
||
Эта таблица `TargetTable` имеет три столбца: | ||
|
||
* `partition` — идентификатор [партиции](../../concepts/glossary.md#partition) топика, из которой получено сообщение; | ||
* `offset` — [порядковый номер](../../concepts/glossary.md#offset), идентифицирующий сообщение внутри партиции; | ||
* `data` — тело сообщения. | ||
|
||
## Шаг 3. Создание трансфера {#step3} | ||
|
||
После создания топика и таблицы следует добавить [трансфер](../../concepts/transfer.md) данных, который будет перекладывать сообщения из топика в таблицу. Это можно сделать с помощью [SQL-запроса](../../yql/reference/syntax/create-transfer.md): | ||
|
||
```yql | ||
$transformation_lambda = ($msg) -> { | ||
return [ | ||
<| | ||
partition: $msg._partition, | ||
offset: $msg._offset, | ||
data: $msg._data | ||
|> | ||
]; | ||
}; | ||
|
||
CREATE TRANSFER Transfer | ||
FROM SourceTopic TO TargetTable | ||
USING $transformation_lambda; | ||
``` | ||
|
||
В этом примере: | ||
|
||
* `$transformation_lambda` - это правило преобразования сообщения из топика в колонки таблицы; | ||
nshestakov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
* `$msg` - переменная, которая содержит обрабатываемое сообщение из топика. | ||
|
||
## Шаг 4. Заполнение топика данными {#step4} | ||
|
||
После создания трансфера можно записать в топик сообщение, например, используя [{{ ydb-short-name }} CLI](../../reference/ydb-cli/index.md). | ||
|
||
{% include [x](../../_includes/ydb-cli-profile.md) %} | ||
|
||
```bash | ||
echo "Message 1" | ydb --profile quickstart topic write SourceTopic | ||
echo "Message 2" | ydb --profile quickstart topic write SourceTopic | ||
echo "Message 3" | ydb --profile quickstart topic write SourceTopic | ||
``` | ||
|
||
## Шаг 5. Проверка содержимого таблицы {#step5} | ||
|
||
После записи сообщения в топик `SourceTopic` спустя некоторое время появится запись в таблице `TargetTable`. Проверить ее наличие можно с помощью [SQL-запроса](../../yql/reference/syntax/select/index.md): | ||
|
||
```yql | ||
SELECT * | ||
FROM TargetTable; | ||
``` | ||
|
||
Результат выполнения запроса: | ||
|
||
```bash | ||
partition offset data | ||
0 0 "Message 1" | ||
0 1 "Message 2" | ||
0 2 "Message 3" | ||
``` | ||
|
||
## Заключение | ||
|
||
Данная статья приводит простой пример работы с трансфером: создание топика, таблицы и трансфера, записи в топик и проверки результата работы трансфера. | ||
nshestakov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
Более подробную информацию о трансфере см. [здесь](../../concepts/transfer.md). | ||
nshestakov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.