Welcome to Data Designer Discussions!

[johnnygreco]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[Pipboyguy]

Hi, I'm Marcel.

I've used this repo already create seed data for my application. It did the job amazingly, though the speed was a bit slow

[johnnygreco]

Hi @Pipboyguy 👋

Thank you for your feedback! Out of curiosity, what model provider did you try? Our research team is actively working on ways to create large datasets as fast as possible, but for sure, inference throughput is a major bottleneck.

[Pipboyguy]

Hey @johnnygreco

I first tried the models listed on data generation but they are clearly meant for robotic use cases. After some experimentation I found most 14B models like ministral-14b-instruct-2512 to be satisfactory ito speed and generated data quality/diversity.

I'm testing this with clients as well for ML applications, so its already proven economically useful! Thank you for this work.

[johnnygreco]

I'm glad to hear you have found Data Designer useful! Thank you for this feedback 🙌

If you want more speedy inference, we have OpenAI set as a default Model Provider. I recommend giving it a spin if you have a chance. You'll need an OpenAI API key, of course.

[SharifMAbdullah]

Hi. Don't know if it's the appropriate place to ask this question but is there a way to pass a value to a prompt from a list? If I try to use the default jinja2 syntax, it leads the framework to think that I'm referencing a column which doesn't exist.

[johnnygreco]

Hi @SharifMAbdullah,

This discussion forum is a great place to ask questions like this – thank you for asking it! And please feel free to start a new discussion if you don't see one on a similar topic.

To answer your question, I'm assuming you have a seed dataset where each element of a column is a list and you'd like to sample an item (or multiple items) from each list and inject it into a prompt. Does that sound right?

If so, the short answer is yes it is possible, though it can be trickier than we'd like at the moment. Fortunately, we have adding support for prompt utilities and more flexible seed dataset sampling strategies on our short-term roadmap.

The longer answer to your question is best give as an example. Below is an example showing how to both sample items from a list and iterate over using jinja:

import pandas as pd

from data_designer.essentials import DataDesigner, DataDesignerConfigBuilder, LLMTextColumnConfig

df = pd.DataFrame(
    {
        "peom_topics": [
            ["Jupiter", "space cowboys", "starships"],
            ["cars", "monster trucks", "motorcycles"],
        ],
        "grocery_list": [
            ["apples", "bananas", "carrots"],
            ["milk", "bread", "eggs"],
        ],
    }
)

data_designer = DataDesigner()

config_builder = DataDesignerConfigBuilder()

# NOTE: It is important that we save our seed dataset as a parquet file so that the lists will be deserialized correctly.
config_builder.with_seed_dataset(data_designer.make_seed_reference_from_dataframe(df, "local_file.parquet"))

# Here we pass peom_topics directly to the random filter. This is why it is important for us to use parquet as 
# our file format to ensure it is rendered as a list and not a string. 
config_builder.add_column(
    LLMTextColumnConfig(
        name="poem",
        prompt="Write a short poem about : {{ peom_topics | random }}",
        model_alias="nvidia-text",
    )
)

config_builder.add_column(
    LLMTextColumnConfig(
        name="store_story",
        prompt=(
            "Write a short story about a guy who goes to the store in search of items in this grocery list:\n"
            "\n### Grocery List ###\n"
            "{% for item in grocery_list %}"
            "* {{ item }}\n"
            "{% endfor %}"
        ),
        model_alias="nvidia-text",
    )
)

# Validating like this as you go can be useful to catch mistakes in your prompts. 
config_builder.validate()

preview = data_designer.preview(config_builder)

Running display_sample_record on the preview, we can see that the lists were used as exepected.

preview.display_sample_record()

                                                   Seed Columns                                                    
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                        ┃ Value                                                                             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ peom_topics                 │ ['Jupiter' 'space cowboys' 'starships']                                           │
├─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤
│ grocery_list                │ ['apples' 'bananas' 'carrots']                                                    │
└─────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────┘
                                                                                                                   
                                                                                                                   
                                                 Generated Columns                                                 
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name        ┃ Value                                                                                             ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ poem        │ **Starship Lullaby**                                                                              │
│             │                                                                                                   │
│             │ In silver hush they pierce the night,                                                             │
│             │ Their hulls reflect a thousand suns—                                                              │
│             │ No wind, just dreams that pulse with light,                                                       │
│             │ And whispers of the worlds undone.                                                                │
│             │                                                                                                   │
│             │ Through constellations they will glide,                                                           │
│             │ Cartographers of the void,                                                                        │
│             │ Each flicker a secret tide,                                                                       │
│             │ Each engine sighs, “We’re un‑employed.”                                                           │
│             │                                                                                                   │
│             │ Yet in their silent, soaring grace,                                                               │
│             │ They keep the galaxy’s refrain—                                                                   │
│             │ A cosmic hymn of time and space,                                                                  │
│             │ Where every star‑ship sings again.                                                                │
├─────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ store_story │ The fluorescent lights of "QuickStop" hummed like tired bees as Ben scanned the produce aisle.    │
│             │ *Apples*, *bananas*, *carrots*. Simple. Easy. He’d barely even looked at the list printed on his  │
│             │ fridge, a scrap of paper with three items in his own looping handwriting. *Easy*, he thought,     │
│             │ pushing the cart forward. *Just grab the basics.*                                                 │
│             │                                                                                                   │
│             │ He reached for the apples first—bright red, shiny, almost too perfect. But then he noticed them.  │
│             │ Tiny bruises, like fingerprints left by a careless hand. Not the kind of flaw you’d notice in a   │
│             │ magazine photo, but real, organic imperfections that made them feel… human. *Like they’d been     │
│             │ picked by someone who actually knew what they were doing*, Ben thought. He picked up one, turned  │
│             │ it over in his hand. It felt heavy, cool, solid. *Okay*, he decided. *This one.*                  │
│             │                                                                                                   │
│             │ Next came the bananas. They were yellow, sure, but one had a little brown speck near the stem.    │
│             │ Ben hesitated. *Brown? Is that… bad?* He remembered his mother’s voice, decades ago: "Brown       │
│             │ bananas are sweeter, honey. Don’t throw them out." He shoved the bunch into the cart, the brown   │
│             │ spot feeling less like a flaw and more like a quiet promise. *Good enough.*                       │
│             │                                                                                                   │
│             │ Finally, the carrots. They were straight, bright orange, crisp. But one had a tiny green shoot    │
│             │ poking from the top—a sign it was *fresh*, not just sitting there. Ben chuckled. *Even carrots    │
│             │ want to grow*, he mused. He grabbed it, the green shoot a tiny rebellion against the sterile      │
│             │ grocery store. *This one’s got character.*                                                        │
│             │                                                                                                   │
│             │ At the checkout, the cashier scanned the items with a quiet *beep-beep-beep*. "That’ll be $3.47," │
│             │ she said, handing him the bag. Ben looked down at the bag: one slightly bruised apple, one banana │
│             │ with a brown speck, one carrot with a stubborn green shoot. It wasn’t perfect. It wasn’t the      │
│             │ gleaming, flawless produce he’d imagined when he’d written the list. But as he walked out into    │
│             │ the afternoon sun, the bag light in his hand, he realized something: *perfect* was boring. These  │
│             │ things were alive. They had stories. They had a little bit of mess, a little bit of life, and     │
│             │ they’d carried him home.                                                                          │
│             │                                                                                                   │
│             │ He took a deep breath of the crisp air, the scent of rain and earth rising around him. *Apples,   │
│             │ bananas, carrots*. He’d found them. Not just items on a list, but… *things*. And for the first    │
│             │ time, the list didn’t feel like a chore. It felt like a beginning.                                │
└─────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘
                                                                                                                   
                                                    [index: 6]                                                     

Please let us know if you have any other questions. As I mentioned, this topic is relevant for our short-term roadmap, so we are keen to hear user feedback 🙌

[SharifMAbdullah]

Thanks for the answer, that solves the issue!

[guox18]

Hi team — I’ve read CONTRIBUTING.md and the docs. I’m happy to start with small contributions while I learn the codebase. I can help with docs improvements, tests, small bug fixes, or add examples/recipes (e.g., a rejection-sampling tutorial). I looked for labels like “good first issue” but didn’t see any — are there areas or issues you’d recommend for a first contribution?