HuggingFaceDataset

"# HuggingFaceDataset\n\n```python\nfrom modal_dojo import HuggingFaceDataset\n```\n\nA dataset loaded from a Hugging Face `datasets` repository.\n\n**Attributes**\n\n<div class=\"tg-param\">\n<p><strong>hf_repo</strong> <code>str</code></p>\n<p>Hugging Face dataset repository ID.</p>\n</div>\n\n<div class=\"tg-param\">\n<p><strong>hf_split</strong> <code>str</code></p>\n<p>Source dataset split. <span class=\"tg-param-default\">Default: \"train\"</span></p>\n</div>\n\n<div class=\"tg-param\">\n<p><strong>hf_config</strong> <code>str | None</code></p>\n<p>Source dataset configuration name.</p>\n</div>\n\n<div class=\"tg-param\">\n<p><strong>input_column</strong> <code>str</code></p>\n<p>Source prompt column.</p>\n</div>\n\n<div class=\"tg-param\">\n<p><strong>output_column</strong> <code>str | None</code></p>\n<p>Source answer column.</p>\n</div>\n\n<div class=\"tg-param\">\n<p><strong>input_format</strong> <code>Literal['text', 'messages', 'raw']</code></p>\n<p>Shape of the source prompt: plain <code>text</code> to convert into messages, preformatted <code>messages</code>, or <code>raw</code> model input. <span class=\"tg-param-default\">Default: \"text\"</span></p>\n</div>\n\n<div class=\"tg-param\">\n<p><strong>system_prompt</strong> <code>str</code></p>\n<p>System message added to formatted examples. <span class=\"tg-param-default\">Default: \"\"</span></p>\n</div>\n\n<div class=\"tg-param\">\n<p><strong>prompt_template</strong> <code>str</code></p>\n<p>Template applied to each source prompt. <span class=\"tg-param-default\">Default: \"{input}\"</span></p>\n</div>\n\n<div class=\"tg-param\">\n<p><strong>always_download</strong> <code>bool</code></p>\n<p>When training, always download the dataset from Hugging Face instead of caching it. <span class=\"tg-param-default\">Default: False</span></p>\n</div>\n\n## Constructor\n\n```python\nHuggingFaceDataset(hf_repo: str, *, hf_split: str = 'train', hf_config: str | None = None, input_column: str, output_column: str | None = None, input_format: Literal['text', 'messages', 'raw'] = 'text', system_prompt: str = '', prompt_template: str = '{input}', always_download: bool = False)\n```\n\n## `apply_chat_template`\n\n```python\napply_chat_template() -> bool\n```\n\nWhether to apply the model's chat template to the input.\n\n## `cache_key`\n\n```python\ncache_key() -> str | None\n```\n\n## `input_key`\n\n```python\ninput_key() -> str\n```\n\nPrompt column name.\n\n## `label_key`\n\n```python\nlabel_key() -> str | None\n```\n\nGround-truth column name, or `None` when rows carry no label.\n\n## `output_format`\n\n```python\noutput_format() -> str\n```\n\nThe on-disk format written by `write()`, either `parquet` or `jsonl`.\n\n## `rows`\n\n```python\nrows() -> Iterable[DatasetRow]\n```\n\nLoad raw examples.\n\n**Returns**\n\nAn iterable collection of raw examples.\n\n## `validate_written`\n\n```python\nvalidate_written(path: str) -> None\n```\n\nValidate the materialized file format and required columns.\n\n## `write`\n\n```python\nwrite(path: str) -> None\n```\n\nMaterialize training data at `path`.\n"

from modal_dojo import HuggingFaceDataset

A dataset loaded from a Hugging Face datasets repository.

Attributes

hf_repo str

Hugging Face dataset repository ID.

hf_split str

Source dataset split. Default: "train"

hf_config str | None

Source dataset configuration name.

input_column str

Source prompt column.

output_column str | None

Source answer column.

input_format Literal['text', 'messages', 'raw']

Shape of the source prompt: plain text to convert into messages, preformatted messages, or raw model input. Default: "text"

system_prompt str

System message added to formatted examples. Default: ""

prompt_template str

Template applied to each source prompt. Default: "{input}"

always_download bool

When training, always download the dataset from Hugging Face instead of caching it. Default: False

Constructor

HuggingFaceDataset(hf_repo: str, *, hf_split: str = 'train', hf_config: str | None = None, input_column: str, output_column: str | None = None, input_format: Literal['text', 'messages', 'raw'] = 'text', system_prompt: str = '', prompt_template: str = '{input}', always_download: bool = False)

`apply_chat_template`

apply_chat_template() -> bool

Whether to apply the model’s chat template to the input.

`cache_key`

cache_key() -> str | None

`input_key`

input_key() -> str

Prompt column name.

`label_key`

label_key() -> str | None

Ground-truth column name, or None when rows carry no label.

`output_format`

output_format() -> str

The on-disk format written by write(), either parquet or jsonl.

`rows`

rows() -> Iterable[DatasetRow]

Load raw examples.

Returns

An iterable collection of raw examples.

`validate_written`

validate_written(path: str) -> None

Validate the materialized file format and required columns.

`write`

write(path: str) -> None

Materialize training data at path.