Tokenize Dataset • AI Agents Framework

tokenize_dataset: API reference for TrainModel.tokenize_dataset

tokenize_dataset

Method

This is a method of the TrainModel class in the train module.

Signature

def tokenize_dataset(dataset: Any) -> Any

Parameters

dataset · Any · required

No description available.

Returns

Returns · Any

The result of the operation.

Uses

• tokenize_function
• tokenized_dataset.remove_columns

Used By

• TrainModel.train_model

Source

View on GitHub

praisonai/train.py at line 195