Tokenize Function • AI Agents Framework

tokenize_function: Tokenizes a batch of text prompts with padding and truncation enabled.

tokenize_function

Function

This function is defined in the train module.

Tokenizes a batch of text prompts with padding and truncation enabled.

Signature

def tokenize_function(examples: Any, hf_tokenizer: Any, max_length: Any) -> Any

Parameters

examples · Any · required

No description available.

hf_tokenizer · Any · required

No description available.

max_length · Any · required

No description available.

Returns

Returns · Any

The result of the operation.

Uses

• hf_tokenizer
• value.tolist

Used By

• TrainModel.tokenize_dataset

Source

View on GitHub

praisonai/train.py at line 83