autogen.retrieve_utils.split_files_to_chunks

split_files_to_chunks

split_files_to_chunks(
    files: list[tuple[str, str] | str],
    max_tokens: int = 4000,
    chunk_mode: str = 'multi_lines',
    must_break_at_empty_line: bool = True,
    custom_text_split_function: Callable[[str], list[str]] | None = None
) -> tuple[list[str], list[dict[str, Any]]]

Split a list of files into chunks of max_tokens.

Parameters:

Name	Description
files	Type: list[tuple[str, str] | str]
max_tokens	Type: int Default: 4000
chunk_mode	Type: str Default: ‘multi_lines’
must_break_at_empty_line	Type: bool Default: True
custom_text_split_function	Type: Callable[[str], list[str]] | None Default: None