Skip to content

Support for GPT-3.5 Tokenizer #14

Open
@iRambax

Description

@iRambax

Hello, thank you for creating the ChatGPTSwift library. I noticed that the tokenizer currently used is the BPE tokenizer for ChatGPT-3, which is different from the Unigram language model tokenizer used by GPT-3.5.

Since we need to manually count the used tokens in stream mode, I was wondering if there is a plan to implement the GPT-3.5 tokenizer in the ChatGPTSwift library.

Thank you for your consideration.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions