OwnAI. A GPT and its BPE tokenizer rebuilt from scratch in NumPy
- 1,055,488 (counted in the OwnAI repository) parameters
- 104 (OwnAI repository test suite) tests
- 59,253,248 parameters (OwnGPT)
Problem
Understand what is inside a language model, step by step, instead of calling a library.
Decisions
- Write everything by hand in NumPy: the BPE tokenizer, then the GPT.
- Test every piece.
- In the same repository, OwnGPT: a larger PyTorch version, trained once on a free Kaggle T4 GPU.
Result
A working GPT, written from scratch, with its test suite. The OwnGPT checkpoint is not in the repository.
What’s not done yet
- I trained OwnGPT only once.
- The checkpoint is not published.
- OwnGPT invents facts: its answers are not yet grounded in the repository’s retrieval (BM25 and word2vec), and it has no gate to refuse an out-of-domain question.