Former OpenAI engineer Andrey Karpaty A new nanochat project was presented on GitHub.
This is an open source toolkit that allows you to train your own language model almost from scratch.
Karpaty showed that almost any developer can now create their own ChatGPT-like bot. Nanochat includes everything you need: a tokenizer, training scripts, code to run the chat, and even a web interface where possible. immediately communicate with the model.
What does the project include?
► Tokenizer: new implementation in Rust
► Preliminary preparation: FineWeb corpus for CORE and other metrics
►Mid-term training; user dialogues with the assistant (SmolTalk), tests and data on the use of tools
► SFT (Supervised Fine Tuning): Tests world knowledge, mathematics (GSM8K) and programming (HumanEval)
► Training in using GRPO to solve GSM8K problems
► The mechanism of the result. caching support, instrument calls (e.g. Python interpreter), ChatGPT-style CLI and WebUI interaction
► Auto reports. the system itself generates Markdown cards with effects and game metrics
To train such a model, you will need a server with eight Nvidia H100 video cards. This will take about 4 hours and approximately $100 if you rent equipment in the cloud. The launch occurs with the command Speedrun.sh.
In 12 hours and approximately $1000, the model will be able to outperform GPT-2 on the CORE metric and will be able to solve basic math problems, programming, and multiple choice tests. [Habr]
Source: Iphones RU
I am a professional journalist and content creator with extensive experience writing for news websites. I currently work as an author at Gadget Onus, where I specialize in covering hot news topics. My written pieces have been published on some of the biggest media outlets around the world, including The Guardian and BBC News.
