Original Summary

I've been building MICA , a tiny byte-level language model that runs through integer rules instead of neural layers. My goal is for it to offer short, useful text completions on ordinary hardware. The prediction score has improved a lot. The latest exported model gets 2.2352 bits per byte on 1,000 held-out text records. An earlier version gets 2.3136 on those same records. Lower is better. Then I tested the thing I actually want people to use it for: completing a few words from a short prompt. That was a reality check. In a set of paired prompts with the same ending but different topics, MICA changed its continuation for only about 3 to 5 of 20 pairs. Most of its generated words are real English words, but they often make no sense together. For example: Thanks so much for → the sum of the sum So the model is learning local text patterns, but it isn't reliably using the earlier words to decide what to say. A better prediction score hasn't fixed that. I've packaged several versions with a CPU inference tool so I can reproduce the results outside the training setup. I'm also comparing MICA against an ordinary n-gram model at the same file size. That's an important check because its current rules behave a lot like hashed context lookups. I still want to release MICA as a small research project people can run and experiment with. Before that, I want it to give at least some useful short completions. If you've built autocomplete or small language models, what would you test next to find out why it loses the prompt's topic? And if you’d like to test MICA when it’s ready, leave a comment. I’ll share a working link here or in a follow-up post. Disclosure: I designed the project and work with Claude on much of the implementation and experiments. I used AI writing assistance for this post too.   submitted by   /u/Silver_Employ2617 [link]   [comments]

  • 情报分类：商业与市场研究
  • 分类依据：内容涉及商业、投资或市场动态
  • 信息来源：Reddit · SideProject
  • 发布时间：2026/9/26 03:02:24