Tech companies are now focusing on developing smaller language models (SLMs) that can rival or even surpass the larger ones. While Meta’s Llama 3, OpenAI’s GPT-3.5, and GPT-4 are well-known for their large parameters, Microsoft’s Phi-3 family and Apple Intelligence are exploring SLMs with significantly fewer parameters. Despite the reduction in size, SLMs offer advantages such as lower energy consumption, the ability to run locally on devices like smartphones, and affordability for smaller businesses and labs.
The shift towards SLMs is evident as tech companies seek to close the performance gap between large and small models. In a recent test conducted by Microsoft, the Phi-3-mini model with 3.8 billion parameters showed promising results compared to larger models like Mixtral and GPT-3.5. The success of SLMs can be attributed to the training dataset used, which includes filtered web data and synthetic data.
Although SLMs demonstrate similar language understanding and reasoning abilities as larger models, they have limitations when it comes to storing vast amounts of factual knowledge. One solution to overcome this limitation is to integrate SLMs with online search engines. The comparison of SLMs to how children learn language highlights the efficiency of human learning in processing language with limited data.
While the exact reason for humans’ efficiency in learning language remains unknown, researchers suggest that reverse engineering humanlike learning at small scales could lead to significant improvements when scaled up to larger models. This approach could potentially enhance the performance of LLMs and bridge the gap between human learning and artificial intelligence.
The development of SLMs opens up new possibilities for businesses and labs that are looking for efficient and affordable language models. By leveraging the advantages of SLMs, companies can enhance their language processing capabilities and explore innovative applications in various sectors. As technology continues to evolve, the integration of SLMs with existing AI systems could revolutionize the way we interact with language and information.
