Small Language Models

Introduction:

• The race towards building large AI models has been building up ever since OpenAI released their 175 billion parameter LLM, GPT-3, in 2020.
• But, in 2024, researchers started to look at language models differently as scaling training data, scoured from the Internet, was giving marginal gains.
• The idea of building smaller language models emerged.
• This is evident in announcements made by Big Tech firms.
• Most of them released a nifty language model alongside their flagship AI models.
• Google DeepMind released Gemini Ultra, Nano and Flash models, while OpenAI and Meta launched their GPT-4o mini and Llama 3 models.
• Amazon-backed Anthropic AI’s launched Claude 3 and Haiku alongside its Opus.

Advantages and drawbacks of Small Language Models:

• Small Language Models (SLMs) are cheaper and ideal for specific use cases.
• For a company that needs AI for a set of specialised tasks, it doesn’t require a large AI model.
• Training small models require less time, less compute and smaller training data.
• French start up Mistral AI, an SLM provider, pitched its AI model to be as efficient as LLMs for specialised, focused applications.
• Microsoft released a family of small language models called Phi.
• Apple Intelligence, the AI system deployed in the latest iPhones and iPads, runs on-device AI models that can sort of match the performance of top LLMs.
• If LLMs are intentionally built to achieve Artificial General Intelligence (AGI), small language models are made for specific use cases.

Use cases difference for large and small AI models:

• Small language models are perfect for edge cases.
• When I am using WhatsApp or any Meta application which is powered by the Llama 8B model, I am trying to learn a new language because its reasonably good at translation and other basic tasks like this.
• But they wouldn’t do well at most benchmarks that large language models are measured against like coding or logical problems.
• There still isn’t a small language model that’s as good at solving more complex problems.
• We still aren’t fully aware why this bottleneck exists.
• But the best way we can understand this is just as human beings have brains with a massive number of neurons, a smaller animal has a limited number of neurons.
• This is why human brains have the capacity for far more complex levels of intelligence.
• This is similar to how small language models and large language models work.

Indian case:

• In a country like India, where the scope of AI adoption is immense but resources are constrained, the diminutiveness of small language models is perfect.
• Another AI initiative from IIIT Hyderabad, Visvam, is building datasets from the ground up to build small language models that can be used in healthcare, agriculture, education and to promote and preserve language and cultural diversity through AI.
• As the world of language model develops, it’s not just enough to build frontier models from scratch. Sarvam AI’s co-founder said, “We want to build GenAI that a billion Indians can use.”

Himalayan ice stupa technique

Context:

• Chile adopting the Himalayan ice stupa technique to save the Andean glaciers

About ice stupas: The artificial glaciers helping combat the effects of climate change

• They are an innovative solution to tackling water shortages caused by drought in the summer months, and can help combat the effects of climate change.
• Artificial glaciers, or ice stupas, are made using a simple irrigation system designed to store fresh water for use during arid summer months.
• Ice stupas have provided a lifeline for farmers during the spring planting season in Ladakh, a high mountain-desert region on the edge of the Himalayas.
• The region has a population of almost 300,000 people, but on average only receives around 10 centimetres of precipitation every year – that’s just a fraction more than the Sahara Desert.
• Agriculture in these cold desert regions is critical to survival, and crops can only be cultivated over a few months each year.
• Due to climate change, this already short window is rapidly declining.
• Retreating – or disappearing – glaciers have provided climate scientists with some of the most alarming evidence that the Earth is warming.
• According to a study published in the journal Cryosphere, the Earth lost more than 1.2 trillion tonnes of ice in the 2010s.
• This is a significant increase from the 760 billion tonnes lost per year in the previous decade, and researchers found that between 1994 and 2017, the planet lost a whopping 28 trillion tonnes of ice.
• Natural glaciers are crucial to life in these arid mountain regions, as they provide an essential freshwater source.
• So engineers have come up with an innovative solution; to freeze, and store winter fresh water in huge, towering structures.
• This water can then be accessed throughout the year and can help to sustain the communities that live in these regions.
• The technique was first developed by engineer Sonam Wangchuk in 2013.
• Researchers from the Cryosphere and Climate Change research group at the University of Aberdeen have been working with Indian universities, locals from the Ladakh region and The Ice Stupa Project to help address the situation.
• In particular, they are looking at ways to avoid water freezing in the pipes, as well as gaining a better understanding of the local micro-climates.

Making artificial glaciers:

• Stage 1: Artificial glaciers are built during the winter months by piping freshwater from a higher altitude downslope using polyethene tubing.
• Stage 2: The water is channelled through a pipe from the base of the ice stupa into a vertical pipe made of galvanised iron.
• Stage 3: When the temperature drops at night, this freshwater is pumped through a sprinkler at the top of this vertical pipe.
• With winter temperatures in the Ladakh region as low as -30°C, the water freezes onto a purpose-built structure made of wood and steel.
• Stage 4: As the water freezes, the result is a huge stalagmite-type structure.
• As the ice accumulates, more piping can be added to increase the height of the artificial glacier and store higher volumes of water.
• Stage 5: When the weather warms and water is scarce, the ice gradually melts to release this freshwater stored in the glacier.
• This provides locals with an invaluable source of water for irrigation in that critical window early on in the planting season.

UPSC Mains PYQ:

• How will the melting of Himalayan glaciers have a far-reaching impact on the water resources of India? (2020)
• Bring out the relationship between the shrinking Himalayan glaciers and the symptoms of climate change in the Indian sub-continent. (2014)

How will AI revolutionize drug development?

Introduction:

• The potential of using artificial intelligence in drug discovery and development has sparked both excitement and skepticism among scientists, investors and the general public.
• “Artificial intelligence is taking over drug development,” claim some companies and researchers.
• Over the past few years, interest in using AI to design drugs and optimize clinical trials has driven a surge in research and investment.
• AI-driven platforms like AlphaFold, which won the 2024 Nobel Prize for its ability to predict the structure of proteins and design new ones, showcase AI’s potential to accelerate drug development.
• AI in drug discovery is “nonsense,” warn some industry veterans.
• They urge that “AI’s potential to accelerate drug discovery needs a reality check,” as AI-generated drugs have yet to demonstrate an ability to address the 90% failure rate of new drugs in clinical trials.
• Unlike the success of AI in image analysis, its effect on drug development remains unclear.
• We have been following the use of AI in drug development in our work as a pharmaceutical scientist in both academia and the pharmaceutical industry and as a former program manager in the Defense Advanced Research Projects Agency, or DARPA.
• We argue that AI in drug development is not yet a game-changer, nor is it complete nonsense.
• AI is not a black box that can turn any idea into gold.
• Rather, we see it as a tool that, when used wisely and competently, could help address the root causes of drug failure and streamline the process.
• Most work using AI in drug development intends to reduce the time and money it takes to bring one drug to market – currently 10 to 15 years and US$1 billion to $2 billion.
• But can AI truly revolutionize drug development and improve success rates?

AI in drug development:

• Researchers have applied AI and machine learning to every stage of the drug development process.
• This includes identifying targets in the body, screening potential candidates, designing drug molecules, predicting toxicity and selecting patients who might respond best to the drugs in clinical trials, among others.
• Between 2010 and 2022, 20 AI-focused startups discovered 158 drug candidates, 15 of which advanced to clinical trials.
• Some of these drug candidates were able to complete preclinical testing in the lab and enter human trials in just 30 months, compared with the typical 3 to 6 years.
• This accomplishment demonstrates AI’s potential to accelerate drug development.
• On the other hand, while AI platforms may rapidly identify compounds that work on cells in a Petri dish or in animal models, the success of these candidates in clinical trials – where the majority of drug failures occur – remains highly uncertain.
• Unlike other fields that have large, high-quality datasets available to train AI models, such as image analysis and language processing, the AI in drug development is constrained by small, low-quality datasets.
• It is difficult to generate drug-related datasets on cells, animals or humans for millions to billions of compounds.
• While AlphaFold is a breakthrough in predicting protein structures, how precise it can be for drug design remains uncertain.
• Minor changes to a drug’s structure can greatly affect its activity in the body and thus how effective it is in treating disease.

Survivorship bias:

• Like AI, past innovations in drug development like computer-aided drug design, the Human Genome Project and high-throughput screening have improved individual steps of the process in the past 40 years, yet drug failure rates haven’t improved.
• Most AI researchers can tackle specific tasks in the drug development process when provided with high-quality data and particular questions to answer.
• But they are often unfamiliar with the full scope of drug development, reducing challenges into pattern recognition problems and refinement of individual steps of the process.
• Meanwhile, many scientists with expertise in drug development lack training in AI and machine learning.
• These communication barriers can hinder scientists from moving beyond the mechanics of current development processes and identifying the root causes of drug failures.
• Current approaches to drug development, including those using AI, may have fallen into a survivorship bias trap, overly focusing on less critical aspects of the process while overlooking major problems that contribute most to failure.
• This is analogous to repairing damage to the wings of aircraft returning from the battle fields in World War II while neglecting the fatal vulnerabilities in engines or cockpits of the planes that never made it back.
• Researchers often overly focus on how to improve a drug’s individual properties rather than the root causes of failure.
• The current drug development process operates like an assembly line, relying on a checkbox approach with extensive testing at each step of the process.
• While AI may be able to reduce the time and cost of the lab-based preclinical stages of this assembly line, it is unlikely to boost success rates in the more costly clinical stages that involve testing in people.
• The persistent 90% failure rate of drugs in clinical trials, despite 40 years of process improvements, underscores this limitation.

Addressing root causes:

• Drug failures in clinical trials are not solely due to how these studies are designed; selecting the wrong drug candidates to test in clinical trials is also a major factor.
• New AI-guided strategies could help address both of these challenges.
• Currently, three interdependent factors drive most drug failures: dosage, safety and efficacy.
• Some drugs fail because they’re too toxic, or unsafe.
• Other drugs fail because they’re deemed ineffective, often because the dose can’t be increased any further without causing harm.
• We propose a machine learning system to help select drug candidates by predicting dosage, safety and efficacy based on five previously overlooked features of drugs.
• Specifically, researchers could use AI models to determine how specifically and potently the drug binds to known and unknown targets, the level of these targets in the body, how concentrated the drug becomes in healthy and diseased tissues, and the drug’s structural properties.
• These features of AI-generated drugs could be tested in what we call phase 0+ trials, using ultra-low doses in patients with severe and mild disease.
• This could help researchers identify optimal drugs while reducing the costs of the current “test-and-see” approach to clinical trials.

Conclusion:

• While AI alone might not revolutionize drug development, it can help address the root causes of why drugs fail and streamline the lengthy process to approval.

UPSC Mains PYQ:

• Introduce the concept of Artificial Intelligence (AI). How does AI help clinical diagnosis? Do you perceive any threat to the privacy of the individual in the use of AI in healthcare? (2023)