Ollama: running Large Language Models locally

I recently discovered a new tool called Ollama, which allows you to run Large Language Models (LLMs) locally, without the need of a cloud service. Its usage is similar to Docker, but it’s specifically designed for LLMs. You can use it as an interactive shell, through its REST API or using it from a Python library.

What is Ollama?

Ollama is a tool to run and manage Large Language Models locally. It’s designed to be easy to use and to be used in different ways:

Interactive shell: you can run Ollama as a shell and interact with it, you will be able to chat with it, ask questions, and simulate a conversation.
REST API: you can run Ollama as a service and send requests to it.
Python library: you can use Ollama from your Python code.

How does it work?

After you installa Ollama, you can start chatting with it by simply using this command:

1

ollama run llama2

ollama will pull the llama2 model from the cloud and start the interactive shell. You can then start chatting with it.

Demo

Using it through its REST API

Ollama comes with an included REST API which you can use to send requests to it. For example to send a question to the llama2:13b-chat model you can use the following command:

1
2
3
4
5


curl http://localhost:11434/api/generate -d '{
  "model": "llama2:13b-chat",
  "prompt": "What can you tell me about Pink Floyd?",
  "stream": false
}'

the response will be a JSON object with the generated text, like this one:

1

{"model":"llama2:13b-chat","created_at":"2024-03-01T17:23:39.094249Z","response":"\nAh, Pink Floyd! One of the most iconic and influential rock bands of all time. The band was formed in 1965 by students Syd Barrett, Nick Mason, Roger Waters, and Richard Wright at the University of Cambridge. They began as a folk-rock group, but soon developed their own unique sound that blended psychedelic rock, progressive rock, and experimental music.\n\nPink Floyd is known for their immersive live shows, elaborate album artwork, and thought-provoking lyrics. Their music explores themes such as life, death, technology, and the human condition. Some of their most famous albums include \"The Dark Side of the Moon,\" \"Wish You Were Here,\" and \"The Wall.\"\n\nSyd Barrett, the band's primary songwriter and lead vocalist, left the band in 1968 due to mental health issues. Roger Waters, who took over as the main songwriter and bassist, became the driving force behind the band's sound and vision. David Gilmour, a guitarist and vocalist, joined the band in 1967 and remained a key member until the band's breakup in 1996.\n\nPink Floyd released several critically acclaimed and commercially successful albums throughout their career, including \"Atom Heart Mother,\" \"Meddle,\" \"Obscured by Clouds,\" and \"The Final Cut.\" They are considered one of the greatest rock bands of all time, with a legacy that continues to inspire new generations of musicians and fans.\n\nSome of Pink Floyd's most famous songs include \"Comfortably Numb,\" \"Another Brick in the Wall (Part 2),\" \"Wish You Were Here,\" and \"Shine On You Crazy Diamond.\" Their music has been covered by countless artists, and their influence can be heard in many different genres of music.\n\nPink Floyd's live shows were known for their elaborate light shows, lasers, and pyrotechnics. They performed at numerous iconic venues, including the London Coliseum, the Royal Albert Hall, and the Earls Court Exhibition Centre. The band also made several memorable appearances at festivals such as the Bath Festival of Blues and Progressive Music and the Knebworth Festival.\n\nIn 1986, Pink Floyd released \"The Wall,\" a rock opera based on Roger Waters' experiences in school and his feelings about isolation and disillusionment. The album was a huge commercial success and spawned several hit singles, including \"Another Brick in the Wall (Part 2)\" and \"Comfortably Numb.\"\n\nPink Floyd continued to experiment with new sounds and techniques throughout their career. They incorporated elements of electronic music, synthesizers, and sound effects into their recordings, creating a distinctive and innovative sound. The band's final studio album, \"The Division Bell,\" was released in 1994 and marked the end of Pink Floyd's active career.\n\nOverall, Pink Floyd is a groundbreaking and influential rock band that has left an indelible mark on the music world. Their unique sound, thought-provoking lyrics, and iconic live shows have made them one of the most beloved and enduring bands of all time.","done":true,"context":[518,25580,29962,3532,14816,29903,29958,5299,829,14816,29903,6778,13,13,5618,508,366,2649,592,1048,349,682,383,18966,29973,518,29914,25580,29962,13,13,17565,29892,349,682,383,18966,29991,3118,310,278,1556,9849,293,322,7112,2556,7679,22706,310,599,931,29889,450,3719,471,8429,297,29871,29896,29929,29953,29945,491,8041,13923,2261,13158,29892,13853,28095,29892,14159,399,10412,29892,322,6123,22927,472,278,3014,310,12585,29889,2688,4689,408,263,19589,29899,20821,2318,29892,541,4720,8906,1009,1914,5412,6047,393,1999,2760,11643,287,295,293,7679,29892,6728,573,7679,29892,322,17986,4696,29889,13,13,29925,682,383,18966,338,2998,363,1009,5198,414,573,5735,3697,29892,19430,3769,1616,1287,29892,322,2714,29899,16123,17223,26627,1199,29889,11275,4696,3902,2361,963,267,1316,408,2834,29892,4892,29892,15483,29892,322,278,5199,4195,29889,3834,310,1009,1556,13834,20618,3160,376,1576,15317,19160,310,278,17549,1699,376,29956,728,887,399,406,2266,1699,322,376,1576,14406,1213,13,13,29903,2941,2261,13158,29892,278,3719,29915,29879,7601,4823,13236,322,3275,20982,391,29892,2175,278,3719,297,29871,29896,29929,29953,29947,2861,304,19119,9045,5626,29889,14159,399,10412,29892,1058,3614,975,408,278,1667,4823,13236,322,12760,391,29892,3897,278,19500,4889,5742,278,3719,29915,29879,6047,322,18551,29889,4699,11788,29885,473,29892,263,11210,391,322,20982,391,29892,8772,278,3719,297,29871,29896,29929,29953,29955,322,9488,263,1820,4509,2745,278,3719,29915,29879,2867,786,297,29871,29896,29929,29929,29953,29889,13,13,29925,682,383,18966,5492,3196,3994,1711,1035,13190,322,7825,5584,9150,20618,10106,1009,6413,29892,3704,376,4178,290,17778,21869,1699,376,19302,29881,280,1699,376,6039,1557,2955,491,14293,29879,1699,322,376,1576,9550,315,329,1213,2688,526,5545,697,310,278,14176,7679,22706,310,599,931,29892,411,263,25000,393,18172,304,8681,533,716,1176,800,310,2301,14722,322,24909,29889,13,13,9526,310,349,682,383,18966,29915,29879,1556,13834,12516,3160,376,1523,3921,2197,405,3774,1699,376,2744,1228,1771,860,297,278,14406,313,7439,29871,29906,511,29908,376,29956,728,887,399,406,2266,1699,322,376,2713,457,1551,887,14279,1537,22904,898,1213,11275,4696,756,1063,10664,491,2302,2222,17906,29892,322,1009,9949,508,367,6091,297,1784,1422,2531,690,310,4696,29889,13,13,29925,682,383,18966,29915,29879,5735,3697,892,2998,363,1009,19430,3578,3697,29892,1869,414,29892,322,11451,4859,3049,1199,29889,2688,8560,472,12727,9849,293,6003,1041,29892,3704,278,4517,1530,895,398,29892,278,7021,10537,6573,29892,322,278,5290,3137,9245,1222,6335,654,11319,29889,450,3719,884,1754,3196,26959,519,21712,472,29482,1338,1316,408,278,28256,8518,310,23434,322,20018,573,6125,322,278,476,484,29890,12554,8518,29889,13,13,797,29871,29896,29929,29947,29953,29892,349,682,383,18966,5492,376,1576,14406,1699,263,7679,14495,2729,373,14159,399,10412,29915,27482,297,3762,322,670,21737,1048,11695,362,322,766,453,3958,358,29889,450,3769,471,263,12176,12128,2551,322,29178,287,3196,7124,22102,29892,3704,376,2744,1228,1771,860,297,278,14406,313,7439,29871,29906,5513,322,376,1523,3921,2197,405,3774,1213,13,13,29925,682,383,18966,7572,304,7639,411,716,10083,322,13698,10106,1009,6413,29889,2688,11039,630,3161,310,27758,4696,29892,14710,267,19427,29892,322,6047,9545,964,1009,2407,886,29892,4969,263,8359,573,322,24233,1230,6047,29889,450,3719,29915,29879,2186,8693,3769,29892,376,1576,7946,10914,1699,471,5492,297,29871,29896,29929,29929,29946,322,10902,278,1095,310,349,682,383,18966,29915,29879,6136,6413,29889,13,13,3563,497,29892,349,682,383,18966,338,263,5962,1030,5086,322,7112,2556,7679,3719,393,756,2175,385,1399,295,1821,2791,373,278,4696,3186,29889,11275,5412,6047,29892,2714,29899,16123,17223,26627,1199,29892,322,9849,293,5735,3697,505,1754,963,697,310,278,1556,1339,8238,322,1095,3864,22706,310,599,931,29889],"total_duration":46945500291,"load_duration":9055150333,"prompt_eval_count":31,"prompt_eval_duration":241905000,"eval_count":737,"eval_duration":37647125000}

Using it from a Python library

Ollama has libraries for different programming languages, including Python. You can use it from your Python code like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


import ollama

response = ollama.chat(model='llama2:13b-chat', messages=[
    {
        'role': 'user',
        'content': 'Can you talk me about Pink Floyd?',
    'stream': False,
    },
])

print(response['message']['content'])

Additional capabilities

With Ollama you can also create a new model based on an existing one. Check the official documentation for more information.

In my examples I used the llama2:13b-chat model, but there are other models available, you can find the full list here.

Conclusion

I think Ollama is a great tool for people who want to experiment with Large Language Models without the need of a cloud service. It’s easy to use and it’s available for different programming languages. I’m looking forward to see how it will evolve in the future.