Introduction
Imagine having the power of ChatGPT right at your fingertips—available 24/7, completely offline, and fully customizable to suit your needs. Whether you’re a developer looking to integrate AI into your applications, a researcher exploring natural language models, or simply a tech enthusiast curious about the possibilities of local AI, setting up a local instance of ChatGPT is easier than you might think.
Before getting into how to set it all up we need to first talk about what we will be doing. To get this set up we need to set up WSL on Windows. WSL (Windows Subsystem for Linux) is a compatibility layer for running Linux-based operating systems directly on a Windows machine, without requiring a virtual machine or dual-boot setup. It allows developers and users to run Linux command-line tools, utilities, and even GUI applications natively on Windows
With WSL we will be using Ollama. Ollama is a platform and tool designed to make it easier to run large language models (LLMs) like GPT on your local system. It provides a streamlined interface for downloading, running, and managing AI models locally, offering users a way to leverage the power of language models without relying on cloud services. This approach is particularly appealing for individuals or organisations concerned about data privacy, latency, or the cost of using cloud-based AI services.
Finally, we will be using OpenWebUI for our Graphical Interface to run our models. OpenWebUI is an open-source platform designed to provide a user-friendly web-based interface for running and interacting with AI models, particularly large language models (LLMs), on local or remote systems. It aims to simplify the process of managing and using AI tools by offering a clean, accessible interface that eliminates the need for command-line interactions.
In this guide, we’ll walk you through the entire process, from understanding the system requirements to configuring the model on your local machine. By the end, you’ll not only have a functional AI at your disposal but also a deeper understanding of how it works and how you can make it your own. Get ready to unleash the power of AI—right on your desktop!
Step 1: Enable Windows Features
Before we can set up WSL, we need to enable the Windows Subsystem for Linux feature on your system. To do this, simply type Windows features into the search bar and select the first option that appears.
Next, scroll down to find Windows Subsystem for Linux, make sure the box is checked, and then click OK to enable the feature.
After clicking OK, the system will begin downloading the necessary files and applying the required changes to complete the setup process.
After the changes have been applied, you’ll be prompted to restart your computer to finalize the setup. Go ahead and restart to ensure everything is configured properly.
After your computer restarts, open the search bar and look for “Terminal.” Launching the terminal will allow you to proceed with the next steps in the setup process.
Step 2: Set Up Ubuntu and Ollama
After opening the terminal, enter the following command to install Ubuntu, the Linux distribution we’ll be using throughout this tutorial. This will set up the foundation for all the steps to follow.
After installing Ubuntu, you’ll be prompted to create a username and password for your system. Simply enter your desired details, and once completed, you’ll be logged into your new Ubuntu environment, ready to start working.
If you close your terminal and later need to access your Ubuntu machine, don’t worry—it’s easy to find. Simply open a new terminal, click on the dropdown arrow, and select the Ubuntu option. This will load your Ubuntu environment and log you in, ready to pick up where you left off.
To confirm that Ubuntu is installed and running, you can use a simple command to check the version. For this tutorial, we’re using Ubuntu version 24.04, but this command will work for verifying any version you have installed.
After setting up your machine, the next step is to update and upgrade the system packages using two simple commands. Think of this as the Linux equivalent of running Windows Update—it ensures that your system is up-to-date with the latest patches and improvements. This is a best practice for any new Linux installation to ensure smooth performance and compatibility.
Next, we’ll download the installation script from Ollama. Once downloaded, simply run the script, and it will take care of downloading and installing Ollama on your system. This step sets the foundation for running and interacting with Large Language Models (LLMs).
This uses curl to silently fetch the script while handling redirects and server errors, then pipes it directly into the sh shell for execution.
After installing Ollama, you’ll notice that it’s not functional just yet because no Large Language Models (LLMs) are installed. An LLM is a type of AI model designed to understand and generate human-like text. These models are built using deep learning techniques and trained on massive datasets to grasp the structure, context, and patterns of natural language. To get started, you’ll need to download and integrate an LLM with Ollama, enabling it to perform tasks like answering questions, generating content, and more.
For this tutorial, we’ll be downloading two powerful models: Llama 3 and Mistral. These advanced Large Language Models (LLMs) are developed by Meta AI and Mistral AI, respectively, and each brings unique capabilities to the world of artificial intelligence. Their cutting-edge performance makes them ideal for exploring the full potential of OpenWebUI. Let’s get started with downloading these models!
While these are two popular models, you’re not limited to just these options. To explore and download other models, visit Ollama’s website, where you’ll find a variety of models available, along with detailed documentation on how to integrate them.
After selecting the models you want to download, use the command below, replacing the placeholder with the specific model name you wish to download. You can find the exact command in the model’s documentation by clicking on its details. The command will pull and download the model to your system. Keep in mind that download times may vary depending on the model size and your internet speed. For example, to download models like Llama 3 and Mistral like in this guide, you can use the following commands:
Once your models are downloaded, you can choose to run them directly from the terminal. To start a model, simply use the following command:
Once the model loads, you can start interacting with it by asking questions and receiving detailed responses. For example, I asked it to write a 2,000-word report on artificial intelligence, and it generated a comprehensive and well-structured answer, showcasing its powerful capabilities.
The speed at which your system can generate text depends largely on your hardware. For optimal performance, it’s recommended to use a GPU, as it significantly accelerates processing compared to a CPU. If you’re relying solely on a CPU, text generation might be noticeably slower. For those with an NVIDIA GPU, there are specific commands available to optimize performance and get the most out of your hardware.
The command watch -n 0.5 nvidia-smi runs the nvidia-smi utility every 0.5 seconds, providing real-time monitoring of your NVIDIA GPU’s performance metrics, such as utilization, memory usage, temperature, and power consumption. It uses the watch command to refresh the output at the specified interval, making it a valuable tool for tracking GPU activity during intensive tasks like AI model training. This allows you to observe GPU behavior and resource usage in real time to optimize performance or diagnose issues.
Step 3: Set Up OpenWebUI
Now that Ollama is up and running, and you’re ready to interact with Large Language Models (LLMs), you’re almost there! While running LLMs in the command line is functional, it has its limitations. That’s where OpenWebUI comes in—a user-friendly, web-based interface that makes interacting with models much easier and more intuitive. To set this up, we’ll be installing Docker, which will create a container to host the OpenWebUI image on your system. Let’s get started!
Before we can install OpenWebUI we need to run some commands.
These commands set up the system to install Docker securely by creating a directory for storing GPG keys, downloading Docker’s GPG key to verify the authenticity of its packages, and adjusting permissions to ensure the key is readable by the package manager. This process ensures that the Docker installation will be authenticated and trusted, preventing unauthorized or tampered software from being installed.
Now we need to run these commands.
These commands add Docker’s official repository to the system’s package sources, ensuring compatibility with the system’s architecture and Ubuntu version, while securing the process with Docker’s GPG key for package verification. The repository is configured dynamically based on the system’s version, and the local package index is updated to include Docker’s packages, preparing the system for secure and seamless Docker installation.
Next, we’ll run a command to download and start the OpenWebUI Docker container, setting up the environment for seamless access.
This command installs Docker’s essential components, including the Docker engine, command-line tools, and plugins for advanced image building and multi-container orchestration. It sets up everything needed to run, build, and manage containers efficiently on your system.
To start your OpenWebUI Docker container, you can use the following command.
This command starts a Docker container named open-webui using the open-webui image from GitHub’s container registry. The container runs in the background, uses the host’s network for direct access to services, and mounts a persistent volume to store data. It is configured to interact with an Ollama service running on the host via a specified environment variable and is set to restart automatically if it stops unexpectedly, ensuring reliable and continuous operation.
To check that your OpenWebUI docker container is up and running we can enter this command.
If your Docker container’s status shows as Up and Healthy, it means everything is running smoothly, and you should be able to access OpenWebUI through your web browser without any issues.
The first step is to ensure that Ollama is running. You can do this by entering http://localhost:11434 in your web browser. If the page confirms that it’s running, then everything is set up correctly, and you’re ready to move forward!
Once you’ve verified that Ollama is up and running, you can access OpenWebUI by navigating to http://localhost:8080 in your web browser. This will take you to the OpenWebUI welcome page. To begin, simply click on the Get Started button and follow the prompts to start exploring the platform!
Next, you’ll arrive at the account setup page to get started with OpenWebUI. Simply enter your name, email address, and a password of your choice. Be sure to remember your password, as there doesn’t appear to be a way to reset it later. Once that’s done, you’re all set to dive in!
After logging in, you’ll be greeted with the release notes. Feel free to skim through them for updates or changes, but you can close them to proceed and get started with the platform.
Once you’re on the main screen, you’ll notice it looks quite familiar if you’ve used ChatGPT before. To select the model you want to interact with, simply use the dropdown menu located in the top-left corner. It’s straightforward and user-friendly, making it easy to get started with the model of your choice.
After selecting your model, you’re ready to start interacting with it and asking questions. To demonstrate its capabilities, I asked the model to write a 2,000-word report on machine learning—and the results were impressive! Whether you’re looking for detailed insights or creative content, the possibilities are endless once your model is set up.
Conclusion And Final Notes
And there you have it—a basic guide to setting up OpenWebUI on WSL! While this post covers the essentials, OpenWebUI offers many additional features, such as adding multiple users, training models on your own notes or documents, and even integrating image generation. I wanted to keep this guide simple to help you get started. Now you don’t need to rely on ChatGPT—you can run your own LLMs right from your system. Happy exploring
One final note: if you close your Ubuntu WSL instance, you won’t be able to access OpenWebUI. Make sure to keep your terminal open. However, there’s no need to manually start your Docker container each time you open your WSL instance—Docker is set up to start automatically, so you can jump right back in without running any additional commands.
In my next blog post, I’ll dive into generating images using Stable Diffusion and show you how to seamlessly integrate it with your OpenWebUI container. It’s going to be an exciting guide packed with practical tips, so make sure to stay tuned!