Local generative Artificial Intelligence LLM gateway for research and teaching

Aalto Research Software Engineers (Science-IT) and the IT Services' AI Task Force are pleased to announce the release of the local Large Language Model (LLM) gateway: a fully local LLM endpoint accessible through an Application Programming Interface (API) and designed for programmatic use in research, teaching and development.

First some clarity about all this jargon: what is an LLM endpoint? And what is an API? 

A large language model (LLM)is the text-generating engine at the core of tools such as chatbots, coding assistants or AI agents. But if the LLM is the engine of the car, you still need the rest of the car for actually using the powerful engine: here comes inference.

Inference is the process that actually uses the AI models and turns your input text (prompt) into more text (response). You are all familiar with this way of interacting with an LLM via a chatbot interface like the Aalto AI Assistant.

An endpoint accessed via an Application Programming Interface (API) is the technical doorway to the language model and inference. You are not limited by the chat user interface, but can now connect to the LLM with a software script, an AI agent, a coding assistant such as Codex or Claude Code, a new custom application, or a teaching exercise, going beyond chatbots and making the LLM a new building block in your research and teaching.

Why is this important? Why do we need this?

Big tech AI tools are powerful and easy to use, but they are not always the right environment for university work. Research data may include confidential interviews, unpublished manuscripts, sensitive project material, source code, and so on. 

Teaching use-cases may involve student work which should not end up in some unclear “cloud” hosting environment. While Aalto has confidential agreements with EU-based Microsoft Azure endpoint, the new, truly local hosting environment makes it an independent alternative that does not depend on any changes of policies or costs from external providers.

A local LLM endpoint gives Aalto more control

A local LLM endpoint gives researchers and teachers the ability to work with local AI models, without needing a powerful computer to run the LLM or worrying about installing disk-space-intensive local AI models. Full control of the AI infrastructure also makes it easier to deal with reproducibility – older versions of the models will not disappear as can happen with major providers – and improves resilience by avoiding vendor lock-in. In addition, having full control ensures digital sovereignty, which is becoming increasingly important in today's global geopolitical context. 

For more information on how to use the local LLM endpoint, please see the Local LLM web APIs page.

Aalto AI strategy toolkit

The local LLM gateway adds another component to Aalto’s growing AI strategy toolkit, driving responsible AI adoption in research and education. A recent update to the Aalto AI Assistant has included a fully local GPT-OSS model option, a perfect entry point to test local LLMs.

Speech2Text gives researchers a way to transcribe interviews and audio material on Aalto's infrastructure using a local OpenAI Whisper model. This is particularly important for confidential interviews and qualitative research workflows, where sending recordings to external tools may not be appropriate.

For researchers who need to experiment with more exotic open LLMs, the Triton Aalto HPC cluster already mirrors nearly 300 open-weights models that can be used for inference on the 280+ GPUs available on the cluster. Triton is also the safest place to get started with AI coding agents. For more information on the AI agents. please see the guideline AI Agents on HPC.

Finally, when it comes to teaching, the MyCourses AI Assistant brings Aalto’s AI infrastructure directly into teaching and learning. It is a Moodle plugin that uses the same backend as the Aalto AI Assistant. Teachers can add it to their courses, allowing students to chat about course topics using selected course materials as context for the large language model, without relying on separate external AI tools. 

Do you have a vision for the next AI tool to add to the Aalto toolkit?

Experimenting fast and testing what works and what does not is easier when done together with a team of experts that can work with you or for you! 

If you have research-specific software development ideas (with or without “AI”) please get in touch with Aalto Research Software Engineers. 

The IT Services' AI task force is happy to hear about ideas that clearly go beyond single research projects. Based on your requests we are able to drive things forward and deliver.