Private On-Premise AI Cloud: A Step-by-Step Guide

According to our understanding, a private AI cloud is a cloud infrastructure where only you as the key holder have access to the data. This means that there is no third party standing between you and your data that can prevent or control access. Neither authorities nor third parties can access your data unless they can crack 256 AES encryption. At the same time, you have access to your data and AI services from anywhere via the internet through a publicly accessible website.

Unlike public cloud services such as ChatGPT, Google Cloud, or Microsoft Azure, where your data is stored on third-party servers and providers potentially have access, with a private AI cloud, you retain complete control over your data and the AI models used.

Advantages of a private AI cloud

The main advantages of a private AI cloud are:

1. Full data sovereignty: You retain complete control over your data.
2. Data protection and data security: Sensitive information does not leave your infrastructure.
3. Compliance: Easier compliance with data protection regulations such as the GDPR.
4. Independence: No dependence on external providers and their price changes.
5. Customizability: Tailored solutions for your specific requirements.

Use cases

In Simpliant's early days, we invested in the best speech recognition model on the market. The license cost was around €1,400 plus a microphone with high-end recording capabilities for about €350 per workstation. This investment paid off because it significantly accelerated our writing of audit reports, memos, and legal analyses.

When ChatGPT came along, especially the ChatGPT phone app, I was amazed at the superior quality of the speech recognition feature. Using my iPhone's regular microphone, it was far more accurate and faster than the expensive solution I had used before.

However, for our use case, it was not a practical day-to-day work tool, as we needed full control over our data, especially sensitive personal and customer information.

The private AI cloud can be used for various use cases:

1. Cloud Storage: Similar to Dropbox or OneDrive, but without access by third parties.
2. Speech recognition and transcription: Converting speech to text for meetings, interviews, or dictations.
3. Text generation and analysis: Creating reports, summaries, or analyses.
4. Document processing: Extracting information from documents.
5. Local AI assistants: Chatbots or assistants tailored to your specific needs.

In the early days of Simpliant, we invested in the best speech recognition model on the market. The licensing costs were around €1,400, plus a high-end microphone with premium recording capabilities for about €350 per workstation. This investment paid off as it significantly accelerated our process of drafting audit reports, memos, and legal analyses.

When ChatGPT emerged, particularly the ChatGPT app for mobile devices, we were impressed by the superior quality of its speech recognition feature. Even with a standard iPhone microphone, it was far more accurate and faster than the expensive solution we had previously used.

However, for our use case, it was not a practical daily work tool, as we required full control over our data.

In our setup, the data is located on a computer in our office. This is, of course, just one possibility. You could also place your computer with a hosting provider (co-hosting) or operate a virtual computer on a hosting platform.

Hardware requirements

To run AI applications on your computer with sufficient performance, you need:

1. Powerful graphics card: A GPU with enough VRAM, such as an NVIDIA GeForce RTX 4070 or better.
2. Sufficient RAM: At least 16 GB, ideally 32 GB or more.
3. Fast processor: A modern multi-core processor.
4. Sufficient storage space: SSD storage for fast access to data and models.

Software components

Our solution is based on the following software components:

1. Operating system: Linux Ubuntu Server
2. AI models:

   • Whisper for speech recognition
   • Llama 8B and Qwen 2.5 as local LLMs
3. Container technology: Docker for easy deployment and scaling.
4. Backend: Flask app for the API interface.
5. Authentication: Custom authentication solution for secure access.

Access and encryption

In our setup, data access works via a Virtual Private Server (VPS) with a third-party provider. This provides the so-called access layer and is accessible via a public IP or a public website over the internet.

The connection between the VPS and the local computer is established via a Virtual Private Network (VPN). This means that an encrypted connection is established between the VPS and the local computer or server. We use WireGuard for this.

Security measures

You should harden your VPS with basic security measures:

1. Installation of a firewall
2. Installation of Fail2Ban
3. Changing your SSH port
4. Deactivation of password access
5. Changing the root user

Additionally, you can protect your domain by using third-party services such as Cloudflare to protect your domain against DDoS attacks. However, we advise against using the Cloudflare Tunnel, as Cloudflare would then have access to your data.

Your access layer should be protected by a service such as Authentik. You should ensure that access is only possible with two-factor authentication. Additionally, you can block IP addresses that do not belong to your country region.

Network isolation

Ideally, your private AI cloud should run isolated from your other devices in a Virtual LAN. This can be done through a physical device with WLAN functionality, such as a router or a switch.

Implementation steps

1. Prepare hardware

Ensure that your computer meets the necessary hardware requirements, especially a powerful GPU for AI workloads.

2. Install operating system

Install a Linux-based operating system such as Ubuntu Server for maximum control and security.

3. Set up AI models

Install the desired AI models such as Whisper for speech recognition and Llama or Qwen for text generation.

4. Configure network

Set up a VPS and configure WireGuard for a secure VPN connection between your local server and the VPS.

5. Implement security measures

Harden your VPS and your local server with the security measures mentioned above.

6. Develop user interface

Develop a user-friendly interface for accessing your AI services, either as a web app or as a native application.

7. Implement backup strategy

Set up regular backups to avoid data loss. Ideally, you can create virtual machines using free services such as Proxmox, on which your services run. You can mirror, copy, and create backup routines for these virtual machines.

Advantages in detail

1. Full data sovereignty: You have complete control over your data and do not need to enter into contracts with third parties, apart from the data processing agreement with your VPS hosting provider.

2. Data protection and data security: Your sensitive data does not leave your infrastructure, which is particularly important for companies working with confidential information.

3. Compliance: Easier compliance with data protection regulations such as the GDPR, as you know exactly where your data is stored and who has access to it.

4. Independence: No dependence on external providers and their price changes or service discontinuations.

5. Customizability: You can adapt the AI models and services exactly to your needs.

Challenges and limitations

1. Technical complexity: Setting up and maintaining a private AI cloud requires technical expertise.

2. Initial costs: The acquisition of the necessary hardware and the setup can be initially costly.

3. Maintenance effort: Your private AI cloud should ideally have redundancies, i.e., you should ideally have two computers at two different locations with different internet connections and various access points via VPS.

4. Power consumption: Powerful hardware for AI workloads consumes correspondingly high amounts of electricity.

5. Limited computing power: Compared to large cloud providers, the available computing power is limited.

Gathering data and strategically planning AI implementation in your organization will be crucial to gaining a competitive advantage in the years to come. In particular, we believe that it is not only the capacities of the large language models that matter as a foundation, but the data that you generate and intelligently integrate based on your specific business model.

To maintain a high level of data security, intellectual property protection, data protection, and compliance while keeping costs low and not relying on big tech monopolists, this could be the most crucial time for your company to invest in AI.

We use our private AI cloud daily to transcribe texts and use AI in a secure and legally compliant way.

Together with our technology partners, we offer turnkey on-premises AI cloud solutions tailored precisely to your requirements—whether in your own data center or a public cloud environment.

As a law firm and consultancy, we ensure that all regulatory requirements, particularly GDPR compliance, are fully met. Contact us for a non-binding consultation!