Nvidia’s new AI supercomputer “Project Digits”

Although AMD and Intel are also involved in the AI chip market, we have only focused on Nvidia for the time being due to their market dominance. You will quickly come across the “Jetson” device family. Nvidia itself advertises these primarily as a portable solution for robotics and local applications.

Firstly, there is the aforementioned Project DIGITS. The big advantage of this machine is that it has much more VRAM than standard graphics cards (128GB compared to 24GB for an RTX 4090). Support for up to 200B (200 billion) parameters was promised. This is an approximate indication of how large an AI model is. For comparison: ChatGPT-3.5 has 175B parameters and ChatGPT-4 has 1000B parameters.

On the other hand, there are smaller devices such as the Jetson Orin Nano Super. This is intended more for testing technologies and is comparable to a Raspberry Pi for the AI world. With 8GB of VRAM, it doesn’t come close to high-end graphics cards. But with a price of only 250$ and 25 Watt power consumption it still plays in a league of its own.

These devices may not be suitable for taking on OpenAI directly. However, they are definitely suitable for specific use cases or for getting your hands dirty for the first time. Especially if you are forced to do without cloud solutions anyway due to data security requirements.

In addition to the benefits mentioned above, there is also a software package called “Jetson AI Lab“. This is a collection of tools and instructions curated by Nvidia and tailored to the various devices in the Jetson family. The applications range from text, image and audio generation to robotics.

One question that quickly arose was how an AI can now be fed with its own data. Roughly speaking, there are currently two approaches. Retrieval augmented generation (RAG) and fine-tuning. Both computers can be run completely isolated. A cloud connection is therefore not mandatory.

Fine-Tuning

Let’s take a look at fine-tuning first. This involves taking an existing AI model and training it further with your own data. Here you take advantage of the fact that the AI model already has a great deal of general knowledge and only needs to be adapted. This is much easier than training a new AI model from scratch. The disadvantages, however, are that the knowledge of the new AI model is frozen after training until it is retrained. In addition, the training data must be prepared and converted into the correct format.

Good applications for this function are, for example, when email responses should be suggested by an AI and the suggestions should be better tailored to the characteristics of the company. Past email conversations can be used as a training data set.

Retrieval Augmented Generation (RAG)

RAG’s approach consists of several steps and components. First, the company’s own data is indexed. This means that it is classified and sorted into a database. This is often a vector database, which makes it possible to search efficiently for relevant data snippets. Let’s take the CVs of all employees as an example. Our system is now ready for use.

As an example of a question, let’s take “Did anyone in our workforce work at Migros in 2012”. The vector database is now used to search for the data snippets that could best match the question. A selection of the most suitable data is forwarded together with the question to a general AI, such as ChatGPT. Together with the forwarded data, the AI can now make a statement that it could not make without the data. The vector database is used here to pass on only the relevant data snippets to the AI. Simply passing on all the data would be so much information that the AI would not be able to process all the data due to the so-called token limit.

RAG is generally slower than pure AI. But it still has some advantages. For one thing, the vector database can be updated much more easily, so the data is more up-to-date. In addition, it is more transparent where the information for the AI answers comes from.

With the specialized computers from Nvidia and the open source tools available, the barriers to building your own local AI are lower than ever before. The unique advantages, such as data security and full control over your own data, are in many cases an argument in favor of standing on your own two feet and building your own AI system.

Do you have exciting use cases in your company that you want to test?

Get in touch with us and we will help you!