For the entirety of 2023, the progress made in the AI section is tremendous. But near the end of 2023, AI has been put in focus once again with the release of the Intel Core Ultra with the new Meteor Lake architecture. Specifically, on-device AI. This distinction is important because currently, ChatGPT or Google Bard or Microsoft Copilot are all processed on the server-side. That means we need to be connected to the internet our prompts will be sent to the server to be processed, then the answer will be sent back to us.
This Intel Core Ultra processor has a built-in neural processing unit, or NPU, that will process our prompts locally on our device, without the need to connect to the internet. I’m sure many people have questions – so let’s go through them in this video.
Having an NPU on your device is not new, actually. Phones have had a built-in NPU chip for many years and even AMD has an NPU built into their SoCs too.
But my biggest question is – how’s the new Meteor Lake processor? Honestly, I had the same question too. This new Intel Core Ultra series of SoCs are using the new Meteor Lake architecture. One of the laptops that was announced alongside the arrival of the Meteor Lake SoCs is the ASUS Zenbook 14 OLED UX3405MA which is powered by the latest Intel Core Ultra 7 155H SoC.
From the list of SKUs provided by Intel, it seems like every single SoC that is based on the Meteor Lake architecture have the same Intel AI Boost neural processor.
So, I decided to do a few benchmarks of my own. I did Geekbench at first but the results seem to be… inconclusive. I needed more freedom to control some parameters. Then, I used UL Procyon instead. This benchmark tool has all of the parameters I wanted to use, and so the benchmarks began.
First, we have to select what API to use. There are two main ones to choose from – Windows ML or Intel OpenVINO.
Windows ML is an API built directly into Windows 11 and it’s open to any hardware maker and software programmer to use. You might have heard of the name “DirectML” but that’s a component under the Windows Machine Learning umbrella.
Intel OpenVINO is an API developed by Intel – obviously – but this API is made specifically to work better with Intel’s own hardware. So, let’s put it to the test.
Let’s compare the AI inferencing performance using the CPU with different APIs first. As we can see, OpenVINO performs a lot better than Windows ML.
Now let’s switch to the GPU. This time, OpenVINO doubles the performance of Windows ML.
Okay, remember that the Meteor Lake SoCs have a built-in NPU? Yeah, unfortunately, we can’t use the NPU with Windows ML so we can’t compare the performance difference. As we can see here, the performance of the NPU is more than 4x better than the CPU using OpenVINO.
In terms of raw AI performance, the GPU is the best, then comes the NPU, and lastly the CPU.
My next big concern is the power consumption. How do they vary? Is the NPU the perfect choice for both speed and power efficiency? I redid the tests using OpenVINO only and this is the graph that we got. Those dips are because the one test has ended and the next test is loading – and using the CPU, we can see both the power consumption and temperature have a lot of spikes, taking about 21 minutes to complete the benchmark.
Switching over to the GPU, it doesn’t have extreme spikes like the CPU does and takes less time to complete the test – about 20 and a half minutes.
Now is time for the star of the show – the NPU. The power draw is significantly lower and the temperature seems to be lower too. However, it takes nearly 25 minutes to complete the benchmark.
I then recompiled the graphs. The first one is this – the average power consumption. From the data that we’ve gathered, we can see that the NPU is the most efficient, then comes the GPU, and finally the CPU.
What about the average temperature then? Surprisingly, they’re all pretty close in terms of the average temperature. Once again, the NPU is the most efficient, then comes the GPU, and last is the CPU.
In conclusion, for the Meteor Lake architecture in particular, we can say that for all AI inferencing tests, using the NPU is always the best. However, if you need something to be done as fast as possible, use the GPU but at the expense of power. And never use the CPU to do AI tests as it is going to be the slowest and also the least efficient.
But why? Why have an NPU in your device? Well, the simplest answer is to run AI tasks on your device locally. There are a lot of benefits by doing this. The most obvious is that we can run everything locally so we don’t need an internet connection. Even at the most remote corners of the earth, we can still use these AI tools. And since we don’t have to send data back to the server, our privacy is protected. I’m sure many of you have seen headlines saying that many companies are banning their employees from using ChatGPT because of data and security concerns.
Since we don’t have to connect to the internet, we might even be more energy-efficient. I don’t have the numbers to compare, though. However, readily-available AI tools that run locally on our devices are still scarce as of now. We do have Stable Diffusion WebUI by Automatic1111 and GIMP, but setting them up isn’t exactly user-friendly. I do think that when the hardware is available, more software that runs AI locally will start to appear.
Perhaps even ChatGPT or Copilot will move on to run locally on our devices so that these companies can save on server costs and also be more efficient at the same time, and we don’t have to worry about our data getting used to help build their dataset.
Also, the new Meteor Lake SoCs also come with a new integrated Intel Arc GPU too – to check out the gaming performance, click the link at the top right corner or the end screen card – same goes to the review of this laptop. It’s a beautiful laptop, honestly.