Many engineers working on embedded AI development may have had similar experiences: the model runs fine on the PC, but once deployed to RK3588, RV1106, or other boards with NPUs, either the accuracy doesn't match or the speed doesn't meet expectations. At this point, the conversion and optimization services of the RKNN toolchain become critical. So what exactly does such a service include? Based on my hands-on project experience, let me break it down for you.
First is model conversion. This is the most basic step. The service converts your trained PyTorch, ONNX, or TensorFlow models into the .rknn format using the RKNN-Toolkit. But it's not just about running a single command; operator mapping issues also need to be handled. Many models contain custom operators or dynamic shapes that the native toolchain does not support, so you need to manually modify the network structure or supplement operator implementations. A reliable service will help you check operator compatibility one by one, resolve errors encountered during conversion, and ensure the model is fully deployed, rather than simply saying "this model is not supported."
Next is quantization and accuracy calibration. RKNN typically uses INT8 quantization to speed up inference, but accuracy drop after quantization is a common issue. The service provider will use your validation set images to run simulated quantization on the PC, analyze the accuracy loss of each layer, identify layers that are sensitive to quantization, and then adjust the quantization strategy accordingly, such as mixed quantization, keeping FP16 for specific layers, or optimizing the data preprocessing pipeline. This step directly affects the actual performance of the product and requires iterative testing.
Then comes performance optimization. A successful conversion doesn't mean it runs fast enough. Real optimization needs to go deep into the hardware characteristics of the NPU. For example, operator fusion combines consecutive operations like Conv+ReLU+BN to reduce memory reads and writes; memory reuse plans buffers for intermediate tensors to avoid frequent allocation and deallocation; and multi-core scheduling, for a multi-core NPU like the RK3588, reasonably allocates the model to different cores to reduce inter-core communication overhead. Experienced service providers also adjust data layouts and DMA transfer methods to achieve higher NPU utilization.
In addition to these, the service also covers end-side integration support in Linux or Android environments, including writing inference example code, handling camera input/output formats, and connecting to peripheral interfaces such as RTSP or V4L2. If you encounter weird issues like logits errors, driver crashes, or the NPU not being fully utilized, a mature service will also help you troubleshoot at the system level, rather than focusing only on the model itself.
Overall, RKNN services are not simply about "converting a format," but a systematic project that spans model analysis, quantization tuning, and hardware adaptation. Partnering with a team experienced in RK platform development can save you a great deal of time avoiding pitfalls.