基于高通公司AI Hub Models的On-Device AI学习:Introduction to On-Device AI

news/2024/7/22 13:41:57/文章来源:https://blog.csdn.net/shizheng_Li/article/details/139121167

Introduction to On-Device AI

本文是学习 https://www.deeplearning.ai/short-courses/introduction-to-on-device-ai/这门课的学习笔记。

在这里插入图片描述

What you’ll learn in this course

As AI moves beyond the cloud, on-device inference is rapidly expanding to smartphones, IoT devices, robots, AR/VR headsets, and more. Billions of mobile and other edge devices are ready to run optimized AI models.

This course equips you with key skills to deploy AI on device:

  • Explore how deploying models on device reduces latency, enhances efficiency, and preserves privacy.
  • Go through key concepts of on-device deployment such as neural network graph capture, on-device compilation, and hardware acceleration.
  • Convert pretrained models from PyTorch and TensorFlow for on-device compatibility.
  • Deploy a real-time image segmentation model on device with just a few lines of code.
  • Test your model performance and validate numerical accuracy when deploying to on-device environments
  • Quantize and make your model up to 4x faster and 4x smaller for higher on-device performance.
  • See a demonstration of the steps for integrating the model into a functioning Android app.

Learn from Krishna Sridhar, Senior Director of Engineering at Qualcomm, who has played a pivotal role in deploying over 1,000 models on devices and, with his team, has created the infrastructure used by over 100,000 applications.

By learning these techniques, you’ll be positioned to develop and deploy AI to billions of devices and optimize your complex models to run efficiently on the edge.

文章目录

  • Introduction to On-Device AI
    • What you’ll learn in this course
  • Why on-device?
  • L2: Deploying Segmentation Models On-Device
    • Exercise: Try another variant of FFNet
    • Setup AI Hub for device-in-the-loop deployment
    • Run on a real smart phone!
    • On Device Demo
  • L3: Preparing for on-device deployment
    • Capture trained model
    • Compile for device
    • Exercise: Try different runtimes
    • On-Device Performance Profiling
    • Exercise: Try different compute units
    • On-Device Inference
    • Get ready for deployment!
  • L4: Quantizing Models
    • Setup calibration/inference pipleline
    • Setup model in floating point
    • Prepare Quantized Model
    • Perform post training quantization
    • Run Quantized model on-device
  • Device Integration
  • Appendix - Building the App
    • TensorFlow Lite
    • Delegation
      • Qualcomm QNN delegate
    • End-to-end examples
    • Semantic segmentation code
  • Afterword

Why on-device?

在这里插入图片描述

applicable use-cases

在这里插入图片描述

Why on-device

在这里插入图片描述

Device in-the-loop deployment

在这里插入图片描述

on-device generative AI

在这里插入图片描述

L2: Deploying Segmentation Models On-Device

设备上AI的应用

在这里插入图片描述

Image segmentation

在这里插入图片描述

types of image segmentation

在这里插入图片描述

image segmentation的应用

在这里插入图片描述

semantic segmentation模型

在这里插入图片描述

FFNet

在这里插入图片描述

FFNet Paper

在这里插入图片描述

from qai_hub_models.models.ffnet_40s import Model
from torchinfo import summary
# Load from pre-trained weights
model = Model.from_pretrained()
input_shape = (1, 3, 1024, 2048)
stats = summary(model, input_size=input_shape, col_names=["num_params", "mult_adds"]
)
print(stats)

Output

Loading pretrained model state dict from /home/jovyan/.qaihm/models/ffnet/v1/ffnet40S/ffnet40S_dBBB_cityscapes_state_dict_quarts.pth
Initializing ffnnet40S_dBBB_mobile weights
==============================================================================================================
Layer (type:depth-idx)                                       Param #                   Mult-Adds
==============================================================================================================
FFNet40S                                                     --                        --
├─FFNet: 1-1                                                 --                        --
│    └─ResNetS: 2-1                                          --                        --
│    │    └─Conv2d: 3-1                                      864                       452,984,832
│    │    └─BatchNorm2d: 3-2                                 64                        64
│    │    └─ReLU: 3-3                                        --                        --
│    │    └─Conv2d: 3-4                                      18,432                    2,415,919,104
│    │    └─BatchNorm2d: 3-5                                 128                       128
│    │    └─ReLU: 3-6                                        --                        --
│    │    └─Sequential: 3-7                                  300,160                   9,797,895,296
│    │    └─Sequential: 3-8                                  1,411,840                 11,542,727,424
│    │    └─Sequential: 3-9                                  3,900,288                 7,977,571,200
│    │    └─Sequential: 3-10                                 7,071,360                 3,617,592,960
│    └─FFNetUpHead: 2-2                                      --                        --
│    │    └─Sequential: 3-11                                 1,208,147                 26,571,541,312
==============================================================================================================
Total params: 13,911,283
Trainable params: 13,911,283
Non-trainable params: 0
Total mult-adds (G): 62.38
==============================================================================================================
Input size (MB): 25.17
Forward/backward pass size (MB): 1269.30
Params size (MB): 55.65
Estimated Total Size (MB): 1350.11
==============================================================================================================

utils.py

import os
from dotenv import load_dotenv, find_dotenvdef load_env():_ = load_dotenv(find_dotenv())def get_ai_hub_api_token():load_env()ai_hub_api_token = os.getenv("AI_HUB_API_KEY")return ai_hub_api_token

Exercise: Try another variant of FFNet

# High resolution variants
from qai_hub_models.models.ffnet_40s import Model
#from qai_hub_models.models.ffnet_54s import Model
#from qai_hub_models.models.ffnet_78s import Model# Low resolution variants
low_res_input_shape = (1, 3, 512, 1024)
#from qai_hub_models.models.ffnet_78s_lowres import Model
#from qai_hub_models.models.ffnet_122ns_lowres import Modelmodel = Model.from_pretrained()
stats = summary(model, input_size=input_shape, # use low_res_input_shape for low_res modelscol_names=["num_params", "mult_adds"]
)
print(stats)

Output

Loading pretrained model state dict from /home/jovyan/.qaihm/models/ffnet/v1/ffnet40S/ffnet40S_dBBB_cityscapes_state_dict_quarts.pth
Initializing ffnnet40S_dBBB_mobile weights
==============================================================================================================
Layer (type:depth-idx)                                       Param #                   Mult-Adds
==============================================================================================================
FFNet40S                                                     --                        --
├─FFNet: 1-1                                                 --                        --
│    └─ResNetS: 2-1                                          --                        --
│    │    └─Conv2d: 3-1                                      864                       452,984,832
│    │    └─BatchNorm2d: 3-2                                 64                        64
│    │    └─ReLU: 3-3                                        --                        --
│    │    └─Conv2d: 3-4                                      18,432                    2,415,919,104
│    │    └─BatchNorm2d: 3-5                                 128                       128
│    │    └─ReLU: 3-6                                        --                        --
│    │    └─Sequential: 3-7                                  300,160                   9,797,895,296
│    │    └─Sequential: 3-8                                  1,411,840                 11,542,727,424
│    │    └─Sequential: 3-9                                  3,900,288                 7,977,571,200
│    │    └─Sequential: 3-10                                 7,071,360                 3,617,592,960
│    └─FFNetUpHead: 2-2                                      --                        --
│    │    └─Sequential: 3-11                                 1,208,147                 26,571,541,312
==============================================================================================================
Total params: 13,911,283
Trainable params: 13,911,283
Non-trainable params: 0
Total mult-adds (G): 62.38
==============================================================================================================
Input size (MB): 25.17
Forward/backward pass size (MB): 1269.30
Params size (MB): 55.65
Estimated Total Size (MB): 1350.11
==============================================================================================================

Setup AI Hub for device-in-the-loop deployment

import qai_hubfrom utils import get_ai_hub_api_token
ai_hub_api_token = get_ai_hub_api_token()!qai-hub configure --api_token $ai_hub_api_token

Output

qai-hub configuration saved to /home/jovyan/.qai_hub/client.ini
==================== /home/jovyan/.qai_hub/client.ini ====================
[api]
api_token = eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJhcHAiLCJzdWIiOiIxNzQ2MDYyIiwiYXVkIjoiV0VCIiwiaWF0IjoxNzE2MzU2MDYxLCJleHAiOjE3MTg5NDgwNjF9.b2yWxfQnX8bVMrncob3vCQX5-g4kduq84m5DlvYoU78
api_url = https://app.aihub.qualcomm.com
web_url = https://app.aihub.qualcomm.com
verbose = True
%run -m qai_hub_models.models.ffnet_40s.demo

Output

在这里插入图片描述

Run on a real smart phone!

%run -m qai_hub_models.models.ffnet_40s.export -- --device "Samsung Galaxy S23"

Output

    ✅ SUCCESS                          ------------------------------------------------------------
Performance results on-device for Ffnet_40S.
------------------------------------------------------------
Device                          : Samsung Galaxy S23 (13)
Runtime                         : TFLITE                 
Estimated inference time (ms)   : 23.3                   
Estimated peak memory usage (MB): [3, 5]                 
Total # Ops                     : 92                     
Compute Unit(s)                 : NPU (92 ops)           
------------------------------------------------------------
More details: https://app.aihub.qualcomm.com/jobs/jn5q228m5/Waiting for inference job (j1glkknlp) completion. Type Ctrl+C to stop waiting at any time.✅ SUCCESS                          
dataset-dd9pg5on9.h5: 100%|██████████| 1.22M/1.22M [00:00<00:00, 11.5MB/s]Comparing on-device vs. local-cpu inference for Ffnet_40S.
+---------------+-------------------+--------+
| output_name   | shape             |   psnr |
+===============+===================+========+
| output_0      | (1, 19, 128, 256) |  62.96 |
+---------------+-------------------+--------+- psnr: Peak Signal-to-Noise Ratio (PSNR). >30 dB is typically considered good.

On Device Demo

%run -m qai_hub_models.models.ffnet_40s.demo -- --device "Samsung Galaxy S23" --on-device

Output

在这里插入图片描述

L3: Preparing for on-device deployment

On-device deployment key concepts

在这里插入图片描述

graph capture

在这里插入图片描述

Capture trained model

from qai_hub_models.models.ffnet_40s import Model as FFNet_40s# Load from pre-trained weights
ffnet_40s = FFNet_40s.from_pretrained()import torch
input_shape = (1, 3, 1024, 2048)
example_inputs = torch.rand(input_shape)traced_model = torch.jit.trace(ffnet_40s, example_inputs)traced_model

Compile for device

在这里插入图片描述

import qai_hub
import qai_hub_modelsfrom utils import get_ai_hub_api_token
ai_hub_api_token = get_ai_hub_api_token()!qai-hub configure --api_token $ai_hub_api_token

Output

qai-hub configuration saved to /home/jovyan/.qai_hub/client.ini
==================== /home/jovyan/.qai_hub/client.ini ====================
[api]
api_token = eyJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJhcHAiLCJzdWIiOiIxNzQ2MDYyIiwiYXVkIjoiV0VCIiwiaWF0IjoxNzE2MzU2MDYxLCJleHAiOjE3MTg5NDgwNjF9.b2yWxfQnX8bVMrncob3vCQX5-g4kduq84m5DlvYoU78
api_url = https://app.aihub.qualcomm.com
web_url = https://app.aihub.qualcomm.com
verbose = True
for device in qai_hub.get_devices():print(device.name)

Output

Google Pixel 3
Google Pixel 3a
Google Pixel 3 XL
Google Pixel 4
Google Pixel 4
Google Pixel 4a
Google Pixel 5
Samsung Galaxy Tab S7
Samsung Galaxy Tab A8 (2021)
Samsung Galaxy Note 20 (Intl)
Samsung Galaxy S21
Samsung Galaxy S21+
Samsung Galaxy S21 Ultra
Xiaomi Redmi Note 10 5G
Google Pixel 3a XL
Google Pixel 4a
Google Pixel 5
Google Pixel 5a 5G
Google Pixel 6
Samsung Galaxy A53 5G
Samsung Galaxy A73 5G
RB3 Gen 2 (Proxy)
QCS6490 (Proxy)
RB5 (Proxy)
QCS8250 (Proxy)
QCS8550 (Proxy)
Samsung Galaxy S21
Samsung Galaxy S21 Ultra
Samsung Galaxy S22 Ultra 5G
Samsung Galaxy S22 5G
Samsung Galaxy S22+ 5G
Samsung Galaxy Tab S8
Xiaomi 12
Xiaomi 12 Pro
Google Pixel 6
Google Pixel 6a
Google Pixel 7
Google Pixel 7 Pro
Samsung Galaxy A14 5G
Samsung Galaxy S22 5G
QCS8450 (Proxy)
XR2 Gen 2 (Proxy)
Samsung Galaxy S23
Samsung Galaxy S23+
Samsung Galaxy S23 Ultra
Google Pixel 7
Google Pixel 8
Google Pixel 8 Pro
Samsung Galaxy S24
Samsung Galaxy S24 Ultra
Samsung Galaxy S24+
device = qai_hub.Device("Samsung Galaxy S23")# Compile for target device
compile_job = qai_hub.submit_compile_job(model=traced_model,                        # Traced PyTorch modelinput_specs={"image": input_shape},        # Input specificationdevice=device,                             # Device
)
# Download and save the target model for use on-device
target_model = compile_job.get_target_model()

Exercise: Try different runtimes

Target runtime

在这里插入图片描述

TensorFlow Lite

在这里插入图片描述

compile_options="--target_runtime tflite"                  # Uses TensorFlow Lite
compile_options="--target_runtime onnx"                    # Uses ONNX runtime
compile_options="--target_runtime qnn_lib_aarch64_android" # Runs with Qualcomm AI Enginecompile_job_expt = qai_hub.submit_compile_job(model=traced_model,                        # Traced PyTorch modelinput_specs={"image": input_shape},        # Input specificationdevice=device,                             # Deviceoptions=compile_options,
)

Expore more compiler options here.

On-Device Performance Profiling

from qai_hub_models.utils.printing import print_profile_metrics_from_job# Choose device
device = qai_hub.Device("Samsung Galaxy S23")# Runs a performance profile on-device
profile_job = qai_hub.submit_profile_job(model=target_model,                       # Compiled modeldevice=device,                            # Device
)# Print summary
profile_data = profile_job.download_profile()
print_profile_metrics_from_job(profile_job, profile_data)

Output

------------------------------------------------------------
Performance results on-device for Job_Jqp4Wxrlg_Optimized_Tflite.
------------------------------------------------------------
Device                          : Samsung Galaxy S23 (13)
Runtime                         : TFLITE                 
Estimated inference time (ms)   : 30.1                   
Estimated peak memory usage (MB): [0, 2]                 
Total # Ops                     : 94                     
Compute Unit(s)                 : NPU (94 ops)           
------------------------------------------------------------

Exercise: Try different compute units

profile_options="--compute_unit cpu"     # Use cpu 
profile_options="--compute_unit gpu"     # Use gpu (with cpu fallback) 
profile_options="--compute_unit npu"     # Use npu (with cpu fallback) # Runs a performance profile on-device
profile_job_expt = qai_hub.submit_profile_job(model=target_model,                     # Compiled modeldevice=device,                          # Deviceoptions=profile_options,
)

On-Device Inference

torch_inputs = torch.Tensor(sample_inputs['image'][0])
torch_outputs = ffnet_40s(torch_inputs)
torch_outputs
inference_job = qai_hub.submit_inference_job(model=target_model,          # Compiled modelinputs=sample_inputs,        # Sample inputdevice=device,               # Device
)
ondevice_outputs = inference_job.download_output_data()
ondevice_outputs['output_0']

对比on-device和local-cpu推理

from qai_hub_models.utils.printing import print_inference_metrics
print_inference_metrics(inference_job, ondevice_outputs, torch_outputs)

Output

Comparing on-device vs. local-cpu inference for Job_Jqp4Wxrlg_Optimized_Tflite.
+---------------+----------------------------+--------+
| output_name   | shape                      |   psnr |
+===============+============================+========+
| output_0      | torch.Size([19, 128, 256]) |  62.96 |
+---------------+----------------------------+--------+- psnr: Peak Signal-to-Noise Ratio (PSNR). >30 dB is typically considered good.

Get ready for deployment!

target_model = compile_job.get_target_model()
_ = target_model.download("FFNet_40s.tflite")

review

在这里插入图片描述

L4: Quantizing Models

Why quantize

在这里插入图片描述

What is quantization

在这里插入图片描述

Scale and zero point

在这里插入图片描述

Types of quantization

weight quantization and activation quantization

在这里插入图片描述

Post Training Quantization (PTQ)

Quantization Aware Training (QAT)

在这里插入图片描述

4 steps in code

在这里插入图片描述

from datasets import load_dataset# Use input resolution of the network
input_shape = (1, 3, 1024, 2048)# Load 100 RGB images of urban scenes 
dataset = load_dataset("UrbanSyn/UrbanSyn", split="train", data_files="rgb/*_00*.png")
dataset = dataset.train_test_split(1)# Hold out for testing
calibration_dataset = dataset["train"]
test_dataset = dataset["test"]calibration_dataset["image"][0]

Output

在这里插入图片描述

Setup calibration/inference pipleline

import torch
from torchvision import transforms# Convert the PIL image above to Torch Tensor
preprocess = transforms.ToTensor()# Get a sample image in the test dataset
test_sample_pil = test_dataset[0]["image"]
test_sample = preprocess(test_sample_pil).unsqueeze(0) 
print(test_sample)

Output

tensor([[[[0.0941, 0.1020, 0.2941,  ..., 0.5176, 0.4784, 0.4510],[0.0980, 0.1059, 0.2000,  ..., 0.5137, 0.4902, 0.4745],[0.1098, 0.1294, 0.2275,  ..., 0.4980, 0.4863, 0.4980],...,[0.4784, 0.5020, 0.5098,  ..., 0.5882, 0.5686, 0.5608],[0.4941, 0.5098, 0.5294,  ..., 0.5020, 0.5098, 0.4824],[0.4980, 0.5137, 0.5333,  ..., 0.4588, 0.4353, 0.4157]],[[0.1098, 0.1020, 0.2431,  ..., 0.5176, 0.4784, 0.4549],[0.1137, 0.1294, 0.1922,  ..., 0.5098, 0.4902, 0.4745],[0.1294, 0.1608, 0.2078,  ..., 0.4980, 0.4863, 0.4980],...,[0.5059, 0.5176, 0.5255,  ..., 0.5647, 0.5333, 0.5294],[0.5137, 0.5333, 0.5451,  ..., 0.4745, 0.4745, 0.4431],[0.5176, 0.5373, 0.5569,  ..., 0.4275, 0.3922, 0.3804]],[[0.0824, 0.0784, 0.2353,  ..., 0.5294, 0.4824, 0.4510],[0.0824, 0.0784, 0.1647,  ..., 0.5216, 0.4980, 0.4863],[0.0667, 0.0902, 0.1843,  ..., 0.5059, 0.4941, 0.5176],...,[0.5412, 0.5412, 0.5490,  ..., 0.5843, 0.5451, 0.5412],[0.5529, 0.5725, 0.5765,  ..., 0.4902, 0.4902, 0.4627],[0.5490, 0.5804, 0.6039,  ..., 0.4353, 0.4039, 0.3882]]]])
import torch.nn.functional as F
import numpy as np
from PIL import Imagedef postprocess(output_tensor, input_image_pil):# Upsample the output to the original sizeoutput_tensor_upsampled = F.interpolate(output_tensor, input_shape[2:], mode="bilinear",)# Get top predicted class and convert to numpyoutput_predictions = output_tensor_upsampled[0].argmax(0).byte().detach().numpy().astype(np.uint8)# Overlay over original imagecolor_mask = Image.fromarray(output_predictions).convert("P")# Create an appropriate palette for the Cityscapes classespalette = [128, 64, 128, 244, 35, 232, 70, 70, 70, 102, 102, 156,190, 153, 153, 153, 153, 153, 250, 170, 30, 220, 220, 0,107, 142, 35, 152, 251, 152, 70, 130, 180, 220, 20, 60,255, 0, 0, 0, 0, 142, 0, 0, 70, 0, 60, 100, 0, 80, 100,0, 0, 230, 119, 11, 32]palette = palette + (256 * 3 - len(palette)) * [0]color_mask.putpalette(palette)out = Image.blend(input_image_pil, color_mask.convert("RGB"), 0.5)return out

Setup model in floating point

from qai_hub_models.models.ffnet_40s.model import FFNet40S
model = FFNet40S.from_pretrained().model.eval()# Run sample output through the model
test_output_fp32 = model(test_sample)
test_output_fp32postprocess(test_output_fp32, test_sample_pil)

Output

在这里插入图片描述

Prepare Quantized Model

from qai_hub_models.models._shared.ffnet_quantized.model import FFNET_AIMET_CONFIG
from aimet_torch.batch_norm_fold import fold_all_batch_norms
from aimet_torch.model_preparer import prepare_model
from aimet_torch.quantsim import QuantizationSimModel# Prepare model for 8-bit quantization
fold_all_batch_norms(model, [input_shape])
model = prepare_model(model)# Setup quantization simulator
quant_sim = QuantizationSimModel(model,quant_scheme="tf_enhanced",default_param_bw=8,              # Use bitwidth 8-bitdefault_output_bw=8,config_file=FFNET_AIMET_CONFIG,dummy_input=torch.rand(input_shape),
)

Perform post training quantization

size = 5  # Must be < 100def pass_calibration_data(sim_model: torch.nn.Module, args):(dataset,) = argswith torch.no_grad():for sample in dataset.select(range(size)):pil_image = sample["image"]input_batch = preprocess(pil_image).unsqueeze(0)# Feed sample through for calibrationsim_model(input_batch)# Run Post-Training Quantization (PTQ)
quant_sim.compute_encodings(pass_calibration_data, [calibration_dataset])
test_output_int8 = quant_sim.model(test_sample)
postprocess(test_output_int8, test_sample_pil)

Output

在这里插入图片描述

Run Quantized model on-device

import qai_hub
import qai_hub_modelsfrom utils import get_ai_hub_api_token
ai_hub_api_token = get_ai_hub_api_token()!qai-hub configure --api_token $ai_hub_api_token
%run -m qai_hub_models.models.ffnet_40s_quantized.export -- --device "Samsung Galaxy S23"

Output

------------------------------------------------------------
Performance results on-device for Ffnet_40S_Quantized.
------------------------------------------------------------
Device                          : Samsung Galaxy S23 (13)
Runtime                         : TFLITE                 
Estimated inference time (ms)   : 6.4                    
Estimated peak memory usage (MB): [1, 9]                 
Total # Ops                     : 97                     
Compute Unit(s)                 : NPU (97 ops)           
------------------------------------------------------------
More details: https://app.aihub.qualcomm.com/jobs/jvgdvreeg/
dataset-dq9k16r52.h5: 100%|██████████| 770k/770k [00:00<00:00, 7.23MB/s]
Comparing on-device vs. local-cpu inference for Ffnet_40S_Quantized.
+---------------+-------------------+--------+
| output_name   | shape             |   psnr |
+===============+===================+========+
| output_0      | (1, 19, 128, 256) |  33.93 |
+---------------+-------------------+--------+- psnr: Peak Signal-to-Noise Ratio (PSNR). >30 dB is typically considered good.

Impact of quantization

在这里插入图片描述

问题: Post-Training Quantization (PTQ)为什么需要calibration_dataset,calibration是什么?

Post-Training Quantization (PTQ) 需要 calibration dataset(校准数据集)的原因是为了确定模型各层的量化参数,特别是 scale(比例)和 zero-point(零点)。Calibration 是指使用校准数据集来统计和计算模型的激活值和权重的分布,从而确定最佳的量化参数,以在量化过程中尽量减少精度损失。

Calibration 的重要性

  • 确定量化范围:浮点数转换为整数表示时,需要确定量化范围。例如,对于8位量化,激活值和权重需要被映射到0到255的范围。Calibration 数据集帮助确定每一层的浮点值范围,以便进行准确的映射。
  • 减少量化误差:通过统计校准数据集上的浮点数值分布,可以更好地选择量化参数,从而减少量化误差,提升量化后模型的精度。

Calibration 过程

  1. 收集统计信息:将校准数据集输入模型,收集每一层的激活值和权重的统计信息。这些统计信息包括最小值、最大值、直方图分布等。
  2. 计算量化参数:根据收集到的统计信息,计算量化所需的 scale 和 zero-point。例如,如果某层的激活值范围是 [a, b],则可以计算出适合该范围的 scale 和 zero-point。
  3. 应用量化参数:将计算得到的量化参数应用到模型中,将浮点数值转换为整数表示。

为什么需要 Calibration Dataset

  • 真实数据分布:校准数据集应尽可能反映模型在实际应用中会遇到的数据分布。这样确定的量化参数更具有代表性,量化后的模型在实际应用中的性能也会更好。
  • 高效和准确:使用校准数据集进行统计,能够在不需要重新训练模型的情况下,快速、准确地确定量化参数,实现高效的量化过程。

Calibration 示例

假设我们有一个简单的神经网络模型,我们需要对其进行8位量化。以下是使用校准数据集进行 calibration 的步骤:

import numpy as np# 模拟校准数据集
calibration_dataset = [np.random.randn(100, 224, 224, 3) for _ in range(10)]# 统计信息收集函数(示例)
def collect_statistics(model, dataset):min_values = []max_values = []for data in dataset:activations = model(data)min_values.append(np.min(activations, axis=0))max_values.append(np.max(activations, axis=0))global_min = np.min(min_values, axis=0)global_max = np.max(max_values, axis=0)return global_min, global_max# 计算量化参数
def calculate_quantization_params(global_min, global_max):scale = (global_max - global_min) / 255.0zero_point = -global_min / scalereturn scale, zero_point# 示例模型(假设已有训练好的模型)
class SimpleModel:def __call__(self, x):return x  # 简单传递输入作为输出(示例)# 模型实例
model = SimpleModel()# 收集统计信息
global_min, global_max = collect_statistics(model, calibration_dataset)# 计算量化参数
scale, zero_point = calculate_quantization_params(global_min, global_max)print(f"Scale: {scale}, Zero Point: {zero_point}")

结论

Post-Training Quantization (PTQ) 的校准(Calibration)过程至关重要,因为它通过校准数据集来统计和计算量化参数,确保在量化过程中尽量减少精度损失。Calibration 数据集提供了模型在实际应用中可能遇到的数据信息,帮助确定准确的量化范围,从而提高量化模型的性能和准确性。

Device Integration

在这里插入图片描述

How is the application implemented

在这里插入图片描述

Runtime dependencies

在这里插入图片描述

Demo

在这里插入图片描述

Appendix - Building the App

Going into the details of the building the final mobile app was slightly outside the scope of this course. We have build this help guide for you to show the main steps and code samples you need to build the app we saw in the last lesson.

TensorFlow Lite

TensorFlow Lite is a runtime that enables on-device machine learning by helping developers run their models on mobile, embedded, and edge devices. The models produced by TensorFlow Lite work on multiple platform support, covering Android and iOS devices, embedded Linux, and microcontrollers. The toolchain also has diverse language support, which includes Java, Swift, Objective-C, C++, and Python.

🔗   TensorFlow Lite guide [ + ]

Delegation

In the context of TensorFlow Lite, “delegation” refers to the use of delegates to enable hardware acceleration of machine learning models on mobile and edge devices. Delegates act as a bridge between TensorFlow Lite and on-device accelerators like GPUs and DSPs, optimizing performance and efficiency by leveraging specialized hardware capabilities. This process can significantly improve the speed and power consumption of running machine learning models on such devices. For more details, you can visit the TensorFlow Lite Delegates page.

🔗   TensorFlow Lite Delegates page [ + ]

Qualcomm QNN delegate

Qualcomm QNN delegate allows you to run models on the NPU.

🔗   Download Qualcomm QNN Delegate – (Zip 724 MB)

End-to-end examples

You can find end-to-end examples, for common machine learning tasks such as image classification, object detection, pose estimation, question answering, text classification, etc. on multiple platforms. Here is the link to some models we have provided for you.

🔗   End-to-end examples – GitHub [ + ]

Models available:

  • ImageClassification
  • ImageSuperResolution
  • SemanticSegmentation

Semantic segmentation code

The code for the semantic segmentation app we developed in this course is available on Github for you to try.

Requirements:

  • Java, android-sdk and sdkmanager is already set at user’s end
  • User should have Linux QNN SDK in local machine.
  • ANDROID_HOME is set to android-sdk path
  • AI-Hub is properly configured with user token.

Note: Please execute build_apk.py. This script will compile and download a model from AI-Hub and paste it in your Android Project and Generate app-debug.apk.

🔗   Semantic segmentation code + Guide – GitHub [ + ]

Afterword

2024年5月22日于上海。

通过学习这门short course,了解了高通公司在on-device AI方面的努力,对AI Hub Models仓库有了一定程度的了解。

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.luyixian.cn/news_show_1053321.aspx

如若内容造成侵权/违法违规/事实不符,请联系dt猫网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

详解Spring MVC

目录 1.什么是Spring Web MVC MVC定义 2.学习Spring MVC 建立连接 RequestMapping 注解介绍及使用 获取单个参数 获取多个参数 获取普通对象 获取JSON对象 获取基础URL参数 获取上传文件 获取Header 获取Cookie 获取Session 总结 1.什么是Spring Web MVC 官⽅对于…

基于.NetCore和ABP.VNext的项目实战二:Swagger

Mag.Blog.Swagger层添加Volo.Abp.AspNetCore和Swashbuckle.AspNetCore包,引用实体层.Domain 添加模块类MagBlogSwaggerModule.cs,依赖MagBlogDomainModule模块,并且重写ConfigureServices和OnApplicationInitialization方法 namespace Mag.Blog.Swagger {[DependsOn(typeof…

【教学类-58-06】黑白三角拼图06(1页3张彩色黑点卡片,一种宫格36张,适合一个班级一次操作)

作品展示 背景需求 【教学类-58-05】黑白三角拼图05&#xff08;2-10宫格&#xff0c;每个宫格随机1张-6张&#xff0c;带空格纸&#xff0c;1页3张黑白3张白卡&#xff09;-CSDN博客文章浏览阅读343次&#xff0c;点赞10次&#xff0c;收藏6次。【教学类-58-05】黑白三角拼图…

vue contextPath的思考

先说我这边的情况&#xff0c;目前项目都是前后端分离开发的&#xff0c;上线有种部署方式&#xff0c;常见的就是前后端分开部署&#xff0c;这是比较常见的&#xff0c;我这边因客户原因&#xff0c;打包一起进行部署比较简单&#xff0c;交付技术运维部方便后期其他现场部署…

5 分钟快速上手图形验证码,防止接口被恶意刷量!

5 分钟快速上手图形验证码&#xff0c;防止接口被恶意刷量&#xff01; 大家好&#xff0c;我是程序员小白条&#xff0c;今天来给大家介绍一个快速实现图形验证码的优秀框架 AJ-Captcha。 需求分析 如果注册接口没有验证码这种类型的限制&#xff0c;很容易会被刷量&#x…

python练习题-反转一个只有三位数的整数

【问题描述】&#xff1a;反转一个只有三位数的整数 [示例]&#xff1a;123 321 完整代码如下&#xff1a; nint(input()) if n<100 or n>999: print("请输入三位数&#xff01;") else: gen%10 shin//10%10 bain//100 m100*ge10*shibai…

电脑同时配置两个版本mysql数据库常见问题

1.配置时&#xff0c;要把bin中的mysql.exe和mysqld.exe 改个名字&#xff0c;不然两个版本会重复&#xff0c;当然&#xff0c;在初始化数据库的时候&#xff0c;如果时57版本的&#xff0c;就用mysql57(已经改名的)和mysqld57 代替 mysql 和 mysqld 例如 mysql -u root -p …

VLDB ’25 最后 6 天截稿,58 个顶会信息纵览;ISPRS 城市分割数据集上线

「顶会」板块上线 hyper.ai 官网啦&#xff01;该板块为大家提供最新最全的 CCF A 类计算机顶会信息&#xff0c;包含会议简介、截稿倒计时、投稿链接等。 你是不是已经注册了顶会&#xff0c;但对截稿时间较为模糊&#xff0c;老是在临近 ddl 时才匆忙提交&#xff1b;又或者…

Linux基础知识点总结!超详细

Linux 的学习对于一个IT工程师的重要性是不言而喻的&#xff0c;学好它是工程师必备修养之一。 Linux 基础 操作系统 操作系统Operating System简称OS&#xff0c;是软件的一部分&#xff0c;它是硬件基础上的第一层软件&#xff0c;是硬件和其它软件沟通的桥梁。 操作系统…

2024-05-28 服务器开发-不同vs版本的std::string的访问出错问题-记录

摘要: 有一个dll库是使用vs2010编译的, 使用这个dll动态库的工程是vs2019. 这个dll动态库返回一个结构体&#xff0c;其中有个成员使用了std::string。但是遇到了std::string的成员显示被赋值为NULL的情况。 本文对进行分析, 重点在于追踪问题的思路。 问题描述: dll使用vs20…

基于Pytorch框架的深度学习RegNet神经网络二十五种宝石识别分类系统源码

第一步&#xff1a;准备数据 25种宝石数据&#xff0c;总共800张&#xff1a; { "0": "Alexandrite","1": "Almandine","2": "Benitoite","3": "Beryl Golden","4": "Carne…

架构师系列---RPC通信原理

RPC通信原理 基于网络的调用 问题&#xff1a;谁来解决这个跨进程调用的问题&#xff1f; RPC&#xff1a;Remote Percedure Call 远程过程调用 定义了一台主机上的程序通过网络调用另外一台主机上的程序的子程序这一行为。 RPC符合CS模型&#xff0c;可以实现进程间的通信&a…

超详细的前后端实战项目(Spring系列加上vue3)前端篇(二)(一步步实现+源码)

好了&#xff0c;兄弟们&#xff0c;继昨天的项目之后&#xff0c;开始继续敲前端代码&#xff0c;完成前端部分 昨天完成了全局页面的代码&#xff0c;和登录页面的代码&#xff0c;不过昨天的代码还有一些需要补充的&#xff0c;这里添加一下 内容补充&#xff1a;在调用登…

vxe-form-design 表单设计器的使用

vxe-form-design 在 vue3 中表单设计器的使用 查看官网 https://vxeui.com 安装 npm install vxe-pc-ui // ... import VxeUI from vxe-pc-ui import vxe-pc-ui/lib/style.css // ...// ... createApp(App).use(VxeUI).mount(#app) // ...使用 github vxe-form-design 用…

Vue学习笔记2——创建一个Vue项目

Vue项目 1、创建一个Vue项目2、Vue项目的目录结构3、模版语法4、属性绑定5、条件渲染 1、创建一个Vue项目 vue官方文档&#xff1a; https://cn.vuejs.org/打开命令行界面&#xff08; “winR"再输入"cmd”&#xff09;&#xff0c;切换位置到指定的位置创建vue项目…

一文详解SpringBoot的自定义starter

目录 一、SpringBoot 二、自定义starter 三、SpringBoot的自定义starter 一、SpringBoot Spring Boot是一个开源的Java框架&#xff0c;由Pivotal团队&#xff08;现为VMware的一部分&#xff09;于2013年推出&#xff0c;旨在简化Spring应用程序的创建和部署过程。它基于S…

民国漫画杂志《时代漫画》第28期.PDF

时代漫画28.PDF: https://url03.ctfile.com/f/1779803-1248635321-5c67ad?p9586 (访问密码: 9586) 《时代漫画》的杂志在1934年诞生了&#xff0c;截止1937年6月战争来临被迫停刊共发行了39期。 ps: 资源来源网络!

Linux一键安装Docker、kkfileviewer

Linux一键安装Docker、kkfileviewer 一、安装docker 安装docker脚本 vi initDocker.sh脚本内容 #安装前先更新yum&#xff0c;防止连接镜像失败 yum -y update#卸载系统之前的docker&#xff08;可选择&#xff0c;我这里直接注释了&#xff09; #yum remove docker docker…

蓝桥杯物联网竞赛_STM32L071_18_长短按键检测

长短按键的检测是国赛题里面遇到的&#xff0c;省赛没出过有两种实方法 定时器配置&#xff1a; 定时器的话要比delay准确&#xff0c;其中tim7定时器的准度最高 定时器预分配配置32 - 1&#xff0c;计数周期是10000 - 1这样做那么32MHZ/32也就是一秒钟记录10^6的数&#xf…

开源远程协助:分享屏幕,隔空协助!

&#x1f5a5;️ 星控远程协助系统 &#x1f5b1;️ 一个使用Java GUI技术实现的远程控制软件&#xff0c;你现在就可以远程查看和控制你的伙伴的桌面&#xff0c;接受星星的指引吧&#xff01; 支持系统&#xff1a;Windows / Mac / Linux &#x1f31f; 功能导览 &#x1f…