YOLO目标检测创新改进与实战案例专栏
专栏目录： YOLO有效改进系列及项目实战目录 包含卷积，主干 注意力，检测头等创新机制 以及 各种目标检测分割项目实战案例
专栏链接: YOLO基础解析+创新改进+实战案例
介绍
摘要
摘要：我们介绍了最新一代的MobileNets，称为MobileNetV4（MNv4），其架构设计在移动设备上具有通用的高效性。核心是我们引入了通用倒置瓶颈（UIB）搜索模块，这是一种统一且灵活的结构，融合了倒置瓶颈（IB）、ConvNext、前馈网络（FFN）以及一种新颖的额外深度可分离（ExtraDW）变体。除了UIB，我们还介绍了Mobile MQA，一种专为移动加速器设计的注意力模块，提供显著的39%速度提升。我们还引入了一种优化的神经架构搜索（NAS）配方，提高了MNv4的搜索效率。UIB、Mobile MQA和改进的NAS配方的整合，产生了一套新的MNv4模型，这些模型在移动CPU、DSP、GPU以及专用加速器（如Apple Neural Engine和Google Pixel EdgeTPU）上大多达到了帕累托最优，这是其他任何测试模型都不具备的特点。最后，为了进一步提高准确性，我们引入了一种新颖的蒸馏技术。在这种技术的增强下，我们的MNv4-Hybrid-Large模型在ImageNet-1K上达到了87%的准确率，在Pixel 8 EdgeTPU上的运行时间仅为3.8毫秒。
文章链接
论文地址：论文地址
代码地址：代码地址
代码地址：代码地址
基本原理
MobileNetV4是MobileNet系列的最新一代，旨在为移动设备提供高效的架构设计。MobileNetV4引入了Universal Inverted Bottleneck和Mobile MQA层，并结合改进的NAS（神经架构搜索）方法。通过这些创新设计和优化技术，MobileNetV4在Pixel 8 EdgeTPU上实现了87%的ImageNet-1K准确率，同时延迟仅为3.8ms，推动了移动计算机视觉的最新发展。
Universal Inverted Bottleneck（通用反向瓶颈）：
- UIB结构融合了Inverted Bottleneck（IB）、ConvNext、Feed Forward Network（FFN）和Extra Depthwise（ExtraDW）变体，提供了统一且灵活的特征提取结构。
- UIB通过引入两个可选的深度卷积层，改进了Inverted Bottleneck结构，实现了更高效的特征提取。
- UIB统一了多种微体系结构，如Inverse Bottleneck（IB）、ConvNext和FFN，提供了更好的性能和灵活性。
Mobile MQA层：
- Mobile MQA是一种专为移动加速器定制的注意力机制层，能够显著提升模型的速度。
- 通过Mobile MQA层，MobileNetV4在移动设备上实现了39%的速度提升，为实时和交互式体验提供了支持。
NAS方法：
- MobileNetV4采用了优化的神经架构搜索（NAS）方法，自动化模型设计过程，提高了搜索效率。
- 通过改进的NAS方法，MobileNetV4创建了一系列普遍优化的移动模型，适用于多种移动设备，包括移动CPU、DSP、GPU以及专用加速器。
Distillation Approach（蒸馏方法）：
- MobileNetV4引入了一种新颖的蒸馏技术，通过模型蒸馏，提高了模型的准确性。
- 借助这种蒸馏方法，MobileNetV4-Hybrid-Large模型在Pixel 8 EdgeTPU上实现了87%的ImageNet-1K准确率，同时仅需3.8ms的运行时间。
核心代码
@tf_keras.utils.register_keras_serializable(package='Vision')
class MobileNet(tf_keras.Model):
"""Creates a MobileNet family model."""
def __init__(
self,
model_id: str = 'MobileNetV2',
filter_size_scale: float = 1.0,
input_specs: tf_keras.layers.InputSpec = layers.InputSpec(
shape=[None, None, None, 3]
),
# The followings are for hyper-parameter tuning.
norm_momentum: float = 0.99,
norm_epsilon: float = 0.001,
kernel_initializer: str = 'VarianceScaling',
kernel_regularizer: tf_keras.regularizers.Regularizer | None = None,
bias_regularizer: tf_keras.regularizers.Regularizer | None = None,
# The followings should be kept the same most of the times.
output_stride: int | None = None,
min_depth: int = 8,
# divisible is not used in MobileNetV1.
divisible_by: int = 8,
stochastic_depth_drop_rate: float = 0.0,
flat_stochastic_depth_drop_rate: bool = True,
regularize_depthwise: bool = False,
use_sync_bn: bool = False,
# finegrain is not used in MobileNetV1.
finegrain_classification_mode: bool = True,
output_intermediate_endpoints: bool = False,
**kwargs,
):
"""Initializes a MobileNet model.
Args:
model_id: A `str` of MobileNet version. The supported values are
`MobileNetV1`, `MobileNetV2`, `MobileNetV3Large`, `MobileNetV3Small`,
`MobileNetV3EdgeTPU`, `MobileNetMultiMAX` and `MobileNetMultiAVG`.
filter_size_scale: A `float` of multiplier for the filters (number of
channels) for all convolution ops. The value must be greater than zero.
Typical usage will be to set this value in (0, 1) to reduce the number
of parameters or computation cost of the model.
input_specs: A `tf_keras.layers.InputSpec` of specs of the input tensor.
norm_momentum: A `float` of normalization momentum for the moving average.
norm_epsilon: A `float` added to variance to avoid dividing by zero.
kernel_initializer: A `str` for kernel initializer of convolutional
layers.
kernel_regularizer: A `tf_keras.regularizers.Regularizer` object for
Conv2D. Default to None.
bias_regularizer: A `tf_keras.regularizers.Regularizer` object for Conv2D.
Default to None.
output_stride: An `int` that specifies the requested ratio of input to
output spatial resolution. If not None, then we invoke atrous
convolution if necessary to prevent the network from reducing the
spatial resolution of activation maps. Allowed values are 8 (accurate
fully convolutional mode), 16 (fast fully convolutional mode), 32
(classification mode).
min_depth: An `int` of minimum depth (number of channels) for all
convolution ops. Enforced when filter_size_scale < 1, and not an active
constraint when filter_size_scale >= 1.
divisible_by: An `int` that ensures all inner dimensions are divisible by
this number.
stochastic_depth_drop_rate: A `float` of drop rate for drop connect layer.
flat_stochastic_depth_drop_rate: A `bool`, indicating that the stochastic
depth drop rate will be fixed and equal to all blocks.
regularize_depthwise: If Ture, apply regularization on depthwise.
use_sync_bn: If True, use synchronized batch normalization.
finegrain_classification_mode: If True, the model will keep the last layer
large even for small multipliers, following
https://arxiv.org/abs/1801.04381.
output_intermediate_endpoints: A `bool` of whether or not output the
intermediate endpoints.
**kwargs: Additional keyword arguments to be passed.
"""
if model_id not in SUPPORTED_SPECS_MAP:
raise ValueError('The MobileNet version {} '
'is not supported'.format(model_id))
if filter_size_scale <= 0:
raise ValueError('filter_size_scale is not greater than zero.')
if output_stride is not None:
if model_id == 'MobileNetV1':
if output_stride not in [8, 16, 32]:
raise ValueError('Only allowed output_stride values are 8, 16, 32.')
else:
if output_stride == 0 or (output_stride > 1 and output_stride % 2):
raise ValueError('Output stride must be None, 1 or a multiple of 2.')
self._model_id = model_id
self._input_specs = input_specs
self._filter_size_scale = filter_size_scale
self._min_depth = min_depth
self._output_stride = output_stride
self._divisible_by = divisible_by
self._stochastic_depth_drop_rate = stochastic_depth_drop_rate
self._flat_stochastic_depth_drop_rate = flat_stochastic_depth_drop_rate
self._regularize_depthwise = regularize_depthwise
self._kernel_initializer = kernel_initializer
self._kernel_regularizer = kernel_regularizer
self._bias_regularizer = bias_regularizer
self._use_sync_bn = use_sync_bn
self._norm_momentum = norm_momentum
self._norm_epsilon = norm_epsilon
self._finegrain_classification_mode = finegrain_classification_mode
self._output_intermediate_endpoints = output_intermediate_endpoints
inputs = tf_keras.Input(shape=input_specs.shape[1:])
block_specs = SUPPORTED_SPECS_MAP.get(model_id)
self._decoded_specs = block_spec_decoder(
specs=block_specs,
filter_size_scale=self._filter_size_scale,
divisible_by=self._get_divisible_by(),
finegrain_classification_mode=self._finegrain_classification_mode,
)
x, endpoints, next_endpoint_level = self._mobilenet_base(inputs=inputs)
self._output_specs = {
l: endpoints[l].get_shape() for l in endpoints}
# Don't include the final layer in `self._output_specs` to support decoders.
endpoints[str(next_endpoint_level)] = x
super(MobileNet, self).__init__(
inputs=inputs, outputs=endpoints, **kwargs)
def _get_divisible_by(self):
if self._model_id == 'MobileNetV1':
return 1
else:
return self._divisible_by
def _mobilenet_base(
self, inputs: tf.Tensor
) -> tuple[tf.Tensor, dict[str, tf.Tensor], int]:
"""Builds the base MobileNet architecture.
Args:
inputs: A `tf.Tensor` of shape `[batch_size, height, width, channels]`.
Returns:
A tuple of output Tensor and dictionary that collects endpoints.
"""
input_shape = inputs.get_shape().as_list()
if len(input_shape) != 4:
raise ValueError('Expected rank 4 input, was: %d' % len(input_shape))
# The current_stride variable keeps track of the output stride of the
# activations, i.e., the running product of convolution strides up to the
# current network layer. This allows us to invoke atrous convolution
# whenever applying the next convolution would result in the activations
# having output stride larger than the target output_stride.
current_stride = 1
# The atrous convolution rate parameter.
rate = 1
# Used to calulate stochastic depth drop rate. Some blocks do not use
# stochastic depth since they do not have residuals. For simplicity, we
# count here all the blocks in the model. If one or more of the last layers
# do not use stochastic depth, it can be compensated with larger stochastic
# depth drop rate.
num_blocks = len(self._decoded_specs)
net = inputs
endpoints = {
}
endpoint_level = 2
for block_idx, block_def in enumerate(self._decoded_specs):
block_name = 'block_group_{}_{}'.format(block_def.block_fn, block_idx)
# A small catch for gpooling block with None strides
if not block_def.strides:
block_def.strides = 1
if (self._output_stride is not None and
current_stride == self._output_stride):
# If we have reached the target output_stride, then we need to employ
# atrous convolution with stride=1 and multiply the atrous rate by the
# current unit's stride for use in subsequent layers.
layer_stride = 1
layer_rate = rate
rate *= block_def.strides
else:
layer_stride = block_def.strides
layer_rate = 1
current_stride *= block_def.strides
if self._flat_stochastic_depth_drop_rate:
stochastic_depth_drop_rate = self._stochastic_depth_drop_rate
else:
stochastic_depth_drop_rate = nn_layers.get_stochastic_depth_rate(
self._stochastic_depth_drop_rate, block_idx + 1, num_blocks
)
if stochastic_depth_drop_rate is not None:
logging.info(
'stochastic_depth_drop_rate: %f for block = %d',
stochastic_depth_drop_rate,
block_idx,
)
intermediate_endpoints = {
}
if block_def.block_fn == 'convbn':
net = Conv2DBNBlock(
filters=block_def.filters,
kernel_size=block_def.kernel_size,
strides=block_def.strides,
activation=block_def.activation,
use_bias=block_def.use_bias,
use_normalization=block_def.use_normalization,
kernel_initializer=self._kernel_initializer,
kernel_regularizer=self._kernel_regularizer,
bias_regularizer=self._bias_regularizer,
use_sync_bn=self._use_sync_bn,
norm_momentum=self._norm_momentum,
norm_epsilon=self._norm_epsilon
)(net)
elif block_def.block_fn == 'depsepconv':
net = nn_blocks.DepthwiseSeparableConvBlock(
filters=block_def.filters,
kernel_size=block_def.kernel_size,
strides=layer_stride,
activation=block_def.activation,
dilation_rate=layer_rate,
regularize_depthwise=self._regularize_depthwise,
kernel_initializer=self._kernel_initializer,
kernel_regularizer=self._kernel_regularizer,
use_sync_bn=self._use_sync_bn,
norm_momentum=self._norm_momentum,
norm_epsilon=self._norm_epsilon,
)(net)
elif block_def.block_fn == 'mhsa':
block = nn_blocks.MultiHeadSelfAttentionBlock(
input_dim=block_def.filters,
num_heads=block_def.num_heads,
key_dim=block_def.key_dim,
value_dim=block_def.value_dim,
use_multi_query=block_def.use_multi_query,
query_h_strides=block_def.query_h_strides,
query_w_strides=block_def.query_w_strides,
kv_strides=block_def.kv_strides,
downsampling_dw_kernel_size=block_def.downsampling_dw_kernel_size,
cpe_dw_kernel_size=block_def.kernel_size,
stochastic_depth_drop_rate=self._stochastic_depth_drop_rate,
use_sync_bn=self._use_sync_bn,
use_residual=block_def.use_residual,
norm_momentum=self._norm_momentum,
norm_epsilon=self._norm_epsilon,
use_layer_scale=block_def.use_layer_scale,
output_intermediate_endpoints=self._output_intermediate_endpoints,
)
if self._output_intermediate_endpoints:
net, intermediate_endpoints = block(net)
else:
net = block(net)
elif block_def.block_fn in (
'invertedbottleneck',
'fused_ib',
'uib',
):
use_rate = rate
if layer_rate > 1 and block_def.kernel_size != 1:
# We will apply atrous rate in the following cases:
# 1) When kernel_size is not in params, the operation then uses
# default kernel size 3x3.
# 2) When kernel_size is in params, and if the kernel_size is not
# equal to (1, 1) (there is no need to apply atrous convolution to
# any 1x1 convolution).
use_rate = layer_rate
in_filters = net.shape.as_list()[-1]
args = {
'in_filters': in_filters,
'out_filters': block_def.filters,
'strides': layer_stride,
'expand_ratio': block_def.expand_ratio,
'activation': block_def.activation,
'use_residual': block_def.use_residual,
'dilation_rate': use_rate,
'regularize_depthwise': self._regularize_depthwise,
'kernel_initializer': self._kernel_initializer,
'kernel_regularizer': self._kernel_regularizer,
'bias_regularizer': self._bias_regularizer,
'use_sync_bn': self._use_sync_bn,
'norm_momentum': self._norm_momentum,
'norm_epsilon': self._norm_epsilon,
'stochastic_depth_drop_rate': stochastic_depth_drop_rate,
'divisible_by': self._get_divisible_by(),
'output_intermediate_endpoints': (
self._output_intermediate_endpoints
),
}
if block_def.block_fn in ('invertedbottleneck', 'fused_ib'):
args.update({
'kernel_size': block_def.kernel_size,
'se_ratio': block_def.se_ratio,
'expand_se_in_filters': True,
'use_depthwise': (
block_def.use_depthwise
if block_def.block_fn == 'invertedbottleneck'
else False
),
'se_gating_activation': 'hard_sigmoid',
})
block = nn_blocks.InvertedBottleneckBlock(**args)
else:
args.update({
'middle_dw_downsample': block_def.middle_dw_downsample,
'start_dw_kernel_size': block_def.start_dw_kernel_size,
'middle_dw_kernel_size': block_def.middle_dw_kernel_size,
'end_dw_kernel_size': block_def.end_dw_kernel_size,
'use_layer_scale': block_def.use_layer_scale,
})
block = nn_blocks.UniversalInvertedBottleneckBlock(**args)
if self._output_intermediate_endpoints:
net, intermediate_endpoints = block(net)
else:
net = block(net)
elif block_def.block_fn == 'gpooling':
net = layers.GlobalAveragePooling2D(keepdims=True)(net)
else:
raise ValueError(
'Unknown block type {} for layer {}'.format(
block_def.block_fn, block_idx
)
)
net = tf_keras.layers.Activation('linear', name=block_name)(net)
if block_def.is_output:
endpoints[str(endpoint_level)] = net
for key, tensor in intermediate_endpoints.items():
endpoints[str(endpoint_level) + '/' + key] = tensor
if current_stride != self._output_stride:
endpoint_level += 1
if str(endpoint_level) in endpoints:
endpoint_level += 1
return net, endpoints, endpoint_level
def get_config(self):
config_dict = {
'model_id': self._model_id,
'filter_size_scale': self._filter_size_scale,
'min_depth': self._min_depth,
'output_stride': self._output_stride,
'divisible_by': self._divisible_by,
'stochastic_depth_drop_rate': self._stochastic_depth_drop_rate,
'flat_stochastic_depth_drop_rate': (
self._flat_stochastic_depth_drop_rate
),
'regularize_depthwise': self._regularize_depthwise,
'kernel_initializer': self._kernel_initializer,
'kernel_regularizer': self._kernel_regularizer,
'bias_regularizer': self._bias_regularizer,
'use_sync_bn': self._use_sync_bn,
'norm_momentum': self._norm_momentum,
'norm_epsilon': self._norm_epsilon,
'finegrain_classification_mode': self._finegrain_classification_mode,
}
return config_dict
@classmethod
def from_config(cls, config, custom_objects=None):
return cls(**config)
@property
def output_specs(self):
"""A dict of {level: TensorShape} pairs for the model output."""
return self._output_specs
@factory.register_backbone_builder('mobilenet')
def build_mobilenet(
input_specs: tf_keras.layers.InputSpec,
backbone_config: hyperparams.Config,
norm_activation_config: hyperparams.Config,
l2_regularizer: tf_keras.regularizers.Regularizer | None = None,
) -> tf_keras.Model:
"""Builds MobileNet backbone from a config."""
backbone_type = backbone_config.type
backbone_cfg = backbone_config.get()
assert backbone_type == 'mobilenet', (f'Inconsistent backbone type '
f'{backbone_type}')
return MobileNet(
model_id=backbone_cfg.model_id,
filter_size_scale=backbone_cfg.filter_size_scale,
input_specs=input_specs,
stochastic_depth_drop_rate=backbone_cfg.stochastic_depth_drop_rate,
flat_stochastic_depth_drop_rate=(
backbone_cfg.flat_stochastic_depth_drop_rate
),
output_stride=backbone_cfg.output_stride,
output_intermediate_endpoints=backbone_cfg.output_intermediate_endpoints,
use_sync_bn=norm_activation_config.use_sync_bn,
norm_momentum=norm_activation_config.norm_momentum,
norm_epsilon=norm_activation_config.norm_epsilon,
kernel_regularizer=l2_regularizer,
)
task与yaml配置
详见： https://blog.csdn.net/shangyanaf/article/details/140364353
