ML之catboost:catboost模型中常用的Pool类型数据结构源代码解读、案例应用之详细攻略(一)-阿里云开发者社区

开发者社区> 一个处女座的程序猿> 正文

ML之catboost:catboost模型中常用的Pool类型数据结构源代码解读、案例应用之详细攻略(一)

简介: ML之catboost:catboost模型中常用的Pool类型数据结构源代码解读、案例应用之详细攻略
+关注继续查看

Pool简介


Pool 是在CatBoost中用作训练模型的数据结构。




Pool的案例应用


train_dataPool = Pool(data=[[12, 14, 16, 18], [23, 25, 27, 29], [32, 34, 36, 38]],

                 label=[10, 20, 30],

                 weight=[0.1, 0.2, 0.3])    # weight各自权重默认设置为1




Pool源代码解读


class Pool Found at: catboost.core

class Pool(_PoolBase):

   """

   Pool used in CatBoost as a data structure to train model from.

   """

   def __init__(

       self,

       data,

       label=None,

       cat_features=None,

       text_features=None,

       embedding_features=None,

       column_description=None,

       pairs=None,

       delimiter='\t',

       has_header=False,

       ignore_csv_quoting=False,

       weight=None,

       group_id=None,

       group_weight=None,

       subgroup_id=None,

       pairs_weight=None,

       baseline=None,

       feature_names=None,

       thread_count=-1):

       """

       Pool is an internal data structure that is used by CatBoost.

       You can construct Pool from list, numpy.ndarray, pandas.

        DataFrame, pandas.Series.

       Parameters

       ----------

Pool 是在CatBoost中用作训练模型的数据结构。

       data : list or numpy.ndarray or pandas.DataFrame or  pandas.Series or FeaturesData or string. Data source of Pool. If list or numpy.ndarrays or pandas.DataFrame or pandas.  Series, giving 2 dimensional array like data.  If FeaturesData - see FeaturesData description for  details, 'cat_features' and 'feature_names' parameters must be equal to None in this case. If string, giving the path to the file with data in catboost  format. If path starts with "quantized://", the file has to contain  quantized dataset saved with Pool.save().

       label : list or numpy.ndarrays or pandas.DataFrame or  pandas.Series, optional (default=None). Label of the training data. If not None, giving 1 or 2 dimensional array like data   with floats. If data is a file, then label must be in the file, that is label must be equals to None  cat_features : list or numpy.ndarray, optional  (default=None). If not None, giving the list of Categ features indices or names. If it contains feature names, Pool's feature names must be defined: either by passing 'feature_names' parameter or if data is pandas.DataFrame (feature  names are initialized from it's column names). Must be None if 'data' parameter has FeaturesData type

       text_features : list or numpy.ndarray, optional . (default=None), If not None, giving the list of Text features indices or  names. If it contains feature names, Pool's feature names must

            be defined: either by passing 'feature_names'  parameter or if data is pandas.DataFrame (feature names are initialized from it's column names).  Must be None if 'data' parameter has FeaturesData type

       embedding_features : list or numpy.ndarray, optional . (default=None). If not None, giving the list of Embedding features  indices or names. If it contains feature names, Pool's feature names must  be defined: either by passing 'feature_names' parameter or if data is pandas.DataFrame (feature names are initialized from it's column names). Must be None if 'data' parameter has FeaturesData type

       column_description : string, optional (default=None). ColumnsDescription parameter. There are several columns description types: Label,   Categ, Num, Auxiliary, DocId, Weight, Baseline, GroupId,  Timestamp.  All columns are Num as default, it's not necessary to   specify  this type of columns. Default Label column index is 0   (zero).  If None, Label column is 0 (zero) as default, all data   columns are Num as default. If string, giving the path to the file with  ColumnsDescription in column_description format.

       pairs : list or numpy.ndarray or pandas.DataFrame or string.The pairs description.  If list or numpy.ndarrays or pandas.DataFrame, giving 2  dimensional. The shape should be Nx2, where N is the pairs' count.  The first element of the pair is  the index of winner object in the training set. The  second element of the pair is the index of loser object in the training set. If string, giving the path to the file with pairs description.

       delimiter : string, optional (default='\t'). Delimiter to use for separate features in file. Should be only one symbol, otherwise would be taken  only the first character of the string.

       has_header : bool optional (default=False). If True, read column names from first line.

       ignore_csv_quoting : bool optional (default=False). If True ignore quoting '"'.

       weight : list or numpy.ndarray, optional (default=None). Weight for each instance. If not None, giving 1 dimensional array like data.

       group_id : list or numpy.ndarray, optional (default=None). group id for each instance.  If not None, giving 1 dimensional array like data.

       group_weight : list or numpy.ndarray, optional .(default=None).Group weight for each instance.  If not None, giving 1 dimensional array like data.

       subgroup_id : list or numpy.ndarray, optional . (default=None) .subgroup id for each instance. If not None, giving 1 dimensional array like data.

       pairs_weight : list or numpy.ndarray, optional .(default=None).  Weight for each pair. If not None, giving 1 dimensional array like pairs.

       baseline : list or numpy.ndarray, optional (default=None).Baseline for each instance. If not None, giving 2 dimensional array like data.

       feature_names : list or string, optional (default=None). If list - list of names for each given data_feature. If string - path with scheme for feature names data to  load. If this parameter is None and 'data' is pandas.DataFrame   feature names will be initialized  from DataFrame's column names. Must be None if 'data' parameter has FeaturesData type

       thread_count : int, optional (default=-1).Thread count for data processing. If -1, then the number of threads is set to the number of  CPU cores.  

data : 列表或numpy。ndarray或pandas.DataFrame或pandas.Series或特性数据或字符串。池的数据源。如果列表或numpy。ndarrays或pandas.DataFrame或pandas.Series,给出类似数据的二维数组。如果FeaturesData -参见FeaturesData描述的详细信息,在这种情况下'cat_features'和'feature_names'参数必须等于None。如果字符串,以catboost格式提供文件的路径。如果path以"quantized://"开头,则文件必须包含保存在Pool.save()中的量化数据集。

label : 列表或numpy。ndarrays或pandas.DataFrame或pandas.Series,可选(默认= None)。训练数据的标签。如果不是None,则给出1维或2维的数组,如带有浮点数的data。如果data是一个文件,那么label必须在文件中,即label必须等于None cat_features: list或numpy。ndarray,可选(默认= None)。如果不是None,则给出类别特征索引或名称的列表。如果包含特性名称,则必须定义池的特性名称:通过传递'feature_names'参数或如果data是pandas。DataFrame(特性名称从它的列名初始化)。如果“data”参数具有FeaturesData类型,必须为None

text_features : 列表或numpy。ndarray,可选的。(default=None),如果不是None,则给出文本特性的索引或名称。如果它包含特性名称,则池的特性名称必须

be defined: 通过传递'feature_names'参数或如果数据是pandas.DataFrame特性名称从它的列名初始化)。如果“data”参数具有FeaturesData类型,必须为None

embedding_features : list或numpy。ndarray,可选的。(默认= None)。如果不是None,则给出嵌入特性的索引或名称。如果包含特性名称,则必须定义池的特性名称:通过传递'feature_names'参数或如果data是pandas。DataFrame(特性名称从它的列名初始化)。如果“data”参数具有FeaturesData类型,必须为None

column_description : 字符串,可选(默认为None)。ColumnsDescription参数。有几个列描述类型:Label, Categ, Num, Auxiliary, DocId, Weight, Baseline, GroupId, Timestamp。默认情况下,所有列都是Num,没有必要指定这种类型的列。默认标签列索引为0(零)。如果为None,则Label column默认为0(0),所有数据列默认为Num。如果是字符串,则以column_description格式给出该文件的路径。

pairs : 列表或numpy。ndarray或pandas.DataFrame或字符串。对描述。如果列表或numpy。ndarrays或pandas.DataFrame,给出二维。形状应该是Nx2,其中N是对的计数。pair的第一个元素是训练集中赢家对象的索引。对的第二个元素是输家对象在训练集中的索引。如果是字符串,则给出文件的路径和对的描述。

delimiter : 字符串,可选(默认='\t')。用于文件中单独特性的分隔符。应该只有一个符号,否则将只取字符串的第一个字符。

has_header : bool可选(默认为False)。如果为True,则从第一行读取列名。

ignore_csv_quoting :bool可选(默认为False)。如果为真,请忽略“”。

weight :列表或numpy。ndarray,可选(默认= None)。每个实例的权重。如果不是None,则给出类似data的一维数组。

group_id : list或numpy。ndarray,可选(默认= None)。每个实例的组id。如果不是None,则给出类似data的一维数组。

group_weight : list或numpy。ndarray,可选的。(默认= None)。每个实例的组权重。如果不是None,则给出类似data的一维数组。

subgroup_id : list或numpy。ndarray,可选的。(default=None) .subgroup每个实例的id。如果不是None,则给出类似data的一维数组。

pairs_weight :列表或numpy。ndarray,可选的。(默认= None)。每双的重量。如果不是None,则给出一个一维数组。

baseline:列表或numpy。ndarray,可选(默认= None)。每个实例的基线。如果不是None,则给出像data这样的二维数组。

feature_names : 列表或字符串,可选(默认为None)。If list -每个给定data_feature的名称列表。If string - path with scheme for feature names要加载的数据。如果此参数为None且'data'为pandas.DataFrame特性名称将从DataFrame的列名初始化。如果“data”参数具有FeaturesData类型,必须为None

thread_count : nt,可选(默认=-1)。用于数据处理的线程计数。如果-1,则线程数设置为CPU核数。

       if data is not None:

           self._check_data_type(data)

           self._check_data_empty(data)

           if pairs is not None and isinstance(data, STRING_TYPES) !

            = isinstance(pairs, STRING_TYPES):

               raise CatBoostError("data and pairs parameters should

                be the same types.")

           if column_description is not None and not isinstance

            (data, STRING_TYPES):

               raise CatBoostError("data should be the string type if

                column_description parameter is specified.")

           if isinstance(data, STRING_TYPES):

               if any(v is not None for v in [cat_features,

                text_features, embedding_features, weight, group_id,

                group_weight,

                       subgroup_id, pairs_weight, baseline, label]):

                   raise CatBoostError(

                       "cat_features, text_features, embedding_features,

                        weight, group_id, group_weight, subgroup_id, pairs_weight, "

                       "baseline, label should have the None type when

                        the pool is read from the file.")

               if (feature_names is not None) and (not isinstance

                (feature_names, STRING_TYPES)):

                   raise CatBoostError(

                       "feature_names should have None or string type

                        when the pool is read from the file.")

               self._read(data, column_description, pairs,

                feature_names, delimiter, has_header, ignore_csv_quoting,

                thread_count)

           else:

               if isinstance(data, FeaturesData):

                   if any(v is not None for v in [cat_features,

                    text_features, embedding_features, feature_names]):

                       raise CatBoostError(

                           "cat_features, text_features,

                            embedding_features, feature_names should have the None

                            type"

                           " when 'data' parameter has FeaturesData type")

               elif isinstance(data, np.ndarray):

                   if (data.dtype.kind == 'f') and (cat_features is not

                    None) and (len(cat_features) > 0):

                       raise CatBoostError("'data' is numpy array of

                        floating point numerical type, it means no categorical features,"

                           " but 'cat_features' parameter specifies nonzero

                            number of categorical features")

                   if (data.dtype.kind == 'f') and (text_features is not

                    None) and (len(text_features) > 0):

                       raise CatBoostError(

                           "'data' is numpy array of floating point

                            numerical type, it means no text features,"

                           " but 'text_features' parameter specifies

                            nonzero number of text features")

                   if (data.dtype.kind != 'O') and (embedding_features

                    is not None) and (len(embedding_features) > 0):

                       raise CatBoostError(

                           "'data' is numpy array of non-object type, it

                            means no embedding features,"

                           " but 'embedding_features' parameter specifies

                            nonzero number of embedding features")

               elif isinstance(data, scipy.sparse.spmatrix):

                   if (data.dtype.kind == 'f') and (cat_features is not

                    None) and (len(cat_features) > 0):

                       raise CatBoostError("'data' is scipy.sparse.

                        spmatrix of floating point numerical type, it means no

                        categorical features,"

                           " but 'cat_features' parameter specifies nonzero

                            number of categorical features")

                   if (text_features is not None) and (len(text_features)

                    > 0):

                       raise CatBoostError(

                           "'data' is scipy.sparse.spmatrix, it means no text

                            features,"

                           " but 'text_features' parameter specifies

                            nonzero number of text features")

                   if (embedding_features is not None) and (len

                    (embedding_features) > 0):

                       raise CatBoostError(

                           "'data' is scipy.sparse.spmatrix, it means no

                            embedding features,"

                           " but 'embedding_features' parameter specifies

                            nonzero number of embedding features")

               if isinstance(feature_names, STRING_TYPES):

                   raise CatBoostError(

                       "feature_names must be None or have non-string

                        type when the pool is created from "

                       "python objects.")

               self._init(data, label, cat_features, text_features,

                embedding_features, pairs, weight, group_id, group_weight,

                subgroup_id, pairs_weight, baseline, feature_names,

                thread_count)

       super(Pool, self).__init__()

     

   def _check_files(self, data, column_description, pairs):

       """

       Check files existence.

       """

       if data.find('://') == -1 and not os.path.isfile(data):

           raise CatBoostError("Invalid data path='{}': file does not

            exist.".format(data))

       if column_description is not None and column_description.

        find('://') == -1 and not os.path.isfile(column_description):

           raise CatBoostError("Invalid column_description

            path='{}': file does not exist.".format(column_description))

       if pairs is not None and pairs.find('://') == -1 and not os.

        path.isfile(pairs):

           raise CatBoostError("Invalid pairs path='{}': file does not

            exist.".format(pairs))

   

   def _check_delimiter(self, delimiter):

       if not isinstance(delimiter, STRING_TYPES):

           raise CatBoostError("Invalid delimiter type={} : must be

            str().".format(type(delimiter)))

       if len(delimiter) < 1:

           raise CatBoostError("Invalid delimiter length={} : must

            be > 0.".format(len(delimiter)))

   

   def _check_column_description_type(self,

    column_description):

       """

       Check type of column_description parameter.

       """

       if not isinstance(column_description, STRING_TYPES):

           raise CatBoostError("Invalid column_description type={}:

            must be str().".format(type(column_description)))

   

   def _check_string_feature_type(self, features, features_name):

       """

       Check type of cat_feature parameter.

       """

       if not isinstance(features, (list, np.ndarray)):

           raise CatBoostError("Invalid {} type={}: must be list() or

            np.ndarray().".format(features_name, type(features)))

   

   def _check_string_feature_value(self, features,

    features_count, features_name):

       """

       Check values in cat_feature parameter. Must be int indices.

       """

       for indx, feature in enumerate(features):

           if not isinstance(feature, INTEGER_TYPES):

               raise CatBoostError("Invalid {}[{}] = {} value type={}:

                must be int().".format(features_name, indx, feature, type

                (feature)))

           if feature >= features_count:

               raise CatBoostError("Invalid {}[{}] = {} value: index must

                be < {}.".format(features_name, indx, feature, features_count))

   

   def _check_pairs_type(self, pairs):

       """

       Check type of pairs parameter.

       """

       if not isinstance(pairs, (list, np.ndarray, DataFrame)):

           raise CatBoostError("Invalid pairs type={}: must be list(),

            np.ndarray() or pd.DataFrame.".format(type(pairs)))

   

   def _check_pairs_value(self, pairs):

       """

       Check values in pairs parameter. Must be int indices.

       """

       for pair_id, pair in enumerate(pairs):

           if (len(pair) != 2):

               raise CatBoostError("Length of pairs[{}] isn't equal to

                2.".format(pair_id))

           for i, index in enumerate(pair):

               if not isinstance(index, INTEGER_TYPES):

                   raise CatBoostError("Invalid pairs[{}][{}] = {} value

                    type={}: must be int().".format(pair_id, i, index, type(index)))

   

   def _check_data_type(self, data):

       """

       Check type of data.

       """

       if not isinstance(data, (STRING_TYPES, ARRAY_TYPES,

        SPARSE_MATRIX_TYPES, FeaturesData)):

           raise CatBoostError(

               "Invalid data type={}: data must be list(), np.ndarray(),

                DataFrame(), Series(), FeaturesData " + " scipy.sparse matrix or

                filename str().".format(type(data)))

   

   def _check_data_empty(self, data):

       """

       Check that data is not empty (0 objects is ok).

       note: already checked if data is FeatureType, so no need

        to check again

       """

       if isinstance(data, STRING_TYPES):

           if not data:

               raise CatBoostError("Features filename is empty.")

       elif isinstance(data, (ARRAY_TYPES,

        SPARSE_MATRIX_TYPES)):

           data_shape = np.shape(data)

           if len(data_shape) == 1 and data_shape[0] > 0:

               if isinstance(data[0], Iterable):

                   data_shape = tuple(data_shape + tuple([len(data

                    [0])]))

               else:

                   data_shape = tuple(data_shape + tuple([1]))

           if not len(data_shape) == 2:

               raise CatBoostError("Input data has invalid shape: {}.

                Must be 2 dimensional".format(data_shape))

           if data_shape[1] == 0:

               raise CatBoostError("Input data must have at least one

                feature")

   

   def _check_label_type(self, label):

       """

       Check type of label.

       """

       if not isinstance(label, ARRAY_TYPES):

           raise CatBoostError("Invalid label type={}: must be array

            like.".format(type(label)))

   

   def _check_label_empty(self, label):

       """

       Check label is not empty.

       """

       if len(label) == 0:

           raise CatBoostError("Labels variable is empty.")

   

   def _check_label_shape(self, label, samples_count):

       """

       Check label length and dimension.

       """

       if len(label) != samples_count:

           raise CatBoostError("Length of label={} and length of

            data={} is different.".format(len(label), samples_count))

   

   def _check_baseline_type(self, baseline):

       """

       Check type of baseline parameter.

       """

       if not isinstance(baseline, ARRAY_TYPES):

           raise CatBoostError("Invalid baseline type={}: must be

            array like.".format(type(baseline)))

   

   def _check_baseline_shape(self, baseline, samples_count):

       """

       Check baseline length and dimension.

       """

       if len(baseline) != samples_count:

           raise CatBoostError("Length of baseline={} and length of

            data={} are different.".format(len(baseline), samples_count))

       if not isinstance(baseline[0], Iterable) or isinstance(baseline

        [0], STRING_TYPES):

           raise CatBoostError("Baseline must be 2 dimensional

            data, 1 column for each class.")

       try:

           if np.array(baseline).dtype not in (np.dtype('float'), np.

            dtype('float32'), np.dtype('int')):

               raise CatBoostError()

       except CatBoostError:

           raise CatBoostError("Invalid baseline value type={}: must

            be float or int.".format(np.array(baseline).dtype))

   

   def _check_weight_type(self, weight):

       """

       Check type of weight parameter.

       """

       if not isinstance(weight, ARRAY_TYPES):

           raise CatBoostError("Invalid weight type={}: must be

            array like.".format(type(weight)))

   

   def _check_weight_shape(self, weight, samples_count):

       """

       Check weight length.

       """

       if len(weight) != samples_count:

           raise CatBoostError("Length of weight={} and length of

            data={} are different.".format(len(weight), samples_count))

       if not isinstance(weight[0], (INTEGER_TYPES,

        FLOAT_TYPES)):

           raise CatBoostError("Invalid weight value type={}: must

            be 1 dimensional data with int, float or long types.".format(type

            (weight[0])))

   

   def _check_group_id_type(self, group_id):

       """

       Check type of group_id parameter.

       """

       if not isinstance(group_id, ARRAY_TYPES):

           raise CatBoostError("Invalid group_id type={}: must be

            array like.".format(type(group_id)))

   

   def _check_group_id_shape(self, group_id, samples_count):

       """

       Check group_id length.

       """

       if len(group_id) != samples_count:

           raise CatBoostError("Length of group_id={} and length

            of data={} are different.".format(len(group_id), samples_count))

   

   def _check_group_weight_type(self, group_weight):

       """

       Check type of group_weight parameter.

       """

       if not isinstance(group_weight, ARRAY_TYPES):

           raise CatBoostError("Invalid group_weight type={}: must

            be array like.".format(type(group_weight)))

   

   def _check_group_weight_shape(self, group_weight,

    samples_count):

       """

       Check group_weight length.

       """

       if len(group_weight) != samples_count:

           raise CatBoostError("Length of group_weight={} and

            length of data={} are different.".format(len(group_weight),

            samples_count))

       if not isinstance(group_weight[0], (FLOAT_TYPES)):

           raise CatBoostError("Invalid group_weight value type={}:

            must be 1 dimensional data with float types.".format(type

            (group_weight[0])))

   

   def _check_subgroup_id_type(self, subgroup_id):

       """

       Check type of subgroup_id parameter.

       """

       if not isinstance(subgroup_id, ARRAY_TYPES):

           raise CatBoostError("Invalid subgroup_id type={}: must

            be array like.".format(type(subgroup_id)))

   

   def _check_subgroup_id_shape(self, subgroup_id,

    samples_count):

       """

       Check subgroup_id length.

       """

       if len(subgroup_id) != samples_count:

           raise CatBoostError("Length of subgroup_id={} and

            length of data={} are different.".format(len(subgroup_id),

            samples_count))

   

   def _check_feature_names(self, feature_names,

    num_col=None):

       if num_col is None:

           num_col = self.num_col()

       if not isinstance(feature_names, Sequence):

           raise CatBoostError("Invalid feature_names type={} :

            must be list".format(type(feature_names)))

       if len(feature_names) != num_col:

           raise CatBoostError("Invalid length of feature_names={} :

            must be equal to the number of columns in data={}".format(len

            (feature_names), num_col))

   

   def _check_thread_count(self, thread_count):

       if not isinstance(thread_count, INTEGER_TYPES):

           raise CatBoostError("Invalid thread_count type={} : must

            be int".format(type(thread_count)))


版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

相关文章
java.util.concurrent解析——ThreadPoolExecutor源码解析
任何一种语言、框架,线程都是非常重要的一部分。要想实现异步就需要通过异步线程,但是频繁地创建销毁线程会带来较大的性能开销,而线程池就是为解决这一问题而出现的
2803 0
+关注
一个处女座的程序猿
国内互联网圈知名博主、人工智能领域优秀创作者,全球最大中文IT社区博客专家、CSDN开发者联盟生态成员、中国开源社区专家、华为云社区专家、51CTO社区专家、Python社区专家等,曾受邀采访和评审十多次。仅在国内的CSDN平台,博客文章浏览量超过2500万,拥有超过57万的粉丝。
1701
文章
0
问答
文章排行榜
最热
最新
相关电子书
更多
《2021云上架构与运维峰会演讲合集》
立即下载
《零基础CSS入门教程》
立即下载
《零基础HTML入门教程》
立即下载