目录
catboost模型中常用的Pool类型数据源结构代码解读
Pool简介
Pool 是在CatBoost中用作训练模型的数据结构。
Pool的案例应用
1. train_dataPool = Pool(data=[[12, 14, 16, 18], [23, 25, 27, 29], [32, 34, 36, 38]], 2. label=[10, 20, 30], 3. weight=[0.1, 0.2, 0.3]) # weight各自权重默认设置为1
Pool源代码解读
class Pool Found at: catboost.core class Pool(_PoolBase): """ Pool used in CatBoost as a data structure to train model from. """ def __init__( self, data, label=None, cat_features=None, text_features=None, embedding_features=None, column_description=None, pairs=None, delimiter='\t', has_header=False, ignore_csv_quoting=False, weight=None, group_id=None, group_weight=None, subgroup_id=None, pairs_weight=None, baseline=None, feature_names=None, thread_count=-1): """ Pool is an internal data structure that is used by CatBoost. You can construct Pool from list, numpy.ndarray, pandas. DataFrame, pandas.Series. Parameters ---------- |
Pool 是在CatBoost中用作训练模型的数据结构。 |
data : list or numpy.ndarray or pandas.DataFrame or pandas.Series or FeaturesData or string. Data source of Pool. If list or numpy.ndarrays or pandas.DataFrame or pandas. Series, giving 2 dimensional array like data. If FeaturesData - see FeaturesData description for details, 'cat_features' and 'feature_names' parameters must be equal to None in this case. If string, giving the path to the file with data in catboost format. If path starts with "quantized://", the file has to contain quantized dataset saved with Pool.save(). label : list or numpy.ndarrays or pandas.DataFrame or pandas.Series, optional (default=None). Label of the training data. If not None, giving 1 or 2 dimensional array like data with floats. If data is a file, then label must be in the file, that is label must be equals to None cat_features : list or numpy.ndarray, optional (default=None). If not None, giving the list of Categ features indices or names. If it contains feature names, Pool's feature names must be defined: either by passing 'feature_names' parameter or if data is pandas.DataFrame (feature names are initialized from it's column names). Must be None if 'data' parameter has FeaturesData type text_features : list or numpy.ndarray, optional . (default=None), If not None, giving the list of Text features indices or names. If it contains feature names, Pool's feature names must be defined: either by passing 'feature_names' parameter or if data is pandas.DataFrame (feature names are initialized from it's column names). Must be None if 'data' parameter has FeaturesData type embedding_features : list or numpy.ndarray, optional . (default=None). If not None, giving the list of Embedding features indices or names. If it contains feature names, Pool's feature names must be defined: either by passing 'feature_names' parameter or if data is pandas.DataFrame (feature names are initialized from it's column names). Must be None if 'data' parameter has FeaturesData type column_description : string, optional (default=None). ColumnsDescription parameter. There are several columns description types: Label, Categ, Num, Auxiliary, DocId, Weight, Baseline, GroupId, Timestamp. All columns are Num as default, it's not necessary to specify this type of columns. Default Label column index is 0 (zero). If None, Label column is 0 (zero) as default, all data columns are Num as default. If string, giving the path to the file with ColumnsDescription in column_description format. pairs : list or numpy.ndarray or pandas.DataFrame or string.The pairs description. If list or numpy.ndarrays or pandas.DataFrame, giving 2 dimensional. The shape should be Nx2, where N is the pairs' count. The first element of the pair is the index of winner object in the training set. The second element of the pair is the index of loser object in the training set. If string, giving the path to the file with pairs description. delimiter : string, optional (default='\t'). Delimiter to use for separate features in file. Should be only one symbol, otherwise would be taken only the first character of the string. has_header : bool optional (default=False). If True, read column names from first line. ignore_csv_quoting : bool optional (default=False). If True ignore quoting '"'. weight : list or numpy.ndarray, optional (default=None). Weight for each instance. If not None, giving 1 dimensional array like data. group_id : list or numpy.ndarray, optional (default=None). group id for each instance. If not None, giving 1 dimensional array like data. group_weight : list or numpy.ndarray, optional .(default=None).Group weight for each instance. If not None, giving 1 dimensional array like data. subgroup_id : list or numpy.ndarray, optional . (default=None) .subgroup id for each instance. If not None, giving 1 dimensional array like data. pairs_weight : list or numpy.ndarray, optional .(default=None). Weight for each pair. If not None, giving 1 dimensional array like pairs. baseline : list or numpy.ndarray, optional (default=None).Baseline for each instance. If not None, giving 2 dimensional array like data. feature_names : list or string, optional (default=None). If list - list of names for each given data_feature. If string - path with scheme for feature names data to load. If this parameter is None and 'data' is pandas.DataFrame feature names will be initialized from DataFrame's column names. Must be None if 'data' parameter has FeaturesData type thread_count : int, optional (default=-1).Thread count for data processing. If -1, then the number of threads is set to the number of CPU cores. |
data : 列表或numpy。ndarray或pandas.DataFrame或pandas.Series或特性数据或字符串。池的数据源。如果列表或numpy。ndarrays或pandas.DataFrame或pandas.Series，给出类似数据的二维数组。如果FeaturesData -参见FeaturesData描述的详细信息，在这种情况下'cat_features'和'feature_names'参数必须等于None。如果字符串，以catboost格式提供文件的路径。如果path以"quantized://"开头，则文件必须包含保存在Pool.save()中的量化数据集。 label : 列表或numpy。ndarrays或pandas.DataFrame或pandas.Series,可选(默认= None)。训练数据的标签。如果不是None，则给出1维或2维的数组，如带有浮点数的data。如果data是一个文件，那么label必须在文件中，即label必须等于None cat_features: list或numpy。ndarray,可选(默认= None)。如果不是None，则给出类别特征索引或名称的列表。如果包含特性名称，则必须定义池的特性名称:通过传递'feature_names'参数或如果data是pandas。DataFrame(特性名称从它的列名初始化)。如果“data”参数具有FeaturesData类型，必须为None text_features : 列表或numpy。ndarray,可选的。(default=None)，如果不是None，则给出文本特性的索引或名称。如果它包含特性名称，则池的特性名称必须 be defined: 通过传递'feature_names'参数或如果数据是pandas.DataFrame特性名称从它的列名初始化)。如果“data”参数具有FeaturesData类型，必须为None embedding_features : list或numpy。ndarray,可选的。(默认= None)。如果不是None，则给出嵌入特性的索引或名称。如果包含特性名称，则必须定义池的特性名称:通过传递'feature_names'参数或如果data是pandas。DataFrame(特性名称从它的列名初始化)。如果“data”参数具有FeaturesData类型，必须为None column_description : 字符串，可选(默认为None)。ColumnsDescription参数。有几个列描述类型:Label, Categ, Num, Auxiliary, DocId, Weight, Baseline, GroupId, Timestamp。默认情况下，所有列都是Num，没有必要指定这种类型的列。默认标签列索引为0(零)。如果为None，则Label column默认为0(0)，所有数据列默认为Num。如果是字符串，则以column_description格式给出该文件的路径。 pairs : 列表或numpy。ndarray或pandas.DataFrame或字符串。对描述。如果列表或numpy。ndarrays或pandas.DataFrame，给出二维。形状应该是Nx2，其中N是对的计数。pair的第一个元素是训练集中赢家对象的索引。对的第二个元素是输家对象在训练集中的索引。如果是字符串，则给出文件的路径和对的描述。 delimiter : 字符串，可选(默认='\t')。用于文件中单独特性的分隔符。应该只有一个符号，否则将只取字符串的第一个字符。 has_header : bool可选(默认为False)。如果为True，则从第一行读取列名。 ignore_csv_quoting :bool可选(默认为False)。如果为真，请忽略“”。 weight :列表或numpy。ndarray,可选(默认= None)。每个实例的权重。如果不是None，则给出类似data的一维数组。 group_id : list或numpy。ndarray,可选(默认= None)。每个实例的组id。如果不是None，则给出类似data的一维数组。 group_weight : list或numpy。ndarray,可选的。(默认= None)。每个实例的组权重。如果不是None，则给出类似data的一维数组。 subgroup_id : list或numpy。ndarray,可选的。(default=None) .subgroup每个实例的id。如果不是None，则给出类似data的一维数组。 pairs_weight :列表或numpy。ndarray,可选的。(默认= None)。每双的重量。如果不是None，则给出一个一维数组。 baseline：列表或numpy。ndarray,可选(默认= None)。每个实例的基线。如果不是None，则给出像data这样的二维数组。 feature_names : 列表或字符串，可选(默认为None)。If list -每个给定data_feature的名称列表。If string - path with scheme for feature names要加载的数据。如果此参数为None且'data'为pandas.DataFrame特性名称将从DataFrame的列名初始化。如果“data”参数具有FeaturesData类型，必须为None thread_count : nt，可选(默认=-1)。用于数据处理的线程计数。如果-1，则线程数设置为CPU核数。 |
| if data is not None: self._check_data_type(data) self._check_data_empty(data) if pairs is not None and isinstance(data, STRING_TYPES) ! = isinstance(pairs, STRING_TYPES): raise CatBoostError("data and pairs parameters should be the same types.") if column_description is not None and not isinstance (data, STRING_TYPES): raise CatBoostError("data should be the string type if column_description parameter is specified.") if isinstance(data, STRING_TYPES): if any(v is not None for v in [cat_features, text_features, embedding_features, weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, label]): raise CatBoostError( "cat_features, text_features, embedding_features, weight, group_id, group_weight, subgroup_id, pairs_weight, " "baseline, label should have the None type when the pool is read from the file.") if (feature_names is not None) and (not isinstance (feature_names, STRING_TYPES)): raise CatBoostError( "feature_names should have None or string type when the pool is read from the file.") self._read(data, column_description, pairs, feature_names, delimiter, has_header, ignore_csv_quoting, thread_count) else: if isinstance(data, FeaturesData): if any(v is not None for v in [cat_features, text_features, embedding_features, feature_names]): raise CatBoostError( "cat_features, text_features, embedding_features, feature_names should have the None type" " when 'data' parameter has FeaturesData type") elif isinstance(data, np.ndarray): if (data.dtype.kind == 'f') and (cat_features is not None) and (len(cat_features) > 0): raise CatBoostError("'data' is numpy array of floating point numerical type, it means no categorical features," " but 'cat_features' parameter specifies nonzero number of categorical features") if (data.dtype.kind == 'f') and (text_features is not None) and (len(text_features) > 0): raise CatBoostError( "'data' is numpy array of floating point numerical type, it means no text features," " but 'text_features' parameter specifies nonzero number of text features") if (data.dtype.kind != 'O') and (embedding_features is not None) and (len(embedding_features) > 0): raise CatBoostError( "'data' is numpy array of non-object type, it means no embedding features," " but 'embedding_features' parameter specifies nonzero number of embedding features") elif isinstance(data, scipy.sparse.spmatrix): if (data.dtype.kind == 'f') and (cat_features is not None) and (len(cat_features) > 0): raise CatBoostError("'data' is scipy.sparse. spmatrix of floating point numerical type, it means no categorical features," " but 'cat_features' parameter specifies nonzero number of categorical features") if (text_features is not None) and (len(text_features) > 0): raise CatBoostError( "'data' is scipy.sparse.spmatrix, it means no text features," " but 'text_features' parameter specifies nonzero number of text features") if (embedding_features is not None) and (len (embedding_features) > 0): raise CatBoostError( "'data' is scipy.sparse.spmatrix, it means no embedding features," " but 'embedding_features' parameter specifies nonzero number of embedding features") if isinstance(feature_names, STRING_TYPES): raise CatBoostError( "feature_names must be None or have non-string type when the pool is created from " "python objects.") self._init(data, label, cat_features, text_features, embedding_features, pairs, weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, feature_names, thread_count) super(Pool, self).__init__() |
|
def _check_files(self, data, column_description, pairs): """ Check files existence. """ if data.find('://') == -1 and not os.path.isfile(data): raise CatBoostError("Invalid data path='{}': file does not exist.".format(data)) if column_description is not None and column_description. find('://') == -1 and not os.path.isfile(column_description): raise CatBoostError("Invalid column_description path='{}': file does not exist.".format(column_description)) if pairs is not None and pairs.find('://') == -1 and not os. path.isfile(pairs): raise CatBoostError("Invalid pairs path='{}': file does not exist.".format(pairs))
def _check_delimiter(self, delimiter): if not isinstance(delimiter, STRING_TYPES): raise CatBoostError("Invalid delimiter type={} : must be str().".format(type(delimiter))) if len(delimiter) < 1: raise CatBoostError("Invalid delimiter length={} : must be > 0.".format(len(delimiter)))
def _check_column_description_type(self, column_description): """ Check type of column_description parameter. """ if not isinstance(column_description, STRING_TYPES): raise CatBoostError("Invalid column_description type={}: must be str().".format(type(column_description)))
def _check_string_feature_type(self, features, features_name): """ Check type of cat_feature parameter. """ if not isinstance(features, (list, np.ndarray)): raise CatBoostError("Invalid {} type={}: must be list() or np.ndarray().".format(features_name, type(features)))
def _check_string_feature_value(self, features, features_count, features_name): """ Check values in cat_feature parameter. Must be int indices. """ for indx, feature in enumerate(features): if not isinstance(feature, INTEGER_TYPES): raise CatBoostError("Invalid {}[{}] = {} value type={}: must be int().".format(features_name, indx, feature, type (feature))) if feature >= features_count: raise CatBoostError("Invalid {}[{}] = {} value: index must be < {}.".format(features_name, indx, feature, features_count))
def _check_pairs_type(self, pairs): """ Check type of pairs parameter. """ if not isinstance(pairs, (list, np.ndarray, DataFrame)): raise CatBoostError("Invalid pairs type={}: must be list(), np.ndarray() or pd.DataFrame.".format(type(pairs)))
def _check_pairs_value(self, pairs): """ Check values in pairs parameter. Must be int indices. """ for pair_id, pair in enumerate(pairs): if (len(pair) != 2): raise CatBoostError("Length of pairs[{}] isn't equal to 2.".format(pair_id)) for i, index in enumerate(pair): if not isinstance(index, INTEGER_TYPES): raise CatBoostError("Invalid pairs[{}][{}] = {} value type={}: must be int().".format(pair_id, i, index, type(index)))
def _check_data_type(self, data): """ Check type of data. """ if not isinstance(data, (STRING_TYPES, ARRAY_TYPES, SPARSE_MATRIX_TYPES, FeaturesData)): raise CatBoostError( "Invalid data type={}: data must be list(), np.ndarray(), DataFrame(), Series(), FeaturesData " + " scipy.sparse matrix or filename str().".format(type(data)))
def _check_data_empty(self, data): """ Check that data is not empty (0 objects is ok). note: already checked if data is FeatureType, so no need to check again """ if isinstance(data, STRING_TYPES): if not data: raise CatBoostError("Features filename is empty.") elif isinstance(data, (ARRAY_TYPES, SPARSE_MATRIX_TYPES)): data_shape = np.shape(data) if len(data_shape) == 1 and data_shape[0] > 0: if isinstance(data[0], Iterable): data_shape = tuple(data_shape + tuple([len(data [0])])) else: data_shape = tuple(data_shape + tuple([1])) if not len(data_shape) == 2: raise CatBoostError("Input data has invalid shape: {}. Must be 2 dimensional".format(data_shape)) if data_shape[1] == 0: raise CatBoostError("Input data must have at least one feature")
def _check_label_type(self, label): """ Check type of label. """ if not isinstance(label, ARRAY_TYPES): raise CatBoostError("Invalid label type={}: must be array like.".format(type(label)))
def _check_label_empty(self, label): """ Check label is not empty. """ if len(label) == 0: raise CatBoostError("Labels variable is empty.")
def _check_label_shape(self, label, samples_count): """ Check label length and dimension. """ if len(label) != samples_count: raise CatBoostError("Length of label={} and length of data={} is different.".format(len(label), samples_count))
def _check_baseline_type(self, baseline): """ Check type of baseline parameter. """ if not isinstance(baseline, ARRAY_TYPES): raise CatBoostError("Invalid baseline type={}: must be array like.".format(type(baseline)))
def _check_baseline_shape(self, baseline, samples_count): """ Check baseline length and dimension. """ if len(baseline) != samples_count: raise CatBoostError("Length of baseline={} and length of data={} are different.".format(len(baseline), samples_count)) if not isinstance(baseline[0], Iterable) or isinstance(baseline [0], STRING_TYPES): raise CatBoostError("Baseline must be 2 dimensional data, 1 column for each class.") try: if np.array(baseline).dtype not in (np.dtype('float'), np. dtype('float32'), np.dtype('int')): raise CatBoostError() except CatBoostError: raise CatBoostError("Invalid baseline value type={}: must be float or int.".format(np.array(baseline).dtype))
def _check_weight_type(self, weight): """ Check type of weight parameter. """ if not isinstance(weight, ARRAY_TYPES): raise CatBoostError("Invalid weight type={}: must be array like.".format(type(weight)))
def _check_weight_shape(self, weight, samples_count): """ Check weight length. """ if len(weight) != samples_count: raise CatBoostError("Length of weight={} and length of data={} are different.".format(len(weight), samples_count)) if not isinstance(weight[0], (INTEGER_TYPES, FLOAT_TYPES)): raise CatBoostError("Invalid weight value type={}: must be 1 dimensional data with int, float or long types.".format(type (weight[0])))
def _check_group_id_type(self, group_id): """ Check type of group_id parameter. """ if not isinstance(group_id, ARRAY_TYPES): raise CatBoostError("Invalid group_id type={}: must be array like.".format(type(group_id)))
def _check_group_id_shape(self, group_id, samples_count): """ Check group_id length. """ if len(group_id) != samples_count: raise CatBoostError("Length of group_id={} and length of data={} are different.".format(len(group_id), samples_count))
def _check_group_weight_type(self, group_weight): """ Check type of group_weight parameter. """ if not isinstance(group_weight, ARRAY_TYPES): raise CatBoostError("Invalid group_weight type={}: must be array like.".format(type(group_weight)))
def _check_group_weight_shape(self, group_weight, samples_count): """ Check group_weight length. """ if len(group_weight) != samples_count: raise CatBoostError("Length of group_weight={} and length of data={} are different.".format(len(group_weight), samples_count)) if not isinstance(group_weight[0], (FLOAT_TYPES)): raise CatBoostError("Invalid group_weight value type={}: must be 1 dimensional data with float types.".format(type (group_weight[0])))
def _check_subgroup_id_type(self, subgroup_id): """ Check type of subgroup_id parameter. """ if not isinstance(subgroup_id, ARRAY_TYPES): raise CatBoostError("Invalid subgroup_id type={}: must be array like.".format(type(subgroup_id)))
def _check_subgroup_id_shape(self, subgroup_id, samples_count): """ Check subgroup_id length. """ if len(subgroup_id) != samples_count: raise CatBoostError("Length of subgroup_id={} and length of data={} are different.".format(len(subgroup_id), samples_count))
def _check_feature_names(self, feature_names, num_col=None): if num_col is None: num_col = self.num_col() if not isinstance(feature_names, Sequence): raise CatBoostError("Invalid feature_names type={} : must be list".format(type(feature_names))) if len(feature_names) != num_col: raise CatBoostError("Invalid length of feature_names={} : must be equal to the number of columns in data={}".format(len (feature_names), num_col))
def _check_thread_count(self, thread_count): if not isinstance(thread_count, INTEGER_TYPES): raise CatBoostError("Invalid thread_count type={} : must be int".format(type(thread_count))) |
|
def slice(self, rindex): if not isinstance(rindex, ARRAY_TYPES): raise CatBoostError("Invalid rindex type={} : must be list or numpy.ndarray".format(type(rindex))) slicedPool = Pool(None) slicedPool._take_slice(self, rindex) return slicedPool
def set_pairs(self, pairs): self._check_pairs_type(pairs) if isinstance(pairs, DataFrame): pairs = pairs.values self._check_pairs_value(pairs) self._set_pairs(pairs) return self
def set_feature_names(self, feature_names): self._check_feature_names(feature_names) self._set_feature_names(feature_names) return self
def set_baseline(self, baseline): self._check_baseline_type(baseline) baseline = self._if_pandas_to_numpy(baseline) baseline = np.reshape(baseline, (self.num_row(), -1)) self._check_baseline_shape(baseline, self.num_row()) self._set_baseline(baseline) return self
def set_weight(self, weight): self._check_weight_type(weight) weight = self._if_pandas_to_numpy(weight) self._check_weight_shape(weight, self.num_row()) self._set_weight(weight) return self
def set_group_id(self, group_id): self._check_group_id_type(group_id) group_id = self._if_pandas_to_numpy(group_id) self._check_group_id_shape(group_id, self.num_row()) self._set_group_id(group_id) return self
def set_group_weight(self, group_weight): self._check_group_weight_type(group_weight) group_weight = self._if_pandas_to_numpy(group_weight) self._check_group_weight_shape(group_weight, self. num_row()) self._set_group_weight(group_weight) return self
def set_subgroup_id(self, subgroup_id): self._check_subgroup_id_type(subgroup_id) subgroup_id = self._if_pandas_to_numpy(subgroup_id) self._check_subgroup_id_shape(subgroup_id, self. num_row()) self._set_subgroup_id(subgroup_id) return self
def set_pairs_weight(self, pairs_weight): self._check_weight_type(pairs_weight) pairs_weight = self._if_pandas_to_numpy(pairs_weight) self._check_weight_shape(pairs_weight, self.num_pairs()) self._set_pairs_weight(pairs_weight) return self
def save(self, fname): """ Save the quantized pool to a file. Parameters ---------- fname : string Output file name. """ if not self.is_quantized(): raise CatBoostError('Pool is not quantized') if not isinstance(fname, STRING_TYPES): raise CatBoostError("Invalid fname type={}: must be str().".format(type(fname))) self._save(fname) |
|
def quantize(self, ignored_features=None, per_float_feature_quantization=None, border_count=None, max_bin=None, feature_border_type=None, sparse_features_conflict_fraction=None, nan_mode=None, input_borders=None, task_type=None, used_ram_limit=None, random_seed=None, **kwargs): """ Quantize this pool Parameters ---------- pool : catboost.Pool Dataset to quantize. ignored_features : list, [default=None] Indices or names of features that should be excluded when training. per_float_feature_quantization : list of strings, [default=None] List of float binarization descriptions. Format : described in documentation on catboost.ai Example 1: ['0:1024'] means that feature 0 will have 1024 borders. Example 2: ['0:border_count=1024', '1: border_count=1024', ...] means that two first features have 1024 borders. Example 3: ['0:nan_mode=Forbidden,border_count=32, border_type=GreedyLogSum', '1:nan_mode=Forbidden,border_count=32, border_type=GreedyLogSum'] - defines more quantization properties for first two features. border_count : int, [default = 254 for training on CPU or 128 for training on GPU] The number of partitions in numeric features binarization. Used in the preliminary calculation. range: [1,65535] on CPU, [1,255] on GPU max_bin : float, synonym for border_count. feature_border_type : string, [default='GreedyLogSum'] The binarization mode in numeric features binarization. Used in the preliminary calculation. Possible values: - 'Median' - 'Uniform' - 'UniformAndQuantiles' - 'GreedyLogSum' - 'MaxLogSum' - 'MinEntropy' sparse_features_conflict_fraction : float, [default=0.0] CPU only. Maximum allowed fraction of conflicting non- default values for features in exclusive features bundle. Should be a real value in [0, 1) interval. nan_mode : string, [default=None] Way to process missing values for numeric features. Possible values: - 'Forbidden' - raises an exception if there is a missing value for a numeric feature in a dataset. - 'Min' - each missing value will be processed as the minimum numerical value. - 'Max' - each missing value will be processed as the maximum numerical value. If None, then nan_mode=Min. input_borders : string, [default=None] input file with borders used in numeric features binarization. task_type : string, [default=None] The calcer type used to train the model. Possible values: - 'CPU' - 'GPU' used_ram_limit=None random_seed : int, [default=None] The random seed used for data sampling. If None, 0 is used. """ if self.is_quantized(): raise CatBoostError('Pool is already quantized') params = {} _process_synonyms(params) if border_count is None: border_count = max_bin dev_efb_max_buckets = kwargs.pop ('dev_efb_max_buckets', None) dev_max_subset_size_for_build_borders = kwargs.pop ('dev_max_subset_size_for_build_borders', None) if kwargs: raise CatBoostError("got an unexpected keyword arguments: {}".format(kwargs.keys())) _update_params_quantize_part(params, ignored_features, per_float_feature_quantization, border_count, feature_border_type, sparse_features_conflict_fraction, dev_efb_max_buckets, nan_mode, input_borders, task_type, used_ram_limit, random_seed, dev_max_subset_size_for_build_borders) self._quantize(params)
def _if_pandas_to_numpy(self, array): if isinstance(array, Series): array = array.values if isinstance(array, DataFrame): array = np.transpose(array.values)[0] return array
def _label_if_pandas_to_numpy(self, label): if isinstance(label, Series): label = label.values if isinstance(label, DataFrame): label = label.values return label
def _read( self, pool_file, column_description, pairs, feature_names_path, delimiter, has_header, ignore_csv_quoting, thread_count, quantization_params=None): """ Read Pool from file. """ with log_fixup(): self._check_files(pool_file, column_description, pairs) self._check_delimiter(delimiter) if column_description is None: column_description = '' else: self._check_column_description_type (column_description) if pairs is None: pairs = '' if feature_names_path is None: feature_names_path = '' self._check_thread_count(thread_count) self._read_pool(pool_file, column_description, pairs, feature_names_path, delimiter[0], has_header, ignore_csv_quoting, thread_count, quantization_params)
def _init( self, data, label, cat_features, text_features, embedding_features, pairs, weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, feature_names, thread_count): """ Initialize Pool from array like data. """ if isinstance(data, DataFrame): if feature_names is None: feature_names = list(data.columns) if isinstance(data, Series): data = data.values.tolist() if isinstance(data, FeaturesData): samples_count = data.get_object_count() features_count = data.get_feature_count() else: if len(np.shape(data)) == 1: data = np.expand_dims(data, 1) samples_count, features_count = np.shape(data) pairs_len = 0 if label is not None: self._check_label_type(label) self._check_label_empty(label) label = self._label_if_pandas_to_numpy(label) if len(np.shape(label)) == 1: label = np.expand_dims(label, 1) self._check_label_shape(label, samples_count) if feature_names is not None: self._check_feature_names(feature_names, features_count) if cat_features is not None: cat_features = _get_features_indices(cat_features, feature_names) self._check_string_feature_type(cat_features, 'cat_features') self._check_string_feature_value(cat_features, features_count, 'cat_features') if text_features is not None: text_features = _get_features_indices(text_features, feature_names) self._check_string_feature_type(text_features, 'text_features') self._check_string_feature_value(text_features, features_count, 'text_features') if embedding_features is not None: embedding_features = _get_features_indices (embedding_features, feature_names) self._check_string_feature_type(embedding_features, 'embedding_features') self._check_string_feature_value(embedding_features, features_count, 'embedding_features') if pairs is not None: self._check_pairs_type(pairs) if isinstance(pairs, DataFrame): pairs = pairs.values self._check_pairs_value(pairs) pairs_len = np.shape(pairs)[0] if weight is not None: self._check_weight_type(weight) weight = self._if_pandas_to_numpy(weight) self._check_weight_shape(weight, samples_count) if group_id is not None: self._check_group_id_type(group_id) group_id = self._if_pandas_to_numpy(group_id) self._check_group_id_shape(group_id, samples_count) if group_weight is not None: self._check_group_weight_type(group_weight) group_weight = self._if_pandas_to_numpy (group_weight) self._check_group_weight_shape(group_weight, samples_count) if subgroup_id is not None: self._check_subgroup_id_type(subgroup_id) subgroup_id = self._if_pandas_to_numpy(subgroup_id) self._check_subgroup_id_shape(subgroup_id, samples_count) if pairs_weight is not None: self._check_weight_type(pairs_weight) pairs_weight = self._if_pandas_to_numpy(pairs_weight) self._check_weight_shape(pairs_weight, pairs_len) if baseline is not None: self._check_baseline_type(baseline) baseline = self._if_pandas_to_numpy(baseline) baseline = np.reshape(baseline, (samples_count, -1)) self._check_baseline_shape(baseline, samples_count) self._init_pool(data, label, cat_features, text_features, embedding_features, pairs, weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, feature_names, thread_count) |
