背景
在使用pandas过程由于文本中存在形如,
、|
等常规字符,所以需要自定义分隔符,特别是自定义由多个字符组成的分隔符。那么此时在使用 pandas.read_csv()的时候要如何设置?
解决
比如当生成文件的时候使用#|#
作为分隔符,直接使用df = pd.read_csv(raw_file, sep='#|#', quoting=3)
会报错:
df = pd.read_csv(raw_file, sep='#|#', quoting=3)
File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 460, in _read
data = parser.read(nrows)
File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 1198, in read
ret = self._engine.read(nrows)
File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 2585, in read
alldata = self._rows_to_cols(content)
File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 3237, in _rows_to_cols
self._alert_malformed(msg, row_num + 1)
File "/data/miniconda3/envs/python36/lib/python3.6/site-packages/pandas/io/parsers.py", line 2998, in _alert_malformed
raise ParserError(msg)
需要将其改为:
df = pd.read_csv(raw_file, sep='\#\|\#', quoting=3)
官方文档是这么说明的:
In addition, separators longer than 1 character and different from '\s+' will be interpreted as regular expressions and will also force the use of the Python parsing engine. Note that regex delimiters are prone to ignoring quoted data. Regex example: '\r\t'.