今天和大家分享一个python入库mongodb的脚本。。。
涉及到python和mongodb，那么安装相应的模块四必不可少的，最简单的安装方法，或者非pip不可了。
|
1
|
# pip install pymongo==3.0.4
|
顺便也记录下源码安装的方式
|
1
2
3
4
|
# wget https://pypi.python.org/packages/source/p/pymongo/pymongo-2.8.tar.gz#md5=23100361c9af1904eb2d7722f2658114 --no-check-certificate
# tar xf pymongo-2.8.tar.gz
# cd pymongo-2.8
# python setup.py install
|
摘自一则日志
|
1
|
35783 s100 android 47 5 192.168.1.100 2015-09-05 08:03:19 strengthenHeroByHeroes {
"consume_gold"
:{
"ogold"
:2893821,
"cgold"
:1700,
"gold"
:2892121,
"tag"
:
"strengthenHeroByHeroes"
},
"taskInfo"
:[{
"id"
:2310033,
"progress"
:2,
"status"
:0}],
"delHeroList"
:{
"id"
:102014,
"id"
:102014,
"id"
:102014,
"id"
:102010,
"id"
:102010},
"id"
:100026,
"olevel"
:46,
"oexp"
:1700,
"cexp"
:1700,
"level"
:46,
"exp"
:3400} 865982021462182 XiaoMi
|
入库mongodb的python脚本
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
|
[root@localhost opt]
# cat analytical.py
#!/usr/bin/env python
#coding:utf8
import
os,sys,json
from
datetime
import
*
from
pymongo
import
MongoClient
def
ConMongo(host,port,cur_db,username,password):
client
=
MongoClient(host,port)
db
=
client[cur_db]
db.authenticate(username,password)
table
=
db.gamelogs
return
table
def
parseLog(logfile,table):
dic
=
{}
dl
=
[]
with
open
(file_log) as fd:
for
line
in
fd:
try
:
tokens
=
line.strip().split(
'\t'
)
uid
=
tokens[
0
]
server
=
tokens[
1
]
system
=
tokens[
2
]
level
=
int
(tokens[
3
])
vip_level
=
tokens[
4
]
ip
=
tokens[
5
]
time
=
datetime.strptime(tokens[
6
],
"%Y-%m-%d %H:%M:%S"
)
#将时间字符串转换成时间格式
action
=
tokens[
7
]
result
=
json.loads(tokens[
8
])
#特殊字符串转换成json格式
uuid
=
tokens[
9
]
if
len
(tokens)
=
=
12
:
channel
=
tokens[
11
]
else
:
channel
=
''
dic
=
{
'uid'
:uid,
'server'
:server,
'system'
:system,
'level'
:level,
'vip_level'
:vip_level,
'ip'
:ip,
'time'
:time,
'action'
:action,
'result'
:result,
'uuid'
:uuid,
'channel'
:channel}
dl.append(dic)
if
len
(dl)
=
=
10000
:
table.insert_many(dl)
dl
=
[]
except
Exception,e:
print
e, line
if
len
(dl) >
0
:
table.insert_many(dl)
if
__name__
=
=
'__main__'
:
table
=
ConMongo(
'localhost'
,
27017
,
'talefundb'
,
'talefun'
,
'123456'
)
try
:
logfile
=
sys.argv[
1
]
parseLog(logfile,table)
except
IndexError,e:
print
e
|
注意事项：
|
1
2
3
|
(1)insert_many参数是mongodb 3.0.4中新加的，允许你将一个大列表直接insert到mongodb数据库中
(2)脚本中做了限制，如果字典中有2000个值，就向mongodb插入一次数据，这样在效率上得到了保证
(3)不建议直接复制脚本测试，很多粘贴出来后，很多制表符等会出现问题。我会吧脚本放在云盘上大家可以下载，测试用。
|
点击可下载：http://pan.baidu.com/s/1qWtbgjq
本文转自zys467754239 51CTO博客，原文链接：http://blog.51cto.com/467754239/1692088，如需转载请自行联系原作者
