开发者社区> 技术小胖子> 正文
阿里云
为了无法计算的价值
打开APP
阿里云APP内打开

Golang实现web api接口调用及web数据抓取[get post模式]

简介:
+关注继续查看

前沿:

   继续扩展我的golang服务端,这边有些数据库是没有权限的,对方给了我webservices的接口,针对异常的数据,我要去抓数据,再次分析,golang貌似没有python那么多的模拟浏览器访问的模块,还好默认的http就支持。 功能一点都不必urllib2 差。。。



   正题!!! 这里是通过golang提供的net/http模块, http.NewRequest来进行数据抓取。 他能实现python下的urllib2的功能 !


原文:http://rfyiamcool.blog.51cto.com/1030776/1384473  


原理不多说了,大家直接套用这两个get post的例子吧。


可以任意的加header头,比如怎么加一个浏览器的标识 !

1
2
3
4
5
client := &http.Client{]
req, err := http.NewRequest("POST""http://127.0.0.1", bytes.NewReader(postData))
req.Header.Add("User-Agent""无敌浏览器")
resp, err := client.Do(req)
defer resp.Body.Close()


下面是完整的例子,可以加更多的Header


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#http://xiaorui.cc
package main
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
import (
    "net/http"
    "io/ioutil"
    "fmt"
    "net/url"
)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
func main() {
    client := &http.Client{}
    reqest, _ := http.NewRequest("GET""http://127.0.0.1/", nil)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
    reqest.Header.Set("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
    reqest.Header.Set("Accept-Charset","GBK,utf-8;q=0.7,*;q=0.3")
    reqest.Header.Set("Accept-Encoding","gzip,deflate,sdch")
    reqest.Header.Set("Accept-Language","zh-CN,zh;q=0.8")
    reqest.Header.Set("Cache-Control","max-age=0")
    reqest.Header.Set("Connection","keep-alive")
    reqest.Header.Set("User-Agent","chrome 100")
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
    response,_ := client.Do(reqest)
    if response.StatusCode == 200 {
        body, _ := ioutil.ReadAll(response.Body)
        bodystr := string(body);
        fmt.Println(bodystr)
    }
//  reqest, _ = http.NewRequest("POST","http:/127.0.0.1/", bytes.NewBufferString(data.Encode()))
/    respet1,_ := http.NewRequest("POST","http://127.0.0.1/",url.Values{"key":"Value"})
//    reqest1.Header.Set("User-Agent","chrome 100")
//    client.Do(reqest1)
}


我们再来测试下 post获取数据 !


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#http://xiaorui.cc
package main
import(
         "fmt"
         "net/http"
         "net/url"
         "io/ioutil"
 )
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
func main(){
        get()
        post()
}
 func get(){
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
         response,_:=http.Get("http://127.0.0.1/")
         defer response.Body.Close()
         body,_:=ioutil.ReadAll(response.Body)
         fmt.Println(string(body))
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
         if response.StatusCode == 200 {=
                 fmt.Println("ok")
         }else{
                 fmt.Println("error")
         }
 }
 func post(){
         //resp, err :=
         http.PostForm("http://127.0.0.1",
                 url.Values{"name": {"ruifengyun"}, "blog": {"xiaorui.cc"},
                 "aihao":{"python golang"},"content":{"nima,fuck "}})
 }



  我们用http加上golang的runtime可以搞成类似 ab的压力测试工具,我昨天写了一个版本,但是在压倒3k以上的链接数的时候,会出现不少的error,原因可能是linux本身没有做tcp的优化,获取是对端的tornado没有用@gen,所以效率跟不上去,我的压力程序没有做defer panic处理。 今天看了下 golang的 gb压力测试工具,发现主要的思路是相同的,但是很多的细节没有做处理,比如channel的同步是用那种for <-c 的土方法实现的。


我的程序是有问题,但是老外有大牛已经构建了一套类似ab的工具,性能差不多,但是这个支持更多的选项和参数,包括代理,基本认证,请求头header信息,长链接,post,gzip压缩,开启几个cpu核心,cookie的插入。


1
2
go get github.com/parkghost/gohttpbench
go build -o gb github.com/parkghost/gohttpbench


用golang实现的搞并发的压力测试工具 !

原文:xiaoruicc

 -A="": Add Basic WWW Authentication, the attributes are a colon separated username and password.        

 -C=[]: Add cookie, eg. 'Apache=1234. (repeatable)                                                      

 -G=4: Number of CPU                                                                                    

 -H=[]: Add Arbitrary header line, eg. 'Accept-Encoding: gzip' Inserted after all normal header lines. (r

epeatable)                                                                                                

 -T="text/plain": Content-type header for POSTing, eg. 'application/x-www-form-urlencoded' Default is 'te

xt/plain'                                                                                                

 -c=1: Number of multiple requests to make                                                              

 -h=false: Display usage information (this message)                                                      

 -i=false: Use HEAD instead of GET                                                                      

 -k=false: Use HTTP KeepAlive feature                                                                    

 -n=1: Number of requests to perform                                                                    

 -p="": File containing data to POST. Remember also to set -T                                            

 -r=false: Don't exit when errors                                                                        

 -t=0: Seconds to max. wait for responses                                                                

 -u="": File containing data to PUT. Remember also to set -T                                            

 -v=0: How much troubleshooting info to print                                                            

 -z=false: Use HTTP Gzip feature                                                                        


wKiom1MybG7Q9OCWAAgXzjPTBo0978.jpg


具体的用法:

wKiom1MybJyxPy2CAAUE_hjCdcE841.jpg


咱们在看看nginx服务端的日志情况:

wKiom1MybvfyFxAyABD8hLgjayk480.jpg






 本文转自 rfyiamcool 51CTO博客,原文链接:http://blog.51cto.com/rfyiamcool/1384473,如需转载请自行联系原作者

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

相关文章
dotweb——go语言的一个微型web框架(三)路由注册
上一章我们讲了如何启动一个dotweb程序,本篇文章将介绍如何注册路由。 router是dotweb用来管理路由的结构体,它提供了一些关于路由操作函数。 app := dotweb.New() router := app.HttpServer.Router() 在上一篇文章中我们讲了dotweb.New()的用处,HttpServer负责处理请求,管理路由、session、中间件等等功能。
983 0
go语言学习与面向对象再思考
据说Smalltalk发明者Alan Kay就曾经说过: 我发明了面向对象,而我可以告诉你C++并不是我头脑里所想的东西. 面向对象的困惑 计算机里面只有数据和算法,数据用于对现实世界抽象建模,算法对数据演算,这已经很好,怎么出来一个面向对象这个"怪胎". 面向对象教学每次都要从对现实世
1912 0
Vue 全家桶实现网易云音乐 WebApp
基于 Vue(2.5) + vuex + vue-router + vue-axios +better-scroll + Scss + ES6 等开发一款移动端音乐 WebApp,UI 界面参考了安卓版的网易云音乐、flex 布局适配常见移动端。
3336 0
21114
文章
0
问答
文章排行榜
最热
最新
相关电子书
更多
低代码开发师(初级)实战教程
立即下载
阿里巴巴DevOps 最佳实践手册
立即下载
冬季实战营第三期:MySQL数据库进阶实战
立即下载