Golang实现web api接口调用及web数据抓取[get post模式]

简介:

前沿:

   继续扩展我的golang服务端,这边有些数据库是没有权限的,对方给了我webservices的接口,针对异常的数据,我要去抓数据,再次分析,golang貌似没有python那么多的模拟浏览器访问的模块,还好默认的http就支持。 功能一点都不必urllib2 差。。。



   正题!!! 这里是通过golang提供的net/http模块, http.NewRequest来进行数据抓取。 他能实现python下的urllib2的功能 !


原文:http://rfyiamcool.blog.51cto.com/1030776/1384473  


原理不多说了,大家直接套用这两个get post的例子吧。


可以任意的加header头,比如怎么加一个浏览器的标识 !

1
2
3
4
5
client := &http.Client{]
req, err := http.NewRequest( "POST" "http://127.0.0.1" , bytes.NewReader(postData))
req.Header.Add( "User-Agent" "无敌浏览器" )
resp, err := client.Do(req)
defer resp.Body.Close()


下面是完整的例子,可以加更多的Header


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#http: //xiaorui.cc
package  main
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
import  (
     "net/http"
     "io/ioutil"
     "fmt"
     "net/url"
)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
func main() {
     client := &http.Client{}
     reqest, _ := http.NewRequest( "GET" "http://127.0.0.1/" , nil)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
     reqest.Header.Set( "Accept" , "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" )
     reqest.Header.Set( "Accept-Charset" , "GBK,utf-8;q=0.7,*;q=0.3" )
     reqest.Header.Set( "Accept-Encoding" , "gzip,deflate,sdch" )
     reqest.Header.Set( "Accept-Language" , "zh-CN,zh;q=0.8" )
     reqest.Header.Set( "Cache-Control" , "max-age=0" )
     reqest.Header.Set( "Connection" , "keep-alive" )
     reqest.Header.Set( "User-Agent" , "chrome 100" )
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
     response,_ := client.Do(reqest)
     if  response.StatusCode ==  200  {
         body, _ := ioutil.ReadAll(response.Body)
         bodystr := string(body);
         fmt.Println(bodystr)
     }
//  reqest, _ = http.NewRequest("POST","http:/127.0.0.1/", bytes.NewBufferString(data.Encode()))
/    respet1,_ := http.NewRequest( "POST" , "http://127.0.0.1/" ,url.Values{ "key" : "Value" })
//    reqest1.Header.Set("User-Agent","chrome 100")
//    client.Do(reqest1)
}


我们再来测试下 post获取数据 !


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#http: //xiaorui.cc
package  main
import (
          "fmt"
          "net/http"
          "net/url"
          "io/ioutil"
  )
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
func main(){
         get ()
         post()
}
  func  get (){
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
          response,_:=http.Get( "http://127.0.0.1/" )
          defer response.Body.Close()
          body,_:=ioutil.ReadAll(response.Body)
          fmt.Println(string(body))
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
          if  response.StatusCode ==  200  {=
                  fmt.Println( "ok" )
          } else {
                  fmt.Println( "error" )
          }
  }
  func post(){
          //resp, err :=
          http.PostForm( "http://127.0.0.1" ,
                  url.Values{ "name" : { "ruifengyun" },  "blog" : { "xiaorui.cc" },
                  "aihao" :{ "python golang" }, "content" :{ "nima,fuck " }})
  }



  我们用http加上golang的runtime可以搞成类似 ab的压力测试工具,我昨天写了一个版本,但是在压倒3k以上的链接数的时候,会出现不少的error,原因可能是linux本身没有做tcp的优化,获取是对端的tornado没有用@gen,所以效率跟不上去,我的压力程序没有做defer panic处理。 今天看了下 golang的 gb压力测试工具,发现主要的思路是相同的,但是很多的细节没有做处理,比如channel的同步是用那种for <-c 的土方法实现的。


我的程序是有问题,但是老外有大牛已经构建了一套类似ab的工具,性能差不多,但是这个支持更多的选项和参数,包括代理,基本认证,请求头header信息,长链接,post,gzip压缩,开启几个cpu核心,cookie的插入。


1
2
go  get  github.com/parkghost/gohttpbench
go build -o gb github.com/parkghost/gohttpbench


用golang实现的搞并发的压力测试工具 !

原文:xiaoruicc

 -A="": Add Basic WWW Authentication, the attributes are a colon separated username and password.        

 -C=[]: Add cookie, eg. 'Apache=1234. (repeatable)                                                      

 -G=4: Number of CPU                                                                                    

 -H=[]: Add Arbitrary header line, eg. 'Accept-Encoding: gzip' Inserted after all normal header lines. (r

epeatable)                                                                                                

 -T="text/plain": Content-type header for POSTing, eg. 'application/x-www-form-urlencoded' Default is 'te

xt/plain'                                                                                                

 -c=1: Number of multiple requests to make                                                              

 -h=false: Display usage information (this message)                                                      

 -i=false: Use HEAD instead of GET                                                                      

 -k=false: Use HTTP KeepAlive feature                                                                    

 -n=1: Number of requests to perform                                                                    

 -p="": File containing data to POST. Remember also to set -T                                            

 -r=false: Don't exit when errors                                                                        

 -t=0: Seconds to max. wait for responses                                                                

 -u="": File containing data to PUT. Remember also to set -T                                            

 -v=0: How much troubleshooting info to print                                                            

 -z=false: Use HTTP Gzip feature                                                                        


wKiom1MybG7Q9OCWAAgXzjPTBo0978.jpg


具体的用法:

wKiom1MybJyxPy2CAAUE_hjCdcE841.jpg


咱们在看看nginx服务端的日志情况:

wKiom1MybvfyFxAyABD8hLgjayk480.jpg






 本文转自 rfyiamcool 51CTO博客,原文链接:http://blog.51cto.com/rfyiamcool/1384473,如需转载请自行联系原作者



相关文章
|
4天前
|
SQL 缓存 测试技术
RESTful API设计的最佳实践:构建高效、可维护的Web服务接口
【6月更文挑战第11天】构建高效、可维护的RESTful API涉及多个最佳实践:遵循客户端-服务器架构、无状态性等REST原则;设计时考虑URL结构(动词+宾语,使用标准HTTP方法)、使用HTTP状态码、统一响应格式及错误处理;确保数据安全(HTTPS、认证授权、输入验证);实施版本控制;并提供详细文档和测试用例。这些实践能提升Web服务接口的性能和质量。
|
8天前
|
JSON 前端开发 API
Apache HttpClient调用Spring3 MVC Restful Web API演示
Apache HttpClient调用Spring3 MVC Restful Web API演示
11 1
|
8天前
|
XML JavaScript 前端开发
基于SVG的web页面图形绘制API介绍
基于SVG的web页面图形绘制API介绍
14 4
|
11天前
|
XML JSON API
RESTful API关键部分组成和构建web应用程序步骤
RESTful API关键部分组成和构建web应用程序步骤
13 0
|
16天前
|
机器学习/深度学习 SQL 人工智能
Web LLM 实验:利用 LLM API 实现命令注入
Web LLM 实验:利用 LLM API 实现命令注入
|
16天前
|
SQL 机器学习/深度学习 人工智能
Web LLM 实验:利用 LLM API 实现 SQL 注入
Web LLM 实验:利用 LLM API 实现 SQL 注入
|
17天前
|
XML API 网络架构
Web Service和Web API理解和使用场景
**Web Service**是一种基于网络、使用SOAP协议和XML的数据封装的重服务,适用于跨平台、跨语言的企业系统集成,尤其在安全性和事务处理严格的场景,如银行系统。而**Web API**是轻量级的HTTP接口,常遵循REST原则,使用JSON格式,适合移动应用、开放平台和微服务间的通信,因其简洁高效。选择哪种取决于项目需求,Web Service适合复杂交互,Web API则流行于现代Web应用。
|
3天前
|
传感器 小程序 搜索推荐
(源码)java开发的一套(智慧校园系统源码、电子班牌、原生小程序开发)多端展示:web端、saas端、家长端、教师端
通过电子班牌设备和智慧校园数据平台的统一管理,在电子班牌上,班牌展示、学生上课刷卡考勤、考勤状况汇总展示,课表展示,考场管理,请假管理,成绩查询,考试优秀标兵展示、校园通知展示,班级文化各片展示等多种化展示。
25 0
(源码)java开发的一套(智慧校园系统源码、电子班牌、原生小程序开发)多端展示:web端、saas端、家长端、教师端
|
10天前
|
关系型数据库 MySQL PHP
PHP的生命周期:从诞生到现代Web开发
本文将探索PHP的发展历程,从其最初的设计目标到成为现代Web开发的核心语言。我们将深入了解PHP如何适应不断变化的技术环境,并保持其在开发者社区中的受欢迎程度。
|
1天前
|
前端开发 JavaScript Java
计算机Java项目|基于Web的足球青训俱乐部管理后台系统的设计与开发
计算机Java项目|基于Web的足球青训俱乐部管理后台系统的设计与开发