我在用webmagic抓取https网站时报错?报错-问答-阿里云开发者社区-阿里云

开发者社区> 问答> 正文
阿里云
为了无法计算的价值
打开APP
阿里云APP内打开

我在用webmagic抓取https网站时报错?报错

2020-06-10 13:37:24 581 1

@黄亿华 你好,想跟你请教个问题:

我在使用你写的webmagic时如果抓去普通http网站是可以的,但是抓取https网站就出错。

16-03-02 21:45:25,487 INFO  org.springframework.context.support.ClassPathXmlApplicationContext(AbstractApplicationContext.java:495) ## Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@fa3ac1: startup date [Wed Mar 02 21:45:25 CST 2016]; root of context hierarchy
16-03-02 21:45:25,582 INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader(XmlBeanDefinitionReader.java:315) ## Loading XML bean definitions from file [C:\Documents and Settings\Administrator\workspace_java\cowpu-crawler\WebRoot\WEB-INF\classes\spring\applicationContext-myBatis.xml]
16-03-02 21:45:25,701 INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader(XmlBeanDefinitionReader.java:315) ## Loading XML bean definitions from file [C:\Documents and Settings\Administrator\workspace_java\cowpu-crawler\WebRoot\WEB-INF\classes\spring\applicationContext.xml]
16-03-02 21:45:26,037 INFO  org.springframework.beans.factory.support.DefaultListableBeanFactory(DefaultListableBeanFactory.java:557) ## Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@51052d: defining beans [sqlSessionFactory,org.mybatis.spring.mapper.MapperScannerConfigurer#0,dataSource,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,jobCrawler,HongLingCapitalConsolePipeline,hongLingCapitalDAO,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; root of factory hierarchy
16-03-02 21:45:27,683 INFO  us.codecraft.webmagic.model.OOSpider(Spider.java:292) ## Spider https://www.my089.com started!
16-03-02 21:45:27,684 INFO  us.codecraft.webmagic.downloader.HttpClientDownloader(HttpClientDownloader.java:77) ## downloading page https://www.my089.com/Loan/default.aspx
16-03-02 21:45:28,140 WARN  us.codecraft.webmagic.downloader.HttpClientDownloader(HttpClientDownloader.java:109) ## download page https://www.my089.com/Loan/default.aspx error
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1591)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:187)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:181)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:975)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:123)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:516)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:454)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:884)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1096)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1123)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1107)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:275)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:254)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:95)
at us.codecraft.webmagic.Spider.processRequest(Spider.java:373)
at us.codecraft.webmagic.Spider$1.run(Spider.java:308)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:285)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:191)
at sun.security.validator.Validator.validate(Validator.java:218)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:126)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:209)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:249)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:954)
... 25 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:280)
... 31 more


取消 提交回答
全部回答(1)
  • 爱吃鱼的程序员
    2020-06-10 13:37:42
    javax.net.ssl.SSLHandshakeException:sun.security.validator.ValidatorException:PKIXpathbuildingfailed:sun.security.provider.certpath.SunCertPathBuilderException:unabletofindvalidcertificationpathtorequestedtarget
    atcom.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
    atcom.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1591)
    atcom.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:187)
    atcom.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:181)
    atcom.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:975)
    atcom.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:123)
    atcom.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:516)
    atcom.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:454)
    atcom.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:884)
    atcom.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1096)
    atcom.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1123)
    atcom.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1107)

    at 


    主要看这段,ssl层没有写好,好好看下example。

    应该是没有设定javax.net.ssl.keyStore,或者路径不对

    该问题已经解决了,我的项目之前是用jdk1.6编译的,只要换成jdk1.7就可以了。

    还是谢谢大家。

    0 0
相关问答

1

回答

网站怎么实现https访问?

2020-06-22 11:38:00 940浏览量 回答数 1

1

回答

网站怎么实现https访问?

2020-06-18 11:14:24 1099浏览量 回答数 1

1

回答

nginx启动http server,监听http请求,代理proxy_pass?400报错

2020-06-08 10:16:04 476浏览量 回答数 1

1

回答

1.java 模拟http post请求报错 2.elasticsearch-?400报错

2020-06-08 11:55:03 810浏览量 回答数 1

2

回答

spring Boot上传excel文件报错。 1、http请求 2、上传类Mu?报错

2020-06-04 21:18:19 714浏览量 回答数 2

1

回答

spring Boot上传excel文件报错。 1、http请求 2、上传类Mu?400报错

2020-06-02 13:43:46 416浏览量 回答数 1

1

回答

nginx+fpm使用curl或者file_get_contents请求http?400报错

2020-05-30 16:57:04 269浏览量 回答数 1

1

回答

nginx+fpm使用curl或者file_get_contents请求http?400报错

2020-05-29 20:20:08 258浏览量 回答数 1

1

回答

Java Web容器是使用一类叫做什么来处理并响应某一个http请求?

2020-04-25 20:20:09 294浏览量 回答数 1

1

回答

HTTP 1.1中好像并没有Patch请求方法?

2020-04-25 14:14:50 343浏览量 回答数 1
+关注
爱吃鱼的程序员
https://developer.aliyun.com/profile/5yerqm5bn5yqg?spm=a2c6h.12873639.0.0.6eae304abcjaIB
2
文章
21564
问答
问答排行榜
最热
最新
相关电子书
更多
低代码开发师(初级)实战教程
立即下载
阿里巴巴DevOps 最佳实践手册
立即下载
冬季实战营第三期:MySQL数据库进阶实战
立即下载