开发者社区> 问答> 正文

我在用webmagic抓取https网站时报错?报错

@黄亿华 你好,想跟你请教个问题:

我在使用你写的webmagic时如果抓去普通http网站是可以的,但是抓取https网站就出错。

16-03-02 21:45:25,487 INFO  org.springframework.context.support.ClassPathXmlApplicationContext(AbstractApplicationContext.java:495) ## Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@fa3ac1: startup date [Wed Mar 02 21:45:25 CST 2016]; root of context hierarchy
16-03-02 21:45:25,582 INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader(XmlBeanDefinitionReader.java:315) ## Loading XML bean definitions from file [C:\Documents and Settings\Administrator\workspace_java\cowpu-crawler\WebRoot\WEB-INF\classes\spring\applicationContext-myBatis.xml]
16-03-02 21:45:25,701 INFO  org.springframework.beans.factory.xml.XmlBeanDefinitionReader(XmlBeanDefinitionReader.java:315) ## Loading XML bean definitions from file [C:\Documents and Settings\Administrator\workspace_java\cowpu-crawler\WebRoot\WEB-INF\classes\spring\applicationContext.xml]
16-03-02 21:45:26,037 INFO  org.springframework.beans.factory.support.DefaultListableBeanFactory(DefaultListableBeanFactory.java:557) ## Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@51052d: defining beans [sqlSessionFactory,org.mybatis.spring.mapper.MapperScannerConfigurer#0,dataSource,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,jobCrawler,HongLingCapitalConsolePipeline,hongLingCapitalDAO,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; root of factory hierarchy
16-03-02 21:45:27,683 INFO  us.codecraft.webmagic.model.OOSpider(Spider.java:292) ## Spider https://www.my089.com started!
16-03-02 21:45:27,684 INFO  us.codecraft.webmagic.downloader.HttpClientDownloader(HttpClientDownloader.java:77) ## downloading page https://www.my089.com/Loan/default.aspx
16-03-02 21:45:28,140 WARN  us.codecraft.webmagic.downloader.HttpClientDownloader(HttpClientDownloader.java:109) ## download page https://www.my089.com/Loan/default.aspx error
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1591)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:187)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:181)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:975)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:123)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:516)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:454)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:884)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1096)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1123)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1107)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:275)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:254)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:95)
at us.codecraft.webmagic.Spider.processRequest(Spider.java:373)
at us.codecraft.webmagic.Spider$1.run(Spider.java:308)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:285)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:191)
at sun.security.validator.Validator.validate(Validator.java:218)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:126)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:209)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:249)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:954)
... 25 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:280)
... 31 more


展开
收起
爱吃鱼的程序员 2020-06-10 13:37:24 1019 0
1 条回答
写回答
取消 提交回答
  • https://developer.aliyun.com/profile/5yerqm5bn5yqg?spm=a2c6h.12873639.0.0.6eae304abcjaIB
    javax.net.ssl.SSLHandshakeException:sun.security.validator.ValidatorException:PKIXpathbuildingfailed:sun.security.provider.certpath.SunCertPathBuilderException:unabletofindvalidcertificationpathtorequestedtarget
    atcom.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
    atcom.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1591)
    atcom.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:187)
    atcom.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:181)
    atcom.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:975)
    atcom.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:123)
    atcom.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:516)
    atcom.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:454)
    atcom.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:884)
    atcom.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1096)
    atcom.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1123)
    atcom.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1107)

    at 


    主要看这段,ssl层没有写好,好好看下example。

    应该是没有设定javax.net.ssl.keyStore,或者路径不对

    该问题已经解决了,我的项目之前是用jdk1.6编译的,只要换成jdk1.7就可以了。

    还是谢谢大家。

    2020-06-10 13:37:42
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
CDN助力企业网站进入HTTPS时代 立即下载
低代码开发师(初级)实战教程 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载