@黄亿华 你好,想跟你请教个问题:
我在使用你写的webmagic时如果抓去普通http网站是可以的,但是抓取https网站就出错。
16-03-02 21:45:25,487 INFO org.springframework.context.support.ClassPathXmlApplicationContext(AbstractApplicationContext.java:495) ## Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@fa3ac1: startup date [Wed Mar 02 21:45:25 CST 2016]; root of context hierarchy
16-03-02 21:45:25,582 INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader(XmlBeanDefinitionReader.java:315) ## Loading XML bean definitions from file [C:\Documents and Settings\Administrator\workspace_java\cowpu-crawler\WebRoot\WEB-INF\classes\spring\applicationContext-myBatis.xml]
16-03-02 21:45:25,701 INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader(XmlBeanDefinitionReader.java:315) ## Loading XML bean definitions from file [C:\Documents and Settings\Administrator\workspace_java\cowpu-crawler\WebRoot\WEB-INF\classes\spring\applicationContext.xml]
16-03-02 21:45:26,037 INFO org.springframework.beans.factory.support.DefaultListableBeanFactory(DefaultListableBeanFactory.java:557) ## Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@51052d: defining beans [sqlSessionFactory,org.mybatis.spring.mapper.MapperScannerConfigurer#0,dataSource,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,jobCrawler,HongLingCapitalConsolePipeline,hongLingCapitalDAO,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; root of factory hierarchy
16-03-02 21:45:27,683 INFO us.codecraft.webmagic.model.OOSpider(Spider.java:292) ## Spider https://www.my089.com started!
16-03-02 21:45:27,684 INFO us.codecraft.webmagic.downloader.HttpClientDownloader(HttpClientDownloader.java:77) ## downloading page https://www.my089.com/Loan/default.aspx
16-03-02 21:45:28,140 WARN us.codecraft.webmagic.downloader.HttpClientDownloader(HttpClientDownloader.java:109) ## download page https://www.my089.com/Loan/default.aspx error
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1591)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:187)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:181)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:975)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:123)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:516)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:454)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:884)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1096)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1123)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1107)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:275)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:254)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:95)
at us.codecraft.webmagic.Spider.processRequest(Spider.java:373)
at us.codecraft.webmagic.Spider$1.run(Spider.java:308)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:285)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:191)
at sun.security.validator.Validator.validate(Validator.java:218)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:126)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:209)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:249)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:954)
... 25 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:280)
... 31 more
at
主要看这段,ssl层没有写好,好好看下example。
应该是没有设定javax.net.ssl.keyStore,或者路径不对该问题已经解决了,我的项目之前是用jdk1.6编译的,只要换成jdk1.7就可以了。
还是谢谢大家。
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。