使用 XPATH 和 HTML Cleaner 解析 HTML/XML

本文涉及的产品
全局流量管理 GTM,标准版 1个月
公共DNS(含HTTPDNS解析),每月1000万次HTTP解析
云解析 DNS,旗舰版 1个月
简介: 使用 XPATH 和 HTML Cleaner 解析 HTML/XML(Using XPATH and HTML Cleaner to parse HTML / XML)太阳火神的美丽人生 (http://blog.csdn.net/opengl_es)本文遵循“署名-非商业用途-保持一致”创作公用协议转载请保留此句:太阳火神的美丽人生 -  本博客专注于 敏捷开发及移动和物联设备研究:iOS、Android、Html5、Arduino、pcDuino,否则,出自本博客的文章拒绝转载或再转载,谢谢合作。

使用 XPATH 和 HTML Cleaner 解析 HTML/XML
(Using XPATH and HTML Cleaner to parse HTML / XML)

太阳火神的美丽人生 (http://blog.csdn.net/opengl_es)

本文遵循“署名-非商业用途-保持一致”创作公用协议

转载请保留此句:太阳火神的美丽人生 -  本博客专注于 敏捷开发及移动和物联设备研究:iOS、Android、Html5、Arduino、pcDuino否则,出自本博客的文章拒绝转载或再转载,谢谢合作。



使用 XPATH 和 HTML Cleaner 解析 HTML/XML
(Using XPATH and HTML Cleaner to parse HTML / XML)

JANUARY 5, 2010
tags:   android,   examples,   HTML,   parse,   scraping,   XML,   XPATH

大家好
Hey everyone,

有时我发现有一种能力十分有用,尤其在 Web 相关的应用中,那就是从 web 站点获取 HTML 并且从 HTML 解析数据,或是任何你要想得到的内容(对于我的情况大多总是数据)。
So something that I’ve found to be extremely useful (especially in web related applications) is the ability to retrieve HTML from websites and parse their HTML for data or whatever you may be looking for (in my case it is almost always data).


I actually use this technique to do the real time stock/option imports for my Black-Scholes/Implied Volatility applications, so if you’re looking for an example on how to retrieve and parse HTML and run “queries” over it using, say, XPATH, then this post is for you.

Now, before we begin, in order to do this you will have to reference an external JAR in your project’s build path. The JAR that I use comes from HtmlCleaner which even gives you an example of how they use it here HtmlCleaner Example, but in addition to that I’ll show you an example of how I use it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
public class OptionScraper {
 
     // EXAMPLE XPATH QUERIES IN THE FORM OF STRINGS - WILL BE USED LATER
     private static final String NAME_XPATH = "//div[@class='yfi_quote']/div[@class='hd']/h2" ;
 
     private static final String TIME_XPATH = "//table[@id='time_table']/tbody/tr/td[@class='yfnc_tabledata1']" ;
 
     private static final String PRICE_XPATH = "//table[@id='price_table']//tr//span" ;
 
     // TAGNODE OBJECT, ITS USE WILL COME IN LATER
     private static TagNode node;
 
     // A METHOD THAT HELPS ME RETRIEVE THE STOCK OPTION'S DATA BASED OFF THE NAME (I.E. GOUAA IS ONE OF GOOGLE'S STOCK OPTIONS)
     public static Option getOptionFromName(String name) throws XPatherException, ParserConfigurationException,SAXException, IOException, XPatherException {
 
         // THE URL WHOSE HTML I WANT TO RETRIEVE AND PARSE
         String option_url = "http://finance.yahoo.com/q?s=" + name.toUpperCase();
 
         // THIS IS WHERE THE HTMLCLEANER COMES IN, I INITIALIZE IT HERE
         HtmlCleaner cleaner = new HtmlCleaner();
         CleanerProperties props = cleaner.getProperties();
         props.setAllowHtmlInsideAttributes( true );
         props.setAllowMultiWordAttributes( true );
         props.setRecognizeUnicodeChars( true );
         props.setOmitComments( true );
 
         // OPEN A CONNECTION TO THE DESIRED URL
         URL url = new URL(option_url);
         URLConnection conn = url.openConnection();
 
         //USE THE CLEANER TO "CLEAN" THE HTML AND RETURN IT AS A TAGNODE OBJECT
         node = cleaner.clean( new InputStreamReader(conn.getInputStream()));
 
         // ONCE THE HTML IS CLEANED, THEN YOU CAN RUN YOUR XPATH EXPRESSIONS ON THE NODE, WHICH WILL THEN RETURN AN ARRAY OF TAGNODE OBJECTS (THESE ARE RETURNED AS OBJECTS BUT GET CASTED BELOW)
         Object[] info_nodes = node.evaluateXPath(NAME_XPATH);
         Object[] time_nodes = node.evaluateXPath(TIME_XPATH);
         Object[] price_nodes = node.evaluateXPath(PRICE_XPATH);
 
         // HERE I JUST DO A SIMPLE CHECK TO MAKE SURE THAT MY XPATH WAS CORRECT AND THAT AN ACTUAL NODE(S) WAS RETURNED
         if (info_nodes.length > 0 ) {
             // CASTED TO A TAGNODE
             TagNode info_node = (TagNode) info_nodes[ 0 ];
             // HOW TO RETRIEVE THE CONTENTS AS A STRING
             String info = info_node.getChildren().iterator().next().toString().trim();
 
             // SOME METHOD THAT PROCESSES THE STRING OF INFORMATION (IN MY CASE, THIS WAS THE STOCK QUOTE, ETC)
             processInfoNode(o, info);
         }
 
         if (time_nodes.length > 0 ) {
             TagNode time_node = (TagNode) time_nodes[ 0 ];
             String date = time_node.getChildren().iterator().next().toString().trim();
 
             // DATE RETURNED IN 15-JAN-10 FORMAT, SO THIS IS SOME METHOD I WROTE TO JUST PARSE THAT STRING INTO THE FORMAT THAT I USE
             processDateNode(o, date);
         }
 
         if (price_nodes.length > 0 ) {
             TagNode price_node = (TagNode) price_nodes[ 0 ];
             double price = Double.parseDouble(price_node.getChildren().iterator().next().toString().trim());
             o.setPremium(price);
         }
 
         return o;
     }
}

So that’s it! Once you include the JAR in your build path, everything else is pretty easy! It’s a great tool to use. However, it does require knowledge of XPATH but XPATH isn’t too hard to pick up and is useful to know so if you don’t know it then take a look at the link.

Now, a warning to everyone. It’s documented that the XPATH expressions recognized by HtmlCleaner is not complete in the sense that only “basic” XPATH is recognized. What’s excluded? For instance, you can’t use any of the “axes” operators (i.e. parent, ancestor, following, following-sibling, etc), but in my experience everything else is fair game. Yes, it sucks, and many times it can make your life a little bit harder, but usually it just requires you to be a tad more clever with your XPATH expressions before you can pull the desired information.

And of course, this technique works for XML documents as well!

Hope this was helpful to everyone. Let me know if you’re confused anywhere.

- jwei



目录
相关文章
|
1月前
|
XML Web App开发 JavaScript
XML DOM 解析器
XML DOM 解析器
|
25天前
|
XML JavaScript 前端开发
如何解析一个 HTML 文本
【10月更文挑战第23天】在实际应用中,根据具体的需求和场景,我们可以灵活选择解析方法,并结合其他相关技术来实现高效、准确的 HTML 解析。随着网页技术的不断发展,解析 HTML 文本的方法也在不断更新和完善,
|
25天前
|
JavaScript API 开发工具
<大厂实战场景> ~ Flutter&鸿蒙next 解析后端返回的 HTML 数据详解
本文介绍了如何在 Flutter 中解析后端返回的 HTML 数据。首先解释了 HTML 解析的概念,然后详细介绍了使用 `http` 和 `html` 库的步骤,包括添加依赖、获取 HTML 数据、解析 HTML 内容和在 Flutter UI 中显示解析结果。通过具体的代码示例,展示了如何从 URL 获取 HTML 并提取特定信息,如链接列表。希望本文能帮助你在 Flutter 应用中更好地处理 HTML 数据。
103 1
|
1月前
|
XML 数据格式
HTML 实例解析
本文介绍了HTML中常见元素的使用方法,包括`<p>`、`<body>`和`<html>`等。详细解析了这些元素的结构和作用,并强调了正确使用结束标签的重要性。此外,还提到了空元素的使用及大小写标签的规范。
|
1月前
|
XML 前端开发 数据格式
Beautiful Soup 解析html | python小知识
在数据驱动的时代,网页数据是非常宝贵的资源。很多时候我们需要从网页上提取数据,进行分析和处理。Beautiful Soup 是一个非常流行的 Python 库,可以帮助我们轻松地解析和提取网页中的数据。本文将详细介绍 Beautiful Soup 的基础知识和常用操作,帮助初学者快速入门和精通这一强大的工具。【10月更文挑战第11天】
60 2
|
1月前
|
XML Web App开发 JavaScript
XML DOM 解析器
XML DOM 解析器
|
1月前
|
XML Web App开发 JavaScript
XML DOM 解析器
XML DOM 解析器
|
1月前
|
XML Web App开发 JavaScript
XML DOM 解析器
XML DOM 解析器
|
1月前
|
XML Web App开发 JavaScript
XML DOM 解析器
XML DOM 解析器
|
1月前
|
XML Web App开发 JavaScript
XML DOM 解析器
XML DOM 解析器
下一篇
无影云桌面