什么是在C＃中parsingHTML的最佳方法？
我正在寻找一个库/方法来parsing一个HTML文件比通用的XMLparsing库更多的HTML特定的function。
Html敏捷包
这是一个敏捷的HTMLparsing器，它构build了一个读/写DOM，并支持普通的XPATH或XSLT（实际上，您不需要理解XPATH或XSLT就可以使用它，不用担心）。 这是一个.NET代码库，可以让你parsing“不在网上”的HTML文件。 parsing器对“真实世界”格式错误的HTML非常宽容。 对象模型与提出System.Xml非常相似，但对于HTML文档（或stream）。
您可以使用TidyNet.Tidy将HTML转换为XHTML，然后使用XMLparsing器。
另一种select是使用内置引擎mshtml：
using mshtml; ... object[] oPageText = { html }; HTMLDocument doc = new HTMLDocumentClass(); IHTMLDocument2 doc2 = (IHTMLDocument2)doc; doc2.write(oPageText); 这允许您使用类似于JavaScript的函数，如getElementById（）
我find了一个名为Fizzler的项目，它采用jQuery / Sizzler方法来selectHTML元素。 它基于HTML敏捷包。 它目前处于testing阶段，只支持CSSselect器的一个子集，但是使用CSSselect器而不是令人讨厌的XPath是非常酷和令人耳目一新的。
你可以做很多事情，而不是疯狂的第三方产品和mshtml（即互操作性）。 使用System.Windows.Forms.WebBrowser。 从那里，您可以在HtmlDocument上执行“GetElementById”或在HtmlElements上执行“GetElementsByTagName”。 如果你想实际上与浏览器交互（例如模拟button点击），你可以使用一点reflection（比Interop小一些）来做到这一点：
var wb = new WebBrowser()
告诉浏览器浏览（与这个问题相关）。 然后在Document_Completed事件上，你可以模拟像这样的点击。
var doc = wb.Browser.Document var elem = doc.GetElementById(elementId); object obj = elem.DomElement; System.Reflection.MethodInfo mi = obj.GetType().GetMethod("click"); mi.Invoke(obj, new object[0]);
你可以做类似的反思的东西来提交表单等
请享用。
我写了一些提供“LINQ to HTML”function的代码。 我以为我会在这里分享。 它基于Majestic 12.它采用Majestic-12的结果并生成LINQ XML元素。 在这一点上，您可以使用所有的LINQ to XML工具来对付HTML。 举个例子：
IEnumerable<XNode> auctionNodes = Majestic12ToXml.Majestic12ToXml.ConvertNodesToXml(byteArrayOfAuctionHtml); foreach (XElement anchorTag in auctionNodes.OfType<XElement>().DescendantsAndSelf("a")) { if (anchorTag.Attribute("href") == null) continue; Console.WriteLine(anchorTag.Attribute("href").Value); }
我想用Majestic-12，因为我知道它有很多关于在野外find的关于HTML的内置知识。 我发现，虽然是将Majestic-12的结果映射到LINQ将接受的东西，因为XML需要额外的工作。 我包括的代码做了很多这种清洗，但是当你使用这个，你会发现被拒绝的页面。 你需要修正代码来解决这个问题。 当引发exception时，请检查exception.Data [“source”]，因为它可能设置为导致exception的HTML标记。 以一种很好的方式处理HTML有时不是微不足道的…
所以，现在的期望是现实的低，这里的代码:)
using System; using System.Collections.Generic; using System.Linq; using System.Text; using Majestic12; using System.IO; using System.Xml.Linq; using System.Diagnostics; using System.Text.RegularExpressions; namespace Majestic12ToXml { public class Majestic12ToXml { static public IEnumerable<XNode> ConvertNodesToXml(byte[] htmlAsBytes) { HTMLparser parser = OpenParser(); parser.Init(htmlAsBytes); XElement currentNode = new XElement("document"); HTMLchunk m12chunk = null; int xmlnsAttributeIndex = 0; string originalHtml = ""; while ((m12chunk = parser.ParseNext()) != null) { try { Debug.Assert(!m12chunk.bHashMode); // popular default for Majestic-12 setting XNode newNode = null; XElement newNodesParent = null; switch (m12chunk.oType) { case HTMLchunkType.OpenTag: // Tags are added as a child to the current tag, // except when the new tag implies the closure of // some number of ancestor tags. newNode = ParseTagNode(m12chunk, originalHtml, ref xmlnsAttributeIndex); if (newNode != null) { currentNode = FindParentOfNewNode(m12chunk, originalHtml, currentNode); newNodesParent = currentNode; newNodesParent.Add(newNode); currentNode = newNode as XElement; } break; case HTMLchunkType.CloseTag: if (m12chunk.bEndClosure) { newNode = ParseTagNode(m12chunk, originalHtml, ref xmlnsAttributeIndex); if (newNode != null) { currentNode = FindParentOfNewNode(m12chunk, originalHtml, currentNode); newNodesParent = currentNode; newNodesParent.Add(newNode); } } else { XElement nodeToClose = currentNode; string m12chunkCleanedTag = CleanupTagName(m12chunk.sTag, originalHtml); while (nodeToClose != null && nodeToClose.Name.LocalName != m12chunkCleanedTag) nodeToClose = nodeToClose.Parent; if (nodeToClose != null) currentNode = nodeToClose.Parent; Debug.Assert(currentNode != null); } break; case HTMLchunkType.Script: newNode = new XElement("script", "REMOVED"); newNodesParent = currentNode; newNodesParent.Add(newNode); break; case HTMLchunkType.Comment: newNodesParent = currentNode; if (m12chunk.sTag == "!--") newNode = new XComment(m12chunk.oHTML); else if (m12chunk.sTag == "![CDATA[") newNode = new XCData(m12chunk.oHTML); else throw new Exception("Unrecognized comment sTag"); newNodesParent.Add(newNode); break; case HTMLchunkType.Text: currentNode.Add(m12chunk.oHTML); break; default: break; } } catch (Exception e) { var wrappedE = new Exception("Error using Majestic12.HTMLChunk, reason: " + e.Message, e); // the original html is copied for tracing/debugging purposes originalHtml = new string(htmlAsBytes.Skip(m12chunk.iChunkOffset) .Take(m12chunk.iChunkLength) .Select(B => (char)B).ToArray()); wrappedE.Data.Add("source", originalHtml); throw wrappedE; } } while (currentNode.Parent != null) currentNode = currentNode.Parent; return currentNode.Nodes(); } static XElement FindParentOfNewNode(Majestic12.HTMLchunk m12chunk, string originalHtml, XElement nextPotentialParent) { string m12chunkCleanedTag = CleanupTagName(m12chunk.sTag, originalHtml); XElement discoveredParent = null; // Get a list of all ancestors List<XElement> ancestors = new List<XElement>(); XElement ancestor = nextPotentialParent; while (ancestor != null) { ancestors.Add(ancestor); ancestor = ancestor.Parent; } // Check if the new tag implies a previous tag was closed. if ("form" == m12chunkCleanedTag) { discoveredParent = ancestors .Where(XE => m12chunkCleanedTag == XE.Name) .Take(1) .Select(XE => XE.Parent) .FirstOrDefault(); } else if ("td" == m12chunkCleanedTag) { discoveredParent = ancestors .TakeWhile(XE => "tr" != XE.Name) .Where(XE => m12chunkCleanedTag == XE.Name) .Take(1) .Select(XE => XE.Parent) .FirstOrDefault(); } else if ("tr" == m12chunkCleanedTag) { discoveredParent = ancestors .TakeWhile(XE => !("table" == XE.Name || "thead" == XE.Name || "tbody" == XE.Name || "tfoot" == XE.Name)) .Where(XE => m12chunkCleanedTag == XE.Name) .Take(1) .Select(XE => XE.Parent) .FirstOrDefault(); } else if ("thead" == m12chunkCleanedTag || "tbody" == m12chunkCleanedTag || "tfoot" == m12chunkCleanedTag) { discoveredParent = ancestors .TakeWhile(XE => "table" != XE.Name) .Where(XE => m12chunkCleanedTag == XE.Name) .Take(1) .Select(XE => XE.Parent) .FirstOrDefault(); } return discoveredParent ?? nextPotentialParent; } static string CleanupTagName(string originalName, string originalHtml) { string tagName = originalName; tagName = tagName.TrimStart(new char[] { '?' }); // for nodes <?xml > if (tagName.Contains(':')) tagName = tagName.Substring(tagName.LastIndexOf(':') + 1); return tagName; } static readonly Regex _startsAsNumeric = new Regex(@"^[0-9]", RegexOptions.Compiled); static bool TryCleanupAttributeName(string originalName, ref int xmlnsIndex, out string result) { result = null; string attributeName = originalName; if (string.IsNullOrEmpty(originalName)) return false; if (_startsAsNumeric.IsMatch(originalName)) return false; // // transform xmlns attributes so they don't actually create any XML namespaces // if (attributeName.ToLower().Equals("xmlns")) { attributeName = "xmlns_" + xmlnsIndex.ToString(); ; xmlnsIndex++; } else { if (attributeName.ToLower().StartsWith("xmlns:")) { attributeName = "xmlns_" + attributeName.Substring("xmlns:".Length); } // // trim trailing \" // attributeName = attributeName.TrimEnd(new char[] { '\"' }); attributeName = attributeName.Replace(":", "_"); } result = attributeName; return true; } static Regex _weirdTag = new Regex(@"^<!\[.*\]>$"); // matches "<![if !supportEmptyParas]>" static Regex _aspnetPrecompiled = new Regex(@"^<%.*%>$"); // matches "<%@ ... %>" static Regex _shortHtmlComment = new Regex(@"^<!-.*->$"); // matches "<!-Extra_Images->" static XElement ParseTagNode(Majestic12.HTMLchunk m12chunk, string originalHtml, ref int xmlnsIndex) { if (string.IsNullOrEmpty(m12chunk.sTag)) { if (m12chunk.sParams.Length > 0 && m12chunk.sParams[0].ToLower().Equals("doctype")) return new XElement("doctype"); if (_weirdTag.IsMatch(originalHtml)) return new XElement("REMOVED_weirdBlockParenthesisTag"); if (_aspnetPrecompiled.IsMatch(originalHtml)) return new XElement("REMOVED_ASPNET_PrecompiledDirective"); if (_shortHtmlComment.IsMatch(originalHtml)) return new XElement("REMOVED_ShortHtmlComment"); // Nodes like "<br <br>" will end up with a m12chunk.sTag==""... We discard these nodes. return null; } string tagName = CleanupTagName(m12chunk.sTag, originalHtml); XElement result = new XElement(tagName); List<XAttribute> attributes = new List<XAttribute>(); for (int i = 0; i < m12chunk.iParams; i++) { if (m12chunk.sParams[i] == "<!--") { // an HTML comment was embedded within a tag. This comment and its contents // will be interpreted as attributes by Majestic-12... skip this attributes for (; i < m12chunk.iParams; i++) { if (m12chunk.sTag == "--" || m12chunk.sTag == "-->") break; } continue; } if (m12chunk.sParams[i] == "?" && string.IsNullOrEmpty(m12chunk.sValues[i])) continue; string attributeName = m12chunk.sParams[i]; if (!TryCleanupAttributeName(attributeName, ref xmlnsIndex, out attributeName)) continue; attributes.Add(new XAttribute(attributeName, m12chunk.sValues[i])); } // If attributes are duplicated with different values, we complain. // If attributes are duplicated with the same value, we remove all but 1. var duplicatedAttributes = attributes.GroupBy(A => A.Name).Where(G => G.Count() > 1); foreach (var duplicatedAttribute in duplicatedAttributes) { if (duplicatedAttribute.GroupBy(DA => DA.Value).Count() > 1) throw new Exception("Attribute value was given different values"); attributes.RemoveAll(A => A.Name == duplicatedAttribute.Key); attributes.Add(duplicatedAttribute.First()); } result.Add(attributes); return result; } static HTMLparser OpenParser() { HTMLparser oP = new HTMLparser(); // The code+comments in this function are from the Majestic-12 sample documentation. // ... // This is optional, but if you want high performance then you may // want to set chunk hash mode to FALSE. This would result in tag params // being added to string arrays in HTMLchunk object called sParams and sValues, with number // of actual params being in iParams. See code below for details. // // When TRUE (and its default) tag params will be added to hashtable HTMLchunk (object).oParams oP.SetChunkHashMode(false); // if you set this to true then original parsed HTML for given chunk will be kept - // this will reduce performance somewhat, but may be desireable in some cases where // reconstruction of HTML may be necessary oP.bKeepRawHTML = false; // if set to true (it is false by default), then entities will be decoded: this is essential // if you want to get strings that contain final representation of the data in HTML, however // you should be aware that if you want to use such strings into output HTML string then you will // need to do Entity encoding or same string may fail later oP.bDecodeEntities = true; // we have option to keep most entities as is - only replace stuff like // this is called Mini Entities mode - it is handy when HTML will need // to be re-created after it was parsed, though in this case really // entities should not be parsed at all oP.bDecodeMiniEntities = true; if (!oP.bDecodeEntities && oP.bDecodeMiniEntities) oP.InitMiniEntities(); // if set to true, then in case of Comments and SCRIPT tags the data set to oHTML will be // extracted BETWEEN those tags, rather than include complete RAW HTML that includes tags too // this only works if auto extraction is enabled oP.bAutoExtractBetweenTagsOnly = true; // if true then comments will be extracted automatically oP.bAutoKeepComments = true; // if true then scripts will be extracted automatically: oP.bAutoKeepScripts = true; // if this option is true then whitespace before start of tag will be compressed to single // space character in string: " ", if false then full whitespace before tag will be returned (slower) // you may only want to set it to false if you want exact whitespace between tags, otherwise it is just // a waste of CPU cycles oP.bCompressWhiteSpaceBeforeTag = true; // if true (default) then tags with attributes marked as CLOSED (/ at the end) will be automatically // forced to be considered as open tags - this is no good for XML parsing, but I keep it for backwards // compatibility for my stuff as it makes it easier to avoid checking for same tag which is both closed // or open oP.bAutoMarkClosedTagsWithParamsAsOpen = false; return oP; } } }
之前已经提到过Html敏捷包 – 如果你要加快速度的话，你可能还想看看Majestic-12的HTMLparsing器 。 它的处理相当笨拙，但它提供了一个非常快的parsing体验。
我认为@Erlend使用HTMLDocument是最好的方法。 不过，我也使用这个简单的图书馆有好运气：
SgmlReader
没有第三方lib，可以在控制台上运行的WebBrowser类解决scheme，以及Asp.net
using System; using System.Collections.Generic; using System.Text; using System.Windows.Forms; using System.Threading; class ParseHTML { public ParseHTML() { } private string ReturnString; public string doParsing(string html) { Thread t = new Thread(TParseMain); t.ApartmentState = ApartmentState.STA; t.Start((object)html); t.Join(); return ReturnString; } private void TParseMain(object html) { WebBrowser wbc = new WebBrowser(); wbc.DocumentText = "feces of a dummy"; //;magic words HtmlDocument doc = wbc.Document.OpenNew(true); doc.Write((string)html); this.ReturnString = doc.Body.InnerHtml + " do here something"; return; } }
用法：
string myhtml = "<HTML><BODY>This is a new HTML document.</BODY></HTML>"; Console.WriteLine("before:" + myhtml); myhtml = (new ParseHTML()).doParsing(myhtml); Console.WriteLine("after:" + myhtml);
parsingHTML的麻烦在于它不是一门精确的科学。 如果是parsing的XHTML，那么事情就会简单得多（正如你所说的，你可以使用一个通用的XMLparsing器）。 因为HTML不一定是格式良好的XML，所以在parsing它的时候会遇到很多问题。 它几乎需要逐个地完成。
我曾经使用ZetaHtmlTidy加载随机网站，然后用xpath（例如/ html / body // p [@ class ='textblock']）对内容的各个部分进行匹配 。 它运作良好，但有一些特殊的网站，它有问题，所以我不知道这是绝对最好的解决scheme。
您可以使用HTML DTD和通用XMLparsing库。
如果您需要查看页面上JS的影响，请使用WatiN [并准备启动浏览器]
根据您的需要，您可能会selectfunction更丰富的库。 我尝试了大部分/所有的解决scheme，但头脑和肩膀突出的是Html Agility Pack。 这是一个非常宽容和灵活的parsing器。
试试这个脚本。
http://www.biterscripting.com/SS_URLs.html
当我使用这个url时，
script SS_URLs.txt URL("http://stackoverflow.com/questions/56107/what-is-the-best-way-to-parse-html-in-c")
它显示了这个线程页面上的所有链接。
http://sstatic.net/so/all.css http://sstatic.net/so/favicon.ico http://sstatic.net/so/apple-touch-icon.png . . .
您可以修改该脚本来检查图像，variables等等。
我写了一些在C＃中parsingHTML标签的类。 如果他们满足您的特殊需求，他们很好，很简单。
您可以阅读关于他们的文章，并在http://www.blackbeltcoder.com/Articles/strings/parsing-html-tags-in-c下载源代码。;
还有一篇关于http://www.blackbeltcoder.com/Articles/strings/a-text-parsing-helper-class的genericsparsing助手类的文章。;
