使用Scalaparsing器组合器来parsingCSV文件

我正在尝试使用Scala分析器组合器编写一个CSV分析器。 语法基于RFC4180 。 我想出了以下代码。 它几乎可以工作，但我无法正确分离不同的logging。 我错过了什么？

object CSV extends RegexParsers { def COMMA = "," def DQUOTE = "\"" def DQUOTE2 = "\"\"" ^^ { case _ => "\"" } def CR = "\r" def LF = "\n" def CRLF = "\r\n" def TXT = "[^\",\r\n]".r def file: Parser[List[List[String]]] = ((record~((CRLF~>record)*))<~(CRLF?)) ^^ { case r~rs => r::rs } def record: Parser[List[String]] = (field~((COMMA~>field)*)) ^^ { case f~fs => f::fs } def field: Parser[String] = escaped|nonescaped def escaped: Parser[String] = (DQUOTE~>((TXT|COMMA|CR|LF|DQUOTE2)*)<~DQUOTE) ^^ { case ls => ls.mkString("")} def nonescaped: Parser[String] = (TXT*) ^^ { case ls => ls.mkString("") } def parse(s: String) = parseAll(file, s) match { case Success(res, _) => res case _ => List[List[String]]() } } println(CSV.parse(""" "foo", "bar", 123""" + "\r\n" + "hello, world, 456" + "\r\n" + """ spam, 789, egg""")) // Output: List(List(foo, bar, 123hello, world, 456spam, 789, egg)) // Expected: List(List(foo, bar, 123), List(hello, world, 456), List(spam, 789, egg)) 

更新：问题解决了

默认的RegexParsers使用正则expression式[\s]+忽略包括空格，制表符，回车和换行符在内的空格。 上面的parsing器无法分离logging的问题是由于这一点。 我们需要禁用skipWhitespace模式。 将whiteSpace定义replace为[ \t]}并不能解决问题，因为它会忽略字段中的所有空格（因此CSV中的“foo bar”变成“foobar”），这是不希望的。 parsing器的更新源是这样的

 import scala.util.parsing.combinator._ // A CSV parser based on RFC4180 // http://tools.ietf.org/html/rfc4180 object CSV extends RegexParsers { override val skipWhitespace = false // meaningful spaces in CSV def COMMA = "," def DQUOTE = "\"" def DQUOTE2 = "\"\"" ^^ { case _ => "\"" } // combine 2 dquotes into 1 def CRLF = "\r\n" | "\n" def TXT = "[^\",\r\n]".r def SPACES = "[ \t]+".r def file: Parser[List[List[String]]] = repsep(record, CRLF) <~ (CRLF?) def record: Parser[List[String]] = repsep(field, COMMA) def field: Parser[String] = escaped|nonescaped def escaped: Parser[String] = { ((SPACES?)~>DQUOTE~>((TXT|COMMA|CRLF|DQUOTE2)*)<~DQUOTE<~(SPACES?)) ^^ { case ls => ls.mkString("") } } def nonescaped: Parser[String] = (TXT*) ^^ { case ls => ls.mkString("") } def parse(s: String) = parseAll(file, s) match { case Success(res, _) => res case e => throw new Exception(e.toString) } } 

Answer 1

你错过的是空格。 我投入了一些红利的改进。

 import scala.util.parsing.combinator._ object CSV extends RegexParsers { override protected val whiteSpace = """[ \t]""".r def COMMA = "," def DQUOTE = "\"" def DQUOTE2 = "\"\"" ^^ { case _ => "\"" } def CR = "\r" def LF = "\n" def CRLF = "\r\n" def TXT = "[^\",\r\n]".r def file: Parser[List[List[String]]] = repsep(record, CRLF) <~ opt(CRLF) def record: Parser[List[String]] = rep1sep(field, COMMA) def field: Parser[String] = (escaped|nonescaped) def escaped: Parser[String] = (DQUOTE~>((TXT|COMMA|CR|LF|DQUOTE2)*)<~DQUOTE) ^^ { case ls => ls.mkString("")} def nonescaped: Parser[String] = (TXT*) ^^ { case ls => ls.mkString("") } def parse(s: String) = parseAll(file, s) match { case Success(res, _) => res case _ => List[List[String]]() } } 

Answer 2

Scalaparsing器组合库从2.11开始的Scala标准库之外，没有理由不使用性能更高的Parboiled2库。 以下是Parboiled2的DSL中的CSVparsing器的一个版本：

 /* based on comments in https://github.com/sirthias/parboiled2/issues/61 */ import org.parboiled2._ case class Parboiled2CsvParser(input: ParserInput, delimeter: String) extends Parser { def DQUOTE = '"' def DELIMITER_TOKEN = rule(capture(delimeter)) def DQUOTE2 = rule("\"\"" ~ push("\"")) def CRLF = rule(capture("\r\n" | "\n")) def NON_CAPTURING_CRLF = rule("\r\n" | "\n") val delims = s"$delimeter\r\n" + DQUOTE def TXT = rule(capture(!anyOf(delims) ~ ANY)) val WHITESPACE = CharPredicate(" \t") def SPACES: Rule0 = rule(oneOrMore(WHITESPACE)) def escaped = rule(optional(SPACES) ~ DQUOTE ~ (zeroOrMore(DELIMITER_TOKEN | TXT | CRLF | DQUOTE2) ~ DQUOTE ~ optional(SPACES)) ~> (_.mkString(""))) def nonEscaped = rule(zeroOrMore(TXT | capture(DQUOTE)) ~> (_.mkString(""))) def field = rule(escaped | nonEscaped) def row: Rule1[Seq[String]] = rule(oneOrMore(field).separatedBy(delimeter)) def file = rule(zeroOrMore(row).separatedBy(NON_CAPTURING_CRLF)) def parsed() : Try[Seq[Seq[String]]] = file.run() } 

Answer 3

RegexParsersparsing器的默认空白是\s+ ，其中包含新行。 所以CR ， LF和CRLF从来没有得到处理的机会，因为parsing器会自动跳过它。