Spark:如何从Spark壳运行Spark文件
我正在使用CDH 5.2。 我能够使用spark-shell来运行命令。 我如何运行包含spark命令的文件(file.spark)。 其次,有没有办法在没有sbt的情况下运行/编译CDH 5.2中的scala程序?
提前致谢,
从spark-shell加载外部文件只需做
:load PATH_TO_FILE
这将调用您的文件中的所有内容。
我没有SBT问题的解决scheme,虽然对不起:-)
在命令行中,您可以使用
spark-shell -i file.scala
运行写在file.scala
代码
你可以使用sbt或者maven编译spark程序。 只需将spark作为依赖添加到maven
<repository> <id>Spark repository</id> <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url> </repository>
然后依赖:
<dependency> <groupId>spark</groupId> <artifactId>spark</artifactId> <version>1.2.0</version> </dependency>
就用spark命令运行一个文件而言:你可以简单地这样做:
echo" import org.apache.spark.sql.* ssc = new SQLContext(sc) ssc.sql("select * from mytable").collect " > spark.input
现在运行命令脚本:
cat spark.input | spark-shell
只是为了给予更多的观点的答案
火星壳是一个斯卡拉击退
你可以input:help来查看scala shell中可能的操作列表
scala> :help All commands can be abbreviated, eg, :he instead of :help. :edit <id>|<line> edit history :help [command] print this summary or command-specific help :history [num] show the history (optional num is commands to show) :h? <string> search the history :imports [name name ...] show import history, identifying sources of names :implicits [-v] show the implicits in scope :javap <path|class> disassemble a file or class name :line <id>|<line> place line(s) at the end of history :load <path> interpret lines in a file :paste [-raw] [path] enter paste mode or paste a file :power enable power user mode :quit exit the interpreter :replay [options] reset the repl and replay all previous commands :require <path> add a jar to the classpath :reset [options] reset the repl to its initial state, forgetting all session entries :save <path> save replayable session to a file :sh <command line> run a shell command (result is implicitly => List[String]) :settings <options> update compiler options, if possible; see reset :silent disable/enable automatic printing of results :type [-v] <expr> display the type of an expression without evaluating it :kind [-v] <expr> display the kind of expression's type :warnings show the suppressed warnings from the most recent line which had any
:在文件中加载解释行