我正在尝试附加处理程序以使用此代码激发 UI:
private def getSparkUI(sparkContext: SparkContext): SparkUI = {
sparkContext.ui.getOrElse {
throw new SparkException("Parent SparkUI to attach this tab to not found!")
}
import org.apache.spark.ui.JettyUtils.createServletHandler
val ui = getSparkUI(sparkContext)
val graphHttpServlet = new GraphHttpServlet("/graph/", sessionUtils)
val handler = createServletHandler("/graph/", graphHttpServlet, "")
ui.attachHandler(handler)
此代码因错误而无法编译:
Error:(20, 19) Symbol 'type org.eclipse.jetty.servlet.ServletContextHandler' is missing from the classpath.
This symbol is required by 'method org.apache.spark.ui.JettyUtils.createServletHandler'.
Make sure that type ServletContextHandler is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'JettyUtils.class' was compiled against an incompatible version of org.eclipse.jetty.servlet.
val handler = createServletHandler("/graph/", graphHttpServlet, "")
我知道 spark 将 jetty 依赖项隐藏到一个新包中:org.spark_project.jetty.servlet.ServletContextHandler
我还反编译了 jar 并检查 jar 是否具有阴影依赖项。
但是为什么编译器看不到呢?我如何附加处理程序?
最佳答案
此外,遇到类似的问题并在几次实验后弄明白了。为 org.eclipse.jetty
库添加阴影规则解决了这个问题。
这解决了以下异常:
java.lang.NoSuchMethodError: 'void org.apache.spark.ui.WebUI.attachHandler(org.eclipse.jetty.servlet.ServletContextHandler)'
java.lang.ClassCastException:类 org.sparkproject.jetty.servlet.ServletContextHandler 无法转换为类 org.eclipse.jetty.servlet.ServletContextHandler(org.sparkproject.jetty.servlet.ServletContextHandler 和org.eclipse.jetty.servlet.ServletContextHandler 位于加载程序“app”的未命名模块中)
sbt 规则
assemblyShadeRules in assembly := Seq(
ShadeRule
.rename("org.eclipse.jetty.**" -> "org.sparkproject.jetty.@1")
.inAll
)
Maven 规则
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<shadedArtifactAttached>false</shadedArtifactAttached>
<artifactSet>
<includes>
<include>org.eclipse.jetty:jetty-io</include>
<include>org.eclipse.jetty:jetty-http</include>
<include>org.eclipse.jetty:jetty-proxy</include>
<include>org.eclipse.jetty:jetty-client</include>
<include>org.eclipse.jetty:jetty-continuation</include>
<include>org.eclipse.jetty:jetty-servlet</include>
<include>org.eclipse.jetty:jetty-servlets</include>
<include>org.eclipse.jetty:jetty-plus</include>
<include>org.eclipse.jetty:jetty-security</include>
<include>org.eclipse.jetty:jetty-util</include>
<include>org.eclipse.jetty:jetty-server</include>
</includes>
</artifactSet>
<relocations>
<relocation>
<pattern>org.eclipse.jetty</pattern>
<shadedPattern>org.sparkproject.jetty</shadedPattern>
<includes>
<include>org.eclipse.jetty.**</include>
</includes>
</relocation>
</relocations>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
https://stackoverflow.com/questions/56681057/