Apache spark-sql brings the required versions of janino
and commons-compiler
. If you're encountering this error, something else in your pom (or parent pom) is over-riding the version. While you can explicitly set the janino
and commons-compiler
versions in your pom to match what spark brings as suggested in other answers this will make long-term maintenance more difficult because maintainers will need to remember to update these explicit versions each-time you update spark. Instead, I recommend what worked well for me:
Figure out what is bringing in the wrong version of janino by running:
mvn dependency:tree #-Dverbose may be helpful
Exclude janino
and commons-compiler
from the offending dependency. In my case it was an in-house hadoop testing framework:
<dependency>
<groupId>org.my.client.pkg</groupId>
<artifactId>hadoop-testing-framework</artifactId>
<version>${some.version}</version>
<exclusions>
<!-- We want only and exactly Spark's janino version -->
<exclusion>
<groupId>org.codehaus.janino</groupId>
<artifactId>janino</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.janino</groupId>
<artifactId>commons-compiler</artifactId>
</exclusion>
</exclusions>
</dependency>
Re-run mvn dependency:tree
and repeat the above process for any other dependencies which are overriding spark's janino version until your versions of janino
and commons-compiler
are coming from your spark-sql dependency as shown in my (abbreviated) mvn dependency:tree
output below:
[INFO] +- org.apache.spark:spark-sql_2.11:jar:2.4.0.cloudera1:provided
[INFO] | +- org.apache.spark:spark-catalyst_2.11:jar:2.4.0.cloudera1:provided
[INFO] | | +- org.codehaus.janino:janino:jar:3.0.9:compile
[INFO] | | +- org.codehaus.janino:commons-compiler:jar:3.0.9:compile
Note, if you see something like:
[INFO] +- org.apache.spark:spark-sql_2.11:jar:2.4.0.cloudera1:provided
[INFO] | +- org.apache.spark:spark-catalyst_2.11:jar:2.4.0.cloudera1:provided
[INFO] | | +- org.codehaus.janino:janino:jar:2.6.1:compile <-- Note old version
[INFO] | | +- org.codehaus.janino:commons-compiler:jar:3.0.9:compile
then someone else is still over-riding spark's janino
version. In my case, the parent pom was explicitly bringing in v2.6.1. Removing that dependency block from the parent pom solved my problem. This is where the -Dverbose
flag may help.
Final note, at least my version of spark could not tolerate any change in the janino
or commons-compiler
versions. It had to be exactly what spark brought with it down to the patch (assuming codehaus follows semver).