Skip to content

Latest commit

 

History

History

spark

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
src/main/scala/io/sqlbenchmarks/sqlbench
src/main/scala/io/sqlbenchmarks/sqlbench
 
 
.dockerignore
.dockerignore
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 
pom.xml
pom.xml
 
 

README.md

Spark SQLBench-H Benchmarks

Pre-requisites

Untar these downloads and set MAVEN_HOME and SPARK_HOME environment variables to point to the install location.

Build the benchmark JAR file

$MAVEN_HOME/bin/mvn package

Standalone Mode

Start a local Spark cluster in standalone mode

$SPARK_HOME/sbin/start-master.sh
$SPARK_HOME/sbin/start-worker.sh spark://ripper:7077

Run Single Query

$SPARK_HOME/bin/spark-submit --master spark://ripper:7077 \
    --class io.sqlbenchmarks.sqlbench.Main \
    --conf spark.driver.memory=8G \
    --conf spark.executor.memory=32G \
    --conf spark.executor.cores=24 \
    --conf spark.cores.max=24 \
    target/sqlbench-spark-0.1.0-SNAPSHOT-jar-with-dependencies.jar \
    --input-path /mnt/bigdata/tpch/sf10-parquet/ \
    --output-path . \
    --query-path ~/git/sql-benchmarks/sqlbench-h/queries/sf\=10/ \
    --query 1

Run All Queries

$SPARK_HOME/bin/spark-submit --master spark://ripper:7077 \
    --class io.sqlbenchmarks.sqlbench.Main \
    --conf spark.driver.memory=8G \
    --conf spark.executor.memory=32G \
    --conf spark.executor.cores=24 \
    --conf spark.cores.max=24 \
    target/sqlbench-spark-0.1.0-SNAPSHOT-jar-with-dependencies.jar \
    --input-path /mnt/bigdata/tpch/sf10-parquet/ \
    --output-path . \
    --query-path ~/git/sql-benchmarks/sqlbench-h/queries/sf\=10/ \
    --num-queries 22

Monitor progress via the Spark UI at http://localhost:8080

Shut down the cluster

$SPARK_HOME/sbin/stop-slave.sh
$SPARK_HOME/sbin/stop-master.sh