Deploying

• Installation
  • Direct Download
  • As a Dependency
  • Optional Components
• Configuration
  • Web Caching
  • Query Optimization
    • Query Optimizer Options
  • Failover
  • Web Client
  • Miscellaneous
• Scaling
• More Information
  • 1-Line installation with Ansible

Installation

Why so complex? All you need is is a single library in your Java/Scala classpath:

groupId: com.tribbloids.spookystuff
artifactId: spookystuff-assembly_2.10
version: 0.3.2

You have 3 options: download it, build it or let a dependency manager (Apache Maven, sbt, Gradle etc.) do it for you.

Direct Download

This pre-built JAR/bundle provide full functionality out-of-the-box, however you need a Apache Spark installation first (including integrated Spark environment, e.g. Notebooks in databricks™ Cloud or Apache Zeppelin). If you haven’t done so, please refer to Apache Spark installation Guide or Integration Section.

As a Dependency

if you want to use SpookyStuff as a library in your source code, the easiest way is to let your dependency manager (e.g. Apache Maven, sbt, gradle) to download it automatically from the Maven Central Repository, by adding the following artifact reference into your build definition:

<dependency>
    <groupId>com.tribbloids.spookystuff</groupId>
    <artifactId>spookystuff-core_2.10</artifactId>
    <version>0.3.2</version>
</dependency>

libraryDependencies += "com.tribbloids.spookystuff" % "spookystuff-core_2.10" % "0.3.2"

'com.tribbloids.spookystuff:spookystuff-core_2.10:0.3.2'

[com.tribbloids.spookystuff/spookystuff-core_2.10 "0.3.2"]

Many integrated Spark environments (e.g. Spark-Shell, databricks™ Cloud or Apache Zeppelin) has built-in dependency manager, which makes deployment much easier by eliminating the necessity of manual download. This is again covered in Integration Section.

Optional Components

SpookyStuff natively supports 2 headless browers for deep web access: HtmlUnit and the much more capable PhantomJS. The binary executable of PhantomJS is NOT included in the pre-built JAR, and if it’s’ not detected on all Spark workers, it will be automatically downloaded from a PhantomJS mirror. To avoid repeated download you can download a permanent copy to each Spark worker:

• on linux https://s3-us-west-1.amazonaws.com/spooky-bin/phantomjs-linux/phantomjs

And tell SpookyStuff to use them by setting its environment variable PHANTOMJS_PATH or system property phantomjs.binary.path to its full installation path (you don’t need to set it on Spark workers). Next time SpookyStuff will use them instead of download. It should be noted that all PhantomJS paths on workers should be identical, otherwise it is impossible to set SpookyStuff to use all of them.

Another optional component for better deep web compatibility is TOR client, it can be used by SpookyStuff as a socks5 proxy (both http and socks proxies are supported by SpookyStuff). Installation of TOR client is OS-specific, users are advised to follow this instruction to install and launch TOR client on their Spark workers.

Configuration

The following options can be set independently for each SpookyContext (by changing its conf.{option-name} property in your program), or set collectively by environment variables and/or Spark/system properties. If multiple values for an option are set by different methods (e.g. environment variable SPOOKY_CACHE and Spark property spooky.cache are set differently), The preceding value in the following list will override the others.

Web Caching

Query Optimization

Query Optimizer Options

• Wide_RDDWebCache is an experimental feature that should be used tentatively, RDDs are designed to be immutable so using this optimizer to collect big dataset may easily results in memory overflow. In addition, some in-memory file systems, e.g. Tachyon and Apache Ignite, may already achieved comparable speed which renders RDD web cache non-competitive.

Failover

Web Client

Miscellaneous

Scaling

SpookyStuff is optimized for running on Spark cluster mode, which accelerates execution by parallelizing over multiple machine’s processing power and network bandwidth, in most cases this is highly recommended in production, and the only feasible way for running query & valorization on big dataset. However, it is important to understand the following fact and ensure that your query’s execution is in compliance with your web service providers, fail to understand the consequence may results in your API key being banned or yourself being prosecuted.

• Despite being able to scale up to hundreds of nodes, SpookyStuff can only approximate linear speed gain (speed proportional to parallelism) if there is no other bottleneck, namely, your concurrent access should be smoothly handled by the web services being queried (e.g. brokered by a CDN or load balancer) and your cluster’s network topology. Otherwise blindly increasing the size of your cluster will only yield diminishing return.

• Your API credential will be shared by multiple IP addresses of your cluster for all API calls, this may max out web server’s connection pool and cause heavy load on its infrastructure, make sure that this is not frown upon by your API provider!

• Web cache and checkpointing directory must be on a distributed file system (HDFS, S3 etc.), as well as other options under SpookyConf.dirs, otherwise neither SpookyStuff nor yourself can access them reliably.

In addition: We also recommend using the following Spark properties for better performance:

• spark.task.maxFailures=100 (or any sufficiently high number): external web services are less stable than in-house web services, so make sure SpookyStuff can retry many times from multiple machines to overcome service downtime and connection error. It should be noted that retry is partition-wise, so make sure your web cache is enabled to avoid repeated remote access.

• spark.serializer=org.apache.spark.serializer.KryoSerializer: Shuffling and broadcasting over mutiple machines are much more expensive so try to enable the more efficient Kryo serializer if you haven’t done so.

• spark.kryoserializer.buffer.max=512m (or a size enough to handle your largest partition): The default value of 64m may be unable to handle large files, increase it if you ran into KryoException.

• spark.kryo.registrator=com.tribbloids.spookystuff.SpookyRegistrator: Not necessary but can be helpful in reducing Serialization size.

More Information

Like other Spark applications, SpookyStuff can benefit from many other option Spark offered, please refer to Spark configuration and Spark Tuning Guide for more information.

1-Line installation with Ansible

The following section propose a fully automated install routine using Ansible, its only tested on Debian-like Linux OS. Please ignore this section if your Spark environment is not using Debian, Ubuntu or their variants.