Is there a way to install apache drill on an EMR cluster on AWS that is already Running

Question

Please Note I have a running EMR cluster that is persistent. I do not wish for a tear down.

Answer 1

Try using Ruby Script. There one for AWS:

You should find one on Github. Make sure there is Ruby installed first. Also, I don't see a reason why it shouldn’t work on an existing cluster. EMR runs bootstrap scripts after all the selected EMR components are installed, Hadoop user created, and config settings applied. Of course, you'd test this on a new cluster, so you don't break your persistent one.

Make sure you use SCP or S3 because your script would run on every node. Also make sure you pick up the latest version to stay updated and installation to happen smoothly: https://drill.apache.org/download/

EMR clusters are bound to tear down. To ease this process, we have bootstrap scripts & cluster configuration. You may want to boot up, run job, tear down. This would be more affordable.

Answer 2

Assuming you have SSM installed you can use ssm.send_command (send the command to all nodes) to call an S3 script which installs and starts drill on a running EMR cluster.

There is an example of this here: https://gist.github.com/fmcmac/a35738376d111fdca45057bd0fb4c79e

Answer 3

It looks to be trying to get apache-drill-1.0.0.tar.gz from getdrill.org/drill/download which should currently be v1.12.0 from the newer site https://drill.apache.org/download/ so at least that part needs to change. It also installs Zookeeper to run Drill, so check if you are already running Zookeeper as Drill will need to install as a new application in there if zookeeper is already there, you might experience issues if you try installing over the top, version compatibility is something to double check.