CLI tool to run a batch jobs with ETL/ExTL framework on AWS or other cloud services.
This page describes how to get started with
hotsub
command.
You need 2 softwares installed on your environment
and 2 permissions on AWS account
You can skip this step if following command already works well on your computer.
% docker-machine version
Otherwise, you need to install docker-machine
.
You can skip this step if following command already works well on your computer.
% aws configure list
Otherwise, you need to install awscli
and configure it with your credentials.
hotsub
is currently stable on MacOS and Linux environment. Windows support is coming soon.
Choose binary for your OS from this release page and, if you want, locate extracted executable file to your PATH
.
You can generate Hello, World
project by using template
command.
% hotsub template
Successfully created template project
at helloworld_201810041451
Go to the directory and check README.md
% cd helloworld_201810041451
% ls
README.md hello.csv hello.sh
These are all you need to run Hello, World
% hotsub run \
--script hello.sh \
--tasks hello.csv \
--verbose
Then you will get following output
2018/10/04 14:53:46 [COMMAND] Your tasks file is parsed and decoded to 2 job(s) ✅
2018/10/04 14:53:46 [COMMAND] See logs here -> /Users/otiai10/proj/hotsub/github.io/helloworld_201810041451/log/20181004_145346
[hello.csv 0] [CREATE] Creating computing instance for this job...
[hello.csv 1] [CREATE] Creating computing instance for this job...
[hello.csv 1] [CONSTRUCT] Constructing containers for this job...
[hello.csv 1] [CONSTRUCT] Constructing workflow container inside the computing instance...
[hello.csv 1] [CONSTRUCT] Constructing routine container inside the computing instance...
[hello.csv 0] [CONSTRUCT] Constructing containers for this job...
[hello.csv 0] [CONSTRUCT] Constructing workflow container inside the computing instance...
[hello.csv 0] [CONSTRUCT] Constructing routine container inside the computing instance...
[hello.csv 0] [EXECUTE] &1> Hello! My name is Tom.
[hello.csv 0] [EXECUTE] &1> I'm a cat.
[hello.csv 1] [EXECUTE] &1> Hello! My name is Jerry.
[hello.csv 1] [EXECUTE] &1> I'm a mouse.
[hello.csv 0] [DESTROY] Terminating computing instance for this job...
[hello.csv 1] [DESTROY] Terminating computing instance for this job...
2018/10/04 15:00:03 [COMMAND] All of your 2 job(s) are completed 🎉
Congrats! You first jobs are completed.
Let’s follow what happened in this trial.
-> How it works