hotsub

CLI tool to run a batch jobs with ETL/ExTL framework on AWS or other cloud services.

View the Project on GitHub github.com/otiai10/hotsub

This page describes how to get started with hotsub command.

0. Prerequisites

You need 2 softwares installed on your environment

and 2 permissions on AWS account

Install docker-machine

You can skip this step if following command already works well on your computer.

% docker-machine version

Otherwise, you need to install docker-machine.

Install AWS CLI

You can skip this step if following command already works well on your computer.

% aws configure list

Otherwise, you need to install awscli and configure it with your credentials.

1. Install hotsub

Choose binary for your OS from this release page and, if you want, locate extracted executable file to your PATH.

2. Hello World!

You can generate Hello, World project by using template command.

% hotsub template
Successfully created template project
    at helloworld_201810041451
Go to the directory and check README.md
% cd helloworld_201810041451
% ls
README.md	hello.csv	hello.sh

These are all you need to run Hello, World

% hotsub run \
    --script hello.sh \
    --tasks hello.csv \
    --verbose

Then you will get following output

2018/10/04 14:53:46 [COMMAND]	Your tasks file is parsed and decoded to 2 job(s) ✅
2018/10/04 14:53:46 [COMMAND]	See logs here -> /Users/otiai10/proj/hotsub/github.io/helloworld_201810041451/log/20181004_145346
[hello.csv 0]	[CREATE]	Creating computing instance for this job...
[hello.csv 1]	[CREATE]	Creating computing instance for this job...
[hello.csv 1]	[CONSTRUCT]	Constructing containers for this job...
[hello.csv 1]	[CONSTRUCT]	Constructing workflow container inside the computing instance...
[hello.csv 1]	[CONSTRUCT]	Constructing routine container inside the computing instance...
[hello.csv 0]	[CONSTRUCT]	Constructing containers for this job...
[hello.csv 0]	[CONSTRUCT]	Constructing workflow container inside the computing instance...
[hello.csv 0]	[CONSTRUCT]	Constructing routine container inside the computing instance...
[hello.csv 0]	[EXECUTE]	&1> Hello! My name is Tom.
[hello.csv 0]	[EXECUTE]	&1> I'm a cat.
[hello.csv 1]	[EXECUTE]	&1> Hello! My name is Jerry.
[hello.csv 1]	[EXECUTE]	&1> I'm a mouse.
[hello.csv 0]	[DESTROY]	Terminating computing instance for this job...
[hello.csv 1]	[DESTROY]	Terminating computing instance for this job...
2018/10/04 15:00:03 [COMMAND]	All of your 2 job(s) are completed 🎉

Congrats! You first jobs are completed.

Let’s follow what happened in this trial.

-> How it works