Running a Skycoin blockchain node in a production environment.

Skycoin is an opensource block-chain implementation which is being used by a number of companies, including Skycoin (SKY) itself.  It has a several advantages over Bitcoin, Ethereum and similar implementations:

  1. There are no transaction costs as such.
  2. Transactions happen in seconds, not hours.
  3. High number of transactions per second (tps)
  4. Resilience (remember Crypto-Kitties)

This blog is about running a server node. You can also easily run a node on your PC/laptop as the wallet download also comes with a full node included.

This blog will get you up and running with Ubuntu 17, but will work equally well for Ubuntu 16 LTS and similar.  The official instructions (for Debian) are here.

1. HW and OS Prerequisites.

It will run on Any recent Linux, windows or OS X machine.

1GB ram, 1CPU and 10GB disk are sufficient, but for production I would recommend minimum of 2 CPU, 2GB ram, 40GB disk.

2. OS preparation

2.1 First make sure your distro is up to date:

$ sudo apt-get update
$ sudo apt-get upgrade -y

NOTE:

  1. when using linode servers, accept the option to keep current grub config file.
  2. you can also consider doing “sudo apt-get dist-upgrade”

2.2 Harden your server

You are responsible to secure your server. Below are recommended additions to your list of hardening steps.

2.2.1 create user accounts for the administrators.  Avoid logging in as root.

$ sudo adduser example_user
$ sudo adduser example_user sudo

2.2.2 Setup certificate based login

Create public/private keys for your login user.
paste your public rsa key into ~/.ssh/authorized_keys

Make sure .ssh dir has 700 permissions and is owned by example_user and that authorized_keys is 600.

See here for more information

2.2.2 Disable root login over SSH

(Warning, create your user accounts first)

Edit /etc/ssh/sshd_config:

PermitRootLogin no

2.2.3 Disable passwords

Make sure you have certificate based login working first.

Edit: /etc/ssh/sshd_config:

PasswordAuthentication no

2.2.4 Setup firewall

make sure firewall is installed and running:

$ sudo ufw status

It will be inactive by default.

$ sudo ufw allow ssh/tcp
$ sudo ufw limit ssh/tcp
$ sudo ufw allow 6000/tcp
$ sudo ufw logging on

start the firewall

$ sudo ufw enable

Note, this will now block external rpc calls.  If you want PRC access, you can open port 8334 with suitable security.

2.2.5 install fail2ban

This bans logsin after too many failed attempts. It will stop the server being spammed with malicious login attempts.  If you decide to keep password based login, this is doubly important to prevent brute force password attacks.

$ sudo apt-get install fail2ban
$ sudo service fail2ban status
$ sudo fail2ban-client status

Consider setting  up sendmail to email the administrator any violations caught.

2.3 Install utils

$ sudo apt-get install -y git curl

3. Skycoin Installation

Install go

There are many ways to do this, including using gvm.  This blog uses the manual install.

$ cd ~
$ export GOV=1.9
$ curl -sS https://storage.googleapis.com/golang/go$GOV.linux-amd64.tar.gz > go$GOV.linux-amd64.tar.gz

 

$ tar xvf go$GOV.linux-amd64.tar.gz
$ rm go$GOV.linux-amd64.tar.gz

$ sudo mv go /usr/local/go
$ sudo ln -s /usr/local/go/bin/go /usr/local/bin/go
$ sudo ln -s /usr/local/go/bin/godoc /usr/local/bin/godoc
$ sudo ln -s /usr/local/go/bin/gofmt /usr/local/bin/gofmt

3.1 Create skyd user

This user will be used as the “owner” of the process.

$ sudo adduser skyd

3.2 Set paths

$ sudo su – skyd
$ vi .profile

add this to the end of .profile:

export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export GOBIN=$HOME/go/bin
export PATH=$PATH:$GOPATH/bin

reload the profile

$ . ./.profile

Note: if you are using a local xterm, not ssh login, then .profile is not called. In this case edit .bashrc rather than .profile.

3.3 Install skycoin

$ sudo su – skyd

$ go get github.com/skycoin/skycoin/…

3.3.1 Installing from a fork

Note: if you installing skycoin from a fork or a clone coin, and you dont want to have to rename the skycoin paths, use the following isntall method to install the fork:

$ sudo su – skyd
$ mkdir go
$ cd go
$ mkdir src
$ mkdir bin
$ mkdir pkg
$ cd src
$ mkdir github.com
$ cd github.com
$ mkdir skycoin
$ cd skycoin
$ git clone git://github.com/YourFork/skycoin
$ go install skycoin/cmd/skycoin/skycoin.go

This will put a new executable “skycoin” in your go/bin dir.

The above is basically what “go get” was doing for us, but it allows us to set the proper import paths.

4. Setup service

This will start the skycoin node on boot, and also if it dies.

Here we use systemd, but you can also use init.d/rc.d, systemctl or other service  manager.  systemd is the default manager on Ubuntu 16 and replaces upstart.

4.1 Create the service description file:

$ sudo vi /etc/systemd/system/skycoin.service

[Unit]
Description=Skycoin node
After=network.target

[Service]
WorkingDirectory=/home/skyd
User=skyd
Group=skyd
ExecStart=/home/skyd/go/bin/skycoin
Restart=on-failure

[Install]
WantedBy=multi-user.target

4.2 enable the service on startup/reboot:

$ sudo systemctl enable skycoin

5. Starting/stopping

$ sudo systemctl start skycoin

$ sudo systemctl stop skycoin

6. Monitoring

$ sudo sytemctl status skycoin

This will output the status (running or not, and the last lines of the log output.

Checking logs:

$ sudo journalctl -u skycoin

$ sudo journalctl -u skycoin -f

 

 

 

 

Advertisements

Cut and paste between macOS and linux guest with Virtualbox.

If you have installed a server version of Linux, without a GUI, but still want to be able to cut and paste text and commands, then follow these steps.

Note, the default terminal you get with virtualbox does not offer cut and paste, you will need to use ssh or ssh client such as putty which support cut and paste. To use ssh or ssh client, you need to be able to connect to your virtual machine on port 22.

How you do this will depend on which type of network adapters you have configured.  The default for Virtual box is NAT

  1. NAT.  Guest can see internet. Host cant see guest.  Guest guest a private IP.
  2. Host.  Guest cannot see internet. host can see guest on private network.
  3. Bridged. Guest can see internet as if it was a physical server connected to your router, and gets an IP on the same network as the host. Host can see guest.

For non NAT setups, skep the next step.

1. Setup port forwarding on VirtualBox.

  1. run your vm, login to the console and do “sudo ifconfig”.  Look for the ip address, it should be something like 10.0.2.15
  2. open the virtualbox control panel
  3. right click on your VM and select “settings”
  4. Click on “Network” icon
  5. this assumes you only have one adapter of type NAT.
  6. Click on Advanced
  7. Click on “port forwarding”
  8. Add a rule as follows:
    1. Protocol: TCP
    2. Host IP: 127.0.0.1
    3. Host Port: 2222
    4. Guest IP: 10.0.2.15
    5. Guest port: 22
  9. Hit OK/Apply

2. Install SSH

check if you have ssh installed:

$ sudo service ssh status

If not install it:

$ sudo apt-get install openssh-server

3. login with ssh

Now you have your port redirection setup, and ssh running on your guest, you can ssh in using a client which supports cut and paste, such as putty, kitty (windows), ssh and iterm (mac).

Cut and paste with the usual Command-C Command-V and selections with the mouse.

4. Installing Guest Additions.

As a bonus, you may want to install guest services. This apparently improves performance, and allows cut/paste in the GUI installs of linux (don’t forget to enable bidirectional cut and paste in Virtualbox settings also).

  1. In the virtualbox guest settings, click Devices | Insert Guest Additions CD Image.
  2. Log in to your guest server.
  3. sudo mount /dev/cdrom /media/cdrom
  4. cd /media/cdrom.
  5. sudo ./VBoxLinuxAdditions.run

5. Bonus: setting up shuttle to remember your ssh settings

If you, like me, have hundreds of ssh server settings, you probably can’t remember them all.   For windows users there is the excellent Putty, and even better kitty.  Mac users have a poor selection of SSH managers, but shuttle is the best I have found so far. There is no GUI to setup the servers, but there is a very nice dropdown menu of servers once you have added them to the shuttle config file.  here is a excerpt for mine:

{ “Virtualbox Servers”: [
{
“cmd”: “ssh -p 2222 youruser@localhost”,
“inTerminal”: “tab”,
“name”: “VM2”,
“theme”: “basic”,
“title”: “Virtualbox VM2 test skywire”
}
]},

 

 

Limitations of Bitrix24 as a project management tool

There are many tools out there, from Jira at one end of the complexity spectrum, to Asana and similar tools at the other.  Most are focused only on task management, and lack the other tools corporations need, including employee directories, wikis, document repositories, messaging, workflow, invoicing etc.

We are currently trialing Bitrix24. Its main focus is CRM, but it also has pretty decent project management features, including subtasks.  It is also far cheaper than the competition ($40/m for 25 users). Compare this with Jira + confluence + various needed plugins = > $40/m per user.

NOTE: we only use the “cloud” version.  If you install and maintain bitrix on your own servers, at significantly higher cost, more customisation is possible.

Rather than go into detail about what it can do, below are the limitations we faced:

  1. No custom fields for users.  E.g. we use telegram, wechat, discord and github across our geographically diverse teams. We want every user to put his telegram ID, wechat ID, github ID etc. in their profile, as maintaining these lists is difficult.  Unfortunately, Bitrix is limited to skype, twitter, facebook, linked in and xing only.  There is no option to add additional fields or social networks/tools.
    1. social
  2. Custom fields cannot be added to all tasks for all projects.  Bitrix tasks are missing some key fields such as priority (which could be a number from 1 to 5, or a string such as “high” “medium” or “low”).  It is easy to add a custom field such as priority to a single task, but this is of little use.  There is no concept of adding fields to tasks for all projects, or for specific projects.  There is an option “set field set to all users”.  This has the rather unfortunate side effect of:
    1. Erasing any custom field anyone else has created.
    2. Applying this field to every task in the system, including peoples private tasks, workflow tasks, workgroup tasks, i.e. tasks outside of the project management system.
    3. If the person who created the custom task fields leaves the company, noone can maintain the custom tasks. Another admin will need to create a similar set, and apply this to everyone.  We have not tested what happens to the existing fields in this case.
    4. Custom fields cannot be made mandatory.
    5. There is the concept of task templates, but again, if you add a custom field to a task template, you need to overwrite every other list of custom fields.
  3. Invoicing only works for physical products, not services.  You can only have products which are measured in “m”, “l” “g”, “kg” and “pcs”. E.g. you cant bill for a specific number of days or hours at a specified rate.  As most of my clients only bill for consultancy, this is a significant limitation.
    1. unit
  4. Tasks do not have levels of priority, only “high priority” or not.  At minimum we would need “critical”, “high”, “medium”, “low” and, ideally, “nice to have”.  There is no usable way to add this.  If adding custom fields via task templates was a tenable solution, the task search and sort system doesn’t support custom fields.  Without the ability to search/report/sort based on priority levels, even using custom fields is not a solution.
  5. Tasks cannot be reordered.  E.g. if you have “test task” then “develop task”, and test task is set as starting after develop task has finished, test task will still be shown on top of develop task.  When you have 30+ tasks, and no way to order them, you get spaghetti in your gantt charts and in your task lists.  The only work around is to make sure you create each task in exact order, which for most projects is impracticable.
  6. If you create a project with tasks, and the project goes into test, the test team will need to create tickets against that project for Bugs etc.  Unfortunately, you can only create tasks on a project. Bugs have must have (and searchable) fields such as version found in, bug type, severity level, component, how to reproduce etc.   When bugs are closed they need a status such as verified, wont fix, fixed, fixed in version X etc.  None of this is possible, unfortunately  Also, new bugs should not appear on the gantt chart as tasks until verified and assigned.

deploying a laravel app to a linux server

Prerequisites:

  1. your mysql DB setup and your schema installed (or use artisan),  the mysql user you are going to put in your .env file must have be granted relevant permissions on the relevant database.
  2. php >= 5.6 for apache
    1. This will depend on your hosting.  for my hosting, I had to add this line to the root .htaccess file: “AddType x-httpd-php56 .php”
  3. php >= 5.6 for command line
    1. When you SSH to your server (e.g. using kitty or putty), see what version you have:
      $ bash-3.2$ php -v
      PHP 5.6.22 (cli) (built: Jun 9 2016 18:53:49)
    2. If the answer is not 5.6 or higher, contact your server hosting provider and ask them how to change the default.  For mine, I edited my root .bashrc and added the following line:
      alias php=/usr/bin/php-5.6
    3. Composer
      1. This one is a bit o a mystery.  There are conflicting instructions.  The official website suggest running the following at the command line:
      2. php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
        php -r "if (hash_file('SHA384', 'composer-setup.php') === '669656bab3166a7aff8a7506b8cb2d1c292f042046c5a994c43155c0be6190fa0355160742ab2e1c88d40d5be660b410') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
        php composer-setup.php
        php -r "unlink('composer-setup.php');"
      3. the result is a composer.phar file.  However, the instructions to run it universally are like this: “# composer xxx” which will not work. I chose to add the following line to my .bashrc file:
        alias composer=/pathtomydir/composer.phar
      4. However, running “# composer xxx” results in the following warning:
        Warning: Composer should be invoked via the CLI version of PHP, not the cgi-fcgi SAPI
      5. To solve this, I found the cli version, and changed the alias to this:
        alias composer="/usr/bin/php-5.6-cli ~/composer.phar"

Deployment:

  1. get the project onto a server.  E.g.
    1. ssh myserver.com
    2. mkdir versions
    3. cd versions
    4. git clone –depth 1 https://someone@app.deveo.com/SomeProject
  2. Now you should have the following dirs:
    1. www (or public_html or similar)
    2. versions
  3. copy the public dir to your www root (including dot files) = don’t use *):
    1. cd www
    2. cp -r ../versions/SomeProject/. .
  4. Modify www/index.php (not Public/index.php)

Install dependencies.

  1. cd yourprojectdir
  2. run “composer install”
  3. This will download a bunch of files.
  4. Unfortunately for me, at the end I got this:
Generating autoload files
> Illuminate\Foundation\ComposerScripts::postInstall
> php artisan optimize
Content-type: text/html

<br />
<b>Parse error</b>: syntax error, unexpected 'class' (T_CLASS), expecting ident ifier (T_STRING) or variable (T_VARIABLE) or '{' or '$' in <b>/var/sites/d/dev.g amificationgiant.com/public_html/versions/platform-new/artisan</b> on line <b>30 </b><br />
Script php artisan optimize handling the post-install-cmd event returned with er ror code 255

The problem seems to be that composer is spawning a new shell which gets the default php which is 5.4, and there is nothing I can do about this.  Dead end for our hosting provider.

The solution is the following:

  1. $ cd
  2. $ mkdir bin
  3. $ ln -s /usr/bin/php-5.6 ~/bin/php
  4. edit .bashrc to have the following two lines:
alias composer="/usr/bin/php-5.6-cli ~/composer.phar"
export PATH=/var/sites/g/gamificationgiant.com/bin:$PATH

Now you can install dependencies:

  1. $ composer install
  2. $ composer dumpautoload -o
  3. $ php artisan route:cache

The last item will give the following error:

Unable to prepare route [api/user] for serialization. Uses Closure.

setup environment

  1. cd to the root of your app
  2. $ cp .env.example .env
  3. edit .env with your database details etc.
  4. $ php artisan key:generate
    1. Do this only if you dont want to use an existing key in your .env
  5. $ php artisan config:cache
    1. This has to be done after adding the key to .env.

Hopefully now your app will be working.

Trouble shooting

Cipher issues

If you see this in your “storage/logs/laravel.log” file:

'The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct key lengths.' in /var/sites/d/dev.gamificationgiant.com/public_html/versions/platform-new/vendor/laravel/framework/src/Illuminate/Encryption/Encrypter.php:43
Stack trace:
#0 /var/sites/d/dev.xxxxx.com/public_html/versions/platform-new/vendor/laravel/framework/src/Illuminate/Encryption/EncryptionServiceProvider.php(27): Illuminate\Encryption\Encrypter->__construct(NULL, 'AES-256-CBC')

Then check the following:

  1. a valid cipher is set in .env APP_KEY
  2. you have run “php artisan config:cache” AFTER setting the key.

 

 

 

Creating JSON APIs in GRAILS 3.3

Creating simple API’s which accept json or get parameters, and respond with json was very easy in grails 2.5.  Unfortunately, many things have subtly changed in grails 3 making porting an app rather difficult, especially as the 3.3 documentation still has 2.5 examples in many places.

Note:

  1. The project you add the API to does not need to be created with the rest-api profile.
  2. The rest-api profile does provide json views, which may or may not be easier (Ill try that another day)
  3. The code below was written from memory, so there will be some typos etc.  Let me know if you find any and update accordingly.

Let us assume we want a simple POST API with loginUser (and other calls)

loginUser will send a JSON request body like this:

{"username":"bob", "password":"pass"}

and expect to get back the following format:

{"result":0, 
 "player":{"username":"bob", "firstname":"bob", "id":1234}, 
 "accounts":[{"balance":1.0, "currencyIso":"GBP", id:1234}]
}

an error would look something like this:

{"result":100, "message":"invalid parameters"}

Here we are using result as a return status/error code, and there are many error codes defined in the API.

Let us assume our domain objects are something like this:

User.groovy

package bla
class User {
  String username
  String password
  String firstname
}

Account.groovy

package bla
Class Account {
  User user
  BigDecimal balance
  String currencyIso
}

Step 1: register the URL

This used to be in Configuration/UrlMappings.groovy. Now it is defined in grails-app/controllers/web.admin/UrlMappings.groovy.

Add your URL thusly:

class UrlMappings {
  static mappings = {
  // stuff
  "/api/v1/user/login"(namespace: "v1", controller: "UserApi", action: "login")
  }
}

Step 2: create your controller (you can do this by simply creating the file or using the create-controller grails command).  Note, here I am putting two classes into one groovy file.  You can split them into two files if you prefer.

UserApiController.groovy

package bla
import grails.converters.JSON

/* The LoginCommand is just to make validating and parsing the JSON input easier
*/
class LoginCommand implements grails.validation.Validateable { 
  String username 
  String password 
  static constraints = { 
     // TODO: share the constraints with the domain object.
     username nullable:false, blank: false, size: 3..32 
     password nullable:false, blank: false, size: 3..32
  }
} // class LoginCommand

class UserApiController {
  static namespace = 'v1'

  def login(LoginCommand cmd) {
  try {
    if (cmd.hasErrors()) { 
        log.error("invalid parameters") 
        render(status: 400, contentType: 'application/json') {
          result 10 
          message "errors in parameters" 
        return
        } 
     } else { 
         // TODO: use a service which will update login failed counters etc
         User user = User.findByUsernameAndPassword(cmd.username, cmd.password)
         if (user == null) { 
            render(status: 404, contentType: 'application/json') { 
                result 100 
                message "could not find user with that username and password" 
            } 
         } else { 
            def userAccounts = Account.findAllByUser(user)
            render(status: 200, contentType: 'application/json') { 
               result 0 
               user {   
                  id user.id   
                  username user.username 
                  firstname user.firstname
               } 
            } 
       } // else all good 
    } // else params ok 
  } catch (Exception e) { 
    log.error("Caught Exception in UserApiController.login()", e) 
    render(status: 500, contentType: 'application/json') { 
        result 9999, 
        message e.toString() 
    } 
  }
}

To test this you can do something like:

http://localhost:8080/api/v1/User/login?username=asdf&password=asdf

Or if you want to use POST, I recommend using https://restlet.com/

The above code works but doesn’t yet return the account info.  Rendering domain objects by hand is easy but hard to maintain – you don’t want to have to duplicate this code in all your apis which return the same objects.  For this there are two options:

  1. object marshallers
  2. json views.

Object marshallers and easy and powerful and work in both grails 2.x and 3.x.  Be aware they are currently not working when you user render(){….} – this is broken. You will need to use JSONBuilder as above and use the respose text parameter.  An object marshaller for our account would be defined in the bootstrap.goovy file thusly:

DecimalFormat df = new DecimalFormat("##0.00"); 
JSON.registerObjectMarshaller(Account) { 
    return [balance: df.format(it.balance), currencyIso: it.currencyIso, id: it.id]
}

Then you can update the login method thusly:

render(status: 200, contentType: 'application/json') { 
  result 0 
  user {   id user.id   username user.username firstname user.firstname } 
  accounts userAccounts
}

 

You can restrict the API to post only using this at the top of the controller:

static allowedMethods = [login: "POST"]

Lastly, don’t forget to run this on SSL only in production to hide the password (not port 80 or 8080), and to encrypt the password in the database (e.g. using a beforeInsert interceptor)

 

How to use font-awesome in Grails 3.2

This was surprisingly difficult. There are several posts, but none were complete or working, so here is a complete version.

  1. download font-awesome zip file from here http://fontawesome.io/
  2. create a directory grails-app/assets/fonts*
  3. unzip the contents of the zip into fonts.  so you should have something like:
    1. grails-app/fonts/css/font-awesome.min.css
    2. grails-app/fonts/fonts/FontAwesome.otf
    3. etc.
  4. Edit application.css under grails-app/assets/stylesheets and add**:
     *= require css/font-awesome
  5. edit build.gradle in the project root and add
assets {
 minifyJs = true
 minifyCss = true
 includes = ["fonts/*"]
}

Now restart your server, and try adding something like this in your gsp pages:

<i class="fa fa-camera-retro fa-4x"></i> fa-4x

And you should see a camera icon.

NOTE:

* personally, I would have put this in a directory called font-awesome rather than fonts, but every post on the web does it this way, and I am assuming they have reasons.

**surprisingly, this is intelligent enough to pull in the minified version, even though the name matches the non-minified file.  This method has the side effect of including the fonts on every page.  In theory, it would be possible to do it via direct include instead. e.g:

<link rel="stylesheet" href="/assets/fonts/css/font-awesome.min.css">

But this does not work. If anyone knows why, let  us know!

configuring grails 3.2 to work with mysql

Sadly, grails 3 uses yaml files for configuration, instead of the goovy files in previous versions.  In my opinion, this is a big step backwards for these reasons:

  1. Yaml is yet another syntax to have to learn for no real benefit.  Groovy is the backbone of grails – every one who uses grails knows it (and most love it).
  2. Existing 2.5 configurations are not compatible.
  3. datasource.groovy was a place for all database stuff. Now database config is mixed in with every other config in the same file, which makes sharing configs trickier.
  4. There are no non-trivial examples documented anywhere using the new format.
  5. Yaml is very finicky, with invisible syntax errors. E.g. if you accidentally have a tab instead of a space, it will fail.  As we are a tab for indentation shop, this was especially painful.
  6. Poor tools support for yaml compared with groovy, especially in Eclipse.  E.g. no automatic yaml formatting or converting tabs to spaces – you will need to edit every line when converting from the old to the new and replace tabs with spaces, or try find external tools.
  7. The out of the box application.yaml fails validation with the online yaml validation tools.
  8. For more complex structures, I find yaml syntax is much harder to manage than groovy DSL.  Especially restructuring and/or cut and pasting is a disaster.
  9. Yaml is full of Gothas.  E.g. “file: c:\hello” is ok, but “file: c:” is not.
  10. groovy dsl is more powerful

To make matters worse, the grails 3 documentation, to date, still only has the old 2.5 groovy database configuration information.

Here is how we did it:

  1. edit build.gradle file in your main projects root.
dependencies {
 :
 runtime 'mysql:mysql-connector-java:5.1.33'

2. Edit your application.yml in grails-app/conf dir.  You will have to translate your old database.groovy into yaml.  We did this by trial and error.

environments:
  development:
     dataSource:
       dbCreate: update
       pooled: true
       url: jdbc:mysql://127.0.0.1:3306/xxx
       username: yyy
       password: zzz
       driverClassName: com.mysql.jdbc.Driver
       dialect: org.hibernate.dialect.MySQL5InnoDBDialect
       properties:
         maxActive: -1
         minEvictableIdleTimeMillis: 1800000
         timeBetweenEvictionRunsMillis: 1800000
         numTestsPerEvictionRun: 3
         testOnBorrow: true
         testWhileIdle: true
         testOnReturn: true
         validationQuery: "SELECT 1"