Copying files from one onedrive account to another without a computer

MS Office 365 offers 1TB per user. The home version offers 5 users.

What happens if you have say 800GB of data in one onedrive users account, and you want to move it to another? This happened recently to me. I filled my 1TB with photos, not leaving any room for my normal files, but could also be because you change accounts or plans.

Downloading 1TB of files, then uploading them to the new account would take months, and require 1-2TB of disk space. As a macbook pro user, I have 10GB free. Clearly not an option.

This is where mover.io comes in. Recently purchased by Microsoft, it offers the ability to transfer from many different storage providers (A3, Google, dropbox etc) onto onedrive. It also offers onedrive to onedrive, and is completely free. It works away server to server, in the cloud, and your computer doesnt need to be on. It took around 5 days to transfer 800GB without error.

Using mover.io is easy. Just create an account with one of your onedrive accounts (I used the “source” one), then add the second onedrive account as the destination. It shows a tree of files on the source, and you can pick all or a subdirectory. One tip is if you want to copy from a directory called say “files” on the source to “files” on the destination, first create a folder called files on the destination and copy to that, as onedrive only copies the contents of directories, not the directory itself.

I made a couple of test transfers before doing anything large – dont want to waste microsoft’s bandwidth given that its free.

Below we can see any transfers in progress or completed.

Creating a fixed IP “VPN” using ssh tunnel on Mac

If you need a fixed ip to access some web resource, and don’t have one at home, there is a quick and easy solution. First, you need a cheap server. I suggest a 5$ lined instance. See https://www.linode.com/pricing/

Once you can ssh to your server, you can create your “VPN”. on the Mac, simply run this command:

$ ssh -D 8123 -f -C -q -N you@yourserver

replacing “you” with your ssh login username and “yorserver” with your servers IP or DSN name.

Now you need to setup a proxy which your browsers will use.

System Peferences->Network->Advanced-> Proxies

Once Now hit “OK” followed by “Apply”

Open a new browser tab, and you should be using your new external IP. If you go to https://www.whatismyip.com/ you should see your servers IP

Updating the database in episerver

When you update the nuget episerver packages then run the site, you may get something like this:

The database schema for ‘EPiServer.Find.Cms’ has not been updated to version ‘13.2.5’, current database version is ‘13.0.4’. Update the database manually by running the cmdlet ‘Update-EPiDatabase’ in the package manager console or set updateDatabaseSchema=”true” on episerver.framework configuration element.

One solution is to edit your web.config in the root of your project. Change this:

  <episerver.framework>
    <appData basePath="App_Data" />
    <scanAssembly forceBinFolderScan="true" />
    <virtualRoles addClaims="true">

to this:

  <episerver.framework updateDatabaseSchema="true">
    <appData basePath="App_Data" />
    <scanAssembly forceBinFolderScan="true" />
    <virtualRoles addClaims="true">

and restart your project. The database will be updated. You probably dont want this setting for production.

Nashorn: using javascript “modules” from java/groovy

Nashorn, which is built into java, allows java to run javascript, and javascript to call java. The main use case of this is avoid duplicating client side javascript as java on the server side. You can run the same javascript on both sides.

There are plenty of basic tutorials which explain how to call a simple function contained in a javascript string or file for example:

engine.eval("function composeGreeting(name) {" +
  "return 'Hello ' + name" +
  "}");
Invocable invocable = (Invocable) engine;
 
Object funcResult = invocable.invokeFunction("composeGreeting", "baeldung");

However, most javascript is written as modules, which do not pollute the global namespace, and hide private functions and variables e.g:

example2.js

var myModule = (function() {
    var privateVar = 42;
    function composeGreeting() {return 'Hello';};
    return {
       composeGreeting: composeGreeting
    };
})(); 

To call composeGreeting in the above example, we need to first “extract” myModule, then call composeGreeting on that (note: we are using grails/groovy, but the java is almost identical)

File file = new File("example2.js")
Reader reader = file.newReader()
scriptEngine.eval(reader)
Invocable invocable = (Invocable) scriptEngine;
Object funcResult
try {
    funcResult = invocable.invokeMethod(scriptEngine.get("myModule"), "composeGreeting");
} catch (NoSuchMethodException e) {
    e.printStackTrace();
} catch (ScriptException e) {
    e.printStackTrace();
}
System.out.println(funcResult) // "Hello"

Spring and Groovy

Grails is the standard for rapid enterprise application development, but sometimes you need to fall back on plain old Java/Spring. However, you don’t need to sacrifice the productivity increase of groovy. You can use groovy classes and code interchangeably with your java in your Spring project.

Here we are using gradle, which is arguably more powerful (scripting language vs XML) and faster.

We don’t even need to install Groovy – gradle handles this for us.

Simple add the following to your spring projects build.gradle:

plugins {
    id 'groovy'
}
dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.3.11'
}

Here is an example of a basic Spring boot web project’s build.gradle file which was generated with Intellij’s Spring Initializr feature:

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:2.1.6.RELEASE")
    }
}

plugins {
    id 'groovy'
    id 'java'
}

apply plugin: 'idea'
apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'

bootJar {
    baseName = 'gs-spring-boot'
    version =  '0.1.0'
}

sourceCompatibility = 1.8
targetCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    compile("org.springframework.boot:spring-boot-starter-web")
    compile 'org.codehaus.groovy:groovy-all:2.3.11'
    testCompile group: 'junit', name: 'junit', version: '4.12'
}

This will generate you a groovy folder under src/main. So you will have:

<projectDir>/src/main/groovy/
<projectDir>/src/main/java/

Now you can create your spring components either as groovy or java.

e.g. create src/main/groovy/hello/MyGroovyController.groovy

package hello
import org.springframework.web.bind.annotation.RequestMapping
import org.springframework.web.bind.annotation.RestController

@RestController
class MyGroovyController  {
    @RequestMapping("/")
    public String index() {
        "Hello from Groovy!"
    }
}

And create a java controller in the file src/main/java/hello/MyJavaController.java

package hello;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestMapping;

@RestController
public class HelloController {
    @RequestMapping("/java")
    public String index() {
        return "Hello from Java!";
    }
}

For completeness, here is the Application class which makes it all work (src/main/java/hello/Application.java)

package hello;

import java.util.Arrays;

import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    @Bean
    public CommandLineRunner commandLineRunner(ApplicationContext ctx) 
        {
        return args -> {
            String[] beanNames = ctx.getBeanDefinitionNames();
            Arrays.sort(beanNames);
            for (String beanName : beanNames) {
                System.out.println(beanName);
            }
        };
    }
}

To run your app, enter the following command:

./gradlew build && java -jar build/libs/gs-spring-boot-0.1.0.jar

Alternatively, you can run the app from the IDE. In intellij, open the gradle tab on the right, open up Tasks, and select application/bootRun.

Now you can hit http://localhost:8080 with your browser and see “Hello from Groovy!” or hit http://localhost:8080/java and see “Hello from Java!”

Exporting excel with grails 3

Usually your controllers will respond with list of data to be show on the screen as paginated tables. However, sometimes your uses will want to download the complete dataset as an excel spreadsheet. This is remarkably easy thanks to the Excel-export plugin here.

Firstly, add this to your dependencies block in your top grails.

dependencies {
    compile 'org.grails.plugins:excel-export:2.1'
}

Let us say you have a domain object books, and a controller with a a search method when the user can select some search criteria:

   def search(Integer max) {
        params.max = Math.min(max ?: 10, 100)
        def books = booklService.find(params)
        respond books, view:"search", model:[books:books, author:params.author, params.title ]
    }

And a gsp something like this:

<g:form name="search" url="[controller:'book',action:'search']">
        <div class="form-row">
            <div class="col-md-5">
                <input type="text" class="form-control" size="20" name="author" value="${author}" placeholder="author" />
            </div>
            <div class="col-md-5">
                <input type="text" class="form-control" size="20"  name="title" value="${title}" placeholder="title" />
            </div>
            <div class="col-md-2">
                <g:actionSubmit value="Search" action="search" />
            </div>
        </div>
</g:form>

        <div id="list-book" class="content scaffold-book" role="main">
            <h1><g:message code="default.list.label" args="[entityName]" /></h1>
            <g:if test="${flash.message}">
                <div class="message" role="status">${flash.message}</div>
            </g:if>
            <f:table collection="${bookList}"/>
            <div class="pagination">
                <g:paginate total="${bookCount ?: 0}" params="[author:params.author, title:params.title]"/>
            </div>
        </div>

Now just add a download button on the search form:

            <div class="col-md-2">
                <g:actionSubmit value="Search" action="search" />
                <g:actionSubmit value="Download" action="download" />
            </div>

And add the download method in your controller:

def download() {
    params.max = 1000 // @TODO put this in a setting.
    def books = booklService.find(params)
    def headers = ['Created','Author', 'title']
    def withProperties = ['dateCreated','author.name', 'title]
    new WebXlsxExporter().with {
        setResponseHeaders(response)
        fillHeader(headers)
        add(books, withProperties)
        save(response.outputStream)
    }
}

That’s it. It will download a nice excel file with todays date and time as the file name.

If you want to get fancy, you can manipulate sheets, rows, columns and cells directly using the underlying Apache POI libs. E.g. if you want to add a sum formula at the bottom of a row, you can do something like this:

def download() {
    params.max = 1000 // @TODO put this in a setting.
    def books = booklService.find(params)
    def headers = ['Created','Author', 'title']
    def withProperties = ['dateCreated','author.name', 'title]
  
    int last = books.getTotalCount() + 1 // for header
    def exporter = new WebXlsxExporter()
    exporter.setResponseHeaders(response)
    exporter.fillHeader(headers)
    exporter.add(deals, withProperties)
    def totalsRow = exporter.getSheet().createRow(last+1)
    def cell = totalsRow.createCell(5)
    cell.setCellFormula("SUM(F2:F" + last +")")
    exporter.save(response.outputStream)
}

grails 3: custom validators and custom error messages

Adding custom validators to grails domain objects is easy. Consider the following domain class:

class Transaction {
    Account account
    BigDecimal debit = 0
    BigDecimal credit = 0

    static constraints = {
        debit nullable: false, min: 0.0, validator: {val, obj -> if ((val == 0.0 && obj.credit == 0.0) || (val > 0.0 && obj.credit > 0.0)) return "transaction.equal" else return true}
        credit nullable: false, min: 0.0
}

This shows how the valiator closure is checking that either credit is not zero or debit is not zero:

validator: {val, obj -> 
     if ((val == 0.0 && obj.credit == 0.0) 
         || (val > 0.0 && obj.credit > 0.0)) 
         return "transaction.equal" 
         else return true
}

The error message to be displayed to the user if the constraint fails will be looked up the following file:

grails-app/i18n/messages.properties

Edit this file and add your message property:

transaction.equal="Either debit or credit must be > zero, but not both"

Grails 3: using enums with Domain Classes

It is common to need somethingStatus or somethingType in your domain classes. e.g. Account.accountStatus might be “open”, “closed”, “suspended” etc.

There are two ways to implement this:

  1. using a separate domain class
  2. using a string or int enum.

If the status has other properties, then use a separate domain class. E.g.

Account.groovy:

class Account {
  User user
  Currency currency
  AccountStatus accountStatus
}

AccountStatus.groovy:

Class AccountStatus {
   String name
   boolean canLogin
}

However, in simple cases, you can use an enum:

Account.groovy:

class Account {
  User user
  Currency currency
  AccountStatus accountStatus
}

enum AccountStatus {
   OPEN,
   CLOSED,
   SUSPENDED
}

Here is an example of usage:

Bootstrap.groovy:


def account1 = new Account(user: user1, currency: gbp, accountStatus: AccountStatus.OPEN).save(failOnError: true)

Grails 3.3. How to fix rounding and truncation of BigDecimal in Fields plugin with scale of more than 3.

If you requite more than 3 decimal places, e.g. to support display crypto currencies (ETH has 18 dp, BTC has 8), and use any of the scaffolded views or fields plugins such as the excellent f:table, f:display etc, you will notice that values of say “1.123456” are shown as “1.12”. Worse still, if you want to update the object using scaffolded views or the fields, even if you don’t change the value, it will be overwritten in the database with the truncated value.

The first step to fixing this is to override the widget used to display either all BigDecimals, or just the specific controller field, or the specific domain object (its your choice).

The documentation to refer to is: https://grails-fields-plugin.github.io/grails-fields/latest/guide/index.html

Lets say this is your domain object:

package me
class Account {
    User user
    BigDecimal balance = 0
    Currency currency
    static constraints = {
        balance nullable: false, scale: 18, precision: 50
    }
}

to override the display of all BigDecimals, create the file:

grails-app/views/_fields/bigDecimal/_displayWidget.gsp

The file should contain something like this:

${value.stripTrailingZeros().toPlainString()} 

Now you should see “1.123456”.

Note:

  1. If you don’t stripTrailing zeros, you will see “1.123456000000000000” if you have scale set to 18 for example.
  2. You can also override the display for a specific field or specific controller, rather than all big decimals (see docs).

Trezor vs Ledger HW wallet for business

There are several comparisons of the trezor and Ledger HW and their corresponding SW wallets. These mainly focus on the end user usability and number of coins supported.

However, when it comes to business, who want to support many customers, and for developers who want to integrate the HW wallets into existing commerce or financial systems, the picture is very different.

The conclusion is that Trezor is far better suited to business than the Ledger, but neither are perfect.

Note, here we don’t make the distinction between Ledger Nano S and Ledger Blue, and likewise between Trezor one and Trezor T, because the later models are basically the same as the former just with larger screen and a few extras.

Architecture

Trezor has everything built into the firmware.  The 500 or so coins it supports are always there and always avialble.  No installation or configuration is required. When Trezor add more coin support, you just upgrade the firmware (which takes less than one minute) and you are done.

Out of the box, the ledger has literally nothing on it, it cant do anything.  You need to install apps using their SW wallet.  Each coin is an app. So there is an app for BTC, one for ETH etc.  You can only install a small number of apps at the same time (something like 4 on the nano and 18 on the blue depending on app size).  If you want to do transactions in more than the allowed number of apps, you have to uninstall and reinstall.

When you upgrade the Ledger, you have to remove all the apps, then reinstall them.

APIs

Trezor has a single unified API which is coin agnostic – e.g. you can generate addresses for any coin with one call, and you can sign transactions for any coin.  The coin (e.g. Bitcoin or Ethereum) is just a parameter.

There are several ways to access the Trezor API. there is a python based command line tool, which allows you to do pretty much anything with a bash script or command line (such as generating 1000 addresses for a given path).  There is also the bridge, which allows web apps to access the Hw through the browser.  This then requires no SW to be installed on the users machine to

Ledger doesn’t have an API as such.  In theory, each app can provide its own API. There is apparently a beta API for the BTC app.  But there is nothing to say that other apps (aka coins) will get APIs, or if they will be unified in any way.

Compatibility

Interestingly, you can restore your 24 word seed on either device.  So they are largely compatible in this regard.  The only issue is that the Ledger uses a different Path for Ethereum than the Trezor by default, so you will need to use a wallet which allows the path to be configured (such as Electrum).

Labels

Labelling accounts and addresses is important for avoiding mistakes and for reconciliation/accounting.  e.g. if you have 10 addresses, you might label them with the 10 customer names you have given them to. Or you might label them for different invoices.

Trezor has an interestingly solution for labels. They dont want to store them on the device, as they will be lost if you lose the Trezor and need to restore it from the Seed. So instead the store the labels in an encrypted file in dropbox. The nice part of this is that other users (in a company) can use their dropbox to securely share the labels between administrators.

Ledger supports labels for accounts, but doesn’t support addresses for accounts so these cannot be labeled.

Trezor can label individual addresses.

Discovery gap limit, accounting, and client funds segregation.

If you have say 20 customers, and you want to invoice each, you would want to give each customer their own account in the form of an address.  This allows proper client fund segregation, easier audit and accounting.  Even better, an account per invoice.  Additionally, you would want to label each account with its function.  At a glance, you can see the balance and transactions on any account.

This should be trivial to do with the HD wallet, as it supports unlimited addresses for a given account path. However, due to the limited SW implementations of the discovery gap, it is not.

Ledger only allow one address to be generated at a time, and only for one account.  It ignores one of the main features of the HD wallet – the ability to generate many addresses, with increasing indexes, for a given account.  It is difficult to compare the ledger wallet to any other wallet, as its feature set is so limited.

Ledger_Live

Above we see an “account” called trstbtc1.  It has one address, which corresponds to index zero. You cannot see this address.

 

You cannot add a second address until there is a transaction on the first.  If you have two customers, and want to give each a separate address for payment, you cant.  You would have to give the first customer an address, wait for them to pay, then the SW will allow you generate the second address.   This goes for their “accounts” also.  The work around I see clients doing every day is the “loading hack“. They generate an addresss, transfer 1 satoshi (or equivalent coin) in to it, wait till it is confirmed (1 hour in the case of BTC), then they can generate a second address.  Next they remove the 1 satoshi, so the customer who gets that address given to him doesn’t get worked that it has a balance (so is a “used” account), and do the same process for the next a account. This is a manual, time-consuming, costly and needless task.

So what is the problem?

Hierarchical Deterministic (HD) wallets have some interesting properties.  Firstly, you only need to backup the master key (e.g. as a 24 word mnewmonic).  From this all child keys can be derived again at any time.  Each coin/token has a well defined sub tree, as defined by the derivation path.  So Bitcoin has one standard path, and Ethereum another. Well, this is not strictly true, bitcoin has at least 2 standard paths, one for “legacy” and one for “segwit”, but that is another post.  Within a specific path, any number of child keys can be derived deterministically (i.e. if you start from the same master key, you will always get the same child keys).  Child keys have an index.  so 0 is the first, 1 is the second etc.  key 0 and key 1 are siblings.

The missing piece in HD wallets is knowing how many child keys were generated, i.e. what index you are on.  As you are only backing up the master key, not the master key + indexes of children for each path.

One solution is the “discover gap limit”.  This setting tells the SW wallet how many addresses for each parent to “load up” (i.e. show the balances for and allow transfers from).  For example a discovery gap limit of 20 means generate the first 20 addresses for the given parent (i.e. index 0 to index 19).  For each of these check the blockchain to see if there were any transactions. If any transactions are round, show the first 20 (with their addresses and balances), and look at the next 20 (index 20 to 39) and so on until no transactions are found for the given block.

The Trezor wallet has a discovery gap limit of 20, allowing 20 new (empty) addresses to be generated without the “loading hack“.

Ledger has a discovery gap limit of 1, alowing only a single new address tone generated (for a given token).

Electrum allows the discovery gap limit to be configured, so you could have 100 or 1000 new accounts to give to your customers in parallel.

The downside to this method is that if you are using a wallet with a high discovery gap limit, then you decide to move to a different wallet with a low discovery gap limit, AND restore your master seed to that new wallet, you wont “see” your accounts. They are there, no money is lost, but the wallet just wont let you view them or use them.

For this reason, all wallets should have a configurable discovery gap limit in the advanced settings.  This is, at most, a few days of development effort for the wallet providers. There is no excuse not to implement this, even as a paid for “pro” feature.

Trezor is far superior to Ledger in this regard (20 vs 1)