Extending Firefly-iii with cron jobs in Docker

[dimaj]

Firefly is an amazing piece of software and I am totally addicted to it. As of right now, the only inconvenience that I have is that I cannot auto-import my transactions as they take place (or soon after).

So, my solution to this was the following:

• Login to each financial institution I am using and configure an email alert for any transaction greater than $0.01
• Configure my gmail to add labels to these alert emails
• Create a Google Script that would
  • scan my inbox and look for labeled emails
  • extract merchant and amount of the transaction
  • add new transaction to Firefly using its extensive API
• Run a shell script ever 15 minutes that would run my rules and delete "temporary" accounts

Without further ado, here are the pieces to make above work:

Google script

When you create your google script, make sure to add 2 project properties:

• firefly-pat - Personal Access Token to authenticate against Firefly
• firefly-endpoint - Endpoint of you Firefly instance.

var userProperties = PropertiesService.getScriptProperties();

/**
 * Get a value associated with a property
 * @param propName {string} Name of the property to get
 * @retuns {string} Value of the property
 */
function getPropertyValue(propName) {
  return userProperties.getProperty(propName);
}

const apiParts = {
  headers: {
    Authorization: `Bearer ${getPropertyValue('firefly-pat')}`,
    'Content-Type': 'application/json',
    Accept: 'application/vnd.api+json'
  },
  endpoint: `${getPropertyValue('firefly-endpoint')}/api/v1`
}


/**
 * Extracts a match at an index
 * @param match {string[]} List of matches
 * @param index {number} Index at which match should be extracted
 * @returns {string} Requested field or undefined is not found
 */
function getValueFromMatchAtIndex(match, index) {
  if (match && match.length > index) {
    return match[index];
  }
}

/**
 * Parses amount from string to float.
 * @param amountStr {string} Amount to parse.
 * @returns {number} Parsed amount.
 * @example `parseAmount("1,234.56"); // returns 1234.56`
 */
function parseAmount(amountStr) {
  return amountStr && parseFloat(amountStr.replace(',', '')) || undefined;
}

/**
 * Map of available accounts.
 */
const accountsMap = {
  citi: {
    match: (from) => from.indexOf('citi') >= 0,
    fields: (body) => {
      const merchant = getValueFromMatchAtIndex(body.match(/Merchant<\/span>(.|\s)+?>\b((.|\s)*?)<\/span/m), 2);
      let amount;
      if (body.indexOf('payment posted') > 0) {
        amount = parseAmount(getValueFromMatchAtIndex(body.match(/A\s*\$(.+?)\s/m), 1));
      } else {
        amount = -1 * parseAmount(getValueFromMatchAtIndex(body.match(/Amount(.|\s)+?\$(.|\s)*?([\d\,\.]*)</m), 3));
      }

      const firefly = { account: '<YOUR FIREFLY ACCOUNT ID>', name: '<YOUR FIREFLY ACCOUNT NAME>', }

      return { firefly, merchant, amount }
    }
  }
}

/**
 * Gets reference to an email label
 * @param labelName {string} Label to get
 * @param shouldCreate {boolean} Should label be created if it doesn't exist
 * @returns {GmailApp.GmailLabel} Reference to a label
 */
function getLabelOrCreate(labelName, shouldCreate) {
  const rv = GmailApp.getUserLabelByName(labelName);
  if (!rv && shouldCreate) {
    rv = GmailApp.createLabel(labelName);
  }

  return rv;
}

const budgetTypes = {
  FIREFLY: 'firefly'
}

const emailLabels = {
  pending: 'Budget Automation/budget-to-process',
  fireflyDone: 'Budget Automation/firefly-processed'
};

const pending = getLabelOrCreate(emailLabels.pending, false);
const fireflyDone = getLabelOrCreate(emailLabels.fireflyDone, true);

function budgetAutomation() {
  if (!pending) {
    Logger.log(`Could not find '${emailLabels.pending}' label. Aborting`)
    return;
  }

  const pendingThreads = pending.getThreads();

  pendingThreads.forEach(email => {
    const messageFieldArr = email.getMessages()
      .map(extractFieldsFromMessage)
      .filter(f => f);

    if (processEmail(email, messageFieldArr, budgetTypes.FIREFLY)) {
      email
        .removeLabel(pending)
        .moveToArchive();
    }
  })
}

/**
 * Processes a single email
 * @param email {GmailApp.GmailThread} Message to process
 * @param messageFieldArr {BudgetAutomation.TransactionFields[]} Fields for given email
 * @param budgetType {string} Type of account to process
 * @returns {boolean}
 */
function processEmail(email, messageFieldArr, budgetType) {
  let functToRun;
  let labelToAdd;
  switch (budgetType) {
    case budgetTypes.FIREFLY:
      if(email.getLabels().indexOf(fireflyDone) >= 0) {
        Logger.log('FireFly has already been processed. Skipping.');
        return true;
      }
      functToRun = processFirefly;
      labelToAdd = fireflyDone;
      break;
    default:
      Logger.log(`Unknown budget type '${budgetType}'`);
      return false;
  }

  const result = messageFieldArr
    .map(functToRun)
    .filter(res => res == false);
  if (result.length === 0) {
    email.addLabel(labelToAdd);
  }

  return result.length === 0;
}

/**
 * Processes a Firefly transaction
 * @param {BudgetAutomation.TransactionFields} Fiedls required to submit transaction
 */
function processFirefly({ firefly: { account }, merchant, amount }) {
  let partial = {}
  if (amount < 0) {
    // we are dealing with a withdrawal
    partial = {
      type: 'withdrawal',
      amount: amount * -1,
      source_id: account,
      destination_name: merchant.toUpperCase()
    };
  } else {
    // we are dealing with a deposit
    partial = {
      type: 'deposit',
      amount,
      source_name: merchant.toUpperCase(),
      destination_id: account
    };
  }

  Logger.log(`About to submit FireFly ('${partial.type}') transaction for '${merchant}' in the amount of '${amount}'`)
  const payload = {
    error_if_duplicate_hash: true,
    apply_rules: true,
    fire_webhooks: true,
    transactions: [{
      ...partial,
      date: new Date().toISOString(),
      description: 'Entered by Google Apps Script',
      order: 0
    }
  ]};

  return addFireflyTransaction(payload);

}
/**
 * Extracts required fields from email (Merchant, Amount, Account)
 * @param message {GmailApp.GmailMessage} Message to extract fields from
 * @returns {BudgetAutomation.TransactionFields}
 */
function extractFieldsFromMessage(message) {
    const accountName = Object.keys(accountsMap)
      .filter(a => accountsMap[a].match(message.getFrom()))
      .shift();
    if (!accountName) {
      Logger.log(`Failed find account for email from ${message.getFrom()}`);
      return false;
    }
    Logger.log(`Found account '${accountName}'`);

    const account = accountsMap[accountName];
    return account.fields(message.getBody());
}

/**
 * Submit transaction to Firefly server
 * @param transaction {Object} Transaction to submit
 * @returns {boolean} True if transaction was added; False otherwise
 */
function addFireflyTransaction(transaction) {
  const url = `${apiParts.endpoint}/transactions`;
  var options = {
    method: 'POST',
    payload: JSON.stringify(transaction),
    headers: apiParts.headers,
    muteHttpExceptions: true
  };
  const response = UrlFetchApp.fetch(url, options);
  const retVal = Math.round(response.getResponseCode() / 100) === 2
  if (!retVal) {
    console.log(response.getContentText());
  }

  return retVal;
}

Cron jobs

Based on the documentation, one of the suggested ways that tasks could be kicked off periodically was to create a new docker container that would be on the same network as Firefly container and would execute shell scripts using container's cron process.
Given that I want to backup my cron scripts, I created an NFS volume that points to a shared folder on my NAS with the following folder structure:

├── 15min
├── daily.hourly
├── monthly
└── weekly

Volume could be created in the following manner:

docker volume create --driver local \
  --opt type=nfs \
  --opt o=addr=[ip-address-of-your-nas],ro \
  --opt device=:[path-to-directory-containing-folders-above] \
  cron-jobs

From there, I can start my container like so:

docker run -d --restart=unless-stopped -v cron-jobs:/etc/periodic/ dimaj/alpine:tools crond -f -l 8

(image dimaj/alpine:tools is essentially an alpine image with jq, curl and bash:

FROM alpine
RUN apk add --no-cache curl jq bash

)

Now, anytime you want to execute a task on regular basis, you can simply add a shell script (with executable bit set) to any of the folders on your NAS and have these scripts get executed automatically.
Your imagination is your limiter as to what you can do there. As of right now, I have 2 scripts:

• Process-Rules - Runs all rule groups every 15 minutes so that my destination accounts would be properly renamed. I am using this to build out my rules library that would rename destination accounts.
• delete-accounts - As destination accounts get renamed from MERCHANT NAME STORE NUMBER 786 to Merchant Name, this script is going to clean up obsolete expense accounts that have their value set to 0.

Hope this is useful!

[JC5]

Wow, this is amazing work! Very cool! 😁 That's going into the docs for sure.

[dimaj]

Thanks!

[dimaj]

How do you feel about getting a repository of cron scripts going?

[JC5]

Not a fan, to be honest. I'm afraid that it will be my job to do upkeep and maintenance on the scripts. I'd rather collect links to GitHub Gists or repositories instead. With the code in people's own repositories, the responsibility to their maintenance falls to them.

There have been some questions on hosting Python scripts in the import configurations repository which i shot down for similar reasons.