BatchSD not publishing to Google Sheets

[BGoto808]

Describe the bug
BatchSD does not publish with Dendrometer code with both LTE and Ethernet usage. Dendrometer uses dirty integrated sensor values added to the json using the add_data() function. Through testing, it's found that whether or not the dirty integrated sensor values are added (whether add_data() is left in or commented out), the data will not publish through Batch. Dendrometer uses LoRa batch_send() to send data to a hub that either has an Ethernet module or an LTE board.

Hardware in Use
Dendrometer node

• Feather M0 LoRa
• HypnosV3.2
• Neopixel
• AS5311 (dirty integrated)
• SHT31D

Hub

• Feather M0 LoRa
• 4G board OR Ethernet module with HypnosV3.3

To Reproduce
Steps to reproduce the behavior:

1. Download Dendrometer node code and upload to dendrometer node, upload dendrometer hub code and upload to hub
2. Let node cycle 5 times before it sends through Batch
3. Watch serial monitor on hub side to see error messages with the getGoogleSheets(Feather).print_config() function

Expected behavior
Data should successfully upload to Google Sheets

Code
Node:

#include <Loom.h>
#include "AS5311.h"

// Include configuration
const char* json_config = 
#include "config.h"
;

// In Tools menu, set:
// Internet  > Disabled
// Sensors   > Enabled
// Radios    > Enabled
// Actuators > Enabled
// Max       > Disabled

using namespace Loom;

Loom::Manager Feather{};

//Pins
#define CS 9 // A3
#define CLK A5
#define DO A4
#define LED A2

#define DELAY_IN_SECONDS 5
#define DELAY_IN_MINUTES 0
#define INT_BUT 11 // A1
#define RTC_INT_PIN 12

#define HYPNOS3 5
//#define HYPNOS5 6
 
//Global Variables
volatile bool flag = false;   // Interrupt flag
volatile bool button = false; // Check to see if button was pressed
uint32_t start = 0;
uint32_t prevTwoSig = 0;
float elapsed = 0;
float prev = 0;
float prevMicro = 0;
const int max_packets = 5; // Number of packets collected before sending through LoRa
int counter = 0;

void setup() 
{
  // Enable waiting for RTC interrupt, MUST use a pullup since signal is active low
  pinMode(RTC_INT_PIN, INPUT_PULLUP);  
  pinMode(INT_BUT, INPUT_PULLUP);
  pinMode(HYPNOS3, OUTPUT);    // Enable control of 3.3V rail
  //pinMode(HYPNOS5, OUTPUT);   // Enable control of 5V rail
  pinMode(13, OUTPUT);

  // Initialize Hypnos
  digitalWrite(HYPNOS3, LOW); // Enable 3.3V rail
  //digitalWrite(HYPNOS5, HIGH);  // Enable 5V rail
  digitalWrite(13, HIGH);

  Feather.begin_serial(true);
  Feather.parse_config(json_config);
  Feather.print_config();

  // Begin Communication with AS5311
  init_AS();

  // LED pin set
  pinMode(LED, OUTPUT);

  // Make sure the magnet is positioned correctly
  //verify_position();
  
  // Flash three times for verification
  green_flash();
 
  getInterruptManager(Feather).register_ISR(RTC_INT_PIN, ISR_pin12, LOW, ISR_Type::IMMEDIATE);
  getInterruptManager(Feather).register_ISR(INT_BUT, ISR_pin11, LOW, ISR_Type::IMMEDIATE);
  delay(500);
  
  // Starting measurement of AS5311
  serial_init_measure();

  // Save 2 most significant bits of start
  prevTwoSig = start & 0xC00;
  
  getNeopixel(Feather).set_color(2, 0, 0, 0, 0); // Turns off Neopixel

  LPrintln("\n ** Setup Complete ** ");
}

void loop() 
{
  // Initialize Hypnos
  digitalWrite(HYPNOS3, LOW); // Enable 3.3V rail
  //digitalWrite(HYPNOS5, HIGH);  // Enable 5V rail
  digitalWrite(13, HIGH);

  delay(100);

  // Protocol to turn on AS5311
  pinMode(CS, OUTPUT);
  pinMode(CLK, OUTPUT);
  pinMode(DO, INPUT_PULLUP);
  digitalWrite(CS, HIGH);
  digitalWrite(CLK, LOW);

  // Protocol to turn on Neopixel
  pinMode(LED, OUTPUT);

  // Initialize magnet sensor  
  init_AS();
  delay(2000); // Warmup AS5311 chip

  if (flag) {
    pinMode(23, OUTPUT);
    pinMode(24, OUTPUT);
    pinMode(10, OUTPUT);
  
    Feather.power_up();
  }

  // Check to see if button was pressed for LED indicator
  verify_LED_button();

  //-------------------------------- DATA MANAGEMENT ----------------------------------------------
  int average = measure_average();
  uint32_t errorBits = getErrorBits(CLK, CS, DO);
  
  // Also updates prevTwoSig to two most significant bits of first param, is being passed by ref
  elapsed = computeElapsed(average, prevTwoSig, elapsed);

  // Computes total distance in mm and um
  float distance = (elapsed + ((2.0 * ((int)average - (int)start)) / 4095.0));
  float distanceMicro = (elapsed * 1000) + ((2000 * ((int)average - (int)start)) / 4095.0);
  float difference = 0;
  float differenceMicro = 0;

  // Reads the movement if any, else it sets the changed distance to 0
  if (distance != prev)
    difference = distance - prev;

  if (distanceMicro != prevMicro)
    differenceMicro = distanceMicro - prevMicro;

  Feather.measure();
  Feather.package();
/*
  Feather.add_data("AS5311", "Serial_Value", average);
  Feather.add_data("Displacement_mm", "mm", distance);
  Feather.add_data("Displacement_um", "um", distanceMicro);
  Feather.add_data("Difference_mm", "mm", difference);
  Feather.add_data("Difference_um", "um", differenceMicro);

  // Logs the status of the magnet position (whether the data is good or not) {Green = Good readings, Red = Bad readings}
  // Ignores the parity bit (last bit)
  
  if (errorBits >= 16 && errorBits <= 18)          // Error bits: 10000, 10001, 10010
    Feather.add_data("Status", "Color", "Green");
  else if (errorBits == 19)                        // Error bits: 10011
    Feather.add_data("Status", "Color", "Yellow");
  else if (errorBits == 23)                        // Error bits: 10111
    Feather.add_data("Status", "Color", "Red");
  else if (errorBits < 16)                         // If OCF Bit is 0
    Feather.add_data("Status", "Color", "OCF_Error");
  else if (errorBits > 24)                         // If COF Bit is 1
    Feather.add_data("Status", "Color", "COF_Error");
  else
    Feather.add_data("Status", "Color", "Other_Error");
  */
  float temp, humid, SVP, VPD;

  float e = 2.71828;

  temp = Feather.get<Loom::SHT31D>()->get_temperature();
  humid = Feather.get<Loom::SHT31D>()->get_humid();
  
  SVP = (0.61078 * pow(e, (17.2694 * temp) / (temp + 237.3)));
  VPD = SVP * (1 - (humid / 100));

  //Feather.add_data("VPD", "VPD", VPD);

  float rssi = Feather.get<Loom::LoRa>()->get_signal_strength();
  //Feather.add_data("RSSI", "RSSI", rssi);

  prev = distance;
  prevMicro = distanceMicro;
  //-----------------------------------------------------------------------------------

  Feather.display_data();

  // Log SD in case it doesn't send
  Feather.log_all();
  counter++;

  // Send to address 0 after 10 data packets
  if (counter % max_packets == 0) {
    getLoRa(Feather).send_batch(0,4000);
    counter = 0;
  } 

  getInterruptManager(Feather).RTC_alarm_duration(TimeSpan(0,0,DELAY_IN_MINUTES,DELAY_IN_SECONDS));
  getInterruptManager(Feather).reconnect_interrupt(RTC_INT_PIN);
  getInterruptManager(Feather).reconnect_interrupt(INT_BUT);

  Feather.power_down();

  // Protocol to shut down AS5311
  pinMode(CLK, INPUT);
  pinMode(DO, INPUT);
  pinMode(CS, INPUT);

  // Protocol to shut off Neopixel
  pinMode(LED, INPUT); 

  // Protocol to turn off Hypnos
  digitalWrite(13, LOW);
  digitalWrite(HYPNOS3, HIGH);
  //digitalWrite(HYPNOS5, LOW); 

  // Protocol to shut down SD
  pinMode(23, INPUT);
  pinMode(24, INPUT);
  pinMode(10, INPUT);
  
  getSleepManager(Feather).sleep();
  while (!flag);
}

//AS5311 functions
void init_AS(){
  // Protocol to turn on AS5311
  pinMode(CS, OUTPUT);
  pinMode(CLK, OUTPUT);
  pinMode(DO, INPUT_PULLUP);
  digitalWrite(CS, HIGH);
  digitalWrite(CLK, LOW);
  delay(2000);
}

int measure_average(){
  int average = 0;
  for (int j = 0; j < 16; j++)
  {
    average += getSerialPosition(CLK, CS, DO);
  }
  average /= 16;
  return average;
}

// Takes 16 measurements and averages them for the starting Serial value (0-4095 value)
void serial_init_measure(){
  for (int j = 0; j < 16; j++)
  {
    start += getSerialPosition(CLK, CS, DO);
  }
  start /= 16; 
}

// Lights up LED is interrupt button is pressed
void verify_LED_button(){
  if (button)
  {
    uint32_t ledCheck = getErrorBits(CLK, CS, DO);

    if (ledCheck >= 16 && ledCheck <= 18)
      getNeopixel(Feather).set_color(2, 0, 200, 0, 0); // Green
    else if (ledCheck == 19)
      getNeopixel(Feather).set_color(2, 0, 200, 200, 0); // Yellow
    else
      getNeopixel(Feather).set_color(2, 0, 0, 200, 0); // Red

    delay(3000);
    getNeopixel(Feather).set_color(2, 0, 0, 0, 0);
  }
  flag = false;
  button = false;
}

// Ensures that magnet starts in good position before continuing program
void verify_position(){
  uint32_t ledCheck = getErrorBits(CLK, CS, DO); // Tracking magnet position for indicator

  while (ledCheck < 16 || ledCheck > 18)
  {
    if (ledCheck == 19)
      getNeopixel(Feather).set_color(2, 0, 200, 200, 0); // Changes Neopixel to yellow
    else
      getNeopixel(Feather).set_color(2, 0, 0, 200, 0); // Changes Neopixel to red

    delay(3000); // Gives user 3 seconds to adjust magnet before next reading
    ledCheck = getErrorBits(CLK, CS, DO);
  }
}

// Flashes green on Neopixel 3 times
void green_flash(){
  for (int i = 0; i < 3; i++)
  {

    getNeopixel(Feather).set_color(2, 0, 200, 0, 0); // Changes Neopixel to green
    delay(500);

    getNeopixel(Feather).set_color(2, 0, 0, 0, 0); // Turns off Neopixel
    delay(500);
  }
}

// Interrupt Functions
void ISR_pin12()
{
  detachInterrupt(RTC_INT_PIN);
  flag = true;
}

void ISR_pin11()
{
  detachInterrupt(INT_BUT);
  flag = true;
  button = true;
}

Node (AS5311.h)

// Returns the serial output from AS533
uint32_t bitbang(int CLK, int CS, int DO) {
  // write clock high to select the angular position data
  digitalWrite(CLK, HIGH);
  delay(1);
  // select the chip
  digitalWrite(CS, LOW);
  delay(1);
  digitalWrite(CLK, LOW);
  // read the value in it's entirety
  uint32_t value = 0;
  for (uint8_t i = 0; i < 18; i++) {
    delay(1);
    digitalWrite(CLK, HIGH);

    if (i < 17) {
      delay(1);
      digitalWrite(CLK, LOW);
    }

    delay(1);
    auto readval = digitalRead(DO);
    if (readval == HIGH)
      value |= (1U << i);
  }
  digitalWrite(CS, HIGH);
  digitalWrite(CLK, HIGH);
  return value;
}

// Isolates the bottom 12 bits position value to decimal
uint32_t convertBits(uint32_t num) {
      uint32_t readval = num & 0xFFF;
      uint32_t newval = 0;
    // Flips bits order
      for (int i = 11; i >= 0; i--)
      {
        uint32_t exists = (readval & (1 << i)) ? 1 : 0;
        newval |= (exists << (11 - i));
      }
      return newval;
}

uint32_t getSerialPosition(int CLK, int CS, int DO){
  return convertBits(bitbang(CLK, CS, DO));
}

// Checks error bits
uint32_t bitCheck(uint32_t num) {
      uint32_t readval = num & 0x3FFFF; // Saves entire value; not sure if there's a better way to do this
      uint32_t newval = 0;
    // Flips bits order
      for (int i = 16; i >= 12; i--)
      {
        uint32_t exists = (readval & (1 << i)) ? 1 : 0;
        newval |= (exists << (16 - i));
      }
      return newval;
}

uint32_t getErrorBits(int CLK, int CS, int DO){
  return bitCheck(bitbang(CLK, CS, DO));
}

// Todo: Make more robust than just checking first two bits
float computeElapsed(uint32_t curr, uint32_t &prevTwoSig, float elapsed) {
  uint32_t currTwoSig = curr & 0xC00;
  if((currTwoSig == 0xC00 && prevTwoSig == 0x0)) {
    Serial.println("ROLLOVER UNDERFLOW");
    elapsed -= 2.0;
  } else if (prevTwoSig == 0xC00 && currTwoSig == 0x0) {
    Serial.println("ROLLOVER OVERFLOW");
    elapsed += 2.0;
  }
  prevTwoSig = currTwoSig;
  return elapsed;
}

Hub (LTE)

#include <Loom.h>

// Include configuration
const char* json_config =
#include "config.h"
;

// In Tools menu, set:
// Internet  > LTE
// Sensors   > Enabled
// Radios    > Enabled
// Actuators > Disabled
// Max       > Disabled

using namespace Loom;

Loom::Manager Feather{};

void setup() {
  
  pinMode(5, OUTPUT);
  digitalWrite(5, LOW); // Sets pin 5, the pin with the 3.3V rail, to output and enables the rail
  pinMode(6, OUTPUT);
  digitalWrite(6, HIGH); // Sets pin 6, the pin with the 5V rail, to output and enables the rail

  Feather.begin_serial(true);
  Feather.parse_config(json_config);
  Feather.print_config();

  LPrintln("\n ** Setup Complete ** ");
}

void loop() {
  
  //if (getLoRa(Feather).receive_blocking(5000)) {
  if (getLoRa(Feather).receive_batch_blocking(5000)) {
    Feather.display_data();
    getGoogleSheets(Feather).print_config();
    getGoogleSheets(Feather).publish_batch();
    getGoogleSheets(Feather).print_config();
  } 
}

Hub (Ethernet)

///////////////////////////////////////////////////////////////////////////////

// This is the simplest example of logging data to Google Sheets

// The only difference between this example an 'Basic' is the LoomFactory
// settings, the line:
//		Feather.GoogleSheets().publish();
// and the configuration, enabling logging to Google Sheets.

// In the config, you need:
// - MAC address for the Ethernet module (you could also replace Ethenet with WiFi)
//		You can use 'default' instead of a parameter list for Ethernet if you
//		are not on a network that restricts to only registered MAC addresses
// - For Google sheets parameters, see:
//   https://github.com/OPEnSLab-OSU/Loom/wiki/Using-Loom-with-Google-Sheets

///////////////////////////////////////////////////////////////////////////////

#include <Loom.h>

// Include configuration
const char* json_config =
#include "config.h"
;

// In Tools menu, set:
// Internet  > Ethernet
// Sensors   > Enabled
// Radios    > Disabled
// Actuators > Disabled
// Max       > Disabled

using namespace Loom;

Loom::Manager Feather{};


void setup()
{
  pinMode(5, OUTPUT);
  digitalWrite(5, LOW); // Sets pin 5, the pin with the 3.3V rail, to output and enables the rail
  pinMode(6, OUTPUT);
  digitalWrite(6, HIGH); // Sets pin 6, the pin with the 5V rail, to output and enables the rail
  
	Feather.begin_serial(true);
	Feather.parse_config(json_config);
	Feather.print_config();

	LPrintln("\n ** Setup Complete ** ");
}


void loop()
{
  if (getLoRa(Feather).receive_batch_blocking(5000)) {
  //if (getLoRa(Feather).receive_blocking(5000)) {
    Feather.display_data();
    getGoogleSheets(Feather).print_config();
    getGoogleSheets(Feather).publish_batch();
    getGoogleSheets(Feather).print_config();
  }
}

Config
Node

"{\
  'general':\
  {\
    'name':'Dendro',\
    'instance':1,\
    'interval':2000\
  },\
  'components':[\
    {\
      'name':'Analog',\
      'params':[8,12,false,false,false,false,false,false,0,0,0,0,0,0,25]\
    },\
    {\
      'name':'SHT31D',\
      'params':'default'\
    },\
    {\
      'name':'DS3231',\
      'params':[10, false, true]\
    },\
    {\
      'name':'InterruptManager',\
      'params':[0]\
    },\
    {\
      'name':'SleepManager',\
      'params':[true,false,1]\
    },\
    {\
      'name':'SD',\
      'params':[true,1000,10,'dend',true]\
    },\
    {\
      'name':'BatchSD',\
      'params':[true,1000,10]\
    },\
    {\
      'name':'Neopixel',\
      'params':'default'\
    },\
    {\
      'name':'LoRa',\
      'params':[255,1,23,7,500]\
    }\
  ]\
}"

Hub (LTE)

"{\
	'general':\
	{\
		'device_name':'Device',\
		'instance_num':1,\
		'interval':3000\
	},\
	'components':[\
		{\
     'name':'LTE',\
     'params':['hologram','','','A5']\
		},\
    {\
     'name':'SD',\
     'params':[true,1000,10,'hub',true]\
    },\
    {\
     'name':'BatchSD',\
     'params':[true,1000,10]\
    },\
    {\
     'name':'DS3231',\
     'params':'default'\
    },\
    {\
     'name':'LoRa',\
     'params':[255,0,23,7,500]\
    },\
		{\
			'name':'GoogleSheets',\
			'params':[\
				'Goog',\
        '/macros/s/AKfycbzySpouxdaHYh6f1e7DI24i4s8XTGDe-X6d-9uaR7HVrpFNWUt7/exec',\
        '11bmZETLyFutZHwZRrpmsNFy0VrWmJ-myskjLZ6cZS-w',\
/*true to autoname tab*/				true,\
/*not used if previous param is true*/	'testTab'\
			]\
		}\
	]\
}"

Hub (Ethernet)

"{\
	'general':\
	{\
		'name':'Device',\
		'instance':1,\
		'interval':10000\
	},\
	'components':[\
    {\
      'name':'LoRa',\
      'params':[255,0,23,3,200]\
    },\
    {\
     'name':'SD',\
     'params':[true,1000,11,'hub',true]\
    },\
    {\
     'name':'BatchSD',\
     'params':[true,1000,11]\
    },\
    {\
      'name':'GoogleSheets',\
      'params':[\
        'Goog',\
        '/macros/s/AKfycbzySpouxdaHYh6f1e7DI24i4s8XTGDe-X6d-9uaR7HVrpFNWUt7/exec',\
        '11bmZETLyFutZHwZRrpmsNFy0VrWmJ-myskjLZ6cZS-w',\
/*true to autoname tab*/        true,\
/*not used if previous param is true*/  'testTab'\
      ]\
    },\
    {\
      'name':'Ethernet',\
      'params':[\
        'Ether1',\
        [134,171,186,10,33,221],\
        [192,168,0,1]\
      ]\
    }\
	]\
}"

Additional context
LTE Error Message:
Note: Whenever I get this LTE issue, I need to reupload the code back to the hub. FeatherFault will prevent it from booting up normally after getting this message.

Ethernet Error Messages:

[udellc]

Hi Bryson,
To clarify: Does the bug only happen if Batch LoRa transmit is used AND the dirty integrated sensor putting data into the Node's transmit packet?

Have you tested this using Batch Transmit and not pushing dirty integrated sensor data into the Node transmit packet?
If so, please describe results as this is needed information.

Have you tested uploading code to Google sheets using a dirty integrated sensor on the Hub side? e.g. no nodes. Just a hub taking Loom integrated and non-loom integrated data into a packet and uploading to Google sheets?

Please test these and respond with behavior.

[udellc]

We can't forget this is still something that needs fixing.

[udellc]

Does the hub log received batch data to SD at least? Or is SD logging having an issue there as well?