IMPORTANT UPGRADE NOTICE #82

333miiko · 2018-07-17T05:12:52Z

IMPORTANT NOTES FOR UPGRADING

The securenode tracker uses a new configuration meaning if you upgrade through the normal restart process it will break.

There are two options for fixing your config:

Re-run the installer again, this will re-generate the new config which is compatible with the latest version (remember to grab the latest version first)
Update the config manually, remember to replace all the variables starting with $

cat << EOF > /mnt/zen/secnode/config.json
{
 "active": "secure",
 "secure": {
  "nodetype": "secure",
  "nodeid": null,
  "servers": [
   "ts2.eu",
   "ts1.eu",
   "ts3.eu",
   "ts4.eu",
   "ts4.na",
   "ts3.na",
   "ts2.na",
   "ts1.na"
  ],
  "stakeaddr": "$stakeaddr",
  "email": "$email",
  "fqdn": "$fqdn",
  "ipv": "4",
  "region": "$region",
  "home": "ts1.$region",
  "category": "none"
 }
}
EOF

The text was updated successfully, but these errors were encountered:

phobos777 · 2018-07-17T10:03:58Z

see my discord PM's to you. most notable last remark

Spiral11 · 2018-07-17T16:50:25Z

I used the second method and the tracker log was outputting this error:

2018-07-17 16:42:53 UTC -- connect ENETUNREACH 127.0.0.0:80 - Local (0.0.0.0:0)

Fix was to stop zend, edit zen.conf and change
rpcallowip=127.0.0.0/24
to
rpcallowip=127.0.0.1

Not really sure how this error is related to the tracker but it happened right after I restarted it so if anyone else encounters it, there's the solution.

sloppycoffee · 2018-07-17T21:06:03Z

I used your new install script and I saw it created config.json but in the logs I am getting this error:

Jul 17 18:48:34 mn1 systemd[1]: Started Zen Secnodetracker Container.
Jul 17 18:48:35 mn1 docker[9370]: Secure node config found OK - linking...
Jul 17 18:48:36 mn1 docker[9370]: Retrieving zen rpc config....
Jul 17 18:48:36 mn1 docker[9370]: /home/node/secnodetracker/app.js:23
Jul 17 18:48:36 mn1 docker[9370]: if (config.ipv === '6') {
Jul 17 18:48:36 mn1 docker[9370]: ^
Jul 17 18:48:36 mn1 docker[9370]: TypeError: Cannot read property 'ipv' of undefined
Jul 17 18:48:36 mn1 docker[9370]: at Object. (/home/node/secnodetracker/app.js:23:12)
Jul 17 18:48:36 mn1 docker[9370]: at Module._compile (module.js:652:30)
Jul 17 18:48:36 mn1 docker[9370]: at Object.Module._extensions..js (module.js:663:10)
Jul 17 18:48:36 mn1 docker[9370]: at Module.load (module.js:565:32)
Jul 17 18:48:36 mn1 docker[9370]: at tryModuleLoad (module.js:505:12)
Jul 17 18:48:36 mn1 docker[9370]: at Function.Module._load (module.js:497:3)
Jul 17 18:48:36 mn1 docker[9370]: at Function.Module.runMain (module.js:693:10)
Jul 17 18:48:36 mn1 docker[9370]: at startup (bootstrap_node.js:191:16)
Jul 17 18:48:36 mn1 docker[9370]: at bootstrap_node.js:612:3

phobos777 · 2018-07-17T23:02:02Z

@333miiko i added some more information with regard to Challenges when your back online. it's cause they moved the Nodetracker (of course), so challenges are not running.

phobos777 · 2018-07-17T23:02:31Z

i commented on Discord i should of said. :)

sloppycoffee · 2018-07-17T23:09:24Z

I noticed that a directory called "local" was created in /mnt/zen/secnode. Is that correct?

Tried many different things but all still come back to the error listed about about config.piv. Not sure how to get around it.

333miiko · 2018-07-18T01:05:13Z

@slayorktc you should have a file called config.json in that folder. Did that not get created?

sloppycoffee · 2018-07-18T01:05:51Z

Yes. There is config.json in that folder.

333miiko · 2018-07-18T01:06:16Z

@phobos777 regarding challenges you can use these steps to try restart the challenge process:

apt-get update && apt-get install jq

Now use this bash file to manually trigger a new challenge:

#!/bin/bash
echo "Starting Testnet Node Challenge Benchmark...time to completion: 15 minutes"
for i in {1..3}; do echo -n "Running Challenge - Wait 300s - Test $i of 3 - Node Status: "; \
curl -s "https://$(cat /mnt/zen/secnode/config.json | jq -r '.testnet.home').zensystem.io/$(docker exec -it zen-node gosu user zen-cli listaddresses | jq -r '.[0]')/$(cat /mnt/zen/secnode/config.json | jq -r '.testnet.nodeid')/send"; \
date -u +' - %Y-%m-%d %H:%M:%S UTC'; sleep 300; done
echo "Benchmark Complete. Check results at:"
echo "https://$(cat /mnt/zen/secnode/config.json | jq -r '.testnet.home').zensystem.io/nodes/$(cat /mnt/zen/secnode/config.json | jq -r '.testnet.nodeid')/chals

333miiko · 2018-07-18T01:06:36Z

@slayorktc could you post the content of config.json (redact your email, hostname and stakeaddr)

sloppycoffee · 2018-07-18T01:08:24Z

{
"active": "",
"secure": {
"nodetype": "",
"nodeid": null,
"servers": [
"ts2.eu",
"ts1.eu",
"ts3.eu",
"ts4.eu",
"ts4.na",
"ts3.na",
"ts2.na",
"ts1.na"
],
"stakeaddr": "x",
"email": "x",
"fqdn": "snode1.convergegroup.tech",
"ipv": "4",
"region": "na",
"home": "ts1.na",
"category": "none"
}
}

Replaced those two with x

333miiko · 2018-07-18T01:10:04Z

Did you use the new parameter at the end of the installer to define it as a secure node?

sloppycoffee · 2018-07-18T01:11:38Z

new parameter? I don't think the install.sh asked for that
I did pop into the container and run setup.js which I completed successfully but the error was still present.

333miiko · 2018-07-18T01:14:02Z

The error seems to be because it can't read the config of what ip format you are using. Did you try restart the service in case you didn't get the most latest image?

sloppycoffee · 2018-07-18T01:14:48Z

Yup, systemctl restart both zen-node and zen-secnodetracker. Multiple reboots. Nothing seemed to fix it..
There is a local directory inside secnode dir. Not sure if that is normal or not.

333miiko · 2018-07-18T01:18:17Z

The local directory is generated by node config

Could you try in the host server clear all the files in /mnt/zen/secnode/

Then try run:

stakeaddr=x
email=x
fqdn=x
region=na

cat << EOF > /mnt/zen/secnode/config.json
{
 "active": "secure",
 "secure": {
  "nodetype": "secure",
  "nodeid": null,
  "servers": [
   "ts2.eu",
   "ts1.eu",
   "ts3.eu",
   "ts4.eu",
   "ts4.na",
   "ts3.na",
   "ts2.na",
   "ts1.na"
  ],
  "stakeaddr": "$stakeaddr",
  "email": "$email",
  "fqdn": "$fqdn",
  "ipv": "4",
  "region": "$region",
  "home": "ts1.$region",
  "category": "none"
 }
}
EOF

Now restart the zen-secnodetracker, obviously replace x with the appropriate variables.

sloppycoffee · 2018-07-18T01:21:05Z

I got this error: Jul 18 03:20:30 mn1 docker[11549]: No secure node config found. exiting
Jul 18 03:20:30 mn1 systemd[1]: zen-secnodetracker.service: Main process exited, code=exited, status=1/FAILURE
Jul 18 03:20:30 mn1 systemd[1]: zen-secnodetracker.service: Unit entered failed state.
Jul 18 03:20:30 mn1 systemd[1]: zen-secnodetracker.service: Failed with result 'exit-code'.
Jul 18 03:20:30 mn1 systemd[1]: zen-secnodetracker.service: Service hold-off time over, scheduling restart.
Jul 18 03:20:30 mn1 systemd[1]: Stopped Zen Secnodetracker Container.
Jul 18 03:20:30 mn1 systemd[1]: zen-secnodetracker.service: Start request repeated too quickly.
Jul 18 03:20:30 mn1 systemd[1]: Failed to start Zen Secnodetracker Container.

sloppycoffee · 2018-07-18T01:21:38Z

root@mn1:/mnt/zen/secnode# pwd
/mnt/zen/secnode
root@mn1:/mnt/zen/secnode# ls
config.json
root@mn1:/mnt/zen/secnode#

sloppycoffee · 2018-07-18T01:51:43Z

Outside the container. Ran it sitting inside /mnt/zen/secnode

333miiko · 2018-07-18T01:52:43Z

systemctl stop zen-secnodetracker
docker run --rm -v /mnt/zen:/mnt/zen --name secnode -it --entrypoint=/bin/bash whenlambomoon/secnodetracker 
cd /home/node/secnodetracker
ls config
node setup
node app

sloppycoffee · 2018-07-18T01:53:37Z

cd /home/node/nodetracker -- Didn't exist but did you mean 'secnodetracker'?

Also no config file in that dir

sloppycoffee · 2018-07-18T01:55:25Z

Configuration for super node saved. Setup complete!
root@46306da1ec86:/home/node/secnodetracker# node app
Retrieving zen rpc config....
STARTING NODETRACKER
CPU Intel(R) Xeon(R) CPU D-1521 @ 2.40GHz count=4 speed=2399
Node.js version: v8.11.3
Tracker app version: 0.3.0
Node type: super
MemTotal: 7.98GB MemFree: 4.48GB MemAvailable: 6.70GB SwapTotal: 8.19GB SwapFree: 8.19GB
2018-07-18 01:55:00 UTC -- Connected to server xns1.eu. Initializing...
2018-07-18 01:55:00 UTC -- ERROR zend get t-address
2018-07-18 01:55:00 UTC -- connect ENETUNREACH 127.0.0.0:80 - Local (0.0.0.0:0)
2018-07-18 01:55:10 UTC -- ERROR zend get t-address
2018-07-18 01:55:10 UTC -- connect ENETUNREACH 127.0.0.0:80 - Local (0.0.0.0:0)

333miiko · 2018-07-18T02:12:35Z

Yep sorry /home/node/secnodetracker there would be a config folder in there now

Now the last issue is you need to exit out and modify /mnt/zen/config/zen.conf

and change the line that starts with rpcallowip to be rpcallowip=127.0.0.1

sloppycoffee · 2018-07-18T02:16:42Z

Okay. Getting closer. I know have this error:

Jul 18 04:16:02 mn1 docker[19097]: External IP address (externalip=) not found in zen.conf. At least one (IPv4 or IP
Jul 18 04:16:02 mn1 docker[19097]: If multiple, add the externalip= for each address on a separate line.
Jul 18 04:16:03 mn1 systemd[1]: zen-secnodetracker.service: Service hold-off time over, scheduling restart

sloppycoffee · 2018-07-18T02:25:08Z

I added externalip=VPS IP
port=9033

Running the zen-secnodetracker is still producing the same ipv === error.

Inside the container, running node setup.js, then node app.js, now produces this message:

018-07-18 02:24:09 UTC -- connect ECONNREFUSED 127.0.0.1:18231
2018-07-18 02:24:19 UTC -- ERROR zend get t-address
2018-07-18 02:24:19 UTC -- connect ECONNREFUSED 127.0.0.1:18231

sloppycoffee · 2018-07-18T02:28:43Z

Well, I think i finally got it up and running.. I think I needed to change these 3 lines:

{
"active": "super",
"super": {
"nodetype": "super",

After the other changes I stated above about externalip and port to zen.conf. Seems to be working now.

Thanks for the all the help! Ill toss some zen your way!

333miiko · 2018-07-18T03:03:28Z

Glad you got it working! I don't think externalip is required, that may have just been due to the earlier rpc issues.

Let me know if you have any other issues.

phobos777 · 2018-07-18T10:36:52Z

@333miiko on reboot of a secure or supernode; the securenode tracker is now requiring one 'externalip' for secure and two external ip (ipv4 and ipv6 for supers). here is an output of the secnode tracker. (i also go a new email this morning from the zen dev team)

root@localhost:~# docker logs -f zen-secnodetracker
Secure node config found OK - linking...
Retrieving zen rpc config....
External IP address (externalip=) not found in zen.conf. At least one (IPv4 or IPv6) required for secure nodes. Both IPv4 and IPv6 required for super nodes.
If multiple, add the externalip= for each address on a separate line.

phobos777 · 2018-07-18T11:36:12Z

**and add port while your there please: LOL:

root@localhost:# docker logs -f zen-secnodetracker
Secure node config found OK - linking...
Retrieving zen rpc config....
Port not found in zen.conf. Add 'port=9033' for mainnet or 'port=19033' for testnet
root@localhost:#**

sloppycoffee · 2018-07-18T19:29:46Z

Agreed. externalip=
port=
change 127.0.0.1

How do I deal with having ipv6 and ipv4? Do I need to have two certs? Not exactly sure how to get that working correctly..

333miiko · 2018-07-20T02:46:35Z

It's not yet finished building, wait for it here https://hub.docker.com/r/whenlambomoon/zend/builds/

sloppycoffee · 2018-07-20T02:47:07Z

Phew.. Was just going to say it didn't work still.. Still a chance.. If this doesn't fix it, I have no clue...

sloppycoffee · 2018-07-20T02:49:41Z

Build "ERROR"

333miiko · 2018-07-20T03:08:22Z

@slayorktc build finished

sloppycoffee · 2018-07-20T03:12:10Z

Weird. I now have this error:
2018-07-20 03:10:59 UTC -- connect ECONNREFUSED 127.0.0.1:18231

I wasn't having that one for awhile.. strange.

333miiko · 2018-07-20T03:13:52Z

Is your zend container running?

sloppycoffee · 2018-07-20T03:19:14Z

Oh. This is why:

/usr/local/bin/entrypoint.sh: line 82: syntax error: unexpected end of file

sloppycoffee · 2018-07-20T03:21:11Z

Are you missing an fi on line 59?

sloppycoffee · 2018-07-20T03:22:02Z

Actually 62-69 not sure which if goes with which fi.. Somewhere in there I think there is a mismatch

333miiko · 2018-07-20T03:23:38Z

https://hub.docker.com/r/whenlambomoon/zend/builds/

sloppycoffee · 2018-07-20T03:36:50Z

Well, I don't know.. Still getting the cert false error.. Did you get it working 33miiko on your vps?

I really have no clue what is causing the issue now..
Jul 20 04:33:25 zen docker[24101]: 2018-07-20 03:33:25 UTC -- Cert check: valid=false. unable to verify the first certificate

333miiko · 2018-07-20T03:40:40Z

Can you paste me the log output of your zen-node and zen-secnodetracker

sloppycoffee · 2018-07-20T03:42:01Z

Logs begin at Fri 2018-07-20 04:39:48 BST, end at Fri 2018-07-20 04:41:28 BST. --
Jul 20 04:42:13 zen systemd[1]: Starting Zen Daemon Container...
Jul 20 04:42:13 zen docker[1517]: Error response from daemon: No such container: zen-node
Jul 20 04:42:13 zen docker[1538]: Error response from daemon: No such container: zen-node
Jul 20 04:42:15 zen docker[1561]: latest: Pulling from whenlambomoon/zend
Jul 20 04:42:15 zen docker[1561]: Digest: sha256:5e57f35d4d8e9482633f74bf60396eb6841023179ffd5658c732a57804ee3939
Jul 20 04:42:15 zen docker[1561]: Status: Image is up to date for whenlambomoon/zend:latest
Jul 20 04:42:15 zen systemd[1]: Started Zen Daemon Container.
Jul 20 04:42:17 zen docker[1624]: Starting with UID/GID : 9001/9001
Jul 20 04:42:17 zen docker[1624]: Symlinking external zcash-params volume...
Jul 20 04:42:17 zen docker[1624]: Using external data volume
Jul 20 04:42:17 zen docker[1624]: Copying additional trusted SSL certificates
Jul 20 04:42:17 zen docker[1624]: Clearing symlinks in /etc/ssl/certs...
Jul 20 04:42:19 zen docker[1624]: done.
Jul 20 04:42:19 zen docker[1624]: Updating certificates in /etc/ssl/certs...
Jul 20 04:42:24 zen docker[1624]: 149 added, 0 removed; done.
Jul 20 04:42:24 zen docker[1624]: Running hooks in /etc/ca-certificates/update.d...
Jul 20 04:42:24 zen docker[1624]: done.
Jul 20 04:42:24 zen docker[1624]: Zcash - fetch-params.sh
Jul 20 04:42:24 zen docker[1624]: Zen uses the same zkSNARK parameters generated by ZCash.
Jul 20 04:42:24 zen docker[1624]: This script will fetch the Zcash zkSNARK parameters and verify their
Jul 20 04:42:24 zen docker[1624]: integrity with sha256sum.
Jul 20 04:42:24 zen docker[1624]: If they already exist locally, it will exit now and do nothing else.
Jul 20 04:42:24 zen docker[1624]: Starting zend ..

04:42:30 zen docker[1618]: STARTING NODETRACKER
Jul 20 04:42:30 zen docker[1618]: CPU Intel(R) Xeon(R) CPU X5670 @ 2.93GHz count=2 speed=2933
Jul 20 04:42:30 zen docker[1618]: Node.js version: v8.11.3
Jul 20 04:42:30 zen docker[1618]: Tracker app version: 0.3.1
Jul 20 04:42:30 zen docker[1618]: Node type: secure
Jul 20 04:42:30 zen docker[1618]: Node Id: 26125
Jul 20 04:42:30 zen docker[1618]: MemTotal: 3.95GB MemFree: 1.21GB MemAvailable: 3.28GB SwapTotal: 3.07GB SwapFree: 3.07GB
Jul 20 04:42:30 zen docker[1618]: 2018-07-20 03:42:30 UTC -- Connected to server ts1.eu. Initializing...
Jul 20 04:42:30 zen docker[1618]: 2018-07-20 03:42:30 UTC -- Zend: Waiting - Loading block index...
Jul 20 04:40:26 zen docker[1618]: 2018-07-20 03:40:26 UTC -- Zend: Waiting - Loading block index...
Jul 20 04:40:36 zen docker[1618]: 2018-07-20 03:40:36 UTC -- Zend: Waiting - Loading block index...
Jul 20 04:40:46 zen docker[1618]: 2018-07-20 03:40:46 UTC -- Zend: Waiting - Verifying blocks...
Jul 20 04:40:56 zen docker[1618]: 2018-07-20 03:40:56 UTC -- Zend: Waiting - Verifying blocks...
Jul 20 04:41:06 zen docker[1618]: 2018-07-20 03:41:06 UTC -- Zend: Waiting - Verifying blocks...
Jul 20 04:41:16 zen docker[1618]: 2018-07-20 03:41:16 UTC -- Zend: Waiting - Loading wallet...
Jul 20 04:41:26 zen docker[1618]: Node t_address (not for stake)=zndLB1RWYn6XpiYUF82BXjqsEF6mvo2Qhxd
Jul 20 04:41:26 zen docker[1618]: Checking private z-addresses...
Jul 20 04:41:26 zen docker[1618]: Balance for challenge transactions is 0.9796
Jul 20 04:41:26 zen docker[1618]: Using the following address for challenges
Jul 20 04:41:26 zen docker[1618]: zcMpuJHxhtc1eP5orrvUFLT5DYdX3nWZkFgNX68PbfCrN1fKw3k4Bqyi5XXn9mYxer864ymG4Q2fibv8v5AxQ54jKxGcdY
Jul 20 04:41:27 zen docker[1618]: 2018-07-20 03:41:27 UTC -- Authenticated. Nice to see you connected again.
Jul 20 04:41:27 zen docker[1618]: 2018-07-20 03:41:27 UTC -- Updated server list
Jul 20 04:41:27 zen docker[1618]: 2018-07-20 03:41:27 UTC -- Stats: send initial stats.
Jul 20 04:41:27 zen docker[1618]: 2018-07-20 03:41:27 UTC -- Saved servers=ts1.eu,ts2.eu,ts3.eu,ts4.eu,ts1.na,ts2.na,ts3.na,ts4.
Jul 20 04:41:27 zen docker[1618]: 2018-07-20 03:41:27 UTC -- Stats received by ts1.eu
Jul 20 04:41:27 zen docker[1618]: 2018-07-20 03:41:27 UTC -- Cert check: valid=false. unable to verify the first certificate
Jul 20 04:41:28 zen docker[1618]: 2018-07-20 03:41:28 UTC -- Down time logged. See server.
Jul 20 04:41:28 zen docker[1618]: 2018-07-20 03:41:28 UTC -- Down time logged. See server.
Jul 20 04:41:28 zen docker[1618]: 2018-07-20 03:41:28 UTC -- Down time logged. See server.

333miiko · 2018-07-20T03:59:54Z

Is there any chance your cert may have already expired? Could you try restart the acme.sh container then restart the node containers to see if it fixes the problem?

Not seeing this happen

sloppycoffee · 2018-07-20T04:11:49Z

I have regenerated the cert multiple times. Your nodes aren't having any cert issues?

sloppycoffee · 2018-07-20T06:20:29Z

Well, I just deleted everything and ran the install script again from scratch and the cert worked.. No clue what got messed up with the old setup.. Thanks for the help.

333miiko · 2018-07-20T06:30:06Z

I didn't have any cert issues.

@phobos777 is your certs ok?

sloppycoffee · 2018-07-20T07:32:08Z

Phew. After some crazy commands and a couple scripts I got everything back up and running. Just a FYI.. I didn't know this but I had like 10gigs worth of old images just laying around..

You can remove untagged images by running:

docker images --no-trunc | grep '' | awk '{ print $3 }'
| xargs -r docker rmi

333miiko · 2018-07-21T08:04:35Z

Yes each time you update the old image won't be removed. As mentioned earlier I'll look into implementing certbot and also include the auto update and image clearing when I'm back.

darkoopa · 2018-07-26T16:12:38Z

Hello,
I had to reboot a VPS and the node didn't connect anymore, I looked at it : "no such container: zen secnodetracker".

So I supposed it comes from this problem. I tried to manually update the config, it created the config.json but the node is still down.
when i try docker exec zen-node gosu user zen-cli getinfo, I get "Secure node config found OK - linking..." but that's all, I don't have the follow up :/

I have this:
{
"active": "secure",
"secure": {
"nodetype": "secure",
"nodeid": null,
"servers": [
"ts2.eu",
"ts1.eu",
"ts3.eu",
"ts4.eu",
"ts4.na",
"ts3.na",
"ts2.na",
"ts1.na"
],
"stakeaddr": "plop",
"email": "plop",
"fqdn": "znode2.darkoopa.pw",
"ipv": "4",
"region": "na",
"home": "ts3.na",
"category": "none"
}
}

I tried to change the home to ts3.na and the region to na as that apparently where it supposed to be https://securenodes4.eu.zensystem.io/nodes/9007/detail but normally the OVH server is in france...

I tried with region eu and hone ts1.eu but it was the same.

thx

333miiko · 2018-07-26T16:17:58Z

Did you restart the node after you updated the config?

darkoopa · 2018-07-26T16:25:00Z

yeah :D I'm really bad at linux but i know to always try to reboot ^^

333miiko · 2018-07-26T16:28:41Z

Could you try re-run the installer again to see if it fixes your issues?

333miiko · 2018-07-26T16:29:16Z

Remember to grab the latest version:

curl -O https://raw.githubusercontent.com/WhenLamboMoon/docker-zen-node/master/install.sh
chmod +x install.sh
./install.sh <stakeaddr> <email> <fqdn> <region> <nodetype>

darkoopa · 2018-07-26T16:29:52Z

it will create a new shielded address right?

333miiko · 2018-07-26T16:31:22Z

No, if one exists it will not. It's safe to re-run the installer multiple times

darkoopa · 2018-07-26T16:32:37Z

ah ok, I was affraid of that, that's why I didn't do it. i'll try, thx

darkoopa · 2018-07-26T17:03:06Z

It worked, thx.
I suppose I can take back 0.95 ZEN from the shielded address as apparently now yoçu only need 0.05?

lutzk87 · 2018-08-10T18:59:33Z

Hey, can someone provide a little tutorial for upgrading the nodetracker to the latest version 0.3.1? i used this tutorial for the secure node because iam a real noob with linux. thanks!

333miiko · 2018-08-11T04:45:35Z

As mentioned above, re-running the installer above is the best way to upgrade:

curl -O https://raw.githubusercontent.com/WhenLamboMoon/docker-zen-node/master/install.sh
chmod +x install.sh
./install.sh <stakeaddr> <email> <fqdn> <region> <nodetype>

IMPORTANT UPGRADE NOTICE #82

IMPORTANT UPGRADE NOTICE #82

Comments

333miiko commented Jul 17, 2018 • edited Loading

phobos777 commented Jul 17, 2018

Spiral11 commented Jul 17, 2018

sloppycoffee commented Jul 17, 2018 • edited Loading

phobos777 commented Jul 17, 2018

phobos777 commented Jul 17, 2018

sloppycoffee commented Jul 17, 2018 • edited Loading

333miiko commented Jul 18, 2018

sloppycoffee commented Jul 18, 2018

333miiko commented Jul 18, 2018

333miiko commented Jul 18, 2018

sloppycoffee commented Jul 18, 2018 • edited Loading

333miiko commented Jul 18, 2018

sloppycoffee commented Jul 18, 2018

333miiko commented Jul 18, 2018

sloppycoffee commented Jul 18, 2018

333miiko commented Jul 18, 2018 • edited Loading

sloppycoffee commented Jul 18, 2018

sloppycoffee commented Jul 18, 2018

sloppycoffee commented Jul 18, 2018

333miiko commented Jul 18, 2018 • edited Loading

sloppycoffee commented Jul 18, 2018 • edited Loading

sloppycoffee commented Jul 18, 2018

333miiko commented Jul 18, 2018

sloppycoffee commented Jul 18, 2018

sloppycoffee commented Jul 18, 2018

sloppycoffee commented Jul 18, 2018

333miiko commented Jul 18, 2018

phobos777 commented Jul 18, 2018

phobos777 commented Jul 18, 2018 • edited Loading

sloppycoffee commented Jul 18, 2018

333miiko commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018

333miiko commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018

333miiko commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018

333miiko commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018

333miiko commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018

333miiko commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018

333miiko commented Jul 20, 2018

sloppycoffee commented Jul 20, 2018 • edited Loading

333miiko commented Jul 21, 2018

darkoopa commented Jul 26, 2018 • edited Loading

333miiko commented Jul 26, 2018

darkoopa commented Jul 26, 2018

333miiko commented Jul 26, 2018

333miiko commented Jul 26, 2018 • edited Loading

darkoopa commented Jul 26, 2018

333miiko commented Jul 26, 2018

darkoopa commented Jul 26, 2018

darkoopa commented Jul 26, 2018 • edited Loading

lutzk87 commented Aug 10, 2018

333miiko commented Aug 11, 2018

333miiko commented Jul 17, 2018 •

edited

Loading

sloppycoffee commented Jul 17, 2018 •

edited

Loading

sloppycoffee commented Jul 17, 2018 •

edited

Loading

sloppycoffee commented Jul 18, 2018 •

edited

Loading

333miiko commented Jul 18, 2018 •

edited

Loading

333miiko commented Jul 18, 2018 •

edited

Loading

sloppycoffee commented Jul 18, 2018 •

edited

Loading

phobos777 commented Jul 18, 2018 •

edited

Loading

sloppycoffee commented Jul 20, 2018 •

edited

Loading

darkoopa commented Jul 26, 2018 •

edited

Loading

333miiko commented Jul 26, 2018 •

edited

Loading

darkoopa commented Jul 26, 2018 •

edited

Loading