Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to ping the other peer #47

Open
saket424 opened this issue Nov 15, 2017 · 26 comments
Open

Unable to ping the other peer #47

saket424 opened this issue Nov 15, 2017 · 26 comments

Comments

@saket424
Copy link

I followed the wonderful instructions in the demo https://screencast-o-matic.com/watch/cbjXbPlmdl with the latest ipop-vpn_rel17.08(ubuntu)

However, when I try it the individual ipop_tap0 interfaces do come up on the respective hosts but they are unable to discover each other and ping each other

Here are the controller logs and it stays stuck in the searching state and does not find the other peer

Any ideas?? I am using my own xmpp server if that matters

root@mike-desktop:/home/mike/tincan-stuff/ipop-vpn_rel17.08(ubuntu)/logs# head ctrl.log
[20171114 22:32:17.420] INFO:TincanInterface: TincanInterface Loaded
[20171114 22:32:17.425] INFO:TincanInterface: Creating Tincan control response link
[20171114 22:32:17.426] INFO:TincanInterface: Setting Tincan log level to DEBUG
[20171114 22:32:17.432] INFO:TincanInterface: Creating Vnet ipop_tap0
[20171114 22:32:17.433] INFO:TincanInterface: Ignoring interfaces [u'ipop_tap0']
[20171114 22:32:17.434] INFO:LinkManager: LinkManager Loaded
[20171114 22:32:17.434] INFO:XmppClient: Key-ring module not installed.
[20171114 22:32:17.435] DEBUG:XmppClient: Started XMPP handling
[20171114 22:32:17.435] INFO:XmppClient: XmppClient module Loaded
[20171114 22:32:17.435] INFO:ArpCache: ArpCache Loaded
root@mike-desktop:/home/mike/tincan-stuff/ipop-vpn_rel17.08(ubuntu)/logs# head -50 ctrl.log
[20171114 22:32:17.420] INFO:TincanInterface: TincanInterface Loaded
[20171114 22:32:17.425] INFO:TincanInterface: Creating Tincan control response link
[20171114 22:32:17.426] INFO:TincanInterface: Setting Tincan log level to DEBUG
[20171114 22:32:17.432] INFO:TincanInterface: Creating Vnet ipop_tap0
[20171114 22:32:17.433] INFO:TincanInterface: Ignoring interfaces [u'ipop_tap0']
[20171114 22:32:17.434] INFO:LinkManager: LinkManager Loaded
[20171114 22:32:17.434] INFO:XmppClient: Key-ring module not installed.
[20171114 22:32:17.435] DEBUG:XmppClient: Started XMPP handling
[20171114 22:32:17.435] INFO:XmppClient: XmppClient module Loaded
[20171114 22:32:17.435] INFO:ArpCache: ArpCache Loaded
[20171114 22:32:17.436] INFO:BaseTopologyManager: BaseTopologyManager Loaded
[20171114 22:32:17.436] DEBUG:BaseTopologyManager: BTM Table::{'discovered_nodes': [], 'p2p_state': 'started', 'uid_mac_table': {}, 'ipop_state': {}, 'ip_uid_table': {}, 'xmpp_client_code': u'XmppClient', 'peer_uid_sendmsgcount': {}, 'GeoIP': '', 'mac_uid_table': {}, 'link_type': {}, 'successor': {}}
[20171114 22:32:17.436] INFO:BaseTopologyManager: ipop_tap0 P2P State: STARTED
[20171114 22:32:17.437] INFO:BroadcastForwarder: BroadcastForwarder Loaded
[20171114 22:32:17.437] INFO:TincanInterface: Received data from Tincan: Operation: CreateCtrlRespLink. Task status::{u'Message': u'Controller endpoint successfully created.', u'Success': True}
[20171114 22:32:17.437] DEBUG:TincanInterface: Tincan Request: {'Request': {'Initiator': 'LinkManager', 'UID': None, 'MAC': '', 'Command': 'QueryNodeInfo', 'ProtocolVersion': 4, 'InterfaceName': u'ipop_tap0'}, 'TransactionId': 4, 'ControlType': 'TincanRequest', 'ProtocolVersion': 4}
[20171114 22:32:17.438] INFO:TincanInterface: Received data from Tincan: Operation: ConfigureLogging. Task status::{u'Message': u'Tincan logging successfully configured.', u'Success': True}
[20171114 22:32:17.438] INFO:TincanInterface: Received data from Tincan: Operation: CreateVnet. Task status::{u'Message': u'', u'Success': True}
[20171114 22:32:17.438] INFO:TincanInterface: Received data from Tincan: Operation: SetIgnoredNetInterfaces. Task status::{u'Message': u'', u'Success': True}
[20171114 22:32:17.438] DEBUG:TincanInterface: Tincan Request: {'Request': {'Initiator': 'BaseTopologyManager', 'UID': None, 'MAC': '', 'Command': 'QueryNodeInfo', 'ProtocolVersion': 4, 'InterfaceName': u'ipop_tap0'}, 'TransactionId': 5, 'ControlType': 'TincanRequest', 'ProtocolVersion': 4}
[20171114 22:32:17.438] DEBUG:TincanInterface: Tincan Request: {'Request': {'Initiator': 'BaseTopologyManager', 'UID': None, 'MAC': '', 'Command': 'QueryNodeInfo', 'ProtocolVersion': 4, 'InterfaceName': u'ipop_tap0'}, 'TransactionId': 6, 'ControlType': 'TincanRequest', 'ProtocolVersion': 4}
[20171114 22:32:17.439] DEBUG:TincanInterface: current state of dd7e18d641c0f18b4f80a1886b589cc2525a931e : {'mac': u'66F5F6C9A3EA', '_uid': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e', '_ip4': u'10.11.0.12', '_fpr': u'sha-1 2B:56:C8:ED:59:11:C1:B5:13:3B:3A:40:96:87:CB:FF:97:40:40:6C', 'interface_name': u'ipop_tap0', 'type': 'local_state'}
[20171114 22:32:17.439] INFO:LinkManager: LM Local Node Info UID:dd7e18d641c0f18b4f80a1886b589cc2525a931e MAC:66F5F6C9A3EA IP4: 10.11.0.12
[20171114 22:32:17.439] DEBUG:TincanInterface: current state of dd7e18d641c0f18b4f80a1886b589cc2525a931e : {'mac': u'66F5F6C9A3EA', '_uid': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e', '_ip4': u'10.11.0.12', '_fpr': u'sha-1 2B:56:C8:ED:59:11:C1:B5:13:3B:3A:40:96:87:CB:FF:97:40:40:6C', 'interface_name': u'ipop_tap0', 'type': 'local_state'}
[20171114 22:32:17.439] DEBUG:TincanInterface: current state of dd7e18d641c0f18b4f80a1886b589cc2525a931e : {'mac': u'66F5F6C9A3EA', '_uid': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e', '_ip4': u'10.11.0.12', '_fpr': u'sha-1 2B:56:C8:ED:59:11:C1:B5:13:3B:3A:40:96:87:CB:FF:97:40:40:6C', 'interface_name': u'ipop_tap0', 'type': 'local_state'}
[20171114 22:32:27.419] DEBUG:LinkManager: Peer Nodes:: {}
[20171114 22:32:27.429] DEBUG:BaseTopologyManager: BTM Table::{'discovered_nodes': [], 'p2p_state': 'started', 'uid_mac_table': {u'dd7e18d641c0f18b4f80a1886b589cc2525a931e': [u'66F5F6C9A3EA']}, 'ipop_state': {'mac': u'66F5F6C9A3EA', '_uid': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e', '_ip4': u'10.11.0.12', '_fpr': u'sha-1 2B:56:C8:ED:59:11:C1:B5:13:3B:3A:40:96:87:CB:FF:97:40:40:6C', 'interface_name': u'ipop_tap0', 'type': 'local_state'}, 'ip_uid_table': {}, 'xmpp_client_code': u'XmppClient', 'mac': u'66F5F6C9A3EA', 'peer_uid_sendmsgcount': {}, 'GeoIP': '', 'mac_uid_table': {u'66F5F6C9A3EA': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e'}, 'link_type': {}, 'successor': {}}
[20171114 22:32:27.430] INFO:BaseTopologyManager: IPOP local state: dd7e18d641c0f18b4f80a1886b589cc2525a931e
[20171114 22:32:27.430] INFO:BaseTopologyManager: ipop_tap0 P2P State: SEARCHING
[20171114 22:32:37.419] DEBUG:LinkManager: Peer Nodes:: {}
[20171114 22:32:37.431] DEBUG:BaseTopologyManager: BTM Table::{'discovered_nodes': [], 'p2p_state': 'searching', 'uid_mac_table': {u'dd7e18d641c0f18b4f80a1886b589cc2525a931e': [u'66F5F6C9A3EA']}, 'ipop_state': {'mac': u'66F5F6C9A3EA', '_uid': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e', '_ip4': u'10.11.0.12', '_fpr': u'sha-1 2B:56:C8:ED:59:11:C1:B5:13:3B:3A:40:96:87:CB:FF:97:40:40:6C', 'interface_name': u'ipop_tap0', 'type': 'local_state'}, 'ip_uid_table': {}, 'xmpp_client_code': u'XmppClient', 'mac': u'66F5F6C9A3EA', 'peer_uid_sendmsgcount': {}, 'GeoIP': '', 'mac_uid_table': {u'66F5F6C9A3EA': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e'}, 'link_type': {}, 'successor': {}}
[20171114 22:32:37.433] INFO:BaseTopologyManager: ipop_tap0 P2P State: SEARCHING
...

@saket424
Copy link
Author

BTW, I am using the sample group-vpn config file as my starting point and filling in the xmpp credentials

The two peers are 10.11.0.11 and 10.11.0.12 and neither can ping the other

Also, my tincan log file is 0 size even when I enabled tincan_logging to 2 or 3
I only see the controller log at this point

@vahid-dan
Copy link
Contributor

vahid-dan commented Nov 15, 2017

Hi @saket424 and thanks for your interest in IPOP VPN!

Make sure your XMPP username is in this format: <your_xmpp_user>@<your_xmpp_server>, not just <your_xmpp_user>.
Could you please copy the XMPP config here? Keep in mind to remove the XMPP password after copying. :-)

Cheers,
Vahid

@saket424
Copy link
Author

Vahid,
I tried with both my ejabberd xmpp server and the xmpp.jp xmpp server that you allude to in your tutorial -- Same issue in both instances.
I definitely am using the correct set of xmpp credentials in the format <your_xmpp_user>@<your_xmpp_server> since the ejabberd server states I am logged in and I get valid ifconfig output for ipop_tap0

So I am confused whether you want the json file that the controller uses or the xmpp config file that ejabberd uses when you ask for the XMPP config

I think you mean you want the former. Here is the gvpn json file
$/home/mike/tincan-stuff/ipop-vpn_rel17.08(ubuntu)/config# cat gvpn.json
{
"CFx": {
"Model": "GroupVPN"
},
"Logger": {
"Enabled": true,
"LogLevel": "DEBUG",
"LogOption": "File",
"LogFilePath": "./logs/",
"CtrlLogFileName": "ctrl.log",
"TincanLogFileName": "tincan.log",
"LogFileSize": 5000000,
"BackupLogFileCount": 5
},
"TincanInterface": {
"ctrl_recv_port": 5801,
"localhost": "127.0.0.1",
"ctrl_send_port": 5800,
"localhost6": "::1",
"dependencies": ["Logger"],
"Vnets": [
{
"Name" : "ipop_tap0",
"IP4": "10.11.0.12",
"IP4PrefixLen": 16,
"MTU4": 1200,
"XMPPModuleName": "XmppClient",
"TapName": "ipop_tap0",
"Description": "Beta 2 Test Network",
"IgnoredNetInterfaces": [ "ipop_tap0" ],
"L2TunnellingEnabled": true
}
],
"Stun": [ "stun.l.google.com:19302" ],
"Turn": [{
"Address": "turn:turnserver-ip:port",
"User": "",
"Password": "
"
}]
},
"XmppClient": {
"XmppDetails": [
{
"Username": "[email protected]",
"Password": "***",
"AddressHost": "xmpp.jp",
"Port": "5222",
"TapName": "ipop_tap0",
"AuthenticationMethod": "password",
"AcceptUntrustedServer": true
}
],
"TimerInterval": 10
}
}


$ifconfig ipop_tap0
ipop_tap0 Link encap:Ethernet HWaddr 16:8d:26:31:01:cf
inet addr:10.11.0.12 Bcast:10.11.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:576 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

@renatof
Copy link
Contributor

renatof commented Nov 15, 2017 via email

@saket424
Copy link
Author

@renatof,
I did use a turnserver in my config but blanked it out before pasting to the issue list just to protect the credentials from being abused.

@renatof
Copy link
Contributor

renatof commented Nov 15, 2017 via email

@saket424
Copy link
Author

should'nt stun config be prefixed with a stun: ?
"Stun": [ stun:stun.l.google.com:19302 ] similar to how the Turn has a turn: prefix
Once you get it working on your end, please share with me the json file minus the specific credentials of course. Also, if there is any additional logging you would like me to enable, do let me know. BTW, I could not have to enable the tincan logging yet. I am guessing that log may have some more clues on what the issue is.

Thanks

@vahid-dan
Copy link
Contributor

@saket424,

STUN server configuration is alright. Have you ever tried without the TURN? If not, please remove the lines related to the TURN server and try again. Don't forget to remove the semi-colon at the end of the STUN configuration line. We'll investigate the problem and get back to you.

Thanks,
Vahid

@saket424
Copy link
Author

@vahid-dan and @renatof
Your initial hunch was correct. I created a bank of 4 xmpp user-ids dq11 through dq14
Till I updated their rosters that each could see each other's presence, they could not receive presence updates and hence they could not get each other's mac address
Now with the roster update, I get further and i have gone beyond the searching state. It now enters the CONNECTING state but the connection does not fully succeed. Any ideas as to why not ?
Your ipop-16.08.0 version of software succeeds while the newer ipop-17.08 version still does not successfully connect. I'll wait till you finish looking at the issue on your end.

@saket424
Copy link
Author

removing the TURN config did not seem to help any. The relay candidates are no longer there but the net result is still it is stuck in the CONNECTING state and never successfully CONNECTED

@vahid-dan
Copy link
Contributor

@saket424,

Thanks for spending time on IPOP. :-)

One more thing to test: Have you ever tried to connect all the nodes with just one XMPP user?

Sometimes it is hard for us to reproduce the errors and that makes the debugging difficult. In those cases, user reports such as what you currently do are very helpful. We really appreciate it.

Bests,
Vahid

@rogora
Copy link

rogora commented Nov 16, 2017

Hi everyone, I'm experiencing exactly the same problem as saket424. I think it might be a problem with TURN servers. I have tried with different TURN servers without success. I have also tried with xirsys TURN server, as suggested by the configuration video.
Also, the peers connect correctly when they are behind the same NAT.

Finally, I confirm that I can connect successfully with version 16.08, in the same scenario where 17.08 fails.

@saket424
Copy link
Author

@vahid-dan ,
Switching both peer config files to use the same xmpp user made no difference -- still stuck in the CONNECTING state

@vahid-dan
Copy link
Contributor

Hi @saket424 and @rogora;

Our team is investigating the problem, and we most probably have an update for you by tomorrow night. BTW, in this case, even if the TURN works, the functionality is not ideal for IPOP VPN. If IPOP needs the TURN server, it means both sides of the connection are behind symmetric NAT or very restricted firewall. So you may want to check what prevents the IPOP nodes to create a direct tunnel. Firewall rules or router settings may be the cause of that.

We'll get back to you as soon as we have an explanation for you.

Bests,
Vahid

@saket424
Copy link
Author

@vahid-dan
The fact that ipopvpn16.08 does work for @rogora and me without TURN is proof that we are not behind a symmetric NAT or restricted firewall. It is some problem specific to the new ipop17.08 build. Glad you are investigating.

Thanks
Anand

@vahid-dan
Copy link
Contributor

@saket424,

So you mean even without setting the TURN server configuration in the config file, IPOP 16.08 works for you? If so, you shouldn't need the TURN server to connect the nodes and if the nodes don't connect, the problem may be something else rather than the TURN server. Anyway, we hopefully know in the next 24 hours. :-)

Vahid

@saket424
Copy link
Author

saket424 commented Nov 17, 2017

@vahid-dan

Precisely my point. Problem is elsewhere in 17.08 since I can connect in 16.08 with no turn server and i cannot connect in 17.08 with or without a turn server. Eagerly awaiting your findings in 24 hours

@kcratie
Copy link
Member

kcratie commented Nov 18, 2017

@saket424 - We are still working to identify the root cause of the TURN bug with the v17.08 and resolve the issue. Independently of this, I would like to assist in determining what is causing your deployment to fail. To do this will need the controller and Tincan logs from both systems. Please ensure that the IPOP config json file sets the Logger LogLevel to DEBUG on both systems. The is just as was done in the previous config file you submitted. Also, feel free to increase LogFileSize if convenient. I may appear to be asking you repeat steps you have already taken, but from following the thread I can see that progressive changes have been made and I need a clear snapshot of the systems as they are now. Since we know tunnels are not being built via TURN, please omit the TURN parameters from the config. Once the peers are on separate subnets and they both have access to the STUN service, they will attempt NAT traversal. As the files will be verbose, it will be more convenient to put them in an archive as an attachment. If possible, please include your config file after removing any sensitive information. This should provide me with sufficient information pinpoint the cause of the failure.
Also, you can try our IPOP test script [https://github.com/ipop-project/Release-Management/tree/master/Test/ipop-scale-test], which creates the IPOP environment within a single Ubuntu VM using containers. I expect this base case to work and can be used a sanity check to validate deploying the build successfully.
Thanks for taking the time to work through this with us.

@saket424
Copy link
Author

201.zip
204.zip

@kcratie
Attached are the logs and associated config files for both ends of the tunnerl. per @vahid-dan 's suggestion, i used the same xmpp id for both config files(no turn involved). I suspect the ipop-scale-test will probably work but i have not tried it. I can confirm to you ipop16.08 worked with no turn specified, hence i expect ipop17.08 to establish the tunnel without any turn needed also but it remains stuck in the connecting state.

If you would like to do a more interactive test, send me a private email and we can do it together

Thanks
Anand

@kcratie
Copy link
Member

kcratie commented Nov 27, 2017

@saket424
I've made a commit to the Tincan and Controller repos to address the problem you have been experiencing. You will have to checkout the branch 'bh1' on both repos and build Tincan using its Makefile until the maintenance release is created. TURN should now work and NAT traversal should be improved. Please let me know the outcome.

@vahid-dan
Copy link
Contributor

Hi @saket424 and @rogora;

While you're building the IPOP on the branch 'bh1', keep in mind that we have recently changed the way to get the WebRTC libraries for building IPOP. The documentation is up to date. Just follow them and you should be fine.

Bests,
Vahid

@saket424
Copy link
Author

@kcratie and @vahid-dan
I tried the 'bh1' branch and I am happy to report that the tunnel did establish without the need for TURN. So NAT traversal with STUN seems to have improved

Tunnel creation takes a little while upto 2 minutes. I think it is probably because xmpp peer discovery is slow?

Also is this a layer2 tunnel if "L2TunnellingEnabled": true and otherwise defaults to layer3 ?

@kcratie
Copy link
Member

kcratie commented Nov 29, 2017

@saket424
It is good to hear the fix worked for you.
We are currently working on improvements to the signalling that's done using XMPP. We expect the new approach will reduce the connection setup time.
Currently we only do layer 2 tunneling based on Ethernet frames. However, there is a road map to integrate layer 3 info into aspects of of the tunneling process.

@saket424
Copy link
Author

saket424 commented Dec 1, 2017

Since only layer2 is supported, What does the json config option "L2TunnellingEnabled":false do? I assume it is just a no-op flag and it does not currently matter if it true or false and results in the same behavior either way?

@saket424
Copy link
Author

@vahid-dan and @kcratie

I am unable to replicate the success i had in late November. From what I can gather, I am having xmpp issues and each peer is not getting the other's IP/mac information and hence does not even attempt a peer connection and stuck in the searching state. Can you think of why I am seeing xmpp unreliability and is anyone else also running into this issue of late?

Thanks
Anand

@vahid-dan
Copy link
Contributor

Hi @saket424;

We will have a maintenance release for our current version of IPOP and also a new release in a few weeks. Do you think you can wait till then? Maybe you should put your effort in the new release.

Thanks,
Vahid

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants