Can not clone the whole remote folder to local

[alejandroivan]

Expected Behavior

Clone (recursively) the full remote folder, to have a 1:1 copy.

Actual Behavior

Only downloads selected folders, but when not selecting any, it should recursively download everything inside the currently seen folder (root of the project), just like when selecting one folder and running SFTP: Download.

Steps to Reproduce the Problem

1. Do NOT select any folder and stay in the root of the project.
   2a. Cmd+Shift+P --> SFTP: Download.
   2b. Cmd+Shift+P --> SFTP: Sync remote to local.

Requisites (important to diagnose the problem!)

This is probably a feature request rather than a bug (or am I missing something?).

Extension Logs

N/A.

Specifications

• Version: Latest (0.8.11)
• VSCode Version: 1.18.1
• Platform: macOS

[liximomo]

Just Cmd+Shift+P --> SFTP: Download. This works with the latest version (0.8.11). It's time to upgrade. If you can't upgrade to 0.8.x. Maybe your vscode is too old.

[alejandroivan]

@liximomo I did that, but it didn't work. It created folders but no files were downloaded.

It was the latest version downloaded directly from VSCode (yesterday). VSCode version was the latest too, downloaded from the official website yesterday too.

I ended up creating a ZIP file in the server and downloaded that (these were A LOT of files, 10 thousand or so).

[liximomo]

There maybe a bug. It works well for me. Can you provide me the logs? Preference -> setting -> sftp.printDebugLog to true, then reload vscode, reproduce the problem, get the logs(view -> output-> sftp)

[alejandroivan]

Hello again, this is the log output:

[info]: config at /Users/dpsmac1/Sites {
    "host": "local.penquistas.cl",
    "port": 22,
    "username": "alejandroivan",
    "password": null,
    "protocol": "sftp",
    "agent": null,
    "privateKeyPath": null,
    "passphrase": null,
    "passive": false,
    "interactiveAuth": true,
    "remotePath": "/Users/alejandroivan/Sites/",
    "uploadOnSave": false,
    "syncMode": "full",
    "watcher": {
        "files": false,
        "autoUpload": false,
        "autoDelete": false
    },
    "ignore": [
        "/Users/dpsmac1/Sites/**/.vscode/**",
        "/Users/dpsmac1/Sites/**/.git/**",
        "/Users/dpsmac1/Sites/**/.DS_Store",
        "/Users/alejandroivan/Sites/**/.vscode/**",
        "/Users/alejandroivan/Sites/**/.git/**",
        "/Users/alejandroivan/Sites/**/.DS_Store"
    ],
    "configRoot": "/Users/dpsmac1/Sites"
}
[debug]: task: download /Users/dpsmac1/Sites


[debug]: 
------
target: /Users/alejandroivan/Sites/
context: transmission dir
error: Error: Failure
------

It is stuck retrieving directory /Users/alejandroivan/Sites/blog/vendor/symfony/finder/Tests/Fixtures/copy/A/B/C, which has one empty file. This is the output in the server's Terminal:

alejandroivan@Mac-mini-de-Alejandro:~/Sites/blog/vendor/symfony/finder/Tests/Fixtures/copy/A/B/C$ ls -al
total 0
drwxrwxr-x  3 alejandroivan  _www   96 Nov  5 13:10 .
drwxrwxr-x  4 alejandroivan  _www  128 Nov  5 13:10 ..
-rwxrwxr-x  1 alejandroivan  _www    0 Nov  5 13:10 abc.dat.copy
alejandroivan@Mac-mini-de-Alejandro:~/Sites/blog/vendor/symfony/finder/Tests/Fixtures/copy/A/B/C$ cat abc.dat.copy 
alejandroivan@Mac-mini-de-Alejandro:~/Sites/blog/vendor/symfony/finder/Tests/Fixtures/copy/A/B/C$ du -h abc.dat.copy 
  0B	abc.dat.copy
alejandroivan@Mac-mini-de-Alejandro:~/Sites/blog/vendor/symfony/finder/Tests/Fixtures/copy/A/B/C$ pwd
/Users/alejandroivan/Sites/blog/vendor/symfony/finder/Tests/Fixtures/copy/A/B/C

Maybe a bug with empty files? Anyway, if I do right-click that C folder and select SFTP: Download from the popup menu, it correctly downloads that abc.dat.copy file.

So far it has only created the directory structure, but without any files:

This is the server status of directories (in this case, application/config):

alejandroivan@Mac-mini-de-Alejandro:~/Sites/application/config$ ls -al
total 96
drwxr-xr-x@ 17 alejandroivan  staff    544 Jun 10 01:45 .
drwxr-xr-x@ 16 alejandroivan  staff    512 Jun 12 01:33 ..
-rwxr-xr-x   1 alejandroivan  staff   4048 Jul  5 14:23 autoload.php
-rwxr-xr-x   1 alejandroivan  staff  18573 Jul  5 14:23 config.php
-rwxr-xr-x   1 alejandroivan  staff   5005 Jul  5 14:23 constants.php
-rwxr-xr-x   1 alejandroivan  staff   4548 Jul  5 14:23 database.php
-rwxr-xr-x   1 alejandroivan  staff   2441 Jul  5 14:23 doctypes.php
-rwxr-xr-x   1 alejandroivan  staff   2993 Jul  5 14:23 foreign_chars.php
-rwxr-xr-x   1 alejandroivan  staff    417 Jul  5 14:23 hooks.php
-rwxr-xr-x   1 alejandroivan  staff    131 Jul  5 14:23 index.html
-rwxr-xr-x   1 alejandroivan  staff    498 Jul  5 14:23 memcached.php
-rwxr-xr-x   1 alejandroivan  staff   3032 Jul  5 14:23 migration.php
-rwxr-xr-x   1 alejandroivan  staff  10057 Jul  5 14:23 mimes.php
-rwxr-xr-x   1 alejandroivan  staff    477 Jul  5 14:23 profiler.php
-rwxr-xr-x   1 alejandroivan  staff   1967 Jul  5 14:23 routes.php
-rwxr-xr-x   1 alejandroivan  staff   3181 Jul  5 14:23 smileys.php
-rwxr-xr-x   1 alejandroivan  staff   6132 Jul  5 14:23 user_agents.php
alejandroivan@Mac-mini-de-Alejandro:~/Sites/application/config$ pwd
/Users/alejandroivan/Sites/application/config

[alejandroivan]

I noticed that, with different SFTP: Download operations, it gets stuck "retrieving" different folders, not only that A/B/C structure. Anyway, a du -h . on the local side, shows these directories and sizes:

dpsmac1@dpsmac1:~/Sites$ du -h .
8.0K	./.vscode
  0B	./application/cache
  0B	./application/config
  0B	./application/controllers
  0B	./application/core
## a lot more folders ##
  0B	./system/libraries/Javascript
  0B	./system/libraries/Session/drivers
  0B	./system/libraries/Session
  0B	./system/libraries
  0B	./system
 20K	.
dpsmac1@dpsmac1:~/Sites$ pwd
/Users/dpsmac1/Sites

On the remote side, the result is:

alejandroivan@Mac-mini-de-Alejandro:~/Sites$ du -h .
8.0K	./application/cache
 96K	./application/config
 44K	./application/controllers
4.0K	./application/core
## a lot more folders ##
 32K	./system/libraries/Javascript
 52K	./system/libraries/Session/drivers
 92K	./system/libraries/Session
696K	./system/libraries
2.2M	./system
154M	.
alejandroivan@Mac-mini-de-Alejandro:~/Sites$ pwd
/Users/alejandroivan/Sites

[liximomo]

What is your server OS? The sftp will download files in parallel. Please check the maximum number of open files of your server is greater than 512.

[alejandroivan]

@liximomo it's macOS High Sierra (10.13.2). I did run the commands from the known issues (both in the server and client). I know it's weird.

That [debug]: task: download /Users/dpsmac1/Sites disturbs me, since dpsmac1 is the local environment. Is there any way to force a localPath in the config, instead of using the project root?

[liximomo]

Is there any way to force a localPath in the config?

The upcoming version(0.9.0) will allow you to set config root manually.

I will make a debug-package for you to debug the download problem.

[liximomo]

Try debug-pacakge and give me some feedbacks.

[liximomo]

@alejandroivan You can config the localPath with sftp 0.9.0!

[alejandroivan]

@liximomo thanks! Sorry for not being able to answer earlier, I've been with lots of work these two past weeks.

I unfortunately deleted some files on the remote path and then synced without issues. It must have been the number of files and several folder levels. At least it is working now without that configRoot at the end of the config JSON file (which is giving me some error messages when it is present, probably it shouldn't have been there).

Thanks for your help!

EDIT: It would probably be a good idea to print every file that's being fetched real time. When it's finished, I'm never sure if it failed or not, so I have to check a couple files to be sure.

[hakudev]

@liximomo

Also having issues with the same goal, but the result is slightly different on my endpoint.
I'm trying to download the entire '/var/www/projectname' folder via
cmd+shift+p -> SFTP: Download.

The probem for me is, it's not downloading the entire folder. It's just downloading the subdirectories inside the root directory.

Files within the root directory, e.g. '/var/www/projectname/' aren't being downloaded.
Files within a subdirectory, e.g. '/var/www/projectname/config/' are being downloaded.

I'm on Windows 10 btw.

// Edit:
If it helps, manually downloading and syncronizing afterwards works like a charm.

[liximomo]

@HakuDouga
What is your root dir of workspace? What did you set to 'context' and 'remotePath'.

[Chrissteven81]

I am having this problem as well, W10 VS Code 1.20.1 Extension version 1.0.4

When I try to download at the root I get just folder structure. I then tried to download each folder with no luck. After enabling debug log I am able to download the sub folders however, I cant get the files in the root. I eventually found out the host we were using was denying permissions of some of the folders. as soon as SFTP ran into the permission denied error it would stop.

[andreyorst]

I have the same issue as @Chrissteven81. Spamming SFTP: Download command is kinda works, but my project is huge (about 13k files). Again, notepad++ with barely same config works, and ftp-simple works too, so I assume it is SFTP problem. My config is:

{
    "protocol": "ftp",
    "passive": false,
    "port": 21,
    "host": "ftp",
    "username": "username",
    "password": "password",
    "remotePath": "/home/username/my_huge_project",
    "uploadOnSave": true,
    "downloadOnOpen": true,
    "concurrency": 1,
    "syncMode": "full",
    "secure": false,
    "ignore": [
        ".vnc",
        ".vscode",
        ".git",
        "*.o",
        "a TON of ignore items here",
        ".DS_Store"
    ]
}

I have a lot of ignore items in the list. About 100 already, I really don't need those files, because it's a compile files, build scripts, etc. It downshiftes amount of files to minimum (about 5k). But can't still download them from remote.

The plugin shows a message in statusline: retrieving directory /path/yo/dir/, and it's not changing. There are like 3-4 files to dovnload from that dir (after ignore removes all unnecessary files).

Also, I still experience error, as I mentioned in #80 (comment):

[error]: Error: Unable to make data connection( Error: connect ECONNRESET ###.###.###.##:#####)
	at Socket.<anonymous> (/home/aorst/.vscode/extensions/liximomo.sftp-1.1.4/node_modules/ftp/lib/connection.js:935:10)
	at Object.onceWrapper (events.js:293:19)
	at emitOne (events.js:96:13)
	at Socket.emit (events.js:191:7)
	at TCP._handle.close [as _onclose] (net.js:510:12)

Can't this be the issue? Again, I can test everything and provide logs.