Unable to Import / Export Application with big file size

[ifytra]

Dear all,

We are using Qlik Sense September 2018 SR2 and with the precious help of Qlik-Cli we are trying to import the application which is being given with Qlik Scalability Tools (File: 200M-PTSalesAnalytics.qvf , Size: 5,45GB).

We use the following commands to import the file

Import-Module Qlik-Cli
Connect-Qlik -computername qserver -username qserver\QlikAdmin
Import-QlikApp -name "200M-PTSalesAnalytics" -file "C:\200M-PTSalesAnalytics.qvf" -upload

Notes: QlikAdmin is a RootAdmin account.

While the file initially is being uploaded (the bytes counter disappear after several minutes) , we get the following error

_Invoke-RestMethod: The remote server returned an error: (500) Internal Server Error.
At C:\Program Files\WindowsPowerShell\Modules\Qlik-Cli\functions\core.ps1:32 char:15

• ... $result = Invoke-RestMethod -Method $method -Uri $path @params -Web ...
• CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
• FullyQualifiedErrorId: WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand_

When we try to import an application with a smaller file size (e.g. 800MB), the operation completes successfully. Do you have any idea on how can we workaround this error?

Thanks for your time and actions!

Best regards,
iLiAS

[Nillth]

You will need to edit the file Core.ps1:

C:\Program Files\WindowsPowerShell\Modules\Qlik-Cli\functions\core.ps1

In this file on line 32 is the command:
$result = Invoke-RestMethod -Method $method -Uri $path @params -WebSession $script:webSession

you will need append a timeout value to this as the upload process for a 5gb file will most likely exceed the default timeout period.
$result = Invoke-RestMethod -Method $method -Uri $path @params -WebSession $script:webSession -TimeoutSec 600

Let us know if this resolves your issue.
Regards,
~MC

[ifytra]

Dear Marc,

Thanks for your response.

We have run the Import-QlikApp command with the following -TimeoutSec values:

• 3600
• 7200

In both cases, we have received the same error message. It is quite strange because the upload seems to complete successfully after approximately 47 minutes.The counter

Writing web request
  Writing request stream... (Number of bytes written: WWWXXXYYYZZZ)

disappears and after awhile (~30 seconds) we get the error message.

Best regards,
iLiAS

[ifytra]

** UPDATE **

In addition we have changed the following settings in the Virtual Proxy:

1st Change

Session inactivity timeout: 15 > 60

2nd Change

Extended security environment: Yes > No

In both cases we have received the same error message.

[ifytra]

** UPDATE **

In addition we have changed the following settings in the Virtual Proxy:

1st Change

Session inactivity timeout: 15 > 60

2nd Change

Extended security environment: Yes > No

In both cases we have received the same error message.

[ifytra]

In addition to my previous post, we get a similar error when we try to Export a big Qlik Sense Application.

The error we get is the following

Invoke-RestMethod : Stream was too long
At C:\Program Files\WindowsPowerShell\Modules\Qlik-Cli\functions\core.ps1:32 char:15

We found out that the maximum size for a Qlik Application in order to be exported successfully is 2,147,483,591 bytes (2 GB).

Finally, as mentioned in

Title: Invoke-RestMethod fails on very large files
URL: PowerShell/PowerShell#4129

'Invoke-RestMethod fails to download a large file but Invoke-WebRequest works'

Any help would be highly appreciated.

Thanks,
iLiAS

[ifytra]

Dear Adam,

Thanks for your efforts. I will download & try the new version today.

Best regards,
iLiAS

[ahaydon]

I suggest also setting $ProgressPreference='SilentlyContinue' as the download will be much faster

[ifytra]

*** UPDATE ***

The new version behaves as described below:

1. Importing an application with big file size does not complete. We get the same error as in my initial message but this time in line 35 of core.ps1 file. I have run the import twice and the second time I have added -TimeoutSec 3600 option.

2. Exporting an application with big file (e.g .~5GB) works fine!

[ahaydon]

That makes sense as I only implemented a fix for the download, the upload seems to be a different issue, as you get a 500 error that suggests it is Sense that is rejecting the request rather than a client-side failure.
Setting the ProgressPreference variable should speed up the import too, so I suggest you do that as it will help to prevent a timeout.
Can you import the same file using QMC, and does that complete successfully?

[ifytra]

Dear Adam,

Can you propose us any other troubleshooting action about the import? The application can be imported successfully via Chrome browser. I will add the $ProgressPreference in my next test runs.

Many thanks once more!

Best regards,
iLiAS

[ahaydon]

If you want to try the same fix that worked for the download you could change line 32 in core.ps1 to the following
} elseif ($params.OutFile -or $params.InFile) {
This will then use Invoke-WebRequest instead of Invoke-RestMethod for uploads as well as downloads.

[ifytra]

I will apply the aforementioned changes and I' ll let you know.

PS: Before your latest answer, I have changed line 35 in the following way and ran once more the import.

$result = Invoke-WebRequest -Method $method -Uri $path @params -WebSession $script:webSession

[ifytra]

*** UPDATE ***

I have made the following changes in the core.ps1 for the import:

1.) line 32 - changed the condition

} elseif ($params.OutFile -or $params.InFile) {

2.) new line 33 - addition

$ProgressPreference = 'SilentlyContinue'

3.) new line 34 - changed the cmdlet used

1st test

$result = Invoke-WebRequest -Method $method -Uri $path @params -WebSession $script:webSession

2nd test

$result = Invoke-WebRequest -Method $method -Uri $path @params -WebSession $script:webSession -TimeoutSec 3600

On both tests, we received the same error message:

Invoke-WebRequest: The remote server returned an error: (500) Internal Server Error
At C:\Program Files\WindowsPowerShell\Modules\Qlik-Cli\functions\core.ps1:34 char:17

Thanks in advance for any corrective actions or suggestions,
iLiAS

[Nillth]

As pointed out in the referenced PowerShell post, Invoke-RestMethod seems to have issues with Large files.

If you are attempting to add a large file into Qlik, one potential work around would be to use the Import Folder.

Once you have connected to the Qlik Sense Environment, you can run the following command to determine the Import folder.

Connect-Qlik 
$ImportFolder = Invoke-QlikGet /qrs/app/importfolder
$ImportFolder

Once you have determined the Import folder you can copy the App Binary into that folder then use
Import-QlikApp without the -upload switch to get around the issue with the Invoke-RestMethod and large files.

Note the quotes "''" around the name, without these, it seems to cause a error 400
Import-QlikApp -file "'example.qvf'" -name Example

https://help.qlik.com/en-US/sense-developer/April2019/Subsystems/RepositoryServiceAPI/Content/Sense_RepositoryServiceAPI/RepositoryServiceAPI-App-Import-App.htm

[ifytra]

Hi Marc,

Your workaround works like a charm!

The steps that I have followed are listed below:

1. Identification of the of ImportFolder path with the following command.

Invoke-QlikGet /qrs/app/importfolder

Result

\\<QLIKSERVER>\<FILE_SHARE_FOLDER>\...

2. Placement of the application with big file size in the aforementioned path.

3. Import of the application with the following command

Import-QlikApp -file "'BIG_APP.qvf'" -name BIG_APP

Finally, I would like to ask you if you plan to provide a fix for the -upload option.

Thanks for your valuable help both Adam & Marc!

Best regards,
iLiAS

[ifytra]

Hi Marc,

Your workaround works like a charm!

The steps that I have followed are listed below:

1. Identification of the of ImportFolder path with the following command.

Invoke-QlikGet /qrs/app/importfolder

Result

\\<QLIKSERVER>\<FILE_SHARE_FOLDER>\...

2. Placement of the application with big file size in the aforementioned path.

3. Import of the application with the following command

Import-QlikApp -file "'BIG_APP.qvf'" -name BIG_APP

Finally, I would like to ask you if you plan to provide a fix for the -upload option.

Thanks for your valuable help both Adam & Marc!

Best regards,
iLiAS

[ifytra]

A final question would be the following...

I am trying to use the workaround in the following way

#
# Building the -file part of the command
#

# First Part of the -file  part
$QS1='"'+"'"

# Application Name
$AppName="ABC_ILIAS"

# Last Part of the -file  part
$QE1=".qvf"+"'"+'"'

# Complete -file part
$Final1=$QS1+$AppName+$QE1

# Verification
write-host $Final1
write-host Import-QlikApp -name $AppName -file $Final1

The script above prints the following result

"'ABC_ILIAS.qvf'"

Import-QlikApp -name ABC_ILIAS -file "'ABC_ILIAS.qvf'"

If I try the following command

Import-QlikApp -name $AppName -file $Final1

I get an error message

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Program Files\WindowsPowerShell\Modules\Qlik-Cli\functions\core.ps1:35 char:17

If I get the the generated command and run it manually

Import-QlikApp -name ABC_ILIAS -file "'ABC_ILIAS.qvf'"

it works fine.

Note: I have copied the ABC_ILIAS.qvf file in the ImportFolder.

Do you have any idea on this issue? Is it a PowerShell restriction?

Thanks,
Ilias

[ahaydon]

@ifytra when using " around the filename on the command line it is interpreted as meaning everything between them is a string value of the parameter and the quotes are not actually part of the value. However, when you concatenate " and ' they are both part of the actual value.
The string value in your variable should not contain ", but must be wrapped in ' for the API to accept it. I will make a note to fix this so that the single quotes are added automatically if omitted to make this command easier to use.

[ifytra]

Hi Adam,

Thanks for the clarification. I will try to apply your suggestion in my scripts.

Meanwhile, I have found the following workaround:

# Exporting the Import command in a Powershell script file
write-output "Import-QlikApp -name $AppName -file $Final1" > c.ps1

# Importing the application by calling the newly created PS file
.\c.ps1

Best regards,
iLiAS

[Nillth]

# First Part of the -file  part
$QS1='"'+"'"
# Application Name
$AppName="ABC_ILIAS"
# Last Part of the -file  part
$QE1=".qvf"+"'"+'"'
# Complete -file part
$Final1=$QS1+$AppName+$QE1
# Verification
write-host $Final1
write-host Import-QlikApp -name $AppName -file $Final1

Personally this feels very complicated, but then again I hate concatenating strings.
Worth a read if you have the time dont-concatenate-strings

The following will work.

# Application Name
$AppName="ABC_ILIAS"
Import-QlikApp -name $AppName -file "'$($AppName).qvf'"

[ifytra]

Thanks for your comment Marc.

I will test your suggestion and I' ll let you know.

iLiAS

[ifytra]

Dear Marc & Adam,

In conclusion,

1. We have a fix for exporting applications which file size is > 2GB. (v.1.15.1)

2. We have a workaround for the import command.

• Placement of the application with big file size in the ImportFolder (run in PowerShell > Invoke-QlikGet /qrs/app/importfolder).

• Import of the application with the following command

Import-QlikApp -file "'BIG_APP.qvf'" -name BIG_APP

I would to thank you both once more.

Greetings from Athens, Greece.

Best regards,
iLiAS