Move chef_gem install to a recipe, so it is only called once.

[scottymarshall]

Moving it to a recipe makes it is only called once. Calling that chef_gem install only once will take a considerable amount of time off of a chef run.

[joekiller]

I'll try this out can you put an end line on recipes/dependencies.rb?

[scottymarshall]

Done.

[joekiller]

I'm not really seeing a performance increase from this.

Test runs without commit:

Chef Client finished, 8/11 resources updated in 27.099885167 seconds
Chef Client finished, 8/11 resources updated in 31.295167196 seconds
Chef Client finished, 8/11 resources updated in 39.797201695 seconds

Test runs with commit:

Chef Client finished, 8/11 resources updated in 29.444784755 seconds
Chef Client finished, 8/11 resources updated in 31.794026351 seconds
Chef Client finished, 8/11 resources updated in 37.628887443 second

[scottymarshall]

Try using s3_file more than once in the same run. I have a recipe that downloads 7 different files from S3, and I was seeing the job delay at the chef_gem run each time for more than 10 seconds as it compared checksums are something. Once I took the call out of the LWRP, I recouped over 60 seconds from my run.

[joekiller]

I did multiple runs across several fresh instances that had three s3 file
references. From the log it appears to call the gem method either way each
time. Do you see only one gem call or multiple?
On Feb 12, 2015 12:26 PM, "Scott Marshall" [email protected] wrote:

Try using s3_file more than once in the same run. I have a recipe that
downloads 7 different files from S3, and I was seeing the job delay at the
chef_gem run each time for more than 10 seconds as it compared checksums
are something. Once I took the call out of the LWRP, I recouped over 60
seconds from my run.

—
Reply to this email directly or view it on GitHub
#43 (comment).

[scottymarshall]

Only one gem call.
Here is a snippet of my logs and you will see that I only see that chef_gem called once even though I use your S3_file LWRP more than once:

          * jmh-utilities_s3_download[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create
       [2015-02-12T22:17:12+00:00] INFO: Processing jmh-utilities_s3_download[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create (jmh-cq::install line 11)
        (up to date)
           * s3_file[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create[2015-02-12T22:17:12+00:00] INFO: Processing s3_file[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create (/tmp/kitchen/cookbooks/jmh-utilities/providers/s3_download.rb line 8)
           Recipe: s3_file::dependencies
             * chef_gem[rest-client] action install[2015-02-12T22:17:12+00:00] INFO: Processing chef_gem[rest-client] action install (s3_file::dependencies line 1)
        (up to date)
             * chef_gem[rest-client] action install[2015-02-12T22:17:50+00:00] INFO: Processing chef_gem[rest-client] action install (s3_file::dependencies line 1)
        (up to date)
             * file[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create[2015-02-12T22:17:50+00:00] INFO: Processing file[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create (/tmp/kitchen/cookbooks/s3_file/providers/default.rb line 87)
.....
* jmh-utilities_s3_download[/usr/local/cq/src/cq-service-pack-5.5.3.zip] action create
 * s3_file[/usr/local/cq/src/cq-5.5.0-hotfix-3659-1.0.0.zip] action create[2015-02-12T22:23:46+00:00] INFO: Processing s3_file[/usr/local/cq/src/cq-5.5.0-hotfix-3659-1.0.0.zip] action create (/tmp/kitchen/cookbooks/jmh-utilities/providers/s3_download.rb line 8)   
 * file[/usr/local/cq/src/cq-5.5.0-hotfix-3659-1.0.0.zip] action create[2015-02-12T22:23:47+00:00] INFO: Processing file[/usr/local/cq/src/cq-5.5.0-hotfix-3659-1.0.0.zip] action create (/tmp/kitchen/cookbooks/s3_file/providers/default.rb line 87)
       [2015-02-12T22:23:47+00:00] INFO: file[/usr/local/cq/src/cq-5.5.0-hotfix-3659-1.0.0.zip] mode changed to 644    
......
           * jmh-utilities_s3_download[/usr/local/cq/src/cq-service-pack-5.5.3.zip] action create  
         * s3_file[/usr/local/cq/src/cq-service-pack-5.5.3.zip] action create[2015-02-12T22:23:47+00:00] INFO: Processing s3_file[/usr/local/cq/src/cq-service-pack-5.5.3.zip] action create (/tmp/kitchen/cookbooks/jmh-utilities/providers/s3_download.rb line 8)    
             * file[/usr/local/cq/src/cq-service-pack-5.5.3.zip] action create[2015-02-12T22:23:55+00:00] INFO: Processing file[/usr/local/cq/src/cq-service-pack-5.5.3.zip] action create (/tmp/kitchen/cookbooks/s3_file/providers/default.rb line 87)
       [2015-02-12T22:23:55+00:00] INFO: file[/usr/local/cq/src/cq-service-pack-5.5.3.zip] mode changed to 644

[joekiller]

Okay let me test it again.

On Thu, Feb 12, 2015 at 5:28 PM, Scott Marshall [email protected]
wrote:

Only one gem call.
Here is a snippet of my logs and you will see that I only see that
chef_gem called once even though I use your S3_file LWRP more than once:

  * jmh-utilities_s3_download[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create

[2015-02-12T22:17:12+00:00] INFO: Processing jmh-utilities_s3_download[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create (jmh-cq::install line 11)
(up to date)
* s3_file[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create[2015-02-12T22:17:12+00:00] INFO: Processing s3_file[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create (/tmp/kitchen/cookbooks/jmh-utilities/providers/s3_download.rb line 8)
Recipe: s3_file::dependencies
* chef_gem[rest-client] action install[2015-02-12T22:17:12+00:00] INFO: Processing chef_gem[rest-client] action install (s3_file::dependencies line 1)
(up to date)
* chef_gem[rest-client] action install[2015-02-12T22:17:50+00:00] INFO: Processing chef_gem[rest-client] action install (s3_file::dependencies line 1)
(up to date)
* file[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create[2015-02-12T22:17:50+00:00] INFO: Processing file[/usr/local/cq/src/CQ_5_5_Quickstart.jar] action create (/tmp/kitchen/cookbooks/s3_file/providers/default.rb line 87)

.....

• jmh-utilities_s3_download[/usr/local/cq/src/cq-service-pack-5.5.3.zip]
  action create

  • s3_file[/usr/local/cq/src/cq-5.5.0-hotfix-3659-1.0.0.zip] action
    create[2015-02-12T22:23:46+00:00] INFO: Processing
    s3_file[/usr/local/cq/src/cq-5.5.0-hotfix-3659-1.0.0.zip] action create
    (/tmp/kitchen/cookbooks/jmh-utilities/providers/s3_download.rb line 8)
    • file[/usr/local/cq/src/cq-5.5.0-hotfix-3659-1.0.0.zip] action
      create[2015-02-12T22:23:47+00:00] INFO: Processing
      file[/usr/local/cq/src/cq-5.5.0-hotfix-3659-1.0.0.zip] action create
      (/tmp/kitchen/cookbooks/s3_file/providers/default.rb line 87)
      [2015-02-12T22:23:47+00:00] INFO:
      file[/usr/local/cq/src/cq-5.5.0-hotfix-3659-1.0.0.zip] mode changed to 644
      ......

   - jmh-utilities_s3_download[/usr/local/cq/src/cq-service-pack-5.5.3.zip]
   action create
    - s3_file[/usr/local/cq/src/cq-service-pack-5.5.3.zip] action
      create[2015-02-12T22:23:47+00:00] INFO: Processing
      s3_file[/usr/local/cq/src/cq-service-pack-5.5.3.zip] action create
      (/tmp/kitchen/cookbooks/jmh-utilities/providers/s3_download.rb line 8)
       - file[/usr/local/cq/src/cq-service-pack-5.5.3.zip] action
         create[2015-02-12T22:23:55+00:00] INFO: Processing
         file[/usr/local/cq/src/cq-service-pack-5.5.3.zip] action create
         (/tmp/kitchen/cookbooks/s3_file/providers/default.rb line 87)
         [2015-02-12T22:23:55+00:00] INFO:
         file[/usr/local/cq/src/cq-service-pack-5.5.3.zip] mode changed to 644
  

—
Reply to this email directly or view it on GitHub
#43 (comment).