Use test3 for refinery page development and automate transfer to production

[PatReynolds]

Priority (0 6 7 4) 17

[Captainkirkdawson]

If really really needed could be done by by a cron task running a rsync of the mysql refinery tables

[PatReynolds]

Methodology: bring everything from test to live. Then when in place, cron task to copy live back to Test 2 (no use for refinery on test 3). Authorisation for cron task in Scrum, to ensure no one has work on Test 2 that would be over-ridden.

Perriodically re-run so that minor textual changes, which do not originate on test 2, are copied back.

[Captainkirkdawson]

test2 has no requirement for help material. It is a test facility for developers working on significant new capabilities or major code changes. The only people using test2 are those involved the functions associated with the capability and are usually members of the FR senior management and senior coordinators. It uses separate databases that are not connected to the production servers and these are often dated and/or damaged.
test3 is the facility that should be used for final development and checkout of code and help materials. Once developed and tested on test3 these are deployed to the production servers.
In this process there should be no requirement for movement of code or help materials in the reverse direction.

[AlOneill]

@PatReynolds What!! If you "bring everything from test to live", as things stand on REG, then a lot of time-consuming formatting/editing will be lost (permanently) on live.

I'm hoping I don't understand what you mean.

[Sherlock21]

What on earth is Pat saying here?

Is the last comment on the right story? ( i.e. the one attributed to Pat)
As AlOneill says: NO.

[PatReynolds]

I prefer to create new pages on test. It allows for the poor html of people like me to be checked before making live. This won't happen much now on FreeREG, but is happening a lot on FreeCEN2.

But as @AlOneill says, there is much more on live which is not on a test server. So it is a matter of knowing what needs to be copied in which direction, rather than wholesale copying.

My understanding (again, may be faulty @Captainkirkdawson is that Alison will need to test things such as changes to tables, rather than working on live. Maybe only on test2@?

That the actual content is of use on a test isn't the issue, I think.

[PatReynolds]

Test 3 - staging is the 'build' for Refinery. Need to find a date/stage when there is nothing on staging that will be lost when Live over-writes staging.

[richpomfret]

To review proposal that we choose a time to 'sync' everything so that Test3 reflects Live.

[richpomfret]

Awaiting documentation before we decide on this.

[benwbrum]

@DeniseColbert to make skeleton document.

[DeniseColbert]

Document sent to @richpomfret and @PatReynolds

[Captainkirkdawson]

Would it not have been better to paste a link so all could see and comment

[DeniseColbert]

As it's a skeleton (i.e. empty) document, there's nothing to comment on just yet, will send around/ link to when there is. Also wary to post the link on github as it's public and I'm not 100% sure that the contents will be suitable.

[lemon-ukgen]

I don't think it's a good idea to cron it, since we might be mid-edit when the sync is scheduled, or we want a bunch of changes to be published one go.

I'm looking at a console from which we can trigger the process on demand, most likely Rundeck.

[lemon-ukgen]

This is mostly assembled. What I need now is confirmation that it's OK for me to test it, by doing repeated transfers of database and assets from test3 to production. Documentation and instructions will follow.

[lemon-ukgen]

@richpomfret @benwbrum @AlOneill Please can someone confirm I am OK to do another test3 -> production transfer to test this?

As before, this is a destructive action to anything that's only in production (sounds like that's understood and intended, but worth being explicit about).

[AlOneill]

@lemon-ukgen @benwbrum @Captainkirkdawson I'm not seeing any of the Refinery images on test3: would appreciate confirmation that all is in order before transfer. Thanks.

To clarify, Refinery Images listing is present but can't see the images there or on Refinery pages.

[Captainkirkdawson]

I agree there are NO refinery images on test3 or test2. I have NO idea when or how they went AWOL
Based on that @lemon-ukgen should NOT attempt to transfer test3 to production as production is still OK

[lemon-ukgen]

The assets symlink on test3 is missing.

Production:

[lemon@colobus ~]$ ls -l /home/apache/hosts/freereg2/production/public/system/refinery
lrwxr-xr-x  1 webserv  webserv  30 Mar 26  2015 /home/apache/hosts/freereg2/production/public/system/refinery -> /raid/freereg2/refinery_assets

test3:

[lemon@vervet ~]$ ls -l /home/apache/hosts/freereg2/production/public/system/refinery
ls: /home/apache/hosts/freereg2/production/public/system/refinery: No such file or directory

I can recreate it... but where did it go?

[Captainkirkdawson]

I think I can shed some light on how it might have gone.
Over the past month the code bases for both test3 and test2 were rebuilt from the github repository.
There is no place in our set up procedures of the need to create such a symlink. Are there others?

[lemon-ukgen]

I expect the symlink was created by me, motivated by needing to put the [uploaded] assets somewhere that gets synced between servers, and the sync process generally operates out of /raid trees.

However, the enclosing directory production/public/system is also missing on test3, and I doubt that was my invention.

We should add both creating the directory and sticking the symlink inside it to the update-freeX scripts.

I've done this by hand on test3 and the example image link above now works.

[Captainkirkdawson]

The myopicvicar code base now runs as different applications. The public files are different between the applications hence there is an initializer that creates the required files in the public folder. I was unaware of the requirement for a public/system folder into which the refinery symlinks were placed hence it is not created. The initializer is easily modified to ensure that there is a public/system folder or it can be done by the update script. Since I am unfamiliar with the symlinks perhaps your solution is better.

[lemon-ukgen]

The requirement is just this config piece, as far as I can tell:

config.dragonfly_url_format = "/system/resources/:job/:basename.:ext"
config.dragonfly_url_format = "/system/images/:job/:basename.:ext"

(Which itself has changed during the course of this issue...)

So I'd suggest the initializer script should create system since the Refinery / Dragonfly config expects it, and that the update script, which is specific to how we deploy this stuff on our servers, should do the symlink.

Alternatively we can update the config to not use system and move the symlink. Equivalent from where I'm sitting.

[Captainkirkdawson]

@lemon-ukgen initializer now creates public/system if not present.

[richpomfret]

@lemon-ukgen can you proceed with this? Let me know if you need anything else.

[lemon-ukgen]

Current status:

• Rundeck installed as a web front end to for humans to initiate the transfer
  • Rundeck jobs created to
  • dump the Refinery tables on vervet from test3
  • transfer that dump to colobus to a staging area
  • transfer the Refinery assets (images, resource) to a staging area
  • backup the production Refinery tables and assets
  • install the transferred assets and tables into production

That last job has a safety catch on it to just dry-run the operation. I'm ready to remove the safety, but have yet to do so.

[lemon-ukgen]

Sample output from a transfer

[lemon-ukgen]

I have now tried this for real, and the transfer worked ok as far as I can tell. The touchstone samuel.jpg is everywhere again. Please could you someone have a look?

To open this up to the group I'll need some usernames and htpasswd-style password hashes. You can generate one online (which amazingly isn't as sketchy as it seems)

I'll write some docs up for how to drive it.

[AlOneill]

Synced pages and images fine on live: my mistake that Coords sub-menu is lacking two of the new page links on some info pages.

However, the two letters for Incumbents are "Not found". Will investigate further …

[AlOneill]

The two letters for Incumbents are also "Not found" on test3. Looks like I didn't check the links (doh!). Did anyone else check that they worked on test3?

[AlOneill]

Links re-inserted on test3 and working as expected. (Issue with opening in OpenOffice will be noted on appropriate card.)

Should I re-do links directly on live? Otherwise, they will not be working until next REG sync.

[lemon-ukgen]

I've redone the transfer to write some docs for the process, so I'm hoping these links are now on production too.

[lemon-ukgen]

Oh no! The "Please be aware that this is a test development website for FreeREG" banner is delivered by Refinery, so when we transfer to production this banner appears there too.

For now I nulled it out in the database (I have no access to Refinery). We'll need to do something better...

For the record I did:

mysql> select id,body from refinery_page_part_translations where id = 204 ;
+-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id  | body

                     |
+-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------+
| 204 | <p>Please be aware that  this is a test development website for FreeREG. The database is NOT updated. It should ONLY be used by by developers and testers.
 It should NOT be used by the public. For access to our records please click on this link <a href="https://www.freereg.org.uk/" title="https://www.freereg.org.uk/
">FreeREG </a>
</p> |
+-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------+
1 row in set (0.00 sec)


mysql> update refinery_page_part_translations set body = "" where id = 204 ;
Query OK, 1 row affected (0.02 sec)
Rows matched: 1  Changed: 1  Warnings: 0

[AlOneill]

@lemon-ukgen I can confirm that links to letters are now working on production. TY

That banner message is done in Refinery so that it can be used, maybe in a hurry, without developer intervention. Can that individual page be excluded from a sync?

[Captainkirkdawson]

That specific warning is no longer needed on test3 now that it is the test sites are behind a log in. Removed from test3 so no need to take special action.

[richpomfret]

@richpomfret to test/look at process for deploying this each sprint (or when necessary).

[richpomfret]

@lemon-ukgen can you post the link to the update process/instructions document?

[lemon-ukgen]

Now landed in the System Documentation/FreeREG folder: FreeREG Refinery CMS transfer test3 to production

Give me a shout if it's unclear or not working.

[Captainkirkdawson]

Used the process following the last release; worked as expected. @AlOneill and @DeniseColbert need to establish passwords if they have not already done so then this can be closed