nabla-servers-bower-sample

Javascript project

Overview

Build

Quality

Info

Tools

Contributing

Why?

Table of contents

// spell-checker:disable

• A bower project sample
  • Create project
  • Tools
    • Internationalization
    • GitHub cards
    • TODO
  • Quality tools
    • python 3.6
    • pre-commit
    • pre-commit specific hook
    • takari maven wrapper
  • NODE/NPM Installation
    • Using nvm
    • Using apt
  • Installation
  • Build & development
    • Docker image
      • Pull image
      • Start container
      • Maven
      • Docker compose
  • Documentation
  • Fix imagemin upgrade
  • List browser compatibility
  • Testing
  • Livereload
  • Proxy / Rewrite jetty
  • Run war in jetty or using cargo
  • ZaProxy
  • Selenium Grid
  • Jenkins
  • Yslow Psi WebPageTest
  • Refresh from upstream
  • Deploy to internet
  • Performance improvements
  • Issues
  • Run End2End tests
  • Eclipse
    • Import as Maven project
    • Java
  • Terraform
    • Init
      • Using terraform cloud to store Terraform State
      • Using consul backend to Terraform State in KV
    • Applying terraform
    • Update terraform documentation
  • Requirements
  • Providers
  • Modules
  • Resources
  • Inputs
  • Outputs
    • Check docker image quality
      • Docker dive
      • Docker linter
      • mega-linter
      • Check secret
      • Check cspell
    • Update README.md
  • Other resources
• Contributing
  • License

// spell-checker:enable

A bower project sample

This project is generated with yo angular generator version 0.11.1. Then I applied uncss generator

This project is also used to test integration of build tools like:

• private-bower (bower repo and caching)
• nexus (npm and java packages repo and caching)
• jenkins (continuous delivery in a jetty embedded)
• sonar multi-languages and coverage (QA metrics except JIRA and perf metrics)
• yeoman/nodejs/npm/grunt/bower and maven integration
• zaproxy (security tests), checkmarx, dependency-check and nsp (thirdparty security check)
• performance (jmx, phantomas, pagespeed, webpagetest, yslow, junitperf in jenkins)
• sonar / stash / jira / jenkins /git /checkmarx integration

languages like :

• javascript
  • unit tests (karma) + LCOV coverage
  • end2end tests (protractor)
• java
  • unit tests (junit and jmockit) + Jacoco coverage
  • integration tests (junit) + Jacoco coverage
• groovy
  • unit tests (junit) + Jacoco coverage
  • integration tests (junit) + Jacoco coverage

jmeter, gatling, mutation testing and arquillian are available in other sample project

Initialize

direnv allow
pyenv install 3.10.4
pyenv local 3.10.4
python -m pipenv install --dev --ignore-pipfile
direnv allow
pre-commit install

Create project

as local user

#npm install generator-angular
#npm install generator-angular-fullstack
yarn config set "strict-ssl" false -g
#yarn config set cafile ~/NABLA.crt
set NODE_TLS_REJECT_UNAUTHORIZED=0
yarn install

as local user

yo angular translate

Initiate

rm -Rf .venv/ .direnv/
direnv allow

Tools

Internationalization

angular-internationalization-tutorials

angular-translate-yeoman

GitHub cards

GitHub cards

TODO

TODO use https://angular-ui.github.io/bootstrap/ for language TODO remvove app/styles/components/_header.scss language-select TODO https://github.com/rackerlabs/angular-bootstrap-nav SEE https://github.com/dlukez/angular-bootstrap-select/blob/master/angular-bootstrap-select.js SEE https://github.com/angular-ui/angular-google-maps

Quality tools

python 3.8

Install python 3.8 and virtualenv

virtualenv --no-site-packages /opt/ansible/env38 -p python3.8
source /opt/ansible/env36/bin/activate
pip install -r requirements-current-3.8.txt

pre-commit

See pre-commit Run pre-commit install Run pre-commit autoupdate

First time run cp hooks/* .git/hooks/ or rm -Rf ./.git/hooks/ && ln -s ../hooks ./.git/hooks && git checkout hooks/

Run pre-commit run --all-files

Run SKIP=ansible-lint git commit -am 'Add key' Commit git commit -am 'TEST' --no-verify

pre-commit specific hook

export JIRA_USER=aandrieu

export JIRA_PASSWORD=XXX

Run custom hook pre-commit run git-branches-check

takari maven wrapper

See takari-maven-wrapper

mvn -N io.takari:maven:wrapper

NODE/NPM Installation

Using nvm

See nvm

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash

nvm ls-remote --lts
nvm --version
nvm version

nvm install 14
nvm install 16
nvm use 16

Using apt

curl -sL https://deb.nodesource.com/setup_16.x | bash - &&
apt-get update && apt-get install --no-install-recommends -y nodejs=16*

#npm cache clean -f
npm install -g n
#n stable
n 11.12.0
node -v
#v11.12.0
/usr/local/bin/node -v
npm install -g [email protected]
env PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true npm install -g [email protected]
env PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true yarn global add puppeteer

Installation

Install NPM modules.

npm install

Install Javascript dependencies.

bower install

Build & development

Docker image

Build image is : See ansible-jenkins-slave-docker

Pull image

docker pull registry.hub.docker.com/nabla/ansible-jenkins-slave-test:latest

Start container

#Sample using container to buid my local workspace
id albandri
#uid=1000(albandri) gid=999(docker) groups=999(docker)
docker run -it -u 1000:999 --userns=host -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v /var/run/docker.sock:/var/run/docker.sock -v /data1/home/albandri/:/data1/home/albandri/ -v /workspace/users/albandri30:/workspace/users/albandri30 -w /sandbox/project-to-build -v /workspace/users/albandri30/bower-fr-integration-test/:/sandbox/project-to-build:rw -v /jenkins:/home/jenkins -v /jenkins:/jenkins -v /etc/bash_completion.d:/etc/bash_completion.d:ro --name sandbox --entrypoint /bin/bash registry.hub.docker.com/nabla/ansible-jenkins-slave:latest

Maven

Run mvnw clean install for building. Run grunt for building and grunt serve for preview. Run mvnw site -Dskip.npm -Dskip.yarn -Dskip.bower -Dskip.grunt for building site.

mvnw clean install -Dserver=jetty9x
#mvnw clean install -Dserver=jetty9x -Dskip.npm -Dskip.grunt -Dskip.bower -Dskip.yarn
mvnw verify gpg:sign -Dgpg.passphrase=thephrase 2>&1 sign.log
npm ls --licenses

Run mvnw site -Dskip.npm -Dskip.yarn -Dskip.bower -Dskip.grunt for building site.

as root

npm uninstall grunt-contrib-imagemin && npm install grunt-contrib-imagemin

Docker compose

cd ./docker-compose
./docker-compose-up.sh
docker-compose -f ./docker-compose.yml -f ./docker-compose.dev.yml -p test up --force-recreate -d web
#docker-compose -f ./docker-compose.monitoring.yml -p monitoring up -d nexus
docker-compose -f ./docker-compose.monitoring.yml -p monitoring up -d
cd /opt/harbor
sudo docker-compose -f ./docker-compose.yml up -d

Documentation

See mkdocs

Run mkdocs for building doc and mkdocs serve for preview.

mkdocs serve
mkdocs build
mkdocs gh-deploy
#mkdocs gh-deploy -b docs

GitHub Pages github-pages

Fix imagemin upgrade

npm install [email protected]
grunt imagemin:dist

List browser compatibility

npx browserslist

Testing

test (dev mode)

grunt serve watch -v

test (deployment mode)

grunt serve:dist

Running grunt test will run the unit tests with karma.

Livereload

Run with grunt watch -v then manually edit a file listed in the watch.livereload.files array. Ensure that it is triggering the file by printing the file name to the console.

Then from the browser end, ensure going to http://localhost:35730/ works and displays {"tinylr":"Welcome","version":"0.2.1"} letting you know the live reload server is running.

Be sure the http://localhost:35730/livereload.js script is being loaded and you see the network socket being established.

Proxy / Rewrite jetty

Proxy can be used to get access to oter URL. In order to be activated it must be reached once : http://localhost:9090/login

Even though it is strongly advice to run this project behind an apache. Please use : ansible-web in order to install this project behind an apache.

Run war in jetty or using cargo

./mvnw jetty:run-war
./mvnw install org.codehaus.cargo:cargo-maven2-plugin:run -Dserver=jetty9x > install.log

ZaProxy

Needed for having /usr/bin/xml2-config in the PATH

sudo apt-get install libxml2-dev libxslt-dev

Note that ZAProxy must be installed and zap.sh must be available on the executable path for this to work. Make sure you are running zap 2.3.0 or 2.3.1

export PATH=/usr/local/zap/zap-2.3.0/:${PATH}

In case of trouble with zap locally. Please deactivate html report in Gruntfile.js html: false

Please use : ansible-zaproxy in order to install zaproxy

Start zap by hand:

sudo su - jenkins
export DISPLAY=:99 && nohup /usr/local/zap/zap-2.3.1/zap.sh -daemon -port 8090 > zap.log &
tail -f zap.log

Selenium Grid

How to start selenium grid

#ssh -X [email protected]
sudo su - jenkins
#start by hand selenium grid
rm nohup.out
nohup java -jar /workspace/selenium-server-standalone-2.52.0.jar -Dwebdriver.chrome.driver=/usr/lib/chromium-browser/chromedriver -role hub -port 4444 -host 192.168.0.29 -debug &
tail -f nohup.out
#check hub is working
INFO - Nodes should register to http://albandrieu.com:4444/grid/register/
#start by hand selenium instance for albandrieu.com
ssh -X [email protected]
#https://github.com/SeleniumHQ/selenium/wiki/Grid2
#export DISPLAY=localhost:99.0 && nohup java -jar /workspace/selenium-server-standalone-2.52.0.jar -role node -hub http://albandrieu.com:4444/wd/register -browser browserName=firefox,version=38.0,firefox_binary=/usr/bin/firefox,maxInstances=1,platform=LINUX -browser browserName=chrome,version=39.0.2171.95,chrome_binary=/opt/google/chrome/chrome,maxInstances=1,platform=LINUX &
#export DISPLAY=localhost:99.0 && nohup java -jar /workspace/selenium-server-standalone-2.52.0.jar -role node -hub http://127.0.0.1:4444/grid/register -browser browserName=firefox,version=40.0,firefox_binary=/usr/bin/firefox,maxInstances=1,platform=LINUX -browser browserName=chrome,version=43.0.2357.125,chrome_binary=/usr/bin/google-chrome,maxInstances=1,platform=LINUX clean install -Dserver=jetty9x -Dsurefire.useFile=false -Psample,jacoco,integration,run-its,arq-weld-ee-embedded -Darquillian=arq-weld-ee-embedded -Darquillian.launch=arq-weld-ee-embedded -Dwebdriver.chrome.driver=/usr/lib/chromium-browser/chromedriver -debug > selenum-hub.out 2>&1 &
#master
export DISPLAY=localhost:99.0 && nohup java -jar /workspace/selenium-server-standalone-2.53.0.jar -Dwebdriver.chrome.driver=/usr/lib/chromium-browser/chromedriver -role node -hub http://127.0.0.1:4444/grid/register -browser browserName=firefox,version=58.0.2,firefox_binary=/usr/bin/firefox,maxInstances=1,platform=LINUX -browser browserName=chrome,version=48.0.2564.116,chrome_binary=/usr/bin/google-chrome,maxInstances=1,platform=LINUX -debug > selenum-hub.out 2>&1 &
#ubuntu
export DISPLAY=localhost:99.0 && nohup java -jar /workspace/selenium-server-standalone-2.53.0.jar -Dwebdriver.chrome.driver=/usr/lib/chromium-browser/chromedriver -role node -hub http://192.168.0.29:4444/grid/register -host 192.168.0.18 -timeout 500 -maxSession 2 -browser browserName=firefox,version=51.0.1,firefox_binary=/usr/bin/firefox,maxInstances=3,platform=LINUX -browser browserName=chrome,version=50.0.2661.102,chrome_binary=/usr/bin/google-chrome,maxInstances=3,platform=LINUX -debug > selenum-hub.out 2>&1 &
#freenas
export DISPLAY=localhost:99.0 && nohup java -jar /workspace/selenium-server-standalone-2.53.0.jar -Dwebdriver.chrome.driver=/usr/lib/chromium-browser/chromedriver -role node -hub http://192.168.0.29:4444/grid/register -host 192.168.0.26 -timeout 500 -maxSession 2 -browser browserName=firefox,version=58.0,firefox_binary=/usr/local/bin/firefox,maxInstances=3,platform=LINUX -browser browserName=chrome,version=61.0.3163.100,chrome_binary=/usr/local/bin/chrome,maxInstances=3,platform=LINUX -debug > selenum-hub.out 2>&1 &
multitail nohup.out selenum-hub.out

curl http://localhost:4444/grid/api/proxy?id=http://172.17.42.1:5555
curl http://albandrieu.com:4444/grid/console

/usr/bin/google-chrome
/opt/google/chrome/chrome --> NOK

#On albandrieu.com check starter script at
#/etc/init.d/selenium_hub

Check result at :

curl http://albandrieu.com:4444/grid/console

http://albandrieu.com:4444/grid/console

http://192.168.0.29:5555/wd/hub/static/resource/hub.html

Please use : ansible-selenium in order to install selenium hub and node

Jenkins

In order to properly configure Jenkins master

ADD /zapSource/build/zap in
sudo nano /etc/init/jenkins.conf
export JENKINS_HOME
export ZAPROXY_HOME=/zapSource/build/zap
export PATH=$PATH:$ZAPROXY_HOME

Yslow Psi WebPageTest

Run yslow, Psi or webpagetest

#grunt yslow_test
npm run pretest
grunt pagespeed
grunt wpt

Refresh from upstream

git remote add upstream https://github.com/AlbanAndrieu/nabla-servers-bower-sample.git
git remote -v
git fetch upstream
git checkout master
git rebase upstream/master
git rebase --continue
git push origin master --force

Deploy to internet

grunt gh-pages --gh-pages-tag 'v1.0.3' --gh-pages-message 'Tagging v1.0.3'

Page deployed at :

GitHub IO

GitHub gh-pages

#VirtualHost *
sudo nano /etc/apache2/sites-available/awstats.conf
#VirtualHost *:7075
sudo nano /etc/apache2/sites-available/sample.conf
#Add the following
Alias /bower_components "/var/www/sample/bower_components"
Alias /fonts "/var/www/sample/fonts"

Performance improvements

What has been done is described at https://www.youtube.com/watch?v=FEs2jgZBaQA

Issues

using-ui-bootstrap-causing-issues-with-carousel

In order to workaround issue of bower_components (cssmin) absolute path instead of releatif : how-to-avoid-grunt-build-not-loading-bootstrap-glyphicons-using-jhipster generator-angular

Run End2End tests

mvnw clean install org.codehaus.cargo:cargo-maven2-plugin:run -Dserver=jetty9x
grunt --gruntfile Gruntfile-e2e.js

Eclipse

Uninstall Nodeclipse Core & Node.js

Add plugins

• Groovy/Grails Tool Suite (GGTS) for Eclipse 3.6.4.RELEASE
• Nodeclipse "Enide Studio 2014" is Tool Suite for Node.js
• AngularJS Eclipse Plugin

Import as Maven project

Java

Project Properties | Java Build Path | Default output folder Change target by target-eclipse bower-sample/target-eclipse/classes

Exclude validation for

• node
• node_modules
• bower_components
• dist
• app
• coverage

Terraform

Init

Using terraform cloud to store Terraform State

terraform login
terraform init -backend-config=config.remote.tfbackend

terraform workspace list

Using consul backend to Terraform State in KV

terraform init -backend-config=config.consul.tfbackend
terraform init -reconfigure
#terraform init \
#-backend-config="address=10.30.0.66:8500" \
#-backend-config="path=uat/terraform_state" \
#-backend-config="scheme=http"
terraform plan

Applying terraform

Apply vault configuration

terraform validate
terraform init
terraform fmt
terraform plan -input=false --out plan
terraform apply
terraform show
terraform destroy

terraform plan --destroy -out=example.plan
terraform show example.plan
terraform state list

terraform output -raw website_endpoint
aws s3 cp /workspace/users/albandrieu30/nabla-site-apache/ s3://nabla-tf-exemple-bucket/ --recursive

terraform graph | grep -v -e 'meta' -e 'close' -e 's3' -e 'vpc' > terraform.gv
terraform graph | grep -v -e 'meta' -e 'close' -e 's3' -e 'vpc' | dot -Tpng > terraform.png

Update terraform documentation

terraform-docs .

Requirements

Name	Version
terraform	1.2.5
aws	~> 4.0

Providers

Name	Version
aws	4.11.0

Modules

Name	Source	Version
site	./modules/site	n/a

Resources

Name	Type
aws_default_vpc.default	resource
aws_dynamodb_table.terraform_locks	resource
aws_s3_bucket.terraform_state	resource
aws_security_group.prod_web	resource

Inputs

Name	Description	Type	Default	Required
CI_PROJECT_NAME	n/a	string	"default"	no
aws_region	AWS region to launch servers.	string	"eu-west-3"	no
github_organization	Github organization	string	"Banbou"	no
site	Site Name	string	n/a	yes
website_bucket_name	Nabla bower site	string	"nabla-tf-site-bucket"	no
website_root	Path to the root of website content	string	"./dist"	no
whitelist	n/a	list(string)	n/a	yes

Outputs

Name	Description
dynamodb_table_name	The name of the DynamoDB table
project_name	n/a
s3_bucket_arn	The ARN of the S3 bucket

Check docker image quality

Docker dive

dive --ci --json docker-dive-stats.json "${DOCKER_ORGANISATION}/pgclient:0.1.0" 1>docker-dive.log 2>docker-dive-error.log

Docker linter

sudo npm install -g dockerfile_lint
dockerfile_lint --json --verbose --dockerfile Dockerfile

brew install hadolint
hadolint Dockerfile

Javascript linter

npm install
# yarn add -D prettier@^2.5.1 eslint@^8.7.0
yarn run eslint

Ansible linter

ansible-lint --write ansible/

Mega linter

npx mega-linter-runner --install
npx mega-linter-runner

Check secret

npx @secretlint/quick-start "**/*"

Check cspell

npx cspell "**/*.{txt,js,md}"

Update README.md

• github-markdown-toc
• With github-markdown-toc

sudo npm install -g markdown-toc
markdown-toc README.md -i
markdown-toc CHANGELOG.md -i
sudo npm install -g markdown-link-check
markdown-link-check ./README.md

Other resources

Yo generator : https://github.com/peterhendrick/openSource

Thanks for reading!

Contributing

The issue tracker is the preferred channel for bug reports, features requests and submitting pull requests.

For pull requests, editor preferences are available in the editor config for easy use in common text editors. Read more and download plugins at http://editorconfig.org.

License

Apache v2

---

Alban Andrieu linkedin