diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..2461d80 --- /dev/null +++ b/.github/CODE_OF_CONDUCT.md @@ -0,0 +1,74 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to making participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, gender identity and expression, level of experience, +nationality, personal appearance, race, religion, or sexual identity and +orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or + advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic + address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces +when an individual is representing the project or its community. Examples of +representing a project or community include using an official project e-mail +address, posting via an official social media account, or acting as an appointed +representative at an online or offline event. Representation of a project may be +further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at [labs@idealista.com](mailto:labs@idealista.com). All +complaints will be reviewed and investigated and will result in a response that +is deemed necessary and appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, +available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html + +[homepage]: https://www.contributor-covenant.org diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md new file mode 100644 index 0000000..4715174 --- /dev/null +++ b/.github/CONTRIBUTING.md @@ -0,0 +1,100 @@ +# Contributing to Idealista + +:+1::tada: First off, thanks for taking the time to contribute! :tada::+1: + +The following is a set of guidelines for contributing to Idealista's repositories, which are hosted in the [Idealista Organization](https://github.com/idealista) on GitHub. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request. + +#### Table Of Contents + +[Code of Conduct](#code-of-conduct) + +[How Can I Contribute?](#how-can-i-contribute) + * [Reporting Bugs](#reporting-bugs) + * [Suggesting Enhancements](#suggesting-enhancements) + * [Pull Requests](#pull-requests) + * [Changelog](#changelog) + +[Styleguides](#styleguides) + * [Git Commit Messages](#git-commit-messages) + +## Code of Conduct + +This project and everyone participating in it is governed by the [Idealista Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to [labs@idealista.com](mailto:labs@idealista.com). + + +## How Can I Contribute? + +### Reporting Bugs + +This section guides you through submitting a bug report for Idealista. Following these guidelines helps maintainers and the community understand your report :pencil:, reproduce the behavior :computer: :computer:, and find related reports :mag_right:. + +Before creating bug reports, please check [this list](#before-submitting-a-bug-report) as you might find out that you don't need to create one. When you are creating a bug report, please [include as many details as possible](#how-do-i-submit-a-good-bug-report). Fill out [the required template](ISSUE_TEMPLATE.md), the information it asks for helps us resolve issues faster. + +> **Note:** If you find a **Closed** issue that seems like it is the same thing that you're experiencing, open a new issue and include a link to the original issue in the body of your new one. + +#### Before Submitting A Bug Report + +* **Check the last version.** Check if you can reproduce the problem in the latest version of the project. +* **Check the FAQ of the project** for a list of common questions and problems. +* **Perform a [cursory search](https://github.com/issues?q=+is%3Aissue+user%3Aidealista)** to see if the problem has already been reported. If it has **and the issue is still open**, add a comment to the existing issue instead of opening a new one. + +#### How Do I Submit A (Good) Bug Report? + +Bugs are tracked as [GitHub issues](https://guides.github.com/features/issues/). Create an issue on the project repository and provide the following information by filling in [the template](ISSUE_TEMPLATE.md). + +Explain the problem and include additional details to help maintainers reproduce the problem: + +* **Use a clear and descriptive title** for the issue to identify the problem. +* **Describe the exact steps which reproduce the problem** in as many details as possible. +* **Describe the behavior you observed after following the steps** and point out what exactly is the problem with that behavior. +* **Explain which behavior you expected to see instead and why.** + +### Suggesting Enhancements + +This section guides you through submitting an enhancement suggestion for Idealista, including completely new features and minor improvements to existing functionality. Following these guidelines helps maintainers and the community understand your suggestion :pencil: and find related suggestions :mag_right:. + +Before creating enhancement suggestions, please check [this list](#before-submitting-an-enhancement-suggestion) as you might find out that you don't need to create one. When you are creating an enhancement suggestion, please [include as many details as possible](#how-do-i-submit-a-good-enhancement-suggestion). Fill in [the template](ISSUE_TEMPLATE.md), including the steps that you imagine you would take if the feature you're requesting existed. + +#### Before Submitting An Enhancement Suggestion + +* **Check the last version.** Check if you can reproduce the problem in the latest version of the project. +* **Check the FAQ of the project** for a list of common questions and problems. +* **Perform a [cursory search](https://github.com/issues?q=+is%3Aissue+user%3Aidealista)** to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one. + +#### How Do I Submit A (Good) Enhancement Suggestion? + +Enhancement suggestions are tracked as [GitHub issues](https://guides.github.com/features/issues/). Create an issue on the project repository and provide the following information by filling in [the template](ISSUE_TEMPLATE.md): + +* **Use a clear and descriptive title** for the issue to identify the suggestion. +* **Provide a step-by-step description of the suggested enhancement** in as many details as possible. +* **Provide specific examples to demonstrate the steps**. +* **Describe the current behavior** and **explain which behavior you expected to see instead** and why. +* **Explain why this enhancement would be useful**. +* **List some other text editors or applications where this enhancement exists.** +* **Specify which version are you're using.** + +### Pull Requests + +* Fill in [the required template](PULL_REQUEST_TEMPLATE.md) +* Any pull request should has **idealista:develop** as base branch. + +### Changelog + +Every project has a CHANGELOG.md file. Once your code is ready to be merged please fill the issue after the **Unreleased** section as explained: + +* For an enhancement, fill the issue after the **Added** subsection (create it if doesn't exists) +* For a fixed bug, fill the issue after the **Fixed** subsection (create it if doesn't exists) +* For an improvement, fill the issue after the **Changed** subsection (create it if doesn't exists) + +Then write the issue info this way: + +- *[#29](https://github.com/idealista/nginx-role/issues/29) Support debian stretch* @jmonterrubio + +## Styleguides + +### Git Commit Messages + +* Use the present tense ("Add feature" not "Added feature") +* Use the imperative mood ("Move cursor to..." not "Moves cursor to...") +* Limit the first line to 72 characters or less +* Reference issues and pull requests liberally after the first line diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md new file mode 100644 index 0000000..b07e35f --- /dev/null +++ b/.github/ISSUE_TEMPLATE.md @@ -0,0 +1,36 @@ + + +### Prerequisites + +* [ ] Put an X between the brackets on this line if you have done all of the following: + * Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista + * Checked that there is not already provided the described functionality + +### Description + +[Description of the issue] + +### Steps to Reproduce + +1. [First Step] +2. [Second Step] +3. [and so on...] + +**Expected behavior:** [What you expect to happen] + +**Actual behavior:** [What actually happens] + +**Reproduces how often:** [What percentage of the time does it reproduce?] + +### Versions + +The version/s you notice the behavior. + +### Additional Information + +Any additional information, configuration or data that might be necessary to reproduce the issue. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..8fa0af8 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,26 @@ +### Requirements + +* Filling out the template is required. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion. +* All new code requires tests to ensure against regressions +* Remember to set **idealista:develop** as base branch; + +### Description of the Change + + + + +### Benefits + + + +### Possible Drawbacks + + + +### Applicable Issues + + diff --git a/.travis.yml b/.travis.yml index 287f5cb..0d8e9f7 100644 --- a/.travis.yml +++ b/.travis.yml @@ -7,8 +7,9 @@ group: deprecated-2017Q4 services: - docker install: - - pip install ansible==2.3.1.0 + - pip install ansible==2.4.1.0 - pip install molecule==1.25.0 + - pip install ansible-lint==3.4.20 - pip install docker script: - molecule test --driver docker diff --git a/CHANGELOG.md b/CHANGELOG.md index ca13139..bd2142b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ This project adheres to [Semantic Versioning](http://semver.org/) and [Keep a ch ## [Unreleased](https://github.com/idealista/nginx-role/tree/develop) - *[#6](https://github.com/idealista/nginx-role/issues/6) Add Travis CI* @jnogol +- *[#42](https://github.com/idealista/nginx-role/issues42) Update Prometheus metrics* @jmonterrubio ## [1.6.0](https://github.com/idealista/nginx-role/tree/1.6.0) (2017-10-26) [Full Changelog](https://github.com/idealista/nginx-role/compare/1.5.0...1.6.0) diff --git a/LICENSE.txt b/LICENSE similarity index 99% rename from LICENSE.txt rename to LICENSE index 20d3ee1..29be8b9 100644 --- a/LICENSE.txt +++ b/LICENSE @@ -72,7 +72,7 @@ To apply the Apache License to your work, attach the following boilerplate notic limitations under the License. /** - * Copyright Idealista S.A. + * Copyright Idealista S.A.U. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/README.md b/README.md index 30f1c23..1bf3473 100644 --- a/README.md +++ b/README.md @@ -23,10 +23,10 @@ These instructions will get you a copy of the role for your ansible playbook. On ### Prerequisities -Ansible 2.2.1.0 version installed. +Ansible 2.4.1.0 version installed. Inventory destination should be a Debian environment. -For testing purposes, [Molecule](https://molecule.readthedocs.io/) with [Vagrant](https://www.vagrantup.com/) as driver (with [landrush](https://github.com/vagrant-landrush/landrush) plugin) and [VirtualBox](https://www.virtualbox.org/) as provider. +For testing purposes, [Molecule](https://molecule.readthedocs.io/) with [Vagrant](https://www.vagrantup.com/) as driver (with [vagrant-hostmanager](https://github.com/devopsgroup-io/vagrant-hostmanager) plugin) and [VirtualBox](https://www.virtualbox.org/) as provider. ### Installing @@ -61,6 +61,15 @@ Look to the [defaults](defaults/main.yml) properties file to see the possible co You can add new servers to nginx by including your server as a file or as a template, setting the server files in the path defined by `nginx_extra_servers_path` or `nginx_extra_servers_template_path` (see tests as example). +If you want to change any version do it at your risk. Some nginx versions and modules are not compatible. +Known compatible versions are: + +``` +Nginx: 1.10.* + lua_module_version: 0.10.7 +Nginx: 1.12.* + lua_module_version: 0.10.11 + +``` + ## Testing ``` @@ -71,7 +80,7 @@ See molecule.yml to check possible testing platforms. ## Built With -![Ansible](https://img.shields.io/badge/ansible-2.2.1.0-green.svg) +![Ansible](https://img.shields.io/badge/ansible-2.4.1.0-green.svg) ## Versioning @@ -94,3 +103,9 @@ This project is licensed under the [Apache 2.0](https://www.apache.org/licenses/ ## Contributing Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us. + +## Acknowledgments + +* [Anton Tolchanov](https://github.com/knyar/) - For the [prometheus metric library for Nginx](https://github.com/knyar/nginx-lua-prometheus) + +* [Sophos](https://github.com/hnlq715/) - For the [metrics](https://github.com/hnlq715/nginx-prometheus-metrics/blob/master/metrics.vhost) diff --git a/defaults/main.yml b/defaults/main.yml index 7280a10..2c34354 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -1,7 +1,7 @@ --- ## General -nginx_version: 1.10.2 +nginx_version: 1.12.2 ## Service options @@ -44,10 +44,11 @@ nginx_logrotate_template_path: "{{ playbook_dir }}/templates/logrotate.j2" headers_more_version: 0.31 # Lua -lua_module_version: 0.10.7 +lua_module_version: 0.10.11 lua_version: 5.1.5 ## Metrics +nginx_lua_prometheus_version: 0.1-20170610 nginx_prometheus_metrics_enabled: true nginx_prometheus_metrics_port: 9145 nginx_prometheus_metrics_role: nginx diff --git a/files/prometheus.lua b/files/prometheus.lua deleted file mode 100644 index c8ba167..0000000 --- a/files/prometheus.lua +++ /dev/null @@ -1,424 +0,0 @@ --- vim: ts=2:sw=2:sts=2:expandtab --- --- This module uses a single dictionary shared between Nginx workers to keep --- all metrics. Each counter is stored as a separate entry in that dictionary, --- which allows us to increment them using built-in `incr` method. --- --- Prometheus requires that (a) all samples for a given metric are presented --- as one uninterrupted group, and (b) buckets of a histogram appear in --- increasing numerical order. We satisfy that by carefully constructing full --- metric names (i.e. metric name along with all labels) so that they meet --- those requirements while being sorted alphabetically. In particular: --- --- * all labels for a given metric are presented in reproducible order (the one --- used when labels were declared). "le" label for histogram metrics always --- goes last; --- * bucket boundaries (which are exposed as values of the "le" label) are --- presented as floating point numbers with leading and trailing zeroes. --- Number of of zeroes is determined for each bucketer automatically based on --- bucket boundaries; --- * internally "+Inf" bucket is stored as "Inf" (to make it appear after --- all numeric buckets), and gets replaced by "+Inf" just before we --- expose the metrics. --- --- For example, if you define your bucket boundaries as {0.00005, 10, 1000} --- then we will keep the following samples for a metric `m1` with label --- `site` set to `site1`: --- --- m1_bucket{site="site1",le="0000.00005"} --- m1_bucket{site="site1",le="0010.00000"} --- m1_bucket{site="site1",le="1000.00000"} --- m1_bucket{site="site1",le="Inf"} --- m1_count{site="site1"} --- m1_sum{site="site1"} --- --- "Inf" will be replaced by "+Inf" while publishing metrics. --- --- You can find the latest version and documentation at --- https://github.com/knyar/nginx-lua-prometheus --- Released under MIT license. - - --- Default set of latency buckets, 5ms to 10s: -local DEFAULT_BUCKETS = {0.005, 0.01, 0.02, 0.03, 0.05, 0.075, 0.1, 0.2, 0.3, - 0.4, 0.5, 0.75, 1, 1.5, 2, 3, 4, 5, 10} - --- Metric is a "parent class" for all metrics. -local Metric = {} -function Metric:new(o) - o = o or {} - setmetatable(o, self) - self.__index = self - return o -end - --- Checks that the right number of labels values have been passed. --- --- Args: --- label_values: an array of label values. --- --- Returns: --- an error message or nil -function Metric:check_labels(label_values) - if self.label_names == nil and label_values == nil then - return - elseif self.label_names == nil and label_values ~= nil then - return "Expected no labels for " .. self.name .. ", got " .. #label_values - elseif label_values == nil and self.label_names ~= nil then - return "Expected " .. #self.label_names .. " labels for " .. - self.name .. ", got none" - elseif #self.label_names ~= #label_values then - return "Wrong number of labels for " .. self.name .. ". Expected " .. - #self.label_names .. ", got " .. #label_values - end -end - -local Counter = Metric:new() --- Increase a given counter by `value` --- --- Args: --- value: (number) a value to add to the counter. Defaults to 1 if skipped. --- label_values: an array of label values. Can be nil (i.e. not defined) for --- metrics that have no labels. -function Counter:inc(value, label_values) - local err = self:check_labels(label_values) - if err ~= nil then - self.prometheus:log_error(err) - return - end - self.prometheus:inc(self.name, self.label_names, label_values, value or 1) -end - -local Histogram = Metric:new() --- Record a given value in a histogram. --- --- Args: --- value: (number) a value to record. Should be defined. --- label_values: an array of label values. Can be nil (i.e. not defined) for --- metrics that have no labels. -function Histogram:observe(value, label_values) - if value == nil then - self.prometheus:log_error("No value passed for " .. self.name) - return - end - local err = self:check_labels(label_values) - if err ~= nil then - self.prometheus:log_error(err) - return - end - self.prometheus:histogram_observe(self.name, self.label_names, label_values, value) -end - -local Prometheus = {} -Prometheus.__index = Prometheus -Prometheus.initialized = false - --- Generate full metric name that includes all labels. --- --- Args: --- name: string --- label_names: (array) a list of label keys. --- label_values: (array) a list of label values. --- Returns: --- (string) full metric name. -local function full_metric_name(name, label_names, label_values) - if not label_names then - return name - end - local label_parts = {} - for idx, key in ipairs(label_names) do - local label_value = (string.format("%s", label_values[idx]) - :gsub("\\", "\\\\") - :gsub("\n", "\\n") - :gsub('"', '\\"')) - table.insert(label_parts, key .. '="' .. label_value .. '"') - end - return name .. "{" .. table.concat(label_parts, ",") .. "}" -end - --- Construct bucket format for a list of buckets. --- --- This receives a list of buckets and returns a sprintf template that should --- be used for bucket boundaries to make them come in increasing order when --- sorted alphabetically. --- --- To re-phrase, this is where we detect how many leading and trailing zeros we --- need. --- --- Args: --- buckets: a list of buckets --- --- Returns: --- (string) a sprintf template. -local function construct_bucket_format(buckets) - local max_order = 1 - local max_precision = 1 - for _, bucket in ipairs(buckets) do - assert(type(bucket) == "number", "bucket boundaries should be numeric") - -- floating point number with all trailing zeros removed - local as_string = string.format("%f", bucket):gsub("0*$", "") - local dot_idx = as_string:find(".", 1, true) - max_order = math.max(max_order, dot_idx - 1) - max_precision = math.max(max_precision, as_string:len() - dot_idx) - end - return "%0" .. (max_order + max_precision + 1) .. "." .. max_precision .. "f" -end - --- Extract short metric name from the full one. --- --- Args: --- full_name: (string) full metric name that can include labels. --- --- Returns: --- (string) short metric name with no labels. For a `*_bucket` metric of --- histogram the _bucket suffix will be removed. -local function short_metric_name(full_name) - local labels_start, _ = full_name:find("{") - if not labels_start then - -- no labels - return full_name - end - local suffix_idx, _ = full_name:find("_bucket{") - if suffix_idx and full_name:find("le=") then - -- this is a histogram metric - return full_name:sub(1, suffix_idx - 1) - end - -- this is not a histogram metric - return full_name:sub(1, labels_start - 1) -end - --- Makes a shallow copy of a table -local function copy_table(table) - local new = {} - if table ~= nil then - for k, v in ipairs(table) do - new[k] = v - end - end - return new -end - --- Initialize the module. --- --- This should be called once from the `init_by_lua` section in nginx --- configuration. --- --- Args: --- dict_name: (string) name of the nginx shared dictionary which will be --- used to store all metrics --- prefix: (optional string) if supplied, prefix is added to all --- metric names on output --- --- Returns: --- an object that should be used to register metrics. -function Prometheus.init(dict_name, prefix) - local self = setmetatable({}, Prometheus) - self.dict = ngx.shared[dict_name or "prometheus_metrics"] - self.help = {} - if prefix then - self.prefix = prefix - else - self.prefix = '' - end - self.type = {} - self.registered = {} - self.buckets = {} - self.bucket_format = {} - self.initialized = true - - self:counter("nginx_metric_errors_total", - "Number of nginx-lua-prometheus errors") - self.dict:set("nginx_metric_errors_total", 0) - return self -end - -function Prometheus:log_error(...) - ngx.log(ngx.ERR, ...) - self.dict:incr("nginx_metric_errors_total", 1) -end - -function Prometheus:log_error_kv(key, value, err) - self:log_error( - "Error while setting '", key, "' to '", value, "': '", err, "'") -end - --- Register a counter. --- --- Args: --- name: (string) name of the metric. Required. --- description: (string) description of the metric. Will be used for the HELP --- comment on the metrics page. Optional. --- label_names: array of strings, defining a list of metrics. Optional. --- --- Returns: --- a Counter object. -function Prometheus:counter(name, description, label_names) - if not self.initialized then - ngx.log(ngx.ERR, "Prometheus module has not been initialized") - return - end - - if self.registered[name] then - self:log_error("Duplicate metric " .. name) - return - end - self.registered[name] = true - self.help[name] = description - self.type[name] = "counter" - - return Counter:new{name=name, label_names=label_names, prometheus=self} -end - --- Register a histogram. --- --- Args: --- name: (string) name of the metric. Required. --- description: (string) description of the metric. Will be used for the HELP --- comment on the metrics page. Optional. --- label_names: array of strings, defining a list of metrics. Optional. --- buckets: array if numbers, defining bucket boundaries. Optional. --- --- Returns: --- a Counter object. -function Prometheus:histogram(name, description, label_names, buckets) - if not self.initialized then - ngx.log(ngx.ERR, "Prometheus module has not been initialized") - return - end - - for _, label_name in ipairs(label_names or {}) do - if label_name == "le" then - self:log_error("Invalid label name 'le' in " .. name) - return - end - end - - for _, suffix in ipairs({"", "_bucket", "_count", "_sum"}) do - if self.registered[name .. suffix] then - self:log_error("Duplicate metric " .. name .. suffix) - return - end - self.registered[name .. suffix] = true - end - self.help[name] = description - self.type[name] = "histogram" - - self.buckets[name] = buckets or DEFAULT_BUCKETS - self.bucket_format[name] = construct_bucket_format(self.buckets[name]) - - return Histogram:new{name=name, label_names=label_names, prometheus=self} -end - --- Set a given dictionary key. --- This overwrites existing values, so we use it only to initialize metrics. -function Prometheus:set(key, value) - local ok, err = self.dict:safe_set(key, value) - if not ok then - self:log_error_kv(key, value, err) - end -end - --- Increment a given counter by `value`. --- --- Args: --- name: (string) short metric name without any labels. --- label_names: (array) a list of label keys. --- label_values: (array) a list of label values. --- value: (number) value to add. Optional, defaults to 1. -function Prometheus:inc(name, label_names, label_values, value) - local key = full_metric_name(name, label_names, label_values) - if value == nil then value = 1 end - if value < 0 then - self:log_error_kv(key, value, "Value should not be negative") - return - end - - local newval, err = self.dict:incr(key, value) - if newval then - return - end - -- Yes, this looks like a race, so I guess we might under-report some values - -- when multiple workers simultaneously try to create the same metric. - -- Hopefully this does not happen too often (shared dictionary does not get - -- reset during configuation reload). - if err == "not found" then - self:set(key, value) - return - end - -- Unexpected error - self:log_error_kv(key, value, err) -end - --- Record a given value into a histogram metric. --- --- Args: --- name: (string) short metric name without any labels. --- label_names: (array) a list of label keys. --- label_values: (array) a list of label values. --- value: (number) value to observe. -function Prometheus:histogram_observe(name, label_names, label_values, value) - self:inc(name .. "_count", label_names, label_values, 1) - self:inc(name .. "_sum", label_names, label_values, value) - - -- we are going to mutate arrays of label names and values, so create a copy. - local l_names = copy_table(label_names) - local l_values = copy_table(label_values) - - -- Last bucket. Note, that the label value is "Inf" rather than "+Inf" - -- required by Prometheus. This is necessary for this bucket to be the last - -- one when all metrics are lexicographically sorted. "Inf" will get replaced - -- by "+Inf" in Prometheus:collect(). - table.insert(l_names, "le") - table.insert(l_values, "Inf") - self:inc(name .. "_bucket", l_names, l_values, 1) - - local label_count = #l_names - for _, bucket in ipairs(self.buckets[name]) do - if value <= bucket then - -- last label is now "le" - l_values[label_count] = self.bucket_format[name]:format(bucket) - self:inc(name .. "_bucket", l_names, l_values, 1) - end - end -end - --- Present all metrics in a text format compatible with Prometheus. --- --- This function should be used to expose the metrics on a separate HTTP page. --- It will get the metrics from the dictionary, sort them, and expose them --- aling with TYPE and HELP comments. -function Prometheus:collect() - ngx.header.content_type = "text/plain" - if not self.initialized then - ngx.log(ngx.ERR, "Prometheus module has not been initialized") - return - end - - local keys = self.dict:get_keys(0) - -- Prometheus server expects buckets of a histogram to appear in increasing - -- numerical order of their label values. - table.sort(keys) - - local seen_metrics = {} - for _, key in ipairs(keys) do - local value, err = self.dict:get(key) - if value then - local short_name = short_metric_name(key) - if not seen_metrics[short_name] then - if self.help[short_name] then - ngx.say("# HELP " .. self.prefix .. short_name .. " " .. self.help[short_name]) - end - if self.type[short_name] then - ngx.say("# TYPE " .. self.prefix .. short_name .. " " .. self.type[short_name]) - end - seen_metrics[short_name] = true - end - -- Replace "Inf" with "+Inf" in each metric's last bucket 'le' label. - ngx.say(self.prefix .. key:gsub('le="Inf"', 'le="+Inf"'), " ", value) - else - self:log_error("Error getting '", key, "': ", err) - end - end -end - -return Prometheus diff --git a/meta/main.yml b/meta/main.yml index 8b678d5..c997ace 100644 --- a/meta/main.yml +++ b/meta/main.yml @@ -2,12 +2,13 @@ galaxy_info: company: Idealista S.A. description: Nginx server - min_ansible_version: 2.2 + min_ansible_version: 2.4.1.0 license: Apache 2.0 platforms: - name: Debian versions: - jessie + - stretch categories: - webserver - proxy diff --git a/molecule.yml b/molecule.yml index efe99b6..b7a0068 100644 --- a/molecule.yml +++ b/molecule.yml @@ -29,9 +29,12 @@ ansible: # configuration options for the internal call to vagrant vagrant: raw_config_args: - - "landrush.enabled = true" - - "landrush.tld = 'vm'" - - "landrush.guest_redirect_dns = false" + - "hostmanager.enabled = true" + # Enable for debug purpose + #- "hostmanager.manage_host = true" + - "hostmanager.manage_guest = true" + - "hostmanager.ignore_private_ip = false" + - "hostmanager.include_offline = true" # molecule's --platform option will look for these names platforms: - name: Debian9 @@ -50,18 +53,18 @@ vagrant: - name: nginx.vm ansible_groups: - group01 - - nginx interfaces: - network_name: private_network - type: dhcp + type: static + ip: 172.28.128.10 auto_config: true - name: nginx-old.vm ansible_groups: - - group01 - - nginx_old + - group02 interfaces: - network_name: private_network - type: dhcp + type: static + ip: 172.28.128.11 auto_config: true docker: network: @@ -71,7 +74,6 @@ docker: - name: nginx ansible_groups: - group01 - - nginx dockerfile: tests/Dockerfile image: nginx_role image_version: latest @@ -86,7 +88,6 @@ docker: - name: nginx-old ansible_groups: - group01 - - nginx_old dockerfile: tests/Dockerfile image: nginx_role image_version: latest diff --git a/tasks/config.yml b/tasks/config.yml index 87aaef9..7497b0d 100644 --- a/tasks/config.yml +++ b/tasks/config.yml @@ -58,6 +58,7 @@ mode: 0640 with_fileglob: - "{{ nginx_extra_servers_path }}/*" + notify: reload nginx - name: NGINX | Copy servers templates template: @@ -68,6 +69,7 @@ mode: 0640 with_fileglob: - "{{ nginx_extra_servers_template_path }}/*" + notify: reload nginx - name: NGINX | Set servers files loaded var find: @@ -106,6 +108,7 @@ state: absent with_items: - "{{ all_servers_available.files | map(attribute='path') | map('basename') | list | difference(nginx_available_servers) }}" + notify: reload nginx - name: NGINX | Remove old servers available file: diff --git a/tasks/dependencies.yml b/tasks/dependencies.yml index 9b9e287..a038bbf 100644 --- a/tasks/dependencies.yml +++ b/tasks/dependencies.yml @@ -19,17 +19,23 @@ url: "{{ lua_url }}" dest: /tmp validate_certs: no - when: 'nginx_force_reinstall or lua_check|failed or "Lua {{ lua_version }}" not in lua_check.stderr' + when: nginx_force_reinstall or + lua_check|failed or + "Lua " + lua_version not in lua_check.stderr - name: NGINX | Untar lua unarchive: copy: no src: /tmp/lua-{{ lua_version }}.tar.gz dest: /tmp - when: 'nginx_force_reinstall or lua_check|failed or "Lua {{ lua_version }}" not in lua_check.stderr' + when: nginx_force_reinstall or + lua_check|failed or + "Lua " + lua_version not in lua_check.stderr - name: NGINX | Build lua command: "chdir=/tmp/lua-{{ lua_version }} {{item}}" with_items: - make linux install - when: 'nginx_force_reinstall or lua_check|failed or "Lua {{ lua_version }}" not in lua_check.stderr' + when: nginx_force_reinstall or + lua_check|failed or + "Lua " + lua_version not in lua_check.stderr diff --git a/tasks/install.yml b/tasks/install.yml index 8c84bb7..33eac68 100644 --- a/tasks/install.yml +++ b/tasks/install.yml @@ -24,41 +24,53 @@ url: "{{ headers_more_url }}" dest: "{{ nginx_download_dir }}" validate_certs: no - when: 'nginx_force_reinstall or nginx_check|failed or "nginx/{{ nginx_version }}" not in nginx_check.stderr' + when: nginx_force_reinstall or + nginx_check|failed or + "nginx/" + nginx_version not in nginx_check.stderr - name: NGINX | Untar headers-more unarchive: copy: no src: "{{ nginx_download_dir }}/headers-more-nginx-module-{{ headers_more_version }}.tar.gz" dest: "{{ nginx_src_dir }}" - when: 'nginx_force_reinstall or nginx_check|failed or "nginx/{{ nginx_version }}" not in nginx_check.stderr' + when: nginx_force_reinstall or + nginx_check|failed or + "nginx/" + nginx_version not in nginx_check.stderr - name: NGINX | Download lua module get_url: url: "{{ lua_module_url }}" dest: "{{ nginx_download_dir }}" validate_certs: no - when: 'nginx_force_reinstall or nginx_check|failed or "nginx/{{ nginx_version }}" not in nginx_check.stderr' + when: nginx_force_reinstall or + nginx_check|failed or + "nginx/" + nginx_version not in nginx_check.stderr - name: NGINX | Untar lua module unarchive: copy: no src: "{{ nginx_download_dir }}/lua-nginx-module-{{ lua_module_version }}.tar.gz" dest: "{{ nginx_src_dir }}" - when: 'nginx_force_reinstall or nginx_check|failed or "nginx/{{ nginx_version }}" not in nginx_check.stderr' + when: nginx_force_reinstall or + nginx_check|failed or + "nginx/" + nginx_version not in nginx_check.stderr - name: NGINX | Download nginx get_url: url: "{{ nginx_url }}" dest: "{{ nginx_download_dir }}" - when: 'nginx_force_reinstall or nginx_check|failed or "nginx/{{ nginx_version }}" not in nginx_check.stderr' + when: nginx_force_reinstall or + nginx_check|failed or + "nginx/" + nginx_version not in nginx_check.stderr - name: NGINX | Untar nginx unarchive: copy: no src: "{{ nginx_download_dir }}/nginx-{{ nginx_version }}.tar.gz" dest: "{{ nginx_src_dir }}" - when: 'nginx_force_reinstall or nginx_check|failed or "nginx/{{ nginx_version }}" not in nginx_check.stderr' + when: nginx_force_reinstall or + nginx_check|failed or + "nginx/" + nginx_version not in nginx_check.stderr - name: NGINX | Define compile-time options set_fact: @@ -82,20 +94,26 @@ command: ./configure {{ nginx_compile_time_options_str }} {{ nginx_compile_time_options_builtin_modules_str }} {{ nginx_compile_time_options_external_modules_paths_str }} args: chdir: "{{ nginx_src_dir }}/nginx-{{ nginx_version }}" - when: 'nginx_force_reinstall or nginx_check|failed or "nginx/{{ nginx_version }}" not in nginx_check.stderr' + when: nginx_force_reinstall or + nginx_check|failed or + "nginx/" + nginx_version not in nginx_check.stderr notify: restart nginx - name: NGINX | Make make: chdir: "{{ nginx_src_dir }}/nginx-{{ nginx_version }}" - when: 'nginx_force_reinstall or nginx_check|failed or "nginx/{{ nginx_version }}" not in nginx_check.stderr' + when: nginx_force_reinstall or + nginx_check|failed or + "nginx/" + nginx_version not in nginx_check.stderr notify: restart nginx - name: NGINX | Make install make: chdir: "{{ nginx_src_dir }}/nginx-{{ nginx_version }}" target: install - when: 'nginx_force_reinstall or nginx_check|failed or "nginx/{{ nginx_version }}" not in nginx_check.stderr' + when: nginx_force_reinstall or + nginx_check|failed or + "nginx/" + nginx_version not in nginx_check.stderr notify: restart nginx - name: NGINX | Remove old upstart daemon script diff --git a/tasks/main.yml b/tasks/main.yml index c1493ed..b7b8bf6 100644 --- a/tasks/main.yml +++ b/tasks/main.yml @@ -1,27 +1,27 @@ --- - name: NGINX | Dependencies - include: dependencies.yml + import_tasks: dependencies.yml tags: - dependencies - name: NGINX | Install - include: install.yml + import_tasks: install.yml tags: - install - name: NGINX | Configure - include: config.yml + import_tasks: config.yml tags: - configure - name: NGINX | Metrics - include: metrics.yml - when: 'nginx_prometheus_metrics_enabled' + import_tasks: metrics.yml + when: nginx_prometheus_metrics_enabled tags: - metrics - name: NGINX | Service - include: service.yml + import_tasks: service.yml tags: - service diff --git a/tasks/metrics.yml b/tasks/metrics.yml index 4a73c7f..14452f1 100644 --- a/tasks/metrics.yml +++ b/tasks/metrics.yml @@ -1,8 +1,15 @@ --- +- name: NGINX | Get nginx-lua-prometheus + unarchive: + src: "{{ nginx_lua_prometheus_url }}" + dest: "{{ nginx_download_dir }}" + remote_src: yes + - name: NGINX | Copy prometheus lib copy: - src: prometheus.lua + src: "{{ nginx_download_dir }}/nginx-lua-prometheus-{{ nginx_lua_prometheus_version }}/prometheus.lua" dest: "{{ nginx_conf_path }}/plugins/prometheus.lua" + remote_src: true owner: "{{ nginx_user }}" group: "{{ nginx_group }}" diff --git a/templates/metrics/prometheus.conf.j2 b/templates/metrics/prometheus.conf.j2 index fcaf4e6..59e1a2f 100644 --- a/templates/metrics/prometheus.conf.j2 +++ b/templates/metrics/prometheus.conf.j2 @@ -1,16 +1,108 @@ # PROMETHEUS METRIC lua_shared_dict prometheus_metrics 10M; lua_package_path "{{ nginx_conf_path }}/plugins/prometheus.lua"; -init_by_lua ' + +init_by_lua_block { prometheus = require("prometheus").init("prometheus_metrics") - metric_requests = prometheus:counter( - "nginx_http_requests_total", "Number of HTTP requests", {"host", "status", "role"}) - metric_latency = prometheus:histogram( - "nginx_http_request_duration_seconds", "HTTP request latency", {"host", "role"}) -'; -log_by_lua ' - local host = ngx.var.host:gsub("^www.", "") - local role = "{{ nginx_prometheus_metrics_role }}" - metric_requests:inc(1, {host, ngx.var.status, role}) - metric_latency:observe(ngx.now() - ngx.req.start_time(), {host, role}) - '; + + http_requests = prometheus:counter( + "nginx_http_requests", "Number of HTTP requests", {"host", "status"}) + http_request_time = prometheus:histogram( + "nginx_http_request_time", "HTTP request time", {"host"}) + http_request_bytes_received = prometheus:counter( + "nginx_http_request_bytes_received", "Number of HTTP request bytes received", {"host"}) + http_request_bytes_sent = prometheus:counter( + "nginx_http_request_bytes_sent", "Number of HTTP request bytes sent", {"host"}) + http_connections = prometheus:gauge( + "nginx_http_connections", "Number of HTTP connections", {"state"}) + http_upstream_cache_status = prometheus:counter( + "nginx_http_upstream_cache_status", "Number of HTTP upstream cache status", {"host", "status"}) + http_upstream_requests = prometheus:counter( + "nginx_http_upstream_requests", "Number of HTTP upstream requests", {"addr", "status"}) + http_upstream_response_time = prometheus:histogram( + "nginx_http_upstream_response_time", "HTTP upstream response time", {"addr"}) + http_upstream_header_time = prometheus:histogram( + "nginx_http_upstream_header_time", "HTTP upstream header time", {"addr"}) + http_upstream_bytes_received = prometheus:counter( + "nginx_http_upstream_bytes_received", "Number of HTTP upstream bytes received", {"addr"}) + http_upstream_bytes_sent = prometheus:counter( + "nginx_http_upstream_bytes_sent", "Number of HTTP upstream bytes sent", {"addr"}) + http_upstream_connect_time = prometheus:histogram( + "nginx_http_upstream_connect_time", "HTTP upstream connect time", {"addr"}) + http_upstream_first_byte_time = prometheus:histogram( + "nginx_http_upstream_first_byte_time", "HTTP upstream first byte time", {"addr"}) + http_upstream_session_time = prometheus:histogram( + "nginx_http_upstream_session_time", "HTTP upstream session time", {"addr"}) +} +log_by_lua_block { + local function split(str) + local array = {} + for mem in string.gmatch(str, '([^, ]+)') do + table.insert(array, mem) + end + return array + end + local function getWithIndex(str, idx) + if str == nil then + return nil + end + return split(str)[idx] + end + local host = ngx.var.host + local status = ngx.var.status + http_requests:inc(1, {host, status}) + http_request_time:observe(ngx.now() - ngx.req.start_time(), {host}) + http_request_bytes_sent:inc(tonumber(ngx.var.bytes_sent), {host}) + if ngx.var.bytes_received ~= nil then + http_request_bytes_received:inc(tonumber(ngx.var.bytes_received), {host}) + end + local upstream_cache_status = ngx.var.upstream_cache_status + if upstream_cache_status ~= nil then + http_upstream_cache_status:inc(1, {host, upstream_cache_status}) + end + local upstream_addr = ngx.var.upstream_addr + if upstream_addr ~= nil then + local addrs = split(upstream_addr) + + local upstream_status = ngx.var.upstream_status + local upstream_response_time = ngx.var.upstream_response_time + local upstream_connect_time = ngx.var.upstream_connect_time + local upstream_first_byte_time = ngx.var.upstream_first_byte_time + local upstream_header_time = ngx.var.upstream_header_time + local upstream_session_time = ngx.var.upstream_session_time + local upstream_bytes_received = ngx.var.upstream_bytes_received + local upstream_bytes_sent = ngx.var.upstream_bytes_sent + -- compatible for nginx commas format + for idx, addr in ipairs(addrs) do + if table.getn(addrs) > 1 then + upstream_status = getWithIndex(ngx.var.upstream_status, idx) + upstream_response_time = getWithIndex(ngx.var.upstream_response_time, idx) + upstream_connect_time = getWithIndex(ngx.var.upstream_connect_time, idx) + upstream_first_byte_time = getWithIndex(ngx.var.upstream_first_byte_time, idx) + upstream_header_time = getWithIndex(ngx.var.upstream_header_time, idx) + upstream_session_time = getWithIndex(ngx.var.upstream_session_time, idx) + upstream_bytes_received = getWithIndex(ngx.var.upstream_bytes_received, idx) + upstream_bytes_sent = getWithIndex(ngx.var.upstream_bytes_sent, idx) + end + http_upstream_requests:inc(1, {addr, upstream_status}) + http_upstream_response_time:observe(tonumber(upstream_response_time), {addr}) + http_upstream_header_time:observe(tonumber(upstream_header_time), {addr}) + -- ngx.config.nginx_version >= 1011004 + if upstream_first_byte_time ~= nil then + http_upstream_first_byte_time:observe(tonumber(upstream_first_byte_time), {addr}) + end + if upstream_connect_time ~= nil then + http_upstream_connect_time:observe(tonumber(upstream_connect_time), {addr}) + end + if upstream_session_time ~= nil then + http_upstream_session_time:observe(tonumber(upstream_session_time), {addr}) + end + if upstream_bytes_received ~= nil then + http_upstream_bytes_received:inc(tonumber(upstream_bytes_received), {addr}) + end + if upstream_bytes_sent ~= nil then + http_upstream_bytes_sent:inc(tonumber(upstream_bytes_sent), {addr}) + end + end + end +} diff --git a/templates/metrics/prometheus.j2 b/templates/metrics/prometheus.j2 index ada069f..34db209 100644 --- a/templates/metrics/prometheus.j2 +++ b/templates/metrics/prometheus.j2 @@ -1,6 +1,14 @@ server { listen {{ nginx_prometheus_metrics_port }}; location /{{ nginx_prometheus_metrics_path }} { - content_by_lua 'prometheus:collect()'; + content_by_lua_block { + if ngx.var.connections_active ~= nil then + http_connections:set(ngx.var.connections_active, {"active"}) + http_connections:set(ngx.var.connections_reading, {"reading"}) + http_connections:set(ngx.var.connections_waiting, {"waiting"}) + http_connections:set(ngx.var.connections_writing, {"writing"}) + end + prometheus:collect() + } } } diff --git a/vars/main.yml b/vars/main.yml index 14e3298..eabaa7b 100644 --- a/vars/main.yml +++ b/vars/main.yml @@ -7,6 +7,7 @@ required_libs: nginx_url: http://nginx.org/download/nginx-{{ nginx_version }}.tar.gz headers_more_url: https://github.com/openresty/headers-more-nginx-module/archive/v{{ headers_more_version }}.tar.gz lua_module_url: https://github.com/openresty/lua-nginx-module/archive/v{{ lua_module_version }}.tar.gz +nginx_lua_prometheus_url: https://github.com/knyar/nginx-lua-prometheus/archive/{{ nginx_lua_prometheus_version }}.tar.gz lua_dependencies: - build-essential