Merge branch 'master' of https://github.com/telefonicaid/fiware-orion

into 2955_quote_in_url_not_escaped # Conflicts: # CHANGES_NEXT_RELEASE
telefonicaid · Oct 11, 2017 · 70801e8 · 70801e8
2 parents cb112b0 + 28f38df
commit 70801e8
Show file tree

Hide file tree

Showing 50 changed files with 533 additions and 172 deletions.
diff --git a/CHANGES_NEXT_RELEASE b/CHANGES_NEXT_RELEASE
@@ -1,11 +1,4 @@
-- Add: self-notification loop protection, based on Fiware-Correlator and Ngsiv2-AttrsFormat headers and lastCorrelator field at DB (#2937)
-- Add: Fiware-Correlator and NgsiV2-AttrsFormat headers cannot be overwritten by the custom notification logic (#2937)
-- Hardening: Mongo driver migrated to legacy-1.1.2 (several bugfixes in the legacy-1.0.7 to legacy-1.1.2 delta)
-- Hardening: Several changes in argument passing in mongoBackend library to avoid passing entire objects on the stack, from "X x" to "const X& x"
-- Fix: several invalid memory accesses
-- Hardening: Several changes in argument passing in mongoBackend library to add 'const' in references to objects that are not altered by the function
-- Fix: bug in parseArg lib that may cause problem printing the error message for wrong CLI usage (#2926)
-- Fix: bug in variable substitution of custom notifications that limited the size of the payload of a custom notification to 1024 bytes (new limit: 8MB)
-- Fix: bug in custom notifications making counters and timestamps not being updated (affected subscription fields: lastSuccess, lastFailure, lastNotifiction, count)
-- Fix: "request payload too large" (>1MB) as Bad Input alarm (WARN log level)
+- Fix: missing lastSuccess/lastFailure associated to initial notification on subscription creation some times when csub cache is in use (#2974)
+- Fix: several invalid memory accesses (based on a workaround, not a definitive solution, see issue #2994)
+- Add: release_date and doc fields are added to the GET /version output to align with FIWARE scheme (#2970)
 - Fix: broken JSON due to unscaped quotes (") in NGSIv2 error description field (#2955)
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -54,8 +54,31 @@ add_definitions(-DLM_ON)
 add_definitions(-fPIC)
 
 # Baseline compiler flags, any change here will affect all build types
-#set (CMAKE_CXX_FLAGS "-Wall -Wno-unknown-pragmas -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -Werror")
-set (CMAKE_CXX_FLAGS "-Wall -Wno-unknown-pragmas -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -Werror -fno-var-tracking-assignments")
+
+#
+# FIXME P4: https://github.com/telefonicaid/fiware-orion/issues/2979
+#   'template class std::auto_ptr' is deprecated in gcc 6.3.0 of Ubuntu 17.04, so -Werror cannot be used, at least not for now.
+#
+#   The *real* fix would be to rewrite the part of the code producing these warnings, but for now,
+#   disabling the -Werror flag makes the problem go away (warnings will still be found and give errors in other distros).
+#
+#   One example of this warning (there are many warnings, but as in header files, just a few places):
+#
+#   mongoBackend/safeMongo.h:
+#   ----------------------------------------------------------------
+#     extern bool nextSafeOrError
+#     (
+#       const std::auto_ptr<mongo::DBClientCursor>&  cursor,
+#       ...
+#   ----------------------------------------------------------------
+#
+# A grep of 'std::auto_ptr<' in the header files show 7 hits, and 13 hits in cpp files
+#
+IF (${DISTRO} STREQUAL "Ubuntu_17.04")
+  set (CMAKE_CXX_FLAGS "-Wall -Wno-unknown-pragmas -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -fno-var-tracking-assignments")
+ELSE()
+  set (CMAKE_CXX_FLAGS "-Wall -Wno-unknown-pragmas -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -Werror -fno-var-tracking-assignments")
+ENDIF ()
 
 #
 # Platform checks

diff --git a/README.md b/README.md
@@ -16,7 +16,7 @@
 * [API Walkthrough](#api-walkthrough)
 * [API Reference Documentation](#api-reference-documentation)
 * [Testing](#testing)
-    * [Ent-to-end tests](#ent-to-end-tests)
+    * [End-to-end tests](#end-to-end-tests)
     * [Unit Tests](#unit-tests)
 * [Advanced topics](#advanced-topics)
 * [License](#license)
@@ -42,9 +42,9 @@ For documentation previous to Orion 0.23.0 please check the manuals at FIWARE pu
 
 ## GEi overall description
 
-Orion is a C++ implementation of the NGSI9/10 REST API binding developed as a part of the FIWARE platform.
+Orion is a C++ implementation of the NGSIv2 REST API binding developed as a part of the FIWARE platform.
 
-Orion Context Broker allows you to manage all the whole lifecycle of context information including updates, queries, registrations and subscriptions. It is an NGSI9/10 server implementation to manage context information and its availability. Using the Orion Context Broker, you are able to register context elements and manage them through updates and queries. In addition, you can subscribe to context information so when some condition occurs (e.g. the context elements have changed) you receive a notification. These usage scenarios and the Orion Context Broker features are described in this document.
+Orion Context Broker allows you to manage the entire lifecycle of context information including updates, queries, registrations and subscriptions. It is an NGSIv2 server implementation to manage context information and its availability. Using the Orion Context Broker, you are able to create context elements and manage them through updates and queries. In addition, you can subscribe to context information so when some condition occurs (e.g. the context elements have changed) you receive a notification. These usage scenarios and the Orion Context Broker features are described in this documentation.
 
 If this is your first contact with the Orion Context Broker, it is highly recommended to have a look to the brief [Quick Start guide](doc/manuals/quick_start_guide.md).
 
@@ -129,7 +129,7 @@ Please have a look at the [Quick Start guide](doc/manuals/quick_start_guide.md)
 
 ## Testing
 
-### Ent-to-end tests
+### End-to-end tests
 
 The functional_test makefile target is used for running end-to-end tests:
 

diff --git a/doc/manuals/admin/database_admin.md b/doc/manuals/admin/database_admin.md
@@ -1,4 +1,4 @@
-#<a name="top"></a>Database administration 
+# <a name="top"></a>Database administration
 
 * [Introduction](#introduction)
 * [Backup](#backup)

diff --git a/doc/manuals/admin/diagnosis.md b/doc/manuals/admin/diagnosis.md
@@ -1,4 +1,4 @@
-#<a name="top"></a>Problem diagnosis procedures
+# <a name="top"></a>Problem diagnosis procedures
 
 * [Resource Availability](#resource-availability)
 * [Remote Service Access](#remote-service-access)

diff --git a/doc/manuals/admin/install.md b/doc/manuals/admin/install.md
@@ -5,7 +5,6 @@
 * [Installation](#installation)
     * [Using yum (recommended)](#using-yum-recommended)
     * [Using RPM file](#using-rpm-file)
-    * [Optional packages](#optional-packages)
 * [Upgrading from a previous version](#upgrading-from-a-previous-version)
     * [Upgrading MongoDB version](#upgrading-mongodb-version)
     * [Migrating the data stored in DB](#migrating-the-data-stored-in-db)
@@ -26,8 +25,7 @@ building from sources, check [this document](build_source.md).
     * In the case of using MongoDB 3.0/3.2/3.4 with its new authentication mechanism (SCRAM_SHA1) you may need to compile from sources using special switches for the MongoDB driver.
       See [this issue](https://github.com/telefonicaid/fiware-orion/issues/1061) for details.
 * RPM dependencies (some of these packages could not be in the official CentOS/RedHat repository but in EPEL, in which case you have to configure EPEL repositories, see <http://fedoraproject.org/wiki/EPEL>):
-    * The contextBroker package (mandatory) depends on the following packages: boost-filesystem, boost-thread, gnutls, libgcrypt, logrotate and libcurl.
-    * The contextBroker-test package (optional) depends on the following packages: python, python-flask, python-jinja2, curl, libxml2, libxslt, nc, mongo-10gen and contextBroker. The mongo-10gen dependency needs to configure MongoDB repository, check [this piece of documentation about that](http://docs.mongodb.org/manual/tutorial/install-mongodb-on-red-hat-centos-or-fedora-linux/).
+    * The contextBroker package (mandatory) depends on the following packages: boost-filesystem, boost-thread, gnutls, libgcrypt, logrotate and libcurl.    
 
 ## Installation
 
@@ -52,20 +50,6 @@ Next, install the package using the rpm command (as root):
 rpm -i contextBroker-X.Y.Z-1.x86_64.rpm
 ```
 
-### Optional packages
-
-Apart from the mandatory RPM described above, you can install the contextBroker-tests package, which contain utility tools:
-
-```
-yum install contextBroker-tests
-```
-
-or
-
-```
-rpm -i contextBroker-test-X.Y.Z-1.x86_64.rpm
-```
-
 ## Upgrading from a previous version
 
 Upgrade procedure depends on whether the *upgrade path* (i.e. from the installed Orion version to the target one to upgrade) crosses a version number that requires:

diff --git a/doc/manuals/admin/logs.md b/doc/manuals/admin/logs.md
@@ -1,4 +1,4 @@
-#<a name="top"></a>Logs
+# <a name="top"></a>Logs
 
 * [Log file](#log-file)
 * [Log format](#log-format)

diff --git a/doc/manuals/admin/perf_tuning.md b/doc/manuals/admin/perf_tuning.md
@@ -1,4 +1,4 @@
-#<a name="top"></a>Performance tuning
+# <a name="top"></a>Performance tuning
 
 * [MongoDB configuration](#mongodb-configuration)
 * [Database indexes](#database-indexes)

diff --git a/doc/manuals/admin/sanity_check.md b/doc/manuals/admin/sanity_check.md
@@ -1,4 +1,4 @@
-#<a name="top"></a>Sanity check procedures
+# <a name="top"></a>Sanity check procedures
 
 * [End to End Testing](#end-to-end-testing)
 * [List of Running Processes](#list-of-running-processes)
@@ -24,12 +24,14 @@ curl --header 'Accept: application/json' localhost:1026/version
 ```
 {
   "orion" : {
-    "version" : "0.23.0-next",
+    "version" : "1.8.0-next",
     "uptime" : "0 d, 0 h, 2 m, 30 s",
     "git_hash" : "c49692a996fb8d23cb2e78992094e26b1ca45dac",
-    "compile_time" : "Tue Sep 8 16:56:16 CEST 2015",
+    "compile_time" : "Wed Sep 27 16:56:16 CEST 2017",
     "compiled_by" : "fermin",
-    "compiled_in" : "debvm"
+    "compiled_in" : "debvm",
+    "release_date" : "Wed Sep 27 16:56:16 CEST 2017",
+    "doc" : "https://fiware-orion.readthedocs.org/en/master/"
   }
 }
 ```

diff --git a/doc/manuals/contribution_guidelines.md b/doc/manuals/contribution_guidelines.md
@@ -19,6 +19,33 @@ planning to contribute to the code you should read this document and get familia
 Note that contribution workflows themselves (e.g. pull requests, etc.) are described in another document 
 ([FIWARE Development Guidelines](https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Developer_Guidelines)).
 
+## Pull Request protocol
+
+As explained in ([FIWARE Development Guidelines](https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Developer_Guidelines)) 
+contributions are done using a pull request (PR). The detailed "protocol" used in such PR is described below:
+
+* Direct commits to master branch (even single-line modifications) are not allowed. Every modification has to come as a PR
+* In case the PR is implementing/fixing a numbered issue, the issue number has to be referenced in the body of the PR at creation time
+* Anybody is welcome to provide comments to the PR (either direct comments or using the review feature offered by Github)
+* Use *code line comments* instead of *general comments*, for traceability reasons (see comments lifecycle below)
+* Comments lifecycle
+  * Comment is created, initiating a *comment thread*
+  * New comments can be added as responses to the original one, starting a discussion
+  * After discussion, the comment thread ends in one of the following ways:
+    * `Fixed in <commit hash>` in case the discussion involves a fix in the PR branch (which commit hash is 
+       included as reference)
+    * `NTC`, if finally nothing needs to be done (NTC = Nothing To Change)
+ * PR can be merged when the following conditions are met:
+    * All comment threads are closed
+    * All the participants in the discussion have provided a `LGTM` general comment (LGTM = Looks good to me)
+ * Self-merging is not allowed (except in rare and justified circumstances)
+
+Some additional remarks to take into account when contributing with new PRs:
+
+* PR must include not only code contributions, but their corresponding pieces of documentation (new or modifications to existing one) and tests
+* PR modifications must pass full regression based on existing test (unit, functional, memory, e2e) in addition to whichever new test added due to the new functionality
+* PR should be of an appropriated size that makes review achievable. Too large PRs could be closed with a "please, redo the work in smaller pieces" without any further discussing
+
 ## Filesystem layout guidelines
 
 ### Directory layout

diff --git a/doc/manuals/index.md b/doc/manuals/index.md
@@ -1,12 +1,8 @@
 # Welcome to Orion Context Broker.
 
-Orion is a C++ implementation of the NGSI9/10 REST API binding developed as a part of the FIWARE platform.
+Orion is a C++ implementation of the NGSIv2 REST API binding developed as a part of the FIWARE platform.
 
-Orion Context Broker allows you to manage all the whole lifecycle of context information including updates, queries, registrations and subscriptions.
-Using the Orion Context Broker, you are able to register context elements and manage them through updates and queries.
-In addition, you can subscribe to context information so when some condition occurs
-(e.g. a context element has changed) you receive a notification.
-These usage scenarios and the Orion Context Broker features are described in this document.
+Orion Context Broker allows you to manage the entire lifecycle of context information including updates, queries, registrations and subscriptions. It is an NGSIv2 server implementation to manage context information and its availability. Using the Orion Context Broker, you are able to create context elements and manage them through updates and queries. In addition, you can subscribe to context information so when some condition occurs (e.g. the context elements have changed) you receive a notification. These usage scenarios and the Orion Context Broker features are described in this documentation.
 
 Github's [README.md](https://github.com/telefonicaid/fiware-orion/blob/master/README.md) provides a good documentation summary.
 The [User Manual](user/index.md) and the [Admin Guide](admin/index.md) cover more advanced topics. 
diff --git a/doc/manuals/user/filtering.md b/doc/manuals/user/filtering.md
@@ -1,4 +1,4 @@
-#<a name="top"></a>Filtering results
+# <a name="top"></a>Filtering results
 
 * [Introduction](#introduction)
 * [NGSIv2 filtering](#ngsiv2-filtering)

diff --git a/doc/manuals/user/ngsiv2_implementation_notes.md b/doc/manuals/user/ngsiv2_implementation_notes.md
@@ -1,4 +1,4 @@
-#<a name="top"></a>NGSIv2 Implementation Notes
+# <a name="top"></a>NGSIv2 Implementation Notes
 
 * [Forbidden characters](#forbidden-characters)
 * [Custom payload decoding on notifications](#custom-payload-decoding-on-notifications)

diff --git a/doc/manuals/user/v1_v2_coexistence.md b/doc/manuals/user/v1_v2_coexistence.md
@@ -1,4 +1,4 @@
-#<a name="top"></a>Considerations on NGSIv1 and NGSIv2 coexistence
+# <a name="top"></a>Considerations on NGSIv1 and NGSIv2 coexistence
 
 NGSIv1 is the API offered by Orion Context Broker from its very first version. 
 [NGSIv2](http://telefonicaid.github.io/fiware-orion/api/v2/stable) development started 

diff --git a/doc/manuals/user/walkthrough_apiv1.md b/doc/manuals/user/walkthrough_apiv1.md
@@ -1,4 +1,4 @@
-#<a name="top"></a>FIWARE NGSI APIv1 Walkthrough
+# <a name="top"></a>FIWARE NGSI APIv1 Walkthrough
 
 * [Introduction](#introduction)
 * [Before starting...](#before-starting)
@@ -157,8 +157,6 @@ command:
 # ./accumulator-server.py --port 1028 --url /accumulate --host ::1 -v
 ```
 
-The accumulator-server.py is also part of the contextBroker-test package (see [optional packages section in how to install](../admin/install.md#optional-packages)). The script is located at `/usr/share/contextBroker/tests/accumulator-server.py` after installation. However, if you only need the accumulator-server.py it uses to be simpler just downloading it from GitHub, as suggested above.
-
 [Top](#top)
 
 ### Issuing commands to the broker

diff --git a/doc/manuals/user/walkthrough_apiv2.md b/doc/manuals/user/walkthrough_apiv2.md
@@ -1,4 +1,4 @@
-#<a name="top"></a>FIWARE NGSI APIv2 Walkthrough
+# <a name="top"></a>FIWARE NGSI APIv2 Walkthrough
 
 * [Introduction](#introduction)
 * [Before starting...](#before-starting)
@@ -131,12 +131,6 @@ command:
 # ./accumulator-server.py --port 1028 --url /accumulate --host ::1 --pretty-print -v
 ```
 
-The accumulator-server.py is also part of the contextBroker-test package (see
-[optional packages section in how to install](../admin/install.md#optional-packages)).
-The script is located at `/usr/share/contextBroker/tests/accumulator-server.py` after
-installation. However, if you only need the accumulator-server.py it's easier to
-just download it from GitHub, as suggested above.
-
 [Top](#top)
 
 ### Issuing commands to the broker

diff --git a/rpm/SPECS/contextBroker.spec b/rpm/SPECS/contextBroker.spec
@@ -136,6 +136,15 @@ grep "tests" MANIFEST > MANIFEST.broker-tests
 
 %changelog
 
+
+# The contextBroker-test package is still taken into account, although it is very old and probably obsolete. If we
+# recover it in the future, dependencies and so on need to be reviewed. The following fragment (removed from
+# install documentation) could be useful:
+#
+#    The contextBroker-test package (optional) depends on the following packages: python, python-flask, 
+#    python-jinja2, curl, libxml2, libxslt, nc, mongo-10gen and contextBroker. The mongo-10gen dependency needs 
+#    to configure MongoDB repository, check [this piece of documentation about that](http://docs.mongodb.org/manual/tutorial/install-mongodb-on-red-hat-centos-or-fedora-linux/).
+#
 %package tests
 Requires: %{name}, python, python-flask, python-jinja2, nc, curl, libxml2, mongo-10gen 
 Summary: Test suite for %{name}
@@ -164,6 +173,18 @@ if [ "$1" == "0" ]; then
 fi
 
 %changelog
+* Mon Sep 11 2017 Fermin Galan <[email protected]> 1.8.0-1
+- Add: self-notification loop protection, based on Fiware-Correlator and Ngsiv2-AttrsFormat headers and lastCorrelator field at DB (#2937)
+- Add: Fiware-Correlator and NgsiV2-AttrsFormat headers cannot be overwritten by the custom notification logic (#2937)
+- Fix: several invalid memory accesses
+- Fix: bug in parseArg lib that may cause problem printing the error message for wrong CLI usage (#2926)
+- Fix: bug in variable substitution of custom notifications that limited the size of the payload of a custom notification to 1024 bytes (new limit: 8MB)
+- Fix: bug in custom notifications making counters and timestamps not being updated (affected subscription fields: lastSuccess, lastFailure, lastNotifiction, count)
+- Fix: "request payload too large" (>1MB) as Bad Input alarm (WARN log level)
+- Hardening: Several changes in argument passing in mongoBackend library to add 'const' in references to objects that are not altered by the function
+- Hardening: Several changes in argument passing in mongoBackend library to avoid passing entire objects on the stack, from "X x" to "const X& x"
+- Hardening: Mongo driver migrated to legacy-1.1.2 (several bugfixes in the legacy-1.0.7 to legacy-1.1.2 delta)
+
 * Wed Feb 08 2017 Fermin Galan <[email protected]> 1.7.0-1
 - Add: HTTPS native notifications (#706), fixing at the same time issue #2844
 - Add: new option to accept self-signed certifications used by HTTPS notification endpoints: -insecureNotif (#706)

diff --git a/scripts/build/compileInfo.sh b/scripts/build/compileInfo.sh
@@ -67,6 +67,7 @@ echo '#define GIT_HASH         "'${hash}'"'
 echo '#define COMPILE_TIME     "'${date}'"'                                         >> $TMP_FILE
 echo '#define COMPILED_BY      "'${user}'"'                                         >> $TMP_FILE
 echo '#define COMPILED_IN      "'${host}'"'                                         >> $TMP_FILE
+echo '#define RELEASE_DATE     "'${date}'"'                                         >> $TMP_FILE
 echo                                                                                >> $TMP_FILE
 echo '#endif  // SRC_LIB_COMMON_COMPILEINFO_H_'                                     >> $TMP_FILE
 

diff --git a/scripts/build/release.sh b/scripts/build/release.sh
@@ -174,6 +174,17 @@ else
 fi
 mv /tmp/README.md README.md
 
+# Adjust Readthedocs documentation link for GET /version response. Note that the procedure is not symmetric 
+# (like in version.h), as dev release sets 'master' and not 'X.Y.Z-next"
+if [ "$BROKER_RELEASE" != "dev" ]
+then
+  sed "s/https:\/\/fiware-orion.readthedocs.org\/en\/master\//https:\/\/fiware-orion.readthedocs.org\/en\/$NEW_VERSION\//" src/lib/common/defaultValues.h > /tmp/defaultValues.h
+else
+  sed "s/https:\/\/fiware-orion.readthedocs.org\/en\/$currentVersion\//https:\/\/fiware-orion.readthedocs.org\/en\/master\//" src/lib/common/defaultValues.h > /tmp/defaultValues.h
+fi
+mv /tmp/defaultValues.h src/lib/common/defaultValues.h
+
+
 # Adjust Dockerfile GIT_REV_ORION. Note that the procedure is not symmetric (like in version.h), as
 # dev release sets 'master' and not 'X.Y.Z-next"
 if [ "$BROKER_RELEASE" != "dev" ]

diff --git a/src/app/contextBroker/version.h b/src/app/contextBroker/version.h
@@ -28,6 +28,6 @@
 
 
 
-#define ORION_VERSION "1.7.0-next"
+#define ORION_VERSION "1.8.0-next"
 
 #endif  // SRC_APP_CONTEXTBROKER_VERSION_H_
Original file line number	Diff line number	Diff line change
Expand Up		@@ -28,6 +28,6 @@



		#define ORION_VERSION "1.7.0-next"
		#define ORION_VERSION "1.8.0-next"

		#endif // SRC_APP_CONTEXTBROKER_VERSION_H_