Bring in changes from dev branch to main (FreeRTOS#214)

* Update ci.yml (FreeRTOS#197)

* Update ci.yml

* Add main branch in the CI YAML

* Add cancel callback API (FreeRTOS#196)

* Add cancel callback command

* Update the function name

* Fix formatting

* Update memory estimates

* Add State to MQTT_ProcessLoop so that it can be called in a non-blocking manner (FreeRTOS#198)

* Add stateful process-loop function

* Add extra checks; fix bugs and add description of functions

* Add index based stateful processloop

* Clean up

* Renamed functions to make them more coherent with their function

* Remove unused function declarations

* Fixed failing CI checks from previous commits except unit-test

* Fixed spell check and updated size-table

* Fix CBMC proofs

* Empty-Commit to trigger CBMC proofs

* Fix loop unwinding values in the Makefile

* Add upper bound on the buffer size of MQTT

* Increase minimum limit on buffer size to >0

* Add upper bound on the size of the buffer as well

* CBMC: Add memmove stub to accelerate coverage

The commit adds a stub for memmove accelerate CBMC coverage
calculation.  Without this stub, coverage for `MQTT_ProcessLoop` and
`MQTT_ReceiveLoop` fails to converge (gets stuck generating the SAT
formula for the memmove in `receiveSingleIteration`).  This stub
checks that src and dst are nonnull pointers and havocs dst.

* Fix formatting

Co-authored-by: Aniruddha Kanhere <[email protected]>
Co-authored-by: Mark R. Tuttle <[email protected]>

* Remove the use of common buffer (FreeRTOS#199)

* Make publish use internal buffers

* Fix comment about packet ID

* Try a different approach for ping and disconnect

* Use writev and flush in conjuction with send

* Update the publish method to use vectors

* Add vectored IO to all functions

* Fix formatting

* Reduce complexity score

* Fix spell check and complexity score

* Fix breaking build

* Add doxygen comments

* Fix doxygen part 2

* Doxygen fix part 3

* Fix doxygen part 4

* Fix some checks

* Fix memory tables

* Fix some small errors

* Fix compiler warnings and breaking CI checks from previous commit

* Fix spell check and doxygen

* Fix a couple of CBMC proofs

* Fix ping and publish proofs

* Update the function name

* Fix more CBMC proofs

* Fix MQTT Connect proof

* Add unwinding loops

* Fix last CBMC proof

* Fix formatting

* Update the Subscribe and Unsubscribe functions

* Fix formatting and doxygen checks

* Fix broken CBMC proofs

* Fix memory statistic table

* Revert changes from serializer source

* update comments to clarify write requirements

* Add a note for write function pointer

* Fix spell check

* Update changelog (FreeRTOS#202)

* Update MQTT logging so that log levels of the library do not leak (FreeRTOS#205)

* Replace publish state arrays with pointers
Added an MQTT_InIt function for QoS > 0 publishes
Fixed functions which were dealing with state arrays

* Fix CI checks and clean up

* Fix CBMC proofs

* Fix sub and unsub CBMC proofs

* Fix remaining proofs

* Fix remaining CI checks

* Fix spell check

* Minor typo fix (FreeRTOS#209)

* Update core_mqtt.h (FreeRTOS#208)

* Update core_mqtt.h

* Update core_mqtt.h

Co-authored-by: Aniruddha Kanhere <[email protected]>

* Update documentation and Doxygen comments in the source (FreeRTOS#206)

* Update comments of the MQTT_InIt function

* Updating documentation of more of functions

* Fix formatting and spell check

* Update core_mqtt.h

* Add hooks to the source code (FreeRTOS#200)

* Add mutex hooks

* Clean up of code

* Add doxygen comments and fix spell check

* Fix LogError call

* Fix formatting and memory table

* Fix dereference failure

* Update the hook names

* Fix broken builds

* Update the macros and variables

* Reword the briefs of hooks and uncrustify

* Fir formatting

* Protect get packet ID

* Fix formatting

* Fix Unit tests (FreeRTOS#212)

* Fix Unit tests

* Update unit tests after new changes

* Fix more UT

* add dummy calls to the transport

* fix build error

* Remove usused variables

* Remove unsused variables

* Remove usused variables

* Unsued parameter

* Fix ut failure

* Fix uninitialized unit test variables

* Fix ut expectation

* Fix unit-tests

* Fix unit test uninitialized variable

* increase unit test coverage

* increase unit test coverage

* Fix unit test build

* State coverage 100%

* Serializer 100% coverage

* 100% UT coverage

* Fix formatting

* Fix size table

* Address PR comments

Co-authored-by: Aniruddha Kanhere <[email protected]>

* Remove unnecessary ternaries (FreeRTOS#211)

Co-authored-by: Aniruddha Kanhere <[email protected]>

* Fix misra deviations  (FreeRTOS#213)

* MISRA compliance update 1

* Fix MISRA errors

* Zero MISRA violations; 13 suppressed

* Fix CI checks

* Update MISRA.md

* Remove deviations

* Fix MISRA.md file

* Fix bug - wasn't incrementing vector length properly.

* Remove unwanted files

* Update comment to clarify the control flow of UT

* Remove 'dev' branch from the CI checks

* Update horrid threshold = 10

Co-authored-by: Aniruddha Kanhere <[email protected]>
Co-authored-by: Mark R. Tuttle <[email protected]>
Co-authored-by: jasonpcarroll <[email protected]>
Co-authored-by: alfred gedeon <[email protected]>
Co-authored-by: Archit Gupta <[email protected]>

.github/workflows/ci.yml

@@ -45,6 +45,7 @@ jobs:
         uses: FreeRTOS/CI-CD-Github-Actions/complexity@main
         with:
           path: ./
+          horrid_threshold: 10
   doxygen:
     runs-on: ubuntu-20.04
     steps:

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog for coreMQTT Client Library
 
+## v1.3.0 (September 2022)
+- [#199](https://github.com/FreeRTOS/coreMQTT/pull/199) Make use of user provided buffer only for packet reception. Stack space will be used for sending packets.
+- [#198](https://github.com/FreeRTOS/coreMQTT/pull/198) Remove timeout from MQTT\_ProcessLoop and MQTT\_ReceiveLoop and make the implementations non-blocking.
+- [#196](https://github.com/FreeRTOS/coreMQTT/pull/196) Add a cancel callback API to cancel sent publish packets.
+- [#191](https://github.com/FreeRTOS/coreMQTT/pull/191) Generate PINGREQ packets on idle input or output.
+
 ## v1.2.0 (November 2021)
 
 ### Changes

MISRA.md

@@ -1,25 +1,39 @@
 # MISRA Compliance
 
-The MQTT library files conform to the [MISRA C:2012](https://www.misra.org.uk)
-guidelines, with some noted exceptions. Compliance is checked with Coverity static analysis.
-Deviations from the MISRA standard are listed below:
-
-### Ignored by [Coverity Configuration](https://github.com/aws/aws-iot-device-sdk-embedded-C/blob/main/tools/coverity/misra.config)
-| Deviation | Category | Justification |
-| :-: | :-: | :-: |
-| Directive 4.5 | Advisory | Allow names that MISRA considers ambiguous (such as LogInfo and LogError) |
-| Directive 4.8 | Advisory | Allow inclusion of unused types. Header files for a specific port, which are needed by all files, may define types that are not used by a specific file. |
-| Directive 4.9 | Advisory | Allow inclusion of function like macros. The `assert` macro is used throughout the library for parameter validation, and logging is done using function like macros. |
-| Rule 2.3 | Advisory | Allow unused types. The `MQTTSubAckStatus_t` enum is unused in our source files, as it is intended for a user to use when parsing a subscription acknowledgment's response codes. |
-| Rule 2.4 | Advisory | Allow unused tags. Some compilers warn if types are not tagged. |
-| Rule 2.5 | Advisory | Allow unused macros. Library headers may define macros intended for the application's use, but are not used by a specific file. |
-| Rule 3.1 | Required | Allow nested comments. C++ style `//` comments are used in example code within Doxygen documentation blocks. |
-| Rule 11.5 | Advisory | Allow casts from `void *`. Fields such as publish payloads are passed as `void *` and must be cast to the correct data type before use. |
-
-### Flagged by Coverity
-| Deviation | Category | Justification |
-| :-: | :-: | :-: |
-| Rule 8.7 | Advisory | API functions are not used by the library outside of the files they are defined; however, they must be externally visible in order to be used by an application. |
+The coreMQTT library files conform to the [MISRA C:2012](https://www.misra.org.uk/MISRAHome/MISRAC2012/tabid/196/Default.aspx)
+guidelines, with the deviations listed below. Compliance is checked with Coverity static analysis.
+Since the coreMQTT library is designed for small-embedded devices, it needs to have a very small memory footprint and has to
+be efficient. To achieve that and to increase the performace of the library, it deviates from some MISRA rules.
+The specific deviations, suppressed inline, are listed below.
+
+Additionally, [MISRA configuration file](https://github.com/FreeRTOS/FreeRTOS-Plus-TCP/blob/main/test/Coverity/coverity_misra.config) contains the project wide deviations.
 
 ### Suppressed with Coverity Comments
-*None.*
+To find the deviation references in the source files run grep on the source code
+with ( Assuming rule 18.2 violation; with justification in point 1 ):
+```
+grep 'MISRA Ref 18.2.1' . -rI
+```
+#### Rule 10.8
+
+_Ref 10.8.1_
+
+- MISRA C-2012 Rule 10.8 states that value of composite expressions should not be cast
+  to variables of different signedness. In this library, array of bytes are used to
+  process data. Functions which fill the arrays with data update an index to an
+  offset. To know the amount of data added to the array, the beginning address of the
+  array has to be subtracted from the index. When the two pointers are subracted, it
+  results in a signed value. It is verified however that the value will always be positive.
+  And thus, can be casted and added to a size_t variable (which is unsigned).
+
+#### Rule 18.2
+
+_Ref 18.2.1_
+
+- MISRA C-2012 Rule 18.2 states that two pointers may only be subtracted if they point
+  to elements of the same array. In this library, array of bytes are used to process
+  data. Functions which fill the arrays with data update an index to an offset.
+  To know the amount of data added to the array, the beginning address of the array has
+  to be subtracted from the index. It is manually verified that the index will always be
+  within bounds of the array. However, Coverity is flagging this as a deviation. Thus, we
+  are suppressing it.

docs/doxygen/include/size_table.md

@@ -9,22 +9,22 @@
     </tr>
     <tr>
         <td>core_mqtt.c</td>
-        <td><center>3.1K</center></td>
-        <td><center>2.7K</center></td>
+        <td><center>3.9K</center></td>
+        <td><center>3.3K</center></td>
     </tr>
     <tr>
         <td>core_mqtt_state.c</td>
-        <td><center>1.4K</center></td>
-        <td><center>1.1K</center></td>
+        <td><center>1.7K</center></td>
+        <td><center>1.3K</center></td>
     </tr>
     <tr>
         <td>core_mqtt_serializer.c</td>
-        <td><center>2.5K</center></td>
-        <td><center>2.0K</center></td>
+        <td><center>2.8K</center></td>
+        <td><center>2.2K</center></td>
     </tr>
     <tr>
         <td><b>Total estimates</b></td>
-        <td><b><center>7.0K</center></b></td>
-        <td><b><center>5.8K</center></b></td>
+        <td><b><center>8.4K</center></b></td>
+        <td><b><center>6.8K</center></b></td>
     </tr>
 </table>

docs/doxygen/pages.dox

@@ -85,8 +85,7 @@ these low-level functions can be used directly:
 The MQTT 3.1.1 protocol allows for a client and server to maintain persistent sessions, which
 can be resumed after a reconnect. The elements of a session stored by this client library consist
 of the states of incomplete publishes with Quality of Service levels of 1 (at least once), or 2 (exactly once).
-These states are stored in the arrays @ref MQTTContext_t.outgoingPublishRecords and @ref MQTTContext_t.incomingPublishRecords;
-the length of each of these arrays can be configured with @ref MQTT_STATE_ARRAY_MAX_COUNT.
+These states are stored in the pointers pointed to by @ref MQTTContext_t.outgoingPublishRecords and @ref MQTTContext_t.incomingPublishRecords;
 This library does not store any subscription information, nor any information for QoS 0 publishes.
 
 When resuming a persistent session, the client library will resend PUBRELs for all PUBRECs that had been received
@@ -149,9 +148,6 @@ Some configuration settings are C pre-processor constants, and some are function
 @section MQTT_DO_NOT_USE_CUSTOM_CONFIG
 @copydoc MQTT_DO_NOT_USE_CUSTOM_CONFIG
 
-@section MQTT_STATE_ARRAY_MAX_COUNT
-@copydoc MQTT_STATE_ARRAY_MAX_COUNT
-
 @section MQTT_PINGRESP_TIMEOUT_MS
 @copydoc MQTT_PINGRESP_TIMEOUT_MS
 

docs/doxygen/porting.dox

@@ -15,7 +15,6 @@ A port to a new platform must provide the following components:
 @see [Configurations](@ref core_mqtt_config)
 
 The following macros can be configured for the managed MQTT library:
- - @ref MQTT_STATE_ARRAY_MAX_COUNT <br>
  - @ref MQTT_PINGRESP_TIMEOUT_MS <br>
  - @ref MQTT_MAX_CONNACK_RECEIVE_RETRY_COUNT
 

lexicon.txt

@@ -43,6 +43,7 @@ connectpacketsize
 const
 copydoc
 coremqtt
+coverity
 csdk
 css
 currentstate
@@ -73,8 +74,10 @@ enums
 eventcallback
 expectprocessloopcalls
 filterindex
+findinrecord
 fixedbuffer
 fn
+freertos
 gcc
 getconnectpacketsize
 getdisconnectpacketsize
@@ -105,14 +108,18 @@ img
 inc
 incomingpacket
 incomingpublish
+incomingpublishcount
 incomingpublishrecords
 ingroup
 init
 initializeconnectinfo
 initializesubscribeinfo
 initializewillinfo
 int
+io
 iot
+ioveccount
+ip
 isn
 iso
 keepaliveintervalsec
@@ -130,6 +137,7 @@ mainpage
 malloc
 managekeepalive
 matchtopic
+maxrecordcount
 md
 mdash
 memcpy
@@ -153,6 +161,7 @@ mqttfixedbuffer
 mqttgetcurrenttimefunc
 mqttillegalstate
 mqttkeepalivetimeout
+mqttneedmorebytes
 mqttnodataavailable
 mqttnomemory
 mqttnotconnected
@@ -190,6 +199,7 @@ mqttsubacksuccessqos
 mqttsubscribeinfo
 mqttsuccess
 msb
+mutex
 mynetworkrecvimplementation
 mynetworksendimplementation
 mytcpsocketcontext
@@ -207,6 +217,8 @@ numcodes
 optype
 org
 os
+outgoingpublishes
+outgoingpublishcount
 outgoingpublishrecords
 packetid
 packetidentifier
@@ -233,23 +245,28 @@ pfilterindex
 pfixedbuffer
 pheadersize
 pincomingpacket
+pincomingpublishrecords
+pindex
 pingreq
 pingreqs
 pingreqsendtimems
 pingresp
 pingresps
+piovec
 pismatch
 plaintext
 pmatch
 pmessage
 pmqttcontext
+pmqttheader
 pnameindex
 pnetworkbuffer
 pnetworkcontext
 pnetworkinterface
 pnewstate
 png
 posix
+poutgoingpublishrecords
 ppacketid
 ppacketidentifier
 ppacketinfo
@@ -277,10 +294,12 @@ psuback
 psubackpacket
 psubscribeinfo
 psubscribes
+psubscription
 psubscriptionlist
 ptopic
 ptopicfilter
 ptopicname
+ptotalmessagelength
 ptr
 ptransport
 ptransportinterface
@@ -328,6 +347,11 @@ serializeack
 serializeconnect
 serializeconnectpacket
 serializedisconnect
+serailizedlength
+serializedpasswordlength
+serializedpayloadlength
+serializedtopiclength
+serializedusernamelength
 serializepayload
 serializepingreq
 serializepublish
@@ -374,6 +398,7 @@ tlssend
 toolchain
 topicfilterlength
 topicnamelength
+totalmessagelength
 tr
 transportcallback
 transportinterface
@@ -388,8 +413,10 @@ tx
 typename
 uint
 un
+unacked
 unsuback
 unsubscribelist
+updatedlength
 updatestateack
 updatestatepublish
 updatestatestatus
@@ -400,6 +427,7 @@ validatesubscribeunsubscribeparams
 validator
 waitingforpingresp
 willinfo
+writev
 xa
 xb
 xc