Fix blockwise response logic #376
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updates the blockwise tests to catch the cases where the payload is smaller than 'maxPacket', but the overall packet size after the headers are added exceeds 'maxPacket' which causes an error to be thrown. Also adds checks to ensure that the response code is 2.05 for the tests that require it, so 5.00 errors can't sneak through.
Fix for blockwise transfer issues raised in coapjs#373 where a coap message could avoid being split into blocks, yet still be larger than the allowable size of 1280 bytes. This applies the recommended maximum payload size from RFC 7252 Section 4.6 of 1024 bytes, and allows it to be reduced in the config parameters as required. The error that this fixes was introduced in d727d43 and this attempts to keep the intent behind that commit, whilst fixing the technical flaws that it introduced. The 'IP_MTU' constant has been renamed to 'MAX_PAYLOAD' to more accurately describe what the it represents. The 'maxPacketSize' parameter has also been renamed to 'maxPayloadSize' for the same reason. Note that the renaming of the parameter is a potentially breaking change.
This fixes the logic for checking the maximum CoAP message size. The check is actually performed in the coap-packet repository in a default parameter on the `generate()` function, however the default value is not appropriate for all (if any) cases. The maximum size that a CoAP message can be is the IP MTU, minus the IP header and minus the UDP header. The value is not constant across all IP network stacks, so the CoAP specification recommends a maximum of 1152 bytes for cases where it is not known. The only way to know for sure is MTU path discovery, which is way outside of the scope of the project. This commit creates a parameter that allows the max packet size to be adjusted as a server parameter for cases where (for example) the server is running on a 6LoWPAN/Thread network and needs a lower maximum message size. Note that the logic for enforcing the size is just to throw an error and crash the server. However, since the maximum payload size is enforced a situation like that should never occur.
JKRhb
approved these changes
Mar 26, 2024
There was a problem hiding this comment.
Hi @edrose, thank you for your (very well-documented) PR, and sorry for the long response time! The changes look good to me, if you are also fine with the PR, @Apollon77, then we could merge it :)
Apollon77
approved these changes
Mar 26, 2024
JimmyBjorklund
pushed a commit
to JimmyBjorklund/node-coap
that referenced
this pull request
May 28, 2024
* Update blockwise tests to catch edge-cases Updates the blockwise tests to catch the cases where the payload is smaller than 'maxPacket', but the overall packet size after the headers are added exceeds 'maxPacket' which causes an error to be thrown. Also adds checks to ensure that the response code is 2.05 for the tests that require it, so 5.00 errors can't sneak through. * Update automatic blockwise transfer logic Fix for blockwise transfer issues raised in coapjs#373 where a coap message could avoid being split into blocks, yet still be larger than the allowable size of 1280 bytes. This applies the recommended maximum payload size from RFC 7252 Section 4.6 of 1024 bytes, and allows it to be reduced in the config parameters as required. The error that this fixes was introduced in d727d43 and this attempts to keep the intent behind that commit, whilst fixing the technical flaws that it introduced. The 'IP_MTU' constant has been renamed to 'MAX_PAYLOAD' to more accurately describe what the it represents. The 'maxPacketSize' parameter has also been renamed to 'maxPayloadSize' for the same reason. Note that the renaming of the parameter is a potentially breaking change. * Perform extra checks on CoAP message size This fixes the logic for checking the maximum CoAP message size. The check is actually performed in the coap-packet repository in a default parameter on the `generate()` function, however the default value is not appropriate for all (if any) cases. The maximum size that a CoAP message can be is the IP MTU, minus the IP header and minus the UDP header. The value is not constant across all IP network stacks, so the CoAP specification recommends a maximum of 1152 bytes for cases where it is not known. The only way to know for sure is MTU path discovery, which is way outside of the scope of the project. This commit creates a parameter that allows the max packet size to be adjusted as a server parameter for cases where (for example) the server is running on a 6LoWPAN/Thread network and needs a lower maximum message size. Note that the logic for enforcing the size is just to throw an error and crash the server. However, since the maximum payload size is enforced a situation like that should never occur.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This is a fix for #375 where the server will attempt to generate a packet greater than the maximum allowable, which was set to 1280 bytes. This would occur if the size of the payload was less than 1280 bytes, but the size of the resulting CoAP message was greater than 1280 bytes after the headers were added.
The bug was introduced 9 years ago in d727d43 and has some incorrect logic for dealing with message sizes. It would be very easy to simply revert those changes and call it a fix, but I wanted to try and understand what the author was trying to implement and to fix the implementation, rather than simply removing functionality.
In addition I've updated the tests to be more robust and to catch errors like this in the future.
Motivation and Context
The intent of the above commit, I believe, was to allow the maximum packet size to be adjusted for cases where the server is running over a network stack that has an MTU different to that of ethernet. A 6LoWPAN/Thread network stack, for example, will have an MTU significantly less than that of ethernet so the server can be configured to perform blockwise transfers earlier to avoid message fragmentation at the IP layer. This is done with an
IP_MTUconstant and amaxPacketSizeparameter that control when blockwise transfers are applied.Unfortunately the phrase 'packet' and 'MTU' are used all over the place and don't really refer to anything meaningful in the context of the application layer. The IP MTU is the size of the whole IP packet, including all headers and payload. The
IP_MTUconstant defaults to 1280 and is used only to check the CoAP payload, missing the CoAP headers, UDP header, and IP header when performing the check.The CoAP specification does not ever refer to anything as a 'packet' - it refers to 'messages'. A 'packet' is the IP layer 3 data unit, and I think this ambiguity of naming in the library has caused confusion in the past when features are implemented.
Changes Introduced
Even though the maximum CoAP message size is based on the IP MTU, it's not practical to try and determine the MTU for every response and apply blockwise logic accordingly. Furthermore the header size of the IP layer is not fixed, and when using 6LoWPAN neither is the UDP header size. So instead the changes here follow the recommendations set out in RFC 7252 Section 4.6 which state:
Those recommendations are reflected in the defaults that are implemented, and can be adjusted from the server parameters.
Tests
Update tests in
test/blockwise.tsto check the response code of the server when checking responses and to move the test payload size to within the bounds of the edge case that this is fixing.Server Code
Change the parameters file to remove the
IP_MTUconstant. This was replaced with two constants:MAX_PAYLOAD_SIZEandMAX_MESSAGE_SIZE. Those changes are reflected in the parameters by removing themaxPacketSize, replacing it withmaxPayloadSizeandmaxMessageSizerespectively.The blockwise logic has been updated to be more efficient and understandable, using the provided
maxPayloadSizeparameter. The check for whether a response needs to be split into blocks is now correct, fixing the initial bug.The
maxMessageSizeparameter is used when callingcoap-packet:generate()and is passed as themaxLengthargument. This argument, when not explicitly passed, defaults to 1280 which (as discussed) does not represent the maximum allowable message size. Now an error will be returned from that function when the total CoAP message size is greater than the max allowable set in the server. This error should now only occur if the server is misconfigured.Potentially Breaking Changes
The changes to the parameters might be breaking changes, depending on how they are used in other projects. I'm happy to rename them if required in order to maintain some form of backwards compatibility, at the expense of the names being less clear.