I am running into a byte buffer that gets bigger than 0x7FFF bytes, while reading the Object List of a meter. From the definition of the function bb_isAttached(), the part return (arr->capacity & 0x8000) == 0x8000, I asume that you use the highest bit to determine if a buffer is attached or not. This is causing problems for me, since the buffer needs to grow.
I see two ways forward: use 32 bit values or adding a separate bool attach_flag to bytebuffer.
Since I am using an ESP32, is there an easy way to use 32 bit values for the DLMS.c library? Everywere the #else part of the switch #if !defined(GX_DLMS_MICROCONTROLLER) && (defined(_WIN32) || defined(_WIN64) || defined(__linux__)) is the 16 bit code, but I see no easy way to use the 32 bit part?
Wouldn't it be better anyways to not "misuse" the capacity field of the bytebuffer also for a flag?
Just for testing I changed the bytebuffer code to have an extra flag 'char attached' and give the highest bit back to the uint16_t capacity. That works, but the Object List response buffer is even bigger than 64k. So I need the 32 bit values for capacity (among other variables).
Is there an easy way with the #defines to use the 32 bit values in stead off the 16 bit values?
Would it be possible to read the object list in parts? Say read 10k of the object list frame, parse it, delete the used bytes and append a new part of lets say 10k, and repeat this process until the full object list is parsed?
I also think there is something wrong with 32 bit indices. I added || defined(ESP32) to all #if .... defined(__linux__) to get 32 bit indices. Still while reading the very large object list, if I log the size of the reply byte buffer in readDLMSPacket(), it resets while reading the response:
19:12:33.824 -> 7e a0 14 02 23 03 ba 8d 0b e6 e6 00 c0 02 81 00 00 00 34 5d 19 7e
19:12:33.858 -> Now ret is 0, reply->data->capacity is 65936, size is 65936, free heap is 298860
19:12:33.957 -> Now ret is 0, reply->data->capacity is 66087, size is 539, free heap is 298708
19:12:33.990 -> 7e a0 08 02 23 03 d1 28 15 7e
19:12:34.024 -> Now ret is 0, reply->data->capacity is 66087, size is 539, free heap is 298676
19:12:34.057 -> Now ret is 0, reply->data->capacity is 693, size is 693, free heap is 364068
I am looking myself where the byte buffer is reduced in size, changed various uint16_t to uint32_t but still can not find it.
Are you sure that the object list is correct? There should be about 1000 objects!
Adding an extra bit for this will increase stack size and we try to decrease it.
I was checking and it would be possible to add GX_DLMS_BYTE_BUFFER_MAX_SIZE_32 to use 32 bit count and size values.
If you read all data to the buffer you can use int cl_parseObjectCount and cl_parseNextObject to parse only objects you want to.
Can you send the total trace to me by email? I can check what is causing this.
You can use GXDLMSDirector to read that easily.
Thanks for your reply! Any idea when GX_DLMS_BYTE_BUFFER_MAX_SIZE_32 will be added? I am using the DLMS.c Arduino version.
I cannot use DLMS director. This meter sends an Identification with a '9' in the suggested baud rate (and needs a '7' in the ACK (ACK '2' '7' '2' CR LF)) and DLMS director does not support higher baud rates.
DLMS.c fails before parsing, while executing readDLMSPacket() it returns a 260, DLMS_ERROR_CODE_OUTOFMEMORY.
I think I could create a trace from code of the first 64k. Is that interesting to you?
Yes, send a trace to me. Baud-rates 38400, 57600, and 115200 are not defined in IEC 62056-21 standard, but support for them is coming in the next GXDLMDirector release.
Thanks for the quick adaptations! However, I think there is still something wrong. If I do NOT #define GX_DLMS_BYTE_BUFFER_SIZE_32 the library works as before (Out of memory on 32k). If I do #define GX_DLMS_BYTE_BUFFER_SIZE_32 there is something wrong with the bytebuffer
I added a printf in the client example:
int readDLMSPacket(
gxByteBuffer* data,
gxReplyData* reply)
{
int resend = 0, ret = DLMS_ERROR_CODE_OK;
if (data->size == 0)
{
return DLMS_ERROR_CODE_OK;
}
reply->complete = 0;
frameData.size = 0;
frameData.position = 0;
//Send data.
Serial.printf("size is %lu, position is %lu, capacity is %lu\n", data->size, data->position, data->capacity);
Without the #define this gives:
size is 10, position is 0, capacity is 58
With the #define this gives:
size is 3131899904, position is 3131922040, capacity is 655418
Adding #define ESP_PLATFORM does not correct this.
Can you get the latest version? There are a few changes that I hope will solve this. We'll build a special meter where we can test this, but it'll take some time.
I just tried the version I downloaded today. This is not working anymore, now also the 16 bit version stopped working. I think at readDLMSPacket no message is sent to the meter, but I am not sure.
It works without problems in our environments. We made a few changes. I hope it works now for you.
It is working if you don't define GX_DLMS_BYTE_BUFFER_SIZE_32?
Just for testing I changed
Just for testing I changed the bytebuffer code to have an extra flag 'char attached' and give the highest bit back to the uint16_t capacity. That works, but the Object List response buffer is even bigger than 64k. So I need the 32 bit values for capacity (among other variables).
Is there an easy way with the #defines to use the 32 bit values in stead off the 16 bit values?
Would it be possible to read
Would it be possible to read the object list in parts? Say read 10k of the object list frame, parse it, delete the used bytes and append a new part of lets say 10k, and repeat this process until the full object list is parsed?
I also think there is
I also think there is something wrong with 32 bit indices. I added || defined(ESP32) to all #if .... defined(__linux__) to get 32 bit indices. Still while reading the very large object list, if I log the size of the reply byte buffer in readDLMSPacket(), it resets while reading the response:
19:12:33.824 -> 7e a0 14 02 23 03 ba 8d 0b e6 e6 00 c0 02 81 00 00 00 34 5d 19 7e
19:12:33.858 -> Now ret is 0, reply->data->capacity is 65936, size is 65936, free heap is 298860
19:12:33.957 -> Now ret is 0, reply->data->capacity is 66087, size is 539, free heap is 298708
19:12:33.990 -> 7e a0 08 02 23 03 d1 28 15 7e
19:12:34.024 -> Now ret is 0, reply->data->capacity is 66087, size is 539, free heap is 298676
19:12:34.057 -> Now ret is 0, reply->data->capacity is 693, size is 693, free heap is 364068
I am looking myself where the byte buffer is reduced in size, changed various uint16_t to uint32_t but still can not find it.
Hi Jeroen,
Hi Jeroen,
Are you sure that the object list is correct? There should be about 1000 objects!
Adding an extra bit for this will increase stack size and we try to decrease it.
I was checking and it would be possible to add GX_DLMS_BYTE_BUFFER_MAX_SIZE_32 to use 32 bit count and size values.
If you read all data to the buffer you can use int cl_parseObjectCount and cl_parseNextObject to parse only objects you want to.
Can you send the total trace to me by email? I can check what is causing this.
You can use GXDLMSDirector to read that easily.
BR,
Mikko
Hi Mikko,
Hi Mikko,
Thanks for your reply! Any idea when GX_DLMS_BYTE_BUFFER_MAX_SIZE_32 will be added? I am using the DLMS.c Arduino version.
I cannot use DLMS director. This meter sends an Identification with a '9' in the suggested baud rate (and needs a '7' in the ACK (ACK '2' '7' '2' CR LF)) and DLMS director does not support higher baud rates.
DLMS.c fails before parsing, while executing readDLMSPacket() it returns a 260, DLMS_ERROR_CODE_OUTOFMEMORY.
I think I could create a trace from code of the first 64k. Is that interesting to you?
Best, Jeroen
Hi,
Hi,
Yes, send a trace to me. Baud-rates 38400, 57600, and 115200 are not defined in IEC 62056-21 standard, but support for them is coming in the next GXDLMDirector release.
BR,
Mikko
Hi Mikko,
Hi Mikko,
I have sent a trace to your personal email address this afternoon. Sometimes my mail is marked as spam, could you please check it?
Best, Jeroen
Hi Jeroen,
Hi Jeroen,
I received it. A new version is planned to release during this week.
BR,
Mikko
A new version for DLMS.c I
A new version for DLMS.c I assume? Supporting 32 bit indices? Thank you very much, looking forward to it!
Hi,
Hi,
Yes, a new version from GuruxDLMS.c
BR,
Mikko
Hi,
Hi,
A new version is released where this is added.
BR,
Mikko
Hi Mikko,
Hi Mikko,
Thanks for the quick adaptations! However, I think there is still something wrong. If I do NOT #define GX_DLMS_BYTE_BUFFER_SIZE_32 the library works as before (Out of memory on 32k). If I do #define GX_DLMS_BYTE_BUFFER_SIZE_32 there is something wrong with the bytebuffer
I added a printf in the client example:
int readDLMSPacket(
gxByteBuffer* data,
gxReplyData* reply)
{
int resend = 0, ret = DLMS_ERROR_CODE_OK;
if (data->size == 0)
{
return DLMS_ERROR_CODE_OK;
}
reply->complete = 0;
frameData.size = 0;
frameData.position = 0;
//Send data.
Serial.printf("size is %lu, position is %lu, capacity is %lu\n", data->size, data->position, data->capacity);
Without the #define this gives:
size is 10, position is 0, capacity is 58
With the #define this gives:
size is 3131899904, position is 3131922040, capacity is 655418
Adding #define ESP_PLATFORM does not correct this.
Hi,
Hi,
Can you get the latest version? There are a few changes that I hope will solve this. We'll build a special meter where we can test this, but it'll take some time.
BR,
Mikko
Yes I got todays version,
Yes I got todays version, downloaded today, with the #if defined(GX_DLMS_BYTE_BUFFER_SIZE_32) in e.g. bytebuffer.h.
Hi,
Hi,
I'm sorry. We made few changes after your post. Did you use that version?
BR,
Mikko
Hi Mikko,
Hi Mikko,
I just tried the version I downloaded today. This is not working anymore, now also the 16 bit version stopped working. I think at readDLMSPacket no message is sent to the meter, but I am not sure.
Best, Jeroen
Hi,
Hi,
It works without problems in our environments. We made a few changes. I hope it works now for you.
It is working if you don't define GX_DLMS_BYTE_BUFFER_SIZE_32?
BR,
Mikko