PatchOptions for RTDB

[eXtr3me26]

Hi mobizt and community!

At the moment I am integrating this library in my project with a communication to RTDB and Firestore at the same time.
I dont have to write extremely much data, but still it would be good to send data always in a json to save unnecessary traffic..
So, for Firestore that is already supported - thank you for those PatchOptions!
But for the RTDB I am missing that feature.

I started to try the Database.update with the JSON of only changed values.
But sadly all the other entries get deleted when the data is send to the RTDB.

Is there already a solution to this or do I really need to send each database entry as a single "Database.set"?

hint: I already studied your examples, but the only solution I found until now is to write the single data separately.

Thank you for any support!

[mobizt]

The RealtimeDatabase::update function works as expected.

Let's see the code below.

object_t json, json1, json2;
JsonWriter writer;
writer.create(json1, "data1/value", random(1000, 2000));
writer.create(json2, "data2/value", random(1000, 2000)); 
        
writer.join(json, 2, json1, json2);

Serial.println("Updating data (JSON object only)...");
Database.update(aClient, "/test/json", json, asyncCB, "updateTask1");

writer.create(json1, "data2/value", "hello");
Database.update(aClient, "/test/json", json1, asyncCB, "updateTask2");

writer.create(json2, "data3/value", "goodbye");
Database.update(aClient, "/test/json", json2, asyncCB, "updateTask3");

The first task will update value to the test/json. If this path is null or not exist, it will create.
{"data1":{"value":1140},"data2":{"value":1919}}

The second task update the value to the same path i.e. test/json which only data2 value will be updated.
{"data2":{"value":"hello"}}

The third task will update the value to the same path test/json which new data3 will be added as it does not yet exist.
{"data3":{"value":"goodbye"}}

Finally, data in test/json will be.

{
   "data1":{
      "value":1140
   },
   "data2":{
      "value":"hello"
   },
   "data3":{
      "value":"goodbye"
   }
}

The second and third updates will not replace your whole data but only its children's nodes.

Then you have to verify the input data before you call update that it contains only portion data of existing one or contains all children data.

[mobizt]

It's not the problems for simultaneous operations from different client or apps to the same database.

There is nothing to do when all operations work (write data) in different locations or nodes.

In case some operations work with the same location or node, you should use Etag to make sure you currently working with the node that is up to date (no one changes the data before you to complete your write).

For simultaneous operations at the same node, before you write (set/update/push) to some node, you can read the Etag from your last operation of get/set/push to that node e.g. using get to get the node's Etag.

When you know that node Etag, you can assign that Etag with the write operation (set/update/push) to that node.

The operation will success if no one (app or client) changes the value of that node while you are currently working with it, and it will fail if someone changes its data before you.

For simultaneous operations, all client apps should use Etag when writing or updating the data.

Here is the Etag example.

For more info about Etag.

[eXtr3me26]

That is a very good hint. Thank you very much mobizt!

Actually I am facing another problem.

My current database looks like this:

root {
sub1 {
subEntry1: 100,
subEntry2: 200,
subEntry3: 300,
subEntry4: "valid"
},
sub2 {
subEntry1: 400,
subEntry2: 500,
subEntry3: 600,
subEntry4: "invalid"
}
}

I have a stream which looks on the root.

My problem is, that the ESP32 writes on:

• sub1->subEntry1
• sub1->subEntry3
• sub2->subEntry2
• sub2->subEntry4

The app writes on:

• sub1->subEntry2
• sub1->subEntry4
• sub2->subEntry1
• sub2->subEntry3

Now the problem is, that every time I write to a subEntry from the ESP32 I get the stream-Callback with the information about the change I made, even when I dont use a asyncCallback in the set-Command.

I did not find a way to set a filter like the dataOption which is available for Firestore.

That would actually be very usefull because then only the relevant changes would be sent to the ESP32.

Do you understand what I mean?

I will adapt my setup to the current implementation and add a toggle-Bit which the app sets to true when new data is available.
The stream will no longer look on root, but on that toggle-Bit.
Each time I receive the toggle-Bit I read the entire root with a callback and then filter the data by myself.

So, the data will look like this:

root {
toggleBit: false,
sub1 {
subEntry1: 100,
subEntry2: 200,
subEntry3: 300,
subEntry4: "valid"
},
sub2 {
subEntry1: 400,
subEntry2: 500,
subEntry3: 600,
subEntry4: "invalid"
}
}

Is that the only way how this can work, or does Firebase support something like a data-filter in the stream?

Thank you in advance for your support!

[mobizt]

Before implement something, you have to understand its feature and capability.

https://firebase.google.com/docs/reference/rest/database
https://firebase.google.com/docs/database

[eXtr3me26]

Thank you. Then that is and will maybe never be supported.

I continue with my concept then. That makes it easy for me.

What I still dont understand is, that I only get 1 data from the Stream-Callback even when I set 50 nodes in one cycle using the Database.set-method.
Shouldnt I be getting all 50 callbacks at once with each changed entry, or even 1 JSON with all changes in it?

Is that a restriction of your library or also from Firebase?

[mobizt]

This can be a network issue. There is no cache in REST API client especially in limited memory device.

This test will show you the performance and reliability of Firebase Realtime Database.
https://github.com/mobizt/FirebaseClient/tree/main/examples/RealtimeDatabase/StreamPerformanceTest

You will get every data as fast as possible. It is still slower than the Firebase SDK client that uses the web socket technology.

If some data is missing when your test, that is because of network.