Has anyone done any benchmark/stress test on Streampipes?

[baoyicai]

Hi, I'm wondering what's the throughput for processing the time-series data, if there's any stress test done before?

It looks like integrating Flink is sitll on the way, I'm a little concerned with the performance with the native processing engine of Streampipes.

[bossenti]

Hi @baoyicai,

Thank you for this interesting question.
Unfortunately, we don't have any official benchmark or stress test numbers. However, we are happy to share our experience.
So far, we haven't seen any performance issues with StreamPipes related to high load. We have seen StreamPipes behave reliably in a variety of industrial contexts. In addition, the underlying microservice architecture makes StreamPipes resilient to high load. However, we have to admit that StreamPipes hasn't yet been proven to be reliable under very high event frequency or a huge number of connected industrial devices (at least to our knowledge).

Are you willing to share what order of magnitude in frequency or connected data sources you are concerned about? We can give you a concrete assessment of StreamPipes' performance for your scenario.

[baoyicai]

Hi @bossenti ,

Thanks for the reply, the magnitude I'm looking at is around ~100K of data streams(tags), for example, get the tag value from OPC at the interval of 5 seconds, store them to influxdb, and do a simple comparion to a threshold value, and output the result.

Potentially there will be more tags to process as time goes, but I guess if StreamPipes can handle this volumn with a single server, it's more than enough for my scenario now.

[baoyicai]

distributed. they will be from multiple opc da/ua sources.

[bossenti]

Just to throw some numbers out there: We have an example where we are reading 500 tags per second, so that would be roughly equivalent to 2,500 tags every 5 seconds. Here we don't have any issues with the performance at StreamPipes at all (although this is certainly still far away from the magnitude of your use case).

But the thing to keep in mind here is that it really depends on what you want to do with your data in StreamPipes. For example, do you just want to persist it in our storage, or do you want to build large pipelines with complex processing operations.

I think a good solution for you might be to use our FileStream connector and roughly simulate your use case. This connector allows you to load data from various file formats (e.g. CSV or JSON) and set the read frequency.
In case you do so, we would be happy to hear from your experiences 🙂

[baoyicai]

hi @bossenti ,thanks for the numbers, I'll give it a try and see how it works.

[baoyicai]

tried to create 5000 pipelines, it seems that the pipeline list page becomes slow. maybe it's my laptop's performance issue.

[bossenti]

Yes, our user interface is not currently designed to display such a number of pipelines, e.g. there is no pagination yet.
Have you encountered any other performance limitations?
Does the current situation prevent you from adopting it?