Discussion

Discussion topics imported from PFN repo.

BQL should support sliding steps of windows. At the same time, it should also refine the computation timing. Possible methods are:

1. SELECT is computed every time a new tuple arrives
2. SELECT is computed only when a window is a full of tuples

• "full" cannot be defined for time-based windows

3. SELECT is cimputed only when a window slides

• the computation delays

Users may select the optimal timing.

We need to think of how 2 and 3 work with joins.

Sources doesn't have to consume resources (CPU/memory/etc) while it doesn't have output destinations. This can be implemented by passing a special Writer to GenerateStream. The Writer implements a method to notify the destination status change.

Ideally, a source or a box shouldn't run any process while it isn't (indirectly) connected to a sink.

-> This feature well be useful but cannot implement it on stream dataflow because SensorBee is adopted dynamic topology model.

This is a mere note for expected requirements:

• QoS management
• Non-blocking software upgrade
  • When upgrading SensorBee, downtime should be 0 or should not cause any data loss at least.
• Backup
• Performance monitoring and alerting
• TODO: more requirements...

Some of these requirements should be implemented as a part of DIMo management tool.

For high availability and falt tolerance, the following paper may describe what we need: High-Availability Algorithms for Distributed Stream Processing.

nobu:

I sometimes want to change a BQL statement a little without rebooting SensorBee to see how its behavior changes. I can easily do it by DROP and CREATE again if the box isn't connected to any other stream or sinks. However, when the box is connected to others, there needs to be a way to update the internal of the box without disconnecting it from them.

Of course, there's a problem like schema change, though.

tgp:

I think in PostgreSQL you can do CREATE OR REPLACE VIEW only as long as the schema doesn't change; that would be a possibility.

tgp:

To debug performance of UDFs, it would be very helpful if we had a function to output execution times of each box in a SensorBee topology. (We already have the possibility to store the processing flow's trace in every tuple, so I think the data is already available, it just needs to be collected, persisted and visualized.) I imagine a diagram that shows the average processing time (plus standard deviation etc.) of tuples per box, queue waiting time and possibly others; maybe even rendered on top of a graph

I wonder if it is also possible to get the memory usage caused by each box (in order to find memory leaks), but even it is possible to obtain that information, I think it is not readily available yet.

nobu:

We're planning to have sensorbee top command. So, let's disucss what information the command should output next week! This feature is very exciting to have.

1. As a source:

    CREATE SOURCE hoge TYPE websocket WITH host='abc', port=8090,
      topology='foo', stream='bar';
    SELECT RSTREAM * FROM hoge [RANGE 1 TUPLES];
   

2. As a UDSF:

    SELECT RSTREAM * FROM remote('bar', 'foo', 'abc', 8090) [RANGE 1 TUPLES];
   

(yet under discussion)

tgp:

In PostgreSQL (see http://www.postgresql.org/docs/9.1/static/xfunc-volatility.html, functions can be declared as one of:

• VOLATILE: can do anything, including DB modifications, and even return a different result on every call
• STABLE: cannot modify the DB and returns the same result for given input parameters in the scope of a statement
• IMMUTABLE: cannot modify the DB and returns the same result for given input parameters forever

Examples for each class are:

• random()
• current_timestamp() (← this returns the start time of a transaction)
• abs()

I was wondering whether we should offer an ability to declare volatility in BQL or assume some class. For example, in the case of SELECT avg(random()) AS r1, avg(random()) AS r2 FROM ... in BQL, the current codebase will use only one value of random() for each input row, therefore r1 and r2 will be the same in the end. (Note: In SELECT random() AS r1, random() AS r2, this is not the case.)

Even though this is a constructed case, I think that when we have functions to modify or query a machine learning model, this may become important, so I was wondering if we should assume some volatility level and/or side-effect behavior?

Similar to what we discussed about detecting UD(A)F type from its parameters, we may not be able to detect automatically whether a UDF is volatile or not.

To deal with this issue independent of whether some particular function is volatile or not, I added a Volatility() method to execution.FlatExpression and just assume for now that all UDFs are volatile. Parameters to aggregate functions that are volatile are then not re-used within a row, but they are computed as often as they mentioned:

• For SELECT sum(a)/count(a), the (immutable) expression a is only evaluated and collected once for every row, so the array passed in to Sum.Call() and Count.Call() is the same object.
• However, in SELECT sum(f(a))/count(f(a)), the (volatile) expression f(a) is computed twice per row and the arrays passed to Sum.Call() and Count.Call() are distinct.

We can deal with the question of how to mark a certain UDF as immutable later, but at least for now we are safe with respect to random().

nobu: I think this will be necessary later but felt it could be too early to work on right now. Changes in the PR seem minimal, though.

In terms of UD(A)F, we'll be able to add the method to them later or define something like Volatilityer (tier?).

tgp: Volatilitier, that would be great :D