Diagnosing and Troubleshooting Multiple Values Standing Queries
Troubleshooting issues with Multiple Values Standing Queries (MVSQ) is a lot like debugging a software program. There’s no “one size fits all” procedure that will enable you to identify and resolve every issue that might arise as a result of using this feature. However, there are some troubleshooting tips you can use to help you get started when an MVSQ doesn’t produce the desired results.
1. Validate ingested data
The first thing to do is to make sure that data is flowing into Quine, and to validate that this data has the expected shape your MVSQ is looking for. Using the Exploration UI, use the recentNodes procedure (CALL recentNodes(100) for example) to get a glimpse of the newest data added to the graph. This will help validate that ingest stream definitions are processing data. To get a better picture, double-click on nodes in the exploration UI to view the edges, or hover over the node to see the data.
Additionally, ensure that the baseline assumptions that the MVSQ in question is making about the graph are valid. Oftentimes an ingest query is structured to create nodes with ids derived from natural or synthetic keys found in the data. Using the exploration UI, query for a node that’s been ingested with that id. Note that the idFrom will always generate the same id from the same inputs, so utilize that knowledge along with known ingested data to query for those nodes.
2. Run as ad-hoc
Having established that data is flowing into the system, run the MVSQ as an ad-hoc query in the exploration UI. All MVSQs are valid ad-hoc queries (note that the inverse is not true). It might be helpful to modify the MVSQ to specifically look for the ids used in step 1 above.
Query returns data
If the query returns data, then there are two possible issues.
1. Not all expected patterns are being matched
Specifically if the MVSQ was modified to look for specific ids, and the query returned results when modified as such, then the issue might be that not all of the ingest data conforms to the pattern described in the MVSQ. Identify missing patterns and use the exploration UI to discover why the pattern fails to match.
2. No patterns are being matched
If no results are returned, the debug.node procedure can be used to help identify where the issue lies. This requires some explanation of the query plan structure and Quine node lifecycle.
Query Plan
Quine uses the actor model of concurrency to achieve the scalability and performance demands of high-volume real-time streaming data. In this model, graph nodes are actors. This impacts how the query plan is generated. Specifically, the query plan is structured to maximize on-node executability. This query plan gets distributed across the graph as new nodes are created.
Each part of the query plan uses a pub-sub model. Nested parts of the query plan might get distributed across edges to related nodes, and at each level the “parent” plan subscribes to the “child” plans, and the “child” plans publish results to the “parent”. This is a key factor in enabling these queries to match data in real time.
Explaining every element of the query plan is outside the scope of this document, but here are some elements that frequently occur.
FilterMap
This query plan operates on the results of the nested query plan by projecting data and optionally filtering data. One thing to note here is that not all predicates in the MVSQ will show up here as an optional filter. Some predicates get more efficient runtime expressions as specialized query plans.
Cross
This query plan operates on the results of a list of query plans by calculating the cross product. This is a fundamental query plan combinator in Quine, and as such it is often used to calculate the cross product of two single result query plans.
LocalProperty
This query plan will generate a single row that contains a property value from the node it was executed on.
Debug Node
Using the debug.node procedure it is possible to get information about the state of a given node’s MVSQ query plan. Specifically, YIELDing the multipleValuesStandingQueryStates will provide a view of the runtime state of one or more MVSQs. Note that it is helpful to ensure that the only running MVSQ is the one that is actively being debugged, otherwise this state will be considerably harder to diagnose.
From the output of this procedure, it should be possible to determine where exactly the failure is happening. As with all troubleshooting activities, it can be helpful to simplify the MVSQ (and potentially the ingest definitions) to help identify the problem.