Repository navigation
Replies: 5 comments 4 replies
|
Hi @vmklvm, The way you're structuring your data is that each event would be overwriting the previous events values since your properties are all the same. You would also have to consider that each consecutive event would have to be written to add the input from the previous event to its own output the keep the data flowing through; it doesn't automatically continue to pass the same data across inside a workflow. What you would need to do is differentiate between events by placing them into their own property. For example: {
"Event1_Output": {
"varkey1": "333",
"varkey2": {
"varkey2-1": "333",
"varkey2-2": "333"
}
}
}This way, you could then reference Event2_Output.varkey2.varkey2-1 from your Event 1 if it ran in the third sequence. But to reiterate, your event scripts need to be aware to populate their output with the input from the previous event in order to keep that data flowing into the next event. Event 1 |
|
@vmklvm Reading through your post again, I think I understand more what you're asking for, which I didn't answer in my previous reply. It sounds like you would benefit from using bucket data and accessing it via the new bucket API. You can write your JSON to a buck data in your first event and then read it from your last event, even if the first event has error, as long as the error occurs after it has written the data, of course. That will be the best way to have data consistency across events like that. |
|
@vmklvm I completely understand where you're coming from but with how xyOps is designed to work with any CLI using STDIN/STDOUT, it's basically impossible to do it any other way that would work across the board. Having a simple way to access the data in the buckets becomes a task for the plugin that's being run to provide the functionality. For example, in the PowerShell plugin that I have, it allows you to set up a bucket with API access to then simply use the Get-xyOpsBucketData and Set-xyOpsBucketData functions to get and set data that persists. Those functions use the API to do the work and since they exist in the plugin code, they're accessible by any of my PowerShell scripts that I run. So depending on the language you're using, you could submit a feature request for the associated plugin to add similar functions to be made available to your scripts. @jhuckaby Does that sound right? |
|
Yup, @nickdollimount is correct. Data buckets are probably the right solution, and adding a "utility" function wrapper is a nice touch (his Powershell plugin is awesome). Also, if you don't want to have to write code to call the API in your scripts, you can use Fetch Bucket and Store Bucket job actions attached to the events (so you can setup the entire get/fetch from the UI):
This illustrates how all 3 events can share the same storage bucket. Each one reads from it at job start time (so the data will be in Okay, I re-read the initial request again. Let me suggest another possible solution here. Since you seem to have a sequence of events where you may need to access the data from all of them, what you may want to utilize is a workflow Join Controller, which joins data together from multiple events, then passes it to a new one:
This is notable because, as Nick points out, your example events are using the same data key names, so they will clobber and overwrite each other in most cases. To help with situations like this, the Join Controller has a neat trick where it joins in two distinct ways, and gives you access to both. Here is the actual data passed to that 4th job on the right side: {
"items": [
{
"varkey1": "222",
"varkey2": {
"varkey2-1": "222",
"varkey2-2": "222"
}
},
{
"varkey1": "333",
"varkey2": {
"varkey2-1": "333",
"varkey2-2": "333"
}
},
{
"varkey1": "111",
"varkey2": {
"varkey2-1": "111",
"varkey2-2": "111"
}
}
],
"combined": {
"varkey1": "111",
"varkey2": {
"varkey2-1": "111",
"varkey2-2": "111"
}
}
}The join controller gives you an array of each individual data object from each of the input jobs wired into it (in the order of completion), AND it also gives you a "combined" version, which is shallow-merged, and the latter prevails on key collision. In this case since all your events use the same keys, they all clobber each other, but event 1 was the slowest, so it "won". I hope this makes sense. Would either of these solutions help your situation? |
|
The reasons why I didn't implement something where "any workflow job can access data from any other job at any time" are:
Due to these footguns, I decided to not implement such a thing, and instead built Storage Buckets, which were literally designed to solve this very issue, and give you complete control over where and how and when the data is fetched and stored. |


Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
For example:
Workflow1
Event1
output_data:
{
"varkey1": "111",
"varkey2": {
"varkey2-1": "111"
"varkey2-2": "111"
}
}
Event2
output_data:
{
"varkey1": "222",
"varkey2": {
"varkey2-1": "222",
"varkey2-2": "222"
}
}
Event3
output_data:
{
"varkey1": "333",
"varkey2": {
"varkey2-1": "333",
"varkey2-2": "333"
}
}
SEQUENCE OF EXECUTING JOBS (probably this don't have big sense): Event 2 > Event 3 > Event 1
I wounder to access data in Event1 from Event2.varkey2.varkey2-1
If there is not Event1 in workflow, through some error / warning
What do you think?
I saw discussion #69, but as I understand there is sequential type logic data flow between events/jobs?
All reactions