Check how long it takes a story to run
When you are in your story, click the 3 dot menu in the top right corner. Select Story Runs from the menu.
You’ll now see the most recent story runs, the route taken, and any actions executed.
In this view, you’ll also see the number of events, start time, finish time and story run duration.
Read more about story runs in our docs here.
Stories in busy state
A story that is noted to be in a "busy state" in Tines will stay that way for one hour. The story will be reviewed after that hour.
The impact is that the story runs slower than other stories until the load reduces.
For more details on job management see here.
Action run orchestration
In Tines, it's important to consider the issue of fairness regarding action runs. How do you ensure a single story doesn't monopolize all of the compute power for action runs on a stack, thus stunting other stories? This question applies to single and multi-tenant stacks, where all of the stories share action run computing capacity.
To address this issue, we have a concept called "fair orchestration." "Fair orchestration" guarantees that each story on a stack receives a fair amount of worker time to process action runs.
How does it work?
The system uses two mechanisms to ensure fair resource distribution: concurrent run limits and token buckets. These are checked in sequence for every action run.
Concurrent run limits
The system allocates a percentage of total Sidekiq workers for concurrent action runs per story:
Multi-tenant stacks: 30% of total workers
Single-tenant stacks: 50% of total workers
This allocation prevents any single story from monopolizing the available workers. The system tracks the number of currently running actions for each story in Redis, and only allows new runs if they won't exceed these limits.
Token bucket system
If concurrent capacity is available, the system then checks token availability. Each story has a token bucket that refills proportionally based on the time elapsed since its last update, up to a maximum capacity of 1.5 million tokens. These 1.5 million tokens represent 25 minutes worth of worker time per minute. For high-priority stories in single-tenant environments, this limit is doubled to 3 million tokens (50 minutes of worker time).
Notes:
1 token = 1 millisecond of worker time
1,000 tokens = 1 second of worker time
1.5 million tokens = 25 minutes of total worker time
High-priority stories (single-tenant only) get 3 million tokens = 50 minutes of worker time
Worker time refers to Sidekiq workers processing capacity the actual CPU time used to execute action runs
As actions in a story execute, tokens are deducted from the story’s token bucket. From the time an action run starts, every second 1,000 tokens are deducted from the story's bucket until the action run is complete. This token consumption is tracked through a dedicated background thread to ensure accurate accounting of active runs.
If a story depletes its tokens, new action runs for that story will be pending until sufficient tokens have accumulated through the time-based refill system.
For example:
Let's say a system has 100 total Sidekiq workers:
For multi-tenant stacks: 30% allocation = 30 maximum concurrent workers
For non-multi-tenant stacks: 50% allocation = 50 maximum concurrent workers
A story starts with 1.5 million tokens and attempts to run 8 actions simultaneously:
1. First, the concurrent capacity check:
System checks if adding another run would exceed the worker allocation (30% or 50% of total workers)
Only actions within this concurrent limit will be allowed to start
The remaining actions wait in a ‘pending’ state until there is capacity (either concurrent action usage falls below the limit, or sufficient tokens are available) wait for capacity
2. Then, for the running actions:
Each action running for 10 seconds consumes 10,000 tokens (1,000 tokens/second × 10 seconds)
Total consumption is 50,000 tokens (5 actions × 10,000 tokens)
The story still has 1.45 million tokens available
3. When any of the running actions complete:
Concurrent capacity becomes available
The system checks token availability (1.45 million remaining)
If tokens are available, the next waiting action can start
Autoscaling to keep up with the demand
We ensure that our workers can auto-scale to match story run demand as needed in the Cloud. This is based on the percentage of workers currently available to process action runs on a stack. If the system notices that the percentage of workers available is less than the defined set of thresholds, workers are automatically scaled up. We do this preemptively to ensure action runs are not queued for too long.
Implementing fair orchestration in our action run logic ensures all stories receive an equitable share of compute time, and to ensure action runs are processed consistently with balanced worker allocation across the stack. Our goal here is to guarantee your action runs are getting enqueued and started as swiftly as possible, always.
High priority stories
Tines allows you to designate specific stories as high priority. This is useful when you have critical workflows, such as incident response or time-sensitive alerting, that must process action runs as quickly as possible, even during periods of high load across your stack.
What does high priority do?
When a story is marked as high priority, its token bucket capacity is doubled from the standard limit:
Story type | Token bucket capacity | Worker time equivalent |
Standard story | 1.5 million tokens | 25 minutes |
High Priority story | 3 million tokens | 50 minutes |
This means a high priority story can sustain significantly more concurrent action run processing before its token bucket is depleted, reducing the likelihood of action runs entering a pending state during heavy usage.
Note: High priority only affects the token bucket limit. Concurrent run limits (50% of total workers for single-tenant stacks) still apply equally to all stories.
How to mark a story as high priority
To designate a story as high priority:
Open your story in the storyboard.
In the properties panel, locate the High Priority setting.
Select High priority.
Note: The High Priority setting is only available on single-tenant Tines deployments. It is not available on multi-tenant (cloud) stacks.
When should you use high priority?
Consider marking a story as high priority when:
The story handles critical security incident response where delays are unacceptable.
The story is a time-sensitive alerting pipeline that must process events ahead of other workflows.
You have identified that a specific story is frequently entering a pending state due to token exhaustion during peak load
Tip: Use high priority sparingly. Since all stories on a stack share the same pool of workers, designating too many stories as high priority may reduce the benefit for each one. Reserve this setting for your most critical workflows.