Since #326 we allow to define operations inline with its enclosing task. The operation defines what API call should be executed and the task defines how it should be executed (e.g. warmup-iterations, iterations, target-throughput). In order to save users some hassle, all of the task properties have default values. Additionally, the operation definition allows to define arbitrary parameters. This leads to a situation where a user can define this task:

{
  "operation": {
    "name": "query-match-car",
    "operation-type": "search",
    ...,
    "clients": 2,
    "warmup-iterations": 1000,
    "iterations": 10000,
    "target-throughput": 100    
  }
}

when they should have done this

{
  "operation": {
    "name": "query-match-car",
    "operation-type": "search",
    ...
  },
  "clients": 2,
  "warmup-iterations": 1000,
  "iterations": 10000,
  "target-throughput": 100
}

In the first case, the properties are attributed to the operation and the task will run with defaults (one client, no warmup iteration, one measurement iteration, no target throughput). This is trappy and surprises users.

We have several ways to address this:

Change track file format

We can change the track file format so task properties need to be defined in their own block, e.g.:

{
  "operation": {
    "name": "query-match-car",
    "operation-type": "search",
    ...
  },
  "task": {
    "clients": 2,
    "warmup-iterations": 1000,
    "iterations": 10000,
    "target-throughput": 100
  }
}

We'd also not define default values and require users to specify properties explicitly.

Make task properties mandatory

A more light-weight approach is to be more strict, remove the default values and require users to specify explicit values. We can prepare the official Rally tracks in advance to conform to that requirement and so this would not affect the majority of the users.

Detect problematic situations

Another option is to detect that the user has specified task-related properties on the operation (but none on the task) and warn the user about it. The problem with this approach is that we can never be sure that a user has intentionally passed the task-related properties. We could declare them as reserved names though. I am only mentioning this possibility for completeness but I think this approach is trappy in itself.

I am favor of option two but this is open for discussion.