Skip to main content

Batch Processing

Batch processing lets you run the same template against many inputs at once. Instead of manually running a template 200 times, select 200 items and let the system create and manage all the tasks automatically.

Why Batch Processing?​

Consider a modernization project with 500 RPG programs to convert. Without batch processing:

  • Run the template manually for each program
  • Wait for completion, review, approve
  • Repeat 500 times over weeks or months

With batch processing:

  • Select all 500 programs
  • Click Run All
  • Tasks execute in parallel
  • Review and approve as they complete
  • Turn months/years of work into weeks

The template ensures every program gets the same careful treatment. The batch operation handles the scale.

How It Works​

Multi-Select Parameters​

Templates can mark parameters as supporting multiple selection. When you encounter such a parameter:

  1. The prompt allows selecting multiple items (files, options, etc.)
  2. A counter shows how many items you've selected
  3. The Run All button shows the total tasks that will be created

Task Creation​

When you click Run All:

  1. The system creates one task per selected item
  2. Each task receives the template with that item's value substituted
  3. Tasks enter the queue and execute as slots become available
  4. Parallel execution is limited by your configured queue slots

Monitoring Batch Progress​

After launching a batch:

  • View all created tasks in the Tasks list
  • Filter by template name to see just your batch
  • Monitor progress as tasks complete
  • Pin tasks that need attention

Running via CLI​

You can also run templates from the command line using the coder CLI:

# Run a template with a single value
coder run convert-to-modern-rpg --source_file=MYPGM.rpgle

# Run with multiple values (creates multiple tasks)
coder run convert-to-modern-rpg --source_file=PGM1.rpgle --source_file=PGM2.rpgle

# Specify environment and branch
coder run convert-to-modern-rpg --environment=pjs-dev --branch=feature-xyz --source_file=MYPGM.rpgle

CLI execution is useful for:

  • Scripted workflows: Integrate template runs into shell scripts or automation
  • CI/CD pipelines: Trigger batch processing from build systems
  • Programmatic input: Generate the list of items to process dynamically

See CLI in the Working with Code section for full command reference.

Cartesian Products​

When multiple parameters support multi-select, you get a cartesian product—every combination of selections.

Example​

A template with two multi-select parameters:

  • Source files: Select 10 files
  • Target formats: Select 3 formats

Result: 30 tasks (10 files × 3 formats)

Each task processes one file in one format. Every combination is covered.

Practical Limits​

The UI warns you when combinations exceed practical limits. Creating thousands of tasks is technically possible but may not be the best approach—consider breaking the work into smaller batches.

Batch Workflow​

1. Validate the Template First​

Before running at scale:

  1. Run the template with a single representative item
  2. Review the results carefully
  3. Verify the output meets expectations
  4. Fix any issues in the template

Never batch-process hundreds of items with an untested template.

2. Start with a Small Batch​

Once the template works for one item:

  1. Select 5-10 similar items
  2. Run the batch
  3. Review all results
  4. Confirm consistency across the batch

This catches edge cases the single-item test might miss.

3. Scale Up​

When confident in the template:

  1. Select the full set of items
  2. Run the batch
  3. Monitor progress
  4. Review and approve as tasks complete

4. Handle Failures​

Some tasks may fail due to:

  • Edge cases the template doesn't handle
  • Unusual input that needs special treatment
  • Transient errors (retry these)

Review failed tasks individually. You may need to:

  • Adjust the template and re-run failed items
  • Handle exceptions manually
  • Exclude certain items from batch processing

Best Practices​

Test before scaling: Always validate with single items and small batches first.

Use meaningful names: Template names and parameters should make batch results easy to identify.

Monitor queue depth: Large batches may take time if queue slots are limited. Plan accordingly.

Review incrementally: Don't wait for all tasks to complete. Review and approve as they finish.

Group related work: Batch similar items together. Mixing very different inputs may produce inconsistent results.

Plan for failures: Expect some percentage of tasks to need individual attention. Build this into your timeline.