patterns-data-processing · diff
git:20260821.ff81a24 to git:20260916.fb6e35b
31 added, 0 removed. Audit A to A.
---
name: patterns-data-processing
description: "Patterns for loops, aggregates, batch processing and list transformation — including which retrieve source is legal where, and why nested loops are the wrong tool. Use when a microflow processes a list, merges data, or does anything in bulk."
---
# Data Processing Patterns
Patterns for loops, aggregates, batch processing, and data transformation.
## Loop Patterns
### Basic Loop
```mdl
/**
* Process all items in a list
*/
create microflow Module.ProcessItems (
$Items: list of Module.Item
)
returns boolean
begin
declare $ProcessedCount integer = 0;
loop $item in $Items
begin
-- Process each item
change $item (ProcessedDate = [%CurrentDateTime%]);
commit $item;
set $ProcessedCount = $ProcessedCount + 1;
end loop;
log info node 'Processing' 'Processed ' + $ProcessedCount + ' items';
return true;
end;
/
```
### Loop with Filtering
```mdl
/**
* Process only active items
*/
create microflow Module.ProcessActiveItems (
$Items: list of Module.Item
)
returns integer
begin
declare $count integer = 0;
loop $item in $Items
begin
if $item/IsActive then
-- Process active item
change $item (LastProcessed = [%CurrentDateTime%]);
commit $item;
set $count = $count + 1;
end if;
end loop;
return $count;
end;
/
```
### Loop with Accumulator
```mdl
/**
* Calculate total value of all orders
*/
create microflow Module.CalculateOrderTotal (
$Orders: list of Module.Order
)
returns decimal
begin
declare $Total decimal = 0;
loop $Order in $Orders
begin
set $Total = $Total + $Order/Amount;
end loop;
return $Total;
end;
/
```
## Retrieve by Association
Use `retrieve $list from $Parent/Module.AssociationName` to retrieve related objects
via association instead of a database XPath query. This is **required** for:
- **Non-persistent entities (NPEs)** — database XPath queries always return empty for NPEs
- **Uncommitted objects** — objects not yet committed to the database
- **JSON mapping results** — imported data structures held in memory
### Persistent Entity Example
```mdl
/**
* Get all orders for a customer via association
*/
create microflow Module.GetCustomerOrders (
$Customer : Module.Customer
)
returns list of Module.Order
begin
retrieve $Orders from $Customer/Module.Order_Customer;
return $Orders;
end;
/
```
### Non-Persistent Entity Example (NPE)
```mdl
/**
* Process imported rows from an in-memory result object.
* Database RETRIEVE would return empty for NPEs — use association retrieve.
*/
create microflow Module.ProcessImportRows (
$ImportResult : Module.ImportResult
)
returns integer
begin
-- Association retrieve is the ONLY way to get related NPEs
retrieve $Rows from $ImportResult/Module.ImportResult_ImportRow;
declare $ValidCount integer = 0;
loop $row in $Rows
begin
if $row/IsValid then
set $ValidCount = $ValidCount + 1;
end if;
end loop;
return $ValidCount;
end;
/
```
### When to Use Which Retrieve
| Scenario | Syntax | Why |
|----------|--------|-----|
| Query persistent entities by attribute | `retrieve $list from Module.Entity where ...` | Database XPath query |
| Get related persistent objects | `retrieve $list from $Parent/Module.Association` | Simpler, no XPath needed |
| Get related NPEs / uncommitted objects | `retrieve $list from $Parent/Module.Association` | **Only option** — database has no data |
| JSON mapping results (import) | `retrieve $list from $Parent/Module.Association` | Mapping creates in-memory NPEs |
**Important:** Association retrieve always returns a list. It does not support WHERE, SORT BY, LIMIT, or OFFSET clauses.
## Aggregate Patterns
Aggregates use **function-call syntax** — there is no `AGGREGATE` keyword.
| Function | Syntax | Returns |
|----------|--------|---------|
| COUNT | `$n = count($list)` | Integer |
| SUM | `$n = sum($list.Attr)` | Decimal |
| AVERAGE | `$n = average($list.Attr)` | Decimal |
| MINIMUM | `$n = minimum($list.Attr)` | Same as attribute |
| MAXIMUM | `$n = maximum($list.Attr)` | Same as attribute |
**Important:** RETRIEVE implicitly declares its variable — do NOT add a separate DECLARE
before RETRIEVE, or you'll get CE0111 "Duplicate variable name".
### Count Items
```mdl
/**
* Count active customers
*/
create microflow Module.CountActiveCustomers ()
returns integer
begin
retrieve $Customers from Module.Customer
where IsActive = true;
$count = count($Customers);
return $count;
end;
/
```
### Sum Values
```mdl
/**
* Sum order amounts for a customer
*/
create microflow Module.GetCustomerTotalOrders (
$Customer: Module.Customer
)
returns decimal
begin
retrieve $Orders from Module.Order
where Module.Order_Customer = $Customer;
$Total = sum($Orders.Amount);
return $Total;
end;
/
```
### Average Calculation
```mdl
/**
* Calculate average order value
*/
create microflow Module.GetAverageOrderValue ()
returns decimal
begin
retrieve $Orders from Module.Order;
$average = average($Orders.Amount);
return $average;
end;
/
```
### Min/Max
```mdl
$MinPrice = minimum($Products.Price);
$MaxPrice = maximum($Products.Price);
```
## List Operations
+ ### One statement per operation — they do not nest
+
+ Every list operation and aggregate is a separate **activity** in Mendix, and an
+ activity stores its list as a **variable reference**. There is no slot for a
+ nested computation, so this is not a shorter spelling — it is a list argument the
+ model cannot hold:
+
+ ```mdl
+ -- WRONG. mxcli check refuses this as MDL-LISTOP02.
+ $n = count(filter($Requests, $currentObject/Status = Module.ENUM_Status.Approved));
+ ```
+
+ Before the rule existed it parsed, passed `check`, and execed with
+ `Created microflow` — then dropped the inner call entirely and wrote an activity
+ with an empty list, which mxbuild rejected with **CE0012** (`The 'List' property
+ is required.`) for an aggregate or **CE0096** for a list operation. The
+ `sort(filter(…), Attr)` shape was worse still: with the list gone the sort
+ attribute has no entity to resolve against, and mxbuild aborts rather than
+ reporting an error.
+
+ Give the inner operation its own statement and pass the variable:
+
+ ```mdl
+ -- RIGHT
+ $Approved = filter($Requests, $currentObject/Status = Module.ENUM_Status.Approved);
+ $n = count($Approved);
+ ```
+
+ The same applies to both operands of `union`/`intersect`/`subtract`, and to any
+ non-variable list argument — `count('nonsense')` fails the same way.
+
### Add to List
```mdl
/**
* Collect matching items into a list
*/
create microflow Module.CollectHighValueOrders (
$Orders: list of Module.Order,
$Threshold: decimal
)
returns list of Module.Order
begin
$HighValue = create list of Module.Order;
loop $Order in $Orders
begin
if $Order/Amount > $Threshold then
add $Order to $HighValue;
end if;
end loop;
return $HighValue;
end;
/
```
### Remove from List
```mdl
/**
* Remove inactive items from list
*/
create microflow Module.FilterActiveItems (
$Items: list of Module.Item
)
returns list of Module.Item
begin
$ToRemove = create list of Module.Item;
-- Collect items to remove
loop $item in $Items
begin
if not($item/IsActive) then
add $item to $ToRemove;
end if;
end loop;
-- Remove collected items
loop $item in $ToRemove
begin
remove $item from $Items;
end loop;
return $Items;
end;
/
```
## Batch Processing
### Process in Batches
```mdl
/**
* Process large dataset in batches
* Commits after each batch to avoid memory issues
*/
create microflow Module.BatchProcess (
$Items: list of Module.Item,
$BatchSize: integer
)
returns integer
begin
declare $Processed integer = 0;
declare $BatchCount integer = 0;
loop $item in $Items
begin
-- Process item
change $item (status = 'Processed');
set $BatchCount = $BatchCount + 1;
set $Processed = $Processed + 1;
-- Commit batch
if $BatchCount >= $BatchSize then
commit $item;
set $BatchCount = 0;
log info node 'Batch' 'Processed ' + $Processed + ' items';
end if;
end loop;
-- Final commit for remaining items
if $BatchCount > 0 then
log info node 'Batch' 'Final batch: ' + $Processed + ' total';
end if;
return $Processed;
end;
/
```
## Data Transformation
### Copy Entity
```mdl
/**
* Create a copy of an order
*/
create microflow Module.CopyOrder (
$source: Module.Order
)
returns Module.Order
begin
$Copy = create Module.Order (
OrderNumber = 'COPY-' + $source/OrderNumber,
Amount = $source/Amount,
status = 'Draft',
CreatedDate = [%CurrentDateTime%]
);
-- Copy association
set $Copy/Module.Order_Customer = $source/Module.Order_Customer;
commit $Copy;
return $Copy;
end;
/
```
### Transform List
```mdl
/**
* Create summary records from detail records
*/
create microflow Module.CreateOrderSummaries (
$Orders: list of Module.Order
)
returns list of Module.OrderSummary
begin
$Summaries = create list of Module.OrderSummary;
loop $Order in $Orders
begin
$Summary = create Module.OrderSummary (
OrderNumber = $Order/OrderNumber,
TotalAmount = $Order/Amount,
CustomerName = $Order/Module.Order_Customer/Name
);
add $Summary to $Summaries;
end loop;
return $Summaries;
end;
/
```
## Error Handling in Loops
### Continue on Error
```mdl
/**
* Process items, log errors but continue
*/
create microflow Module.ProcessWithErrorHandling (
$Items: list of Module.Item
)
returns integer
begin
declare $Processed integer = 0;
declare $Errors integer = 0;
loop $item in $Items
begin
if $item/data = empty then
log warning node 'Process' 'Skipping item with empty data: ' + $item/Code;
set $Errors = $Errors + 1;
else
change $item (status = 'Processed');
commit $item;
set $Processed = $Processed + 1;
end if;
end loop;
log info node 'Process' 'Completed: ' + $Processed + ' processed, ' + $Errors + ' errors';
return $Processed;
end;
/
```
## Best Practices
1. **Commit inside loops carefully**: Can cause performance issues on large sets
2. **Use batch commits**: Commit every N records for large datasets
3. **Log progress**: Add logging for long-running operations
4. **Handle errors gracefully**: Don't let one bad record stop the whole process
5. **Return counts**: Help callers know what was processed
6. **Use meaningful variable names**: `$ProcessedCount` not `$c`