To cancel a transaction via SuiteScript, you cannot simply update the record's status value.
Instead, you have to close each line.
Below is a snippet of SuiteScript that shows how to do this.
recordObj = record.load( { type: record.Type.SALES_ORDER, id: transactionID, isDynamic : true } );
var lineCount = recordObj.getLineCount( { sublistId: 'item' } );
for ( var i = 0; i <= ( lineCount - 1 ); i++ ) {
var sublistLine = recordObj.selectLine( { sublistId: 'item', line: i } );
recordObj.setCurrentSublistValue( { sublistId: 'item', fieldId: 'isclosed', value: true } );
recordObj.commitLine( { sublistId: 'item' } );
}
recordObj.save( { enableSourcing: true, ignoreMandatoryFields: true } );
Note that this doesn't actually cancel the transaction. Instead, it closes it. By closing the lines, you're indicating that you do not intend to fulfill them. And by closing all of the lines, you're effectively closing the entire order.
The difference between a cancelled and a closed transaction is that you can re-open a closed transaction (by un-checking the "Closed" value for each line).