Inefficient Code

I have a rule that pulls values form an external data source and I'm thinking that I didn't code the rule efficiently because when I go to compose it takes quite a while for my computer to process the information. I wondering if anyone out there has any advice to make my code run quicker.

 
if (FusionPro.inValidation)
 Rule("OnJobStart");
 var test_val = Field("Member Number");
 var matches = new Array();
for (var i = 0; i <= Test_File.recordCount; i++)
{
 var ID_Variable = Test_File.GetFieldValue(i, "Member Number");
 var Charge_Description = Test_File.GetFieldValue(i, "Description");
 var Charge_TransDate = Test_File.GetFieldValue(i, "Transaction Date");
 var Charge_PostDate = Test_File.GetFieldValue(i, "Post Date");
 var Charge_SiteNum = Test_File.GetFieldValue(i, "Site Number");
 var Charge_InvNum = Test_File.GetFieldValue(i, "Invoice Number");
 var Charge_Charges = Test_File.GetFieldValue(i, "Charges");
 var Charge_Payment = Test_File.GetFieldValue(i, "Payments");
 if (test_val == ID_Variable)
   matches.push(Charge_TransDate + "\t" + Charge_PostDate + "\t" + Charge_SiteNum + "\t" + Charge_InvNum + "\t" + Charge_Description + "\t" + Charge_Charges + "\t" + Charge_Payment);
}
return matches.join("\n");

Dana,

Is this rule within another CallBack rule or a normal rule? Also, any reason why you are calling the OnJobStart rule each time? That maybe your problem. Once you load the external data file at the beginning of your job (in the OnJobStart rule) then you don't need to load it again. You can just reference it over and over again via a counter as you do in your for-loop.

Good Luck

Is this rule within another CallBack rule or a normal rule? Also, any reason why you are calling the OnJobStart rule each time? That maybe your problem. Once you load the external data file at the beginning of your job (in the OnJobStart rule) then you don't need to load it again. You can just reference it over and over again via a counter as you do in your for-loop.

Actually, what Dana is doing here in terms of calling the OnJobStart rule, guarded by a test of the FusionPro.inValidation property, is correct. At rule validation time (in the Rule Editor dialog), the callback rules are not automatically invoked like they are in a composition. So you need to call the callback rule explicitly if your rule depends upon it, which, in this case, it does because it's loading up an ExternalDataFileEx object. The FusionPro.inValidation property ensures that the OnJobStart rule is not called more than once at composition time. So I think that's a red herring here.

I think that the slowdown is occuring is because you're doing all this work to populate an array once for every record. My question is, what are you doing with this data once it's in the array? The answer to that will help me to suggest a way to optimize the logic. You probably want to at least save the results of this search in another object, by the field value, so that at worst you're doing all this lookup in the external data file only once for each unique field value in your main data file.

I've included a sample of what the code outputs into my layout. Some records return so many matches that overflow to another page would be required. Thank you for the help.

http://www.pipalaska.com/tmp/addresses-Output-1.gif

I've included a sample of what the code outputs into my layout. Some records return so many matches that overflow to another page would be required. Thank you for the help.

Well, one inefficiency that I can see right away is that you're accessing all the values from the external file every time through the loop, even if it's not the record you're looking for. So this minor change should speed things up a bit:

if (FusionPro.inValidation)
 Rule("OnJobStart");
var test_val = Field("Member Number");
var matches = new Array();
for (var i = 0; i <= Test_File.recordCount; i++)
{
 var ID_Variable = Test_File.GetFieldValue(i, "Member Number");
 if (test_val == ID_Variable)
 {
   var Charge_Description = Test_File.GetFieldValue(i, "Description");
   var Charge_TransDate = Test_File.GetFieldValue(i, "Transaction Date");
   var Charge_PostDate = Test_File.GetFieldValue(i, "Post Date");
   var Charge_SiteNum = Test_File.GetFieldValue(i, "Site Number");
   var Charge_InvNum = Test_File.GetFieldValue(i, "Invoice Number");
   var Charge_Charges = Test_File.GetFieldValue(i, "Charges");
   var Charge_Payment = Test_File.GetFieldValue(i, "Payments");
   matches.push(Charge_TransDate + "\t" + Charge_PostDate + "\t" + Charge_SiteNum + "\t" + Charge_InvNum + "\t" + Charge_Description + "\t" + Charge_Charges + "\t" + Charge_Payment);
 }
}
return matches.join("\n");

The other problem, as I mentioned earlier, is that for each record in your composition (in your main input data file), you have to run this whole algorithm again and refetch the data for each "Member Number" field. You could cache these results in another global object, which you would need to declare either in the JavaScript Globals or in OnJobStart like so:

CachedMemberCharges = new Object();

Then you can do something like this in your rule:

 
if (FusionPro.inValidation)
 Rule("OnJobStart");

var test_val = Field("Member Number");
if (typeof(CachedMemberCharges[test_val]) != "undefined")
{
 Print("Found cached data for member: " + test_val);
 return CachedMemberCharges[test_val];
}

var matches = new Array();
for (var i = 0; i <= Test_File.recordCount; i++)
{
 var ID_Variable = Test_File.GetFieldValue(i, "Member Number");
 if (test_val == ID_Variable)
 {
   var Charge_Description = Test_File.GetFieldValue(i, "Description");
   var Charge_TransDate = Test_File.GetFieldValue(i, "Transaction Date");
   var Charge_PostDate = Test_File.GetFieldValue(i, "Post Date");
   var Charge_SiteNum = Test_File.GetFieldValue(i, "Site Number");
   var Charge_InvNum = Test_File.GetFieldValue(i, "Invoice Number");
   var Charge_Charges = Test_File.GetFieldValue(i, "Charges");
   var Charge_Payment = Test_File.GetFieldValue(i, "Payments");
   matches.push(Charge_TransDate + "\t" + Charge_PostDate + "\t" + Charge_SiteNum + "\t" + Charge_InvNum + "\t" + Charge_Description + "\t" + Charge_Charges + "\t" + Charge_Payment);
 }
}
Print("New data for member: " + test_val);
CachedMemberCharges[test_val] = matches.join("\n");
return CachedMemberCharges[test_val];

This way, at worst you're iterating through the entire external data file only once for each unique Member Number, instead of once for each record in your main data file.

If you want to optimize beyond that, I would use an external editor (Excel would probably work) to pre-sort the external data file by member number. Then you can greatly reduce the number of iterations in the loop:

 
if (FusionPro.inValidation)
 Rule("OnJobStart");

var test_val = Field("Member Number");
if (typeof(CachedMemberCharges[test_val]) != "undefined")
{
 Print("Found cached data for member: " + test_val);
 return CachedMemberCharges[test_val];
}

var matches = new Array();
var i = Test_File.FindRecord("Member Number", test_val);
for (; i <= Test_File.recordCount; i++)
{
 var ID_Variable = Test_File.GetFieldValue(i, "Member Number");
 if (test_val != ID_Variable)
   break;

   var Charge_Description = Test_File.GetFieldValue(i, "Description");
   var Charge_TransDate = Test_File.GetFieldValue(i, "Transaction Date");
   var Charge_PostDate = Test_File.GetFieldValue(i, "Post Date");
   var Charge_SiteNum = Test_File.GetFieldValue(i, "Site Number");
   var Charge_InvNum = Test_File.GetFieldValue(i, "Invoice Number");
   var Charge_Charges = Test_File.GetFieldValue(i, "Charges");
   var Charge_Payment = Test_File.GetFieldValue(i, "Payments");
   matches.push(Charge_TransDate + "\t" + Charge_PostDate + "\t" + Charge_SiteNum + "\t" + Charge_InvNum + "\t" + Charge_Description + "\t" + Charge_Charges + "\t" + Charge_Payment);
}
Print("New data for member: " + test_val);
CachedMemberCharges[test_val] = matches.join("\n");
return CachedMemberCharges[test_val];

This has two advantages in terms of speed: One, the lookup to find the first matching line in the external file using the internal ExternalDataFileEx.FindRecord method is faster than iterating and doing all the comparisions in JavaScript, and Two, since all the lines for each member number are consecutive, it stops looking as soon as it finds one that doesn't match, instead of iterating to the end of the file each time. Again, though, this last version will only work if you pre-sort the external data file by member number.