Mashup quality control #4

In the previous post about my Mashup quality control tool for Infor Smart Office Mashups, I used the Smart Office API to list the Mashups and their XAML files; today I will load the XAML files and run my predicate rules.

GetStream

Mashup files are packed with ZIP compression. The Smart Office API has methods to unpack the XAML files with System.IO.Packaging, and to get their stream of bytes so we can load them with System.Xml.XmlDocument:
1

The Smart Office Mashup SDK documentation does not have much useful information, but I show it anyway:
2

Source code

Here is the source code to run a predicate rule on each XAML file of each Mashup; for illustration purposes, this example will list the <m3:ListPanel> controls that are missing the property IsListHeaderVisible, but you can replace this rule with whatever rule you wish:

 import System;
 import System.IO;
 import System.Xml;
 import Mango.UI.Services.Mashup;
 import Mango.UI.Services.Mashup.Internal;
 
 package MForms.JScript {
     class Test {
         public function Init(element: Object, args: Object, controller : Object, debug : Object) {
             var mashups /*IList<FileInfo>*/ = PackageHelper.GetSharedMashupList();
             for (var mashup: FileInfo in mashups) {
                 debug.WriteLine(mashup.Name);
                 var baseUri: Uri = UriHelper.CreateBaseUri(new Uri(mashup.Name, UriKind.RelativeOrAbsolute));
                 var manifest: Manifest = PackageHelper.GetManifest(mashup);
                 var list /*IList<FileInformation>*/ = manifest.CreateFileInformationList();
                 for (var information: FileInformation in list) {
                     if (information.MimeType == Defines.MimeTypeXAML) {
                         var relativeUri: String = information.Path;
                         var stream: Stream = PackageHelper.GetStream(baseUri, new Uri(relativeUri, UriKind.Relative));
                         var document: XmlDocument = new XmlDocument();
                         document.Load(stream);
                         var nsmanager: XmlNamespaceManager = new XmlNamespaceManager(document.NameTable);
                         nsmanager.AddNamespace("m3", "clr-namespace:MForms.Mashup;assembly=MForms");
                         var nodes: XmlNodeList = document.SelectNodes("//m3:ListPanel[not(@IsListHeaderVisible=\"True\")]", nsmanager);
                         if (nodes.Count == 0) {
                             debug.WriteLine("\t" + relativeUri + " PASSED");
                         } else {
                             debug.WriteLine("\t" + relativeUri);
                             for (var e: XmlElement in nodes) {
                                 debug.WriteLine("\t\t" + e.Attributes["Name"].Value + " FAILED");
                             }
                         }
                         stream.Close();
                     }
                 }
             }
         }
     }
 } 

It was inspired by:
Mango.UI.Services.Mashup.Internal.MashupApplication.InitMashups
Mango.UI.Services.Mashup.Internal.PackageHelper.GetStream
Mango.UI.Services.Mashup.MashupInstance.CreateInstance.

Note: For the namespace, I should use the fields NamespacePrefix and NamespaceUri, or even better the field NamespaceManager, from MForms.Mashup.MashupUtil, but they are internal; I may use Reflection next time.

Result

In my case, the result is the list of <m3:ListPanel> controls and whether they PASSED or FAILED the predicate rule:

For the metrics:

  • Number of Mashups: 13 (7 FAILED, 54% error)
  • Number of XAML files: 178 (40 FAILED, 29% error)
  • Number of <m3:ListPanel>: 347 (115 FAILED, 33% error)

In other words, with this tool I was able to quickly scan about 200 files, covering over half of the Mashups, and find errors in a third of the <m3:ListPanel>. I would not have been able to do the same manually so fast with such accuracy.

Future work

There is more work to be done. Next time I will explore Mango.UI.Services.Mashup.Internal.MashupNameScope.CreateInstance. Stay tuned.

That’s it.

Please comment, like, subscribe, share, author. Thanks for your support.

Related posts

Mashup quality control #3

I am attempting to develop a quality control tool for Infor Smart Office Mashups. In my original post I was using Python. This time I will use the Smart Office API to get the list of Mashups.

List of Mashups in Smart Office

Smart Office automatically installs the Mashups on each user’s computer:
3

The list is based on what is installed in LifeCycle Manager:
4

Smart Office uses the Mango web services CategoryFilesManager.ListAllCategoryFileInfo and ListCategoryFileInfo to get the list of Mashups:
6

And it stores the files locally in a temporary user based path:
5

The administrator can fine tune the Mashups access:
7

That is more than sufficient information for my needs. Let’s move on.

Smart Office API

The Smart Office API has a class PackageHelper for Mashups:
1

I could not find it in the Smart Office Mashup SDK Documentation, but here is a screenshot of the documentation anyway:
1_

Source code

Here is the source code to get the list of Mashups, the Manifest, the XAML files, and the resources:

import System.IO;
import System.Xml;
import Mango.UI.Services.Mashup.Internal;

package MForms.JScript {
  class Test {
    public function Init(element: Object, args: Object, controller : Object, debug : Object) {
      var mashups /*IList<FileInfo>*/ = PackageHelper.GetSharedMashupList();
      for (var i: int = 0; i < mashups.Count; i++) {
        var mashup: FileInfo = mashups[i];
        debug.WriteLine(mashup.ToString());
        var manifest: Manifest = PackageHelper.GetManifest(mashup);
        var xml: XmlDocument = manifest.Document;
        var files: XmlNodeList = xml.SelectNodes("/Manifest/Files/File");
        for (var j: int = 0; j < files.Count; j++) {
          var node: XmlElement = files[j];
          var relativeUri: String = node.GetAttribute("Path");
          debug.WriteLine(relativeUri);
        }
        debug.WriteLine("");
      }
    }
  }
} 

It was inspired by PackageHelper.GetSharedMashupList and Manifest.CreateFileInformationList.

In addition to GetSharedMashupList(), we can also use GetPrivateMashupList()GetLocalMashupList() to get the various Mashups origins: shared, private, and local.

Reminder: To deploy a shared Mashup, use LifeCycle Manager > Admin > Upload Products > Upload; to deploy a Mashup privately, use Mashup Designer > Deploy > Private; and to deploy a Mashup locally, use Smart Office > Show > My Local Applications > Install.

Result

Here is the resulting list of files: *.mashup, *.xaml, *.png, etc.:
2

Future work

The next steps are:

  • Load the XAML and run my predicate rules
  • Deal with the name scoping across multiple files, e.g. <mashup:Event SourceName=”CustomerList”> where CustomerList is in a separate file
  • Instead of simple static analysis of the XAML files, do dynamic analysis as well, i.e. when the Mashups are running in Smart Office.
  • Explore the UriHelper.CreateMashupUri, Package.Open, Package.GetParts, PackageHelper.GetStream, MashupInstance.CreateInstance, MashupInstance.Load, MashupNameScope, etc.

That’s it.

Please comment, like, subscribe, share, author. Thanks for your support.

Related posts

Missing columns? Restore columns

Somehow I am doing a lot of maintenance and troubleshooting of Infor Smart Office Mashups these days.

Missing columns

Today I had the unusual case of a user that was missing many columns in a list. The list is an <m3:ListPanel> of STS300/B1 in a Mashup. The problem did not occur when running STS300 directly, it did not occur for other users, and it did not occur with his userid on another computer. The problem occurred only with the specific combination of his userid, on his computer, on that M3 program, in that Mashup. Good luck troubleshooting that.

Here is an illustration of missing columns, shown here simply with CRS046/B:

Troubleshooting in vain

We wasted time troubleshooting in vain: the XAML source code, the sorting order, the view, personal views. the filters, the personalizations, the Bookmark in MTS043, the M3 program Java source code, the interactive subsystem, and we uninstalled/re-installed Smart Office, without progress.

Duh, restore the columns

Then, the user selected right-click > Restore Columns in the list, and that fixed the problem [FACEPALM] It is one of those little things that are easy to forget. We do not know how 32 columns accidentally went missing in the first place, it is not something one does by mistake without remembering. That is still a mystery.

Anyway, here is the reminder for next time:

User specific data storage

Where are those settings stored you ask? They are stored in the file MFormsColumnDefinitions.xml in the user specific data storage path, and are persisted to disk when the user logs out of Smart Office:
b5

And those settings are loaded and saved in MForms.List.ListColumnManager:
b6

That’s it.

Please comment, like, subscribe, share, author. Thanks for your support.

New blog! Beyond the Limits of M3

I was doing some web search about Infor M3, and I discovered this new blog that was started over a year ago, Beyond the Limits of M3, by Kasun Sandaruwan, Consultant at Brandix i3, Sri Lanka. The blog is about M3 technology: M3 Enterprise Collaborator, M3 Web Services, M3 API, LifeCycle Manager, Smart Office Scripts, etc. Congratulations Kasun!

Go check out his blog, say hi, and subscribe to it. Check out the other blogs too.

Mashup quality control #2

Today I had to troubleshoot why a refresh icon in a Smart Office Mashup was not refreshing, and after correcting it, I included the correction as a new predicate rule in my Mashup quality control tool to automatically spot similar errors in other Mashups.

IconButtons and CommandBarButtons

Here is a screenshot of the icon bar and its XAML code:

You can find more information about the Smart Office Design System’s Icons, IconButtons and CommandBarButons on norpe’s blog post.

The error

After some manual troubleshooting, I found that the icon was refreshing the wrong list, i.e. the Click event had the TargetName property set to the wrong <m3:ListPanel>, it was set to PurchaseOrderList (incorrect) instead of RequisitionList (correct). Probably the developer copy/pasted the code from another tab and forgot to change the TargetName. That’s hard to find, quick to fix.

<ds:CommandBarButton IconName="Refresh" ToolTip="{mashup:Constant Refresh, File=Mango.UI}">
    <ds:CommandBarButton.CommandParameter>
        <mashup:Event TargetName="PurchaseOrderList" TargetEventName="Refresh" />
   </ds:CommandBarButton.CommandParameter>
</ds:CommandBarButton>

Finding other errors

Assuming the icons are grouped in a <StackPanel> followed by their <m3:ListPanel>, I can quickly find similar errors with the following XPath expression that lists all the icons TargetName:

//*[name()='ds:IconButton.CommandParameter' or name()='ds:CommandBarButton.CommandParameter']/mashup:Event/@TargetName

The Python code would be:

import os
import glob
import lxml.etree as etree

for f in glob.glob(os.path.join(r'C:\BuyerPortal', '*.xaml')):
    tree = etree.parse(f)
    r = tree.xpath("//*[name()='ds:IconButton.CommandParameter' or name()='ds:CommandBarButton.CommandParameter']/mashup:Event/@TargetName", namespaces={'mashup': 'clr-namespace:Mango.UI.Services.Mashup;assembly=Mango.UI', 'ds': 'clr-namespace:Mango.DesignSystem;assembly=DesignSystem'})
    for element in r:
        print(f, element)

Result

Then I visually inspect the result for outliers. In my case, I see the error I found earlier, and a new error I did not know about:
3

This is a quick way to help identify errors before users have to.

Future work

This still requires a visual inspection of the result. A better solution would be to calculate the distance between the icon and its target m3:ListPanel in the XAML tree, where a minimum distance would indicate a low probability of error, and a maximum distance would indicate a high probability of error.

That’s it!

Please comment, like, subscribe, share, author. Thanks for your support.

Related posts

Mashup quality control #1

It has become difficult for me to manually maintain the Smart Office Mashups of my customer – there are about 50 files, 1,000 controls, and 10,000 lines of XAML – so I am developing a software verification tool that does automatic quality control for me.

How?

I defined a set of predicate rules, and I use XPath to validate the Mashup against those rules. I am using Python for now because of its expressiveness and interactivity, but I will port it to JScript.NET or C# soon to benefit from the Smart Office API.

Sample rule

As a sample rule, I want all the <m3:ListPanel> controls to have the property IsListHeaderVisible=”True” such that users have the ability to expand the list header and change the sorting order, view, and apply filters. If one of the list panels does not have that property I want to know about it and correct it. Note this is my own preference, and other developers may have the opposite preference.

Here is the property in Mashup Designer:
3

The following XPath expression will return the list panels not validating the rule:

//m3:ListPanel[not(@IsListHeaderVisible="True")]

Here is a Python code to validate that rule:

import os
import glob
import lxml.etree as etree

for f in glob.glob(os.path.join(r'C:\RentalCounterMashup', '*.xaml')):
    tree = etree.parse(f)
    r = tree.xpath('//m3:ListPanel[not(@IsListHeaderVisible="True")]', namespaces={'m3': 'clr-namespace:MForms.Mashup;assembly=MForms'})
    for element in r:
        print(f, element.attrib['Name'])

The result is the following, a list of XAML filenames and <m3:ListPanel> names that fail the rule:
4

Result

In my example, out of 63 list panels, 46 had the property, and 17 were missing the property, that’s 27% of list panels not passing the quality control. In other words, I was able to quickly identify a third of the list panels to correct.

Future work

I have many more ideas to implement, for example:

  • Ensure there are no hard-coded values in the MForms Bookmarks, Links, and MForms Automation, such as hard-coded CONO or DIVI
  • Automatically correct the Mashup, e.g. set IsListHeaderVisible=”True” if missing, and save

That’s it!

Let me know in the comments below if you have other rules to control the quality of Mashups.

Please click Like, share this post with your colleagues, click Follow to subscribe, come write the next blog post with us, and send some love to the other M3 blogs as well. This is a volunteer-based community, and your participation keeps the community alive and growing. Thank you.

Related posts

Who is the author of an M3 mod?

Today I needed to find who is the developer of an M3 customer modification. There is a bug in the MForms Bookmark of program M3 Customer. Connect Addresses – OIS002 which the developer modified for our customer’s needs, and I needed to report the bug to that developer. But I do not have the MAK training nor tool, so I cannot easily see the list of modifications and their authors. My colleague Shashank remind me of the following answer in M3 Server View.

  1. Go to M3 Server View (from the Grid Management Pages, or from LifeCycle Manager):
    0
  2. Find the interactive subsystem that is running the program (in my case OIS002), and select Tools:
    1
  3. Select Find Class:
    2
  4. Search for the M3 program (in my case OIS002):
    3
  5. The line with the customer class will give the file path, version, author, date, and unique ID (in my case I found the author, Rajesh):
  6. Note: Up to here, we can access this page anonymously, without being an authenticated user nor an administrator (security vulnerability anyone?)
  7. Now, if we have access to the M3 Business Engine file system, we can open the file itself and see the full list of developers; in my case the file is at:
    D:\Infor\M3BE\env\M3BE_15.1_TST\Fix\CUS\VFix\src\mvx\app\pgm\customer\OIS002.java:
    5

That’s it.

Let me know in the comments below if you have other tips. Click Like. Share this post with your colleagues. Click Follow to subscribe. And come write the next blog post with us. This is a volunteer-based community, and your participation keeps this blog going. And send some love to the other M3 blogs too. Thank you.

Inspect tool for Mashups

How great the Inspect tool is for developing Mashups in Infor Smart Office! I mentioned it a long time ago in another post about developing scripts, and I want to mention it here again.

I am currently maintaining a Mashup that other developers created. That Mashup has 30 XAML files, 10,000 lines of code, 500 controls, 80 tabs in three levels, etc. Any time I need to modify the Mashup, I have to follow a thread to find the relevant line of source code.

With the Inspect tool, I can point at a control in the Mashup (watch cursor) to find any of its parts, find its name, its parent tab, its XAML file, etc. That saves me valuable time.

Here are some screenshots:
1 2 3

I wish the Mashup Designer had the same watch cursor feature. Maybe it is easy to implement with the Smart Office SDK. To be explored.

There is also the Snoop tool I mentioned in the other post. Try that too.

That’s it!

Let me know in the comments below what other tools you use. Share this post. Click Like. Follow this blog. And come write the new blog post with us. This is a volunteer-based community, and your participation keeps it going. Thank you.

Default country in Mashup ComboBox

Quick illustration of how to set the user’s country as the default selection in a Mashup ComboBox in Infor Smart Office.

Suppose you have a <m3:MIComboBox>, and you want it to be a list of countries (e.g. FR-France, SE-Sweden, US-United States, etc.), populated from the M3 API CRS045MI.LstByCode, displaying CSCD and TX40, and you want the default selection to be the user’s country (e.g. US).

For that, I will use the System.Globalization.RegionInfo.CurrentRegion’s property TwoLetterISORegionName and assume that CRS045 uses the same two-letter ISO codes.

Here is the code with the relevant lines:

<StackPanel
   xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
   xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
   xmlns:mashup="clr-namespace:Mango.UI.Services.Mashup;assembly=Mango.UI"
   xmlns:glob="clr-namespace:System.Globalization;assembly=mscorlib"
   xmlns:m3="clr-namespace:MForms.Mashup;assembly=MForms">
    <TextBlock Name="MyCountry" Text="{Binding Source={x:Static glob:RegionInfo.CurrentRegion}, Path=TwoLetterISORegionName}" Visibility="Hidden" />
    <m3:MIComboBox Name="Countries" SortField="CSCD" SelectedValuePath="[CSCD]" SelectedValue="{Binding ElementName=MyCountry, Path=Text}" Width="200">
       <m3:MIComboBox.Events>
          <mashup:Events>
             <mashup:Event SourceEventName="Startup" />
          </mashup:Events>
       </m3:MIComboBox.Events>
       <m3:MIComboBox.DataSource>
          <m3:MIDataSource Program="CRS045MI" Transaction="LstByCode" OutputFields="CSCD,TX40" IsCacheable="True" />
       </m3:MIComboBox.DataSource>
       <m3:MIComboBox.ItemTemplate>
          <DataTemplate>
             <StackPanel Orientation="Horizontal">
                <TextBlock Text="{Binding Path=[CSCD]}" />
                <TextBlock Text=" - " />
                <TextBlock Text="{Binding Path=[TX40]}" />
             </StackPanel>
          </DataTemplate>
       </m3:MIComboBox.ItemTemplate>
    </m3:MIComboBox>
 </StackPanel>

Here is the result:
blog

blog_

That’s it. Please like, share, subscribe, author.

Poll: How to add 80 million item/warehouse records in M3?

My customer is doing data conversion in preparation for go live, of which 80 million item/warehouse records, they want to complete the upload over a week-end, but M3 is being a bottleneck, i.e. it cannot absorb the data as fast as they are feeding it to.

What would you do? I am not an expert at item/warehouse, nor at M3 performance, and from what I have seen in the past, each project handles it at their own sauce.

Synopsis

My customer has about 80,000 items to add to MMS001, and each item needs to be connected to about 1,000 warehouses in MMS002.

The result is about 80 million item/warehouse records to create.

M3 can handle gazillions of records. So this should be a piece of cake, right?

Failed tentative 1

Because the M3 API MMS200MI does not cover all the fields, my customer initially used M3 Web Services (MWS) of type M3 Display Program (MDP) to simulate a user entering data in all the desired fields.

It takes about 10ms per record, or 100 records per second. Loop 80 million times, and that would take about 800ks, which is about 222h, which is about 10 days to complete.

Yikes!!! 10 days to upload???

We need faster. We need one or two orders of magnitude faster.

Failed tentatives 2, 3, 4

I did A/B testing of various alternatives:

  • Use MWS MDP, versus use M3 API (I thought this would be the silver bullet, nope)
  • Create the item from scratch, versus create the item from an item template
  • Call the Add transaction, versus call the Copy transaction
  • Let M3 auto-create (we create the item oursleves but let M3 auto-create the item/warehouse), versus manually create (we create both ourselves).
  • Remote Desktop Connect inside the M3 BE server or database server to avoid any potential network issues
  • etc.

Either way, we only get a marginal, insignificant benefit in some cases, but the order of magnitude is equally bad.

Concurrency

Then, I decided to create the records concurrently (in parallel) rather than sequentially (one after the other).

With the help of some command line Kung fu I was able to automatically split my big CSV file into 80 smaller chunks, create 80 M3 Data Import (MDI) descriptions, and run 80 parallel threads that each upload one million records:
poll2

It turns out M3 can comfortably handle the throughput. It created many jobs in the API subsystem to process the records in parallel:
poll

Each job takes about 1% CPU. So that should max out the server to about 80% CPU utilization.

However, M3 seems to have capped well before the 80 jobs. I guess that is Amdahl’s law of maximum expected performance in parallel computing.

The result was about 1,100 records per second, which would complete in about 20h, less than a day. That is one order of magnitude improvement!!!

With some more benchmarking we could plot the curve of Amdahl’s performance and find the optimum number of jobs and have a pretty accurate estimate of the expected duration.

I also automatically generated a cleanup script that deletes everything, and I noticed the delete takes four times longer than the add.

Reality check

By the time I had come up with my results, the customer had completely changed strategy and decided to upload the records directly with SQL INSERT. [Cough] [Cringe] <your reaction here>. They consulted with some folks at Infor, and apparently it is OK as long as the data has been pre-validated by API or by MWS MDP, and as long as M3 is shut down to avoid collisions with the M3 SmartCache. It is important to get clearance from Infor as SQL INSERT will void your Infor Support warranty.

Future work

I have also heard of the Item Data Interface (IDI) and the Warehouse Interface (WHI). I do not know if those could help. To be explored.

Poll

What would you do? How have you uploaded millions of records at once? Leave me a comment below.

Disclaimer

Do not take these results as a reference. I was just providing some help to my customer and to the colleagues that are handling this.

I am not a reference for item/warehouse in M3, and I am not a reference for M3 performance, there are special teams dedicated to each.

Also, these results are specific to this customer at this time. Results will vary depending on many factors like CPU, memory, disk, tweaks, etc.