Tag: Process Automation

I re-discovered a technique to synchronize users between Infor M3 and Infor Process Automation (IPA) using IPA’s web based administration tools.

It is now my preferred technique for the initial mass load of users, whereas previously in part 2, I had been using command line tools.

IPA web admin tools

The IPA web admin tools are located at http://host/UserManagement/home?csk.gen=true for the gen data area, to create the Identities, Actors, Actor-Identity, and Actor-Roles; and at http://host/lmprdlpa/LpaAdmin for the environment data area – PRD in my case – to create the Users, Tasks, and User-Tasks. It is the same as the IPA Rich Client Admin but web-based instead of Java GUI based.

To find the tools, use the shortcuts ValidationURLs.htm in your IPA installation folder:

Replaying the actions

I manually created users in the IPA web admin tools, intercepted in Fiddler the HTTP requests of each action, replayed the POST requests in Fiddler Composer, and repeatedly pruned the unnecessary data away until I identified the minimum set of values.

I then translated the result into the equivalent JavaScript code.

Result

Here below is the result. It supposes you already have an array of users which you can generate with SQL or M3 API (from MNS150 and CRS111).

Identities:

Actors:

Actor-Identities:

Actor-Roles:

Users:

Source code

I posted the source code WebAdmin.js on my GitHub repository. I will update the source code there.

Future work

• Stitch the JavaScript code together
• Create the Tasks (from MNS405)
• Create the User-Tasks (from MNS410)

Conclusion

This is yet another technique to synchronize users between M3 and IPA. I find it easier than the command line tools I was using previously. Follow the progress on the GitHub repo.

Related articles

• Part 1, overview of user synchronization between M3 and IPA

That’s it! Please like, comment, subscribe, share, and author with us.

Now I will do the incremental backup of M3 users and roles.

Refer to part 1 and part 2 for the background.

Design decisions: Pull or Push?

We can either make IPA pull the data from M3, or make M3 push the data to IPA.

Pull

We can make a process flow with SQL queries that pull data from M3, creates or updates the respective record in IPA, scheduled to run at a certain frequency, e.g. once an hour.

It is the simplest strategy to implement. But it is not efficient because it re-processes so frequently thousands of records that have already been processed. And it does not reflect the changes of M3 sooner than the scheduled frequency. And each iteration causes unnecessary noise in the IPA logs and database, so the less iterations the better. And IPA is already slow and inefficient for some reason, so the less work in IPA the faster it is. And after an undetermined amount of work, IPA becomes unstable and stops responding after which we have to restart it, so the less work in IPA the more stable it is.

Push

We can also make a series of Event Hub subscriptions and a process flow with Landmark activity nodes. It will reflect the changes in IPA virtually immediately, faster. And it will operate only on the record being affected, not all, which is efficient.

Mix

I will use a mix: I used the pull strategy for the initial mass load (see part 2), and here below I will use the push strategy for the incremental backup.

Process flow

Here is my process flow that does the incremental backup:

• The top section handles the M3 user (MNS150, CMNUSR) and the respective IPA identity, actor, actor-identity, actor-role, in the gen data area, and the user in the environment data area (e.g. DEV, TST).
• The second section handles the M3 email address (CRS111, CEMAIL, EMTP=4) and the respective IPA actor email address.
• The third section handles the M3 role (MNS405, CMNROL) and the respective IPA task.
• The bottom section handles the M3 user-roles (MNS410, CMNRUS) and the respective IPA user-tasks.
• Each section handles the M3 operations: create, update, and delete.
• I merged everything in one big flow, but we could split it in individual flows too.
• I upload the process with logs disabled to avoid polluting the logs and database.

Source code

You can download the process flow source code on my GitHub here.

Event Hub subscriptions

I created the corresponding Event Hub subscriptions in the IPA Channels Administrator:

• M3:CMNUSR:CUD
• M3:CEMAIL:CUD
• M3:CMNROL:CUD
• M3:CMNRUS:CUD

Result

Here are the resulting WorkUnits:

Repeat per environment

Deploy the process flow, and setup the event hub subscriptions on each environment data area (e.g. DEV, TST).

IMPORTANT

Refer to the challenges section in part 1 for the limitations, notably the data model dissonance which will cause collisions, the out-of-order execution which will cause inconsistencies, and the constraint mismatch which will cause failures.

Future work

• Prevent deleting the administrative users M3SRVADM and lawson.
• Recursively delete dependencies (e.g. in IPA we cannot delete a user that has pending WorkUnits)

Conclusion

This three-part blog post was a complete solution to do unidirectional synchronization of users between M3 and IPA, by means of an initial mass load via the command line, and an incremental backup via a process flow and Event Hub subscriptions. Unfortunately, Infor does not have complete documentation about this, there are serious shortcomings, and IPA is defunct anyway. Also, check out the valuable comments of Alain Tallieu whom takes user synchronization to the next level.

Related articles

• Part 1, overview of user synchronization between M3 and IPA
• Part 2, initial mass load
• Part 3, incremental backup
• Comments by Alain Tallieu where he shares his experience and valuable tips
• Event Hub for Infor Process Automation (IPA)

That’s it!

Please like, comment, follow, share, and come author with us.

Now I will do the initial mass load of users.

As a reminder, we create the identities and actors in the gen data area, and the users and tasks in the environment data area (e.g. DEV, TST). For more information, refer to part 1.

Design decisions: Command line or process flow?

We could use either the command line or the Landmark activity node in a process flow. I will explore the former now, and the latter next time. Note the command line is not available in Infor CloudSuite.

1. Identities and actors

I will generate a file of identities and actors, reading from M3, and I will use the secadm command to import the file to the gen data area in IPA.

1.1. Documentation

There is some documentation for the secadm command at Infor Landmark Technology Administration Guides > Infor Landmark Technology User Setup and Security > Landmark for Administrators > Using the Administrative Tools > The Security Administration Utility (secadm):

1.2. Extract the data

Extract all the users (MNS150) and email addresses (CRS111) from M3, and save them to a file somewhere (e.g. semi-colon separated users.csv):

SELECT DISTINCT JUUSID, JUTX40, CBEMAL
FROM MVXJDTA.CMNUSR U
LEFT OUTER JOIN MVXJDTA.CEMAIL E
ON U.JUUSID=E.CBEMKY AND E.CBEMTP='04'

Note: If you already know the subset, you can filter the list to only the users that will participate in approval flows, and discard the rest.

PROBLEM: M3 is environment specific (e.g. DEV, TST), but the gen data area is not. And M3 is company (CONO) specific, whereas IPA is not. So we will have collisions and omissions.

1.3. Transform

Transform the list of users to a list of secadm commands, where for each user, we have the commands to create the identity, actor, actor-identity, and actor-role, e.g.:

identity add SSOPV2 USER123 --password null
actor add USER123 --firstname Thibaud --lastname "Lopez Schneider" --ContactInfo.EmailAddress thibaud.lopez.schneider@example.com
actor assign USER123 SSOPV2 USER123
role assign USER123 InbasketUser_ST

Not all attribute keys are documented, but you can find them all here:

For the transformation you can use the following DOS batch file (e.g. users.bat):

@echo off
for /f "tokens=1-3 delims=;" %%a in (users.csv) do (
    echo identity add SSOPV2 %%a --password null
    for /f "usebackq tokens=1,* delims= " %%j in ('%%b') do (
        echo actor add %%a --firstname %%j --lastname "%%k" --ContactInfo.EmailAddress %%c
    )
    echo actor assign %%a SSOPV2 %%a
    echo role assign %%a InbasketUser_ST
)

Note 1: Replace delims with the delimiter of your file (e.g. semi-colon in my case).

Note 2: The command will naively split the name TX40 in two, where the first word is the first name and the rest is the last name; this will be an incorrect split in many cultures.

Save the result to a text file (e.g. users.txt):

users.bat > users.txt

We now have a list of commands ready to be executed:

1.4. secadm

Execute the secadm command to import the file to the gen data area:

cd D:\Infor\LMTST\
enter.cmd
secadm -f users.txt -d gen

1.5. Result

Here is the resulting list of identities, actors, actor-identity, and actor-role, in the gen data area:

1.6. Repeat per environment

Repeat from step 1.2 for the next environment (e.g. DEV, TST). Due to the data model dissonance between M3 and IPA, there will be collisions; see the challenges section in part 1.

1.7. Delete

To delete the records, proceed in reverse with the remove and delete sub-commands and the –complete argument. Be careful to not delete the administrative users M3SRVADM and lawson.

@echo off
for /f "tokens=1-3 delims=;" %%a in (users.csv) do (
    echo role remove %%a InbasketUser_ST
    echo actor remove %%a SSOPV2 %%a
    echo actor delete %%a --complete
    echo identity delete SSOPV2 %%a
)

1.8. Update

I could not find a command to update the Actor; for future work; meanwhile, delete and re-add.

1.9. More

Here is some more help for the secadm command:

D:\Infor\LMTST>enter.cmd

D:\Infor\LMTST>secadm
Usage: Utility for security administration.
Syntax: secadm [secadm-options] command [command-options]
where secadm-options are global secadm options
(specify --secadm-options for a list of secadm options)
where command is a secadm command
(specify --help-commands for a list of commands
where command-options depend on the specific command
(specify -H followed by a command name for command-specific help)
Specify --help to receive this message
FAILED.

D:\Infor\LMTST>secadm --secadm-options
-c Continue on error
-d dataarea
-? Print help meesage
-i Enter interactive shell mode
-H <command> Command-specific help
-f <filename> File to use as for commands
-r Recover Secadm Password
-q Run quietly
--secadm-options For a list of secadm options
-s Run silently
--help-commands For a list of commands
-m Enter interactive menu mode
-p Password for secadm
--help Print this message
-v Print version information
[-p >password>] -u Upgrade AuthenDat
FAILED.

D:\Infor\LMTST>secadm --help-commands

Valid sub-commands are:
accountlockoutpolicy Maintain system account lockout policies.
actor Maintain system actors
httpendpoint Maintain system HTTP endpoints and HTTP endpoint assignments.
identity Maintain system identities.
load Load data from a file.
provision Provision Lawson users
loginscheme Maintain system login schemes.
migrate Migrate supplier identities from default primary SSO service to domain primary SSO service
passwordresetpolicy Maintain system password reset policies.
role Maintain system roles
secanswer Maintain system security answers.
secquestion Maintain system security questions.
service Maintain system services.
ssoconfig Maintain Single Sign On Configuration
ssodomain Maintain system domain.
security Assign security classes to roles and control Security activation
admin Lawson Security Admin Configuration
passwordpolicy Maintain system password policies.
generate Secadm script generation from data
agent Migrate system agents and actors
principalresolver Maintain custom Principal Resolver code.
report Security Data Reports
mitrustsetup Set up trusted connections for an MI socket service.
keys Key Management
SSOCertificate Manage Federated Server Certificates
wsfederation Manage WS Federation Settings
proxy Proxy
class SecurityClass
FAILED.

D:\Infor\LMTST>secadm -H identity
identity Maintain system identities.

Valid sub-commands are:
privileged Maintain privileged identities.
add Add identity to the system.
update Update identity in the system.
delete Delete identity from the system.
display Display identity in the system.
pwdResetByIdentity Password reset by identity in the system.
pwdResetByService Password reset by service in the system.
listIdentities List all identities in the system.
listBadPasswords List identities with bad passwords by service in the system
overrideBadPasswords Override password for identities with bad password by service in the system
DONE.

D:\Infor\LMTST>secadm -H actor
actor Maintain system actors

Valid sub-commands are:
add Add actor to the system.
delete Delete actor from the system. !This option is temporarily unavailable
assign Assign Identity to an actor.
remove Remove Identity from an actor.
accountenable Enable actor account in the system.
accountdisable Disable actor account in the system.
enablerunas Enable Run As for Actor in the system.
disablerunas Disable Run As for Actor in the system.
actorenable Enable actor in the system.
actordisable Disable actor in the system.
context Actor context maintenance
ctxproperty Context property maintenance
list List all actors in the system.
link Actor to Agent link maintenance
DONE.

D:\Infor\LMTST>

2. Users and tasks

Now, for each M3 environment (e.g. DEV, TST), I will generate a file of users and tasks, and I will call the importPFIdata command to import the file to the respective data area.

2.1. Documentation

There is some documentation for the importPFIdata command at Infor Landmark Technology Installation Guides > Infor Lawson System Foundation Using Infor Process Automation Configuration Guide > Post-Installation Procedures > Run migration scripts:

2.2. Extract the data

For each environment (e.g. DEV, TST), extract all the roles (MNS405) and user-roles (MNS410) from M3, and save them to files somewhere (e.g. roles.csv and user-roles.csv):

SELECT KRROLL, KRTX40 FROM MVXJDTA.CMNROL
SELECT KUUSID, KUROLL FROM MVXJDTA.CMNRUS

Note: If you already know the subset, you can filter the list to only the users and roles that will participate in approval flows, and discard the rest.

2.3. Transform

Transform the list of roles and user-roles to the XML syntax of importPFIdata, e.g.:

<?xml version="1.0" encoding="UTF-8"?>
<ImpExpData Version="1">
    <Tables>
        <Table Name="WFTASK">
            <Rows>
                <Row>
                    <Column Name="TASK"><Value>FLEET_MGR</Value></Column>
                    <Column Name="WF-DESCRIPTION"><Value>Fleet manager</Value></Column>
                </Row>
            </Rows>
        </Table>
    </Tables>
</ImpExpData>

Not all table names and columns are documented, but you can find them all here:

For the transformation you can use the following DOS batch file (e.g. user-roles.bat):

@echo off
echo ^<?xml version="1.0" encoding="UTF-8"?^>
echo ^<ImpExpData Version="1"^>
echo ^<Tables^>
echo ^<Table Name="WFUSRPROFL"^>
echo ^<Rows^>
for /f "tokens=1,* delims=;" %%a in (users.csv) do (
echo ^<Row^>^<Column Name="WF-RM-ID"^>^<Value^>%%a^</Value^>^</Column^>^</Row^>
)
echo ^</Rows^>
echo ^</Table^>
echo ^<Table Name="WFTASK"^>
echo ^<Rows^>
for /f "tokens=1-2 delims=;" %%a in (roles.csv) do (
echo ^<Row^>^<Column Name="TASK"^>^<Value^>%%a^</Value^>^</Column^>^<Column Name="WF-DESCRIPTION"^>^<Value^>%%b^</Value^>^</Column^>^</Row^>
)
echo ^</Rows^>
echo ^</Table^>
echo ^<Table Name="WFUSERTASK"^>
echo ^<Rows^>
for /f "tokens=1-2 delims=;" %%a in (user-roles.csv) do (
echo ^<Row^>^<Column Name="WF-RM-ID"^>^<Value^>%%a^</Value^>^</Column^>^<Column Name="TASK"^>^<Value^>%%b^</Value^>^</Column^>^<Column Name="START-DATE"^>^<Value^>00000000^</Value^>^</Column^>^<Column Name="STOP-DATE"^>^<Value^>00000000^</Value^>^</Column^>^</Row^>
)
echo ^</Rows^>
echo ^</Table^>
echo ^</Tables^>
echo ^</ImpExpData^>

Save the result to an XML file (e.g. user-roles.xml):

user-roles.bat > user-roles.xml

We now have an XML file ready to be imported:

 

2.4. importPFIdata

Execute the importPFIdata command to import the file to the specified data area (e.g. lmdevipa):

cd D:\Infor\LMTST\
enter.cmd
env\bin\importPFIdata.bat lmdevipa -f user-roles.xml

2.5. Result

Here is the resulting list of users, tasks, and user-tasks, in the specified data area:

2.6. Repeat per environment

Repeat from step 2.2 for the next environment (e.g. DEV, TST).

2.7. Delete

I do not yet know how to delete via the command line; for future work.

2.8. Update

The importPFIdata command will automatically update the record if it already exists.

Source code

I made a unified PowerShell script m3users.ps1 that I put on my GitHub.

Conclusion

That was the initial mass load of users from M3 to IPA using the command lines secadm for identities and actors in the gen data area, and importPFIdata for users and tasks in each environment data area (e.g. DEV, TST).

See also

See part 1 for the overview of user synchronization between M3 and IPA.

And read the comments by Alain Tallieu where he shares his experience and valuable tips.

Future work

Here is some future work:

• What to do about the lack of environment and CONO in IPA
• How to update actors
• How to delete users and tasks
• Prevent deleting the administrative users M3SRVADM and lawson.
• Finish the PowerShell script

To be continued…

I will continue in part 3 for the incremental backup.

I return to Infor Process Automation (IPA) 😱 to develop approval flows for purchase orders in Infor M3 and to setup the users and roles that will take action in the Infor Smart Office (ISO) Inbasket.

Note: The IPA product is dead and Infor is replacing it with ION, so my endeavor is obsolete; furthermore, the integration of IPA and ION with M3 is flawed by design at many levels, ergo working on it is flawed too; but my customer still uses IPA for M3, so I return.

UserAction activity node

The UserAction activity node defines who can take what actions:

Inbasket

The Inbasket displays the work to the respective users so they take the action (e.g. approve/reject in this purchase order to branch managers):

Email actions

Email addresses are needed to be able to allow actions via emails (e.g. Accept/Reject in this sale approval):

PROBLEM 1

If the user is not setup in IPA, they will get the error “Inbasket Error, Unable to populate Inbox for the LPA server”:

To avoid the error, we have to setup the user in IPA, even if the user does not need the Inbasket.

There are some workarounds: 1) setup two Smart Office Installation Points, with or without the Infor.PF10 feature, 2) setup two Smart Office profiles, with or without the PF configuration enabled, 3) setup the Mango.Core application settings to enable roaming profiles with or without the Process Server inbasket, 4) maybe use the Category File Administration for profile.xml with or without the Inbasket and Connect Roles and Users, etc. Each workaround has advantages and disadvantages. Nonetheless, we still have to setup at least those users that need the Inbasket.

PROBLEM 2

No users are setup in IPA upon installation, IPA is blank, and M3 and IPA do not have a built-in synchronization mechanism. Hence, for every M3 user that needs the Inbasket, we have to setup the user in IPA. If we have thousands of users, it is too much work to do manually. I need an automatic solution. Note: as a consolation, at least we do not need to manage passwords in IPA as IPA has LDAP binding.

Design

I will implement a solution to automatically synchronize users between M3 and IPA.

My design decisions are:

• Simple mirror, one-way synchronization, from M3 (primary) to IPA (replica), and ignore changes in IPA
• One-time initial mass load + incremental backup
• Per environment (e.g. DEV, TST)
• Synchronize all users to avoid problem #1 above; otherwise filter to synchronize only the users that will use the Inbasket, not the others; but then must implement one of the workarounds above

Challenges

• IPA does not have the concept of CONO, whereas M3 does; thus we will have to choose one of the CONOs and discard the rest, i.e. possible data loss
• IPA stores the email addresses in gen for all data areas, whereas M3 stores the email addresses per environment (e.g. DEV, TST); thus we will have to choose one of the email addresses and discard the others, i.e. possible data loss
• IPA stores the user’s name in two fields firstname and lastname, whereas M3 stores it in a single field TX40; thus we will have to split the field in two, and make an assumption of where to split for names with more than two words, i.e. possible incorrect split
• IPA is slower than M3, and there are no concurrency guarantees; thus there could be race conditions where two consecutive changes in M3 are executed in the wrong order by IPA, i.e. possible data inconsistency
• There is a constraint mismatch between M3 and IPA; e.g. let’s take the case of M3 users and roles (that’s users and tasks in IPA), and let’s suppose a role has one or more users connected to it (one-to-many relationship): in M3 we can delete the role, and M3 will automatically and recursively delete the user-roles associations; whereas if we try to delete the corresponding task in IPA, it will throw an error that associations exist, so we would have to recursively delete the associations ourselves. I just tested that; I have not tested the rest: identities, actors, etc.
• M3 is encoded in UTF-8, whereas IPA is encoded in ISO-8859-1, i.e. possible data loss

Documentation

There is some documentation about the setup, but it is not everything we need:

Setting up users for Infor Process Automation and Infor Smart Office:

Mass-loading actors for Infor Process Automation: Overview:

User management in M3

Here are the programs, tables, and fields for M3:

Users (MNS150, CMNUSR), CONO, DIVI, USID, TX40:

Roles (MNS405, CMNROL), ROLL, TX40:

User-Roles (MNS410, CMNRUS), USID, ROLL:

Email address (CRS111, CEMAIL), CONO, EMTP, EMKY, EMAL:

Everything is stored in the M3 database in the respective tables:

MetaData Publisher (MDP) has information about the tables, columns, and keys:

User management in IPA

Here are the programs involved for IPA:

In the general data area:

Identity (with Service SSOPV2):

Actor (FirstName, LastName, and Email address):

Actor-Identity (one-to-one):

Actor-Roles (at least InbasketUser_ST):

In each data area (e.g. DEV, TST):

Users (one-to-one with Actor):

Tasks (equivalent of M3 Roles):

User-Tasks:

The data is stored in the IPA database in the respective tables (gen: IDENTITY, ACTOR, IDENTITYACTOR, ACTORROLE; DEV/TST: PFIUSERPROFILE, PFITASK, PFIUSERTASK):

To be continued…

I will continue in part 2.

Today I will illustrate authorization hierarchies for approval flows in Infor M3.

Approval flows

A common requirement in M3 projects is to implement approval flows, for example for purchase orders; where a buyer creates a purchase order of a certain amount; where one or more approvers must review the order, and either approve it, one approver after the other, either reject it for some reason; and where the approvers are selected from a hierarchy of managers and their maximum order amount.

There are many variations of these approval flows, each being specific to the requirements of the M3 project. Here is a simple approval flow with a single approver: Approval flows get complex quickly, with many decisions to take, many levels of approval to have, many design trade-offs to consider, and all sorts of scenarios to support.

Infor Process Automation

To implement the approval flows we use Infor Process Automation (IPA). We can also use the former Lawson ProcessFlow Integrator (PFI). As for Infor ION, it is the new standard for implementing M3 approval flows, but its usage for M3 is still young, and it does not yet have as many features as IPA does.

M3 Purchase Authority – PPS235

To store the hierarchy of approvers and their maximum order amounts, we use the program M3 Purchase Authority – PPS235. It stores the user (AURE), the maximum order amount this user is authorized to approve (MPOA), and their manager who is the next level for authorization (MNGR). For orders that exceed this amount, authorization is required by a user with authorization rights for a higher amount.

Huh? I am not an expert on PPS235, but it looks like it does not enforce integrity, and it is not in normal form, and that can result in logical inconsistencies. Indeed, the hierarchy of managers and maximum order amount may contradict each other. For instance, PPS235 allowed me to set a user whose maximum order amount was higher than that of its manager, in which case routing the approval to that manager will result in a logical anomaly. Also, there is a field to set the user’s authorization level (AUTL) which results in an alternate hierarchy of approvers, and PPS235 allowed me to enter illogical values there too. That results in several possible hierarchies: either based on the managers, either based on the maximum order amounts, either based on the authorization levels, all possibly contradicting each other. Also, PPS235 erroneously allows cycles in the hierarchy. For instance, it allowed me to set a user to be the manager of its manager. This will cause an infinite loop in the graph traversal. Also, users can be unreachable if there does not exist a connection to that user in the hierarchy. There is probably a logical explanation for these design decisions. Meanwhile, you must ensure the integrity of your data before proceeding.

MPAUTD

The data of PPS235 is stored in table MPAUTD – Authorization distribution as an adjacency list of users and their managers, for example:

AURE	MNGR
Marie	Eric
Keith	Daniel
Eric	Daniel
Charles	Daniel
John	Daniel
Daniel	Joe
Joe	Jeff

The resulting tree looks something like this (I use Graphviz to visualize the hierarchy and ensure it is correct):

To retrieve the hierarchy of managers for a certain user we traverse the adjacency list recursively using SQL’s common table expressions (CTE), for example for user Marie and for a purchase order in company 100:

WITH CTE AS (
SELECT ATCONO, ATAURE, ATMPOA, ATMNGR FROM MPAUTD WHERE ATCONO=100 AND ATAURE='Marie'
UNION
SELECT M.ATCONO, M.ATAURE, M.ATMPOA, M.ATMNGR FROM MPAUTD M JOIN CTE C ON M.ATCONO=C.ATCONO AND M.ATAURE=C.ATMNGR
)
SELECT ATAURE FROM CTE
ORDER BY ATMPOA

That results in the hierarchy of approvers and their maximum order amount starting at the specified buyer:

You can now create a loop of UserAction activity nodes in IPA to iterate thru that hierarchy. Here is an excerpt (you will need to add the SQL activity node and everything else that may be needed; you can also use the new ForEach activity node instead of my loop with an if-then-else Branch):

Conclusion

That was a quick illustration of authorization hierarchies for approval flows for purchase orders in M3, more specifically how to store the hierarchy of approvers in PPS235, how to recursively query it from MPAUTD, and how to do a loop of UserAction activity nodes.

(I meant to write about this many years ago. I finally got around to doing it after I learned how to do the recursive SQL portion for a customer last week. I hope it helps you.)