Removing BEGIN statements from Streamfiles with Java

In this example, we have a standard M3 Streamfile (PPS307PF – M3 Put-Away document) that contains separate BEGIN statements (the keyword that signifies a new event to Streamserve) for each item received against a purchase order.  My client would like to see one document with all received items shown as separate lines.  This will save a tremendous amount of paper and allow for easier processing.  With the help of Java and Streamserve filter chains, we can remove unwanted BEGIN statements and quickly achieve a single event document without modification in M3.

In addition to Java knowledge, you’ll also need to have:

  1. Experience with the M3 Output Solution including Streamserve Control Center and Design Center (version 5.6 is used in this example).  Specifically:
    • Creating new Streamserve Applications
    • Modifying, Exporting, and Deploying Streamserve Projects
  2. Administrator access to the Streamserve server.

Now that we’re armed with some basic knowledge, let’s get started.

Step 1:  Examine the Streamfile

Here is my example Streamfile:

*SERVER	server.company.com
*PORT	 22110
*NAME	PPS307PF
*USER	USER
*JOBID	135979716657926744PPS307PF
*MAIL	YES
*TOMAIL1	mnjones@ciber.com
*FRMAIL1	m3apps@hertz.com
*ARCHIVE1	0
*COUNTRY1	US
*ReportFiletype1	PDF
*LANGUAGE	GB
*ZDCONO	111
*ZDDIVI	AAA
*ZDFACI	FAC
*ZDWHLO	DS5
BEGINPPS3070H
*TIME	20160317153921
*LANGUAGE	GB
*ZZCONO	0
*ZZDIVI
*ZZFACI	FAC
*ZZWHLO	WHS
*LAYOUT	LETTER
0HINDCTR	000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000010000000
0HDSUSS	USER
0HMWLNCD	GB
0HPADAT	03-17-16
0HPATIM	15:39:21
0HWDROW3	COMPANY US (111/AAA)
0HZDROW2	M3 13.2 DEV

1MINDCTR	000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000010000000
1MIABUYE	USER
1MIBECVE
1MIBIRCV
1MIBITNO	100000099
1MIBPITD
1MIBPLPN	0000000
1MIBPNLI	010
1MIBPNLS	000
1MIBPRCS
1MIBPROD
1MIBPUNO	8888888
1MIBPURC	USER
1MIBPUUN	EA
1MIBRORC	0
1MIBRORL
1MIBRORN
1MIBSITE
1MIBSUFI
1MIBSUNO	99999999
1MIBWHLO	WHS
1MICREPN	   2111030001
1MMLBANO	7777777
1MMLBREF
1MMLBRE2
1MMMHAC1
1MMMHAC2
1MMMHAC3
1MMMITDS	Excavator
1MMMPPUN
1MMMSLDY
1MMMUNMS
1MMUCOFA
1MMWWHNM	Main Warehouse
1MWTBUYE	Patty Purchaser
1MWTHAC1
1MWTHAC2
1MWTHAC3
1MWTPROD
1MWTPURC	Patty Purchaser
1MWWCAWE
1MWWFUDS	Excavator
1MWWPQTY
1MWWRPQA	         1
1MWWRPQT
1MWWSUNM	Excavator Supply
1MWWWHSL	SERVICE

0NINDCTR	000000000000000000000000000000000000000000000000001000000000000000001000000000000000000000000000000
0NIBGRMT	DP0
0NP7QCLV	4
0NWTGRMT	Direct Put-Away (No Doc)
0NWWNEAC	No more activity
0NWWQCQT	

CFINDCTR	000000000000000000000000000000000000000000000000001000000000000000001000000000000000000000000000000
CFWWTX60	MAKE BOBCAT

CFINDCTR	000000000000000000000000000000000000000000000000001000000000000000001000000000000000000000000000000
CFWWTX60	MODEL D12A
BEGINPPS3070H
*TIME	20160317153921
*LANGUAGE	GB
*ZZCONO	0
*ZZDIVI
*ZZFACI	FAC
*ZZWHLO	WHS
*LAYOUT	LETTER
0HINDCTR	000000000000000000000000000000000000000000000000001000000000000000001000000000000000000000010000000
0HDSUSS	USER
0HMWLNCD	GB
0HPADAT	03-17-16
0HPATIM	15:39:21
0HWDROW3	COMPANY US (111/AAA)
0HZDROW2	M3 13.2 DEV

1MINDCTR	000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000010000000
1MIABUYE	USER
1MIBECVE
1MIBIRCV
1MIBITNO	100000099
1MIBPITD
1MIBPLPN	0000000
1MIBPNLI	010
1MIBPNLS	000
1MIBPRCS
1MIBPROD
1MIBPUNO	8888888
1MIBPURC	USER
1MIBPUUN	EA
1MIBRORC	0
1MIBRORL
1MIBRORN
1MIBSITE
1MIBSUFI
1MIBSUNO	99999999
1MIBWHLO	WHS
1MICREPN	   2111030002
1MMLBANO	8191811
1MMLBREF
1MMLBRE2
1MMMHAC1
1MMMHAC2
1MMMHAC3
1MMMITDS	Excavator
1MMMPPUN
1MMMSLDY
1MMMUNMS
1MMUCOFA
1MMWWHNM	Main Warehouse
1MWTBUYE	Patty Purchaser
1MWTHAC1
1MWTHAC2
1MWTHAC3
1MWTPROD
1MWTPURC	Patty Purchaser
1MWWCAWE
1MWWFUDS	Excavator
1MWWPQTY
1MWWRPQA	         1
1MWWRPQT
1MWWSUNM	Excavator Supply
1MWWWHSL	SERVICE

0NINDCTR	000000000000000000000000000000000000000000000000001000000000000000001000000000000000000000000000000
0NIBGRMT	DP0
0NP7QCLV	4
0NWTGRMT	Direct Put-Away (No Doc)
0NWWNEAC	No more activity
0NWWQCQT	

CFINDCTR	000000000000000000000000000000000000000000000000001000000000000000001000000000000000000000000000000
CFWWTX60	MAKE BOBCAT

CFINDCTR	000000000000000000000000000000000000000000000000001000000000000000001000000000000000000000000000000
CFWWTX60	MODEL D12B

CFINDCTR	000000000000000000000000000000000000000000000000001000000000000000001000000000000000000000000000000
CFWWTX60	Air Condition:Yes

We want to process PPS307PF as one document with multiple lines as opposed to the standard document which has multiple documents (indicated by the multiple BEGINPPS3070H statements in the Streamfile) each with a single line.

There is only one type of BEGIN statement in this Streamfile, but some Streamfiles have more than one, so we’ll take that into account in the code.

Step 2:  Create Java filter to remove the unwanted BEGIN statements

In the Streamfile above we found two instances of BEGINPPS3070H.  In order to remove the second (and any subsequent appearances) while keeping the first, we’ll create a class that will remove these lines as the Streamfile gets passed into Streamserve.  We’ll also take into account that we may need to filter Streamfiles other than PPS307, so we’ll use a regular expression to find the instances of BEGIN in the file that end in ‘H’ (This way we’re excluding 0A BEGIN statements while including 0H, 1H, AH, etc.).

import java.io.*;
public class SingleEvent {
 public static void main(String[]args) {
  BufferedReader br = null;
  int counter = 0;
  String regex = "BEGIN[a-zA-Z0-9]+H$";
  try {
   String sCurrentLine;
   br = new BufferedReader(new InputStreamReader(System.in));
   while ((sCurrentLine = br.readLine()) != null) {
    if (sCurrentLine.matches(regex)) {
     counter++;
     if (counter > 1) {
      sCurrentLine = "***BEGIN STATEMENT SKIPPED***";
     }
    }
    System.out.println(sCurrentLine);
   }
  } catch (IOException e) {
   e.printStackTrace();
  }
  finally {
   try {
    if (br != null)
     br.close();
   } catch (IOException ex) {
    ex.printStackTrace();
   }
  }
 }
}

Our code creates a class called SingleEvent which reads the incoming Streamfile one line at a time and checks the current line to see if it matches our regular expression.  If so, a counter is increased and if the counter is larger than one, it replaces the current line with new text.  This text can be anything other than BEGINPPS3070H as Streamserve is looking for the BEGIN statements to create new documents.  Each (new) line is then written to the console and Streamserve picks up each of the lines before processing the Streamfile.

When we replace the BEGIN statements with alternative text (in this case replacing them with the text “***BEGIN STATEMENT SKIPPED***”).  Streamserve rejects the line, notes it in the Control Center log, and moves to the next line. Later in the process, we’ll see our skipped lines in the log.

Next, compile the code and create a SingleEvent.jar using your favorite tool.  Copy the SingleEvent.jar to an accessible folder on the Streamserve server.  Now we can begin prepping Streamserve to use Java.

Step 3:  Prep Streamserve to use Java

Because we’ll be adding a filter on both the TCP/IN port and DirScan folder, it’s best to create a new Streamserve Application specifically for our new SingleEvent Streamfiles.  We don’t want to place this filter on our base Streamserve Application and filter all Streamfiles.   Follow the standard steps to create the new application in the correct domain in Control Center.  For this example, we named our application “Dev_SingleEvent_22104” to indicate the domain, application purpose, and incoming port number.   Next, right click on the application and select Java Configuration.

beginsts_31

When the Java Configuration screen opens, we see the properties list.  Click in the Value box and select the appropriate vendor.  In my case, this was Oracle.

beginsts_32

Next, we’ll need to import our SingleEvent.jar and add the new filter chain in the Streamserve Global_Resourceset. In the Global_Resourceset, right click and select “Import…”.

beginsts_33

Navigate to the folder on the Streamserve server where you stored SingleEvent.jar.  Highlight SingleEvent.jar and click Open.

Streamserve will automatically load the correct Resource Type (Java).

beginsts_34

Click OK and your new Java resource will appear in the Global_Resourceset.

Now, let’s create our new filter chain.  The filter chain is the heart of this transaction as Streamserve will filter the Streamfile through the Java program as it comes into the TCP port or is dropped into the DirScan folder.

First, navigate to the Filter Chain folder in the Global_Resourceset, right click and select New > Filter Chain.

beginsts_35

We’ll name the filter chain “Platform – SingleEvent”.  It will be used at both the TCP/IP and Directory input connectors.

beginsts_36

Then, right click and select Start Editor.

There will be an existing Codepage Filter already in place.  We won’t make any changes here.  From the Filter Chain menu, select Add Filter > External Filter.   Type “java.exe –jar ..\data\java\SingleEvent.jar” in the Value box, save and close.

beginsts_37

Navigate to the Platform next, where we’ll copy/paste the standard TCP_MOVEX and DirScan_MOVEX input connectors and rename them as “SingleEvent_TCP_MOVEX” and “SingleEvent_DirScan_MOVEX”, respectively.  Make the necessary changes to ports and input folders for the physical platform you’re using.

Now, we’ll add the Platform – SingleEvent filter chain to each of these input connectors. In the Platform, right click on the SingleEvent_TCP_MOVEX input connector and select Settings.

beginsts_38

From the logical platform, click on the “Filter chain” button and add the new filter chain.

beginsts_39

Repeat these steps for the SingleEvent_DirScan_MOVEX connector.

Verify all the changes, save and export the project and then deploy in Control Center to your new application.

Step 4:  Test

To test, drop the Streamfile into the DirScan folder in the Management Gateway and see the Filter Chain at work in the Control Center log. Here is a snip from the log:

beginsts_41

We can see our “Unknown line (ignored):***BEGIN STATEMENT SKIPPED***” message line and each duplicate header fields that has been ignored.  We also see the “ExternFilter exitCode:0” message indicating the filter completed successfully.

Here is our new document with one event and multiple lines:

beginsts_42

Final Thoughts

We’ve successfully written a Java class that works as a filter to remove unwanted BEGIN statements from a Streamfile, setup Streamserve to use Java and our new filter, and processed a test Streamfile through the filter.  Let me know how it works for your project and any tweaks you made for it to deliver the document you want.

Special thanks to the posters at StreamShare whose excellent variety of posts on filtering and using Java with Streamserve allowed me to combine them to achieve a solution.  If you work with Streamserve, make sure to check out the forums.

Now that we’ve seen how Java works with Streamserve, next time let’s discuss using Java to connect to M3 APIs and add information to a Streamserve PageOut.

Good luck,

Neil

4 thoughts on “Removing BEGIN statements from Streamfiles with Java”

    1. Double check that you have a main method in your class and that you’ve compiled your class from the correct directory. Also verify that the class has been added to the resource set.

      Like

  1. Great article!
    Also, Really looking forward to the “using Java to connect to M3 APIs and add information to a Streamserve PageOut”.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s