HTTP channels in MEC (part 5)

Continuing the guide on how to setup HTTP channels in Infor M3 Enterprise Collaborator (MEC), I will illustrate how to setup the HTTPSOut channel for MEC to make requests using HTTP Secure (HTTPS), i.e. HTTP over SSL/TLS; I will investigate the channel’s features and drawbacks, and I will verify if it is safe to use (it is not).

Why it matters

It is important to use SSL/TLS in partner agreements that need to transfer via HTTP sensitive information such as names, addresses, bank account numbers, purchase orders, credit card numbers, financial transactions, health records, user names, passwords, etc. More generally, it is important to accelerate the adoption of cryptography.

HTTPS in brief

SSL/TLS is a security protocol that provides privacy for a client and a server to communicate over insecure networks. It is a communication layer that sits between a transport layer (usually TCP) and an application layer (for example HTTP). Secure Sockets Layer (SSL) is the original protocol and is not considered secure anymore. Transport Layer Security (TLS) is the successor and everybody should upgrade to its latest version 1.2. https is the scheme token in the URI that indicates we are using a dedicated secure channel often on port 443.

Properties

SSL/TLS provides the following properties of secure communication:

  • Confidentiality, using a cipher to encrypt the plain text and prevent eavesdropping
  • Integrity, using signed hashes to detect tampering
  • Authentication, using digital certificates to affirm the identity of the entities

It can also provide non-repudiation, using digital signatures to assert the sender cannot deny having sent the message, provided the sender’s private key is not compromised or changed.

And depending on the key agreement protocol, it can also provide perfect forward secrecy, using ephemeral key exchanges for each session, to ensure that a message cannot be compromised if another message is compromised in the future.

Handshake

During the SSL/TLS handshake, the client and server exchange digital certificates and establish cipher suite settings. The connection bootstraps with asymmetric cryptography using public and private keys (which eliminates the problems of distributing shared keys), and then switches to symmetric cryptography using a temporary shared key for the session (which is several orders of magnitude faster).

Digital certificates

A digital certificate is a public key and the owner’s identity that have been verified and digitally signed by a certificate authority (CA). Public key infrastructure (PKI) is used to create, distribute, and revoke certificates. If an entity trusts a CA, then by transitive relation it will trust that any certificate the CA issues authenticates the owner of the certificate. There can be any number of intermediate CAs in the chain of certificates, and root CAs use self-signed certificates.

The client must verify the server certificate, and optionally the server may verify the client certificate:

  • Verify the chain of trust
  • Verify the hostname in the certificate using SubjectAltNames and Common Name; this could get tricky with wildcard patterns, null characters, and international character sets
  • Verify the certificate’s activation and expiration dates
  • Verify the certificate’s revocation status using a certification revocation list (CRL) or OCSP
  • Check the X.509 certificate extensions (e.g. can this key sign new certificates?)
  • Check that the certificates of the intermediate CAs have the CA bit set in the “Basic Constraints” field

Despite all that, validating certificates is difficult, and the CA model is broken.

Test

You can test HTTP over SSL/TLS and see the chain of certificates,

with cURL:

curl http://curl.haxx.se/ca/cacert.pem > cacert.pem
curl --verbose --cacert cacert.pem https://www.example.com/

with OpenSSL:

openssl s_client -connect www.example.com:443 -showcerts
GET / HTTP/1.1
Host: www.example.com:443

Proxy

A web proxy is a program that makes requests on behalf of a client. There are different types of web proxies, for example tunneling proxies, forward proxies, and reverse proxies. And there are different implementations, for example, explicit proxies and transparent proxies. And they have different purposes, for example caching, load balancing, securing, monitoring, filtering, bypassing, and anonymizing. Proxies may require authentication.

An explicit web proxy is for example when you set your browser to use a proxy at a certain host and port number. With that proxy, the client makes an HTTP request to the proxy using the CONNECT method, then the proxy establishes a TCP connection to the destination server, and then the proxy acts as a blind tunnel that is forwarding TCP between client and server, seeing only the SSL/TLS handshake and then encrypted traffic.

You can test HTTP over SSL/TLS via an explicit proxy,

with cURL and Fiddler:

curl --verbose --cacert cacert.pem --proxy 127.0.0.1:8888 https://www.example.com/

with proxy authentication:

curl --verbose --cacert cacert.pem --proxy http://username:password@proxy:port/ https://www.example.com/

with GNU Wget and Fiddler:

export https_proxy=localhost:8888
wget --debug --https-only https://www.example.com:443/

You will see this request and response between client and proxy, then the SSL/TLS handshake, then the HTTP traffic:

CONNECT www.example.com:443 HTTP/1.1
Host: www.example.com:443
HTTP/1.1 200 Connection Established

On the other hand, a transparent proxy intercepts the traffic at the lower network level without requiring any client configuration. It acts as the SSL termination for the client, and establishes a second encryption channel with the destination server thereby being able to monitor and filter the SSL/TLS traffic in transit; in that case the client is not aware of the presence of the proxy but must trust the proxy’s certificate, and the proxy must trust the server’s certificate.

For MEC

In our case, we want MEC to communicate with partners over a secure channel with confidentiality, integrity, and authentication to protect the data being exchanged, optionally via proxy.

Problem

MEC does not provide secure channels for HTTP out of the box. For instance, there are no HTTPSIn, HTTPSOut, HTTPSSyncIn, or HTTPSSyncOut channels. There is an HTTPSOut channel, but it is only a sample in an appendix of the documentation (does that mean it is safe to use?), and it does not come out of the box in the Partner Admin, it must be added manually (although it is available in the Java library). Also, there is a WebServiceSyncIn channels that uses WS-Security specifically for XML over SOAP, but in my case I am interested in HTTPS in general, for example for flat files, I am not interested in SOAP. Ideally, I would prefer to use the Infor Grid which already communicates via HTTPS, but unfortunately it does not have a native connection handler for MEC.

None of this means MEC is insecure, it just means you have to add the security yourself.

Documentation

The useful documentation

The MEC Partner Admin Tool User Guide contains the necessary source code and some explanation for the HTTPSOut channel and user interface:
doc1 doc2doc3

The obstructing documentation

However, the documentation references HTTPS Communication, HttpServer.xml, and HTTPSOut at the same time, yet it does not clearly explain the relationship between them, so it is confusing:
doc4 doc5

I had to find the Communication Plug-in Development User Guide of an old version of MEC to find the answer. It turns out the HttpServer.xml was a file that existed in old versions of MEC to configure the web user interface. I think the sample in the documentation used that server to test the HTTPSOut channel; I do not know. Anyway, that file does not exist anymore after MEC was ported to the Infor Grid, so that chapter is irrelevant, confusing, and unrelated to HTTPSOut. We can ignore it, and Infor should remove it from the Partner Admin guide.

Java classes

The MEC Java library ec-core-x.y.z.jar includes the Java classes sample.HTTPSOut and sample.HTTPSOutPanel:
4.1

Features

Looking at the source code, the HTTPSOut channel is similar to the HTTPOut channel: they both make a POST HTTP request to a destination server, at a host, port number, and path defined in the Partner Admin, with the message sent in the request body.

HTTPSOut uses javax.net.ssl.HttpsURLConnection which in turn uses javax.net.ssl.SSLSocket to establish a secure socket with the destination server. From the HTTPSOut point of view, it just reads/writes plain HTTP to the socket, and the socket underneath takes care of the handshake and encryption/decryption.

HTTPSOut has an advantage over HTTPOut. Given the entire HTTP traffic is encrypted over SSL/TLS, we can safely set a username and password for HTTP Basic authentication, unlike with HTTPOut where we should not set a username and password because they transit in clear text (trivial Base64 decode).

Drawbacks

HTTPOut and HTTPSOut share the same drawbacks: they do not use Java NIO, and there are no proxy settings.

HTTPSOut has additional drawbacks: there are no settings for Content-type, multi-part, and keep alive, unlike HTTPOut. Worse, the response is completely ignored unlike HTTPOut where the response is at least added to the debug log.

Test

You can test the HTTPSOut class with the following code and with a message.txt file:

javac -cp log4j-1.2.17.jar;ec-core-11.4.1.0.0.jar;. Test.java
java -cp log4j-1.2.17.jar;ec-core-11.4.1.0.0.jar;. Test < message.txt
import java.util.Properties;
import sample.HTTPSOut;

public class Test {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.setProperty(HTTPSOut.HOST, "www.example.com");
        props.setProperty(HTTPSOut.PORT, "443");
        props.setProperty(HTTPSOut.PATH, "/");
        props.setProperty(HTTPSOut.USER, "username");
        props.setProperty(HTTPSOut.PASSWORD, "*****");
        props.setProperty(HTTPSOut.TRUST_STORE, "C:\\somewhere\\keystore");
        props.setProperty(HTTPSOut.TRUST_STORE_PWD, "*****");
        HTTPSOut client = new HTTPSOut();
        client.send(System.in, null, props);
    }
}

HTTPSOut will send the following request over SSL/TLS:

POST /Hello HTTP/1.1
Content-Type: multipart/form-data; boundary=----------------------------afatudbu6sb5-875oo9dmn52fzfypuig0x1kv
Authorization: Basic: dXNlcm5hbWU6cGFzc3dvcmQ=
User-Agent: M3 Enterprise Collaborator v11.4.1.0
Cache-Control: no-cache
Pragma: no-cache
Host: www.example.com:443
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 256
------------------------------afatudbu6sb5-875oo9dmn52fzfypuig0x1kv
Content-Disposition: form-data; name="name"; filename="output.xml"
Content-Type: application/xml

Hello, World!
------------------------------afatudbu6sb5-875oo9dmn52fzfypuig0x1kv--

Activity diagram

Here is a sample activity diagram of the HTTPSOut channel:

diagram

Authentication bug

There is a bug in the HTTPSOut class regarding the HTTP Basic authentication header that will cause the server to return HTTP/1.1 401 Unauthorized, and that is easy to fix:

// incorrect
urlConn.setRequestProperty("Authorization: Basic", Base64.encode (to64, "ISO-8859-1"));
// correct
urlConn.setRequestProperty("Authorization", "Basic " + Base64.encode (to64, "ISO-8859-1"));

The bug is located in both the HTTPSOut source code in the Partner Admin guide, and in the HTTPSOut class in the JAR file. And the JAR file is located in both the Partner Admin tool and in the MEC server in the Infor Grid.

That code is only executed if you need HTTP Basic authentication (i.e. if you specify a username and password); you can ignore it otherwise.

Proxy

If you need to use a proxy, add the following code to the HTTPSOut class and recompile it:

import java.net.Authenticator;
import java.net.InetSocketAddress;
import java.net.PasswordAuthentication;
import java.net.Proxy;

[...]
// get the proxy properties
String proxyHost = props.getProperty(PROXY_HOST);
int proxyPort = Integer.parseInt(props.getProperty(PROXY_PORT));
String proxyUser = props.getProperty(PROXY_USER);
String proxyPassword = props.getProperty(PROXY_PASSWORD);

// connect via proxy
InetSocketAddress proxyInet = new InetSocketAddress(proxyHost, proxyPort);
Proxy proxy = new Proxy(Proxy.Type.HTTP, proxyInet);
HttpsURLConnection con = (HttpsURLConnection) url.openConnection(proxy);

// authenticate to proxy
Authenticator.setDefault(new Authenticator() {
    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication(proxyUser, proxyPassword.toCharArray());
    }
});

//con.setRequestProperty("Proxy-Authorization", "Basic " + Base64.encode(proxyUser + ":" + proxyPassword, "ISO-8859-1")); // this did not work for me

You will need to add some if-then-else so the user can choose to use a proxy or not, and add some exception handling around parseInt.

Also, for the user interface, you will need to add a group widget, a checkbox, four labels and four text boxes to the HTTPSOutPanel class and recompile it:
test___

Security holes

Unfortunately, the HTTPSOut class does not verify the certificate and as such the connection is susceptible to man-in-the-middle attack.

For instance, it does not verify the hostname of the certificate, it just returns true for any hostname:

HostnameVerifier hv = new HostnameVerifier() {
    public boolean verify(String s, SSLSession sslSession) {
        return true;
    }
};

More so, HTTPSOut does not verify the certificate’s dates, revocation, basic constraint, extensions, etc.

Thus, HTTPSOut is not safe to use out of the box. You have to implement certificate validation yourself. That is probably why it is only a sample.

Compilation

To compile the HTTPSOut and HTTPSOutPanel classes:

  1. Recover the source code (either copy/paste it from the Partner Admin guide, or decompile the classes).
  2. Change the code as needed (e.g. fix the authentication bug, add the proxy settings, add certificate validation, etc.)
  3. Compile the source code with:
    javac -cp lib\log4j-1.2.17.jar;lib\ec-core-11.4.1.0.0.jar sample\HTTPSOut.java
    javac -cp lib\x86-3.3.0-v3346.jar;lib\log4j-1.2.17.jar;lib\ec-core-11.4.1.0.0.jar;. sample\HTTPSOutPanel.java
    4.2
  4. Replace the Java classes in both the Partner Admin tool folder and in the MEC server folder of the Infor Grid, for example:
    D:\Infor\MECTOOLS\Partner Admin\classes\sample\
    D:\Infor\LifeCycle\host\grid\M3_Development\grids\M3_Development\applications\MECSRVDEV\MecServer\lib\sample\
    4.2_ 4.2__
  5. Restart the Partner Admin and restart the MEC server in the Infor Grid.
  6. The class loader will give precedence to the classes you put in the folders over the classes in the JAR file.

1. Preparation

Before we can use HTTPSOut, we have to prepare the terrain for SSL/TLS.

As a reminder, the HTTPSOut channel is an HTTP client that will make an HTTP request to a destination HTTP server over SSL/TLS, e.g. to https://www.example.com/.

1.1. Server authentication

The short instructions to setup server authentication are:

openssl s_client -connect www.example.com:443 > example.cer
keytool.exe -import -file example.cer -keystore example.ks -storepass changeit

The long instructions are the following.

We need to setup server authentication, i.e. for HTTPSOut to affirm the identity of the server it is connecting to; it will rely on SSLSocket to verify that the server certificate is signed by a certificate authority that is stored in a trusted keystore.

Internet Explorer, Google Chrome, Fiddler and other programs that use WinINet ultimately rely on the Windows certificate manager, certmgr.msc, but Firefox and Java each use their own key management. The JRE has its default keystore at %JAVA_HOME%\lib\security\cacerts that SSLSocket uses. HTTPSOut ignores the default keystore and requires the user to specify one. To create a Java keystore we use keytool.exe and import the server’s certificate in it.

The steps are:

  1. Open a browser to the destination server over HTTPS and open the certificate:
    2.2_
  2. Inspect the certificate, any intermediate certificate authorities, and the root certificate authority, up the chain, and make sure you trust them (if your browser does not trust them it will throw a big warning sign):
    2.5 2.6
  3. Now we need Java to trust that certificate. Check if one of the certificates in the chain is already in your Java keystore. Java already comes with a list of trusted certificates in its keystore. If the destination server presents a certificate that is already trusted by one of the trusted certificates of the JRE, it will automatically be trusted by the JRE, i.e. the friends of my friends are my friends. In my case, my JVM already trusts the certificate authority DigiCert, so by chain of trust it will automatically trust the certificate presented by http://www.example.com. It may be the same case for you:
    keytool.exe -list -keystore cacerts -storepass changeit | findstr /i DigiCert
    keytool.exe -list -keystore cacerts -storepass changeit -v -alias digicerthighassuranceevrootca

    2.8

  4. If you already have the server certificate, good, just keep in mind the location of that keystore, and skip the rest of these steps; otherwise continue reading.
  5. Export the certificate to a file, e.g. example.cer:
    2.6_ 2.6__
  6. Import the certificate to a keystore, e.g. example.ks:
    keytool.exe -import -file example.cer -keystore example.ks -storepass changeit

  7. Now you have a keystore ready for HTTPSOut.

1.2. Client authentication

As for client authentication, the HTTPSOut channel does not support that, i.e. it does not have a public/private keys and a digital certificate to present to the server if the server requests it, so we have to forget about that. If the server needs to authenticate the client, we can use Basic authentication (provided we fix the authentication bug).

2. How to setup

Now let’s setup the HTTPSOut channel in MEC.

The steps will be:

  1. Setup the Send protocol
  2. Setup the Send channel configuration
  3. Setup the Agreement with detection and processes

2.1. Setup the Send protocol

  1. Open the Partner Admin tool
  2. Go to Manage > Advanced
  3. Select the Send Protocols tab
  4. Click New, and set:
    Protocol: HTTPSOut
    Send class: sample.HTTPSOut
    UI class: sample.HTTPSOutPanel
    4.3
  5. Click OK
  6. Click Close

2.2. Setup the Send channel configuration

  1. Go to Manage > Communications
  2. Select the Send tab
  3. Click New:
    4.4.1
  4. Select protocol HTTPSOut and enter the information host, port number and path to the destination server, as well as the keystore where you saved the certificate and the keystore password, and optionally set user and password for HTTP Basic authentication to the destination server (provided you corrected the authentication bug):
    4.4.2
    Note: The keystore path is relative to the JVM that runs MEC server, i.e. java.exe must be able to access the file and have read permission to it. Also, given the Infor Grid is distributed make sure the keystore file is distributed accordingly to the correct server.
  5. Optionally set the proxy settings if you added that feature to the classes (see proxy section above)
  6. Click Send test message to ensure everything is correct; if you get an error message, test the HTTPSOut class as explained in the test section above, or check the troubleshooting tips in the section below
  7. Click OK
  8. Click Close

2.3. Setup the Agreement with detection and processes

  1. In Communications > Receive, setup any incoming channel such that we can simply trigger the partner agreement for the purposes of illustration; the protocol is not important right now; for instance I have a simple DiskIn channel that has status RUNNING in my Infor Grid, and I have a partner agreement with it in a channel detection.
  2. Go the agreement > Processes, right-click select Send, and select the HTTPSOut channel:
    5.1 5.2
  3. Click OK
  4. Click Save
  5. The HTTPSOut channel is now ready to use

3. How to test

To test the HTTPSOut channel, follow the same instructions as in the test of Part 4.

4. …on the Internet

If you are doing the HTTPS requests to a server out on the Internet you should consult with your security team. I have MEC in a LAN behind a NAT, firewall, content based filter, security appliance, and transparent proxy.

5. Troubleshooting

In case HTTPSOut does not work as expected, here are some troubleshooting tips.

Check the logs

The MEC server logs are useful for troubleshooting problems. For example, in one case I got the following exception:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

error1

That means the server certificate validation failed, either because we put the incorrect server certificate in the keystore that we defined in PartnerAdmin, either because a man-in-the-middle server presented a different certificate.

Test SSL/TLS connection

Some network and security administrators setup firewalls, proxies, and gateways that interfere with SSL/TLS. In my case, I will ensure that my host can access the destination server over SSL/TLS. For that:

  1. Locate on which node is MEC server running:
    2.0.2_
  2. From that server (for example via Remote Desktop Connection), make a test connection to the destination server over SSL/TLS, for example use OpenSSL s_client:
    openssl s_client -connect www.example.com:443 -showcerts
  3. Make sure the server certificate that is presented is the correct one.

Test HttpsURLConnection

  1. Check what JVM the MEC server is using:
    2.0.3_
  2. Use that JVM to test a HttpsURLConnection; that will check if that particular JVM of that host has access to read that keystore, it will check if it can make a SSL/TLS connection to that destination server, and it will show the server certificates received:
    // javac Test.java && java -cp . Test
    
    import java.io.OutputStream;
    import java.net.URL;
    import java.security.cert.Certificate;
    import javax.net.ssl.HttpsURLConnection;
    
    public class Test {
        public static void main(String[] args) throws Exception {
            System.setProperty("javax.net.ssl.trustStore", "D:\\java\\jre\\lib\\security\\cacerts");
            System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
            URL url = new URL("https://www.example.com/");
            HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
            con.setDoOutput(true);
            OutputStream out = con.getOutputStream();
            Certificate[] scerts = con.getServerCertificates();
            for (Certificate scert: scerts) {
                System.out.println(scert.toString());
            }
            con.disconnect();
        }
    }
    

Test HTTPSOut

See the section at the top to test the HTTPSOut class.

Conclusion

That was an illustration of how to setup the HTTPSOut channel in MEC so that MEC can act as a client making secure HTTP requests to servers over SSL/TLS. For that, we reviewed the basics of HTTPS, we reviewed the features and drawbacks of HTTPSOut, we learned how to fix the authentication bug, how to add proxy settings, we identified security holes, we learned how to recompile the classes, how to setup the channel in PartnerAdmin, how to test, and how to troubleshoot.

HTTPSOut lacks features, it has a bug, and it has security holes; that is probably why it is just considered a sample channel that is unsafe to use in a production environment. Take it as a starting point from which you can build your own safe channel with production quality.

Hopefully, Infor will remove the deprecated documentation from the Partner Admin guide, will enhance the HTTPSOut channel from unsafe sample to safe channel with production quality, and will eventually provide a native connection handler in the Infor Grid for MEC.

In the next part of this guide, I will illustrate how to setup HTTPS for inbound messages.

(Congratulations if you made it this far! Let me know what you think in the comments below.)

Related articles

HTTP channels in MEC (part 4)

Here is the fourth part of the guide on how to setup HTTP channels in Infor Enterprise Collaborator (MEC), this time with how to setup and use the HTTPOut channel such that MEC acts as a client making HTTP requests to servers; that is the opposite of the HTTPIn and HTTPSyncIn and HTTPSyncOut channels where MEC acts as a server.

Documentation

The M3 Enterprise Collaborator Partner Admin Tool User Guide does not mention anything about the HTTPOut channel (no matches found in search), but if we read the pages attentively there is a chapter Setting Up Send Channels with a section on HTTP. After some studying of the HTTPOut Java code and testing in PartnerAdmin I confirm that section is effectively referring to the HTTPOut channel. Unfortunately it is just one sentence with no substantial information, and it is incorrect in that “the content type returned” should be “the content type of the request” and in that there is no such thing as “content type web browser”.

doc

Java code

By reading the partially recovered Java source code of class HTTPOut in package com.intentia.ec.communication we determine it is indeed an HTTP client (not a server, and not a response like the word out in HTTPSyncOut suggests). It uses a java.net.HttpURLConnection to make the request to the server defined in PartnerAdmin with host, port number and path.

Java

It supports connection keep alive, Basic authentication, no-cache directives, and multi-part content types.

Unfortunately, there are some drawbacks. The scheme is hard-coded to “http” so there is no HTTP-Secure possible; whereas it is possible with javax.net.ssl.HttpsURLConnection; although there is an HTTPSOut channel provided in the sample chapters of the documentation. The user and password transit in clear text (trivial Base64-decode). It is a departure from the benefits of Java NIO; see part 1 of the guide. The response is ignored (at best it is logged in the debug files) so I do not think there is a possibility for MEC to process the response. And it does not have support for a proxy.

Activity diagram

The HTTPOut channel is a Send process in a partner agreement, in other words it is a message sender, or an outgoing channel. More precisely it is an HTTP client making a request to a server. For the HTTPOut channel to run, the partner agreement must be triggered by an incoming channel such as DiskIn, Event Hub subscriptions, HTTPIn, and others.

Here is a simple activity diagram of the HTTPOut channel:

activity

Again, the word pair in/out does not reflect the word pairs client/server and request/response, and that can cause confusion.

How to setup

To setup an HTTPOut channel in MEC:

  1. Start the PartnerAdmin.
  2. In Communications > Receive, setup any incoming channel such that we can simply trigger the partner agreement for the purposes of illustration; the protocol is not important right now; for instance I have a simple DiskIn channel that has status RUNNING in my Infor Grid, and I have a partner agreement with it in a channel detection:
    1.11.4_3.1_
  3. In Communications > Send, add a new HTTPOut channel by selecting Protocol HTTP:
    2.1__
  4. Enter the host, port number, and path of the server; that will construct the URL http://host:port/path:
    2.2
    Note 1: The Content type is optional as HTTPOut will attempt to guess it based on the contents of the file using the Java method URLConnection.guessContentTypeFromName.
    Note 2: The checkbox Keep connection alive is to keep the TCP connection persistent and not close it, probably if there are multiple Send processes to eliminate the three-way handshake during connection establishment and reduce latency, although that is only useful for millisecond requirements, or to reduce network congestion (fewer TCP connections) although that is only useful for requirements with tens of thousands of concurrent connections; for the rest of us we can leave unchecked; I have not tested this.
    Note 3: The checkbox Post as multi-part is for sending multiple messages into one request; I have not tested this. By reading the Java code of the HTTPOut class, I can see the boundary is hard-coded to —————————, i.e. simply 27 repetitions of the dash character, which means if your file contains the same string (plausible) there could be unexpected results, like the request will fail or the message will be truncated.
    Note 4: The user/password is for HTTP basic authentication (Base64 encoded), but because this is for HTTP only (not HTTPS) the user/password transit in clear text (trivial Base64 decoding), i.e. it can be viewed by eavesdroppers, and it has no encryption and no message verification, i.e. it can be tampered by man-in-the-middle, so do not use this over unsecure networks.
  5. You can click Send test message for the HTTPOut channel to send a sample HTTP POST request to your server; I have Fiddler running on default port 8888 and it has a neat Fiddler Echo Service that I use as a simple HTTP server:
    2.3_
    Quirk: I notice the HTTPOut channel (or something else in MEC) generated two content type headers, they do not have the same case, they have different values from each other, and neither value is text/plain as I had specified in the PartnerAdmin; that again causes an HTTP protocol violation:
    2.3__
  6. Go to the Partner Agreement > Processes, add a Process Send:
    3.2
  7. Select the HTTPOut channel created earlier:
    3.2_
    Note 1: I suppose the encoding is for the binary encoding of the message in the HTTP response body; I have not tested this.
    Note 2: I suppose the Zip-output checkbox is for compressing the message in the HTTP response body; I have not tested this.
    Note 3: The Zip entry name is probably referring to java.util.zip.ZipEntry to compress the message and give it a file name; I did not know HTTP compression needed a file name.
    Note 4: I do not know what Routing Run On Host means.
  8. You can choose actions for Error Handling; in my case I leave it blank to keep it simple here:
    3.3
  9. The HTTPOut channel is now ready to use.

How to test

To test the new HTTPOut channel:

  1. Prepare a sample trigger for the partner agreement; in my case I have a simple message.txt file with content Hello, World! that I drop in the folder to trigger my DiskIn channel detection.
  2. Prepare the HTTP server on the other end to log the request so we can see the result; in my case I have Fiddler AutoResponder that will return the built-in HTTP 200 with a sample HTML page:
    4.2_
  3. Trigger the partner agreement to execute the Send Process; in my case I drop the message.txt file in the folder.
  4. The HTTPOut channel makes the request to the server, and we get the result; in my case in Fiddler:
    4.2__
  5. We can see the Event in the MEC Management Pages in the Infor Grid “Message successfully sent using protocol HTTP”:
    4.3_
    Note: I see no trace of the HTTP response in MEC; as suspected, MEC seems to ignore the response body.
  6. Also, for testing purposes, I set my Fiddler AutoResponder to respond with HTTP 404 Not Found, and thankfully the MEC Events show “error processing”, ERROR, and a Java stack trace with java.io.FileNotFoundException which is correct:
    HTTP404
  7. Also, for testing purposes, I set my Fiddler AutoResponder to respond with HTTP 502 Unreachable Server, and likewise the MEC Events show “error processing”, ERROR, and a Java stack trace with “java.io.IOException: Server returned HTTP response code: 502 for URL” which is correct:
    HTTP502
  8. You can now use your third party applications to receive HTTP requests from MEC.

Conclusion

That was how to setup the HTTPOut channel for MEC to act as a client and make requests to servers. For that, we setup the HTTPOut channel in PartnerAdmin, we setup the partner agreement with the Send process, and we tested with Fiddler. The HTTPOut channel is almost undocumented, not a state-of-the art HTTP client, and has drawbacks, but it suffices for most simple cases. In the next part of this guide I will illustrate how to setup a secure incoming HTTP channel for MEC, i.e. HTTP over SSL/TLS for MEC.

Related articles

UPDATES

2015-03-31: I removed my incorrect note about the boundary as in fact there is a random string appended to the boundary.

2015-03-19: I updated the documentation section about the content type, proxy, and HTTPSOut, and updated the quirk section about the content type.

HTTP channels in MEC (part 3)

Continuing the guide on how to setup HTTP channels in Infor M3 Enterprise Collaborator (MEC), here is the third part with how to setup and test the HTTPSyncIn and HTTPSyncOut channels, and how to use the MEC Mapper to process the incoming message and return a customized response.

HTTPSyncIn + HTTPSyncOut

The HTTPSyncIn and HTTPSyncOut channels are used in concomitance to accomplish both HTTP request and response. The HTTPSyncIn channel creates a simple HTTP server to accept requests, and the HTTPSyncOut channel returns custom responses, in the same connection. This is unlike the HTTPIn channel which only responds with a hard-coded acknowledgment of receipt.

Here is a simple activity diagram of both channels jointly in action:
activity

And here is an excerpt of the decompiled Java classes HTTPSyncIn and HTTPSyncOut of package com.intentia.ec.communication:
HTTPSyncIn HTTPSyncOut

Both Java classes HTTPSyncIn and HTTPSyncOut live in separate threads, and the keyword sync refers to thread synchronization for the threads to communicate with each other and access the socket channel in shared memory; see class SyncComPool. I do not think the keyword sync refers to synchronous and asynchronous, as in JavaScript XMLHttpRequests, as there are no event handlers in the client here; for the client it is all the same connection.

That means the entire processing of the message in MEC must be completed before the client times out. MEC usually responds within milliseconds, and the client usually times out in tens of seconds, so that leaves plenty of time.

How to setup

We need to setup the following:

  • Messages
  • MEC mapping
  • Receive channel
  • Send channel
  • Detection
  • Processes
  • Test

Prepare the messages

Let’s prepare two messages: one for the request and one for the response. I will keep it simple for the purposes of illustration, so for the request I will simply send a name, and for the response I will return “hello” + name; it is like an echo request (“ping”) and an echo reply (“pong”) containing the exact data received in the request message.

I have to choose my message format. Remember, MEC is XML-centric. If I choose flat file format I have to create a flat file definition which is also XML. I might as well just do XML from the beginning.

My incoming sample message request.xml will be:

<?xml version="1.0" encoding="UTF-8"?>
<name>Thibaud</name>

And my outgoing sample message response.xml will be:

<?xml version="1.0" encoding="UTF-8"?>
<message>Hello Thibaud</message>

Then, we have to create an XML Schema (*.xsd) for each XML file. For that, I will use the XML Schema Definition Tool (Xsd.exe) from the Microsoft .NET Framework with the command:

C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\
xsd.exe request.xml
xsd.exe response.xml

2.1_ 2.2_ 2.3_

The tool creates the files request.xsd and response.xsd.

There is a quirk. The files are encoded in UTF-8 and contain a Byte order mark (BOM) like most Unicode files should contain. But when we generate the mapping the MEC Mapper will trip on the BOM and will throw:

com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '?' (code 63) in prolog; expected '<'
 at [row,col {unknown-source}]: [1,1]
Generate failed

3.10_

So we have to remove the BOM, for example with Notepad++ select Encode without BOM and save the file:
2.4

XML

Setup the mapping

Now, let’s create a mapping in MEC Mapper to take the input message and create the customized response:

  1. Start the MEC Mapper.
  2. Create a new Project and a new Mapping, browse to the request.xsd and to the response.xsd, and click Finish:
    3.1_
  3. MEC mapper will display the XML elements of the request on the left and those of the response on the right. We have to do the mapping in between.
  4. From the palette, add a Java function to the mapping, then add an Input parameter and an Output parameter to the function, and connect the elements together from left to right:
    3.4__
  5. Then, right-click on the Java function, select Edit Java code, enter the following code in function(), save the file, and close the tab:
    oParameter = "Hello " + iParameter;

    3.5_

  6. Right-click in the mapping, select Save to Database, select the mapping database location, and click Finish:
    3.7_ 3.7__ 3.9_
  7. Right-click in the mapping again, select Generate, select the server location, and click Finish:
    3.10b 3.10c 3.11_
  8. Right-click in the mapping one last time, select Publish, select the server location, and click Finish:
    3.12_ 3.13 3.14
  9. Go to the Infor Grid Management Pages for MEC > Server > Mappings, locate the mapping, click Activate for the state to become Active:
    4.1_
  10. Go to the Server tab, click Reload Server Control, and click Yes to confirm:
    4-2__
  11. The mapping is now ready to be used in Partner Admin.

Setup in Partner Admin

Now let’s setup the HTTPSyncIn and HTTPSyncOut channels in MEC:

  1. Start the Partner Admin.
  2. Go to ManageCommunication.
  3. In the Receive tab, create a new inbound channel with protocol HTTPSyncIn and a port number, and set the checkbox Enabled:
    1.1
  4. In the Send tab, create a new outbound channel with protocol HTTPSync (it means HTTPSyncOut):
    1.2_ 1.2___
  5. Create a new Agreement.
  6. In the Detection tab, select the HTTPSyncIn channel just created:
    1.5_
  7. In the Processes tab, add process XML Transform, and select the Mapping created earlier:
    1.6_ 1.7
  8. Then add a process Send and select the HTTPSyncOut channel created earlier:
    1.8_ 1.9_
  9. Stop and start the MEC application in the Infor Grid, and the HTTPSyncIn channel will be in status RUNNING:
    4.2__
  10. Now the HTTPSyncIn and HTTPSynOut channels and agreement are ready to be used.

Test

Let’s test the new HTTPSyncIn and HTTPSyncOut channels:

  1. Prepare a sample HTTP request with header and body:
    POST http://localhost:8085/ HTTP/1.0
    Content-Type: application/xml
    Content-Length: 62
    Host: localhost:8085
    
    <?xml version="1.0" encoding="UTF-8"?>
    <name>Thibaud</name>
    
  2. Use an HTTP client like Fiddler Composer to send the request to MEC, and in return we get the customized response:
    5a 5b
  3. You can now use your third-party applications to communicate with MEC via HTTP.

Conclusion

That was how to setup and use the HTTPSyncIn and HTTPSyncOut joint channels for a client to send a request to MEC via HTTP and receive a custom response. For that, we setup the XML and XSD files for the request and for the response, we created the mapping in MEC Mapper, we setup the agreement in Partner Admin with the receive channel, the detection, the processes, and the send channel, and finally we tested with Fiddler Composer. It is admittedly quite a lot of work for in the end it is just a dynamic HTTP server, but the power of MEC lies in that it is an EAI product tailored to Infor M3 wherein HTTP is just one of the parts of the bigger puzzle. In the next part of the guide I will illustrate the HTTPOut channel for MEC in turn to become an HTTP client.

Related articles

HTTP channels in MEC (part 2)

Continuing the guide on how to setup HTTP channels in Infor M3 Enterprise Collaborator (MEC), here is the second part with how to setup and test an HTTPIn channel.

HTTPIn channel

The HTTPIn channel is the easiest to use of the incoming HTTP channels. It creates a simple HTTP server for MEC.

By reading the Java source code of method HTTPIn.runChannel(), we can tell the channel will open a non-blocking Java NIO ServerSocketChannel on the port number that we setup in Partner Admin, and it will wait for incoming connections. Upon receiving a request, the channel will start a thread worker HTTPInSubThreadWorker, then method runWork() will call class HTTPRequest to parse the HTTP request (remember the MEC HTTP channels are not fully HTTP/1.1 compliant so only specific HTTP requests will work), then it will extract the message, it will create a manifest, it will persist the message, it will add the message to the MEC queue, it will return a hard-coded HTTP response with status code 200 OK “The request was successfully processed” regardless of the incoming message (it is also called an acknowledgment received response), then it will close the socket connection, and then the MEC agreement will do the detection and processing of the message.

In EAI, this pattern is called “fire and forget”. There is no possibility of changing the HTTP response, it is a fixed response. I suppose this pattern was designed for high throughput by MEC, to process incoming requests as fast as possible to be able to accept the next requests without bottleneck. This reminds me of the Snd transactions of MI programs that accept requests and do not return a response.

Here is a simple activity diagram of the HTTPIn channel:

HTTPIn

Here is an excerpt of the decompiled Java class HTTPIn of package com.intentia.ec.communication:
HTTPIn_

How to setup

To setup an HTTPIn channel in MEC in an agreement with incoming channel, detection and process:

  1. Start the Partner Administrator.
  2. Select Manage > Communications:
    3.2
  3. In the Receive tab, click New:
    3.3
  4. Enter a Name, and select Protocol HTTPIn:
    2.3
  5. Find an available port number on the MEC server, for example use the following command to see which port numbers are already in use:
    netstat -an | findstr LISTEN

    2.4

  6. Enter that port number in the Property Port, and click OK:
    2.5
  7. Check the box Enabled:
    2.6
  8. Leave the other tabs – Send, M3 API, Databases, DAF – to their default values and click Close.
  9. In the Agreement View, right-click and select Insert group, enter a name for the group, then right-click the group and select Insert agreement, enter a name for the agreement, then in the Basic tab enter a Name:
    3.1_
  10. In the Detection tab, select either XML Detection, Channel detection, or Flat detection. I select Channel detection, Channel Group, and the HTTPIn channel I created earlier:
    3.3_
  11. When we click Save, the Partner Admin will show the warning message “No Body XPath is defined. Are you sure you want to continue?”, it is a false positive because we are not using XML Detection, so we can click Yes to ignore and continue:
    3.3__
  12. In the Process tab, select a process – in order to keep the illustration simple I choose to only Archive the message and not process it – and click Save:
    3.5_
  13. Now Stop and then Start the MEC application in the Infor Grid so that MEC starts our new channel:
    4.14.1_
  14. In the MEC Management Pages, in the Communication tab, verify that the channel is in State RUNNING:
    4.2
  15. The HTTPIn channel is now ready to use.

How to test

To test the new HTTPIn channel:

  1. Prepare a sample HTTP request with header and body:
    POST http://localhost:8082/ HTTP/1.1
    Host: localhost:8082
    Content-Type: text/plain
    Content-Length: 12
    
    Hello World!
    

    Quirk 1: We send data in the request body using method POST but somehow MEC will also accept method GET even if it is a protocol violation.
    Quirk 2: The HTTP request header Content-Type is mandatory, and without it MEC will throw a java.lang.NullPointerException without further details.

  2. Use an HTTP client like Fiddler Composer to send the request:
    Fiddler
    Quirk 3: Somehow when I use a telnet client to send a request to MEC I get a java.nio.BufferUnderflowException on the first byte whereas the telnet client otherwise works correctly with other HTTP servers.
  3. In return, MEC sends the hard-coded response “HTTP 200 OK […] e-Collaborator HTTP Reply […] The request was successfully processed”:
    Test2
  4. Back in the Infor Grid Management Pages, go to the Message tab and click show on your message:
    6.2_
  5. Click on the icon to open the Archived .rcv file:
    6.3_
  6. That will open the archive file of the message we sent:
    6.4_
  7. The archives files are stored in the file system somewhere at D:\Infor\MECDEV\archive\doc\f\:
    6.5_
  8. You can now use your favorite programming language or application to make the HTTP request.

Conclusion

That was an illustration of how to setup an HTTPIn channel in MEC using the Partner Administrator and Infor Grid so that clients can send HTTP requests to MEC in a “fire and forget” style. For that, we setup the agreement with the incoming channel, detection, and process, then we tested with Fiddler Composer, then we received the hard-coded acknowledgement of receipt, and finally we confirmed in the Infor Grid. In order to keep the illustration simple, I just archived the message and did not process it. In the next part of the guide I will illustrate the HTTPSyncIn and HTTPSyncOut channels and the MEC Mapper to process the message and return customized responses.

Related articles

HTTP channels in MEC (part 1)

Here is a multi-part guide on how to setup HTTP channels in Infor M3 Enterprise Collaborator (MEC) using the HTTPIn, HTTPOut, HTTPSyncIn, HTTPSyncOut channels such that MEC can communicate with other applications via HTTP.

About this guide

In this first part of the guide I will give an overview of MEC and the tools we will need. In upcoming parts of the guide I will illustrate each HTTP channel: HTTPIn, HTTPOut, HTTPSyncIn, and HTTPSyncOut. And at the end of the guide I will illustrate how to setup MEC for HTTP-Secure (HTTPS), i.e. HTTP over SSL/TLS.

The latest version of MEC that is available for download on Infor Xtreme is MEC version 11.4.2.0, but I do not have that version available to play with, so I will use the previous MEC version 11.4.1.0 from 2013 for which I have a server to play with and to learn.

I do not know the actual design decisions of MEC, and I have not been to a MEC training. In addition, it is difficult to learn MEC from its documentation because it only has partial explanations about the HTTP channels. Therefore, I will decompile the MEC server Java libraries to discover how MEC works internally, and I will ask my colleague Sheryll Limmex for help. As a result, what I write on this guide about MEC may or may not be accurate.

HTTP overview

HTTP is a stateless application protocol for communication between clients and servers usually over TCP/IP. There is an HTTP client and an HTTP server. The client makes an HTTP request to the server with a GET or POST method and a message, and the server responds with an HTTP response to the client with a status code and a message; in brief. The request and response each have a header and a body separated by a blank line. To maintain a session across requests the client and server must use a session identifier usually as a unique id in cookie or a parameter in the URL.

Here is a simple activity diagram between HTTP client and server:
0

Here is a sample HTTP request:

POST http://www.example.com/ HTTP/1.1
Host: www.example.com
Content-Length: 12

Hello World!

Here is a sample HTTP response:

HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 81

<!doctype html>
<html>
    <body>
        Example Domain
    </body>
</html>

I like to use a telnet client or Fiddler Composer to simulate a simple HTTP client and forge HTTP requests. And I like to use Netcat or Fiddler AutoResponder to simulate a simple HTTP server and forge HTTP responses.

Here is a sample HTTP request/response using the Windows telnet client:
y

Here is a sample HTTP request/response using Fiddler Composer:
0_ 0__

Documentation

To download the MEC documentation, go to Infor Xtreme > Downloads > Products > Product Search, search for MEC Server and download the documentation:
1.2

The M3 Enterprise Collaborator Partner Admin Tool User Guide trickles a few sentences on the HTTPIn and HTTPSyncIn channels, but nothing on the HTTPOut or HTTPSyncOut channels, and nothing on how to secure the communication either:
1.3

And the training material does not provide any information on the HTTP channels either:
Course

Java libraries

The most interesting Java library to learn about MEC channels is ec-core-11.4.1.0.0.jar. You can find it in the product download by unzipping products\MEC_11.4.1.0\components\MEC_11410.gar and continuing to MecServer\lib\ . You can also find it in the LifeCycle Manager folder of the MEC server at D:\Infor\LifeCycle\<host>\grid\<grid>\grids\<grid>\applications\<MEC>\MecServer\lib\ :
2

Client + server

MEC can act either as a server listening to incoming requests from clients (HTTPIn and HTTPSyncIn channels) and returning a response (HTTPSyncOut), either as a client making requests to servers (HTTPOut). The suffixes in/out in the channel names can lead to confusion as they do not exactly reflect the pairs of words client/server nor request/response.

Java NIO

By looking at the Java source code for the HTTP channels in MEC, I can tell they use Java Non-blocking I/O (NIO) and buffers, I suppose it is for high scalability and high throughput of its HTTP channels, for good reason, and it does not use the former and unfavorable Java I/O with its serialization, one-thread per connection and other bottlenecks; that is wondrous as I suppose that allows MEC to handle Gigabytes of data received, and tens of thousands of messages or connections simultaneously; to be confirmed.

Not HTTP/1.1

Unfortunately, when I look at the source code, I can tell the HTTP channels in MEC do not fully implement the HTTP 1.1 specification. They are just in-house subsets of the specification for the purposes of MEC; that is wary if we need specific parts of HTTP 1.1.

Partner agreements

MEC works with partner agreements. They are a combination of:

  • Incoming channels of various sorts to accept incoming messages
  • Detections to detect the type of incoming messages
  • Processes to take action on the messages, for example to decode the message
  • Mappings to act on the message, for example to call M3 API, SQL statements or Java code
  • Message senders to send a response back to the client

Here is a sample activity diagram:

agreement

Partner Admin

We will use the Partner Administrator tool to setup channels and agreements:
3.1

The Communication’s Receive tab reflects the Java classes that extend com.intentia.ec.shared.IncommingChannel:
Receive_ IncommingChannel

The Communication’s Send tab reflects the Java classes that implement com.intentia.ec.shared.IMessageSender:
Send_ MessageSender
The Agreement’s Process tab reflects the Java classes that extend com.intentia.ec.server.process.AbstractProcess:
3.5_ 3.5___

Mapping development

We will use the MEC mapper (plugin for Eclipse) to create the mappings that will process the incoming requests (and eventually transform XML, call M3 API, SQL statements, and Java code) and send customized responses.

Mapper Mapper

Grid management

We will administer the HTTP channels and messages from the Infor Grid Applications and the MEC Management Pages:
app Grid

Conclusion

That was a quick introduction of the HTTP channels and tools in MEC that we will use in this guide to setup channels and agreements to communicate with MEC via HTTP.

Related articles