Augmented Reality

M3 picking lists in Google Glass @ Inforum

I am very pleased to announce that after months of working here and there in the evenings voluntarily after work hours, I finally completed and presented both my demos of M3 picking lists in Google Glass and Augmented Reality at Inforum. They were a success. I showed the demos to about 100 persons per day during six days flawlessly with very positive reception. The goal was to show proof of concepts of wearable computers and augmented reality applied to Infor M3. My feet hurt.

Features

This is my second Glass app after the one for Khan Academy.

This Glass app has the following features:

It displays a picking list from Infor M3 as soon as it’s created in M3.
For each pick list line it shows the quantity (ALQT), item number (ITNO), item description (ITDS), and stock location (WHSL) as aisle/rack/level.
It displays the pick list lines as a bundle for easy grouping and finding.
It shows walking directions in the warehouse.
It has a custom menu action for the picker to mark an item as picked and to change the status of that pick list line in M3.
It uses the built-in text-to-speech capability of Glass to illustrate hands-free picking.
It’s bi-directional: from M3 to Google’s servers to push the picking list to Glass, and from Google’s servers to M3 when the picker confirms a line.
The images come from Infor Document Management (formerly Document Archive).
I developed the app in Java as an Infor Grid application.
I created a custom subscriber and added a subscription to Event Analytics to M3:MHPICL:U.
It uses the Google Mirror API for simplicity to illustrate the proof-of-concept.

I have been making the resulting source code free and open source on my GitHub repository, and I have been writing the details on this blog. I will soon post the remaining details.

Acknowledgements

I want to specially thanks Peter A Johansson of Infor GDE Demo Services for always believing in my idea, his manager Robert MacCrate for providing the servers on Infor CloudSuite, Philip Cancino formerly of Infor for helping with the functional understanding of picking lists in M3, Marie-Pascale Authié of Infor Pre-Sales for helping me setup and create picking lists in M3 and for also doing the demo at Inforum, Zack Makris of Infor Labs for providing technical support, Jonathan Amiran of Intentia Israel for helping me write the Grid application, and some people of Infor Product Development that chose to remain anonymous for helping me write a Java application for Event Hub and Document Archive. I also want to specially thank all the participants of Inforum whom saw the demo and provided feedback, and all of you readers for supporting me. And I probably missed some important contributors, thank you too. And thanks to Google X (specially Sergey Brin and Thad Starner) for believing in wearable computers and for accelerating the eyewear market.

Screenshots

Here below are the screenshots from androidcast. They show the bundle cover, the three pick list lines with the items to pick, the Confirm custom menu action, the Read aloud action, and the walking directions in the warehouse:

Vignettes

Here below are three vignettes of what the result would look like to a picker:

Inforum

Here are some photos at Inforum:

In the Manufacturing area:

In front of the SN sign:

Holding my Augmented Reality demo:

Playing around with picking lists in virtual reality (Google Cardboard, Photo Spheres, and SketchFab):

Playing around with picking lists in Android Wear (Moto 360):

That’s it! If you liked this, please thumbs up, leave a comment, subscribe to this blog, share around you, and come help me write the next blog post, I need you. Thank you!

I believe Google is silently killing Glass

I am starting to believe that Google is silently killing Glass for lack of success, rumors on The Interwebs Inc. are starting to confirm it, so I too am phasing out of my Glass project and moving on to Augmented Reality. I call Glass the Palm Pilot of 2013.

Palm

For those of you whom remember 1996 when Palm Pilot was released, it instantly became the new must-have gadget in geek universe, it was the future right now in a world of phone bricks…except it never took off, it came too early for its time, it was clunky, it needed to be manually connected, and it suffered from the what-can-I-do-with-it syndrome. A small faction of resistant users kept it afloat for many years, and then it was strolled and bounced around from bad to worse investment at HP to the surprise of most.

After that, phone manufacturers continued to fool us with so-called innovations for another decade.

And then came the iPhone in 2007, it blew everybody away, and it secured smartphones as obvious and unquestionable in every category.

The rumors

The Intertubes (TM) is starting to confirm the rumors that Glass is going away. I heard from a friend that a friend who works at Google said Google had asked some of their employees to return their Glass devices and had assigned them to other projects. Then, at the I/O conference this year Google played radio silence on Glass, no BMX, no parachutes, not even a peep. Then, the video for the release of Glass in London got ~~a whopping~~ 7k hits when I watched it, and now it is stalling at 300k hits. Then, my Google Searches for my Glass software development questions seem to return less and less hits. Finally, another friend of other Googler friends said the topic of Glass shutting down came up during a conversation. That is definitely solid proof, don’t you think?

My experience

I can speak of my own experience with Glass. I originally bought Glass because I was excited to finally try software development for wearable computers and Augmented Reality. I grew up reading in the 90s about the pioneers of the MIT Media Lab, Steve Mann and Thad Starner of the Wearable Computing Group and Hiroshi Ishii of the Tangible Media Group, and Professor Steven Feiner of Columbia University with his research on Augmented Reality. It seemed Glass was set to be the first of such devices ready for the mass market.

I have had Glass for 9 months, and from this gestation emerged the reality. I sadly came to admit I never wear it, it stays in the drawer. Even since October I continue to feel like Robocop with a thing on my face, it changes my behavior as if everybody was looking suspiciously at me, and that makes me uncomfortable. I get many positive reactions from people whom are curious of this novelty. Mostly I get too many negative reactions from Glass haters misconceiving it as an always on surveillance camera with face recognition. False. I sympathize with them because like them I value the protection of our freedom and privacy, and then I cannot help to satirically warn them I can turn on the X-ray vision. Then, in May at the Augmented World Expo 2014, I understood clearly that everybody had tried Glass for Augmented Reality and everybody gave up; in hindsight, they all admitted Glass had never been intended for Augmented Reality. Reality check. It changed my view of Glass as a wearable device. I kept Glass for software development. And then the technical problems. The battery exhausts too quickly, the device heats up and slows down to a freeze, and it is limited in terms of applications. And I am not that good of an Android developer to squeeze the juice out of it. Now I am using it just for pictures and videos, it is excellent for point-of-view shots. And so it has become the most expensive camera I possess.

I am still glad I had Glass, I killed a fever of want, I boosted my software development skills in the process, I anchored my confidence that I can still implement new technologies at 37, and I confirm wearable devices and Augmented Reality are here.

As for my smartphone, I have this weird dream-like feeling of needing to wrap the phone around me like a cloth, dive jump inside the screen and swim in a giant world of digital information. That is my need for the holodeck and Glass does not come close to an inch of fulfilling that.

What’s next

It is a fact wearable computers are here to stay. It just will not be with Glass. Glass was a milestone in history that will remain in the archives as one of the first general wearable devices. Glass also helped spawn the industry of eyewear, and there are valid niche markets where Glass-like devices fit perfectly, for example this safety device for motorcyclists from FUSAR Technologies that displays a rear camera inside the helmet.

I feel sad for Thad Starner and Sergey Brin whom really believed in Glass; they have other awesomeness in their sleeves. Steve Mann does not seem to be affected as he is doing great sensitizing us to sousveillance and working for META. If I project the analogy of the Palm versus iPhone history to Glass, we will see the natural heir of Glass, the obvious leader of the wearable computers, in 11 years from now, in 2025. Yikes! I say it will be a holodeck, light-guided into the eye, mixed with the Minority Report of John Underkoffler (he was one of Hiroshi’s students), and Tony Stark’s helmet of Iron Man.

Meanwhile, I think Google will push full throttle with Project Tango, after all, anything that Johnny Chung Lee touches becomes a hit. Tango is a dream for Augmented Reality enthusiasts. Also, I keep an eye on castAR and Projective Augmented Reality, and I am eagerly awaiting for their first device; Jeri Ellsworth is a self-taught pioneer with many followers that does not come from academics.

As for me, I will finish my Glass proof-of-concept to honor the commitments I made with my partners, and after that I will learn to implement Augmented Reality with Metaio SDK, Unity3D, Qualcomm Vuforia, and OpenCV.

Augmented Reality for M3 – Hello World with Metaio Creator

Here is a Hello World illustration of Augmented Reality (AR) for Infor M3 using Metaio Creator and the Junaio Browser on my iPad. The demo shows a 3D warehouse with aisles, racks, and levels, where I highlighted one of the boxes in red. This new result complements my previous demo of AR for M3 which was implemented programmatically in JavaScript. This time I am using Metaio Creator.

Why it matters

The idea is to highlight the stock location of the the next item to pick in a picking list so the picker can quickly identify where to go in the warehouse. This scenario is specially useful for temporary workers that are hired for campaigns on short notice and are not yet familiar with the warehouse thus saving costs in training and picking time.

Also, Augmented Reality is predicted to be one of the next multi-billion dollar industries in five years from now, so this is one of the learning steps I am taking in that direction.

Preview the demo

To preview the demo on your device (PC, Mac, iPad, Android) follow these instructions:

Print the following satellite picture in full page or bigger, and place it on a flat surface; that will be the trackable AR marker:
Install the Junaio Augmented Reality Browser app on your device (from junaio.com for PC/Mac, from the App Store for iPad, or from the Google Play Store for Android).
Open the app and click Scan.
Scan the following QR code; Junaio Browser will identify the QR code, and will download the resources from my channel ThibaudWarehouse3D:
Point your device’s camera towards the printed satellite picture. Junaio Browser will track the satellite picture and will register the warehouse 3D accordingly. Here is a screenshot of the result:

How I built it

The creation process is simple.

I used my previous 3D model of a warehouse with racks, aisles, levels, and boxes that I had created in SketchUp for a demo three years ago. I removed the walls and roof. I removed unnecessary 3D elements that slow down the 3D rendering pipeline on iPad. And I hard-coded an arbitrary box in red.

Here is a screenshot of the trackable and 3D model in Metaio Creator:

Here is a screenshot of the channel creation:

Here is a video of the entire creation process and preview:

Summary

That was how to create a simple Hello World demo of Augmented Reality for M3 using a 3D warehouse and Metaio Creator to highlight the stock location of the next item to pick in a picking list to save training time and picking time.

Future version

In a future version, I will un-hard-code the red box, and I will highlight it programmatically using Metaio SDK.

That’s it! Like. Comment. Subscribe. Share. Author.

Thank you.

Augmented World Expo 2014

Last week I attended the Augmented World Expo (AWE) 2014 [1] in Santa Clara, one of the world conferences on Augmented Reality, Virtual Reality, Augmented Virtuality [2], and smart glasses [3]. There, I saw Steve Feiner, pioneer of Augmented Reality in the 1990s [4] [5], Professor of computer science and director of the Computer Graphics and User Interfaces Lab at Columbia University, and adviser for Space Glasses at Meta [6]. I also saw Mark Billinghurst, director of the HITLab in New Zealand [7] whom created the AR Toolkit which I later used (JavaScript port) for my prototype M3 + Augmented Reality. I didn’t see Steve Mann, also adviser for Meta, and one of the pioneers of the Wearable Computing group in the Media Lab in the 1980s [8]; Thad Starner was in that group and later went on to design Google Glass for Sergey Brin [9]. I got inspiration from their work when I was younger, and I was excited to see them.

I went to the conference to learn more about the future. I’m currently working on a personal project to develop an app to display picking lists in Google Glass with data from Infor M3.

Here are some pictures of me at the conference, dreaming my vision of future picking lists 😉

M3 + Augmented Reality

In this article I introduce the first implementation that I know of Augmented Reality for Infor M3. Augmented Reality is the ability to superpose digital information on top of real world objects. This is achieved by locating the user’s head in space, by determining the user’s point of view, by registering real world objects, and by projecting virtual 3D objects accordingly. Implementing it has been a deer dream of mine. In this example I use fiducial markers and data coming from Item Master – MMS001.

Applications

Augmented Reality for M3 could be used for many applications. For example, it could help a worker find an Item in the warehouse by showing optimized walking directions and distance to possible picking locations. Also, it could help a worker show contextual information at a glance.

I believe Augmented Reality to be a disruptive technology and one of the next big revolutions in the software industry, with positive impacts similar to those of the Internet and mobile devices, that will reshape entire industries in the next 10 years.

Timeline & motivation

In 1998 I got a summer job in a warehouse for a company that sold car brakes. Every few minutes a printer spit out a picking list of items that I had to collect. As a temporary worker unfamiliar with the place, I spent most of my time wandering through the warehouse, searching for the items, and asking the more seasoned workers for help; I found that inefficient and I wished the computer gave me a map with directions of where to go. Also, the picking lists were un-ordered and I often had to go back to a previous location I had just visited; I found that inefficient and I wished the computer optimized the picking lists. Also, once I found the location, I often discovered the boxes were empty and I had to ask a forklift driver to replenish the stock location from a box of a higher shelf; I found that inefficient and I wished the computer planned replenishment ahead of time. That was in 1998 and nowadays ERP and Warehouse management systems are more common. Yet, I kept my wish to make better systems.

Then, In 2001 I read about Professor Steven Feiner’s Augmented Reality KARMA project from 1992 at Columbia University. The system fit in a backpack and had portable computer, batteries, GPS, compass, and head-mounted display. It would give detailed instructions to a user on how to repair a printer. That was my first exposure to Augmented Reality and ever since I have been wanting to implement it.

In 2007 Apple introduced the iPhone, with a stunning user interface, graphics, and processing power, blowing everybody’s mind about mobility and redefining an industry. And in 2009 Apple added a camera to the iPhone 3GS. The hardware technology for Augmented Reality started becoming accessible to the masses.

In 2009 I met with Brad Neuberg of Google at the Google I/O conference and I started working on a client-side search engine for M3 source code. That was my first exposure to HTML5.

In 2010 I implemented my first Warehouse 3D demo using Google Earth, with real data fed from the ERP, and I projected the result on a large touch screen for an immersive experience. That was my first step towards implemented Augmented Reality for M3.

In 2011 I proposed an idea for an internal project for M3 + Augmented Reality on mobile devices.

In parallel, WHATWG and W3C have been working hard to standardize HTML5 with the ability to use the webcam in JavaScript with WebRTC, to access pixel data, to paint on the canvas, and to use WebGL for 3D rendering. The software technology for Augmented Reality is becoming accessible to the masses.

More recently I started working on geo-locating Stock Locations in M3. This opens the door to new applications for geo-coded data in M3.

Then, at the Google I/O conference this year, I met with Ilmari Heikkinen whom pointed me to his article in HTML5 Rocks on Writing Augmented Reality Applications using JSARToolKit. That was the last push I needed to implement actual Augmented Reality for M3. So I did.

Implementation

I used Ilmari’s source code and I added a few lines of code to call an M3 API using REST in JavaScript when a marker is detected. In this example, the marker is mapped to an Item number (ITNO), but it could also be mapped to a Stock Location (WHSL) for example. Then, for that Item number I call the M3 API MMS200MI.GetItmBasic and I display the Name (ITDS), Description (FUDS), Basic unit of measure (UNMS), Volume (VOL3), Net weight (NEWE), Gross weight (GRWE).

Result

Here is a video of the result. Note the section below the canvas that shows M3 data coming from MMS200MI.GetItmBasic for the detected marker. We can see an activity indicator flickering as the markers are detected. For best viewing, watch the video in YouTube, in HD, and in full screen.

Source code

I provide the result for download at http://ibrix.info/ar/demo.zip with HTML and JavaScript source code, sample fiducial markers, and sample images.

Future work

With the simple example I introduced in this article I illustrate that hardware and software technology for Augmented Reality have have already become accessible for the masses. The technology is still maturing. There are on-going projects to provide registration without the use of markers. Also, sensors are becoming better for indoor location.

That’s it for now.

Please click ‘Follow’ to subscribe to my blog.

M3 + Augmented Reality (idea)

Here is an idea that would be great to implement: M3 + Augmented Reality. I believe AR to be one of the next big revolutions in the software industry, and the technology is available today. We have mobile phones with cameras, GPS, compass, and millimetric indoor radio positioning, fast CPU for feature registration, localization and mapping, REST Web Services, etc. It went from being mostly reserved to research labs, to being the hype of emerging start ups. Get ready for the future 🙂