Autonomy Reloaded
With the influx of open, interoperable autonomous mining solutions will come new possibilities... meet some of the companies that are ushering in this change.

By Carly Leonida, European Editor

An autonomous drill rig at FMG’s Christmas Creek mine in the Pilabara, Australia. (Photo: FMG)
Despite autonomous haulage systems (AHSs) having been commercially available for more than 20 years and some incredible results publicized in terms of utilization and cost reductions, the technology is still yet to achieve mainstream adoption. But that could be set to change thanks to a new wave of open, interoperable solutions. E&MJ asked autonomy consultant Ben Miller, partner and principal at Autonomous Correct, to help it explore the market and its trajectory.

“Today, there are only three commercially available AHS options,” he explained. “There are two from OEMs — Caterpillar and Komatsu — which both have a bunch of deployments, and one from a third-party provider — ASI — which has a few smaller deployments plus a major contract to convert trucks for Roy Hill in Australia. Both Hitachi and Liebherr have trials under way, but their systems are slightly different.” The latter OEMs have opted to produce trucks and/or systems that are ready to accept other companies’ autonomous hardware and software. It’s important because those two manufacturers generally only operate on mixed fleet sites; neither offer a large bulldozer, for instance, so their trucks and shovels need to be able to be “open” in order to participate in autonomous ecosystems.

“There are other companies that offer solutions for smaller machines like Bobcats, but there’s a very distinct difference between making one truck drive around and making many things drive around,” Miller explained. “Volvo’s approach is interesting; their autonomous technology is focused on trucks with less than 100 tons of capacity more often utilized in industrial mineral operations than large metallic mines. “With traditional AHSs, the cost per conversion is so high that companies will have a hard time pushing into the 100- ton truck class with an autonomy layer. Currently, mines are limited on how small they can go with deployments.”

How could the use of next-generation interoperable systems change that? “The market is beginning to demand interoperability and, because of that, AHSs are becoming more of a commoditized product,” Miller explained. “Now, because there are three companies offering these systems, mines can put their projects out to tender and see which bid they like best. Ten years ago, that wasn’t an option, mines would have to create a strategic partnership with either Caterpillar or Komatsu. Today, the process is similar to buying equipment, and that means that mines have a little more sway in how things are done.”

Interoperable Hardware
This shift combined with the fact that the vast majority of mines worldwide run mixed fleets, means that things have to change. “Most fleets aren’t all the same shade of yellow and often their trucks aren’t even the same model; for instance, a mix of Komatsu 930E-4s and 930E-5s,” said Miller. “Often, you’ll see Komatsu trucks with Komatsu shovels and then all the pushing gear is from Caterpillar. That’s OK to deal with in the current AHS situation. However, if a mine had some Cat trucks and some Hitachi trucks then it’s much harder to introduce an AHS. Caterpillar might encourage the mine to replace all of the trucks, or at least the Hitachi portion of the fleet, with newer Cat models, because their AHS kit wouldn’t be compatible with the existing ones. That’s just an example, but that limitation increases the cost of autonomy significantly for the mine.”

The concern for many mines when considering autonomous haulage is that, if they pick one of the two-leading solutions, Caterpillar’s Command for Hauling or Komatsu’s FrontRunner, then, when it comes to the next iteration of truck purchases (which is constantly occurring) they won’t be able to go to an open market if the autonomy layer won’t support other OEM’s trucks. Or that the AHS provider will support other trucks, but it could cost an extra million dollars to build special conversion kits. “No mines want that situation,” Miller said. “They don’t want to be beholden to the OEM that they’ve chosen for their dispatch or AHS system.”

An autonomous mine haul truck simulation using
Oxbotica’s autonomy platform. (Photo: Oxbotica)
Software obsolescence. We’ve all been there, usually with cell phones. You go to do a system update, but your phone model doesn’t support the latest version, so you have to upgrade your handset. But all your music is on a certain platform and you use a certain computer for work, so it’s easier to stick with the same phone brand rather than move to another. The purchasing decision has essentially been made for you. “It’s exactly the same sensation,” Miller said. “And you don’t want that sensation to occur over a $5 million truck. Mines are paranoid that’s how this is going to go, so that’s why there’s this pressure on interoperability.”

Interoperable Software
The ability to put autonomous hardware on to any truck is just one piece of the interoperability puzzle, there is also the software to consider. “With an AHS there needs to be interaction between the different layers of software,” Miller explained. “So, at the top of stack is probably the dispatch system — like DISPATCH from Modula Mining or Wencomine from Wenco International Mining Systems — which tells the machines where they are supposed to go. Then the next layer is the traffic control or traffic management system, like ASI’s Mobius, Cat Command or Komatsu’s FrontRunner, then there’s a machine health layer. There’s an opportunity to add APIs between those software layers in that stack so that they can talk to one another.”

Currently, there are supported APIs available that allow interoperability between Komatsu’s FrontRunner and Modular’s DISPATCH, and Modular released a public API in May to allow openness between third-party technologies and its suite of fleet management solutions. It’s also possible for mines to create unsupported APIs i.e., applications that pay attention to the data generated by the different layers and translate it for use in others. However, it takes a lot of time and effort to develop and maintain these APIs and, if the layer above the API receives an update, then the API might need to be rewritten. It’s a work around as opposed to an optimized solution. “Software interoperability is probably the most important piece,” Miller said. “Because it allows the mine to choose which dispatch system it uses alongside the AHS, rather than replace it for one that is selected by the AHS provider.”

It’s worth bearing in mind that the physical conditions at some mine sites dictate the use of certain fleet management (FMS) or positioning systems. Currently, if those mines want to run an AHS, they must maintain their existing system alongside a new dispatch system that works with the AHS stack. This adds a further layer of complexity into an operation and poses significant challenges in terms of change management. If not managed properly, this could set the project up to fail before it’s even started. “The cost to replace or add a new dispatch system at an existing operation is massive,” said Miller. “And it usually has to happen in advance of the AHS implementation. We’re talking maybe a year ahead and it could cost anywhere between $10 million and $100 million depending upon the size of the operation. If there were an API that allowed the mine to keep its dispatch system in place, whatever brand it might be, and just add an AHS… that would significantly reduce the cost and complexity of the conversion project.” The overall impact of hardware and software interoperability is the same; they open up the market and drive down the overall cost of autonomy.

“I think it will be a win-win situation,” said Miller. “The industry leaders might be resistant initially because, right now, they have the ability to control the market if they wish. The push for autonomy will level the playing field a little and everyone will be back to selling trucks for the value of trucks. Because interoperability in autonomy removes the potential for a vendor to say, ‘That truck and shovel don’t match, we don’t know if we can support that.’” The widespread introduction of interoperable technologies will probably see AHSs become more mainstream over the next 5-10 years, and not just at large mine sites, but smaller ones too. Miller agreed: “Interoperability not only strips out a significant amount of cost from the front end of AHS projects, which reduces risk, it also brings down the overall cost of deployment. Together, those things are going to make it much easier to get the business case for the technology across the line.”

From Truck to Shovel
The influx of small, agile technology providers into the mining autonomy market could also accelerate the development of solutions for loading equipment, too. “Technologically, the autonomous control of trucks is not that difficult,” Miller said. “My hope is that some of the small, innovative companies that are coming through find novel solutions for the other equipment. There’s a reason why shovel operators are at the pinnacle of the operator level. It’s because they are doing a huge amount of visual processing while they’re looking at the face they are digging; they’re evaluating rocks, dig ability and safety. They’re gauging whether the ore markers are correct and making decisions based on that.”

SafeAI recently signs an agreement with Japanese firm Obayashi to automate a construction site. The pair are currently
working on load-haul-dump cycles with an autonomous ADT. (Photo: SafeAI)
How far away are we from fully autonomous shovels? “There have been autonomous functions on excavators for some time, such as auto-grade control, which prevents the bucket from going beneath the bench level without warning,” said Miller. “And there are teleop solutions available for excavators and dozers today, but the productivity is much lower on a teleop unit than a manned one.” Part of the challenge is that good shovel operators need to “feel” the material they’re digging and how hard the machine is working. “There’s haptic feedback, which allows remote operators to ‘feel’ the material they’re digging, but we also need to envision a way to provide enhanced feedback to the operator in a tele-situation and give them all of the feelings they would have if they were in the machine. Like perception of balance, vibration and how hard the hydraulics are working,” Miller said.

“One of the guys who works for me is a Ph.D. student at the Colorado School of Mines and he’s looking at hyperspectral analysis of geology at the face to provide real- time feedback to loading unit operators.” Theoretically, by mounting a hyperspectral analyzer on a drone or on an excavator cab, processing the data and feeding it back to the operator in real time through a screen or even augmented reality glasses, then they could get a more accurate idea of the ore grade and mineral content than if they were sat in the cab. “Grade improvements delivered through functions like that could make up for the lower productivity that we see with teleop shovels and could potentially make the case to pull operators out of cabs,” Miller said. “It’s a really interesting space, and I hope that some of these new players try to figure out the hard problems. Because the agile sort of development that goes on underneath little nimble companies like that… that’s where we’re going to find solutions using new technology.” Let’s meet some of these companies...

ASI Mining
Although one of the three current big autonomy providers, ASI’s technology is fundamentally different to the other two. The company received an investment injection from Epiroc in 2018, with a view to expanding Epiroc’s scope of interoperable, autonomous solutions and the companies continue to work side-by-side today. Drew Larsen is director of business development at ASI Mining, and he explained the company’s ethos. “Our approach is to be OEM agnostic and customer-centric, above all else,” he said. “Because we don’t supply vehicles or FMSs ourselves, we have no business interest in replacing vehicles or legacy dispatch systems, nor in restricting future vehicle purchases to specific makes/models. “Additionally, we want to help future- proof mine automation, so that when AHS operators desire to expand to additional applications, such as autonomous drilling or blasting, they can leverage our autonomy platform — Mobius — to enable more seamless integration. Providing an open architecture that allows us to include the autonomy for all mobile equipment is our goal.” ASI’s solutions are scalable. The company offers remote operation and teleop solutions all the way up to full automation for single or multiple machines. “For some operations, like haulage, full autonomy is already happening, and the payoffs are huge in terms of increased utilization, efficiency, and cost savings,” Larsen said. “In other areas, such as dozing, full autonomy isn’t quite there yet. But we can auto-tram or provide basic navigation and situational awareness for the dozer. By applying semiautonomous functionality there is a seamless integration between teleop and autonomous control, enabling back and forth transition as needed based on the application. Over time, we anticipate more and more autonomous functionality in areas like dozers, wheel loaders, etc.”

Whereas most companies are focused on automating one or two aspects of an operation, such as drilling and hauling, the end goal for the Mobius platform is to track, monitor and automate all aspects of mobile equipment operation in a mine. This allows better management of a mine’s assets, as they are aware of each other. Also, by automating and tracking all aspects of a mine, operators can find new areas of improvement and optimization. Larsen explained: “A big part of our effort is focused on establishing standard interfaces and connection points for OEMs and other technology providers. Examples of this include the integration of various FMS systems such as Wenco and Jigsaw into Mobius, and interfaces with truck OEMs such as Liebherr.” The modular architecture of ASI’s hardware and software allows integration points with Mobius as well as onboard systems that enable increasing participation opportunities for OEMS that don’t offer a full autonomy stack. This benefits mining operators by eliminating the need for FMS replacement on brownfield sites or eliminating fleet replacements for unsupported models or OEMs. “We continue to work with new OEMs and even other solution providers, such as Enaex’s blasting program to participate in the Mobius autonomy platform. We’re excited to be able to announce new participants from time to time,” Larsen added.

Oxbotica’s autonomy platform offers a number of different software solutions that partners can mix and match as they choose or implement as a
package. (Photo: Oxbotica)
In July, ASI announced a compatibility agreement with MS4M, a mine management and optimization solution provider, to ensure compatibility between Mobius and MS4M’s ControlSense FMS for autonomous mine vehicles. “As companies such as MineSense supply and maintain FMS to their customers, our agreement assures their MS4M system will leverage standard interfaces with Mobius to enable integration between their FMS and ASI’s Mobius traffic management system,” Larsen said. “This assures MineSense customers the ability to implement autonomy under the MSF4M dispatch system having future compatibility with ASI’s autonomous systems.” As mentioned earlier, in March, ASI signed a subcontract with Epiroc for the supply of 77 autonomous vehicle conversions for the Roy Hill iron-ore mine in Western Australia. The companies are currently working on phase two of a threestage deployment.

“We have successfully automated both Cat 793F and Hitachi EH5000 haul trucks and completed integration of the Wenco dispatch system with Mobius,” Larsen said. “Overall, the project calls for the automation of 77 haul trucks and situational awareness on all other manned vehicles operating at site.” What’s next for Mobius? “We continue to expand the value proposition of autonomy for our customers by continuously improving KPI performance. We are also working to provide additional tolerance of suboptimal infrastructure including comms, and expansion into more applications and integration opportunities,” Larsen explained. “Over the next 5-10 years, we see brownfield mines and smaller mines introducing automation to their sites at much higher rates than today. Larger mines that have already automated their haulage fleets will expand to other services such as blasting and drilling, even ancillary activities like water trucks will be automated.”

Another company that made the mining headlines in June was U.K.-based autonomous software specialist Oxbotica, when it announced a partnership with mine FMS provider Wenco. Co-Founder and CTO Paul Newman joined the discussion. “Oxbotica was founded in 2014 by myself and Oxford University professor, Ingmar Posner,” he said. “Since then, it has grown from a startup into one of the world’s leading autonomous driving software companies.”

The company is founded on the principle of universal autonomy — the idea that having machines know where they are, what is around them and what they should do are universally required competencies for autonomous vehicle’s whether they’re in mines, ports or cities. “The universality point has some remarkable advantages,” said Newman. “If the software can span domains then learning in one domain naturally transfers to others bringing remarkable advantages. For example, the use of vision, radar and laser for obstacle detection and tracking on busy streets has transferred to the mining domain superbly. Similarly, with our partner Navtech, we have developed a system that uses radar as a replacement for GNSS in the mining industry, but it finds immediate application in ports and airports.”

Oxbotica’s product portfolio for the mining sector includes software that spans the entire technological spectrum, from low-level sensor managers, through calibration, four-modal localization (vision, laser radar and GNSS), mapping, perception (with vision, lidar or radar), 3D mapping and planning and control. “It’s vertical integration even covers user interface and data export systems including cloud-based autonomy APIs,” said Newman. “What is unique about Oxbotica is we offer these pieces as parts or as a whole integrated system. Oxbotica has built an autonomy platform from a number of solutions, and partners can mix and match those solutions as they choose. “A great example of this is our partnership with Wenco to develop an open autonomy solution for the industry. It will allow mining operators to deploy autonomy in a way that suits them, using hardware they already have and own and or operate.”

What makes your technology different to more traditional autonomous solutions? “There are some obvious and immediate differences, as well as some subtle ones,” Newman explained. “Firstly, there is the modularity argument; our solutions (e.g., localization, obstacle detection, etc.) have open APIs. Then underpinning that are some fundamental technology differentiators. Examples include localization and obstacle detection with any combination of vision, radar and or cameras. And, of course, our complete stack is remarkably low power (circa 300 W) and agnostic to choices of vehicle, sensor and compute platforms. “More subtly, our software-in-the-loop simulation and data synthesis tools allow us to rehearse and train and fake scenarios and sensor data across weather, lighting and scene types. This accelerates testing, secures safety cases and allows extremely rapid product iteration. This means we can react to customer needs in days.”

Oxbotica described its solutions as “universal.” In practice, this means the software can be installed on to any vehicle with a drive-by-wire interface, in any setting, to make it drive autonomously with a wide choice of sensors. The technology is able to work indoors, outdoors, underground, in any weather condition with zero dependence on external systems or reliance on GPS, third-party mapping or any single sensor modality or supplier. Newman outlined the next steps for the technology. “There is a growing business case for open autonomy in mining and innovative new solutions are being introduced to the market,” he said. “We will service that market by providing optionality and high-performance autonomy solutions. From machine vision, through to planning and full site simulation. We will continue to add new functionality and make autonomy easier to use, easier to deploy and easier to add to. “There are never going to be fewer autonomous vehicles in mining. We see the horizon of autonomous vehicles expanding in the sector far beyond current deployments. We see a world in which, if there is an advantage to having a vehicle being autonomous, whatever its function, it can be. The advantages are, of course, diverse — economic, operational and safety — but they are all served by our single vision of universal autonomy.”

SafeAI founder and CEO, Bibhrajit Halder, was part of the Caterpillar-BHP team that masterminded one of the first commercial AHS deployments back in 2008. Having taken his experience and software knowledge to Silicon Valley to work on self-driving cars, he came back to his roots in 2017, establishing SafeAI with a view to accelerating the transition to autonomous mining and construction. “I have been working in autonomy for nearly 15 years,” he said. “Being part of that massive Caterpillar-BHP project and helping to deliver the AHS system to site was a huge learning opportunity for me. In 2015, when I moved to Silicon Valley, autonomous self-driving cars were just starting to take off and, in the five years since, there has been an explosion in investment and technology development in that space. To give you an idea of the numbers, there’s about $20 billion invested in that ecosystem in the U.S. alone. Autonomous systems — the hardware, software and infrastructure — have all improved exponentially thanks to that.

SafeAI retrofits heavy equipment for autonomous applications in both mining and construction. It’s platform currently supports
trucks from 30-ton ADTs to 220t rigid-frame haul trucks, as well as dozers and skidsteers. (Photo: SafeAI)
“My team and I came from mining originally, and now we’re bringing our knowledge and experience from Silicon Valley back to heavy industry. We know that the mining sector is looking to scale up deployments and use autonomy more widely and that’s really where our vision is.” SafeAI retrofits heavy equipment for autonomous applications in both mining and construction. It’s full stack, task-specifi c autonomy platform currently supports trucks from 30-ton ADTs to 220-ton rigid-frame haul trucks, as well as dozers and skidsteers. “In addition to the autonomous kit (hardware), which can be easily taken off of one machine and transferred to another, we also provide cloud-hosted, AI-based software that handles machine coordination, assignment and fleet management,” Halder explained. “Our APIs are open, and we provide customers with full access to their operational data. We believe that mining companies own their own data, and they should be able to use it in any way they see fit.”

This interoperable approach means that SafeAI can supply any or all elements of an AHS. “Even though we have our own full stack AHS system, we have also integrated our hardware and software with other vendor’s systems,” Halder said. “For example, if a mine that already had Modular Mining’s DISPATCH FMS installed came to us and asked us to retrofit their Cat 777 trucks for autonomy which, by the way, Cat doesn’t currently offer, then we could do that and make our technology to work with DISPATCH too.” In October, SafeAI announced it had partnered with Japanese construction firm Obayashi Corp to create autonomous construction sites, beginning with a pilot program on a test site in California. “We are currently working on automating a Cat 725 ADT,” Halder said. “It will debut in the U.S. first and we will expand the availability of that solution over the next six months.”

Where does Halder see the technology heading? “We’ve got lots more work to do, but our goal is to work with mining companies, our end users, directly to deploy solutions that meet their needs exactly,” he said. “AHSs were originally developed by OEMs according to their vision of autonomy, but we’re doing things the other way around. “We believe the key isn’t in just using technology to automate mine sites, it’s about giving mining companies what they really want. For example, if a mine wants to reduce its haul road lane widths because that will help them with costs, then we will tailor our technology to enable their trucks to navigate narrow road widths safely. If they want to implement autonomous haulage but can’t afford to replace their network, then we will make our technology work with their current network. We work very closely with customers and that really makes for a better product.”

Halder hinted that SafeAI may expand its solutions in the coming years to incorporate automation for light vehicles and mine water trucks too. “The mining industry is at a tipping point with autonomous technology,” he said. “That tip may not happen for 2-3 years yet, but it’s coming. Autonomous technologies have improved exponentially. We can do things today that we never even thought of 10 years ago. There’s also a lot of venture backed capital flowing into the autonomy space, so we will see more players coming to the market with open and interoperable solutions. “From the customer side, mining companies are getting access to better technology at reduced prices. Today, there are approximately 600 autonomous mining trucks across the globe, and I think we’re going to see that number accelerate very quickly. The drivers are all there from both outside and inside the industry.

As featured in Womp 2020 Vol 12 -