Rio Tinto has embraced the all-or-nothing approach with Komatsu for its Mine of the Future program in Western Australia.
(Photo courtesy of Rio Tinto)

In a relatively short period of time, ASI has automated 50 different kinds of vehi-cles—not 50 vehicles, 50 different kinds of vehicles. The company has a good understanding of what it takes to automate a vehicle and integrate it into a process. Most of the work involves surface applica-tions, using GPS-based positioning for nav-igation. The company has some experience with factory and underground applications, or what is referred to as GPS-denied posi-tioning. That knowledge is helpful in addressing deficiencies with GPS.

There are two approaches to introduc-ing autonomous technology in the mine environment, an all-or-nothing grand vision approach, where the mine goes from manned-vehicles to a fully-automated unmanned fleet in one step, or a moderat-ed user-assisted approach. The user-assist-ed approach has been implemented fairly successfully in the agriculture industry for the last several years. Before a mine makes that decision though, understanding autonomous vehicles, how they work and robotics, sheds light on the technology’s strengths and weaknesses.

Who is looking at robotics? The interest seems to fall into two camps, which are not mutually exclusive, but the interest tends to be dominated by one or the other. Many major mining companies are considering autonomous mining to improve efficiency and productivity. They have a long range vision and they are expecting to make an investment to significantly change the min-ing process. These mining companies may also have “excessive manpower problems,” or they might operate in remote or unpleas-ant locations or both.

Then, there are also those mine opera-tors with immediate, non-routine safety problems, such as geotechnical issues. A highwall or bench failure would be a good example, where the mine has been told by regulators or an insurer that they cannot operate in a certain area and they need an unmanned vehicle to perform the work. These companies have safety concerns and are not necessarily looking to make an investment. They are not expecting to improve productivity. If anything, they are expecting to take a hit on productivity by using a remote control machine to deal with a circumstance.

Automation can be used to improve the mining process. The technology could allow equipment to work through breaks/shift changes, to mitigate the effects of weather or fatigue, and to elimi-nate mistakes by improving the accuracy and repeatability. Accurately repeating processes is an important aspect for min-ers. Haul trucks could be spotted correctly every time (dumping and loading). They would travel at the right speed all of the time. They would place materials in the appropriate place consistently. The machines would move/rip as much rock as possible. The system could maximize hole drilling speed and accuracy.

By being able to improve all of the pieces in the process, ultimately with the grand vision of a fully autonomous mine, engineers begin to redesign the mine and take a whole new approach.

The all-or-nothing approach is the approach currently being embraced by the OEMs. It is a viable option that requires a significant investment in one single inte-grated solution from one OEM. The poten-tial upside is obvious. This approach will work best for new mines in remote loca-tions. Aside from the big investment, the downside would be the reliability risks associated with a big process change and working with a single supplier.

Miners could also opt for an incremen-tal approach. They could begin with a user-assisted approach before transitioning to full autonomy over time. John Deere had an all-or-nothing vision. They were going to go from where they were to unmanned farming in one step. That hasn’t happened in the last 10 years. Instead, they have taken an incremental approach where they intro-duced user-assisted systems. Year-by-year, the user-assisted systems took more control of the functions of the vehicles until ulti-mately farmers have unmanned agriculture.

What is an Autonomous Vehicle?
To truly understand the grand approach as well as the incremental approach, users must first get a feel for how the technology works. Most people have some hazy notion that it has something to do with GPS. As the technology on an unmanned vehicle matures to an autonomous state with see-and-avoid behavior, it progresses through several stages. It starts with remote control, which electronically controls all the degrees of freedom on the vehicle. The operators have a controller in their hands and they are looking at the vehicle. The next level is tele-remote operation, the distinction is that the operators can’t see the vehicle so they are dependent on a video feed. For the next level, autonomous (blind), GPS and map-ping software give the machine the ability to execute a sequence of paths and actions commanded by the user, but it’s blind. If there is an obstacle in the way, the vehicle will plow into it or over it in the case of a haul truck. This would be the lowest level of autonomy. Vehicles can perceive their envi-ronment using an array of sensors. The sophistication varies from see-and-stop behavior, where a vehicle has a sensor hori-zon that tells it there is something there, but not enough sensors to tell it how to safely plan around it, to see-and-avoid behavior.

This Cat 793 haul truck is equipped with autonomous technology for hauling ore.

The first step in automating a vehicle is to convert it to drive-by-wire by adding actu-ators and hydraulic controls. Once it is drive-by-wire enabled, computers can talk to it. Most vehicles are not equipped with drive-by-wire controls. The drive-by-wire system ties into a controller, which acts as the brain for the vehicle. The controller is getting data from position sensors, which are primarily GPS-based, telling the vehicle where it is located. It also receives data from obstacle detection (OD) sensors. It knows where it is and what’s around it. The operator uses that information to control the vehicle, commu-nicating with the base station command-and-control software over a radio network. The video system operates in parallel.

The most vehicle-specific aspect of automating the vehicle is the drive-by–wire system. When ASI is approached with a new vehicle, the engineers determine what devices will be used to control the vehicle. Everything else—all of the other black boxes—remain the same from vehicle to vehicle. To a robotics engineer involved in autonomous systems, a utility vehicle is no different than a haul truck even though the vehicles are worlds apart as far as size and weight. Typically, the drive-by-wire package controls the steering, brakes, throttle, transmission, attachments and ancillary functions (horn, lights, etc.). It receives data from OEM systems (speeds, engine RPM, health, etc.) The goal in designing a good drive-by-wire system is to minimize the impact on the base platform and always allow a human to easily take control and operate the vehicle.

The division of labor between the com-mand-and-control software and the vehicle controller is straightforward. On the com-mand-and-control side, the software re-ceives high level goals from the user, e.g., go from point A to point B and do some-thing. It’s creating a global path plan to execute the goal taking into account the map and the vehicle’s driving capabilities. It sends the plan either all at once or in chunks at a time over the radio network to a vehicle control unit (VCU). On the VCU locally, the controller takes the goal and does all of the low level computing to deter-mine how it has to manipulate the throttle, steering and transmission to execute the goal. The VCU closes control loops on the vehicle. It performs onboard safety checks for off-path error, over-speed, over RPM, steering limits, redundant sensing, etc.

GPS is not perfect and it requires fil-tering and augmentation, especially in robust environments such as mining. It fits hand-in-hand with good map data. Adding perception and intelligence to the system mitigates the need for high accu-racy GPS and/or map data. An inertial measurement device, which calculates with a six-axis set of accelerometers and gyros, tells the system how the vehicle is moving. With vehicle odometry, the system knows the speed of the vehicle and the angle of the steering wheel. If it loses GPS, the system can dead reckon where the vehicle should be until it gets a new GPS point. A variety of other sensors per-form tasks of similar complexity.

For GPS-denied situations, infrastruc-ture-based solutions, such as reflectors and RFID tags, can be positioned through-out the pit. Another technology currently under development is vision- or laser-based recognition, where the system recognizes landmarks. The military is interested in this technology because of the threat that GPS could go away. That is the direction the industry will move toward for unmanned robotic systems.

Obstacle Detection and Avoidance
ASI uses three types of sensors for obsta-cle detection: lasers, vision and radar. They all have strengths and weaknesses.

What stays the same is the obstacle detection processing software, which takes the data from whatever suite of OD sensors make sense for a given application and converts that into a traversable map for the vehicle, locations it can and cannot drive. The spe-cific selection of sensors depends on a variety of factors based on definition. Would the vehicle be looking for other vehi-cles, people or debris on the road? The vehicle’s speed and the look-ahead dis-tance would also be a consideration for safety purposes. Environmental factors, such as rain, snow, dust and fog are also considerations. The sensors are fairly expensive, so mining companies would want to use as few of them as possible on each vehicle.

On the command-and-control side, the software should perform multi-vehicle coor-dination. The software has to be able to take high-level goals from a dispatch sys-tem and coordinate the activities of multi-ple vehicles to achieve those goals. The best situation avoids deadlocks or stale-mates along the route. Instead, the soft-ware prioritizes the vehicles and prevents them from ever coming near each other.

A whole new set of safety issues arise from autonomous mining. Does the mine allow humans and manned vehicles to interact with unmanned vehicles? How will-ing is the mine to rely on the OD system or tagging process to govern safety? One option is to use GPS-based safety perime-ters and “keep out” zones. That would probably be impractical for haul trucks that traverse the entire pit on haul roads shared with other vehicles. There are also a couple layers of safety measures built into the sys-tem, while it is working. As previously dis-cussed, long-range path planning for multi-ple vehicles prevents them from encounter-ing each other. If they do encounter anoth-er, then the system moves from a central-ized approach to a decentralized peer-to-peer approach where the vehicles can ulti-mately see each other. Even if they are not communicating with each other or they are not updating their position to a central con-troller, they can still see each other.

From a command center, one operator supervises multiple trucks. (Photo courtesy of Cat)

When the system isn’t working properly for some reason, it needs to fail in a safe manner. There are a lot of low level fea-tures that assist with the fail-safe system. An autonomous system must have a soft-ware-independent, stored-energy emer-gency brake system. In the case of com-plete power loss, it will bring the vehicle to a complete stop. There are several triggers, such as communications loss, off-path error, and system failure, which include redundant sensing on vehicle controls.

If radio communications is lost, then the big red button in the control room no longer works. Theoretically, the vehicle can run without radio communications. It could get the entire mission from the global con-troller and then off it goes. It does not need to talk to the global controller again. That’s probably not the safest way to operate. Typically, there is some kind of time-out on the vehicle if it loses communications.

There are many questions surrounding autonomous mining. At what point will the technology be proven? There is a big dif-ference between seeing a demo and seeing a system run 24-7, 365 days a year. How would a mine recover from failure if it has no operators? Reliability is obviously a huge issue. How does the mine know an OD system is safe enough? What type of test(s) does it run? There is no industry standard at this point. Can the system be integrated into existing operations?

Incremental Options
Working with a user-assisted program, min-ers could move toward the big vision of autonomous mining incrementally over time. The 24-7, 365 performance goal would be tabled for a while, but the mine could implement the technology at existing operations and retrofit the technology to current fleets. Humans are still in the loop and can be safely integrated into manned operations. Some of the current user-assist-ed systems for mining include auto-spot and auto-pilot solutions for haul trucks, col-lision alert-avoidance systems, remote con-trol/tele-remote systems with autonomous assist and ripping path assist for dozers.

Many machines, such as drills, shovels and dozers, have GPS and mapping tools to assist the operator that are also tools that could be used for autonomous opera-tions. The next move would be to shift up to a drive-by-wire or steer-by-wire technolo-gy. The users would have the ability to implement auto-spotting or auto-pilot. When the truck driver gets close to the shovel and they are able to communicate with each other, the autonomous system takes over and performs the spotting. Similarly, when the driver enters a long-haul corridor, the auto-pilot takes over and maintains optimal speed. Data from mine operators indicates that average speed tends to slowly decrease over the course of a shift and they could theoretically gain a measurable payback for that investment.

One of the problems with automating a haul truck fleet is the number of vehicles, which is a big investment. They cannot be effectively isolated. There are sets of equip-ment that do operate in isolated zones, such as drills and dozers. The mines could start by automating multiple unmanned vehicles, which do not have to interact with manned vehicles. In certain situations, such as mul-tiple drills working one pattern or multiple dozers ripping a leach pad, a single opera-tor could supervise two to five vehicles. Eventually as the group (operators, produc-tion managers and engineers) become more comfortable with the technology and equip-ment, it can be phased in over time.