JP7694885B2 - Method and computing system for performing or facilitating physical edge detection - Patents.com - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63/034,403, entitled "ROBOTIC SYSTEM WITH VISION MECHANISME," filed on June 4, 2020, which claims priority to U.S. Provisional Patent Application No. 17/331,878, entitled "METHOD AND COMPUTING SYSTEM FOR PERFORMING OR FACILITATING PHYSICAL EDGE DETECTION," filed on May 27, 2021, the entire contents of which are incorporated herein by reference.

The present disclosure relates to computational systems and methods for performing or facilitating physical edge detection.

As automation becomes more common, robots are used in more environments, such as warehousing and retail environments. For example, robots may be used to interact with objects in a warehouse. The robot's actions may be fixed or may be based on inputs, such as information generated by sensors in the warehouse.

One aspect of the present disclosure relates to a computing system, or a method performed by a computing system. The computing system may include a communications interface and at least one processing circuit. The communications interface may be configured to communicate with a robot and a camera having a camera field of view. At least one processing circuit may be configured to perform the following: receiving image information representing the group of objects, generated by the camera when the group of objects is within the camera field of view; identifying from the image information a plurality of candidate edges associated with the group of objects, the plurality of candidate edges being or including respective sets of image locations or physical locations forming respective candidates for representing a physical edge of the group of objects; if the plurality of candidate edges includes a first candidate edge formed based on a boundary between a first image region and a second image region, determining whether the image information satisfies a darkness condition defined by the first candidate edge, the first image region being darker than the second image region, the first image region and the second image region being respective regions described by the image information; and selecting a subset of the plurality of candidate edges to form a selected subset of candidate edges for representing the physical edge of the group of objects, the selecting including determining whether to retain the first candidate edge as a candidate for representing at least one of the physical edges of the group of objects by including the first candidate edge in the selected subset of candidate edges based on whether the image information satisfies the darkness condition defined by the first candidate edge.

1 illustrates a system for performing or facilitating physical edge detection consistent with embodiments herein. 1 illustrates a system for performing or facilitating physical edge detection consistent with embodiments herein. 1 illustrates a system for performing or facilitating physical edge detection consistent with embodiments herein. 1 illustrates a system for performing or facilitating physical edge detection consistent with embodiments herein.

1 provides a block diagram illustrating a computing system configured to perform or facilitate physical edge detection consistent with embodiments herein. 1 provides a block diagram illustrating a computing system configured to perform or facilitate physical edge detection consistent with embodiments herein. 1 provides a block diagram illustrating a computing system configured to perform or facilitate physical edge detection consistent with embodiments herein. 1 provides a block diagram illustrating a computing system configured to perform or facilitate physical edge detection consistent with embodiments herein.

1 illustrates an environment in which physical edge detection may be performed, according to embodiments herein. 1 illustrates an environment in which physical edge detection may be performed, according to embodiments herein. 1 illustrates an environment in which physical edge detection may be performed, according to embodiments herein.

1 provides a flow diagram illustrating a method for processing candidate edges according to an embodiment herein.

1 illustrates example image information in which candidate edges may be identified, according to embodiments herein. 1 illustrates example image information in which candidate edges may be identified, according to embodiments herein.

4 illustrates candidate edges identified from image information according to embodiments herein. 4 illustrates candidate edges identified from image information according to embodiments herein.

4 illustrates candidate edges identified from image information according to embodiments herein. 4 illustrates candidate edges identified from image information according to embodiments herein. 4 illustrates candidate edges identified from image information according to embodiments herein.

1 illustrates an environment in which an image may be generated based on diffuse reflection of incident light off objects within the camera's field of view, according to embodiments herein.

1 illustrates an image intensity profile around a candidate edge according to an embodiment herein. 1 illustrates an image intensity profile around a candidate edge according to an embodiment herein. 1 illustrates an image intensity profile around a candidate edge according to an embodiment herein. 1 illustrates depth values of regions around locations corresponding to candidate edges, according to embodiments herein.

1 illustrates an image intensity profile around a candidate edge according to an embodiment herein. 1 illustrates an image intensity profile around a candidate edge according to an embodiment herein. 1 illustrates an image intensity profile around a candidate edge according to an embodiment herein.

FIG. 11D illustrates an image intensity profile of an image representing a physical edge, while FIG. 11D illustrates depth values of positions around a physical edge, according to embodiments herein. 1 illustrates depth values for positions around a physical edge, according to embodiments herein. 1 illustrates depth values for positions around a physical edge, according to embodiments herein. 1 illustrates an image intensity profile of an image representing a physical edge, according to embodiments herein.

1 illustrates an image intensity profile for an image depicting an object having an image region that is darker than adjacent image regions, according to embodiments herein. 1 illustrates an image intensity profile for an image depicting an object having an image region that is darker than adjacent image regions, according to embodiments herein. 1 illustrates an image intensity profile for an image depicting an object having an image region that is darker than adjacent image regions, according to embodiments herein.

1 illustrates an image intensity profile for an image representing two objects separated by a physical gap, one object being darker than the other, according to embodiments herein. 1 illustrates an image intensity profile for an image representing two objects separated by a physical gap, one object being darker than the other, according to embodiments herein. 1 illustrates an image intensity profile for an image representing two objects separated by a physical gap, one object being darker than the other, according to embodiments herein.

One aspect of the disclosure relates to detecting or otherwise identifying physical edges of a group of objects using image information representing the group of objects. For example, a 2D image may represent a group of boxes and may include candidate edges that may potentially represent physical edges of the group of boxes. A computing system may use the candidate edges in the image information to distinguish between individual objects represented in the image information. In some instances, a computing system may use information identifying the individual boxes to control robotic interactions involving the individual boxes. For example, the robotic interactions may include an unpalletizing operation in which an end effector device of a robot approaches one of the objects, picks up the object, and moves the object to a destination location.

In some scenarios, a 2D image or other image information may include candidate edges that are false edges, which may be candidate edges that do not correspond to actual physical edges of objects in the camera field of view. Thus, one aspect of the present disclosure relates to evaluating candidate edges to determine a confidence level that the candidate edge corresponds to an actual physical edge as opposed to being a false edge. In an embodiment, such a determination may be based on an expectation or prediction regarding how a particular physical edge is likely to appear in an image. More specifically, such a determination may be based on an expectation that when a physical edge is associated with a physical gap between objects (e.g., the physical edge forms one side of the physical gap), such a physical gap may appear very dark in the image and/or may have an image intensity profile characterized by a spiked decrease in image intensity in the image region corresponding to the physical gap. Thus, a method or computational system of the present disclosure may operate based on an expectation that a physical gap between objects, particularly a narrow physical gap, may be represented by an image having certain characteristics related to how dark the physical gap is in the image. Such features or characteristics of an image may be referred to as dark priors, and the present disclosure may relate to detecting dark priors, the presence of which may increase the confidence level as to whether a candidate edge corresponds to an actual physical edge.

In an embodiment, a method or system of the present disclosure may determine whether an image satisfies a candidate edge-defined darkness condition, which may be related to detecting a dark prior. More specifically, the defined darkness condition may be defined by a darkness threshold criterion and/or a spike intensity profile criterion, which are discussed in more detail below. In this embodiment, if the computing system or method determines that an image satisfies a candidate edge-defined darkness condition, there may be a greater confidence level that the candidate edge corresponds to an actual physical edge, such as a physical edge that forms one side of a physical gap between two objects. In some instances, if an image does not satisfy a candidate edge-defined darkness condition, the candidate edge may be more likely to be a false edge.

One aspect of the present disclosure relates to using 2D image information to compensate for limitations of 3D image information, and vice versa. For example, when multiple objects, such as two or more boxes, are placed closely next to each other and separated by a narrow physical gap, the 3D image information may not have a high enough resolution to capture or otherwise represent the physical gap. Thus, the 3D image information may have limitations in its ability to be used to distinguish individual objects of a group of objects, especially when the objects have the same depth relative to the camera generating the 3D image information. In such an embodiment, the physical gap between the objects may be represented in the 2D image information. More specifically, the physical gap may be represented by image regions that meet a defined darkness condition. Thus, the candidate edges associated with such image regions may represent the physical edges of the objects with a high level of confidence. In such a situation, the candidate edges in the 2D image information may be useful in distinguishing individual objects of a group of objects. Thus, the 2D image information may enhance the ability to distinguish individual objects in certain situations.

In certain circumstances, the 3D image information may compensate for limitations of the 2D image information. For example, the 2D image may not meet a darkness condition defined at a particular candidate edge in the 2D image. In such an example, the candidate edge may have a low confidence level that corresponds to any real physical edge object in the camera field of view. The 3D image information may be used to compensate for this limitation in the 2D image information if the candidate edge in the 2D image information corresponds to a candidate edge in the 3D image information. More specifically, a candidate edge in the 2D image information may be mapped to a location or set of locations in the 3D image information where there is an abrupt change in depth. In such circumstances, the 3D image information may be used to increase the confidence level that the candidate edge in the 2D image information corresponds to a real physical edge.

In an embodiment, the 3D image information may be used to identify surfaces (e.g., top surfaces) of an object, and a candidate edge may be identified based on a location where there is a transition between two surfaces. For example, a surface may be identified based on a set of locations having respective depth values in the 3D image information that do not deviate from each other by more than a defined measurement variance threshold. The defined measurement variance threshold may account for the effects of imaging noise, manufacturing tolerances, or other factors that may cause random variations in the depth measurements of the 3D image information. The identified surface may be associated with a depth value that is the average of the respective depth values. In some implementations, a candidate edge may be detected in the 3D image information based on identifying a depth transition between two surfaces identified in the 3D image information that exceeds a defined depth difference threshold.

FIG. 1A illustrates a system 1000 for performing or facilitating physical edge detection, which may involve using image information representative of one or more objects to detect or otherwise identify the physical edges of the one or more objects. More specifically, system 1000 may include a computing system 1100 and a camera 1200. In this example, camera 1200 may be configured to generate image information that depicts or otherwise represents an environment in which camera 1200 is located, or more specifically, represents an environment in a field of view (also referred to as a camera field of view) of camera 1200. The environment may be, for example, a warehouse, a manufacturing plant, a retail space, or some other facility. In such an instance, the image information may represent objects located in such a facility, such as boxes, bins, cases, crates, or other containers. The system 1000 may be configured to generate, receive, and/or process image information, such as using the image information to distinguish between individual objects in the camera field of view, perform object recognition or object registration based on the image information, and/or perform robotic motion planning based on the image information, as discussed in more detail below (the terms "and/or" and "or" are used interchangeably in this disclosure). Robotic motion planning may be used to control a robot at a facility, for example, to facilitate robotic interaction between the robot and a container or other object. The computing system 1100 and the camera 1200 may be located at the same facility or may be located remotely from each other. For example, the computing system 1100 may be part of a cloud computing platform hosted at a data center remote from a warehouse or retail space and may communicate with the camera 1200 via a network connection.

In an embodiment, the camera 1200 (also referred to as an image sensing device) may be a 2D camera and/or a 3D camera. For example, FIG. 1B shows a system 1000A (which may be an embodiment of the system 1000) including a computing system 1100, and a camera 1200A and a camera 1200B (both of which may be embodiments of the camera 1200). In this example, the camera 1200A may be a 2D camera configured to generate 2D image information that includes or forms a 2D image describing the visual appearance of the environment in the camera's field of view. The camera 1200B may be a 3D camera (also referred to as a spatial structure sensing camera or a spatial structure sensing device) configured to generate 3D image information that includes or forms spatial structure information about the environment in the camera's field of view. The spatial structure information may include depth information (e.g., a depth map) that describes the respective depth values of various positions relative to the camera 1200B, such as positions on the surface of various objects in the field of view of the camera 1200. These locations on the camera's field of view or the surface of the object may also be referred to as physical locations. The depth information of this example may be used to estimate how the object is spatially arranged in three-dimensional (3D) space. In some instances, the spatial structure information may include, or be used to generate, a point cloud that describes locations on one or more surfaces of an object in the field of view of camera 1200B. More specifically, the spatial structure information may describe various locations on the structure of the object (also referred to as the object structure).

In an embodiment, the system 1000 may be a robotic manipulation system for facilitating robotic interaction between the robot and various objects in the environment of the camera 1200. For example, FIG. 1C shows a robotic manipulation system 1000B, which may be an embodiment of the system 1000/1000A of FIGS. 1A and 1B. The robotic manipulation system 1000B may include a computing system 1100, a camera 1200, and a robot 1300. As described above, the robot 1300 may be used to interact with one or more objects, such as boxes, crates, bins, or other containers, in the environment of the camera 1200. For example, the robot 1300 may be configured to pick up containers from one location and move them to another location. In some cases, the robot 1300 may be used to perform an unpalletizing operation, where a group of containers or other objects are unloaded and moved, for example, to a conveyor belt. In some implementations, the camera 1200 may be attached to the robot 1300, such as to a robotic arm of the robot 1300. In some implementations, the camera 1200 may be separate from the robot 1300. For example, the camera 1200 may be mounted to the ceiling of a warehouse or other structure and may remain stationary relative to the structure.

In an embodiment, the computing system 1100 of FIGS. 1A-1C may form or be part of a robot control system (also referred to as a robot controller) that is part of the robot manipulation system 1000B. The robot control system may be a system configured to generate commands for the robot 1300, such as, for example, robot interaction move commands for controlling the robot interaction between the robot 1300 and a container or other object. In such an embodiment, the computing system 1100 may be configured to generate such commands based on, for example, image information generated by the cameras 1200/1200A/1200B. For example, the computing system 1100 may be configured to determine a motion plan based on the image information, where the motion plan may be, for example, intended to grasp or otherwise pick up an object. The computing system 1100 may generate one or more robot interaction move commands to execute the motion plan.

In an embodiment, the computing system 1100 may form or be part of a vision system. The vision system may be, for example, a system that generates visual information describing the environment in which the robot 1300 is located, i.e., more specifically, the environment in which the camera 1200 is located. The visual information may include the 3D image information discussed above, and/or 2D image information, or some other image information. In some scenarios, when the computing system 1100 forms a vision system, the vision system may be part of the robot control system discussed above, or may be separate from the robot control system. When the vision system is separate from the robot control system, the vision system may be configured to output information describing the environment in which the robot 1300 is located. The information may be output to a robot control system that can receive such information from the vision system and, based on the information, perform motion planning and/or generate robot interaction movement commands.

In an embodiment, the computing system 1100 may communicate with the camera 1200 and/or the robot 1300 by a direct connection, such as a connection provided through a dedicated wired communication interface, such as an RS-232 interface, a universal serial bus (USB) interface, and/or a local computer bus, such as a peripheral component interconnect (PCI) bus. In an embodiment, the computing system 1100 may communicate with the camera 1200 and/or the robot 1300 over a network. The network may be any type and/or form of network, such as a personal area network (PAN), a local area network (LAN), e.g., an intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The network may utilize different technologies and layers or stacks of protocols, including, for example, the Ethernet protocol, the Internet Protocol suite (TCP/IP), Asynchronous Transfer Mode (ATM) technology, Synchronous Optical Networking (SONET) protocol, or Synchronous Digital Hierarchy (SDH) protocol.

In an embodiment, the computing system 1100 may communicate directly with the camera 1200 and/or the robot 1300, or may communicate via an intermediate storage device, or more broadly, an intermediate non-transitory computer-readable medium. For example, FIG. 1D illustrates a system 1000C, which may be an embodiment of the system 1000/1000A/1000B, including a non-transitory computer-readable medium 1400 that may be external to the computing system 1100, and may act as an external buffer or repository for storing image information generated by the camera 1200, for example. In one such example, the computing system 1100 may retrieve or otherwise receive image information from the non-transitory computer-readable medium 1400. Examples of the non-transitory computer-readable medium 1400 include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. The non-transitory computer readable medium may form, for example, a computer diskette, a hard disk drive (HDD), a solid state drive (SDD), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read only memory (CD-ROM), a digital versatile disc (DVD), and/or a memory stick.

As described above, the camera 1200 may be a 3D camera and/or a 2D camera. The 2D camera may be configured to generate a 2D image, such as a color image or a grayscale image. The 3D camera may be a depth-sensing camera, such as a time-of-flight (TOF) camera or a structured light camera, or any other type of 3D camera. In some cases, the 2D camera and/or the 3D camera may include an image sensor, such as a charge-coupled device (CCD) sensor and/or a complementary metal-oxide semiconductor (CMOS) sensor. In an embodiment, the 3D camera may include a laser, a LIDAR device, an infrared device, a light/dark sensor, a motion sensor, a microwave detector, an ultrasonic detector, a radar detector, or any other device configured to capture depth information or spatial structure information.

As described above, the image information may be processed by computing system 1100. In an embodiment, computing system 1100 may include or be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop computer, etc.), a smartphone, a tablet computing device, and/or any other computing system. In an embodiment, all of the functionality of computing system 1100 may be performed as part of a cloud computing platform. Computing system 1100 may be a single computing device (e.g., a desktop computer) or may include multiple computing devices.

2A provides a block diagram illustrating an embodiment of a computing system 1100. The computing system 1100 includes at least one processing circuit 1110 and a non-transitory computer-readable medium (or media) 1120. In an embodiment, the processing circuit 1110 includes one or more processors, one or more processing cores, a programmable logic controller ("PLC"), an application specific integrated circuit ("ASIC"), a programmable gate array ("PGA"), a field programmable gate array ("FPGA"), any combination thereof, or any other processing circuit.

In an embodiment, the non-transitory computer readable medium 1120 that is part of the computing system 1100 may be an alternative or addition to the intermediate non-transitory computer readable medium 1400 discussed above. The non-transitory computer readable medium 1120 may be a storage device such as an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, such as a computer diskette, a hard disk drive (HDD), a solid state drive (SSD), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, any combination thereof, or any other storage device. In some instances, the non-transitory computer readable medium 1120 may include multiple storage devices. In certain implementations, the non-transitory computer-readable medium 1120 is configured to store image information generated by the camera 1200 and received by the computing system 1100. In some instances, the non-transitory computer-readable medium 1120 may store one or more object recognition templates used to perform object recognition operations. The non-transitory computer-readable medium 1120 may alternatively or additionally store computer-readable program instructions that, when executed by the processing circuit 1110, cause the processing circuit 1110 to perform one or more techniques described herein, such as the operations described with respect to FIG. 4.

FIG. 2B depicts an embodiment of the computing system 1100, the computing system 1100A, including a communication interface 1130. The communication interface 1130 may be configured to receive image information generated, for example, by the camera 1200 of FIGS. 1A-1D. The image information may be received via an intermediate non-transitory computer-readable medium 1400 or network discussed above, or via a more direct connection between the camera 1200 and the computing system 1100/1100A. In an embodiment, the communication interface 1130 may be configured to communicate with the robot 1300 of FIG. 1C. If the computing system 1100 is external to the robot control system, the communication interface 1130 of the computing system 1100 may be configured to communicate with the robot control system. The communication interface 1130 may also be referred to as a communication component or communication circuitry, and may include, for example, communication circuitry configured to communicate via a wired or wireless protocol. As an example, the communications circuitry may include an RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth (registered trademark) controller, a PCI bus controller, any other communications circuitry, or a combination thereof.

In an embodiment, in FIG. 2C, the non-transitory computer-readable medium 1120 may store edge detection information 1126 that may describe a plurality of candidate edges identified from the image information generated by the camera 1200. As discussed in more detail below, when the image information represents a group of objects, each of the candidate edges may be or form a candidate for representing at least one of a plurality of physical edges of the group of objects. In some instances, the computing system 1100/1100A/1100B may determine whether a particular candidate edge in the edge detection information 1126 should be used to represent at least one of the physical edges of the group of objects. Such a determination may involve evaluating a confidence level associated with whether the candidate edge actually represents a physical edge, as opposed to being a false edge. In one example, such an evaluation may be based on whether the candidate edge is associated with image characteristics that result from representing a physical edge. Such characteristics may be associated with an image feature referred to as a dark prior, which is discussed in more detail below. In some scenarios, the computing system 1100 may select a subset of candidate edges from the plurality of candidate edges that have a sufficiently high confidence level that they actually represent a physical edge of the group of objects, while the candidate edges excluded from the subset may not have a sufficiently high confidence level that they represent a physical edge of the group of objects. Thus, if the computing system 1100/1100A/1100B determines that a particular candidate edge is to be used to represent at least one of the physical edges, the computing system may include the candidate edge in the subset. If the computing system 1100/1100A/1100B determines that a particular candidate edge is not to be used to represent at least one of the physical edges, the computing system may determine not to include the candidate edge in the subset. Candidate edges that are not included in the subset may be removed from the edge detection information 1126, or more broadly, may be excluded from further consideration as candidates for representing at least one of the physical edges of the group of objects.

In an embodiment, the processing circuit 1110 may be programmed by one or more computer-readable program instructions stored on a non-transitory computer-readable medium 1120. For example, FIG. 2D illustrates computing system 1100C, an embodiment of computing system 1100/1100A/1100B, in which the processing circuit 1110 is programmed by one or more modules, including a physical edge detection module 1125, an object recognition/registration module 1128, and/or a motion planning module 1129.

In an embodiment, the physical edge detection module 1125 may be configured to determine which candidate edge should be used to represent the physical edge of the group of objects from among multiple candidate edges appearing in the image information representing the group of objects. In some implementations, the physical edge detection module 1125 may perform such a determination based on whether a defined darkness condition is met and/or whether a depth discontinuity condition is met, as discussed in more detail below. In some instances, the physical edge detection module 1125 may also be configured to identify multiple candidate edges from the image information. In some instances, the physical edge detection module 1125 may be configured to perform image segmentation (e.g., point cloud segmentation), which may involve distinguishing individual objects represented by the image information. For example, the module 1125 may extract or otherwise identify an image segment (also referred to as an image portion) of the image information representing one object of the group of objects. In some implementations, the image segmentation may be performed based on, for example, the candidate edge that the module 1125 has determined should be used to represent the physical edge of the group of objects.

In an embodiment, object recognition/registration module 1128 may be configured to perform an object recognition operation or object registration based on results from physical edge detection module 1125. For example, if physical edge detection module 1125 identifies an image segment that represents an object of a group of objects, object recognition/registration module 1128 may be configured, for example, to determine whether the image segment sufficiently matches an object recognition template and/or to generate a new object recognition template based on the image segment.

In an embodiment, the motion planning module 1129 may be configured to perform robot motion planning based on the results of the physical edge detection module 1125 and/or based on the results of the object recognition/registration module 1128. As described above, the robot motion planning may be for a robotic interaction between a robot (e.g., 1300) and at least one object of a group of objects. In some instances, the robot motion planning may involve, for example, a movement by a component of the robot (e.g., an end effector device) to pick up an object and/or a determination of a trajectory of a subsequent component after picking up an object.

In various embodiments, the terms "computer-readable instructions" and "computer-readable program instructions" are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, the term "module" refers broadly to a collection of software instructions or code configured to cause the processing circuitry 1110 to perform one or more functional tasks. The modules and computer-readable instructions may be described as performing various operations or tasks when the processing circuitry or other hardware component is executing the module or computer-readable instructions.

3A-3C show an exemplary environment in which the processing of candidate edges, or more specifically, physical edge detection, may be performed. More specifically, FIG. 3A depicts a system 3000 (which may be an embodiment of systems 1000/1000A/1000B/1000C of FIGS. 1A-1D) including a computing system 1100, a robot 3300, and a camera 3200. Camera 3200 may be an embodiment of camera 1200 and may be configured to generate image information representing a scene within a camera field of view 3210 of camera 3200, or more specifically, representing objects within camera field of view 3210, such as objects 3510, 3520, 3530, 3540, and 3550. In one example, each of objects 3510-3540 may be a container, such as, for example, a box or crate, while object 3550 may be, for example, a pallet on which the container is placed.

3B, which more specifically shows the physical edges of the objects. More specifically, the figure shows physical edges 3510A-3510D of the top surface of object 3510, physical edges 3520A-3520D of the top surface of object 3520, physical edges 3530A-3530D of the top surface of object 3530, and physical edges 3540A-3540D of the top surface of object 3540. The physical edges of FIG. 3B (e.g., 3510A-3510D, 3520A-3520D, 3530A-3530D, and 3540A-3540D) may be the outer edges of the top surfaces of each of objects 3510-3540. In some instances, the physical edges of the object's surfaces (e.g., 3510A-3510D) may define the contours of the surface. If the object forms a polyhedron (e.g., a cube) with multiple non-coplanar surfaces (also called faces), the physical edge of one surface may form the boundary where the surface meets another surface of the object.

In an embodiment, an object within the camera field of view may have visual details, such as visual markings (also referred to as visible details), on the outer surface of the object. For example, in FIGS. 3A and 3B, objects 3510, 3520, 3530, 3540 may have visual markings 3512, 3522, 3532, 3542, respectively, printed or otherwise disposed on the outer surface (e.g., top surface) of each of objects 3510-3540. As examples, the visual markings may include visible shapes, such as visible lines (e.g., straight or curved lines), polygons, visual patterns, or other visual markings. In some scenarios, the visual markings (e.g., visible lines) may form or be part of a symbol or drawing displayed on the outer surface of the object. Symbols may include, for example, logos or characters (e.g., alphanumeric characters). In some scenarios, visual details on the outer surface of a container or other object may be formed by the contour of a layer of material (e.g., a strip of packaging tape or a sheet of mailing labels) disposed on the outer surface of the container.

In an embodiment, the system 3000 of FIG. 3A may include one or more light sources, such as light source 3600. Light source 3600 may be, for example, a light emitting diode (LED), a halogen lamp, or any other light source, and may be configured to emit visible light, infrared light, or any other form of light toward the surface of objects 3510-3550. In some embodiments, computing system 1100 may be configured to communicate with light source 3600 to control when light source 3600 is activated. In other implementations, light source 3600 may operate independently of computing system 1100.

In an embodiment, as shown in FIG. 3C, system 3000 may include multiple cameras, including camera 3200A (which may be an embodiment of camera 1200A) having camera field of view 3210A, and camera 3200B (which may be an embodiment of camera 1200B) having camera field of view 3210B. Camera 3200A may be, for example, a 2D camera configured to generate 2D images or other 2D image information, while camera 3200B may be, for example, a 3D camera configured to generate 3D image information. The 2D images (e.g., color or grayscale images) may describe the appearance of one or more objects, such as objects 3510-3550, in camera field of view 3210/3210A. For example, the 2D images may capture or otherwise represent visual details, such as visual markings 3512-3542 disposed on the outer surfaces (e.g., top surfaces) of objects 3510-3540 and/or the contours of their outer surfaces. In an embodiment, the 3D image information may describe the structure of one or more of the objects 3510-3550, which may also be referred to as the structure of the object or the physical structure of the object. For example, the 3D image information may include a depth map, or more generally, depth information that may describe respective depth values of various positions of the camera field of view 3210/3210B relative to the camera 3200B or relative to some other reference point. The positions corresponding to the respective depth values may be positions (also referred to as physical positions) on various surfaces of the camera field of view 3210/3210B, such as positions on the top surface of each of the objects 3510-3550. In some instances, the 3D image information may include a point cloud, which may include a plurality of 3D coordinates that describe various positions on one or more outer surfaces of the objects 3510-3550, or some other object within the camera field of view 3210/3210B.

In the example of Figures 3A and 3B, a robot 3300 (which may be an embodiment of robot 1300) may include a robot arm 3320 having one end attached to a robot base 3310 and another end attached to or formed by an end effector device 3330, such as a robot gripper. The robot base 3310 may be used to mount the robot arm 3320, while the robot arm 3320, and more specifically, the end effector device 3330, may be used to interact with one or more objects (e.g., 3510/3520/3530/3540) in the environment of the robot 3300. The interaction (also referred to as robot interaction) may include, for example, grasping or otherwise picking up at least one of the objects 3510-3540. For example, the robot interaction may be part of an unpalletizing operation in which the robot 3300 picks up objects 3510-3540 (e.g., boxes) from an object 3550 (e.g., a pallet or other platform) and is used to move the objects 3510-3540 to a destination location.

As discussed above, one aspect of the disclosure relates to performing or facilitating detection of one or more physical edges of a group of objects, such as a group of boxes, based on image information representing one or more objects. FIG. 4 illustrates a flow diagram of an example method 4000 for performing or facilitating physical edge detection, or more specifically, for determining whether a candidate edge should be used to represent at least one of the physical edges of the group of objects. More specifically, the method may involve receiving image information having a candidate edge that may represent a physical edge or may be a false edge. A false edge may be, for example, a candidate edge that represents a visible line or other visual marking displayed on a surface of one of the group of objects. The visual marking may have an appearance that resembles a physical edge but does not actually correspond to any physical edge. Thus, the method 4000 may be used in embodiments to evaluate a confidence level or likelihood as to whether a candidate edge corresponds to an actual physical edge, or whether the candidate edge is likely to be a false edge. If a candidate edge is likely to be a false edge and/or does not have a high enough confidence level that it corresponds to a real physical edge, the method 4000, in an embodiment, may remove or more broadly exclude the candidate edge from further consideration for representing any physical edges of the group of objects.

In an embodiment, the method 4000 may be performed, for example, by the computing system 1100 of FIG. 2A-FIG. 2D or FIG. 3A or 3C, or more specifically, by at least one processing circuit 1110 of the computing system 1100. In some scenarios, at least one processing circuit 1100 may perform the method 4000 by executing instructions stored on a non-transitory computer-readable medium (e.g., 1120). For example, the instructions may cause the processing circuit 1110 to perform one or more of the modules shown in FIG. 2D, which may perform the method 4000. As an example, one or more of steps 4002-4008 discussed below may be performed by the physical edge detection module 1125. If the method 4000 includes steps of performing object recognition and/or object registration, the steps may be performed, for example, by the object recognition/registration module 1128. Where the method 4000 involves planning robot interactions or generating robot interaction movement commands, such steps may be performed, for example, by the motion planning module 1129. In an embodiment, the method 4000 may be performed in an environment in which the computing system 1100 is in communication with a robot and camera, such as the robot 3300 and camera 3200/3200A/3200B of FIG. 3A and FIG. 3C, or with any other camera or robot discussed in this disclosure. In some scenarios, such as those shown in FIG. 3A and FIG. 3C, the camera (e.g., 3200) may be mounted on a stationary structure (e.g., the ceiling of a room). In other scenarios, the camera may be mounted on a robot arm (e.g., 3320), or more specifically, on an end effector device (e.g., 3330) of the robot (e.g., 3300).

In an embodiment, one or more steps of method 4000 may be performed when a group of objects (e.g., 3510-3550) is currently in the camera field of view (e.g., 3210/3210A/3210B) of a camera (e.g., 3200/3200A/3200B). For example, one or more steps of method 4000 may be performed immediately after the group of objects enters the camera field of view (e.g., 3210/3210A/3210B), or more generally, while the group of objects is in the camera field of view. In some scenarios, one or more steps of method 4000 may be performed while the group of objects is in the camera field of view. For example, when a group of objects is in a camera field of view (e.g., 3210/3210A/3210B), the camera (e.g., 3200/3200A/3200B) may generate image information representative of the group of objects and communicate the image information to a computing system (e.g., 1100). The computing system may perform one or more steps of method 4000 based on the image information while the group of objects is still in the camera field of view, or even when the group of objects is no longer in the camera field of view.

In an embodiment, method 4000 may begin with or otherwise include step 4002, in which computing system 1100 receives image information representing a group of objects within a camera field of view (e.g., 3210/3210A/3210B) of a camera (e.g., 3200/3200A/3200B). The image information may be generated by a camera (e.g., 3200/3200A/3200B) when the group of objects is (or was) in the camera field of view, and may include, for example, 2D image information and/or 3D image information. For example, FIG. 5A shows 2D image information, or more specifically, 2D image 5600 generated by camera 3200/3200A and representing objects 3510-3550 of FIGS. 3A and 3C. More specifically, the 2D image 5600 (e.g., a grayscale or color image) may describe the appearance of the objects 3510-3550 from the viewpoint of the camera 3200/3200A. In an embodiment, the 2D image 5600 may correspond to a single color channel (e.g., a red, green, or blue channel) of a color image. If the camera 3200/3200A is positioned above the objects 3510-3550, the 2D image 5600 may represent the appearance of the top surfaces of each of the objects 3510-3550. In the example of FIG. 5A, the 2D image 5600 may include respective portions 5610, 5620, 5630, 5640, and 5650 (also referred to as image portions) that represent the surfaces (e.g., top surfaces) of each of the objects 3510-3550. In FIG. 5A, each image portion of image portions 5610-5650 of 2D image 5600 may be an image region, or more specifically, a pixel region (where an image is formed by pixels). More specifically, an image region may be a region of an image, and a pixel region may be a region of pixels. One or more image portions 5610-5550 may capture or otherwise represent visual markings or other visual details that are visible or appearing on the surface of an object. For example, image portion 5610 may represent visual marking 3612 of FIG. 3B, which may be printed or otherwise disposed on the top surface of object 3610.

5B shows an example in which the image information of step 4002 includes 3D image information 5700. More specifically, the 3D image information 5700 may include, for example, a depth map or point cloud indicating respective depth values for various locations on one or more surfaces (e.g., top surfaces, or other exterior surfaces) of the objects 3510-3550. For example, the 3D image information 5700 may include a first portion 5710 (also referred to as an image portion) indicating the depth values of each of a set of positions 5710 ₁ to 5710 _n (also referred to as physical positions) on the surface of the object 3510, a second portion 5720 indicating the depth values of each of the set of positions 5720 ₁ to 5720 _n on the surface of the object 3520, a third portion 5730 indicating the depth values of each of the set of positions 5730 ₁ to 5730 _n on the surface of the object 3530, a fourth portion ₅₇₄₀ indicating the depth values of each of the set of positions 5740 ₁ to 5740 _n on the surface of the object 3540, and a fifth portion 5750 indicating the depth values of each of the set of positions 5750 1 to 5750 _n on the surface of the object 3550. Each depth value may be relative to the camera (e.g., 3200/3200B) generating the 3D image information, or may be relative to some other reference point. In some implementations, the 3D image information may include a point cloud including respective coordinates for various positions on the structure of the object within the camera field of view (e.g., 3210/3210B). In the example of FIG. 5B, the point cloud may include respective sets of coordinates describing positions on the surface of each of the objects 3510-3550. The coordinates may be 3D coordinates, such as [X Y Z] coordinates, and may have values relative to the camera coordinate system, or some other coordinate system. As an example, the camera coordinate system is defined by X, Y, Z, as shown in FIGS. 3A, 3C, and 5B.

In an embodiment, step 4002 may involve receiving both 2D image information and 3D image information. In some instances, the computing system 1100 may use the 2D image information to compensate for limitations of the 3D image information, or vice versa. For example, if multiple objects in a camera field of view are positioned close to each other and have substantially equal depths relative to the camera (e.g., 3200B), the 3D image information (e.g., 5700) may describe multiple positions with substantially equal depth values and may lack detail to distinguish individual objects represented in the 3D image information, especially if the spacing between the objects is too narrow for the resolution of the 3D image information. In some instances, the 3D image information may have erroneous or missing information due to noise or other error sources that may further increase the difficulty of distinguishing individual objects. In this example, the 2D image information may compensate for this lack of detail by capturing or otherwise representing physical edges between individual objects. However, in some instances, the 2D image information may include false edges, which may be candidate edges that do not correspond to actual physical edges, as discussed below. In some implementations, the computing system 1100 may evaluate the likelihood that a candidate edge is a false edge by determining whether the 2D image information satisfies a darkness condition defined at the candidate edge, as discussed below with respect to step 4006. In some implementations, the computing system 1100 may determine whether the candidate edge corresponds to a physical edge in the 3D image information, such as when the candidate edge corresponds to a physical location where the 3D image information describes an abrupt change in depth. In such scenarios, the 3D image information may be used to check whether the candidate edge is a false edge, which may complement or replace the use of the defined darkness condition to provide a more robust method of determining whether the candidate edge is a false edge or whether the candidate edge corresponds to an actual physical edge.

Returning to FIG. 4, method 4000 may include step 4004 in which, in one embodiment, computing system 1100 identifies from the image information of step 4002 a number of candidate edges associated with the group of objects (e.g., 3510-3550). In an embodiment, the candidate edges may be or include image locations or sets of physical locations that form candidates for representing a physical edge of an object or group of objects. In one example, where the image information includes 2D images for representing one or more objects, the candidate edges may refer to a set of image locations, such as pixel locations (e.g., pixel locations [u ₁ v ₁ ]-[u _k v _k ]). The set of pixel locations may correspond to a set of pixels that collectively resemble a physical edge. For example, FIG. 6A illustrates computing system 1100 identifying candidate edges 5601 ₁ , 5601 ₂ , 5601 ₃ , 5601 ₄ , 5601 ₅ , 5601 ₆ , . 5601 _n . Each of candidate edges 5601 ₁ -5601 _n may include or be formed, for example, by a respective set of pixel locations where the 2D image defines a line or line segment having an abrupt change in image intensity. An abrupt change in image intensity may occur, for example, between two image regions immediately adjacent to one another, where one image region is darker than the other. As discussed in more detail below, the candidate edge may be formed based on a boundary between the two image regions. In such an example, the boundary may be formed by the line or line segment discussed above. A candidate edge identified from a 2D image (e.g., 5600) may be referred to as a 2D candidate edge or a 2D edge.

In an embodiment, the image information may include some candidate edges that correspond to actual physical edges and some candidate edges that are false edges. For example, candidate edges 5601 ₁ , 5601 ₂ , 5601 ₅ , 5601 ₆ in FIG. 6A may correspond to actual physical edges of the group of objects 3510-3550, or more specifically, to physical object 3510, while candidate edges 5601 ₃ and 5601 ₄ may be false edges. Candidate edges 5601 ₃ and 5601 ₄ may represent, for example, visible lines or other visual markings displayed on the surface of object 3510. These visible lines may resemble physical edges but do not correspond to actual physical edges of objects 3510-3550. Thus, as discussed below with respect to step 4008, method 4000, in one embodiment, may involve determining whether a particular candidate edge should be used to represent at least one of the physical edges of the group of objects.

In one example, if the image information includes 3D information, the candidate edges may refer to a set of image locations or a set of physical locations. As an example, if the image locations are pixel locations, they may correspond to a set of pixels that appear to be physical edges. In another example, if the 3D image information includes a depth map, the candidate edges may include a set of pixel locations that may form a boundary where there is an abrupt change in depth in the depth map, e.g., defining a line or line segment. If the 3D image information describes 3D coordinates of physical locations on the surface of an object (e.g., via a point cloud), the candidate edges may include a set of physical locations that may form a boundary where there is an abrupt change in depth of the point cloud or other 3D image information, e.g., defining a virtual line or line segment. For example, FIG. 6B illustrates an example where the computing system 1100 has identified candidate edges 5701 ₁ , 5701 ₂ , 5701 ₃ , 5701 _n from the 3D image information 5700. Each candidate edge 5701 ₁ -5701 _n may, for example, include a set of physical locations [X ₁ Y ₁ Z ₁ ] - [X _p Y _p Z ₁ ] that define a boundary where an abrupt change in depth occurs. Candidate edges identified from the 3D image information may be referred to as 3D candidate edges or 3D edges.

In an embodiment, if the computing system 1100 identifies both 2D and 3D candidate edges, the computing system 1100 may be configured to determine whether any of the 2D candidate edges (e.g., 5601 ₅ ) represent a common physical edge with one of the 3D candidate edges (e.g., 5701 ₁ ), or vice versa. In other words, the computing system 1100 may determine whether any of the 2D candidate edges map to one of the 3D candidate edges, or vice versa. The mapping may be based, for example, on transforming the coordinates of the two-dimensional candidate edge from being represented in the coordinate system of the 2D image information to being represented in the coordinate system of the 3D image information, or on transforming the coordinates of the 3D dimensional candidate edge from being represented in the coordinate system of the 3D image information to being represented in the coordinate system of the 2D image information. Mapping of 2D candidate edges to 3D candidate edges is discussed in more detail in U.S. patent application Ser. No. 16/791,024, entitled "METHOD AND COMPUTTING SYSTEM FOR PROCESSING CANDIDATE EDGES" (Attorney Docket No. MJ0049-US/0077-0009US1), the entire contents of which are incorporated herein by reference.

As described above, the computing system 1100 may identify candidate edges from a 2D image or other 2D image information by identifying image locations (e.g., pixel locations) within the 2D image information where there is an abrupt change in image intensity (e.g., pixel intensity). In some implementations, the abrupt change may occur at a boundary between two image regions where one image region is darker than the other image region. For example, the two image regions may include a first image region and a second image region. The first image region may be a region of the 2D image that is darker than one or more immediately adjacent regions, which may include the second image region. The darkness of an image region may indicate how much reflected light has been detected or otherwise sensed from the corresponding physical region by a camera (e.g., 3200/3200A) generating the image information (e.g., 5600). More specifically, a dark image region may indicate that the camera has sensed a relatively small amount (or no) reflected light from the corresponding physical region. In some embodiments, the darkness of an image region may indicate how close the image intensity within the image region is to the minimum possible image intensity value (e.g., zero). In these implementations, a dark image region may indicate that the image intensity value(s) of the image region are closer to zero, and a less dark image region may indicate that the image intensity value(s) of the image region are closer to the maximum possible image intensity value.

In an embodiment, the second image region may have an elongated shape such as a rectangular band or a line or line segment. As an example, Figure 7A shows examples of image regions ₅₆₀₃₁ , ₅₆₀₃₂ , ₅₆₀₃₃ , _{56034, 56035 , 56036} that are darker than immediately adjacent image regions ₅₆₀₅₁ , ₅₆₀₅₂ , ₅₆₀₅₃ , 56054, ₅₆₀₅₅ , _{56056 ,} respectively. If image 5600 contains pixels, the image regions of Figure _7A may be pixel regions. Image regions _56031-56034 may each be a band of pixels, such as a rectangular band, and may have a width of multiple pixels, while image regions ₅₆₀₃₅ and ₅₆₀₃₆ may each be, or form, a line of pixels having a width of one pixel. As discussed above, a candidate edge may be formed by or based on a boundary between a first image region (e.g., one of image regions 5603 ₁ -5603 ₆ ) and a second image region (e.g., one of image regions 5605 ₁ -5605 ₆ ), where the first image region may be immediately adjacent to the second image region and may be darker than the second image region, and an abrupt change in image intensity may occur at the boundary between the two image regions. For example, FIG. 7B shows candidate edges 5601 ₁ -5601 ₆ defined or formed by respective boundaries between image regions 5603 ₁ -5603 ₆ and corresponding immediately adjacent image regions 5605 ₁ -5605 _6. As an example, computing system 1100 may identify candidate edge 5601 ₁ as a set of pixel locations that define a boundary between one image region 5605 ₁ and another dark image region 5603 ₁ . As an additional example, the computing system 1100 may identify a candidate edge 5601 ₅ as a set of pixel locations that define a boundary between one image region 5605 ₅ and a dark image region 5603 _5. In some instances, the pixel locations of the candidate edge 5601 ₅ may be located in the dark image region 5603 _5. More specifically, the candidate edge 5601 ₅ may be or may coincide with the image region 5603 ₅ , which may be, for example, a line of pixels having a width of a single pixel.

In an embodiment, computing system 1100 may detect or otherwise identify a candidate edge, such as one of candidate edges 5601 ₁ -5601 ₆ , based on an image edge detection technique, which may, for example, detect an abrupt change in image intensity. For example, computing system 1100 may be configured to detect candidate edges or other image information in the 2D image by applying a Sobel operator, a Prewitt operator, or other technique for determining intensity gradients in the 2D image, and/or by applying a Canny edge detector or other edge detection technique.

In an embodiment, where the computing system 1100 identifies an image region in a 2D image that is a band of pixels darker than one or more immediately adjacent image regions, the image region may in some circumstances be wide enough to form _more than a candidate edge. For example, Figure 7C shows computing system 1100 identifying additional candidate edges _56017-560110 based on image regions _56031-56034 . More specifically, _the additional candidate _{edges 56017-560110} may be respective sets of pixel locations that define respective boundaries between image regions _56031-56034 and immediately _adjacent image regions _{56071-56074 .} As a more specific example, image region 5603 ₂ in this example may be sufficiently wide such that image edge detection techniques identify a candidate edge 5601 ₂ formed by the boundary between one side (e.g., the right side) of image region 5603 ₂ and immediately adjacent region 5605 ₂ , as shown in FIG. 7B, and further identify another candidate edge 5601 ₈ formed by the boundary between the opposite side (e.g., the left side) of image region 5603 ₂ and immediately adjacent region 5607 ₂ , as shown in FIG. 7C. In an embodiment, the image region may be very narrow such that image edge detection techniques may identify only a single candidate edge from the image region. In such an embodiment, the image region may have a width of a single pixel, or a few pixels. For example, as described above, image region 5603 ₅ may form a line of pixels and may have a width of one pixel. In this example, the computing system 1100 may identify only a single candidate edge 5601 ₅ based on the image region 5603 ₅ , and the candidate edge 5601 ₅ may match the image region 5603 ₅ , for example, such that the image candidate edge 5601 ₅ may be a line of pixels that form the image region 5603 ₅ or may overlap.

Returning to FIG. 4, method 4000 may include step 4006, which may be performed in an embodiment when the plurality of candidate edges of step 4004 includes a first candidate edge formed by a boundary between a first image region and a second image region, where the first image region may be darker in image intensity than the second image region and may be immediately adjacent to the second image region. In this example, the first image region and the second image region may be regions described by a 2D image (e.g., 5600) or other image information. For example, FIGS. 7A-7C provide examples of a plurality of candidate edges 5601 ₁ -5601 _n , and the first candidate edge of step 4006 may be any one of the plurality of candidate edges 5601 ₁ -5601 _n . As discussed above, the first candidate edge may be formed by or based on a boundary between a first image region and a second bright image region. As an example, if the first candidate edge is candidate edge _5601-1 , the first image region may be image region _5603-1 and the second image region may be _5605-1 . As another example, if the first candidate edge is candidate edge 5601-2, the first image region may be image _{region 5603-2} and the second image region may be image region _5605-2 .

At step 4006, the computing system 1100 may determine whether the image information (e.g., 2D image 5600) satisfies a defined darkness condition at the first candidate edge (e.g., 5601 ₂ ). Such a determination may more specifically include, for example, determining whether the first image region (e.g., 5603 ₂ ) satisfies the defined darkness condition. In an embodiment, the defined darkness condition may be used to determine whether the first candidate edge (e.g., 5601 ₂ ) corresponds to an actual physical edge of the object (e.g., 3510) in the camera field of view (e.g., 3210/3210A) or whether the first candidate edge is a false edge.

In an embodiment, the defined darkness conditions may be used to detect image priors, or more specifically, dark priors. Image priors may refer to image features that have the potential to appear in an image during a particular situation or that may be expected in an image. More specifically, image priors may correspond to expectations, expectations, or predictions of what image feature(s) will be present in an image generated in such a situation, such as a situation where an image is generated to represent a group of boxes or other objects placed next to each other in a camera field of view. In some instances, dark priors may refer to image features that have a high level of darkness and/or have a spike-shaped image intensity profile (e.g., pixel intensity profile). A spike-shaped image intensity profile may involve a spike increase in darkness and/or a spike decrease in image intensity. A dark prior may correspond to a situation where a group of boxes or other objects in a camera field of view are placed close enough to each other that there are only narrow, physical gaps between some or all of the objects. More specifically, the dark prior may correspond to an expectation, expectation, or prediction that a physical gap will appear very dark in an image when the image is generated in such a situation to represent a group of objects. More specifically, the dark prior may correspond to an expectation or prediction that an image region in an image representing a physical gap will have a high level of darkness and/or may have a spike-shaped image intensity profile, as discussed in more detail below. In some implementations, the dark prior may be used to determine whether a candidate edge corresponds to a physical edge by evaluating whether an image region associated with the candidate edge corresponds to a physical gap between two objects.

In an embodiment, the defined darkness condition, which in some scenarios may be a condition for detecting a dark prior, may be based on a model of what the physical gap between two objects (e.g., 3510 and 3520 in FIGS. 3A-3C) is, and should or is likely to be visible in the 2D image, especially if the physical gap is narrow (e.g., less than 5 mm or less than 10 mm). For example, the defined darkness condition may be based on a Lambertian model of diffuse reflectance. Such a model of reflectance may estimate how light reflects from one or more surfaces or regions, especially surfaces or regions that cause diffuse reflection of incident light. Thus, the model may estimate the intensity of reflected light from a surface or surface region, and may indicate how bright or dark the surface or surface region is in an image generated by a camera (e.g., 3200/3200A) that senses the reflected light.

この実施例では、物体３５１０および３５２０は、互いに隣接して配置されてもよく、物体３５１０の物理的エッジ部３５１０Ｂと物体３５２０の物理的エッジ部３５２０Ｄとの間の空間であり得る狭い物理的ギャップｇによって分離され得る。図８のシナリオでは、画像（例えば、５６００）中の物理的ギャップｇの画像強度は、物体３５１０の表面３５１０Ｅ（例えば、側面）などの物理的ギャップを形成するさまざまな表面から反射光の量に依存し得る。拡散反射率のランバートモデルの下で、特定の表面からの反射光の強度は、ｃｏｓαに比例するか、または他の方法でそれ基づいてもよく、ここで、αは、入射光のベクトル

と表面（例えば、３５１０Ｅ）の法線ベクトル

との間の角度である。図８Ａでは、ベクトル

と表面３５１０Ｅの法線ベクトル

との間の角度αは、実質的に９０度であってもよく、その結果、表面３５１０Ｅからの反射光の強度が極めて低くてもよく、または一部の状況ではゼロであり得る。従って、二つの物体間の物理的ギャップを表す画像領域は、極めて高いレベルの暗さを有し得る。従って、ステップ４００６の定義された暗さ条件は、いくつかの実装では、画像領域が、二つの物体間の物理的ギャップを表すことを示すために十分に暗いかどうかを評価するために使用され得る、定義された暗さ閾値基準を含んでもよい。 As an example of how the Lambertian model is applied to a group of objects (e.g., 3510-3550 in FIGS. 3A-3C), FIG. 8 depicts a camera 3200A configured to generate an image (e.g., 5600) representing at least the objects 3510 and 3520 by sensing reflected light coming from various surfaces of the objects 3510, 3520. In some scenarios, the reflected light may be a reflection of emitted light from a light source 3600. More specifically, the light source 3600 may emit light toward the objects 3510, 3520 at least along a vector.

In this example, objects 3510 and 3520 may be positioned adjacent to one another and separated by a narrow physical gap g, which may be the space between physical edge 3510B of object 3510 and physical edge 3520D of object 3520. In the scenario of FIG. 8 , the image intensity of physical gap g in an image (e.g., 5600) may depend on the amount of reflected light from various surfaces that form the physical gap, such as surface 3510E (e.g., sides) of object 3510. Under the Lambertian model of diffuse reflectance, the intensity of reflected light from a particular surface may be proportional to or otherwise based on cos α, where α is the vector of incident light

and the normal vector of the surface (e.g., 3510E)

In FIG. 8A, the vector

and the normal vector of the surface 3510E

The angle α between may be substantially 90 degrees, such that the intensity of reflected light from surface 3510E may be extremely low, or in some circumstances zero. Thus, an image region representing a physical gap between two objects may have an extremely high level of darkness. Thus, the defined darkness condition of step 4006 may, in some implementations, include a defined darkness threshold criterion that may be used to evaluate whether an image region is sufficiently dark to indicate that it represents a physical gap between two objects.

In some circumstances, a physical gap may appear darker in its middle than its periphery. That is, if any reflected light leaves the physical gap, more reflected light may come from the periphery of the physical gap than from the middle of the physical gap. The periphery may refer to a location within the physical gap, for example, closer to physical edge 3520D or physical edge 3510B. In some scenarios, a peak level of darkness may occur in the middle of the physical gap. Thus, an image region representing a physical gap may have a spike-shaped image intensity profile (e.g., pixel intensity profile), where the image intensity profile has a spike increase in darkness or a spike decrease in image intensity within the image region. Thus, the darkness condition defined in step 4006 may, in some circumstances, include a spike intensity profile criterion defined to evaluate whether the image region has, for example, a spike-shaped image intensity profile (as opposed to a step-shaped image intensity profile).

In an embodiment, the defined darkness condition may be defined by one or more rules, criteria, or other information stored, for example, on the non-transitory computer-readable medium 1120 or elsewhere. For example, the information may define whether the darkness condition is met only by meeting a darkness threshold criterion, only by meeting a spike intensity profile criterion, only by meeting both criteria, or only by meeting either the darkness threshold criterion or the spike intensity profile criterion. In some instances, the information may be predefined manually or otherwise and stored on the non-transitory computer-readable medium 1120 such that the defined darkness condition may be a predefined darkness condition. In some instances, the information about the darkness condition may be dynamically defined.

In an embodiment, the defined darkness threshold criteria and/or the defined spike intensity profile criteria may be defined by information stored on the non-transitory computer readable medium 1120 or elsewhere. The information may be predefined such that the defined darkness threshold criteria and/or the defined spike intensity profile criteria may be a predetermined criteria or criteria. In an embodiment, the various predetermined thresholds or other predetermined values of the present disclosure may be manually defined as stored values on the non-transitory computer readable medium 1120 or elsewhere. For example, the defined darkness thresholds or defined depth difference thresholds discussed below may be values stored on the computer readable medium 1120. These may be predetermined values or may be dynamically defined.

9A-9C illustrate an embodiment for evaluating whether image information (e.g., 5600) satisfies a darkness condition defined by a candidate edge 5601 ₂ , and more specifically, whether image region 5603 ₂ satisfies the defined darkness condition. Image region 5603 ₂ may represent a physical gap between a first object, such as object 3510 of FIG. 8, and a second object, such as object 3520. In an embodiment, candidate edge 5601 ₂ may represent a physical edge 5610B of object 3510, and may be formed by or based on a boundary between image region 5603 ₂ and an immediately adjacent image region 5605 _2. In this example, image region 5603 ₂ may be a first image region and image region 5605 ₂ may be a second image region. More specifically, image region 5603 ₂ may be a first pixel region forming a band of pixels, while image region 5605 ₂ may be a second pixel region, and thus candidate edge 5601 ₂ may include or be formed by, for example, a set of pixel locations that define a boundary between the first pixel region and the second pixel region. As discussed above, computing system 1100 may identify candidate edge 5601 ₂ by, for example, detecting an abrupt change in image intensity (e.g., pixel intensity) between image regions 5603 ₂ , 5605 ₂ .

In an embodiment, a candidate edge is formed based on a boundary between a first image region and a second image region, and if the first image region is darker than the second image region, the computing system 1100 may determine that the first image region meets the defined darkness condition if it meets the defined spike intensity profile criteria. More specifically, the computing system may determine that the first image region (e.g., 5603 2 ) meets the defined spike intensity profile criteria if it has a particular shape to its image intensity profile (e.g., pixel intensity profile), such as a shape in _which the image intensity increases in darkness within the first image region toward a peak level of darkness at a location within the first image region, and then decreases in darkness. Such criteria may be consistent with a spike-shaped intensity profile, where the image intensity profile has a spike increase in darkness within the image region, or a spike decrease in intensity within the image region. Such criteria may be associated with detecting dark priors, where any physical gap that appears in an image is expected to appear darker in the middle of the gap relative to the perimeter of the gap.

FIG. 9B illustrates an image intensity profile 9001, or more specifically, a pixel intensity profile, that may meet the defined spike intensity profile criteria. More specifically, the image intensity profile may include information describing how image intensity, or more specifically, pixel intensity, varies as a function of image location, such as pixel location. In some embodiments, the image intensity profile may be represented by a curve or graph describing the value of the image intensity as a function of location in the image. For example, FIG. 9B depicts as image intensity profile 9001 a curve or graph describing the value of the image intensity, or more specifically, pixel intensity value, as a function of pixel location in a particular direction along axis 5609. Axis 5609 may be an axis extending across and along a width dimension of image region 5603 ₂ , and the direction along axis 5609 may be a particular direction along axis 5609. In the example of FIG. 9B, the width dimension may be aligned, for example, with the coordinate axis u of image 5600 of FIG. 5A, and the direction along axis 5609 may be the positive direction along which pixel coordinates [u,v] increase in value of u.

In an embodiment, the computing system 1100 can determine whether the image region 5603 ₂ meets the defined spike intensity profile criteria by determining whether the image region 5603 2 has a pixel intensity profile (e.g., 9001) that includes (i) a first profile portion (e.g., 9011) in which the image intensity (e.g., pixel intensity) increases in darkness in the first image region as a function of position along _a first direction (e.g., a positive direction along the axis 5609) and reaches a peak level of darkness (e.g., 9002) at a position u 1 in the first image region, followed by (ii) a second profile portion (e.g., 9012) in which the image intensity decreases in darkness in the first image region away from the peak level of darkness as a function of position along the same direction (e.g., a positive direction). The image intensity profile 9001 of FIG. _9B can more specifically be a spike-shaped intensity profile having a spike decrease in image intensity in the image region 9503 ₂ .

In some implementations, an image intensity profile of increasing darkness may correspond to an image intensity profile having values of decreasing image intensity. For example, an image (e.g., 5600) may have pixel intensity values that range from a minimum possible pixel intensity value (e.g., zero) to a maximum possible pixel intensity value (e.g., 255 for 8-bit coded pixel intensity values). In this example, lower pixel intensity values may have higher levels of darkness due to lower levels of brightness, while higher pixel intensity values may have lower levels of darkness due to higher levels of brightness. Further in this example, a peak level of darkness in the image intensity profile (e.g., 9002) may correspond to a minimum image intensity value in the image intensity profile (e.g., 9001).

In the above example, the computing system 1100 may determine whether an image region meets the defined spike intensity profile criteria by determining whether the image intensity profile has a shape where the image intensity values (e.g., pixel intensity values) begin decreasing in image intensity towards a minimum image intensity value and then switch to increasing image intensity away from the minimum image intensity value. For example, the image intensity profile 9001 of FIG. 9B may describe respective pixel intensity values for a series of pixels extending across the width dimension of the image region 5603 _2. The computing system 1100 may determine whether the image region 5603 ₂ meets the spike intensity profile criteria by determining whether the image intensity profile has a shape where the respective pixel intensity values decrease towards a minimum pixel intensity value in the image region 5603 ₂ and then switch to increasing in image region 5603 ₂ away from the minimum pixel intensity value. In this example, the minimum pixel intensity value may correspond to a peak level of darkness 9002 in the image intensity profile 9001.

In an embodiment, a candidate edge is formed based on a boundary between a first image region and a second image region, and if the first image region is darker than the second image region, meeting a defined darkness threshold criterion may involve a comparison with a defined darkness threshold. Such a criterion may correspond to detecting a dark prior, where any physical gaps present in the image are expected to be very dark in appearance. Figure 9C shows another image intensity profile 9003 for image region 5603 ₂ and an immediately adjacent image region. In this example, the first image region may be image region 5603 2 that is immediately adjacent to a second lighter image region (e.g., 5605 ₂ ), while the dark image region 5603 ₂ may be the first _{image region. As mentioned above, image region 5603 2 may} form a band of pixels. 9C, the computing system 1100 may determine whether the image region 5603 ₂ meets the defined darkness threshold criteria by determining whether the image region 5603 ₂ has at least one portion with an image intensity darker than the defined darkness threshold τ _{dark_prior} . As discussed above, a higher level of darkness may correspond to a lower image intensity value in some examples. In such an example, the computing system 1100 may determine whether the image region 5603 ₂ has an image intensity profile with image intensity values (e.g., pixel intensity values) that are less than the defined darkness threshold τ _{dark_prior} . In some circumstances, the computing system 1100 may more specifically determine whether the minimum intensity value of the image intensity profile 9003 is less than or equal to the defined darkness threshold τ _{dark_prior} , which may correspond to a peak level of darkness 9004 of the intensity profile 9003. In an embodiment, if an image region 5603 ₂ has an image intensity profile 9003, it may satisfy both the defined darkness threshold criteria and the defined spike intensity profile criteria.

In an embodiment, the computing system 1100 may determine that the defined darkness condition is met for a candidate edge and/or image region if at least one of the defined darkness threshold criteria or the defined spike intensity profile criteria is met, such that any one of the above criteria may be used to meet the defined darkness condition. In an embodiment, the computing system 1100 may determine that the defined darkness condition is met only in response to a determination that the defined spike intensity profile criteria is met (regardless of whether the defined darkness threshold criteria is met), only in response to a determination that the defined darkness threshold criteria is met (regardless of whether the defined spike intensity profile criteria is met), or only in response to a determination that both the defined darkness threshold criteria and the defined spike intensity profile criteria are met.

In an embodiment, the computing system 1100 may identify candidate edges (e.g., 5601 ₂ ) based on 2D image information, such as 2D image 5600, and determine whether the candidate edges meet a defined darkness condition. As described above, when the computing system 1100 receives both 2D image information and 3D image information, the computing system 1100 may use the 2D image information to compensate for limitations of the 3D image information or compensate for the lack of 3D image information, or vice versa. For example, when a camera (e.g., 3200B) generates 3D image information to represent a group of objects, the 3D image information may lack information to distinguish individual objects in the group, especially when the group of objects has equal depth values relative to the camera. More specifically, the 3D image information may lack information to detect narrow physical gaps between objects and therefore may have limited usefulness in identifying physical edges associated with physical gaps.

As an example, Figure 9D illustrates depth values associated with portion 5715 of 3D image information 5700 of Figure 5B. More specifically, portion 5715 may describe respective depth values for physical locations 5720a _{through 5720a +5} on the top surface of object 3520, and locations 5710b through 5710b ₊₄ on the top surface of object 3510. These physical locations may be mapped to or otherwise correspond to image locations within or around image region ₅₆₀₃₂ , or around candidate edge _56012. As shown in Figure 8, image region ₅₆₀₃₂ may represent a physical gap g between objects 3510, 3520. As discussed above, computing system 1100 may use the 3D image information to attempt to detect one or more locations where there is an abrupt change in depth. However, the physical gap of Figure 8 may be too narrow or otherwise too small for the resolution of the 3D image information captured by the 3D image information 5700. Thus, in the example of Figure 9D, the computing system 1100 may determine that there is no abrupt change in depth at locations _5720a through 5720a ₊₅ and _5710b through 5710b ₊₄ , and therefore determine that the 3D image information does not show any candidate edges at those locations. Furthermore, in some circumstances, the 3D image information may lack depth information for some of locations 5720a _through 5720a _{+5 and} 5710b through 5710b ₊₄ , or more specifically, for one or more locations that correspond to the candidate edge _5601-1 . In some circumstances, a portion of the 3D image information that maps to or otherwise corresponds to the candidate edge _5601-1 may be affected by a level of imaging noise that is greater than a defined noise tolerance threshold, which may be a value defined in the non-transitory computer-readable medium 1120. In the above example, the 2D image information may compensate for these limitations of the 3D image information because the 2D image information may include candidate edge 5601 ₁ that represents a physical edge 3510B of object 3510, and may also include image regions 5603 ₂ that do not represent a physical gap between object 3510 and object 3520. In the above example involving limitations of the 3D image information, the computing system 1100 may further use the defined darkness condition to determine whether candidate edge 5601 ₂ should be used to represent one of the physical edges (e.g., 3510B) of the group of objects.

10A-10C show an example for determining whether a candidate edge 5601 ₄ and/or image region 5603 ₄ meets a defined darkness condition. Image region 5603 ₄ may represent visual marking 3512 of FIG. 3B, such as a visible line printed on the top surface of object 3510. In this example, image region 5603 ₄ may be darker than immediately adjacent image regions, such as image regions 5605 ₄ and 5607 _4. Candidate edge 5601 ₄ may be formed based on the boundary between dark image region 5603 ₄ and immediately adjacent region 5605 ₄ .

10B illustrates image region 5603.4 having an image intensity profile 10001 that does not meet the defined spike intensity profile criteria because image intensity profile ₁₀₀₀₁ does not vary within image region 5603.4 in a manner that increases in darkness as a function of position toward a peak level of darkness and then decreases in darkness away from the peak level of darkness. Image intensity profile 10001 in this example may describe pixel intensity values as a function of pixel position along axis ₅₆₀₈ , which may be aligned with the width dimension of image region _5603.4 . As discussed above, the spike intensity profile criteria may correspond to an expectation that a physical gap between physical edges of objects may appear darker in the middle of the physical gap relative to the perimeter of the physical gap. Thus, an image region representing a physical gap may have an image intensity profile that varies as a function of image position within the image region, more specifically, increasing in darkness as a function of image position along a particular direction from image locations corresponding to the periphery of the physical gap to image locations corresponding to the center of the physical gap, and then decreasing in darkness as a function of position along the same direction. More specifically, the image intensity profile may have a spike of increasing darkness or a spike of decreasing image intensity in the image region representing the physical gap. In embodiments, an image region representing a visual line or other visual marking instead may lack such an image intensity profile, and instead have a more uniform level of darkness within the image region. Thus, as shown in FIG. 10B, an image region _{5603_4} representing a portion of a visual marking 3512 may have an image intensity profile 10001 that is more uniform within image region _{5603_4} , such that profile 10001 does not vary substantially within image region _{5603_4} . Additionally, image intensity profile 10001 may have a step-shaped change in image intensity at the boundaries of image region _{5603_4} from the image intensity associated with a brighter adjacent image region (e.g., _{5605_4} ) to a uniform image intensity within image region _{5603_4} . Thus, image intensity profile 10001 does not have a shape that begins by increasing darkness along a particular direction across image region _{5603_4} to a peak level of darkness and then switches to decreasing darkness along that direction. More specifically, image intensity profile 10001 does not exhibit a spike decrease in image intensity. Thus, computing system 1100 of this example may determine that image region _{5603_4} does not satisfy the defined spike intensity profile criteria, which may result in a determination that 2D image 5600 does not satisfy the defined darkness condition at candidate edge _{5601_4} and/or image region _{5603_4} .

10C depicts image region _{5603_4} having an image intensity profile 10003 that may fail to meet a defined darkness threshold criterion because image intensity profile 10003 may indicate that image region _{5603_4} is not dark enough. More specifically, computing system 1100 may determine that most or all of the pixel intensity values in image intensity profile 10003 of image region 5603_4 exceed a defined darkness threshold τ _{dark_prior} . Thus, computing system 1100 of FIG. _10C may determine that image region _{5603_4} does not meet a defined darkness threshold criterion, which may result in a determination that image 5600 does not meet a defined darkness condition at candidate edge _{5601_4} and/or image region _{5603_4} . Image intensity profile 10003 may also fail to meet a defined spike intensity profile criterion, as discussed above with respect to FIG. 10B.

In embodiments, an image region may have a width that is too small to make a reliable assessment of whether the image region meets the defined spike intensity profile criteria. For example, the image region may be only a single pixel wide, or only a few pixels wide. In some instances, the computing system 1100 may determine that such an image region does not meet the defined darkness condition. In other instances, the computing system 1100 may determine whether the image region meets the defined darkness condition based on whether the image region meets the defined darkness threshold criteria. In some instances, the computing system 1100 may decide not to evaluate such an image region or associated candidate edges for the defined darkness condition.

As mentioned above, one aspect of the disclosure relates to situations where the computing system 1100 identifies a plurality of candidate edges based on at least 2D image information, such as the 2D image 5600. In such an embodiment, the plurality of candidate edges may include at least a first candidate edge (e.g., 5601 ₁ /5601 ₂ /5601 ₃ /5601 ₄ ) identified based on the 2D image. For example, the first candidate edge may be formed based on a boundary between two image regions of the 2D image. In some instances, the computing system 1100 may identify a plurality of candidate edges based on the 2D image information and the 3D image information. In such an example, the plurality of candidate edges may include a first candidate edge from the 2D image information, as described above, and may further include a second candidate edge (e.g., 5701 ₁ of FIG. 6B ) identified based on the 3D image information.

11A-11B illustrate computing system 1100 identifying candidate edge _5701.1 as a second candidate edge of a plurality of candidate edges based on 3D image information 5700. In this example, computing system 1100 may identify candidate edge _5701.1 based on detecting an abrupt change in depth at candidate edge _5701.1 between first portion 5710A and second portion 5750A of 3D image information 5700. First portion 5710A may represent, for example, a region of locations on a top surface of object 3510 of FIGS. 3A-3C, while second portion 5750A may represent, for example, a region of locations on a top surface of object 3550. More specifically, as shown in Figure 11B, a first portion 5710A of the 3D image information may include respective depth values for locations _5710c through 5710c ₊₅ on the top surface of the object 3510, while a second portion 5750A may include respective depth values for locations 5750d _through 5750d ₊₄ on the top surface of the object 3550. In this example, locations 5710c _through 5710c ₊₅ and _5750d through 5750d ₊₄ may be a series of locations aligned along the Y-axis in Figure 11A.

In an embodiment, the computing system 1100 may identify the candidate edge 5701 1 of FIG. 11B based on detecting an abrupt change in depth between two consecutive or otherwise adjacent positions in the sequence of positions described by the 3D image information 5700. Such an abrupt change may be referred to as a depth discontinuity. The abrupt change may be detected, for example, when the difference between the respective depth values of the two positions exceeds a defined depth difference threshold. For example, _the computing system 1100 may determine that the difference between the depth value of the position 5710 _c+5 and the depth value of the position 5750 _d exceeds a defined depth difference threshold. As a result, the computing system 1100 may identify the candidate edge 5701 ₁ based on these two positions 5710 _c+5 , 5750 _d . For example, the candidate edge 5701 ₁ may be identified to include a position that is between the positions 5710 _c+5 , 5750 _d on the Y-axis.

In an embodiment, the computing system 1100 may identify candidate edges by identifying two surfaces having a depth difference that exceeds a defined depth difference threshold based on the 3D image information. For example, as shown in FIG. 11C, the computing system 1100 may identify a first surface of the group of objects 3510-3550 based on a first set of locations described by the 3D image information 5700, the first set of locations having respective depth values that do not deviate from each other by more than a defined measurement variance threshold. Similarly, the computing system 1100 may identify a second surface of the group of objects 3510-3550 based on a second set of locations described by the 3D image information 5700, the second set of locations having respective depth values that do not deviate from each other by more than a defined measurement variance threshold. In the example of FIG. 11C , the first set of positions may include positions 5710 _c through 5710 _c+5 , which may represent the top surface of object 5710, while the second set of positions may include positions 5750 _d through 5750 _d+4 , which may represent the top surface of object 5750.

In this embodiment, the defined measurement variance threshold may describe the effects of imaging noise, manufacturing tolerances, or other factors that may result in random variations in the depth measurements taken by the camera (e.g., 3200B). Such sources of random variation result in some natural variation in the depth values of various locations, even if the various locations are part of a common surface and actually have the same depth relative to the camera. In some instances, the defined measurement variance threshold may be equal to or based on a nominal standard deviation, which is used to describe the expected random variation in the depth measurements, or more broadly, how susceptible the camera is to noise or other error sources. The nominal standard deviation may describe a baseline standard deviation or other form of variance expected in the depth values or other depth information generated by the camera. The nominal standard deviation, or more broadly, the defined measurement variance threshold, may be a value stored, for example, in the non-transitory computer-readable medium 1120, and may be a predetermined value or a dynamically defined value. In an embodiment, the computing system 1100 may determine that a set of locations are part of a common surface if the set of locations have respective depth values that do not deviate from one another by more than a defined measurement variance threshold. In a more specific embodiment, the computing system 1100 may determine that a set of locations are part of a common surface if the standard deviation (e.g., Std ₅₇₁₀ or Std ₅₇₅₀ ) of their respective depth values is less than a defined measurement variance threshold.

In the above embodiment, the computing system 1100 may identify candidate edges from the 3D image information based on two surfaces having sufficient depth difference. For example, the first set of locations in FIG. 11C describing the top surface of the object 5710 may have or be otherwise associated with an average depth value Avg _5710. Similarly, the second set of locations describing the top surface of the object ₅₇₅₀ may have or be otherwise associated with an average depth value Avg 5750. The computing system 1100 may determine whether the difference between Avg ₅₇₁₀ and Avg ₅₇₅₀ is greater than or equal to a defined depth difference threshold. In some instances, the defined depth difference threshold may be determined as a multiple of a defined measurement variance threshold (e.g., twice the defined measurement variance threshold or five times the defined measurement variance threshold). If the difference between Avg ₅₇₁₀ and Avg ₅₇₅₀ associated with the two surfaces is greater than or equal to a defined depth difference threshold, then computing system 1100 may determine that a depth discontinuity condition is met. More specifically, computing system 1100 may determine that a candidate edge (e.g., 5701 ₁ ) exists at a location between the two surfaces, and more specifically may identify the candidate edge based on the location at which the transition between the two surfaces is located.

As mentioned above, one aspect of the present disclosure relates to using 2D image information and 3D image information to compensate for each other, such that the 3D image information can compensate for the limitations of the 2D image information (and vice versa). In some instances, a physical edge detected from the 3D image information can be associated with a higher level of confidence than a physical edge detected only from the 2D image information. In some instances, when a physical edge (e.g., 3510A in FIG. 3B ) is represented in both the 2D image information and the 3D image information, the computing system 1100 can identify a candidate edge (e.g., 5601 ₅ ) that represents a physical edge of the 2D image information and identify a corresponding candidate edge (e.g., 5701 ₁ ) that represents a physical edge of the 3D image information. As mentioned above, corresponding candidate edges can map to each other. For example, a candidate edge in the 3D image information (e.g., 5701 ₁ ) can be mapped to a candidate edge in the 2D image information (e.g., 5601 ₅ ). A candidate edge (e.g., 5601 ₅ ) may be formed based on a boundary between two image regions, such as 5605 ₅ and 5650. However, in some circumstances, the computing system 1100 may not be able to determine with a high level of confidence whether a candidate edge (e.g., 5601 ₅ ) from the 2D image information corresponds to an actual physical edge. For example, in FIG. 11D , the 2D image 5600 may have a step-shaped change in image intensity at the candidate edge 5601 _5. In this example, the computing system 1100 may determine that the 2D image 5600 does not meet a defined darkness condition at the candidate edge 5601 ₅ , or more specifically, at the two image regions 5605 ₅ and 5650. Thus, the computing system 1100 may determine that there is not a sufficiently high level of confidence associated with the candidate edge 5601 ₅ that represents a physical edge.

In such a situation, the computing system 1100 may use the 3D image information to provide additional input. More specifically, the computing system 1100 may identify the candidate edge 5701 ₁ based on the 3D image information as described above with respect to FIGS. 11A-11C, and the candidate edge 5601 ₅ in the 2D image information may be mapped to or otherwise correspond to the candidate edge 5701 ₁ in the 3D image information. In some instances, since the candidate edge 5701 ₁ is identified based on the depth information, the computing system 1100 may determine that there is a sufficiently high probability that a physical edge, i.e., the candidate edge 5701 ₁ representing the physical edge 3510A of FIG. 3B, exists. Thus, the 2D image information may not lead to the detection of the physical edge 3510A or may lead to the detection of the physical edge 3510A with a low confidence level, while the 3D image information may be used by the computing system 1100 to detect the physical edge 3510A with a higher confidence level.

Returning to FIG. 4, method 4000 may include step 4008, in which in one embodiment, computing system 1100 may select a subset of the plurality of candidate edges (e.g., a subset of the plurality of candidate edges 5601 ₁ -5601 _n ) to form a selected subset of candidate edges for representing a physical edge of the group of objects (e.g., 3510-3540). In an embodiment, this step may involve excluding from the subset one or more candidate edges, each of which is likely to be a false edge. The one or more candidate edges that are likely to be false edges may be removed from the subset of candidate edges, or more generally, may be ignored from further consideration for representing a physical edge of the group of objects (e.g., 3510-3540). In one example, computing system 1100 may select the subset of the plurality of candidate edges by determining which candidate edge(s) to remove from the plurality of candidate edges, which, after removal, form the resulting subset. In one example, when multiple candidate edges are represented or described by the edge detection information 1126 of FIG. 2C , removing a candidate edge may involve deleting information about that candidate edge from the edge detection information 1126.

As discussed above, the plurality of candidate edges (e.g., 5601 ₁ -5601 _n , or 5601 ₁ -5601 _n and 5701 ₁ -5701 _n ) may include at least a first candidate edge (e.g., 5601 ₁ or 5601 ₄ ) formed based on a boundary between a first image region and a second image region that is darker than the first image region. Additionally, the first candidate edge may be identified from the 2D image information. In an embodiment, step 4008 may involve determining whether to include the first candidate edge in the subset (also referred to as a subset of candidate edges). Including the first candidate edge (e.g., 5601 ₁ ) in the subset may enable the first candidate edge to be used to represent at least one physical edge (e.g., 3510B) of a group of objects in the camera field of view. More specifically, if the first candidate edge (e.g., 5601 ₁ ) is included in the subset, such inclusion may be an indication that the first candidate edge (e.g., 5601 ₁ ) is a candidate that remains under consideration for representing at least one of the physical edges of the group of objects. In other words, the computing system 1100 may determine whether to retain the first candidate edge as a candidate for representing at least one physical edge. If the computing system 1100 determines to retain the first candidate edge as such a candidate, it may include the first candidate edge in the subset (which may also be referred to as a selected subset of the candidate edges). This determination may be part of the step of selecting a subset of the multiple candidate edges, and may be made based on whether the image satisfies a darkness condition defined by the first candidate edge. In some instances, the inclusion of the first candidate edge in the subset may be an indication that the first candidate edge is sufficiently unlikely to be a false edge. In some cases, including the first candidate edge in the subset may indicate that the first candidate edge (e.g., 5601 1 ) has a sufficiently high confidence level that it corresponds to an actual physical edge of the group of objects such that the computing system 1100 continues to use the first candidate edge to represent at least one physical edge of the group of objects or to consider at least the first candidate edge to represent at least one physical edge of the group of objects. If the computing system 1100 determines not to include the first candidate edge (e.g., 5601 ₄ ) in the subset such that the first candidate edge (e.g., 5601 ₄ ) is filtered or otherwise removed from the subset, such removal may be an indication that the first candidate edge (e.g., 5601 ₄ ) is no longer a candidate for representing at least one physical edge of the group of objects. In some instances, removal _of the first candidate edge from the subset may be an indication that the first candidate edge (e.g., 5601 ₄ ) is likely to be a false edge.

In an embodiment, the determination of whether to include the first candidate edge in the selected subset of candidate edges may be based on whether the image information (e.g., 5600) satisfies the darkness condition defined by the first candidate edge, as described above. In some implementations, if the image information satisfies the darkness condition defined by the first candidate edge, such a result may indicate that the first candidate edge is sufficiently unlikely to be a false edge, because the first candidate edge in such a situation is likely to be associated with an image region that represents a physical gap between two objects. Thus, the first candidate edge may represent a physical edge that forms one side of the physical gap. In such a situation, the computing system 1100 may determine to include the first candidate edge in the selected subset. In some instances, if the image information does not satisfy the darkness condition defined by the first candidate edge, the computing system 1100 may determine not to include the first candidate edge in the selected subset. In some instances, if the computing system determines that the 2D image information does not satisfy the darkness condition defined by the first candidate edge, the computing system 1100 may further evaluate the first candidate edge using the 3D image information. For example, if the computing system 1100 determines that the 2D image 5600 does not meet the darkness condition defined by the candidate edge 5601 ₅ , the computing system 1100 can determine whether that candidate edge 5601 _{5 maps to a candidate edge 5701 1 described} by the 3D image information, and whether the candidate edge 5701 ₁ of the 3D image information exhibits a depth change greater than a defined depth difference threshold, as discussed above with respect to FIGS. 11A-11D.

In an embodiment, method 4000 may perform steps 4006 and/or 4008 multiple times (e.g., via multiple iterations) to determine whether the image information satisfies the defined darkness condition at multiple candidate edges and select the subset discussed above based on these determinations. As an example, if the multiple candidate edges include at least candidate edges 5601 ₁ -5601 _n , computing system 1100 may perform step 4006 multiple times to determine whether 2D image 5600 satisfies the defined darkness condition at, for example, candidate edges 5601 ₁ -5601 _n . Computing system 1100 may further perform step 4008 multiple times to determine which of these candidate edges are included in the subset and remain candidates for representing a physical edge, and which of these candidate edges are excluded from the subset and therefore are no longer candidates for representing a physical edge. For example, the computing system 1100 may determine that the subset includes candidate edges 5601 ₁ and 5601 ₂ because the 2D image 5600 meets the darkness conditions defined at those candidate edges, and that the subset does not include candidate edges 5601 ₃ and 5601 ₄ because the 2D image does not meet the darkness conditions defined at those candidate edges. In some circumstances, the computing system 1100 may determine not to include candidate edge 5601 ₅ in the subset because the 2D image 5600 cannot meet the darkness conditions defined at the candidate edges. In some circumstances, the computing system 1100 may determine to still include candidate edge 5601 ₅ in the subset if candidate edge 5601 ₅ maps to candidate edge 5701 ₁ of the 3D image information that exhibits a depth change that exceeds a depth difference threshold.

In an embodiment, method 4000 may include a step in which computing system 1100 outputs a robot interaction move command. The robot interaction move command may be used for robot interaction between a robot (e.g., 3300) and at least one object of a group of objects (e.g., 3510-3550). The robot interaction may involve, for example, the robot (e.g., 3300) picking up an object (e.g., a box) from a pallet, moving the object to a destination location, unloading the object from the pallet, or performing other actions.

In an embodiment, the robot interaction movement command may be generated based on the selected subset of the candidate edges of step 4008. For example, the computing system 1100 may use the selected subset of the candidate edges to distinguish individual objects from among a group of objects described by the image information. In some instances, the computing system 1100 may use the selected subset to perform segmentation of the image information. For example, if the image information includes a point cloud, the computing system may use the selected subset of the candidate edges to perform point cloud segmentation, which may involve identifying portions of the point cloud that correspond to individual objects from among a group of objects. Point cloud segmentation is described in U.S. Patent Application Serial No. 16/791,024 (Attorney Docket No. MJ0049-US/0077-0009US1), the entirety of which is incorporated herein by reference. In one example, if the image information includes 2D image information, the computing system 1100 may use the selected subset of the candidate edges to separate portions of the 2D image information that correspond to individual objects from among a group of objects. The separated portions may be used, for example, as target images or target image portions used to perform object recognition or object registration operations (e.g., by module 1128). Object registration and object recognition are discussed in more detail in U.S. Patent Application No. 16/991,466 (Attorney Docket No. MJ0054-US/0077-0012US1) and U.S. Patent Application No. 17/193,253 (Attorney Docket No. MJ0060-US/0077-0017US1), the entire contents of which are incorporated herein by reference. In such examples, robot interaction movement commands may be generated based on the results of the object recognition or object registration operations. For example, the object recognition operation may generate a detection hypothesis, which may be an estimate of which object or object type is represented by the image information, or a portion thereof. In some instances, the detection hypothesis may be associated with an object recognition template, which may include information describing the physical structure of one of the objects 3510-3540, for example. This information may be used by the computing system 1100 to plan the movements of a robot (e.g., 3300) to retrieve and move objects (e.g., via module 1129).

While the above steps of method 4000 are illustrated with respect to objects 3510-3550 in Figures 3A-3C, Figure 12A illustrates the above steps with respect to object 12510, while Figure 13A illustrates the above steps with respect to objects 13510-13520. In an embodiment, object 12510 in Figure 12A may be a box having a top surface with a first physical area 12512 that is darker than a second, immediately adjacent physical area 12514. For example, first physical area 12512 may have more ink printed thereon compared to second physical area 12514. Object 12510 may be placed on object 12520, which may be a pallet on which the box is placed. Figure 12B illustrates a 2D image 12600 that may be generated to represent object 12510. More specifically, the 2D image 12600 may include a first image region 12603 representing a first physical region 12512, and may include a second image region 12605 representing a second physical region 12514. The computing system 1100 of the present embodiment may identify a first candidate edge 12601 ₁ based on a boundary between the first image region 12603 and the second image region 12605.

In an embodiment, the computing system 1100 may determine that the 2D image 12600 does not satisfy the darkness condition defined at the first candidate edge 12601 _1. For example, the computing system 1100 may determine that the 2D image 12600 has an image intensity profile 12001 that has a step-shaped change in image intensity at the first candidate edge 12601 _1. The image intensity profile may be measured along an axis 12609 that extends along the u-axis of the image. In some implementations, the computing system 1100 may determine that the image intensity profile 12001, or more specifically, image regions 12603 and 12605, do not satisfy the spike intensity profile criteria. The computing system 1100 may further determine that the defined darkness condition is not satisfied at the first candidate edge 12601 _1. As a result, the computing system 1100 may remove the first candidate edge 12601 ₁ from the edge detection information 1126.

In the embodiment of Fig. 13A, objects 13510 and 13520 may each be a box and may be placed on object 13530, which may be a pallet or other platform. In this embodiment, object 13510 may be darker than object 13520 (e.g., as a result of being made of darker cardboard or other material). Furthermore, the two objects may be separated by a narrow physical gap g. Fig. 13B shows a 2D image 13600 including a first image region 13603 representing a first object 13510 and a second image region 13605 representing a second object 13530. The computing system 1100 of this embodiment may identify candidate edges 13601 ₁ based on the boundary between the two image regions 13605, 13605.

As shown in Figures 12B and 13B, images 12600 and 13600 may have similar appearances. However, as shown in Figure 13C, image 13600 may have an image intensity profile that includes a spike decrease in image intensity. More specifically, image region 13603 of image 13600 may include image region _13603-1 to represent the physical gap g between objects 13510, 13520, and may include image region _13603-2 to represent object 13520. In this embodiment, image region _13603-1 may include a spike decrease in image intensity and may have a minimum pixel intensity value that is less than a defined darkness threshold. Thus, computing system 1100 may determine that image region _13603-1 meets the defined spike intensity profile criteria and/or the defined darkness threshold criteria. As a result, the computing system 1100 may determine that the image 13600 satisfies the darkness condition defined by the first candidate edge 13603 _1. Thus, the computing system 1100 may determine that the first candidate edge 13603 ₁ is used to represent one of the physical edges of the objects 13510, 13520.

Additional considerations regarding various embodiments:

Embodiment 1 includes a computing system or a method implemented by the computing system. The computing system may include a communications interface and at least one processing circuit. The communications interface may be configured to communicate with a robot and a camera having a camera field of view. The at least one processing circuit receives image information representing a group of objects generated by the camera when the group of objects is within the camera field of view, and identifies from the image information a plurality of candidate edges associated with the group of objects, the plurality of candidate edges being or including respective sets of image locations or physical locations that form respective candidates for representing physical edges of the group of objects, and if the plurality of candidate edges includes a first candidate edge formed based on a boundary between a first image region and a second image region, determines whether the image information satisfies a darkness condition defined by the first candidate edge, the first image region being darker than the second image region, and the first image region and the second image region being darker than the respective candidate edges described by the image information. The image processing system may be configured to perform the following: selecting a subset of the plurality of candidate edges to form a selected subset of candidate edges for representing a physical edge of the group of objects, where selecting includes determining whether to retain the first candidate edge as a candidate for representing at least one of the physical edges of the group of objects by including the first candidate edge in the selected subset of candidate edges based on whether the image information satisfies a darkness condition defined by the first candidate edge; and outputting a robot interaction move command, where the robot interaction move command is for a robot interaction between the robot and at least one object of the group of objects and is generated based on the selected subset of candidate edges. In this embodiment, the at least one processing circuit is configured to determine that the image information satisfies the darkness condition defined by the first candidate edge in response to determining that the first image region satisfies at least one of the defined darkness threshold criterion or the defined spike intensity profile criterion. Further in this embodiment, the at least one processing circuit is configured to determine whether the first image region satisfies the defined darkness threshold criterion by determining whether the first image region has at least one portion with an image intensity darker than the defined darkness threshold. Further, in this embodiment, at least one processing circuit is configured to determine whether the first image region meets the spike intensity profile criteria by determining whether the first image region has an image intensity profile including (i) a first profile portion where the image intensity increases in darkness within the first image region as a function of position and reaches a peak level of darkness at a position within the first image region, followed by (ii) a second profile portion where the image intensity decreases in darkness within the first image region away from the peak level of darkness as a function of position.

Embodiment 2 includes the computing system of embodiment 1, where the first image region is a first pixel region forming a band of pixels representing a physical gap between a first object and a second object of a group of objects, and the second image region is a second pixel region immediately adjacent to the first pixel region such that a boundary forming a first candidate edge is between the first pixel region and the second pixel region.

Embodiment 3 includes the computing system of embodiment 2, where at least one processing circuit is configured to determine whether the first image region meets a defined darkness threshold criterion by determining whether the first image region has pixel intensity values less than a defined darkness threshold.

Embodiment 4 includes the computing system of embodiment 2 or 3, wherein an image intensity profile of the first image region describes pixel intensity values for each of a series of pixels extending across a width dimension of the first image region, and at least one processing circuit is configured to determine whether the first image region meets a spike intensity profile criterion by determining whether the image intensity profile has a shape in which each pixel intensity value decreases toward a minimum pixel intensity value in the first image region and then switches to increase away from the minimum pixel intensity value, the minimum pixel intensity value being associated with a peak level of darkness in the first image region.

Embodiment 5 includes a computing system according to any one of embodiments 1 to 4, in which at least one processing circuit is configured to determine that the first image region meets a defined darkness condition only in response to determining that the first image region meets the spike intensity profile criterion.

Embodiment 6 includes a computing system according to any one of embodiments 1 to 5, wherein at least one processing circuit is configured to determine that the first image region meets a defined darkness condition only in response to determining that the first image region meets a defined darkness threshold criterion.

Embodiment 7 includes the computing system of embodiment 1, where at least one processing circuit is configured to determine that the first image region meets the defined darkness condition only in response to determining that the first image region meets both the defined darkness threshold criterion and the defined spike intensity profile criterion.

Embodiment 8 includes the computing system of any one of embodiments 1 to 7, and at least one processing circuit is configured to identify a first candidate edge formed based on a boundary between a first image region and a second image region based on the 2D image information when the image information includes 2D image information and 3D image information, and the 3D image information includes depth information of a position within the camera field of view.

Embodiment 9 includes the computing system of embodiment 8, wherein at least one processing circuit is configured to determine whether to retain a first candidate edge as a candidate for representing at least one physical edge of a group of objects when (i) the 3D image information is missing depth information at one or more locations corresponding to the first candidate edge, and (ii) a portion of the 3D image information corresponding to the first candidate image is subject to a level of imaging noise greater than a defined noise tolerance threshold.

Embodiment 10 includes the computing system of embodiment 8 or 9, wherein at least one processing circuit is configured to determine whether to retain a first candidate edge as a candidate for representing at least one of the physical edges of the group of objects when the 3D image information does not satisfy a defined depth discontinuity condition at one or more locations corresponding to the first candidate edge.

Embodiment 11 includes the computing system of embodiment 10, wherein at least one processing circuit is configured to determine that the 3D image information does not satisfy a defined depth discontinuity condition at one or more locations corresponding to a first candidate edge in response to determining that the 3D image information does not describe a depth change at one or more locations that exceeds a defined depth difference threshold.

Embodiment 12 includes any one of the computing systems of embodiments 8 to 11, in which at least one processing circuit is configured to identify a second candidate edge of the plurality of candidate edges based on the 3D image information.

Embodiment 13 includes the computing system of claim 12, wherein at least one processing circuit is configured to identify a second candidate edge based on the 3D image information by: identifying a first surface of the group of objects based on a first set of locations described by the 3D image information having respective depth values that do not deviate from each other by more than a defined measurement variance threshold; identifying a second surface of the group of objects based on a second set of locations described by the 3D image information having respective depth values that are within the defined measurement variance threshold; determining an average depth value associated with the first surface as the first average depth value; determining an average depth value associated with the second surface as the second average depth value; and, in response to determining that the difference between the first average depth value and the second average depth value exceeds the defined depth difference threshold, identifying a second candidate edge based on a location at which there is a transition between the first surface and the second surface.

Embodiment 14 includes the computing system of embodiment 12 or 13, wherein at least one processing circuit is configured to identify a second candidate edge based on the 3D image information when the second candidate edge is mapped to a candidate edge in the 2D image information and formed based on a boundary between two image regions that do not satisfy a defined darkness condition.

Embodiment 15 includes the computing system of embodiments 1-14, wherein at least one processing circuit is configured to perform an object recognition operation or an object registration operation based on a selected subset of the candidate edges, and a robot interaction move command is generated based on a result of the object recognition operation or the object registration operation.

Embodiment 16 includes the computing system of any one of embodiments 1 to 15, wherein at least one processing circuit is configured to select a subset of the plurality of candidate edges by determining which candidate edges to filter from the plurality of candidate edges, and the plurality of candidate edges form the subset of candidate edges after filtering.

It will be apparent to those skilled in the relevant art that other suitable modifications and adaptations to the methods and applications described herein can be made without departing from the scope of any of the embodiments. The embodiments described above are illustrative examples, and the present invention should not be construed as being limited to these particular embodiments. It should be understood that the various embodiments disclosed herein may be combined in different combinations than those specifically presented in the description and accompanying figures. It should also be understood that, by way of example, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, added, combined, or omitted entirely (e.g., not all acts or events described may be necessary to implement a method or process). In some instances, method 4000 may be modified to omit step 4002. Although the various embodiments described above relate to steps 4002-4008 of method 4000, another method of the present disclosure may include identifying candidate edges based on 3D image information, as discussed with respect to FIG. 11B or 11C, and may omit steps 4002-4008. In addition, although certain features of the embodiments herein are described for clarity as being performed by a single component, module, or unit, it should be understood that the features and functions described herein may be performed by any combination of components, modules, or units. Thus, various changes and modifications may be effected by one skilled in the art without departing from the spirit or scope of the invention, as defined in the appended claims.

Claims (15)

1. A computing system comprising:
a communication interface configured to communicate with the robot and a camera having a camera field of view;
at least one processing circuit;
The at least one processing circuit, when a group of objects is within the camera field of view,
receiving image information generated by the camera and representative of the group of objects;
identifying candidate edges from the image information;
determining whether a portion of the image information adjacent the candidate edge satisfies an intensity profile criterion based on whether the portion of the image information includes (i) a first profile portion where image intensity increases in darkness followed by (ii) a second profile portion where image intensity decreases in darkness;
Outputting a robot interactive movement command;
configured to run
A computing system, wherein the robot interaction move command is for robot interaction between the robot and at least one object of the group of objects and is based on the candidate edge. The computing system of claim 1, wherein if the intensity profile criterion is met, the portion of the image information adjacent the candidate edge is a first pixel region that forms a band of pixels representing a physical gap between a first object and a second object of the group of objects. an image intensity profile of the portion of the image information adjacent the candidate edge describing respective pixel intensity values for a series of pixels extending across a width dimension of the portion;
2. The computing system of claim 1, wherein the at least one processing circuit is configured to determine whether the portion of the image information adjacent the candidate edge meets a spike intensity profile criterion by determining whether the image intensity profile has a shape in which the respective pixel intensity values decrease toward a minimum pixel intensity value and then switch to increase away from the minimum pixel intensity value, the minimum pixel intensity value being associated with a peak level of darkness in the portion of the image information adjacent the candidate edge. The at least one processing circuit comprises:
only in response to determining that the portion of the image information adjacent the candidate edge satisfies the spike strength profile criterion;
The computing system of claim 3 , configured to determine that the portion of the image information adjacent the candidate edge satisfies the intensity profile criterion. The at least one processing circuit comprises:
When the image information includes 2D image information and 3D image information,
configured to identify the candidate edges based on the 2D image information;
The computing system of claim 1 , wherein the 3D image information includes depth information for locations within the camera field of view. The at least one processing circuit comprises:
(i) the 3D image information lacks depth information at one or more locations corresponding to the candidate edge, or (ii) a portion of the 3D image information corresponding to the candidate edge is subject to a level of imaging noise greater than a defined noise tolerance threshold;
The computing system of claim 5 , configured to determine whether to retain the candidate edge as a candidate for representing at least one physical edge of the group of objects. The at least one processing circuit comprises:
when the 3D image information does not satisfy a defined depth discontinuity condition at one or more locations corresponding to the candidate edge;
The computing system of claim 5 , configured to determine whether to retain the candidate edge as a candidate for representing at least one physical edge of the group of objects. The at least one processing circuit comprises:
in response to determining that the 3D image information does not describe a depth change at one or more locations that exceeds a defined depth difference threshold;
The computing system of claim 7 , configured to determine that the 3D image information does not satisfy the defined depth discontinuity condition at the one or more locations corresponding to the candidate edge. The computing system of claim 5, wherein the at least one processing circuit is configured to identify a second candidate edge based on the 3D image information. The at least one processing circuit comprises:
identifying a first surface of the group of objects based on a first set of locations described by the 3D image information having respective depth values that do not deviate from each other by more than a defined measurement variance threshold;
identifying a second surface of the group of objects based on a second set of locations described by the 3D image information having respective depth values that are within the defined measurement variance threshold;
determining an average depth value associated with the first surface as a first average depth value;
determining an average depth value associated with the second surface as a second average depth value;
in response to determining that the difference between the first average depth value and the second average depth value exceeds a defined depth difference threshold, identifying the second candidate edge based on a location of a transition between the first surface and the second surface;
10. The computing system of claim 9, configured to identify the second candidate edge based on the 3D image information by: The at least one processing circuit comprises:
when the second candidate edge is mapped to a candidate that is in the 2D image information and is formed based on a boundary between two image regions that do not satisfy the intensity profile criterion;
The computing system of claim 9 , configured to identify the second candidate edge based on the 3D image information. The computing system of claim 1, wherein the at least one processing circuit is configured to perform an object recognition operation or an object registration operation based on the candidate edges. A non-transitory computer-readable medium having instructions,
The instructions, when executed by at least one processing circuit of a computing system, cause the at least one processing circuit to:
receiving image information by the at least one processing circuit of the computing system, the computing system being configured to communicate with (i) a robot and (ii) a camera having a camera field of view, the image information being representative of a group of objects within the camera field of view and generated by the camera;
identifying candidate edges from the image information;
determining whether a portion of the image information adjacent the candidate edge satisfies an intensity profile criterion based on whether the portion of the image information includes (i) a first profile portion where image intensity increases in darkness followed by (ii) a second profile portion where image intensity decreases in darkness;
Outputting a robot interactive movement command;
The above is executed.
A non-transitory computer-readable medium, wherein the robot interaction move command is for robot interaction between the robot and at least one object of the group of objects and is based on the candidate edge. The non-transitory computer-readable medium of claim 13, wherein if the intensity profile criterion is met, the portion of the image information adjacent the candidate edge is a first pixel region that forms a band of pixels representing a physical gap between a first object and a second object of the group of objects. 1. A method performed by a computing system, comprising:
receiving image information by the computing system, the computing system configured to communicate with (i) a robot and (ii) a camera having a camera field of view, the image information being for representing a group of objects within the camera field of view and generated by the camera;
identifying candidate edges from the image information;
determining whether a portion of the image information adjacent the candidate edge satisfies an intensity profile criterion based on whether the portion of the image information includes (i) a first profile portion where image intensity increases in darkness followed by (ii) a second profile portion where image intensity decreases in darkness;
Outputting a robot interactive movement command;
Including,
The method, wherein the robot interaction move command is for robot interaction between the robot and at least one object of the group of objects and is based on the candidate edge.

Images