Multi-label feature selection based on HSIC and sparrow search algorithm

1.   Introduction

Any TVSS system consists mainly of an image sensor, an image processing unit and a tactile display. The tactile display involves an array of tactile stimulators which transfer the scene visual image to the user's skin with minimum details to avoid sensory overload while respecting the spatial resolution of the skin's sensory receptors ^[1,2,3,4]. The main job of the image processing unit is to generate a tactile pattern, for the scene image, without losing its important features. One of the main used ways to do that is by converting the captured image to an edge, boundary or contour image ^[5,6,7,8,9]. This was enhanced by research results showing that the human retina is related to extraction of edge information from an incident image ^[10,11]. The desired features of the edge images for TVSS system are essentially accomplished by applying one of the first order edge detection algorithms in order to enhance the main edges and exclude the detail edges ^{[12,13,14,15,16,17,18,19]}. Anyway, some research has applied a different approach to finding edges of the scene image ^[20].

To take a step towards the production of an all-in-one TVSS device, this paper presents the design and simulation of an EDIS. EDIS is mostly tailored for a certain demand. The benefits of using EDIS are reviewed here in addition to the review of some literature. History of microelectronics tells that integration leads to greater reliability, lower system power, and plummeting system cost/performance ratios. CMOS-based imaging systems on a chip can be expected to reflect these long-standing trends as they will be developed over the next years ^[21]. Low-level image processing on the sensor focal plane produces a smart sensor device called a retina which makes vision systems more compact, offering fast and parallel processing, but needs optimal design architecture for every vision algorithm ^[22], with improved image quality in comparison with the standalone digital processing, ^[23] offering lower manufacturing cost ^[24]. Research in the field of image sensor design, simulation, and implementation continues to produce a smart image sensor that mimics the biological retina where the pixel-level processing promises high SNR ^[25]. Markus Loose et al. ^[26] have implemented a CMOS Image Sensor (CIS) having (20 × 20) pixels of adaptive photoreceptors, modeled on a biological example, to be used for tactile vision. The design of a CMOS active pixel sensor (APS) chip has been described by R.H. Nixon et al. with integrated digital control functions forming a camera-on-a-chip having a 256 × 256 APS sensor ^[27]. System-level architectures and design methodology was presented ^[28]. An event detection CIS based on relative change in intensity was produced ^[29]. CIS readout strategy performing edge detection, to overcome the excessive output data flow bottleneck, was emulated by MATLAB, with circuit simulations using the Cadence Spectra Simulator ^[30]. The Bioinspired CIS may add visual tracking, detection, recognition, simultaneous localization and mapping, reconstruction, stereo matching, and control ^[31].

Roberts has presented one of the earliest first order edge detection methods, the cross operator ^[32,33]. Roberts' method "fails to detect fine edges" ^[34], which is considered an asset thing in the generation of the tactile patterns. Operator's convolution with the image is widely used for edge detection using software or DSP tools while Applying convolution in analog processing earns little interest and consumes more hardware elements. This paper presents the implementation and simulation of Roberts' method inside the CMOS image sensor to get an edge image that can be converted to a tactile pattern.

1.1.   CMOS Image Sensor (CIS) simulation

CIS simulation has been applied using SPICE models for Si and GaAs pin photodetectors ^[35]. An optoelectronic simulation that included time, power, and wavelength-domain behaviors was presented ^[36]. On the other hand, and to overcome the long time needed for simulation, a graphical toolbox was developed in Cadence Analog Design Environment (ADE). It permitted to manipulate images as input and automatically generate images at output ^[37], which is improved later ^[38,39]. Another approach was also introduced by M. R. Jha et al ^[40]. CIS with 8T global shutter and the 3T APS was simulated using Virtuoso analog design environment ^[41]. Other works have presented a combination of the circuit level and functional level simulation ^[42,43].

1.2.   CMOS Image Sensor (CIS) structure

The typical parts of the CIS (Figure 1) are the photodetector (PD), charge to voltage converter, source follower (SF), and analog to digital converter (ADC). The CIS basic block diagram is sketched in Figure 2, it consists of an array of pixels, vertical access part, horizontal access part, and the readout part. The PD array is a two-dimensional matrix of pixels that resemble the function of the retina. A readout circuit is a one-dimensional array of switches, sample & hold (S/H) circuits, and some storage elements. Access units are used to reach a certain pixel by enabling a certain row and a certain column.

1.3.   Photo pixels (sensor cells)

The photodetector is integrated with other components to form the sensor cell or the photo pixel. Many types of pixels were introduced over the last three decades. They are classified mainly as passive pixel sensors (PPSs), active pixel sensors (APSs), and digital pixel sensors (DPSs). While PPS directly transfers the accumulated signal charges to the outside of the pixel, APS converts the accumulated signal charges to a potential in the attached transistor ^[44]. APS was first produced by E. Fossum et al. at Jet Propulsion Laboratory (JPL) using a photogate (PG), and then using a photodiode (PD), as a photodetector ^[45,46]. PS can also be classified according to the number of transistors per pixel, like three transistors APS (3T-APS), 4T-APS, 5T-APS ^[47], 2.5T-APS, and so on. Decimal fraction of transistors' number per pixels is due to transistor sharing with adjacent pixels which depends on the design and readout mechanism. As the number of transistors per pixel increases, the fill factor (FF) decreases.

1.4.   Photo detector (PD)

The photodetector is mostly a p-n junction photodiode, but phototransistor, pinned PD (PPD), photogate (PG), or Gate/Body-Tied Photodetector (GBT ^[48,49,50] can be used. PPD CIS represent today the main solid-state optical sensor technology for imaging applications ranging from mass-market smartphones to high-end scientific instruments ^[51]. It is used in most of CCD and CIS due to its low noise, high quantum efficiency, and low dark current ^[52,53]. The PPD is almost used in the 4T-APS.

2.   Materials and methods

This paper presents the design and simulation of an Edge Detection Image Sensor (EDIS), which is an IS with a built-in edge detector. It was based on Roberts' algorithm and was applied to the binary image. The proposed way was verified and compared with some other ways of edge detection using MATLAB. Then designed and simulated using the compact SPICE models. The simulation was implemented using a circuit level model and a small size CIS model via LTSPICE. A third simulation was done using both MATLAB and LTSPICE to perform a fast simulation with image-in image-out for any resolution of the input image.

2.1.   Edge detection scheme

The proposed technique was based on applying XOR function between every two vertices of the diagonals of 2 × 2 subarray, then obtaining the OR result of them (Figure 3). It was applied using a sliding window, having a height of two rows of pixels, slides from top to the bottom of the photo pixels array as shown in Figure 4.

2.2.   Verification of the proposed method using MATLAB

The following steps were applied, using MATLAB, to check the validity of the proposed way:

1. read the input image and if it is an RGB image convert it to the gray level,

2. convert the gray to binary using a threshold level,

3. detect the change of state between the (i, j)th pixel and the (i+1, j+1)th pixel by XOR function using the formula (1),

4. detect the change of state between the (i, j)th pixel and the (i-1, j+1)th pixel using XOR function using the formula (2),

5. combine the two outputs using OR function (3),

6. repeat the above three steps to the end of the current row, and

7. slide down one row and repeat the above 4 steps while moving down until the last row.

The flow of an image through this verification steps is shown in Figure 5.

2.3.   CMOS image sensor design

Some parts of the image sensor that are used in this design are explained here. These units are the photo pixel circuit, the column control circuit, the comparator, the XOR circuit, the NOR circuit, and the readout unit. The design efforts were mainly focused on the array of the photo pixels and the readout circuitry to fulfill the edge detection demand.

The 4T-APS pixel with Pinned PD (PPD) was selected to be used in the proposed design. The accumulated photocharge in the junction capacitor CPD (Figure 6) is passed to the floating diffusion capacitor CFD via the transfer gate M2. M1 transistor, when activated by the RST signal, pushes up the FD node to (VDD - Vth(M1)) voltage level. While when M1 is OFF, the voltage on the FD node is transferred to M4 via the NMOS Source Follower (NMOS SF) transistor M4. M3 is the output stage of the pixel that transfers the pixel signal to M3 and is activated via the ROW control line. This process is repeated to all the rows of the image sensor array. The timing diagram of the photo pixel control signals is sketched in Figure 7. The time interval from two TX pulses is called the integration time because it is the time for photocurrent to accumulate or integrate the charge inside the PPD.

The column control circuit (CCNT) contains the sample and hold (S/H) circuit, column line biasing transistor, and output buffering as shown in Figure 6. The output of the CCNT unit is fed to a comparator unit along with a reference voltage (VR) in order to convert the analog signal level to a binary level.

2.3.1.   The readout unit

The simplified block diagram of the proposed IS is illustrated in Figure 8 without the edge detection circuit. An entire row of pixels is selected using the signal ROWi, hence, the voltage of each pixel in that row (only) is passed to its corresponding column line, cached into the (S/H) capacitor, passed to the comparator circuit, and compared with a reference voltage (VR) controlled by the user. The output is a binary signal that represents the photo-detected signals of the pixels at the ith row for all the CIS columns, stored temporarily in M-number D latches (for an N × M PD array). At each frame, only one row is selected, and its data is stored at the D-FF latches using the CLK signal as shown in Figure 9. In this design, there is no need for the Analog to Digital Conversion (ADC). The stored values form a line of the scene image in binary. To apply the proposed edge detection technique, a second line of D-FFs was added to store the data of the neighbored row. Therefore at a given row time the first set stores the previous row, while the second stores the current row. The outputs of these two D-FFs are fed to the XOR-OR circuits to generate the points of the edge image and the results are stored in a third set of D-FFs (Figure 10). The whole operation is finished in a single-row interval. The XOR circuit used in this design is composed of three transistors ^[54]. A CMOS inverter and a PMOS pass transistor were used to act as a two inputs XOR gate. A complementary NOR gate design was selected due to its low power, high noise margins, and ease of design.

2.4.   The simulation

We have applied three ways of simulation: the first; was the simulation of the individual circuits via LTSPICE electronic circuit simulation environment, the second was also via LTSPICE to simulate a CIS having a PD array of 8 × 8 pixels. While the third was a fast way through a combination of both MATLAB and LTSPICE.

2.4.1.   Individual circuit simulation

The simplified (pinned) photodiode model circuit diagram shown in Figure 11 was used in the simulation. The shunt resistor (RSH) and the series resistor (RS) were neglected, as in the most of the image sensor functional simulations, i.e. the shunt resistor is considered to be very high and the series resistor is very low. One of the advantages of the pinned photodiode is the very low dark current (IDark), therefore it was not included in the simulation model. The IPD current source was used to model the photocurrent. CPD is the parasitic capacitance of the P-N junction. It was found that CPD value changes with the voltage change at the photodiode cathode during the photocurrent integration period, and typically it is a few tens of fFs ^[55]. The control signals were generated using simulated voltage sources. Figure 12 shows the signal diagram, and Figure 13 demonstrates the simulated signals for five different photocurrent (IPD) values showing the signal voltages decrease with the increase of IPD input for the PD, FD and Sout nodes that were depicted in Figure 6.

The output of the photo pixel circuit (Sout) is passed to the input of the column control circuit (CCNT), and the simulation signals are sketched in Figure 14, in synchronization with the ROW signal. The comparator, XOR, NOR, and the remnant circuits were all simulated too.

2.4.2.   The simulation of CIS that possesses 8 × 8 photo pixels

An overall functional simulation was performed using a PD array of 8 × 8 pixels. Signals of the important nodes of the CIS circuits were monitored for various inputs and conditions. Each pixel of the PD array was equipped with its own IPD variable current source, to simulate the reflected light from a certain point of the scene. This current source is, essentially, a part of the PD simulation model, and was drawn off to the outside of the pixel to simplify the simulation procedure, thus a certain input pattern representing a scene image could be given to the CIS. We have found empirically that IPD value range that sustains linear characteristics of the designed APS was from 0.0 nA to 1.66 nA. Figure 15 illustrates the circuit diagram used in this simulation. Many input patterns were used in the simulation, one of them was a diagonal line consisting of the pixels P (3, 6), P (4, 5), P (5, 4) and P (6, 3). Where all the pixels of the PD array were fed with an IPD = 1.0 nA and the diagonal line pixels with an IPD = 0.5 nA. An extra control signal, the frame start (Fstr), was added to synchronize the periphery unit with the first pixel of the current image frame. The timing diagram for this case is sketched in Figure 16. It is not easy to judge the result through this form, therefore, a new look of the diagram is presented in Figure 17, by rotating right the previous timing diagram by 90̊, and paste a small black rectangle to each zero-level of the output waveform. The results had proved the functionality of the proposed EDIS.

2.4.3.   Fast simulation using MATLAB and LTSPICE

Fast simulation is used to overcome the limitations of CIS simulation with its full PD array size. It is based on the reduction of the number of calculations from the variable number of pixels, forming the array, to a fixed number depends on the gray levels (256) for the input image and on the resolution of the ADC (if used) ^{[37,38,39,43]}. The output of the CIS is an edge image, where every pixel relates to an input-pixel, or pixels, by a certain relationship or function. This relationship was extracted from the previous LTSPICE simulation of the proposed design. The simulation was repeated for 256 times by sweeping the input IPD from 0.0 nA to 1.66 nA in a step of 6.484 pA (1.66nA/256). For each step of the input, the output of the column control circuit was recorded in lock-up-table 1 (LUT1) and the comparator output was recorded in LUT2. These two LUTs were used as a real substitution of the CIS-SPICE macro models and inserted in a MATLAB simulation program that was written for that purpose (Figure 18). The input image was converted to gray levels, then each gray level was mapped to an IPD level (0–1.6 nA), then converted (via LUT1 or LUT2) to a voltage level, at last, the XOR-OR function is applied to find the resultant edge image can select either hardware or software comparison.

3.   Results

The simulation of the 8 × 8 pixels EDIS has given edge's pattern for any input image as depicted in Figure 17 for example. Parametric results were listed in Table 1, in comparison with previous works.

Results of the fast simulation can be demonstrated in many forms. The first was performed to check the validity of the proposed way when subjected to downscaling which is essential for TVSS applications. Figure 19 shows 5 images with different original resolutions passed to the fast simulation process for two times to get the 256 × 256 and 64 × 64 output pixels edge images respectively.

Another test was done to check the device response to the reference voltage (VR) variation. User's controlled VR, can give the blind person the required tactile image detail's level. An example is shown in Figure 20.

Results of visual comparisons with the results of some of the previous works are presented in Figures 21–24.

4.   Discussion

The proposed way had given acceptable edge images, and it was validated by the simulation using LTSPICE and MATLAB. Parametric comparison with some previous works have shown acceptable results, as well as the visual comparison of the edge images.

5.   Conclusion

This paper has presented the design and simulation of an Edge Detection CMOS Image Sensor (EDIS) to satisfy the demands of TVSS systems. An edge detection technique based on Roberts's has been converted to a simple hardware scheme conserving APS-fill factor as well as eliminating the use of ADC. The design functionality has been verified using a fast simulation technique. The design of this IS can be considered a step towards the production of an all-in-one TVSS device. The design can be improved by adding a Correlated Double Sampling (CDS) circuit to improve SNR in the future.

Conflict of interest

The authors declare that there is no conflict of interest in this paper.