Advanced Lane Finding

Lane line detection is a critical technique for the design of systems that allows a car to drive itself. The detection of these lines allows our car to stay on right path and follow it, in the same way that we use our eyes to stay on the lane. This project uses an approach based on image processing, it uses Python as the main language and OpenCV as complementary framework for image analysis and processing.

---

1. Approach

The approach consisted of 6 steps:

1. Distortion correction: Light rays often bend a little too much or too little at the edges of the curved lenses in real cameras, this creates a radial distortion and if the camera’s lens is not aligned perfectly parallel to the imaging plane a tangential distortion also occurs. In this project the distortion coefficients were calculated to correct radial distortion using a 9 x 6 corners chessboard images.

• Radial distortion:

  

  

• Tangential distortion:

  

  

2. Perspective transform: A perspective transform maps the points in a given image to different, desired, image points with a new perspective. In this project a bird’s-eye view transform that let’s us view a lane from above was used (it is useful for calculating curvature in the next steps).

• Calculate transformation and inverse matrices from four pair of points:

  

  

• Apply perspective transform using the matrix:

  

• Points used in this project:

  

3. Thresholding: Thresholding is a method of segmenting image. In this project we use a combined threshold (sobel and colorspaces) to create a binary image where the lane line pixels are activated.

• Convolute a sobel of kernel size of 5 in the x direction with the image:

  

• Get S channel from HLS colorspace transform:

  

  

• Get B channel from LAB colorspace transform:

  

  

  

  

  

  

• Every channel is then binarized using the next criteria:

  

• Finally, the binary image is made by the union of the 3 channels:

  

4. Polynomial fit: After the threshold, we fit a quadratic polynomial to each of the lane lines, this step will allow us to detect curves in the road.

• First we need to calculate a histogram in the bottom half of the image by summing all the pixels in the y axis, two peaks will be detected and will be the base for the left and right lines. To avoid noise, margins () in the left and right edges were added.

  

  

• Next step we use the sliding window algoritm to fit 15 windows in each line starting from the base lines until the top of the image in order to look for all the pixels related to the lines. Once all pixels for each line are detected we fit a second degree polynomial in each set of the lines by minimizing the squared error.

  

  

• For optimization, once on the next frame we don't need to blindly look again for the lines, but use previous information. Adding a margin in the polynomial equation we can create a region of interest customized for every frame that follows the line in the previous frame.

  

5. Polynomial fit validation: In order to determine if the fit found is valid we need to met some criteria.

• In case that a previous fit exists we need to compare the difference between coefficients, if the difference is too big, then it's not a valid one (differences in line coefficients shouldn't vary that much from frame to frame).

  

• In order to make a smooth transition the fit in the current frame will be equal to an average of the previous 5 fits.

  

6. Measuring curvature: Having the coefficients we can calculate the radius of curvature at any point of the function .

2. Shortcomings

There are some flaws in this approach:

• It is not illumination invariant, so the lack of contrast and light would result in a malfunction of the program.

• Thresholding is without a doubt the core of the problem, playing with multiple thresholdings make the model more robust but also can detect undesired patterns and objects that will lead to a wrong lane detection.

• A fixed camera position and consistent image size are assumed, so positioning the camera at a different angle and handling different image sizes would have undesirable results.

3. Improvements

Some points of the pipeline can be improved.

1. Perspective Transform: The images and videos have a fixed size and perspective, right now the 4 selected points for the transform are fixed, so a better way to select them automatically based on features to handle multiple sizes and camera angles will be necessary for a more robust approach

2. Thresholding: Without a doubt trhesholding is the core of the algorithm, a more robust method for illumination, rotation and translation invariance should be developed for a real application. This approach showed a good performance for the project_video and a fairly average for the challenge_video, but its completely useless for the harder_challenge_video.

3. Polynomal fit: Sliding windows algorithm and its optimization for looking around previous polynomial are pretty robust methods for lane line detection, minor improvements could be done here. However, these methods are strongly dependant on the thresholding for finding a good fit, i.e. we are as strong as the weakest link.