First I started off with importing the Times Square Subway with almost all the trains, and the 59th street station trains images onto MATLAB.
This plate displays numerous colors, but in order to make it easier for the camera to capture them, I have to pick up the colors individually.
First I started off with changing the color of the whole image. Using the following MATLAB code, I turned the whole image to gray.
>> rgb = imread('NQR345.png');
>> figure
>> imshow(rgb)
>> gray_image = rgb2gray(rbg);
imshow(gray_image);
After, I was flexible with changes the colors around, I tired to make the change it to black and white. Since there are so many other letter and numbers on the plate itself, we need to detect the area we have to focus on. So using MATLAB, I was able to make the image black and white and able to capture NQR456.
The code:
>> center6 = center(1:6,:);
>> radius = radii(1:6);
>> metric6 = metric(1:6);
>> viscircles(center6, radii6, 'EdgeColor', 'r');
**The 6 in the code above could be named anything else, for instance, train.
So far I been successful at taking the colors away, now it's time to detect colors. Any colors I want.
MATLAB has an app called Color Thresholding App designed to mix the color or separate the colors. I played around with it using the Times Square Subway image.
The polynomial to the right in the image helps you separate or join the colors. So in the image above, I took away all the colors expect for red and Orange. It even took away the white letters and numbers. We can still see the numbers 1, 2, 3, and letters N, Q, R, and W because initially they were white, and the background is all black.
This is another example of Color Thresholding. In this image I made the polynomial in a way so that we can see all the trains in one look. Although, it;s not completely clear because we can still see some of the white letters,
To take this to the next level, using MATLAB, I took all the colors away, and took away all the numbers and letters away except for 1, 2, and 3. The code to get this answer is the following.
%image = imread('/Users/admin/Pictures/subwaysignall.jpg');
image = imread('subwaysign.jpg');
J = imresize(I, 0.5);
figure
imshow(image)
title('Original Image')
figure
imshow(J)
title('Resized Image')
%123
%image = imread('subwaysignall.jpg');
figure(1), imshow(image), title('Original');
%image = im2double(image);
%[r c p] = size(image);
%imageR = squeeze(image(:,:,1));
%imageG = squeeze(image(:,:,2));
%imageB = squeeze(image(:,:,3));
%imageBWR = im2bw(imageR, graythresh(imageR));
%imageBWG = im2bw(imageG, graythresh(imageG));
%imageBWB = im2bw(imageB, graythresh(imageB));
%imageBW = imcomplement(imageBWR&imageBWG&imageBWB);
%figure(2), imshow(imageBWR); title('Color Thresholded');
%figure(3), imshow(imageBWG); title('Color Thresholded');
%figure(4), imshow(imageBWB); title('Color Thresholded');
%figure(5), imshow(imageBW); title('Color Thresholded');
red = @createMask;
imageBWRed = red(image);
figure(6), imshow(imageBWRed); title('Red Train');
imageBWRed = bwmorph(imageBWRed,'clean')
% ocrAnn = @insertOCRAnnotation;
ocrEval = @evaluateOCRTraining;
[ocrimageBWRed, results] = ocrEval(imageBWRed);
figure(7), imshow(ocrimageBWRed); title('OCR Red Train');
results.Text;
text = results.Text;
words = ['say ' '-v' 'Victoria' ' subway train ' text(1:1) ' ' text(2:2) ' ' text(3:3)];
%system('say -v "Victoria" words');
system(words);
results.CharacterConfidences;
function [BW,maskedRGBImage] = createMask(RGB)
%createMask Threshold RGB image using auto-generated code from colorThresholder app.
% [BW,MASKEDRGBIMAGE] = createMask(RGB) thresholds image RGB using
% auto-generated code from the colorThresholder App. The colorspace and
% minimum/maximum values for each channel of the colorspace were set in the
% App and result in a binary mask BW and a composite image maskedRGBImage,
% which shows the original RGB image values under the mask BW.
% Auto-generated by colorThresholder app on 03-Dec-2016
%------------------------------------------------------
% Convert RGB image to chosen color space
I = rgb2hsv(RGB);
% Define thresholds for channel 1 based on histogram settings
channel1Min = 0.917;
channel1Max = 0.045;
% Define thresholds for channel 2 based on histogram settings
channel2Min = 0.000;
channel2Max = 1.000;
% Define thresholds for channel 3 based on histogram settings
channel3Min = 0.000;
channel3Max = 1.000;
% Create mask based on chosen histogram thresholds
sliderBW = ( (I(:,:,1) >= channel1Min) | (I(:,:,1) <= channel1Max) ) & ...
(I(:,:,2) >= channel2Min ) & (I(:,:,2) <= channel2Max) & ...
(I(:,:,3) >= channel3Min ) & (I(:,:,3) <= channel3Max);
% Create mask based on selected regions of interest on point cloud projection
I = double(I);
[m,n,~] = size(I);
polyBW = false([m,n]);
I = reshape(I,[m*n 3]);
% Convert HSV color space to canonical coordinates
Xcoord = I(:,2).*I(:,3).*cos(2*pi*I(:,1));
Ycoord = I(:,2).*I(:,3).*sin(2*pi*I(:,1));
I(:,1) = Xcoord;
I(:,2) = Ycoord;
clear Xcoord Ycoord
% Project 3D data into 2D projected view from current camera view point within app
J = rotateColorSpace(I);
% Apply polygons drawn on point cloud in app
polyBW = applyPolygons(J,polyBW);
% Combine both masks
BW = sliderBW & polyBW;
% Initialize output masked image based on input image.
maskedRGBImage = RGB;
% Set background pixels where BW is false to zero.
maskedRGBImage(repmat(~BW,[1 1 3])) = 0;
end
function J = rotateColorSpace(I)
% Translate the data to the mean of the current image within app
shiftVec = [0.035456 0.001129 0.241630];
I = I - shiftVec;
I = [I ones(size(I,1),1)]';
% Apply transformation matrix
tMat = [-0.488370 -0.361528 0.000000 0.683321;
0.012842 -0.028299 0.658630 -0.491234;
0.282769 -0.623110 -0.029912 8.146246;
0.000000 0.000000 0.000000 1.000000];
J = (tMat*I)';
end
function polyBW = applyPolygons(J,polyBW)
% Define each manually generated ROI
hPoints(1).data = [0.191929 0.018162;
0.548562 -0.007794;
0.502281 -0.306294;
0.257266 -0.341984];
% Iteratively apply each ROI
for ii = 1:length(hPoints)
if size(hPoints(ii).data,1) > 2
in = inpolygon(J(:,1),J(:,2),hPoints(ii).data(:,1),hPoints(ii).data(:,2));
in = reshape(in,size(polyBW));
polyBW = polyBW | in;
end
end
end
function [ocrI, results] = evaluateOCRTraining(I, roi)
% Location of trained OCR language data
trainedLanguage = '/Users/admin/Documents/MATLAB/myLang/tessdata/myLang.traineddata';
% Run OCR using trained language. You may need to modify OCR parameters or
% pre-process your test images for optimal results. Also, consider
% specifying an ROI input to OCR in case your images have a lot of non-text
% background.
layout = 'Block';
if nargin == 2
results = ocr(I, roi, ...
'Language', trainedLanguage, ...
'TextLayout', layout);
else
results = ocr(I, ...
'Language', trainedLanguage, ...
'TextLayout', layout);
end
ocrI = insertOCRAnnotation(I, results);
end
%--------------------------------------------------------------------------
% Annotate I with OCR results.
%--------------------------------------------------------------------------
function J = insertOCRAnnotation(I, results)
text = results.Text;
I = im2uint8(I);
if isempty(deblank(text))
% Text not recognized.
text = 'Unable to recognize any text.';
[M,N,~] = size(I);
J = insertText(I, [N/2 M/2], text, ...
'AnchorPoint', 'Center', 'FontSize', 24, 'Font', 'Arial Unicode');
else
location = results.CharacterBoundingBoxes;
% Remove new lines from results.
newlines = text == char(10);
text(newlines) = [];
location(newlines, :) = [];
% Remove spaces from results
spaces = isspace(text);
text(spaces) = [];
location(spaces, :) = [];
% Convert text array into cell array of strings.
text = num2cell(text);
% Pad the image to help annotate results close to the image border.
I = padarray(I, [50 50], uint8(255));
location(:,1:2) = location(:,1:2) + 50;
% Insert text annotations.
J = insertObjectAnnotation(I, 'rectangle', location, text);
end
end
As we can see in the image, all the colors have vanished, and we are only left with what we want, 1, 2, 3, in black and white. By using the color thresholding app, I inputed the function to the code. Using the same method, I used the function for the blue line (A, C, E) from the color thresholding app, and the image gave me A, C, E in black and white.
Code for the image above:
%image = imread('/Users/admin/Pictures/subwaysignall.jpg');
image = imread('subwaysign.jpg');
%image = imread('subwaysignall.jpg');
figure(1), imshow(image), title('Original');
%image = im2double(image);
%[r c p] = size(image);
%imageR = squeeze(image(:,:,1));
%imageG = squeeze(image(:,:,2));
%imageB = squeeze(image(:,:,3));
%imageBWR = im2bw(imageR, graythresh(imageR));
%imageBWG = im2bw(imageG, graythresh(imageG));
%imageBWB = im2bw(imageB, graythresh(imageB));
%imageBW = imcomplement(imageBWR&imageBWG&imageBWB);
%figure(2), imshow(imageBWR); title('Color Thresholded');
%figure(3), imshow(imageBWG); title('Color Thresholded');
%figure(4), imshow(imageBWB); title('Color Thresholded');
%figure(5), imshow(imageBW); title('Color Thresholded');
red = @createMask;
imageBWRed = red(image);
figure(6), imshow(imageBWRed); title('Red Train');
imageBWRed = bwmorph(imageBWRed,'clean')
% ocrAnn = @insertOCRAnnotation;
ocrEval = @evaluateOCRTraining;
[ocrimageBWRed, results] = ocrEval(imageBWRed);
figure(7), imshow(ocrimageBWRed); title('OCR Red Train');
results.Text;
text = results.Text;
words = ['say ' '-v' 'Victoria' ' subway train ' text(1:1) ' ' text(2:2) ' ' text(3:3)];
%system('say -v "Victoria" words');
system(words);
results.CharacterConfidences;
function [BW,maskedRGBImage] = createMask(RGB)
%createMask Threshold RGB image using auto-generated code from colorThresholder app.
% [BW,MASKEDRGBIMAGE] = createMask(RGB) thresholds image RGB using
% auto-generated code from the colorThresholder App. The colorspace and
% minimum/maximum values for each channel of the colorspace were set in the
% App and result in a binary mask BW and a composite image maskedRGBImage,
% which shows the original RGB image values under the mask BW.
% Auto-generated by colorThresholder app on 03-Dec-2016
%------------------------------------------------------
% Convert RGB image to chosen color space
I = rgb2hsv(RGB);
% Define thresholds for channel 1 based on histogram settings
channel1Min = 0.917;
channel1Max = 0.045;
% Define thresholds for channel 2 based on histogram settings
channel2Min = 0.000;
channel2Max = 1.000;
% Define thresholds for channel 3 based on histogram settings
channel3Min = 0.000;
channel3Max = 1.000;
% Create mask based on chosen histogram thresholds
sliderBW = ( (I(:,:,1) >= channel1Min) | (I(:,:,1) <= channel1Max) ) & ...
(I(:,:,2) >= channel2Min ) & (I(:,:,2) <= channel2Max) & ...
(I(:,:,3) >= channel3Min ) & (I(:,:,3) <= channel3Max);
% Create mask based on selected regions of interest on point cloud projection
I = double(I);
[m,n,~] = size(I);
polyBW = false([m,n]);
I = reshape(I,[m*n 3]);
% Convert HSV color space to canonical coordinates
Xcoord = I(:,2).*I(:,3).*cos(2*pi*I(:,1));
Ycoord = I(:,2).*I(:,3).*sin(2*pi*I(:,1));
I(:,1) = Xcoord;
I(:,2) = Ycoord;
clear Xcoord Ycoord
% Project 3D data into 2D projected view from current camera view point within app
J = rotateColorSpace(I);
% Apply polygons drawn on point cloud in app
polyBW = applyPolygons(J,polyBW);
% Combine both masks
BW = sliderBW & polyBW;
% Initialize output masked image based on input image.
maskedRGBImage = RGB;
% Set background pixels where BW is false to zero.
maskedRGBImage(repmat(~BW,[1 1 3])) = 0;
end
function J = rotateColorSpace(I)
% Translate the data to the mean of the current image within app
shiftVec = [0.035456 0.001129 0.241630];
I = I - shiftVec;
I = [I ones(size(I,1),1)]';
% Apply transformation matrix
tMat = [-0.488370 -0.361528 0.000000 0.683321;
0.012842 -0.028299 0.658630 -0.491234;
0.282769 -0.623110 -0.029912 8.146246;
0.000000 0.000000 0.000000 1.000000];
J = (tMat*I)';
end
function polyBW = applyPolygons(J,polyBW)
% Define each manually generated ROI
hPoints(1).data = [0.191929 0.018162;
0.548562 -0.007794;
0.502281 -0.306294;
0.257266 -0.341984];
% Iteratively apply each ROI
for ii = 1:length(hPoints)
if size(hPoints(ii).data,1) > 2
in = inpolygon(J(:,1),J(:,2),hPoints(ii).data(:,1),hPoints(ii).data(:,2));
in = reshape(in,size(polyBW));
polyBW = polyBW | in;
end
end
end
function [ocrI, results] = evaluateOCRTraining(I, roi)
% Location of trained OCR language data
trainedLanguage = '/Users/admin/Documents/MATLAB/myLang/tessdata/myLang.traineddata';
% Run OCR using trained language. You may need to modify OCR parameters or
% pre-process your test images for optimal results. Also, consider
% specifying an ROI input to OCR in case your images have a lot of non-text
% background.
layout = 'Block';
if nargin == 2
results = ocr(I, roi, ...
'Language', trainedLanguage, ...
'TextLayout', layout);
else
results = ocr(I, ...
'Language', trainedLanguage, ...
'TextLayout', layout);
end
ocrI = insertOCRAnnotation(I, results);
end
%--------------------------------------------------------------------------
% Annotate I with OCR results.
%--------------------------------------------------------------------------
function J = insertOCRAnnotation(I, results)
text = results.Text;
I = im2uint8(I);
if isempty(deblank(text))
% Text not recognized.
text = 'Unable to recognize any text.';
[M,N,~] = size(I);
J = insertText(I, [N/2 M/2], text, ...
'AnchorPoint', 'Center', 'FontSize', 24, 'Font', 'Arial Unicode');
else
location = results.CharacterBoundingBoxes;
% Remove new lines from results.
newlines = text == char(10);
text(newlines) = [];
location(newlines, :) = [];
% Remove spaces from results
spaces = isspace(text);
text(spaces) = [];
location(spaces, :) = [];
% Convert text array into cell array of strings.
text = num2cell(text);
% Pad the image to help annotate results close to the image border.
I = padarray(I, [50 50], uint8(255));
location(:,1:2) = location(:,1:2) + 50;
% Insert text annotations.
J = insertObjectAnnotation(I, 'rectangle', location, text);
end
end
This is just the Red Line. Using different functions, the same tactic can be implied to all the lines that exist in the New York City Subway. I did the same thing with A,C,E,4,5,6,N,Q,R,W,7, and S.
Now, the next step is to actually make this useful for a visually impaired individual. In order for us to make this useful, this image above will actually have to speak to us. So the following code will actually say the subway train 1, 2, 3, N, R, Q.
MATLAB was not able to give the results I had expected. The problem was that the memory of the macs was not enough. Most of the laptops had up to 4GB only.
Dell:
04: 4GB
09: 4GB
01: 4 GB
Mac:
A12: 4GB
07: 4 GB
B2: 4 GB
A10: 4GB
B1: 4GB
B9: 4GB
The individual functions can be very tedious, and inconvenient, it's best if I combine all of them so that one function will do the same job as five functions.
And the same thing, with the video one. Taking all the different functions, and combining it into one saves so much time.
The most convenient way to make all the trains, the whole board rather black and white, using the color thresholding app, you can change the colors in any way you want. If you select the color properly, you can have a perfect image.
The problem is that I have to get rid of the little unnecessary dots. The little white pieces can be taken out by erosion and dilation in MATLAB.
Now I want to work on video for the whole board.
I used the tool box to clean up the image of all the trains.
Code for the image above:
image = imread('subwaysign.jpg');
figure(1), imshow(image), title('Original');
red = @createMask;
imageBWRed = red(image);
figure(6), imshow(imageBWRed); title('Red Train');
imageBWRed = bwmorph(imageBWRed,'clean')
% ocrAnn = @insertOCRAnnotation;
ocrEval = @evaluateOCRTraining;
[ocrimageBWRed, results] = ocrEval(imageBWRed);
figure(7), imshow(ocrimageBWRed); title('OCR Red Train');
results.Text;
text = results.Text;
words = ['say ' '-v' 'Victoria' ' subway train ' text(1:1) ' ' text(2:2) ' ' text(3:3)];
%system('say -v "Victoria" words');
system(words);
results.CharacterConfidences;
function [BW,maskedRGBImage] = createMask(RGB)
%createMask Threshold RGB image using auto-generated code from colorThresholder app.
% [BW,MASKEDRGBIMAGE] = createMask(RGB) thresholds image RGB using
% auto-generated code from the colorThresholder App. The colorspace and
% minimum/maximum values for each channel of the colorspace were set in the
% App and result in a binary mask BW and a composite image maskedRGBImage,
% which shows the original RGB image values under the mask BW.
% Auto-generated by colorThresholder app on 22-Mar-2017
%------------------------------------------------------
% Convert RGB image to chosen color space
I = RGB;
% Define thresholds for channel 1 based on histogram settings
channel1Min = 0.000;
channel1Max = 255.000;
% Define thresholds for channel 2 based on histogram settings
channel2Min = 0.000;
channel2Max = 255.000;
% Define thresholds for channel 3 based on histogram settings
channel3Min = 0.000;
channel3Max = 255.000;
% Create mask based on chosen histogram thresholds
sliderBW = (I(:,:,1) >= channel1Min ) & (I(:,:,1) <= channel1Max) & ...
(I(:,:,2) >= channel2Min ) & (I(:,:,2) <= channel2Max) & ...
(I(:,:,3) >= channel3Min ) & (I(:,:,3) <= channel3Max);
% Create mask based on selected regions of interest on point cloud projection
I = double(I);
[m,n,~] = size(I);
polyBW = false([m,n]);
I = reshape(I,[m*n 3]);
% Project 3D data into 2D projected view from current camera view point within app
J = rotateColorSpace(I);
% Apply polygons drawn on point cloud in app
polyBW = applyPolygons(J,polyBW);
% Combine both masks
BW = sliderBW & polyBW;
% Initialize output masked image based on input image.
maskedRGBImage = RGB;
% Set background pixels where BW is false to zero.
maskedRGBImage(repmat(~BW,[1 1 3])) = 0;
end
function J = rotateColorSpace(I)
% Translate the data to the mean of the current image within app
shiftVec = [56.132587 47.697461 47.632722];
I = I - shiftVec;
I = [I ones(size(I,1),1)]';
% Apply transformation matrix
tMat = [-0.002278 0.001021 0.000000 0.241278;
0.000002 0.000004 0.002411 -0.501110;
-0.001065 -0.002185 0.000004 9.324087;
0.000000 0.000000 0.000000 1.000000];
J = (tMat*I)';
end
function polyBW = applyPolygons(J,polyBW)
% Define each manually generated ROI
hPoints(1).data = [-0.194331 -0.578767;
-0.178443 -0.637376;
-0.038094 -0.622724;
-0.040742 -0.540671;
0.149920 -0.414661;
0.306156 -0.356052;
0.385599 -0.335538;
0.380303 -0.171432;
0.300860 -0.171432;
0.290268 -0.344330;
0.250547 -0.285720;
0.213474 -0.294512;
0.229362 -0.373634;
0.165808 -0.394148;
0.123439 -0.317955;
0.075773 -0.332608;
0.107550 -0.414661];
% Iteratively apply each ROI
for ii = 1:length(hPoints)
if size(hPoints(ii).data,1) > 2
in = inpolygon(J(:,1),J(:,2),hPoints(ii).data(:,1),hPoints(ii).data(:,2));
in = reshape(in,size(polyBW));
polyBW = polyBW | in;
end
end
end
