Last modified by stu122612 on 2025/01/30 12:18
Show last authors
1 The code for the live detection of the ball in the camera feed is located in the files //detection.cpp //and //detection.h// and uses constants from //magic.h// that are being determined from calibration.
2
3 The purpose of this code is:
4
5 * detect the ball in one of multiple regions of the camera frame
6 * output the balls real world coordinates (in meters)
7 * determine if the launch lamp is flashing (can the ball be launched to resume the game?)
8 * determine if the flippers are currently up (for timing measurements)
9
10 == Camera Settings ==
11
12 * 30 Hz refresh rate → 33.3 ms between frames
13 * 640x480 resolution
14 * 10 ms shutter speed (to avoid deformation and blurring of the ball)
15
16 == Regions of Interest ==
17
18 [[image:attach:Unbenannt4.PNG]]
19
20 There are three so called regions of interest (ROIs) defined in the algorithm.
21
22 The main ROI is the important region for keeping the ball in the game. The purpose of the top region is to be able to use the top flipper to deflect the ball into parts of the playfield that are otherwise hard to reach.
23
24 The ball detection algorithm can be run on any defined ROI. However, the ROIs currently in use are chosen to include as few lamps as possible (which must be masked or the detection algorithm will be confused) and fit the time window of at most 33.3 ms that is available well. Too many or too large regions will become problematic at some point.
25
26 == Detection Algorithm ==
27
28 The following is a step by step description and visualization of the actual detection algorithm for the ball.
29
30 === Cropping ===
31
32 The given ROI is cropped out of the full frame:
33
34 \\
35
36 [[image:attach:0_roi.png]]
37
38 === Grey scaling ===
39
40 Three color channels are not required, reduce to single grey channel for more efficient processing.
41
42 [[image:attach:1_grey.png]]
43
44 === Masking ===
45
46 Ignore lamps and areas outside of the playfield using polygon mask:\\
47
48 [[image:attach:mask.png]]
49
50 [[image:attach:2_mask.png]]
51
52 === Gauss Filter ===
53
54 Ignore high frequency noise.
55
56 [[image:attach:3_gauss.png]]
57
58 === Absolute Difference ===
59
60 Compute pixel wise difference to fixed reference frame, makes movement visible.
61
62 Areas with no movement are black, areas with movement are grey/white:
63
64 [[image:attach:4_absdiff.png]]
65
66 === Thresholding ===
67
68 Set all pixels with gray value above certain threshold to full white, rest to full black
69
70 → Binary image of moving areas:
71
72 [[image:attach:5_thresh.png]]
73
74 === Dilation ===
75
76 Widen contours by several pixels. Fill in holes, join close areas together.
77
78 [[image:attach:6_dilate.png]]
79
80 === Erosion ===
81
82 Narrow contours by several pixels. Roughly restores original size of contours but holes remain filled, small gaps remain closed.
83
84 Operation is called “Closing”:
85
86 * First Dilation
87 * Then Erosion
88
89 [[image:attach:6_erode.png]]
90
91 === Detection ===
92
93 * Find all contours (here: 1)
94 * Iterate over contours
95 * Compute minimal area bounding rectangle
96 * Drop contours that don’t fit the expected ball contour
97 (((
98 * Contour area
99 * Aspect ratio
100
101 * Should leave only contour of the ball
102 * Center of bounding rectangle is ball position in pixel coordinates
103 * Finally transform pixel coordinates into world coordinates
104 )))
105
106 [[image:attach:8_ball_detect.png]]
107
108 \\
109
110 == Additional Processing ==
111
112 Apart from ball detection in the main and top ROI, the algorithm does the following:
113
114 === Mask Flippers ===
115
116 The algorithm detects if the two bottom flippers are up and will mask them if that's the case. This is done to avoid that the contour of the flipper merges with the ball contour so that the ball isn't detectable anymore.
117
118 === Detect Launch Lamp Flashing ===
119
120 The launch ROI is a small area at the red lamp that is flashing when a ball is ready to be launched into the game. This is detected by looking at how many pixels in that region are white, if the number is above a certain threshold, the lamp is assumed to be on.
121
122 == Discarded Approaches ==
123
124 === Automatic Masking of Lamps ===
125
126 We tried to observe the whole camera frame with the ball detection algorithm (not only regions) and mask flashing lamps automatically. This has been tried by observing the flashing lamps for a while before the actual game starts and saving the information about the lamps (based on their purple color in the no-IR filter image) in a buffer image that then was used to compute a reference frame free of blinking lamps and also to mask lamps from the current frame. While that did work, it introduced too many blind spots due to the sheer amount of lamps on the playfield, so that not much was gained compared to the approach of just looking at the bottom part of the pinball machine that has a black cover for the lamps. Also the ball was colored in the same purple color as the lamps itself when it passed a flashing lamp, making it almost impossible to detect.
127
128 \\
129
130 \\
131
132 \\
133
134 \\
135
136 \\
137
138 \\
139
140 \\
141
142 \\
143
144 \\
145
146 \\
147
148 \\
149
150 \\
151
152 \\

Navigation

Quick Links

KIELER on GitHub
RTSYS Homepage